Skip to content

[feat](jni-writer) Support JNI-based write framework for custom writers#60756

Open
morningman wants to merge 2 commits intoapache:masterfrom
morningman:tvf-jni-writer
Open

[feat](jni-writer) Support JNI-based write framework for custom writers#60756
morningman wants to merge 2 commits intoapache:masterfrom
morningman:tvf-jni-writer

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 14, 2026

What problem does this PR solve?

Introduce a symmetric JNI writer mechanism that complements the existing
JNI scanner framework, enabling custom file writers to be implemented in
Java and invoked through JNI from the C++ query executor.

Key changes:

  • Add VJniFormatTransformer for delegating writes to Java via JNI
  • Extend TTVFTableSink with writer_type and writer_class fields
  • Add JniWriter base class and LocalFileJniWriter reference implementation
  • Integrate with file_format_transformer factory routing logic

The implementation follows the same patterns as JniScanner for consistency:

  • Reuses class loading mechanism (Jni::Util::get_jni_scanner_class)
  • Mirrors constructor signature: (int batchSize, Map<String,String> params)
  • Supports schema caching and performance instrumentation

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 14, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman morningman changed the title [feat](demo) support JNI write framework [feat](jni-writer) Support JNI-based write framework for custom writers Feb 14, 2026
@morningman morningman marked this pull request as ready for review February 14, 2026 02:20
@morningman
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.33% (1796/2264)
Line Coverage 64.80% (31992/49372)
Region Coverage 65.47% (15961/24378)
Branch Coverage 55.97% (8488/15164)

Copy link
Contributor

@hubgeter hubgeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉
Increment coverage report
Complete coverage report

fileWriter.write(output);
writtenRows += numRows;
writtenBytes += output.getBytes().length;
LOG.info("LocalFileJniWriter wrote " + numRows + " rows, totalWrittenRows=" + writtenRows
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L65 and L92
so many logs in write, maybe could remove next time

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 14, 2026
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 29187 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1bdfe4761251aa7e66ff5f414d183bcc03987f1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17668	4481	4394	4394
q2	q3	10721	767	521	521
q4	4723	361	253	253
q5	8164	1227	1012	1012
q6	232	178	149	149
q7	828	857	655	655
q8	10825	1466	1378	1378
q9	6639	4762	4736	4736
q10	6874	1875	1641	1641
q11	472	250	243	243
q12	747	562	474	474
q13	17796	4229	3416	3416
q14	233	233	203	203
q15	932	795	787	787
q16	734	711	684	684
q17	822	876	412	412
q18	6502	5329	5187	5187
q19	1156	977	885	885
q20	597	561	424	424
q21	4522	1905	1474	1474
q22	351	312	259	259
Total cold run time: 101538 ms
Total hot run time: 29187 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4809	4523	4521	4521
q2	q3	1867	2300	1871	1871
q4	870	1234	787	787
q5	4041	4382	4337	4337
q6	181	176	140	140
q7	1779	1641	1525	1525
q8	2466	2702	2538	2538
q9	7479	7592	7350	7350
q10	2619	2823	2570	2570
q11	510	443	424	424
q12	513	621	442	442
q13	4169	4374	3602	3602
q14	284	314	287	287
q15	889	793	790	790
q16	738	770	711	711
q17	1208	1575	1300	1300
q18	7197	6941	6589	6589
q19	938	891	923	891
q20	2124	2152	2018	2018
q21	4008	3452	3380	3380
q22	488	473	412	412
Total cold run time: 49177 ms
Total hot run time: 46485 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants