lib: use utf8 fast path for streaming TextDecoder #61549

ChALkeR · 2026-01-27T18:17:16Z

Tracking: #61041

A continuation of #61409

This is based on the logic in https://github.com/ExodusOSS/bytes

Unifies stream and non-stream codepath for UTF-8 both for intl and no-intl variants.

Previously, ICU and string_decoder were used on with-intl and without-intl variants correspondingly for streaming UTF-8 implementation

Instead, just do minor quick slices on JS side and use the single stateless native decoding API to be fast on large chunks

This:

Improves streaming UTF-8 TextDecoder performance
Brings streaming fatal UTF-8 TextDecoder support to without-intl builds
Unifies the logic, using a single codepath everywhere for UTF-8

Benchmarks

Previously: #61131

Non-chunked, 25.5.0

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	36.69 GiB/s	0.002 ms
Arabic lipsum	79.771 KiB	2.03 GiB/s	0.037 ms
Chinese lipsum	68.203 KiB	4.10 GiB/s	0.016 ms
Arabic + 2 * ASCII	249.577 KiB	2.72 GiB/s	0.092 ms
Non-ASCII char + ASCII	84.905 KiB	3.84 GiB/s	0.024 ms

Non-chunked, PR

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	36.19 GiB/s	0.002 ms
Arabic lipsum	79.771 KiB	2.02 GiB/s	0.038 ms
Chinese lipsum	68.203 KiB	4.00 GiB/s	0.016 ms
Arabic + 2 * ASCII	249.577 KiB	3.32 GiB/s	0.075 ms
Non-ASCII char + ASCII	84.905 KiB	4.71 GiB/s	0.022 ms

(should not be affected)

Chunked (1000 byte chunks), 25.5.0

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	1.31 GiB/s	0.062 ms
Arabic lipsum	79.771 KiB	0.90 GiB/s	0.084 ms
Chinese lipsum	68.203 KiB	0.99 GiB/s	0.066 ms
Arabic + 2 * ASCII	249.577 KiB	1.14 GiB/s	0.209 ms
Non-ASCII char + ASCII	84.905 KiB	1.31 GiB/s	0.062 ms

Chunked (1000 byte chunks), PR

Test	Size	Throughput	Mean Time
Latin lipsum (ASCII)	84.902 KiB	7.45 GiB/s	0.011 ms
Arabic lipsum	79.771 KiB	1.24 GiB/s	0.062 ms
Chinese lipsum	68.203 KiB	1.54 GiB/s	0.042 ms
Arabic + 2 * ASCII	249.577 KiB	2.67 GiB/s	0.091 ms
Non-ASCII char + ASCII	84.905 KiB	7.26 GiB/s	0.011 ms

The benchmark creates about ~70-256 chunks on each test
The improvement is significant

lib/internal/encoding/util.js

anonrig · 2026-01-27T20:21:16Z

@ChALkeR can you run benchmark CI that applies to this pull-request?

ChALkeR · 2026-01-27T20:33:00Z

@anonrig I can't do anything until #61553

Also, we don't have a stream: true TextDecoder benchmark.
The bench results above are from https://github.com/lemire/jstextdecoderbench with this modification:

function decodeChunked(bytes) {
  const chunk = 1000
  const max = bytes.length - chunk
  let i = 0
  for (; i < max; i += chunk) decoder.decode(bytes.subarray(i, i + chunk), { stream: true })
  decoder.decode(bytes.subarray(i))
}

And using that instead of decoder.decode

anonrig

Thank you for the kind responses and amazing work.

codecov · 2026-01-27T21:08:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.77%. Comparing base (e155415) to head (1559954).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #61549    +/-   ##
========================================
  Coverage   89.77%   89.77%            
========================================
  Files         672      673     +1     
  Lines      203755   203875   +120     
  Branches    39167    39190    +23     
========================================
+ Hits       182922   183038   +116     
- Misses      13164    13172     +8     
+ Partials     7669     7665     -4

Files with missing lines	Coverage Δ
lib/internal/encoding.js	`100.00% <100.00%> (ø)`
lib/internal/encoding/util.js	`100.00% <100.00%> (ø)`

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nodejs-github-bot · 2026-01-27T22:48:04Z

CI: https://ci.nodejs.org/job/node-test-pull-request/71082/

ChALkeR · 2026-01-27T23:16:20Z

@anonrig I added a benchmark for streaming Unicode TextDecoder.
Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1787

ChALkeR · 2026-01-27T23:34:37Z

Benchmark CI output. utf-16le are flakes as that codepath hasn't changed.

Chunks of 25 bytes are slightly slower than previously, larger chunks are significantly faster than previously.

util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                *     -1.79 %       ±1.65%  ±2.20%  ±2.88%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'                 ***    267.01 %       ±9.19% ±12.38% ±16.42%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'              ***     -2.84 %       ±1.45%  ±1.93%  ±2.52%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'                 ***    264.65 %       ±7.72% ±10.41% ±13.81%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'              ***     -4.11 %       ±2.02%  ±2.69%  ±3.49%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'                 ***    267.92 %       ±7.64% ±10.29% ±13.63%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'              ***     -3.67 %       ±1.84%  ±2.45%  ±3.19%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'                 ***    266.98 %       ±9.07% ±12.22% ±16.21%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'               **     -2.97 %       ±1.96%  ±2.62%  ±3.41%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'                 ***     34.06 %       ±1.47%  ±1.97%  ±2.59%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'              ***     -4.82 %       ±1.54%  ±2.05%  ±2.68%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'                 ***     35.53 %       ±1.13%  ±1.51%  ±1.97%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'              ***     -3.71 %       ±2.01%  ±2.68%  ±3.49%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'                 ***     23.26 %       ±1.28%  ±1.71%  ±2.24%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'              ***     -3.48 %       ±1.66%  ±2.21%  ±2.88%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'                 ***     22.98 %       ±1.06%  ±1.42%  ±1.85%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                       -2.18 %       ±2.67%  ±3.56%  ±4.63%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'                  ***     68.48 %       ±3.39%  ±4.55%  ±5.99%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'                **     -4.99 %       ±3.26%  ±4.36%  ±5.72%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'                  ***     71.15 %       ±4.89%  ±6.52%  ±8.52%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'                       -0.58 %       ±2.92%  ±3.89%  ±5.07%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'                  ***     65.57 %       ±4.40%  ±5.92%  ±7.83%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'                 *     -3.81 %       ±3.03%  ±4.03%  ±5.25%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'                  ***     65.41 %       ±6.05%  ±8.09% ±10.64%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                       -0.91 %       ±1.68%  ±2.23%  ±2.91%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'                  ***     11.54 %       ±1.77%  ±2.36%  ±3.07%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'                 *     -3.42 %       ±2.81%  ±3.73%  ±4.86%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'                  ***     12.49 %       ±1.83%  ±2.44%  ±3.21%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'                       -0.40 %       ±3.20%  ±4.27%  ±5.59%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'                  ***      4.22 %       ±2.18%  ±2.90%  ±3.78%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'                 *     -3.27 %       ±2.94%  ±3.93%  ±5.18%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'                  ***      4.72 %       ±1.96%  ±2.61%  ±3.41%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                         -0.43 %       ±3.97%  ±5.28%  ±6.88%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'                            -0.60 %       ±4.44%  ±5.91%  ±7.70%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'                          2.78 %       ±8.25% ±10.98% ±14.31%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'                            -4.37 %       ±6.42%  ±8.54% ±11.13%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'                         -1.01 %       ±5.05%  ±6.72%  ±8.74%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'                            -1.37 %       ±5.84%  ±7.77% ±10.11%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'                          3.16 %       ±4.27%  ±5.68%  ±7.40%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'                            -3.63 %       ±9.19% ±12.22% ±15.91%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                         -2.25 %       ±5.58%  ±7.44%  ±9.73%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'                    ***    -21.49 %       ±5.17%  ±6.88%  ±8.96%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'                          5.46 %       ±6.03%  ±8.08% ±10.65%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'                    ***    -19.70 %       ±8.21% ±10.95% ±14.30%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'                          5.43 %       ±5.97%  ±7.99% ±10.50%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'                    ***    -19.80 %       ±6.25%  ±8.33% ±10.89%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'                   *      6.59 %       ±5.88%  ±7.87% ±10.34%
util/text-decoder-stream.js type='ArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'                    ***    -20.68 %       ±5.28%  ±7.05%  ±9.23%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                    **      1.49 %       ±1.08%  ±1.43%  ±1.87%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'                      ***    486.50 %      ±17.42% ±23.47% ±31.16%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'                            0.22 %       ±1.44%  ±1.91%  ±2.49%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'                      ***    482.01 %      ±21.29% ±28.69% ±38.08%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'                            0.82 %       ±1.55%  ±2.07%  ±2.71%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'                      ***    495.12 %      ±14.58% ±19.65% ±26.09%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'                            0.16 %       ±1.58%  ±2.11%  ±2.77%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'                      ***    496.46 %      ±15.79% ±21.28% ±28.24%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                            1.18 %       ±1.29%  ±1.72%  ±2.23%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'                      ***     42.18 %       ±1.07%  ±1.44%  ±1.90%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'                            0.86 %       ±1.24%  ±1.65%  ±2.15%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'                      ***     42.70 %       ±0.79%  ±1.06%  ±1.40%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'                     *      1.60 %       ±1.36%  ±1.80%  ±2.35%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'                      ***     29.50 %       ±1.01%  ±1.35%  ±1.76%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'                            1.35 %       ±1.64%  ±2.19%  ±2.85%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'                      ***     29.22 %       ±1.09%  ±1.45%  ±1.89%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                    ***     25.62 %       ±7.98% ±10.74% ±14.20%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'                       ***    165.75 %      ±17.49% ±23.55% ±31.21%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'                    ***     16.01 %       ±6.72%  ±9.03% ±11.92%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'                       ***    153.29 %      ±22.05% ±29.70% ±39.37%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'                     **     11.84 %       ±7.93% ±10.57% ±13.80%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'                       ***    174.61 %      ±19.32% ±26.03% ±34.55%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'                    ***     17.35 %       ±8.00% ±10.69% ±14.01%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'                       ***    151.66 %      ±20.67% ±27.83% ±36.91%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                    ***     18.16 %       ±7.87% ±10.50% ±13.73%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'                       ***     11.68 %       ±4.59%  ±6.11%  ±7.97%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'                    ***     17.06 %       ±7.64% ±10.24% ±13.48%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'                       ***     13.33 %       ±4.43%  ±5.93%  ±7.77%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'                    ***     12.99 %       ±6.13%  ±8.24% ±10.90%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'                        **      5.74 %       ±3.79%  ±5.05%  ±6.59%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'                      *      7.59 %       ±6.16%  ±8.20% ±10.69%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'                       ***      6.09 %       ±3.15%  ±4.20%  ±5.49%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                              13.62 %      ±17.29% ±23.00% ±29.94%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'                                  2.69 %      ±14.15% ±18.86% ±24.62%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'                              15.01 %      ±20.40% ±27.15% ±35.34%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'                                  8.72 %      ±13.33% ±17.81% ±23.32%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'                              -4.80 %      ±16.50% ±21.96% ±28.60%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'                                 -9.38 %      ±14.11% ±18.79% ±24.51%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'                               4.03 %      ±18.87% ±25.11% ±32.68%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'                                 -2.59 %      ±13.91% ±18.53% ±24.16%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                              -9.15 %      ±17.20% ±22.89% ±29.80%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'                          **    -18.41 %      ±11.99% ±16.10% ±21.25%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'                             -10.74 %      ±15.03% ±20.01% ±26.07%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'                         ***    -21.23 %       ±9.79% ±13.19% ±17.49%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'                               3.62 %      ±16.45% ±21.89% ±28.49%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'                         ***    -29.36 %       ±8.93% ±11.95% ±15.72%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'                        *    -15.44 %      ±14.73% ±19.62% ±25.58%
util/text-decoder-stream.js type='Buffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'                         ***    -25.76 %      ±10.24% ±13.74% ±18.13%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                -0.19 %       ±0.99%  ±1.32%  ±1.72%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'           ***    162.73 %       ±2.08%  ±2.79%  ±3.69%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'        ***     -2.34 %       ±1.08%  ±1.44%  ±1.87%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'           ***    160.58 %       ±2.76%  ±3.71%  ±4.90%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'        ***     -2.68 %       ±0.77%  ±1.02%  ±1.33%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'           ***    162.40 %       ±2.99%  ±4.02%  ±5.33%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'        ***     -2.65 %       ±1.21%  ±1.60%  ±2.09%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'           ***    163.47 %       ±1.36%  ±1.81%  ±2.39%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                -0.93 %       ±1.19%  ±1.58%  ±2.07%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'           ***     29.86 %       ±1.28%  ±1.72%  ±2.26%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'         **     -1.57 %       ±1.14%  ±1.52%  ±1.98%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'           ***     30.56 %       ±1.44%  ±1.92%  ±2.50%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'         **     -1.27 %       ±0.93%  ±1.24%  ±1.62%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'           ***     20.36 %       ±1.27%  ±1.70%  ±2.23%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'        ***     -2.54 %       ±1.39%  ±1.84%  ±2.40%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=131072 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'           ***     19.83 %       ±1.04%  ±1.40%  ±1.83%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                  0.14 %       ±1.96%  ±2.61%  ±3.41%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'            ***     59.13 %       ±2.27%  ±3.03%  ±3.96%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'          **      3.37 %       ±2.30%  ±3.06%  ±3.99%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'            ***     58.65 %       ±3.62%  ±4.82%  ±6.29%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'                  0.59 %       ±2.07%  ±2.75%  ±3.58%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'            ***     60.49 %       ±3.32%  ±4.42%  ±5.75%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'                  1.05 %       ±1.26%  ±1.67%  ±2.18%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'            ***     61.06 %       ±3.05%  ±4.06%  ±5.30%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                  1.80 %       ±1.99%  ±2.65%  ±3.45%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'            ***      6.17 %       ±1.38%  ±1.84%  ±2.40%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'                  1.81 %       ±2.63%  ±3.50%  ±4.56%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'            ***      6.65 %       ±1.87%  ±2.49%  ±3.24%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'                  0.56 %       ±1.86%  ±2.47%  ±3.23%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'              *      2.28 %       ±1.92%  ±2.56%  ±3.33%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'                  2.03 %       ±2.18%  ±2.90%  ±3.78%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=16384 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'             **      2.39 %       ±1.47%  ±1.96%  ±2.55%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-16le'                    5.39 %       ±8.64% ±11.50% ±14.96%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=0 encoding='utf-8'                *     -8.11 %       ±6.34%  ±8.49% ±11.16%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-16le'                    2.34 %       ±7.99% ±10.65% ±13.89%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=0 ignoreBOM=1 encoding='utf-8'                       0.88 %      ±10.43% ±13.88% ±18.07%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-16le'                    5.91 %       ±7.12%  ±9.48% ±12.37%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=0 encoding='utf-8'                      -1.71 %       ±8.67% ±11.54% ±15.02%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-16le'                    3.82 %       ±6.01%  ±8.00% ±10.41%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=0 fatal=1 ignoreBOM=1 encoding='utf-8'                      -1.29 %       ±7.43%  ±9.88% ±12.86%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-16le'                    3.56 %       ±8.87% ±11.80% ±15.36%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=0 encoding='utf-8'              ***    -12.89 %       ±7.12%  ±9.54% ±12.55%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-16le'                    2.24 %       ±8.41% ±11.19% ±14.57%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=0 ignoreBOM=1 encoding='utf-8'              ***    -13.09 %       ±6.15%  ±8.27% ±10.96%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-16le'                    1.08 %       ±8.55% ±11.38% ±14.82%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=0 encoding='utf-8'               **    -11.34 %       ±8.32% ±11.16% ±14.73%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-16le'                   -0.76 %       ±8.14% ±10.83% ±14.10%
util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8'              ***    -21.55 %       ±4.82%  ±6.42%  ±8.36%

util/text-decoder.js output also exists on the link, but it's useless as (1) that codepath wasn't actually changed and (2) it tests on Buffer.allocUnsafe which is basically random garbage and mostly zeros but not always, so is heavily flaking between executions. Also the try-catch for fatal makes it even more flaky.
All the results are random there.

ChALkeR · 2026-01-28T08:15:21Z

cc @nodejs/performance perhaps

gurgunday

lgtm

I reviewed and tested to the best of my ability, and this is indeed great work @ChALkeR

nodejs-github-bot · 2026-01-28T10:46:10Z

CI: https://ci.nodejs.org/job/node-test-pull-request/71094/

nodejs-github-bot · 2026-01-28T15:17:24Z

CI: https://ci.nodejs.org/job/node-test-pull-request/71102/

ChALkeR · 2026-01-28T15:23:13Z

@gurgunday Once everything is fixed, the plan is to update WPT + bring in my extra tests. They'll fail now due to other bugs though, so it can't be done yet without sorting the tests, and that's likely not worth it if we can just fix stuff first.

anonrig · 2026-01-28T15:51:53Z

Can we find a middle ground and avoid regressions like this with this pull-request?

util/text-decoder-stream.js type='SharedArrayBuffer' n=1000 chunks=10 len=256 unicode=1 fatal=1 ignoreBOM=1 encoding='utf-8' *** -21.55 % ±4.82% ±6.42% ±8.36%

ChALkeR · 2026-01-28T16:18:31Z

@anonrig That's streaming with chunks of 25 bytes. Which is not a realistic use-case of this API.
And not worth having a whole separate impl to back it.
Moving util methods to native later would be more worth it (and could give even larger perf benefit).

nodejs-github-bot added encoding Issues and PRs related to the TextEncoder and TextDecoder APIs. needs-ci PRs that need a full CI run. labels Jan 27, 2026

ChALkeR force-pushed the chalker/decoder/unify/2 branch from a61503c to 4331d20 Compare January 27, 2026 18:18

ChALkeR mentioned this pull request Jan 27, 2026

lib: unify ICU and no-ICU TextDecoder #61409

Merged

ChALkeR force-pushed the chalker/decoder/unify/2 branch from 4331d20 to 916cd32 Compare January 27, 2026 18:25

anonrig reviewed Jan 27, 2026

View reviewed changes

lib/internal/encoding/util.js Outdated Show resolved Hide resolved

anonrig added the needs-benchmark-ci PR that need a benchmark CI run. label Jan 27, 2026

ChALkeR force-pushed the chalker/decoder/unify/2 branch 5 times, most recently from 9ebe8ce to 111b99e Compare January 27, 2026 19:22

ChALkeR requested a review from anonrig January 27, 2026 19:23

ChALkeR marked this pull request as ready for review January 27, 2026 19:25

ChALkeR force-pushed the chalker/decoder/unify/2 branch 3 times, most recently from 1a77e7a to cd5c966 Compare January 27, 2026 20:12

lib: use utf8 fast path for streaming TextDecoder

e9e9252

ChALkeR force-pushed the chalker/decoder/unify/2 branch from cd5c966 to e9e9252 Compare January 27, 2026 20:13

anonrig approved these changes Jan 27, 2026

View reviewed changes

ChALkeR added the request-ci Add this label to start a Jenkins CI on a PR. label Jan 27, 2026

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 27, 2026

benchmark: add streaming TextDecoder benchmark

1559954

ChALkeR added the request-ci Add this label to start a Jenkins CI on a PR. label Jan 27, 2026

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 27, 2026

github-actions bot added the request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. label Jan 27, 2026

This comment was marked as outdated.

Sign in to view

ChALkeR requested review from avivkeller, gurgunday and richardlau January 28, 2026 00:10

This was referenced Jan 28, 2026

lib: add utf16 fast path for TextDecoder #61559

Draft

TextDecoder is wrong and very slow #61041

Open

gurgunday approved these changes Jan 28, 2026

View reviewed changes

ChALkeR added the request-ci Add this label to start a Jenkins CI on a PR. label Jan 28, 2026

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 28, 2026

cjihrig approved these changes Jan 28, 2026

View reviewed changes

ChALkeR added request-ci Add this label to start a Jenkins CI on a PR. and removed request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. labels Jan 28, 2026

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 28, 2026

RafaelGSS added the performance Issues and PRs related to the performance of Node.js. label Jan 28, 2026

Uh oh!

lib: use utf8 fast path for streaming TextDecoder #61549

Are you sure you want to change the base?

lib: use utf8 fast path for streaming TextDecoder #61549

Conversation

ChALkeR commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Non-chunked, 25.5.0

Non-chunked, PR

Chunked (1000 byte chunks), 25.5.0

Chunked (1000 byte chunks), PR

Uh oh!

Uh oh!

anonrig commented Jan 27, 2026

Uh oh!

ChALkeR commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anonrig left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nodejs-github-bot commented Jan 27, 2026

Uh oh!

ChALkeR commented Jan 27, 2026

Uh oh!

This comment was marked as outdated.

ChALkeR commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChALkeR commented Jan 28, 2026

Uh oh!

gurgunday left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jan 28, 2026

Uh oh!

nodejs-github-bot commented Jan 28, 2026

Uh oh!

ChALkeR commented Jan 28, 2026

Uh oh!

anonrig commented Jan 28, 2026

Uh oh!

ChALkeR commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ChALkeR commented Jan 27, 2026 •

edited

Loading

ChALkeR commented Jan 27, 2026 •

edited

Loading

codecov bot commented Jan 27, 2026 •

edited

Loading

ChALkeR commented Jan 27, 2026 •

edited

Loading

ChALkeR commented Jan 28, 2026 •

edited

Loading