Add heap-based BPE merge path for large inputs (>128 bytes)#7580

Merged

tarekgh merged 16 commits intomainfrom

copilot/fix-byte-pair-encoding-performance

Feb 14, 2026

Commits on Feb 13, 2026

Remove timing-based performance test to prevent CI flakiness

Copilot
and
stephentoub
committed
Add explanatory comments for threshold and heap capacity choices

Copilot
and
stephentoub
committed
Apply suggestion from @stephentoub
stephentoub
authored
Use default capacity for PriorityQueue instead of pre-allocating to max

Copilot
and
stephentoub
committed
Apply suggestion from @stephentoub
stephentoub
authored
Update src/Microsoft.ML.Tokenizers/Utils/BytePairEncoder.cs

stephentoub
and
Copilot
authored
Update src/Microsoft.ML.Tokenizers/Utils/BytePairEncoder.cs

stephentoub
and
Copilot
authored
Fix CompareTo ordering, revert BOM changes, and update test comments

Copilot
and
stephentoub
committed
Add parameterless PriorityQueue constructor and use it in BytePairEncodeLarge

Copilot
and
tarekgh
committed
Add comment noting that CurRank assumes rank == token Id (Tiktoken-specific)

Copilot
and
tarekgh
committed
Remove stackalloc for State array; always use ArrayPool since method is only called for >128 bytes

Copilot
and
tarekgh
committed
Replace List+ToArray with ArrayPool for result buffer in BytePairEncodeLarge

Copilot
and
tarekgh
committed