forked from openai/tiktoken
-
Notifications
You must be signed in to change notification settings - Fork 0
Pull requests: AvalancheHQ/tiktoken
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
perf: build the
decode output bytes in place: one pass, no side buffer
#89
opened Aug 11, 2026 by
codspeed-hq
Bot
Loading…
perf: drop the per-token validity test from the decode lookup
#88
opened Aug 11, 2026 by
codspeed-hq
Bot
Loading…
perf: decode token lists in one pass into a single right-sized buffer
#87
opened Aug 11, 2026 by
codspeed-hq
Bot
Loading…
perf: give the tokeniser split a literal-prefixed fast alternative for leading-space words
#86
opened Aug 11, 2026 by
codspeed-hq
Bot
Loading…
perf: upgrade fancy-regex to 0.19 for pooled VM state in the tokeniser split
#85
opened Aug 10, 2026 by
codspeed-hq
Bot
Loading…
perf: slice split pieces out of the haystack's bytes
#84
opened Aug 10, 2026 by
codspeed-hq
Bot
Loading…
perf: compile the shipped split patterns into a single
regex-automata delegate
#83
opened Aug 10, 2026 by
codspeed-hq
Bot
Loading…
perf: merge byte pairs over a compact 8-byte part array
#82
opened Aug 9, 2026 by
codspeed-hq
Bot
Loading…
perf: build the Rust core without unwinding (
panic = "abort")
#81
opened Aug 9, 2026 by
codspeed-hq
Bot
Loading…
perf: drop the per-token
Vec and slice bookkeeping from the decode fast path
#80
opened Aug 9, 2026 by
codspeed-hq
Bot
Loading…
perf: store vocabulary byte strings as
Box<[u8]> to shrink the hot lookup tables
#79
opened Aug 8, 2026 by
codspeed-hq
Bot
Loading…
perf: match disallowed special tokens with the stdlib
re engine
#78
opened Aug 8, 2026 by
codspeed-hq
Bot
Loading…
perf: compile the shipped split patterns without possessive quantifiers
#77
opened Aug 8, 2026 by
codspeed-hq
Bot
Loading…
perf: split with a single delegated automaton instead of a possessive alternation
#76
opened Aug 8, 2026 by
codspeed-hq
Bot
Loading…
perf: split tokeniser text with one streaming
regex pass per document
#75
opened Aug 7, 2026 by
codspeed-hq
Bot
Loading…
perf: write decode_bytes results through cursors instead of re-checking bounds per token
#74
opened Aug 7, 2026 by
codspeed-hq
Bot
Loading…
perf: decode token ids straight out of contiguous integer buffers
#73
opened Aug 7, 2026 by
codspeed-hq
Bot
Loading…
perf: resolve the default
disallowed_special matcher once per Encoding
#72
opened Aug 7, 2026 by
codspeed-hq
Bot
Loading…
perf: decode a whole batch inside the Rust core instead of one thread-pool task per sequence
#71
opened Aug 6, 2026 by
codspeed-hq
Bot
Loading…
perf: reuse one byte-pair merge workspace per
encode call
#70
opened Aug 6, 2026 by
codspeed-hq
Bot
Loading…
perf: drive the tokeniser split loop directly instead of through fancy_regex's Matches iterator
#69
opened Aug 6, 2026 by
codspeed-hq
Bot
Loading…
perf: try the split pattern's letter and number alternatives before the contraction alternative
#68
opened Aug 5, 2026 by
codspeed-hq
Bot
Loading…
perf: reject the tokeniser split's contraction alternative with a byte test
#67
opened Aug 5, 2026 by
codspeed-hq
Bot
Loading…
perf: stop the tokeniser split at the end of the haystack
#66
opened Aug 5, 2026 by
codspeed-hq
Bot
Loading…
perf: merge long byte-pair pieces with the heap-based algorithm
#65
opened Aug 5, 2026 by
codspeed-hq
Bot
Loading…
Previous Next
ProTip!
Mix and match filters to narrow down what you’re looking for.