CoreBPE::new has an unconstrained NSE type parameter, making normal downstream calls fail type inference

Author: fallintoplaceCreated May 24, 2026Updated May 24, 2026

Verified on current main at 08a5f3b (May 24, 2026).

CoreBPE::new currently has this signature in src/lib.rs:

rust

pub fn new<E, SE, NSE>(
    encoder: E,
    special_tokens_encoder: SE,
    pattern: &str,
) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
where
    E: IntoIterator<Item = (Vec<u8>, Rank)>,
    SE: IntoIterator<Item = (String, Rank)>,
    NSE: IntoIterator<Item = (String, (Rank, Rank))>,

The NSE type parameter is not used in the function arguments or return type, so normal downstream calls cannot infer it.

Minimal repro from an external crate:

rust

use tiktoken::{CoreBPE, Rank};

fn main() {
    let _ = CoreBPE::new(
        vec![(b"a".to_vec(), 0 as Rank)],
        Vec::<(String, Rank)>::new(),
        r".",
    );
}

cargo check fails with:

error[E0283]: type annotations needed
 --> src/main.rs:4:13
  |
4 |     let _ = CoreBPE::new(
  |             ^^^^^^^^^^^^ cannot infer type of the type parameter `NSE` declared on the associated function `new`

This seems like a real public Rust API bug rather than an internal-only issue, since Cargo.toml publishes an rlib in addition to the Python extension.

Expected behavior:

A normal downstream call to CoreBPE::new(...) should compile without requiring unrelated generic annotations.

Likely fix:

Remove the unused NSE type parameter from CoreBPE::new.

Thanks!

Source: openai/tiktoken

View original on GitHub View discussion on GitHub