How is language/multilinguality represented in Byte Latent Transformer?
Probing what latent patches in Byte Latent Transformers encode, and whether byte-level models build more language-agnostic linguistic hierarchies.
About the project
Most language models rely on tokenizers that often represent low-resource languages with many rare or poorly trained subword tokens. Byte-level models such as the Byte Latent Transformer instead operate on bytes and dynamically group them into latent patches, avoiding a fixed vocabulary and potentially providing more uniform multilingual representations.
This project asks how linguistic structure emerges without predefined token boundaries. We first study what patch representations encode,whether they correspond to characters, morphemes, words, or other units; and then trace whether this is higher-level linguistic abstractions such as syntax and semantics emerge across layers. Ultimately, we aim to understand whether byte-level models reconstruct token-like representations internally or develop a different, potentially more language-agnostic hierarchy of linguistic representations.
What mentees potentially work on
- Understand how Byte Latent Transformers work
- Learn interpretability techniques, from early ones like probes to recent mechanistic interpretability techniques like patching
- Some linguistics knowledge is good to have
Requirements and recommended background
- Python and PyTorch, comfort with linear algebra
- Some prior interpretability knowledge or tinkering with LLM architecture is a plus
- Interest in linguistics
There are no formal eligibility limits. See who can apply; you can still apply if you don't match every item.