How is language/multilinguality represented in Byte Latent Transformer?

Mechanistic interpretabilityMultilingual NLP

Probing what latent patches in Byte Latent Transformers encode, and whether byte-level models build more language-agnostic linguistic hierarchies.

About the project

Most language models rely on tokenizers that often represent low-resource languages with many rare or poorly trained subword tokens. Byte-level models such as the Byte Latent Transformer instead operate on bytes and dynamically group them into latent patches, avoiding a fixed vocabulary and potentially providing more uniform multilingual representations.

This project asks how linguistic structure emerges without predefined token boundaries. We first study what patch representations encode,whether they correspond to characters, morphemes, words, or other units; and then trace whether this is higher-level linguistic abstractions such as syntax and semantics emerge across layers. Ultimately, we aim to understand whether byte-level models reconstruct token-like representations internally or develop a different, potentially more language-agnostic hierarchy of linguistic representations.

What mentees potentially work on

  • Understand how Byte Latent Transformers work
  • Learn interpretability techniques, from early ones like probes to recent mechanistic interpretability techniques like patching
  • Some linguistics knowledge is good to have

Requirements and recommended background

  • Python and PyTorch, comfort with linear algebra
  • Some prior interpretability knowledge or tinkering with LLM architecture is a plus
  • Interest in linguistics

There are no formal eligibility limits. See who can apply; you can still apply if you don't match every item.

Relevant publications

Mentors

Ruochen Zhang
Ruochen Zhang
Brown University
PhD Student
Frederikus Hudi
Frederikus Hudi
NAIST
PhD Student