SEATauBench: Progressively Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

Build an environment and evaluation benchmark for agentic LLMs in low-resource languages and underrepresented regions.

Mentors:
Samuel Cahyawijaya Patomporn Payoungkhamdee
Mentees (5):
Aulia AdilaKittiphat LeesombatwathanaMy (Chiffon) NguyenSaksorn RuangtanusakVissuta Gunawan Lim

Project Proposal

In this work, we address the gap in enabling LLMs with agentic capabilities for low-resource languages and underrepresented regions. Most existing environments and evaluation benchmarks (e.g., Taubench) are Anglocentric, leaving a critical void in assessing performance across diverse linguistic contexts.

This initiative aims to develop a comprehensive environment and evaluation benchmark that measures agentic LLM effectiveness in underrepresented regions, ensuring these technologies are inclusive and accessible to a broader global audience.

Relevant publications:

  • τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
  • τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment

Publications

  1. SEATauBench: Progressively Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages
    My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak, Kittiphat Leesombatwathana, Vissuta Gunawan Lim, Patomporn Payoungkhamdee, Samuel Cahyawijaya
    EMNLP Findings • 2026
    Abstract

    While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly understood despite its importance to sovereign AI. To fill this gap, we introduce SEATauBench, the first agent-focused evaluation framework for SEA sovereign AI. It adapts τ^2-Bench (Barres et al., 2025) to five languages—Mandarin, Vietnamese, Thai, Indonesian, and Filipino—and evaluates agents cross three progressively localized settings that vary the language of user-agent interaction, tool specifications, and task domains. Across three models, we find that English agent capabilities transfer reasonably well when only the conversation language changes, but quality and robustness degrade sharply as more task contexts are localized, with the largest losses in full domain adaptation. We also highlight the limits of English-only agent assessment for predicting agent capabilities in SEA languages. More broadly, SEATauBench provides a diagnostic benchmark and reusable adaptation pipeline for building reliable multilingual agents for linguistically diverse regions.

    PDF Code Website