Code-switching is a widespread practice among the world's multilingual majority, yet few benchmarks accurately reflect its complexity in everyday communication. We present PingPong, a benchmark for natural multi-party code-switching dialogues covering five language-combination variations, some of which are trilingual. Our dataset consists of human-authored conversations among 2 to 4 participants covering authentic, multi-threaded structures where replies frequently reference much earlier points in the dialogue. We demonstrate that our data is significantly more natural and structurally diverse than machine-generated alternatives, offering greater variation in message length, speaker dominance, and reply distance. Based on these dialogues, we define three downstream tasks: Question Answering, Dialogue Summarization, and Topic Classification. Evaluations of several state-of-the-art language models on PingPong reveal that performance remains limited on code-switched inputs, underscoring the urgent need for more robust NLP systems capable of addressing the intricacies of real-world multilingual discourse.
@misc{farhansyah2026pingpongnaturalbenchmarkmultiturn,
title={PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues},
author={Mohammad Rifqi Farhansyah and Hanif Muhammad Zhafran and Farid Adilazuarda and Shamsuddeen Hassan Muhammad and Maryam Ibrahim Mukhtar and Nedjma Ousidhoum and Genta Indra Winata and Ayu Purwarianti and Alham Fikri Aji},
year={2026},
eprint={2601.17277},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2601.17277},
}