Vishnu Pulipaka

2026

1. BabyLM

   Can Translation Bridge the Data Gap for Low-Resource Languages? A Case Study of Hindi and Telugu

   Venkata Naga Sai Vishnu Rohit Pulipaka and Libby Barak

   In Proceedings of the Second BabyLM Workshop, pp. 279–288, Budapest, Hungary. Association for Computational Linguistics, Oct 2026

   AbsHTMLPDFDOICodeHindi dataTelugu dataBib

   Low-resource languages often lack sufficient training data for developing small language models, limiting the application of recent language acquisition benchmarks beyond English. Moreover, comparison across languages is made harder given differences in training data composition and quality. We investigate whether machine translation can be used to create BabyLM-scale training corpora for low-resource languages by translating the English BabyLM corpus into Hindi and Telugu using IndicTrans2. We train GPT-BERT and GPT-Wee models and evaluate them using perplexity and linguistic benchmarks. GPT-BERT consistently outperforms GPT-Wee across both monolingual and bilingual settings. Although perplexity is lower on in-domain test data, this pattern is consistent across English and the translated languages, suggesting that the effect reflects domain similarity rather than artifacts introduced by translation. Furthermore, the models achieve competitive performance on linguistic evaluations relative to pretrained language models. These findings suggest that translation warrants further analysis to obtain data for developing and evaluating small language models in low-resource settings.

   @inproceedings{pulipaka-barak-2026-translation,
     title     = {Can Translation Bridge the Data Gap for Low-Resource Languages? A Case Study of {H}indi and {T}elugu},
     author    = {Pulipaka, Venkata Naga Sai Vishnu Rohit and Barak, Libby},
     booktitle = {Proceedings of the Second {B}aby{LM} Workshop},
     month     = oct,
     year      = {2026},
     address   = {Budapest, Hungary},
     publisher = {Association for Computational Linguistics},
     url       = {https://aclanthology.org/2026.babylm-main.22/},
     doi       = {10.18653/2026.babylm-main.22},
     pages     = {279--288}
   }

2. arXiv

   Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages

   Venkata Naga Sai Vishnu Rohit Pulipaka

   arXiv preprint arXiv:2608.07727, Aug 2026

   AbsarXivPDFCodeBib

   Dravidian languages, mainly Tamil, Telugu, Kannada, and Malayalam make up only a small part of the data used to train multilingual language models, so it's not clear how much per-language ability these models actually keep. I have trained five GPT-2 architecture models from scratch to compare four monolingual models (one each for Tamil, Telugu, Kannada, and Malayalam, each with its own 32K-vocabulary subword tokenizer) against one multilingual model sharing a 64K-vocabulary subword tokenizer across all four languages. All the 5 models are trained on cleaned CC-100, Wikipedia, and Samanantar data. I have tested the models on perplexity, bits-per-byte, tokenizer efficiency, and fine-tuning results which are compared against mGPT. The monolingual models outperform mGPT on sentiment classification and named entity recognition, and their tokenizers proved more efficient than the shared multilingual model across all the languages tested.

   @misc{pulipaka2026dravidian,
     title         = {Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for {D}ravidian Languages},
     author        = {Pulipaka, Venkata Naga Sai Vishnu Rohit},
     year          = {2026},
     eprint        = {2608.07727},
     archivePrefix = {arXiv},
     primaryClass  = {cs.CL},
     url           = {https://arxiv.org/abs/2608.07727}
   }