BanglaDual: A 1.24-Million Pair Multi-Source Dataset for Banglish Transliteration and Translation

To support immediate model development, the dataset includes a pre-computed 80/10/10 train/validation/test split (seed = 42). This dataset is roughly 30 times larger than previously available benchmarks like BanglaTLit and is the first to support both transliteration and translation tasks from a single format. It aims to support researchers addressing the challenges of code-mixed Bengali text in low-resource environments.

Repository File Structure:To facilitate ease of use, the repository is organized into five directories containing the data files in Apache Parquet format (with Snappy compression for efficiency) and standard CSVs for quick inspection. 📁 Task-Specific-Datasets/banglish_bangla_transliteration.parquet: Contains 1,122,189 cleaned and deduplicated transliteration pairs. banglish_english_translation.parquet: Contains 119,727 translation pairs, including 470 manually translated by the authors. 📁 Merged-Dataset/banglish_merged_dual_task.parquet: Contains 1,122,998 rows produced by an outer join of the Bangla and English datasets, supporting multi-task learning. 📁 Model-Splits/train.parquet: Training set partition (898,024 rows). validation.parquet: Validation set partition (112,253 rows). test.parquet: Testing set partition (112,254 rows). 📁 Sample-Data/samples_bangla.csv: 60 randomly sampled transliteration records for quick inspection. samples_english.csv: 40 randomly sampled translation records for quick inspection. sample_merged_data.csv: A representative sample showcasing the outer-joined dual-task format structure. 📁 Processing-Scripts/banglish_to_bangla_pipeline.py: Code to process and merge the six Bangla sources. banglish_to_english_pipeline.py: Code to process the four English sources. merge_datasets.py: Code to perform the outer join and generate the 80/10/10 splits.