Zhaoming213/baiduwiki

数据集介绍 这是百度百科数据集，对原始数据集进行了简单清洗，最终变成了类似:{"text":....}对结构，jsonl文件。 清洗代码 import json input_file = "百度百科.jsonl" output_file = "filtered_data.jsonl" with open(input_file, 'r', encoding='utf-8') as infile, \ open(output_file, 'w', encoding='utf-8') as outfile: for line in infile: if not line.strip(): continue try: data = json.loads(line) outfile.write(json.dumps(data, ensure_ascii=False) + '\n') except… See the full description on the dataset page: huggingface.co/datasets/Zhaoming213/baiduwiki.