Table · dataset · 2024
PleIAs/common_corpus
Listed in Hugging Face Datasets
Common Corpus Full paper - ICLR 2026 oral Common Corpus is the largest open licensed text dataset, comprising 2.27 trillion tokens (2,267,302,720,836 tokens).
Description
It is a diverse dataset, consisting of books, newspapers, scientific articles, government and legal documents, code, and more. Common Corpus has been created by Pleias in association with several partners.
Common Corpus differs from existing open datasets in that it is: Truly Open: contains only data that is either… See the full description on the dataset page: huggingface.co/datasets/PleIAs/common_corpus.
Links
Where it is published
- Hugging Face dataset page huggingface.co/datasets/PleIAs/common_corpus ↗
landing page · from Hugging Face
Documentation and papers
- arXiv:2410.22587 arxiv.org/abs/2410.22587 ↗
publication · from Hugging Face
Catalogue records · 1
- Hub API huggingface.co/api/datasets/PleIAs/common_corpus ↗
metadata API · from Hugging Face
Topics
- From keywords
- Computer Science & AI
- Inferred from text
- Text 75%
Provenance · 1 source records, 10 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Hugging Face Datasets | PleIAs/common_corpus | 10 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · Hugging Face | connector:huggingface@1.0.0 | /gated |
| concepts[field].local:field:computer-science-ai | mapping · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:tabular | source · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:text | source · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].local:modality:text | enrichment · Hugging Face | keyword-concept-rules@1.0.0 | title+description (75%) |
| created_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| description | source · Hugging Face | connector:huggingface@1.0.0 | /description |
| publication_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| title | source · Hugging Face | connector:huggingface@1.0.0 | /id |
| updated_date | source · Hugging Face | connector:huggingface@1.0.0 |