Text · dataset · 2024
FineWeb
Listed in Hugging Face Datasets
Description
🍷 FineWeb 15 trillion tokens of the finest data the 🌐 web has to offer What is it? The 🍷 FineWeb dataset consists of more than 18.5T tokens (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 datatrove library, our large scale data processing library. 🍷 FineWeb was originally meant to be a fully open replication of 🦅 RefinedWeb, with a… See the full description on the dataset page: huggingface.co/datasets/HuggingFaceFW/fineweb.
Links
Where it is published
- Hugging Face dataset page huggingface.co/datasets/HuggingFaceFW/fineweb ↗
landing page · from Hugging Face
- DOI doi.org/10.57967/hf/2493 ↗
DOI / persistent id · from Hugging Face
Documentation and papers
- arXiv:2306.01116 arxiv.org/abs/2306.01116 ↗
publication · from Hugging Face
- arXiv:2109.07445 arxiv.org/abs/2109.07445 ↗
publication · from Hugging Face
- arXiv:2406.17557 arxiv.org/abs/2406.17557 ↗
publication · from Hugging Face
Catalogue records · 1
- Hub API huggingface.co/api/datasets/HuggingFaceFW/fineweb ↗
metadata API · from Hugging Face
Topics
- Stated by source
- tabular · text · text generation
- From keywords
- Computer Science & AI
Provenance · 1 source records, 11 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Hugging Face Datasets | HuggingFaceFW/fineweb | 13 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · Hugging Face | connector:huggingface@1.0.0 | /gated |
| concepts[field].local:field:computer-science-ai | mapping · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:tabular | source · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:text | source · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[task].hf_task:text-generation | source · Hugging Face | connector:huggingface@1.0.0 | /tags[task_categories:*] |
| created_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| description | source · Hugging Face | connector:huggingface@1.0.0 | /description |
| license_text | source · Hugging Face | connector:huggingface@1.0.0 | |
| publication_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| title | source · Hugging Face | connector:huggingface@1.0.0 | /id |
| updated_date | source · Hugging Face | connector:huggingface@1.0.0 |