Data · dataset · 2022
loubnabnl/data-filtering-statistics
Listed in Hugging Face Datasets
Description
Filterings on top of near-dedup + line filtering: Comments filtering (at least 1% of the number of lines should be comments/docstrings) Stars filtering (minimum of 5 stars) (on top of near-dedup + line filtering) Language Before filtering Stars Comments ratio More near-dedup Tokenizer fertility Python 75.61 GB 26.56 GB 65.64 GB 61.97 GB 72.52 GB Java 110 GB 35.83 GB 92.7 GB 88.42 GB 105.47 GB Javascript 82.7 GB 20.76 GB 57.5 GB 65.09 GB 76.37 GB
Links
Where it is published
- Hugging Face dataset page huggingface.co/datasets/loubnabnl/data-filtering-statistics ↗
landing page · from Hugging Face
Catalogue records · 1
- Hub API huggingface.co/api/datasets/loubnabnl/data-filtering-statistics ↗
metadata API · from Hugging Face
Topics
- From keywords
- Computer Science & AI
Provenance · 1 source records, 7 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Hugging Face Datasets | loubnabnl/data-filtering-statistics | 13 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · Hugging Face | connector:huggingface@1.0.0 | /gated |
| concepts[field].local:field:computer-science-ai | mapping · Hugging Face | connector:huggingface@1.0.0 | |
| created_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| description | source · Hugging Face | connector:huggingface@1.0.0 | /description |
| publication_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| title | source · Hugging Face | connector:huggingface@1.0.0 | /id |
| updated_date | source · Hugging Face | connector:huggingface@1.0.0 |