Data · dataset · 2022
MultiParaCrawl
Listed in Hugging Face Datasets
Description
Parallel corpora from Web Crawls collected in the ParaCrawl project and further processed for making it a multi-parallel corpus by pivoting via English. Here we only provide the additional language pairs that came out of pivoting. The bitexts for English are available from the ParaCrawl release. 40 languages, 669 bitexts total number of files: 40 total number of tokens: 10.14G total number of sentence fragments: 505.48M Please, acknowledge the ParaCrawl project at paracrawl.eu.
This version is derived from the original release at their website adjusted for redistribution via the OPUS corpus collection. Please, acknowledge OPUS as well for this service.
Links
Where it is published
- Hugging Face dataset page huggingface.co/datasets/Helsinki-NLP/multi_para_crawl ↗
landing page · from Hugging Face
Catalogue records · 1
- Hub API huggingface.co/api/datasets/Helsinki-NLP/multi_para_crawl ↗
metadata API · from Hugging Face
Topics
- Stated by source
- translation
- From keywords
- Computer Science & AI
- Inferred from text
- Artificial intelligence 73%
Provenance · 1 source records, 10 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Hugging Face Datasets | Helsinki-NLP/multi_para_crawl | 13 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · Hugging Face | connector:huggingface@1.0.0 | /gated |
| concepts[field].anzsrc:group:4602 | enrichment · Hugging Face | taxonomy-embedding@1.1.0 | title+keywords+description (73%) |
| concepts[field].local:field:computer-science-ai | mapping · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[task].hf_task:translation | source · Hugging Face | connector:huggingface@1.0.0 | /tags[task_categories:*] |
| created_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| description | source · Hugging Face | connector:huggingface@1.0.0 | /description |
| license | source · Hugging Face | connector:huggingface@1.0.0 | /tags[license:*] |
| publication_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| title | source · Hugging Face | connector:huggingface@1.0.0 | /id |
| updated_date | source · Hugging Face | connector:huggingface@1.0.0 |