Table · dataset · 2022

codeparrot/codeparrot-valid-near-deduplication

Listed in Hugging Face Datasets

CodeParrot 🦜 Dataset after near deduplication (validation) Dataset Description A dataset of Python files from Github.

Description

We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files.

We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more… See the full description on the dataset page: huggingface.co/datasets/codeparrot/codeparrot-valid-near-deduplication.

Topics

Stated by source: tabular · text
From keywords: Computer Science & AI

Provenance · 1 source records, 9 field assertions

Source	Key	Last seen	Raw
Hugging Face Datasets	codeparrot/codeparrot-valid-near-deduplication	11 d ago	JSON v1

Field	Assertion	Extractor	Evidence
access_level	source · Hugging Face	connector:huggingface@1.0.0	/gated
concepts[field].local:field:computer-science-ai	mapping · Hugging Face	connector:huggingface@1.0.0
concepts[modality].hf_modality:tabular	source · Hugging Face	connector:huggingface@1.0.0
concepts[modality].hf_modality:text	source · Hugging Face	connector:huggingface@1.0.0
created_date	source · Hugging Face	connector:huggingface@1.0.0
description	source · Hugging Face	connector:huggingface@1.0.0	/description
publication_date	source · Hugging Face	connector:huggingface@1.0.0
title	source · Hugging Face	connector:huggingface@1.0.0	/id
updated_date	source · Hugging Face	connector:huggingface@1.0.0

codeparrot/codeparrot-valid-near-deduplication

Description

Links

Where it is published

Topics