Data · dataset · 2025
Pairwise Duplicate News Dataset
Listed in Teesside University Research Data Repository
The Pairwise Duplicate News Dataset consists of 10,836 article pairs (5,418 duplicate and 5,418 non-duplicate) collected from seven major Pakistani English news outlets, DAWN, The Express Tribune, GEO News, The News International, The Nation, 24 News, and 92 News HD.
Description
The dataset was created from a larger corpus of 54,929 news articles covering eight categories: National, World, Business, Sports, Entertainment, Crime, Technology, and Health.
The dataset was constructed using Sentence-BERT embeddings and FAISS cosine similarity search, with a similarity threshold of 0.85 for identifying duplicate articles. Each duplicate cluster represents a single real-world news event, and one pair was sampled per cluster. Non-duplicate pairs were formed by randomly pairing articles from different clusters and stratified into three similarity levels: easy (0.2-0.3), medium (0.3-0.5), and hard (0.5-0.7) to ensure balanced evaluation difficulty.
Links
Where it is published
- DOI doi.org/10.17632/jmrhxb5666.1 ↗
DOI / persistent id · from researchdata tees ac uk
Catalogue records · 1
- OAI-PMH record data.mendeley.com/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai%3Adata… ↗
metadata API · from researchdata tees ac uk
Topics
- From keywords
- Artificial intelligence · Computer Science & AI · Earth & Environmental Science
Provenance · 1 source records, 8 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Teesside University Research Data Repository | oai:data.mendeley.com/jmrhxb5666.1 | 16 h ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · researchdata tees ac uk | connector:researchdata_tees_ac_uk@1.0.0 | |
| concepts[field].anzsrc:group:4602 | mapping · researchdata tees ac uk | vocabulary-mapper@1.0.0 | keywords['Artificial Intelligence'] |
| concepts[field].local:field:computer-science-ai | mapping · researchdata tees ac uk | connector:researchdata_tees_ac_uk@1.0.0 | |
| concepts[field].local:field:earth-environmental | mapping · researchdata tees ac uk | connector:researchdata_tees_ac_uk@1.0.0 | |
| description | source · researchdata tees ac uk | connector:researchdata_tees_ac_uk@1.0.0 | /metadata/dc/description |
| license | source · researchdata tees ac uk | connector:researchdata_tees_ac_uk@1.0.0 | /metadata/dc/rights |
| publication_date | source · researchdata tees ac uk | connector:researchdata_tees_ac_uk@1.0.0 | |
| title | source · researchdata tees ac uk | connector:researchdata_tees_ac_uk@1.0.0 | /metadata/dc/title |