German News Portals: The Linked Domains In Their Articles

Methods

To collect the data, we employed a web crawling and scraping approach, where the homepages of the 12 news portals were crawled daily at 12 noon during the survey period (from 2024-08-27 until 2024-10-31). We used the Scrapy and Selenium libraries in Python to implement the crawler and scraper, which allowed us to handle cookie banners, interactive elements, and dynamic content when necessary. The crawler extracted all news articles in text form on the homepage, along with the links contained within the article text.

The code is available on Github at github.com/EstherKuerbis/NewsArticlesScraper.git, allowing other researchers to review, use, and modify the code for their own purposes. To exclude irrelevant links, we filtered out links that appeared alongside the article text, such as advertisements. Where possible, we extracted the articles’ departments from the scraped keywords or URLs.

To categorize the domains, we used a Python script to automatically subdivide them into internal and external domains based on the domain name in the URL. This approach enabled us to collect a comprehensive dataset of news articles and links, which can be used to analyze the linking behavior of news portals. During the survey period (from 2024-08-27 until 2024-10-31), the homepages of the news portals were crawled daily at 12 noon.

All news articles in text form on the homepage were also crawled. The article texts and the links contained therein were scraped. Links that appeared alongside the article text (e.g. advertisements) were excluded.

The crawler and scraper were implemented in Python using the Scrapy and Selenium libraries. This made it possible to handle cookie banners or other interactive and dynamic elements when necessary. Where possible, the articles' departments were extracted from the scraped keywords or the URLs.

The domains were automatically subdivided into internal and external using a Python script based on the domain name in the URL.