WebInsight Data Repository

This is the dataset used in the paper: "Look back, look around: a systematic analysis of effective predictors for new outlinks in focused Web crawling". The original paper can be found on: arxiv.org/abs/2111.05062. The dataset contains information on 384,325 unique URLs.

For each URL, there are more than 230 attributes that describe that URL and its corresponding page for 10 consequent weeks, for example, page content length, language, text size, number of internal and external links and so many other attributes. There are also 192 columns (starting with the "SV" prefix) which represent the semantic vector related to the page text. The target column is linkInternalChangeRate and linkExternalChangeRate.