Wikipedia

Download and extract text from Wikipedia Dumps using Curator.

Wikipedia releases compressed dumps of all its content in XML format twice per month. Curator provides a complete pipeline to automatically download, parse, and extract clean text from these dumps.

How it Works

The Wikipedia pipeline in Curator consists of four stages:

1. URL Generation: Automatically discovers Wikipedia dump URLs for the specified language and date
2. Download: Downloads compressed .bz2 dump files using wget
3. Iteration: Parses XML content and extracts individual articles
4. Extraction: Cleans Wikipedia markup and converts to plain text

Before You Start

Wikipedia publishes new dumps around the first and twentieth of each month. Refer to the English Wikipedia dumps index at https://dumps.wikimedia.org/enwiki/ for available dates.

Curator uses wget to download Wikipedia dumps. You must have wget installed on your system:

• On macOS: brew install wget
• On Ubuntu/Debian: sudo apt-get install wget
• On CentOS/RHEL: sudo yum install wget

Usage

Here’s how to download and extract Wikipedia data using Curator:

1
from nemo_curator.core.client import RayClient
2
from nemo_curator.pipeline import Pipeline
3
from nemo_curator.stages.text.download import WikipediaDownloadExtractStage
4
from nemo_curator.stages.text.io.writer import JsonlWriter
5
6
# Initialize Ray client
7
ray_client = RayClient()
8
ray_client.start()
9
10
# Create and configure pipeline
11
pipeline = Pipeline(
12
    name="wikipedia_pipeline",
13
    description="Download and process Wikipedia dumps"
14
)
15
16
# Create the Wikipedia processing stage
17
wikipedia_stage = WikipediaDownloadExtractStage(
18
    language="en",
19
    download_dir="./wikipedia_downloads",
20
    dump_date=None,        # None uses latest dump
21
    url_limit=5,           # Optional: limit number of dump files (useful for testing)
22
    record_limit=1000,     # Optional: limit articles per dump file
23
    verbose=True
24
)
25
pipeline.add_stage(wikipedia_stage)
26
27
# Create writer stage to save results
28
writer_stage = JsonlWriter(
29
    path="./wikipedia_output"
30
)
31
pipeline.add_stage(writer_stage)
32
33
# Execute the pipeline
34
results = pipeline.run()
35
36
# Stop Ray client
37
ray_client.stop()

For executor options and configuration, refer to Execution Backends.

Parameters

Output Format

The processed Wikipedia articles become DocumentBatch objects, with each line containing the following fields:

• text: The cleaned main text content of the article
• title: The title of the Wikipedia article
• id: Wikipedia’s unique identifier for the article
• url: The constructed Wikipedia URL for the article
• language: The language code of the article
• source_id: Identifier of the source dump file

If you enable add_filename_column, the output includes an extra field file_name (or your custom column name).

Example Output Record

1
{
2
  "text": "Python is a high-level, general-purpose programming language...",
3
  "title": "Python (programming language)",
4
  "id": "23862",
5
  "url": "https://en.wikipedia.org/wiki/Python_(programming_language)",
6
  "language": "en",
7
  "source_id": "enwiki-20240401-pages-articles-multistream1.xml"
8
}