The Data-Prep Supercomputer: Offline Data Processing

There are two supercomputers in a frontier training program, and only one of them is GPU-dominant. The first — the one this part of the guide has been about — keeps thousands of accelerators saturated with a clean, shuffled, tokenized stream. The second runs before that, often weeks before, and its job is to manufacture the stream in the first place: ingest raw web crawls and licensed corpora, strip boilerplate, deduplicate at corpus scale, filter for quality and safety, scrub out anything that overlaps your evaluations, and tokenize the survivors into the exact binary format the loader will memory-map. This is the data-prep supercomputer, and it is the most consistently undersized, under-instrumented, and under-respected machine in the building.

The reason it gets disrespected is that it looks like ETL — and ETL is something every org thinks it already knows how to do. But web-scale data prep for a frontier run is not a nightly Spark job. It is a multi-petabyte, trillion-token batch pipeline whose stages have opposite hardware appetites — some embarrassingly parallel and CPU-bound, some metadata-storm small-file workloads, some GPU-accelerated classifier sweeps, some all-to-all shuffles that look like a network benchmark. Size it as one thing and you bottleneck on whichever stage you got wrong. This chapter walks the pipeline stage by stage and ends by sizing the cluster that runs it, which has a power, network, and storage profile deliberately unlike the GPU fleet next door.

The strategic case for taking prep seriously is a goodput case, and it cuts two ways. First, prep is on the critical path to the run. If your corpus takes six weeks to prepare and your GPU cluster is energized and idle for four of them, you have paid frontier-cluster lease or depreciation for a month of string processing. The prep cluster's wall-clock is a direct deduction from the training program's schedule, and the GPUs cannot start that training release until its authorized, versioned input is ready; incremental corpus releases need their own quality gates. Second, the quality of prep sets the ceiling on what the GPUs can achieve. The dominant lesson of the open-data era — FineWeb, RefinedWeb, DataComp-LM, Dolma, Nemotron-CC — is that careful filtering and deduplication beats raw scale: a smaller, cleaner corpus trains a better model than a larger, dirtier one at the same token budget. Every duplicate you fail to remove is a GPU-hour spent memorizing boilerplate; every leaked benchmark sample is a leaderboard number you have to caveat.

The decision is rarely whether to do prep at all; it is how much engineering to invest, and on what hardware. The expensive failure mode is the one nobody plans: discovering, two weeks before a launch, that the prep pipeline cannot tokenize the licensed corpus fast enough on the CPU pool you have, and either delaying the run or — far worse — borrowing GPU nodes to brute-force tokenization, which is an expensive way to run a Rust regex engine if the exact tokenizer and normalization pipeline still execute on the CPU while the accelerators sit idle.

A web-scale prep pipeline is a directed sequence of stages, each with a distinct hardware personality and a distinct fork. The canonical order — text extraction → language ID → quality/heuristic filtering → exact dedup → fuzzy dedup → classifier filtering → decontamination → tokenization → shuffle/shard — is not arbitrary: you filter cheaply before you filter expensively, and you dedup before you tokenize so you never tokenize a duplicate. The table below maps each stage to what binds it, so you can see where the prep cluster's bottleneck actually lives.

The web mirrors, syndicates, and templates itself relentlessly. Removing those duplicates is the single biggest lever on token efficiency, because a model wastes capacity memorizing anything it sees too many times, and over-represented duplicates skew the distribution. But dedup is also a deletion from the derived training set; retained source IDs and an authorized immutable input make that exclusion reproducible and reversible, and the aggressiveness knob — how similar is "the same" — is the most consequential single number in the pipeline.

There are three tiers, and they compose. Exact dedup hashes whole documents or lines and drops identical copies; it is cheap, CPU-bound, embarrassingly parallel, and catches the trivial cases. Fuzzy / near-duplicate dedup is the real work: compute a MinHash signature per document (a compact sketch of its n-gram set), then use Locality-Sensitive Hashing (LSH) to bucket documents whose signatures collide in any band, so you only compare plausibly-similar pairs instead of the quadratic all-pairs explosion. The banding configuration sets a probability curve, not a hard Jaccard cutoff. FineWeb used word 5-grams and 14 bands of eight MinHash values; its method and per-snapshot comparison are documented by the authors in 2024. At similarity s, the candidate probability is 1 − (1 − s⁸)¹⁴; at s = 0.75 it is about 77%. Preserve normalization, candidate handling and survivor policy with the resulting corpus. Semantic dedup goes further: embed every document, cluster in embedding space, and drop near-neighbors that are paraphrases or translations a MinHash would miss — at the cost of an embedding-model inference pass over the entire corpus.

The hardware fork inside dedup carries the biggest 2026 swing. Fuzzy dedup at trillion-token scale is an all-to-all shuffle of billions of signatures, and on a CPU cluster it is routinely the slowest stage in the entire pipeline — days of wall-clock on a large cluster. GPU-accelerated dedup (NVIDIA's NeMo Curator on RAPIDS, and the FED-class GPU dedup frameworks) collapses that. NVIDIA’s NeMo Curator deduplication stages put GPU implementations on the shortlist, but compare signature generation, bucketing, shuffle and survivor selection separately at fixed corpus, normalization and dedup policy. Keep end-to-end wall-clock, peak memory, spill and accepted output with the CPU/GPU comparison; a fast signature kernel does not time the complete corpus release. That is the rare case where the prep cluster legitimately wants a few GPUs — not to train, but to keep the dedup stage from being the long pole. You either pay for a large CPU pool to grind the shuffle, or borrow a small GPU partition to do dedup in hours and free the CPUs for tokenization.

After dedup, you decide what is worth keeping. The fork is heuristic filtering vs classifier-based curation, and most serious pipelines use both — heuristics first because they are cheap and interpretable, classifiers where a measured quality gain justifies their cost.

Heuristic filters are the C4/Gopher/FineWeb lineage: drop documents by language-ID confidence, mean line length, fraction of alphabetic characters, repetition ratios, presence of boilerplate or blocklisted terms, perplexity under a reference model. They are CPU-cheap, fully auditable, and every threshold is a knob you can explain to a regulator or a model-behavior reviewer. Classifier filters score each document with a trained model — a fastText or transformer quality classifier (often trained to recognize "educational" or instruction-like text, as in FineWeb-Edu and the DataComp-LM curation), plus safety classifiers for toxicity, CSAM signals, and policy categories. These can capture learned quality signals beyond a hand-authored regex, but corpus-scale cost depends on the selected CPU/GPU implementation and their decisions can be harder to interpret than explicit rules; their false-positive rate silently shapes the model: a quality classifier that down-weights a dialect, a domain, or a non-English register is making a model-behavior decision, not a data-cleaning step.

For the technical stratum, A rejects 8/1,000 = 0.80% and passes this sample gate; B rejects 35/1,000 = 3.5% and fails. The crossover is 20 eligible rejections; 21 fails even if the pooled global score improves. Select A only for the illustrated sample comparison. Real corpus release remains HOLD until every required stratum, retained-corpus quality test and eligibility gate passes, with uncertainty and adjudicator disagreement reported.

Compare dedup scope and filter thresholds at fixed token budget, tokenizer and evaluation protocol; inspect source/language retention as well as model outcomes. A sample pass is not a population bound. Preserve excluded IDs and reason codes under authorized retention so the corpus can be regenerated after an approved policy change. FineWeb's controlled curation comparisons supply the experimental method; Chapter 10.10 owns policy and this chapter owns its execution.

Decontamination is the stage that protects the meaning of every benchmark you will ever report. If your pretraining corpus contains the test items from GSM8K, MMLU, HumanEval, or whatever you benchmark on, your model can memorize the answers and your reported scores are fiction. This is not hypothetical: MMLU questions, like other evaluation material, can enter a training corpus as exact copies or transformed questions and make a model's score reward leakage rather than generalization. Test both kinds of overlap against the versioned evaluation registry before interpreting the score. The fork is how aggressively you scrub, and against which eval sets.

The standard mechanism is n-gram / substring overlap: index every eval set you care about, then drop or flag any training document that shares a long enough contiguous span (commonly an 8- to 13-gram match) with a test item. It is CPU-bound, index-lookup-heavy, and conceptually simple. The catch is that it is necessary but not sufficient: paraphrase, translation, and reformatting slip straight past a literal n-gram match, so a corpus can pass decontamination and still be semantically contaminated. What actually works is procedural rather than algorithmic: maintain a frozen, versioned registry of every benchmark and decontaminate against the entire registry before training; when you add an eval after training, scan the frozen training corpus referenced by its manifest for overlap and, if overlap exists, mark the original score contaminated and use a clean mirror or held-out replacement; then add the eval to the registry and remove overlaps before the next training run. Get it wrong and the cost is a credibility event: an outside party reproduces your eval on a clean split and your numbers collapse.

Tokenization is where the prep cluster's hardware profile reveals itself, and where the most common sizing surprise lives. BPE applies ranked pair merges; Unigram selects the best path through a scored segmentation lattice. Widely deployed production tokenizers are often CPU-oriented and parallelize across documents, but GPU BPE implementations can also parallelize merge work within a document. Benchmark the exact tokenizer, normalization and regex path, document-size distribution, batch size, and available CPU/GPU implementation; those measurements — not an assumed CPU-only limit — size the tokenization fleet.

At frontier-corpus scale, tokenization is a large compute pass whose wall-clock depends on measured end-to-end throughput. The decision is whether to size a dedicated CPU pool, use a GPU implementation, or lean on a managed distributed framework (Spark, Ray, Dask, or NeMo Curator's streaming executor that overlaps CPU and GPU stages). Choose the hardware path by throughput, cost, and pipeline compatibility on the real corpus.

The prep supercomputer is sized against the opposite constraints from the GPU fleet, and the headline is that CPU-bound extraction and tokenization can make prep CPU-dense, RAM-heavy and storage-bandwidth-bound; GPU dedup or classification changes that mix, so size each stage’s compute, memory, storage and network before calling the hall power-light. A CPU-heavy prep rack can fit an air-cooled row where a dense training GPU rack needs liquid cooling; its populated CPU/GPU mix still sets the actual power and cooling envelope, and its scarce resources are CPU cores, DRAM capacity (dedup hash tables and shuffle buffers are memory-hungry), fast local scratch (NVMe for intermediate shuffle spill), and sustained sequential read bandwidth from the object/capacity tier where the raw corpus lives (→ Chapter 9.6). The network it stresses is the data-center fabric for all-to-all shuffles, not a non-blocking GPU back-end.

The two structural decisions are dedicated vs shared and co-located vs remote. A dedicated prep cluster (or a large pool of general CPU compute) decouples prep wall-clock from GPU availability — you prepare the next corpus while the current run trains — at the cost of standing up and powering a second cluster. Sharing idle GPU-node CPUs is capital-efficient but couples your prep schedule to GPU idleness and wastes accelerators. On location: prep wants to run where the data already is, because raw corpora are multi-petabyte and data gravity makes them economically immovable (cloud egress at ~$0.087–0.12/GB first-tier list price in 2026 makes moving a petabyte a ~$90–120k line item; negotiated rates lower) — so prep is the textbook case of move compute to data, the same gravity logic developed in Chapter 9.8.

The stage ledger below exposes which resource sets each duration. Its rates are qualification targets from the stated assumptions; a faster isolated kernel changes the release date only if it changes the limiting term.

The same prep failures recur, because each comes from treating prep as ETL rather than as a sized supercomputer:

• Tokenizing without implementation benchmarks. Sizing a CPU pool or borrowing GPU nodes before benchmarking the exact normalization, regex, and tokenizer pipeline. The wrong hardware mix leaves either CPU cores or accelerators idle.
• Treating a post-training scan as a repair. When a new eval overlaps the frozen training manifest, mark the original score contaminated and use a clean mirror or held-out replacement; add the eval to the registry and remove overlaps before the next training run.
• Treating dedup threshold and classifier operating point as hygiene, not hyperparameters. Picking a similarity threshold and a quality-classifier cutoff because they sounded reasonable, never ablating them, and discovering a quality regression or a silent capability gap after a frontier run. Global dedup that deletes the good recurring web is the canonical case.
• Sizing the prep cluster with GPU-fleet instincts. Liquid cooling, a non-blocking back-end, accelerator-first — buying that package before measuring the prep stages can overspend on idle GPUs while starving the cores, DRAM, scratch and read bandwidth the CPU-bound stages need. Derive each resource from the complete pipeline budget. → Chapter 9.8.

Choose the prep resource mix by the stage that controls delivery, and the curation policy by accepted output quality at fixed training budget. Keep corpus release behind both gates: finishing early with an untraceable or over-filtered dataset does not feed the intended run. Under-sizing memory or scratch delays the program; accepting an untested exclusion policy spends the GPU budget on a different corpus than the one the program meant to train.