of already-filtered RefinedWeb documents removed by fuzzy+exact dedup (~12% of original Common Crawl documents) — dedup runs after language/quality filtering
~50%observed
| Value kind | observed — Reported measurements, counts, and specifications keep the precision and scope stated by their source; an exact specification is not treated as a range. |
|---|---|
| Scope | corpus dedup removal share |
| As of | 2024 |
| Source | RefinedWeb (NeurIPS 2023), Figure 2 / Table 5 |
| Review | checking…review by 2026-10-24 · standard cadence |
| Recorded changes | last 2026-07-14 · 2 revisions tracked |
| Claim id | duplicate-share-of-multi-snapshot-common-crawl |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.