FineWeb 2024 MinHash/LSH deduplication operating configuration
112 hashes (14 × 8) over 5-gramsobserved
| Value kind | observed — Reported measurements, counts, and specifications keep the precision and scope stated by their source; an exact specification is not treated as a range. |
|---|---|
| Scope | Word 5-grams and MinHash banding; 0.75 is a similarity target, not a hard cutoff. Collision probability 1−(1−s^8)^14 is about 77% at s=.75. Per-snapshot dedup is an experiment result for that corpus, not a universal policy. |
| As of | 2024 |
| Source | FineWeb (2024) method description. · Our deduplication parameters: 112 hashes, 14 bands of 8; collision-probability equation and per-snapshot experiments |
| Review | checking…review by 2027-09-05 · standard cadence |
| Recorded changes | last 2026-09-16 · 2 revisions tracked |
| Claim id | fineweb-fuzzy-dedup-config-minhash-over-5-grams |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.