← Publications

8-Bit Quantization and a Floating-Point Trap — Two Structural Pitfalls That Are Easy to Miss When Comparing AI Model Numbers

A plain-language explanation of DOI 1.16 and DOI 1.21, written for readers without specialized background.

This isn't a quirk specific to this project's four models. Any model trained on, or applied to, ordinary 8-bit images sits on top of the same two facts described below — one about how 8-bit images work, one about how computers do arithmetic. We think it's worth writing down clearly, as a record, even though what (if anything) to do about it is a question for the wider field, not something we're trying to settle here ourselves.

In short: Part 1 is about why there are only finitely many true values to begin with. Part 2 is about how those true values can get contaminated — miscounted — by the computer's own floating-point arithmetic.

Part 1: why unrelated patches of image can land on the exact same number (DOI 1.16)

The birthday paradox

Picture a room of 23 strangers. It feels like it should be extremely unlikely that any two of them share a birthday — there are 365 possible days, after all. But the actual probability is over 50%. This is the famous “birthday paradox”: it isn't about any two specific people sharing a birthday, it's about any pair at all among the group, and the number of possible pairs grows much faster than the number of people.

How this applies to pixel brightness

An 8-bit image works the same way as the birthday example above, just with a much larger “room.” Each pixel's brightness can only be one of 256 possible values (0–255) — not any real number, only an integer in that narrow range. This project measures, for every small square (“block”) of an image — 8×8 pixels, 64 of them — a brightness-spread value called σ_gt, computed from all 64 pixel values inside that block (how spread out they are, similar to a standard deviation). Combining 64 inputs, each one of 256 possible integers, can produce a huge number of different exact results — far more than 256 — but because every one of those 64 inputs is still restricted to an integer between 0 and 255 rather than any real number, the set of possible σ_gt outputs, however large, is still a specific, finite, countable number. That number turns out to be about 2.3 million. This 2.3 million is this project's version of “only 365 possible birthdays” — not 256 itself, but the limited (if much larger) set of outputs that 256-valued inputs boil down to once combined. And instead of just 23 people, there are roughly 79 million blocks — so by the same logic as above, many of them land on the exact same value purely by coincidence, with no actual connection between what's pictured in them. Not a bug, not a sign the measurement is broken — the direct, unavoidable consequence of there only being 256 possible pixel brightnesses to begin with, combined with having far more “people” (≈79 million blocks) than “days” (≈2.3 million possible σ_gt values) to go around.

How often this actually happens in practice

DOI 1.16 measured just how often this coincidence happens in practice, in this project's own data: of roughly 79 million blocks, only about 2.3 million distinct σ_gt values exist — on average, over 34 unrelated blocks share each exact value. (This 2.3 million figure itself turns out to have been slightly inflated by how computers do floating-point arithmetic — Part 2 below shows that the true count is actually smaller.)

Part 2: a trap in the computer's own arithmetic (DOI 1.21)

A quick example you can run yourself

Here's a separate, second effect, layered on top of the first one — not about images at all, but about how computers do arithmetic. Try this in any Python interpreter:

>>> (0.1 + 0.2) + 0.3
0.6000000000000001
>>> 0.1 + (0.2 + 0.3)
0.6

Both expressions add up the same three numbers. Mathematically, they're both exactly 0.6. But computers store numbers with decimal points (“floating-point” numbers) using a fixed, finite number of bits, which means most everyday decimal values can't be represented with perfect precision — only approximated, extremely closely. Depending on which addition happens first, that tiny imprecision rounds off slightly differently, and the computer ends up storing two technically different bit patterns for what is, mathematically, the exact same number.

How the same σ_gt ends up counted as two separate values

This matters here because of Part 1 above: since so many unrelated blocks are supposed to land on the exact same σ_gt, whether the software actually computed each one via the exact same sequence of additions determines whether they end up stored identically too. If two blocks' calculations happened to take a very slightly different path through the arithmetic, both can be mathematically exactly the same σ_gt, yet get stored as two bit patterns that are technically different by the smallest possible amount — we call this “ghost duplication.” A direct, naive way of counting (“how many different exact values appear in this column?”) then miscounts these as two separate values instead of one, inflating the apparent count.

A real example: how much this inflated the distinct-value count

A real example, found in this project's own data: two unrelated blocks in one image, sharing the same true σ_gt (≈0.0004863513439456968), were found stored with a gap of 5.421010862427522e-20 between them — exactly 2^-64, a clean IEEE 754 machine constant that comes directly from how computers represent floating-point numbers, not a real difference in content. DOI 1.21 measured how much this trap inflates DOI 1.16's distinct-value counts: about 1.95× too high for σ_gt (1.18 million was the true figure, not the roughly 2.3 million first reported), and about 4.2× too high for the related brightness measure μ_gt (16,320 was the true figure, not the 68,649 first reported) — the same ghost-duplication mechanism, just a different degree of inflation for each quantity.

Why this is bigger than one project

Part 1 (8-bit quantization) applies to any model trained on, or evaluated against, ordinary 8-bit images — which is most of them. Part 2 (floating-point ghost duplication) applies to any software, in any field, that counts exact distinct values computed from floating-point arithmetic, which is a far broader category than image models alone. Anyone measuring how “discretized” a model's behavior is, or comparing exact values across many computed results, is exposed to the same pair of structural effects described here.

We're recording this clearly, with working code anyone can re-run (see the links below), because we think the problem is worth having on the record at this level of precision. What to do about it — whether it matters for a given use case, and if so how to handle it — is a judgment we're leaving to the wider research community, not something we're claiming to have settled ourselves.

Want more detail?

Technical detail (verification SQL, screen recordings, file hashes) is published in the Zenodo records for DOI 1.16 and DOI 1.21, with a query-by-query walkthrough on the DOI 1.21 SQL page and term-level definitions in the glossary.