Running 49 fingerprint tests unattended, and what that takes

To automate browser fingerprint testing, you load each detector in the real browser under test and read its verdict off the page with a script. Reading is the easy half. Our catalogue lists 49 public detectors, and the hard half is that a run nobody watches still has to say, for every row, which of four things happened: the profile failed, the site was down, the page could not be read, or our own code was wrong.

Three kinds of row you need to automate browser fingerprint testing

Each row in src/shared/detection-tests.ts has a verdict field, and it takes one of three values.

• automatic: the page renders a verdict and our runner reads it. Only these rows can pass or fail a run.
• reference: the page has no verdict to read. deviceinfo.me lists sixty browser properties and never says whether one is wrong, so the screenshot is the result and no code is owed.
• manual: the page renders a verdict and nobody has written the code to read it yet.

reference and manual used to be one word. A tally of "N manual" then mixed work we owed with work that cannot exist, and nobody reading the report could tell how much was really unverified. There are no manual rows in the catalogue today: a row has an extractor, or it is a page with nothing to extract. We also refuse to invent a pass threshold over somebody else's table of values to promote a reference row, because a threshold we made up is a guess, not a measurement.

How it works: Machines that exist

The catalogue lied in both directions at once

The app reads that verdict field, and so do the REST API, the MCP tool and the totals at the top of the report. What actually produces a verdict is a switch in run-detection-tests.mjs. That is one fact written down in two files, and the two had drifted apart both ways:

amiunique, fingerprintjs, ipleak   claimed automatic, had no extractor
12 rows, every challenge site      claimed manual, had a working one

The first line is the dangerous one. Our report said "34 automatic" when three of the 34 could not be read by anything. A row with no extractor fell through to the default branch, which reported "manual", and the totals added up what the catalogue declared instead of what the runner got.

Every run now calls reconcile(), which holds each row's claim against what its extractor returned:

1. If a row marked automatic produced nothing, it is filed as a bug in our catalogue, by name, and never as a result about the profile.
2. If a row marked manual produced a real verdict, the verdict stands (it is genuine) and the stale label is reported.
3. If the two disagree anywhere, the run exits non-zero, because every total in the report is computed from the catalogue's copy.

npm run check:catalogue runs that logic against fixed cases in about a millisecond. It needs no browser and no network, so it runs in our build.

A site that was down, a page we could not read, a bug of ours

Each way a row can go wrong sends you to a different place to look, so each gets its own word:

pass / FAIL    a verdict was read and met the bar, or did not
attention      read, but short of the bar
unreachable    the site did not serve a page
unread         the page loaded and no verdict was where we expected it
flaky          different verdicts across rounds
error          our runner threw

unreachable belongs to the site or to the exit. unread is usually ours: a stale selector, or a page we read while it was still scanning. An automatic row that comes back empty is reported as a catalogue bug, in those words. Fold any of these into pass or fail and the run looks better or worse than it was. Nobody goes digging after a run that looks better.

Splitting the statuses out turned up three dead or misread rows. A DataDome row pointed at a demo page its author had retired; the path now served his homepage with HTTP 200, and our challenge extractor would have read a page with no challenge on it as a pass. Another row already covers DataDome, so we deleted that one. A DNS-leak row's domain answered SERVFAIL from both Cloudflare's and Google's DNS-over-HTTPS resolvers, while two control domains in the same batch resolved fine, so the site was down for everybody and that row went too. And dnsleaktest had been reported unreachable while it was alive. Its test buttons are <input> elements, and a probe reading textContent from a, button finds nothing on an input. Reading value as well fixed it.

One tab per row, closed when the row is done

Our first runner sent one tab through every detector. The sites opened popups and ad windows of their own, nothing closed them, and by the end the browser held a few dozen live tabs, which is a state no profile is ever in. Worse, the page target went away three tests into one run and the loop carried on, because it was watching the socket rather than the target. It wrote a report from three real rows and forty-six holes, over a file that had held thirty-five passes.

Now every row gets a fresh tab, and anything still open when the row ends is closed: a target that appeared during the test was opened by the site. We polled the profile's DevTools endpoint 145 times over a full run and counted 3 page targets in every sample (the startup tab, a scratch tab and the tab under test).

What an unattended run still cannot tell you

One round is one observation of somebody else's live service, so run more than one. Read the unreachable rows before the failures, because a flagged exit gets refused before any fingerprint is read. If a detector fails in every configuration you try, suspect your extractor before the profile. The Pixelscan scan that never finishes is a case where the page decided the outcome and the browser did not, and what a CreepJS lie is covers the detector whose verdict is a count of contradictions. How each profile's values are drawn to begin with is on the real fingerprints page.