Tardelli Stekel PRO

m5 read 2 after today's update, so we both called it. Without hf_xet, snapshot_download sends a HEAD and a GET to /resolve for the .gguf and both count, like a first hf_hub_download. With hf_xet installed it's 0, since the bytes never go through /resolve. Both cases are in the data card now: https://huggingface.co/datasets/modelpulse/model-pulse-data

Thanks for reading whole tensors. That's a stronger check than mine.

First, a correction to my post: the sample isn't 128 bytes. Tensors up to 32,768 values are read whole and compared byte for byte, and that covers every norm vector in both of those models. Bigger tensors get a 512-value window from the middle row. So if a single norm value moves by 1 ulp, the tensor counts as changed. Your embedding rows go further than the engine does, since it reads only 512 values of one row there. I've fixed the "128 bytes" on the card and the method page.

On the 1,365: the count is right, but the direction splits. In 601 of them the child is a quantization of the parent its card names, which is your reading. In the other 764 it goes the other way: the card names a quantized repo (FP8, AWQ, I8…) as the parent, and the child holds full-precision weights that match it after quantization. Those are the original or a dequantized copy. So about 20% of the 2,923 are quants labeled as fine-tunes.

1 ulp: yes. I went through the norms of 14,088 declared fine-tunes, using the 64 values the store keeps per tensor. Of the ~8,000 pairs where any norm moved, 60 differ by exactly 1 ulp everywhere I can see. Closest to your case: Nabbers1999/Mini-Llama-8B-Chat-SFT-0129 vs Ministral-3-8B-Base-2512. 1 of 69 norm tensors moved, and only one value, by 1 ulp. One more rounding and it reads as lora_merge. 48 full fine-tunes have only 1 or 2 norm tensors moved. So your case is real at the edge, and some lora_merge pairs are probably full fine-tunes whose norm updates all rounded away. Weights alone can't separate those from frozen norms, which is why the verdict reads "a merged LoRA (or a fine-tune with frozen norms)".

Your numbers reproduce here. Using the observed tie values, with 2-decimal triples drawn uniformly and summed in random order, I get 3.66 expected first-listed labels on the 25 multi-option ties (observed 3) and 0.23 expected "false" on the 10 noul ties (observed 1, P(≥1) ≈ 0.21).

On the samples: the public release has only the means. The repo holds the parquets, the card and bench/, and nothing per sample. The closest thing is label_agreement, which keeps per-question summaries (argmax_agree, argmax_majority, total_variation). That doesn't recover the bit either: on the 35 ties argmax_majority always lands inside the tied pair, but in the 31 cases where label and the scorer split it sides with label only 21 times. Whether the generator run kept the raw samples is a question for the maintainers. I'll ask on #68 so it sits next to the tie-rule question.

Reproduced your three-reading split on test, and train shows the same pattern (61 ties, label on the later-listed option in 53).

My best guess for the 4, which only the generator code can confirm: label was taken from the unrounded means, before the 6-decimal rounding, with ties going to the later option. Two things point that way. The gold values are means of three 2-decimal samples (p x 3 sits on the 0.01 grid for 98.5% of the 28,400 probabilities). And the noul ties are all exactly 0.5, with label "true" in 9 of 10 on test and 21 of 22 on train, which is what a ">= 0.5 is true" rule gives. The exceptions then come from float noise: summing three 2-decimal values in a different order changes the last bit in about 13% of permutations, so two options that tie after rounding can differ by ~1e-16 before it, and that decides it. customer_service_000066, the one noul tie labelled "false", is one of your 4.

If that's right, label is deterministic but not meaningful on ties, which is one more reason for symmetric split credit. I opened the PR that saves per-decision probabilities (#68 on the dataset) and left the tie rule out of it, so the maintainers can pick it separately.

Confirmed on c9e8c416: 35 ties, all two-way, 17 of them including the first-listed option. One more thing in the same spot: the gold label field breaks those ties the other way in 31 of the 35, so scoring against label and running score_with_jobs.py give different numbers for the same predictions. Rows on the card use both: Decider-4B, od1 and OpenDecider match their card values under label; nullspan and Qwen3.8-27B (#60) match under the scorer's argmax. Across the 12 runs the two rules differ by 0.001 to 0.007.

On Jev, the 0.006 to 0.0065 gap splits into 9 decisions from gold ties and 3 from ties in the prediction itself. Jev returns probabilities rounded to two decimals, so 20 of its 2,000 answers have a tied top option, and list.index breaks those by option order too (OpenDecider-medium has 104).

So yes to split credit, and I'd apply it on both sides: the expected hit when gold ties and predicted ties are each broken at random, |top_gold ∩ top_pred| / (|top_gold| × |top_pred|). It is deterministic, independent of option order, and equal to the current rule when nothing is tied. For Jev's whole-case run it gives 0.7328, between the scorer's 0.7305 and label's 0.7350.

Agreed on the PR: save the normalized probabilities per decision, keyed on case id and question id, not the hits, so the tie rule becomes a scoring choice anyone can rerun.

Per-rule table for all 12 runs: https://huggingface.co/spaces/tardellirs/typed-decisions-explorer/tree/main/analysis

Checked all of it in the 2.4.0 install: line 1899 returns None without Xet, is_xet_available reads HF_HUB_DISABLE_XET and then looks for hf_xet (utils/_runtime.py, line 157), and the METADATA marker is the one you quote. Without hf_xet, xet_file_data still comes back from the HEAD, so line 1818 keeps url_to_download on /resolve, and http_get goes there (2026–2034). HEAD plus GET on the query file.

My bet is 2 as well. The one call m5 adds that we haven't priced is a GET on /resolve that answers 302 to the Xet bridge, and m4's 302 HEAD counted, so I don't see a way to 1.

The default still decides the count, though. hf_xet comes with huggingface_hub on x86_64, amd64, arm64 and aarch64, which covers Macs, almost every Linux box and Windows. The 2 needs HF_HUB_DISABLE_XET=1, an install without hf_xet, or a machine outside that list. So for a GGUF repo the counter should still miss nearly every snapshot pull. It just isn't true everywhere.

I'll run m5 as m3 with HF_HUB_DISABLE_XET=1 and read it after the next batch. The card's line gets "with hf_xet installed" either way, plus the m5 number once it's in.

The batch ran: m3 = 0, m4 = 1. That's the first row, so it lands neither on your (1, 2) nor on the token reading.

Every request in the log carried X-HF-Download-Counter: 1, so:

• The header doesn't decide anything. m3 sent seven flagged calls (revision, tree, HEAD+GET on .gitattributes and README.md, the xet-read-token GET), and none counted.
• The token GET doesn't count. Both repos sent one, and m3 stayed at 0.
• m4's 1 is the HEAD on /resolve for model.gguf, answered with a 302. A 302 HEAD counts like m1's 200 HEADs did.
• A GGUF on Xet fetched with snapshot_download counts zero, because its bytes never touch /resolve.

That last one is wider than scripts. In 2.4.0, hf download <repo> with --include, or with more than one file name, also goes through snapshot_download (cli/download.py, line 187). Only a single file name takes hf_hub_download. So hf download <repo> --include "*Q4_K_M.gguf", the usual way to pull one quant, shouldn't count either. That's from the source; I haven't run it.

As promised, the card keeps its sentence and gets one line:
"A .gguf stored on Xet and fetched with snapshot_download (huggingface_hub 2.4.0) counted 0: the client takes the file's Xet hash from the tree listing and pulls the bytes from Xet storage, with no request to /resolve. The same file fetched with hf_hub_download counted 1, for its HEAD on /resolve."

For Model Pulse it means a GGUF repo's counter undercounts by an amount nobody can see from the Hub. Requests on /resolve still count; snapshot pulls of Xet GGUFs don't.

Thanks for re-running it. Since then I found more predictions: the OpenDecider repo (#8) publishes per-decision files for its own models and for its runs of Jev 1.13 (TypeSafe API), microsoft-decision-1 and clm-8b (the last two are not on the card), so the check now covers 12 runs. Numbers below use the analytic case-level SE, which matches the bootstrap and gives 1.35 for od1 as you found.

On the top: deff does grow with accuracy (Spearman 0.82 over the 12), from 1.02–1.18 below 0.67 to 1.28–1.40 above 0.78. The highest is Decider-4B at 0.827 with 1.40, n_eff about 1,430. So it rises, but in this sample it levels off near 1.4.

Paired against Jev (OpenDecider's whole-case run, 0.7305 under the dataset scorer's argmax rule, 0.737 in their own report, which breaks the 35 tied gold labels differently; the card's 0.727 is a separate run), the paired SE is 0.72–0.93x the independent one, median 0.79. d3-mini's +0.016 stays inside the band even at 0.72x.

On showing the paired band when both rows ship predictions: I'd rather not mix two kinds of band in one table yet. Of the 90 rows, only about ten have public predictions, two of them zero-shot, and the only Jev predictions come from a third-party run rather than the card's. The cleaner fix is upstream: score_with_jobs.py writes only the summary to results.json; if it also wrote the per-decision answers, every new row would ship them and the explorer could move every comparison to a paired case bootstrap at once. @sandeep85 @codelion , would that be welcome as a PR?

Script, sources and output: https://huggingface.co/spaces/tardellirs/typed-decisions-explorer/tree/main/analysis

This comment has been hidden

Thanks, Dipankar, both points are fair to raise.

On the post line: the explorer leaves nanoGentzen-Jev-v3 out of the zero-shot view, and the footer gives the reason (its own eval YAML in #59 describes it as fitted on train). The post followed that view without saying so; I should have. And yes, "above Jev" there is a point estimate: by the explorer's own rule neither 0.731 nor 0.743 separates from 0.727.

On n: you're right that the 5 decisions per case are not independent, so I ran a case bootstrap on the submissions that ship per-decision predictions (od1-typed-decisions, Decider-4B, nullspan v1.3.0, Qwen3.8-27B from #60), using the scorer's own accuracy rule. Design effects came out at 1.0–1.4, so n_eff is about 1,450–2,000, not 400. For comparing two models it helps further that both answer the same cases: the paired, case-resampled SE of the difference is 0.64–0.90× the independent n = 2,000 SE the explorer uses, so its tie band is, if anything, conservative. Caveat: only four models, and none of the current top zero-shot rows publish predictions, so I can't check those directly. If more submissions add predictions, a paired case bootstrap would be the right rule and I'll switch to it.

Line 575 checks out, and so does the split. A second snapshot_download in the same process sends only repo_info: the tree cache hits on the commit at line 282, and each file returns at the pointer check. If it lands on (1, 1), I'll run that.

On the card: it now says only requests to a library's query files count. (0, 1) leaves that true and adds one line: a GGUF fetched with snapshot_download counts zero, because its bytes never pass through /resolve. Every other row breaks that sentence, since what counted wasn't a request to a query file. The card would have to say the counter reads the client's X-HF-Download-Counter header, and which calls it counts. Under (1, 2), a snapshot would also count once per process however many GGUFs it pulls, since the token is cached per revision.

For Model Pulse that matters more than the card. Under (0, 1), a GGUF repo's counter misses everyone who uses snapshot_download. Under (1, 2), it counts them once per run, not per file. Either way it isn't a file count.

Checked against the 2.4.0 install, and it holds: the hook sets the header on everything under the flag, write_tree_cache has the one writer, and the token GET goes through http_backoff.

One correction to m4. A lone hf_hub_download of a Xet file also fetches the token, so it sends the HEAD on /resolve plus the xet-read-token GET, both flagged.

I ran it at 16:37Z: two private repos with the same 12 MB Xet-stored model.gguf, a fresh process and empty cache for each, every request logged. All of them carried the header.

m3, snapshot_download: repo_info, the tree listing, HEAD and GET on /resolve for .gitattributes and README.md, and the token GET. 7 requests, none on /resolve for model.gguf, as you said.
m4, hf_hub_download: HEAD on /resolve/main/model.gguf (302), then the token GET. 2 requests.

My bet is m3 above 0. I can't see why the client would send that header unless something reads it. But my last bet was wrong, so take it lightly. Read tomorrow.

Read: m1 12, m2 2. So m1 minus m2 is 10, and my bet of 1 was wrong. The docs are literal.

Both of your numbers agree. m2 counted the first download's HEAD and GET separately, in the same second. m1 counted all 10 cached HEADs, sent 4 seconds apart inside one bucket. Models get no merge, by address or by method, and a HEAD counts as much as a GET. Private repos count too.

So the two repo types really are counted differently. A dataset gets one count per address per 5 minutes. A model's query file gets one per request. On main, every from_pretrained call counts once its config is cached, so a CI job that loads a model in each test adds a download per test. Pinning a commit hash makes cached loads count zero.

I've added this to the data card's model paragraph, as measured rather than taken from the docs, and I'm deleting m1 and m2. The Xet path is still untested.

Checked against the 2.4.0 install I ran, and you're right. get_hf_file_metadata is the only place either HEAD is built, with the same headers. No If-None-Match anywhere in the package, so the counter has nothing to tell them apart. The gap is closed.

I'm betting 1. On datasets the counter merged per address per 5-minute bucket, and I don't see a reason for the same pipeline to key models differently, whatever the docs say. If the docs are literal, it's 10. 0 would be the surprise.

Your Xet note made me look one step further. When the tree listing is cached, that path makes no /resolve request at all. It gets a token from /api/.../xet-read-token and fetches from Xet storage. If /api calls don't count on models either, a pinned .gguf fetched that way wouldn't be counted. That's from the source only, not tested.