AtomicChat/Ling-3.0-flash-Sante-GGUF

How to Run Ling 3.0 flash Sante Locally

Built from inclusionAI's original weights with our own importance matrix. The calibration corpora behind our builds are public.

• You can now run Ling in Atomic Chat.
• See our quantization analysis below for measurements and instructions.

---

Pick a file

Every number below is measured, not estimated. How we measured it is at the bottom, and the raw logs are in the metrics repo so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the original weights. Lower is better, and zero means identical. top-1 is how often it picks the same next word the original would have picked.

File	Size	KL divergence	top-1
AD-Q6_K	110.8 GB	0.00748	98.00%
AD-IQ4_XS	78.1 GB	0.01497	96.66%
AD-IQ3_S	61.1 GB	0.03525	94.47%
AD-IQ2_M	52.4 GB	0.06086	92.45%
AD-IQ1_S	35.7 GB	0.22337	84.75%

AD- marks an Atomic Dynamic layout. The name says what the two largest tensor groups got: AD-<ffn_down>-<ffn_up>, collapsed to one name when both match. Nothing is named after a type it does not contain.

Which one fits your card

The file has to fit, and so does the context. This model keeps 560 KB of attention cache per token, which is 4.4 GB at 8k context and 17.5 GB at 32k. Budget for both.

Your card	File	Leaves room for
48 GB	AD-IQ1_S	around 24k context
64 GB	AD-IQ3_S	around 8k context
80 GB	AD-IQ4_XS	around 8k context
128 GB	AD-Q6_K	around 40k context
256 GB and up	AD-Q6_K	full context

If you are choosing between two neighbouring files, take the larger one. The steps between them cost one or two gigabytes and buy noticeably fewer wrong words.

Running it

llama-server -m Ling-3.0-flash-Sante-AD-IQ4_XS.gguf -ngl 99 -c 8192

The chat template is inside every file, llama.cpp applies it on its own.

The model ships a multi token prediction head. It is inside every file here and needs no extra download:

llama-cli -m Ling-3.0-flash-Sante-AD-IQ4_XS.gguf --spec-type draft-mtp -ngl 99 -c 8192

Speed

How these compare to other builds

What we found while building these

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe ling3-moe. It is 4,685,467 tokens across 2,726 documents:

Part	Share
agentic and tool use	23.6%
multilingual	19.4%
code	17.1%
reasoning	12.8%
long context	11.4%
vocabulary sweep	6.7%
structured data	4.6%
graphics	4.3%

Every conversation is rendered through this model's own chat template, so the chat markup appears as the single tokens the model actually sees.

If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out set is 3.1334 plus or minus 0.01199.

Held-out text: eval_neutral from our calibration dataset, never used for calibration. 144 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement.

Hardware: 8x A100-SXM4-40GB.

The raw logits of the reference are published in the metrics repo, split into parts because of the file size limit. With them you can measure your own build against exactly the same point we did:

cat base-neutral.kld.*.part > base-neutral.kld

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, in one run:

llama-imatrix -m Ling-3.0-flash-Sante-BF16.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special

Then, for AD-IQ4_XS:

llama-quantize \
  --imatrix imatrix.gguf \
  --tensor-type '^blk\.42\.=q8_0' \
  --tensor-type '^blk\.(2|3|39|40|41)\.ffn_down_exps\.weight$=q5_k' \
  --tensor-type '^blk\.(2|3|39|40|41)\.ffn_(gate|up)_exps\.weight$=iq4_xs' \
  --tensor-type '^blk\.\d+\.ffn_down_exps\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_(gate|up)_exps\.weight$=iq4_xs' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)_shexp\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.attn_\w+\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ssm_\w+\.weight$=q8_0' \
  --output-tensor-type \
  f16 \
  --token-embedding-type \
  q8_0 \
  Ling-3.0-flash-Sante-BF16.gguf Ling-3.0-flash-Sante-AD-IQ4_XS.gguf IQ4_XS

The rules for every other file are in ladder/ in the metrics repo, next to its quantization log. The first rule that matches a tensor wins.

Model details

42 layers, hidden size 2560, feed-forward size 6144, vocabulary 157,184, context 262,144. One multi token prediction head.

Needs a recent llama.cpp with bailingmoe3 architecture support. Built and tested with llama.cpp b11571 (097cf33d9).