Qwen/Qwen3.5-4B Brain Atlas — A 4B Professional That Wants You to Know It Tried Hard

This is not a benchmark. I ran the same GWIQ-style activation census on the 4B Qwen3.5 model. Now my pick for most likely to receive an employee or the year award

model: Qwen/Qwen3.5-4B
atlas type: activation census + Sub-Zero brain atlas + OV-circuit SVD + logit lens + compliance/behavior contrast
corpus: 8,965 prompts
layers: 32
attention layers: 3, 7, 11, 15, 19, 23, 27, 31
linear-attention / MLP layers: literally everything else
sacred (fully probed) layers: all 32
datasets: juiceb0xc0de/qwen3.5-4b-atlas

What this is

Qwen3.5-4B keeps a sparse attention model with full multi-head attention only on one layer in four, and linear-attention / MLP blocks everywhere else. The question is whether doubling the parameters deepens the same circuits or just makes them louder.

Short answer: both. The 4B is louder, but it is also more fragile in one specific place. There is a landmine at layer 0 that the 2B does not have, and the late MLP gates have become serious vocabulary routers.

What was run

• Activation census over 8,965 prompts.
• Per-layer feature taxonomy for every component.
• Per-head analysis on the 8 full attention layers.
• OV-circuit SVD (W_V @ W_O) on every head.
• Logit-lens pass to see which internal directions predict output tokens.
• Coactivation and code-analysis passes.
• Compliance/behavior contrast pass.
• Sub-Zero surgery pass with capability fence across code, math, reasoning, factual, and multilingual.

The shape of the thing

What the numbers suggest

Features are broader but more dead than the 2B

The taxonomy shifts toward partial_shared and picks up a small specific tail.

Compared to the 2B, the 4B has fewer universally-active all_shared directions and more partial_shared and non_activated ones. That reads as the model learning narrower context detectors but also leaving more directions idle. The specific_* tail is still tiny as a fraction, but it now includes real categories: creative_writing (82), ml_ai (50), introspection (45), tool_use (43), core_technical (37).

Attention is still mostly an induction engine, but the final layer compresses

OV-circuit metrics show strong induction across all attention layers. The top heads are brutal:

Layer 31 is different. Its average induction score is still high (0.86), but its OV spectral concentration jumps to 0.077 and effective rank drops to 60.3. The final attention layer has compressed its value-output path into a tighter, more focused circuit. My read is that the model parks a distilled copy-paste machine right at the exit, then lets the gates handle the broader vocabulary routing.

MLP gates dominate the logit lens even harder

The strongest output-vocabulary predictors are almost all gate features. Late layers run the show:

In the 2B the logit-lens peak was 589 at layer 15. Here it is 713 at layer 23, and the runner-up is at the final layer. The MLP gating path is doing most of the output vocabulary routing, and the peak moves deeper as the model grows.

Code directions are extremely selective

Code-analysis labels are heavily selective across the board, especially in the MLP family:

The 2B also walled code off, but the 4B is even cleaner about it. Whatever training pressure separated code from chat is still present and stronger.

Coactivation is still business-first, but the bucket map is more diverse

Business is still the largest labeled bucket, but introspection and ml_ai are close behind. The 4B is less monomaniacally corporate than the 2B was. The uncategorized share is high, which means a lot of feature pairs coactivate across multiple buckets rather than locking to one topic.

Compliance/behavior separation is even stronger and still silent

The top compliance/behavior features have absurd F-stats, and they are still sparse switches:

Layer 26 up feature 8754 and layer 26 mlp feature 8754 are the same index in different components, which means they are two views of the same direction. One view leans authentic, the other leans corporate. That is a clean style separator that the Sub-Zero pass should be looking at.

Surgical headroom is good unless you touch layer 0

197 Sub-Zero tests per domain, 985 rows total. 169 pass the capability fence per domain, an 85.8% pass rate. Average damage is low (~0.08 nats/token). But the worst damage is catastrophic:

Removing the top compliance/behavior direction at layer 0 makes the model assign roughly e^10.6 (~40,000×) less probability to the correct next token on average in multilingual. That is not a data error; it is a real entanglement between a style direction and core capability machinery. The fence correctly rejects it.

Layer 1 has the next-worst linattn_out_proj axis, with damage up to 1.93. After that, the network is forgiving.

v heads carry the cleanest signal

Per-head F-stats on the 8 attention layers:

Same pattern as most dense models: value vectors store content, query/key machinery routes. The k heads are weakest, which is consistent with the induction-heavy attention signature — keys are doing positional/contextual matching, not semantic storage.

The classifier is rock-solid everywhere

Sub-Zero classifier accuracy is 0.9688–1.000 across all 32 layers. The style boundary is linearly separable at every depth, which is why the compliance/behavior F-stats are so high.

The stuff I deliberately skipped

• I did not try to interpret the linear-attention SSM-style recurrent state directly. The probed components tokenize language cleanly, but the internal state updates between tokens need different tools.
• SAE features were not extracted for this run.

Caveats

• This is a 4B sparse-attention model. Some patterns are capacity-limited or architecture-limited, not deliberate design choices.
• The layer 0 linattn_out_proj damage is a real cross-entropy loss delta in nats/token. The magnitude reflects genuine catastrophic entanglement, not a bug.
• I do not have a base-vs-instruct split here. If this is the instruct checkpoint, some findings are post-training artifacts.
• Sub-Zero measures compliance/behavior candidate axes, not every load-bearing direction in the model.

Bottom line

Qwen3.5-4B is a bigger, more capable version of the 2B office worker. It keeps code in a separate room, routes most vocabulary decisions through late MLP gates, and runs a compressed induction engine at the final attention layer. The business-professional subnetwork is still there but less dominant. And it has one serious surgical landmine at layer 0: an early linear-attention output projection that you should not touch.

The 4B tells the same story as the 2B, just with bigger numbers and one new catastrophic direction. Scale here does not make the model more mysterious. It makes the interesting parts more visible and the dangerous parts more dangerous.