Mechanistic interpretability · Unified multimodal models
Causal Tests of Where Task Differences Live in a Unified Multimodal Model
Unified multimodal models such as Harmon route image understanding and image generation through one visual encoder with bit-identical weights. Because the encoder never sees the question or the generation regime, its task-blindness is a property of the architecture, not a discovery — so we use it as a controlled testbed for two questions weight sharing leaves open: is the shared representation causally interchangeable between tasks, and where do task differences actually come from? Three causal results give one answer: the encoder writes an editable scene code, and task differences emerge from how each downstream consumer recovers and uses that code.
A unified model doesn't split into a "seeing" network and a "drawing" network. Both read one shared scene code. What differs is the consumer — and where the architecture injects task context.
Substituting the understanding pass's rows at a visual fact's token positions with rows from a content-flipped scene moves the answer 63–66% toward the donor at every encoder layer (random positions: 3–11%). Rows computed by the generation pass repair deleted understanding computation at 12 of 20 layers (71–99%), while wrong-content donors anti-rescue. One task can consume what the other computed.
On 98 real COCO scenes, deleting one annotated object instance while others survive lets generation recover .82 of the answer signal, vs. .54 when every instance is deleted — the one-shot understanding reader collapses to .05 under full removal. With the object entirely gone and both consumers seeing the same input, generation still retains +.48 more at every layer: the iterative consumer re-derives missing evidence from spatial backups. Yet neither consumer survives corrupted evidence (0.74 → 0.07). Robustness is a property of consumer × evidence-redundancy.
In the task-conditioned trunk, cross-pass similarity declines gradually (CKA 0.997 → 0.421). On Show-o, which mixes task context from layer 0, a causal cross-pass swap collapses at every layer even where CKA reads 0.995 — similarity is not interchangeability. The mechanism: the passes share one representational geometry in rotated bases. A single frozen orthogonal map, fit on held-out natural images, causally repairs 99/93/75% of the collapse at layers 0/2/4, while wrong-target rotations and low-rank maps recover ≈0. Only deep, where CKA itself collapses, does the geometry diverge.
@article{zhang2026onescenecode,
title = {One Scene Code, Two Consumers: Causal Tests of Where
Task Differences Live in a Unified Multimodal Model},
author = {Zhang, Tong and Peng, Yun and Xie, Tao},
year = {2026}
}