When should a model trust the image, the text, or both?

A photo

Reading the picture on its own

Show a model this picture. An image expert would call it a holiday: sun, sea and a beach umbrella. Nothing in the pixels is ambiguous, and an image-only classifier would be confident.

That confidence is the first thing to be wary of. A model can be completely sure and still be answering only half of the question. Real inputs arrive with words attached (a caption, a tweet, the text printed across a meme), and the words can change what the picture means.

A caption that agrees

When both modalities say the same thing

“Great weather for my day off.” The text says what the image says. Either expert alone gets this right, so trusting both equally costs nothing.

Many examples look like this, which is why a fixed fusion rule works well on clean benchmarks: when the sources agree and neither is damaged, averaging them is hard to beat. Our results come back to this later, because on clean data routing matches static fusion almost everywhere.

A caption that flips it

When the meaning lives in the combination

“Can’t wait to be back at my desk on Monday.” Alone, the text is about work; alone, the image is a holiday. Together they are sarcasm, which only a model of the interaction can see.

In hateful memes this is the hard case: a harmless picture and a harmless sentence that become hateful together. Neither half is evidence on its own.

That is why we give the model a third expert. Besides an image expert and a text expert, an interaction expert reads the two together, because when the signal sits in the pairing, neither single-modality expert can supply it.

A broken input

When one side cannot be trusted

Now the photo fails to load, or the caption is empty, or the text describes something else entirely. Only one side is reliable, yet a fixed fusion rule keeps giving the image its usual share of the vote.

The right weighting depends on the input in front of it, and it can change from one example to the next. That is what we set out to learn: a router that decides, for every input, how much each expert should count.

The rest of this page follows that question: how the router is built, how we tested it fairly, and where it helped and where it did not.

Static fusion

One recipe for every input

The simplest rule averages the experts with the same weights for every input. It cannot react to a sarcastic caption or a missing photo.

It is a strong baseline, and we treat it as the one to beat: every claim later on is a comparison against static fusion built from the same three experts, on the same frozen features, splits and seeds.

A learned gate

Our first version, V3

Our earlier version (V3) learned one gate α per input, sliding between the image and the text.

Where static fusion fixes the weights in advance, V3 learned them: a small network looked at both feature vectors and produced a single number α between 0 and 1, the image’s share, with 1 − α left for the text.

The first paper reported this ahead of an equal-weight baseline on the Hateful Memes validation split (AUROC 0.752 against 0.712).

The gate collapsed

Dynamic in name, static in practice

In our last V3 runs α settled at 0.5 for every input, with a standard deviation of about 0.002. It had learned to be static fusion.

Two things in the design pushed it there. One scalar scaled two already-normalised vectors that the next layer could simply rescale, and an entropy regulariser nudged α toward 0.5.

A gate that never moves cannot be dynamic, so the question became how to give a router a signal worth following.

And lost to the fixed rule

What a stricter look showed

A fixed α = 0.5 scored higher: 0.737 against 0.720 AUROC.

Three more problems surfaced: “cross-attention” over a single pooled token (softmax over one key is always 1), one 500-sample split used both to select the model and to report it, and fine-tuned CLIP overfitting (train AUROC 0.98, validation ≈ 0.70).

None of this is a reason to drop the idea. It is a reason to rebuild it carefully, so version 4 rebuilds both the method and the evaluation protocol.

Frozen features

Computed once, reused everywhere

CLIP ViT-L/14@336 turns the image into 144 pooled patch tokens plus a summary token, and the text into up to 77 word tokens. CLIP stays frozen and the features are cached once, so one training run takes about five minutes.

Freezing matters twice. It makes experiments cheap enough to repeat across ten seeds, and it avoids the overfitting we saw when fine-tuning CLIP on 8,500 memes, where training AUROC reached 0.98 while validation stayed near 0.70.

Three experts

Image, text, and their interaction

The image expert reads image tokens, the text expert reads word tokens, and the interaction expert runs two layers of image↔text cross-attention plus an outer product of the two summaries. Each predicts class probabilities p_k.

The first two see one modality only, so when one side fails the other is untouched. The interaction expert is the only one that can see sarcasm, a contradiction, or a meme whose punchline needs both halves.

A router

Deciding how much each expert counts

The router reads the two summary embeddings, how unsure each expert is (its entropy), and how well image and text agree (their cosine similarity). It outputs weights π over the experts, eq. (1).

Three kinds of evidence go in: what the input looks like, how sure each expert already is, and whether the two modalities agree. The weights it returns are positive and sum to one, one per expert, and are recomputed for every input.

Mix

One probability from three opinions

The prediction is the π-weighted mixture of the experts’ probabilities, eq. (2). With π = ⅓ for every input this is exactly static fusion.

Because the weights sum to one, the mixture is itself a proper probability. At the extremes it reduces to something familiar: equal weights are static fusion, and all the weight on one expert is hard selection.

Train toward reliability

Teaching the router who was right

Each expert’s loss on the true label measures how reliable it was on that sample. A softmax over the losses at temperature τ gives the target t, eq. (3), and a KL term pulls the router toward it, eq. (4).

The target is computed from the experts’ own losses and held fixed during training, so the router chases it without the experts moving to meet it.

As τ → ∞ the target is uniform: static fusion. As τ → 0 it picks the single best expert. At τ = 1 it equals the E-step responsibility of a classical mixture of experts. One parameter spans static to hard routing: that is the sense in which the framework is unified.

Clean memes: near the centre

Hateful Memes, undamaged

Each mark is one test meme, placed by its router weights; the centre is static fusion. On clean Hateful Memes the router barely leaves it: the three experts are strongly correlated and usually agree.

That is a result, not a failure. On clean data the router matches static fusion on every dataset except Harm-C, where it is significantly better, and Food-101, where it is slightly worse. When nothing is damaged, there is little for a router to route around.

Harm-C: the image wins

One modality that is reliably better

On Harm-C (COVID-19 memes) the image is consistently the more reliable modality, and the router learns it: median π_img = 0.79. It is the one clean dataset where routing beats static fusion significantly.

Nobody told the router this. It was trained only toward whichever expert was right on each sample, and it found the pattern itself, while a fixed rule would keep giving the weaker sources an equal say.

Break half the inputs

A test fixed before we looked

Before running it, we pre-registered a test: half of every split loses one modality (a quarter the image, a quarter the text). Follow the same Harm-C memes: those without an image slide toward the text corner, those without text toward the image.

Which samples are broken depends only on each sample’s ID, so every method, seed and epoch sees exactly the same damage. A missing modality is replaced by the average training token, so nothing is tuned after seeing the outcome.

Every dataset, same direction

The weights respond to the damage

On all six datasets, the weight on a modality falls when that modality is missing. Every line slopes down.

On Harm-C the effect is dramatic: with the image missing, its weight drops from 0.65 to 0.03; with the text missing, the text weight drops from 0.18 to 0.04. On Harm-P it is slight (0.34 to 0.31) but points the same way.

And it pays off, sometimes

Small, consistent, and not the whole story

Under mixed quality, routing beats static fusion significantly on FHM, Harm-C and CrisisMMD after Holm correction. On clean data, only on Harm-C, and on Food-101 it is slightly worse.

The gains are modest: +0.003 to +0.007 AUROC, ahead of static fusion in 8 or 9 of 10 seeds on those three datasets.

We do not claim it is the best method. Under mixed quality, PDF or QMF scores higher than ours on five of the six datasets (their means are over 5 seeds, ours over 10). What we can say is when routing helps, and how much.