QACD: Corrupting Exactly What the Question Asks

In an earlier post I walked through Self-Aug, a contrastive decoding method that lets a vision-language model pick its own image corruption to reduce hallucination. This post is about Query-Aware Contrastive Decoding (QACD), a method that pushes that idea a good deal further.

The one-line version: instead of letting the model pick which corruption to apply and then applying it to the whole image, QACD lets the model plan the whole thing. What to corrupt, how hard, and crucially where, all by looking at the actual image and the actual question.

The amateur branch is everything

Contrastive decoding runs a model twice on the same question: once on the clean image (the "expert") and once on a corrupted version (the "amateur"), then subtracts the amateur's output distribution from the expert's. Whatever survives is the part of the answer that genuinely depends on looking at the image, rather than on the language model's priors about what a plausible answer sounds like.

The model runs on the clean image and a distorted copy. On the beach scene it hallucinates a "surfboard"; because the distorted run hallucinates it too, subtracting the two distributions cancels the surfboard out and keeps the objects that are really there.

Everything hinges on how you build that amateur. The original method, VCD, used one global Gaussian-noise corruption for every question: blunt, and blind to what's being asked. Self-Aug improved on this by letting the model choose the corruption type from a menu. But that choice is made from the question text alone, the operation is still applied globally, and always at a fixed strength. The model never actually looks at the image when deciding how to sabotage it.

Two principles for a corruption worth subtracting

Before any mechanism, I found it useful to pin down what actually makes a corruption good for contrastive decoding. Two principles.

What to corrupt: anti-correlation. Because you subtract the amateur from the expert, the corrupted run must be pushed away from the correct answer. If the corruption accidentally produces something that looks like a plausible ground-truth label, you end up subtracting correct signal instead of hallucinated signal.

Where to corrupt: localization. Vanilla VCD nukes the whole image, so the model loses all its visual footing and falls back on language priors about the question in general. If you instead confine the corruption to the query-relevant region and leave the surroundings intact, the rest of the scene stays available as a reference, and the amateur becomes a targeted signal about the specific evidence the question depends on, not about diffuse global degradation.

The pipeline: the model as a corruption planner

QACD runs in three stages, all on the same 7B model, with no external detectors and no training.

One planner reads the image and the query, then emits two things: an adversarial recipe (operation + intensity) and, from its own cross-attention, a binary mask. The operation is applied only inside the mask, and the region-corrupted image becomes the amateur branch of the contrastive decode.

Stage 1: Plan the recipe. The model is given the image, the question, and an adversarial meta-prompt: you are a red-team adversary; propose the single edit that would most mislead another model trying to answer this question. (The GAN-flavored framing is just prompt engineering; there's no adversarial training here.) It emits a small, parseable recipe:

TARGET:    the umbrella
OPERATION: invert
INTENSITY: 2

One greedy forward pass. A parser pulls out the fields; if it ever fails, we fall back to a fixed recipe.

Stage 2: Ground the region in attention. Here's the key move. LVLMs are bad at describing where something is ("top-left, roughly a third in…"), so QACD never asks. Instead it reads the model's own cross-attention from the generated TARGET tokens to the image patches during that same forward pass, averages across heads at a mid-layer, and denoises the heatmap into a clean binary mask. The division of labor plays to each component's strength: language decides the semantic what, attention reveals the spatial where.

Stage 3: Corrupt the region, then decode. The planned operation is applied only inside the mask and blended back with the untouched surroundings (a sub-millisecond pixel-space edit), and the result becomes the amateur branch of the contrastive decode.

The operation menu is curated to respect anti-correlation: five hide-the-object degraders (blur, downsample, noise, obscure, r-noise) and two color flippers (desat, invert). Masking and mean-fill are deliberately excluded.

The payoff: the mask moves with the question

My favorite result is qualitative. Take one image, an aerial view of a snowy slope with a tiny snowboarder, and ask two different questions.

"Is there a snowboard?" The mask (center, red patches) concentrates on the rider, and blur softens exactly that region (right).

"Is there a dining table?" (there isn't). On the same image the mask shifts to a different, plausible surface, and downsampling hits there instead, nowhere near the snowboarder.

Same pixels, different masks, driven purely by the question. That's the property a salience-based method fundamentally can't provide: salience gives you one "most important region per image," but QACD gives you a region per query. And on the negative question the final answer stays correct: the corrupted amateur also sees no dining table, so the subtraction doesn't flip the decision.

Does it move the numbers?

I evaluated on POPE (the object-existence benchmark) against VCD and Self-Aug, using LLaVA-1.5-7B for both the planner and the target model.

The VCD, Self-Aug, QACD progression is monotone on both datasets. QACD edges ahead while using roughly an order of magnitude fewer planner tokens than Self-Aug: the reported numbers use a 64-token planner with no verbalized rationale, whereas Self-Aug's selection step writes out a full chain of thought. And the pipeline turned out to be genuinely robust. Across all 18,000 inferences, the planner produced a valid recipe every single time and attention grounding produced a usable mask every single time, a 0% fallback rate.

Where it's honest about its limits

The gains are modest, and I want to be straight about why.

• Compute. Everything ran on a single L4 GPU on a tight budget, so these are single-seed results on POPE only: indicative, not statistically conclusive.
• POPE is friendly to the old method. On yes/no existence questions, global noise is already a strong amateur, since the model loses almost all its visual footing anyway. Localized corruption deliberately leaves most of the image intact, so QACD's headroom here is narrow. I'd expect the advantage to widen on tasks where the queried evidence is genuinely local: attributes, counting, spatial relations.
• Anti-correlation caps aggressiveness. On existence questions, the principle rules out the single strongest attack (masking the object), leaving only the softer degraders.
• The planner bounds everything. QACD delegates both the operation choice and the region to a 7B model that is, frankly, a weak reasoner, and it occasionally picks an operation poorly suited to the question. A stronger backbone is the most direct lever, and a with-reasoning planner variant I prototyped looked noticeably better, though I haven't fully evaluated it yet.

Takeaway

QACD is training-free, adds no external models, and costs one planner pass plus a cheap attention read per question. It reframes the vision-language model as its own query-conditional corruption planner, and it makes the anti-correlation principle (don't build an "attack" that secretly agrees with the truth) explicit, where prior work only followed it by accident. The absolute numbers on POPE are small, but the mechanism it validates, localized per-query masks that a salience method can't produce, is the part I think generalizes.

QACD builds directly on the VCD and Self-Aug line of contrastive decoding. See my earlier post on Self-Aug for the background.