Ground truth: Accelerated DAT
Ground truth: 6
Vision language models (VLMs) are expensive. Open-source VLMs like Qwen are easy to run for free on consumer hardware, but are also notoriously unreliable.
What's the best way to boost the accuracy of open-source vision language models? Average the outputs over multiple calls? Use another VLM judge to verify the output of the original model? Give the model a zoom tool to better inspect the image?
To find out, I ran 13 open-source VLMs for over 500 GPU hours with different test-time compute methods.
TLDR: the best way to boost a VLM's accuracy is by using another VLM from a different model family to judge the original model's answers, but you're better off just using the biggest VLM you can fit on your GPU.
I tested models across 4 different families: Gemma at E2B, E4B and 12B, Qwen3-VL at 2B, 4B and 8B, InternVL3.5 at 1B, 2B, 4B, 8B and 14B, LLaVA-1.5 at 7B and 13B. Each model was run locally through vLLM on a single RTX 3090.
To test both simple perception and visual reasoning, I evaluated every model on both CharXiv and CountBenchQA. CharXiv is a popular benchmark that tests the model's ability to logically interpret real-world scientific figures. CountBenchQA is a simple dataset where the model has to count the number of objects in an image. Despite the simplicity of the questions, smaller VLMs can surprisingly struggle at counting everyday objects.
The simplest method to increase the accuracy of VLMs is to run the same model N times and take the response that is generated most often (Maj@N). Similarly, you can ask the same question to multiple different models and take the most common answer across them for a cross-model majority vote.
Another popular technique for improving VLMs is to use a separate VLM to judge the original model's answers. Following Lu et al., I ran the VLM judge and the generation model in a loop where if the judge rejects an answer as incorrect, the generation model produces a fresh response to try again (with a maximum of 5 attempts). I tested the VLM judge under three different settings:
Finally, I also tried giving the model a tool to zoom into the image. There's surprisingly little information out there about how to run open-source models with this type of zoom tool and how much it changes the accuracy. To test this, I ran each model with a zoom tool multiple times with different maximum budgets of 2, 4 and 8 total crops and report the best accuracy. LLaVA was not trained to have multi-turn conversations with multiple images, so I did not test the zoom tool calls with it.
| Solver Model | Base Acc | Zoom Tool | Avg Cross Family judge | Maj@5 |
|---|---|---|---|---|
| Qwen3-VL-2B-Instruct | 0.54 | +4% | +5% | +9% |
| Qwen3-VL-4B-Instruct | 0.63 | -4% | +2% | +6% |
| Qwen3-VL-8B-Instruct | 0.65 | -4% | +2% | +4% |
| InternVL3.5-1B | 0.42 | -8% | +9% | +11% |
| InternVL3.5-2B | 0.52 | +3% | +5% | +10% |
| InternVL3.5-4B | 0.61 | -5% | +3% | +4% |
| InternVL3.5-8B | 0.63 | -31% | +2% | +1% |
| InternVL3.5-14B | 0.64 | -1% | +2% | +4% |
| gemma-4-E2B-it | 0.44 | +8% | +5% | +8% |
| gemma-4-E4B-it | 0.54 | +4% | +2% | +5% |
| gemma-4-12B-it | 0.77 | +1% | +0% | +1% |
| llava-1.5-7b-hf | 0.25 | NA | +26% | +6% |
| llava-1.5-13b-hf | 0.27 | NA | +26% | +12% |
| average | 0.53 ± 0.2 | -3 ± 10.4% | +7 ± 8.8% | +6 ± 3.6% |
Relative change in accuracy over the base model, averaged across CharXiv and CountBenchQA.
Small models aren't good at tool usage. Qwen and InternVL frequently mess up the format of the tool call, and even when they do it correctly, they zoom in to irrelevant parts of the image. Gemma benefits the most from the zoom tool1, but the performance gains are minimal.
In this example, InternVL3_5-8B zooms in to the incorrect region in the image. Then it hallucinates that the crop shows the correct "hopper:stand" sublpot and confidently gives the wrong answer
Question: "For the subplot labeled hopper:stand, what is the maximum episode return approximately achieved at 10h?"
Okay, let's take a closer look at the 'hopper:stand' subplot to determine the maximum episode return at around 10 hours. I'll start by examining the graph where the x-axis represents learner walltime and the y-axis shows episode return. Let me zoom in on the 'hopper:stand' subplot focusing on the area around 10 hours.
<tool_call>
{"name": "zoom", "arguments": {"box": [0.35, 0.4, 0.45, 0.6]}}
</tool_call>
The zoomed-in image shows the 'hopper:stand' subplot around the 10-hour mark. From the graph, the Distributed: 16 actors line reaches its peak near this time. The episode return appears to be around 800 based on the y-axis scaling.
Given this observation, the maximum episode return approximately achieved at 10 hours is likely close to 800.
Final Answer: \boxed{800}
Final answer: 800
Incorrect (Ground Truth: 750)
Using a VLM judge gives the highest average gain, but also has a very high variance. The small 2 billion parameter models and LLaVA benefit the most from an external judge that can easily point out their hallucinations, but the larger models don't gain much by having smaller models verify their work.
There's also a large amount of variance across different judges. Having a model from another VLM family be the judge can significantly increase the accuracy when compared to self judging or having another model of the same family judge the answer. This is supported by Tong et al. who find that VLM judges work best when their predictions are less correlated, since models that make different mistakes provide more useful signals.
Figuring out the best judge / verifier combo for a task would be very expensive if you wanted to brute force through every combination. By estimating the gain from a cheap static judge grid, we can predict what the final rejection-sampling performance should be. Lu et al. address this problem by introducing a metric called verifier gain. After determining the precision of the judge through its false positive rate and true positive rate, we can estimate how well the judge will do.
Taking the most common answer over 5 runs increases accuracy by a solid 6%. It's by far the most consistent method as well given that the increase in accuracy does not vary as much with model size like it does for the VLM judge.
If majority vote over one model's samples helps, what about a majority vote across all the models at once? Interestingly, sampling from the best and the worst models and averaging over their answers still does worse than just using Gemma-4-12B even when you average over all 13 model answers.
The whole point of using a small VLM is that it is efficient to run. Is the added cost of test-time compute even worth it? Interestingly, just taking the default response of a strong model dominates the Pareto frontier of the compute cost to accuracy tradeoff.2 In almost all situations, the best course of action is to use a bigger model rather than to use a smaller model + test-time compute.
Smaller models are not currently strong enough to reliably use a zoom tool. Using a VLM judge can boost accuracy, but make sure that it is from a different model family than the original solver model. The simplest and most consistent way to improve accuracy is to just have the same model answer multiple times. However, using the best model you can fit on your GPU is still the best bang for your buck.
I uploaded the solver, judge, rejection-sampling, self-consistency, and agentic-zoom traces to a public Hugging Face dataset. This could be useful fine-tuning data for future VLMs if anyone would like to use it. 😃
If you have any questions or comments, feel free to reach out to me on X or email!