AMALIA vs. smaller reasoning models

A few weeks ago there was a bit of a scandal back home. Everyone was mad that the government had taken so long to grade this year's high school exams. At the time, I saw a weird post on LinkedIn saying we should have AMALIA grade all those exams because that would be faster.

Let's be serious. Using an LLM to grade high school exams is a horrible idea. But it made me think for a bit. How good is AMALIA at answering those questions? What the author might not know is that the AMALIA team also thought of this. PT Exams is one of the benchmarks the team created to test exactly that. It contains 1,819 questions taken from high school exams, formatted as multiple-choice questions.

The question stands. How good is AMALIA at doing this? I was not particularly interested in comparing it with much bigger/closed models. I was more interested in understanding how smaller models perform vs. AMALIA. The results are interesting. In the following interactive chart, you can see how different models perform.

If you navigate to the PT Exams benchmark, you'll notice that Qwen3.5 2B actually beats AMALIA. This is surprising. That model has a fourth(!) of AMALIA's parameters/size.

If you know a thing or two about LLMs, you'll probably call bullshit on this right away. Yes, the other models are smaller, but they're also reasoning models (most of them), while AMALIA is not. They use something called test-time scaling to improve their performance. This basically means that they let the LLM output more tokens when it's time to answer the question, leading to better performance.

You'll say it's a little unfair to compare AMALIA with these. They come from big labs, from NVIDIA to Alibaba's Qwen team. AMALIA has a smaller team, a smaller budget, and admittedly a narrower goal. I agree. Still, it's interesting to see much smaller models outperform it even on Portuguese-focused tasks.

But reasoning is not free. Models produce many more tokens before reaching an answer. AMALIA DPO averaged about 374 output tokens per question while LFM2.5 used 1,733 (about 4.6x more). Qwen3.5 2B used 6,954 tokens per question, nearly 19x more(!). Nobody knows the exact recipe of models like LFM and Qwen (they are open-weight, not open source). But it will likely come down to extensive pretraining (almost overtraining) and some advanced post-training recipe. We can only speculate.

What's the takeaway? In practice, if you work at a law office and are interested in a small model that can help you respond to legal questions, you might be better off starting with Qwen3.5 4B than AMALIA (select LegalBench PT above). It's thanks to the AMALIA team and their great work that we now have something to judge these models against.

Still, I think this tells us something else that is interesting for the future of AMALIA. Could we have a much smaller model that everyone could run on their own hardware easily, but that still outperforms smaller models? Should the next AMALIA be a reasoning model?

Note: All the evaluations and code are open in this repo. I adapted some of the evaluations from AMALIA-LM-eval.

Acknowledgement: Thanks to Gonçalo for providing compute to make this happen!