Ginjinha: Pretraining LLMs on European Portuguese

For the past year, I've been very curious about the intersection of Large Language Models (LLMs) and the European Portuguese language. I've built evaluations, classifiers, and eventually the largest pretraining dataset in European Portuguese.

Now, that's all fine and dandy, but it raises the obvious question: Can we pretrain an LLM fully in European Portuguese? And is it any good?

Welcome to project Ginjinha.

But before we start training models, we need a harness.

A pretraining harness: NanoChat

In 2025, Andrej Karpathy released NanoChat: a small, hackable codebase for training GPT-2-style language models. It's easy to understand and modify. Exactly what I needed.

The first step was to adapt it to Ginjinha (code). NanoChat includes the three key stages of training LLMs:

• Pretraining: We train a text-completion model to predict the next token.
• Supervised fine-tuning (SFT): We further train the model to follow instructions (i.e., chat).
• Reinforcement learning (RL): We "align" the model through preference tuning or verifiable rewards (very hyped nowadays).

We could do all of these, provided we had the data. But Bagaço v2 is a pretraining dataset, so we are only interested in the first stage here. If we use Bagaço to pretrain an LLM, how good a model can we get?

To measure how good a base model is, NanoChat uses something called the CORE metric.

From CORE to PTCORE

The CORE metric is an evaluation suite designed to measure the capability of a language model on downstream tasks. For CORE, there are 22 tasks. These tasks are things like ARC, HellaSwag, and BIG-bench. Each one is made up of questions (usually multiple choice). We take each answer and measure the loss that the model assigns to it. If the correct answer gets the lowest loss, we consider the model answered correctly.

A trivial example:

Question:
What is the capital of Le Marche?

Candidate completions:
A. Camerano
   → average loss: 1.76
B. Ancona
   → average loss: 1.12

Prediction: B (lowest loss)
Correct answer: B ✓

The CORE metric is a great way to evaluate a model if you are building a general-purpose LLM (think ChatGPT).

For our use case, since we're training on European Portuguese data, we're interested in capabilities that measure knowledge of Portugal and the Portuguese language: enter PTCORE.

PTCORE is a collection of six tasks that measure the capabilities of base models across Portuguese language and culture (e.g., think CORE for Portuguese). It draws on recent work from the AMÁLIA team and a couple of other datasets I curated. Here's the full list:

And here's an example question from the SAUDADE task:

Question:
Qual dos dois acontecimentos sobre Fortios ocorreu primeiro?

Candidate completions:
A. Registo populacional de 1785 habitantes em Fortios
   → average loss: 1.76
B. Fortios foi desanexada da freguesia de São Lourenço
   → average loss: 1.12

Prediction: B (lowest loss)
Correct answer: B ✓

For each task, we can measure two scores: accuracy and a centered score. Accuracy is self-explanatory: how many questions the model got right. The centered score is a bit more interesting: 0 means random performance (for example, 33% accuracy on a multiple-choice question with three options gives 0), while 1 is a perfect score. Scores below 0 indicate worse-than-random performance. We can then aggregate the centered scores for each task, giving us a single number—the PTCORE score—for a base model.

Educational ablations

Now that we have a way of measuring how good a model is at European Portuguese, the next step was to run some experiments: How good is the Bagaço v2 dataset for pretraining language models? If we filter data by educational score, do we see a change in the capability of the base model?

If you remember, every document in the Bagaço v2 dataset has an educational score associated with it:

I don't work for a big lab and don't have an H100 GPU sitting under my desk, so I wanted to run an experiment that wouldn't cost thousands of dollars. I trained 12 small language models of around 73 million parameters each:

About $50 and five hours later, I had the results. For each filter, we get a PTCORE score and a validation BPB (bits per byte, lower is better). PTCORE is shown as a percentage, and each value is the mean ± standard deviation across three seeds. You can also expand the full task-level table to see every individual run.

Expand all D6 task scores

Two interesting findings:

1. Filtering the pretraining data for documents with higher educational scores improved the base model's performance, but only up to a certain threshold. Filtering by ≥2 performed better than filtering by ≥3.
2. The base models trained without filtering (the first row) achieved the lowest validation BPB. This likely reflects how closely the unfiltered training data matches the validation distribution.

The idea that higher-quality pretraining data creates better language models is not groundbreaking. It's shown in research such as FineWeb and FineWeb-Edu, as well as models such as Microsoft's Phi. Still, it's very interesting to see it in practice!

The best base model will need to balance both PTCORE score and validation BPB (quality AND ability to model language).

Training larger and for longer

During this work, I trained and tested a lot of small language models on the Bagaço dataset. More than I should have, to be very honest.

The two highest-scoring runs were:

• ginjinha_d8_ratio80_ptcore5_education_score_gte2 — PTCORE ≈ 0.163 (126M parameters @ 80 tokens per scaling param)
• ginjinha_d11_ratio130_ptcore5_education_score_gte1 — PTCORE ≈ 0.162 (279M parameters @ 130 tokens per scaling param)

It's interesting to see a model half the size perform just as well. Filtering the pretraining data can help smaller models close the gap. For reference, I stored all the Ginjinha runs in this repository.

Here are all 22 runs, sorted by PTCORE. Click an available model to open its weights and evaluation files.

These models can complete Portuguese text pretty well! Below are two unedited completions from the D8 ratio-40 model trained on scores ≥3, generated with a temperature of 0.8 and a top-k of 40:

Prompt: O bairro de Alvalade

O bairro de Alvalade em Lisboa, um bairro de ruas largas e estreitas, onde os lisboetas passam 2/3 do tempo que passa em Lisboa, não tem nada a ver com a sua dimensão.
A partir de um mapa, o percurso que liga Lisboa ao bairro de Alvalade passa por pequenas ruas com traçado irregular e estreitas, com alguns metros de largura, e com pequenas praças a marcarem a malha urbana dos pequenos núcleos urbanos do bairro.
O percurso que liga Alvalade ao bairro de Alvalade passa por pequenas ruas com traçado irregular e estreita, com algumas metros de largura, e com pequenas praças a marcarem a malha urbana dos pequenos núcleos urbanos do bairro.
Há

Prompt: A cidade do Porto

A cidade do Porto possui uma gastronomia única e uma arquitetura única. Desde os pratos de peixe grelhado ao bacalhau guisado e à doçaria conventual. A cidade do Porto oferece uma experiência culinária única, não se afastando muito dos restaurantes, mas convidando a uma boa esplanada enquanto se aprecia o pôr-do-sol.
Para os viajantes que procuram uma experiência gastronómica de excelência, uma viagem ao Porto é indispensável, uma vez que a cidade é muito conhecida pela sua gastronomia rica e requintada. Desde as famosas caves de Vila Nova de Gaia até às famosas caves de vinho do Porto, cada refeição, à refeição principal, é uma oportunidade para conhecer

Not AGI, but you can see how these models could become quite capable with some SFT and RL on top.

Lessons learned and next steps

This was a fun exercise. We already knew that better-quality data results in better—or at least more capable—models. But running these experiments on a tight budget taught me how structured the research needs to be. I started with a lot of YOLO-style runs until I had to stop and think: "OK, what do I want to test here?"

The Ginjinha project also showed me what an enormous advantage labs with access to compute have. If I had a single H100 for a year, I could conduct many more ablations and experiments—and do so much faster. Perhaps I should invest in one.

And even though I haven't trained on the entire Bagaço v2 dataset, I did realize that it has a major shortcoming: data quality. Approximately 50% of the documents in Bagaço v2 have an educational score of 0 (i.e., they have zero educational value). Yes, you read that right. That's not going to get us where we need to be. It might add some diversity, but we need much more high-quality data.

Yes. I'm working on it!