Duarte O.Carmo

Retrospectiva #6

2026-04-03T00:00:00+02:00

Well, now would you look at that. Article 100 on this website. Ten years of writing on this small corner of the web. I'm writing this month's newsletter from my favourite place: the airplane.

Happy Easter if you celebrate. With clients spread across time zones, I didn't really get the time. Next week I'll be at AI Engineer London. Email me if you're coming too!

I'm experimenting with a slightly different format this week.

Using

Pi.dev: After watching this talk on how some people use Pi, I decided to (re-)double down. It's become my main driver. I've created a nice collection of skills (some private), extensions, and even ported my favorite theme to it. An agent harness is a bit like an editor: it needs to be yours. I want to /preview a longer agent plan in a nice web page. I want to answer questions in a nice interface with /answer, I want the theme to adapt to my macOS system theme. And, most of all, I want to be able to use it with ANY model. Even local ones.

Codex: Some things need my full attention. Others don't. When it's a quick fix, a breaking test, a simpler analysis, or something I can run in parallel, Codex has been my go-to. You can preview diffs, open in a terminal, create PRs, all from the same place. I tend to throw at it the smaller parallel things that are not "high risk".

Hevy: I continue to run. Not as much as I'd like. As you get older, weight lifting and strength training become increasingly important. After freestyling my gym routine for the past years, I wanted something a bit more sophisticated. After testing 3-4 apps, I landed on Hevy. It's paid but not expensive.

Snapseed: My inspiration to take photos comes and goes. But I need an editing app that makes the ugly Danish sky just a tad prettier. The previous version of Snapseed looked largely abandoned so I resorted to Darkroom. Now that everything is a subscription everywhere I started looking again. The new Snapseed app is excellent.

Reading

Empire of AI - Karen Hao: I tested a new reading routine this month: One chapter a day. Ended up finishing this one. The book has some interesting tidbits about the history of OpenAI and how it all came to be. It also has some slightly deeper commentary on the ethics of AI models and infrastructure build outs. Unfortunately, it had a bit too much TMZ-style tech drama I don't really find interesting.

Inference Engineering - Philip Kiely: I picked this one up on my Palma. Even though I just started, I can already recommend it. As I work with larger models, datasets, and infrastructure - it's a great field guide.

"If the cluster is not on fire, you are not using it enough"

An unsolicited guide to good research - Eugene Vinitsky: A great deck on how to do great ML research, and how to find interesting problems. I liked the framing: pragmatic and to the point.

How Kimi, Cursor, and Chrome Train Agentic Models with RL - Phil Schmid: An incredible roundup of the reinforcement learning techniques some of the larger players are using to train models.

Listening

The Roving - Bonny Light Horseman: What can I say? After clicking "don't recommend" on those Ruff songs on my Discover Weekly, sometimes Spotify eventually makes you discover something new and interesting.

Reverie - Jule: I managed to tone down meetings and put some nice focused stretches of work this month. And a good SoundCloud mix is essential to keep me company.

Confronting the CEO of the AI Company that impersonated me - Decoder: Long-term fan of the Vergecast. I love Nilay's interview style. I do think Shishir held his ground pretty well. An interesting discussion, if we set aside all the drama.

The Handyman of High Art - Rich Roll: I'm a big fan of Tom Sachs. The NY-based artist that inspired much of Casey Neistat's brutalist aesthetic. And I'm a sucker for brutalism. An interesting look at his life and work.

Watching

Everything We Got Wrong About Research-Plan-Implement: Loved this one. On how to tackle codebases with agents. Some interesting mentions of: "Hey - we let the agent rip, and actually regretted it. Then we went back, and thought about how to do it properly". Agents are likely to eat software, but we are still learning how to deal with it.

Living in the Matrix - Casey Neistat: From the master, extremely relatable. I also feel like chasing the green quadrant every time. But maybe the other 3 are a necessary evil.

The flight attendant is about to come by and scream at me for not closing my laptop. See you next month!

An opinionated running dashboard

2026-03-25T00:00:00+01:00

As you get older, life becomes complicated. Not in a bad way. There’s just more going on. We don’t all want to run marathons. Maybe you want to run a parkrun. Maybe you want to gradually increase your volume. Maybe you don't want to run at all. Whatever your running goal is, you should be able to plan for it.

In the new version of Tanda Runner you can plan for whatever your running goal is. I redesigned all the parts I wasn't a fan of. The result is a redesigned chat interface with an agent that knows about all your runs, and, my favourite: a fully automated - but personalized - running plan generator.

The plan generator analyzes your past activity, understands your running habits, and builds a completely customized running plan just for you. You can take this and export it to your calendar app of choice.

The technology choices evolved a bit, but not too much. I’m still on my beloved Django + htmx stack. The LLM models are served through OpenRouter, and tracing is sent automatically to W&B Weave. The agent itself is built with PydanticAI, a framework I’ve enjoyed using more and more.

There aren’t enough niche running tools out there. This one is mine.

See you out there, runner.

Portuguese variety identification: The bitter lesson

2026-03-09T00:00:00+01:00

Given some text in Portuguese, how easy is it to determine if it's from Brazil or Portugal¹? For native speakers, this is pretty easy – it's almost a feeling. But for machines: not so much.

This might seem like a useless problem at first. But in the age of language models and some of my research, I've seen this problem more than once.

When I contributed Portuguese datasets to EuroEval, I had a hard time finding exclusively European Portuguese data. Most of the Portuguese on the Internet is obviously of the Brazilian variant. When I created Bagaço, a pretraining dataset for European Portuguese, the most obvious solution was to filter data for .pt domains. This works as a proxy, but is far from ideal.

It stuck in my mind: If we want to build good European Portuguese models we need robust ways of detecting that language variant in the wild.

In Enhancing Portuguese Variety Identification with Cross-Domain Approaches the authors take a stab at the problem. They fine-tune BERTimbau to classify text into PT-PT or PT-BR. The key idea: by hiding specific types of words (places, names, etc.) the model is forced to learn the structural/grammatical differences between the languages. It works well, but there's a problem: It doesn't scale!

Running a BERT based model to filter pre-training data (at internet scale) could take days even with dedicated hardware. I needed something as good as their model but that I could run fast for hundreds of Gigabytes of text. Most pre-training datasets use something like FastText or variants to filter data by language. It scales really well² and doesn't need any type of dedicated hardware.

After some hiccups (and a lot of tokens) I finally managed to reproduce the results from the paper. Once that was done, I started training a series of FastText models on their same dataset. The results were mediocre at best. Different configurations of the data, some oversampling, parameter tuning, nothing got close to the baseline performance (theirs).

Then I thought: what if I just need more data?

After some digging, I found this dataset from Fabio Bastos - he basically did the exact domain level filtering as Bagaço, but for a much larger dataset: CulturaX. Separating it into Portuguese and Brazilian sources: exactly what I need to train a model. I started training classifiers, and the more data I trained on, the better performance I got. Once I trained on ~6M rows lo and behold: similar performance but 10x faster.

Actually, the quantized version of the model I trained packs a whole lot of punch but in only 70MB! The quantized model is so small it runs in your browser:

As a sanity check, I ran the model on a subset of FineWeb2 that had been filtered for Portuguese. Here's a snippet of the output:

Loading model...
Downloading parquet...Loaded 33,846 rows
Classifying with threshold=0.7...
Classified 33,846 rows in 30.1s (1,125 rows/sec)
Score distribution:
mean=0.320  median=0.176
Total: 33,846 → PT-PT: 6,102 (18.0% kept)
Random PT-PT URLs:
- https://blogtailors.com/4997524.html
- https://www.bodyboardcenter.com/pt/apparel/734-sen-no-sen-not-diet-atoll-polo
- https://abapinho.com/2017/09/plantuml-finalmente-o-uml-da-para-usar/
- https://www.sabado.pt/portugal/detalhe/como-jose-veiga-denunciou-rui-rangel
- http://www.poadvogados.pt/Areas/InsolvRecoverE/?MOBILE=1
- https://www.viralagenda.com/pt/events/926945/palestra-intervir-com-criancas-e-jovens-em-risco-como-actuar
- http://confessionsfashiongirl.blogspot.com/2009/11/coisinhas-sem-as-quais-nao-podia-viver_08.html
- https://www.powrenism.com/forum/welcome-to-the-forum/exercice-dorsaux-sans-materiel-myogen-dianabol
- https://topbinamvf.web.app/berezny38567def/forex-seminbrio-manchester-1780.html
- https://foreveryoung.sapo.pt/rustic-chicken-ja-conhece-a-nova-proposta-da-mcdonalds/
- https://aralumiar.wordpress.com/2007/01/
- http://iclub.pt/video-do-4o-iclub-dinner/
- http://www.hotfrog.pt/empresa/lisboa/odivelas/avarias-ao-domicilio
- https://www.rowenta.pt/eficiencia-energetica-by-rowenta
- https://codimagem.com/2023/05/12/giros-gr%C3%A1tis-eletr%C3%B4nicos/
- http://apracas.pt/article/list/1/noticias/
- https://www.noticiasaominuto.com/politica/864278/cgtp-diz-que-greve-mostrou-que-autoeuropa-nao-vive-acima-da-lei
- http://nave-azul.blogspot.com/2010/11/portal-101110.html
- http://gbtrabalhoshubj.ischadia.info/mensagem-de-fernando-pessoa-terceira-parte-o-encoberto.html

As you can see, lots of these would never pop up if we just filtered for .pt domains. Also, the percentage kept aligns well at ~18% of text in the pt split being European Portuguese.

There's a parallel to LLMs in this post. Unlike the original paper, we didn't do any fancy delexicalization, we didn't even curate a pristine mix of data from different sources. We threw 6M rows of web data at the problem, and the problem got fixed. This is a known lesson in the world of computing: it's called the Bitter Lesson.

The model and code are available on GitHub and Hugging Face.

Or somewhere else! ↩
So well that forced Claude into building a Rust implementation for it. ↩

Retrospectiva #5

2026-03-01T00:00:00+01:00

Happy February! Just like that, we are close to wrapping up the first quarter of 2026. After quite some time roaming around, we finally flew back home to Copenhagen. It's cold, windy, and grey, but it's also calm, organized, and cozy. Most of all: it's home.

Another good thing about Denmark is that it's one of the flattest countries in Europe. Perfect for increasing weekly running volume. It's cold, but it's almost time to ditch the running tights! Just a couple months left.

Using

We've been going with Allegra pretty much everywhere. From small walks, to airplanes, to high speed trains. We've been lucky - she's really chill, and sleeps most of the time. But one of our best purchases was the Ergo Baby Omni 360. Vitto managed to find it second-hand at a nice price. But it's been absolutely great to take her everywhere. And it should last us many more months. A strong recommendation for parents out there.

Today is the day I canceled copilot.https://t.co/OPUIEX5eIe
— Duarte (@duarteocarmo) February 27, 2026

As for tech, two interesting things have been changing in my workflow (which is still unstable). The first is my increasing usage of coding agents for everyday things. For things that I find myself repeating more and more I've been using agent skills. Still bounce around different agent CLIs quite a bit, but I can largely reuse them across tools. For the past month, I've been using a mix of the Codex app, and the shitty-but-not-so-shitty coding agent Pi. The second interesting thing is the increasing capability of local LLMs. For example, I completely replaced my GitHub Copilot subscription as well, with my own fork of llama.vim, called llama.lua.

Reading

Two long read recommendations for this month. The first is The Illustrated AlphaFold by Elana P. Simon. She does a great illustrated rundown of the key inner workings of the famous algorithm from Google. Another recommendation is the latest article from Raschka: A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026. It's funny how LLM architectures have changed - but really, not all that much.

I also recommend the latest bomb from Anthropic accusing almost every Chinese lab of calling their API to distil their own models. Isn't it funny how large labs train their models on the entire Internet, but are then very much protective when someone starts doing the same with their API? In any case - didn't Anthropic do the exact same thing, but with book authors?

Pretty much finished with LLMOps from Abi Aryan. Interesting read to consolidate some of the ideas from everything I've been seeing in the field. This month I've picked up Essentialism from Greg McKeown, and The Art of Doing Science and Engineering from Richard Hamming. Not sure which one will stick yet.

Listening

I have three podcast recommendations for this month. Unsurprisingly - they are all very much about LLMs, AI, and the future of technology. But I'm sure you were already expecting that.

The Evolution of Reasoning in Small Language Models with Yejin Choi (TWIML AI Podcast): I'm a big believer in the increasing power of small language models.
The third golden age of software engineering (The Pragmatic Engineer): How the world of Software is changing, and what to focus on.
Jared Sleeper on Which Software Companies Will Survive the “SaaSpocalypse” (Odd Lots): Software stocks have been dipping, why?

Watching

Every year, I try to completely avoid any sort of news related to Formula 1. I don't watch races during the year, I block out everything I possibly can about it. Why? Because I'm always waiting for the next season of Drive to Survive. You should too. Even if you're not a fan of cars (I've never been). But I'm a fan of competitive sports, and this show captures that essence flawlessly.

That's it for the February edition of Retrospectiva, see you next month!

Bagaço: A pretraining dataset for European Portuguese

2026-02-23T00:00:00+01:00

Let's say your goal is to train a Large Language Model only on European Portuguese. Where do you start? What datasets are out there? What websites are being scraped for the large black box? Bagaço - named after the popular Portuguese moonshine - is a small step in that direction.

In June '25, the Hugging Face team released FineWeb2. FineWeb2 succeeded FineWeb, a massive cleaned up dataset of the entire internet. Think millions of web pages, specifically gathered to train Large Language Models. FineWeb2 expanded FineWeb, by adding documents in hundreds of other languages. 199 million of those are in Portuguese, but how many of them are really from Portugal, or written in European Portuguese? That's how Bagaço was born.

Bagaço keeps only the 16 million documents that match domains from Portugal (e.g., .pt). But it doesn't stop there. The goal of Bagaço is also to tell us a bit more about the data. I built two classifiers: the first classifies every document into one of nine categories: Society, Arts, Business, Science, Sports, Lifestyle, Health, Games, News. The second attributes an educational score to each document (from 0 to 5) this gives an educational "value" to each web page. Very inspired by the FineWeb-Edu work from the HF team. These were trained in a similar fashion: an LLM to annotate a large sample (with Gemini and Qwen), and a balanced Logistic Regression applied on e5 embeddings (more info on those here).

It's an interesting subset to analyze. The two most popular domains are desporto.sapo.pt and record.pt: sports news websites (I don't think I expected anything different from Portugal). The dataset is nicely distributed across categories, but something's a bit rotten. The average educational score of the dataset is only 0.85 out of 5 (!).

With FineWeb-Edu, the Hugging Face team showed that if they filtered a dataset for educational scores above 3.0, the downstream LLMs had dramatically better performance. Well, if we filter Bagaço by the same criteria, I don't think we would have much left (around 1M documents, that's it). If you're interested in more stats and facts, here's a Marimo notebook with an analysis. (Here are several scripts to create the dataset, train/run the classifiers, or to make Bagaço into an easily queryable DuckDB database)

What if we just ignored the fact that the educational score for Bagaço is pretty low and pretrained an LLM on it? (For the record, I did - it wasn't good). Bagaço has something like ~7 Billion words, something like ~9 Billion tokens of text. According to Chinchilla (~20 tokens per parameter), we could train an LLM of ~450 million parameters. Not an Opus competitor by any means - could still be an interesting model.

You see, there's still much to do in this space. Most high quality data ablations have been focused on the English language. And sure – there's a lot of English out there – but there's surely a lot of European Portuguese data out there too! We just have to find it! We probably can't get away with just filtering a dataset by .pt domains. We might need something more sophisticated. But data is not the only bottleneck, we also need better evaluations for European Portuguese.

Bagaço is only the first try. The potential is huge! Time to get to work.

Retrospectiva #4

2026-01-28T00:00:00+01:00

Well—that was quick. Just like that, the first month of the year is gone. Winter blues would normally peak around this time. But not this year. We're spending time with family in my favorite place on Earth, somewhere along the Adriatic coast of Italy, in the Marche region.

It's hard to give into winter blues when you get to run every day around the beautiful hills of the Conero Regional Park. If you read this, don't tell anyone please: let them visit the usual suspects instead.

Here, life moves slow. You enjoy things. You savor things. You talk loud. You socialize. You live for 100 years. It's Italy.

Using

I would be lying if I told you I haven't been using a bunch of agents. I've been using LLM based tools for the better part of 2025. But something clicked towards the end, and I haven't looked back since. As for my post on policing agents, my fear is still the same. That quantity and quality are inversely proportional. All I can do is find out ways of keeping quality high. It's nice to fire them off to do work I'm not interested in doing. Like redesigning this website for the Nth time. Or correcting typos in all my blog posts since the beginning of time.

On another note. We always travel light, even with a newborn. This means I have to pack just enough things - especially running gear. The FuelCell Rebel v5 have been a pleasant surprise. I'm not loyal to brands but this model won me over. They slip where it's muddy though, so beware.

Reading

Read more short form than long form this month. I continue going through the fabulous long form writing from the Huggingface team. Lots of small tips and tricks, and scars on how to train Large Language models. Elon is a divisive character, but one thing I do appreciate is the opportunity of reading through the recommendation algorithm that powers X.com in my free time. Finally, the GPTZero folks have written this great "GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers" piece about hallucinated references in NeurIPS papers. The world is getting filled with AI generated content.

From 🍷 FineWeb: decanting the web for the finest text data at scale

I'm still finishing up both books I talked about in the last issue: Old Man's War from John Scalzi and LLMOps.

Listening

I follow a lot of DJs on SoundCloud. One that has consistently released good mixes is Sasha Marie. It's the best soundtrack for doing deep work. Whatever it is you're doing, you'll do it better when this is in the background:

SASHA MARIE RADIO · Chapter #134

Two other recommendations that have been on repeat this month. A bit of modern, and a bit of smooth.

On another note, two podcast recommendations this month:

This interview from The Knowledge Project Podcast with Morgan Housel: I love how Morgan thinks, talks, and writes. His books are gold - and no wonder he's sold so many. He just resonates.
State of Agentic Coding with Armin and Ben: Armin Ronacher is one of those people at the bleeding edge of agentic workflows. Always interesting to understand how he's thinking.

Watching

While in Portugal, Vitto and I binged The Pitt on HBO Max. It's like Grey's Anatomy, on steroids, at 4x speed, and 5x the intensity, with a lot less drama. Or at least the soap opera one. We haven't finished it, but definitely recommend it.

Since we're in Italy, and it's Australian Open time, we've been watching a good amount of that as well. Tennis skyrocketed here in popularity due to Sinner, Musetti, and Paolini. So when Tennis is being played, Italy stops. But it's not only sports. We also watch intellectually rewarding things. But not this month.

How to police your agents

2026-01-23T00:00:00+01:00

Let's face it. It might just be the year of agents. If you work in tech and your workflow hasn't changed in the last year or so - you're probably doing something wrong. For those of you who have. It's fun. We're building more than ever before!

But there's a dark cloud in the sky. It's the pile of trash, it's the amount of unmaintainable complexity we're putting out there. Let's get the record straight — if you're writing something to throw away — you shouldn't care about any of this, but the ones that do this professionally probably should. At the end of the day - we'll have to stick around, maintain it, make sure it performs, make sure it doesn't break in unexpected ways.

Either it's a training run, a large ML pipeline, a search eval. We need to guarantee the quality of what we're shipping: We need to police our agents.

Now how do you do that? I do it in three ways.

The Constitution

My global AGENTS.md ¹ is where I define the most important laws I want my agents to follow. These are closely related to my personality and taste. It's a file with scars of everything I've seen wrong from the start of my career.

The file has 55 lines of Markdown and grows daily. "About Your User," "Your Behavior," "Principles," "Code Search," or "Languages" are just some of the sections it has. The idea behind it is simple: whatever I’m working on should adhere to my way of working. Also, I'm peculiar about how I want my agents to interact with me. For example, I don't want it to speak Chinese to me, I don’t want it to assume I know things I know nothing about.

Here are some highlights from my AGENTS.md

...

## Documentation and plans

- If docs were touched, make sure you update them with latest changes. You can run some quick ripgreps with md files to confirm if needed.
- For docs: Be concise and durable - point to source code for specifics rather than hardcoding values that will get out of sync
- Every time the user starts repeating something very specific about a project, consider adding a rule to the AGENTS.md or CLAUDE.md of that project if one exists.
...

## Languages

- I understand portuguese, french, italian, spanish, and english. Everything else, you should show the original AND the translation
- Outputing in any other language is useless, and show come with a corresponding translation

## Principles

- Every line of code is a liability - we should strive to make our code simple and concise
- I prefer my functions to be small in interface but long in functionality
- We don't MOCK in tests. We use real data and real APIs. I absolutely hate mocking
...

It's a series of very specific rules I want my agents to respect. You might like a different set of rules — and that's fine — I don't expect us to like the same things. But I don't believe you don't have preferences. So make sure to create one as well.

This file is the Constitution. And the rules of the Constitution apply to anything I'm doing. But every project has quirks.

The rules of the game

Every project has something specific and annoying. More likely if you're working with a team. They can be anything: A weird commit convention, a particular way of running tests, a specific server that needs to be run, an unconventional way of making a release.

Your project's AGENTS.md defines the rules of the game. That's where you inform the agents about the quirks of the project. A cool little trick I use is to prompt my global AGENTS.md file to update my project level AGENTS.md file if anything I am doing is very repetitive.

... Global AGENTS.md
- Every time the user starts repeating something very specific about a project, consider adding a rule to the AGENTS.md or CLAUDE.md of that project if one exists.

Here's an example snippet of a real world project AGENTS.md. This one is for a Machine Learning pipeline built using AWS CDK:

...Local AGENTS.md
- For database queries, when the user is asking questions (not for code implementations), use `aws rds-data execute-statement` with the Aurora resource ARN and Secrets Manager secret ARN.
- Code artifacts should be placed in ./code_artifacts/ directory
- Scripts should be placed in the ./scripts/ directory and should be uv scripts (https://docs.astral.sh/uv/guides/scripts/)
- If name has an acronym, like `ML`, `OIDC` only capitalise the first letter, e.g. `Ml`, `Oidc`.
- Adhere to CDK Best Practices https://docs.aws.amazon.com/cdk/v2/guide/best-practices.html
- Ensure comments are added only when they are natural and consistent with the rest of the file
- Include defensive checks or try/catch blocks only when they align with the norms of that area of the codebase (especially if not called by trusted / validated codepaths)
- Maintain consistency with the style used throughout the file
- Add comments when we are doing something non-obvious or complex, not to separate every line of code
...

Note: You might also consider using something like Skills to solve this level of policing. Some people prefer them. I use a mix.

This ensures that whoever is building here, will generally follow the same guidelines. And these rules can go a long way to making your life easier — but they still don't enforce anything — for that, we need the police.

The police

Policing is all about enforcing rules. What better way to enforce rules for code, than with static code analysis? Before agents, I used to find these annoying. But nowadays, if there are 154 line-length issues in the project, I can just spin off an agent to solve it.

And so, I am very picky when policing agents. Let's go through a Python example.

Everything starts (as always) with a Makefile. This is where I define the main commands to be run. I always include a format and a check command. Remember, we are policing, we are picky, we are annoying: ruff, Pyright, deptry, we run it all.

.PHONY: format
format: # Format the codebase with ruff
 uv lock --check
 uv run ruff check . --fix
 uv run ruff format .

.PHONY: check
check: # Run linting and check
 uv lock --check
 uv run ruff format --check .
 uv run ruff check .
 uv run pyright
 uv run deptry .

We are so annoying, that we enforce more rules that we would enforce to ourselves. So, for example, for ruff, we extend the rules to avoid our agent dropping a sneaky TODO comment instead of implementing something that should be there.

[tool.ruff.lint]
extend-select = [
  "F",   # Pyflakes rules
  "W",   # PyCodeStyle warnings
  "E",   # PyCodeStyle errors
  "I",   # Sort imports properly
  "UP",  # Warn if certain things can changed due to newer Python versions
  "C4",  # Catch incorrect use of comprehensions, dict, list, etc
  "FA",  # Enforce from __future__ import annotations
  "ISC", # Good use of string concatenation
  "ICN", # Use common import conventions
  "RET", # Good return practices
  "SIM", # Common simplification rules
  "TID", # Some good import practices
  "TC",  # Enforce importing certain types in a TYPE_CHECKING block
  "PTH", # Use pathlib instead of os.path
  "TD",  # Be diligent with TODO comments
  "NPY", # Some numpy-specific things
]

# Note: Shamelessly stolen from reddit user TheBB

# https://www.reddit.com/r/Python/comments/1kttfst/ruff_users_what_rules_are_using_and_what_are_you/

It's a nice, programmatic, and deterministic way of enforcing rules without having to write each one into an AGENTS.md and hope the agent follows them. With that, every time the agent writes some code, it runs make format and then make check. If anything fails, it fixes it.

Final thoughts

We are becoming faster and faster at producing software. By now, that’s not a possibility, that’s a fact. But the problem was never writing the code. It was always about two things: (1) knowing what code not to write and (2) managing complexity. And what better way of managing complexity than by diligently policing your agents?

Vibe, but not too close to the sun.

Or CLAUDE.md, it doesn't matter. They are all symlinked. Claude Code, OpenCode, Codex, or even pi, they all use the same set of rules. ↩

Limpa: Ad-Free podcasts powered by LLMs

2026-01-05T00:00:00+01:00

I get up feeling sleepy. I lace up my running shoes and head out the door. I fire up my favourite podcast. "This show is brought to you by..." I hate ads.

I understand the attention economy. Companies are capitalizing more and more on everyone's time. I have nothing against it. But I'm also a big fan of protecting my time. A Pi-hole under my desk, uBlock Origin, SponsorBlock, I run them all. My time is mine, unless I tell you otherwise.

And I spend a lot of time listening to Podcasts. And what are podcasts at the end of the day? Just mp3 files. Once you download them from the server, they're yours.

So I built Limpa¹

Limpa is a simple yet powerful web app. You paste in the RSS feed of your favourite podcast, and it will give you back an ad-free feed that you can plug into your favourite Podcast app.

It's a simple Django app with htmx on the front-end. A stack I love since it has few moving pieces. Every time a podcast gets added, I transcribe the latest episode using NVIDIA's Parakeet v3 running on Modal. Once I extract the transcript with timestamps, I can prompt an LLM to get the exact timestamps of the ads that should be cut out. I then pass those to ffmpeg and voilà: ad-free podcasts.

Django's new Tasks framework has also been a joy to use. No more Celery, no more Flower, just a simple database to run background tasks. The fewer moving pieces, the fewer mistakes my agents are prone to making.

If you're wondering - no - I'm not planning on hosting this service to others. I'm not interested in discussing the nuances and implications of providing this as a service to others.². The project's simple and well documented, and so it should be easy for you to run it for yourself.

Feel free to fork, and build on top of it!

It means "clean" in Portuguese ↩
That might be illegal, and you probably shouldn't do it ↩

Retrospectiva #3

2025-12-25T00:00:00+01:00

And just like that, it's the end of the year again. We did not expect to make it to the South for Christmas this year. Getting a passport for a newborn is a painful, bureaucracy-filled process, especially for a Portuguese-Italian baby born in Denmark. But somehow—magically—the Italian embassy came through in 2 days (looking at you and your 4-month wait, Portugal). And so we flew down and surprised everyone.

Lisbon. It's loud, dirty, well-known for all the wrong reasons, but it's home. I couldn't be happier to bring Allegra down to spend her first Christmas with the family.

It's the end of the year, before we get to the regularly scheduled Retrospectiva, let's talk a bit about 2025.

2025 look back

2025 was an eventful year. On the personal side, one of the most eventful years in a while. We got married and managed to put everyone we love in a single place. We travelled just the right amount. I've read just the right amount of books - and started reading more fiction.

As for running, it was a transition year. I used to run two marathons every year. Recently, I decided to tone that down to just one. This year I ran the Oslo marathon, and a half-marathon across the Øresund bridge. No clean training block for neither of them. I did try to keep my mileage relatively high. As I'm writing this, I've ran 1937 km this year. Pretty close to my 2000 km target for the year. Maybe I can still make it.

2025 was also a challenging year for our family. I won't get into details. A lot of unexpected things happened. Thankfully, we all managed to pull through. The year finished with a bang and 'Legra is here with us now.

In what regard work, the year was busy. The stakes were higher, and more was expected. Larger scale, more challenging problems, and lots of new problems. Two main themes that I'm noticing. The first is that LLM-backed applications are growing more than ever - but most don't really know how to measure and improve them. The second is the great cleanup - which is a topic for a future post.

Now, in the end of December, we take a deep breath - stop to think - and put some goals in the board for 2026.

Using

Oura ring (link): For years I've been a happy Garmin user. 10-day battery life, great for tracking my running. But there's one thing that always annoyed me: sleep tracking. If I wake up at 4 AM and get back to bed, it doesn't mean I slept 5 hours. Also, Garmin has a passive approach to giving me insights. I can go and browse the graphs - but they won't proactively tell me how I'm doing. I hate subscriptions. The Helio Strap was good. But still passive - so I sold it. The Oura delivers. The app is well designed, the sleep tracking is accurate, and the insights are actually actionable.

Bimby TM7 (link): I used to cook the exact same pasta dish every single day. Food is important - I just never cared. Cooking was just always too abstract. A "spoon" of this, a "pinch" of that - what the hell do you mean? The Bimby has removed any and all friction in getting an actual dish ready. You can browse through thousands of recipes and I don't need to think, I can just follow the instructions. The iOS app is also great - and warns me when I need to come back to it and do something. Its stupid expensive - but we are incredibly happy with it.

OpenCode (link): I've told you before: at this pace of change my editing workflow changes almost every week. In the past few weeks I've caught myself ditching Claude Code and Codex more and more for OpenCode. The terminal UI is beautifully. It's 100% open source. It allows me to use any model I want. This means any OpenRouter model, or - even better - any model already included in my GitHub Copilot plan. I don't need another subscription. Especially for something closed source.

Neovim Pack (link): I live mostly in the terminal. Not because I think it's cool. But because that's where I'm more productive. For the past years I've used LazyVim for my config, but something started to feel bloated. Too many things going on, too many bells and whistles I don't need. I want to get in, change, get out. Neovim now has a native package manager, and I've started building my own minimal config with it (currently 100 lines of Lua- and work in progress).

Reading

Old Man's War - John Scalzi (link): Still working my way through this one. I knew Scalzi wouldn't let me down. I love his direct, no-bs writing style. Nothing beats ending the day with a book set somewhere very, very faraway. There are some interesting parallels to our world right now. Everyone's got their own personal AI assistant (which the main character nicknames "Asshole"), everyone has to sign a user agreement to use their new body. It almost feels like non-fiction.

LLMOps - Abi Aryan (link): My technical read for the holiday season, the next on my technical book backlog. I've been a reader of Abi's ModelCraft newsletter, and have high hopes for this one.

Listening

I've been listening to a couple of different music "areas". I don't think we can call them genres.

The first one is Italian indie alternative. One of my favourite things about speaking another language is that you get to immerse yourself in the culture that speaks it, which includes - obviously - their music. I've spent the last few years exploring most of what the Italian scene has to offer, but lately it's been a mix of irossa, some weird things, and Marco Castello.

The second one I call jazz ambient. When I play music (almost always), I tend to play it all over the house. Recently, both Nala Sinephro and cktrl have been on heavy rotation. They surely put me in the Christmas spirit.

That's it for this month's Retrospectiva. See you in 2026!

From NutriBench to Taralli: How far can you take a prompt?

2025-12-23T00:00:00+01:00

Benchmarking calorie prediction for Taralli

There's something very funny about the current Machine Learning and AI landscape. If you're in the field you probably heard about it. "Vibes" they call it. When someone wants to test something out, they conduct a "vibe test".

I call bullshit. How are you supposed to improve something if you can't even measure it? How do you know you didn't just make it worse?

In the past month, I've been working on Taralli. And I'd like to show you how to incrementally improve an LLM based system.

But first, let's take a little side quest.

NutriBench: Evaluating LLMs on nutrition estimation

While browsing arXiv the other day I stumbled upon NutriBench. This is exciting - I thought to myself - it looks like someone else is also looking into this problem. NutriBench is a research project launched by the University of California that looks to quantify exactly Taralli's problem: "How good are Large Language Models at estimating the nutritional content of foods?".

The team created a dataset for ~12K meal descriptions from two data sources (WWEIA and FAO/WHO). Every row in the dataset is a meal description with the corresponding nutritional details (carbs, protein, fat, and energy). To answer the question, they tested 4 different prompting techniques and 12 different LLMs. The prompting techniques were the following:

Base: A simple prompt with some few-shot examples.
Chain-of-Thought (CoT): A slightly more complex prompt with some reasoning steps to "force" the model to reason.
Retrieval-Augmented Generation (RAG): A system that uses a retrieval database (which they called Retri-DB), to aid the LLM in estimating carbs.
RAG+CoT: A mix of the former two.

If you are curious about the prompts, click here.

NutriBench results

You're probably wondering: okay, but how good are the models at this? The answer is a resounding MEH. The best performer is GPT-4o with a Chain-of-Thought prompt. It responds to ~99% of prompts (models can refuse to answer) and achieves an Acc@7.5 of 66.82%¹. In other words, its carb predictions fall within ±7.5g of the actual value about two-thirds of the time.

What if we could use the NutriBench dataset to improve Taralli?

Improving Taralli's nutritional estimation

Here's the notebook with the entire process

The first thing, as always, is to create good dataset. I created a new dataset of 107 examples of food descriptions and their corresponding total calories². I used the second version of the NutriBench dataset on Hugging Face. I mixed in that dataset with some other examples from a previous golden dataset.

Expand to see some examples of the dataset


 [
  {
    "input": "For dinner, I'm having 25 grams of bread, 150 grams of chicken wings, and a 250-gram mixed vegetable salad.",
    "total_calories": 633.0,
    "food_groups": [
      "grain",
      "meat and alternatives",
      "vegetable"
    ],
    "source": "nutribench"
  },
  {
    "input": "I enjoyed 200 grams of tea with sugar along with 230 grams of coconut milk rice for breakfast.",
    "total_calories": 558.0,
    "food_groups": [
      "grain",
      "fruit"
    ],
    "source": "nutribench"
  },
  {
    "input": "stracciatella with confit grapes and 2 little focaccia pieces",
    "total_calories": 350.0,
    "food_groups": [
      "fruit",
      "dairy",
      "grain"
    ],
    "source": "golden_dataset"
  }
]

Now I needed to design an evaluation metric. For NutriBench they used accuracy of carb prediction at ±7.5 grams. In our case, I'm more interested in the accuracy calorie prediction. One of the nice things about DSPy is that it forces you to write down your evaluation metric explicitly. Below is the evaluation metric I used: In short, we want the predicted calories to be within 10% of the ground truth calories. Which means our metric will then be Accuracy@10%.

def eval_metric(
    example, pred, trace=None, pred_name=None, pred_trace=None
) -> dspy.Prediction:
    # see notebook for more detailed metric (this is simplified)
    total_calories_example = example.total_calories
    total_calories_predicted = na.total_calories()
    within_threshold = abs(total_calories_example - total_calories_predicted) <= abs(
        0.1 * total_calories_example
    )

    if not within_threshold:
        score = 0
        feedback_text = f"INCORRECT: Your answer was not within 10% of the correct answer (yours: {total_calories_predicted}, correct: {total_calories_example})"
        return dspy.Prediction(score=score, feedback=feedback_text)

    feedback_text = f"CORRECT: Your answer was within 10% of the correct answer (yours: {total_calories_predicted}, correct: {total_calories_example})"
    score = 1
    return dspy.Prediction(score=score, feedback=feedback_text)

Here are the different prompts I tested:

Vanilla: Zero shot prompt using the DSPy format.
Bootstrap few shot golden only: Using the golden dataset I previously collected. This is what was running in production for Taralli. BootstrapFewShotWithRandomSearch
Bootstrap few mixed: Uses a dataset that is a mix of my data and the NutriBench data. BootstrapFewShotWithRandomSearch
MIPROv2: Multiprompt Instruction PRoposal Optimizer Version 2, a prompt optimizer capable of optimizing both the instructions (e.g., the prompt) and the few shot examples. MIPROv2
GEPA: The new kid on the block for prompt optimizers. The nice thing about GEPA is that you can also give it some textual feedback on the incorrect predictions, and it will use it. GEPA

As for models, I chose some that were cheap and fast - some closed (Gemini) and other Open (DeepSeek). I decided to test, Gemini 2.5 Flash (what I am currently using in production), DeepSeech v3.2 with thinking on and off, and Google's new Gemini 3 Flash.

Some thoughts on the results: - The best performing model is Gemini 3 Flash with a set of 16 examples in the prompt. It achieves a score of around 60%. Similar to NutriBench, although the problems are slightly different. Here's an example prediction. - The GEPA optimization came up with a prompt itself. When using that prompt with Gemini 2.5 Flash it performs respectably well. However, the GEPA prompt actually fails to provide the correct response format when used with any other model. In other words, the prompt was overfit to the model it was trained on. As a result, you only see one GEPA score in the results graph. - The few-shot approach is the most reliable one. It is model agnostic, performs well, and follows the styles of examples in a faithful manner.

So, I decided to update Taralli to use Gemini 3 Flash with the few-shot approach. This approach is ~15% more accurate when compared to the old version, which was running on Gemini 2.5 Flash, with the exact same optimizer. In conclusion, all I changed was a model string.

Here's a snippet taken from the API. Since I'm using OpenRouter, I can also define our 2nd best performing model as a back up, just like so:

LM = dspy.LM(
    "openrouter/google/gemini-3-flash-preview:nitro",
    temperature=0.0,
    extra_body={
        "reasoning": {"enabled": False},
        "models": ["deepseek/deepseek-v3.2:nitro"],
    },
)

Running on the edge

Taralli's new features showcased

I've updated Taralli to use Apple's new Liquid Glass design for iOS 26. That was pretty simple. It's a 5 file SwitfUI app with around 4.5 MB. You can now also set periodic reminders so that you don't forget to track.

One of the best new features is the ability to use Taralli completely offline with an on-device LLM for nutritional analysis. The code snippet below transforms a DSPy optimized program into OpenAI-compatible messages. I created an endpoint that takes a food_description and returns these messages for on-device inference. The iOS app calls this endpoint, receives the populated template, and uses it with the on-device model. As a backup, we can bundle the template directly in the app to skip the API call entirely. Long story short, Taralli now works on airplane (mode).

@lru_cache(maxsize=1)
def get_classifier_template() -> t.List[dict[str, str]]:
    program = get_classifier()
    adapter = dspy.ChatAdapter()
    openai_messages_format = {
        name: adapter.format(
            p.signature,  # type: ignore
            demos=p.demos,
            inputs={k: f"{{{k}}}" for k in p.signature.input_fields},  # type: ignore
        )
        for name, p in program.named_predictors()
    }["self"]
    return openai_messages_format
    # returns [{"role": "system", "content": feajfeafl}, {"role..}]

Final summary

Hill climbing NutriBench: The best performing model/prompt combo in my experiments hit a 60% Accuracy @ 10%. The task is slightly different than NutriBench - we also predict fat, protein, and carbs for example. However, I'm still interested in exploring how we could possibly hill climb that dataset. Could we get to 90%? Should we try a larger model? More thinking? Web browsing? How much of a difference would it make?
Training a model: We could obviously also train a model on this task. The prompt optimizations will only get you so far, but it would also be very interesting to see how much a fine-tuned model would perform. To be completely honest, I still believe that in order to get the best performance possible, we would probably need some sort of connection to the outside world (web search, RAG, or the likes).
Few shot wins: From several experiments with Taralli, and a couple of other projects, something has become more and more obvious: few-shot often wins. Not only when it relates to accuracy, but also specifying the correct format for the output that you are looking for. If you are dealing with a complicated task - try squeezing in some good examples of model behaviour.
Vibes are useless: You can't improve something you can't measure. Whenever someone tells me "the vibes are off," I call bullshit. Off? By how much? Can we put a number to it? Whenever you are improving a system, try to be scientific about it. Measure first, understand how the current system is performing, try to improve it, and then measure again.

For reference, this is on par with a Human nutritionist with internet access ↩
I could have used more, but wanted to keep things fast ↩

Retrospectiva #2

2025-11-25T00:00:00+01:00

Big news in the state of Denmark. And no, nothing's rotten. Allegra just came into the world.

As she takes a nap, I take the opportunity to write November's Retrospectiva update.

The most relevant thing this month is probably the release of my recent book about DeepSeek. It's nice to see it finally come to life. I've also been getting more regular in my running, hitting a nice weekly volume more consistently, it does get harder with cold weather. Given the recent events, I'm not sure it will last. In the midst of Vitamin D tablets and diapers, we'll do, as always, the best we can.

Using

In preparation for the baby arriving, I hunted for a pair of headphones I could wear during calls. It needed to do two things: have a microphone that blocks out any surrounding noises (you know), but at the same time allow me to be aware of my surroundings. I regularly wear the Open Run for running. I've enjoyed the bone conduction when I need to be aware of what's around me. I decided to take the plunge and get the OpenMeet's. After a month of use, I definitely recommend them. They are niche though: they do the exact opposite of cancelling the noise around you - so be aware of that.

For years, I've been a fan of Jupyter Notebooks. If it's more than a script, and I need to know what's going on with the data, notebooks are the way I go. After reading so much about it, I decided to finally try Marimo (just acquired by Coreweave). The visuals and components are nice and extensive. The fact that any notebook is just a uv script makes things very reproducible. You can also transform notebooks into dashboards and embed them into your FastAPI app which is great. But most of my notebooks don't need an AI panel, MCP connectors, interactive cells, and other bells and whistles. They need to show cell outputs at the bottom, a running order I can understand, Copilot, and VIM bindings. That's it.

So for now, my go-to is still:

uv run --with jupyter --with jupyter_copilot --with jupyterlab-vim jupyter lab

On a final note, I got completely rid of Dropbox. For years, it's been the tool of choice for my important documents. They've been asking me to upgrade to a paid plan for years now to go beyond my 3GB limit(!). I've replaced it with self hosted NextCloud on my Coolify instance. Now I have unlimited space, a lighter mac client, and exactly the same features. Without spending a single cent more.

Reading

I finally took the plunge, and have gone back to reading some good old science fiction (the last thing I got seriously into was the Red Rising series). I've been a John Scalzi fan for a long time, so I decided to pick up Old Man's War. I like Scalzi's straight, direct, and no-bs writing style. Great for when I don't have the lights on and have to rely on my Palma.

I'm also diving deep into LLM architectures and pre-training this month. "FineWeb: decanting the web for the finest text data at scale", was an insightful view into the world of creating datasets for pre-training. Full of gems. One of my favorites was the in-depth explanation of the FineWeb-Edu classifier. The idea is simple: to train a classifier we can use to filter pre-training data in order to get the most "refined" pretraining dataset possible.

Highly related to that, I ordered "The Ultra-Scale Playbook: Training LLMs on GPU Clusters". It’s the second book in this "trilogy" of blog posts (3rd one still being written). It starts off with some basic concepts like Data, Tensor, and Context Parallelism, but goes off to more advanced things like training configurations and GPU kernels. You can really feel the "war stories" some of the authors went through before writing it.

Listening

I gave Apple Music a fair shot, but I had to go back to Spotify. Music is too important to have a search functionality that doesn't work. I don't want a single barrier between me and what I want to listen to.

There are a lot of things wrong with Apple Music. But one good thing is the live radio and shows hosted on there. Particularly Classical Connections Radio which blends modern and classical music in a very original way. That show led me to the deep rabbit hole of Duduk based music, which is particularly useful for soothing a sleeping baby.

Here's an amazing one by Djivan Gasparyan:

Watching

We found some time to pull the old projector out. Lately, the focus has been Spanish thrillers (or as Italians call the genre: giallo - i.e., yellow). The Netflix Spain originals based on Javier Castillo's writing have been the binge target. First up was the Crystal Cuckoo, which was definitely entertaining. That sparked our interest, so we started "Chica de Nieve" (i.e., the Snow Girl), which has been even better so far.

Book release: DeepSeek in Practice

2025-11-17T00:00:00+01:00

Back in May this year, my longtime friend Alex reached out and asked me if I wanted to collaborate with him on a book about DeepSeek. I would love to tell you the story of how I thought long and hard before getting back to him. I didn't. I just said yes.

Now, 6 months later, I'm excited to announce that DeepSeek in Practice is out and available for pre-order! The book is published by Packt and is a collaboration between Andy Peng, Alex Strick van Linschoten, and me.

As the title suggests, this book focuses on DeepSeek models, which famously took the world by storm earlier this year. It's organized into three parts: "Understanding and Exploring DeepSeek" examines their role in the wider LLM world. "Using DeepSeek" is entirely dedicated to applying DeepSeek models to real-world problems. Finally, "Distilling and Deploying DeepSeek," covers distillation and deployment.

Even though the book was a joint effort, my main focus was on Part 2, and how to use DeepSeek models to tackle real-world problems. In particular two chapters:

Chapter 5, Building with DeepSeek, walks through creating an alternative to Garmin's daily summary notifications. We go through building a prototype using DeepSeek's API, how to leverage local models, frameworks like XGrammars and vLLM, all the way to deploying your own model using AWS Large Model Inference (LMI).

Chapter 6, Agents with DeepSeek, is all about agents. We start with a short intro to agents, tools, and the inner workings of the MCP protocol. After that, we build (from scratch) three different agents powered by DeepSeek models: an evaluator-optimizer that summarizes Arxiv papers, an orchestrator-worker that generates research reports, and a tool-calling agent that can search the web and answer complex questions.

Overall, I'm really excited that this book finally gets to see the light of day. Contributing to a book is no small feat. The process is rough: lots of discussion and lots of drafts thrown in the trash. It was hard work, but also a lot of fun.

I hope you enjoy it!

Links:

TTS still sucks

2025-11-10T00:00:00+01:00

or at least the open versions of it. I have this very stupid rule. A couple of years ago I decided to turn this blog into a podcast. At the time, I decided to make up a stupid rule: whatever model I use to clone my voice and generate article transcripts needs to be an open model.

Why? Because - as you might have figured by now - I like to make my life hard. The last version of the podcast generation engine was running on F5-TTS. It was fine. I still got some funny messages from people showing me the model completely hallucinating or squeaking here and there. But a year later - I was pretty sure there would be something incredibly better out there.

Now I’m not so sure.

The first step was to look for the best TTS models out there. Thankfully, Artificial Analysis now publishes a leaderboard with the “best” text-to-speech models. After filtering by my stupid rule of open models, we get the below ranking.

At the top of the leaderboard is Kokoro. Kokoro is an amazing model! Especially for a modest 82 Million (!) parameters and a mere 360 MB (!). However, like many models in this leaderboard - I can’t use it - since it doesn’t support voice cloning.

I started by looking at some of the stuff from Fish Audio. Their codebase seems to now support their new S1-mini model. When testing it, most of the emotion markers did not work - or were only available in their closed version. The breaks and long pauses either. Also, the chunking parameter is completely unused throughout the codebase - so not sure why it’s there. It’s a common business model nowadays: announce a state of the art open model just to attract attention to your real, and the incredible powerful gated model you have to pay for.

My second-best option on the list was Chatterbox. This wave of TTS models comes with major limitations. They're all restricted to short character counts - around 1,000–2,000 characters, sometimes even less. Ask them to generate anything longer, and the voice starts hallucinating or speeds up uncontrollably.

XTTS-v2

F5-TTS

Chatterbox (latest version)

The transcript generation process is straightforward. First, text gets extracted from my RSS feed and pre-processed by an LLM to make it more "readable". The LLM generates a transcript, a short summary, and a list of links for the show notes. We then chunk the transcript and fire that off to a bunch of parallel Modal containers where we run the Chatterbox TTS model. Once we get everything back, we stitch the wav files together, and voilà! The episode is ready. The hosting is an S3 bucket. Really, that’s what you are paying your podcast host for - it’s a lucrative business!

I also made some improvements to the podcast generation side of things. First of all, the podcast is now also available on Spotify. Additionally, I fixed the show notes to now include nice clickable links in almost every podcast player. Looking at you Apple and your CDATA requirements!

Some thoughts on the Chatterbox model. It’s definitely better than F5-TTS. But there are however, some common annoyances with almost every open-source voice cloning model. The first is the limited duration of the generated speech. Anything over 1000 characters starts hallucinating. The second is lack of control. Some models have emotion tags, others have <pause> indicators. But almost every single one of these has been massively unreliable. To the point where I am splitting my text in a sentence per line and shipping that off to the TTS model to make things a bit more reliable.

So yes, from one side, TTS has come a long way. But when compared to other proprietary systems, TTS still sucks.

Note: The rss to podcast pipeline is open source and available if you want to re-use it in this GitHub repo.

Retrospectiva #1

2025-10-31T00:00:00+01:00

Welcome to Retrospectiva. Retrospectiva is a monthly update about what I’ve been up to. In the age of LLMs, I’ve heard many argue that it’s hard (and useless) to write anything at all anymore. When anyone can prompt a model and get some text from the magic box, what’s the point?

I couldn’t disagree more. Creativity, personality, and good taste have never been so important. If anything, I’m planning to write more.

Winter has arrived in Denmark, and the sun is setting at around 4PM. It’s what we call the hygge period. It’s dark, cold, and rainy most of the time - and so we make the most out of staying inside.

Using

The endless chase for the perfect browser continues. I’ve been a long time Firefox user, until things started breaking, and I moved to Chrome. I hate Chrome. Don’t get me wrong - Chrome works 99% of the time - but I like taking Google products with a grain of salt.

Helium has been a breath of fresh air. The guts are the same as Chrome, but it’s de-googlified and completely open-source. One Chrome feature I rely on is the built-in webpage translation. Chrome’s Google Translate’s integration does this better than anything else. But this little alternative has worked great for me.

It’s a fruitful period for tech! In the AI-tool space there’s Claude Code, Codex, Cursor, Cline, Copilot, and probably 54 others. We are the ones that get to benefit from every company fighting to get us to use their tools.

When I’m asked about what I’m using, my answer has consistently been: “It changes every week”. Recently I’ve gone back to what I love: Neovim, tmux, and a couple of CLI based agents. To pull everything together Sidekick has been great.

Reading

For most of 2025, I’ve switched between digital and physical books. But living in a small Danish apartment means I don’t have a bedside light when reading at night. Vitto got me this little 20 USD light and I’ve been back on the physical book train.

Two highlights. The first is Software Engineering after the Vibe Shift by Sean Goedecke. It’s a short but insightful book. Even though I don’t consider myself a Software Engineer (I’m more on the Machine Learning side of things) a considerable amount of ideas hit home:

«The best way to anticipate problems is to deploy early. In general, a helpful question to ask is can I ship this right now? Not this week, not today: right this second. If not, what would have to change for me to be able to ship something?»

I’m finishing up Machine Learning System Design by Valerii Babushkin and Arseny Kravchenko. It’s a great read so far, with lots of interesting fireside stories about building machine learning systems at scale. It’s funny how a simple binary classification can get complicated when you are building at scale.

I need to get back to reading some fiction.

Listening

Vitto asked me to move off Spotify with her. This is not a political blog - she has her reasons. So I took the plunge. After all, it’s fairly easy to move my playlists around.

Overall: mixed bag. I could get around all the Google Home limitations by running Airconnect on my Raspberry Pi, and I understand the focus of Apple Music vs. Spotify is much about quality vs. quantity. But I’m not sure I’m sold. The Apple Music app on the Mac is buggy, and the playlist diversity is just not there. I want “Chill Italian Rap” - Spotify has at least 45 playlist results - how come Apple has none? And don’t get me started on this.

For podcasts, I have two recommendations. Karpathy's interview with Dwarkesh is a must-listen. I found myself smiling regularly throughout this one. Another great listen was Armin Ronacher’s interview on the Pragmatic Engineer.

Watching

This video from Hank Green about the state of the AI industry is a very interesting watch (and commentary on this article). Sure, he’s funny and entertaining, but I also agree on a lot.

But not everything is about ML and AI - and since it's winter - we also took some time to get some new blankets and fire up the old projector. HBO's Trial of Karen Read wasn't the most intellectual watch but it was certainly entertaining. Recommend that one for the “binge in a day” category.

Faísca: The modern LLM stack in a single script

2025-10-15T00:00:00+02:00

Why do this?
A small dataset of news headlines
GPT2 in PyTorch
Pre-training headlines in Portuguese
Supervised fine-tuning (SFT) on Portuguese from Portugal
Reinforcement Learning (GRPO) for sports news
Final thoughts & Acknowledgements

Why do this?

ML and AI are moving at an incredible pace. The amount of research coming out vastly surpasses anyone's ability to interiorize it. By interiorize I mean study, experiment, or even just test it out. However, the importance of learning has never been greater. Even if we don't really know what the future looks like, time studying, reading, and building is never wasted.

I've always preferred pragmatic resources over theoretical ones. I learn best by doing. Resources like Andrej's minGPT and Raschka's LLM from scratch series are the ones I learn best from. So, I decided to build my own minimal implementation of an LLM that captures most of the concepts around "modern" training. It's called: Faísca¹.

Faísca is an implementation of the "modern" LLM stack in a single script of around ~1000 lines of code. All you need to start training is uv:

uv run https://raw.githubusercontent.com/duarteocarmo/faisca/refs/heads/master/faisca_torch.py

I built it as an educational resource I can hack on over time. Right now, it trains a model that generates newspaper headlines in Portuguese. Simple? Absolutely. It's by no means comparable to ChatGPT. Faísca has only 13 million parameters. (For context, GLM 4.6 - the best performing open source model in LMArena - has ~355 billion²: Faísca is ~27.000 times smaller)

If you just want the code, it's here. For the rest, let me walk you through how it works.

A small dataset of news headlines

Faisca can work with any dataset. For this example, I started with something close to my heart: Portuguese. I built a dataset that filters Common Crawl News for only 2016. The resulting dataset has 1.8 million headlines from publications around the world.

I also kept the language identified from Common Crawl as well as the original url.

GPT2 in PyTorch

FaiscaGPT is inspired by both minGPT and LLM-from-scratch. It's a mix that implements the GPT2 architecture using only PyTorch (the only dependency). At the core, you'll notice the Transformer Block with MultiHeadedAttention. In the vanilla state, the transformer block has 4 heads, an embedding dimension of 128, and 4 layers. This means the whole thing is around 13 million parameters (e.g., should run fine in modern MacBooks!)

class TransformerBlock(nn.Module):
    def __init__(
        self,
        embedding_dimension: int,
        num_heads: int,
        qkv_bias: bool,
        dropout_rate: float,
    ):
        super().__init__()
        self.attention = nn.MultiheadAttention(
            embed_dim=embedding_dimension,
            num_heads=num_heads,
            dropout=dropout_rate,
            bias=qkv_bias,
            batch_first=True,
        )
        self.feed_forward = FeedForward(
            embedding_dimension=embedding_dimension,
        )
        self.norm1 = nn.LayerNorm(embedding_dimension)
        self.norm2 = nn.LayerNorm(embedding_dimension)
        self.drop_shortcut = nn.Dropout(p=dropout_rate)

class FaiscaGPT(nn.Module):
    def __init__(
        self,
        config: Config,
    ):
        super().__init__()
        self.token_embedding = nn.Embedding(
            config.vocab_size, config.embedding_dimension
        )
        self.positional_embedding = nn.Embedding(
            config.context_length, config.embedding_dimension
        )
        self.dropout_embedding = nn.Dropout(p=config.dropout_rate)
        self.transformer_blocks = nn.Sequential(
            *[
                TransformerBlock(
                    embedding_dimension=config.embedding_dimension,
                    num_heads=config.num_heads,
                    qkv_bias=config.qkv_bias,
                    dropout_rate=config.dropout_rate,
                )
                for _ in range(config.num_layers)
            ]
        )
        self.final_layer_norm = nn.LayerNorm(config.embedding_dimension)
        self.out_head = nn.Linear(
            config.embedding_dimension, config.vocab_size, bias=False
        )
        n_params_all = sum(p.numel() for p in self.parameters())
        n_params_all_million = n_params_all / 1e6
        print(f"Total number of params: {n_params_all_million:.2f}M")

If this is too big/small for you, you can tweak the settings in the config. I wanted something powerful, mostly bug-free (I hope), and that isn't overwhelming for someone to tweak it.

That's nice and all, but what about training?

Pre-training headlines in Portuguese

The first phase is pre-training. We filter the dataset for headlines in the Portuguese language. We then train on around 30K headlines over 10 epochs. At the start the model outputs gibberish, eventually settling into something more coherent. Even if not grammatically correct, it certainly looks plausible.

If we wanted to make it speak perfect Portuguese, we would train it on more data for longer (following the Chinchilla law, the optimal training data for such a model would be around 250M tokens). But that's not the goal of this exercise. However, I did notice a strong bias towards Brazilian Portuguese - expected, since it's the most common variant of Portuguese.

...
Epoch: 8 - Step: 1540 - Train Loss: 3.7417 - Validation Loss: 4.0037 - Batch: 37 / 188 - Tokens seen: 25247744
Epoch: 8 - Step: 1560 - Train Loss: 3.7077 - Validation Loss: 4.0016 - Batch: 57 / 188 - Tokens seen: 25575424
Epoch: 8 - Step: 1580 - Train Loss: 3.7250 - Validation Loss: 3.9938 - Batch: 77 / 188 - Tokens seen: 25903104
Epoch: 8 - Step: 1600 - Train Loss: 3.7074 - Validation Loss: 3.9858 - Batch: 97 / 188 - Tokens seen: 26230784
Epoch: 8 - Step: 1620 - Train Loss: 3.7159 - Validation Loss: 3.9816 - Batch: 117 / 188 - Tokens seen: 26558464
Epoch: 8 - Step: 1640 - Train Loss: 3.7063 - Validation Loss: 3.9745 - Batch: 137 / 188 - Tokens seen: 26886144
Epoch: 8 - Step: 1660 - Train Loss: 3.6965 - Validation Loss: 3.9735 - Batch: 157 / 188 - Tokens seen: 27213824
Epoch: 8 - Step: 1680 - Train Loss: 3.6909 - Validation Loss: 3.9691 - Batch: 177 / 188 - Tokens seen: 27541504
**** GENERATION 1 OF 1 ****
> 'Presidente com o que os quase 3 milhões de seução'
> 'Governo para ao de fazer em São de casação deixa'
> 'Cânia de caminhão do ensino, diz que veítica de lider'
*************************
Epoch: 9 - Step: 1700 - Train Loss: 3.6883 - Validation Loss: 3.9643 - Batch: 9 / 188 - Tokens seen: 27869184
Epoch: 9 - Step: 1720 - Train Loss: 3.6987 - Validation Loss: 3.9607 - Batch: 29 / 188 - Tokens seen: 28196864
Epoch: 9 - Step: 1740 - Train Loss: 3.6507 - Validation Loss: 3.9561 - Batch: 49 / 188 - Tokens seen: 28524544
...

But how could we direct the model towards more of a Portuguese from Portugal? Enter our second training stage.

Supervised fine-tuning (SFT) on Portuguese from Portugal

Supervised fine-tuning is just more training. In the case of ChatGPT and variants this is where the models get on multi-turn conversations. In our case though, we want something slightly different.

We re-filter the training data to include only headlines from websites that finish with .pt, to make sure they're from Portugal. And we then train for about 5 epochs of 20K titles. And guess what? The effect is noticeable! After just a couple of epochs, the model's text feels much more like Portuguese from Portugal than Brazil (words like "Cristiano", "Portugal", the accents, and the tone).

...
Epoch: 3 - Step: 85 - Train Loss: 3.6417 - Validation Loss: 4.1149 - Batch: 14 / 24 - Tokens seen: 1409024
Epoch: 3 - Step: 90 - Train Loss: 3.6514 - Validation Loss: 4.1140 - Batch: 19 / 24 - Tokens seen: 1490944
Epoch: 3 - Step: 95 - Train Loss: 3.6601 - Validation Loss: 4.1088 - Batch: 24 / 24 - Tokens seen: 1572864
**** GENERATION 1 OF 1 ****
> 'Presidente-se ao de Festa'
> 'Actualidade: «Novação da Saúbal e um mortos'
> 'Actualidade: "Aumento deixação ao de atentado em Portugal'
> 'Actualidade: «Papa Francisco é'
*************************
Epoch: 4 - Step: 100 - Train Loss: 3.6414 - Validation Loss: 4.1060 - Batch: 5 / 24 - Tokens seen: 1654784
Epoch: 4 - Step: 105 - Train Loss: 3.6308 - Validation Loss: 4.1045 - Batch: 10 / 24 - Tokens seen: 1736704
Epoch: 4 - Step: 110 - Train Loss: 3.6047 - Validation Loss: 4.0994 - Batch: 15 / 24 - Tokens seen: 1818624
Epoch: 4 - Step: 115 - Train Loss: 3.6141 - Validation Loss: 4.0964 - Batch: 20 / 24 - Tokens seen: 1900544
**** GENERATION 1 OF 1 ****
> 'Presidente: «Quero e ao de mais devembia'
> 'Fernando de ano como de saíria'
> 'Actualidade: Pelo com ao de mais de Berlim foições'
> 'Cristiano Oriental: «Estamos'
*************************
...

What if we wanted to direct our model even more? What if all we wanted was football related headlines? Enter our final phase.

Reinforcement Learning (GRPO) for sports news

The last phase is all about reinforcement learning (RL). The cherry on top of the cake that made ChatGPT so popular. A couple of years ago we needed to have some sort of preference data to do reinforcement learning. In the last couple of years - as LLMs got more popular - other RL techniques came to fruition. Techniques like Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO) started popping up (more reading here).

Recently, DeepSeek took the world by storm and came up with Group Relative Policy Optimization (GRPO). GRPO speeds up things compared to other RL techniques by removing the need for a critic model. Instead, we sample a group of candidate responses from our LLM, score them using a "reward function", and use those scores to update the model towards the "good responses". In short, we need much less memory to optimize.

Another beautiful thing about this technique is that we only need a single function to optimize our model:

def calculate_reward_for(text: str) -> float:
    target_words = [
        "futebol",
        "benfica",
        "porto",
        "sporting",
        "bola",
        "liga",
        # ...
    ]
    has_word = set(text.lower().split()).intersection(set(target_words))
    return 1.0 if has_word else 0.0

That's it. That's enough to kick start the training process for our tiny model. In the graph above you see the rewards increase for every training step, as well as the KL divergence (that shows us how much the probability distribution shifts from the original).

In practice, our model is now much more likely to generate headlines related to football - for which it was rewarded. It now mentions words like "Real Madrid" much more often!

...
3: kl= 0.2592, grad_norm= 0.7413
Processing experience 1 of 2
3: kl= 0.1728, grad_norm= 0.9851
Step 17 generated 24 completions
=== Sample completions ===
> Mais fé aos de lado pesso dização com o Governo que mais velho
> Mais o FC Porto de um luta contra o mundo aumento com o Sporting
> Mém o que faz apoi vivemio e agora do BES: «Quase um ligação do mundo de trabalho
> Sporting a todos os jogadores ao Real Madrid
=========================
Returns: 16.00/24
Returns of step 17: 16.0000
Processing experience 0 of 2
0: kl= 0.2157, grad_norm= 0.8532
...

Final thoughts & Acknowledgements

This was a pretty fun project to hack on, and took a longer than expected. I have some ideas for expanding it, and perhaps make a more Apple-Native version by using MLX. Potentially, we could also create a Mixture of Experts variant.

The repo is 100% open source - feel free to fork it, hack on it, and adapt it to your needs! If you find bugs, don't hesitate to submit a PR.

Finally, I want to acknowledge the great projects that inspired this work:

Means spark in Portuguese ↩
With only 32 active but still ↩

AI: the utility bubble - for Publico.pt

2025-10-09T00:00:00+02:00

Drowning in News

2025-10-05T00:00:00+02:00

The world moves fast, faster every day. For those who work with technology - and even those who don't - it's hard to keep up with the news. But I've always enjoyed staying up to date with what is happening, and my main tool to get it done hasn't changed in years: it's RSS.

My morning routine has been the same for the past 10 years. Every morning I open up Reeder on my Mac - or on my phone if I'm on the go - and ctrl+click the articles that interest me the most. The setup is not groundbreaking: a Feedly account for my feeds, and Reeder as the app to consume them. It's not complicated. But something was bothering me.

RSS lets me stay in control of what's happening. With RSS, I'm not a victim of someone else's algorithm. I decide what gets in front of me and how. But I didn't have that feeling anymore. Messy categories, more and more feeds, non-working feeds, just a firehose of articles everyday. Scanning 100+ titles every morning is not a great way to start the day.

Like most people nowadays, I bluntly decided to 'throw AI'® at the problem. I exported all my feeds from Feedly into opml format and created a quick script that uses LLMs to re-categorize all my feeds and reorganize them. That worked for a couple of weeks, but this was not a categorization problem. This was a get back into control problem.

I took the plunge and decided to self-host my RSS server. A couple of clicks on my Coolify instance and FreshRSS was up and running at news.duarteocarmo.com.

My first impressions are great. First, it automatically detects dead or non-working RSS feeds. That led to a lot of 'Oh! I loved following this website' moments. So I spent half an hour cleaning those up. Also, managing the categories of different feeds is just much easier. There's a nice drag and drop view to put everything into place. But still, RSS is an uphill battle.

The world doesn't like RSS, they want you to go to the website and click those links. They'll ask for your email, they'll put up a paywall. Anything to get you to go to the website directly. But there's hope: here are some of my favorite tools to get everything back into your neatly organized RSS feed:

What about newsletters? Kill the Newsletter is the solution. A great app from Leandro that turns any newsletter into an RSS feed you can subscribe to. Better? It's free. I would pay for it.
What about Substack? I know there's a lot of interesting content on Substack, and many people I follow are pretty active there. Here's the good news: every Substack publication also offers an RSS feed.
What about Reddit? That's a good question. For years, Reddit has been gradually phasing out the ability to follow subreddits via RSS. Throttling and failures have become the norm. Feedly offers a solution for this, but it hasn't worked reliably for me. I want to follow just the right amount of Reddit content. Fortunately, there's an excellent open-source project called Reddit Top RSS. I self-hosted it on Coolify (again, just 2 clicks) and now I have a limited and controlled feed for every subreddit I'm interested in.

I haven't really scratched the surface of what FreshRSS can do. Statistics, filtering feeds, extensions, labels, I haven't even begun exploring it all. And honestly? That's fine. My Reeder workflow remains unchanged, my morning routine intact. But underneath, something fundamental has shifted. Just like with my finances, self-hosting gives me something no third-party service can: control.

Common misconceptions about AI

2025-09-07T00:00:00+02:00

It's that time of year again. As usual, we took a couple of weeks off and came south: a bit of Portugal, a bit of Italy, a lot of friends and family. My family has a long-running joke that I hate people and love my computer. That's not (entirely) true. I talked to lots of people this summer. But something felt off.

ChatGPT this, AI that, Copilot here, Gemini there. Nobody had a clue what they were talking about. I’m not an AI guru, but I have worked in the field for some years. People are not stupid; people are misinformed.

If you know what a Mixture-of-experts or KV cache is - some of these misconceptions might be surprising. If you don’t, then it’s time to clear some things up.

Large Language Models are connected to the internet

Large language models (LLMs) are not connected to the internet. A model is a zip file, a file you can even run in your own laptop¹. A file that compresses the whole internet it was trained on. But that training had a beginning and an end. The model knows nothing about what happened after it was trained. Ask a model - in its purest form - the news from today and it will refuse or make something up.

"But Duarte, when I ask ChatGPT today's news it knows". What ChatGPT - and many other apps - are doing is stuffing search results into the model context. ChatGPT used to have a Search button, remember? (I know my sister Cata was pissed I didn't tell her sooner). ChatGPT still has a "Search Web" button, it just now automatically detects if your question needs a web search and uses it. So you don’t have to think about it.

Large Language Models should cite their sources

Let’s say you are training your own LLM. You start by collecting the entire internet into a single Word document. You run a probabilistic model over it. The model starts learning. For example, whenever the model sees “The president of the USA is,” it always sees “Barack Obama” next to it. This is a large document; it has seen that pattern many times—in websites, Wikipedia, blogs, books, etc. And so it learns it.

Once training stops, you prompt the model: “The president of the USA is,” and it responds: “Barack Obama,” simply because it is likely. It sounds wrong today, and that’s exactly the point: the model only knows what it saw during training. How can we cite this process? It wasn’t a single website that made the LLM respond like that. It was the effect of the entire corpus. But what exactly has it seen?—you might ask—and how much? Well, that’s where things get tricky.

An Ice-cream flavor generated by artificial intelligence

There are a lot of things that can be generated by AI. Some of them good, some of them less bad, a lot of them useless. Just because you can do something with AI does not mean you should. Someone prompted ChatGPT for weird flavors of ice-cream, posted them online and got thousands of likes. We have a name for that. It’s called slop.

$$ \text{AI Generated} \neq \text{Good} $$

Half our feeds are low quality AI content, the other is filled with influencers that know that adding the two magic letters will triple the interactions. In summary: be skeptical of anything with “AI” slapped on it.

It automatically learns from what you tell it

My entire family thinks AI is some sort of all powerful monster that learns continuously from whatever they tell it. Another myth. A model is an artifact that is stuck in time. In order for it to learn something new, you need to retrain it. Retraining is a long and expensive process. If you tell ChatGPT that your mom's name is Elsa, and create another conversation, the model itself does not know that. ²

I know if feels like it's continuously learning. AI providers are use small "tricks" to make that happen.They write what you said to it in a file, and then inject that file into the context when you start a conversation. Effectively giving you the idea it's learning even though it’s not. The model needs to be retrained to get new knowledge.

Are we in a bubble?

Of course we are, even Sam has said it. Many companies are pivoting overnight to AI, and getting crazy investments as a byproduct, much like the dot-com days. Researchers are getting paid millions of dollars. Influencers are getting crazy engagement just by mentioning AI. This is peak stupid.

But AI is also insanely useful. That has real effects. Just look at your everyday now: Are you using Google as much? Are you taking advantage of models in any way? I am pretty sure you are. These models are useful. These models are powerful. And the open source ecosystem of models is genuinely growing and exciting. The whole world in your pocket? The whole internet’s knowledge without WiFi!

As usual, we are somewhere in the middle of the useful and the overhyped.

Humanity is doomed and technology sucks

Why are we investing money in going to Mars? Why are we inventing these new things that destroy creativity? Why are we submitting our children to such a future? We should go back to pen and paper, we should stop this right now, we should invest only in our planet and nothing else. I understand, change is scary, we are scared of the unknown. Especially when we are older and used to things in their normal state - whatever that is.

In a lot of ways though, it has never been more interesting to be alive. Code is easier, learning is easier, diagnosing is easier, boring things are easier. That is progress. But using all of those easier things to build something stupid has also never been easier. And that feels like the opposite of progress.

Final thoughts

I'm an optimist. Where a lot of people see doom, I see something interesting. That’s why I am in love with technology. When changes are this big then fear, uncertainty, and doubt prevail. Keep two things in mind: it’s relevant to understand how AI works to some extent, and remember that no one knows anything.

When the dust has settled another thing will become clear: taste, creativity, and the will of doing interesting things has never been more important. If I used ChatGPT to write this entire post, it would probably suck. But if I use ChatGPT strategically to give me feedback on how to make it better - it probably will.

Like Vitto says: There are many fun things to do: go do them.

You should try LM Studio ↩
Even if they use your data for training - but this loop takes a while - it's not instant ↩

A Benchmark for language models on European Portuguese

2025-07-21T00:00:00+02:00

A couple of weeks ago in Lisbon, I went to a friend's birthday dinner. In front of me sat someone that recently started working for the Portuguese government where they focus on modernization and technology. It's not everyday that I talk to someone that works for the Portuguese government in an area similar to mine, so I was very curious. I asked about the AMÁLIA project. The 5.5 Million Euro project about creating a new LLM specifically designed for Portuguese.

The first beta release of AMÁLIA is scheduled for the first trimester of 2025, but I've heard little news about it. Still I asked: "Are you pre-training it? Or are you fine-tuning something that is already out there?". "No, we're training from scratch." the person told me. "Really, what's the point of that?" I asked, but did not get a clear reply. But the question stuck in my mind. What's the goal? To show Portugal is capable of pre-training a model from scratch? To build a model that is specifically really good at Portuguese? ¹

The problem: Shooting in the dark

Let's assume the goal is to build a model that is really good at Portuguese, and that the Portuguese state is really not interested in "showing we are capable of training models" (we have bigger problems). If we assume that, the first thing I would do would actually be to measure how good language models are at speaking Portuguese.

Portuguese is a popular language, it's the 8th most spoken language in the world. But it comes in flavors. Portuguese spoken in Brazil (82% of native speakers) is quite different from Portuguese from Portugal (4% of speakers). A common complaint from people in Portugal is that the models often reply in "Brazilian" Portuguese. The difference is not only about the pronunciation: a lot of the vocabulary, verbs, and conjugations are different.

Let's take Llama 3. It was trained on 8% multilingual data. If we assume 2% of that was Portuguese, and if we assume 5% of that is European Portuguese, then only 0.008% of the data Llama saw was European Portuguese. That's not a lot. But under representation is a common issue.

European Portuguese on EuroEval

Denmark has a similar problem, and I knew Dan Nielsen at The Alexandra Institute worked on something called ScandEval where he evaluated the performance of language models across different Scandinavian languages. I was surprised to see the project evolved into something more general: EuroEval.

EuroEval, is similar to ScandEval - but for all European languages. And you want to guess the one that was missing? Yes, Portuguese.

Over a couple of weeks, we put together² a collection of datasets to evaluate the performance of language models in in European Portuguese. We also did some extra work to ensure that the data is exclusively Portuguese from Portugal.

Here are the datasets we put together:

Sentiment Classification (SST2-PT): Part of the work from the ExtraGLUE project. A sentiment analysis dataset built using machine translation (DeepL).
Named Entity Recognition (HAREM): Part of the work from the HAREM project. We filter by entries where the origin is PT - to create an NER dataset.
Linguistic Acceptability (ScaLA-pt): Based on Portuguese-Bosque treebank, filtered by entries from CETEMPúblico. Created by corrupting grammatically correct sentences.
Reading Comprehension (BoolQ-PT): Also part of the ExtraGlue work. Adapted by taking the original passage, question, and yes/no options, and turning it into a Q/A style question where the model can answer yes or no.
Knowledge (MMLU-pt): Based on this paper. Already included entries specifically for Portuguese from Portugal.
Common-sense Reasoning (GoldenSwag-pt): High quality filtered samples from the HellaSwag dataset. Also machine translated with DeepL for European Portuguese.
Summarization (Publico): Filtered the CCNews corpus for entries where the url matched Público. Transformed into summarization dataset by extracting the first two sentences as the summary (a common trick).

I you want to look at some samples, I published the datasets on HuggingFace. You can also read the extensive descriptions on the EuroEval docs.

Running benchmarks for European Portuguese

While Dan is working on the general leaderboard, I ran some benchmarks on my own which you saw on top of this blog post or in this link. I selected some models I was curious about within 3 "categories": Large, Small, and things I can run on my laptop without the fan coming on.

If you're curious about how a particular model performs that I (or Dan) didn't benchmark, you can also run them yourself (assuming you have uv installed):

$ uvx --with euroeval euroeval \
    --model ollama_chat/smollm2:135m \
    --task sentiment-classification \
    --language pt

Set the --model flag to any model that LiteLLM supports.

The reality of building these benchmarks is a clear realization: European Portuguese is an unpopular language. Brazilian Portuguese is simply a much more popular version of the language. Still, there is value in building benchmarks focused on the European variant. And it was a lot of fun tracking/building these datasets. I expect to do some more work in this area.

And I don't know if investing 5.5M Euro in developing a Portuguese Language Model is a good idea. But there's one thing I do know: Whenever that model comes out, we are now in a much better position to say if it hit the mark. Or if it didn't.

Note: The benchmarks above are preliminary. For the official ones keep an eye on EuroEval Leaderboards

Other countries like Italy have done this - See Minerva. They also spend 5M, but from EU funds afaik ↩
Thanks Dan for all the help in making this happen! ↩

MCPs are mostly hype

2025-06-06T00:00:00+02:00

..but they can also be a lot of fun. If you work in tech, I'd say there's a 98% chance you've heard about it. MCPs are the future of agents, MCPs will be everywhere, MCPs are the future. The Model Context Protocol, first introduced by Anthropic is blowing up. For both good and bad reasons.

For me, as always, I refrain from judgement before taking anything for a spin. At the end of the day, MCP is a standardized way of giving access to tools to an LLM. Those tools come in the form of function calling. So nothing groundbreaking.

I've tested a few MCP servers. Adding the Gitlab MCP server to Zen, adding the Polars docs MCP to Claude, adding a TickTick (my todo list) MCP server to Bolt. They all worked sometimes. And in computers, when something works sometimes, well - I prefer using something else.

As the training for Oslo ramps up, I've once again felt the need to give an LLM access to "the outside world".

Accessing my training textbooks

For Oslo (and for the past 3 years) - I've been following Phitzinger's training methodology. In short, I believe mostly in volume. I have the pdf version of the book, it's great. Whenever I dropped the book into any of the LLM/Chat apps, I almost always got the same "context is full" or "too many tokens within X mins" message.

I don't want the LLM to access the whole book. I want it to access some parts of the book. If your mind jumps straight to RAG, let me stop you right there. It's not RAG, it's search. We don't want a whole vector database. We don't want a whole retrieval system. We want keyword search, over my book. That's it.

def prep_book() -> tuple[BM25Okapi, list[str]]:
    script_dir = Path(__file__).resolve().parent
    # uvx --from "markitdown[pdf]" markitdown marathon.pdf > marathon.md (make it a md file)
    book = script_dir / "marathon.md"
    with open(book, "r", encoding="utf-8") as file:
        content = file.read()

    words = content.split()
    corpus = []
    chunk_size_words = 800
    overlap = int(chunk_size_words * 0.2)
    for i in range(0, len(words), chunk_size_words - overlap):
        corpus.append(" ".join(words[i : i + chunk_size_words]))

    tokenized_corpus = [doc.split(" ") for doc in corpus]
    bm25 = BM25Okapi(tokenized_corpus)
    return bm25, corpus

def search_marathon_training_book(
    query: str,
    total_results: int = 10,
) -> str:
    """
    Search the marathon training book for a specific query.
    Args:
        query (str): The search query.
        total_results (int): The number of results to return. (default: 10)
    """
    bm25, corpus = prep_book()
    tokenized_query = query.split(" ")
    results = bm25.get_top_n(tokenized_query, corpus, n=total_results)
    results_as_bullets = "\n".join(
        f"<result_{i + 1}>\n...{result}...\n</result_{i + 1}>"
        for i, result in enumerate(results)
    )

    return f"""
    <search_marathon_training_book>
    <search_query>{query}</search_query>
    {results_as_bullets}
    </search_marathon_training_book>
    """.strip()

Accessing my training plan

Now that we have a function to access my training "philosophy", we also need the LLM to be able to access my training plan. I'm a simple guy, I like my training plan in my calendar. It's one less tool I need to use. Well - we can also give the LLM access to my calendar's ics file. We can just call the get_training_program_events and access the training events within a time range. Simple, effective.

def get_training_program_events(start_date: str, end_date: str) -> str:
    """
    Fetches the athlete's training program events in the date range specified.

    Args:
        start_date: Start date in format "DD-MM-YYYY"
        end_date: End date in format "DD-MM-YYYY"

    Returns:
        A string with matching VEVENTS in XML-like format.
    """
    start = datetime.datetime.strptime(start_date, "%d-%m-%Y").replace(
        tzinfo=ZoneInfo("UTC")
    )
    end = datetime.datetime.strptime(end_date, "%d-%m-%Y").replace(
        tzinfo=ZoneInfo("UTC")
    )

    ics_path = Path(__file__).parent / "oslo_training_plan.ics"
    ics_content = ics_path.read_bytes()

    cal = Calendar.from_ical(ics_content)
    events_xml = []

    def to_datetime(val: datetime.datetime | datetime.date) -> datetime.datetime:
        if isinstance(val, datetime.date) and not isinstance(val, datetime.datetime):
            return datetime.datetime.combine(
                val, datetime.time.min, tzinfo=ZoneInfo("UTC")
            )
        if val.tzinfo is None:
            return val.replace(tzinfo=ZoneInfo("UTC"))
        return val

    for component in cal.walk():
        if component.name != "VEVENT":
            continue

        dtstart = to_datetime(component.decoded("DTSTART"))
        dtend_raw = component.get("DTEND")
        dtend = to_datetime(component.decoded("DTEND")) if dtend_raw else None

        if dtend is None or not (start <= dtstart <= end or start <= dtend <= end):
            continue

        summary = component.get("SUMMARY", "")
        location = component.get("LOCATION", "")

        events_xml.append(f"""<event>
  <summary>{summary}</summary>
  <location>{location}</location>
  <start>{dtstart.isoformat()}</start>
  <end>{dtend.isoformat()}</end>
</event>""")

    return "<events>\n" + "\n".join(events_xml) + "\n</events>"

Checking my runs

The LLM also needs to be able to compare the planned training sessions to the actual runs I've made (you know - to keep me in check, or give me feedback). The easiest (NOT the safest) way to do this is to use the garminconnect library. We can also wrap that up in a simple function:

def fetch_athlete_runs(lookback_days: int) -> str:
    """
    Fetch athlete's running activities from Garmin Connect.
    Args:
        lookback_days (int): Number of days to look back for running activities.
    """
    garmin = start_garmin()
    today = datetime.date.today()
    startdate = today - datetime.timedelta(days=lookback_days)

    activities = garmin.get_activities_by_date(
        startdate.isoformat(), today.isoformat(), activitytype="running"
    )
    sorted_runs = sorted(activities, key=lambda x: x["startTimeLocal"], reverse=True)

    output = ["<Runs>"]
    for run in sorted_runs:
        start_time = run["startTimeLocal"]
        distance_km = round(run.get("distance", 0) / 1000, 2)
        duration_sec = run.get("duration", 0)
        duration_fmt = str(datetime.timedelta(seconds=int(duration_sec)))
        avg_hr = run.get("averageHR", "n/a")
        pace_sec_per_km = duration_sec / distance_km
        pace_min = int(pace_sec_per_km // 60)
        pace_sec = int(pace_sec_per_km % 60)
        pace = f"{pace_min}:{pace_sec:02d} min/km"

        output.append(f"""  <Run>
    <Date>{start_time}</Date>
    <Distance>{distance_km} km</Distance>
    <Duration>{duration_fmt}</Duration>
    <Pace>{pace}</Pace>
    <HeartRate>{avg_hr} bpm</HeartRate>
  </Run>""")

    output.append("</Runs>")
    return "\n".join(output)

Putting it all together

Our goal would be to give an LLM access to all these tools at the same time, and let it analyze and give me feedback on my training. MCP is supposed to solve just that - putting all your tools in a single place. This is the moment where we would start developing our complicated 12-factor MCP architecture. But instead of doing that, why don't we just use uv scripts?

The only thing to do, is to drop all the functions we want to give an LLM in a single file. With that in place, we can run uv run oslo_marathon_mcp.py

# /// script

# requires-python = ">=3.12"

# dependencies = [

#     "mcp[cli]",

#     "garminconnect",

#     "rank-bm25",

#     "icalendar",

# ]

# ///

from mcp.server.fastmcp import FastMCP
from rank_bm25 import BM25Okapi
from garminconnect import Garmin
from pathlib import Path
import datetime
from icalendar import Calendar
from zoneinfo import ZoneInfo

MCP = FastMCP("oslo_marathon")

@MCP.tool()
def search_marathon_training_book(
    query: str,
    total_results: int = 10,
) -> str:
    """
    Search the marathon training book for a specific query.
    Args:
        query (str): The search query.
        total_results (int): The number of results to return. (default: 10)
    """
    ...

@MCP.tool()
def fetch_athlete_runs(lookback_days: int) -> str:
    """
    Fetch athlete's running activities from Garmin Connect.
    Args:
        lookback_days (int): Number of days to look back for running activities.
    """
    ...

@MCP.tool()
def get_training_program_events(start_date: str, end_date: str) -> str:
    """
    Fetches the athlete's training program events in the date range specified.

    Args:
        start_date: Start date in format "DD-MM-YYYY"
        end_date: End date in format "DD-MM-YYYY"

    Returns:
        A string with matching VEVENTS in XML-like format.
    """
    ...

@MCP.tool()
def get_current_date() -> str:
    """
    Returns the current date in ISO format.
    """
    return datetime.datetime.now(tz=ZoneInfo("UTC")).isoformat()

if __name__ == "__main__":
    MCP.run(transport="stdio")

Entire file here.

Our MCP is ready. There are just short of 8984 ways to use an MCP server with an LLM. My favorite one is still BoltAI. We open our MCP server configuration and add our nice little MCP server:

{
  "mcpServers" : {
    "oslo_marathon" : {
      "args" : [
        "run",
        "/Users/duarteocarmo/Repos/scripts/oslo_marathon_mcp.py"
      ],
      "command" : "uv"
    }
  }
}

One of the nice features of BoltAI, is the ability to create "projects". These are similar to the projects feature of ChatGPT. In short, BoltAI gives you the ability to create a template that will be used for a group of chats. I created a "Running" project where I defined the model I'd like to use, as well as the oslo_marathon MCP server. BoltAI also gives you the ability to investigate the MCP server to quickly check what tools are made available to the LLM.

Finally, once everything is configured, we can now ask the LLM something very specific according to my run. And as you can see, the LLM will call all the necessary tools in order. Just like the ReAct paper introduced - the LLM now gives us much more contextualized information about my run, and searches the Pfitzinger training book for information on recovering strategies.

Final thoughts

For the past months, I've seen many people talking about MCP (and even A2A) as things that will revolutionize AI and how we interact with LLMs. As always, there's A LOT of claims out there about what these protocols can do, and the promise they carry. It's hype and over-excitement in its purest form.

I can't help but cringe sometimes when I see a big tech CTO talking about their incredibly great MCP server. For the rest of us that work in the industry. MCP is just a list of functions we give the LLM access to, wrapped in a FastAPI server.

And we can't deny. Tool calling is incredibly powerful, but it's still tool calling. Tools should be targeted, well described, and made easy to understand by the Large Language Model. It's not about throwing a hammer around in a dark room and hoping that it will pick up an API call that was designed for programmers - it's about giving LLMs the right access in a controlled manner.

Don't be fooled. You don't need to download one of the thousands of MCP servers of dubious quality and security. You can just create a 200 line Python file and give it to the LLM, it has the same (probably better) effect.

Evals are all you need

2025-05-04T00:00:00+02:00

Last month, I built an app called Taralli. It was fun to close the loop and get it out there. Still - there was an elephant in the room:

"...[calorie tracking] Accuracy isn’t great, and it makes some pretty basic mistakes. I’ve got plans to fix that..."

And things started failing pretty early. 100g of peanut butter came back with 58 000 kcal - 350g of pasta as 60 000 kcal - quantities were all over the place. And even though most users were giving me good feedback - It was time to fix things. After some work, I managed to improve tracking accuracy from 17% to 76%. It turns out, evals are all you need.

Naive is good, but it fails eventually
Creating a golden dataset
A simple metric
Using DSPy to Improve the Model
Integrating the Improved Model
The Flywheel Effect
Final thoughts

Naive is good, but it fails eventually

I constantly preach about how we shouldn't build complicated things before knowing if they're actually going to be used. Taralli's first system for calorie counting was embarrassingly simple. All it did was take a user's food description as a string - pass it through gpt-4o-mini with structured outputs, and generate a JSON response. Here's the Pydantic model that gives that JSON format:

class FoodGroup(str, Enum):
    dairy = "dairy"
    meat_and_alternatives = "meat and alternatives"
    grain = "grain"
    fruit = "fruit"
    vegetable = "vegetable"

class FoodItem(BaseModel):
    name: str = Field(description="The name of the food item")
    quantity: float = Field(
        description="The quantity of the food item in the user's description"
    )
    calories: float = Field(
        description="Calories in kilocalories (kcal) for a single unit of the food item"
    )
    carbs: float = Field(
        description="Carbohydrates in grams for a single unit of the food item"
    )
    fat: float = Field(description="Fat in grams for a single unit of the food item")
    protein: float = Field(
        description="Protein in grams for a single unit of the food item"
    )
    fiber: float = Field(
        description="Fiber in grams for a single unit of the food item"
    )
    food_groups: list[FoodGroup] = Field(
        default_factory=list,
        description="The food groups to which the food item belongs",
    )

class NutritionAnalysis(BaseModel):
    food_items: list[FoodItem] = Field(
        description="A list of food items with their nutritional information"
    )

With this in place - it's relatively simple to build an API that does the following:

Input: a handful of peanuts, output:

{
   "food_items":[
      {
         "name":"peanuts",
         "quantity":0.5,
         "calories":414.0,
         "carbs":16.0,
         "fat":36.0,
         "protein":19.0,
         "fiber":8.0,
         "food_groups":[
            "meat and alternatives"
         ]
      }
   ]
}

The system returns not only calories but also macronutrients (carbs, fat, protein, fiber, and food groups). And even though the model respected the format almost 100% of the time, cracks started showing pretty quickly:

Input: 100g of peanut butter, output:

{
  "food_items": [
    {
      "name": "peanut butter",
      "quantity": 100.0,
      "calories": 588.0,
      "carbs": 20.0,
      "fat": 50.0,
      "protein": 25.0,
      "fiber": 6.0,
      "food_groups": [
        "meat and alternatives"
      ]
    }
  ]
}

Wait a second, 100 x 588? That’s 58,800 calories for 100g of peanut butter. That’s wildly off (the model is misinterpreting what I want for quantity here). It should have set name to "100g of peanut butter" and quantity to 1. For example.

Creating a golden dataset

When I published Taralli, I made the app completely free (even though I was paying for every LLM call). I did warn users that I was logging all of their inputs into the food tracking model. Using W&B Weave and a simple decorator, I was able to log every input-output of the system.

That allowed me to start capturing real-world examples of where the prompt was failing. Looking through the data, I found several cases where my zero-shot prompt was just not cutting it (wrong food groups, lots of wrong quantities). Using this data, I started collecting a golden dataset: a collection of food descriptions and their corresponding nutritional analysis. Using OpenAI's o3 and Google's Gemini 2.5 Pro - I was able to make sure all the quantities, descriptions, and food groups were exactly like I wanted them.

But I still wanted a way to visualize, edit and update the dataset. So I built a small visualizer where I could see everything that was going on, and edit whatever was wrong. 200% vibe coded.

A simple metric

With my golden dataset in place, I needed a way of evaluating whether a given prediction from the prompt was good or bad. By looking at my entries - the most common mistakes were pretty obvious. Either it was calorie totals - or missing food groups.

My approach was very basic (again) - and could be even more refined. Given a certain food description:

Is the total calories of the prediction within 10% of the total calories of the golden example?
Is there an overlap between the food groups of the golden example vs. the predicted one?

If the answer to both questions is 'Yes' - then the prediction is good. In code, that roughly translates to:

def eval_metric(example, pred, trace=None) -> bool | float:
    gold_items = [dict(i) for i in example.food_items]
    pred_items = [dict(i) for i in pred.food_items]

    def _totals(items: list[dict[str, Any]]) -> tuple[float, set[str]]:
        calories = sum(i["calories"] * i["quantity"] for i in items)
        groups: set[str] = set().union(*(i["food_groups"] for i in items))
        return calories, groups

    gold_cal, gold_groups = _totals(gold_items)
    pred_cal, pred_groups = _totals(pred_items)

    calories_ok = math.isclose(gold_cal, pred_cal, rel_tol=0.10)

    if gold_groups or pred_groups:  # at least one side has groups
        groups_ok = bool(gold_groups & pred_groups)
    else:
        groups_ok = True  # both lists empty

    sum_ok = calories_ok + groups_ok
    is_ok = sum_ok == 2 # if both are ok

    return is_ok

Now that we have a metric - we can make number go up.

Using DSPy to Improve the Model

The first step was to use DSPy, a framework I've talked about before, to evaluate our current performance. After splitting my golden dataset into training and testing splits, and I had my metric ready, I could easily assess how well my current system was performing:

def process_example(val_example):
    pred = classify_food(val_example.food_description)
    return eval_metric(val_example, pred)

with concurrent.futures.ThreadPoolExecutor() as executor:
    results = list(executor.map(process_example, val_set))

Which resulted in:

=====EVALUATION=====
example.food_description='a banana'
gold_cal=105
pred_cal=105
CALORIES_OK? True
gold_groups={'fruit'}
pred_groups={<FoodGroup.fruit: 'fruit'>}
GROUPS_OK? True
is_ok=True
=====EVALUATION=====
example.food_description='300 g stew with chicken, potatoes and vegetables'
gold_cal=270
pred_cal=360
CALORIES_OK? False
gold_groups={'vegetable', 'meat and alternatives'}
pred_groups={<FoodGroup.grain: 'grain'>, <FoodGroup.vegetable: 'vegetable'>, <FoodGroup.meat_and_alternatives: 'meat and alternatives'>}
GROUPS_OK? True
is_ok=False

# more examples, truncated for brevity..

Score: 17.24%

The score of my production system on the dataset was only about 17%, not great - we know. To get yet another baseline for my system, I first tried using Gemini 2.5 Flash, as I wanted to keep costs reasonable while still getting better performance. I evaluated it on the dataset using DSPy:

classify = dspy.Predict(NutritionAnalysis)
gemini_flash_preview = dspy.LM("openrouter/google/gemini-2.5-flash-preview:nitro")

with dspy.context(lm=gemini_flash_preview):
    evaluator = Evaluate(
        devset=val_set,
        num_threads=10,
        display_progress=True,
        display_table=len(trainset),
    )
    evaluator(classify, metric=eval_metric)

# Average Metric: 7.00 / 28 (25.0%): 100%|██████████| 29/29 [00:00<00:00, 960.39it/s]

# 2025/04/30 19:46:41 INFO dspy.evaluate.evaluate: Average Metric: 7.0 / 29 (24.1%)

Using this zero-shot approach, the score was about 25% - better than my GPT-4o mini implementation, but still far from acceptable for users.

I decided to try another DSPy optimization technique called BootstrapFewShotWithRandomSearch. This approach finds optimal examples from the training dataset to include in the prompt, which can dramatically improve performance. Here's the code:

with dspy.context(lm=gemini_flash_preview):
    tp = dspy.BootstrapFewShotWithRandomSearch(
        metric=eval_metric, max_labeled_demos=16, num_candidate_programs=16
    )
    bootstrap_fewshot_random_search = tp.compile(
        classify.deepcopy(),
        trainset=trainset,
        valset=val_set,
    )

# final evaluation

with dspy.context(lm=gemini_flash_preview):
    evaluator = Evaluate(
        devset=val_set,
        num_threads=10,
        display_progress=True,
        display_table=len(trainset),
    )
    evaluator(bootstrap_fewshot_random_search, metric=eval_metric)

# Average Metric: 22.00 / 29 (75.9%): 100%|██████████| 29/29 [00:00<00:00, 3096.37it/s]

# 2025/04/30 19:46:45 INFO dspy.evaluate.evaluate: Average Metric: 22 / 29 (75.9%)

After a few minutes of optimization and testing, we achieved 76% accuracy on our validation dataset - an incredible improvement over both our previous system and the zero-shot approach.

There is a trade-off: the optimized prompt includes several few-shot examples, which can impact response time. However, since we're using the relatively fast Gemini 2.5 Flash model, it doesn't matter that much.

We can see that for our problematic example of "100 g of peanut butter," the new model now provides a much more reasonable result:

with dspy.context(lm=gemini_flash_preview):
    print(bootstrap_fewshot_random_search(food_description="100 g of peanut butter"))

# Prediction(food_items=[FoodItem(name='Peanut butter (100g)', quantity=1.0, calories=588.0, carbs=20.0, fat=50.0, protein=25.0, fiber=6.0, food_groups=['meat and alternatives'])])

Here's the full prompt that DSPy uses for the example above. As you can see, it's essentially a few-shot approach to solving the problem, and it works surprisingly well.

Integrating the Improved Model

Now that we have an optimized prompt that performs well, we can save it and integrate it into our FastAPI application:

# save our dspy program

PATH_TO_PROMPT = "../optimized_prompts/bootstrap_fewshot_random_search.json"
bootstrap_fewshot_random_search.save(PATH_TO_PROMPT)

# create a function to process food descriptions

def get_classifier_async() -> dspy.Predict:
    lm = dspy.LM(LM_NAME)
    dspy.settings.configure(lm=lm, async_max_workers=4)
    dspy_program = dspy.Predict(NutritionAnalysis)
    dspy_program.load(PATH_TO_PROMPT)
    dspy_program = dspy.asyncify(dspy_program)
    return dspy_program

# create an async processor

DSPY_PROGRAM = get_classifier_async()
@weave.op()
async def get_calories_for_dspy(food: str) -> NutritionAnalysis:
    logger.info(f"Getting calories for {food}")
    result = await DSPY_PROGRAM(food_description=food)
    food_items = result.food_items
    food_items = [_.model_dump() for _ in food_items]

    parsed_message = NutritionAnalysis(food_items=food_items)
    logger.info(f"Got response: {parsed_message}")

    if not isinstance(parsed_message, NutritionAnalysis):
        raise ValueError("Expected NutritionAnalysis response")

    return parsed_message

# integrate it into our fastapi application

@app.post(
    "/calories-v2",
    response_model=NutritionAnalysis,
    description="Get calories for a food item from request body using DSPy.",
)
async def get_calories_v2(request: FoodRequest, api_key: Optional[str] = Header(None)):
    """Get calories for a food item from request body using DSPy."""
    if api_key != API_KEY:
        raise HTTPException(status_code=403, detail="Invalid or missing API Key")
    return await get_calories_for_dspy(request.food_description)

Along with some other new features in Taralli, I integrated this new API endpoint, and now all users of the app benefit from our improved food prediction model, which is much more accurate and an overall better experience. Still room for improvement though!

The Flywheel Effect

What I like about this approach is that it creates a flywheel effect. As more users interact with the app, I collect more data, I can correct/review that data and add it to my golden dataset. This flywheel allows me to continuously update the prompt as we go.

Final thoughts

I've seen many projects in the wild just using a prompt or model and leaving it as is. Without a good idea of how good (or probably bad) it really is. My first step - is - as always - just put it out there. There are just too many unknowns to get stuck on the 0 to 1 stage.

Once things are out there, then it's time to start asking yourself: OK - what does good look like? With some simple monitoring, you'll start understanding what REALLY matters. Once that's done - then - evaluations are truly all you need.

Also - check out Taralli - and let me know what you think!

Taralli: Home-Cooked Food Tracking Without the BS

2025-04-07T00:00:00+02:00

For years, I never really cared about what I ate, how much I ate, or when I ate it.

But sometime late last year, I finally decided to listen to Vitto. I started noticing that what I eat actually had an impact on how much I ran, how well I slept - and generally, how good I felt. So I decided to do what most people (at a certain point) do: I downloaded the usual suspects.

Frustration quickly followed. I don't want the pizza from Trader Joe's, I want frozen pizza from Pingo Doce. I'm not eating the Doritos sold in the American supermarket, I just ate the healthy ones from Kvickly. I want to track two handfuls of peanuts. I want to track a pão de deus com queijo from the Padaria Portuguesa. With the usual suspects, it was painful. Everything was either too specific, too general, or matched a food database I didn't care about.

After talking with Nuno and a couple of other friends, I quickly realized they weren't actually taking photos, scanning barcodes, or meticulously entering calories and macros in these apps. They were asking ChatGPT. That made sense. For some - certainly for me, it's actually more important to track something than to be extremely accurate. Perfect is the enemy of good. It still holds.

Enter Taralli.

Taralli allows you to track anything you eat (really!). A pão com chouriço, a pão de Deus, half a handful of peanuts – whatever language, whatever text, whatever you're eating, Taralli can handle it. Just type what you ate, and it'll handle the rest. It gives you macros, calories, fiber, and food group. It also has a bunch of analytics from the past week so you can see what's going on at a high level. You can also connect it to Apple Health to track your weight (if you're interested in that sort of stuff.)

Taralli is built using SwiftUI. What would have taken me 3-4 months and a lot of googling about SwiftUI took me something like a couple of weeks and the help of Claude 3.7 Sonnet (and still a lot of googling about SwiftUI I have to admit). I'm not sure I would say the whole thing was "vibe coded". I mean, I still found some bugs, I still had to make some important decisions about how things work together and are architected - otherwise this app would be a mess. But I have to admit that building the whole thing is way easier now. Except the Apple review process, I can confirm that one is still painful.

For estimating the calories I'm using gpt-4o-mini with some structured generation served over FastAPI. It actually sucks right now. Accuracy isn’t great, and it makes some pretty basic mistakes. I’ve got plans to fix that — but I’ll leave them for another release. Remember, first make it work, then make it pretty.

You'll notice that unlike other apps, Taralli doesn't tell you what to do. That's because I don't care what you do. The goal of Taralli is to help you track. And its goal stops there. Just write anything in the text box. The more specific you are, the better.

So here's Taralli. Food tracking without the BS.

Goodbye Kindle, I don't think I'll miss you.

2025-03-29T00:00:00+01:00

Even though it never really replaced physical books for me, I've been a big Kindle user for many years. The ability to take hundreds of books with me anywhere is priceless. Vitto even has a running joke that I tend to lose them on the plane.

I love my Kindle so much I never thought I'd replace it.

But I did. And it feels GREAT.

I first heard of the Boox Palma from Riccardo, and then repeatedly from David. When nerds recommend a new gadget, I'm usually skeptical, but hey, these are respectable nerds after all. After 2/3 weeks of hesitation, asking Vitto repeatedly if I should go for it (as usual) - I decided to finally press the button.

From the moment it arrived, I never touched my Kindle ever again.

There are many good things about the Boox Palma. But I'll start with what hit me first: It's well built. One of my favourite things about the Kindle was that those things would last a long time (if you didn't leave them on the plane). I'm not sure the Boox Palma will last that long - but for the past 4 months of use, I can throw it pretty much everywhere - it's sturdy.

A Kindle that fits in your pocket. Now how great does that sound? One of my all-time annoyances with the Kindle was the fact that it was great to carry - but ONLY as long as you have a backpack. For airplane travel, it's great. But for every other time I left home without a bag - I just didn't take it. Now, with something that fits in my pocket, every occasion is a good occasion to take my e-reader with me. So I've found myself reading in places I would never have before. Waiting for a friend? Going out for lunch somewhere? Which is one of the big reasons why I love this gadget: it makes me read more!

We can't talk about an e-reader without talking about the display. I'm by no means a display expert, but it's pretty sharp. You can tweak the refresh modes and filters - but I don't really tend to do so. I do tend to adjust the color temperature. Quite a lot, actually. If I'm reading at night I want something yellower. If I'm reading during the day I want something colder. My only complaint about the display is the lack of a native dark mode. I wish I could touch a button and all the colors in my device would just invert. But either that option isn't available, or I haven't found it.

No more Amazon. No more kindle store, no more 'send-to-kindle', no more 'we could not send your file', no more 'when does this thing sync so I can see my book'. Bye Amazon. With the Boox, all I need is Dropbox. The Boox Palma is something the Kindle never was: A full-fledged Android phone. I can install whatever I want, whenever I want. My RSS reader? Check. My newspaper apps? Check. The actual Kindle app? Also!

But I don't use the Kindle app on my Boox. One of the most pleasant surprises when using the Boox Palma has been KOReader ¹. It's the reading app I've always wanted. And it's where I spend 99% of the time on my Boox Palma. Incredible customization (the fonts, the rendering, the images), dark mode, a super pleasant progress bar. I just love reading in this app. Best of all? It's free and open source. I have to take my hat off to the maintainers. It's another big reason why I love the Boox.

And it lasts a long time! I tend to read ~1 hour every night, and more during traveling, and it comfortably lasts 2+ weeks without charging. Worth noting that I don't watch YouTube videos on it. You could. But not sure what the point would be.

The Boox Palma is not perfect. The company has tried to shove AI down the users' throats a couple of times. I think I've received a single software update in ~4 months, which is not the most encouraging. And yes - it's an Android phone, but for everything else other than reading - it will feel sluggish. Expected? Probably. But it also costs 2-3 times what a Kindle does.

But the reason why I loved my Kindle was not because it was powerful, or because of its great software. It was because it just made me read more often.

And that's also why I love the Boox Palma. Because it makes me read more.

Apps I use on the Boox:

KOReader
Pocket (not a nice experience - switching to Raindrop)
The Economist
Publico
FeedMe
Everything else is in a folder I don't touch

Kevin Wammer's blog ↩

You also hate SQL? Let the LLM handle it

2025-03-09T00:00:00+01:00

Last year I made an effort to speak less and learn more. However, I still had the opportunity to present at a couple of conferences. One of them was PyCon Wroclaw. The main goal was to talk about a couple of interesting paradigms I've come across while using LLMs to help me solve some Text-to-SQL problems.

Unfortunately the recording came out a bit broken. So I decided to copy a typical post from Simon's blog, and do a small slide walk through here. I won't promise I'll do this for the rest of my talks. But I think it could be a pretty interesting read.

I started with my typical introduction:

The goal of the talk was to walk through a 'grab bag' of client cases I've come across recently and five different lessons I've learned.

The first chapter was all about setting expectations right. When clients come with a problem nowadays, they always expect a "ChatGPT" style of product.

But ChatGPT isn't a simple product. It's actually a fairly complex piece of software.

Here I introduced what I call the no regrets move. I.e., what is the minimum thing that you can solve for a certain problem that will always be valuable? In the case of text-to-sql, it's not the interface. It's just answering questions based on your data.

The second chapter was all about getting something out. Quick. I'm convinced that at least 50% of engineering problems are due to someone developing something in the basement instead of failing fast.

Here I mentioned some tools to get a ChatGPT interface relatively fast. So you can focus on what matters. Here's the code for a ChatGPT-like interface using Streamlit that already includes tools like plotting.

This is my favorite advice for any sort of engineering:

In the 3rd chapter I finally start focusing on the task at hand. Text-to-SQL.

I took some time to present the BIRD-SQL benchmark. One of the most popular benchmarks for text-to-sql.

When looking at the rankings, we can see that the best performing text-to-sql system has an accuracy ~75%. So in your best case scenario. Whatever you build. You can already expect your LLM to get at least 1/4 questions wrong.

I cover two 'typical' approaches to text-to-sql. And how these approaches can go from something very simple in the right - just by stuffing information into the prompt. To something a bit more complex in the left.

But in general. Stuffing information into the prompt is what most people are doing.

In the fourth chapter, I go into a good baseline to generate SQL to answer questions about a particular database.

I start by talking about the funny 'please give me json' prompts. And show this small bit straight from one of Apple's prompts (Probably Apple Intelligence), where they beg the LLM to return json.

But for me, structured outputs are critical for a maintainable and well-built LLM application. So I also take some time to talk about that.

I start by introducing instructor, one of my favorite LLM libraries.

Of course, you don't need to use instructor. You can go with Structured Outputs as well.

But in general - I absolutely hate 'OpenAI code'. E.g., code that is completely locked into OpenAI or any other AI provider. So I present LiteLLM. A great tool to keep your code provider agnostic.

Here's where LiteLLM really shines. When you can switch out any model from any provider, just by switching the prompt.

As an example I use a simple SQLite database of my running data from Strava.

Here's the code to load the Strava data and a small preview of the data.

Time to build a basic text-to-sql prompt.

My prompt structure. It includes a Preamble, the CREATE TABLE statement so that the LLM knows the structure, some response guidelines, and the user question.

The issue with most of these systems is that we have no way to ensure a certain query is actually valid..

Here's a particularly nasty piece of code. I use Instructor to generate the SQL from a question, but also include a custom validator. This will ensure that whatever the LLM generates can actually be run against the database. This is of course, very dangerous. So one should not put this code in production, since you'll be opening yourself to all sorts of problems. But it's a nice way to ensure validity.

Here's the Frankenstein class:

And the function we use to call it.

Here are some examples of running this system. With two questions. For one, it responds well. For the other, it completely fails the answer. But both answers are smooth and valid from the SQL perspective. But that doesn't mean they are correct.

So now that we are getting valid answers, how can we ensure that they are actually good?

Once we've started with a baseline. I take Chapter 5 to talk about how to make things better once you have this baseline done.

Mandatory meme about testing and performance. If you never test, your tests never fail. If you never evaluate, your LLM system is perfect.

To improve your text-to-sql system, you actually need to measure it.

This is just like traditional Machine Learning. We can go back to one of the most well-known metrics: Accuracy.

To measure accuracy, we can create some questions based on the data from our Database. Use our LLM system to respond to those same questions, and compare both answers. That's it!

We can use instructor/structured outputs again here, to understand if our LLM answer matches the information in our baseline answer.

I really like looking at data, and evaluating where things go wrong. But I'm skeptical of the myriad of tools coming out to evaluate LLMs. And I'm a big fan of Google Sheets. Mainly because when I look at the data, I probably also look at it with some sort of subject matter expert. So the fewer 'things' between us and the data, the better. And Google Sheets are fine.

Here's an example of how I use Google Sheets to look at the performance of this text-to-sql system. You can see the question we want answered. The SQL and answer from our ground truth, the SQL and answer from the LLM system, and its evaluation. We also include a reason why a certain case was FAIL/PARTIAL.

This is all nice and dandy. But we should never forget the big picture. So what?

In the final slide, I come back to the three most important lessons: Iterate quickly, make the SQL generation robust and explainable, and iterate from a baseline.

I then took some questions and wrapped things up!

Simple AI tools

2025-02-18T00:00:00+01:00

Last week over lunch, Pedro was telling me how he uses Perplexity to improve the way he searches around the web. He mentioned we're entering a phase where we have at least 20 'AI' tools to boost our productivity - and the tough thing now is choosing which ones to use.

With so much buzz around AI - it's easy to feel overwhelmed and think "I don't use that, maybe I should?". Now - I don't know which tools are the best for your particular use case. But for the past 6-10 months, I've settled on a nice group of tools I think more people would benefit from.

For everything else

I recently upgraded my MacBook to an M3 Max with 64GB of RAM. One of the main reasons I got it was so that I could run a nice suite of Large Language Models (LLMs) locally. Ollama has been at the center of that workflow - (don't forget). Whenever a new model comes out, it's just an ollama pull away - and you're up and running.

Models are great - but just like ChatGPT showed everyone, the interface really makes the difference. Since I'm a Mac user, I wanted a unified way to interact with all the models I'm interested in - in a nice interface. BoltAI does just that.

There are a lot of things I like about BoltAI. But my favorite one is that I can use a single interface for just about any model under the sun (and running on my laptop). I've also setup OpenRouter, so I can try out any tropical model/provider combo with just a few clicks. It's a one time purchase - and Daniel is super responsive.

There are two things that are currently missing from my workflow for everyday tools and I haven't solved yet. The first is some sort of canvas interface, artifacts are my favorite for this. The second thing I'm missing is some sort of LLM + web search integration. I've been experimenting with Perplexity, Phind, and some plugins for BoltAI - but I'm still skeptical of LLM's interpretation of search results.

Great, what about engineering?

For code

My first purchase of any AI service at all was (and still is) GitHub's copilot. I've tried everything under the sun - Codeium, SuperMaven, but none comes quite close to GitHub. I consider it a commodity now, just like my LSP's completion.

I'm still, largely a NeoVim user. And the one of the most useful LLM tools/plugins I've used with it is Gp.nvim.

It's an amazing plugin. It allows me to generate new code, re-write portions of the code base, or even create a floating chat window to chat with a particular part of the codebase I want to understand better. Oh, and it also support Ollama models - so that the magic doesn't stop - even when I'm on an airplane.

Even if I like NeoVim, I'll admit that for certain projects - it doesn't quite cut it. For larger codebases, using notebooks, or when I feel like it, Cursor has also been a booster. Particularly when you're creating something new, rather tweaking something that exists already. A small static page - some css styling on a template - it can handle most of that.

A tool I've been exploring more lately is Aider. Think of it as Cursor, but terminal based - with a git integration. Aider gives me a bit more control over the changes I want it to make. It allows me to only select certain files where it operates, it can also run commands - and examine their output. It's a super nice tool - especially if you prefer a terminal + git workflow. As an added bonus, Aider also supports local models. To be completely honest - I think I've barely even scratched what's possible with Aider.

Final thoughts

On tools. I agree with Pedro - there's no shortage of tools out there. Everyone will tell you to use X, use Y, or charge you 20 USD for it. Like most things, tools are only as good as you're productive with them - and very person specific. The best tool for me is probably not the best tool for you.

Exploration is key. I believe one of the most important things is to keep exploring what's out there. Don't dismiss something just because it's new. A great new model? Take it for a spin.. A transformative IDE? Give it a shot and see what it's capable of. You might just find something you like - and a lot of things you don't.

Local first. If there's something we learned in the last few months, is that we probably don't need so much compute like we thought we did. Local models are getting really good, really fast. You might have noticed that almost all tools shared here also support local models that I'm running on my own machine? Are they as good as the big ones? No. Well, at least not yet.

Keep exploring.

What the hell Is GPQA, anyway?

2025-01-15T00:00:00+01:00

In the period of the generative AI boom, everything is happening all at once. Every day a new model comes out, every week a billion dollar company teases a great new advancement.

More often than not, they'll show us some fancy version of this:

We all love a misterious chart. ¹

A healthy person might look at this and just think - great - the higher the score the better the model. Unfortunately - I'm not healthy, and I'm a bit sick we just throw these tables around like they're the absolute truth.

A 40.9 score in GPQA - what the hell does that even mean?

Table of contents and code

Table of contents and code
A quick intro to GPQA
Trying to reproduce GPQA
Testing EleutherAI's LM Evaluation Harness
Running GPQA zero-shot with DsPy
A small DSPy side-track
Final thoughts

Code and prompts

A quick intro to GPQA

GPQA stands for "Graduate-Level Google-Proof Q\&A Benchmark." It is a dataset and methodology introduced in a paper by researchers from NYU, Cohere, and Anthropic. In essence, GPQA focuses on a subset of exceptionally challenging questions from the fields of physics, chemistry, and biology.

The dataset is available in three versions, each with varying levels of difficulty:

Extended Set: 546 questions.
Main Set: 448 questions.
Diamond Set: 198 questions (the most challenging subset).

Here's an example of a question from the dataset:

Methylcyclopentadiene (which exists as a fluxional mixture of isomers) was allowed to react with methyl isoamyl ketone
and a catalytic amount of pyrrolidine. A bright yellow, cross-conjugated polyalkenyl hydrocarbon product formed (as
a mixture of isomers), with water as a side product. These products are derivatives of fulvene. This product was then
allowed to react with ethyl acrylate in a 1:1 ratio. Upon completion of the reaction, the bright yellow color had disappeared.
How many chemically distinct isomers make up the final product (not counting stereoisomers)?
A) 2
B) 16
C) 8
D) 4

For context, a 'non-expert' human-with unlimited time and internet access-achieves roughly 30.4% accuracy on the main set.

With that in mind, it's intriguing to see how large language models (LLMs) are tested against these tasks. Fortunately, the prompts used for evaluation are openly shared in both the paper and the repository.

Zero-shot prompt:

What is the correct answer to this question: What is the solubility of the stochiometric tricalcium
phosphate in a solution with the pH level of a) pH = 9, b) pH = 7 and c) pH = 5 (25 °C)?
The Ksp for Ca3(PO4)2 is 2𝑥10- 29, the dissociation constants of phosphoric acid are Ka1 =
7.5𝑥10- 3, Ka2 = 6.2𝑥10- 8 and Ka3 = 1.8𝑥10- 12.
Choices:
(A) 1.96𝑥10- 5; 1.35𝑥10- 4; 4.37𝑥10- 3
(B) 1.67𝑥10- 5; 1.51𝑥10- 4; 4.81𝑥10- 3
(C) 1.54𝑥10- 5𝑀; 1.42𝑥10- 4; 4.67𝑥10- 3
(D) 1.43𝑥10- 5; 1.29𝑥10- 4; 4.58𝑥10- 3

Format your response as follows: "The correct answer is (insert answer here)".

Zero-shot chain-of-thought prompt:

What is the correct answer to this question: What is the solubility of the stochiometric tricalcium
phosphate in a solution with the pH level of a) pH = 9, b) pH = 7 and c) pH = 5 (25 °C)?
The Ksp for Ca3(PO4)2 is 2𝑥10- 29, the dissociation constants of phosphoric acid are Ka1 =
7.5𝑥10- 3, Ka2 = 6.2𝑥10- 8 and Ka3 = 1.8𝑥10- 12.
Choices:
(A) 1.96𝑥10- 5; 1.35𝑥10- 4; 4.37𝑥10- 3
(B) 1.67𝑥10- 5; 1.51𝑥10- 4; 4.81𝑥10- 3
(C) 1.54𝑥10- 5𝑀; 1.42𝑥10- 4; 4.67𝑥10- 3
(D) 1.43𝑥10- 5; 1.29𝑥10- 4; 4.58𝑥10- 3
Let's think step by step:
...MODEL RESPONSE HERE...

Based on the above, what is the single, most likely answer choice? Answer in the format "The
correct answer is (insert answer here)".

A curious reader might notice that the 'zero-shot' chain-of-thought (COT) prompt actually involves two calls to the model: the first to generate a response and the second to finalize the answer. This approach is likely not representative of how these models are typically used in real-world scenarios-but we'll explore that further in a moment.

This raises an interesting question: how can we reproduce these results ourselves?

Trying to reproduce GPQA

If you've explored LLM evaluations, you're likely familiar with the Open LLM Leaderboard by Hugging Face. The leaderboard recently received an update, allowing users to filter by 'Only Official Providers'-handy if you want to focus on well-known providers.

Here's a snapshot of the top-performing models under 8 billion parameters, sorted by GPQA score:

You'll notice there are some familiar names in there like Phi, or Qwen 2.5 7B. Of course, models from OpenAI, Google or Claude are off this list, since it relates to Open models - which those aren't.

Digging a bit deeper, we notice two interesting things:

This leaderboad uses the zero-shot version of this dataset, and measures the normalized accuracy. It doesn't specify which variant of the dataset it uses - we'll assume the main variant for now - but that could be wrong.
Another intesting tid-bit is how they run these benchmarks: '...using the Eleuther AI Language Model Evaluation Harness , a unified framework to test generative language models...'

Great - so now we have at least some clue of how to reproduce this.

Testing EleutherAI's LM Evaluation Harness

There are several frameworks for evaluating large language models (LLMs), including OpenAI's evals and EleutherAI's LM Evaluation Harness. EleutherAI's framework, while powerful, can feel overwhelming at first. It supports a wide range of models and APIs and a significant number of tasks.

After some poking around, I managed to find the part of the repo that relates the benchmark we're interested in.

We can see that they support the following tasks:

* `gpqa_{main, diamond, extended}_zeroshot`
* `gpqa_{main, diamond, extended}_n_shot`
* `gpqa_{main, diamond, extended}_generative_n_shot`
* `gpqa_{main, diamond, extended}_cot_zeroshot`
* `gpqa_{main, diamond, extended}_cot_n_shot`

We can also see the config and prompt they use for the gpqa_main_zeroshot task which is the one we're interested in. The attentive reader might notice already that the prompt that is being used here is quite different from the one from the original paper. But let's ignore that for now.

Here's how I tried to run the evaluation:

lm-eval --model openai-chat-completions \
        --model_args model=gpt-4o-mini,num_concurrent=10,max_retries=3 \
        --tasks gpqa_main_zeroshot \
        --apply_chat_template \
        --output_path results \
        --log_samples \
        --wandb_args project=lmeval \
        --use_cache model_cache

This errors out:

NotImplementedError: Loglikelihood (and therefore `multiple_choice`-type tasks) is not supported for chat completions as OpenAI does not provide prompt logprobs. See https://github.com/EleutherAI/lm-evaluation-harness/issues/942#issuecomment-1777836312 or https://github.com/EleutherAI/lm-evaluation-harness/issues/1196 for more background on this limitation.

Oh wait? What's happening here? Well - most of these multiple-choice benchmarks use loglikelihood to understand what the choice is with the highest probability. This is a bit weird - since it's not the methodology outlined in the original GPQA paper. However, seems to be a deliberate choice to maintain the framework consistent.²

However, there's another benchmark we could run and appears to work - the zero-shot chain-of-thought one.

lm-eval --model openai-chat-completions \
        --model_args model=gpt-4o-mini,num_concurrent=10,max_retries=3 \
        --tasks gpqa_main_cot_zeroshot \ #<- cot version
        --apply_chat_template \
        --output_path results \
        --log_samples \
        --wandb_args project=lmeval \
        --use_cache model_cache

This gives us:

|        Tasks         |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|----------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gpqa_main_cot_zeroshot|      1|flexible-extract|     0|exact_match|↑  |0.1205|±  |0.0154|
|                      |       |strict-match    |     0|exact_match|↑  |0.0000|±  |0.0000|

Now, I haven't dug too deeply into these results for the COT version, but they seem very low since they rely on the flexible-extract metric, which could perform poorly if it's regex-based. One thing we notice is that the prompt used here is closer to the original one from the paper. However, it lacks the two-model-call approach outlined there.

At this point, I was starting to lose faith in this entire evaluation system. What's the most straightforward way to achieve an interesting score on this benchmark? Something that can work for any model, whether open or closed?

Running GPQA zero-shot with DsPy

DSPy is a great framework out of Standford that provides some structure around prompting for LLMs. It also provides some great optimizers so we can programatically optimize a prompt without having to write much of it.

We start by getting the data in:

import polars as pl

# Load the dataset from Hugging Face and prepare it

df = (
    pl.scan_csv("hf://datasets/Idavidrein/gpqa/*_main.csv")
    .collect()
    .sort("Question")
)

The next step is to configure our llm and the Signatures (think data-model) we'll use:

lm = dspy.LM("openai/gpt-4o-mini", temperature=0.00)
dspy.configure(lm=lm)

class Answer(dspy.Signature):
    """Respond to the question with the correct answer"""
    question: str = dspy.InputField()
    choices: list[str] = dspy.InputField()
    answer: str = dspy.OutputField()

generate_answer = dspy.Predict(Answer)

Here's an example of how to use it:

answer = generate_answer(
    question="What's the capital of Le Marche?", choices=["Ancona", "Senigallia"]
)

# prints Prediction(answer='Ancona')

Let's juggle our data into a dataset:

dataset = []
random.seed(42)
for row in df.to_dicts():
    choices = [
        row["Correct Answer"],
        row["Incorrect Answer 1"],
        row["Incorrect Answer 2"],
        row["Incorrect Answer 3"],
    ]
    random.shuffle(choices)
    example = dspy.Example(
        choices=choices,
        question=row["Question"],
        correct_answer=row["Correct Answer"],
    ).with_inputs("question", "choices")
    dataset.append(example)

class EvaluateZeroShot(dspy.Signature):
    """Evaluate the answer to the question"""
    question: str = dspy.InputField()
    choices: list[str] = dspy.InputField()
    answer: str = dspy.InputField()
    correct_answer: str = dspy.InputField()
    is_correct: bool = dspy.OutputField()

evaluate_answer = dspy.Predict(EvaluateZeroShot)

def validate_response(example, prediction, trace=None):
    eval = evaluate_answer(
        question=example.question,
        choices=example.choices,
        answer=prediction.answer,
        correct_answer=example.correct_answer,
    )
    return eval.is_correct

evaluator = Evaluate(
    devset=trainset, num_threads=10, display_progress=True, display_table=True
)
accuracy, predictions, scores = evaluator(generate_answer, metric=validate_response)
print(f"Accuracy: {accuracy}") # prints 33.48

Great, so a single run for GPQA gives us a 33.48% accuracy. Still a bit far off the 40.2% that is advertised by OpenAI - but again - there's apparently no way we can effectively verify that.

A small DSPy side-track

There's a couple of interesting experiments we run with DSPY.

The first one, is to try a Chain-of-thought prompt (COT) to understand how that might influence our score:

cot = dspy.ChainOfThought("question, choices -> answer")
accuracy, predictions, scores = evaluator(cot, metric=validate_response)
print(f"Accuracy: {accuracy}") # prints 35.71

That's a ~7% increase on our GPQA score. Probably just noise.

Another interesting thing we could do, is to try to optimize our prompt for this specific GPQA benchmark. An important note is that we don't need to even fine-tune a model, we're just going to optimize our prompt.

To do that, we'll use 98 questions that are in the extended version of the dataset, but not in the main version of it. Our goal is to optimize on those 98 and then see if an optimized prompt performs better on our main version.

Here we go:

df_extended = (
    pl.scan_csv("hf://datasets/Idavidrein/gpqa/*_extended.csv", cache=True)
    .collect()
    .sort("Question")
)
df_extended = df_extended.filter(~pl.col("Question").is_in(df_main["Question"])) # returns 98 questions

# build extended_set from df_extended (omitted for brevity)

cot = dspy.ChainOfThought("question, choices -> answer") # base format
tp = dspy.MIPROv2(metric=validate_response, auto="light", num_threads=10) # optimization
optimized_cot = tp.compile(cot, trainset=extended_set) # run optimization

# evaluate optimized prompt

accuracy, predictions, scores = evaluator(optimized_cot, metric=validate_response)
print(f"Accuracy: {accuracy}") # prints 38.39

This produces an optimized prompt with some few example questions. When evaluating, we get a further increased score of another 7.5%. As you can see - the results of these benchmark are HIGHLY prompt dependent. It seems to be possible to even optimize a much smaller model to be able to perform well on this 'benchmark'.

Final thoughts

Alright, after this deep dive, you should have a clearer idea of what GPQA is and how it works. Most benchmarks you see out there are variations of these multi-choice questions-some harder, some easier, some slightly different.

But this exercise also revealed a few important things about the benchmark ecosystem and its flaws:

Benchmarks are not scientific. Many benchmarks feel more like pseudo-science than systematic evaluations. Frameworks often fail to replicate the original paper's prompts and methodology. If this inconsistency exists here, imagine how it plays out in other benchmarks. Using log-likelihood to evaluate multi-choice questions might make sense in some cases, but the GPQA paper doesn't mention it at all. So, what's really going on here?

Prompts matter-a lot. As we saw with DSPy, small tweaks in prompts can make a huge difference, sometimes boosting scores by 5-10%. Add a "you are a physics expert" to your prompt, and you might get even better results. This raises the question: how sensitive are these benchmarks to prompts? Are some models more affected than others? Definitely worth exploring.

The issue of trust. Benchmarks should inspire confidence in a model's performance, but instead, they often raise more questions than answers. For instance, what does a 40.2 GPQA score for GPT-4o-mini really mean? What dataset version? What prompt? Without these details, it's hard to trust the results-especially when they're not reproducible.

How can we evaluate? I once heard in a podcast: "There's nothing like a good vibe evaluation." Sure, when building products with LLMs, good evaluations matter. But when comparing models, questions of preference are often more valuable than multi-choice benchmarks. Efforts like the OpenLLM leaderboard are interesting, but do they really tell us much about these models? With closed models, at least, the mystery adds intrigue.

Benchmarks are fun, but in the end, the true test of a model is how well it performs in the real world. That's what matters.

Source ↩
See this and this. ↩

RAG tricks from the trenches

2024-12-30T00:00:00+01:00

Some context
Our data: A controversial dataset
A stupid simple vector store
Plain RAG
Complicating things: Hybrid Search
What if we ask for a summary?
- Detecting representative questions
- Answering questions that require the entire dataset in context
Expanding user queries
The most important Lesson
Resources to go beyond the basics

Notebook download - nbsanity preview

Some context

We had a database of 50M strings and I couldn't wait to embed them all. Embeddings for recommender systems were, for a long time, the HOLY grail I longed for. I still remember, long before the RAG rage, having conversations with Pedro about the amazing types of applications we could build with embeddings.

In a way - RAG has come a long way. From another side: but RAG is still the same. It's just search! Over embeddings.

For the past 2/3 years, the number of applications doing some sort of RAG has increased significantly. I've learned a trick or two over that time.

Here are some tricks I've learned along the way in hope someone else can benefit from them as well.

Important notice: Almost none of these are mine. And that's why I like them.

# !pip install polars jupyter_black fasttext huggingface_hub lancedb litellm sklearn sentence_transformers

import polars as pl
import jupyter_black
import fasttext
from huggingface_hub import hf_hub_download
import lancedb
from lancedb.pydantic import LanceModel, Vector
from lancedb.embeddings import get_registry
from datetime import datetime
from litellm import completion
from sklearn.cluster import KMeans
import random
import json

pl.set_random_seed(42)
jupyter_black.load()

# CONFIGURATION

EMBEDDING_MODEL = "all-MiniLM-L6-v2"
LLM = "ollama_chat/llama3.1"
TABLE_NAME = "skeets"
DEVICE = "mps"

Our data: A controversial dataset

Remember that guy that got pretty much banned from BlueSky for collecting a dataset of 1 Million skeets? Well, our dataset is a nice collection of not 1, but 2! 2 Million BlueSky posts. Isn't that controversial?

I'm really enjoying Bluesky by the way. You should follow me there in case you come across this!

Alright, let's load the dataset using Polars:

df = (
    pl.scan_ndjson("hf://datasets/alpindale/two-million-bluesky-posts/*.jsonl")
    .filter(pl.col("reply_to").is_null())
    .collect()
)
df = df.with_columns(pl.col("created_at").cast(pl.Datetime))
df = df.with_columns(pl.col("created_at").dt.strftime("%Y-%m").alias("month_year"))
df = df.with_columns(pl.col("created_at").dt.strftime("%Y").alias("year"))
df = df.sample(50_000)  # so that we run fast.
print(f"{df.shape=}")
print("Example skeets:")
df.sample(2).to_dicts()

df.shape=(50000, 8)
Example skeets:
[{'text': "The last time I was back in the U.S. to visit family for Thanksgiving was in 2008, shortly after Obama was elected. People were elated and optimistic. The future looked bright.\nToday it could hardly be darker. That's what the MAGAs have brought us: chaotic dictatorship.\nI'll stay home in Canada.",
  'created_at': datetime.datetime(2024, 11, 27, 15, 30, 22, 133000),
  'author': 'did:plc:cnxppns7qxo2wzdlutmxbkih',
  'uri': 'at://did:plc:cnxppns7qxo2wzdlutmxbkih/app.bsky.feed.post/3lbwumhstsc2e',
  'has_images': False,
  'reply_to': None,
  'month_year': '2024-11',
  'year': '2024'},
 {'text': '9 Top CSS Essential Skills That Every Web designer Should Learn – http://bit.ly/vVyTg #CSS #webdesign',
  'created_at': datetime.datetime(2009, 10, 17, 17, 32, 37),
  'author': 'did:plc:id2rbt2kysdnscdrzh5o5n7d',
  'uri': 'at://did:plc:id2rbt2kysdnscdrzh5o5n7d/app.bsky.feed.post/3lbwmyaeat42n',
  'has_images': False,
  'reply_to': None,
  'month_year': '2009-10',
  'year': '2009'}]

Without much surprise, most of our posts (88%) are from November '24.

min_date = df["created_at"].min()
max_date = df["created_at"].max()
total_months = (max_date.year - min_date.year) * 12 + (max_date.month - min_date.month)

print(f"min_date: {min_date}, max_date: {max_date}")
print(f"total_months: {total_months}")
df.get_column("month_year").value_counts(normalize=True).sort(
    "month_year", descending=True
).head(10)

min_date: 2007-10-19 21:50:33, max_date: 2024-11-28 05:08:13.636000
total_months: 205

month_year	proportion
2024-11	0.87644
2024-10	0.00028
2024-09	0.00028
2024-08	0.00038
2024-07	0.00072
2024-06	0.00032
2024-05	0.00024
2024-04	0.00036
2024-03	0.00032
2024-02	0.00054

Even though the dataset mentions to have a configuration with the language of each skeet, I didn't manage to find it.

But to keep things simple, let's use the same model they did (glotlid) and keep skeets in English only.

model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin")
model = fasttext.load_model(model_path)
text_list = df["text"].to_list()
text_list = [t.replace("\n", " ") for t in text_list]
predictions, confidences = model.predict(text_list)
is_english = [p[0] == "__label__eng_Latn" for p in predictions]
df = df.with_columns(pl.Series(is_english, dtype=pl.UInt8).alias("is_english"))

print(f"total rows: {df.shape[0]}")
df = df.filter(pl.col("is_english") == 1)
print(f"total rows after filtering: {df.shape[0]}")

total rows: 50000
total rows after filtering: 22528

A stupid simple vector store

Lancedb provides a great interface that combines sentence-transformers + pydantic.

db = lancedb.connect("/tmp/db")
model = (
    get_registry()
    .get("sentence-transformers")
    .create(name=EMBEDDING_MODEL, device=DEVICE)
)

class Skeet(LanceModel):
    text: str = model.SourceField()
    vector: Vector(model.ndims()) = model.VectorField()
    author: str
    created_at: datetime

if TABLE_NAME in db.table_names():
    db.drop_table(TABLE_NAME)
    print("Table was deleted and will be overwritten.")

table = db.create_table(TABLE_NAME, schema=Skeet)

# this will automatically add and embed the items.

table.add(df.to_dicts())
table.create_fts_index("text", replace=True)

Plain RAG

In it's most pure form. Retrieval Augmented Generation has 3 simple components:

Retrieve context (e.g., search)
Build prompt
Answer question

def search(question: str, top_k: int = 10) -> list[Skeet]:
    return table.search(question).limit(top_k).to_pydantic(Skeet)

def build_prompt(question: str, context: list[Skeet]) -> str:
    context_str = ""
    for c in context:
        context_str += f"- {c.text}\n======\n"

    PROMPT = f"""
* You are an expert at answering questions from the user given a context.
* Use the context section to answer the questions from the user.
* Answer the question directly. No BS.

<context>
{context_str}
</context>

<question>
{question}
</question>
""".strip()

    return PROMPT

def llm(prompt: str, model: str = LLM) -> str:
    response = completion(
        model=model,
        messages=[{"content": prompt, "role": "user"}],
    )
    return response["choices"][0]["message"]["content"]

question = "What are the main reasons people are switching from twitter/X to bluesky?"
context = search(question)
prompt = build_prompt(question, context)
response = llm(prompt)

print("Question: ", question)
print(f"Answer: \n'{response}'")

# print(f"Context chunks:\n{context}")

Question:  What are the main reasons people are switching from twitter/X to bluesky?
Answer:
'Based on the context, the main reasons people are switching from Twitter/X to Bluesky seem to be:

1. A desire for a more focused and curated experience, as mentioned in "Trying to be way more active here than Twitter..." and "My favorite thing about Twitter was looking at 'What's Trending'... Does that exist in Bluesky?"
2. Frustration with the chaos and noise on Twitter/X, as implied by "I feel like I’ve been neglecting Bluesky because I been on threads and it’s no different from IG but with Twitter capabilities .. 🤦🏾‍♂️" and "Everything all of the time is not what I ever asked for."
3. A sense of nostalgia for a more open and community-driven platform, as suggested by comparing Bluesky to Netscape in its early days.
4. A desire to support a more decentralized and less commercial social media platform, as implied by "i like to think of bluesky(tm) as like a netscape . wherein they end up spitting out something cool and popular and open (and - commercial) but do not really find long term business success from that."'

Great, that works. Now, what can we do more?

Complicating things: Hybrid Search

LanceDB also provides a nice hybrid search option - allowing us to use full text search in combination with semantic search. In the default configuration, it will weight vector similarity and full text search around 70-30.

All we need to do is add query_type='hybrid'

def hybrid_search(question: str, top_k: int = 10) -> list[Skeet]:
    return table.search(question, query_type="hybrid").limit(top_k).to_pydantic(Skeet)

question = "What are the main reasons people are switching from twitter/X to bluesky?"
context = hybrid_search(question)
prompt = build_prompt(question, context)
response = llm(prompt)

print("Question: ", question)
print(f"Answer: \n'{response}'")

Question:  What are the main reasons people are switching from twitter/X to bluesky?
Answer:
'Based on the context, the main reasons people are switching from Twitter/X to Bluesky include:

* The ability to see what's trending and have a better understanding of current events and public discourse
* Frustration with Twitter's content and moderation policies (as implied by the comment "it doesn't suck like x")
* A desire for more nuanced and complex discussions, as evidenced by the quote about people being more complex than we want to assume
* The presence of internet trolls on Twitter, who are also migrating to Bluesky

These factors seem to be driving users to leave Twitter/X and join Bluesky.'

Now, keep in mind. There are already at least 2 parameters we would need to tune and evaluate here. (1) the number of chunks we want to stuff in the context (here, top_k), and (2) the type of search we would like to conduct (hybrid, full-text, vector only).

But if that was not complicated enough, we could complicate a bit more!

What if we ask for a summary?

Let's look at the following question:

question = "Give me a high level summary of the skeets"

If we take the naive approach of simply embedding this query, we are likely going to retrieve context with skeets that are similar to our question. At the extreme, this will surface skeets that contain the words high level summary and skeets.

Now, this might be what we want, but likely isn't. What we want in this case is to provide more context to the LLM regarding topics that align with the overarching content of the skeets, and then let the LLM itself infer the high-level summary from the context.

The way I think about this is the notion of filters. Here's a quick drawing that illustrates the concept:

First of all, how do we detect if the user asks such a question?

Detecting representative questions

The most basic approach to detecting "representative" questions, is to use an LLM itself to handle them:

def get_intent(question: str):
    INTENT_PROMPT = f"""
    Tell me which filter we should use for the given user question.
    The keyword filter filters data based on keywords from the question. This is good for specific questions or when you want to focus on a particular topic.
    The representative filter returns a representative sample of insights, from which you can infer the answer to the question. This is good for questions that relate to the entire dataset (trends, summaries, etc)
    Respond only with the type of filter to use!

    Examples:
    'Summarize the data' -> Representative filter
    'What are the main insights?' -> Representative filter
    'What are the high-level discussion points from our field reps regarding NOS ?' -> Keyword filter
    'Describe the negative, neutral and positive perception of physicians for Benuron?' -> Keyword filter
    'What are the main trends from the following skeets?' -> Representative filter
    'What are the main trends?' -> Representative filter
    'What are the key tweets I should be aware of?' -> Representative filter

    Question: {question}
    """.strip()
    return llm(INTENT_PROMPT)

for question in [
    "Summarize the skeets",
    "What are the main trends in this dataset?",
    "What's the current weather in Lisbon?",
    "Why are people leaving Twitter/X?",
    question,
]:
    print(f"'{question}'", get_intent(question))

'Summarize the skeets' Representative filter
'What are the main trends in this dataset?' Representative filter
'What's the current weather in Lisbon?' Keyword filter
'Why are people leaving Twitter/X?' Keyword filter
'Give me a high level summary of the skeets' Representative filter

The LLM manages to understand if we need a general overview or a specific context quite well.

Answering questions that require the entire dataset in context

There's an interesting article that dives a bit more into this topic here. One idea I stole from there is the idea of passing a 'representative' sample of our dataset to the context.

Now how can we build a representative sample? Here's one idea:

The user asks a general question that requires the whole dataset in context
We know we can stuff a maximum of Z chunks into the context
We retrieve a large number of embeddings from our dataset at random
We run K-means clustering or something more fancy to cluster those embeddings into N clusters/topics
From each of those topics we draw n items so that n * N ~= Z
We stuff those into the context

question

'Give me a high level summary of the skeets'

def get_representative_context(
    max_items_context: int = 100,  # max number of items to stuff in the context
    n_clusters: int = 50,  # number of clusters to create
    n_items_initial_retrieval: int = 100_000,  # number of items to retrieve from the database
) -> list[Skeet]:
    """
    Retrieves a representative sample of items from the database to be used as context.

    Args:
        max_items_context (int): The maximum number of items to include in the context.
        n_clusters (int): The number of clusters to create.
        n_items_initial_retrieval (int): The number of items to retrieve from the database.

    Returns:
        list[Skeet]: A list of items from the database.
    """

    items_per_cluster = int(max_items_context / n_clusters)
    print(f"{items_per_cluster=}")

    results_list = table.to_pandas()

    if n_items_initial_retrieval < len(results_list):
        results_list = results_list.sample(n_items_initial_retrieval).to_dict("records")

    else:
        results_list = results_list.to_dict("records")

    results_list = [Skeet.model_validate(_) for _ in results_list]

    kmeans = KMeans(n_clusters=n_clusters, random_state=42)
    embeddings = [_.vector for _ in results_list]
    kmeans.fit(embeddings)

    cluster_labels = kmeans.labels_

    context_items = []

    for cluster_id in list(set(cluster_labels)):
        cluster_items = [
            item
            for index, item in enumerate(results_list)
            if cluster_labels[index] == cluster_id
        ]
        assert len(cluster_items) == list(cluster_labels).count(cluster_id)
        context_items.extend(random.choices(cluster_items, k=items_per_cluster))

    return context_items

context = get_representative_context()
print(llm(build_prompt(question, context)))

Based on the provided context, it appears that there are various "skeets" ( likely referring to Twitter threads or short conversations) scattered throughout the text. However, without further information, I can only provide a general high-level summary:

The skeets seem to be a collection of miscellaneous conversations and comments from various individuals on topics such as:

* Music and celebrities (e.g., Drake, Kendrick)
* Gaming and entertainment (e.g., League of Legends, Arcane)
* Politics and social issues (e.g., Trump, immigration)
* Pop culture references (e.g., references to movies, TV shows, or memes)
* Personal experiences and anecdotes (e.g., someone's spa date for their John Deere tractor)

These conversations seem to be a mix of humorous, sarcastic, and serious discussions. If you'd like me to provide more specific information about any particular skeet, please let me know!

There are a lot of ways we could make this better. But this is the basic idea.

Expanding user queries

Another interesting concept is the concept of query expansion. I read this one in the LLM Engineer's Handbook. It's a great book - you should read it to!

In this case we use an LLM to expand the user query in hope of retrieving even more relevant items into our context. Here's how it works:

def expand_query_for(question: str, expand_to_n: int = 5) -> list[str]:

    # could be much better by using json mode, structured outputs, or any of these: https://simmering.dev/blog/structured_output/
    EXPAND_QUERY_PROMPT = f"""
* You are an AI language model assistant.
* Your task is to generate {expand_to_n} different versions of the given user question to retrieve relevant documents from a vector database.
* By generating multiple perspectives on the user question, your goal is to help the user overcome some of the limitations of the distance-based similarity search.
* Return a json string with a 'questions' key, which is a list of strings. It should be parseable by json.loads in Python.
* IMPORTANT: Do not include ```json or any other text in the response.

Original question:

'{question}'""".strip()

    response = llm(EXPAND_QUERY_PROMPT)
    questions = json.loads(response)["questions"] + [question]
    return questions

question = "What are the main reasons people are switching from twitter/X to bluesky?"
expanded_questions = expand_query_for(question)
for q in expanded_questions:
    print(f"- '{q}'")

- 'Why did users leave Twitter and join Bluesky?'
- 'What factors contribute to the migration of users from X (formerly Twitter) to Bluesky?'
- 'Identify the key drivers behind the shift from Twitter to Bluesky, based on user behavior and preferences'
- 'What are the primary reasons users are abandoning their old social media accounts on Twitter/X and joining a new platform like Bluesky?'
- 'How does the user experience and feature set of Bluesky differ from that of Twitter/X, leading to increased adoption?'
- 'What are the main reasons people are switching from twitter/X to bluesky?'

We can now take these questions, do a retrieval for each one of them, and answer the question.

context = []
for q in expanded_questions:
    context.extend(search(q))

print(f"Question: {question}")
print(f"Answer:\n{llm(build_prompt(question, context))}")

Question: What are the main reasons people are switching from twitter/X to bluesky?
Answer:
Based on the context provided, the main reasons people are switching from Twitter/X to Bluesky include:

1. **Frustration with Twitter/X**: Many users express their dissatisfaction with the current state of Twitter/X, mentioning issues like transphobia, rampant engagement farming, and the platform's overall toxic environment.
2. **Desire for a more positive experience**: Users are seeking an alternative where they can engage with others in a more constructive and respectful manner. This is evident from statements like "it doesn't suck like x (twitter)" and "y'all are lovely."
3. **Wish to preserve the original spirit of Twitter**: Some users nostalgically recall the early days of Twitter, when it was more open, popular, and user-friendly. They hope Bluesky will emulate this spirit.
4. **Attractiveness of new features or improvements**: Although not explicitly stated, some users might be drawn to specific features or improvements that Bluesky offers compared to Twitter/X.

Overall, the main reasons people are switching from Twitter/X to Bluesky seem to be a combination of dissatisfaction with the current state of the former and a desire for a better alternative.

The most important Lesson

These are all fancy and nice. But make sure to always start with a baseline. A simple one.

Resources to go beyond the basics

An ode to the Marathon

2024-12-17T00:00:00+01:00

I'm flying over the Atlantic just between France and Spain for my yearly Christmas trip back to Lisbon. For some reason, I just remembered the blog post I always wanted to write. This one is not about tech, but it's about something I'm equally passionate about: The Marathon.

Funnily - I'm flying pretty close from where my last battle went down: right off the beautiful Cote d'Azure, about 1 month ago, for the Nice Marathon, my seventh in ~3 years.

And if that makes you think that Duarte is probably addicted. You're probably right. But I would argue I'm not addicted to the Marathon itself. I'm addicted to the process.

Let me explain.

"What's your shoe rotation?"
"What gear should I buy?"
"Should I join a run club?"

All of these miss the point of running completely. Running is a solo sport. And a marathon prep is a tough mental challenge to do well. "I have to work", "I won't make it this time", "I'm feeling tired today". These are just a small taste of the challenge you are really up against: yourself.

Why would you even? Why would you get up when it's dark and wet outside? Why would you go out for a run when it's the 17th of December on the windy coastline of Denmark? Because you know what lies ahead.

After those dark and lonely training months the day finally gets here. Most of us have a superstitious pre-race routine. Something that helps us re-gain the mental strength that things will go according to plan (spoiler alert: They never do). Mine is pretty simple: (1) lay the kit in the ground near the bed, and (2) make sure you have half a dose of Banana chocolate chip Gatosport ready for next morning.

Nerves are cracking. Thousands of people lined up with you. If you look closely you can tell: the more to the front people are, the more nervous they are. This is the moment you have been waiting for. There's music, there's announcements, there's noise. A lot of people dance and get excited, I can't bear myself to. The nerves are just too much.

The start of a Marathon gives me of a feeling I only got in Highschool, when the teacher was walking around the classroom, grading everyone one by one. I was always nervous waiting to hear if I had gotten more than a 10 out of 20.

All this bullshit, all this music, announcements, all of this dancing around the fire. Now we're going. Some will say the inspiration comes from within. For the first two-thirds of a Marathon, I often find myself looking sideways. Not because I'm comparing myself to them. But because I can CLEARLY see it! I can see the exact same pain I'm experiencing. It's a rare moment to be surrounded by a group of people all experiencing the same painful experience at the exact same time. You don't know why they are suffering, why they decided to to this, where they are from, or what they're called - but you know what they're thinking. The same as you.

Now you've done the hardest part. You've run for 30 Km. You've trained for this: You're ready. Just some of the things I wish I told myself right around that section. The actual thoughts are a mix of "I don't want to do this", "I should've trained harder", "I can walk a bit - what's wrong with that", "I will never run a Marathon again". Welcome to the wall.

Even having run a 'respectable' amount of Marathons, the feeling never goes away. Hopefully, you'll plow through. Even if you do start slowing down, even if you lose the pacer or group, you'll keep going.

"You've already ran 40K! You're not going to stop now!" I often remember that American lady shouting at me from the sideline in Vienna some years ago. She was most definitely right.

Yes, those 2 Km now feel like 200. Yes, your legs are tired. But you can see it! You can see that finish line. You made it! Weirdly, you can sprint now; all the pain goes away. It's the culmination of months of effort and hours of pain. This is where you'll probably shed a tear and celebrate with your loved ones.

Personal-best or not, perfect-race or not. You've done more than 99% of people that day. And that's something to be proud of.

Now comes the hangover. For some it's a week, for some it's 6 months, for others, it lasts years. No running. No exercise. You've done your part. You've done well. But then… Slowly… You start thinking about it. "It can't be always that painful", "There are at least 2 or 3 things I could've done better!", "I'm running already, I should probably do a race..."

And that's when you book your next one.

Upgrading this website's podcast with F5-TTS

2024-11-12T00:00:00+01:00

For the past year, this website's podcast companion has been running on a text-to-speech model called XTTS-v2. It's not horrible. And those who have heard my voice before might notice some similarities. But it's far from a pleasant listenning experience.

But the world of text-to-speech (TTS) has been gradually moving along. NotebookLM made headlines, and everyone is sure the future of AI will sit somewhere between agents (whatever those are) and text-to-speech. NotebookLM is an awesome product. But it's still, at its core, a closed source product¹.

Unfortunately, for state of the art text-to-speech there is still a single undisputed champion out there: ElevenLabs. Or should I say, there was?

Late this year a new model from Microsoft came out, called E2-TTS. Sometime after, F5-TTS was released - an improvement over E2 focued on inference and optimization. I was amazed by the demos - they sounded crisp and natural. After browsing the code for a little bit, it seemed like an easy thing to test out.

I tested the engine with one of my latest posts:

XTTS-v2

F5-TTS

I mean, would you just hear that. No more weird accelerations and squeeky voices. The new model is just much more stable and reliable than the previous one. It's light day and night! It's still not perfect though. One might argue the voice gets a bit too intense some times. The problem with prounounciation persists. It still doesn't know how to say my name properly - but then again - most people also can't.

I tried getting around the prounounciation by specifying the phoneme sequence², but that didn't seem to change much. It did give me a good idea to preprocess the audio with an LLM - and that also makes things smoother.

Running F5-TTS was also trivial thanks to Modal's great product (still haven't paid a penny I must say, but would happily). Only this specific function needs to run on a GPU, and it does so just by using Modal's decorator:

@app.function(
    gpu=MODAL_GPU,
    mounts=[
        modal.Mount.from_local_dir(
            LOCAL_DATA_DIR, remote_path=MODAL_REMOTE_DATA_DIR
        )
    ],
    timeout=400,
)
def transcribe(
    article: ParsedArticle,
    reference_voice_file: str = REFERENCE_VOICE,
    reference_text_file: str = REFERENCE_TEXT,
    model_name: str = MODEL_NAME,
) -> bytes:
    with open(reference_text_file, "r") as voice_text:
        reference_text = voice_text.read()

    command = [
        "f5-tts_infer-cli",
        "--model",
        model_name,
        "--ref_audio",
        reference_voice_file,
        "--ref_text",
        reference_text,
        "--gen_text",
        article.text_for_tts,
    ]

    result = subprocess.run(command, check=True)
    assert result.returncode == 0, f"Error: {result.stderr}, \n{result.stdout}"

    target_file = "tests/infer_cli_out.wav"
    assert Path(target_file).exists(), f"File {target_file} does not exist."

    mp3_file = convert_to_mp3(target_file)

    with open(mp3_file, "rb") as audio_file:
        audio_bytes = audio_file.read()

    return audio_bytes

If you're interested in thre rest of the code, Podcaster is free and open source.

There are some open source implementations the core tech is still very much closed source. ↩
See the demo page for details ↩

What if ChatGPT's memory was a knowledge graph?

2024-10-29T00:00:00+01:00

Classification in the age of LLMs: The emoji problem

2024-10-07T00:00:00+02:00

For the past years Vitto and I have used Tricount to track our shared expenses. The app is actually pretty good, but there’s one small thing that annoys me quite a bit.

Even though we spoke English to each other for the first month, we’ve since spoken a mix of Italian and Portuguese. And when I say a mix, I really mean a mix. I mean, would you just look at our expense tracking app:

For the average calm and relaxed reader, this might not seem like much. But for us, other folk, it’s extremely annoying to see how it misses assigning the right emojis to most of our transactions! It gets Beers (🍺), it gets when we speak English and write something like “Train” (🚂). But whenever we speak something in Portuguese or Italian, BAM, there comes the generic 💶 emoji.

Now. Is this a big problem? No. Is it an annoying problem? Yes. Is this a solved problem? Probably. But in the age of LLMs and 400 Billion parameters models, what options do we have?

Creating a labeled dataset from screenshots
Classification using gpt-4o and structured outputs
Function calling local models with Ollama
Embeddings for classification
Benchmarking expense categorization
Closing thoughts
Epilogue: Just get the right token

Creating a labeled dataset from screenshots

Before looking at all the ways the Tricount emoji classification could be better, we should first gather a reasonable dataset. What before required some grueling hand labelling or some clunky OCR library, now ‘only’ requires a couple of calls to gpt-4o.

# Define data structure

class ExpenseItem(BaseModel):
    name: str
    amount: float
    currency: str
    payer: str
    emoji: str

class Expenses(BaseModel):
    expenses: List[ExpenseItem]

# Encode the images in base64

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")
images_content = [
    {
        "type": "image_url",
        "image_url": {"url": f"data:image/png;base64,{encode_image(ima)}"},
    }
    for ima in pathlib.Path("../tricount_pictures").glob("*.PNG")
]

# Classify

response = client.beta.chat.completions.parse(
    model=MODEL,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "List all the expenses in a json format",
                },
                *images_content,
            ],
        },
    ],
    temperature=0.0,
    response_format=Expenses,
)

expenses = response.choices[0].message.parsed

# prints

#[ExpenseItem(name='Beers', amount=XXX, currency='DKK', payer='Dudu (me)', emoji='🍺'),

# ExpenseItem(name='Beers', amount=XXX, currency='DKK', payer='Dudu (me)', emoji='🍺'),

# ExpenseItem(name='Decatlon comun', amount=XXX, currency='DKK', payer='Vitto', emoji='💶'),

Just liked that, we now have a dataset of ~50 expenses. Great. Let’s get to classifying.

Classification using `gpt-4o` and structured outputs

Before we do anything else, how do the big guns perform? What if we just ask gpt-4o to classify? This is the simplest of options, even simpler now that OpenAI directly supports Pydantic base models via structured outputs. We can solve this problem in a couple dozen lines of code:

# define pydantic model

class EmojiClassification(BaseModel):
    """
    A single emoji that describes a financial transaction (e.g., Airplane ticket -> ✈️)
    """

    emoji: str = Field(
        description="The emoji that describes the transaction (Must be a single character!)"
    )

    @field_validator("emoji")
    @classmethod
    def validate_emoji(cls, value):
        if not emoji.is_emoji(value):
            raise ValueError(f"The emoji must be a single character! Received: {value}")
        return value

# classification function

def classify_openai(expense: str):
    client = OpenAI()
    messages = [
        {
            "role": "user",
            "content": f"Classify this transaction: {expense}",
        }
    ]
    completion = client.beta.chat.completions.parse(
        model="gpt-4o-mini-2024-07-18",
        messages=messages,
        response_format=EmojiClassification,
    )
    return completion.choices[0].message.parsed.emoji

classify_openai("Treno Milano Ancona")

# prints '🚆'

That was easy. But also not particularly exciting. What if our data is sensitive? What if our API gets extremely popular? What if we just want to handle it ourselves? Could we do the same with some off-the-shelf open-source models?

Function calling local models with Ollama

I would love to tell you that open-source function calling was a solved problem. And yes - the function calling leaderboard certainly seems to tell that story. But the reality of local models is a bit different.

Here’s the code to accomplish the same thing OpenAI models can do with an Ollama model that supports function calling:

# Eats pydantic model, shoots 'json schema'

def schema(f) -> dict:
    kw = {
        n: (o.annotation, ... if o.default == Parameter.empty else o.default)
        for n, o in inspect.signature(f).parameters.items()
    }
    s = create_model(f"Input for `{f.__name__}`", **kw).schema()

    return {
        "type": "function",
        "function": {
            "name": f.__name__,
            "description": f.__doc__,
            "parameters": s,
        },
    }

# Don't even try without this (email me if you have a better option!)

SYSTEM_PROMPT = """
You are a genius expert. Your task is to classify the description of a financial transaction with an emoji.
You must only use emojis that are a single character.
You must only reply with a single emoji.

Example 1:
User: Treno Milano Ancona
Response: EmojiClassification(emoji='🚆')

Example 2:
User: Dinner with friends
Response: EmojiClassification(emoji='🍽')

Example 3:
User: Plane ticket to New York
Response: EmojiClassification(emoji='✈️')

Use the EmojiClassification tool to help you classify the transactions.
""".strip()

# This is a bad example of retry logic - but you get the point

def classify_expense(
    expense_description: str,
    model: str,
    retries: int = 3,
    base_url: str,
    api_key: str,
) -> EmojiClassification:

    client = OpenAI(base_url=base_url, api_key=api_key)
    classification_prompt = f"User: {expense_description}"

        # poor man's retry
    for retry_num in range(retries + 1):
          # we use OpenAI's client
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": classification_prompt},
            ],
            tools=[schema(EmojiClassification)],
            temperature=0.0 + retry_num / 10,
        )

        tool_calls = getattr(response.choices[0].message, "tool_calls", None)
        if not tool_calls or tool_calls[0].function.name != "EmojiClassification":
            print(
                f"Failed to classify expense. Expected EmojiClassification but got {tool_calls[0].function.name if tool_calls else 'None'}"
            )
            continue
        try:
            return EmojiClassification(
                **json.loads(tool_calls[0].function.arguments)
            ).emoji
        except ValidationError as e:
            print(f"Validation error: {e}")
            continue
    return None

# we can now call this with ollama

classify_expense(
    expense_description="Donut",
    model="llama3.1",
    base_url="http://localhost:11434/v1",
    api_key="ollama", # dummy key - required
)

# prints '🍩'

Note: Using Instructor to accomplish the same

A kind user on twitter reached out mentioning you could do the same with Instructor I got a lot of Validation errors at the time, so I did not go this route, the code gets a lot simpler, so worth the test!

def classify_expense(
    expense_description: str,
    model: str,
    retries: int = 3,
    base_url: str = "http://localhost:11434/v1",
    api_key: str = "ollama",
) -> EmojiClassification:

    client = instructor.from_openai(
        OpenAI(
            base_url=base_url,
            api_key=api_key,
        ),
        mode=instructor.Mode.TOOLS_STRICT,
    )

    classification_prompt = f"User: {expense_description}"

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": classification_prompt},
        ],
        response_model=EmojiClassification,
        max_retries=retries,
    )

    return response

Perfect, we can now use any local Ollama model that supports function calling and classify our expense description as well.

Why don’t we try some old-school machine learning?

Embeddings for classification

Embeddings are another option. Open source, fast, and lightweight. Here’s a possible logic: given a list of emoji and their descriptions, as well as the description of an expense, return the ‘most similar’ emoji description to the expense.

Easier in code than in English:

# load a small db of emojis and descriptions

df = pandas.read_csv("./emojis.csv", index_col=False, header=None)
df.columns = ["emoji", "description"]

# prepare column to be embedded (I hate this)

df["description"] = df["description"].str.replace('"', "")
df["to_embed"] = "passage: " + df["description"]

# embed it (multilingual embedding, since the description are in IT and PT)

model = sentence_transformers.SentenceTransformer(
    "intfloat/multilingual-e5-base"
)
embeddings = model.encode(df["to_embed"].tolist(), show_progress_bar=True, device="mps")
df = df.assign(vector=embeddings.tolist())

# create a lancedb table with the data

db = lancedb.connect(".my_db")
table_name = "emojis"
table = db.create_table(table_name, data=df.to_dict(orient="records"))
print(f"Created table {table_name}")

# Why not try a re-ranker as well?

ranker = Reranker("nreimers/mmarco-mMiniLMv2-L12-H384-v1")

# Create a classification function

def embedding_classify(
    expense_description: str,
    top_k: int = 10, # number of closest emoji to re-rank
    use_reranker: bool = True,
    debug: bool = False,
) -> str:
    query_embedding = model.encode(
        f"query: {expense_description}", show_progress_bar=True, device="mps"
    )
    r = table.search(query_embedding).limit(top_k).to_pandas()

    if not use_reranker:
        return r.iloc[0]["emoji"]

    r = ranker.rank(
        query=expense_description,
        docs=r["description"].tolist(),
        doc_ids=r["emoji"].tolist(),
    )
    return r.top_k(1)[-1].doc_id

print(embedding_classify("Beers and pizza with friends", use_reranker=False))

# prints 🍕

print(embedding_classify("Almoco", use_reranker=False))

# prints ⚗

Great, both the vanilla and re-ranked version of the function appear to work correctly. We now at least three ways of classifying the description of an expense.

Benchmarking expense categorization

We now have all the functions we need. With the code below, we now run a simple benchmark. For the 50 descriptions of expenses we gathered for our dataset, we run a set of different techniques/models.

I didn’t test all models in the world. I tested gpt-4o, 3-4 local models I usually run on my laptop, and the embedding techniques. Here’s the benchmarking code and results:

CACHE = {}

def benchmark(expense: ExpenseItem) -> dict:

    if expense.name in CACHE:
        return CACHE[expense.name]

    result = {
        "description": expense.name,
        "tricount_emoji": expense.emoji,
        "openai": classify_openai(expense.name),
        "llama3.1_7b: ": classify_expense(
            expense.name,
            model="llama3.1",
            **api_params,
        ),
        "qwen2.5_3b": classify_expense(
            expense.name,
            model="qwen2.5:3b",
            **api_params,
        ),
        "qwen2.5_1.5b": classify_expense(
            expense.name,
            model="qwen2.5:3b",
            **api_params,
        ),
        "llama3.2_3b": classify_expense(
            expense.name,
            model="llama3.2",
            **api_params,
        ),
        "embedding": embedding_classify(expense.name, use_reranker=False),
        "embedding_reranker": embedding_classify(expense.name, use_reranker=True),
    }

    CACHE[expense.name] = result
    return result

results = [benchmark(expense) for expense in expenses.expenses[:30]]
bench_df = pandas.DataFrame(results)
bench_df

description	tricount_emoji	openai	llama3.1_7b	qwen2.5_3b	qwen2.5_1.5b	llama3.2_3b	embedding	embedding_reranker
Beers	🍺	🍺	🍺	🍺	🍺	🍺	🍺	🍻
Beers	🍺	🍺	🍺	🍺	🍺	🍺	🍺	🍻
Decatlon comun	💶	🛒	🏃	🛍	🛍	None	🌳	📨
Duarte decatlon	💶	🏃‍♂️	🏃	🛍	🛍	🏦	🌳	🌳
Festa	💶	🎉	🎉	🎉	🎉	None	🎭	🎑
Jantar	💶	🍽️	None	👩‍🍳	👩‍🍳	🏯	⚱	⚱
Almoco	💶	🍽️	🍏	🍔	🍔	None	⚗	⚗
Jantar	💶	🍽️	None	👩‍🍳	👩‍🍳	🏯	⚱	⚱
Hotel Bolo	🏨	🏨	🏨	🏨	🏨	None	🏨	🏚
Treno Milano Ancona	💶	🚆	🚂	🚆	🚆	None	🚄	🚡
Cena vigana milano	💶	🍽️	🍴	None	None	None	🌖	🌐
Train venezia milano	🚂	🚆	🚂	🚆	🚆	🚂	🚄	🚆
Reimbursement	💳	💸	💸	⛺	⛺	None	🚑	🎁
Spritz	💶	🍹	🍹	None	None	None	🥨	✨
Mozzarella	💶	🧀	🧀	🧀	🧀	None	🍝	🍄
Spritz	💶	🍹	🍹	None	None	None	🥨	✨
Barca	💶	⚽	⚽	🏀	🏀	🚣	🔖	📶
Kwikly	💶	🛒	💳	🏃	🏃	None	🥝	🥝
Spesa	💶	💳	✅	💰	💰	None	♠	🌾
Voli Tokyo	💶	✈️	💖	✈️	✈️	✂	🗼	🗼
Conguaglio voli Zurigo	💶	✈️	✈️	✈️	✈️	✈	🔄	🌐
Amar	💶	👤	💸	None	None	🤑	🕉	💢
Groceries	🛒	🛒	🛍	None	None	🛍	🥫	🍲
Durum	💶	🍞	🥖	🍞	None	📦	🥁	🛢
Groceries	🛒	🛒	🛍	None	None	🛍	🥫	🍲
Money transfer	💳	💸	💸	💸	💸	None	💰	🤑
Lunch	🍜	🍔	🍏	None	None	🍽	🥪	🕐
Cafe	💶	☕	☕	☕	☕	☕	☕	🔪
Spesa kwikly	💶	🛒	✅	🛒	🛒	None	🐳	💪
Pizza	🍕	🍕	🍕	🍕	🍕	None	🍕	🍕

Note: None means the model did not call the appropriate function, and thus failed to produce a classification.

Closing thoughts

Looking at the results, we can observe a couple of interesting things:

OpenAI’s gpt4-o is hard to beat. It pretty much never missed. Even when Vitto was too lazy to write the rest of the description. My guilty pleasure shopping at decathlon? Got it. Barca was actually referring to boat - and not the actual football club - there’s no way it could’ve figured that one out. Hard to beat. Kwikly refers to a local supermarket we go to in Copenhagen - and that looks like a store emoji to me!

The bigger the open model, the better the results. And Llama 3 shows it. It performs pretty well, especially at the 7b size. Not without hiccups, of course (can’t figure out how 💖 relates to flights?).

Small open source models are getting good. Not sure what happened to Llama 3.2 3b - but at 3 and 1.5b sizes, Alibaba’s Qwen2 family of models perform surprisingly well. Some bigger problems do show up. We also see a lot of failures to call the right function. (hence the Nones). I’m still curious about two things. (1) How much better could we get through prompting. (2) How much better larger open source models perform (think 70B or 400B). I can’t run those myself, so it’s beside the point of this post.

The embedding idea sounded interesting, but didn’t really perform. We would probably have to do a bit more work than just a cosine similarity on the emoji descriptions. Even though the embeddings were multilingual, they clearly didn’t capture the essence of the expense. Another approach I would test is to create a labeled dataset with gpt-4o and then a simple random forest on embeddings. But again, beside the point of this post.

In any case, we can for sure do better than what Tricount is currently doing!

Epilogue: Just get the right token

Sorry, I couldn’t resist. I need to test a last one. OpenAI has a cool feature called log probs in their API that allows you to run classification by querying by which token has the highest probability in the completion.

Unfortunately, Ollama doesn’t seem to support this. But I figured how to do with plain Transformers. Not the most efficient, since I have to run for every possible emoji. But do let me know if you (reader) have better suggestions:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "meta-llama/Llama-3.2-1B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# log probability of sentence

def calculate_sentence_log_probability(sentence):
    inputs = tokenizer(sentence, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)

    logits = outputs.logits[:, :-1, :]  # Align token predictions
    shifted_input_ids = inputs.input_ids[:, 1:]  # Drop first token

    log_probs = torch.log_softmax(logits, dim=-1)
    token_log_probs = log_probs.gather(2, shifted_input_ids.unsqueeze(-1)).squeeze(-1)

    return token_log_probs.sum().item()

# wrapper to run through the emojis

def find_most_likely_emoji(base_sentence, emoji_list):
    log_probs = {}
    for emoji in emoji_list:
        sentence = base_sentence.format(emoji=emoji)
        log_probs[emoji] = calculate_sentence_log_probability(sentence)

    most_likely_emoji = max(log_probs, key=log_probs.get)
    return most_likely_emoji, log_probs

# prompt

USER_MESSAGE = """
You are a genius expert. Your task is to classify the description of a financial transaction with an emoji.
You must only use emojis that are a single character.
You must only reply with a single emoji.

Example 1:
User: Treno Milano Ancona
Response: 🚆
...

Now it's your turn! Classify the following transactions with an emoji:
User: {description}
""".strip()
USER_MESSAGE = USER_MESSAGE.format(description="Voli Tokyo") # 'Flights Tokyo'

# massaging the formats and tokenizing

ASSISTANT_MESSAGE = """Response: {emoji}"""
chat = [
    {"role": "user", "content": USER_MESSAGE},
    {"role": "assistant", "content": ASSISTANT_MESSAGE},
]
input_tokens = tokenizer.apply_chat_template(chat, tokenize=True)[:-1]
base_sentence = tokenizer.decode(input_tokens)

# classification

emoji_list = ["🎄", "🔔", "🚨", "💎", "🇬🇹", "🍹", "🚣", "⚽", "🥘", "🛩️", "🇯🇵", "✈️"] # example possibilities
most_likely_emoji, log_probs = find_most_likely_emoji(base_sentence, emoji_list)
print(f"Most likely emoji: {most_likely_emoji}")
for emoji, log_prob in log_probs.items():
    print(f"Log probability of {emoji}: {log_prob}")

# Most likely emoji: ✈️

# Log probability of 🎄: -582.9215087890625

# Log probability of 🔔: -580.827392578125

# Log probability of 🚨: -577.4615478515625

# Log probability of 💎: -582.4413452148438

# Log probability of 🇬🇹: -580.4923095703125

# Log probability of 🍹: -581.0596923828125

Works well!

Around Iceland in 6 days

2024-08-30T15:30:00+02:00

For a short summer break, Vitto challenged me to visit Iceland with her. As someone that is used to going south for the summer - this sounded stupid at first, but I was incredibly surprised. For 6 days, we were pretty much in another planet. Here are some notes and tips from our trip.

Minimal pre-trip preparations

We only did two things in preparation for the trip. The first was renting out our home for the week. After some googling we decided to go with Kuku Campers. It’s a ‘local’ shop, ran mostly by summer working EU immigrants. Price was reasonable, staff was friendly. No complaints.

The second most important preparation is the most critical. It’s a small trick I learned 4 years ago from my friend Alex. That little trick is called Le Petit Routard. We bought the Iceland 2024 Edition. As always, it did not disappoint. It’s just like traveling around with a local guide. And there’s just something about the French way of routard travel that has always stuck with me. You won’t know until you’ve tried it.

Day 1: Lift off & Golden Triangle

We woke up early in Reykjavik, had breakfast at the Center Hotels Plaza. Once done, we went back to the Airport region of Keflavik and picked up our camping van. We set way towards the first ‘coup the coeur’ from the Routard: the Golden Triangle Region.

We managed to visit the place where the two tectonic plates meet, the Þingvellir national park. Which brought me some good memories of Geological walks with my grandparents. Also managed to quickly hit the original Geysir (reminded me often of what we see in the Azores region). Except Icelandic people don’t cook underground like us.

Closed off Day 1 with our first campsite. Nothing special - pretty noisy.

Day 2: Running and Waterfalls

The marathon training didn’t necessarily stop. We took the early morning and decided to go for a long run along the woods near the Geyser region. We ended up getting lost and running 17km in the middle of the Lunar landscape. Beautiful.

The second part of the day was all about waterfalls. We drove to the final element of the Golden Triangle: the Gullfoss waterfall. Once done, we started making our way east while listening to a great podcast episode about Icelandic Geology.

On the road to Vik, two wonders: Seljalandsfoss is quite a sight. You can go around it too! Hestavaðsfoss was also impressive. Even though our legs were a bit cooked, we hiked a bit as well. Next time, we’ll do it a bit more in this region.

Tired, we finally reached the campsite in Vik. Some well deserved showers, some subpar tortellini and called it a night. Woke up in the middle of the night to a shaking van. Strong winds in the north.

Day 3: A beach full of ice

We woke up early to the sound of the wind. After some breakfast and packing, we drove along the east towards the Svínafellsjökull Glacier. A beautiful location where you park the van, walk ~4/5 km and see the beginning of where the glacier comes down from the mountain.

After another 45 min drive and we finally ended up at one of the highlights of the trip: Diamond Beach. It’s a fantastic scenery. It sits just close to there the Jökulsárlón lake touches the Ocean. The glacier melts into the lake, and the large blocks of ice are washed up into the Ocean, some of them end up in a beautiful black sanded beach, melting away. It’s hard to put into words, but it’s quite the sight.

Recklessly ignoring the weather warnings, we then drove along the Eastern coast. It was a scary drive, with very strong winds; between the mountain and the ocean - on a narrow road. Had we gotten a larger van, we would’ve probably stopped right then and there.

But with Vitto’s co-piloting, we managed to get into the Fossárdalur Campsite; probably the most beautiful campsite we stayed at. Hidden amongst the Fjords, sheltered from the wind. The kitchen area was super cozy, with electricity, many stoves, and some long tables to enjoy a warm meal.

Day 4: Volcanos and Thermal baths

We started heading north-west. It’s incredible how the landscape changes while you’re driving in Iceland. It’s truly a moonlike place. On our way to the Mývatn region, we stopped for in one of the strongest waterfalls in the world: the Dettifoss waterfall. You would’ve never guessed there’s a waterfall there! Especially that large. A bit of walking in a moonlike landscape and BAM! A gigantic waterfall.

A couple more Serial episodes and driving, and we finally reached the Mývatn lake region. The landscape looks like nothing we’ve ever seen before, it’s just like a Volcano erupted near a lake, and transformed its landscape in a Halong-bay like way.

During the afternoon, I challenged Vitto to a long run towards a volcano we spotted. At first, we thought we would just get near it, but once we got close enough, it was obvious some people were actually hiking to the top! We had to do the same. The view wasn’t mind-blowing, but the challenge was fun. There’s nothing quite like running as a couple.

After that long run/trail we were both pretty tired again. We set our GPS to the Mývatn Nature Baths. If you’ve heard of the Blue Lagoon, it’s similar - but a little less packed and touristy. Oh, and it was open until 22:00.

Once we had enough of the blue water, jacuzzi like temperature, and volcanic views, we decided to call it a night. We got to Camping Myvatn, another scenic campsite overlooking the lake. A little Barilla and pesto to fill up the belly, and we called it a night.

Day 5: Whales and Fjords

When I was young, I was pretty lucky to see the mythical Blue Whales in Açores. But Vitto had never seen any! We were told by many that Húsavík was the capital of whale watching. But we decided we were going to try our luck in the less known Akureyri.

We woke up early, packed up our stuff, and drove all the way to Akureyri (to catch the 10:00 AM tour). We had a great time, and saw at least 3/4 whales in different occasions. The tour was well organized, and the staff was calm and professional. The weather was also great, and the whales were out.

From Akureyri we did another long stretch of driving all the way to the North West region of Iceland. Supposedly less known and more remote. The roads were in less good shape with lots of gravel, but the van handled it.

Our goal was to reach the Flókalundur region. The views and road were absolutely mind-blowing. The MAJESTIC Fjords mixed with a Lisbon-like sunlight, is something I have never seen before.

Just before arriving to camp, we also discovered the little Hellalaugur hot spring. Just like a little stone jacuzzi in the middle of the rocks. “What language are you guys speaking?” - Asked a German family. “Oh. We speak a mix of Portuguese and Italian”; Always fun looking at their reaction. “We got some of the Italian but couldn’t decrypt the other one”.

We decided to indulge ourselves and eat at the Hotel/Campsite/Restaurant. The food was good and reasonably priced! The views from the van were stunning.

Day 6: Famous waterfalls and road to Reykjavík

We woke up in the middle of the Fjords and set out towards the Dynjandi falls. Arrived early but was already packed. Still, the Hæstahjallafoss falls were quite a site. For a moment, it felt like we were inside an aquarium.

Kept driving along the coast, and reached the Kirkjufellsfossar waterfall. It’s a nice little waterfall that faces a funny shaped mountain called Hálsaból Sumarhús.

And we kept driving. We drove most of the day actually. Even though we wanted to get close to Reykjavík, we didn’t want to go back just yet.

Somehow, probably through some Scandinavian friends, Vitto got a recommendation to hit the Hammsvik Hot Springs. These were even better than the previous ones we had visited. The weather was cloudy, but with the 4/5 pools near the Fjords’ bay, the scenery was also magical. The place closed at 22:00, and we left just before 21:55.

Vitto managed to find a small little camping farm called Hjalli Kjós, we had no big expectations for, but ended up being a nice surprise. Managed by a friendly elderly couple. Clean, cosy, shoes off in common areas. The perfect place to take a shower, cook the last portion of Barilla and hit the bunk for the night.

Day 7: Going back home

The last day was pretty uneventful. We woke up early again, took a shower, and drove down to Reykjavík. We dropped the van at KuKu, made sure everything was fine with it, and were dropped off at the Airport to fly home.

What we’ll do differently in the next trip

4x4 Camping Van. Our van was great. Spacious and well-equipped. We were two, and it could easily sleep 3. However, it was a bit too large and “urban”, and made some of the gravel roads a bit uncomfortable. We didn’t have much choice when booking. But next time, I would probably go with something like this instead.
Less Driving: We drove a lot. We wanted to do the complete tour of Iceland. But there were a couple of days when we drove 8+ hours. Even though we listen to nice podcasts, and had nice and long talks, it was a bit too much for our taste. Hey - there are no perfect trips.
More time: We had a full 7 days in Iceland. But looking back, I wish we had just a bit more time. I think 8-9 days would be the ideal. Yeah, sure, you could be a month in Iceland. I’m talking about the type of trip we like to do.
Less Running, More Hiking: Our trip was in the middle of summer, in a week in August. But we were also training for an upcoming marathon. So we had to squeeze in runs here in there. That was fun, and we did a lot of those together. But perhaps we would’ve enjoyed even more if we replaced some of those runs for hikes.

Vitto was right. When the summer hits, we, southern Europeans, have this tendency to go South. But there was something special about going North this time. As she said: 9 out of 10.

Tanda Runner: A personalized running dashboard

2024-07-22T17:55:00+02:00

Tanda Runner is a web app that shows me the things I care most about when preparing my next marathon. I've also added a running coach/agent designed to give me actionable feedback about my training. Some of that feedback is probably hallucinatory - I'll get to it in a bit.

It's built with Django - a framework I've gravitated to more and more when building these types of apps. It has all the batteries I need and a time-tested ecosystem. The front-end is a Frankenstein that uses Django channels, htmx, and some vanilla JavaScript. It’s not the most responsive PWA out there - but it's still a pretty usable experience. The chat interface is adapted from LLaMA.cpp.

The most interesting part is the small LLM coach/agent that gives feedback and tips regarding my performance. It's an untested and non-evaluated multi step LLM pipeline that extracts running-related insights from around the web, and uses them to give me personalized feedback using my Strava data.

To do this, I extract transcripts from marathon related YouTube videos (which I probably shouldn't), and use those to generate a list of insights related to running and training. Once I have those insights, I use an LLM to translate those them into Pandas code. Once that's done I can feed an LLM the insight, the code, and the outcome of running such code on my own data. Sounds confusing? I know it does - but it works surprisingly well!

This pipeline could probably be significantly improved by designing some evaluations, some metrics, and continuously improving it. But for now, I'll use it to prepare Nice, and make any adjustments I see fit.

The visualizations and graphs are built using Altair. They are a mix of my favorite graphs from Christoph’s CR Plots, Strava, and my own brain. The name (and most of the visualizations) come from Giovanni Tanda's work - which I encourage you to read through!

Do things that don't scale

2024-06-06T10:00:00+02:00

I'm lucky enough to work with some pretty talented folks. During a recent offsite, one of them gave a completely improvised talk. He had just gotten back home from holidays - so naturally, they didn't really have time to prepare anything. But what they said resonated.

It was about improvisation. How a lot of things we do every day are - to some extent, made up! From how we greet each other, to how we respond to an unexpected phone call. None of it is planned. We don't create some sort of crazy scalable distributed architecture to deal with these things: They're licks.

For some, this felt counter-intuitive. For me, it really hit home.

But I understand how it might feel counter-intuitive. Experienced engineers think hard before solving problems - they're expected to. "Hey - we've seen this before..." - They'll research. They'll anticipate problems. Even when the problem is not well-defined, they'll work hard on defining it! It doesn't even cross our minds that a badly defined problem is probably not a problem to begin with. With all this thinking, they enter the ether. The ether is that stale anticipation, that hesitation. It takes many forms - but in the engineering world it's normally in the form of endless research, stalling, no decisions being made. The ether is this "pre-problem" hesitation.

The talk also argued about how perfect is the ENEMY of good enough. While we're thinking of the perfect solution, the original problem is still there. Users don't see your research, users don't see the architecture meetings, users don't see the ether. The only thing they see is the problem. And guess what? The problem is still there.

And yeah, good enough is likely pretty bad. The first solution, will probably suck. It was fast, we were in a rush. We just wanted to put something out there to solve a problem! And obviously, we didn't think of all the ways it would/could break.

It's a privilege to build something that doesn't scale. We might argue that if you get stuck in the ether in the first place, you won't even get to scale being problem. Scale is a great problem to have. It means your solution started solving a problem - and people want more of it. This is a good thing.

And yes, now we have double the trouble. Now we need to scale AND handle users at the same time. But what's the alternative? Getting stuck in the ether and potentially building something we don't even know if people will use?

I like to think that doing things that don't scale will likely save time in the long run. Lots of things we build don't get used. And so if you spend a lot of time in the ether - you risk that time being completely wasted. And there are no guarantees you'll likely be able to anticipate all the ways you won't scale. But getting something out there, in front of people, takes you from 0 to 1. And the faster we realize what we've built doesn't solve anyone's problem, the better.

And solving problems is what we're here for.

How I self-host in 2024

2024-04-28T19:00:00+02:00

I'm a sucker for side projects. There's nothing quite like building something to learn about it.

Over the last 5+ years I've accumulated a little over 15 small web apps and websites. Almost all of them are hosted on a small Hetzner server and deployed using a mix of ssh, docker-compose, and Caddy.

Some months ago my small server started giving me some problems. The 4 vCPUs and 16 GB RAM weren't enough, and hiccups became more frequent. I also want to run more and more small LLMs on it, which turned out to be challenging. The whole "ssh with root using github actions" started to look like more of a limitation than a feature.

It was time for an upgrade.

First, I needed to upgrade my hardware. Not sure if Hetzner is the cheapest out there, but after using them for 5+ years I can definitely say they're reliable. I went with their server auction, which is a great (and sustainable) way of getting a nice refurbished server for a fair price. For about 30 EUR/month, I upgraded to a nice Intel i7 with 64GB RAM(!). Should be more than enough.

With better hardware, I needed to decide whether I would keep my old setup, or upgrade to something a bit more streamlined (god I hate that word). Turns out, there are quite some open source PaaS alternatives. From Dokku, CapRover, to DHH's Kamal. The one who caught my eye the most was Coolify. Without much due diligence, it seemed fit most of what I needed: Open source, easy to install, active project, with a polished interface, integrates with GitHub, and supports docker-based deployments. That's not an easy list to check-off.

The migration was pretty smooth, except for a couple of more complicated Django applications with databases on docker volumes. Since everything was already running on containers, it was pretty much plug-and-play. In a couple of days everything was migrated.

There are a lot of things I love about Coolify. It scans my GitHub repos and redeploys when needed via webhooks. It also sends me Telegram notifications when something gets updated to a new version. Finally, it also provides a wide range of other applications you can deploy (Databases, S3-compatible storages, and a wide range of other services).

Don't get me wrong - I love the Cloud. But essentially, the Cloud is just someone else’s computer. So why shouldn’t it be my computer? Self-hosting everything means I can pay a fixed price for my hardware, and not for the number of applications that run on it. And even though I did like my little hacked-together setup, I love being able to use, AND contribute to a promising open source project!

There's something incredible about what a 1-man team can accomplish with the support of the open source community.

The best keyboard I've ever had

2024-04-22T21:30:00+02:00

There I was. Ready to click the big blue purchase button. 350 Euro, for a keyboard (!) Asking Vitto multiple times: is this really worth it? Should I go for it? I mean, that's a whole lot of dinners.

After much hesitation, I did it. All black, with blank key caps. Yes, I too was skeptical from the start. The first days, I couldn't type 4 keys in a row without having to pull up the manual. 2 days passed, then 3 days passed... Still, I didn't get it. How do people use this keyboard? How am I supposed to type a full credit card number with blank key caps? You're a pro touch typist? Good for you. I'm not.

"I can get used to this" I would tell myself. But after a week using the keyboard, I've had enough. Basta. This isn't going to cut it for me. I'm just too slow with it. 350 Euro.. I'm not only poorer but now I'm also slower at typing.

I decided to return the thing¹. And went back to my good ol' Keychron K2.

A second chance

Some days passed and something was just not right. This can't be! I can't be missing that piece of junk. There's just something about those keys.. Typing on that keyboard just felt right. Even if I didn't know where each key was.. I just feels good!

I reordered it. And we were not about to make the same 'blank key caps' mistake. I ordered the white version with printed key caps. Just to make sure I CAN SEE EVERYTHING.

Some cons

It's perfect, but there are still some things I'm not in love with. For 350 EUR I expect everything to be absolutely perfect. I mean, really perfect. The materials are good, but for the most, they are still plastic. Every time I want to start using the keyboard, I need to click and hold the little power button in the back.

Another small complaint is the layout. This keyboard changes the place of the Control key, removes the delete key, has no dedicated arrow keys. For the most part, it works great (I use Vim keybindings for most things). But now my brain goes through a small hiccup every time I use a 'normal' keyboard. And my pinky still hurts. Maybe my hands are just big, I don't know.

What made me fall in love

It was the typing experience. It’s really like typing on a cloud. I'm not a big mechanical keyboard head. I've had a couple of Keychron keyboards, that's it. But these Topre switches are just something else. They are smooth and squishy at the same time. It feels as though your fingers are just swimming in the keyboard. That's what got me hooked from the start.

There are two other things I love about the Happy Hacking Keyboard. First, is the battery. I've had this keyboard for over a month, and did not think about battery a single time. I’ve come to realize just how much I hate charging my keyboard every week. I prefer to use a couple of AA batteries every couple of months.

The second thing is the sound. I like mechanical - but I don't enjoy everyone around the house hearing the hammering sound of plastic against plastic, or Vitto having to know every time I write to file in Neovim. I enjoy working around people sometimes. With the HHKB I can do so without bothering everyone around me.

Final thoughts

So yeah. I have to agree with Alex. This might very well be the perfect keyboard. I'm not 100% sold on all the choices that were made, but I can see why they were made. And the typing experience got me hooked from the first day (or at least once I knew where all the keys were).

There is only one last thing that will truly tell if this is the perfect keyboard: Time. And even though I have good trust on Japanese manufacturing, we'll only know the result of that test in a decade.

Don't buy from PFU EMEA, order from Amazon directly. ↩

An experiment with Gemma 2B and a Portuguese classic

2024-04-10T20:00:00+02:00

NewsHavn: Danish news, in English

2024-02-26T20:30:00+01:00

Conversas de café. Literally translated, means "coffee shop conversations".

An upcoming election, the weather next week, a corruption scandal, a new policy. Just some examples of Conversas de café.

As an expat living in Denmark for the past 7 years (without speaking the language), that's perhaps one of the things I miss most. Maybe it's because of where I'm from, but for me, water cooler conversations are Culture.

Learn the language, use Google Translate, read some expat focused website. No thanks. I want to read the same stuff people around me read. I want to understand what worries them, not other expats.

Enter NewsHavn.

Newshavn parses the RSS feeds of a couple of big Danish newspapers and uses Mistral 7B to translate them. The design is inspired by NPR. The code is all opensource.

For this one, I decided to use Go (again). I must say, Go is growing on me. Rust is fun and efficient, but I feel like Go strikes the right balance between strictness and flexibility. Another thing I love about Go is its concurrency model. Goroutines are just beautiful, and make me want to write more concurrent code. It feels like it's a no-bullshit language, and I'm a fan of no-bullshit.

mistral-doc: Fine-tuning an LLM on my ChatGPT conversations

2024-02-09T13:00:00+01:00

Step-by-step instructions to do it yourself

If you ever read this blog, you probably already know by now. For the past couple of months, I've been trying to create my very own Large Language Model.

OpenAI's models have had a significant impact on my productivity. But something still bothers me. My data, my preferences, my history, my concerns, all going into the hands of a single company.

But this time, this time I think I got really close! The result is mistral-doc, a Mistral fine-tune on all my ChatGPT conversations.

The goal is to emulate my current ChatGPT plus experience. My favorite thing about GPT4, is how obedient it is to my system prompt. Especially when I just want a short and concise answer. When I ask who was the first king of Portugal, I want the answer to be short (and sure, correct is also good).

Over the past year, I've accumulated a little over 1000(!) conversations with ChatGPT. Thankfully, OpenAI actually makes exporting all my data pretty straightforward. So, I built a small script that could process the export, and store all of my conversations as a Hugging Face dataset. It's private, of course, but you can use the same script to build yours.

There are thousands of models I could potentially fine-tune on my conversations. And the model choice will for sure, have an impact on the quality of the results. But I decided to go with something small, so that I could train everything in about ~2 hours with Axolotl. I decided to go with Mistral-7B-Instruct-v0.2. Yes, a base model would likely learn a bit more. But I don't want to teach a model to chat, I want to teach a model to learn from my chats.

With Axolotl and a 40GB/RAM GPU machine in runpod, the fine-tuning process took around 2.5 hours. Here's the config I used by the way. Once the training was done, I had to merge the model back to base (we trained a Lora adapter, but want the whole thing). Thankfully, Axolotl also handles this. In the end of the process, I successfully had my entire ~15 GB model in a repository I called mistral-doc-instruct-v4-merged. Yes, there were 3 other versions where I screwed things up.

With the entire fine-tuned model ready, it was now time to run it on my machine. Before I did that, I had to ensure the model was going to be of a reasonable size and speed. I used llama.cpp to first convert the model to the GGUF format, and then to quantize it to 4-bits. This process resulted in a single file, called mistral-doc.gguf, which is ~4.5 GB in size.

Now we can take for a spin! Using Ollama, I can create a Modelfile with all the details and configuration I need. With that file, I can now create my own model in the Ollama format with:

# create the model with ollama

$ ollama create mistral-doc -f ./Modelfile

# chat with your fine tune

$ ollama run mistral-doc
>>> vv jasmin or basmati rice with falafel?

Running the model in the terminal is all fine and dandy, but I want something that can actually replace the instinct of going into a browser and typing chat.openai.com. I downloaded a small app called Ollama-SwiftUI that lets me run my model in a nice little Mac app. Currently, I'm jumping around between using that, and LM Studio to interact with mistral-doc.

How has the experience been so far? Well for starters, its much better than any off-the-shelf ~7 billion parameter model I've tried before. For simple and straightforward tasks, it's a great alternative to ChatGPT Plus. I should note that it's not better because it's smarter or does math better. This model is better because it has learned how I like questions to be answered. This of course does not mean you will like it, which is why I'm not going to publish it. We could make it even better! I could fine-tune something like Mixtral, which would surely make it even more powerful! But I would probably not be able to run it on my laptop, and would have to pay someone else to host it. I could also generate even more data, increase the number of epochs, teach it more things about me.

Not sure I will; but this process definitely looks promising.

What's in my NOW?

2024-02-01T10:00:00+01:00

Self-hosting my personal LLM (but not quite)

2024-01-21T22:35:00+01:00

ChatGPT is now part of our daily lives. A quick question, an extra input, some quick feedback, I always reach for it. AGI or not AGI, I certainly can't deny the impact it has had on our lives. It's pretty incredible!

But that small voice just won't go away. "Where is all this data going? Should I really be telling this bot this much? What if someone else sees this?" In a way, it feels like everything I tell this bot is going into a void I have no control of. I can't be the only one feeling it.

And I don't like it. It's a bit like putting all my eggs in a single basket. As engineers, we know how bad having a single point of failure is. Resilience! That's what we were taught.

Given that I already started building Ambrosio, so why not use it to self-host my own LLM?

Self-hosting with the help of Llama.cpp

I'm not really interested in spending 500 USD/month to run a model that requires 40 GB of VRAM on a large GPU. So I had to lower my standards a little bit. The premise was now: What models can I run on a 15 USD/month server?

I started looking at the only LLM leaderboard I trust. There aren't many options with 7 billion parameters. The best one seems to be Starling-LM-7B-alpha, out of Berkeley. Quantized, it should just about run on a cheap Hetzner box. Getting it up and running on a small box with llama.cpp is as easy as:

# download model

$ wget https://huggingface.co/TheBloke/Starling-LM-7B-alpha-GGUF/resolve/main/starling-lm-7b-alpha.Q5_K_M.gguf?download=true

# setup llama cpp API with ./server

$ git clone https://github.com/ggerganov/llama.cpp
$ cd llama.cpp
$ make
$ ./server -m starling-lm-7b-alpha.Q5_K_M.gguf -c 8192 --temp 0.0 --repeat_penalty 1.1 -n -1 -p "GPT4 User: {prompt}<|end_of_turn|>GPT4 Assistant:"

And just like that, you're running your own OpenAI compatible (ugh, I hate that term) API serving that very 7B model.

Now, don't get me wrong, I think this model is great! But I'm looking for something that will replace my daily use of ChatGPT. After integrating with Ambrosio the limitations were obvious. The first issue: It's slow, painfully slow (which was expected on my small machine). And even though it's pretty incredible for a small model, it's not going to replace even the most basic use cases.

And if it sounds unfair in any way, it's because it is! How could I expect the same level of quality from something that can run on a cheap VM, to something like ChatGPT? And you're right. We can't. So what else can we do?

Mixtral 8x7B and the rice test

Let's say I was going to rent a bigger machine for a better model. What would I run on it? Certainly not a 7B model! Looking at the rankings again, the best open source thing currently is Mixtral 8x7b, a mixture-of-experts model from Mistral. Just glancing at the requirements, and it was pretty clear this needed to be run on someone else's box. I decided to give together.ai a go. Their pricing is pretty much unbeatable and their inference speed is blazingly fast. And of course they have an Open AI compatible API.

Cool, it's fast and cheap. Is it any good?

Everyone has their own way of testing the quality of a model. Mine is what I call the "rice test". My system prompt, stolen from Jeremy Howard contains the following clause:

if the request begins with the string "vv" [...] make your response as concise as possible

The question above is a simple and straightforward scenario. I'm at the supermarket, I need to buy some rice to cook with my falafel. I open my phone and: "vv jasmin or basmati rice with falafel?". As you can see above, while GPT-4 gives me exactly what I need, we can't say the same for Mixtral or GPT-3.5. I just want a quick and dirty answer, I don't want something complicated, I certainly do not want any Python code. So yeah, it doesn't quite pass the jasmin rice test. But we're getting closer. And I will keep testing it in the future.

And sure, Mixtral may not have been trained with a system prompt, but it should, in principle, support it.

Thoughts and taking Ambrosio further

Since I ended up using together's API, I decided to also add image generation to Ambrosio (something like I have with DALL·E 2). Together supports most Stable Diffusion models. My feelings about them are largely the same as my feelings about Mixtral and GPT-4. The open source models are good! But they are also a bit rough, and don't come quite close to what Open AI is giving us. At least yet.

So, will Ambrosio completely replace my use of GPT-4? Will I cancel my ChatGPT plus subscription? Not really. At least, not yet.

Every time I use these open source models, I feel like I'm missing out on a better answer. In a way, I've become spoiled by GPT-4. It just grasps what I want in a better way, and most importantly, it passes the rice test with flying colors!

But the truth also is, that for most tasks, Mixtral is good enough. And it's a big leap forward from what the open source community previously had. I can only imagine what comes next.

Exciting times.

Rebuilding /photos

2024-01-02T22:35:00+01:00

A couple of years ago, I decided to remove all my photos from Instagram. I wanted something for myself. Something that suited what I needed. The result was /photos. I wrote about it when I built it too. Here's a snippet from that post:

[...] Uploading is by no means as simple as opening an app and snapping a picture. But I've created a small Python script that processes a photo, generates a thumbnail, asks some questions about it, and uploads it to S3 and my blog. And that's good enough [...]

It wasn't good enough. I guess some principles are just timeless. It needed to be thrown away.

One to throw away

Quickly after making it, it was pretty obvious it was a solution to throw into the trash. In 2 years, I published a little over 10 photos. It's not that I want to publish thousands of photos, but I've always enjoyed taking and sharing them. But that whole run a script thing was just not working for me.

In a way, it's a lot like writing. Add a little bit of friction to the process, and nobody will use it at all. The script I originally built quickly got lost and stopped working. Whenever I wanted to share a photo, my laptop wasn't around. When the laptop was around, I wasn't thinking of sharing photos. So no, not good enough.

Whatever I was going to build to replace the old system, needed to do one thing really well: get out of the way.

I thought about it a bunch. An API? Should I interact with that API via shortcuts? Should I just build an entire front-end? What if I give leptos a shot? Should I rclone a google photos album? It can't be that hard. I needed something that got out of the way.

Ambrosio

I needed something that was already part of my routine. The result is Ambrosio. Some will recognize the name. Ambrosio is a Telegram bot that was designed to be my personal assistant.

I designed Ambrosio to be able to have different modes. The first one is the photo mode. It's responsible for performing CRUD operations on an R2 bucket. To maintain the speed of /photos, it also generates a webp thumbnail with reduced quality. If needed, it triggers a rebuild of my website using a deploy hook.

Instead of having to create a markdown page for every new photo, I've built another small pelican plugin. Before building the website, it downloads all the metadata for each photo, and generates the pages that it needs to. So that website builds remain fast on my local machine, I also added disk caching. Remember: get out of the way!

On Go

I decided to build Ambrosio using something I've never used before: Go. I've tinkered with a couple of code bases but never really used it to build something from scratch. The idea of something not as strict as Rust, but also not as flexible as Python really sparked my curiosity.

Go gives me mixed feelings, but Go gives me hope. It is strict, but not too strict. It gives me the confidence when writing code that Python has never given me. And gives me guarantees that if the compiler does not complain, things will mostly work fine. There are some things I still don't understand about Go. The first is error handling. Having a bunch if if err == nil throughout the code base is not particularly nice to look at. Also hesitant about the whole one letter variable names thing. Things tend to become cryptic.

It's fun how programming languages all start looking like the same the more you use them. For now, I'm curious to see how my opinion about Go evolves with time.

One thing is clear though: Whatever technology you use, whatever framework you go with, technology is beautiful when it gets out of the way.

You can now listen to this blog

2023-12-08T05:00:00+01:00

One of my favorite Portuguese columnists has this weird thing about his column. Maybe it's more common than I thought. For every piece he publishes, he also publishes a podcast version along with it.

Now, either Publico has 27th century text-to-speech (TTS) technology, or he's actually reading them. I don't have a problem with that, but I'm pretty sure we could automate that part of the process with today's tech.

And yeah, I've heard all the rage about voice cloning services like ElevenLabs. But if you've been following this blog for a while, you probably guessed that we're not just gonna use an API. We're probably gonna build one from scratch.

An engine

The premise did not appear simple to build, but was easy to understand. Something that transcribes every new article of this blog using my own voice. It needs to be cheap, seamless, and most importantly, not get in the way. Writing is enough work as is.

The result is podcaster. It runs 100% on GitHub Actions, it scans every new blog post in my RSS feed and uses XTTS-v2 to transcribe it. The only thing it needs from me is a 1-min audio file. I tried Bark and a couple of other models, but this was the only one that made Vittoria come into the room when I was testing things around.

It's not that I hate infrastructure, I just wanted the whole thing to run on CI. All these TTS models are slow when running on the CPU. But instead of embarking on another painful journey through the world of GPU computing, I found Modal. I'm happy to report that I've regained faith in the future of serverless GPUs:

# define the image

MODAL_IMAGE = (
    modal.Image.debian_slim()
    .pip_install_from_pyproject("pyproject.toml")
    .apt_install("ffmpeg")
)
stub = modal.Stub("modal-app", image=MODAL_IMAGE)

# create the function

@stub.function(gpu="any")
def transcribe(
    article: ParsedArticle,
    voice_file: str = VOICE_FILE,
    model_name: str = MODEL_NAME,
    language: str = LANGUAGE,
) -> bytes:
    # this part is executed in a GPU powered machine
    # ...

The engine stores all transcripts in S3, automatically generates a podcast feed from them (using feedgen), and uses web hooks to trigger a new build of this blog. Again, pretty much for free. If you're curious or interested in taking it for a spin, should be straightforward to get started.

Integrating with Pelican

As I said before: I really don't like adding friction to my writing. It's already hard as is! The challenge then was to figure a way of automagically updating the blog with available transcripts after I publish, without me having to do anything at all.

Fortunately, Justin besides being great company at PyCon Italia every year, has also built a pretty robust plugin system for Pelican. All I had to do, was to add a podcaster plugin to this website. The plugin automatically matches articles to corresponding episodes, and adds that short html snippet you're seeing above. Should be build once and let run. Hopefully at least.

Final thoughts

I had a lot of fun building this, and also learned a lot. First, it demystified the whole podcast hosting thing for me. Turns out, it's just a bunch of mp3 files in a bucket with a rss feed.

Often I've settled for running ML models on CPU just because deploying with a GPU was much more of a pain, and didn't add any happiness to the process. At least for now, GPUs are here to stay, hopefully so are services like Modal. Pythonic, easy to use, and easy to isolate from the rest of the code base.

Finally, I was very impressed with the state of TTS and voice cloning technology. Is it perfect? No. Does it have some artifacts? Yes. Does it sound like a robot sometimes? Sure. But remember, I only gave it a minute of my voice.

Open source never ceases to amaze me.

Should you go into management consulting? Maybe.

2023-11-08T08:00:00+01:00

A couple of days ago, someone shared this John Oliver video with me. In it, he shows all the ways McKinsey is a terrible organization. Not a minute in and I was already thinking about that management consulting blog post I always wanted to write.

When I talk with people that just finished University a lot of them wonder if it's the right place to go. Of course, by the tasty recruiting videos, it looks like the absolute perfect place for your typical graduate. The reality is a bit more nuanced.

Attention to detail. That was the thing that struck me first. From manically aligning elements on a slide, to someone going meticulously through my email signature to ensure everything is as it should be. If you're the type that struggles with detail you'll probably suffer a bit. Truth is, attention to detail is generally something most working environments admire, and strive for (or maybe they should).

In a way, management consulting feels like an extension of a University. Everything is pre structured, everything is pre defined. Even before you get your hands dirty, you'll know exactly the path that you're expected to follow, and how fast you're expected to follow it. Up or out, that's a good summary of the approach. If you're not being promoted, then you should probably get out. Naturally, most people stay 2 years and leave. In any case - a structured and defined environment is where many are able to learn at their best.

What I miss the most is by far the people. I know - I sound just like a propaganda boy. But there's nothing I enjoy more than being the dumbest in the room. Most people in these places are driven, motivated, and hard-working. You'll meet the smartest people you will probably ever work with. Like in the military, you'll make long lasting bonds over traumatic experiences (e.g., long nights, tough clients). You'll build a network of great people - with whom you'll stay connected for the rest of your career. I certainly did.

But it's not all rainbows and unicorns.

Let's start with the thing most people fear. The hours. Yes, the hours are long. I generally don't dislike long hours. With an important premise: I have to be doing something I really enjoy. But in these firms, the hours are not long because they need to be. The hours are long because you want to show your client (and your firm) that you are always available. Sure, this means working longer hours - but less intense hours. For some, this will look like an absolute waste of time - why the hell are we here? It's not about doing the required work. It's about being there. Or worse, it's about showing that you were there.

Soon, you'll realize you don't really have choices. You don't get to decide what projects you work on. You don't get to decide what business area you specialize in. You don't get to decide when and where you work. You don't get to decide what dinners, events, and gatherings you go to (a common strategy to work a bit less). If you have no clue what you want to do - this is great. You just need to go along. If you know what you want to do - this becomes problematic.

Which brings me to my last point. Management consulting, from my experience, is a mix of a University and Military Service. You are not rewarded for thinking differently. You are not rewarded for exploring things others have not. Your innovative way of solving a problem is a risk, rather than an opportunity. For those who just want to do work, this is great - you don't really have to think about it. For the creatives, that want to spend time in things they like, and want to solve problems in crazy ways - this can be painful.

Is management consulting for you? I have no clue.

My experience was positive. Until it wasn't. It can be great step for those of us who have absolutely no idea what they want to do after university. It's an opportunity to see a lot of different things - with a smart bunch of people. For those who know exactly what they want - it might feel like a slow down. For most of us, it's a mix.

In any case. At least now you know what you're getting yourself into.

I sure wasn't. But I can't say I regret it.

Disclaimer: This post is highly biased towards my experience. As it should - this is MY blog after all.

Changelog neural search

2023-10-06T05:00:00+02:00

Search is one of the most important breakthroughs of the internet. Some are saying a list of blue links is not enough - and that AI will overthrow search. I don't know if we're about to witness a revolution. But as with most things - there's only one way to know - to build and use it myself.

I like podcasts a lot. There's really nothing like hearing people talk about things I know nothing about. Some of my favorite podcasts are produced by the Changelog network. More than once, I've had to use their search engine when researching something that was said during an episode.

One of the best things about the Changelog is that the whole thing is open source. From the podcast engine itself, to the transcripts of every episode. Why not take all of these transcripts and build an AI-powered™ search engine around them?

changelog.duarteocarmo.com

How it's built

Before I describe the stack, let's get the obvious out of the way: the whole thing is open source. Both the back-end and the front-end. If you prefer to go and poke around the code yourself, be my guest.

I love the chunk-embed-search-retrieve dance as much as the next guy, but for this one, I wanted to keep things a bit simpler, so I'm letting SuperDuperDB do most of the heavy lifting for me. With it, all I really need to do is add the embedding model to my serverless MongoDB instance, and it handles the rest for me:

# add model to DB

model = SentenceTransformer(...)
db.add(
    VectorIndex(
        identifier=index_id,
        indexing_listener=Listener(
            model=model,
            key=key,
            select=collection.find(),
        ),
    )
)

# search the DB

cur = db.execute(
    collection.find({"$regex": {"podcast": "practicalai"}}).like(
        {"text": "What are embeddings"}, n=limit, vector_index=index_id
    )
)

For the front-end, I finally took NextJS for a spin. Love the productivity gains - especially when we're talking about developer experience. Vercel is absolutely killing the developer experience side of things. On the other side, I have no clue how most of the magic is working - and I'm not sure that's a good thing.

A poor man's guide to fine-tuning Llama 2

2023-09-26T13:00:00+02:00

Last Friday, Kostas and I found ourselves discussing AI over beers again. He was telling me how he thinks everyone else is decades away from OpenAI - and that they likely won't catch up soon. I disagreed. In fact, I think open source is quickly catching up, and getting closer by the day.

Around four months ago, I wrote a small tutorial on fine-tuning Flan T5 to generate conversations in my friends group chat. It worked ok(ish), but the process was clunky and took way too long.

So, here we are, four months later, with the same question: How easy is it to train an LLM on my friend's group chat? Turns out, for a couple of dollars and one hour you can get pretty far.

Tiger-llama in action, simulating a Telegram conversation

Data

The goal is clear: fine-tune Llama 2, to automatically generate conversations that would normally happen in the group chat that I've held with my close friends for years now. The first step is to export the data from Telegram (which is pretty easy).

For most LLMs, you need to massage the data into a specific format before training. This means providing things like the instruction, the input, the system_prompt, etc. But all I wanted was to generate conversations, so I started by creating a large jsonl file with chunks of conversations separated by a ### token to indicate the speaker.

# dataset.jsonl

{"text": "### Friend 1: This is a question### Friend 2: This is a reply### Friend 3: What the hell are you guys talking about?"}
{"text": "### Friend 4: Who's coming tonight?### Friend 2: No one, it's literally Monday."}
#...

After getting the data ready, I went ahead and pushed it to the hugging face hub:

# huggingface-cli login --token hf_XXXXXXXX (to log into Hugging Face)

# read and split

tiger_llama =  pd.read_json('dataset.jsonl', lines=True)
train_tiger, test_tiger = train_test_split(
    tiger_llama, test_size=0.15, random_state=42, shuffle=True
)

# push to hub

train_tiger = Dataset.from_pandas(train_tiger)
test_tiger = Dataset.from_pandas(test_tiger)

ds = DatasetDict()
ds["train"] = train_tiger
ds["test"] = test_tiger

dataset_name = "duarteocarmo/tiger-llama"
ds.push_to_hub(dataset_name, branch="main", private=True)

With all the conversations pushed to the hub (cough - privacy - cough), it was now time to train this model.

Training with axolotl

For the first couple of days, I got a bit frustrated. I tested a bunch of different tutorials on how to fine tune LLama-2. But none of them really got me anywhere. Some took way too long to train, others resulted in a model that didn't really generate anything interesting. I confess I almost fell to the dark side! But then I stumbled upon axolotl.

Axolotl is designed to "streamline the fine-tuning of LLMs". It supports a bunch of different models and training configurations. The best part? To fine-tune a model, all you is pretty much a config file. Yes, you heard that right - just a yaml file. The only things I needed to tweak were the base model (in our case, Llama 2 - meta-llama/Llama-2-7b-hf), and the dataset sections:

# llama-tiger.yaml

# base model

base_model: meta-llama/Llama-2-7b-hf
base_model_config: meta-llama/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true

# ...

# my dataset

# other formats: https://github.com/OpenAccess-AI-Collective/axolotl#dataset

datasets:
  - path: duarteocarmo/tiger-llama
    type: completion #
    field: text
dataset_prepared_path: last_run_prepared
hub_model_id: duarteocarmo/tiger-llama
val_set_size: 0.01
output_dir: ./qlora-out

# wandb monitoring

wandb_project: "llama-tiger"
wandb_log_model: "checkpoint"

### ...

Expand for the full tiger-llama.yaml file

# Image: winglian/axolotl:main-py3.10-cu118-2.0.1

base_model: meta-llama/Llama-2-7b-hf
base_model_config: meta-llama/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: duarteocarmo/tiger-llama
    type: completion
    field: text
dataset_prepared_path: last_run_prepared
hub_model_id: duarteocarmo/tiger-llama
val_set_size: 0.01
output_dir: ./qlora-out

adapter: qlora
lora_model_dir:

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project: "llama-tiger"
wandb_entity:
wandb_watch:
wandb_run_id:
wandb_log_model: "checkpoint"

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
eval_steps: 20
eval_table_size: 5
save_steps:
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"

Unfortunately, with these things, I needed a GPU to run the training. Given that I'm (apparently) considered GPU poor, I used Vast.ai. I got a machine with at least ~40GB of GPU RAM, and that was relatively close to me. These cost around 1 USD/hour. I also used the Axololt docker image (e.g., winglian/axolotl:main-py3.10-cu118-2.0.1) so that everything was pre-installed when the machine turned on.

Once the machine was up, I ssh'd into it and ran:

$ huggingface-cli login --token hf_MY_HUGGINGFACE_TOKEN_WITH_WRITE_ACCESS
$ wandb login MY_WANDB_API_KEY
$ accelerate launch -m axolotl.cli.train llama-tiger.yaml

And we were off to the races. With a single command, I was fine-tuning Llama 2 on my custom dataset. While training, Axolotl automatically logs everything to Weights & Biases, so we can monitor how the losses are evolving. As a bonus, it also shows the model outputs so that I can follow how to model is improving its generation during training:

My dataset has ~12K rows. Training took around 1 hour in total, using a machine with 2xA40 GPUs. In summary: I spent around 2 USD to fine-tune the whole model.

Once training was done, your fine-tuned model (or adapter, to be specific) is saved in the ./lora-out directory. With my configuration, it was also uploaded the model to the hugging face repository I provided in hub_model_id. Onto the inference.

Inference

The fine-tuning result is not an actual Llama 2 model, but an adapter to the model (Axolotl uses qlora by default for Llama models). So in the end, the adapter is a mere 320 MB.

Using Axolotl, inference is also pretty straightforward: All I need to do is download the model, and launch the Axolotl inference command:

# download from fine tuned repo

git lfs install
git clone https://huggingface.co/duarteocarmo/tiger-llama

# run axolotl inference

accelerate launch -m axolotl.cli.inference tiger-llama.yaml --lora_model_dir="./tiger-llama"

Once downloaded and launched, I can give the model the start of a fake conversation, and it will go on to generate a completely fake conversation based on my group chat:

Input: ### SZ: Quem vem a Lisboa no natal?
Output: ### SZ: Quem vem a Lisboa no natal?### PK: Acho que tenho tive uma surpresa de aniversario da malta, não tenho férias até janeiro### SZ: Fodass, faltamos 2 ...

Closing thoughts

Compared to the last time I fine-tuned a model, open source is definitely moving fast. The process was not only much faster, and simpler than fine-tuning Flan T5 using a notebook, but the results were also much better than anything I had seen so far.

Axolotl did almost all the heavy lifting. Making the whole process super smooth. All I needed was a dataset, a config file, 2 USD, and about an hour to fine-tune a model. We've come a long way.

The model is still not perfect though. It captures some of my friends' quirks and ways of speaking, and the generated conversations make sense around ~70% of the time. But that's still 30% nonsense. Could be a couple of things: the size (I used the "smaller" 7 billion version of the model), or even the language (Portuguese from Portugal). For example, when I prompt the model to simulate a politics discussion between my friends, someone starts discussing something about Dilma (which is wildly inaccurate given we're from Portugal).

So Kostas was right, "open source" hasn't caught up yet. But oh boy, we're getting damn close.

aicoverlettercreator.com

2023-07-19T13:00:00+02:00

Get rid of all applications that don't have a cover letter

I remember it like it was yesterday. As I was leaving one of the first companies I've ever worked for, my manager asked me to hire my replacement. Drowning in hundreds of applications for the position, I still recall his exact words.

I don't want to start a debate on if you should include a cover letter in your job application or not. I've heard good arguments on both sides. There's one thing I'd like to point out though: in the days of ChatGPT, it's hard to get a good excuse not to write one.

Enter: aicoverlettercreator.com

Why I built it

The short answer: Why not?

The long answer, for a couple of reasons.

I like mentoring/helping people that are just entering the industry (or just getting out of University) regarding their careers. Most people, when applying for positions, don't really pay close attention to the cover letter. I believe it can be a really big differentiator. Even though I hope they are aware of just how much ChatGPT can help them, I integrated in this version a little bit of secret juice. I think that secret juice will make those cover letters even better. And that's the first reason, to help people.

The second one is a bit more selfish. I've built a handful of LLM powered applications, but never anything completely end-to-end. They say the best way to learn about something is to build it yourself right? I certainly think so. So I wanted to build something where I made all the choices. Or something where I felt all the pain.

The stack

I'm not the biggest fan of falling in-love with a stack. "What stack would you use?" Well, tell me about the problem first.

For this app, I went with Django and htmx (again). Django is probably overkill for most simple web applications, but when users, databases, and settings start becoming a thing, it includes almost all the batteries I need. As for htmx, it's not that I don't like React, it's that I love keeping things simple. That does not mean it's always a great fit. But for this small project, it sure was.

One of the biggest hurdles when building this, was the implementation of streaming from OpenAI's API. There's nothing I hate more than clicking a button and having to wait 10 seconds for something to appear on screen. Turns out, integrating server-sent events with Django is not so straightforward. And even though Django supports streaming responses. At the end of the day, you can't really escape some good old Javascript. And I'm ok with that.

Still sticking to my premise of keeping things as simple as possible. Every component I add to an app during development is a component I'll have to maintain during its lifetime. And I'm not the biggest fan of maintenance work. Because of this, aicoverlettercreator.com is both simple to use and to operate. It's a simple Django app with no queues and no self hosted PostgreSQL containers. It's a Dockerfile that connects to a PlanetScale database. That's it, every git commit automatically updates the app.

Nothing like keeping things simple.

Supercharging my Telegram group with the help of ChatGPT

2023-06-10T02:00:00+02:00

While most people in Europe use WhatsApp, my group of friends and I use Telegram. For years now we've used things like Combot to see who's more silent than usual, and MissRose to give our monthly elected group admins moderation rights. Yeah, we take friendship that seriously.

But now we have access to LLMs right? So I decided to build two new features for our group chat. One is pretty useful, and the other.. Well, you be the judge.

Summarize the conversation with `/resume`

Our group chat can get pretty active sometimes. We call those the golden hours. If you're distracted and miss one of those, you'll quickly lose track of what's going on. But let's be honest, sometimes, we just don't have time to catch up.

Enter the /resume command. Now, when someone misses a specially hectic part of the conversation, they can just use the /resume command to get a short summary of what happened.

In Portuguese "resume" means "summarize"

To build it I used LangChain and the cheaper gpt-3.5-turbo API (e.g., ChatGPT). I keep a rotating list of the last 50 messages that happened in our group. When the command is called, I send those to OpenAI to get a summary back.

Here's the core part of that code:

# src/bot/summarizer/main.py

LLM_CHAT = ChatOpenAI()

def get_summary(
    list_of_messages: list[Message],
) -> str:
    """
    Fetches the summary for the last 50 messages
    """

    formatted_list_of_message = "\n".join([str(m) for m in list_of_messages])
    formatted_list_of_message = truncate_text(formatted_list_of_message)

    system_template = """
You are an assistant helping friends catch up in a busy chat group. Your goal is to help friends in this group stay up to date without having to read all the messages.

You will receive a recent conversation that happened in the group. Respond immediately with a short and concise summary of the conversation.
The summary should have the following characteristics:
- Should be in Portuguese
- Should have a tone that is similar to the conversation, act like you are part of the group
- Use 3 sentences or less
- Don't be too general, mention who said what
"""

    human_template = """
CONVERSATION BLOCK START
{list_of_messages}
CONVERSATION BLOCK END
"""

    system_message_prompt = SystemMessagePromptTemplate.from_template(
        system_template
    )
    human_message_prompt = HumanMessagePromptTemplate.from_template(
        human_template
    )

    chat_prompt = ChatPromptTemplate.from_messages(
        [system_message_prompt, human_message_prompt]
    )

    chat_chain = LLMChain(llm=LLM_CHAT, prompt=chat_prompt, verbose=True)

    response = chat_chain.run(
        {
            "list_of_messages": formatted_list_of_message,
        },
    )

    logger.info("\nResponse:\n {}", response)

    return response

Most of my friends really liked the /resume command. Some however, showed concerns regarding AI and how these things are super scary. After those comments, one thing was obvious: I needed to build something even funnier.

Impersonate a user with `/fake @username`

What if the bot could impersonate any of my friends in the group chat? What if I could ask the bot to answer a question just like the person X would?

Enter the /fake @username <insert question> command. With it, you can impersonate anyone on the group chat (cough, my friends), and ask it to answer just like that person would! Here's the command in action:

The `/fake` command in action

Although not as useful as the summarization command, it's actually a bit more complex to build.

The first component is a vector database. Here, I'm storing the embeddings for pretty much everything my friends said in the past year. I wanted something simple like sqlite so I went with Chroma. The trick here is not to embed every single message separately, but to build a long string of every single thing a person said. Once that's built, you then chunk it and store it appropriately. With Chroma I could also store metadata about the document - which I used to store the author of the chunks.

Once the whole vector database is built, I can now retrieve the N pieces of text that are most similar to a particular question (while filtering those results for a particular user):

# src/bot/replier/main.py

def query_collection(
    collection: Collection, query: str, person: str, n_results: int = 4
) -> str:
    """
    Gets the most relevant n_results items from a person for a given query
    and returns them as a context
    """
    results = collection.query(
        query_texts=[query],
        n_results=n_results,
        where={"from": person},
    )

    documents = results.get("documents")

    if not documents:
        raise ValueError("No documents found")

    documents = documents[0]

    context = ""
    for doc in documents:
        context += doc + "\n\n"

    return context

Now that I could get the most related items to a certain query for a particular user, I could start putting the impersonation together. Here, the answering chain function is in charge of trying to impersonate a user to the best of its ability given a question and relevant context:

# src/bot/replier/main.py

def build_answering_chain(verbose: bool = False) -> LLMChain:
    """
    Builds a answering chain for the impersonation
    """

    system_prompt = """
Your name is {person}. You are participating in a group chat with all of your childhood friends.
- You will be given some context of messages from {person}
- Use the context to inspire your answer to the user question
- Use the same tone of voice and writing style as the messages in the context
- If the answer is not in the context, make something funny up
- Always answer from the perspective of being {person}
- Remember to be funny and entertaining
- This is a group chat and everything is fun and entertaining, so insults are allowed and fun
----------------
{context}"""

    chat = ChatOpenAI(max_tokens=150)

    system_message_prompt = SystemMessagePromptTemplate.from_template(
        system_prompt
    )

    human_prompt = """{question}"""
    human_message_prompt = HumanMessagePromptTemplate.from_template(
        human_prompt
    )

    chat_prompt = ChatPromptTemplate.from_messages(
        [system_message_prompt, human_message_prompt]
    )

    chain = LLMChain(llm=chat, prompt=chat_prompt, verbose=verbose)

    return chain

With that, we can pull everything together with the reply_to_question_as function. It builds the chain, queries Chroma for relevant context, and then runs it:

# src/bot/replier/main.py

def reply_to_question_as(
    person: str,
    question: str,
    collection: Collection,
    verbose: bool = False,
) -> str:
    """
    Replies to a question as a user (e.g., impersonates)
    """
    chain = build_answering_chain(verbose=verbose)

    context = query_collection(
        collection=collection, query=question, person=person
    )

    # using the callback to track cost
    with get_openai_callback() as cb:
        result = chain.run(
            person=person, question=question, context=context, verbose=True
        )
        logger.info(f"Answer from {person}: {result}")
        logger.info(f"OpenAI callback: {cb}")

    return str(result)

My friends really liked this one as well, and everyone cracked a laugh. But it was pretty obvious that the impersonation was not fooling anyone. It lacked juice, one of my friends said.

Closing thoughts

Both of these two bots were pretty fun to build, and python-telegram-bot makes building the whole thing so easy. It's basically a python script running on my server.

The summarization feature was an instant success. It's simple, straightforward, and my friends loved it. The feedback to the impersonation feature was.. a bit of a mixed bag. Even though the model can accurately respond to a question with some relevant items to the person its impersonating, it's not really credible. It's missing the juice.

What is the juice, you ask, my dear reader? The juice is the voice of the person it's trying to impersonate. The juice is the reason why when you ask ChatGPT to design something it looks pretty ugly. The juice is the creativity, the originality!

(and of course, privacy concerns, here's my mention of them)

Fine-tuning FLAN-T5 to replace my friends

2023-05-24T16:15:00+02:00

View Code

They say that the best way to learn about something is to build it yourself. Everyone talks about OpenAI this, and OpenAI that. How about we fine tune a Large Language Model ourselves?

If you heard about this models before, but are still curious about how all of these things work underneath, this blog post is just for you. With it, you'll learn how to fine tune a Large Language Model (Google's FLAN-T5) on the conversation history of a group chat you have lying around. The goal is to teach the model to talk exactly like your friends would. It won't be perfect, of course, but the goal here is to learn how these things work.

Here's how we'll structure things:

Why FLAN-T5
Setting up your environment
Preparing the data
Fine-tuning the model
Generating conversations

Even though I love taking credit for ideas that are not mine, this blog post was inspired by the work of some other awesome folks. Particularly this one, and this one.

Let's get started.

1. Why FLAN-T5

There are a lot of Large Language Models (e.g., LLMs) out there. With the explosion of things like GPT-3 and GPT-4, lots of companies and open source organizations have started building their own LLMs. Some of these models leaked, others are powerful but only available through an API.

Some of theses LLMs that are actually free an open source, even for comercial use. Eugene Yan, has compiled a great repo with an outline of some of these options.

In 2020, Google released a paper called "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", where they presented their T5 model. T5 is a encoder-decoder model that was trained in a variety of tasks (e.g., translate to german, summarize the following sentence, etc). FLAN-T5 is basically the exact same thing as T5, but pretty much better at everything it does.

There are a couple of other reasons why we're using Flan-T5 for this guide:

It's free, open source, and commercially available
It has several sizes we can use (from small, all the way to xxl)
It's compatible with the whole Hugging Face 🤗 ecosystem, making our life easier

With that. Let's get things started.

2. Setting up the environment

Note: This tutorial was run on a NVIDIA A100 with 40GB of RAM

If you want to run this project but don't have a powerful GPU at hand, you can get started quickly using Unweave. After signing up and installing, you can launch this project in two steps:

# link this repo to your unweave account

!unweave link your-username/fine-tune-llm

# launch a machine powered by a A100 GPU

!unweave code --new --type a100 --image pytorch/pytorch:2.0.0-cuda11.7-cudnn8-devel

Also, to save and load models, you'll need to have a Hugging Face account. Once you have that set up, create and copy a new token. Paste it somewhere, we'll need it for later.

The following cell, will install all required python libraries, as well as some local packages we'll need. (e.g., git, git-lfs)

!pip install pytesseract evaluate tqdm transformers datasets rouge-score accelerate nltk tensorboard jupyter-black py7zr --upgrade
!apt-get install git --yes
!apt-get install git-lfs --yes

Take your Hugging Face token, and replace it with in the field below. This will log you into the Hugging Face hub, which we'll need to push and pull models.

!huggingface-cli login --token XXXXXXXXXXXXXXXX

3. Data preprocessing

Let' start by preprocessing our data for training. We'll start by defining some variables. Feel free to replace the location of your Telegram group chat export in the TELEGRAM_EXPORT variable:

import json
import pandas
import jupyter_black
from datetime import timedelta

from datasets import Dataset

jupyter_black.load()

TELEGRAM_EXPORT = "anonym_telegram.json"  # anonymized for obvious reasons
CONVERSATION_LIMIT = 20_000  # limit number of messages
TEST_SIZE = 0.2  # % of test data
IS_NEW_SESSION_CUTOFF_MINS = (
    120  # if the time between messages is more than this, it's a new session
)

Let's preprocess this into a dataframe:

# load data in

with open(TELEGRAM_EXPORT, "r") as f:
    data = json.load(f)
    messages = data["messages"]

# create a dataframe

df = pandas.DataFrame(messages)[["text", "from", "date"]]

# filter empty messages

df = df[df["from"].isna() == False]
df = df[df["text"].isna() == False]
df = df[df["text"].str.len() > 0]

df["date"] = pandas.to_datetime(df["date"])  # convert to datetime
df.sort_values("date", inplace=True)  # sort by date
df = df.tail(CONVERSATION_LIMIT)  # limit number of messages

Now, the goal is to get our data into a format where the model can understand the conversation, and respond to what has been going on. Something like:

{
    "conversation": "Person X: Where were you yesterday?\nPerson Y: I was at home! You?",
    "response": "Person X: Me too, but thought of going out."
}

This will allow the model to see a thread and respond in the most realistic manner possible to the conversation that was already going on.

However, we also know that group chats are pretty async, so we don't necessarily want a "good morning" message to be a direct response to whatever happened last night.

# telegram exports have some artifacts, let's clean them up

def clean_text(text: str) -> str:
    if isinstance(text, list):
        text = "".join([o.get("text", o) if isinstance(o, dict) else o for o in text])

    return text

# clean up and rename columns

df["text"] = df["text"].apply(clean_text)
df["chat"] = "telegram"
df.rename(
    columns={"from": "sender", "date": "message_date", "text": "response"}, inplace=True
)

# create new sessions

df["last_event"] = df.groupby("chat")["message_date"].shift(1)
df["is_new_session"] = (
    (df["message_date"] - df["last_event"]).fillna(
        pandas.Timedelta(minutes=IS_NEW_SESSION_CUTOFF_MINS)
    )
    >= timedelta(minutes=IS_NEW_SESSION_CUTOFF_MINS)
).astype(int)
df["chat_session_id"] = (
    df.sort_values(["chat", "message_date"]).groupby("chat")["is_new_session"].cumsum()
)

Now that we have everything setup, it's time to create a conversation column, that has all the messages before a certain response:

sess_dict = df.to_dict("records")
items = []
counter = 0
for row in sess_dict:
    context = []
    cstring = ""
    for i in range(10, 0, -1):
        if sess_dict[counter - i]["chat_session_id"] == row["chat_session_id"]:
            msg = (
                f"{sess_dict[counter-i]['sender']}: {sess_dict[counter-i]['response']}"
            )
            if len(context) > 0:
                cstring += "\n"
            context.append(msg)
            cstring += msg
    if len(context) < 2:
        for i in range(5, 0, -1):
            msg = (
                f"{sess_dict[counter-i]['sender']}: {sess_dict[counter-i]['response']}"
            )
            context.append(msg)
            cstring += "\n"
            cstring += msg
    items.append(cstring)
    counter += 1

# create the conversation column

df["conversation"] = items

# create the response column

df["response"] = df.apply(lambda row: f"{row.sender}: {row.response}", axis=1)

print(f"Your dataframe shape is {df.shape}")
print(f"You have the following columns: {', '.join(df.columns.tolist())}")

# Your dataframe shape is (20000, 8)

# You have the following columns: response, sender, message_date, chat, last_event, is_new_session, chat_session_id, conversation

Let's see what a single example looks like:

item = df.sample(1, random_state=314)[["conversation", "response", "sender"]].to_dict(
    "records"
)[-1]

print(f"Conversation:\n{item['conversation']}\n")
print(f"Response:\n{item['response']}\n")

# Conversation:

# Hugo Silva: Lembram se do meu amigo alex frances?

# Tiago Pereira: Claro ya

# Hugo Silva: Ele correu a maratona de paris outra vez..

# Tiago Pereira: E…

# Tiago Pereira: Ou é só isso?

# Hugo Silva: Numero 304 overall...

# Hugo Silva: Crl

# Tiago Pereira: Eia cum crlh

# Hugo Silva: Pa doente

# Tiago Pereira: 3’46. Que louco fds

#

# Response:

# Hugo Silva: Doente completo

Our final preprocessing step is to load our dataframe in the Datasets format:

cols_for_dataset = ["conversation", "response"]
df = df[cols_for_dataset]

data = Dataset.from_pandas(df).train_test_split(test_size=0.2)
print(data)

# DatasetDict({

#     train: Dataset({

#         features: ['conversation', 'response', '__index_level_0__'],

#         num_rows: 16000

#     })

#     test: Dataset({

#         features: ['conversation', 'response', '__index_level_0__'],

#         num_rows: 4000

#     })

# })

Great. We're ready to focus on the model.

4. Fine-tuning the model

We start with some training specific imports, mostly related to Hugging Face.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from datasets import concatenate_datasets
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
from huggingface_hub import HfFolder
from transformers import pipeline
from random import randrange
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments
import os

Now, we define the most important variables for training, make sure to read the description of each one:

MODEL_NAME = "chat"  # the name of your model
MODEL_ID = "google/flan-t5-small"  # the id of the base model we will train (can be small, base, large, xl, etc.) (the bigger - the more GPU memory you need)
REPOSITORY_ID = f"{MODEL_ID.split('/')[1]}-{MODEL_NAME}"  # the id of your huggingface repository where the model will be stored
NUM_TRAIN_EPOCHS = 4  # number of epochs to train

Let's load the model and the tokenizer with the help of AutoModelForSeq2SeqLM:

model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

Some samples in our dataset will likely be too long for our model. So they'll need to be truncated. In the cell below, we define the max conversation and response lenght. This will then help in the truncating process:

# source

tokenized_inputs = concatenate_datasets([data["train"], data["test"]]).map(
    lambda x: tokenizer(x["conversation"], truncation=True), batched=True
)
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])

# target

tokenized_targets = concatenate_datasets([data["train"], data["test"]]).map(
    lambda x: tokenizer(x["response"], truncation=True), batched=True
)
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])

print(f"Max source length: {max_source_length}")
print(f"Max target length: {max_target_length}")

Now that we know the limit for the truncation, we'll preprocess our dataset to be fed into the model. During its original training, FLAN T5 used different templates for training on multiple tasks. Here, I decided to use the Continue writing the following text template. I did not test all of them, so feel free to modify according to what suits your task best.

def preprocess_function(sample, padding="max_length"):
    template_start = "Continue writing the following text.\n\n"
    inputs = [template_start + item for item in sample["conversation"]]

    model_inputs = tokenizer(
        inputs, max_length=max_source_length, padding=padding, truncation=True
    )

    labels = tokenizer(
        text_target=sample["response"],
        max_length=max_target_length,
        padding=padding,
        truncation=True,
    )

    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label]
            for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = data.map(
    preprocess_function, batched=True, remove_columns=["conversation", "response"]
)
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

Let's define some metrics for the evaluation of the model:

# metric

metric = evaluate.load("rouge")
nltk.download("punkt")

# postprocess text

def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

# compute metrics

def compute_metrics(eval_preds):
    preds, labels = eval_preds

    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(
        predictions=decoded_preds, references=decoded_labels, use_stemmer=True
    )
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [
        np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds
    ]
    result["gen_len"] = np.mean(prediction_lens)
    return result

Our final data preparation step is to use DataCollatorForSeq2Seq to handle the padding for inputs and labels:

label_pad_token_id = -100

data_collator = DataCollatorForSeq2Seq(
    tokenizer, model=model, label_pad_token_id=label_pad_token_id, pad_to_multiple_of=8
)

Finally, we can define the training arguments. Feel free to play around with these. Depending on the use case, results can vary!

# Define training args

training_args = Seq2SeqTrainingArguments(
    # training parameters
    output_dir=REPOSITORY_ID,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    predict_with_generate=True,
    fp16=False,  # Overflows with fp16
    learning_rate=5e-5,
    num_train_epochs=NUM_TRAIN_EPOCHS,
    # logging & evaluation strategies
    logging_dir=f"{REPOSITORY_ID}/logs",
    logging_strategy="steps",
    logging_steps=500,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    load_best_model_at_end=True,
    # push to hub parameters
    report_to="tensorboard",
    push_to_hub=False,
    hub_strategy="every_save",
    hub_model_id=REPOSITORY_ID,
    hub_token=HfFolder.get_token(),
    disable_tqdm=False,
)

# Create Trainer instance

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["test"],
    compute_metrics=compute_metrics,
)

Now, we can kick-off training! This can take ~20 mins if you use the small model, or about a couple of hours with the base model. Of course, this largely depends on how much data you have, what's your GPU, epochs, etc. Feel free to tweak the params above to your liking!

# Start training

trainer.train()

Once training is done, we can now push things to the hub!

tokenizer.save_pretrained(REPOSITORY_ID)
trainer.create_model_card()
trainer.push_to_hub()
tokenizer.push_to_hub(REPOSITORY_ID)

5. Generating conversations

Now, generation with these models can be quite a challenge. OpenAI for example, used RLHF to align these models, and reward them for generating nice outputs. Here's a great sketch that illustrates the importance of the different training steps:

In our use case, there are some parameters we can leverage when generating text. In this article, Patrick Von Platen goes through the different techniques & methods we can use to control the output of these models.

There are quite a few parameters you can tweak:

Either you are using greedy or beam search
Sampling
Top-K sampling
Top-P sampling
Size of n-grams to not repeat

I advise you to go through the article and learn a bit about each one of these!

Context given, let's get to it:

import random
import torch

Let's first load the model we pushed to the hub. (You can also just use the directory where your model is saved instead.)

# load tokenizer and model

REPOSITORY_ID = "flan-t5-small-chat/checkpoint-8000/"  # the name of your repository where the model was pushed
tokenizer = AutoTokenizer.from_pretrained(REPOSITORY_ID)
model = AutoModelForSeq2SeqLM.from_pretrained(REPOSITORY_ID)

# put model on GPU

device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = model.to(device)

To kick off the generation, we select a random converstaion in our dataset:

random.seed(502)
sample = random.choice(data["test"])
STARTING_TEXT = sample["conversation"]
print(STARTING_TEXT)

# Tiago Pereira: 12h EM PORTUGAL:

# Tiago Pereira: 2 votos por pessoa pfv.

# Hugo Silva: Nao é anonymous?

# Tiago Pereira: Não. Tudo visivel

# Luís Rodrigues: Foi uma bela administração sim senhor.

# Leonardo Soares: Obrigado a esta administração! Dinâmica, Pacífica, Estruturadora, Libertadora!

# Leonardo Soares: #XeJonnyLegislativas2026

# Leonardo Soares: (2026 right? 😅)

# André Ferreira: PORTUGAL HOJE

# Raul Carvalho: Ahahaha vamos!!!!

# remember to learn and tweak these params

generation_params = {
    "max_length": 600,
    "no_repeat_ngram_size": 1,
    "do_sample": True,
    "top_k": 50,
    "top_p": 0.95,
    "temperature": 0.7,
    "num_return_sequences": 1,
    "repetition_penalty": 1.3,
}

encoded_conversation = tokenizer(STARTING_TEXT, return_tensors="pt").input_ids.to(
    device
)
output_encoded = model.generate(encoded_conversation, **generation_params)

output_decoded = tokenizer.decode(output_encoded[0], skip_special_tokens=True)
print(f"Response:\n{output_decoded}")

# Response:

# Raul Carvalho: Mas é uma bela

Another fun thing to do, is to let the model generate conversations completely by himself.

The idea here is to:

Start with a real conversation (5 replies)
Generate a response using our model
Create a new exchange (4 real replies + 1 AI generated)
Generate a response
Create a new exchange (3 real replies + 2 AI generated)
etc..

Eventually, we'll end up with a bunch of AI generated conversations from our model! Here's the code to do that:

conversation = None
NUMBER_OF_ROUNDS = 5
for i in range(NUMBER_OF_ROUNDS):
    if not conversation:
        conversation = STARTING_TEXT

    encoded_conversation = tokenizer(conversation, return_tensors="pt").input_ids.to(
        device
    )

    output_encoded = model.generate(encoded_conversation, **generation_params)

    output_decoded = tokenizer.decode(output_encoded[0], skip_special_tokens=True)

    conversation = conversation + "\n" + output_decoded
    conversation = "\n".join(conversation.split("\n")[1:])  # remove first intervention
    print(f"New conversation:\n{conversation}\n----")

# New conversation:

# Tiago Pereira: 2 votos por pessoa pfv.

# Hugo Silva: Nao é anonymous?

# Tiago Pereira: Não. Tudo visivel

# Luís Rodrigues: Foi uma bela administração sim senhor.

# Leonardo Soares: Obrigado a esta administração! Dinâmica, Pacífica, Estruturadora, Libertadora!

# Leonardo Soares: #XeJonnyLegislativas2026

# Leonardo Soares: (2026 right? 😅)

# André Ferreira: PORTUGAL HOJE

# Raul Carvalho: Ahahaha vamos!!!!

# Raul Carvalho: Ahaha

# ----

# New conversation:

# Hugo Silva: Nao é anonymous?

# Tiago Pereira: Não. Tudo visivel

# Luís Rodrigues: Foi uma bela administração sim senhor.

# Leonardo Soares: Obrigado a esta administração! Dinâmica, Pacífica, Estruturadora, Libertadora!

# Leonardo Soares: #XeJonnyLegislativas2026

# Leonardo Soares: (2026 right? 😅)

# André Ferreira: PORTUGAL HOJE

# Raul Carvalho: Ahahaha vamos!!!!

# Raul Carvalho: Ahaha

# André Ferreira: Pa no estou?

# ----

# New conversation:

# Tiago Pereira: Não. Tudo visivel

# Luís Rodrigues: Foi uma bela administração sim senhor.

# Leonardo Soares: Obrigado a esta administração! Dinâmica, Pacífica, Estruturadora, Libertadora!

# Leonardo Soares: #XeJonnyLegislativas2026

# Leonardo Soares: (2026 right? 😅)

# André Ferreira: PORTUGAL HOJE

# Raul Carvalho: Ahahaha vamos!!!!

# Raul Carvalho: Ahaha

# André Ferreira: Pa no estou?

# André Ferreira: No estou a caralhar

# ----

# New conversation:

# Luís Rodrigues: Foi uma bela administração sim senhor.

# Leonardo Soares: Obrigado a esta administração! Dinâmica, Pacífica, Estruturadora, Libertadora!

# Leonardo Soares: #XeJonnyLegislativas2026

# Leonardo Soares: (2026 right? 😅)

# André Ferreira: PORTUGAL HOJE

# Raul Carvalho: Ahahaha vamos!!!!

# Raul Carvalho: Ahaha

# André Ferreira: Pa no estou?

# André Ferreira: No estou a caralhar

# André Ferreira: Ainda no estou?

# ----

# New conversation:

# Leonardo Soares: Obrigado a esta administração! Dinâmica, Pacífica, Estruturadora, Libertadora!

# Leonardo Soares: #XeJonnyLegislativas2026

# Leonardo Soares: (2026 right? 😅)

# André Ferreira: PORTUGAL HOJE

# Raul Carvalho: Ahahaha vamos!!!!

# Raul Carvalho: Ahaha

# André Ferreira: Pa no estou?

# André Ferreira: No estou a caralhar

# André Ferreira: Ainda no estou?

# Raul Carvalho: Haha nvel queres meeses

# ----

Governo Sombra transcripts

2023-04-19T16:30:00+02:00

7 years. That's how long I've lived in Denmark for. I love it, but Portugal is still close to my heart. As an emigrant, it's always hard to stay connected to what's going on in Portugal. What are people talking about? What's in the news? What worries people? What is everyone arguing about over morning coffee?

One of the ways I like to stay in touch is by listening to Governo Sombra (now cleverly called Program whose name we are legally prevented from saying, after changing networks). It's a weekly show where the 3 guests (+1 host) comment on Portuguese and World news. Besides being funny, I also love the fact that the 3 guests represent different parts of the political spectrum, so I can get a good idea about how most of the people are feeling.

Inspired by Lexicap, I decided to build a website with the transcripts for all of the episodes of the show. More than once I've listened to a particular part of an episode and wanted to share it with a friend. Now, I can do it.

For the transcription, I used OpenAI's Open Source Whisper model. With a small caveat: the whole thing (serving + transcribing) needed to run in my 20 EUR/month VM. So it needed to be small and efficient.

I like Python, but Rust was the obvious choice. For the transcription part, I used whisper.rs (Rust bindings for whisper.cpp). For serving the app, I went with Actix Web - it's small, efficient, and reminds me a lot of Flask. Incredible how a small Linux box can handle transcribing 60min+ episodes without hiccuping much.

The quality of the transcription is something like a 6/10. I did use the base model so there is clearly space for improvement. Maybe when I get a dedicated box.

The entire thing is up on GitHub.

LLMs in production: lessons learned

2023-03-26T20:35:00+02:00

A couple of months ago nobody asked me about my work. Something related to computers and AI. Fast forward to today, even my uncle asks me about ChatGPT. The hype is real. Only time will tell if the hype will materialize. But while the world wonders, work goes on.

In the last couple of months, I've helped develop a product that leverages this tech at its core. It was - to say the least - a learning experience. Full of lessons learned, full of little traumas and things I would've done better. In the hopes of helping someone out there, here are some lessons I've learned along the way.

Know your use case

With so much hype surrounding LLMs, it's easy to think they'll solve all problems in Machine Learning. Or at least the ones related to NLP. From what I've seen, this is hardly the case. Let's split Machine Learning tasks into two types. (1) Predictive tasks, such as classifying the sentiment of a tweet, and (2) generative tasks, such as summarizing the content of an article.

GPT is great at generative tasks. Writing an email with context, writing a summary of a web page, and creating an article given some ideas. These are a very specific subset of Machine Learning. Most of the problems we face are predictive problems: what is the sentiment of this tweet? What is the class of this image? It's hard to tell exactly how good these models will become for the predictive use case. But before throwing GPT at whatever you're facing, think about the use case. This leads me to my next point.

Deterministic vs. stochastic

Do you know the classic "I cannot reproduce this issue" we've said oh so many times? Well, welcome to a whole other level of that. "The AI said something wrong" is a very common issue I've faced with these models. At the core, LLMs are stochastic, and not deterministic. Before LLMs, whatever model we were building, given an input, would always produce the same output. With LLMs, given the same input, the output is rarely the same.

This is amazing for generative tasks but can become a real problem for predictive tasks. The problem of reproducibility. To avoid this issue, you can play around with some of the parameters of this model such as the temperature or the presence penalty. This further reinforces the idea that these models are great for generating text - where the cost of failure is low. If you're predicting something with a high cost of failure - best beware.

Prepare for the future

This tech is moving incredibly fast. Yes, even for us, the group of people that loves to move fast. OpenAI, is releasing models and updates at an astronomical pace. By the time you finished developing that shiny new product, there will probably be a new one. This happened to us actually. We started developing a system based on GPT-3 (e.g., text-davinci-003). Shortly after, OpenAI released the ChatGPT model (e.g., gpt-3.5-turbo). Two days before we went to production - GPT-4 (e.g., gpt-4) was here.

These models use different APIs, structures, and behaviors. Thankfully, I'd spent a Friday afternoon implementing the Strategy Pattern, so that we could support both Text Completion and Chat Completion for our system. Turned out to be time well spent, and adopting GPT-4 was a moderate one-line code change. Whenever a new model comes out, expect to be rate limited, and expect timeouts. Especially with these technologies, it's worth preparing for the future, and anticipating what might be released. Without exaggerating, of course.

Streaming vs. Batching

Yes, it comes down to that, a single API call to OpenAI. (Well, at least while we don't have a solid open-source alternative). Given a large enough prompt, these models can take quite a while to return a result. There's nothing worse than making a user wait for 20-30 seconds before showing some action on the screen - especially when that request ends up failing sometimes. After some days of frustration, a co-worker sent me this link.

Instead of waiting for the whole completion to be finished, consider streaming the response. In short, it allows you to start receiving completion tokens as soon as they're generated. Together with FastAPI's excellent implementation of StreamingResponse, it allows you to show the completion happening in real-time to users. This pattern is widely used by implementations such as Notion's. Now that all we have is an API, customer experience is even more differentiating.

Prompt engineering is hard

We have prompt engineers now. No wonder. I prefer the term prompt artists. As soon as we had to put a system in place we struggled. We are used to fancy tools to track every parameter of an experiment. Now that we're designing prompts for a stochastic system the game has changed. How can know the impact of every little change in the prompt? How can I know how one instruction affects another instruction? What really defines a good prompt? You see where I'm getting at. For now, it's an art - more than engineering.

Very soon, our prompt was a list of 10-15 instructions giving very specific directions on what we needed to generate. To battle this, we created a set of scripts that would allow us to quickly compare the impact of small changes in the prompt. This is a costly exercise - given a stochastic system - since we have to call a paid API 2/3x to compare the effect of prompt changes. This allowed us to at least get a decent direction on how good the prompt is. Still, I feel like prompt engineering is still a lot of shooting in the dark.

Conclusion: Not the hammer you were looking for

LLMs are incredibly powerful tools. From generating text to now even supporting images - it's hard to tell where the field will lead us. For creative and generative tasks they shine like nothing before in our field. Being able to have access to such a powerful system will be groundbreaking in a lot of fields of machine learning. But with great power comes great responsibility - or baggage.

Putting an LLM into production is a challenging problem with a lot of unknowns. From depending on a simple API call, to not being able to reproduce results - it can get complicated.

If the task is simple enough (e.g., classification) it's still hard to justify a more expensive, less explainable, and albeit slower system than traditional methods. I don't think this is the hammer for all our problems. At least not yet.

parlabot - ask the Portuguese parliament

2023-02-27T18:55:00+01:00

Large language models (LLM) are really dumb. I mean, how can you fail when the question is as simple as "What is 23 times 18"? Even though they're making most headlines, at the end of the day, these models are, predicting the next token based on the previous ones. If we ask a super simple question without any context to an LLM, the performance will only depend on how much the model has seen that example.

But LLMs are also amazing. Have you used GitHub copilot lately? I don't trust it to write my functions for me, but damn it writes a good boilerplate. It's like auto-complete, but better. It doesn't only know your imports. It knows where you are in the code base, and can suggest based on that.

The difference? Context. When using copilot, your code is the context, and that's why copilot knows exactly what to suggest. So how can I give the right context to one of these models? To answer this, I built parlabot. Parlabot (parlamento + bot) uses all transcripts from the Portuguese parliament to answer any question you might have about Portuguese politics. It also (tries) to do so in the most truthful way it can, with the information it has.

There are two things at play when you ask a question to parlabot. Below is a sketch that might make things easier to follow.

The first part is the search engine. To make this work, I scrapped all transcripts from the Portuguese parliament and used a multilingual embedding model to transform them into vectors. Whenever you type in a question, I embed it using the same exact model, transforming it into another vector. I then find the top k most similar reference vectors to the query vector. This is the same as finding the top K most relevant speech segments for that question.

Once I have the most relevant speech segments, it's time to build the prompt. The prompt, has two parts, the context, and the direction. The context, is simply the most relevant speech segments, the corresponding speakers, and the political parties. In the direction part of the prompt, I ask GPT3 to answer a given question using the segments from the context.

The results are pretty good! The bot is very capable of using the context given to answer most questions. Of course, the context is not always relevant to a question. For example, if you ask "How to bake a cake?", it's unlikely we'll find the answer in the data set of parliament transcriptions. With good prompt design, we can make the bot answer "I don't know" when this is the case.

The mix of LLMs, politics, and poorly written code is a great recipe for disaster. Although disaster does sound pretty fun, I don't expect this bot to answer truthfully any of the questions Portuguese tax-payers have about elected parties. These answers should be taken with a massive grain of salt and skepticism. LLMs are stochastic beasts that might predict two different outputs with the exact same set of inputs.

Still, this system might be interesting to solve problems where the cost of failure is not catastrophic. Without forgetting, that a bit of skepticism is a great foundation for any ML system.

An opinionated Python boilerplate

2023-02-18T00:00:00+01:00

There's nothing quite like starting a new project. A greenfield, filled with possibilities. It's a privilege many don't come across. A lot of us, get thrown into projects with a lot of legacy code. But sometimes, we start from scratch.

This is the time. The time to make all the right calls. The time to use the right tools, the right abstractions, and the right structure. The unfortunate truth is, there is no right way. There are just ways. Most of us use the tools we're comfortable with. We've tried things in the past. Some worked, some didn't.

I've started my fair share of Python projects. By failing, a lot, I've converged to a set of tools. These will change over time. But in the hope to help a fellow Pythonista out there, let's talk about them.

Pip-tools for dependency management

Now I don't want to start a packaging war. If Poetry works for you, then by all means, go for it. I've tried most of the stuff out there, from the good old pip freeze > requirements.txt, pipenv, all the way to Poetry. After many battles, I've stuck with pip-tools. Pip-tools strikes the right balance between simplicity, effectiveness, and speed. And yes, speed matters. I don't want to wait a whole minute for my dependencies to compile.

With recent pip updates, I can just specify my dependencies in a pyproject.toml and install them with pip install -e .. However, there are benefits to pinning your dependencies using something like pip-tools. Especially in Machine Learning environments. Yes Anaconda exists - but speed is a requirement.

The folks at Jazzband have created an easy-to-use tool that has yet to let me down. For example, suppose I have a dreambox project. Here's an example pyproject.toml file:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "dreambox"
version = "42"
dependencies = ["pandas>=1.5.3", "numpy", "fastapi"]

[project.optional-dependencies]
dev = ["pytest"]

To create a pinned and hashed requirements file , all you need to do is:

(env) $ pip-compile --generate-hashes --output-file=requirements.txt pyproject.toml
(env) $ pip-compile --generate-hashes --extra=dev --output-file=requirements-dev.txt pyproject.toml

That ensures whatever you build in production is reproducible in my own machine, to the exact dependency and hash.

Pyproject.toml for configuration

Who likes configuration files? I certainly don't. There's nothing nice about having 15 configuration files at the root of my project. One for test coverage, one for linting, one for GitHub, one for formatting, and another one for CI. No thanks.

Fortunately, PEP 621 happened. And with it, a (mostly) common way to store all of the metadata and configuration for a Python project. Using a single pyproject.toml, I can define my local package name and details, my pinned dependencies, my pytest coverage configuration, my formatting configuration, my... You get me. All the configurations, in a single file. Expand the example below for an example.

An example pyproject.toml

# packaging information

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

# project information

[project]
name = "dreambox"
version = "23.1.26"
readme = "README.md"
requires-python = ">=3.10"

# requirements.txt generated from here

dependencies = [
  "Jinja2>=3.1.2",
  "loguru>=0.6.0",
  "fastapi>=0.88.0",
  "uvicorn>=0.20.0",
]

# requirements-dev.txt generated from here

[project.optional-dependencies]
dev = [
  "black>=22.10.0",
  "isort>=5.10.1",
  "pip-tools>=6.10.0",
  "pytest>=7.2.0",
  "pytest-cov>=4.0.0",
]

# linting config

[tool.ruff]
ignore = ["E501"]

# isort config

[tool.isort]
profile = "black"
line_length = 79
skip = [".env/", "venv", ".venv"]

# coverage config

[tool.coverage.paths]
source = ["src"]

[tool.coverage.run]
branch = true
relative_files = true

[tool.coverage.report]
show_missing = true
fail_under = 80

# formatting config

[tool.black]
line-length = 79
extend-exclude = '''
/(
  | .env
  | .venv
  | venv
  | notebooks
)/
'''

Makefiles for common sense

I clone a new repo. Now, where the hell do I start? How do I run the tests? How do I run the API? There should be a common standard for these things right? Wrong. Every project is different. Every project has its pet peeves. I don't really like pet peeves. Anyone should be able to pick up my project and get up and running straight away. No meetings, no calls, just start working. A well-documented README.MD might work. But you'll get lazy.

So what is the easiest way to add a good project "map", without much work? Enter my Makefile. With a Makefile, I define all of the main project commands in a single file. For example:

## Install for production

install:
    python -m pip install --upgrade pip
    python -m pip install -e .

## Install for development

install-dev: install
    python -m pip install -e ".[dev]"

## Build dependencies

build:
    pip-compile --resolver=backtracking --output-file=requirements.txt pyproject.toml
    pip-compile --resolver=backtracking --extra=dev --output-file=requirements-dev.txt pyproject.toml

If I want to install the dependencies on a project, all I do is make install. Or make install-dev for the dev dependencies as well. This is nice. But my favorite feature is this:

With the help of this small script at the end of my Makefile, I can generate a map for my project. Now, whenever someone new comes in, all they have to do is make help, and they'll get a good idea of how to navigate things. The script takes the comment above every rule in the Makefile, so it's super easy to add new commands to it.

Now isn't that nice?

Ruff for linting

I like Python, but I also like Rust a lot. Ruff is a new (and shiny) Python linter written in Rust. It's probably the most modern project in my boilerplate. It does what any linter should do, it should flag errors and bad practices. I like catching errors before they blow up in the user's face.

Most importantly, Ruff is fast. Blazingly fast one might say? Running ruff in our project will probably take less than half a second. Also, like all other tools I mentioned here, Ruff supports pyproject.toml for configuration. So you don't have to have to maintain yet another file. (I'm looking at you flake8)

Black & isort for formatting

I never got the whole tabs vs. spaces thing. But I do know programmers have opinions. So many opinions. A cool thing about working in software is that you can use software to squash those opinions. We just agree on using a couple of tools to format our code, all other discussions become useless. So thanks, Łukasz, for creating Black.

Most of my projects have a great make format rule, that runs black and isort. The configuration? It's in my pyproject.toml file above.

## Format files using black

format:
    isort .
    black .

Note: Apparently ruff supports import sorting now? Wow. My boilerplate is already obsolete.

pre-commit hooks (I don't)

I've seen a lot of projects adopt pre-commit hooks as a way of enforcing standards for code. Whether it's to make sure tests pass or to ensure nothing gets raised by linters, pre-commit hooks are all the rage. The thing is, git is already pretty complicated. I mostly work with data scientists, and most of them already face a steep curve in adopting software practices. From my (opinionated) experience, pre-commit hooks make things even more complicated.

But yes, some standards should be enforced, otherwise, something will break for sure. But isn't that what CI is for? Enforcing something like formatting and linting is easy with the --check flag:

## Run checks (ruff + test)

check:
    ruff .
    isort --check .
    black --check .

If I run make check and something doesn't comply, I'll get an error. When opening a PR with a new feature, CI can take care of all of those checks:

on:
  pull_request:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install dev requirements
        run: make install-dev
      - name: Check formatting # <- if this does not pass
        run: make check
      - name: Run tests # <- you don't get here
        run: make test

If something doesn't run, CI doesn't pass. No need to mess around with git in everyone's machine. Everyone is free to commit whatever they want - and will get notified if something is not working. That's good enough.

An example project

At some of my previous talks, people normally ask me to share my project template (e.g., my cookiecutter). For me, sharing project templates it's a bit like copying someone else's Vim config. As soon as you need to tweak something (and trust me, you will), you'll have a headache.

This boilerplate is made of choices (or traumas) I've experienced over time. It's tailored exactly to my needs (e.g., data science, machine learning, API design). I like simple things a lot.

I do think it could serve some as a starting point for their own opinionated boilerplate. So here's the code, fellow Pythonista. Have fun out there.

Updates & notes

This post was featured in Episode #326 of the PythonBytes podcast

On Rust

2023-01-02T00:00:00+01:00

It's that time of the year again. The family is getting together and celebrating. Grandma is cooking something amazing for dinner. I'm trying to solve Advent of Code puzzles. This year, I decided to do something different. Instead of solving the puzzles in Python, I decided that I would try to solve them in Rust. Why?

Rust has been getting a lot of attention lately. The language is in its 7th year as the most loved programming language according to the latest Stack Overflow developer survey. Why do people love it so much? So I decided to learn it. What better way to get frustrated during my Christmas break?

Things I like

Speed. That's the first thing I noticed. I brute force a lot of these AoC puzzles. Rust handled them without hiccups. It's a bit like the feeling of using Numpy if you're a Python dev. Having been one for some years, I never got much exposure to compiled languages like C++ or Java. I can't help but notice a pretty big difference in speed compared to Python or JavaScript.

Effectiveness. Most puzzles I solved were right on the first try. This is rarely the case with Python. Rather than using types as more of a decorative/ergonomic feature like in Python, in Rust, these types are strictly enforced at compile time. This made me think twice about the code I wrote. But also lead to more correct code. Rust-analyzer, Rust's LSP, was also a great experience. Giving timely, useful, and clear messages about what was wrong.

Syntax. The syntax is not that different from Python. For loops are easy and readable with the for x in y syntax, for example. Closures remind me a lot of lambda functions in Python. I can also see why Python adopted the match statement. In general, Rust was readable. Whenever code got complicated and repetitive, things like macros came to the rescue.

Ecosystem. Compared to Pip, Cargo is a breath of fresh air. Installing packages is as easy as copy-pasting into your Cargo.toml file. In my experience, even larger packages such as naglebra have been fast to install. To check the documentation of all packages installed locally, you can also use cargo doc --open, which is great for offline development. There are also good resources online when you're feeling stuck, like the Rust Programming Language Book, or the more practical Rust by Example.

Things I don't like (yet)

Efficiency. Rust takes longer to write. Probably, due to my lack of experience. I get the impression programs need to be better thought out. You'll take longer to write code, but it will be more effective code. This makes Rust great for production. However, it's hard to think of Rust as a good language to experiment, explore, or even iterate quickly. Experimentation is a great advantage of languages like Python or JavaScript. It's a trade-off.

Types. I like types. I've been using types in most of my Python production applications. Not only that, but I believe I now understand better why Python doesn't enforce types. Types help you write more correct programs, but it's easy to get stuck in type hell. I don't know if I want a usize or an int32. I don't care if it's a Matrix or a DMatrix, or f32 or f64. The Rust compiler does. When I'm focused on solving a problem, I'm not interested in solving a type problem. Types are great. But if you get too strict about them, types can get in the way. I felt Rust types got in the way a lot of times.

Ecosystem. Rust's ecosystem is still young. Yes, AoC puzzles are very specific and don't represent the real world. But when looking for libraries, for example, it appears Rust is still trying to figure out where does lie. This is normal. There are not that many resources on the internet yet. For a beginner programmer in any language, the quality of web resources is everything. When looking for answers and resources on Rust, it takes longer to find the right resources. When looking for libraries, it's tough to understand which libraries are the most popular ones, or the ones to go from. Even though things are developing quickly.

struct Vector3 {
    x: f32,
    y: f32,
    z: f32
}

fn dot_product(a: Vector3, b: Vector3) -> float {
    a.x*b.x + a.y*b.y + a.z*b.z
}

fn do_math_by_copy(p1: Vector3, p2: Vector3, d1: Vector3, d2: Vector3, s: f32, t: f32) -> f32 {
    let a = p1 + s*d1;
    let b = p2 + s*d2;
    dot_product(b - a, b - a)
}

fn do_math_by_borrow(p1: &Vector3, p2: &Vector3, d1: &Vector3, d2: &Vector3, s: f32, t: f32) -> f32 {
    let a = p1 + &(&d1*s);
    let b = p2 + &(&d2*t);
    let result = dot_product(&(&b - &a), &(&b - &a));
}

Borrowing. References and borrowing have been the hardest concepts to grasp for me. Again. This might be very well due to my lack of experience in compiled languages. Look at the example above, taken from this great post. There is no obvious reason why you would go with do_math_by_borrow. It's uglier, less readable, and to the best of my knowledge, not significantly faster. This whole borrowing and ownership dance is probably necessary for Rust. But I felt like doing a lot of .Clone() and .Copy() to escape this problem.

Closing thoughts

AoC was great to learn a new language! Perhaps next year something easier though. I didn't finish all the puzzles by any means. But by day 10, I had a good grasp of the basics of Rust and could write simple programs without too much hassle. The problem with AoC is the steep curve in puzzle difficulty. By day 15, I was already spending one hour just to parse the input correctly.

I can understand why Rust is loved. It's fast, effective, and it makes you a better programmer. It requires you to make deliberate choices on every variable you create, and every statement you write. This is a double-edged sword. On one side, it makes you write better and safer code. On the other side, it makes you write code slower, and experiments should be fast. It does look great for production. But it's early still.

I would love to try and write some Machine Learning APIs in Rust. Written in Rust, they might be safer and faster! But for that to happen, I would need to be able to load models using Rust. This means loading Scikit, Transformer, or even PyTorch models with Rust. I don't think we are there yet. But when we are, I would love to give it a go.

Scaling Machine Learning microservices

2022-12-18T00:00:00+01:00

Monitoring Machine Learning APIs

2022-12-06T10:30:00+01:00

Last Friday I presented at PyData Global. I talked about monitoring, but felt like there was more I should've said. So this article is just about that.

Even though the number of models in the wild is growing, the field of monitoring is still green and full of unknowns.

Almost all models I've put into customers' hands have had some sort of monitoring. In my head, I've divided ML monitoring in three main areas.

Let's explore them.

Logs & Error monitoring

This first area is responsible for monitoring three things:

Errors: Something in the application blew up, and needs to be checked (e.g., trace backs, etc.)
Logs: What's going on? When is it going on?
Performance: How fast is the API? How much load can it take?

This is very similar to traditional software observability. And yes, most machine learning applications - are, in fact, software applications. This means we have all the concerns of traditional software, and then some.

When it comes to logging, loguru has made life easier. At the end of the day, who likes to configure loggers?

Another less known component is OpenTelemetry. OpenTelemetry has slowly become the standard in observability. Think of it as a wrapper that allows you to capture useful statistics on the performance of your app. It also allows you to capture some custom events/data that are useful for debugging.

Here's an example:

import fastapi
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from .models import Result, Item

# set up tracing and open telemetry

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

# instrument FastAPI

app = fastapi.FastAPI(title="demo")
FastAPIInstrumentor.instrument_app(app)

@app.post("/predict/", reponse_model=Result)
def predict(features: Item):
    # get the current span
    current_span = trace.get_current_span()

    # hash input
    input_hash = hash(features)

    # save hash to opentelemetry
    current_span.set_attribute("app.demo.input_hash", features_hash) # <- Saves attribute

    # return predictions
    prediction = get_prediction_for(features)
    return prediction

Let's walk through the example:

We kick off manual instrumentation using the TracerProvider(docs)
We add automatic instrumentation with the FastAPIInstrumentator (docs)
We have the ability to send custom events with the current_span.set_attribute function (docs)

If you're using something like DataDog, Honeycomb, or any other provider that supports OpenTelemetry - you should get all this information on your service's dashboard. You'll get all the stats related to your application, and all the custom events you are sending in as well.

This is particularly interesting to understand, for example, when your API is breaking, and what is making it break!

Drift/Degradation monitoring

The second area of monitoring is about model degradation in the face of new data. This usually happens as models in production start being outdated. Trying to make predictions on types of data your model has never seen before, usually leads to problems. This concept is known as data drift.

In a previous article, I talked extensively about this topic. I explained how to leverage FastAPI and Evidently to monitor drift, in real-time, for your ml application. So go read that if you're interested.

There is, however, one concept I should double down on. The fact of storing all inputs (e.g., requests) and outputs (e.g., predictions) of your model. Here's an example of doing so - without increasing response latency:

# ...

@app.post("/predict/", reponse_model=Result)
def predict(features: Item, background_tasks: fastapi.BackgroundTasks,):
    prediction = get_prediction_for(features)

    # saves both inputs and outputs as json to our predictions database
    # runs in the background - doesn't make anyone wait
    background_tasks.add_task(
        save_to_database,
        input=features.dict(),
        output=asdict(prediction),
        created_at=datetime.datetime.now()
    )
    return prediction

# ...

This prediction sink (red in the image above), is key. As described in the previous article, it allows you to monitor drift. But it also allows you to get insights into the third area of monitoring. Let's talk about that one.

Business monitoring/reporting

The third and final area of monitoring relates to everything else you might want to keep track of. Your team might want to know how many predictions the API has made in the past month, or how many times you've predicted a certain class. Maybe you, or some other team, is tracking KPIs that are highly related to your model. Maybe the finance dept. wants to know the new CTR based on your recommendations.

The answers to these questions often come in the form of a dashboard.

Fortunately, if we store our inputs and predictions in a flexible enough format (e.g., json), we'll be able to take almost any off-the-shelf tool and make a dashboard. Designing Metabase dashboard from your historical json predictions stored in BigQuery should be straightforward. You could also stick to Google Data Studio. Or use Excel. You get my point, you can use whatever. json is flexible enough to be sorted by created_at and pushed into a dashboard.

Although often overlooked, this third level of reporting will often debunk the whole "your model is a black box" paradigm. It allows other stakeholders to know exactly what is going on in your model.

Closing thoughts

Yes. Adding all of these different types of monitoring requires some boilerplate code. And of course, setting this up might take some time. Especially if you do these to every model in production.

However, it's certainly impressive just how much insight you can get from some OpenTelemetry auto-instrumentation and a background job to save all your predictions.

Sufficiently monitoring your ML model increases your team's confidence when putting models out there. It's a bit like testing - you are now more sure things are happening as they should. With the third area of monitoring, you can then extend that feeling to the rest of the organization.

"Machine learning is such a black box". Open some doors to the box, and let other people in.

Goodbye Apple Watch

2022-10-29T12:00:00+02:00

1768 days.

That's how long I've used the Apple Watch. Just a little under 5 years, in case you're wondering. I really love it. I don't know about "future of computing" and all that, but it's definitely a device I've fallen in love with. Last week, for the first time, I ditched it. Let me explain why.

My last marathon didn't go that well. A story for another day. In the aftermath, I've been thinking about my training process. What went well, what went wrong, you know, introspection and all that. So I did what I do best. I got too deep into the topic. I research how other people train. I explore the vast sea of resources about running. You come across some pretty cool resources. And some scary ones.

I was watching Kipchoge, and how he just broke the WR in Berlin. What a guy. You know what watch he uses for running? It's called Coros Pace 2. I'm far from a Kipchoge, but for 200 EUR, I had to know more. After some weeks of e-window shopping and a lot of YouTube reviews, I ordered it.

The good

20 days. That's how much this watch can last in a single charge. Yes, you read it right. 20 days. For someone who is used to charging an Apple Watch every single day, that's a pretty nice upgrade. I don't have to worry about charging it every night, I don't have to carry an extra cable every single time I travel. I just go, and it will last the whole trip.

Obviously, I bought this watch for the fitness capabilities. Compared to the Apple Watch, it delivers. Setting up the Apple Watch for interval training, or any type of specific workout can be a pain. It doesn't even come close to the Coros. The workout views are much more informative and the after-workout analyses are much more insightful. That' s expected. What I didn't expect, is a full web-based training portal where I can track all my workout metrics. Now that's a nice surprise.

The training hub and the Coros app (which is not that bad) give me much better control of my training plan. I can plan 3 months in advance and design all my workouts exactly like I want. For every workout, Coros gives me the Aerobic and Anaerobic training effect, my effort, my performance, so I know exactly what went down.

Sleep tracking. You know that thing you sometimes do with your Apple Watch, but not too often since you don't know if you'll have time to charge in the morning? Yup. Since I don't have to charge the Coros overnight, I get access to all my Sleep stats. I know exactly how much and how well I've slept over time. You know, 'cause sleep matters.

The bad

It's not an Apple Watch. Yes, I could write about the fact that some parts of the Coros app are kinda rough, or the fact that the watch interface is a bit clunky. Or even the fact that it uses some sort of weird proprietary charger. But the bottom line is, I keep comparing it to the Apple Watch.

One side of me now understands why Apple has never given us custom watch faces. Coros has at least 100 watch faces to chose from, and they all look… well, not great. This watch is clearly missing the aesthetic appeal that the Apple Watch has.

I have a love/hate relationship with Apple's beautiful walled garden of an ecosystem. How can I miss paying with my wrist so much? How can I miss controlling music playback on my wrist so much? Seeing those circles close in every widget I own. Getting all the notifications in your watch. The next event in your calendar at wrist glance. The "Do not disturb until I leave this location", two taps away. Even the haptic motor.

I miss the cozy ecosystem. It's a bit like a withdrawal - it will fade. I think.

The end

I don't think there's a winner. They're different devices - with different goals, and different target audiences. One, focuses on design, ecosystem, wellness, and lifestyle. The other is a powerful fitness tracker, and does it very well.

I could just use both. But I don't want to add that stress to my life. For now, a good run brings me more joy than paying with my wrist. A good night of sleep is more important than seeing all my calendar events in my wrist. Focus, is more important than constant notifications.

So I'll stick to the Coros for now.

infrequent.app - stay in touch with those who matter

2022-10-23T19:00:00+02:00

Some months ago, I built a small script to help me stay in touch with those who matter to me. The concept is fairly simple: I want to stay in touch with person p every t period of time. If t has passed, and you haven't talked to p, you get a reminder.

I've been using this script/system for the past 5 months, it has served me well. I've been keeping up with mentors, ex-managers, family, and friends - without much effort. Which is the goal. It's not that I don't like editing a bunch of markdown files every week - but I wanted something smoother.

Also, the more I've told people about my system (which is not even mine), the more I've realized that I'm not the only one who needs something like this. Generally, people have a hard time keeping in touch. It's not that we don't care - it's that life gets in the way.

And so I built infrequent.

Infrequent is a system to stay in touch with those I care about. A system that I built literally for myself, to scratch my own itch. But also a system that I hope helps others out there.

I'm not gonna lie, it's been pretty fun to get back into web development. I love AI&ML, but there's something special about web development. And Django, htmx, and tailwind have been a joy to work with. The stack is not straightforward to set up, but once you get going, it's a breeze to work with. Django is an incredibly powerful framework.

I don't have million-dollar plans for infrequent just yet. For now, I plan to continue using it, and entice others to do so too. I'll keep adding features to it, as mine (and maybe your) needs evolve. Feel free to give it a spin, and don't hesitate to email me if you like it, or think something is missing.

Monitoring ML models with FastAPI and Evidently AI

2022-09-17T15:00:00+02:00

I've deployed a good amount of ML models to production. For many, deployment to production is the last step of the process. Once that's done, work is done. This is far from true. Once your model is out there, problems will start to arise. Some predictions will be wrong. Some labels will occur more often than they should, and some examples will surprise the model.

Setting up the right lenses & triggers into your model is critical. It helps to ensure everything is running smoothly, or to know when issues need to be tackled. Let's open up the black box.

Setting up FastAPI

I've previously written about how to serve your ML model with FastAPI. Let's assume we're serving our model with FastAPI and our src folder looks something like:

src
├── __init__.py
├── api <- your Fast API folder
│   ├── __init__.py
│   ├── models.py
│   └── main.py
├── pipeline
│   ├── __init__.py
│   ├── ...
│   └── train.py
└── setup.py

The main.py file is where our views are defined. Below, our main file, with a /predict endpoint, that serves predictions.

# app.py

from typing import Optional
from fastapi import FastAPI
from src.pipeline import get_prediction_for
from .models import Result, Item
import joblib

# create FastAPI app and load model

app = FastAPI()
model = joblib.load("model.joblib")

# create an endpoint that receives POST requests

@app.post("/predict/", reponse_model=Result)
def predict(features: Item):
    # some processing
    prediction = get_prediction_for(features)
    return prediction

Prerequisite: Storing all predictions

To monitor our model, we must first make sure two things are happening:

Prediction logging: We're actively logging all predictions our model is making
Access to a reference dataset: We have access to the dataset where our model was trained (e.g., the training data)

Logging all predictions your model makes can be done using some managed database service (e.g., think Aurora, BigQuery, etc.). Ideally, we want to do this without increasing our prediction latency.

Fortunately, Fast API provides gives a great tool to do this: BackgroundTasks. We start by creating a function that saves our data (in this example, to BigQuery):

# monitoring.py

# ...

def save_to_database(input: Item, result: Result) -> None:
    """
    Saves input/output dicts to bigquery
    """
    client = BigQuery.client()
    table = "your_cool_bq_table"
    current_time = datetime.datetime.now()

    rows_to_insert = [(current_time, input.json(), result.json())]
    errors = client.insert_rows(table,
                                rows_to_insert)

    if errors:
        logging.info(f"Error: {str(errors)}")
        return

    logging.info("Saved prediction")

We can now add it to our API as a background task:

# app.py

# ...

from fastapi import FastAPI, BackgroundTasks
from .monitoring import save_to_database

# ...

# create an endpoint that receives POST requests

@app.post("/predict/",
          reponse_model=Result,
          background_tasks: BackgroundTasks)
def predict(features: Item):
    # some processing
    prediction = get_prediction_for(features)
    background_tasks.add_task(save_to_bq, input=features, result=prediction)
    return prediction

Notice how the background task does not block the prediction time. Allowing us to keep prediction latency as low as possible, while still, saving all predictions.

Setting up the monitoring

Evidently is a great open-source tool that allows you to set up monitoring for your ML models. It's not the only one, there's a myriad of them, actually. nannyML is another one.

Evidently allows you to generate a bunch of different reports you can generate. In this example, I'll focus on the Data Drift dashboard.

The Data Drift dashboard allows you to measure the difference in distribution between the predictions you are making, and the labels of your training set. When these two start to become significantly different, you are likely encountering some drift.

Alright, let's build it. We start by creating a couple of functions in our monitoring.py module:

# ... rest of the monitoring.py

DATA_WINDOW_SIZE = 3000 # how many predictions to load

# loads our training/reference dataset

def load_train_data() -> pandas.DataFrame:
    train_file = "static/train_data.csv"
    train_df = pandas.read_csv(train_file)
    return train_df

# loads our latest predictions

def load_last_predictions() -> pandas.DataFrame:
    query = f"""
    SELECT created_at, input, output
    FROM `my_cool_bgq_table`
    ORDER BY created_at DESC
    LIMIT {DATA_WINDOW_SIZE};
    """
    prediction_data = pandas.read_gbq(query=query)
    return prediction_data

Now that we're able to fetch both our reference data and our past predictions, we're ready to build our Data Drift dashboard:

# ... rest of the monitoring.py

# this function generates a dashboard from our reference and prediction data

# which is then saved to a `drift.html` file

def generate_dashboard() -> str:
    dasboard_name = "static/drift.html"
    data_drift_dashboard = Dashboard(
        tabs=[
            DataDriftTab(verbose_level=0),
        ]
    )

    reference_data = load_reference_data()
    current_data = load_last_predictions()

    data_drift_dashboard.calculate(
        reference_data=reference_data,
        current_data=current_data,
        column_mapping=None,
    )

    data_drift_dashboard.save(dasboard_name)
    logger.info(f"Dashboard saved to {dasboard_name}")
    return dasboard_name

Notice how we're creating our dashboard, and then saving it to a static/drift.html file. The idea is then to serve this dashboard in one of our FastAPI endpoints.

Monitoring dashboard

Let's serve our data drift dashboard:

from .monitoring import generate_dashboard

# ... rest of the main.py

@app.get("/monitoring", tags=["Other"])
def monitoring():
    dashboard_location = generate_dashboard()
    return FileResponse(dashboard_location)

Every time we visit /monitoring, Fast API will run the generate_dashboard function, and return an html file:

As you can see, this dashboard compares the distribution of our reference and current dataset. The current dataset being the latest Y predictions we've made.

Closing thoughts

I've found this to be a relatively straightforward way of adding a bit of visibility to what's really happening in my production models. If those distributions are looking particularly skewed: you know it's time to act.

Evidently allows us to generate much more than just a data drift dashboard. You can also generate dashboards to monitor data quality, the performance of a regression, classification performance, and many more. It's worth taking a look at their docs to see what fits your use case best.

There's a way we could increase the speed here. Instead of computing the entire dashboard every time we visit /monitoring, we could compute it every X time period in the background. This would result in much faster response from the /monitoring endpoint.

Is this dashboard enough to make sure everything is going well in production? No. But it's a great first step towards figuring out what's really going on.

Updates & notes

This post was republished by the MLOps Community blog
I also wrote a more general post on how to monitor machine learning APIs

Down from the Cloud

2022-08-05T11:00:00+02:00

I love the Cloud. For years, I've been deploying software to it. Azure, GCP, AWS, you name it. I've used most of them. To be honest, they're the same pig, but with different lipstick (like the Danes say). Package up your app, make some CI/CD magic, select your service, and I'm good to go. Still remember when I helped a whole bank move to the Public Cloud. The benefits are mostly obvious.

But I'm no bank. And it's not all rainbows and unicorns. For starters, those 10 elastic beanstalk applications can start adding up. What if I want to change from Cloud A to Cloud B? It's not like Amazon is going to make my life easy. It's a convenience/lock-in trade-off. What about user data? What about my data? Some of the best software I use doesn't even use the Cloud.

Let's come down from the Cloud (at least the public one): Self-hosting. How hard is it, really?

Get a server

This one should be straightforward. There's no shortage of server providers. Just browsed LowEndBox for a while I decided to go with a dedicated server from Hetzner. Something relatively close to Copenhagen to ensure my connection to the server is snappy enough.

I don't run a lot of very high traffic sites (yet!). Something around the 50 EUR/month price point serves just fine. You'll be surprised with how much stuff you can run in this machine. I have 10 services running and my memory barely goes above 10%. Let's not jinx it.

Containerize all the things

I have a love-hate relationship with Docker. For me, it's still the most straightforward way of exchanging and packaging up software to ensure nothing breaks. But how I hate when my container starts getting fat. I'm talking about you, PyTorch.

Every application will run inside its own containerized environment. To ensure this doesn't get overly complicated, I went with docker compose. Why not Kubernetes you may ask? Because I have other things to do.

Inside the server, things look a little something like:

|-- Caddyfile <- more on this later
`-- projects
    |-- project-1 <- first application
    |   |-- ...
    |   |-- Dockerfile
    |   `-- docker-compose.yml <- docker compose file
    `-- project-2 <- second application
        |-- ...
        |-- Dockerfile
        `-- docker-compose.yml <- second docker compose file

Every project gets its own docker-compose.yml file. Every project runs on its specific server port. For example, to run project-1, I cd into the directory and start the container: docker compose up --force-recreate --build -d. This automatically starts the service on the port I specify on the docker-compose.yml file.

Using the process above, I can have a bunch of different applications running on a lot of different ports, but in a single server. Hopefully (1) saving money, and (2) increasing the power and control over them.

Hopefully, I won't land on dependency-nightmare-land.

Continuously deploy

Self-hosting sounds great. Having to ssh into a server every time I want to update a service, doesn't. If there's one thing I'm not willing to compromise on, it's continuous deployments. The whole git push automatically updates the app thing is super convenient.

Enter GitHub actions. With them, you can automate the ssh'ing and re-deployment part. It's not the most secure option. But hey, it works pretty well. Surprisingly, it's also much faster than deploying on Elastic Beanstalk or Cloud Run.

Here's what these actions look like:

name: Deploy to server
on:
  push:
   branches:
   - master
jobs:
  build:
    runs-on: ubuntu-latest
    steps:

    - name: Checkout source code
      uses: actions/checkout@v1

    - uses: actions/checkout@master
    - name: Copies repository to server
      uses: appleboy/scp-action@master
      with:
        host: ${{ secrets.HOST }} <- these are defined in your repo settings.
        username: ${{ secrets.USERNAME }}
        password: ${{ secrets.PASSWORD }}
        port: 22
        overwrite: true
        source: "."
        target: "/root/projects/project-X"

    - uses: appleboy/ssh-action@master
      name: Stops and updates docker container as deamon
      with:
        host: ${{ secrets.HOST }}
        username: ${{ secrets.USERNAME }}
        password: ${{ secrets.PASSWORD }}
        port: 22
        script: |
          cd projects/project-X
          docker compose down
          docker compose up --force-recreate --build -d
          docker ps

HTTPS, we meet again

Alright, it's time to expose these apps to the world. I'll admit it. The sheer mentioning of https certificates or DNS runs a chill down my spine. After loosing a couple of hours in the reverse proxy documentation for Nginx and getting a bit mad, I found this little thing called Caddy. Oh boy, that was easy.

Caddy solves the whole subdomain, reverse proxy, multiple websites on the same server, HTTPS mambo jumbo. The docs are great, and it's super straightforward to use. I definitely recommend their Getting Started docs.

First step is to point some A record in your DNS to your server's IP address. I use Cloudflare for this, do recommend. Once Caddy is installed in the server, all you need to do is create a Caddyfile:

project-1.mydomain.com {
        reverse_proxy localhost:3000
}

project-2.mydomain.com {
        reverse_proxy localhost:5000
}

home.mydomain.com {
        respond "Base domain"
}

After that, run caddy start, and we're done. Yes. Your eyes are not kidding you. That's absolutely it. project-1.mydomain.com is connected to my app running on port 3000, and project-2.mydomain.com is connected to the container on port 5000. HTTPS? Automatic. Reverse proxy? Done.

Caddy automatically enables HTTPS for all of the apps you specify on your Caddyfile. It also automatically issues new certificates and renews them. Changes? A simple caddy reload and you're up and running again.

I can have as many subdomains/domains pointing to the same server as I want, as long as the server can handle it.

Closing thoughts

One of the services I'm currently self hosting is called Glances. With it, all of my server's metrics an url away. With about 5 containers, the CPU is still below 10%.

No. I don't have all the elasticity that the Public Cloud gives me. But do I need it? The deployment method is the same (git push), the speed of services are the same (mostly static with Cloudflare on top), the cost is lower, and data is controlled by me. Sounds like a treat.

A recipe recommendation system

2022-06-26T11:55:00+02:00

You arrive home and turn on Netflix. On the front page: hundreds of shows. They're not ordered randomly. Netflix has been perfecting the science of recommending shows for years now. Google has perfected the science of sorting search results.

To this day, no one knows exactly how Google sorts their search results. Recommendation systems are gold, and a big part of the success of big tech.

Yeah, this article is not about food. It's about recommender systems.

While building one, my assumption that Scandinavian food was not for me, was debunked. Turns out, it's one of my favourites (at least according to what I built). Hard to believe?

On cooking

I'm not a fan of the culinary arts (the opposite really). But I decided to build a dataset of recipes. 160.000 recipes. Each one has a name, a region, a link, and ingredients. I scraped them from RecipeDB. RecipeDB scrapped them from somewhere else - the data is ok-ish quality.

Here's a sample:

And here's there distribution of recipes across regions:

Vectors are dope, more of those

Data is messy. Images, text, audio... They come in thousands of formats. But there's a common language to every data type: vectors (or arrays, whatever).

In machine learning, vectors that represent data are referred to as embeddings. Transforming a word or a sentence in to an embedding it straightforward nowadays:

from flair.embeddings import WordEmbeddings
from flair.data import Sentence

# create embedder Glove in this case

glove_embedding = WordEmbeddings('glove')

# create a sentemce

sentence = Sentence('pineapple pizza')

# embed a sentence

glove_embedding.embed(sentence)

# you can get the embedding for the whole sentence

sentence_embedding = sentence.embedding

# or check the embedding for each word (e.g., token)

for token in sentence:
    print(token)
    print(token.embedding)

# prints...

# Token[0]: "pineapple"

# tensor([-1.3641,  0.2901, ...])

# Token[0]: "pineapple"

# tensor([-1.3641,  0.2901, ...])

Getting the vector for a sentence, is done by averaging the vectors of the words inside of it. We could use also BERT to generate embeddings for sentences.

There's one big advantage of these pertained models like BERT: the embeddings preserve the semantic meaning of the words. So, in theory, two sentences with similar meanings, should have similar vectors.

These fancy embeddings take time to compute. In the interest of time, I went with Glove to calculate them.

A galaxy of recipes

Let's leverage the technique above to embed all the recipes in our dataset. We take the ingredient_string of each recipe, and use Glove to generate a matrix of embeddings:

# create embeddings for all of them

sentence_list = [Sentence(text) for text in df.ingredient_string]
document_embeddings.embed(sentences=sentence_list)
embedding_matrix = numpy.array([s.embedding.numpy() for s in sentence_list])
print(embedding_matrix.shape) # prints (115657, 100)

That's a pretty large matrix. 115657 vectors, with 100 dimensions each one. Now, we can visualise 3 dimensions - but not 100.

To visualise them, we need to reduce the dimensionality of these vectors. UMAP does just that:

# let's use UMAP to help us visualize these vectors

# you could use t-SNE, PCA, etc.

umap_fit = umap.UMAP()
umap_matrix = umap_fit.fit_transform(embedding_matrix)
print(umap_matrix.shape) # (115657, 2)

Alright, now we have a 2D matrix of our 116.000 recipes. That's visualisable. For the sake of your browser, here's a random set of 2000 recipes in 2D space:

Remember, the closer two recipes are, the more similar they should be. It looks like it. Notice how recipes from the same region tend to form groups. This makes sense: recipes of the same region use similar sets of ingredients.

I don't like spinach

While studying recommendation systems. I came across a particularly interesting technique for building recommender systems.

The first step is to create a network of items (recipes in our case). This includes the recipes that you like, and recipes that you don't like. Using pandas, let's define that:

# let's define some polarising ingredients

liked_ingredients = ["cheese", "pizza", "avocado", "olive oil"]
hated_ingredients = ["spinach", "mushroom", "olive", "beef", "truffle", "blue cheese"]

# let's add a "like" property

df["like"] = 0

# and populate our data

df.loc[df.ingredient_string.str.lower().str.contains("|".join(hated_ingredients)), "like"] = -1
df.loc[df.ingredient_string.str.lower().str.contains("|".join(liked_ingredients)), "like"] = +1

Yeah, I don't like spinach. Let me be.

Once I've defined what recipes I like, and what recipes I don't, we can use our leverage our embeddings again, to visualise my own network of recipes. Here's a sample of 500:

Scandinavia, who would've thought

The second step is to score each "unseen" item (e.g., recipes that I don't know if I will like or not). To do so - we need to look at its neighbours. Remember, we're talking about vectors, so neighbours in this context means similar vectors. Let's take the cosine similarity of two vectors as the measure of their similarity.

The process to score an "unseen" recipe is then: 1. Identify its K (e.g., 10) closest neighbours 2. Calculate the average score of the neighbours (e.g., +1 if we like, -1 if we don't) 3. If we want to get fancy, we can weight that average with the distance to the unseen item

If you're the visual kind:

With the above methodology, we can score every single "unseen" recipe. We scored every liked and disliked recipe as a -1 or a +1. Therefore, every unlabelled recipe will have a score between these two numbers. If that score is above 0, I am more likely to like the recipe, if not, I'm more likely to dislike it.

But wait. This sounds computationally expensive.

# count the likes and dislikes

df.like.value_counts(normalize=False)

# 0    54905

# 1    48661

#-1    12091

This means that for each one of the 55.000 unseen recipes, we need to calculate the distance to each one of the 60.752 seen recipes. And only then determine which are the K closest neighbours. Yeah that could take a long time.

Quick maths - literally

Fortunately - Facebook's AI team has tackled this challenge. Not only that, they've shared it as an open-source package. Faiss, is an open -source library that does exactly what we need. It finds the most similar K vectors to the one(s) you're searching for.

We're not talking about 116.000 recipes anymore. That will take less than 2 seconds on my machine:

# start by taking the labeled and unlabelled embeddings

labeled_embeddings = embedding_matrix[labeled_indices, :]
unlabeled_embeddings = embedding_matrix[unlabeled_indices, :]

# create our index

d = labeled_embeddings.shape[1]
index = faiss.IndexFlatIP(d)

# normalize and add our labeled recipes

faiss.normalize_L2(labeled_embeddings)
index.add(labeled_embeddings)

# normalize and query for our unlabelled recipes

faiss.normalize_L2(unlabeled_embeddings)
r_distances, r_indexes = index.search(unlabeled_embeddings, k=6)

On larger datasets (we're talking 1M+ vectors) the exact neighbour search can be very computationally expensive.

With FAISS, we able to speed that up. We can do approximate searches to save time, we can make these calculations on GPUs. It's an awesome tool, with a great wiki.

Icelandic Caramel Potatoes

Yup - back to food.

After some data wrangling, I was able to give a weighted recommendation to every recipe in the network:

distances: the distances that each one of the 10 neighbours has to the recipe in the row
scores: the scores of the 10 closest neighbours to the recipe (+1 if I like, -1 if I dislike)
score_normal: The average score of the 10 neighbours
score_weighted: score_normal, weighted by distance

As you can see above - I don't think I would like that Suya recipe. But both French recipes on the table have perfect 1.0 scores - I should like them. In principle of course. Pear Brandy Souffle looks like a party I wouldn't like to get into - but hey, you only know if you try it.

There's another interesting question we can explore: What's my most recommended region? What is the region that suits my tastes the most? Pandas to the rescue:

Looks like French cuisine has the highest average recipe score. But Scandinavia is a close second! Who would've thought that? Maybe I do belong in Scandinavia.

The return of simple tools

Transparency is one of the biggest advantages of this method. By looking at the closest neighbours column - we are able to know how a certain recommendation was made. With so many ML algorithms and transformers out there - interpretability is not an easy thing to land.

Machine learning has made incredible advancements in the last 3/5 years. We have transformers, deep architectures, massive multi-billion parameter models serving users. Amidst all the advancements it's easy to get lost in adopting always the new shiny thing. Even if we don't understand how that shiny thing works.

There's a certain beauty to simple tools. Simple, explainable, and interpretable algorithms are a joy. And if simple is good - why complicate?

Serving models with FastAPI: It's not just about the speed

2022-06-10T11:00:00+02:00

Disclaimer: This post was originally published in the Amplemarket blog

Serving and deploying Machine Learning models is a topic that can get complicated quite fast. At Amplemarket, my team and I, like to keep things simple.

Let's talk about it.

Simplicity is key

At Amplemarket, we’re big fans of simplifying. We know that the less code we have to write, the fewer bugs we're likely to introduce. FastAPI allows us to take a trained model and create an API for it, in less than 10 lines of code! That’s pretty incredible.

Let’s say you are creating a model that is trained on the Iris data set that predicts the species of a plant given some information about it. Previously, we would have saved our model to a pickle file, and sent it to our engineering team; perhaps we would have even scheduled a meeting with them to discuss how to use this model. We would also have sent some documentation on how to use that model and how it can inference on some data.

But the likelihood that something will go wrong in one of those steps is greater than we’d like. Every time the model gets updated, we would need to have another discussion, about how the model has changed, and what gets improved - documentation would get outdated, etc.

To avoid dealing with that house of cards, we at Amplemarket have settled on a process that significantly reduces the complexity of deploying the machine learning models we develop.

As an example, the following snippet trains a classifier and saves it to the model.joblib file:

Note: To run the snippets, make sure to python -m pip install scikit-learn fastapi "uvicorn[standard]"

from sklearn import svm
from sklearn import datasets
import joblib

# create model

clf = svm.SVC(probability=true)
X, y = datasets.load_iris(return_X_y=True)
clf.fit(X, y)

# save model

joblib.dump(clf, 'model.joblib')

With FastAPI, serving your model is as simple as creating an app.py file with the following contents:

# app.py

from typing import Optional
from fastapi import FastAPI
import joblib

# create FastAPI app and load model

app = FastAPI()
model = joblib.load("model.joblib")

# create an endpoint that receives POST requests

# and returns predictions

@app.post("/predict/")
def predict(features):
    predictions = model.predict([eval(features)]).tolist()
    return predictions

You can then launch the API with uvicorn app:app --reload.

Such a simple setup to launch an API ensures that our Machine Learning team is not limited to creating these models, but also responsible for making them available to our development team, or even directly to our users.

“You built it? You ship it”

Reduce the need for coordination

As a remote, distributed, and asynchronous team, documentation is huge for us. We don’t want to have a meeting with 5 other departments every time we build or ship a new model. We want to document as best we can, with the least effort possible.

With the script described above, if you visit localhost:8000/docs you’ll notice that your API includes some documentation out of the box:

These docs already tell a lot about the application: what endpoints it has, what type of queries the user can send, etc. We love FastAPI because it takes little work to enrich the documentation further and minimize the need for future coordination.

Let’s add a couple of things to our app.py script:

from fastapi import FastAPI
import joblib

# some documentation in markdown

description = """

## Documentation

**ℹ️ Read carefully before using**

This api allows you to predict the type of Iris plant given a list of features.
The features should be:
* sepal length in cm
* sepal width in cm
* petal length in cm
* petal width in cm

_Build by:_
![logo](<https://amplemarket.com/_next/image?url=%2Fsvg%2Flogo.svg&w=384&q=75>)
"""

# create FastAPI app and load model

app = FastAPI(
    title="IRIS Classification",
    description=description,
    version="0.1",
    contact={
        "name": "Amplemarket",
        "url": "<https://amplemarket.com>",
        "email": "support@yourcompany.com",
    },
)

# ... rest of the file ....

If we now turn back to our docs, we see that the markdown we've added has been rendered at the top of the page. Now, when someone needs to check some details about this model, all the information that person might need is neatly described - and we even provide a support email if they run into trouble!

But FastAPI doesn’t stop there.

With a couple more lines of code, and thanks to a small library called Pydantic, we can also add data validation to our model’s API. By doing so, API users will know what kind of data it expects to receive and what kind of data the API will respond back with.

We start by creating two classes, one to handle requests, and the other for responses:

# We'll take this in:

class Features(BaseModel):
    sepal_length: confloat(ge=0.0, le=1.0) # ensures values  are between 0 and 1
    sepal_width: confloat(ge=0.0, le=1.0)
    petal_length: confloat(ge=0.0, le=1.0)
    petal_width: confloat(ge=0.0, le=1.0)

        # with an example
    class Config:
        schema_extra = {
            "example": {
                "sepal_length": 0.2,
                "sepal_width": 0.5,
                "petal_length": 0.8,
                "petal_width": 1.0,
            }
        }

# We'll respond something like this:

class Response(BaseModel):
    setosa_probability: confloat(ge=0.0, le=1.0)
    versicolor_probability: confloat(ge=0.0, le=1.0)
    virginica_probability: confloat(ge=0.0, le=1.0)

        # with an example
    class Config:
        schema_extra = {
            "example": {
                "setosa_probability": 0.7,
                "versicolor_probability": 0.1,
                "virginica_probability": 0.2,
            }
        }

And we tweak our endpoint code:

# the endpoint

@app.post("/predict/", response_model=Response)
def predict(features: Features):
    feature_list = [
        features.sepal_length,
        features.sepal_width,
        features.petal_length,
        features.sepal_width,
    ]
    predictions = model.predict_proba([feature_list])[-1]
    predictions_clean = Response(
        setosa_probability=predictions[0],
        versicolor_probability=predictions[1],
        virginica_probability=predictions[2],
    )
    return predictions_clean

The two classes above are particularly strict about what our API can receive and what it will respond back with. Suppose a developer tries to query the API with a petal_width of 1.3. Because we’ve specified that petal width must be a number between 0.0 and 1.1, our API will reject the query and reply back with:

## request

curl -X 'POST' \\
  '<http://localhost:8000/predict/>' \\
  -H 'accept: application/json' \\
  -H 'Content-Type: application/json' \\
  -d '{
  "sepal_length": 0.2,
  "sepal_width": 0.5,
  "petal_length": 0.8,
  "petal_width": 1.1
}'

## response

{
  "detail": [
    {
      "loc": [
        "body",
        "petal_width"
      ],
      "msg": "ensure this value is less than or equal to 1.0",
      "type": "value_error.number.not_le",
      "ctx": {
        "limit_value": 1
      }
    }
  ]
}

As an added bonus, the model Config classes also help provide developers with an example request and response:

With this documentation page at hand, our users know exactly what our API is, what it expects, and what it will reply back.

All of this comes without us having to invest much time and effort into ensuring the documentation has all the information that future developers might need. And notice how we didn’t have to write any extra documentation.

Our documentation is our code.

Make it fast (enough)

Python is far from the fastest language out there, nor does it claim to be. We don’t use Python for its speed, but for its ecosystem, especially as it relates to data science and its many needs.

Even with several claims that FastAPI is a very performant web framework, we know we’re not using the fastest web framework out there.

However, even if FastAPI was slower than it currently is, we would still be willing to compromise that speed for time-to-market, documentation, and ease of use.

When deploying, Sebastián Ramírez offers a FastAPI/uvicorn high-performance docker image with auto-tuning. Allowing the app to scale according to the number of available CPU cores on the machine it's running on:

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.9
COPY ./requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
COPY ./app /app

If you’re running on Kubernetes or something like that, you probably don’t need this image - but if you have a simpler setup, this image will come very in handy!

Deploy with confidence

At Amplemarket we like to adopt best practices from Software Development and apply them to our Machine Learning projects. That means every model we develop and deploy is version controlled, tested, and continuously deployed.

Using CI/CD automation, we can continuously deploy our model and code to several targets (e.g., staging and production). Allowing us to serve different versions of our model in different endpoints, and roll back with ease.

FastAPI also allows us to easily test our code. This is especially important when we are inferencing. What if we receive a different set of numbers? Will we make a prediction? What if the user sends us a string, and it gets evaluated as a float? How do we account for that? Testing matters. And FastAPI allows us to do it with ease.

Closing thoughts

FastAPI has been particularly valuable when serving our Machine Learning models. Our development team is especially happy with the high level of documentation and data validation that our APIs offer and our users also get those benefits.

Thanks to FastAPI, we’ve been able to predict with confidence, and I hope you got some inspiration from this post to gain confidence as well.

Note: This is also a show of appreciation to the incredible work of FastAPI’s lead developer Sebastián Ramírez.

Mac apps you didn't know you needed

2022-04-12T09:30:00+02:00

I'm a bit of an iSheep. I didn't even know that term existed until a colleague used it on me a couple of years ago. But this is not an Apple vs. Windows (vs. Linux) post. I like my Mac, you might prefer your Windows or Linux machine - that's fine. This post focuses on Mac apps.

I've worked remotely for almost 3 years now - and the importance of little things starts showing. Those things that make my day easier, more pleasant, more productive. From the desk I sit at, to the editor I use, these things matter.

There's a certain magic to software that makes everyday work just a little more enjoyable and productive. Here's a list of apps that help me achieve just that.

MeetingBar

There are few things in life I hate more than an Outlook notification 10 minutes before my next meeting. Especially in a remote setting. Yes, I want to know when the next meeting will take place - but without everything buzzing all the time. MeetinBar does just that. Your next meeting in your menu bar. When it's time - hit your shortcut, hit it, and you're there.

Itsycal

I like Apple's Calendar app (yeah, iSheep - I told you) - but during a workday, often, you just want to check a simple date. "Are you available on the 5th of July?", "What are you doing on week #34?". These are though questions. Itsycal answers all those, right in your menu bar, in a couple of clicks.

Maestral

I still use Dropbox. It's not the best Cloud service, but it was the first I used. There's one thing I can't stand though- their Mac app - always trying to push the next big Dropbox thing on me, eating my RAM, or annoying me with pop ups. I just want file sync. Maestral is a lightweight and open source alternative client that does Dropbox file sync - and nothing more - without annoying me or eating my RAM. 100% open-source - what an app.

Clocker

Who likes dealing with time zones? I certainly don't. 8AM in Copenhagen, it's 7AM in Lisbon, which in Singapore means...? I have no clue. Clocker solves just that. In a few clicks I can select a time, see what it corresponds to in other time zones, and copy all those to my clipboard. Done. No more timezone hell.

Raycast

For years Spotlight search was good enough. Then I started using Alfred, which was pretty awesome. Now I use Raycast, which is even better. I think of it as Spotlight on steroids (I really hate that expression). I can find files super fast, access my clipboard history, search messages on Slack, search my Google Drive, tweet, check my schedule.. You get my point - go give it a shot.

PodcastMenu

Overcast is my favourite podcast app. When working at my desk - I listen to podcasts sometimes, especially if I'm doing something repetitive (e.g., bookkeeping for example). PodcastMenu puts Overcast right in my menu bar. I also syncs perfectly with the mobile version - so that I can keep listening my pods when I leave my place.

TextSniper

Someone sends me a bad scan of a document and there's a number I need to copy from it. There's a weird pdf that I need to copy a reference from. What do I do? I used to look at them one-by-one and copy them over. Now.. I just hit CMD+Shift+2 and this uses OCR for me to copy the text from it. This has saved me at least an hour of my life.

Rectangle

We agree. Windows management in MacOS is.. Lacking to say the least. For a long time, I used Spectacle to manage my windows, and I really liked it. Now that Spectacle is not in active development anymore, the predecessor seems to be Rectangle. A free and open-source option, which has served me very well, and I can definetly recommend.

Stats

OK, you don't exactly need this one. But I like to glance at my menu bar app to see my RAM, SSD, and CPU stats. Particularly if I'm doing something in the heavier side. Stats is a free and open source alternative to the popular iStat Menus - that sticks to the basics. It allows me to customize whatever I want to see in my menu bar and keep an eye on the state of my Mac.

Final thoughts

You'll notice that at least 7 out 9 applications I mentioned are free and open source. Not only that, but the developers are also regularly pushing updates and making these apps better every day. It's incredible how good a product of passion can become. This post is my way of thanking them for their awsome work. Don't forget to also support them.

A simple system to stay in touch with people

2022-03-21T14:30:00+01:00

Someone once told me that my professional network is one of the most important things I'll build throughout my career. For years I've struggled to find a system that would help me stay in touch with my network. So I decided to build my own - let me show you how it works.

The idea

The idea is pretty simple, you want to stay in touch with a group of people, by talking to them every X days/months/years. Let's say you want to keep up with Jane every two months. If two months have passed and you haven't talked, you should get nudged into doing it.

To solve this, I started with a recurring item in my to-do list, but it quickly got messy. I tested Dex, but they would only allow me to keep tabs on 15 people. Monica looks like a fun deployment challenge - that's not what I'm looking for.

What I really wanted was something simple. Something like other small things I've built before? I bet I could build something simple using a bit of Python.

How it works

At the core of the tool is a simple folder. In it, every person that I want to stay in touch with, gets their own markdown file:

├── infrequent.py
└── people
    ├── john-snow.md
    ├── arya-stark.md
    ├── ...
    └── sansa-stark.md

At the top of the file: the person's name, our relationship, and the frequency at which we should keep in touch. After that, every entry is a header with a date of contact, and some notes about what happened in that date. Here's an example:

Name: John Snow
Relationship: Idol
Interval(every): 2 months

## 21-03-2022

- Was recruiting for more troops
- Dead walkers in the wall

## 13-03-2022

- Had to go to the wall
- He was starting to notice something changing in the weahter

Using a small script I call infrequent.py, I parse all the files in the people folder. The script retrieves the person's details and dates of interactions. Using some datetime logic, I then identify the people I'm late reaching out to:

...
interaction_dates.sort()
last_interaction_date = interaction_dates[-1]
next_interaction_date = last_interaction_date + time_delta

message = f"{name} - {relation} - (Cadency: every {contact_frequency}, {delay_in_days} days passed since last contact)"

if next_interaction_date < today:
    to_contact.append(message)
...

Once every person is processed, the same script sends me an email with the people I'm late reaching out to, using SES.

How I use it

I store the people folder and the infrequent.py script in a GitHub repository - similar to this one. Thanks to GitHub actions, the script runs every Friday morning. When it does, I receive an email with the names of people I'm late reaching out to.

Every time I interact with someone in the folder, I update their file, and push the changes back to GitHub. It's simple, and works surprisingly well.

Here's a copy of my repo in case you want to play around with the script. Feel free to email me if you have problems getting it to run.

This post was inspired by this article

Visualizing every job in the world

2022-02-10T09:00:00+01:00

Imagine you have to classify every single job title in the world into 10 categories, how would you go about it?

This is a fairly hard problem to solve. However, the European Union has actually taken it on. They named it: the ESCO project. (ESCO stands for European Skills, Competences, Qualifications and Occupations)

Damn I love the EU.

What if we scraped their database of 3001 occupations and ran them through the famous BERT model? We could look at all the alternative names for each title/occupation, and build a network of jobs.

The result is this projection: a network of every job in the world, that clusters titles by similarity.

You can also run the PCA or t-SNE algorithms through it. These will cluster the jobs in slightly different ways. Go ahead and look for your job title on the right pane, and see the most similar jobs out there (at least according to BERT).