The Pulse: tech companies move to open AI models

Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of five topics from last week’s The Pulse issue. Full subscribers received the article below seven days ago. If you’ve been forwarded this email, you can subscribe here.

Before we start, as a reminder, I’ll be in New York, next week. Should you be in the city, you can meet me at:

• At the LDX3 conference: at the keynote and afterward, at the Antithesis booth on Tue, 15 Sep & Wed, 16 Sep

• tubopuffer tech talks on Wed, 16 Sep, 6pm

• An evening with The Pragmatic Engineer on Thu, 17 Sep, 5:30pm

In May, I covered an emerging trend of companies wanting to cut back their AI spending, starting with engineering departments. Different approaches were being tried:

• Experimentation with running cheaper, open models on inference providers

• More investment in model routing to route simpler requests to cheaper models

• Knowledge-sharing sessions on how to use AI models cost-effectively

• Setting per-developer monthly AI usage limits

A few months later, it seems that several companies have managed to achieve this, according to sources I’ve spoken with.

Uber: AI costs down 50%

Uber managed to blow through its annual AI budget in the first three months of this year, and it wasn’t a surprise to hear, in May, Uber’s COO say that it was getting harder to justify spending on tools like Claude Code without seeing benefits from the leading models. It wasn’t long until engineering teams at the ridesharing giant set to work on how to optimize AI spend, and their efforts weren’t in vain.

Uber cut the cost per AI request by 34%, and the cost per AI session by 52%:

Of course, Uber keeps using more AI tokens and starting more AI sessions, but thanks to optimizations the cost has been flat since March, despite significantly more usage:

How did they pull it off at Uber? It was via a combination of different approaches:

• Open weight models, run using inference: run open weight models on inference services, which are significantly cheaper than frontier ones.

• Optimized model selection: benchmark all available frontier and open models, to build an accurate picture of their present capabilities

• Ongoing benchmarking: run benchmarks every week based on real work, and update them

• Cheaper subagent models: subagents do smaller tasks not requiring the most expensive models

• Reduce model effort: Uber found that defaulting to Medium effort gives the best cost-to-output ratio with advanced models

• Optimize requests: trigger automatic compaction above 400K tokens, even for models with 1M context windows

• Cache prompts: cache prompts to save money when using Uber’s own harness, Minions

• … and more: Uber wrote an engineering blog post detailing the dozens of optimizations taken to reduce token cost without noticeable change in the quality of code generated by agents

From the outside, the single biggest win seems to be Uber’s transition to using open models for certain tasks. Open models cost 2-20x less, compared to frontier ones: