|
 |
The Pulse: We need to talk about migrations with AI
Asana migrated off testing framework Enzyme in two weeks: without AI, this work would surely have been kicked down the road. Airbnb and Uber share similar stories and AI seems excellent at migrations.
|
| ||||||||||||
Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of four topics from last week’s The Pulse issue. Full subscribers received the article below seven days ago. If you’ve been forwarded this email, you can subscribe here.
OpenAI put an impressive-sounding case study about how they helped Asana save $5.9M with a single migration. From OpenAI (emphasis mine.)
“Asana cleared 5 years of engineering work in 2 weeks with Codex. Using OpenAI Codex, Asana replaced an outdated testing system in two weeks for about $12K.
For this project, Codex helped Asana’s engineers remove Enzyme, an outdated testing system that had made the company’s code harder to upgrade. Model and infrastructure costs came to about $12,000, compared with Asana’s roughly $6 million estimate for the previous staffing plan.
After 1.5 weeks of engineering effort spread across two calendar weeks, Enzyme was fully removed. Model and infrastructure costs totaled about $12K. For comparison: the previous plan was expected to take at least five years and estimated to cost roughly $6M. The experience changed which long-running software projects the company believes are practical to take on.”
For context, Asana migrated from Enzyme to React Testing Library, which indeed would not be a simple migration. However, their estimate of four engineers (each on circa $300K/year, according to OpenAI’s arithmetic), spending five years on the project, had me like:
 |
But thinking about this for longer raised the question: what does a migration from Enzyme to React Testing Library even look like?
Enzyme to React Testing Library migration
Let’s take a simple test and see how it looks in various testing libraries.
For our test, we want to verify that a button increments a counter. Here’s our button in React:
 |
Now, the test to verify this, in Enzyme:
 |
And let’s rewrite this test in React Testing Library:
 |
The two tests do the same, but they have completely different syntax! Let’s see just how different they are, with a side-by-side comparison:
 |
The main reason for this difference is that the two frameworks use a fundamentally different approach to testing. Whereas Enzyme is oriented towards component testing (notice how the test operates on a component instance), the React Testing Library operates on the rendered Document Object Model (a data structure representing the HTML shipped to the client) so the test sees the whole rendered page, not just the component its written for. That’s why the testing approach will differ radically between the two frameworks, especially when testing complex user journeys. You can learn more here on the tradeoffs between the two approaches from the React Testing Library author.