Engineering video
1:15 · 21 October 2026
Running all 950 integration tests is too expensive, so a critical subset (smoke testing) checks core functionality like login. Following this, a QA agent acting with user context tests new features and handles ticket closure and production merging upon success.
Hear it at 0:00 →Building a software factory involves duct-taping environments and preparing reproducible data for agents, such as anonymizing heavy users by hiding names and emails while maintaining semantic alternatives like lorem ipsum text of the same length.
Hear it at 0:46 →In on staging, we run a subset of our integration tests. As I mentioned, we have 950 integration tests that's too expensive to run in terms of GitHub actions. I think program it will take around $1.50 the last time I checked. So that's just not sustainable, especially when you release this much, not for a small company, and not for a big company either, like for small teams. So what we do is we made a subset of those tests that are critical.
We call that the smoke testing, but it's actually a bit more than smoke testing. And it checks things like login and core functionality doesn't break. We run that every time and if that's okay, then there's a QA agent. Even if everything passed, we still want an agent that acts like a user. So this agent only has the context of the user and the feature that needs to be tested, and if everything goes well, it will close the ticket and lean directly and we merge to production.
So a lot of what you'll have to do if you want to build your own software factory, it's doc taping. Like they're making sure that environments are reproducible, deciding which data you want to give to the agent. Sometimes we anonymize data I'm like, hey, take these five users that are heavy users, hide their names and emails and hide everything they've done, and just build a semantic alternative of it.
So add lower ipsum on their community posts, but with lower ipsum fake text, create a post that's has the same length, so we can reproduce exactly the way they use the tool.
