software factory explained by a Senior Engineer (step by step)

An agenic software factory. It's basically a way to integrate agents all the way through your software development lifecycle. So you automate most of the repetitive part. And as an engineer right now, I spend most of my time either building the factory or reviewing part of the code that comes out. But you're seen in a second that I didn't even review most of the code, I only review critical code. And what we do as engineers now, it's instead of writing a component, you define a design system.

Instead of writing an endpoint, you define what kind of APIs will we build, is going to be rest, or graphql, how will we do caching. And based on those directive that you give, you have these little robots, these little agents, pretty much everywhere implementing. So a lot of the boring stuff gets delegated and you have to work at a higher level. So once you will build a software factory, you are more worried about building the production line or acting like a scientist where you go in and you make a certain test to check certain hypothesis on once that hypothesis is checked, you throw it into the engineering line.

But you are not in deep production line, you are not in the assembly line manually building in every line of code. You're actually what you're doing is to build the factory itself. So you go from building the product to building the factory that builds the product. How do we build an agenic software factory? So just to give you a high level intro, most of our stuff runs on AWS and you can do this with a lot of the free tier services.

You don't need to pay for all of it and I did at least with a personal cloud max subscription. And I'm not even using all of it, so you could probably do it with a lower tier. So all this is to say that if you are a developer right now, you could build this like those tools are available, there's nothing, you don't need to work for a frontier lab to build something like this. And you don't need an unlimited token budget, just because the current cloud subscriptions are very generous anyways, so you get millions of tokens.

So you can build this yourself and I'm going to give you a very high level. Overview and then we can dive into all the specific components. But at the high level, in our case, our architecture is basically a monolithic front end and some backend microservices. They are all on GitLab. And what happens right now is we have a triaz agent. ingesting issues that come either through Sentry or they are user reported.

So all those are hooks and whenever a user reports a bug or we see an error popping up on Sentry, it will call to send all the context to an agent. And then the agent will try agencies is this a real error? Is there something already in the backlog? Was it already fixed? So that's the first step, we automate this kind of bug collection. Then we have a refinement agent that will try to estimate scope, blast radius, how many services are impacted if we make the change, what's the priority? Is it a live production bug that we need to fix ASAP and the size.

So all this is the product work that we used to do. And the beautiful thing is this used to take days in a team. Like you'd wake up, you're like, okay, what's let me reproduce the issue and is this a real issue and it was fairly mechanical. Like there's very few bugs that really need you to be present. So I really love this because it took away part of the work that me as an engineer, I didn't even like.

Again, all those are agents, so they try to estimate, they try to guess. Don't trust this as, oh, it's going to give the estimate as good as mine. But it will be good enough and for most changes, most small bugs, it's good enough. You don't need to spend more time on it, it's not worth it. You just said that this refinement, it's going to do the work you don't like, which is basically kind of trying to understand the ticket. Okay.

Okay. Yes, so basically the refinement agent. We'll try to scope it, figure out the blast radius, reproduce it, and then it ends up creating a ticket here in our board. And as you see here, all those tickets have been created by agent. So I usually give them a very off screenshot and we're like, okay, can you check this? Is this a bug? Somebody reported this. I have a button in our application where if I get a user message, I just click report a bug and it gets tracked into this pipeline and they do all the hard work.

Uh they use Cypress and Playwright and they can visualize the whole application, they have marked production data so they can try and reproduce the exact same bug. And they usually take a video with Playwright, so I can even see they reproduced it and they attach that to the ticket. And this is how our ticket looks like. Now, a big problem with AI is that tickets are very pose right now and that's something I would work on.

Later, but as you see, it's trying to go over to steps to reproduce. it's code, acceptance criteria and repo evidence, and usually it has screenshots and videos trying to figure out okay where the issue is. So, rather than me reviewing even code, I usually give you high level bugs and figure out, do I really need to get into the code. So that's what the refinement agent would be. That's the final result of the refinement agent, it's the actual scope down ticket.

Now once we have that queue, we basically have also a prioritizer that will try to figure out which tickets should be implemented, which are not ready to be implemented, which need human input. Because a lot of those things are like, is it worth doing? Does it change the architecture? Is it really a bug? If it's a very clear bug and it's easy to fix, for example with a small front end tweak usually, then it goes straight to an instance of what we call the implementer agent.

Now, the implementer agent, it's actually the developer and is the one writing the code. So it has access to a cloud MD where we define, okay, whenever you build something, take a look at the take a look at our design system, or whenever you write an endpoint, use this. Or check this out and I can define there how we do caching and the database and pretty much all the stack, so it doesn't have to think too much.

Then we have certain skills, I usually prefer skills from the stack we use. Things like Vercel skills for React, AWS skills from AWS. So I want to get as close to the source because I know that the provider usually has the best knowledge on how that technology works, and they package that as a skill. It has access to all our MCP servers, so in in our case, it's entry post hog, so it can run its own queries.

Linear, of course, in GitHub, and there's also defense sub agents. So the beautiful thing is you can work in parallel on on so many tickets. Um, and so we will go back to this implementer agent, but basically once the code is written, it goes to our building quality pipeline. And this is probably one of the most important pieces where I'm actually still working right now, and we will talk about antipatterns also, agent antipatterns and so how agents get stuck and make mistakes, and how you can actually control that when you build a factory.

But your building quality pipeline is not writing the code. So a lot of what I do these days is kind of looking at our end result. I go into the source code and I go to components, I'm like, okay, what do I need to implement in the building quality pipeline or the context to make this better? So I'm not fixing the component, I'm fixing again the factory. That's a big mindset shift you need to take. And you always think in second or third order consequence. It's not about if we change this and we make this a custom hook, how would the component change? But rather, if the agent will start writing custom hooks all over the codebase, what will be the end consequences? Is that going to increase the bug density? Is that going to increase our testing efforts? So that's how you think. As opposed to back in the days where I had to do all the ground work, so I'll spend a lot of time just building a specific end point or component or a view. And then when I finish that, then I put on my architect hat and then I look at the overall kind of system. Bug number when I look at all this so far, I mean correct me if I'm wrong, but because of all the work was manual in the past, right? You thought about something as an engineer and you had to actually spend a lot of time, many a few hours to to implement it. Whenever you have all these agents that you can direct, which means that this is kind of an engineer's wet dream, right? There's nothing to be afraid of these agents and AI taking, you know, eliminating jobs for engineers when actually they finally allow us to make this vision. That was in all these clean code books, you know, all the theory of proper software development can now actually be implemented because one of the main reason why it wasn't implemented is because it it was way too much time, right? So this is actually something that that engineers should be excited about, right? %? So I can tell you it's the first time, for example, where our code base it's covered by integration test. It was just so hard to get it done and maintain it and we have 950 in our full independent test suits, testing everything every night, okay? You need a dedicated testing team to write test every day. And so a lot of the things that were hard to do are now democratized and a lot of people are saying.

think more than ever, but I code less than ever. Yeah, so when it comes to this building quality pipeline, going me if I'm going, but this is probably the most important piece of the system. Can you walk us to it and how you build one? So when building the build and quality pipeline and adding all those quality guard rails to make sure that code actually does what it's supposed to do, that there's no dead code. You need to be a lot more aggressive than back in the days. Just because not only AI makes you mistakes, but also you are shipping a lot more volume, a lot more code.

And so the main idea here is you want deterministic measures on your code, because AI already introduces a lot of probabalistics, and that's too much risk. We cannot just let it hallucinate. So we need to go back to traditional software engineering knowledge. And when we talk about deterministic measures is the first thing we use is basically having a code linter with your own set defined, you know, just use Slint defaults are pretty good.

And then we have TypeScript, and that's how we use that's used to be enough in back in the days. But now, we also have better tools like Fallow for static analysis. And if you don't know what it is, check out the project, 100% recommended. But Fallow it's a tool where I can run it and it can tell me if there's any code duplication. Try to statistically identify duplicated functionality in the code. And I know there's a lot of people out there saying, okay, if code is duplicated.

But it turns out that if you have duplicated code, you're going to spend a lot more tokens because you need to write units on that duplicated code, you're going to have to ingest it next time you extend your codebase. So, the guiders of AI, duplicated code and duplicated features are not a good thing. Then it will also check out the cyclomatic complexity. Basically that's a way to measure functions and how complex they are.

So we don't have this monolithic components that are 1000 lines long. And the cool thing is that you can run those static measures and they feed their input back into the agent. Then you have the agent correcting its own code using those static measure. And we also look at dead code. Dead code it's it happens quite often with AI that you remove a certain feature, but there's like the backend endpoint that it didn't delete.

Maybe there is a database table that you don't need anymore. and the faster you go, the more it accumulates. It's very important that you clean up and you remove as much code as you add. And we're going to enter and test. So of course, a lot of people are talking about unit testing right like do TDD with the AI. I believe TDD it's a waste of tokens for the most part, especially when you integrate very fast.

Because components and code change so fast, and the fact that there's a unit test on a new function that the model build doesn't give me any confidence. But end to end tests written against our features that run in our real environment, they make me sleep well at night. We do that and then we also have a visual difference in all the PIs. So our agents are instructed to take screenshots before the feature then they check out on the fix branch, and then they take screenshots and videos of the new feature.

And you need that because the reality is you're not going to read T pages of AI generated text. And you are also not going to read the code unless it's something important because you just don't have the time. So visual checking it's one of the best ways I found to really get confidence. And the other thing is whenever you run those tools and you run test, agents will fail and they try to fix their own stuff and then they'll fail again.

And that's what we call turns, how many turns did an agent took to really fix this bug or feature. And at the beginning, you think they do that three times or four times. But there are edge cases where no matter what model you use, you can use fable, you can use Oppus 5.5, they just go, they drive themselves into loops, maybe because they miss some context or they keep contradicting themselves. And I had agents trying to solve a certain bug a hundred times.

And that's my hard limit right now in the pipeline, which is 100 turns. If it takes more than 100 turns, you stop and you go back to human review. Because you're not it's going to burn through your whole subscription and token limit. This is where the real engineering comes in, because whenever you have one of those issues, I could spend a day just thinking about what happened and how do I change the factory.

Or the code architecture, how do we modularize our code? Do we go more into event-driven microservices? Where is the context missing to really understand why couldn't the model fix the issue. So models they have a lot of knowledge, but you are definitely smarter as an engineer, okay? people say models actually still make very dummy mistakes. And when they make mistakes, they go all the way to the bottom.

So, the picks are very high with AI, but the bottom it's all way low. And that's why UI there. And you can basically scale your judgment by embedding it into agents and skills once you figure out what the problem was. The result of this will be a lot of pull requests. And a lot of people are talking about co-review or not. And I'll be honest with you, I only give you based on the blast radius and the services affected.

So, you know, this in the very beginning, we talked about scope and blast radius, which was done by the refinement agent. And if the blast radius is very small and it only affects one service, then I would just have AI merging it. What do we mean by blast radius? Well, you can imagine it almost as a layer. If it touches the database, that's a huge blast radius. If it's just a tweak on a front component that's not even in the design system, that's a very small blast radius.

If it goes to design system because it affects the whole thing, then it's a bigger blast radius. And right now, just to demo a bit, my mornings look like this. So, I wake up to a lot of pull requests created by agents. And depending on the ticket that I look at, then I'll decide to either merge it or not. I do not do a lot of auto merge. But I think as we get more and more confidence and more and more releases are successful, if we get to a 99% ratio where I just came through the PR and looks good, then we're probably going to move also into the auto merge where again, if the blast radius is small, you just merge, take it and deploy it to prod.

And by the way, this was just yesterday. Right? A lot of those we usually average with the software factory, 10 to 20 PRs per developer in a day. Perfect. So those are the pull requests and this is where we do have an AI reviewer. Also, that's a separate agent that doesn't have the context of the others, and it's trying to check if there was any architecture regression. Like sometimes you we build certain microservices and they just, you know, the AI is just going to solve the issue.

So sometimes they would high code things into tests. They will violate module boundries. And these are the so then the actual engineering goes in. And then we also check if there's any under the system state. Again, AI loves to add things. They would rewrite the whole thing. They would make up a function even if it's already declared. They would forget that we have a module where this functionality is composed, so you need to take that logic from a component and move it there.

And the same happens in the back end with the database, like it takes so many shortcuts. I've never seen someone taking so many shortcuts than AI. Like you'd have JSON stringify to store objects in database. Like that would be okay. And like no, no, no, we're going to we're going to structure that. This is not the MovoDB database. This is a relation database. You cannot do this kind of things. You're violating your functioning integrity when you do that.

So that's where the engineering goes in and it's quite exciting because you spend a lot of time thinking about how we store data. Clean code and let's be honest, back in the days, this would be something that would you ask your interviews and you touch upon one day of work in five years. Most of the time you're just grinding out features and writing small bugs and talking to product. And depending on the blast radius as I mentioned, we are moving towards low blast radius, it goes into auto merging, it goes to staging and on staging we run a subset of our end to end test.

As I mentioned, we have 950 end to end test suits that's too expensive to run in terms of GitHub actions. I think per run, it would take around $1.50 the last time I checked. So that's just not sustainable, especially when you release this this much, not for a small company. And not for a big company either, like for small teams. So what we do is we made a subset of those tests that are critical, we call that the smoke testing.

But it's actually a bit more than smoke testing and it checks things like login and core functionality doesn't break. We run that every time and if that's okay, then. There's a QA agent also kind of this is something I introduced later this week. Even if everything passed, we still want an agent that acts like a user. So this agent only has the context of the user and the feature that that needs to be tested.

And if everything goes well, it will close the ticket and lean or directly and we merge to production. I use AWS amplifier for our front end, which is really cool because you can create environments based on your front end. On the back end, it's a bit more complex. Because everything I mentioned allows you to give the ability to the agent to reproduce the environment. They need a copy of the database that they can spin up.

They need the backend. They need to be able to check out on a specific branch on the frontend, pull all the code. And if you have several services, this can get very complicated. So a lot of what you'll have to do if you want to build your own software factory, it's doc taping, right? And making sure that environments are reproducible, deciding which data you want to give to the agent. Sometimes we anonymize data, I'm like, hey, take this five users that are heavy users, hide their names and emails, and hide everything they've done, and just build a semantic alternative of it.

So add logsum on their community post, but with logsum, fake text, create a post that's has the same length. So we can reproduce exactly the way they use the tool. So that's a lot of what you will be doing. And what we are also working on, and this is where the buzz link is right now, it's formal verification. And this is a term that has been very popular on X lately. But basically it's a way for you to write higher level hardcore rules in your software.

Like a user shouldn't be able to make two payments in within 5 seconds. Or a user shouldn't be able to submit two code challenges within 2 minutes. And you encode this in your software, and that's like the knowledge that's actually static. And whenever agents build lower level code, like they write or JavaScript or TypeScript, they run those rules. So they make sure that the software itself, the manifestation of the software can be formally verified with high level abstractions.

And I think a lot of software engineering will move into this direction. And there's languages like bend that try to do that. It's all very early. But I feel whenever you have a bigger architecture, where you need to maintain certain certain logic, it's unrealistic that you'll do that by querying agents. You will need a layer of static architecture, static rules that cannot be broken. And bend it's one of those that's coming up.

And I think in 2027 we'll see a lot more about it. We also use Jeff as a model for classification because it has static typing. It gives you static type safe outputs from classification tasks. So a lot of it is moving towards how do we get these very big systems, very probabilistics into type safe fty and predictability, which is what you need for enterprise software. And finally, you know, um that's where engineering reviews goals in big APIs, big features. And we also have a observability layer that I'm adding. So basically what I'm trying to figure out is whenever you see that there's a lot of bugs around the feature, it's usually a sign that the architecture has changed. There's probably a problem. That's what we're looking at it. You can use an agent for that, you go through all the linear tickets that we close in the last month. It's usually a lot and ask yourself, okay, what piece of infrastructure did they touched? Why is this happening? Is this secure and can we do something about it? So, you are moving the two things, okay? You're building a factory, but you're also looking at the code base and designing a code base that it's what they call now agent friendly. Design a code base that's modular, design a code base that's easy to extend, design clear boundaries so you limit failure and do everything that we learn in good old software. Microservices, micro front ends, going to event driven and and so on. Now, as future work, I know to a couple of things, having a model router. So, we can choose between a chip model versus a strong model right now, I'm using all 5.5 for everything. And I think this will work until IPO, you know, until the company goes to the IPO and they finally start charging us the real price of inference. Having docs and a changelog agent. Right now, agents put too many comments in the code, they have this crazy habit of just adding a comment which you know, with the issue name in the code, and so the code base looks terrific. And we're going to clean this up, we're going to remove all the comments from the code. We do want to have feature flags, you know, we just have a per user flag. And then what I mentioned before, right, we go from incident doing a pausemortem and understanding if there's any architecture change we need to implement, not just fixing the bug. Those are things that agent don't do really well, they over optimize for a specific constraint. And so as you see a lot of my work, after you implement this, it's you delegate the low level bugs and writing and typing out the code, but a lot of thinking comes to you. And you have the chance to go into into the database, look at the schema, be like, okay, how do we apply data normalization here? Which are things that you wouldn't even have the time. You literally have to

work the weekend back in the days to be able to do some clean code or refactoring because products will always be pushing features and you always have to fight, but now you don't have to do that because they get the feature and you get to work on the staff you actually like. So there's a very positive side of building a software factory, and I think there's a positive side of working with AI. But if you look at how it was marketing and pushed onto us developers, I understand why people are frustrated and maybe even angry with certain companies because it was all about replacing engineers. And now they realize, oh, we made a very bad marketing, nobody wants to use our tools.

So now they change your marketing and maybe they succeed, but leaving that aside, I think there's a lot of wonderful things about the technology, as long as you understand that you're doing the engineering work and you value your judgement and you invest in your coding skills and you become a better developer. That's what I wanted to ask you. I mean, this is not just a new CSD pipeline. This is a fundamental change in how we build software, but even more than that, it's a fundamental change in the software engineering profession as a whole.

And the things you do on a daily basis, they change. And that's what I wanted to ask you like, if you look at even yourself, three, four, let's say five years ago, in terms of the mindset and the technical skills you were focused on, like the stuff you were doing on a daily basis, and you look at now after this agentic software factory, which is a pattern that I'm sure, a pattern and an architecture that comes with it, will take over software engineering.

More and more companies are implementing it, and it makes sense, it's economically valuable, right? So if you look at those two versions, if you look at yourself three, four years ago and now, what are the major differences that you see in terms of skills and mindset of software engineers after the agentic software factory takes over? So the number one mindset shift when you are going from traditional software development to working and building your agentic software factories, I feel like most of the times I have to roles. One of them is a scientist where I go in and I look at algorithms and I try to test hypotheses very abstract, very high level.

I'm like, okay, if we use a hash map for this state of this bigger feature or we move to arrays or we change the state definition, how that change how many state mutations do I need to make this UI happen. And that's more like scientific. If you think about it, you'll be like, how is this has to do with product? It's very far away. But then once the hypothesis is proved, then I go into my productizing engineering kind of mindset. Where is like, okay, there's this cool material or whatever we found out, how do we put this in the product? And that you go to the factory, right? And that's how I think the additional engineering works these days. Where they built a lot of engineering, it's around the manufacturing and building the factory and they design a product that it's easy to manufacture. The first thing you learn in engineering school is, yeah, you could build anything, but what's the effect of that? How expensive it'll be, what materials you're going to use? And how will you design even the specific treatment you need to do to all the materials? Is this even feasible? So I feel like, in a way, we got closer to traditional engineering, we need to be even more sharp when it comes to software fundamentals. But I definitely spend less time on the implementation layer. Back in the days, I think 80% of my time, if you're working in a front end team would be talking about React and what kind of class components or functional components you write, or do you use arrow functions, or this low-level stuff, which is, to be honest, very annoying. Nowadays, that's usually just a discussion at the beginning of the project. And then if things go very south, then you go in again, you're like, wait, let's look at this component. What is the problem? How do we change the factory to improve the quality? But in a way, it got more fun. And a lot of people are talking about, you need to let go of the coding and stop coding, but I think the people that will build great genetic software factories are the exact same people that were very passionate about coding back in the days. A lot of my thinking, a lot of my fundamentals, I use them every day, I use them more often, and a lot of the job also got more fun, even of some things got let go. I mean, if you look at the whole thing, you'll see that a lot of automation is done into the insting classification and prioritizing, which used to be this endless meetings that you would be forced into as an engineer. So, yes, there's some things I love that I didn't do, but there's also a bunch of things I hated that I didn't have to do anymore. But then, would it be a lot for me to say that the age of software craftsmanship, it's over. And uh, we are entering the age

so for proper engineering, more like mass production. There's definitely an aspect of the everyday craftsmanship that is gone, but you still need to be at the core very passionate and very crafty. And so, building a state machine by yourself, I think it's a good exercises. You'll always do these kind of things. But it's not something you'll do to deliver. You just do that to improve your judgment, which in the end improves the whole software.

So, you scale yourself a lot more. Well, I think that craft person, when you build a product and you design a product, we are not churning out the same car over and over. So, this idea of there's going to be mass production. It's not this is still a design exercise, you're building a product that's constantly changing. Not the same thing moving on and on. I think in terms of if you look at the skill set, you definitely need the fundamentals.

But now you also need to think about you need to know cloud, you need to know a bit of architecture. You need to know algorithms, you need to know a little bit of system design, but a lot about API design, for example, or end-to-end testing. And so, you need to become more end-to-end. The age of hyper specialization, it's over. Now, what I've seen after running this in production is that whenever things go very bad, you do need that specialized knowledge.

So, if you look at the UI layer, for example, you could patch it and keep going. But if you want something much better, you need to go into engineering mode and be like a front end engineering going in and designing a design system. And in the backend, it's pretty much the same. If you want a much better system, you need to go from a mono to micro services. So, whenever you start solving hard problems, you still need to specialize.

And my feeling is that people are talking about vibe coding and they're just vibe coding everything because they don't really care about the quality, are able the user and they're putting a lot of the weight on the user, like they release buggy stuff and then the user reports that and they feel it into the agent. But I think what will happen in the future is users will be a lot more hardcore. They will want things that are high quality and if you want quality and predictability, you need engineering.

Yeah, because in the end, it's users who pay for the cost of you not building quality software and every time that happens, a tiny bit of the user trust is going away until you lose the customers, right, which a lot of companies are going through right now. And I I double user is like going to be a lot more prone to be a lot more skeptic than before. Now, if I look at this, this is kind of the perfect merge between software engineering fundamentals, both high-level and low-level, and what AI and agents can do.

It's kind of the the perfect symbiosis, right? And it's beautiful, if you look at it, right? If you give this to a developer from 5 years ago, they would be more productive. Assuming that they are building the right thing, their company would would get ahead. So it's an exciting future. Now, I would love to to go into each of those components, cuz each of those components has again, the automation and AI part, and then the engineering fundamentals part and how do you build solid in order for these components to be solid in the whole agentic software factory.

But we don't have time for this in this video. So folks, let us know in the comments if you want Bugton and I to do a second part of this video where we go more in depth into each of those components. Also, if you want to become the kind of senior AI native engineer who is able to build agentic software factories, which is what companies are looking for right now, then check the link in comments there is a free training on how to do that.

And if you want Bugnite to personally help you get there, then check out the mentorship we offer. Or if you are just looking to get hired, there's also a free assessment for you folks to find your tinkle gaps. And thank you so much, Bugden. We will see you folks in the next one.