7 charts that will change your mind about AI coding

So, number one is the exaggerated accuracy of current models. One of the most accurate benchmarks is the SVD verified, which is a set of 3,000 problems that are standardized. They have unit tests and they tell us how many problems whereas the model able to realistically solve. If we dig deeper into these numbers, you will find out that although they do claim an 18.83% accuracy for GPT 4.0, researchers have found that if you take out suspicious fixes, those numbers drop really lower.

And so, in the case of GPTO, we see a drop from 18% all the way down to 3%. Now, we didn't found data for the latest model, but applying the same method, we assume you'll get around a 4% accuracy in the case of GPT 4.0, which means those models can roughly solve four in a 100 issues. And the next thing people usually say is that, hey, the models are not great or perfect, but they will get much better with size.

And there's this idea that quality of output, it's in some sense proportional to the size of the model in terms of parameters. So, you know, these models are made out of these little weights and they put billions of them, you know, each one has a weight and then you kind of get an input in, get an input out. Now, the thing is from what we see looking at that size of the model is that the accuracy, it's not directly proportional to the model size.

And what you see here is the model performance against one of the scales where it kind of measures general knowledge in English. And we see here that Pan, which is a very small model, has around 72 billion parameters, has roughly the same performance as deepseek 3, where deepseek 3 has 10 times more parameters. So, we increase the number of parameters 10 times, which is a lot of computing, a lot of memory, and we got a marginal increase in model performance.

So, bigger models with trillions of parameters do not guarantee that the currency will be better or that we will fix the design issues that current models have. Grab number three is that the answer quality does not until increase with the consecutive prompts. So, what OpenAI pioneered, it's an architecture model where the AI kind of prompts itself. So, it kind of verifies the answer with itself. And the graph I have here comes straight from OpenAI where they just publish deep research.

And deep research will basically prompt itself until it comes out with a full research paper for you. And that can take around 1 hour or even more in some cases. So, you'll have a lot of calls to itself. We're talking about 60 right, 60 to 20 calls, and they usually are quite intensive. So, it will take around 1 hour to get a final lens. Now what they've noticed is that after a certain number of monologues, right, after a certain number of prompts, the efficiency of the model just platues.

And so, a change in design or any of those fancy terms that you hear being thrown around like time to compute or a chain of thought, they might marginally improve the performance of the model, but they will not radically change the output. The one thing they also mention is that the more the model has a monologue with itself trying to figure out from all the different possibilities it came up with which one is the best, the more you increase the hallucination rates.

Not to mention that this comes with a huge cost in compute. Bringing this back to practice, if you worked with Cursor, the code editor or with Copilot, what you'll see, it's a plateau in the result of your prompts. So, the usual developer experience is that we get in and you get a lot done, but then all of a sudden, you start making prompts and prompts, and you're going around and you end up fixing that issue yourself.

And so, the answer quality plateau at some level. And then that's pretty much shit. At that point, you usually either quit and fix the issue yourself or move on. The number four is the real context distribution. A lot of people say, well, wait until they pull out the next model and you can fit to it a lot, a lot more. You can fit to it, you know, your whole database code base and your backlog is software field, a lot of it is not written. I am yet to know a software company where you go to the backlog, you look at a J ticket and you know exactly what to do. Usually, you kind of have a rough idea of what to do and you go in and there's a lot of back and forth and you talk to POs, and you talk to the designer, and you make a hundred decisions yourself and then you ship something. That's the average developer experience. That's why people hire senior developers, because they can deal with extremely incomplete specifications, they can infer from their past experience and they can make smart decisions fast without writing anything down. And so when they say we will add more context to it, the reality is that software products knowledge is very vast. And then some of that, a very small subset of that gets encoded in our specification, in our gear boards and then from that we extract and write a bit of it in code. And that's what the model is dealing with. The difference between the real world knowledge that a human has and a software engineer has and what's embedded in text in the organization, it's an order of magnitude. And so no matter how big the context window, the model are inherently limited by the exposure they have because they don't live in the real world. And picking up on that one, Bogdan, I have never been in a software company that had even proper documentation about, you know, the software itself. Which is, let's say, an achievable target. To be honest, most companies don't even have a system diagram, I'm not even talking about like function call documentation or anything. They don't have like well written product specification and that's very high level. That's not implementation. Now, number five, let's talk about how AI coding assistants changed the way developers actually code. And so from the official sources, from basically GitHub and GitHub Copilot, we know that on average, developers accept around 24%. of the suggestions that Copilot gives them. So, what they are actually claiming is that one out in four things that, you know, Copilot suggests to us, we actually can accept that. Which is quite a lot, right? It's like, basically, as you were saying that 25% of the work is done by Copilot, which is review that and move it to production ideally. Now to go even deeper, we found another study from the MIT where they claimed that the AI

code suggestions also increase software developer productivity by 26%. And this estimates is based on observing probably over a years the output of almost 5,000 developers at three different companies as part of their regular job, which strongly supports its external validity. Okay, so like this 5,000 devs and they somehow measure productivity and they come up with this. Now, the reality is that if you use cursor or co-pilot, not all suggestions are born equal.

And we actually found a study where they actually broke down suggestions by the different categories. Of course, there's a lot of the suggestions that co-pilot and cursor give us that you could get with a traditional IDE, like inline completion of a class or a function method or an import. Those ones were pretty okay already. But one that's very interesting is C8, which is what they call the function body completion.

That means you can write a signature to a function and then co-pilot or Tab9 or code GX would give you the body of the function. And what we see is that the acceptance ratio for that one, it's somewhere under 10% for co-pilot. So, as we see, the number it's a lot lower than the initial 25% estimate. And going back to this first study, this 26% was estimated based on a success criteria. And the success criteria for the suggestions that they use in the study, it's surprisingly to build success rate.

That means, basically, if you would use co-pilot, write some suggestions, and then you would build your application and it still builds, then that's valid code. Yeah, I'm working, and I think it's very interesting because if you go back to that graph, developers on an average are roughly accepting, I don't know, 26, 24%. There is 75% of suggestions that the developer has to assess and say no to. And that will lead to some burnout for sure because you have to think about this stuff.

And the other thing is, you know, Intellisense and VS Code, I mean, we had auto completes. And I think it's very dangerous to look at these numbers because we were talking about this, the Salesforce CEO, how their team increase 30% productivity. And a lot of developers are scared and I think the reason is because we tend to take this to the face value. We don't think about, wait a minute, what do you mean, 30%?

But when you get into numbers, you reality is quite different. The one thing nobody talks about is that most of suggestions you get and accept, you usually need to edit. The one reason I do think using AI coding tools is a good idea is because sometimes they just write a function ahead and it's just sometimes easier to edit something than to write it from scratch. The bottom line is we see these numbers thrown around, which is a 24%.

But, you know, are these suggestions referring to markdown files or one line completions, or are they really AI writing and making decision and writing code? And what we see is that the real estimates are a lot, lot lower in most cases. Number six, the developer intervention versus the number of prompts. If you built an application with AI from scratch, you'd probably kind of notice what we see here, which is in the beginning you just get so much down with very little prompts.

You can literally go to any of those new AI tools like Sky Surtride with composer and Lovable and there's a couple more that will scaffold the whole application. And they'll pull in tailwind and they'll implement features that you as a developer might think, wow, this would take me a day to implement. But then what usually happens is that there it's a bug and something just doesn't work. And what happens at that point is that you need to go in and use all your debugging skills to fix it.

And then you fix that and you move on to the next feature and with a few prompts, you get a lot done and then again, boom, another stop in the road. And this is kind of the intermittent fasting, let's put it that way, kind of the intermittent cycle to which you go as you use AI to generate your code. Where, yes, there are sporadic periods where you get a lot of code generated and you feel like you're getting done a lot, but then you need to stop and debug and understand all the code to keep extending and continuing that.

So at the end of the day, you're still doing 80% of the work because 80% of the work is done in this intervention points. And that's where the difference between a senior and a junior will shine in debugging, refactoring, skills, solid principles in clean code. And I think this matches the reality of most developers working with AI. There is this initial excitement, it feels like you you get dann reicht, denn especially if you're in a greenfield, is very tempting based on the data we analyzed, junior devs and mid level devs.

We're the ones that use the AI the most. Which is I think junior and mid level devs are not aware, not 100% aware, mostly because of the the number of projects they've seen. But they are not aware of this last mile, the height surface to finish things. And when you go through this cycle a few times, then you get a bit of more realistic feeling of actually how much more productive can AI actually makes me.

Sometimes you stop using it so much. You just just keep this in mind, do not give in to the to the, you know, the the the instant gratification at the beginning. You will go through these loops and then you'll get a better feeling of how it can uh it can actually help you. Now, finally, some reality, though. Zuckerberg said they're working to replace developers in 2025. So, a lot of people are considering going into plumbing.

But if we look at the uh US Bureau of statistics, the projection is that software developers, testers and computer occupations overall will grow much faster than any others. And plumbing will grow at 6% where developers, you know, the software dev positions will grow at 18. So you might want to think twice if you want to get into the plumbing sector. Yeah, and this is no joke because you see people saying things like, oh, we need to be AI safe, it's not worth it, you know, there's tons of competition.

And it's true. You know, part of this is true, the market it's slightly more competitive. The whole hiring process got a bit more complicated, I would say, not not complicated for the software engineers who are doing things well. And if you're looking to be one of those engineers that can make it uh in in today's market, then, you know, this is the the channel you want you want to follow. Because this is nice job to bring you the best tools for you to make it.

What what I mean is, you see a lot of people kind of, I think very quickly reaching complasions based on all this noise that's right now in the market. And making life-changing decisions. If you quit coding and you go into some other job, you know, coming back in a year, of course you can do it. Depending on how much experience you had as a software engineer, but coming back in a year, it's not going to be the best move you can make right now.

Right? The best move you can make right now is to just have a cold head. Do not freak out, do not fall into desperation. Look at the data, right? Do do integrate all these AI tools, but do not think for a second that AI will replace software engineers in the near future. If you look all the big tech firms, which are the ones spreading this message, are actually hiring. And if we look at this graph, we see this total jobs that are down.

But if you look at the growth from the low, it's over 34%. So we are 34% better than we were. And I'm 100% sure the market will recover because markets, like everything in life, are cyclical and we will see the other side of the cycle. There's one way to lose the game, it's by quitting. The other thing, and I think this is very useful to think about in first principles is who adds more value to society and who adds more productivity to the world?

A software engineer working on a piece of software that 600 companies use and their staff, or a plumber? Right? The problem with plumbers is they don't scale. Software scales, what happens? Software is very capital intensive because it's a very human intensive business. And that's why you have all these CEOs just praying one day they will replace developers. But if you replace the developer, you replace the value added in the freaking software.

So you don't have a job as a CEO. There's going to be one company maybe building the best model and we are all doing something else. Keep on coding, do not get distracted, do not fall into procrastination, this is an amazing field to be in, and we will still need you, software engineers, for probably years to come. So if you decide to still give software dev a chance and not jump to plumbing yet, then make sure you take the free triangle assessment to see where your skills are, we build this with interview questions from the 2025 market, so you'll get a clear idea of where you are right now, what are your strengths and weaknesses,

and you can actually work on them and use your time to get better.