the disturbing reality of AI coding

So OpenAI just released their latest model, which is at this point, O3 Mini. And they are claiming that this model, it's extremely good at coding. And basically what they are saying is that on the software engineering SW Bench verified benchmark, it has 49% accuracy. Which literally means it can solve one in two problems. Now, as a developer, this is really exciting, but at the same time very scary for me.

So, I decided to go a bit deeper. And so the first question that came to my mind was, what is really SVE bench verified? And going down the rabbit hole, I figured out it's a collection of almost 3,000 open issues in 12 Python repositories. And all of them have literally tests to check the issue status. So what they usually do is they run the model on these issues and see, okay, how well did the model did? Because they have unit tests and you can actually see.

Was the model able to code a solution that solves the issue? So, we have our typical get issue, and then the model goes through, it solves the problem and then see, okay, did it pass the test or not? All those were standardized by OpenAI, you can find it on Hugging Face and you can run this benchmark on any coding model and get your own results. So, Bogdan, I mean, judging after this, it all seems pretty legit. I mean, you have the open issues, you have your Python code, you have your unit tests.

And um you just look at the percentage, feels like it can solve one out of two. Should I be worried as a as a software engineer? What's what's the catch? Well, the first thing that came up to my mind is that this test was built by OpenAI, which is already kind of sketchy. Testing your own model and making claims about it. But the second thing, which is even more surprising, it's the quality of the experiment itself.

Now, a couple of weeks ago, a independent research team from York University decided to go a bit deeper into this benchmark. And they found out that the benchmarks were corrupted. Basically, it turns out that from all the AI fixes, a lot of them were what they call suspicious. And they pretty much were able to classify these suspicious fixes into four categories. Number one, it's a solution leak. Number two, an incorrect fix.

Number three, a solution in a different file or all together an incomplete fix. If I understand correctly, means that the model was not that good at solving issues, but what does this solution leak mean? Sure, let's break them down. So, a solution leak means that the solution to the issue was clearly outlined either in the description of the issue or in a comment. And we see example here where in that very issue somebody actually said, well, you should use this and this and paste it a code some.

And so obviously the model took that and wrote the solution. The second type of suspicious solution, it's an incorrect fit. Now, the incorrect fixes refer to the cases where the model will provide some sort of solution that will make the test fast, but it will actually be incorrect. So, it tells more about the quality of the test suit than the accuracy of the model. And we can see here it's a click sample where the gold patch, the good solution was to return the reversed version of a dictionary and what it did, it's actually return the keys, but somehow the test passed.

So basically, number one was that, you know, you give the model the solution to the issue, which is kind of a fake way to to to analyze it. And the second one, it's that it the test is so weak that even a a solution that's not correct can pass. Exactly. Yeah. Sounds like a very confused junior dev that this was me back in the days. I could solve issues that I had a solution for and I could solve issues in the wrong way if the test were written poorly.

Number three was when the model went ahead and just modified something random in the code base that had nothing to do maybe with the issue, but somehow again, it made the test. And so it would go ahead and modify like a totally different file and then all of a sudden the test would pass, but who knows? I mean, this is Python code. If you're doing JavaScript, you can imagine the model going around and changing dumb or window object definitions or prototypes and then somehow the test pass, but it didn't really fix the issue.

So basically he's changing the the rules, the the boundary conditions of of the problem and then of course he gets the the right test. Exactly. So it's definitely thinking outside the box. And then the last one would be an incomplete fix, where it will generate code that will lead to an incomplete implementation compared to the solution, the human solution, the gold patches what they call. It would often just omit critical details.

It would really code against the test. But the quality of the solution would be limited to the quality of the test. I haven't added the code snippet here, but I'll link you up to the research paper and you can see the examples there. Now, if we take all these four things into account and we start adjusting the real performance based on the fact that a lot of the solutions were not actually valid solutions, the performance of the models changes dramatically.

And so for GPT-40, the claimed performance on the SV bench website, it's 18.83%. You can actually check that over here, go to SVbench.com and find it yourself. I took the full one, but the reality is that it goes down to 3.83% when we actually look at valid solutions. Which is a very big difference. In the last model, Open AI claims they have a 49% difference. If we were to apply the same criteria, we'll probably bring that down to around 5 or 6%.

So basically the the benchmark it's it's fake. Because these numbers won't impress any investor, that means the AI model it's fixing about maybe four maximum, optimistically speaking, four out of 100 issues that is being confronted with. And we're still talking about, you know, a collection, we're not talking about the messy environment, which is a real environment, a human developer action. We're still talking about a very structured way, right? You have the GitHub issue and it's documented, that's that's not how most developers work. Let's be honest, right?

So, uh yeah, 3.20%.l I think my our job as software engineers are pretty safe. Exactly, anyone working in a dev team knows that fixing issues, it's usually the lowest effort work. It's a lot easier than building real feature. So, this is not the hardest thing in coding. But, I think at the end of that paper, there was something about the the cost, the the financial cost. like the monetary cost. How much computing power are we talking about for GPT40 to solve those issues, right? You have the table there, nice. So, what does this mean exactly? So, basically, they try to monitor the API cost that you'd incur of really solving that issue in the time. We see that GPT4 with SV agent didn't do pretty well, it took around 600 bucks to uh to solve a single issue. But autocode rover and GPT4 did a bit better with 12.61. to solve an issue, of course, if we adjust that to a 3.83 success rate, it would take you around $400 per issue solved after 30 trials and 2.5 hours of compute. But again, you know, if I'm a tech CEO, I pay 400 bucks and I get one issue solved. For me, like compute power, the time, I mean, it can read while people are asleep, right? It can work at night. It's not a human, right? Humans are the problems, humans are complex, it's what we are trying to get rid of. They are very expensive. I mean, you know, average senior dev in the US, it's 160k, up to, you know, 350, 450. So, it still kind of makes sense financially. If you show this to a bunch of MBAs, which are the people running these companies, then like, wow, wow, this looks good and it's only going to get better. But there's a catch, right? Yes, the thing is you need a complete test suit for those issues. You need someone to open the issues and you also need someone to tell you which solution was actually the correct one. Again, this was humans looking at the actual code. You couldn't really just make the UI if if you make the model just commit to master without human review, you'll get an incorrect solution. And you'd have no way to know if that it's actually the correct or the wrong one. So in this case, you either have an engineer, engineer would be the best, maybe you have a product manager that knows the requirements, or if you really want your automated dev team, you'll have the user directly. You deploy the stuff to production, you get burned and then you go back to the AI to fix it, but then you need to describe the AI what's exactly wrong and you have all these layers of complexity. So what what I see from my perspective is we created this very controlled environment, right, within software engineering, which is this very well tested

did uh GitHub issues. It doesn't matter if there are 2,000, it can be 10,000. If someone wrote the test and basically wrote the solutions inside them, it doesn't matter, right? That the sample is not relevant. And then we are measuring right the the performance of those models against this this very controlled environment that was built by humans, by human developers who completed the test suite, opened the issues, reviewed the you know, and they tag the solution.

And then we are saying we go out and say, hey, we're not going to have any software engineer in, you know, 10 years from now. And I think here's where the stupidity comes from. And I understand it. Why? Because behind those companies there are huge valuations. Uh there's a big promise of AI solving humanity's problems, right? And AI actually being smarter than humans at solving these problems, right?

So, I understand why companies are pushing this propaganda, but I think the damage is done on the human side, on the software engineers who are here on YouTube, looking for information, every influencer, it's kind of selling that this thing is huge, that AGI is coming. And people are making career choices based on this on these skewed numbers. And I think we are one of the few people that actually went, you know, down the rabbit hole and looked at the numbers, but I think the average the problem is the average out there, it's not equipped to understand these numbers.

Because when you do understand the numbers, you don't really care. You get back to get in bed. And the problem with with the ordinary developer is they don't have the time to go to the numbers, identify them and they're just getting I see a lot of people getting scared and thinking, hey, I invested so much into this career and now it's all going down the tubes. Well, we are here to tell you it's not.

You can look check the numbers, check the the files, check the benchmarks yourself. And we invite you to look at them yourself because there is, you know, the emperor, this emperor has no clothes and you don't want to fall victim of these companies. propaganda. Because my final message is, whenever a new model comes out, people throw out to this arbitrary numbers of performance. And it's really interesting to stop and really go down and analyze those numbers and really try to understand what do they mean.

And remember that the result of a benchmark, it's as good as the benchmark itself. And the first thing you folks are probably going to realize when you check the numbers, just like we did, it's yeah, the emperor has no clothes. This is a big inflated valuation behind big companies. And you're going to come back to you, and what you have to do, it's you still have to get better. And that was the point of this video, the point of bug and I and these videos is to get your folks focused back on the things that matter.

And I'm telling you, I don't know where AI, I don't know if that 3.8% is going to be 4.5 or you know, 20%. But as, you know, this thing has been released a few years ago, it feels like we are reaching a plateau in model development. Uh, there's a lot of stuff doing like post training, uh, to to make the the models better. But in the end, human supervision, it's still very much needed. And I think a lot of developers are wasting this opportunity, are getting distracted and they are wasting this time instead of getting better.

At a point where the market itself, it's more competitive than ever. And it's not more competitive because of AI, it's more competitive because of many other factors. So, what I would invite you, it's to get back to working on your skills, get back to planning your career, get back to thinking what, you know, where am I at right now. And the best way for you to to improve, it's for you to get some clarity on what your tinkle gaps are.

This is why I put together a 10 minutes free tinkle assessment. You can check it out in comments, you can go to the Cendev, click on Tinkle assessment. Fill out your name and email so we can send you the results. Uh, it should take you about 10, 15 minutes. Be honest, don't Google, don't use Chat GPT. What you will get back, it's an in depth analysis of your tinkle skills and you will also see how you stand against other developers who took this test.

Bogdan, thank you so much for digging into the numbers, we will see you folks in the next one.