AI does NOT make developers more productive… Okay. Maybe a bit.

Okay folks, so we finally have real data on how much more productive software developers are when using AI. And this is coming from Stanford University, they basically did a study with over 100,000 engineers, 600 companies participated in the study, and millions of commits made with AI. You have numbers what they found out is that whenever the task complexity its high, the productivity increase around 10 to 15% maximum.

This is very different from what people like the tropic CEO said where they claim that in 3 to 6 months AI will be writing 90% of the code, and I think he made that statement around January. And this is uh Dario Amode, I believe, and then uh Satya Nadala, the Microsoft CEO claimed that right now 30% of the Microsoft code, it's written by AI. We don't really know what that means, but again a very, very bold statement and you can see Zuckerberg in the picture said in January that they are trying to replace middle level engineer engineers with AI purely at meta.

Again, very bold statements made by tech CEOs that are trying to pump up the valuation of their companies and also AI companies that are trying to sell their products. But we finally have this study, which is somehow neutral, it comes from a from a third party, which is the Department of Stanford University responsible for research on software developer productivity. In this video we will dive into the numbers behind that study and what they found out.

We will uh look at why I strongly believe it will not get exponentially better, they've been claiming that models are getting exponentially better every months because they really are looking for investments. But if you look at the probabilistic and mathematics laws behind those models, more power, more compute power or more context, bigger context windows will not fix the problems they have. So I do believe strongly it will not get exponentially better and how you can survive this toxic environment that AI created.

Because when you have this big tech CEOs making these strong statements, well then you have the management, you have your software company coming in and saying, well, how do we get our developers to, you know, to be 100% more productive or how do we get rid of half of our developers? These conversations are really happening. You know, the CEO goes to the CTO and tells them, well, how much AI are you using? It created this extra toxicity in teams where, you know, developers are kind of being uh pushed to to displace themselves or at least that's what sometimes a business and management thinks.

Now, who am I? I'm Bogdan, I'm the uh full co-founder. That's my twin brother Rush uh 5 years ago, we founded a senior dev. In the last 5 years, we helped around I think 320 developers get to the senior level, mostly on the JavaScript and kind of the web stack. And some of them got beyond the senior level to principle and beyond by achieving technical mastery. And learning helps themselves very well. In the last three years, I used AI for coding almost every day.

I do code every day. I love coding and I did use AI. I was one of the first because obviously when it came out, it felt like an existential threat to myself. I invested around 10 to 12 years in this career already and so obviously when AI came out, I was like, okay, let's let's see what it can do. I wrote around 150,000 lines of code with AI, started with copilot when no one was using it. I switched to uh then the uh cursor and Windsurf came out and then Claude code and I even tried deep seek and other models.

So let's dive into the numbers behind the AI coding that they discovered in the study. So they analyzed over 100,000 engineers, 600 companies and millions of commits and they measure productivity based on functionality delivered. Not lines of code. And this is really important. A lot of the claims that you see people saying, oh, I feel like I'm so much faster, it's number one, based on the line of code or features they they shipped.

But many times they're just kind of copying, you know, functionality, they're not really building software. It's more like, oh, I replicated this and I built all this in a day, but obviously you have no users because replicating functionality is usually very easy. or just pushing commits or lines of code. And they focused on real life enterprise software. Not open source, not hobby projects, because they really wanted to see, okay, what's the actual business increment value of shipping that code?

you know, is that really valuable? Do you need to rework the code? Are you actually building valuable features or is it just you're shipping threading code? And so they've been doing this since 2024. And what they found out is that the net productivity is usually around 0 to 30%. It depends on the language, it depends on the size of the codebase, and it depends on the task. But in general, yes, maybe you can write 50% or even 100% more code, right? You can write twice more code with AI, but what they found out is that you will probably need to rework that code and you end up reworking or redoing or refactoring that code in the next couple of commits, that obviously doesn't add to your productivity.

right? Your net productivity will not be substantially increased if you need to go back, review and and fix bugs from what you just committed. And so they kind of try to average those things out, and and you can look at his video to see exactly the analysis method they used. But I think it's one of the most rigorous piece of research we've seen so far. And so 0 to 30% is definitely something, it's definitely cool, but again, it's basically equivalent of eliminating like two meetings that you have that maybe they're not very valuable.

So it's not like they double people productivity. AI is very good at boilerplate in popular languages like JavaScript, especially React. That's why people are so scared of it, because a lot of the code we write nowadays are it's front end and if it's front end and 90% will be React and so if you look at the scenarios, we are using a very common language. or on you start from scratch, then you go much, much faster with AI. So what they saw is that on greenfield project, you can go as as high as 100% faster.

If you use AI, so basically twice faster, whereas if you go to Greenfield where you have legacy code and you know, the system is distributed, you have several constraints. then the productivity drops to around 20%, which is basically, you know, you are out the room plate is probably a bit better, but that's pretty much it. I mean, you still need to solve the problem yourself. Now, AI does struggle with bigger code bases. So the bigger the code base, the worse the benefit. People will say when I say is that well, you know, the context window will just get, you know, 50 million times bigger, you will be able to fit everything in it. But we'll see in a second that context does not actually solve any problem, and what people are finding out is that the more context you provide, the more the model will hallucinate. So context is not the solution, but bottom line, the bigger the system, the more distributed it is, the less productivity gains you will get from AI. Now, let's talk about why it's not going to get exponentially better. And I believe it will not get exponentially better because no matter how much the investing computer and context windows, and it's a fundamental design problem that LLMs have. which is that they are statistical machines, and so the more context we give, the more instruction we give to a model, the more it will hallucinate. The more the probabilities will stack up, and what they found out is that even if you have Cloud 3.5 sonnet with a context window of 200,000 tokens, as soon as you go beyond 8,000 tokens, the performance of the model drops significantly. So, just keep this numbers in mind, the context window right now is 200,000 tokens. So, you could fed that many, but the more you fed past 4,000, the least quality you get in your output. So, we've seen this red craze between AI companies trying to build models with bigger context windows, because they are not really taking into account the probabilistic nature of those LLMs. And so, what they found out by coding with AI is that the more cru and rules I would add, the more context I would give, the worse the quality of the code that it will produce, I would have to prompt it again and again, and again, and again and it may be start from scratch. And so, you get to this point where it's just kind of like a slot machine. Like, yes, sometimes you just get lucky and you get exactly what you needed. But that happened after you prompting it 5, 6, 10 times at that point, you're like, okay, can I just implement the feature? Like, if you know what you're doing, you're probably just going to go and fix it a lot faster than it took for the AI to do it. And AI bands to the law of compound probability. This is something that most of us studied.

high school, which is you know, whenever you flip a coin, if you flip a coin and you want it to always give a certain outcome, every time you do it, the probabilities they compound and it gets smaller. So Like if you flip once you have 50-50, but if you flip twice, it's 25% and so and so on, right? So, with AI, the more the agent prompt itself, the more subsequent prompt you add, the more context, then the higher the probability that something wrong will happen.

And so, context will not solve that problem. This is a mathematical limitation of the model. But what we are seeing with a lot of those companies is that they make bold statements that have no scientific or engineering foundation, just because they need to raise capital. It's getting like a hype machine to get marketing in and to convince some investors that, yeah, they they should invest in this.

Now, how can you survive the toxic environment that AI created as a developer? Because the reality is you'll be forced to use AI regardless of what I'm saying right now. One in three dev job postings are mentioning AI and LLMs as a desirable skill for developers. And management is pushing to use AI literally everywhere and for everything. We've actually had people in our program where management had everybody learning Python, even if they are JavaScript developers, just because they might be using AI, so they all need to learn Python.

Which is literally nonsense. Now, how can you win in this scenario? Well, number one, you want to get crazy good at the craft. Because there will be a lot of messy code, there will be a lot of slot generated with AI. People in general are getting more lazy, the quality of world it's decreasing, and errors will happen. Right? We are running production software, enterprise software, you pay for mistakes. If errors happen in the live environment, there's usually a customer that will, you know, which to customer carry, right? Things that have to be undone.

And so, in general, for production software, there are consequences for mistakes. You cannot just throw some AI slot there and expect that's going to work. And so, stuff will break, you'll have to fix it, and that's when you'll stand out. You want to get crazy good at debugging. Is that skill nobody nobody it's uh interested in anymore? people are like please fix this. Are you want to get crazy good at getting the error stack traces?

I know people just copy it, they throw it to GPT, not good behavior. Read it, you will probably be able to fix it before the AI. And get crazy good at getting code. Because again, it's so easy to generate a lot of code. People are just like tap, tap, tap, tap. But in reality, uh you'll have to read a lot of code, you'll have to read a lot of PRs, you'll have to debug very fast. And that's how you will stand out.

And push AI to the limit, so do use it, try to build features with AI. And you'll find the limitations I personally use it as much as I could. I was like, okay, let me let me just do like the product management part, right? Let me just prompt it and and see what's up and I research as much as I cared until I just got to a point where I'm like, okay, they're actually making claims that have no scientific.

or uh any kind of engineering background, they're just throwing it around. They definitely haven't worked with the model or if they did work with the model, they just kind of jumped on the vapor where bandwagon and they just trying to get everybody to use the model. That it's good, it definitely helps, but it doesn't do the work for you. Add LLM including tools to your resume. Keep all your existing stack, but do add, you know, at least one bullet point where you mention that you're using LLMs.

They have the names there, you know, cursor, GPT and so on. So they see that, yeah, you're open to it and you're using it and you know how to do it. And remember, if you cannot fight them, join them. So, even if I'm telling you all this, even if the folks at Stanford are claiming it, we probably see companies going along and tagging along for the next six months and a half. Until they realize, oh, okay.

It's actually making us more productive, but we still need great people that build software. And at that point, they'll get back to hiring developers and realize, um, okay, maybe we we fall too deep into this and then we'll move on to the next hype train. Okay? So, you can't fight that, you got to ride the wave. But do it from a perspective of an engineer that knows the truth and then decides to just play along because, of course, uh sometimes that's the reality of the workplace.

Now if you're a JavaScript developer, there's a free training in the description where uh we explain what we do and what is the fastest way to get to senior and full stack developer. as a JavaScript developer. Check it out on our website, and I'll see you in the next one.