1. Impact of Context Length on AI Performance
Studies show that AI model performance decreases as context length increases beyond 32,000 tokens, making larger context windows ineffective for big codebases.
Hear it at 0:35 →Engineering video
1:22 · 18 August 2025
AI does not do well on Big Codebases
Key concepts in this video:
Transcript
A is struggles with big code bases. And so the quality of the output you get, it's worse the bigger the code base. Now, a lot of people will say that, well, you know, this is just a temporary problem and the models will get much more context windows, and you'll be able to feed your whole thing to it, and so we will fix this in no time. We will see why that assumption is wrong from a technical point of view.
If you go past k lines of code, and 100k of lines of code is not a big code base. That's a medium to small code base. Like, production code bases in enterprise software go way beyond that. Another thing that this study found out is that the model performance decreases as you increase context length. And so it doesn't really matter how big the context window for the model is. With Cloud it, the context window is 200,000 tokens.
But as soon as you feed more than 32,000 tokens to the model, you get very bad results. And there's many reasons for that. But it's a fallacy in the AI world that says that if we throw in more compute, if we add more parameters to the models, or if we increase the context window, we'll get better results. Again, those are statistical models, and so the more statistical parameters you have, the more divergent they will be.
And so increasing context will actually decrease your accuracy.