System Design Mock Interview: Build Amazon (Load Balancing, Microservices, CAP Theorem, etc)

Now today we're going to system design Amazon.com and we will focus on the catalog page and the product page. And so this is for you if you're a mid or senior level engineer that is worried about AI getting a job and knows that you definitely need to upskill in terms of system design. In this video, I'll touch upon all the essential system design mental models like load balancing microservices eventdriven architectures, and we'll also go deeper in very concepts like the captorium and consistent hashing. So, let's get started. Now, the most important thing about the system design interview is that you are expected to lead it and in order to do so effectively, you need a mental structure. In general, we want to first focus on requirements and then jump into the highle design. The reason for that

is because you definitely want to get to the system design as fast as possible. You don't want to spend 30 minutes talking about requirements and then interview. Then we'll talk a bit about API, data modeling. And in this case, things like state management and web performance. And finally, we'll do some deep dives both into the architecture, the database, and so on. Those ones you will coordinate with your interviewer. And finally, I added a small architecture review just for you to wrap up the design and talk about the future work you can do to scale it even farther. So, as I mentioned, the first

thing you want to focus on are the functional requirements. And this is basically what will the users be able to do with your system. So, as an outcome of the functional requirements, you'll come up with user stories. If they give you a screenshot, you can do that using the screenshot of the application. In this case, we have the catalog page of amazon.com and the product page. And what you want to do with your interviewer is to decide what it's in scope of the design and what is outside of the scope. For this particular system design exercise, I want to focus on the catalog feature, basically rendering results, being able to search, and I'll pretty much know everything else. Like for example, this add thing over here or everything that has to do with order or even adding to cart. That's a very interesting feature, but I'll probably tackle it in a future video. I'll also

take out all these other kind of sorting and filters. I'll just keep it simple. We just want to make the search fast render results be able to show different pages. So, this is also in scope. This other ad, it's also out of scope and the suggestions. And then going to the product page itself, I really focus on rendering this page. So, basically the images, the description, and so being able to click add to card, but that's pretty much it. We will not focus on the card functionality itself, which is also very interesting, especially if you have several devices. But for now, we'll keep it simp. So, users should be able to search for products. They should be able

to view your product page, and merchants should be able to add and modify products. This is more of an optional kind of nice to have that I will tackle if we have enough time. Now, once you're done with the functional requirements, we need to talk about the nonfunctional requirements. And non-functional requirements are not about what the system does, but how it does it. how fast, how efficient, how many users can it support. So in this case, I selected loading speed as a nonfunctional requirements that we definitely need. This is an e-commerce use case. So the website has to load very fast both on the product page on the search page. We

want to be under 200 milliseconds. And we see that this changes our architecture especially on the front end. The other metic is one of the core web vitals. We'll go in depth into them, but it's basically having a large contentful paint of under 1.5 seconds. When it comes to product pages, it's probably the product images or description. The second one is availability. So, how much downtime can we afford? So, I went for a 59 availability. And we'll see that this also changes our architecture. The third one and a very important one is traffic.

And you want to ask your interviewer, are there any picss? And what is the difference between search and product page? I made some very fast assumptions. I assume that there's very big PS on Black Friday and Prime Day. I was very conservative. I multiply the traffic by 10. But I think on Black Friday, for example, you probably have 8,000 more traffic than on a normal day or even more. For every search a user does, I assumed on average five product views. Now, we don't want to go too deep because it will require you to spend more time here and you really want to move on and start designing the system as fast as possible. what you want to tell your interview that you'll come

back to those and do some back of the envelope analysis as you go to the system itself. I also mentioned SEO and accessibility as requirements that's standard for an e-commerce websites and for every feature I will let the interviewer know that we will talk about consistency versus availability because as we mentioned here we want to be very available but that means we might have to sacrifice some consistency according to the cap theorem but we'll get back to that later. Now let's talk about highle design and I added here what I call the junior level system. This would be a very basic system that I would like to

believe everyone can design. So basically here we have our users that make requests to our front end that would be in a modern JavaScript framework because we definitely want to reuse components and we want reactivity and then we'll have a backend monolith. I chose NodeJS with a REST API for it. Those two things live in the same instance but the application is client side render. So it means that the users will download the JavaScript HTML and then make a request to the back end. And we also have static storage to store specific assets. The most important ones are product images. We also have a

database and I went with a posgress database because there's a lot of relationships between products. You could probably build this also with but in this case I would go with the database that you are mostly familiar with and letting the viewer know that as you scale your system you might change this or you might add different types of databases. Now let's start scaling our highle design. Now the first and most simple optimization we can do is to load balance our back end. So basically we add a load balancer here that will distribute traffic between different instances and this is one of the smartest way to scale if you want to

support variable traffic that means there's definitely a variation we mentioned we will have a peak and so this kind of setup allows us to add more instances when we have more traffic but then remove them so we are cost efficient. Of course, you could go into vertical scaling and just put a bigger server here. But the problem is you'll be paying for it quite a lot and it would be mostly underutilized and you'll only take advantage of it when you have high traffic. So your costs will go up very high. You're also not very flexible. It's hard to shut it down and resize it. So it's much better if instead we just use smaller instances

and distribute the traffic. To be able to do this, you got to make sure that those instances are stateless. So you got to make sure that they can independently generate an answer. And that means all data needed lives in the database. Finally, this load balancer has an algorithm. There's different types of them. You can distribute traffic based on how those instances are doing. How's the CPU? How is their memory? Or just do round robin, which is basically kind of going one by one and giving a request to the first one, the second one, the third, or you could forward the next request to the instance that has the list connections and so on.

If you mention round robin as an algorithm or list frequencies, you should be fine here. The next optimization we want to add is basically replicating our database. As we mentioned, we were able to scale our back end, but now the bottleneck moved and all those requests are waiting on our database. And as I mentioned, we only have one single instance. And so as we start getting more and more traffic, we will see it slowing down. Another very important aspect is that in most systems you usually have a lot less rights than reads. This is very common whenever you have one of those web applications that receives a lot of traffic. In the case of Amazon.com, a merchant will update a product every now and then, but at the same time, a lot of people might view that product per second. So, we have a lot more reads than rights. And whenever you have this

kind of asymmetry, you can scale your database by using replication basically by having a single primary database where rightes happen. and that is basically propagated to read only replicas which are the ones being read by the back end. Now the only disadvantage is that you'll have some lag. So it will basically take a while for all the replicas to contain new products or updates in a product. And this is a good example where the captorum comes into discussion. And basically the captorum tells us you can only have two guarantees at the same time. Consistency availability or partition tolerance. Now because our nodes are right now distributed across the network we already satisfy partition tolerance which means we are left with one of the other and in our case right now the system it's available because if any of those copies goes down we still

have all our data and we can promote one of the readonly replicas to primary but we are only eventually consistent. That means for a short 200 milliseconds the system it's inconsistent which basically means that data that they read it's stale it's the old one it's not the latest right now the captorium is probably one of the most important mental models in system design and if you're able to really understand it and present the trade-offs to your interviewer you will really stand out now database replication definitely helps us handle a lot of traffic but it still doesn't solve one problem which is as merchants start adding more and more products to our database, it will be slower and slower to find a specific product between all those millions rows. And the reason for that is because every time we look for a specific product or

customer, we need to do a full table scan. That means we need to go through each single record. So that means we have to do in this case 7 million lookups. Full table scan has a time complexity of O where N is the number of rows. So in the worst case scenario, you have to look at all of them. Amazon.com has around 600 million products live. So imagine checking 600 million records every time you need to find a single one. Luckily, there's a very easy way to fix this, which is basically creating indexes on specific tables. When we index, progress will create a new data structure for that specific table. The underlying data structure, it's a B3. I will not go deep into how exactly it works. But the important thing is that in a B3, you will be able to find a record much much faster. So if we have 10 million records, a full table scan

would be doing 10 million lookups. Whereas if you have a B3 index, you'll only do 23 lookups. So as you see, it's logarithmically smaller. And this is one of those examples where your algorithmic and data structure knowledge really helps you in system design interviews. Now, database indexing, it's one of those senior mental models that you really need in order to stand out in senior level interviews. And if you want to know exactly how you do that, take our free technical assessment, link it in the comments. We build this over the years with real interview questions and you can really figure out where do you stand. Now, with that being said, we've done a lot of optimizations here in the back end, but regardless of how fast our back end is, our front end might still be slow because of different reasons. And this is what we tackle next. Now let's talk about latency in static

assets. As we saw our page has a lot of images. We also have fonts. We also need to bring in our CSS and we had a hard requirement for a page loading speed of 100 200 milliseconds. So web performance here becomes very very important. Now not only those assets are very heavy but they also might be very far away from our users. Imagine a user is here on our east coast and normally they'll have to go all the way to Virginia if that's where our server is to get an image. This round trip is already very expensive and it's even more expensive if you download a big asset. So the easy solution for that is to distribute our assets into a CDN. So basically whenever we upload it to the blob storage it stays there but we have this edge locations and whenever a user tries to request an image they will go to the

edge location it'll ask is the asset there and if it's not there it will actually go to our blob storage or history whatever that's stored and then come back they get it and it's also returned to the cache. So the next user that comes through will get the resource from a location that is close to them. And so all in all adding a CDN to host all your static assets, your fonts, your JavaScript, it's one of the easiest way to improve the web performance of your front end. So our new system looks like this. We have our CDN that basically pulls from our static storage or blob storage. And then once the users have the JavaScript application or the

browser, they start making requests to our load balancer which finally goes to the back end monolith. And when it comes to writing and reading to the database, we have our replication. So we have our primary database and the read only replicas. And we also have specific indexes on specific tables to make sure that our reads are fast. Now the second layer of caching we can add is at the server side. So we saw how we added caching at client side with the CDN but we could also add a radius cache to avoid so many trips to the database. remember we had this huge spikes in traffic and what happens there is that imagine on Black Friday Amazon.com promotes a specific product and so a lot of people get an email with that product and all of a sudden on a specific product we get all those reads we could send all that to the database directly of course and maybe our reader replicas

can take it but it's a cheaper and more performant way to handle it which is basically to write the whole answer of this query into a reddish cache that's a very fast key value storage that will sit very close to our servers and so when our server servers get a request they will first check if that thing is in the cache and if it's there they will return the cache version so the request never hits our database now the hardest thing about caching it's cache invalidation so when do we decide we need to clear the cache now common sense would tell us whenever there's a right we can maybe invalidate this cache and

that can be done but it would require a bit more application code so you need to kind of write that code in your back end and you need to maintain that code and you need to make sure that everywhere in your application where you upload certain on a V card, you also invalidate a cache. An easier way is to just declare a TTL. That's the time to leave, which means how long an item will be fresh on the cache. We say 60 seconds. So, this kind of protects you against those spikes in traffic because if someone starts requesting a product, they'll get that product and will stay in a cache for the next 60 seconds. So, if you have a lot of users doing the same, you're kind of protecting against,

of course, you could be more aggressive. You could say it can stay in a cache for 1 hour. And it all depends on how often do users edit the product and how much consistency you are able to trade off because the more something stays in the cache, the more inconsistent your system will be because they might read a cache version that is stale. And with that being said, we close our mid-level system. We have a CDN. We have load balancing. We have read only replicas for database. So replication is in place. We have static storage for images and fonts. and we have a reddish cache to avoid overloading our database whenever traffic increases. This is decent for you to show mid-level competency. But if you want to show them you're a real senior, you need to go a lot more in depth. And that's what we do in the next part of the video. And by the way, if you are a software engineer

and you're interviewing right now and you're looking for your next position, we have a special fast track program which guarantees that you'll get a software engineering offer in 90 days or less. Our last cohort had amazing results and if you want to know more about it, the link is in the comments. In most interviews, your interviewer will have you writing the API declaration also because they really want to know that you can program to interfaces and that you can go deeper into the components, not just painting them on the screen. The easiest way to do that is to start with a REST API, especially in our case where it's literally just listing products and then getting the product detail. A rest API is usually enough and rest is very simple. And the one thing you want to make sure is that you're able to write the rest endpoints for a specific API. So rest it's all about nouns and they go

in plurals. So if we want to build something about products that's a product entity well we'll call it / products. And normally in rest we use a stable URL. As you see they all start with products. The only thing that changes is if there is a slash and what comes after we also have an HTTP verb that will dictate the type of operation. So get it's mostly reading records. post it's basically adding like create in patch report it's basically updating a request and then it's also delete which I didn't add it because it's very self-explanatory whenever you want to get products you either get a list of products and we see here I use a cursor

pageionation but we'll talk about pageionation later on so ignore that for now or you can get the specific product by ID so for products you could get different offers so that's important to know that you can actually nest also resources or you could get products reviews for example and then to create a product the agents will make a post to products and in the body of that post they'll add the product information and as a response they'll get the 2010 one resource created if they want to update a product they'll make a patch or a put I'm not going to explain the difference between patch and put right now but make sure you look that up the second part

they might ask you to do it's a bit of data modeling and so data modeling basically means understanding what are the different entities that are in your system design and so looking back at a screenshot of our product page we could actually model, our first and most important entity, which is the product. So, it's pretty obvious that our product has several images. It has a title or a description. It looks like it has a subtitle. Uh it says here, "Visit the Keycom store." So, it has some sort of a store or brand. It has reviews. And uh then it also has a price. And there's a lot more in here. But we also have this. This is what we call in a comma is the S keys, which are basically variations of that product. Each one of them has an image and a price. There's also brand compatible devices which sold text a keyboard description recommended use for

which means like a collection of labels. So basically this is the art of breaking down the screenshot into actual data types. So going back to how your final result might look like. In this case I have a tire bill that is products. It has an ID a title description as we saw and then it has a foreign key to another table which contains the brand information. That's because you can have many products that share the same brand. So it doesn't make sense to share them all together in a single table. This might change when we are optimizing our RITs. So for now consider we'll have this kind of relationships to really

separate the entities. As I mentioned before, we also have SKUs which are a typical way to show a variation of a product. So we have the product ID. It will have a code and then there's different options that it actually verbalizes which I left as a structured JSON B. Finally, it has a price and if it's in stock. So that's kind of the complex thing about an e-commerce like this is that we have the product but then the product has these variations that we also have to express. All in all in this part make sure you're able to number one extract the entities and number two explain the relationships between them. You don't need to be

proficient in SQL. You just need to make sure you understand onetoone relationships one to many and many to many. Now one of the most important thing about the API in this case it's the pageionation part. And a very typical question is should you use offset pagionation or cursor pagenation to pageionate your product. In this case we see this pages different results and then we have this kind of handler where users might click and choose to go to the first page or page number 20. Now that we know the customer requirements let's talk about offset and cursor. So offset pagionation would basically be

the case in which you have a page and a limit like that's how many items you have for a page. So in this case we want the third page and there 24 per page. So what the back end will do it's make the query like select all the products and be like okay you want to skip the first two pages each one of them has 24 so we want to skip the first 48 return the next 24 let's imagine you sorted this by created at so basically the query will look something like this like select all the products order by uh we could choose a different score that can be compound like created by or relevance and we will add a limit and an offset give me the next 24 starting with number 48 the only problem with this approach is that when you have multiple pages, let's imagine I have one, two, three, four products here and page number two has another four products. Just to keep things simple, what happens is let's say I am in page

number one and then I want to navigate to page number two. But just when I navigate, somebody will add a new product and if you follow our sorting order, that new product will actually be added here. So that means this is actually now my first page and this last product that I had in page one becomes the first one in page two and so on. So this one also moves to the next page. What happens when I navigate is that I end up seeing this last product that I already saw twice because I see it again here and also I'm not aware that this one was added here. So that's kind of the problem with the pageionation that

you might end up in an inconsistent mode if you're navigating as the ad products. The other way to pageionate it's curs pagionation. And in cursionation you basically sort your items. So the way it looks like it's you know we're just going to have all our items in a continuous list. And what we keep it's the specific ID of this last item we saw, we get the next 24. And so even if something was added here, we don't really see it. Adding new products will not change what I actually see after them unless it's added after, which makes sense. So it's an easier way to maintain consistency. And basically with

cursor, you just say, you know, give me the 24 and this is the lowest one I saw. Now cursor pagenation is really good whenever you have one of those lazy loading infinite scrolling and offset pagionation. It's usually better for catalog based. And because Amazon.com it's a product catalog, I would go with after pageionation here. But being able to explain the difference between those two and how the API endpoint will look like, it's a seniority signal. And by the way, if you're a software engineer and you want to get to next level, you want to become more senior, you want to get to a better job, check out the free resources that we link into the description and our free training about

how to get to senior in the most efficient way. Okay, now let's keep scaling our system. And the first concept I want to talk about it's sharding and consistent hashing. So basically as we add more and more products our database might run out of space. So this primary database might be so limited that we need to split into different instances that store different records. Now a quick note normally posgress databases are very efficient and you can scale them very well. Unusual product for amazon.com it's usually around 10 to 20 kilobytes of data. It's mostly text data and one single posgress database instance can store about 1 billion products. And just for reference, Amazon.com has around 600 million products listed. So that means you could possibly run Amazon.com with a

single database and a very efficient reading layer. Now before we move on, let's quickly explain how sharding works. And so basically whenever you do sharding instead of writing directly to database you have a router component that will dictate where the right will be because you might have different shards. So are basically different subsets of data. So you can imagine if we shard by ID and our ID would be a number you know you have one to 10 million records in the first shard and then you know 10 million to number one goes in the next shard right all the way up to 20 million and then you might have a third shard. And so what this router does, it's look at the ID and then redirect the query to that specific shard. Now remember the cap theorem. So whenever you kind of distribute your data again, you will use consistency or availability. In this case, we'll use a

bit of availability because it takes longer to write things to the shard just because you know we need to go through this router first. Now the other problem is you might want to add another shard. you need to move all those records and if they had read only replicas or caching attached you need to clear that and back fill it again. Um the way to avoid that is by using consistent hashing. I do explain that in one of my other videos so not get in depth in here but do remember that shing it's kind of complex and for the most cases you don't really need it. However you need to be able to talk about it and explain it in a system design interview to really showcase seniority. Now the next thing I want to talk about it's front end performance. So basically how fast we load and we've seen so far that we have a CDN and we are rendering on the client but that will still not be enough for

the 200 millisec we had and I did run a lighthouse diagnosis on amazon.com page and we see that their largest content for paint is 2.9 and the first content for paint it's 1.3 seconds. So that's pretty slow. Of course, all this depends on your network and it depends on where you're based, but in general, the performance score was pretty low. So, how can we improve performance and can we still do client side rendering? Well, let's take a look at how our request looks right now. So, basically, we have our browser. They want a specific product. So, they'll get a specific product. Normally, here we have the UU ID. Now, whenever they hit our domain, they will be redirected to our CDN, which will give us the empty index HTML. So that will be an empty div because again all this is client side rendered. So you still need to pull that empty

HTML and then render on the client. So they'll download they'll pause the JavaScript and then finally they'll get to a point where they pretty much render the application but they still need to make a get to the back end to get the latest information of that product. So they make a get to the back end. They select the product from the database. It comes back and then finally they show the information of the product and that leads to a first contentful paint of around 2.5 seconds. Now, of course, here they could have shown some loading indicators in the title and description and the images and so on. You could move the first content paint here, but at the

end of the day, here is when you actually show the information to the user. So, that's pretty slow. And how can we solve this? Well, you probably heard about server side rendering. And so, in the case of serverside rendering, our request life cycle looks a bit different. We go from the browser and make a get. Now, this will usually be done to something like Nex.js, which will call the back end, get the product information that will make the request to the database and then come back. But again, those three pieces of infrastructure, they sit very close. So those requests are very fast. And finally, the next JS will pre-render, give us back a full HTML page. So we still are looking at an empty screen for a short while. Like the first version we see comes in much faster because we get plain HTML. We don't get that empty div and then we have to render on the

client. The only problem with this HTML is that it's not yet interactive. To make it interactive, we need to hide it and that will take a bit more time. The other optimization we could do is if we want to update prices live, we could actually stream the updates. So we get an initial price from the back end. But if they change the price later on, we can listen to that with server sent events. So the backend might do a live change of prices, which is pretty cool because we could listen to those updates and show them to the user without having to refresh the page. So serverside rendering is pretty decent. But it turns

out in the case of e-commerce, we can go even farther. So what we could do is to have a mix of static and incremental static regeneration. How does this work? Well, basically way before we start getting users when we deploy our application, we could have a request in our CI/CD pipeline getting top 10 key products. And then for those 10k products, we could actually pre-render the product detail page and push it to the CDN. And so we know that those are the most seen products. And what will happen when a user requests that product is is that they immediately get the pre-under HTML. And that's how we bring the first content paint to under 150 milliseconds. Of course, they'll need to hydrate this. So it becomes interactive around here. But this is a lot smaller than it used to be. And the other advantage is we don't make those

requests to the back end all the time. So we don't need to spend so many comput resources to render. Now a good question would be well wait a second. What happens if we update a product? Do I need to redeploy the whole application? And the answer is that whenever you update a product, you could trigger a product updated. So you need to revalidate that and what will happen is that the NexJS application will rerender that specific product and push that new version to the CDN and our server send events work pretty much the same. So in this case, we're able to be much much faster. The trade-off is we have more

complexity and takes longer to deploy the application. So if we were to look at our product detail picture right now, we could see that we could basically pre-render most of the information except for the very dynamic one. These ones at the top, we will leave them out of the scope because they're not even in the scope of our uh design, but they cannot be pre-rendered because they need backend data. Uh the sponsor ads are also out the scope. But when it comes to the product detail page, the only things we cannot pre-render and have statically on the CDN are basically the price, which changes very often and probably the add to cardart and inventory widget.

As I said, we'll not get too deep into how those two works. And so for the most part, for all the static information, we could use incremental static rendering, render it when we deploy it, and push it to the CDN. And that's how you get the best performance and the most scalability. Now let's talk about microervices. So right now we have our statically rendered application but the traffic still mostly goes to a backend monolith and this will become pretty heavy to run. And as this application grows there will be just too much complexity. Even if we try to make this a modular monolith, it will get to a point where there's just too many people pushing changes to this backend monolith. So it becomes pretty dangerous because everybody can bring it down. The other problem is we have a lot of code running. So each one of the single instances will have to be vertically

scaled. So they take up more resources and so it will get to a point where it makes sense to separate this because there's a lot of a symmetry also between different modules and they get different level of traffic and requirements data intensive in different ways and that's when we start thinking about microervices. So basically we take that monoly that we had and we split it on fully independent servers that handle a subset of the functionality. The big advantage here is that all of a sudden all those services can expose a different API depending on their needs and they can use a different database

and we see here that they chose elastic search for the search service but the product service and the pricing service they are relying on different databases. In this case for products I chose a document database for the pricing service I chose posgress and for the inventory service we have dynamo. All this is to illustrate that those services are independent. They deploy independently. they expose an API and the teams that own those services can work in parallel. You know, like these two teams, they don't really need to communicate. It's really just communicating with the back end. So, all of the sudden you get a lot more

efficiency. The trade-off with microservices is that now you need to pay the microser tax, which means you need a deployment strategy for each one of them. You need distributed logging and usually a request goes from one to the other. So you know the front end will go and make a specific request and in a normal production ready architecture you have the request going through here and then coming back to this other service and then to this one and then to this one and then coming all the way back up. If there's any failure or error here you need a lot of mechanisms to debug that. And keep in mind that this picture right here is

very simple. I only have four microservices. But in production applications, you might end up having hundreds or thousands of microservices. Deploying all of them, maintaining them, and having observability kind of monitoring and logging for all of them becomes a real problem. Now, the one component I added without really telling you was the API gateway. And so, why is it useful to have an API gateway when we talk about microservices? Well, the simple reason for that is because right now we have a front end requesting all those microservices. And whenever the front end does this, it needs to authenticate. For example, a front end might have a JWT token. It might have an HTTPS certificate. It might have a specific caching policy. And all those microservices have to also implement that. They only need to handle HTTPS. They only need to handle caching. And so

all those teams keep reinventing the wheel and implementing the same functionality for those edge functions as we call them. And a way to work around that is to push all that functionality to an API gateway. And so as the name says, this would be the gate to our system. And this is where most of that shared functionality will be kind of handling authentication, caching, HTTP. And so in that sense, those services and the teams that work on them will only have to worry about the specific business logic for that subset of the product, not about reinventing the wheel or implementing HTTPS and so on. And so an API gateway component is extremely good for security because usually the API gateway will be the one that is open to the internet and it will check JWT. It will also check HTPS and once the request is fine it will allow

that request to go into our VPC that's our virtual private cloud which is a private network. So all of the sudden we are protected. Nobody from outside can go in and make a request to the product service or the pricing service. they all have to go to the API gateway which is really where the security team implements you know those hard security checks or implements authentication and so on. So all of the sudden it's a lot safer to deploy our microervices. The second security optimization we can do is to apply the list privilege principle which means each one of those microservices they have a surrounding firewall that only allows the connections they need. So you can imagine every micros service as having open doors like network ports. Well, we close all of them and we only leave open the ones that are actually used. So

basically in this case, we'd only allow connections to the search service from the load balancer component or the React server side render application. So no one else can request the search service. This list privilege principle, it's one of the best when it comes to security. Now, you don't need to be an expert in cyber security to really pass a system design interview, but you do need to know a little bit about it. So, just to review really quickly, our API gateway component will handle the HTTPS and the TLS handshake and it will forward a request to our VPC under HTTP which is much faster and you don't need to worry about certificates anymore. And inside the VPC, the services themselves have their own firewalls and they really restrict as much as they can the open doors that they left. So they can only receive requests from services that really need them and they can only make

requests to their dependencies. The last security optimization I want to talk about it's rate limiting and that's protecting us against distributed denial of service attacks. Basically attackers might control different machines and start requesting your API at the same time trying to overload your servers. And to avoid that you can tell your API gateway to implement a rate limiter. And so basically you are storing how many requests you received from a specific client that you identify with their IP over a specific period of time. So for example, you'd allow them to have 20 requests per second. And if you receive

more than 20 requests per second from a specific IP address, then you deny the request and you send back a 4 to9 request. This is a typical functionality that is much easier to implement on an API gateway level. And so it makes sure that your product teams, your backend teams don't need to worry about this. And at the same time once it's implemented it really protects the whole system. So you don't need to reinvent the wheel for every single micros service. Now one of the most popular questions when designing this kind of system it's also how will you handle the search autocomplete. What do I mean?

Well whenever you go to amazon.com and you type something you will get a couple of suggestions. And so the way they look like it's pretty much like this where you're like okay you type and you start getting those auto suggestions. So how do we make sure that this is very very fast? Can we still handle this with our posgress database? Well, let's see how this will actually work. So basically I'm a user. I start typing. I probably have a debounce so I do not make a request every time I type a letter. We wait for 50 milliseconds or 100 millconds and once they stop typing we make a request to the back end to get those suggestions. And so a very simple setup would be that whenever the users type, we get that word they typed and we run the search query. it goes to the back end and into our posgress database where we make an I like query for that

specific word being in the name or description of the product. The only problem is with this kind of query we cannot really index as we mentioned before. So if we have around 10 million products remember we have 600 million in the case of Amazon it will take around 1 second to do that query and finally to render those full suggestions it will take us around 1 second and that's quite a lot. How can we make this more efficient? And also, how can we protect our database? Because again, this is quite a lot of requests that will happen. Well, there's different layers of optimizations that we can add here.

And the first one, as I mentioned, we basically debounce it. So, we wait for 100 milliseconds. We don't just make a request on every keystroke. So, users type and then they think about what they want and that's when we kind of make the request and show the suggestions. We can also store the results of the most common queries in a CDN. So that's how we make the most common queries very fast. Imagine it's Halloween. Everybody's looking for a Halloween costume. We know the suggestions that'll get more or less. We pre-arch them when we deploy. We fill up our city. And when they make that search, they get the auto suggestions in under 10 milliseconds.

Now, what if the result is not in the Cydia? Well, in that case, we have a cache miss. So, the record is not in a cache. So, we forward it to the search microser. But instead of posgress, we can use elastic search and it's extremely good for search. So it can give you the top 10 searches by popularity in under five milliseconds. I'm not going to go in depth on how it works, but just keep in mind that it's basically a read layer on top of our database that's very much optimized for fast reads and availability. Although we do sacrifice again a bit of consistency.

And so as we get the results from elastic search, we populate the CDN and then we return them back to the user. So the next time the same user looks for the same exact thing or some other user well we go back to this super low latency from the CDN. So all in all our type optimization has basically three layers debouncing the CDN and then using elastic search as a database and that's profoundly more performant than the old request we are doing our system became more and more complex and now we have a lot of microservices which are load balanced. So deployment will become very painful and this is why we need to talk a bit about the infrastructure layer. So the easiest way to deploy a microser architecture is to basically dockerize it. Now what does that mean? Well basically for every single service being the front end or any backend microser

will have a build pipeline that will run the code quality measures. It will type check. It will run the unit test and finally it will package that in a docker container. The docker container contains the code the runtime and the Linux image. And it's basically a packaged version of that service that everyone who runs Docker can run without needing to know what's inside. And the advantage here is that all of a sudden we standardized all our deployment because the problem of deployment now becomes how do I deploy a Docker image. And that's the same one for all our services. The easiest way to do so is to push our image to a container registry.

That's an image registry. And so the new version of our service package with everything that it needs, it's in the cloud and anyone can pull it and run it if they run docker. And so the only thing we need to run our whole system, it's a way to run those images. One of the easiest way is to use manage services like ECS. That's the elastic container service from AWS. Inside this service, you will define a task or service and you'll attach that to a container. So basically, they'll pull that container and they will run it for us. We don't need to worry about anything. they'll end up giving us like an IP address that is static and behind

that IP address they are running our service and the cool thing about ECS is that you just need to specify some highle requirements like take that container and run it on for example two 1 GB instances and be elastic. So if it receives more traffic horizontally scale it all the way to eight instances. So all that maintenance work, deployment work gets simplified because all of the sudden all these services use the exact same system to be deployed. And this is how some of the biggest companies work with microservices. In a lot of job postings, you probably heard about Kubernetes, which is basically an open-source alternative to ECS as an orchestration platform for containers. In general, ECS can be a bit more expensive, but it has the upside that you don't need to learn Kubernetes and you don't need to have a person that

knows Kubernetes in your team, which over time becomes cheaper. Anyhow, this is the simplest way to scale not only your system, but also your operations and your deployment. Now, finally, let's do some deep dives. And the first one I want to do is about the back end for front end. So, as we saw, our users come mostly from desktop. So obviously we get a lot of users that are on the web but we might also start having mobile users and at some point maybe we even decide to build a mobile app. The way a mobile application fetches data is very different for what you need on desktop.

And so the easiest way to satisfy the requirements for these both platforms is to have two different backend for front ends for each one of them. So backup for front end would be an intermediate service that the front end will request and it will forward that request to the different microservices depending on the data needs put them together in a single response and then send it to the front end. And so the advantage here is that the client developers have a lot less complexity. whenever they need to fetch data for a specific page, they only make one request or a couple of them to the back end for front end and is the back end for front end that figures out which service do I need to call and then stitch the data together and sends it back to the front end. Because of the nature of this component, the best technology to build it with, it's usually GraphQL. I talk a lot about GraphQL versus REST in one of my

previous videos about system design. Make sure you check that one out. And this is a clear example where you can actually not only switch databases but also move around APIs and that is the final indicator of seniority which is that you're able to use the best tool for the use case. So you always listen to the specific use case and then use the best API. You're not just like use graphical for everything or use rest API because they are better. You really make decisions like an engineer and understand what are the exact requirements. The next thing I want to mention is switching to an event-driven architecture where not only we have our

different microservices over here requesting each other but they also listen and publish to a unique event hub. And so whenever something happens in architecture like a merchant will upload a new product instead of calling all the different services that need to update themselves invalidate their caches and so on. We will just publish a new event and that is product created in our product stream. All the services that need to react to that event are listening to the stream and they will react to it in isolation and maybe when they are finished they would even publish a new event in a different stream. For example, let's imagine we create a product. So that means we upload a bunch of images and now those images have to be optimized. One thing we need to do for example is to convert them from PNG to WEBP because that's better for web performance or we need to

create different thumbnails. Well, there's a service that is the image resize service that will take our images and do those transformations. And when that service is done, it will basically publish a different event in our media event stream. That would be hey the images are ready. And that's when maybe the application that the merchants use can show that that product has been uploaded and the new image thumbnails are also ready and distributed. And so the big advantage of an event driven architecture is that you have a lot less request between services and that depending on which technology you use,

you can have a lot more flexibility. What do I mean? Well, if we use something like CFKA, it almost works like a database where all the events are stored from the very beginning. Which means if at any time we want to connect an analytics service here, we just basically hook up our analytic service to all these event cues and it will have all the information from the beginning of history all the way to now and we don't really need to change any logic in those services. The only thing that we might have to do is to ask all these product teams to publish certain events. But there is no direct communication

line between our analytics service and their microervices which means there is no coupling. And because there's no coupling, we can also go much faster because again all those teams are working independently. The biggest problem with big architectures is that it's very hard to coordinate all those different teams. And instead of having like a top- down management where it's like a architect that kind of controls everything, it's much better to have independent teams that work against APIs. And that's exactly the mentality Amazon has introduced where back in the days they mandated that teams should

communicate through an API. No email, no meetings. Just show me the API and we will build against it. Now the next deep dive I want to make it's in micro frontends and we saw how we were able to split our backend in different backend teams that build different services but our front end it's pretty much still a single team and the problem is as we add more and more features we'll have more and more people working on the same front-end application and so everything becomes slower because they all need to coordinate. And an easy way to do this is to apply the same mentality that we had in order to split our microservices but apply to the front end. And that's what microrends do. And so in micro front ends, we basically have a single front end that is a shell. And then we have many front-end applications that render an independent part of the UI. In this case, we could have a microrend for

product info. We can have another microrend for our price over here that was communicating with the back end using server sent events. We might have another one for the stock and the buy now. This here it's usually owned by a single team because it's very sensitive and they can make different changes and they could run AB test without having to communicate with the team that just renders the product. Those domains are fairly independent when you scale. And same goes for the search or the top bar over here. They could all be isolated in specific microrins. As we spread those out between different teams, we still need to make sure that we maintain visual cohesion and we can do that by having a design system that is independent, it's built by the front-end platform team and that they all consume. So the product engineers working on

those features are basically assembling different features based on the specific business logic with the components built by the system design team. And in a lot of companies nowadays, a lot of those product engineers are full stack people that use coding agents and they figure my MCP to connect to the design system and basically have the agent insembling the feature with existing components. And that is why if you are a developer that's more full stack, you'll basically do a lot more insembling and working with the customers. But if you're someone that likes a front end, you probably work more on the things like

design system. And that's what nowadays they like to call a product engineer. The beauty of microrends is that you end up being organized in those vertical slices. Which means if you are someone that works on the search, you will own the search micro front end which is lally that input field where we type with the type head autocomp completion and then you will own this section of that in the BFF and finally the search micros service. So all of the sudden you can have full stack product teams. So that's how your team will look like and the advantage here is that there's no more blockers. There's no need to

coordinate and you can deliver features directly to our user. Right? So what we avoid here is people working in sillos having backend developers just doing backend work or database developers optimizing the database and the whole team it's disconnected from the user. So this is kind of the last optimization where all of the sudden our product it's a set of vertical slices that deliver features independently straight to our user and the microrends and BFFs and API layers are responsible of giving this coherence. So it's all still one system. Our next deep dive it's again about infrastructure. We talked about how we will deploy most of our services as containers. But as we scale, it still gets more and more expensive for us to go into our cloud platform and provision

all those services. So one of the optimizations you can add to your software development life cycle is to add something like infrastructure as code which basically means that you have a declaration of the services you need and in your CI/CD pipeline you check your actual infrastructure. So you have your declaration this is what I need and then tools like Terraform will connect to your cloud account and figure out okay what do you have right now in production and they'll figure out a difference that you need to make like okay we need to add these two resources that you declare unless this one you no longer use and so this is what we call

our plan and what will happen when we deploy is that Terraform will calculate all the different transformation we need to apply and then you could do something like terapform apply and it will basically you know start changing all the infrastructure that you need and when you have hundreds of infrastructure pieces. This is very very useful. Now, infrastructure as code is one of those technologies that I'm super excited about and really deserves its own video, but I will not do this now. Do let me know in the comments if you want me to get deeper. Now, the last deep dive I want to make it's availability. And one of the easiest way to be very available is to have what we call the active

passive model in which you could replicate your whole infrastructure in a different availability zone. And if something happens, if there's a tornado that destroys the data center, you could immediately at DNS level switch traffic to that other region. Now again, this can be very expensive because you need to synchronize those two at all times. You need to pay double the cost for your infrastructure, but it does give you the highest confidence in terms of availability. Basically, it means that if the whole system goes down, you are still up. This is very useful when you are building critical services like finance. In the case of e-commerce and something like Amazon, I would only do this for a subset of services that are really critical and that cannot go down because again this makes your deployment complicated. This duplicates infrastructure cost and it's just a lot

more things to maintain and think about. And when it comes to complexity, you always want as least as possible. And the final parenthesis I want to make is that we talked about the type hat completion and elastic search, but we didn't really talked about how data gets into elastic search. And so a typical setup would be that we have our product microservices that's where merchants will add a new product and what happens then is that it's persisted to our primary posgress database but we also publish an event to our event stream. There's different indexes services those are lambda they are synchronous workers they look at that event consume it and then persist that to elastic search. So as you see here the only problem here is that there is certain latency between the operations of creating or updating a product and that product really

appearing in our search and that's why we say that the search feature right now it's only eventually consistent. That means at some point in time it will come back to being consistent but for a certain time interval between the creation of the product and the propagation of the change the system will be inconsistent. This is a decent trade-off to live with in the search micros service. And if you really want to show that you are a real senior developer, you can evaluate consistency and availability, not for the whole system, but rather per feature because depending on the feature, merchants and users might need different levels of availability and consistency. A good example would be in the inventory service. We definitely want strong consistency because when you add to cart and buy a product, you want to make sure that you don't have users buying a

product that's not available anymore. So your product quantities have to have strong consistency. But for product information, images, text descriptions, it's okay to have more relaxed consistency models like eventual consistency. And that's pretty much it. And congratulations, you made it. This was a very fun video to make, but also very in-depth one. I do recommend that you watch it once or twice if you need. And I wish you all the best with your upcoming interviews.