System Design Interview Full Course: API Design, Caching, Scalability, Databases (Mid to Senior)

So, today we are building the LinkedIn job search page and our final system design will look like this. So, we will have things like API gateways, load balancing, we will touch on scalability, API design, what kind of database to choose, when to use caching and load balancing, and even more in-depth topics like database indexing, using Elastic Search to scale your search, and then scaling the whole architecture by moving towards an event-driven approach. The reason why I chose this specific system is because number one, it's a real system that one of the engineers that we mentored as a senior dev got, and number two, because it really touches on all the essentials that you need to know to pass a senior system design interview.

So, let's get started. Now, the most important thing about this kind of interview is that you are supposed to lead it. You're supposed to go from fairly unstructured requirements like a problem statement or a bunch of screenshots to a full system. And in order to do that, you need a structure. Now, the commonly accepted structure is that you want to start with the requirements and then move on to data modeling and API design. Now, this has to be fairly quick so that it gives you enough space to talk about the real thing, which is the high-level design.

Finally, you probably have to talk about scalability at all levels including load balancing, things like caching, scaling the database, and there might be some space for deep dives at the very end when you wrap it up. The interviewer might tell you, "Let's focus on this specific part." And you'll have to go deep. So, let's start with the requirements. Now, when it comes to system requirements, you can split them between functional requirements and non-functional. Functional is pretty much what the application has to be able to do for the users. And in this case, we want users to basically type something in the search, get some results, we're going to have those results, and we have a pagination over here. That's pretty much it. And so, in terms of the user stories that you want to extract at this stage, I wrote a couple of them down here. So, job seekers should be able to search jobs by text and location. They should be able to view a job and similar jobs, and they should be able to apply to a job. Now, as you see, this the scope here is pretty big. So, in general, in an interview, you want to make the scope as small as possible, and you want to ideally negotiate and agree with your interviewer on one core feature. In my case, it would be the search. I also added an optional feature, which is employees should be able to post and update a job, but this is more in the case there's more enough time, and we can talk about that. And everything else it's out of scope. So, looking at the design, you know, how we'll cut out all this, it's out of scope, notifications, messages, my profile, authentication, all this is kind of out of scope, and we'll focus on the main thing, which is over here. Being able to return results for the query, and being able to render them. Now, the non-functional requirements are more about the how. So, how fast should we load? How much traffic should we support? And breaking down the traffic is there a peak in the traffic? Where are the users based? And all this would allow you to kind of understand how scalable the system should be. Now, in general, you just want to mention those. Don't spend too much time breaking those down, because maybe you get tangled up in high school math, and you end up spending 30 minutes breaking those numbers down, and by the end of the interview, you have nothing on the screen. So, in general, I just want to ask, okay, how many users we have? It's around 10 million a day. When do they log in? In our case, it's on weekdays between 7:00 to 10:00 p.m. And we multiply our traffic by six in those peak. And in our case, we are going to consider that they are distributed across the East Coast and West Coast in the United States. Now, one of the most important requirement you want to mention is for each single feature you're building, which one is more important? Is it consistency, or is it availability? So, consistency and availability will dictate how we scale our service, and which kind of trade-offs we are willing to accept. And you want to do this on a per feature basis. Just as a general idea, if you're building payments, you want them to be strongly consistent. And if we are building pretty much anything else, availability is usually the priority.

So, in the case of our job posting website, if somebody posted a job and it takes 2 minutes to appear in the search, is not a crime. That's something we can live with in order to have a very scalable search. So, that's the kind of trade-off we'll make later on. Again, right now you just want to mention those and it should take you around 5 minutes and you should be able to move on. You don't want to spend half an hour here.

The next thing you want to do based on those requirements is a back-of-the-envelope estimate. Now, again, you can do this later on as you start scaling your system, but in general you want to break down how many searches you get per day. Okay, what does that means per second? How many people view a job per day? We got 120 million and that's around 1,400 per second. How many people post a job? It's 330k per day on average. So, on average it's four per second. And our read-to-write ratio, for example, for the jobs view, it's 500 to 1. And that's a really important number we'll come back later. But again, you can do this for single endpoint or single feature and then tell your interviewer that for the rest of the numbers, you'll break them down as you get there. Now, once we've got our non-functional requirements and functional requirements, we can move on and do a bit of data modeling and API design just to understand the shape of our data.

Now, again, just like in section before, you want to do this in under 5 minutes. So, we will explain them at a high level. Don't go too deep. Don't go into the actual sequel if you're writing sequel, because what will happen is you will not have enough time to scale the system. So, in terms of data modeling, you want to look at your main entities. And it doesn't matter if you worked a lot on the front end or back end, you want to be able to break this down into something like TypeScript types, for example. Basically, entities. Of course, uh will look different if they are persisted in the database in SQL language or if they are handled in the backend, but in general you want to extract all the different fields that you see there here and understand what are the relationship between the different entities. So, in this case, we definitely have a job posting that has a title. It definitely has like some description and I see there's different types, you know, that a job could be on site, full-time. And then there's a company that posted it, so that's a different entity and it has a logo, so I'll definitely need some image storage to store that. Um and this I'll take out of the scope, okay? This is looks like some sort of micro fronted that gets rendered in between, but it's definitely not related to the job itself. And so, breaking that down into some entities, uh I kind of went way above and beyond here. Usually just keep it to one. For example, we have a job posting, this one over here, and then a company entity.

And uh of course, the uh you know, there's a relationship between the company ID and the job and you want to be able to talk about one-to-many, many-to-many, or one-to-one relationships. Uh pretty much the rest of the fields are based on what we just saw. We have a title, that's text, description, location, workplace, and a job type. In this case, uh I went for text, but normally job type should be something like an enum that you could filter by. So, that's something to take into account. We have a posted at which should be uh timestamp, probably normalized into UTC. And then the salary mean and max. The one thing I want to raise out is that whenever I use IDs, I use UUIDs just because they are safer.

And of course, each one of those entities has a primary key, which is unique and it has a domain ID of that table, and then it might have foreign keys into the respective sub tables. Again, you don't want to go too deep. Uh you want to mention things like, you don't have any data redundancy, that's why we took companies outside the jobs. So, whenever I duplicate the job, uh I don't need to duplicate the company.

That will change as we scale our search. And you want to maintain referential integrity. So, that's why you have all those foreign keys. Again, this doesn't have to be perfect. Take this as a first pick and tell your interviewer, "Hey, this is just a first version. And of course, as I scale the system and I know more about the system, we'll come back to this view." The same goes about your API design. And in this case, I went for a REST API because of simplicity. If they ask you why you guys do this versus GraphQL versus TRPC, you just say, "Hey, you know what? I'm going to assume uh REST is usually enough for the first version of the system. But we might integrate this GraphQL later on for specific services or even remote procedure calls." In general, REST is usually the easiest one to talk about.

And also, most interviewers will want you to be able to write down one REST endpoint fully, which is what I did here. So, basically, in order to get different jobs and apply search, we make a get to the jobs endpoint. Now, a lot of you might be tempted to do {slash} search or {slash} jobs {slash} search. But remember that adding the verb of the actions you're trying to do, like search, into the URL it's not RESTful.

So, in REST, the URL it's only the plural noun of the entity that we're querying. In this case, it's the jobs. And so, to get a list of jobs, we make a get. And here we add all our filters and sometimes also the parameters we want to sort by and also our pagination. Now, in this case, I went with a offset pagination, where basically we show, you know, "Give me page five and you know, the next 10." A typical system design interview question is, "When would you use offset pagination versus cursor?"

And in general, as very often answer, "Cursor is really good when we have a feed and you're scrolling down, whereas pagination is better when you want to really navigate to a catalog." They both have trade-offs. And if you want me to make a more in-depth video about it, let me know in the comments. Just type API design. And if it gets enough upvotes, I'll dive deep into it. Now, to get the specific job, we usually make a get to jobs and we add the ID as a second URL parameter. And if you want to get similar jobs in that case, we can add them also at that particular level. Are there my guys give a status code? How would you answer? Normally, it's a 200 if it was success. And everything that starts with four, there was a client error. Everything that starts with five, it's a server error. In order to create jobs, you'll need a post. And to edit jobs, we use a put just because it's a bit more scalable than a patch. A typical question might be here, what is put versus patch. Uh but again, you want to spend around 5 minutes defining the first endpoints, and you'll have to write these down, so it's going to take you a bit longer. So, do make sure you manage your time. And with this guy definition, we are ready to move on to the high-level design. Now, high-level design would be where we go from all those requirements to the actual final system. But, I will not start drawing all these things at once. I'll actually start one by one and kind of building like an, you know, like the different layers of an onion. We'll add abstraction on top of an abstraction, a system on top of system, and this will be the final result. But, what's important is that you're able to start very simple and be able to defend any choice that you make at this point. And so, one thing you want to avoid is to draw things here that you cannot defend and just go in and be like, "Well, I'm just going to add a bunch of API gateways here, and it's going to be two of them." And then you move on to the next component, and what the interviewer would do would be like, "Okay, why why did we add this? What was the assumption? What was the problem you're trying to solve?" So, do not introduce solutions before you define the problem.

Now, your first system is based on your functional requirements, which are basically your user stories. And for this specific system, we literally have our clients that are users on a browser. They hit a server, they got back over your HTML or SPA. And uh the server would be the one that requests everything in a database. I chose Postgres as a database because it supports most queries that we really need. It's widely supported, and because I'm very familiar with it. And this is something you can let your interviewer know. It's like, "Hey, I chose Postgres because I know Postgres and I work before with it." Of course, for this specific use case, you could also use and add some acid properties to it for the persistence of Java applications, and that would work, too.

But, it's okay. Don't spend too much time debating which one is it. In general, for 99% of business applications, Postgres Postgres will be just fine. And the big difference between Postgres or it's actually the difference between having base transaction or acid transactions. Now, uh to summarize it really quickly, when you have acid transactions, it's easier to guarantee the integrity of the data, and it's easier to make predictable changes, but it's not as scalable or available as a base database, b a s. If you wanted to break this down in a video about databases, which is like the API in this one, tell me and I'll add it to the backlog. We also added a simple email service to send some emails when people apply to jobs, and a blob storage to store logos and images of the company, and maybe also resumes if people start applying. You usually want to separate that from the server, because not your server will not run out of memory. Now, our next step would be taking this simple system and start scaling it and adding more complexity as we start fulfilling those non-functional requirements that we talked about. Now, sometimes some interviewer will ask you to describe exactly what happens with a request. And for this simple system, in terms of a sequence diagram, I really drafted it so you get an idea of how to look like, but basically we have a client, it will make a get request to our API that we defined before, that will hit the server, and the server will make a select from the table jobs where the title might include software engineer. This is the most basic level of search that you can provide. Again, I left out the sorting and ranking and all that. Um results will be uh returned, usually in following the offset and pagination, and then we get a 200 back with, uh, you know, the array of different job postings as, you know, JSON objects. Um, so, basically, the reason why this is important is that you want to make sure that you know things also a bit more in depth, not just at the surface level, like, "Okay, there's a load balancer calls this." Um, because nowadays with AI, everybody going to give you like a high level of what the system is, but the difference between me, senior, and staff members is the depth. And so, seniors and staff, they can go very in-depth. They can design URLs, they can go into writing even a pseudo code for some of the endpoints.

Whereas, mid-levels or, uh, even juniors, they, uh, like, "Oh, load balancer." But, they only know it at a very superficial level. Okay, now, looking at our system at our initial requirements, at the peak level, we had around 12,000 reads to the job endpoint. So, basically, this will create a lot of pressure. You have all these people connecting to the server, those requests have to be processed and sent to the database. And the way I usually have problems is following traffic. So, we're going to this direction. So, the first scalability bottleneck I'll look at is the server. The first optimization you can make to scale your server is to basically load balancing it. And so, the reason we do this is because, of course, when you get more traffic, you could buy a bigger server, but then traffic will decrease, as we mentioned, there's a lot of peaks, and you want to be able to react to those changes. And reacting to those is almost impossible if every time traffic changes, you buy a smaller or a bigger server. Plus, you'll have downtime. So, it's much easier to distribute the load between cheaper, smaller servers. And for that, we need a load balancer component. Now, in order to be able to load balance and horizontally scale our system, you got to make sure that all those instances are stateless. What does stateless really means in this context? Well, basically, if I'm a user, user number one here, and I make my first request, right? And the load we use a load balancing algorithm like round robin, and it will assign my request to this instance, right? And so, that request gets sent to the database, and then it comes back, and then I make a second request. And it gets redirected to another instance, let's say instance two, goes to the database, and answers.

As you see, those two requests, the answers that I get cannot be dependent on the instance they get redirected to. The only state that I need only data that to be included in the logic of my request should be in the database, which is by definition shared between those instances. So, your servers have to be stateless, and if there was any state in there, you got to move that to the database. That's very important to take into account that scalability introduces these kind of constraints.

There's a lot more constraints in the way you deploy your application. You got to be able to spin up those servers really quickly, maybe add them and register them with the load balancers when you're on the scale, but I'll leave that out of scope. Usually, that's taken care in our cloud platform if you're using something like Kubernetes or EC2 on AWS. They take care of that. And that would be our first optimization, and that's scaling the server and doing a bit of load balancing. Now, no matter how well we load balance our application, we will still have a problem with the database, which is that, as I mentioned in the beginning, we usually create around 300 K new job postings a day. And so, they will end up accumulating, and that select statement to find a specific query, like, let's say you look for your developers, will start becoming very slow. So, as I said, as our database gets bigger and bigger, and we have more and more records, it's just harder to search. It takes a lot more time to go over 10 million rows than it used to take over 10 K rows.

Now, this is a read-only problem. Whenever a user hits the search endpoints to get a couple of jobs, they will run into it. And if we are talking about 10 million rows, even the best, most optimized Postgres server will need around 2.1 seconds just to answer to the query. If you add DNS lookups and all the other latencies, you'll end up with a very slow application. Now, the easiest way to scale this, because it's a read-only and it's a database problem, is by adding a database index. And at a high level, what the database index does is that it creates a parallel data structure for a specific table. So, we normally have our table with all these other records, right? And we follow a linear search. We go from top to bottom trying to find the ones that satisfy our query. Well, what happens when we create an index on, for example, the title is that we go from this linear search to parsing actually a tree, which is a lot faster to parse and find something.

Like, let's say we want to find something in this node, then having to go through each single record every time. So, we basically avoid a full table read. Now, of course, there's a lot more to database indexing and it's a whole topic by itself, but for the purpose of the system design interview, you really want to just kind of mention it. Just keep in mind that because we have this new data structure that we need to maintain every time we write to the database, number one, we'll need more space on our disk for our database, but also it'll take a bit longer to write records. Now, that's a trade-off we decided to live with because, as we mentioned in this design, we'll favor availability over consistency. And just sum it up, the difference between a full table scan and an index, it's huge the more you scale. So, it basically goes exponentially. That means over a 10 million records, a full scan will take around 2.1 million seconds, whereas if you use an index, it'll take 15 milliseconds. So, you can imagine that at scale, these numbers are huge because the relationship is totally different.

Here, the complexity grows linearly, and here it's basically a logarithm of n, which is a lot smaller. It's exponentially smaller. Now, going back to our current system, we have a load balancer, we have a indexed database. So, things have probably improved, but we are still not at the level where we can support the 10 million users a week that we mentioned in the beginning. Now, common problem that usually happens is that you might have news or certain events that trigger a specific query.

So, all of the sudden, everybody is looking for a specific job in a specific geography. Uh for example, lately the forward deploy engineer, it's pretty much everywhere. So, there's been a lot of news and YouTube video that says, "Well, you have to become a forward deploy engineer. Everybody's hiring forward deployed engineers." So, that would basically cause a lot of people to look for this specific search, and it's usually concentrated. So, most of them will probably be based in San Francisco or New York or most of the, you know, software capitals in the United States.

And so, all of the sudden, for that specific query, we will receive a lot of traffic. And there's a very smart way to solve this problem and protect ourselves from these kind of peaks in read traffic without putting too much pressure on our system by introducing a caching. So, basically, what we can do is that whenever the first person searches for that specific search, which is, for example, software engineer in Boston or forward deploy engineer in San Francisco, we store the results of that in a Redis cache. And if the next person, you know, 10 seconds later, they hit the same search, rather than going to the database and putting a lot of pressure on the database, we have Redis handle it. This is again another layer of scalability. And the biggest, most important question here is, "Okay, but then when do you clean the cache?" The easiest way here is just to set a fixed time to live. And let's say we go for 60 seconds. That means every search will live in Redis for 60 seconds. So, if user A goes in, you know, and there's nothing in the cache, well, they go to the database. They get the result, they fill up the cash, and they go back. That would be a cash miss. But then, user number B, you know, they go to another instance, but they uh the instance where they read the cash will be like, "Okay, we already have the results." And return it to us for the next 60 seconds. I mean, that happens in that interval.

After that, it gets cleared. Why do we do this? Well, this is a pretty robust, but it's extremely easy to do cache invalidation, which tends which tends to be pretty hard if we want to be more specific. And it's protecting us from any spike in the read traffic. Our current system it's growing. We have a Redis database. We have Postgres that's index, and it's pretty fast. And we were basically able to also make our server pretty fast by adding a load balancer.

However, there's one problem we did not tackle, which is latency. So, what's latency? Latency is a problem whenever your user is very far away from your servers. So, in this case, let's say we deploy it to Virginia on the East Coast, and we are receiving some traffic from the West Coast. The problem is that the first request between these two it's at the very least 220 milliseconds. Why? Because a round trip is usually on 70 milliseconds at the speed of light, and we need to do three to five only to establish the connection. Of course, the second and third request are faster because we use the same connection, but in general, this is quite big. And when you're doing things at scale and you're going to have plenty of users, I can promise you it's going to be a lot slower than that. The other problem is that there is no optimization we can do to our system that can actually decrease this distance because this is just the limitations of physics. There is a lot of distance between these two points.

The fastest we can travel is at the speed of light, and even that is not enough when we talk about such a big distance. And so, the easiest way to solve that is by distributing our server and database. So, basically, we could actually have a replica of our server and a replica of our database, a read-only replica, closer to our users in specific locations. So, we have one in Oregon, we can have one in you know, somewhere around Texas, where we redirect users based on the distance to the closest server. And our primary database still stays around Virginia.

So, that would be our primary cluster. Now, do you remember that in order to maintain integrity, our replicas are read-only. So, basically, we always write to the primary database. Our writes will still be slow, but our reads got a lot faster. And so, this is how our system looks like right now, where we'll have a primary database where we can write, and then we have these replicas that basically get synchronized. Now, there is a bit of a lag between those two. And the lag is around 200 milliseconds. And this is the first time, when we go back to our discussion about consistency and availability. So, as you distribute our database into these read-only replicas, we're moving from strong consistency to high availability, but only eventual consistency. What do I mean? Well, let's look at the sequence diagram of this request. So, we might have an employer, they decided to update a specific job by changing the max salary. So, they make a pull request to that initial API we defined. That hits the server that's closer to them, and then triggers an update operation on the database. Now, updates are only done on the primary database, and once the update is successful, they will get a 200 okay, like, hey, we finished that update. But in the background, the replicas are still being updated. I'll let us assume a 200 millisecond lag between those two.

Well, what happens is that for these 200 milliseconds, if somebody would request a copy of the job from the replica, they still get the old version. So, for a small amount of time, the 200 milliseconds, we can see that our system is inconsistent. The data is inconsistent because I made a change, but somebody else doesn't see that change. Now, what will happen after the 200 milliseconds is that eventually the data will be synchronized, and then we are consistent. And this is why in the consistency patterns, this would be eventual consistency. And again, for most use cases, and specifically for our use case, this is good enough. Now, if you're working on a financial system like payment provider, usually you cannot even live with this. Because if you're building a stock exchange, for example, if prices don't update fast enough, you might end up doing some very bad actions based on a lower price that's not actually real, and it can be a total disaster. So, whenever you talk about financial systems, usually they will require stronger consistency, which comes at the price of availability. Why?

Because if we wanted stronger consistency here, we could also just signal that the table was updated after all the updates were finished here, but we would have a bigger gap. So, the data would not be available, right? You might just have to wait a bit more. And that's why we say that you can always choose between more availability on one side and consistency on the other side. Now, the same latency problems applies to our static assets, too. And in this case, I just want to touch upon a CDN.

Basically, a CDN is a content distribution network that will run different edge nodes, as we call them, very similar to the back-end and database replicas we had. And what happens is whenever we upload a static assets to our blob storage, to our cloud folder, which is our S3 database, it will get pushed to all these different locations. So, whenever a user loads a job posting, and they need to load the logo of our application, they will get it from the closest location to them.

And the cool thing about static content is that consistency is usually not as important, and but availability is. So, if a logo is basically uploaded 100 milliseconds later, that's okay as long as all the users get it much faster. So, that's when we basically introduce a CDN. And uh really important here is that you can use the CDN to also distribute your static and pre-generated HTML, and you can use your CDN to even distribute a copy of your static response. We used to get this before to cache it between the server and the database, but you could also cache it between the client and the server by putting it in a CDN. So, it's not only about images and fonts and CSS files, but you can also use your CDN to cache actual responses, and we use those. Now, with all those moving pieces, this is how our system looks right now. We have our users that could be job seekers or employers. They usually make requests.

The first request, if applicable, it's served from the CDN, which usually gets synchronized from blob storage. After that, it hits our load balancer, which will distribute the load between this fleet of six server instances. If they take a query to the database and it was already cached on Redis copy, then it will go to Redis. If not, it hits the primary database or the read-only replicas if it's a read-only.

And if it's a write, it will be done in the primary database. Of course, all this is also distributed geographically, which increases complexity, but overall, this kind of system is already pretty good. Like, you'll probably go very far with this kind of setup. And that's why it's really important for you to make sure that you really understand the essentials, right? Get the basics covered. Because if you get this far, you probably pass like a mid, almost senior interview. They might ask you a couple of things about going in depth maybe in the schema of the database or how do you set up Redis. But for the most part, in the 45 minutes, you cannot go much, much deeper. Now, if you're going for a very senior position or you're interviewing the company where there's a lot of competition, then yes, you will have to go much, much deeper.

Or if you're going for a principal staff level, then yes, uh this kind of complexity you would tag around mid maybe early senior. But again, it's good enough to pass the interview. And this is when we will start with our deep dives. And the first deep dive I want to take into is basically microservices. Now so far we basically treated our server as a monolith. But the endpoints that we have which is basically creating job applications, the login for employers, people looking at job applications, they have different requirements. And that's the point where in order to scale even further, it's about time to break this monolith into independent services. And so looking at our backend monolith, we usually draw some service boundaries. I feel like, "Okay, which logic could be actually separated?" And the way you draw out these boundaries is by looking at asymmetries. So you look at the part of the system that takes the most traffic.

You're like, "Well, if we take that one out, we could scale that one independently and leave these other ones on cheaper services without so much complexity. So remember that complexity it's always a price you pay. Uh the difference between like seniors and juniors is that junior mid-level they will talk about the fancy stuff. They'll be like, "Oh, API gateways. Oh, microservices." But the senior will be like, "Okay, we we really want to keep this as simple as robust as cheap as possible uh because we never know what happens in the future and how we like to change it. And the more moving pieces we have, the harder it will be to change the system later on." And so right now I made a very quick split and I said, "Well, there's going to be a service for the job search and a different service to actually persist the jobs, right?

Because job search will mostly rely on those read-only replicas, but the jobs will be the actually the one that hits the primary database. So this also simplifies things in terms of permissions. And we might also have a different service for job applications. That's where users create a job application and apply to a job. This again is a very rough separation. And based on this I split it into three different microservices. We have a job search service that again has this read-only copy of Redis and it's looking at the read only replicas of the database. We have the job service. This is job application. That's where employers actually create those job applications. They use a primary database and that gets synchronized with the read only copies that the search service looks at. And finally, we have an application service that also writes into primary. So, make no mistake, this database is still shared. You know, a lot of you will tell me, "Hey Bogdan, but when you build microservices, they they should be totally independent.

Like, they should have a separate database, too." But, in the first phase where you're still separating your services, it's okay to still rely on this primary database until you actually are able to fully migrate and split it. The advantage of having a shared database is that there's zero latency when you know, the job service makes a change, the application service has it. But, the disadvantage is that maybe those two teams are very independent and they end up breaking each other's database. You might lose data. They may want to scale things differently. So, it introduces a lot of communication overhead by the fact that they actually share this piece of infrastructure. They need to talk to each other. Now, as an entry point to our microservice architecture, we added an API gateway. As you see, there is two of them right now. We split the traffic between the two. And the reason we chose to have two, even if it's more expensive, is because if any of them would fail, we can rely on the other one. If we just leave one of them, we'd have a single point of failure. So again, they both receive traffic and the traffic is split between them and if one of them goes down, the other one fully takes over the load. So, the API gateway will look at the URL the user will request and redirect them to the specific microservice. And this is how our initial split to microservices looks like. Now, in general, when you're drawing this yourselves on the board, you're not going to go that far. So, if your drawings don't look as fancy, just don't panic. Now, one of the things we left out at the very beginning was the email service. So, basically we mentioned that there must be emailing going back and forth whenever you apply to a job, but it turns out you also want to send emails to employers when they modify job, and also the job search service has a recommendation service that will send you similar jobs to the ones you searched for a week later or a month later. And so, this is one of those deep dives where they might ask you, "How would you optimize these kind of tasks?" And the first red flag with the current setup is that if this service, if the simple email service, is down or if we have a speak in traffic, let's say a lot of people apply to plenty of jobs and we have to send many, many emails, we end up losing a lot of those emails if the service is down. Uh the only way not to lose those emails is to have specific application code where you're like, "Okay, I'm going to try to send an email and then store it if it wasn't sent and have a database specific to it." And that's a very robust way, uh but you also need a cron job to then pick up the ones that were not sent, send them again, and retry, and you basically have to reinvent the wheel.

And so, the first question uh they would ask, especially going for a senior or principal engineer, would be, "What kind of change would make to this kind of architecture to make this kind of asynchronous tasks much better and to be able to scale them without having a lot of application code and a lot of debugging and testing effort." Because again, all the application code you add here will be debugged if it doesn't work or you need tests on it. And in general terms, the fix is to move all your synchronous work into event-driven. That means those services, right, they will answer to synchronous requests directly to the REST APIs, but whenever they're done with something, they will send an event. And in this case, we will use Kafka to manage our events. Now, what do I mean? Well, basically let's imagine in the application service somebody applies to a job. And we store that application in the database. We're like, "Okay, this candidate applied to this job, so we store it in our tables, but we will also send an event to our Kafka cluster. And so basically, we send an event that says this user applied to this job. Now, to our event hub, like in this case, it's Kafka, which I'll go into in a minute why I chose it, there will be services that just listen to the events that happens, and they can react to it independently. And that's the best thing about event giver, which is you go into a mentality which is more related to choreography, where you don't need to explicitly tell every service what they need to do. They listen to the events and they react. They don't see anything else. So, for example, the analytics service doesn't need to be coupled now with the application service to know how many people applied to a specific job or how many people applied to jobs this week versus the next week. Or you might have a data science team that just extract their own data over here from the event stream, and they analyze it with DynamoDB, which would be a no sequel database, because they don't need so many relationships. Uh the same goes for notifications. So, our notifications uh synchronous workers could be listening to the event stream, and be like, "Okay, user applied, and I'm going to send a message to that user." Now, why do we use Kafka? Well, the thing with Kafka is that uh at a high level, it will store all the events. So, we can read from the beginning of history. Like, if we connect a new service, let's say we want to connect an extra service that does some sort of machine learning on it, you could start at the beginning of history and read all the events from the very first time we deployed our Kafka set.

So, events are basically stored. It's almost like I mean, I'm not I don't want to say that Kafka is a database, but it's definitely a data storage. So, you can read them all up and apply your analytics from it. That's the first advantage. The second advantage is that to a specific event stream, let's say um user applied, you can have several instances of the notification service listening to that stream, and Kafka has an internal way to know that we only process one event once. So, imagine we have these two workers and an event comes to us like, okay, somebody applied to a job. If you have two workers, they might both pick up that email and end up sending two emails. They're like, okay, that didn't happen and they both sent the email. But, if you use Kafka, the moment one of them reads it from the topic, because they identify as the same consumer, that event is gone. So, Kafka will allow you to really scale this in a very clean way. And uh it also has incredible durability and scalability.

But again, this is not an interview about Kafka, but that's something you can be posted. If you're not familiar with Kafka, don't worry. And if you want me to make a video about specifically event-driven architectures, I can also make that later on. You could do this with a simple queue, but you will have again to write a lot more application code to store your events and have that the exact same reliability. Now, last but not least, let's talk about search.

So, as of now, our search service will basically have its own load balancer. So, we hit their API, they look at Redis, and then finally they look at the primary database, which is somehow here, mostly owned by the job service. Now, the problem right now is that in terms of sophisticated fast search, things like ranking job offers or having a smarter search are quite hard using only Postgres. The other problem is that again, there's a lot of pressure and a certain latency, because as I mentioned, most of the reads we do from this replicas. So, if you really want to scale your search, if you have millions and millions of job postings, we need something more specialized in this specific problem. And in general, it's a good idea to introduce Elasticsearch.

Now, Elasticsearch, you can think of it almost as a read-only database that is basically based on our primary database that has much higher search capability. So, Elasticsearch would support different kinds of paginations and offsets, and it's also a lot more scalable. It introduces database sharding internally. So, the performance that we get from something like Elasticsearch, it's much, much higher than if we only use Postgres. Now, you never want to use Elasticsearch as your main database. Use it as a read-only copy, and you only need also a service to synchronize between your primary database and the Elasticsearch cluster whenever an update happens. So, usually, we push those updates to Kafka, which is again extremely good for anybody that wants to listen in, like analytics with notifications, but also there's these serverless indexes. Those are very small functions that will copy that event and persist to Elasticsearch. Elasticsearch has a REST API that's extremely performant, and we'll get results very fast, but because of this lag, remember we got to go from the primary database, persist that through uh running instantiating these serverless indexes, and then finally, it's in Elasticsearch, there will be a bit of a lag. So, as I mentioned at the beginning of this uh video, it will take a bit of time between when you publish a job and somebody sees it in search. Now, the cool thing about Elasticsearch is that it supports very good ranking algorithms, so you are able to rank all those job positions depending on what the user searched for. They have an inverted frequency index, so if somebody's looking for a React JS developer, it will actually rank the jobs that have a React versus jobs that have developer, because if you look for React JS developer, then maybe a lot of jobs that match developer, but it could be a real estate developer. So, they actually have a much better out-of-the-box ranking algorithm in search, and it's super performant. So, instead of reinventing the wheel or writing procedures here to make this fast and still deal with all this complexity, it's much easier to have a single version of truth as a primary database, a couple of read-only replicas for the jobs microservice itself, but when it comes to the search, that's very heavy and it most of the traffic, you can actually add this extra layer of hyper scalable elastic search. Now, going back to consistency, do you remember that we are in now, if we update one of the job postings, we'll have to call our server that will update the primary database, send that to Kafka, and then finally an indexer will listen to that, and it will hit it into elastic search, and when this is done, we actually persisted the job. It's actually distributed, right? So, we finally have that change. So, the lag, it's much bigger. The time lag it's now this big, whereas back in the days it used to be this big. So, there's definitely a price in terms of data being available fast, but we get a lot more in terms of scalability. So, it's pretty clear that right now our consistency model for the job search it's eventual consistency. We will be eventually consistent, but as we add more and more pieces to make it more scalable, it will take a bit more lag to actually be consistent. And the final optimization I want to talk about it's introducing a back end for front end.

So, we pretty much ignore the front end in this interview. There's a lot of interesting things that happen in there, like server-side rendering, but right now, from the client side, we need to request all these different microservices, and that's pretty tough because the front end developers themselves will need to coordinate to all these different teams whenever they need to change a feature. They're like, "Okay, can you add this extra field in the jobs microservice, and in the job search I need this other one?" And there's a lot of communication overhead because these things are kind of tangled up. And so, the best way to give those front end teams more autonomy and go even faster, but also be able to scale a lot more, is by adding back end for front end. So, those back end for front ends are specialized applications. I will see over here I chose GraphQL as an API just because it's easier for the front end. And so, this web back end for front end is developed by the web front end team, and it communicates directly with the next JS, let's say with the front end part of that application. So, basically the front end team owns all these. And the advantage here is that whenever they build a feature, they can act as full stack developers and go really fast. Now, when they need a bigger feature and any bigger changes, they need to talk to the back end teams.

But, as of now in this picture, everybody can work pretty much independently. The back end developers from the job search can just expose their rest API and so on. You know, the know, the job microservice, and this is all aggregated into a single unique typed interface specific for the specific client under GraphQL. So, we have a web BFF, and we might have a mobile BFF because data in mobile looks totally different, the kind of data we want to request than the one in the web part. And so, if we don't do this, we'll have a lot of problems like underfetching and overfetching, we'd have latency in our client. So, this is all to help the client offload a lot of the logic that you normally have in the front end to the BFF and simplify also front end state management and reduce the number of requests and data usually have to request if you don't have a BFF.

So, this is how our final architecture looks like. And just to give you, we have our clients, they hit some requests. Of course, that will go through some DNS. A lot of the static assets are served directly from very close to them from CDN edges that solve the static asset problem. We have an API gateway that has obviously two copies in standby, and that gets forwarded to our actual application. I chose the React and server side rendering. I didn't have enough time to actually talk about why, but the main reason is because most of those job websites have SEO as a non-functional requirement that you might have to justify. And so, it makes sense if you want to get indexed by Google on organic or specific searches to use server side rendering. And this also implies that we can cache a lot of the jobs in the CDN as I mentioned in the beginning.

Behind it there's the back end for front ends. We could have as many as you want but of course those are expensive pieces of infrastructure. They do have their own load balancer and they usually run GraphQL because it's easier for the client to consume a GraphQL database and they aggregate behind them a lot of other back end microservices which at their time they orchestrate themselves with an event hub that they can also use to delegate all the asynchronous work.

Now imagine to this architecture one of the features we want to add is being able to apply to the job with a cover letter. Whenever you upload a PDF and we need to parse that when we need to send it over it's extremely easy to do so by taking an event in Kafka adding the URL of that in where we told I would probably interest me and then have this all these different asynchronous workers doing pricing and processing that PDF without interrupting the synchronous flow. So asynchronous and event driven architectures are really good for any kind of delaying the work. You really want to do only the work that's essential at the very beginning. And this is basically our final architecture. As a final piece of advice there's two anti patterns that people fall into here which is number one just putting things on the board that they don't really know. So the interviewer will end up questioning you and you won't be able to defend that. Or the other one is basically running out of things to say. So you do want to make sure that you don't fall in any of those. And to do so it's best to really make sure that your fundamentals are very strong when it comes to system design. So you can put together a coherent system that can cover 80% of the use cases. So do let me know in the comments what do you think and if you want me to make a specific video about any other specific system that you probably got in the interview also let me know in the comments. Me and Dragos do read all of them and with that being said I'll see you in the next one.