Or: How We Learned to Stop Worrying and Love the Engineering Interview Process

Skroutz is hiring! We write this phrase on our social media and blog posts, and we discuss it internally within our hiring teams. We are growing rapidly and hiring people is one of our top priorities.

Even when the pandemic hit and we started working from home, not only did we not stop our hiring efforts, but we doubled down on them and quickly adapted our interview process to the new realities. To put it into perspective, of the 29 engineers we have hired in 2020, 25 were hired after March! In this post we will discuss how we think about interviewing engineers, how our approach has evolved over time, and how with a few tweaks the same approach worked well when it became fully remote.

When this big hiring initiative started, our engineering hiring process was in need of rethinking. It wasn’t really broken - we have hired many good people - but with more colleagues joining our hiring team and more positions open than ever, it had started to show its age. So we wrote down the things that concerned us about it and spent a couple of weeks thinking, discussing, and reading relevant articles and book chapters. In the end, we came up with a process that, while not radically different, seemed to iron a few kinks out.

Of course, a hiring process that looks good on paper might not withstand colliding with the real world. So we decided to try this process out on one of our openings first and then share the results. Meanwhile, other divisions within the company are experimenting with different variations of the process, which means we get to meet and exchange experiences after!

Before moving forward, we must note that hiring is an inherently flawed process. Judging individuals for their skills is messy, hard, and at times unfair. Candidates are called to compete in something that barely resembles their everyday work. In practice, the whole hiring process boils down to minimizing “false negatives” in the early stages - candidates that should have moved forward but didn’t - and “false positives” in the hiring decision - candidates that were hired but weren’t a good fit after all. This is tricky and you’re bound to make mistakes. Andy Grove wrote that “careful interviewing doesn’t guarantee you anything, it merely increases your odds of getting lucky”. In this post we share our experience and our current understanding which might change in the future, so please take everything we say with a grain of salt.

Search Team Job Opening

The search team was looking to hire two engineers, and we chose that job opening to try our process. We received the first resume on 7 February and the last (we removed the job listing) on 12 March. 83 candidates applied in total, and we eventually hired 3 of them - two joined the search team and one joined the content engineering team. Overall we were quite happy with the way it turned out: we are confident that we made the right decisions and are excited to start working together with our new colleagues.

What didn’t go that well was our response times: on average we needed 10 days from the day we received a resume to the day we conducted the first screening with a candidate. The average time from resume to job offer was 44 days. We can attribute this to three main reasons: the first is the fact that we had many open positions simultaneously and our HR department was, at the time, understaffed for the candidate load. The second reason is the fact that only four people were involved in the interviews, which put a cap on the total interviews we could arrange per day. The third reason is of course the elephant in the room: the COVID-19 pandemic and the resulting lockdown which made us switch the whole recruitment process online.

Interview Process Revamped

Our old process consisted of three parts: first filter through the incoming resumes. Then, do a screening call with those that we think might fit the role. Finally, do an onsite interview with the most promising candidates. The onsite interview consisted of two parts: a coding exercise and some database-related questions.

Before the lockdown happened, we were thinking of adapting that process in order to address a few issues we had identified. First, the screening call would be more structured, with specific things to check for.

Furthermore, we decided to introduce a second screening call that would include a simple coding exercise. The reasoning behind this was that onsite interviews are very “expensive” both for the candidate that would have to come over to our offices and spend a few hours there, as well as for the interviewers who would end up spending a large portion of their day preparing for and conducting the interview. It made sense then to only call the most promising candidates for onsite interviews, and we had identified that the coding exercise would help us do that.

Finally, the onsite interview would be split into three distinct parts: first another coding exercise, a bit harder this time. Then, a question about system design. Finally, a chat around the candidate’s past experience.

Interview Process Turned Remote

…then one day we stopped going to the office altogether!

Fortunately, the process above was easy to adapt for a remote setting. First of all, there would only be one screening call. Onsite didn’t make sense anymore since there was no “site” to go to and the whole process was just a series of video calls on Google Meet. So we decided to split that into two calls: first the coding challenge, and then the system design and past experience discussions.

This is an overview of the full hiring process, assuming the candidate always reaches the next step:

1. Screen resume
2. Do a screening call
3. Do a coding exercise call
4. Do a system design/past experience call
5. Make an offer they can’t refuse

Note that all interviews are conducted by engineers who are members of the recruiting team and there are always at least two interviewers in each call to reduce bias. The whole process is facilitated by the HR department, members of which also join the call on the last step to make a few questions themselves.

Screening Call

The goal of a screening call is to get a first impression of the candidate and try to establish whether they might be a good fit for the position. We spoke with 17 people (20% of the total applicants), and 10 of them went forward to the next step.

We try to keep the length of screening calls at around 45 minutes, and having a predetermined structure helps a lot. First, we ask the candidate to talk a bit about their experience, urging them to be concise. This serves as a nice icebreaker and also becomes the starting point for follow up questions so we can understand a bit more about their background.

We then ask them a bit about their motivation, the reason they applied for the specific position, and also what they want to primarily focus on for the next couple of years. Of course there is no “right” answer here, but this is a nice way to move the conversation forward and learn more about the candidate and their expectations.

Finally, and depending on the position the candidate is interviewing for, we ask a couple “knowledge” questions. These can be of two types, the first is checking whether the candidate knows about something that is considered a requirement for the role. For example, we might ask about Ruby symbols if the position requires Ruby background. The second type is something based on the candidate’s experience. For example, a candidate might talk about (or mention on their resume) their experience with a distributed systems project and we could follow up with asking about consistency or availability issues. This way we can get an impression of the depth of their knowledge, but also of their skills in communicating technical topics.

Coding Exercise

Of the 17 people we did a screening call with, we moved forward with the coding exercise call with 10 (59%).

What’s nice about having the coding exercise as a separate step, is that you can be less strict in the screening call, because they can get pretty messy; there were times that we were left uncertain whether we actually learned something useful about the candidate. In such cases, moving forward with the coding exercise was an easy decision, since it would give an opportunity to the candidate to do well, while being easier for us to judge. An example would be giving a chance to people with little interviewing experience that were visibly stressed during the screening call.

The goal of this step is to determine the problem-solving, coding, and communication skills of the candidate. First, we let them know that while reaching a solution is important, we also care about communicating their thinking out loud. We also note that the quality of the code is important and that we want to simulate a scenario where we work as a team to solve a problem but the candidate takes the lead and we just follow along. Finally, we let the candidate know that there is a time limit of 45 minutes and we actually try to conclude the call within that range.

In practical terms, we use Coderpad for this step. What’s nice about it is it allows us to watch the candidate’s progress in real time, and it offers an environment on which we can actually run and test the code. In preparation for the call we will have created a “pad” with the exercise description, and some boilerplate code in the language the candidate is most experienced with.

As for the exercise itself, we try to pick problems that are not trivial but also that do not require a certain “aha moment” to figure out the solution. Rather, we prefer problems that are amenable to the sort of incremental problem solving that is common in day to day work. During the call, we encourage the candidate to take some time to think about the problem, even use pencil and paper if they want to. We also try to give them hints if they are stuck, and try to steer them towards a solution, sometimes by giving them specific examples to work with.

In order to gain confidence about the exercises we picked, we actually did a couple simulations with Skroutz engineers: we actually logged on Coderpad, gave them the exercise, and watched them trying to solve it within the predetermined 45 minute limit. While of course this isn’t realistic as there is no stress involved, it gave us a nice (albeit optimistic) baseline and reduced the doubt about the quality of the exercise significantly. What’s more, we have decided to adopt this approach before approving any exercise to be used in Skroutz interviews.

Finally, we gave the same exercise to every candidate. While this might sound a bit risky (it might leak, or they might already be familiar with it), in practice it was very helpful in that we were able to judge the candidates compared to each other, rather than in absolute terms. This helped increase our confidence about the people we chose to move to the next step of the process.

System Design & Past Experience

Of the 10 people that did the coding exercise, 5 went forward to the next step. We should note here that all five candidates were good engineers, able to communicate their thinking, and we would probably be happy working with any of them.

The final session has two parts. The first and most technical is a 45 minute discussion around a system design topic. The second is a 30 minute discussion based on the candidate’s past experience.

In the system design, we ask the candidate to assume that we are an engineering team that gets assigned to create a new system/service/website, etc. We want them to take the lead and tell us how they would approach this task. These are intentionally open ended: for example “create a twitter clone”, “design a system that enables users to ‘like’ posts”, and “design a bit.ly clone”, are all potential topics.

Note that there is not a single right answer and that what we are looking for can be adapted based on the candidate’s experience and seniority. For a junior candidate we could focus more on database schema design, API endpoints, and queries. On the other hand, we would expect a more senior candidate to do requirement gathering and trade-off discussion before diving into a design proposal.

What’s nice about this type of question is that we can dive as deep as we want: we can explore potential issues with the proposed systems, e.g. “what happens if there’s a spike in traffic?”, we can ask how the proposed systems could be extended, e.g. “how can we support lists in our twitter clone?”, or even alternative technologies, e.g. “is a relational database ideal for this scenario?” It goes without saying that this type of question requires preparation from the interviewers as well.

We conclude this call with a 30 minute discussion based on the candidate’s past experience. This is mostly informal and includes questions on technical topics, for example we could ask “what is a project you worked on you are especially proud of?”, “what is the weirdest bug you have encountered?”, but also possibly touch on teamwork-related topics, for example “what was for you a good experience of a well-functioning team?”, or “how did you resolve disagreements with your lead?”, etc.

Touching on such topics can let us determine the seniority of a candidate in various areas, not strictly technical. This can play a role in determining the team and the manager we assign them to, should they join us. Moreover, many of the answers we get are truly interesting, informative and sometimes even surprising.

Conclusion and Lessons Learned

We hired 3 of the 5 people that made it to the last interview stage, a 4% of the total applicants. All in all we are quite happy with the process and we felt we learned a lot along the way.

Other teams have started adopting some key parts of this process:

• The coding & design step is permanently split into two calls instead of a very long (~3 hours) one. We believe this helps with scheduling and can reduce fatigue for both the candidates and the interviewers.
• The interviewers are always suitably prepared and are expected to take notes during the interview and submit their feedback on the candidate within a couple of days.
• For each open position we try to determine “knowledge requirements” beforehand. That is, things a candidate must know in order to be considered for the position.
• When researching coding exercises we take care that they are not trivial and do not require a single aha moment to solve.
• We have a common pool of coding exercises, so we can experience how different people try to solve them and judge their performance compared to each other, rather than in absolute terms.
• Before a coding exercise enters the pool, we do a simulation where our colleagues try their hand in solving them!
• Similarly, we have prepared a pool of system design questions.

Of course the process is continuously evolving and we try to get better in time. Asking the candidates for feedback is very helpful in that regard. We want to treat the whole process as we would a product proposal: first develop some assumptions on what we can improve, then try the changes out, and finally adapt the process or the assumptions accordingly, based on the outcome.

We believe that people are what matters most in an organization. A proper hiring process then is critical for growing the organization successfully - maintaining the core values intact and an excellent level of technical aptitude. It also shapes the candidate’s initial impression of the organisation and its people. Thus we believe that we should keep working on it, and that sharing our experience is important.