In Defense of Coding Interviews

What did whiteboards ever do to you?

Welcome from Hacker News! I have 10+ years experience in software engineering at a variety of companies from “Big N" to startups. I write a weekly blog post discussing various topics about working as a software engineer, running a company, and ideas on tech. It’s completely free to subscribe, so please do so if you find this content interesting.

---

The “traditional” coding interview is structured such that a candidate gets a programming question, like “write a function to move all zeros in an array to the left” or “write a function to do level order traversal of a binary tree”. And they write on a whiteboard the code for this function, while the interviewer asks questions to probe why they are doing things a certain way, what the running time is, etc. Of course in our remote-friendly world this might not be a physical whiteboard but some online system like coderpad, but the basic idea remains the same.

This type of interview has often been criticized, and it certainly has many limitations:

1. Doesn’t reflect realistic coding conditions, like having an IDE, Google, StackOverflow, and more than 45 minutes to think about what you are doing.

2. Doesn’t ask realistic problems, but rather data structure/algorithm heavy artificial problems. 

3. Problems often have some kind of “trick” that you either get or you don’t. It’s not a good way to evaluate a continuous variable (developer quality).

4. Easy to “game” by “grinding leetcode” or other practice and questions are often reused or repeated. Performance reflects how much you trained/memorized rather than how good you’ll do your job. 

5. Favors candidates that are earlier in their career and so remember what they learned in college better.

To do things better there have been many alternative interviews proposed. And while I don’t doubt that they can provide additional signal, it’s still important to include some form of the coding interview while hiring. After all, ultimately writing code is why you want to hire a developer, so shouldn’t this be something you actually check for? And while knowledge of various obscure data structures is kind of specialized, surely it is useful to know when to use a list vs. a dictionary (this can have huge real-world performance implications!). With some small modifications traditional coding interviews can be structured to avoid or minimize some of the issues above:

1. Tell the candidate you don’t require perfect syntax or any specific language and don’t require knowledge of any libraries or functions. The point isn’t to test anything that can be found easily on Google. You can also allow candidates to type on a computer, if they prefer. I still don’t like executing the code because I think this can bog people down in syntax.

2. Use questions that can be iteratively improved. This allows you to adapt the question to the candidate, if they are struggling you can leave them satisfied with just the first part, but if they do a great job you can probe them more. This is important for getting an accurate sense of the candidate’s skill. Avoid “all or nothing” questions where you either get the answer or you don’t.

3. Don’t ask “trick” questions, like anything where you have to use an obscure data structure or algorithm in a clever way (or dynamic programming). Instead, prefer to ask questions that use common data structures (array/list, set, map/dictionary) and standard library algorithms (sorting, binary search). Be careful of anything requiring recursion, it’s not super common to use it in practice and it often has performance issues in any case. It’s better if there is an equally acceptable iterative solution (as a result I think questions involving trees are generally bad).

4. Instead of the aforementioned tricks, the challenge in the question should come from:

   1. Modeling the problem correctly

   2. Structuring code in a reasonable way

   3. Handling all the relevant edge cases

5. Avoid any specific domain knowledge unless it’s specific to the role. For example, questions about date/time formatting, unicode, networking, etc. may greatly benefit people that have experience in those specific areas since they are hard to fully explain in an interview.

6. Don’t get too hung up on asymptotic runtime or memory analysis. I would only be concerned about this if the candidate is doing something obviously inefficient, like some loop with a nested linear search. 

7. Don’t treat the candidate adversarially when asking them questions. Instead of saying “you forgot to handle this case” or “your code is wrong”, say something like “What are some test cases you could use to see that it works correctly?”

This requires using different types of questions than the traditional LeetCode ones. It can also be good to pull them from work you are doing, so that they feel a bit more specific to the job at hand. For example:

1. Implement a feature for a search that handles common misspellings via autocorrect. You are given a dictionary of correct spellings, and a list of common misspellings. Your code should preprocess this in some way and then have a function that converts a potentially misspelled query into one that uses only correct words.

2. Implemented a “suggested friends” feature as some social networks have. This should take a user, find all their friend-of-friends, and then return the top 5 by mutual friend count. This can be extended to support other methods of ranking, doing precomputation to speed it up, etc.

3. If you are hiring a React dev, you could ask them to implement a React component to render a To Do list with “add to do” support (or, really, anything with nesting and some state management). (Or similar for other frontend work, just make sure they are familiar with the framework involved).

4. Shorten a long piece of text to create a preview that fits into N characters, ending in “...” and without breaking a word. For example “The quick brown fox jumps over the lazy dog” -> “The quick brown fox…” but not “The quick brown fox ju…”. Then extend this to handle HTML, for example “The <b>quick brown fox jumps</b> over the lazy dog” -> “The <b>quick brown fox</b>…”

These are just examples and I don’t claim that they are the perfect questions. I do think that they are pretty reasonable, and far superior to the common “trick” questions that get used. You can certainly come up with your own questions based on work your company is doing, this will be more interesting to the candidate and make them more involved in your business. Just be careful to calibrate first by testing on some coworkers so you know what kind of performance is reasonable.

Now that we’ve seen how coding interviews can be done better, in the rest of this article I’ll go over some commonly suggested alternatives, and why I think they fall short in fully replacing the traditional coding interview, although they can be valuable to be used in addition to it. Note that I am not going over other types of interviews software engineers face (such as systems design or behavioral) since it doesn’t seem like anyone is proposing to replace those.

Reading Code

One alternative interview type is “reading code”. In this interview you give the candidate some existing code (either artificially created for the interview, or perhaps pulled from some real code). You then ask them to predict the output of the code, i.e. to figure out what it’s doing. Then you show them the real output and if they got it wrong they get a chance to explain why. This starts out with simple functions, and then you expand it to more complex functions with recursion, multiple functions, etc. You could make it as complex as you want depending on how much time you have to spend.

The main reason this approach is suggested is that developers spend a large % of their time reading code (though this probably depends a lot on what you are working on), it provides a higher amount of signal than writing code, and it’s less stressful for the candidate. I think these are a reasonable claims, but nevertheless I think there are some downsides that make this a poor choice to entirely replace a coding interview:

1. Reading code is not writing code. Sure, developers spend a lot of time reading, but hopefully not all of their time. The skills are related, but not the same, and you will be missing out on things like determining if they can convert an abstract description into code and figure out how to structure code properly, which are really important. 

2. Provides an advantage to those more familiar with the language you happen to be using. This may be good if you only want people who are experts in that language, but could also be a downside if you’d rather have a generalist. If you are hiring for multiple languages you need multiple versions.

3. Works better for more experienced developers. Typically in college you don’t spend much time at all reading code, this comes with more experience. Thus this type of interview wouldn’t work as well for new grads as for experienced people, and may just be directly measuring “years of experience”, which you can get for free.

Debugging Code

This is a variant of “Reading Code” in which you are given (perhaps real) code to debug. You are told what it’s supposed to do, and then either just that it contains a bug (for simpler code), or perhaps some sample input that causes wrong output (for more complex code). The motivation for this is that a large % of developer time is spent debugging, and so evaluating that is important.

I don’t doubt that debugging is an important skill, but I think this type of interview has a lot of issues, and it’s arguably worse than the regular “Reading Code” version. All the criticisms above still apply, and additionally there are others:

1. Relies on familiarity of the language/libraries/frameworks. Unless you only want to hire an existing expert in that system you will be unfairly biased against someone that hasn’t worked with whatever you are using for the code. And it’s probably impractical to create ten equal versions in ten different languages. This is a much bigger issue than for “Reading Code” because debugging usually relies on a more precise understanding.

2. Too prone to random luck. If you have 100 lines of code and line 42 has a bug, then some people will, just by chance, find it immediately because they happened to look at line 42. While others might take a lot longer to find it just because they started at a different line. While there is some skill involved, we should all be familiar with how often debugging time can vary even for similar bugs just because of random chance. This makes it a poor choice for an interview in which you only get a single sample.

Sharing Code

One unusual type of interview I learned about recently is to “bring your own code to share”. In this interview the candidate picks a sample of code they wrote, and shares their screen to the interview. They then walk the interviewer through the code. The interviewer asks “hard” questions such as why they made certain technical decisions, how they would do things differently, what worked well and what didn’t, etc.

I actually think this is probably the worst type of “coding” interview I’ve ever heard. It seems designed on purpose to maximize bias:

1. Some strong candidates might not have code to share (they only coded for a previous employer, and are forbidden from showing this code, or they recently graduated and haven’t worked on anything interesting yet). Limiting your pool of candidates in this capricious way is a terrible approach considering how hard it is to hire right now. 

2. You need to be able to understand the code they are sharing. This means you can only hire people that work on similar things you have experience with. If you only have experience with web dev and are hiring a systems programmer it’s going to be hard for you to understand what they are doing.

3. Because it’s so subjective you will be maximally subject to any bias you have. This includes not only bias in terms of the candidate’s background, but also bias in terms of coding style, the work they did, etc. This means if you have multiple interviewers there’s no way to standardize feedback, you can’t even compare candidates directly because they each show something different.

4. It misses many important aspects of work (similar to “reading code”), and yet “sharing code” is barely even important at most jobs. Most of the time people that rely on your code either understand it pretty well (because they work with you) or don’t care (because they treat it as a black box). So you’re directly testing a skill that doesn’t matter very much. Indeed reasoning about architecture and design is certainly important, but those are better left to a “systems design” type interview specifically focused on that.

Take Home Project

The origin of “take home” coding assignments seems reasonable enough. We can get rid of all the unrealistic aspects of interview coding questions by making it a real project that candidates get to actually work on in their own environment. They can use their preferred setup of IDE, documentation, Google, and other resources in order to produce a realistic view of the output. This can work, especially for candidates without any traditional experience wherein you need higher confidence that they can tackle bigger work. However, there are a number of downsides:

1. You may scare off strong candidates that don’t want to spend several hours on your assignment. It can be unclear whether it replaces a traditional interview or whether it will be followed by a second round. These can be mitigated somewhat by keeping it short and explaining the process very clearly.

2. There’s no way to restrict time spent. You can’t tell apart someone that did a great solution in 1 hour vs. someone that spent 5 hours. However you’d certainly rather hire someone that did the work significantly faster in practice, so missing out on this signal is problematic.

3. There is inherent bias in favor of candidates that have a better environment for working at home. With many people working remotely this is less of a concern, but some candidates might only code at their office and thus not even have a proper setup on their home computer. Or they might not have a good working environment at home (e.g., many distractions, less quiet time, etc.) Of course, if hiring for remote work you might consider this valuable signal instead.

4. It can be much harder to evaluate a bigger project than a specific coding problem. If they spent 4 hours writing code, you will probably need a significant amount of time reading it to make sure it’s well-written, structured nicely, etc. There should probably be some standardized checklist to make sure different interviewers are evaluating it similarly.

5. There is inherent bias in favor of specific languages. Unless you only want to hire experts in a particular language, anyone with more familiarity will do a lot better than someone that has to Google every small detail. This type of difference often doesn’t matter as much on the job since they will get more familiarity (and have to learn your specific setup anyway). Again, this could be considered a valuable signal if you absolutely need someone to get off the ground immediately.

6. It’s easy to cheat. This is unfortunate, but it certainly needs to be considered. Anyone working without being monitored can easily get more help than you’d like them to, up to and including having a more competent person do some or all of the work for them. Most people won’t be so dishonest, but it’s risky to rely so much on something easy to fake.

“Realistic” Coding Interview

I save this for last because I think it’s probably the best alternative to a traditional coding interview, and yet also the most similar to it. In this interview you give the candidate a coding problem that is more “realistic”, and let them solve it using a real programming setup. They then do this while you observe, perhaps asking questions about their process during the interview (like pair programming). For example, for a full stack web dev, you might have some existing sample web app, and ask them to add some specific API endpoints and create a small UI that calls them. 

This is a pretty good alternative if you don’t like the structure of traditional coding interviews. However, it does have some downsides that need to be considered:

1. There is a lot of bias in favor of people familiar with the development setup. What if the candidate prefers a specific editor, like vim? If you allow them to bring their own device to code on, then, again, you are favoring those that have it set up nicely. 

2. It can be hard to make it work well for different languages, and favor those with more familiarity. If one candidate needs to Google various library functions because they haven’t used that language in a few years, but another has them memorized, the latter has a big advantage.

3. Similar to the Take Home Assignment, it can be harder to evaluate because they are writing a lot more code. Observing the whole time does help, since you get to see more of the process. But because there is often more variety in approaches for a more complex problem it can be harder to compare candidates.

4. It takes longer to do this than a traditional coding interview. Because the problem is longer you probably need to give them 60-90 minutes rather than the 30-60 minutes that a traditional interview takes. This means you have less time for other interviews.

Conclusion

There’s no perfect way to interview, every method has its pros and cons. A good approach will combine different types of interviews to get a more rounded picture of the candidate. Despite many people recently being against including traditional coding interviews, I have argued that it is an integral part of any interview loop, and that it can’t be properly replaced by any other commonly suggested types of interviews. This is because it’s the best way to get a minimally-biased look at how the candidate actually writes code, and the somewhat artificial setting of the whiteboard problem actually minimizes bias and allows for more uniform evaluation. You should then add to that other types of interviews, including some of the types listed above, as well as system design, behavioral, etc. Having this variety will help you to hire the best people.