Step 3: Coding Interview Question Checklist 


So far, we’ve been focusing on the diverse categories of challenges, based on what you need. Now let’s get down to the practical ways of designing challenges. We looked at the question banks of various companies to pinpoint the patterns of what makes a code challenge successful. Based on these patterns, here’s a checklist of 5 common mistakes to avoid and  ensure each challenge will draw the best candidates:

 

Do you have the right answer?

 

This sounds like a no-brainer. But you’d be surprised as to how many companies just get their own code challenge wrong. CareerCup CEO Gayle McDowell frequently consults with tech companies of all sizes on their hiring process.

“When I’ve reviewed companies’ question banks, about 10% of answers are wrong,” McDowell says.

And it’s not a matter of carelessness or minor bugs in typing up the solutions. The challenge designer genuinely believed their algorithm was right until they were proven wrong. Take the time to ensure that 100% of your answers are correct. Otherwise, you’ll turn away a lot of strong candidates who were actually answering the question right.

 

Are your algorithm challenges challenging enough?

 

When it comes to algorithm code challenges, McDowell estimates that if 5% of your candidates instantly know the solution to your question, then it’s likely to be too easy. Easy challenges cluster the mediocre and the great, which of course makes identifying great developers very difficult. Your challenge should be hard enough to separate the average from the top.

 

In an algorithm code challenge, you should have at least one challenging question where only about 20% of your candidates solve it perfectly. A good algorithm question would have 2 (or more) different solutions, one more optimized than the other. This leads to 3 tiers of performances:

 

Tier 1: The best candidates will pass all test cases because they implemented a correct and optimal solution.
 

Tier 2: The okay candidates will pass most of the test cases with an algorithm that’s correct but suboptimal. Their code gives the right output, but times out on the large test cases.
 

Tier 3: The candidates you don’t want to hire won’t develop a correct algorithm at all.
 

If you use a platform like HackerRank, for instance, you can evaluate based on the time to complete a test case, taking into account both asymptotic complexity and the lower constraints. But it doesn’t impose a time condition on all problems. It’s up to you as the problem designer to decide if the efficiency is important.

 

Are your problem statements detailed and tailored to your company?

 

Engineers who can identify the right sub-problems, solve smaller problems and then merge them together are more likely to be successful as leaders in your org. Being able to break things down, spot patterns and attack problems systematically are crucial for any strong engineer. The more tailored and specific the problem statements are to your industry, the higher the chances of high-quality candidates completing code challenges as part of their  application.

 

We did an experiment in which we compared two similar companies’ challengesone with a generic problem statement and another with a more tailored challenge. We saw a nearly 10% higher completion rate with the latter. Even anecdotally, we consistently see this direct correlation time and time again.

 

Candidates are more drawn to and interested in challenges themed to your mission versus generic, cold challenges. VMware, for instance, created a host of very detailed, complex problem statements that delved deep into virtualization. You can see the Logical Hub Controller problem is richly tailored to a typical virtualized datacenter problem they grapple with daily. By replacing resumes with tailored code challenges for two years, VMware:

 

  • Saved VMware engineers 100+ of hours per quarter by only calling candidates who passed the coding challenges

  • Increased the success rate for candidates who come onsite (compared to manual resume screenings).


Did you rule out luck or bias?

 

If a candidate gets stuck on some aspect of a problem, were they just unlucky? It’s important to distinguish between these factors, and the best way to do this is by getting multiple data points. That is, ask a question that involves multiple hurdles. This way, if a candidate gets stuck on one aspect, you can help them through it and they still have other logical leaps to overcome (and thus additional data points for your evaluation).

 

For example, consider this question, which has a bunch of solutions, each of which presents a new hurdle:

 

Given an array of people, with their birth and death years, find the year with the highest population.

 

  • First hurdle: The 1st hurdle is coming up with any correct solution. Most candidates should be able to come up with a brute force algorithm (e.g., for each year that a person is alive, walk through all the people to count the number of people alive in that year), but a few won’t. If they don’t, give them some help and see if this was just a brief lapse in reasoning or a consistent issue.
     
  • Second hurdle: The 2nd hurdle might be noticing that you don’t need to check the same year repeatedly. You can use a hash table to cache this data.
     
  • Third hurdle: A 3rd hurdle might be noticing that you only need to check the years between the first birth year and the last death year. So a candidate might first get that information, and then proceed with checking that range of years.
     
  • Fourth hurdle: A 4th hurdle might be noticing that, actually, you only need to check the first birth year through the last birth year. You don’t need to check years in which people only died; they certainly won’t have the highest population.
     
  • Fifth hurdle: And so on, until we arrive at an optimal solution. Great candidates might even leap past several hurdles at once, and that’s a great sign. A problem like this has so many hurdles that you can more effectively see if there’s a pattern in the candidate’s performance. For good measure, test out questions on your peers first. Ask at least 2-3 engineers to solve the problem before unleashing onto your candidates. This helps you see what sort of hurdles are in the question and what to expect from candidates.
     

Are you sure there aren’t any CS jargon words?

 

There’s a common misconception that you need to have a computer science (CS) degree in order to be a good programmer. In reality, only 40% of working software engineers in one survey said they had a CS degree. When we wrote about this in TechCrunch, Wavefront CEO Sam Pullara told us he agreed.

I’ve worked with a lot of great programmers over the years who don’t have computer science degrees. Their biggest disadvantage is that they often lack the jargon of computer science.
— Sam Pullara, CEO of Wavefront and formerly Yahoo and Twitter


 

 

Senior candidates, particularly, are most likely to be unfamiliar with CS textbook words. For instance, avoid dropping terms like “state machine” or “dependency injection.” Likewise, in order to avoid ruling out senior engineers who haven’t taken CS fundamental classes (or don’t remember them), avoid questions that involve more obscure algorithm knowledge.

This brings us to not only the most important step but also the most overlooked component of hiring senior engineers.

 

This brings us to not only the most important step but also the most overlooked component of hiring senior engineers.

 

Continue to Step 4: Setting Expectations and Warming your Candidates

  


              Step 1:                                        Step 2:                            Step 3:  (this piece)                           Step 4:                                            Step 5:  Before You Do Anything               Designing Challenges             Challenge Checklist                 Setting Expectations                             Calibrate 

              Step 1:                                        Step 2:                            Step 3:  (this piece)                           Step 4:                                            Step 5: 
Before You Do Anything               
Designing Challenges             Challenge Checklist                 Setting Expectations                             Calibrate