Step 2: Designing Impactful Coding Challenges 

In Step 1, we made the argument that you need to think about which you value more: intelligence or knowledge. Now we’ll get to the core of the actual design of each of these types of questions.


Broadly speaking, we've found there are 3 facets of tech skills to test for:


  • How well does he or she know the fundamentals?

  • How deep is his or her tech knowledge?

  • How deeply can he or she think about problems?

The first two can be tested well before even interacting with the candidate using coding challenges. The third facet is best measured in person. This is the longest step of this guide because we offer detailed examples of each category of challenges, as well as a crucial checklist of biggest mistakes that companies make when designing challenges.

I. Gauging Problem Solving Skills

There’s one thing you need to account for off the bat: Can they code? It sounds obvious. But for years, notable engineers have repeatedly (20072012 and as recent as 2015) pointed out that a major chunk of applicants can’t write simple programs. You can weed these folks out by asking them to solve a basic coding challenge. Your first challenge should be light and breezy.

Before asking algorithm questions, Soham Mehta, founder of a technical interview preparation site Interview Kickstart, includes quick multiple-choice challenges that test for some very basic CS fundamentals. e.g.:

  • Given N distinct numbers, how many subsets can you form? [Answer: 2^N]

  • Given string of length N, how many permutations of that string exist? [Answer: N!, or simply N x (N-1) x (N-2) … x 1]

a. Multiple Choice

Multiple choice coding challenges are lightweight and are easier on the candidates. Offer several questions, each with plenty of answer choices. That minimizes the chance of getting correct answers by clicking randomly.


Here’s an algorithm-based multiple choice example: 

b. Code Completion

Code completion is another lightweight style to consider as the first impression with candidates. A code completion challenge is when candidates have to fill-in-the-blank of a given piece of code. Some of the greatest programming minds of all time spend a great deal of time reading source code. If you provide most of the code, and ask candidates to fill in the lines, it’s an interesting way for candidates to solve a challenge while gauging critical thinking and their ability to read other people’s code. Plus, it’s often more fun.


We initiated a contest series exclusively for code completion for our community of developers recently and it’s been one of the most engaging contests we’ve ever held. Check out the contest for examples of sentence completion coding challenges which will help guide you in creating some of your own.  


II. Gauging Knowledge


Unlike fundamental algorithm challenges, knowledge-based coding challenges are much more straightforward because they’re dependent on knowledge of a particular domain or technology. For instance, challenges based on Client-Server, Sockets, Multi-threading are good signals of seniority in distributed systems. We asked Dr. Memelli to offer a great example of a knowledge-based coding challenge:



Here’s a knowledge-based multiple choice example:



III. Gauging Depth of Thinking


While intelligence-based challenges are the best predictors of success, there’s one major differentiator between novice and an expert. And it’s not the years of experience. Rather, it’s how deeply can you think about problems? 


Josh Tyler, who runs engineering at Course Hero, defines experts as engineers who have proven ownership or thought leadership of large systems, like services, apps or frameworks. With this experience of ownership and tools, experts are scientifically proven to intuitively look at problems in deeper context than novice programmers.


There are quite a few studies that support this. Here's one:


Novice: “Novices operate from an explicit rules and knowledge-based perspective. They are deliberate and analytical, and therefore slower to take action, they decide or choose.”

Expert: Experts operate from a mature, holistic well-tried understanding, intuitively and without conscious deliberation. This is a function of experience. They do not see problems as one thing and solutions as another, they act.


So, how do you really test for depth in thinking? How do we know if engineers can think about software rather than just see its code? Unlike the first two types of challenges (algorithm and knowledge-based), which can be clear-cut automated challenges, questions that measure depth of thought are usually best accomplished in person. This way, you can start off simple and proceed to make the problem statement more complex as you proceed to work it out. There are two different types of questions that can help you measure seniority by depth of thought:


a.) Open-ended questions

After you have a vetted stream of candidates who pass the questions for technical aptitude, asking open-ended questions about big picture strategy is a good way to gauge depth of thought. Taylor finds success with questions like:


Tell me something you did that had a big impact in a positive way as a result of time to think and strategize. How about the negative impact?

I find that mistakes are made when technical people run straight into the how we build something instead of challenging what the right thing to build is for the customer.
— Josh Tyler, who runs engineering at Course Hero

Other examples of open-ended questions that are critical to test seniority-level and leadership skills:

  • How do you build search for Gmail?
  • Describe to me some bad code you’ve read or inherited lately.
  • When do you know your code is ready for production?

b.) Infuse multiple parts in your challenges

Tyler says the most effective way to design challenges for senior folks is to create lots of room for depth in the problem statement. “Most of our interview questions are designed in a way that has many parts. A junior candidate usually only gets through the first part, whereas senior folks get through the second or third parts,’ Tyler says.

For instance, here’s an example of a good optimization question.

You are given many words and you have to find frequencies of each word. Here simple maps, arrays, lists will not work when a huge number of words are given. Instead, you should go for Trie, an efficient data structure here.

Greg Badros, who founded Prepared Mind Innovations, would break up this problem into the following parts:

  • Tell the candidate that you’ll start simple and make it more complex as you work through the problem
  • Ask for word frequencies
  • Make sure they get the simple map solution without coding it
  • Tell them how much RAM they’ve got and how big the dataset its
  • Ask them to estimate how big the process will get for the language they’d write this in

If they get this far, then ask them to propose an alternative data structure that would be more memory efficient because they’re out of RAM. When they’re out of design ideas, have them code as far as they go.


Continue to Step 3: Coding Challenge Checklist



                Step 1:                                  Step 2: (This piece)                          Step 3:                                        Step 4:                                              Step 5:    Before You Do Anything                  Designing Challenges              Challenge Checklist                   Setting Expectations                               Calibrate  

             Step 1:                                 Step 2: (This piece)                         Step 3:                                      Step 4:                                            Step 5: 
Before You Do Anything               
Designing Challenges             Challenge Checklist                 Setting Expectations                             Calibrate