Step 5: Calibrate After Screening

We've made it to the final step after strategizing on the best types of challenges to tease the engineers you need. But what good is a coding challenge screening if you’re testing different candidates with different questions? Consistency can make the data you’re collecting meaningful.


HackerRank’s head of challenge curation Dr. Heraldo Memelli finds that all too often, companies change the question after it’s already gone out to some candidates, ruining your data set. Periodically review which questions have been directly correlated with successful candidates.


What good is a coding challenge screening if you’re testing different candidates with different questions
— Dr. Heraldo Memelli

After the initial screening, it’s crucial to mirror your coding challenges with the rest of your interviewing rounds. It’s logical — folks who clear the automated coding challenges online are more likely to perform well in-person if the questions are consistent. Automated code challenges are the best tool to predict who will perform well in the in-person interview. So, target your questions to mirror the style of questions asked in the actual interview as much as possible.


Choose questions that you’ve actually asked in on-site interviews so that you can see how candidates might react. This can help you frame and word the question appropriately as well. Of course, this means you’ll have to retire those questions from your in-person interviews. Aim to keep your coding challenge banks fresh. Trying out new code challenges on your own time is a great way to not only benchmark your own team against newcomers but also keep your engineering teams’ skills sharp. Investing the time in revamping and optimizing your set of interview code challenges is minuscule compared to the cost of hiring wrong engineers. 



Tying it All Together


Asking senior engineers to revisit fundamentals in an interview sounds outrageous. And it is if you simply send a cold email without any proper preparation. But automated coding challenges are the most objective and successful predictors of hiring we have to filter through candidates at scale.


Gauging not only their intelligence but also how much they value fundamentals through algorithm and data structure questions are strong instruments to find the best engineering talent. If you set expectations and are mindful of senior engineers’ complicated hubris, you’ll have a stronger pool of senior engineers. And you’ll be set up with a replicable system to build and scale your engineering team. A set of well-designed mix of questions that test fundamentals, knowledge and depth of thinking, is a good way to weed out candidates who don’t have the basic skills you need to build revolutionary software.

               Step 1:                                          Step 2:                                      Step 3:                                        Step 4:                                    Step 5: (this piece)    Before You Do Anything                   Designing Challenges               Challenge Checklist                   Setting Expectations                            Calibrate

            Step 1:                                        Step 2:                                    Step 3:                                      Step 4:                                   Step 5: (this piece)
Before You Do Anything               Designing Challenges             Challenge Checklist                 Setting Expectations                           Calibrate