By popular demand we are introducing candidate feedback into our code challenges.
We took our inspiration from conversations with developers who had taken our code challenges and clients who'd looked through our code reviews. We were also partly inspired by Andy Davis's post about failing a code challange and having the decision overturned once he spoke to the hiring team.
However hard you try, it's nigh on impossible to create a decent code challenge with zero ambiguity without creating instructions that read like legal briefs. Throw in 'exam pressure' and good old fashioned: 'I don't read instructions, I jump in feet first' and you have a recipe for misinterpretation and misunderstanding.
We believe code challenges should be rewarding for the candidate. If someone is going to give up 2+ hours of their time to take your challenge the very least they should get back is the results. And when we speak about results we don't mean a score, we mean a thorough evaluation of the code; line by line analysis, with summary points consisting of constructive feedback pointing out the good and not so good elements of their solution. The trouble is, without any mechanism to reply back to the reviewer this feels a little like being back at school. We felt the natural evolution of this process is to allow the candidate to justify their decisions and to be able to explain the choice(s) they made, by responding to the reviewer's comments.
At Geektastic we try and make the code challenge experience a 'real' as you can whilst allowing constraints to measure and compare candidates. This means creating challenges that are aligned to the company candidates are applying to, creating product features or solving problems that they might experience when they land the role. We acknowledge that a 2 hour time constrained code challenge creates an unreal situation, it adds pressure that doesn't usually exist during the average working day. Sure, businesses put engineers under pressure to deliver, commitments made by the team during sprint planning inevitably put demands on individual members, but nothing compares to the clock ticking away (even the thought of it takes me back to cold exam rooms, teachers pacing up and down, staring at the analogue clock on the wall that seemed to be running at double time). Even in open-ended challenges where the time element is removed there is still the monkey on your shoulder encouraging you to do things you might not do if you were sitting relaxed at your desk with your team.
Every so often we'd see a developer write into the team asking if they could respond to the peer review. Most of the time it's calm, collected and just wanting to explain their solution (we have had one developer go ape$hit, but luckily it's pretty rare). Sometimes they are disagreeing with the comments, but other times it's because they interpreted the problem differently.
Here's an example.
Most of our challenges say the code should be 'of production quality'. Our Python code challenge requires the solution to be run once to establish if it works. The candidate used cached lookup tables to improve ('real world') performance as this was his interpretation of 'production quality code'. The reviewer marked the candidate down as he felt he had over engineered the solution. (one might argue it's OTT to mark down but that's another matter). Once the developer fed back on these comments, everything made sense and the results were adjusted accordingly - that developer is now VP of engineering at a very exciting prominent AI firm).
Allowing these feedback loops inside the challenge allows for interaction between the reviewer (whether that's one of our reviewers or the hiring team's) to take place, speeding up conversations that traditionally either didn't happen or if they did, were delayed until a follow up call, email back and forth or face-to-face interview.
As always we'd love to hear your thoughts on this and any other experiences you have had with code challenges.
If you'd like a demo or want to hear more about Geektastic and how we are changing the way software engineers are hired please email email@example.com