Guest post by Andy Davis
Anyone who has devised a coding challenge will agree with me when I say:
It’s REALLY hard to create that perfect coding challenge that filters out bad developers but keeps good developers interested.
Or is it?
Well, it turns out, you can dramatically increase the effectiveness of a coding challenge by using one simple rule.
And in today’s post I’m going to show you what that is… and exactly how apply it to the next coding challenge you design.
But before I tell you exactly what that is, let me tell you a little story...
Several years ago, a good friend of mine took his driving test for the 3rd time. He’d done a tonne of lessons and he’d had more than enough practice - he is a good driver. On the day of the test, the examiner asked him to turn left out the test centre. Under the pressure of the test, the instructions didn’t register and he turned right.
Now you could assume he failed for not following instructions. Instead some common sense was applied and his driving skills were still assessed - NOT his ability to follow instructions under intense examination pressure.
It should be this way with coding challenges.
Clean, well written code that communicates its intention, should be a pass even if it is not exactly what the specification asked for. Unfortunately it’s not always the case.
You or I might argue that understanding requirements is a big deal. And in some contexts it really is. If the job was to be fed requirements by email, have no interaction with anyone, and deliver within 1 hour, then I would have to agree with you. But how many jobs are like that? And let’s be honest, would we really want to work there? or shall we leave those jobs for elancers?
We all know that the real world isn’t like this. You don’t get fed a bunch or algorithms to code up during the job, so why do they send them to us as a coding challenge?
It’s because it is easier isn’t it? There is more less a set answer, does it work or not? And potentially an order of complexity (big O, anyone?)
Since algorithm questions have mathematically predictable answers, these sorts of tests can even be marked by a machine.
This is disrespectful. We took the time to do your coding challenge and you couldn’t take the time to have it marked by a human.
If you have your coding challenges marked by a machine - PLEASE STOP.
Machine marking is a binary solution to a non binary problem. People are not binary (not matter how geeky we talk).
Let me side-track you for a moment and tell you another story… (I’ll get back on topic soon, I promise)
At my university, there was a revolutionary new system called Coursemaster. The boffins that be had devised an automatic programming coursework marking system. It became our friend, foe and arch nemesis over those 3 years. At first it was fun, then we realised that our grades depended upon it. That’s right, our degree was marked by a machine (well, part of it anyway).
The professors built-in some flexibility, it allowed a 2nd and 3rd attempt. So what did students learn to do? That’s right, Group up and code it together, pooling their 2 spare lives as a sacrifice to the cause. A great application of Pair Programming - others might call it cheating!
Anyhow, like it or not, the Coursemaster system was definitely gamed by some students - but let’s be fair here, the system was out to game the students, it wasn’t really very good at marking their code - correctness is a very small part of the battle - so it probably levelled the playing field.
Flash forward 15 years and the technology has not moved on!
Only last week I was requested to do a test that was marked by a computer…
I got 0%....Why?
...I misinterpreted one word in an aggressively timed coding challenge and it left me with not enough time to correct my error. Total frustration. I couldn’t even prove it was good code, it wasn’t, I hadn’t refactored yet, I ran out of time -
Now there’s an idea, if the test is timed maybe there could be a refactoring window at the end where it doesn’t allow functionality to be changed but you can improve the code quality and make it communicate (which in my opinion would be a pass if you know how to do that).
Let’s cut to the chase...
The single biggest factor that will improve your coding challenge
So here is is… the killer idea you have been waiting for. I spent the last few weeks speaking to and surveying some of my peers on this topic. One guy stood out amongst the crowd, he’d done as many tests as me, but better than this, he’s also designed several coding challenges and saved a tonne of time for HR in the process.
It is with great pleasure that I mention Zeno Foltin, a London-based (polyglot) software developer. Zeno is a high quality developer and I rate his opinion on this topic extremely highly.
I met Zeno when both of our companies were working with WorldRemit, a London FinTech money transfer company. They were on a major hiring binge and Zeno took responsibility for making a coding challenge to help them in their quest for hiring good developers. Introducing this stage to their process was vital for them. The quality of candidates through the door went through the roof. I caught up with Zeno and asked him about World Remit coding challenge and his thoughts on how to design a good challenge.
We compared notes about the wealth of challenges we’ve done between us and some key points stand out.
So here it is, the number 1 stand out idea that you should apply immediately to any challenge you create:-
Make the challenge as close to the real world as you can.
It sounds too obvious doesn’t it? I wish it was. I can count on one hand the number of challenges I’ve done where I’ve been able to see the connection between the coding challenge and the company once I have started the role.
This is the single biggest factor that has kept us interested in various challenges over the years. There is nothing more frustrating that receiving a coding challenge to do, and seeing it is another boring algorithm like checking a string for parenthesis counts, or sorting a list without using inbuilt functions.
Generic coding challenges can be a drag, especially if you are applying for multiple roles. When the challenge is closer to the real role, it solves the “interest” problem. Candidates who aren’t that interested in the challenge will naturally not do their best - therefore it kills multiple birds with one stone.
One of the best ways of making your challenge close to real world is to make sure the challenge poses a problem within the general business domain of the company.
IMPORTANT NOTE: If you take this to too far and make this too domain specific, you’ll end up alienating people and could end up with a coding challenge that only lets you hire people who already work for you - so remember to strike a balance. And remember if you use Geektastic for your coding challenge, there is an opportunity to license this challenge out of multiple companies to use. So if it covers a bit of the tech that needs testing as well as the general domain area, you’ll be onto a winner. If I were looking for a coding challenge to use for a logistics company and there were 2 available challenges but one was about sending parcels and one was about phone bills, I know which one I would chose.
To give you an idea. Challenges we’ve liked in the past, all fulfil the following
Several BIG Topics came up in the process of our discussions:
So let’s look at each of those points in detail.
Challenges with a hard cut off time. It seems obvious why companies might impose a time limits:
Zeno and I put our heads together on this one and we think not enough thought is put into the downsides of that time limit. For example:
“I don't think timed challenges provide any value in measuring how well the candidate will do on the job. If it's about trying to find out how someone handles pressure, do it in a pair-programming session maybe. I think it's silly to think in our creative industry that putting such pressure on people will bring the best out of them.”
So whilst we don’t think timed challenges are completely useless, it just might not be the type of comparison they think they are making - possibly just a superficial comparison.
The compromise here is to have a time limit but make it generous enough that the candidate has time to try out a few ideas. The last thing you want to do is cut everyone off mid flow - this would remove the benefit of having a test in the first place.
Next up was the IDE. Believe it or not, some coding challenge systems actually force you to write and compile code in the browser - it seems to have become a bit of a fad recently. It’s pretty nifty but it only really works for simple code. It seems to have benefits of scale for the companies using them as the computer tests the inputs and outputs for correctness - but this shows very little respect to the candidate and actually makes their life much harder. It is unable to provide rich intellisense or access to commonly used frameworks or testing tools. There wasn’t really much debate here, we both hated coding in the browser. Developers like their tools and we are no different. Writing code in the browser is not part of our weaponry and will not be something required on the job. Browser based code editors are great for tutorials and quickly trying something out you’ve just been taught but as a candidate testing tool, this should seriously be reconsidered.
“They must use an IDE they are comfortable in. I used to ask candidates to bring their laptop with them for the face-to-face interview to do some pair-programming. You want to find out how the candidates work in their usual environment, not how they cope with a new and unfamiliar one. Unless the job is about writing code in the browser?”
Compromise might be that you use your own IDE but then upload or paste the final solution.
Likeness to the actual job
We know that is it impossible to condense the real nature of the job down into a coding challenge. But there really doesn’t seem to be any point in asking a developer to complete something like an in-memory algorithm to return them the correct change from a vending machine. Sure, there are some algorithms in there and order of complexity to think about. However, if the job isn’t about solving little puzzles, it really doesn’t appear relevant.
I was once sent a zip file from an online fashion retailer and told to refactor it. Whilst this sort of request is a little bit lazy - at least this is probably closer to the real job. Refactoring legacy code is part of almost every job. Furthermore it was actually in their domain. So, big points to them for this challenge, it was enjoyable. The only downside of that is that I never heard anything back (after spending 2 hours on the test too) - this is one of the biggest bugbears we have as developers.
The bottom line around getting challenges close to the real world is:
Case Study: JUST EAT
One example we discussed in detail was the JUST EAT recruitment test. Zeno worked with them for a year and I went through the process last year, so I had also taken the same test.
We both had a great experience with it and rated it the number 1 test we’d done. You can see their test here:- https://github.com/justeat/JustEat.RecruitmentTest
“I really liked the JUST EAT recruitment test because it is simple - list the takeaways for a given postcode using the API. It requires you write a simple UI to present the information, but it doesn’t tell you much more on how to go about it. It is directly to do with their business, it covers UI and logic (I don't believe in back-end or front-end only roles), and it requires more effort than a couple of minutes figuring out or googling an algorithm. A well designed challenge, can give an opportunity for the candidate to showcase the best quality work that they can do without hard time constraints.”
The JUST EAT recruitment test is also a very enjoyable challenge. So let’s take a deeper look at it:
It’s great that even at the coding challenge stage, there is a bit of two way flow of information, just by being open to open-sourcing. There aren’t many places where you get to see some of the code before you go there. It is often a bait and switch of great stuff spoken about at interview followed by a shocking reality of legacy code once you start.
What can we learn from JUST EAT even if you don’t have a public API?
You might be fooled into thinking JUST EAT have unfair advantage of their domain being highly public facing. Even if you don’t “need a balti”, you still know some of the takeaways in your area that they service and when these familiar venues are returned by the API for your postcode. Not everyone can do this as their domain is not so publicly known - but I bet with a little thought, you can be creative enough and throw the candidates a little bone about the real job rather than have them implement a vending machine, phone bill or shopping cart. There is nothing worse than seeing another challenge that isn’t even related to the business sector that the company is in.
Another lesson that can be learned from JUST EAT is not be too strict about the solution. If you can allow the candidate to be creative, you will get a better picture than from an algorithm implementation with a right or wrong answer. Sure, it takes more effort to have a developer mark the solution, but that time will be infinitely better spent if you discover better candidates from it.
Here are some ideas for you, even if you aren’t as publicly glamourous as JUST EAT
It is a continual learning process
In agile software development, we are told to embrace change. Things will change, so build change into the process and not make it an afterthought.
Like anything else, accept that you won’t your challenge right the first time.
Developers WILL complain.
Some good candidates will slip away - refusing to do it.
Some good developers will ‘fail’ it.
Some bad candidates, will find their way past the challenge and waste precious time that you could be spending with Ninjas
Just a joke, there’s no such thing as Ninjas…
...and even if there is you don’t want them
...and even if you think you do, you just alienated them and half the developer community by using the word Ninja, whoops.
You should plan to get feedback from candidates taking your test and LISTEN to this feedback.
Warning: FEEDBACK WILL BE EMOTIONAL, especially if they ‘fail’.
Use any little nuggets inside their feedback to evolve and improve the challenge.
Finally, if they have a valid point - reconsider the candidate, I have successfully failed a challenge before and turned it into a pass - you should do the same (under the right circumstances)
Continue to screen out cheats and chancers, however, is googling the answer really cheating? After all, good coders are good googlers and part of the job is knowing how to define problems well enough that finding the right answer on stackoverflow comes naturally. Pasting a snippet of code and using it effectively, is definitely not cheating, But I’ll leave this one up to you.
Don’t put all your eggs in one basket with a single piece of code.
A coding challenge should only be used to screen out really unsuitable candidates. Everyone else progresses to the next stage. And yes, there must be multiple stages if you have included a coding challenge. You cannot base a hiring decision on code alone.
Uncle Bob thinks a coding challenge is useful but only as part of an interview process, and values pair programming over and above it - I tend to agree, but that is another story - and probably a long one so I’ll save that for another day
Let’s wrap things up (that timer is ticking down)
I have covered a few key points here. I hope it sparks some ideas. If you are making a coding challenge or thinking about changing your current challenge, drop me a line, I would be interested to hear about it and be happy to pass comment on your current challenge.
I think it is important to remember that developers do like coding challenges. We’re coders, we like to code. We’re not so fond of having someone looking over our shoulder but we do like solving business problems and we do like writing code - so don’t be afraid to have one in your process.
Let’s conclude by reminding ourselves of some of the important points that we’ve explored
Good luck in producing a great coding challenge. Personally, I recommend an open and creative challenge with a guide time limit. - where the candidate can show off their creativity and flare, domain modelling and good clean code. But then again, I would say that… because I once failed FizzBuzz.
Let me know your thoughts in the comments and I’ll be happy to respond.
About Andy Davis
Andy is a software developer. He’s been writing software for over 20 years and helps businesses solve interesting business problems, by helping them create the right software. He also helps teams with their developer hiring strategy as he believes that hiring the right people has a multiplier effect on awesome software teams.
Want to create your own? Perhaps you have a coding challenge you have created already already, or maybe you feel you can create a good one now from scratch. Whichever of those two, Geektastic make it easy to load it onto their platform and manage the whole process of inviting your candidates (who can complete it in their own IDE), submitting it and getting it reviewed by a real human.