This might come across a bit like an advert for TransferWise, but we felt it's important to share why we chose them to transfer payments to our global UberGeek team every month for the code challenge reviews they perform on our platform. We were looking for a partner to facilitate fast, secure, cost effective money transfers to our team based all across the globe from San Francisco to Sydney. We looked at PayPal, Moneygram and a few others but felt that Transferwise had the edge with their cheaper fees, constant innovation and serious backers. Here are some fact and figures (all of these can be seen on their website) TransferWise launched in January 2011 by two Estonians Kristo Käärmann and Taavet Hinrikus (Skype's first employee). It is headquartered in London and has eight offices around the world, including New York, Sydney, Singapore and Tallinn, Estonia. TransferWise has over 1 million customers, sending more than £800m using the platform every month. TransferWise supports more than 645 currency "routes" across the world. They have a 5/5 star rating, 36329 reviews on TrustPilot They raised $25m investment from Peter Thiel’s Valar Ventures, Index Ventures, Andreessen Horowitz, IA Ventures, Sir Richard Branson and other high profile technology investors. Authorised by the UK Financial Conduct Authority (The FCA) **UPDATE** We have now added Payoneer to our payments solution. We found that Transferwise didn't support international payments to Brazil and since we were seeing some amazing engineers join the platform and join our elite review team we needed to work out a way to transfer funds. Payoneer offer a bank transfer system using Barclays. The money gets transferred from our bank to Barclays and then our team withdraw from their Payoneer account a few days later. We have also used this for payments to our Chinese team.
3 Comments
How to create an awesome code challenge, a really really awesome one. The very best, in fact.16/2/2017 Guest post by Andy Davis - super talented software engineer, guest blogger and awesome code challenge creator! Recently I’ve been talking about coding challenges a lot. What’s makes for a good one? What makes for a bad one? How to mark one? And even, how to make one. Plenty of people out there are asked by their hiring team to come up with a challenge that will test candidates and find them their perfect matches. If only it were that simple... Since you are reading this, you have probably made one in the past, or maybe even need to make one now. So let’s spend a bit of time on it. It doesn’t have to be complicated and you don’t have to grab the first one you find online.
You shouldn’t rush it and you don't need to create the best one in the world. Here’s why... No one gets hired off the back of one coding challenge solution. That said, it is usually the first opportunity to see what someone is like when actually writing code. So it is still totally worth the investment. A reliable developer hiring process shouldn’t only be about talking. So, you need to make one, how should you do it? Firstly, decide the goals behind your challenge. Goals are the easiest and most obvious way you can assess candidates quicker and more fairly. You’d be surprised at how many challenges out there are just Fizz Buzz - has any thought really been put into what they are looking for? or did they just google and use the first thing they found? For example, do you want people who test drive their code, do you want hardcore optimisation specialists, do you just want people who can get the job done as fast as possible. Once you have the goals, it is much easier to design a challenge. I’m going to design a challenge, to show you what I mean…. But before I do… I want to leave you with one last suggestion to keep in mind when you make your challenge. You should be trying to separate the wheat from the chaff, not the wheat from the wheat. What I mean by this is, filter out the people who can’t write code. Don’t filter at this stage by coding style or architectural approach. Those are the things that carry the value later in your hiring process when you are actually in the same room and talking to each other. Without further ado, here is my attempt at creating a challenge from scratch. I wanted to pick something topical, so I scoured hacker news to see what a common topic was and if I could come up with something relevant. I settled on Autonomous Vehicles (or Self Driving Cars). Not only was it popular, it seemed kinda fun too. I wondered if I could code one up. Or better still, put together a part built framework/api and let the candidates come up with something. After a few iterations and feedbacks, I had something quite useful in assessing candidates and working out which bucket they fall into. I say 'bucket' for a reason. Despite all companies thinking they only hire the best, I believe developers aren’t actually good or bad. There are no ‘best’ Developers. They just fall into different categories or 'buckets'. The best for one company is not the best for another. A junior, fresh into the industry, with no commercial experience are not bad developers for example. Ok ok, there are some bad developers out there. We allI know who there are, but someone hires them don't they ? Let's move on :) TL;DR - If you think you are up to the challenge, please try out the Self Driving Car Challenge for yourself right now - We have C Sharp and Java versions for now, but we'll add more if there is demand. Simply send an email to [email protected] stating whether you would like the C Sharp or Java version. And now comes the science…. This is my detailed thinking and the process behind how I came up with the challenge. The challenge is still a work in progress, but that is how I think all challenges should be and here is the detail and the journey of how I came up with it. Where do I start? Should I just google “coding challenges” and pick one at random, and maybe tweak it a little bit? Here’s the snag… Coding challenges are much more popular now, it’s getting more and more common to find:-
Let’s address the point A. As long as we twist it a little, it won’t matter if they already know it. As long as the challenge isn’t 100% like for like with the one chosen from online, grabbing code from Google and StackOverflow is totally acceptable. It will still show how the candidate likes to write code. After all in production code, a good developer rarely copies code without tweaking it to it their needs. On point B, if they are too busy to do it, that means we don’t want to hire them anyway, right? Well...Not exactly! Too busy can be a good thing if it means they are in demand. After all, a good developer won’t necessarily be looking for their next role right now. However, they might be curious - we don’t want to quash that curiosity. We want developers to actually do our challenge! So we need to make the bar of entry low enough keep the good guys interested despite how busy they are. That’s not to say the challenge should be easy! We are knowledge workers. There must be some knowledge that the good guys know and the bad guys don’t? Something that comes with experience. Like a series of keys that get unlocked with experience? The problem is that different hirers have different keys. For example, when I set challenges, all I really want to know is if they have the right type of attitude and can collaborate effectively with the team I’m hiring for. I favour TDD over Test after. I favour clean code and SOLID principles over other schools of thought. That’s not to say other schools of thought are wrong, it’s just not the way that I have found (over the years) to be the most efficient. Let’s turn our attention to food for a second (because I'm hungry :)). Do you know Masterchef The Professionals? I see the code challenge as the equivalent of a Monica's or Marcus’ “Skills test”. For those who don’t know, Masterchef is a cooking competition and “The Professionals” version of the show is for people who already work professionally as chefs, this is not amateurs. You see, it’s just like us - we’re professionals too In the skills test, a specific skill is put under test for around 15 minutes. It might be making a posh omelette, it might be filleting a whole salmon. The contestants don't know in advance. The skills test is rapid. Contestants mess up all the time, the nerves and the pressure makes them do silly things, but that is ok, the judges are looking for seeds of potential, not the finished article. So if we think about this code challenge as a quick skills test like Masterchef, it puts it into context. Submissions don't need to be perfect. Which means the test itself doesn't need to be super hard in order to challenge them and separate them from each other. So what am I looking for in a “Skills test” ? Personally, I want to see:-
Remember: this list might not be the same for where you are hiring. I also want to make it REALLY easy for a good candidate to breeze through. Therefore I want the finished code to be
Pro Tip: I want working code, one way of proving to yourself (and me) that it works is to have good tests around the code. I might not ask for tests in the challenge, but unless you know a better way of proving your code works, write tests. Now we’re getting somewhere… To get cracking with my design of a new challenge, I need to:-
What? Pick a pattern? Yes, I will pick a pattern first and design the challenge after. One ideal solution would use that pattern - however, there might be even better solutions, only time will tell. NOTE: If I was hiring for an online casino then my domain would be casino related. If I am writing a general challenge that is more about testing their skill in technology, I would try to pick something that everyone has heard of in order to avoid domain confusion. I must keep reminding myself : How can I make it as easy as possible for the good developers to pass my challenge? Clean code and TDD are cross cutting concerns in this context, I always want to see this. So I won’t make any template code that is full of hard dependencies as this would be unfair to expect them not to follow suit. Even though great developers would fix my poor template, this would be a risk so I will avoid that. For extensibility, I’m also keeping in mind that I want something that can easily have “Feature requests” later - to give the candidate a chance to write code that lends itself to being open for extension (by the candidate predicting obvious next feature request and leaving the code open to that). Pro Tip: Don’t actually implement that prediction (remember YAGNI) Here are some options Shopping Cart - done to death Mars Rover - a bit rarer but most good developers know it (not such a bad thing) A Phone Bill - nice little domain that can be modelled multiple ways, gets messy and long winded if parsing the file needed Bowling Kata - takes far far too long to do it well and solutions easy to download online. Any of those can work if we tweak them to be short enough and focused enough. However that just seems too easy Let’s go wild and pick something topical at the moment, a quick scan of hacker news tells me that Self Driving Cars is pretty topical - eg George Hotz, an ex Google developer just open sourced his project and there is of course Tesla who make the news almost daily at the moment. The big players seem to think this is the future. A future they saw in Total Recall perhaps? First shot at the description "Your Luxury Limousine company has purchased its first Self Driving Car. The car has radar, cameras, GPS, and a computer onboard and accepts simple commands that control all movements of the car including steering, acceleration, braking etc. Your car has an onboard assistant (that needs writing) called ALFIE. The passenger in the car only needs to say a location and the car should drive itself there. Right now these options are A and B Your first passenger gets in the car. Requests to go to Destination A. When sShe gets there, she changes her mind and realises she wants to go to Destination B. ALFIE takes her to destination A, then to destination B. Background information From HQ….Destination A is NORTH (4 blocks) From HQ….Destination B is NORTH (2 blocks), then EAST (2 blocks), then, NORTH (2 blocks) Right now there are only 4 commands that work. FORWARD (this moves the car forward 1 block) TURN-LEFT (this turns the car 90 degrees left) TURN-RIGHT (this turns the car 90 degrees right) STOP At HQ the car automatically turns round and faces NORTH NOTE: This is an early prototype of the car and the manufacturer is releasing new available commands every week. Assumptions: You are only writing ALFIE. You do not need to deal with voice control When the destination of A or B is chosen, the car sets off The car needs a new command for each block You can queue up these commands in one go. The car will keep driving unless it has an explicit stop command The car will automatically turn and face the opposite way if you send one of the turn commands when Stopped. ” There, that wasn’t too hard. Does it satisfy my criteria?
First, solving the closed system issue. We need to be able to measure success. Perhaps some logging? That sounds too vague. I think that something in the domain would be better. What would tell us in the real world we are at the destination or where we are? GPS! Yes GPS is the answer… I need to add GPS to the question for verification purposes “As the car moves, the GPS should update its current position. For simplicity, the GPS can be an [X,Y] coordinate. Good - that means some tests can be put in place too, which is good. Next, find some guinea pigs….. Rick, the CEO for Geektastic, kindly put me in touch with 3 good developers who agreed to take the challenge. Here's how they did... Firstly, thank you to all 3 of you for taking the challenge. There is no job at the end of it, sorry, just this article instead. However, having now seen your code… I think I might just keep you in mind for the future. The solutions submitted were all good in their own ways - but it was tackled in 3 different ways. One was quite familiar to me and quite similar to the way I code, another was a way I didn’t think of doing it and the 3rd really focused in on the algorithm. Although the pool is small, getting 3 different solutions is good news to me. It means the challenge can be used in different contexts to find different types of developers (all good ones of course). The overwhelming feedback from the 1st candidate was that the challenge was decent. There was enough things to code and enough decisions to make about it and he was able to demonstrate how he writes code. However he missed the deadline - so, clearly, not enough time, fair enough, I only allowed him 30 minutes (plus geektastic allows a little grace, which is good, human-to-human process after all). I sent it to him again, and got it back within the hour. So in reality he had about an hour and a half. Not bad. We decided that we need to increase the time limit as the challenge was ok for him but just not enough time. He also didn't realise the API was a library reference (no source code), so said I should mention this as it was a little confusing. Consider it done :) Changes made:
Now we’re ready for the other 2 candidates to try. Let's call them A and B. This time, the time limit was no problem - fantastic. 2 hours is enough. It isn’t originally what I had in mind, as I think 2 hours can be off-putting. Since I think this challenge can be done in a hour or so, let’s stick with 2 hours. Candidate A’s solution was very algorithm centric. Candidate B’s used a very basic algorithm but focused more on the coding style and expressiveness of the code. Both are 100% acceptable solutions in the right context. It really just depends what the client wants - ie these 2 developers fall into 2 different categories, but they are both good. One of them (I won’t say which) gave feedback that they really didn’t know if what they had produced is what was wanted. Almost like it was too easy to just fill in a bit of code and a couple of tests, cleanup a few things and submit it. They were left not really feeling challenged. At first, this worried me, what if everyone thinks this? what if it really is too easy? It turns out, that developer is just really good! So, honestly, that is kind of what I want… a good developer to just come in and write the code and it seem like not a big deal. When I review the code, I want to think “ok yep, that’s not bad, I see what they did there. Yep, they’ve cleaned up a few things quite nicely”. To me, that is a sign that the challenge is working - as long as everyone doesn’t feel this of course. It has hit the right balance. What next? It appears the challenge is working well now. Still, I am a perfectionist and there is always room for improvement. I do want to iterate the challenge and improve it more. To self-critique my work, I think it is too wordy… under exam conditions your mind seems to race and seeing a long page for the description can be like a big mind fuzz. I don’t want the candidates to feel this - I think my next iteration would be to cut the length of instructions in half. For now, it is what it is... I would love to see you have a go at the challenge. We have C Sharp and Java versions for now, but we'll add more if there is demand. Simply send an email to [email protected] stating whether you would like the C Sharp or Java version Once you have submitted your solution we'll get it reviewed by one of the review team. By popular demand we are introducing candidate feedback into our code challenges. We took our inspiration from conversations with developers who had taken our code challenges and clients who'd looked through our code reviews. We were also partly inspired by Andy Davis's post about failing a code challange and having the decision overturned once he spoke to the hiring team. However hard you try, it's nigh on impossible to create a decent code challenge with zero ambiguity without creating instructions that read like legal briefs. Throw in 'exam pressure' and good old fashioned: 'I don't read instructions, I jump in feet first' and you have a recipe for misinterpretation and misunderstanding.
We believe code challenges should be rewarding for the candidate. If someone is going to give up 2+ hours of their time to take your challenge the very least they should get back is the results. And when we speak about results we don't mean a score, we mean a thorough evaluation of the code; line by line analysis, with summary points consisting of constructive feedback pointing out the good and not so good elements of their solution. The trouble is, without any mechanism to reply back to the reviewer this feels a little like being back at school. We felt the natural evolution of this process is to allow the candidate to justify their decisions and to be able to explain the choice(s) they made, by responding to the reviewer's comments. At Geektastic we try and make the code challenge experience a 'real' as you can whilst allowing constraints to measure and compare candidates. This means creating challenges that are aligned to the company candidates are applying to, creating product features or solving problems that they might experience when they land the role. We acknowledge that a 2 hour time constrained code challenge creates an unreal situation, it adds pressure that doesn't usually exist during the average working day. Sure, businesses put engineers under pressure to deliver, commitments made by the team during sprint planning inevitably put demands on individual members, but nothing compares to the clock ticking away (even the thought of it takes me back to cold exam rooms, teachers pacing up and down, staring at the analogue clock on the wall that seemed to be running at double time). Even in open-ended challenges where the time element is removed there is still the monkey on your shoulder encouraging you to do things you might not do if you were sitting relaxed at your desk with your team. Every so often we'd see a developer write into the team asking if they could respond to the peer review. Most of the time it's calm, collected and just wanting to explain their solution (we have had one developer go ape$hit, but luckily it's pretty rare). Sometimes they are disagreeing with the comments, but other times it's because they interpreted the problem differently. Here's an example. Most of our challenges say the code should be 'of production quality'. Our Python code challenge requires the solution to be run once to establish if it works. The candidate used cached lookup tables to improve ('real world') performance as this was his interpretation of 'production quality code'. The reviewer marked the candidate down as he felt he had over engineered the solution. (one might argue it's OTT to mark down but that's another matter). Once the developer fed back on these comments, everything made sense and the results were adjusted accordingly - that developer is now VP of engineering at a very exciting prominent AI firm). Allowing these feedback loops inside the challenge allows for interaction between the reviewer (whether that's one of our reviewers or the hiring team's) to take place, speeding up conversations that traditionally either didn't happen or if they did, were delayed until a follow up call, email back and forth or face-to-face interview. As always we'd love to hear your thoughts on this and any other experiences you have had with code challenges. If you'd like a demo or want to hear more about Geektastic and how we are changing the way software engineers are hired please email [email protected] Guest post by Andy Davis Anyone who has reviewed a coding challenge will agree with me when I say: The best way to judge a developer's skill, is to get them to write some code. But there is one problem... The code submitted to a coding challenge is NOT the developer’s best code. It is their code that was done within the pressure of a hiring process. We could just review the code, see if it works, see if the design matches our own idea of the design and then decide based on that if they are any good. But is that a good approach? I will suggest to you that there is no right or wrong solution. The code isn't really code, it is much better than that. It is a collection of clues. These clues can easily be overlooked. I want to show you how to get the most out of them. Including:
But first, let me tell you a little story from a few years ago: I once worked with a 'Software Developer in Test' who told me that he does something called MDD. I was intrigued, what is MDD? I’m into TDD, BDD, and even ATDD, but what is MDD? “Mortgage Driven Development” he said “I write the code in a way that only I can understand. That way I will keep my job as long as I want and I’ll continue to be able to pay my mortgage”. He laughed. We all laughed. The comment was made in jest. The unfortunate thing is, it turned out to be true. It wasn’t long before he was looking for a new company to pay his mortgage. If only he’d done a coding challenge, perhaps we could have spotted MDD earlier? As developers, we are the best people to review coding challenge submissions. So what can we expect? Let’s consider some of the types of submissions we will receive as reviewers:-
Good code What do we do when we receive really good code? No brainer. Praise all the positives, give it is great review. On the flip side. It is completely plausible that a bad 'real-world' developer has become good at coding challenges, but it's fairly rare. Bad code Complete spaghetti code, that doesn't work and doesn't make sense. No brainer. Pick out a few of the major negatives, give some constructive feedback and move on. This reminds me of two of my Geektastic reviewing experiences. One submission left 3 of the 4 methods completely unimplemented (I later worked out why, no data was stored in the first method so it would be impossible to retrieve it with the other 3). The other strange submission was a full blown Windows application when the challenge just asked to implement 4 small methods. Neither of those solutions had tests and neither of them worked. So both were obvious, nil points. Average code This leaves us with average code. These are the hardest to mark and unfortunately the most common. It is easy to dismiss this code as bad code, because it isn't as easy to read and follow as the 'good code'. And it isn't a disaster zone like the 'bad code'. So it takes a bit more time. But that is why we are here, that is why it needs our developer brains. We can unpick the code and come up with positives and negatives. To do it justice, we must overlook the comparison to a perfect solution. What we are interested in is spotting little gems of the good stuff and some smells of the bad stuff (see Martin Fowler’s book for an introduction to Code Smells). These little gems and smells are the factors that will give us the bigger picture. For average code, we MUST take interest in the bigger picture. Hiring costs money, so we need to look beyond statements like “it’s crap code”. If we can find a diamond in the rough - a productive developer - this will have a huge saving for the company hiring and this value will be repaid over and over. Productive developers are lurking within average code submissions as well as good code submissions, we just have to find them. We all know that hiring a bad developer can be destructive to the team, code and company so it is good to be cautious. However, missing out on a great developer is potentially an even greater risk, especially at “screen-out stage”. Accidentally screening someone out could mean tens or hundreds more people who need to apply and go through the process before finding someone that good again. Steve jobs once said: “The difference between the best worker on computer hardware and the average may be 2 to 1, if you’re lucky. With automobiles, maybe 2 to 1. But in software, it’s at least 25 to 1. The difference between the average programmer and a great one is at least that“ This tells us that we need to look harder for signs of a good developer. This will mitigate the risk of accidentally screening out that 25x developer. We owe it to the hirer to looks for clues that they might have 25x (or even 5x) potential. Luckily we’re human, so we are able to do this. What are some clues that point to a good developer? To find good developers, we need to first agree on the things that make a good developer:-
Since we are focussing purely on code and challenges, we can distil this list to:- Writes code that is maintainable Maintainable code is:-
*Code maintainability is the primary thing I look for when marking a coding challenge. Sometimes I won’t even run the code because the fact of whether it works 100% or not is as important as whether I could pick up the code and make a quick change to make it work. Code that works There are several ways to know if the code works.
The cheapest proof is automated tests that live alongside the code. Tests that were written by the developer at the time the code was written and are runnable at any given time to prove if the system is still working. There are commonly referred to as Unit Tests. However there are also Integration tests and Acceptance tests, but I won’t get into that here, let’s just call them Tests. Tests prove code works as it is supposed to. Without tests, we can break the code and not know we’ve broken the code until the bug is discovered in production, or if we are lucky, by a manual tester. So a coding challenge solution that is submitted without any tests or with code paths written that don’t have tests, is a HUGE ALARM BELL. As the reviewer, I cannot know the system works without running through various scenarios myself manually, or if I am sensible, by writing tests that cover all the requirements. Here is a simple guideline
Code that works, but has no tests is bad because
Code without tests, hides its intentions. The intention is locked securely in the head of the original developer, which is unfortunately no use in a team environment. How do we know if the code communicates its intention? There is a wonderful quote that I read on Jeff Atwood’s blog, but apparently originates a bit earlier than that Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live. Or put less dramatically. Write code for the maintainer, not for yourself. If we receive some code, we would hope they had done the same for us as it will make our job much easier. The simplest way to spot code that communicates intention are:
Well tested In the context of a coding challenge, this is where we start award points for workings. Even if the code doesn’t quite work, if the is intention expressed in the tests, it means the requirements are documented with the code in a live and runnable way. Runnable documentation covering the whole system means that changing the code is simple a case of changing the documentation (the test) in the right place. This leaves a failing test and allows a targeted code change without fearing the rest of the system will break. That is an example of Test Driven Development (TDD) - For more information, read Test Driven Developer by Example - Kent Beck Coding challenges that were completed using TDD are quite easy to spot. They will have a regression suite of tests that can be run (as this is a side effect of driving the code). Because the The code will be and the code itself will be naturally more API style, with less hard dependencies and with l TDD is not the only way of having well tested (and therefore working) code. A candidate can arrive at this without writing the tests first. But if they don’t write the tests at all then that is a different story. And I do NOT agree that TDD is dead by the way, even if DHH shouts loudly about it. Good naming There is no better way to express intention than using well named variables, objects, functions. Variables named with a single letter like a,b,c,x or the annoying geek habit of foo/bar are of absolutely no use in expressing intention. Also there is no need these days for shortening words, we don’t have much screen width limit, there is nothing worse than seeing variables like fname lname. One very poor name I saw recently was foreach (var itemExists in personList) {...} You’d think that itemExists was a boolean, bit it’s not, it was a string of the person’s name. The code inside the for loop then used that itemExists variable. Obviously the line of code didn’t read well - a simple name fix to person tidies the whole thing up. foreach (var name in personList) {...} We spoke earlier about the importance of writing tests, and tests are a prime example of where good naming is extremely important. Consider the two following test method names:- test_name_splitter() { … } splitting_a_name_with_a_single_space_splits_it_into_first_and_last_name() { ... } To someone new to this type of naming, the second one looks stupidly verbose. If this is you then take the time to think about it. The method name is our one shot at expressing intention to the next person to maintain the code. So it stands to reason that this method name should be a sentence and the sentence should express the intention as well as it can. Since code changes, we can think about a next requirement. Maybe it is to split names with two spaces into first middle and last name - how would we fit that into test_name_splitter() naming style? test_name_splitter2()? an extra assert inside the same test? Or maybe a new test: splitting_a_name_with_two_spaces_splits_it_into_first_middle_and_last_name() { ... } This is where tests start to get powerful. Documentation starts to emerge. Furthermore, this documentation is live with the code, if the code changes then the documentation changes (providing the software maintainer knows what they are doing). Some people like to run document generators over the top of test names but to be honest that is just window dressing, a document isn't needed at code level. This stuff is best kept in the code solely for developers to find and change code with confidence. Comments Comments are something we should talk about. Historically, comments were thought to be good. Phrases like “All code must be commented” used to get banded around and in some companies you could even get official warnings or fired if you didn’t comment your code. In this day and age, we are more likely to hear the opposite. A blanket rule of “no comments allowed” is arguably a better rule as it forces us to name things well and make the code itself readable rather than just throw a comment block on confusing code. The real solution is only use a comment if it really needs a comment and if that comment serves a real purpose. // this is not the obvious way of doing this but the obvious way doesn’t work for reason X GetPerson(); The comment example above serves a purpose. However the following serves no purpose at all. // initialise a new Person object instance Person person = new Person() This comment is not wrong, it’s just litter. It gives no more information than the line it decorates. However, the major downfall of these type of comments is that they can mislead us, take this example from one of the coding challenges I reviewed recently + //if not found return null + if (found) return null; The comment tells me the exact opposite of what it does - clearly the developer flipped it at somepoint, I'm sure it was right when it was first written. But before it was even checked in (submitted), it was out of sync with the code. Fortunately this one was easy to spot, but imagine a large codebase with hundreds of examples of this - what a minefield! Tip: if you think you need to comment the code to explain what it does, consider renaming some of the variables, or wrapping functionality in a new well-named function or class - now see if that comment is still needed. Go on, delete that comment, I dare you. There must be more things to look out for? What else? We've talked about Naming, Comments, Code that works. Why haven't we touched on Design patterns, immutability, value types, object models, proper understanding of SOLID principles? Well, those are all great, but those submissions fall well inside the category of “Good Clean Code that works” and those submissions are going to be easy to spot well before we get to that level of detail. But there must be more bad stuff? Yes, of course there is. But again it's going to be pretty obvious before you need to get to that level. For the record, here are some other smells to look out for:
Enough about the code? Let’s turn our attention to ourselves and think about the reviewer mindset and how to get into it. The mindset we should have as code reviewersSome candidates will think of things that others don’t. They will even think of things that we didn’t think of at all. The more we mark the same challenge, the more we start to learn about it and the more material we have in our pockets to mark people down on. This is grossly unfair and more importantly it is likely do what exactly what we don’t want - screen out good developers. So what is the most important mindset when reviewing a challenge submission? Answer: Empathy. The fact that we, human beings, have been chosen to mark the submission is because we can see things that a machine cannot. We can give the candidate the benefit of the doubt, and put ourselves in their shoes. We can overlook things that need overlooking and we have the potential to see what the developer was trying to do, rather than what they actually did. A machine marker would need some serious AI to try and figure out the things that we can figure out in just 1 minutes of looking over the code. Therefore it is important that we don’t start acting like machines deciding if it is right or wrong. It goes far deeper than right or wrong. The simplest way to do this is to do the challenge ourselves (under the same conditions as the candidates). We do this, not to find the ideal solution, but to make us realise that we will not come up with perfect code. We probably won’t even come up with code that we ourselves are happy with, let alone a reviewer. We will compromise, we might not finish it, and we might even make a complete mess of it - it happens. Days later we will realise something that is better than the way we did it, we will have the urge to change our solution. WE MUST NOT CHANGE IT. Once again, this slowly leads to treating a candidate unfairly and screening out potentially decent candidates. Our brains like to change our memories, over time we will start to believe we made a better solution than we actually did and we will unfairly use this against the candidates we are marking. Luckily there is an easy way out of this trap. Save a copy of our first attempt at the solution including all its shortcomings and make an agreement with ourselves that we will not change that code. Every time we receive a challenge to review, we should take 30 seconds to scan over our own solution. This will snap us back into the open and fair mindset and let us get down from our ivory towers. We owe this to the candidate and to the hirer. We are biased - what should we do? We have strong opinions and we believe our opinions are right. Truth is we might be, but these views won’t always help us when we are reviewing code challenges. Opinions on big topics like SOLID and design patterns are ok. We are often paid for our opinions and experience on these things so this isn’t what I’m talking about. What I’m talking about is suggesting that superficial preferences are better one way that another. Examples
We do not need to comment on personal (or even team) preferences. We can find out later if a developer is open to doing it the way the team does it. It is hard to do, but we have to resist the urge to write the review comment suggesting one way is better than the other way when all it really comes down to is a preference. Let's recap and wrap up Good code probably means they are good - Good clean, well tested, production-like code covering many or all of the requirements means they are good. Short of plagiarism, they are unlikely to be a bad developer who is somehow a super duper coding challenge specialist - and even if that is the case, they will be found out at the next stage. Dreadful code that doesn't work and doesn't make any sense probably means they are bad - Still, be constructive, it is very possible they are a junior and have a passion to learn better ways, even if the code tells you they are bad right now. Average (and sometimes less than average) code doesn’t necessarily mean they are a terrible developer. There are things that can be found in bad code that mean the code is not that bad… e.g. tests expressing intention even if the code doesn’t quite work, good variable and function naming, the decision to use an object model, thoughts about DRY, KISS and the boy scout rule, there are lots of potential things to pick out - Don’t give a blanket ‘no’ just because the code wasn’t perfect or was a little tricky to get your head round. Make our best effort to look for the positive - Hiring is expensive, finding a diamond in the rough is extremely valuable. Even great developers won’t write amazing code under the pressure of a hiring process. Realise that and starting looking for good stuff. When Critiquing code, it is all too easy to get into the negative mode of only fault finding. Don’t let the positives slip by unnoticed. Comment on the positives. List out the positives. When positives are put alongside the negatives, the code submission may not look as bad as we first thought. Empathy - we are not machines, we can put ourselves in the candidate’s shoes. We should have a proper go at the challenge ourselves and keep a copy of that solution forever. We must refer back to it to snap our minds into the openness and fairness needed when marking a challenge. Overlook our own biases - you don’t like underscores, that’s ok, it doesn’t mean the candidate is wrong, and you do not have to tell them you prefer it your way without underscores - even though underscores are evil :). Finally… Be a human Let’s remind ourselves that we are human. We are a human beings reviewing other human beings’ code. Have a human-to-human experience. This is chance to exercise benefit of the doubt, to find those rough diamonds, especially if the code is less than ideal. I’ve done plenty of code submissions where my code was far less than my best - yet I still got hired and still did a great job for the client. I’ve also been screened out and in some cases completely ignored when I wanted feedback - I’m sure you have too. Equally, I’ve seen and reviewed hundreds of pieces of code. I’ve seen some absolute shockers and I’ve seen some pure elegance (and everything in between). I’ve been kind, I’ve been rude, and I’ve even got a review completely wrong to the point of embarrassment - but it happens - after all, we’re human. Let’s make a stand. Let’s review coding challenges the right way, with an open mind. We can do this. We must do this. Our industry needs us. Tell me, what’s your experience of being the coding challenge reviewer? Any tips or takeaways? I’ve love to hear them in the comments. About Andy Davis
Andy is a software developer. He’s been writing software for over 20 years and helps businesses solve interesting business problems, by helping them create the right software. He also helps teams with their developer hiring strategy as he believes that hiring the right people has a multiplier effect on awesome software teams. Design a high quality coding challenge that will test your candidates and not alienate them27/7/2016 Guest post by Andy DavisAnyone who has devised a coding challenge will agree with me when I say: It’s REALLY hard to create that perfect coding challenge that filters out bad developers but keeps good developers interested. Or is it? Well, it turns out, you can dramatically increase the effectiveness of a coding challenge by using one simple rule. And in today’s post I’m going to show you what that is… and exactly how apply it to the next coding challenge you design. But before I tell you exactly what that is, let me tell you a little story... Several years ago, a good friend of mine took his driving test for the 3rd time. He’d done a tonne of lessons and he’d had more than enough practice - he is a good driver. On the day of the test, the examiner asked him to turn left out the test centre. Under the pressure of the test, the instructions didn’t register and he turned right. Now you could assume he failed for not following instructions. Instead some common sense was applied and his driving skills were still assessed - NOT his ability to follow instructions under intense examination pressure. It should be this way with coding challenges. Clean, well written code that communicates its intention, should be a pass even if it is not exactly what the specification asked for. Unfortunately it’s not always the case. You or I might argue that understanding requirements is a big deal. And in some contexts it really is. If the job was to be fed requirements by email, have no interaction with anyone, and deliver within 1 hour, then I would have to agree with you. But how many jobs are like that? And let’s be honest, would we really want to work there? or shall we leave those jobs for elancers? We all know that the real world isn’t like this. You don’t get fed a bunch or algorithms to code up during the job, so why do they send them to us as a coding challenge? It’s because it is easier isn’t it? There is more less a set answer, does it work or not? And potentially an order of complexity (big O, anyone?) Since algorithm questions have mathematically predictable answers, these sorts of tests can even be marked by a machine. This is disrespectful. We took the time to do your coding challenge and you couldn’t take the time to have it marked by a human. If you have your coding challenges marked by a machine - PLEASE STOP. Machine marking is a binary solution to a non binary problem. People are not binary (not matter how geeky we talk). Let me side-track you for a moment and tell you another story… (I’ll get back on topic soon, I promise) At my university, there was a revolutionary new system called Coursemaster. The boffins that be had devised an automatic programming coursework marking system. It became our friend, foe and arch nemesis over those 3 years. At first it was fun, then we realised that our grades depended upon it. That’s right, our degree was marked by a machine (well, part of it anyway). The professors built-in some flexibility, it allowed a 2nd and 3rd attempt. So what did students learn to do? That’s right, Group up and code it together, pooling their 2 spare lives as a sacrifice to the cause. A great application of Pair Programming - others might call it cheating! Anyhow, like it or not, the Coursemaster system was definitely gamed by some students - but let’s be fair here, the system was out to game the students, it wasn’t really very good at marking their code - correctness is a very small part of the battle - so it probably levelled the playing field. Flash forward 15 years and the technology has not moved on! Only last week I was requested to do a test that was marked by a computer… I got 0%....Why? ...I misinterpreted one word in an aggressively timed coding challenge and it left me with not enough time to correct my error. Total frustration. I couldn’t even prove it was good code, it wasn’t, I hadn’t refactored yet, I ran out of time - Now there’s an idea, if the test is timed maybe there could be a refactoring window at the end where it doesn’t allow functionality to be changed but you can improve the code quality and make it communicate (which in my opinion would be a pass if you know how to do that). Let’s cut to the chase... The single biggest factor that will improve your coding challenge So here is is… the killer idea you have been waiting for. I spent the last few weeks speaking to and surveying some of my peers on this topic. One guy stood out amongst the crowd, he’d done as many tests as me, but better than this, he’s also designed several coding challenges and saved a tonne of time for HR in the process. It is with great pleasure that I mention Zeno Foltin, a London-based (polyglot) software developer. Zeno is a high quality developer and I rate his opinion on this topic extremely highly. I met Zeno when both of our companies were working with WorldRemit, a London FinTech money transfer company. They were on a major hiring binge and Zeno took responsibility for making a coding challenge to help them in their quest for hiring good developers. Introducing this stage to their process was vital for them. The quality of candidates through the door went through the roof. I caught up with Zeno and asked him about World Remit coding challenge and his thoughts on how to design a good challenge. We compared notes about the wealth of challenges we’ve done between us and some key points stand out. So here it is, the number 1 stand out idea that you should apply immediately to any challenge you create:- Make the challenge as close to the real world as you can. It sounds too obvious doesn’t it? I wish it was. I can count on one hand the number of challenges I’ve done where I’ve been able to see the connection between the coding challenge and the company once I have started the role. This is the single biggest factor that has kept us interested in various challenges over the years. There is nothing more frustrating that receiving a coding challenge to do, and seeing it is another boring algorithm like checking a string for parenthesis counts, or sorting a list without using inbuilt functions. Generic coding challenges can be a drag, especially if you are applying for multiple roles. When the challenge is closer to the real role, it solves the “interest” problem. Candidates who aren’t that interested in the challenge will naturally not do their best - therefore it kills multiple birds with one stone. One of the best ways of making your challenge close to real world is to make sure the challenge poses a problem within the general business domain of the company. IMPORTANT NOTE: If you take this to too far and make this too domain specific, you’ll end up alienating people and could end up with a coding challenge that only lets you hire people who already work for you - so remember to strike a balance. And remember if you use Geektastic for your coding challenge, there is an opportunity to license this challenge out of multiple companies to use. So if it covers a bit of the tech that needs testing as well as the general domain area, you’ll be onto a winner. If I were looking for a coding challenge to use for a logistics company and there were 2 available challenges but one was about sending parcels and one was about phone bills, I know which one I would chose. To give you an idea. Challenges we’ve liked in the past, all fulfil the following
Several BIG Topics came up in the process of our discussions:
So let’s look at each of those points in detail. Timed challenges Challenges with a hard cut off time. It seems obvious why companies might impose a time limits:
Zeno and I put our heads together on this one and we think not enough thought is put into the downsides of that time limit. For example:
Zeno said: “I don't think timed challenges provide any value in measuring how well the candidate will do on the job. If it's about trying to find out how someone handles pressure, do it in a pair-programming session maybe. I think it's silly to think in our creative industry that putting such pressure on people will bring the best out of them.” So whilst we don’t think timed challenges are completely useless, it just might not be the type of comparison they think they are making - possibly just a superficial comparison. The compromise here is to have a time limit but make it generous enough that the candidate has time to try out a few ideas. The last thing you want to do is cut everyone off mid flow - this would remove the benefit of having a test in the first place. IDE Next up was the IDE. Believe it or not, some coding challenge systems actually force you to write and compile code in the browser - it seems to have become a bit of a fad recently. It’s pretty nifty but it only really works for simple code. It seems to have benefits of scale for the companies using them as the computer tests the inputs and outputs for correctness - but this shows very little respect to the candidate and actually makes their life much harder. It is unable to provide rich intellisense or access to commonly used frameworks or testing tools. There wasn’t really much debate here, we both hated coding in the browser. Developers like their tools and we are no different. Writing code in the browser is not part of our weaponry and will not be something required on the job. Browser based code editors are great for tutorials and quickly trying something out you’ve just been taught but as a candidate testing tool, this should seriously be reconsidered. Zeno said: “They must use an IDE they are comfortable in. I used to ask candidates to bring their laptop with them for the face-to-face interview to do some pair-programming. You want to find out how the candidates work in their usual environment, not how they cope with a new and unfamiliar one. Unless the job is about writing code in the browser?” Compromise might be that you use your own IDE but then upload or paste the final solution. Likeness to the actual job We know that is it impossible to condense the real nature of the job down into a coding challenge. But there really doesn’t seem to be any point in asking a developer to complete something like an in-memory algorithm to return them the correct change from a vending machine. Sure, there are some algorithms in there and order of complexity to think about. However, if the job isn’t about solving little puzzles, it really doesn’t appear relevant. I was once sent a zip file from an online fashion retailer and told to refactor it. Whilst this sort of request is a little bit lazy - at least this is probably closer to the real job. Refactoring legacy code is part of almost every job. Furthermore it was actually in their domain. So, big points to them for this challenge, it was enjoyable. The only downside of that is that I never heard anything back (after spending 2 hours on the test too) - this is one of the biggest bugbears we have as developers. The bottom line around getting challenges close to the real world is:
Case Study: JUST EAT One example we discussed in detail was the JUST EAT recruitment test. Zeno worked with them for a year and I went through the process last year, so I had also taken the same test. We both had a great experience with it and rated it the number 1 test we’d done. You can see their test here:- https://github.com/justeat/JustEat.RecruitmentTest Zeno said: “I really liked the JUST EAT recruitment test because it is simple - list the takeaways for a given postcode using the API. It requires you write a simple UI to present the information, but it doesn’t tell you much more on how to go about it. It is directly to do with their business, it covers UI and logic (I don't believe in back-end or front-end only roles), and it requires more effort than a couple of minutes figuring out or googling an algorithm. A well designed challenge, can give an opportunity for the candidate to showcase the best quality work that they can do without hard time constraints.” The JUST EAT recruitment test is also a very enjoyable challenge. So let’s take a deeper look at it:
It’s great that even at the coding challenge stage, there is a bit of two way flow of information, just by being open to open-sourcing. There aren’t many places where you get to see some of the code before you go there. It is often a bait and switch of great stuff spoken about at interview followed by a shocking reality of legacy code once you start. What can we learn from JUST EAT even if you don’t have a public API? You might be fooled into thinking JUST EAT have unfair advantage of their domain being highly public facing. Even if you don’t “need a balti”, you still know some of the takeaways in your area that they service and when these familiar venues are returned by the API for your postcode. Not everyone can do this as their domain is not so publicly known - but I bet with a little thought, you can be creative enough and throw the candidates a little bone about the real job rather than have them implement a vending machine, phone bill or shopping cart. There is nothing worse than seeing another challenge that isn’t even related to the business sector that the company is in. Another lesson that can be learned from JUST EAT is not be too strict about the solution. If you can allow the candidate to be creative, you will get a better picture than from an algorithm implementation with a right or wrong answer. Sure, it takes more effort to have a developer mark the solution, but that time will be infinitely better spent if you discover better candidates from it. Here are some ideas for you, even if you aren’t as publicly glamourous as JUST EAT
It is a continual learning process In agile software development, we are told to embrace change. Things will change, so build change into the process and not make it an afterthought. Like anything else, accept that you won’t your challenge right the first time. Developers WILL complain. Some good candidates will slip away - refusing to do it. Some good developers will ‘fail’ it. Some bad candidates, will find their way past the challenge and waste precious time that you could be spending with Ninjas Just a joke, there’s no such thing as Ninjas… ...and even if there is you don’t want them ...and even if you think you do, you just alienated them and half the developer community by using the word Ninja, whoops. You should plan to get feedback from candidates taking your test and LISTEN to this feedback. Warning: FEEDBACK WILL BE EMOTIONAL, especially if they ‘fail’. Use any little nuggets inside their feedback to evolve and improve the challenge. Finally, if they have a valid point - reconsider the candidate, I have successfully failed a challenge before and turned it into a pass - you should do the same (under the right circumstances) Continue to screen out cheats and chancers, however, is googling the answer really cheating? After all, good coders are good googlers and part of the job is knowing how to define problems well enough that finding the right answer on stackoverflow comes naturally. Pasting a snippet of code and using it effectively, is definitely not cheating, But I’ll leave this one up to you. Don’t put all your eggs in one basket with a single piece of code. A coding challenge should only be used to screen out really unsuitable candidates. Everyone else progresses to the next stage. And yes, there must be multiple stages if you have included a coding challenge. You cannot base a hiring decision on code alone. Uncle Bob thinks a coding challenge is useful but only as part of an interview process, and values pair programming over and above it - I tend to agree, but that is another story - and probably a long one so I’ll save that for another day Let’s wrap things up (that timer is ticking down) I have covered a few key points here. I hope it sparks some ideas. If you are making a coding challenge or thinking about changing your current challenge, drop me a line, I would be interested to hear about it and be happy to pass comment on your current challenge. I think it is important to remember that developers do like coding challenges. We’re coders, we like to code. We’re not so fond of having someone looking over our shoulder but we do like solving business problems and we do like writing code - so don’t be afraid to have one in your process. Let’s conclude by reminding ourselves of some of the important points that we’ve explored
Good luck in producing a great coding challenge. Personally, I recommend an open and creative challenge with a guide time limit. - where the candidate can show off their creativity and flare, domain modelling and good clean code. But then again, I would say that… because I once failed FizzBuzz. Let me know your thoughts in the comments and I’ll be happy to respond. ______________________________________________ About Andy Davis Andy is a software developer. He’s been writing software for over 20 years and helps businesses solve interesting business problems, by helping them create the right software. He also helps teams with their developer hiring strategy as he believes that hiring the right people has a multiplier effect on awesome software teams. Want to create your own? Perhaps you have a coding challenge you have created already already, or maybe you feel you can create a good one now from scratch. Whichever of those two, Geektastic make it easy to load it onto their platform and manage the whole process of inviting your candidates (who can complete it in their own IDE), submitting it and getting it reviewed by a real human. Hiring the right employees is critical to the success of any business. It is also an incredibly time-consuming process in what is, more often than not, a time-sensitive situation. This is never more true than when hiring software engineers. Developer interviews can often consist of several stages, so filtering out any candidates who won’t make the cut as early as possible is all-important in ensuring enough time is spent on the ones who do. This is why many firms opt for a coding test in the early stages of the software engineering interview process. The fluffier interview stuff is nice and can be valuable, but ultimately it always, or should always, come down to whether the engineer can write the code. The problem is, so many software engineers hate those technical tests. And, to be fair, there are some very good reasons why: 1) No level playing field So, you’re an experienced developer looking for a new role and suddenly you’re bombarded with 5 or 6 requests to do code challenges with no defined time limit: “get it back to us in your own time - next week is fine...”. So effectively it is something you could easily spend 6-8 hours or more completing. Most software engineers realise that any piece of software is never complete. It can always be improved. How do you know how long to give to a challenge like that? Maybe you’ll be up against a developer who is not as talented as you but who has much more time to spend taking the test. That’s not to say a timed code challenge is necessarily the easy answer either. Andy Davis of Tenkaisolutions, a development and agile specialist with over 15 years in the industry, finds timed coding challenges can accidentally promote poor quality: “I don't mind a timed coding challenge but it is something that makes me rush, and if I rush then I make mistakes. “Often you’re expected to be all set up and ready to work on code, but actually this setup time is time that you need to contribute to doing the test. I actually know plenty of great developers who don't have a dev environment setup at home - especially true in the .NET world where historically a development environment is really expensive,” he says. It’s always good practice to make sure that your candidates have the basic development environment setup before their challenge timer starts. Let them know what they need to have installed and running - and in generally keep these pre-requisites to a minimum. 2) Old or poorly constructed coding challenges It’s surprising how many times we’ve heard software engineers tell us about being faced with ancient or poorly conceived code challenges that have very little to do with the day-to-day role they’re applying for. A challenge that’s been around for years: “it’s the one we always use” - but no one ever questions that as roles, responsibilities and technologies within the company change and evolve. What’s the point in spending hours completing a test that will provide almost no insight into the skills you’ll need to make a success of the actual role? The code challenge that’s set is a reflection of the company and the current development team. If the challenge is poor that’s a poor reflection on the company. If the instructions are unclear or ambiguous or show that no one in the current team has actually thought it through from the perspective of the person trying to complete the challenge in two hours then that reflects badly and the candidate’s perception of that development team is going to be tarnished. 3) Reviews are done by a machine Code challenges that are marked as a pass or fail by a machine allow for no benefit of the doubt. Often the candidate’s approach to solving a problem is just as illuminating as the solution itself. As every developer knows there are always ten different ways to solve a problem and some of those approaches will have different strengths and weaknesses. That’s why challenges assessed by a human will always seem more reasonable to a software engineer than those that work from a score calculated by an algorithm. 4) Code challenges are notoriously difficult to send over email This one is really straightforward but a surprisingly common problem. Code challenges compressed into zip files often get stripped out by mail servers, or flagged as a “security risk”. This is one of the key reasons why so few of the code challenges that are sent out by employers get completed – the software engineer didn’t know it was there in the first place. At Geektastic, we spend a lot of time talking to recruiters as well as our global network of skilled developers. Like Robin Beattie, Director of technology recruiter Mortimer Spinks who knows all too well the problems in persuading developers to take code challenges: “We are increasingly experiencing a difficult scenario between the client's desire for all candidates to be tested and candidates (understandably) pushing back on doing multiple challenges, when each of those challenges can take 3-4 hours or more to complete,“ he says. Beattie adds that sometimes as little as 10% of contractors actually take the code challenges resulting in both clients and candidates missing out on a perfect match. While we’re not bold enough at Geektastic to claim we’ve found a solution to the whole problem, we do think we’ve come up with something that works a lot better. The answer? Code challenges that are tailored to the specific technologies required for the role, that allow the software engineer to choose whether to complete it as an open-ended (“next week...”) or shorter time-constrained test. The challenges are created by our developer network and reviewed by them. No robots, real humans who have taken these challenges themselves and have produced reference or benchmark solutions. All challenges are handled within the Geektastic platform, so there is no need for email file attachments. Geektastic also looking to provide a level of quality for the code challenges that will allow us to say to employers for example: “this candidate has already demonstrated their skills in Java and knows how to setup a Spring application. Here is their challenge submission with the reviewers comments and appraisal. We don’t think that there is much need for you to have them complete your own Java / Spring challenge which tests similar skills.”. We think this will be a strong benefit for software engineers: complete a couple of Geektastic challenges and that will qualify from a technical perspective for multiple roles. Our code challenges evaluate both overall technical ability and specific software skill sets. This allows employers to filter their final list of candidates quickly, eliminating hours of time wasted in evaluating CVs and reviewing code challenges. Not quite a revolution then, but definitely a good start. You’ve found a role that seems a great fit and passed the code challenge, now it’s time for the interview. Over the years the management team at Geektastic has hired hundreds of developers. In that time we’ve found there are certain types of interview questions that will get discussions going and allow a skilled interviewing team to see how much you really know and how you might fit in. If you’re a developer looking to prepare for your next interview please take a few minutes out your busy day. 1. Why did you make decision X in your code challenge submission? The very fact you’re interviewing face-to-face means you passed the code challenge, the next step is to make sure you’ve come prepared with a rationale for the decisions you made and be ready to talk them through. “Oh, I wrote that? I don’t remember”, could lead to some difficult moments. While it’s unlikely you’ll be expected to write out perfect code on a whiteboard (many developers use IDEs and this is a reasonable mitigation for imperfect syntax), being able to justify your choices and the ability to rationalise and compare alternative solutions is particularly important in agile development teams. 2. What happens between you typing a URL into your browser address bar, hitting enter and seeing a web page? This is an example of a broad type of question that could lead in any direction. You can talk about all sort of things: maybe DNS to start with – what are DNS servers? How do they work at a high level or a lower level? Lookups. Primary and Secondary servers. Caching. TTLs. Change propagation. DNS records. A records. CNAMEs. MX records. And that’s just DNS. We’ve not hit a load balancer or web server or app server yet. Although “there is a DNS lookup that tells the browser the IP address of the server that will respond to this request” is a good starting point, skillful interviewers might lead you down one or two rabbit holes to see whether you know more than the bullet point. Thirty seconds into your response you might be onto TCP, HTTP, transport layers, SSL, certificate chains – who knows? You’re not expected to be Google and if the role involves building JavaScript libraries, a gap (or abyss) in your knowledge of transport layers is likely not going to kill your chances, but it’s always good to show that you know there’s more going on than just how much of the internet npm downloads when you build that 50 line JS file you’ve just written. If you are a JavaScript engineer working on client-side code, it would not be unreasonable to expect that you have some depth to your understanding of how a web browser works – since that’s effectively the platform that you’re writing your code for. 3. What are the things you should consider if you were writing your own database server? Most software engineers, particularly those working with backend applications, will use data stores. But do you understand something more about the application underlying that nice GUI interface that you double click on? How would you write an application like MySQL or MongoDB or Neo4j? Questions like these could lead into general database principles such as ACID; SQL / NoSQL; different types of NoSQL; transactions; logging; file storage; blocking/non-blocking I/O; threading; indexing; sharding; query optimisers etc. As with question two, the interviewer is asking an open-ended question in order to investigate the bounds of your knowledge. 4. How does role Y fit in with role Z within the development team? Many developer interviews will involve meeting lots of members of the team – in start-ups especially it’s not unusual to meet upwards of eight different people, all keen to check that you understand how your work will influence their work and how you’ll work together. Do your research, spend some time before the interview thinking about how the IT flow works in the company and demonstrate you have an idea of how the pieces fit together. 5. What environment are you looking for? Hopefully you’ve done some research and you know the company only has 10 employees or it has a development team of 150. If you’re moving from large to small or vice versa it’s probably good to show that you know what you might be in for. If you like a nice subsidised canteen and powering your machine down at 5:30pm, then a 10 employee company might not be for you. Be prepared to talk about the environment you’re expecting and how environments you’ve experienced in the past have worked. How do you see yourself fitting in? Do you understand how IT projects tend to work in smaller or larger development teams? It may be obvious, but if interviewing at a startup, don’t make the mistake of thinking there are no rules. All smart businesses, no matter how laid-back and new, will want you to adhere to certain principles and take responsibilities for your choices. Similarly, a financial institution for example, will be looking for you to show you can follow strict protocols and understand how the work you’re doing is often directly quantifiable in financial terms for the business. Some bonus tips:
You have just spent two months (and a lot of money) hiring the next member of your dream team but don't forget, this is a journey and it's only just begun. There are few things more intimidating than the first day in a new office (especially if it's your first job). It's so easy to forget what it's like and just assume that your new hire will just get on with it and the team will assimilate them into the fold. Fail to onboard a new hire properly and you risk them leaving as fast as they joined (especially if they are a software engineer, you can bet your bottom dollar they will be being tapped up by unscrupulous recruiters as soon as they update their linked in profile with their cool new role "Hey Sue, if it doesn't work at XYZ Ltd let me know, I can find you somewhere far better") There are some golden rules to follow which need to be applied regardless of age and experience but special consideration should be made when on-boarding graduates or those where it's their first job.
Some golden rules - and these apply across the board. 1. Be honest about the role. It's all very well selling the company culture, but overselling it to the extent that the reality bears little resemblance to the interview process will cause trust issues you will never recover from. 2. Introduce them to the rest of the team. This sounds so obvious but the number of jobs I have started and just been left to say hello to the guys sitting next to me. Encourage team members to arrange face to face sessions where they can explain their roles and how they will interact in the future. 3. Have their desk all set up. Unless you are Amazon and your on boarding involves building your own desk from a door (not sure if this is even true but sounds like the sort of thing they would do!). It shows how little you value someone if you are not even ready for them to start. 4. Set up 1:1's. To be honest this should apply across the board but is especially important for new (especially younger) team members. Yes, you are super busy and they will be too, but setting aside 30 mins for a catch up is invaluable. You might think that you know what's going on with everyone in the team but it's not until you sit down, 1:1 and ask direct questions that things come out in the open. 5. Explain the company line. The contract says 9:5 but half the team slope in at 10:30 - what's with that? It's vitally important to explain the company rules; hours of work, dress code, protocol for being late - it could take the form of a handbook but a quick informal chat is just as good. It's much easier to have these conversations up front rather than 3 months down the line when bad habits and frustration has been built up because your new starter assumed things incorrectly. Want to try something different – try starting your new hires on a Friday – as Fastcompany recently reported, this is what Austin based tech company Spredfast have introduced. "People are running around trying to get caught up from the weekend," says Sam Baber, vice president of talent and development for the social media marketing. "When you’re starting someone new at your company, you want an atmosphere with less chaos." Makes sense, people are generally in better moods on a Friday – first impressions count and your new hires will feel better too having probably had Monday to Thursday off rather than leaving their last job on a Friday and starting the new one on a Monday. As a start-up, taking on technical debt is as inevitable as a fresher taking out a student loan. But like a lot of students, this is the first of many debts that start piling up and unless managed properly can cripple future growth.
Here's a great quote from Martin Fowler "Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future" Some things to consider about technical debt which make it even more scary than financial debt. 1. There is no crystal ball - you can't predict the true cost of the debt you are taking on. 2. Technical debt suffers from compound growth with an increasing rate of interest. What might take a few weeks to fix in the early days can easily take months later on. 3. Some debt is never paid off. 4. It becomes the excuse later down the line. Teams change, the liability shifts over time. Those that took on the debt leave and newcomers are left carrying the loan, they then use this debt as a (legitimate) excuse for long delivery times. All said and done, as a start-up you need to take on technical debt. You need to ship product quickly, you need to test the market for product market fit, you only have a small team so you can't deliver the perfect clean design, you need to throw in libraries (although not like they are going out of fashion) rather than building from the ground up. But all of these decisions add up and at some point you are going to need to retrench. I tried explaining this to my wife; since we are having some building work done I used the analogy of building a house. You can build a house very quickly. You can start building the walls, put in some doors, windows, add a roof. You move in and you love it. At some point in the future you want to have a family, you need more space so you look to build an extension. You have a bigger budget now so you hire architects, designers, structural engineers etc who take one look at the foundations, the electrics, the materials used in the original construction. And then come the deep breaths, the chin-stroking, the worried looks and the excuses. "Well 'love', if you'd built the house properly in the first place this job would be a piece of cake, but you don't have any foundations, your walls are made of cheap breeze blocks and your original builders were probably better suited to lego" And where are the original builders to help out, they are long gone, off working on a new project. You have two choices. 1. You build on top of the original poor construction. You add another floor, this new construction is all properly constructed and architected but it takes twice as long to build as it should (but not as long as the alternative). When complete you still get issues, walls move, electrics keep blowing, the original plumbing starts to leak and you are constantly calling in support. 2. You rip down the original build (or at least some of it), ie you start again. Of course this increases your cost and your project timelines. Once you start again you build the foundations, you use the correct size boiler, the right fuse box and use bricks not breeze blocks to support the new size house, and even more importantly this supports further growth without leaving you with the headache of constant calls out to the plumber, the electrician and builders. From our experience there is no hard and fast rule, but at some point you are going to have to bite the bullet and pay off some of you technical debt (or at least part of it). This will mean product development grinds to a halt, but in the greater scheme of things you'll be in a better position afterwards and when you look back, you'll be grateful. At first glance you might just say "Well that's easy, take their salary (e.g. £50,000) and multiply by the average recruiter revenue share of say 20% and bingo it's £10,000!" But it's not as simple as that, when you look at the internal costs they quickly add up. This post looks the hiring flow and tries to put some costs against the different steps to try and get a breakdown of the costs in hiring just one software engineer. You'll be shocked at number 6. We have worked on an internal cost of £100 /hour/person. Clearly if you feel your cost is higher / lower then you'll need to adjust accordingly. Recognising you will have different costs depending on whether you use an agency recruiter or do it direct (in house) we've broken things down for each. 1. Create a proper job description. It's worth spending a few hours on this, even if you work from a template to start with - it makes a HUGE difference in response rates to your ads / will help the recruiter no end. We have seen some shocking JDs that don't differentiate the hirer / explain the role at all. Spend = £200 2. CV screens - You can expect around 30 CV's per role - any more and your recruiter(s) are sending over too much fluff - say 5mins per cv = £249. If you are taking the direct approach (job boards / ads) you will be sifting through way more so 100 is probably not far off the mark £830 3. Phone screens - (these should really be called 'sales pitches' - from experience this is more the snr team pitching the business to the engineer to get them sold in and engaged enough to take a tech screen) You will probably need to do at least 18 calls (60% of CVs) at 30mins via a recruiter £896 or 40 calls (40% of CVs) if going direct £1,992 4. Code Challenges - to save a lot of lot time on the next two steps we and many other tech teams use code challenges to evaluate a candidate's tech skills (please have a read of our blog on code challenges here). Assume of phone screens you carry out, you will send out Code Challenges to 80% of them (you will issue 14 if going via a recruiter, 32 if doing it direct). You'll spend 15mins faffing around with the recruiter/candidate sending zip files back and forth (unless you use our platform :). Of the ones you get back (60% - if you are lucky - asking an in demand engineer to spend 2hrs on your code challenge is a big ask) it takes 45 mins to evaluate the code submission by your engineers, then another hour with the team (let's say 4 people) talking through candidates and deciding who to take through to the next round of phone interviews. Costs add up to £3,351 via recruiter or £6,957 direct. 5. Phone Interviews - aka tech follow ups - 30 mins call with 3 engineers from hiring team. Circa 50% of those that take the tech screen pass and get taken through to a phone interview. £645 via a recruiter and £1,434 direct. 6. Final Face to Face interviews - From the phone interviews you should aim to get to 3-4 final face to face interviews. We always bought in 5-6 team members and these final interviews took up to 3hrs - this is a huge overhead but of course ultimately worth it (and justifies the need to do code challenges to reduce the volume of final face to faces) = £7,200 These internal costs add up to £12,541 via a recruiter and £18,613 direct. And don't forget, this is on top of the revenue share you pay to the recruiter or the ad spend on your job boards etc. |
AuthorRick Brownlow Archives
March 2017
Categories
All
|