Interviewing Engineers

  This past week, I participated on the interview committee for three candidates at our company.  I've been on both sides of the interview process, both at big and small companies, and am convinced that the industry does a poor job of identifying stellar candidates.  Too many unqualified candidates are able to slip through our preliminary screens because people will fill out their resume with buzz words which they don't really understand.  Too many people slip through phone screens because the questions asked are too simple, and don't adequately separate good from the bad.  Too many good candidates are screened out by questions testing development in a way that doesn't match real programming.  Finally, too many bad candidates are simply given a pass because a binary evaluation  in a vacuum of information favors the candidate.  So what kind of steps could the industry take to insure the best candidate gets the job?

Good candidates should be found by looking for the right things

  First, where do we find our candidates?  Companies pull from a wide list of sources, some of which are more reliable than others.  Recruiting services and job fairs represent the shotgun approach, leading to the full gambit of qualified and unqualified candidates.  It is far too expensive to adequately reduce this list to candidates who would be a good fit, so they should just be eliminated.  Niche communities of programmers represent candidates who get involved in their craft at a greater depth, and could serve as a more effective source of recruiting.  Job boards on developer sites like StackOverflow, or from communities like TopCoder have a built in bread crumb trail to help validate a candidate before ever even looking at their resume.  Good candidates stick out, and that should be your first screen.  Another primary source of candidates should be your dev evangelists.  Engineers who attend tech talks outside of working hours have demonstrated a level of commitment beyond the average developer.  And if they are going to a talk about your own product, they probably already like you, and are ahead on the learning curve.

Good candidates should be able to make good designs

  Second, what questions should we ask them during a screen?  I agree with Jeff Atwood on this point; the phone screen should weed out candidates with gaping holes in their knowledge.  There are just some broad areas of knowledge that all good software engineers know something about.  First, algorithms.  I'm not talking about remembering the details of an algorithm taught in a classroom, but the on the fly problems like reversing the order of all words in a string.  Candidates should be able to analyze the efficiency of their algorithms, and a good candidate with a  handle on different types of data structures will know the trade-offs    that lead to more optimal solutions.  Next, candidates need to show they know how to design systems.  Ideally, a candidate has a grasp of common design patterns, so if you give them a problem that just screams for a solution, good candidates will lean on that past experience.  Candidates without explicit experience with design patterns still can preform well, since design patterns just give a language to ideas that all engineers should be familiar with anyways.  Third, scripting and regular expressions.  Parsing data turns out to be an important part of so many tasks, so a good candidate will have these tools in their belt.  Fourth, bits and bytes.  If you ask a candidate to find the number of bits set to 1 in an integer, they should be able to whip through a solution that picks them out.  If you have a candidate who does well with these questions, it will go a long way to assuring that your candidate is qualified.

  Next comes the onsite interview.  I'd have three goals of things to learn about the candidate at this stage.  First, does the candidate have any specialized knowledge relevant to the position you are hiring for?  For this, you should get an interviewer who knows the technology stack in question, and can answer.  Second, can a programmer work through a more complicated problem, something that is going to take a significant amount of time?   Third, are they a good culture fit with your company?  The biggest issue I have with current industry practices is that most developers are asked to solve problems at the whiteboard, which in my opinion bares little resemblance to how a developer actually writes code.  I would ask a candidate to bring in their laptop, and solve the problem using any environment they want to use, and any online resources they need.  The problem asked can't be easy to solve, so I might pick a question similar to those used in a TopCoder SRM, or in a round of Google Code Jam, which are intended to be solved in about a half hour.

Good candidates should not be evaluated in a vacuum, but against other candidates

  Finally, comes the hire/no hire decision being asked of the interviewer.  In most instances, they aren't given any metrics against which to compare for evaluating a candidate, other than their own gut feelings.  And software engineers have a track record about getting things wrong when they go with their gut.  That is a big motivation why the Agile process was developed, as a way of giving engineers mechanisms for estimating complexity of code by comparing against the complexity of past stories.  I think similar lessons could be applied to the interview process.  If interviewers were instead asked to rank candidates they had seen before, or assign planning poker numbers to the candidates, we would gain a better understanding of how an interview experience went.  Due diligence means each interviewer should be interviewing a wide range of candidates before a final hire no-hire decision is made for a position.  It may be a timely and expensive process, but I assure you hiring the wrong candidate is much more so.

  While there may not be a foolproof way to identify good teammates, there are many steps that can be taken to make the process more efficient.  Limiting your sources to places where more good candidates live, vetting candidates based on their online bread crumb trail, asking relevant questions that test their skills in a way that compares well to how real development is done, and evaluating candidates against other candidates are just a few ideas I have about how we can approach those goals.  Better developers means better teams means better products.  This is all something we should strive to provide.