A San Francisco startup has a way for employers to test a job candidate’s ability to write computer code.
Here’s how it works. An employer or recruiter either chooses a ready-made “challenge” or creates their own.
Let’s say the challenge is that candidates have to write a program showing all the prime numbers up to a million. Candidates are given a set amount of time, such as two to four hours, to complete the challenge. The CodeEval tool then can rank them in a “pass/fail” sort of way, or by percentages, depending on what the challenge was. In addition to seeing whether candidates passed the test or not, the employer can also take a look at the code that candidates have written. “There are million ways to write a program,” the site’s founder, Jimmy John, says. “You can write a program elegantly or write it in a really crappy way and still get the right answer.”
CodeEval runs a cheating-detection script to see if two candidates’ responses are strikingly similar, a sign they could have collaborated.
John hoped that professors would use the tool to create assignments and automatically grade them. But he’s removing the support for this function, because the response, he says, was “lukewarm. Nobody really wanted to use it.” That’s probably because, he says, it cut into teaching assistants’ jobs, and those jobs are baked into the university system and funding.
Jimmy John (who says he goes by Jim because Jimmy sounds too much like the sandwich shop) is backed logistically but not financially by a San Francisco-area startup incubator, where he can share space, have mentors, and chat with other startups. “It’s like an ecosystem,” he says.
He has a master’s in computer science from Arizona State, and started this off as a weekend project. Over time, he realized that evaluating IT candidates is “a huge pain point that everybody faces,” and last August/September launched it full time. Startups have shown interest, big companies not as much. “For big companies to go and evaluate a new tool and bring that into the process takes time,” he says.
Right now, it’s free, and a few people are trying it out. When the free beta ends, the monthly fee is likely to vary based on whether easy, medium, or difficult challenges are chosen, and perhaps based on whether someone uses a challenge already in the system or creates their own. Probably, it’ll start around $49. CodeEval has partnered with one job board and is looking to partner with others. He’s also using the site to create what he calls an “ecosystem of developers” who go in and compete against each other.
Zachary Gottlieb, founder of the q-and-a site Realpha, has used CodeEval to evaluate the skills of a developer he was thinking of hiring (and did hire). Actually, Gottlieb has used CodeEval more than once, and though he says it’s just a piece of the hiring puzzle and not comprehensive, he’d likely keep using it, even when he has to pay.