90% of AI Engineers I Interviewed Think a Better Model Fixes a Broken Pipeline

We built a fake company run by nine AI agents, hid 35 problems inside it, and told six candidates almost nothing. Counter intuitively, the strongest predictor of a bad result was how confident the candidate sounded.

We did a lot of hiring interviews and testing to hire someone new to help bolster our OpenClaw fleet engineering capacity.

Resumes could not tell me what I needed to know. Certifications turned out to be worse than useless. So we built a system and let people work on it.

The system

The system was a working replica of an OpenClaw AI-agent fleet running a fictional 15-person outdoor-gear e-commerce company. Within each system we loaded it with 9 AI agents handling all of the business day-to-day operations. The system itself also included ~90 days of generated history. A 21,000-row billing ledger, theoretical run logs, generated conversation transcripts, a mock 197-commit git history, and 16 active and running cron jobs. For the test: each candidate got their own clone of this on their own fully permissioned docker container on one of our EC2 instances.

The audit planted a lot of easter eggs, some harder to find than others. Including a few traps and a few “hidden in plain sight” eggs that made them in some ways harder (like things that don’t fail but are still major problems) or something that says it is broken, but then it says it fixed itself, but then if you look deeper, it actually didn’t…

What I did not tell anyone was if there was one problem on the system, or 10 or more… Because I fully expected [and wanted] people to use AI, it was really important to think through how I could create a puzzle that AI alone would not solve without someone who knew how to prompt and direct it. I think because I have been using AI daily for several years now it has also become pretty clear what the limits of it are, and even though we like to think models like Fable are extremely intelligent, tricking them and getting them stuck on superficial problems is not all that hard once you’ve played with it for a few hours.

What candidates were told

I should say that for the record, all I basically said was: Here is an OpenClaw system, audit it.

If a candidate read our job description they would see we need someone to do a pretty serious audit, so that alone should have been enough context to help them know how deep it should be checked.

No timebox was given, intentionally and it was not a paid job. Candidates were informed everything they did on the server was time stamped and logged for performance measurement purposes.

The funnel and the scores

Sample size: ~100 resumes read, 25 1st round interviews, 6 people sent the test.

We tracked eggs found but more importantly we did scoring on the audit, their methodology and the issues found. Not all issues were equal in importance or complexity. Some candidates also found unintentional eggs that we did count towards their scores.

Scoring went from 40 to 89 out of 100, and was distributed pretty evenly between these two ends.

Two people re-tested and went from 53 > 62 and from 71 > 80. Not everyone opted to retest and we did not offer the retest to someone if we felt they were disqualified from their performance on the first test.

Our worst performance was 1 out of 35 eggs found, and our best result was 25 out of 35 eggs found. No one hit 100%.

We got back everything from 5 bullet point audit reports to 30 page document audits.

Confidence ran inversely to score

The candidates in general seemed to do much worse the more confident they seemed. I found an inverse correlation between a candidate’s confidence and their test scores, which was the predominant indicator of their performance. Candidates who thought I was a more experienced OpenClaw developer than them, scored much higher than candidates who assumed I was not very experienced at OpenClaw or AI engineering; and they were the expert in the room. This lack of effort extended to not only how many issues they found, but how they presented their findings. Candidates who said they had the most experience tended to write low effort reports.

Confidence is not something I can say is objective. I noticed for example if I asked a question about local inference experience, instead of telling me their experience, I would be told lots of different GPU model and AI model names and numbers, stats on ram etc. My impression was I was supposed to be impressed by the specs they were quoting. I had a different candidate who struck me as confident, ask me mid way through the call if they could use “technical terms” and I would understand them, clearly not understanding that I was a technical person interviewing them.

The people who were more humble would say things like “I read your blog and I think I can learn a lot from you” and then proceeded to score the highest test of anyone on the practical.

Why did it go this way? It felt like the most confident candidates probably just assumed a few problems was all I was checking for, and had no idea this person who was hiring them meant serious business when he built the trial system.

The retest: priming raises effort, not coverage

I spent a full day worried that my test was not well designed because of this inverse correlation. Meaning surely candidates who are confident should score higher? So I decided to build a second, new variation of the test and offer it to candidates who now knew how hard it was, to see how well they would do.

So we did some re-tests.

Everyone who re-tested scored better, but all the same holes in their process appeared in each person’s re-test. Meaning if someone never noticed API costs changing on a specific date in the first test, they didn’t notice it on the second test either. Someone who forgot to look at a specific log the first time around, still didn’t look at it on the second pass.

So my theory that priming candidates with better expectations on the test complexity would result in better scores, was right in the sense that it did improve people’s performance, but it still didn’t result in dramatic changes in the patterns of how well or poorly specific people performed on the test. More awareness just meant better audit reports, more bugs found and reported, but within the same general nature and areas where the person was already looking. Basically a 9 out of a 100 point increase in performance between test1 and test2. Not a jump from 40 to 90/100.

Telling someone the test is hard makes them try harder, not more things. Effort increased but coverage remained relatively the same.

How good is the driver at the seat of the bus?

How well did it work? For testing motivation, very well. It also shows me who has what standards inherently. What it doesn’t show me though, is how good the driver is at the seat of the bus.

What I mean by this is, you can be very good at asking AI to do an audit. You can provide a lot of context, perhaps you provided not just the system, but also the job description (and most candidates did not) and that increased the quality of the types of things that were audited and found greatly. You perhaps also asked your AI to go a few more rounds than most people, and used a top model like Fable when someone else only used Sonnet.

You get back an excellent report because you pushed AI to do more, with more and for longer. But does this mean you are the best person to hire?

Fundamental theory is the biggest gap

What I found on the subsequent round of interviews, where I pushed harder on theoretical questions, is that theory is generally weak among most if not all candidates.

More broadly I noticed in my final review that almost no one can answer theoretical questions, even people who have passed certifications that relate to that theory.

Fundamental theory is the biggest gap by far. 90% of the people I interviewed answered that the way to increase the performance of a multi-step process that Opus runs with a 95% success rate is to start using Fable. (me: terrified)

Here are the questions most candidates could not answer across the board:

• If you have a multi-step process using the LLM Anthropic Opus and only 95% success rate, what can you do to improve this to a higher success rate?
• If your MEMORY.md file is 600 lines long, and growing by 50 lines a day, what should you do?
• When do you use a hook and what kinds of hooks are there?
• How do you determine which process you should send to a local model vs. a smaller model like Haiku or a heavier model like Sonnet?

To name a few…..

Who you are actually looking for

In actuality, I really don’t want to hire someone who looks up to me. I want to hire someone I can pass the work to confidently, and know they will take care of it. I love helping, but that is not why I am hiring someone. So it has been confusing to see that the people who sounded like they will take charge, put in much lower effort. And the people who seem like they want to learn, put in much more effort and do a much better job.

It made me realize that the ideal here isn’t necessarily the person who already knows how to do the job, it is the person who is almost ready to do the job, but not quite ready yet. That’s a very narrow window.

Once someone is in, that’s when you see if they have drive, a desire to learn and they take ownership. Arguably the most important long-term skills of anyone you are hiring because it means they will stick, will grow and will be responsible.

Where we landed

Yes, we hired. They showed a strong eagerness to learn, and very high test scores. We are now in the work-start phase and I am really excited to see how good they are and can become.

If you are hiring for this

OpenClaw experts. Are extremely rare. And most who think they are qualified have little to no experience, and lack theoretical knowledge which they overcome with AI.

Build a test. A real one, with real history and real noise. You cannot assess agent operators from conversation, and you certainly cannot assess them from a resume.

Plant traps, not just bugs. The things that don’t fail but are still major problems, and the things that claim they fixed themselves and didn’t, are what separate an operator from a prompt writer. The best traps are the ones that are designed to trick AI, who is driving the bus?

Do not tell them how hard it is. Their calibration of how deep to go is partly what you are measuring.

Score methodology, not hit count. 25 of 35 found matters far less than how they got there and what they did with it. How good is their prioritization? How clear do they articulate the issues? Do they make false assumptions?

Then interview the theory separately. The practical measures how well they steer AI. The theory measures whether they know where they are going. They are not the same skill, and in our sample they did not correlate. Without strong theory you have someone pressing the enter-key at the helm, not an AI engineer.