Purpose: every engineer we hire should already work AI-native, or be provably capable of getting there fast. This is now a scored dimension, not a nice-to-have. It sits inside the existing process, no extra interview stage.
1. What we are actually measuring
Not "do you use AI." Almost everyone says yes. We measure depth of integration into their real workflow. Five levels, scored by recruiters with no deep tech needed:
| Level | What it looks like | Verdict |
|---|---|---|
| L0 None | Doesn't use it, or "my last company blocked it" with no personal use | Below bar |
| L1 Copy-paste | Opens a chat window in the browser, pastes code in, pastes answers back | Below bar for mid and above |
| L2 In-editor | Uses an agentic tool in the IDE or terminal (Claude Code, Codex, Cursor, Copilot). Lets it read the repo, run tests, make multi-file edits | Baseline for mid |
| L3 Configured | Has shaped their environment: project instruction files (CLAUDE.md / AGENTS.md), skills, subagents, MCP servers, hooks, custom commands, permission settings. Has opinions on when not to use the agent | Target for senior |
| L4 Multiplier | Builds workflows others use: CI automation, review agents, internal tooling, prompt and skill libraries for the team | Lead or staff signal |
Key distinction to listen for: L1 and L2 candidates describe AI as something they ask. L3 and L4 candidates describe it as something they configure and delegate to, and can tell you where it fails.
2. Recruiter screening call (5 minutes, non-technical)
Ask these as normal conversation, not a quiz. Note verbatim answers into the scorecard.
Q1. "Walk me through how AI shows up on a normal working day for you."
Weak: "I use ChatGPT when I'm stuck." Vague, no tooling named.
Strong: names a specific agentic tool, describes a loop (plan, let it work, review the diff), mentions what it's bad at.
Q2. "Which tools, and how are they set up?"
Listen for: Claude Code, Codex, Cursor, Copilot, Windsurf, Aider. Tool brand doesn't matter, setup depth does.
Follow-up that separates real from performative: "Have you customized it in any way for your projects?" A real user will immediately mention instruction files, rules, commands, or MCP connections. A performative user will say "not really, it just works."
Q3. "Do you use skills, MCP, or subagents?"
You don't need to understand the answer. You need to know whether they can answer. Skills are reusable packaged instructions or procedures the agent loads when relevant. MCP is connecting the agent to external systems (Jira, DB, docs). Subagents means delegating a sub-task to a separate agent context.
Score: can't define any is L1 or L2. Defines one and gives a concrete example they built or use is L3 or above.
Q4. "Where has AI produced something bad, and how did you catch it?"
This is the highest-signal question in the set. Anyone who genuinely ships with agents has scar tissue: hallucinated APIs, plausible but wrong refactors, tests written to pass rather than to verify, security-sensitive code accepted too fast.
No answer, or "it's usually fine," means they aren't reviewing output. That is a hard flag regardless of claimed usage.
Q5. "Any personal projects or workflows you've built with it?" Optional but useful for junior candidates, where L4 potential shows up before employer exposure does.
3. The screen-share walkthrough (technical interview, 10 to 15 min)
Add this to the front of the existing technical interview. Send it in the invite so candidates can prepare and choose a safe repo:
"We'll ask you to share your screen for around 15 minutes and walk us through your own AI development setup on a project you're comfortable showing. A personal or open-source repo is ideal. Please don't show client or confidential code."
That NDA line is mandatory. Never let a candidate open a former client's codebase.
Prompt to give them: "Show us how you'd add a small feature or fix a real bug in this repo, using your setup the way you normally do."
What the interviewer looks for:
Fluency of the tool itself: keyboard speed, not fumbling through menus
Do they give the agent context, or one vague sentence? Do they have a plan and review step, or accept everything blind?
Is there an instruction file in the repo? Is it maintained and specific, or boilerplate?
Custom commands, skills, hooks, MCP connections, permission config
Do they run tests and read the diff before accepting?
Do they intervene when it goes wrong, or keep re-prompting hopefully?
Green flags: narrates why they're delegating this and not that; corrects the agent mid-task; has an opinionated setup they can justify.
Red flags: default everything; accepts diffs unread; can't work without the agent (breaks when asked to reason about the code themselves); over-delegates trivial work.
4. Scoring and the bar
Add one scorecard field: AI Fluency, 1 to 5, mapped to L0 through L4. The interviewer must paste one sentence of evidence, not just a number.
| Role | Minimum | Target |
|---|---|---|
| Junior | L1 plus strong learning signal | L2 |
| Mid | L2 | L3 |
| Senior | L2 with clear L3 trajectory | L3 |
| Lead or Architect | L3 | L4 |
Treat as a hard fail at any level: actively dismissive of AI tooling; claims heavy usage but cannot demonstrate it on screen; accepts AI output without review.
5. Fairness rules, non-negotiable
Score the setup, not the brand. A deeply configured Cursor or Codex user is L3. Tool-specific familiarity is trainable in a week, the underlying working model is not.
Don't punish blocked environments. Many enterprises banned these tools until recently. If a candidate had no access at work, ask what they do on personal projects, and weight the answer to Q4 and adaptability instead.
Consent and notice. Screen share is announced in the invite, never sprung. Recording only with explicit written consent.
This is a dimension, not a gate. An exceptional engineer at L1 with visible curiosity is still hireable. Flag them for accelerated onboarding rather than rejecting.
Interviewers must be at L3 themselves. An L2 interviewer cannot assess an L3 candidate and will over-reward buzzwords. Calibrate the panel before this goes live.
Rollout
Recruiters get a one-page crib of Q1 to Q5 with the good and weak answer patterns from section 2.
Add the AI Fluency field to the scorecard.
Add the screen-share paragraph to the technical interview invite template.
Panel calibration session: walk the technical interviewers through the rubric and one mock walkthrough.
After around 15 candidates, review score distribution. If everyone scores 4 or 5, the questions are too easy or the interviewers aren't probing. Recalibrate.