Guides
AI Strategy
How Do You Evaluate Your Engineering Team’s AI Skills?
CH2 Solutions
.
9
min read
The question isn’t whether your engineers use AI. It’s whether they know how to use it without giving up the judgment that makes them valuable.
LEADERSHIP CHALLENGE
How can technology leaders determine whether their engineers are actually skilled at working with AI—not simply using AI tools?
Most engineering organizations know their teams are using AI. What they don’t necessarily know is whether they’re using it well.
An engineer can use Claude, ChatGPT, Copilot, Cursor, or another AI development tool every day and still lack the judgment required to use it effectively in a production engineering environment.
That creates a new challenge for technology leaders.
AI adoption is relatively easy to measure. AI competency is not.
As AI becomes embedded in software development, leaders need a better way to understand whether engineers can use these tools to improve their work without outsourcing the thinking, judgment, and accountability that make them valuable in the first place.
Executive Summary
AI proficiency is not the same as tool proficiency.
Knowing how to use an AI coding tool tells you very little about whether an engineer can frame a problem correctly, provide the right context, evaluate what AI produces, recognize when it is wrong, validate the result, and take responsibility for the final solution.
Those are increasingly important engineering skills.
For technology leaders, that means AI capability should not be evaluated through self-reported tool usage, certifications, or knowledge-based quizzes alone.
A meaningful assessment should put engineers into realistic situations and evaluate how they work when AI is available to them.
The objective isn’t to identify who uses AI the most.
It’s to understand who uses it well.
Problem Framing
Defines the problem, constraints, and desired outcome before turning to AI.
AI Orchestration
Gives AI the right context and direction and uses it deliberately throughout the work.
Engineering Judgment
Recognizes weak assumptions, flawed approaches, and technically incorrect AI output.
Verification
Tests, reviews, and validates AI-generated work rather than accepting it at face value.
Ownership
Understands the solution, can explain the decisions made, and takes responsibility for the result.
CH2 evaluates AI capability through realistic engineering scenarios rather than tool proficiency alone. We look at how engineers frame problems, work with AI, apply engineering judgment, verify outputs, and take ownership of the final result. The goal is to identify engineers who can use AI effectively without giving up the judgment that makes their work valuable.
Key Takeaways
• AI adoption and AI competency are not the same thing.
• Tool proficiency is a weak proxy for an engineer’s ability to work effectively with AI.
• AI skills are best evaluated through realistic engineering scenarios, not self-reporting or knowledge-based tests.
• AI competency and engineering competency should be measured separately.
• As AI produces more implementation, engineering judgment, verification, and ownership become increasingly important.
AI Skills and Engineering Skills Are Not the Same Thing
AI competency and engineering competency should be evaluated separately.
A strong engineer may initially use AI inefficiently but still produce excellent engineering work because their fundamentals and judgment are strong.
Another engineer may be highly proficient with AI tools but lack the technical depth necessary to recognize when the output is flawed.
And an engineer who is strong in both can use AI to dramatically increase their effectiveness.
Those are very different talent profiles.
If the assessment only looks at the final code, you miss how the engineer worked with AI.
If it only evaluates AI usage, you miss whether the resulting engineering is actually good.
Leaders need visibility into both.
Five Capabilities to Evaluate
Stage 1 — Define the Capability You Actually Care About
Don’t begin with a list of AI tools. Start with the behaviors you want from an AI-enabled engineer. What should they be able to do better, faster, or more effectively because AI is available?
Stage 2 — Assess Through Realistic Work
Knowing AI terminology is not the same as working effectively with AI. Give engineers a scenario with enough complexity and ambiguity that judgment is required. Allow them to use AI. Then observe what happens.
Stage 3 — Evaluate the Process, Not Just the Output
The final answer tells only part of the story. Two engineers can arrive at similar solutions through very different processes. Understanding how the engineer reached the result is increasingly important.
Stage 4 — Separate AI Competency From Engineering Competency
Evaluate the engineer’s use of AI and the quality of the engineering independently. This distinguishes strong engineers who need better AI skills from sophisticated AI users who need stronger engineering fundamentals—and from engineers who are strong in both.
Stage 5 — Turn the Results Into Talent Decisions
Assessment should lead to action. Use the results to inform hiring, professional development, team composition, and engineering strategy—not simply to generate a score.
What This Looks Like in Practice
At CH2, we started thinking about this problem because we were seeing the same disconnect many technology leaders are seeing.
Engineers were using AI more frequently, but tool adoption alone didn't tell us whether someone had become a more effective engineer.
So we built a scenario-based AI skills assessment specifically for software engineers.
Rather than asking candidates or engineers how proficient they are with a particular tool, the assessment puts them into realistic engineering scenarios with AI available to them.
We evaluate how they frame the problem, work with AI, apply professional judgment, verify the output, and take ownership of the final result.
We also evaluate the quality of the engineering separately.
The goal is not to reward the person who uses AI the most.
It is to identify engineers who know how to use AI without giving up the judgment that makes their work valuable.
What Leaders Should Be Asking
If you're trying to understand the AI capability of your engineering organization, start with a different set of questions.
Don’t just ask: Which AI tools are our engineers using?
Ask:
Are they solving problems better because of them?
Can they recognize when AI is wrong?
Are they validating what AI produces?
Do they understand and own the work they submit?
Is AI amplifying strong engineering judgment—or replacing it?
Those questions tell you much more about whether your organization is actually becoming AI-enabled.
Leadership Lens
AI will continue to make code easier and faster to produce.
That makes human judgment more valuable, not less.
The engineers who create the most value won’t necessarily be the people who generate the most code with AI. They’ll be the people who know what to ask, what to trust, what to challenge, what to verify, and when the technology should not be making the decision at all.
For technology leaders, the challenge is learning how to identify those capabilities before they become invisible behind faster output.
FAQ
How do you measure AI skills in software engineers?
The most useful approach is to evaluate engineers through realistic development scenarios where AI is available. Assess not only the final output but also how the engineer frames the problem, directs AI, evaluates its recommendations, validates the work, and takes ownership of the solution.
Should companies test engineers on specific AI tools?
Tool-specific knowledge can be useful, but it should not be the primary measure of AI competency. AI tools change quickly. The more durable capability is knowing how to work effectively with AI regardless of the specific interface or model being used.
Is prompt engineering an important skill for software engineers?
Giving AI useful instructions and context matters, but prompt writing is only one part of effective AI use. Engineers also need to decompose problems, evaluate outputs, recognize errors, validate results, and make sound engineering decisions.
Can AI skills and engineering skills be measured separately?
Yes, and they should be. An engineer’s ability to use AI effectively and the quality of their underlying engineering judgment are related but distinct capabilities.
How can companies use AI skills assessments?
The results can help inform hiring, internal training, professional development, team composition, and broader AI adoption strategies.
Sources and Further Reading
DORA — State of AI-assisted Software Development 2025
Research on how AI is affecting software development and the organizational conditions that influence whether AI adoption improves software delivery.
Microsoft Research — The SPACE of AI: Real-World Lessons on AI’s Impact on Developers
Research examining how task complexity, individual usage patterns, team adoption, organizational support, and peer learning affect the value developers realize from AI.
Microsoft Research — The Effects of Generative AI on High-Skilled Work
Field research examining the effects of generative AI coding tools on developer productivity across several organizations and experience levels.
METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
A randomized study examining AI-assisted development among experienced developers working in mature codebases they already knew well.
