Use case
Agent evals that catch cheating.
Run benchmark suites on isolated computers with credential controls and response filtering so agents cannot fetch leaked answers.
Why this is hard
Everyone is tuning prompts on vibes
A change feels better on the three examples someone tried. Whether it is better across the work you actually do is unknown.
Public benchmarks are not your codebase
A score on someone else’s repositories tells you little about a model’s behavior in your monorepo with your conventions.
Agents cheat when you let them
Given network access and a scored objective, an agent may find the answer key rather than solve the task. If the environment is open, your score is fiction.
How an eval run works
You fix a task set from your own work
Real tickets and real pull requests with known-good outcomes, so a score means something to your team.
Each attempt gets an identical sealed computer
Same starting snapshot every time. Runs cannot see each other, and nothing carries over to contaminate the next attempt.
A separate model grades each run
An LLM judge scores against the rubric you wrote, with no access to the agent under test. It evaluates what happened in the session instead of trusting a self-reported answer.
Results are graded and compared
Pass rate per task, with the trajectory kept for the failures, so you can see how a model got it wrong and not just that it did.
What you get back
- Pass rate per task
- Score deltas between runs
- Failure trajectories
- Comparison across models
What makes a score comparable
Byte-identical snapshots for every attempt
Two scores only mean something if both runs started from the same environment.
A separate LLM judge with no shared context
The grader scores against your rubric without the agent under test seeing it or shortcutting through the session.
Same task, any harness
Compare Claude Code, Codex, Cursor, and your own agent on identical computers and policies.
Hundreds of parallel runs from CI
Sweep a prompt or model change while the diff is still fresh.
Who runs this
Setting up was seamless and I love the experience.
Shay DahanTech Lead, Public software companyAll customer stories →
I have run plenty of tests and evals and it is amazing.
Volodymyr BihunenkoSoftware Engineer, ARIMLABSAll customer stories →
Where teams go next
Common questions
How do content filters help evals?
Gateway scans response bodies and blocks agents from fetching solutions during benchmark runs.
Can I use my own harness?
Yes. Islo is infrastructure; bring Claude Code, Codex, Cursor, or a custom setup.
Are eval computers isolated from each other?
Yes. Each run gets its own computer with scoped network policies.