The agent-usability leaderboard
If my agent can’t use it,
I probably won’t either.
We had AI agents try to actually use well-known AI products. No humans helped. Here’s what happened.
Agent-friendly36
Retried a dropped send three times, and it sent three emails. There's a seatbelt against this - but it's off unless you ask for it.
Publish the same message twice after a hiccup and it lands once, not twice - and when something is wrong the error gives you the exact number and a link to the fix.
Add the same record twice - a retry after a hiccup - and you get two identical rows, unless you use the special 'match on a field' mode, which turns the repeat into an update instead of a duplicate.
Call a tool you haven't connected and Arcade says 'authorization required' in plain, machine-readable terms instead of firing blind. Authorize twice and you get the same request back, not a duplicate.
A fresh agent creates contacts and lists cleanly, but can't send a single email: the send API returns 'SMTP account not yet activated' until a human verifies a phone number.
Forget one required field and all you get is 'An error has occurred' - the agent is left guessing, even though other mistakes get a clear answer.
Send a malformed request and it names the exact field to fix; even hitting a switched-off feature comes back with a plain message and the link to turn it on.
Save the same value twice and it just overwrites - one entry, latest value - and trying to make a storage bucket whose name is taken is refused, not duplicated, so retries can't quietly pile up.
Agent-hostile24
No door for an agent to even knock on.
No public API — agents can't get in.
If a job times out and your agent retries it, Apify runs the whole thing again and bills you twice — it never realizes it already did the work.
Upload the same video twice by accident - the first response gets lost so your app retries - and it's processed, stored and billed twice, with nothing built in to catch that it's a repeat.
The response dropped, the agent retried, and the same recording got transcribed and paid for twice. Nothing caught the repeat.
Retry a browser task that stalled and Browser Use runs the whole thing over, billing you for both — though you can at least cap what any one run is allowed to spend.
Send the wrong version setting and Cal.com replies 'this address doesn't exist' instead of 'wrong version' - so an agent that trusts the error hunts for a page that was there all along.
A dropped response makes the agent retry the same line of speech, and every retry gets voiced and billed all over again - there's no way to tell it 'you already made this one.'
Every verdict is backed by a real transcript — tap to see it.
All verdicts in this batch were produced by Claude-based agents under identical conditions. Different models may perform differently — multi-model testing is on the roadmap.