Readiness scorecard

Eight questions, about 10 minutes, for a CTO or VP of Engineering. Each comes from eight months of running AI agents on a commercial Java product with paying customers.

How to answer. Pick what is true today, not what is planned; if torn between two, pick the lower. No agents yet? Answer for what is in place now, and where one would run. Your total appears as you go, with a summary to copy and send me. Your answers stay in your browser: nothing on this page is stored or sent anywhere.

1. How long does a full build take, and when it goes red, how often is the cause real?

0 Over an hour (or nobody knows), and flaky tests aren't logged and fixed.
1 One of the two: under an hour, or flaky tests logged and fixed.
2 Both: under an hour, and flaky tests logged and fixed.
3 All of 2, under 30 minutes, and changes to docs or tests alone skip the full build.

Why it matters: every agent mistake costs a build, and builds slow as agents add tests: my full build doubled, from about 30 minutes to about an hour.

2. Could an agent take your next ten backlog items without asking anyone a question?

0 Nobody could name the next ten without a meeting.
1 They're ranked, but not all ten have acceptance criteria someone could test.
2 All ten have testable acceptance criteria, but one or more has an open question.
3 All ten have testable acceptance criteria, and none has an open question.

Why it matters: every question an agent asks waits for a person, and in my fleet, once the budget was paced, the binding constraint was me.

3. Who approves what an agent raises, and what happens to the queue when they're away?

0 Nobody is named to approve what agents raise.
1 Named people approve, with no time set aside for it.
2 Named people with time set aside each day; the queue waits while they're away.
3 All of 2, a named stand-in, and only a named person's approval can merge.

Why it matters: five agents produce more than one person comfortably approves. While I was away in October, an estimated 104 pull requests queued.

4. Could an agent read a production credential, or reach a host you haven't approved?

0 It could read a production credential.
1 It could read none, but it could reach any host.
2 It could read none, and a filter blocks some hosts, but not every unapproved one.
3 It could read none, and it could reach only hosts you have approved.

Why it matters: an agent runs the code it writes and reads what it fetches, so a mistake, or a page written to mislead it, puts at risk whatever it can read or reach.

5. What stops the agents when the week's budget runs out on a Wednesday?

0 Nothing: no budget is set for agent usage, or nobody owns it.
1 A budget is set and owned, but nothing stops the agents before it runs out.
2 Something stops the agents when the budget is spent, and tells its owner.
3 Each day releases its share of the budget, and the agents pause when that share is spent.

Why it matters: in July my agents' weekly budget kept running out mid-week, and July had 12 days without a single merge.

6. Who or what notices an agent that is stuck, idle or duplicating work, and can it act without waking you?

0 Nothing: you'd find out when someone happened to look.
1 A named person checks the agents by hand, on a schedule.
2 An alert tells a person, who then acts.
3 Something restarts or reassigns the agent, and wakes a person only when it can't.

Why it matters: with their supervisor switched off, my agents committed work for four days, then almost nothing, and nothing noticed they had stopped.

7. Which of your rules are enforced by a check that refuses, and which only by a document?

0 A red build or an unreviewed change can still be merged.
1 A merge needs a green build and an approval; nothing else is enforced.
2 Checks also refuse other breaches, such as missing tests or a committed secret.
3 All of 2, and rules that keep being broken become checks: you can name the last one.

Why it matters: an agent reads a rule, believes it, then forgets it. Mine were told to tick off work items by hand: of 431 items, they ticked 7.

8. Can you count merged pull requests, cycle time and failed changes from your own records, and are you counting the thing you mean?

0 None of the three without a manual exercise.
1 One or two of them, from a query or a dashboard.
2 All three, from a query or a dashboard.
3 All three, each defined in writing and checked by hand against a sample.

Why it matters: agents generate so much activity that the wrong count is easy: my first count took merge commits for merged work and overstated throughput about 2.4 times.

Your total:

Paste them to me on LinkedIn

Your total, out of 24

0 to 8: foundations first. Agents would spend your budget waiting for builds, answers and approvals. Fix three things, then score again: a full build under an hour, with flaky tests logged and fixed (question 1); ten ranked backlog items, each with testable acceptance criteria (question 2); a named approver, with time set aside for it (question 3).

9 to 16: prepare, then pilot. Some foundations are in place, not all. The discovery assessment pinpoints what to prepare.

17 to 24: ready for a pilot. Your pipeline can take agents now. The discovery assessment confirms it and plans the pilot.

Gates. Whatever the total, a 0 on question 3 or 4 means "prepare, then pilot" at best. With nobody to approve (3), the agents' work only queues: a pilot keeps a person approving every merge. An agent that can read a production credential (4) is a risk no total outweighs: a pilot runs agents where they can read none, so that comes first.

A pilot sets up what questions 5 and 6 ask about, so 0 or 1 there is normal.

What happens next

Whatever your total, send me your eight scores and we'll set up a 30-minute call on your answers. If a pilot makes sense, the discovery assessment comes first (2 weeks). I review your pipeline and codebase against the scorecard, checking each answer. You get a readiness report, a pilot plan with its first pieces of work, the design of your human gate, and a cost model.

Send me your eight scores on LinkedIn Back to the offer