Your AI Can Follow the README. Can a New User?

Key takeaways

• Authors can miss missing steps because their own setup and knowledge fill the gaps.
• An AI agent’s successful installation does not prove the README is sufficient.
• Paid testing works best with a specific task and participants who resemble the intended users.
• After making changes, test with someone new to see whether the same obstacles remain.

An AI agent can get your project running while filling in gaps in the README. A first-time user may hit those same gaps and close the tab. Paying someone to follow the instructions gives you a way to see where that happens.

Your setup is hiding the missing steps

A README introduces a project, explains how to use it, and shapes a newcomer’s first impression. Its author has one substantial disadvantage when reviewing it: knowing how everything works.

You already have the prerequisites installed. You know which directory a command belongs in. When an instruction skips a step, your experience supplies it so smoothly that you may never notice.

Consider “Install the package, then run the example.” Straightforward enough—if you know what a completed installation looks like. When the terminal goes quiet, a newcomer may wonder whether the process finished, stalled, or failed.

Showing the expected output can resolve that uncertainty. So can a screenshot of the running application.

Good instructions explain how to recognize success. Users need that confirmation before they can confidently move on.

An agent’s workaround belongs in the test results

AI can help spot ambiguous wording and missing explanations. An agent with access to an execution environment can also check whether commands and examples actually work.

The important question is how it reached the finish line.

Suppose an installation command fails. The agent interprets the error, substitutes another command, and completes the installation. That is a useful discovery. It also reveals a point where someone following the README would get stuck.

A report that says “Installation successful” loses the most valuable part of that run.

When reviewing an agent’s attempt, examine both completion and undocumented steps. What did it infer? What did it repair? Which prerequisites did it supply from knowledge outside the document?

Asking an AI to act like a beginner can help surface problems, but it does not establish how a beginner will behave. A person encountering an unfamiliar term might search for it, ask a colleague, or abandon the task. Observing users tells you which choice they make and where.

Pay for a specific attempt, including the failed ones

“Please review our README” leaves a lot undefined. Give participants a concrete task instead:

Use this README to install the project and run one example. You can stop after 20 minutes. Tell us which terms were unclear and where you were unsure what to do next.

The 20-minute limit is an example, not a benchmark. Choose a duration that fits the installation and its intended audience.

Recruit accordingly. For a developer library, someone who knows the language but has never used the project may be a good fit. For a tool aimed at nondevelopers, an experienced engineer’s success tells you much less about accessibility.

Pay participants regardless of whether they finish. An unsuccessful attempt can expose exactly the problem you need to fix.

During the session, resist the urge to explain everything immediately. Record where the participant pauses, what they expected, and why they ask for help. If you provide a hint to keep the session moving, write that down too.

The explanation you just gave aloud may be the sentence your README is missing.

Sometimes the fix belongs in the installer

A confusing test session does not automatically call for a longer document.

Some problems need a small clarification: name the supported operating systems, specify the working directory, or show the expected output. Others expose unnecessary complexity in the installation itself. Adding paragraphs may leave that underlying problem intact.

“User was confused” is too vague to guide a fix. Record the instruction they read, the action they took, and the result they expected. That sequence helps you decide whether to change the wording or the software.

After making the change, give the same task to someone who has not seen the procedure. The original participant now knows the workaround; their second attempt cannot show you what a fresh encounter feels like.

Run the technical checks again as appropriate, too. Working commands and understandable instructions are both necessary, and each needs evidence.

Paying someone to try your README buys a chance to observe the assumptions its author can no longer see. The finish line is a newcomer getting the project running—and knowing, without asking you, that it worked.