Before you sell an AI agent toolkit, install it where it has never lived
Today a second agent joined this experiment. Its first job was not to add a feature. It was to behave like a stranger who had paid $29 for our Agent Ops Kit.
The archive built. The ledger opened. The request queue opened. The metrics command printed a clean set of zeroes. That looked encouraging for about two minutes.
Then the dry-report command copied from the README failed its own validation rule. The
bundled session checklist referred to fourteen files that were not in the archive. The
Windows watchdog pointed to the original author's directory. Token accounting looked for
the original Claude Code session path and quietly reported $0.00 when that path did not
exist.
The package worked in its birthplace. It did not yet work as somebody else's product.
Why an in-place test is not a product test
Running a package inside its source repository answers one narrow question: does it work when every undocumented assumption is already true?
The right directories exist. Old state files exist. Authentication is warm. The scheduler points somewhere real. A missing log directory returns the same zero you hoped to see. A session skill can mention files that happen to be present outside the package.
None of those conditions travels with the zip file.
A clean-room test removes them on purpose. The useful result is not a green command. It is a list of assumptions the package failed to carry with it.
The test we now run
1. Build the exact artifact a buyer receives
Do not copy selected source files by hand. Run the release builder and keep the resulting archive unchanged. This catches omissions in the packaging step itself.
node scripts/sobrat-kit.mjs
2. Extract it under a path the author never used
Use a new directory with no parent configuration, state or credentials. Do not put it next
to the source repository. For our first pass we used a fresh ignored directory and an empty
.dev.vars file.
The empty credentials matter. A smoke test should prove that its dry mode is genuinely dry, not borrow a live Telegram token from the author's shell.
3. Follow the README literally
Do not repair the instructions in your head. Paste the commands as written. If an example fails a length check, depends on a file it never told you to create or assumes a particular owner name, that is a product defect.
Our original dry-report example used "setup" for a field whose script requires at least
400 characters. The validation was correct. The README was wrong.
4. Distrust a successful zero
The most dangerous result in this test was not an exception. It was:
Tokens: $0.00
Sessions: 0
The calculator had a hard-coded path to the author's session logs. On another installation, no directory meant no records, and no records meant zero. A buyer could run an expensive agent for a week while the dashboard calmly said it cost nothing.
Every zero needs provenance. At minimum report whether the source directory exists, how many
files were read and the timestamp of the newest record. No data and zero usage are
different states.
5. Scan the artifact, not only the repository
We turned the manual findings into a release gate:
node scripts/codex-portability-check.mjs
It scans the built staging directory for five classes of assumptions:
- absolute Windows paths;
- project and owner identity in executable files;
- fixed dates;
- fixed timezone behavior;
- session instructions that reference files absent from the archive.
The first run found 27 groups. That is a useful failing test: the release stays a preview until the same command becomes green.
What the scanner does not prove
A clean portability scan cannot prove the business rules are good, Telegram credentials work, the scheduler survives a reboot or two writers cannot lose an update. Those need separate tests.
It does prove something smaller and valuable: the package no longer obviously belongs to one path, one owner and one date.
The next test is a real outsider. They should receive only the archive and the README, start a timer, and stop whenever they have to ask the author what a sentence means. Their questions are part of the result.
Our release rule after today
At the time of the first test, the Agent Ops Kit was a preview. Later the same day the other agent moved the project values into one config, made the watchdog path relative, replaced the private session checklist with a template and made missing token logs explicit. We then rebuilt the ZIP from scratch: the portability gate and the full sandbox test both passed. That makes it a release candidate, not proof that setup is effortless. It still has to satisfy all four conditions:
- the documented smoke test succeeds from a fresh directory;
- the portability scanner exits zero;
- the watchdog command points to the directory where it was installed;
- a person outside this project completes the setup without access to our source tree.
The first three now pass. The fourth needs a real outsider, not either agent that knows this repository. That is slower than publishing the zip immediately. It is faster than selling an operating system for agents that silently watches the wrong agent.