Achieving a feedback improvement loop with documentation by simulating user queries
- The logs don’t show what the agent did with your docs
- What Vercel found about skills that don’t fire
- A system prompt reaches only one platform
- Recreating user sessions with a build log
- What the recreated sessions turned up
- Fresh docs help only if the agent fetches them
- One actionable doc bug
- A live example: embedding a biking map on my own blog
- Agents as users we can observe
- Is anyone else doing this?
- Follow-up: 20 more scenarios
- Posts on agent readability
- Related posts
Update (Nov 3, 2026): I ran a larger follow-up test with 20 scenarios. It confirmed some of what I describe here and complicated other parts, especially how much you can trust an agent’s build log. See Follow-up: 20 more scenarios at the end.
A couple of weeks ago, I wrote about a documentation strategy I’m pursuing: mining chat logs from AI app builders to find doc problems and fix them in a continuous loop. (See My AI strategy to fix docs based on chat logs.) Since you might not have read that post, let me briefly set up the larger goal.
For years, technical writers have written documentation largely in the dark, with little visibility into where developers get stuck when trying to use an API. Now that developers are using AI coding agents and app builders to build apps with our products, those chat sessions should tell us a lot.
If we can look at the logs of how users are trying to build with our APIs, identify the frictions and failure points where the agent or user goes off the rails, turn those failures into actionable doc bug tickets, and feed them into an automated doc-bug-fixing skill, we can create a feedback loop that steadily improves documentation quality.
That was the idea, anyway. In this post, I’ll go deeper into what got in the way when I tried to put this loop into motion, and describe a workaround that turned out to be more useful than the logs themselves.
The logs don’t show what the agent did with your docs
To test this workflow, I first got access to sanitized logs from an AI app builder where developers describe an app they want and an agent writes the code to build it (AI Studio). I expected the logs to show where the agent read our Google Maps Platform documentation, misunderstood an instruction or code sample, and produced broken code that we could trace back to a doc gap.
However, the logs didn’t give me a clear picture of what was wrong with the docs. There’s no neat log entry that says the agent read this page, misused this parameter, and the build failed as a result. The logs I had were sanitized summaries of each conversation, and they didn’t record which docs or skills the agent loaded, which tools it called, or which errors it hit.
A bigger problem showed up first, though. In most sessions, the agent never consulted our docs at all:
- Many sessions had nothing to do with the product I document (Google Maps APIs).
- In sessions that did involve the product, our product skill rarely fired. It loaded in only a small fraction of sessions.
- The agent usually built from memory. It relied on what it already knew from training, and it often chose an open-source mapping library instead of ours.
Agents are confident (some would say arrogant). They assume they know how to build whatever the user asks for, so they rarely stop to look anything up. (Tech writers are probably groaning now. For years we complained that human users needed to RTFM; now, agents are adopting the same behavior of not reading the manual.)
What Vercel found about skills that don’t fire
Vercel ran into the same problem (product skills not being used by agents, which then relied on outdated training data) and wrote about it in AGENTS.md outperforms skills in our agent evals. Vercel, which makes the Next.js framework, tested whether coding agents could correctly use Next.js 16 APIs that aren’t in the models’ training data.
When they gave agents a Next.js docs skill, the agents didn’t invoke it in 56% of the eval cases. The pass rate (53%) was no better than having no docs at all.
Adding explicit instructions to AGENTS.md to use the skill raised the pass rate to 79%, but the wording mattered a lot. One instruction, telling the agent it must invoke the skill, made it anchor on the doc patterns and miss details in the project itself. Another, telling it to explore the project first and then invoke the skill, worked better.
What worked best was putting a compressed docs index (about 8 KB) directly into the project’s AGENTS.md file, which the agent loads into its context on every turn. The index points to version-matched doc files in the project and includes this line:
IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning for any Next.js tasks.
With that in place, the pass rate went to 100%, because the agent no longer had to decide whether to look something up. Vercel can do this because it owns the project scaffolding. Starting with Next.js 16.2, create-next-app adds an AGENTS.md file to new projects. An API provider usually doesn’t create the developer’s project, so it doesn’t have that channel.
The same idea applies to the system prompt of an AI app builder. If you control that system prompt, you can add something like this:
Your training data about Google Maps APIs is out of date. Before you build a solution that uses maps or location data, load the Google Maps Platform skill and use its docs instead of relying on memory.
I tried a version of this. I can’t change the builder’s system prompt, so I put a two-sentence hint like this at the start of the user prompt instead. Short prompts that clearly describe a map app loaded our skill with or without the hint, so I made longer prompts (which might simulate user behavior more closely) that didn’t mention our product by name.
Without the above hint, the agent loaded our skill in 0 of 2 runs. With the hint, it loaded the skill in 2 of 2 runs. A prompt for an app that didn’t need a map still didn’t load the skill, so the hint didn’t make the agent load it everywhere. These are small samples, but they point the same direction as Vercel’s results.
Sponsored
A system prompt reaches only one platform
Adding instructions to a prompt, as I did, has limitations. Even if a hint like this works, it fixes the problem only on the platform whose system prompt you control. People build apps with Claude Code, ChatGPT, Cursor, Replit, and many other tools, and each one has its own system prompt that you can’t change. On those platforms, the agent’s choice depends on its training data, what it finds on the web, and which skills or MCP servers the developer installed (which help only developers who already chose your product).
Still, fixing one platform is better than nothing. Once your skill fires and the agent starts building, you can get to the second (and more interesting) part of the problem: identifying what the agent struggles with in your docs.
Recreating user sessions with a build log
If your logs record what docs the agent struggled with, great. Mine didn’t. What I did have were sanitized, high-level descriptions of the kinds of apps people were trying to build — for example, a site for a bakery that shows its store locations on a map.
So I had my analyze-chat-logs skill go through a week’s worth of these descriptions (a sample, not the full set) and write 10 prompts that roughly simulated the most common scenarios. Then I ran those prompts myself in the same AI app builder tool (AI Studio). To each prompt, I added a block of instruction asking the agent to keep a build log file that records, for each step, what the agent did, which source it used, why, and the result or error. The block also asks the agent to note where the docs were hard to use.
Before you put too much stock in the agent’s build log file, note that agents are unreliable narrators. In one build, the log said the agent used a skill that it never downloaded. In others, the agent downloaded doc files and never opened them. Because I ran these sessions myself, I could also check each one against the platform’s action history. (In Google AI Studio’s Build mode, the person running a session can see its action history.) Just to be clear, every session and action history I describe in this post is one I ran, not a user’s.
What the recreated sessions turned up
Across the 10 scenarios, I ran about 15 builds (some failed because of quota limits or platform errors, so I reran them), and 9 apps were built. I fed the built apps and build logs into AI for analysis. It identified 8 doc-related issues. Seven of them were in the product skills the agent loaded, not in the docs on our site.
The issues fell into a few broad patterns, which I suspect show up with many APIs:
- Setup steps the agent had to discover by trial and error. A missing configuration step cost a fix cycle in almost every build.
- Broad data requests. Agents often requested more fields than the app used. The guidance on requesting only the fields an app needs wasn’t prominent enough to change that behavior.
- Example code treated as a spec. Agents copied examples line for line, shortcuts and placeholders included.
- Instructions that didn’t hold everywhere. A rule that worked for some endpoints caused errors on others.
- Thinner guidance for frameworks. Guidance for building with a popular framework like React was weaker than guidance for the core API, partly because framework samples often live outside the main docs.
- Policy rules in the wrong place. Requirements like attribution lived on pages the agent didn’t open while it was writing code.
Fresh docs help only if the agent fetches them
I expected agents to pull fresh docs on demand by calling a docs tool, such as an MCP server that searches the official docs. A docs MCP server searches the docs when the agent asks, so its results can stay close to the current version of each page, depending on how often its index is refreshed.
In the builder I tested, the agent rarely made those tool calls. It got docs in three ways:
- Product skills (most of the time). The agent downloaded static Markdown files that a pipeline generates from the published docs.
- A docs retrieval tool (occasionally). The agent also had access to a Maps-specific docs retrieval tool, but it seemed to call it only once in a while.
- Doc pages (rarely). Once in a while, the agent fetched a doc page directly.
A docs API can still play a part, because a skills pipeline can use it to pull the pages that an LLM then rewrites into skills. In that case, though, the API feeds the pipeline, not the agent.
Problems with generated skills between the docs and the agent
Having generated skill files sit between the docs and the agent creates two problems:
- Slower doc updates. A skill changes only when someone reruns the skill generation pipeline and publishes the output, and I’m not sure when the skill files my test agents loaded were last regenerated. A doc fix I publish today won’t reach agents in this builder until the next skill regeneration.
- Harder debugging when the agent makes a mistake. Tracing a build failure back to a specific doc page takes detective work. The problem could come from several places:
- The source doc page has an error or a gap.
- The LLM introduced an error when rewriting the doc page into a skill.
- The skill’s example code has a flaw or shortcut that the agent copied.
- The model ignored a rule that was right in front of it.
This setup applies to the one builder I tested. In IDE agents like Claude Code, Cursor, or Gemini CLI, a developer can connect a docs MCP server, such as the Developer Knowledge MCP server, so the agent queries live docs directly. Even in those tools, though, developers have to install the MCP server first. When a skill is also installed, the agent might read the skill first and call the docs tool only when the skill points to it or doesn’t cover something, as the build log later in this post shows.
That’s a strong argument for tech writers to take a bigger role in product skills instead of treating them as someone else’s generated artifact. Writers can review what the generator produces (especially the example code, which agents copy line for line), have a say in how often skills are regenerated, and test whether a doc fix makes it all the way to the agent.
One actionable doc bug
When I sent the 8 bugs through my doc bug-fixing workflow for another agent to fix, only one became a doc change. A couple of API reference pages didn’t explain how the fields you request in a call affect the pricing tier for that request, so the fix adds that explanation to those pages. The other seven issues were in the skills, which are maintained separately from the docs, so they became proposed changes rather than doc edits.
So my plan of mining chat logs for a steady stream of doc fixes was perhaps too dreamlike. Accessing, curating, and parsing the data to find doc problems is like looking for special seashells on the ocean floor. And most of what I found pointed to the skills, not the docs. Still, simulating the same user queries and having agents describe their actions in build log files told me more than trying to mine the same information (often missing) from user chat logs.
A live example: embedding a biking map on my own blog
To show what a build log looks like in practice, I ran the same experiment on my own blog using Gemini in Antigravity IDE. I asked the Gemini coding agent to embed a Google Map of a 17.5-mile bike ride into an older post of mine, The most perfect 17-mile biking loop in Santa Clara, and to keep a build_log.txt file describing every step it took to retrieve information, whether from the Google Maps Platform plugin, remote skill files, a docs API, or memory.
Across 8 logged steps, the agent found the post, extracted the route coordinates from an older KML link, prompted me to install the official Google Maps Platform plugin, loaded 4 sub-skills and 9 reference files, fell back to the Maps Code Assist API when a routing reference file contradicted the main skill, and embedded a <gmp-map> web component with the route polyline and bicycling layer.
View the full build_log.txt (Steps 1–8)
=== AGENT BUILD LOG ===
Task: Embed a Google Map of the Santa Clara loop in https://idratherbewriting.com/blog/biking-the-santa-clara-loop/
Started: 2026-11-01T15:20:37-08:00
Step 1: Locate the target blog post and check the Agent Marketplace
- Action: Ran `find _posts -name "*biking-the-santa-clara-loop*" -o -name "*santa-clara-loop*"` via `run_command` to locate the markdown file in the Jekyll repository.
- Action: Called `search_marketplace` with queries `["google maps", "maps", "geo"]` to check if an official Google Maps skill or MCP plugin exists before consulting external docs or improvising from memory.
- Result (File search): Found `_posts/2020/11/2020-11-28-biking-the-santa-clara-loop.md`.
- Result (Marketplace search): Found an official uninstalled plugin: `antigravity-plugins-official/plugin/google_maps_platform` ("Google Maps Platform: Integration skill and tools for Google Maps Platform APIs").
Step 2: Inspect the existing post and route data
- Action: Read `_posts/2020/11/2020-11-28-biking-the-santa-clara-loop.md` using `read_file`.
- Observation: Line 19 already contains a legacy Google My Maps iframe (`https://www.google.com/maps/d/embed?mid=16lrGzUFJMukBvARwRrwCY2JBjvlDOu_D`).
- Action: Tested the existing My Maps embed and KML export endpoint (`https://www.google.com/maps/d/kml?mid=16lrGzUFJMukBvARwRrwCY2JBjvlDOu_D&forcekml=1`) using `curl` via `run_command`.
- Result: The KML endpoint returned the full route geometry (`<name>Santa Clara loop - biking trail</name>`, 1,157 lines of XML) with the exact `<LineString>` coordinates (1,047 points) and start/end `<Point>` placemarks at `Monroe @ San Tomas Creek, Santa Clara, CA 95051, USA` (`37.36404, -121.96868`).
Step 3: Pause for Google Maps Platform plugin installation
- Reason: Rather than relying on model training memory or scraping general Google Maps documentation manually, the Agent Marketplace provides the official `google_maps_platform` plugin.
- Action: Surfaced `<plugin-card catalog-id="antigravity-plugins-official/plugin/google_maps_platform"></plugin-card>` so the user could install it.
- Result: User installed the `google_maps_platform` plugin (`google_maps_platform/skills/SKILL.md`).
Step 4: Load the Google Maps Platform Main Skill & Dynamic Skills Index
- Action: Read `google_maps_platform/skills/SKILL.md` using `read_file`.
- Observation: The main skill enforces a strict Grounding and Routing Workflow:
1. Never rely on training memory.
2. Fetch the remote Skills Index (`https://www.gstatic.com/googlemapsplatform-agent-skills/index.json?client=antigravity`).
3. Match and fetch relevant sub-skills (`SKILL.md` and reference docs) from `www.gstatic.com`.
4. Use the Maps Code Assist endpoint (`https://mapscodeassist.googleapis.com/v1:retrieveContexts`) as a fallback for uncovered API details.
5. Follow mandatory compliance checkpoints (review `compliance-review` skill, include `gmp_git_agentskills_v1` attribution ID, avoid deprecated APIs like `google.maps.Marker`, `DirectionsService`, `DirectionsRenderer`, and `KmlLayer`, and request a Google Maps API key or Maps Demo Key rather than hardcoding credentials).
- Action: Fetched `https://www.gstatic.com/googlemapsplatform-agent-skills/index.json?client=antigravity` via `curl`.
- Result: Retrieved 42 official Google Maps Platform sub-skills.
Step 5: Fetch Matched Sub-Skills and Reference Guides from gstatic.com
- Action: Matched four sub-skills from `index.json` and fetched their `SKILL.md` files with `?client=antigravity`:
1. `maps-javascript-api-javascript/SKILL.md` (interactive 2D/3D maps, polylines, GeoJSON data layers, bicycling layer, routing)
2. `maps-embed-api-javascript/SKILL.md` (iframe-based `directions` and `view` modes)
3. `gmp-common-api-keys/SKILL.md` (API key management, restrictions, and Maps Demo Key quickstart)
4. `compliance-review/SKILL.md` (mandatory Terms of Service and deprecation review methodology)
- Action: Fetched the specific capability reference files linked inside those sub-skills:
- `maps-javascript-api-javascript/references/add-customizable-interactive-map-web-page-mobile-app.md`
- `maps-javascript-api-javascript/references/add-marker-map.md`
- `maps-javascript-api-javascript/references/add-kml-layer-map.md`
- `maps-javascript-api-javascript/references/add-geojson-data-map.md`
- `maps-javascript-api-javascript/references/add-shape-line-map.md`
- `maps-javascript-api-javascript/references/add-bicycling-layer-map.md`
- `maps-javascript-api-javascript/references/specify-the-travel-mode-drive-transit-walk-two-wheeled-for-route.md`
- `maps-javascript-api-javascript/references/specify-stop-pass-through-point-for-route-request.md`
- `maps-embed-api-javascript/references/embed-standard-google-maps-directions-view-into-iframe-web-page.md`
Step 6: Query the Maps Code Assist Grounding API (`mapscodeassist.googleapis.com`)
- Reason for Fallback: The sub-skill reference files for routing in `maps-javascript-api-javascript` had direct contradictions and outdated/broken code, forcing a fallback query to `https://mapscodeassist.googleapis.com/v1:retrieveContexts`.
- Action: Sent POST request with `"llmQuery": "Maps JavaScript API Route.computeRoutes bicycling route with createPolylines and createWaypointAdvancedMarkers"` and `"source": "antigravity"`.
- Result: Retrieved current reference documentation (`developers.google.com/maps/documentation/javascript/reference/3.65/route`) and official sample code (`samples/routes-markers/index.ts`) showing the actual `Route.computeRoutes()` signature and `result.routes[0].createPolylines()`.
Step 7: Confirm Architecture with User & Prepare Route Assets
- Action: Asked the user via `ask_question` to choose the embed method and how to handle the API key.
- Result: User selected `<gmp-map>` with the exact KML route polyline, markers, and `BicyclingLayer`, and requested a central config variable in `_config.yml` for experimenting with their Maps Demo Key.
- Action: Extracted the 1,047 coordinate points from the My Maps KML export into `/assets/files/santa-clara-loop-path.json` and saved `/assets/files/santa-clara-loop.kml`.
Step 8: Implement `<gmp-map>` in Blog Post & Configure Central API Key Setting
- Action: Added `google_maps_api_key: "YOUR_GOOGLE_MAPS_API_KEY"` to `_config.yml` so the user can drop their Maps Demo Key in one place and reuse `` across any blog post.
- Action: Updated `_posts/2020/11/2020-11-28-biking-the-santa-clara-loop.md` to embed `<gmp-map>` with explicit CSS height, `map-id="DEMO_MAP_ID"`, `internal-usage-attribution-ids="gmp_git_agentskills_v1"`, a `<gmp-advanced-marker>` at the trailhead, and dynamic library loading (`google.maps.importLibrary`) for `BicyclingLayer`, `Polyline`, and `LatLngBounds`.
How to read a build log for doc and skill bugs
The build log gives a readable, step-by-step account of what the agent says it did: which sources it used and in what order. It’s still the agent’s own account, though. In my later 20-scenario test, roughly one in five docs that build logs listed as sources were never opened (see Follow-up: 20 more scenarios), so check the log against an action history when you can.
The next obvious question: If you’re a tech writer looking at a build log like this, how do you know whether there are doc issues to fix? After all, the build succeeded and the map renders on the page. If you only looked at the final output, you would move on.
However, when I asked the agent what it ran into while reading those files, five signals in the log pointed to 8 specific issues in the skills and reference files:
- Fallback escalation (Step 5 to Step 6). The reference files from
gstatic.comare the primary path, and the Code Assist endpoint is only a fallback for uncovered topics. When the agent fetches reference files for a capability and then calls the fallback API for the same capability, the reference files failed or contradicted the rules. - Dead-end reads. In Step 5, the agent downloaded 9 reference files, but in Step 8 it used only 4 of them. A high fetch-to-discard ratio suggests the index descriptions in
SKILL.mdpromised one thing while the reference file delivered something deprecated or unrelated. - Policy collisions between the main skill and child reference files. The main skill lists legacy APIs that the agent must never use. When a child reference file teaches one of those banned APIs, the agent has to abandon the file and search elsewhere.
- Mixed patterns across files. When the initialization files use modern web components (
<gmp-map>andgoogle.maps.importLibrary) while the feature files still use global callbacks (window.initMap = initMap), the agent has to invent glue code to bridge the two. - Local script workarounds (Step 7). The agent dropped into Python to parse the KML file into a JSON coordinate array. When an agent writes its own script to transform data before handing it to your SDK, it usually marks a deprecation gap or friction point in the docs.
The 8 documentation and skill issues the log uncovered
When the agent logged the friction points behind those signals, 8 concrete issues came out of that single task, ranging from a sub-skill link that routed to a disabled DistanceMatrixService doc to a hallucinated routeService.computeRoutes() method and conflicting <gmp-map> attribute names.
View all 8 documentation and skill issues uncovered in the log
- Issue 1: A sub-skill link for route travel modes points to a disabled legacy Distance Matrix doc. In
maps-javascript-api-javascript/SKILL.md, the checklist item “Allows setting the desired mode of transportation (driving, walking, transit, bicycling) for route calculation” links toreferences/specify-the-travel-mode-drive-transit-walk-two-wheeled-for-route.md. The same URL is linked a second time further down under the Distance Matrix section. When the agent fetched that file to see how to setBICYCLINGmode on a route, the file was titled “Distance Matrix Travel Mode Configuration (Legacy)” and taughtnew google.maps.DistanceMatrixService()usingvar. Meanwhile, the mainSKILL.mdstates thatDistanceMatrixServicewas disabled in March 2025 and forbids writing code against it. - Issue 2: A non-existent
routeService.computeRoutes()instance method in the waypoints reference. Inreferences/specify-stop-pass-through-point-for-route-request.md, the sample code defines arouteRequestWithStopoversobject and ends with two commented-out lines:// Assuming you have a RouteService instance: // routeService.computeRoutes(routeRequestWithStopovers).then(response => { ... });There is no
RouteServiceclass in the modern Maps JavaScript APIrouteslibrary. When the agent fell back to the Code Assist API in Step 6, the 3.65 reference doc and official sample (samples/routes-markers/index.ts) showed thatcomputeRoutesis a static method on theRouteclass (const { Route } = await google.maps.importLibrary('routes'); const result = await Route.computeRoutes(request);). An agent that trusts the sub-skill reference file without verifying against Code Assist will either call a class that doesn’t exist or leave the code commented out. - Issue 3: Incompatible initialization patterns across reference files. The primary initialization guide (
add-customizable-interactive-map-web-page-mobile-app.md) teaches the dynamic bootstrap loader (google.maps.importLibrary) and the<gmp-map>web component (accessingmapElement.innerMap). However, secondary feature guides (add-bicycling-layer-map.md,add-geojson-data-map.md, and parts ofadd-shape-line-map.md) still use the older callback pattern (window.initMap = initMap; new google.maps.Map(...)) and instantiatenew google.maps.BicyclingLayer()off the global namespace without showing how it works withimportLibrary('maps')or<gmp-map>. - Issue 4: Truncated Markdown table cells from unescaped pipe characters (
|). Inmaps-embed-api-javascript/references/embed-standard-google-maps-directions-view-into-iframe-web-page.md, the parameter table rows forwaypointsandavoidexplain that multiple values should be separated by the pipe character (|). Because the literal|inside backticks wasn’t escaped when the Markdown table was generated, the parser treated it as a column delimiter and truncated both cells mid-sentence:Multiple values must be separated by the pipe character (\|). - Issue 5: HTML attribute casing mismatch on
<gmp-map>(mapIdvs.map-id). Setting a Map ID is required whenever you useAdvancedMarkerElement; without it, markers fail to appear. Yetmaps-javascript-api-javascript/SKILL.mdandadd-marker-map.mduse<gmp-map map-id="DEMO_MAP_ID">, whileadd-customizable-interactive-map-web-page-mobile-app.mduses<gmp-map mapId="DEMO_MAP_ID">(camelCase in the HTML attribute), and the Code Assist sample setsinnerMap.setOptions({ mapId: 'DEMO_MAP_ID' })in JavaScript. Because the docs disagreed on the HTML attribute name, the agent set bothmap-id="DEMO_MAP_ID"on the element andinnerMap.setOptions({ mapId: "DEMO_MAP_ID" })in JavaScript just to be safe. - Issue 6: Missing
solution_idin the Maps Embed API reference file. The parent skillmaps-embed-api-javascript/SKILL.mdmarks&solution_id=gmp_git_agentskills_v1as a mandatory query parameter for all Maps Embed API URLs. However, the child reference file (embed-standard-google-maps-directions-view-into-iframe-web-page.md) omitssolution_idfrom the base URL structure, the parameter table, and the example URL. This is the exact “policy rules in the wrong place” pattern from my earlier runs: if an agent reads the reference file for the parameters, it drops the attribution parameter. - Issue 7: Inconsistent relative paths in
SKILL.mddependency lists. Inmaps-embed-api-javascript/SKILL.md, links to reference files include thereferences/folder in the link text, whereas inmaps-javascript-api-javascript/SKILL.mdthe link text and*Dependencies*arrays omit thereferences/prefix (for example,["add-custom-geospatial-dataset-map.md"]). - Issue 8:
KmlLayerdeprecation without a concrete client-side migration example. Inmaps-javascript-api-javascript/SKILL.md, the checklist entry says “Displays geographic data supplied in a KML (Keyhole Markup Language) file as an overlay on the map. Read add-kml-layer-map.md” without mentioning deprecation. Only after fetchingadd-kml-layer-map.mddid the agent learn on line 3 thatKmlLayeris deprecated. The file tells the agent to suggest migrating to either cloud-hosted Datasets or third-party KML-to-GeoJSON conversion withmap.data, and then spends the remaining 180 lines showing legacyKmlLayercode anyway. That’s why the agent wrote a Python script in Step 7 to extract the KML coordinates into JSON and draw aPolyline.
The bugs agents surface are much more specific and technical than the bugs humans usually file. In fact, if you were able to successfully follow and understand each of the issues, my hat is off to you.
My next step would be to log these bugs in our issue tracker with as much context as possible, and then pass them to my fix-doc-bugs skill, which applies a detailed algorithm to fix the issues.
The question for the technical writer is how many of these issues are real bugs, how many are fixable improvement opportunities in the docs, and how many should go to other teams. As Dave Rensin argues in Elephants, Goldfish and the New Golden Age of Software Engineering, as agents take over more of the mechanical execution, judgment becomes our primary role. The technical writer’s job now becomes looking at each highly technical, specific, and opaque bug the agent identifies and figuring out whether it’s the right change to make.
Agents as users we can observe
Above all, most tech writers care about producing clear, accurate docs that support the tasks users want to accomplish. When we come back to these “first principles” of tech comm, agents give us a real advantage. For the first time, we have a user we can follow and observe in new ways.
Imagine if, a decade ago, you could ask a dozen users to do common tasks with your docs and observe their thoughts, friction, and errors as they struggled to build something with your APIs. That kind of record would have been a goldmine, and it almost never happened. In my 20+ years as a tech writer, I’ve watched users work with my documentation only a few times. I wrote about those experiences in Best tech writing tip ever: Watch a user try to follow your instructions (2007) and A few notes from usability testing: Video tutorials get watched, text gets skipped (2011).
With agents, we can give them tasks and have them log what they’re doing. If you can’t see an action history and have to rely on the agent’s narration, treat the build log as a set of leads to check, not as a record of what happened. In my follow-up test, build logs were wrong often enough to mislead. Unlike a usability lab, though, you can run these sessions at scale and on a schedule.
For example, say a month of logs shows that many people are building store locators. You write a prompt that captures the typical request, such as “Build a store locator for a chain of 12 bike shops, with a search box and directions to the nearest shop,” and add the build log instruction block. You run it every week.
If agents keep losing a fix cycle on the same setup step, or keep requesting data the app doesn’t need, that’s a pattern worth a bug. After the fix ships, the same prompt becomes your regression test. If the friction disappears from the builds, the fix reached the agent. If it doesn’t, the fix is stuck somewhere between your docs and the agent, perhaps in a skill that hasn’t been regenerated yet. Multiply that by the 20 or 30 most common scenarios, and you have something close to a usability lab that runs on a schedule.
This is close to the approach OpenAI describes in Testing Agent Skills Systematically with Evals. You run a prompt, capture the agent’s trace and the files it produced, apply a small set of checks, and compare scores over time. The difference is where the prompts come from. Engineers usually write their own test prompts, while tech writers can base them on what people are trying to build.
Is anyone else doing this?
I’ve found people and tools working on parts of this. Docs chatbot platforms like Kapa.ai group the questions their bot couldn’t answer into coverage gaps. Mintlify shows which AI agents read your docs and which pages they read. Fabrizio Ferri-Benedetti built Impersonaid, a tool that tests docs against LLM-simulated user personas. Vercel ran evals on how agents use its docs.
Most of this work looks at how people use docs: what questions they ask, and whether they find answers. With agents, I think we’ve moved past that. People now ask agents to build solutions with your products, and those agents don’t come to your help site to ask questions.
Instead of studying the questions users ask in chat, I think we need to capture what people are trying to build, recreate those sessions with agents, and check what the agents actually read and did along the way.
Follow-up: 20 more scenarios
A few days after I published this post, I ran a larger version of the experiment with 20 common map-heavy app scenarios, such as a business website with a “find us” map, a delivery tracker, a store locator, a weather dashboard, a real estate map, and a navigation app. I deliberately chose map-heavy scenarios, because those are the builds where an agent reads our docs and skills.
For each scenario, I wrote a realistic prompt, added the two-sentence hint and the build log block, and ran it myself. For each run, I saved the action history, the exported code, and the skill files the agent downloaded. Then I had agents analyze each run and trace each problem back to the doc or skill passage behind it.
Nearly all of the apps built and ran. A couple hit runtime errors, and one of those needed the builder’s automatic “fix” button.
The hint kept the skill loading
With the hint at the top of every prompt, the agent loaded our skill in every run and built with current APIs instead of from memory. That’s consistent with the hint working, but it isn’t proof. I didn’t run a no-hint comparison this time, and most of these prompts clearly describe a map app, which loaded the skill even without the hint in my first round. The cleaner evidence is still the earlier comparison with long prompts that don’t name the product.
Build logs aren’t reliable evidence
When I compared each build log against its action history, the logs overstated how much the agent consulted docs:
- Roughly one in five docs that a build log listed as a source never appeared as opened in the action history.
- Agents often downloaded skill and reference files and never opened them, or read only the first part of a long file. The logs still cited those files.
- Each log also had several other claims the history contradicted, such as reporting a clean build after several fix cycles.
For the second half of the runs, I added an instruction asking the agent to check its build log against its action history before finishing and not to make anything up. Phantom citations dropped only slightly, and the other false claims didn’t drop at all. The two halves used different prompts, so even that small drop isn’t a clean comparison.
So the build log is useful in one way: its notes about friction often point to real confusion, such as two example files that contradict each other. But the reliable evidence came from the action history, the folder of downloaded skill files, and the code. If your platform doesn’t show an action history, I’d be cautious about drawing conclusions from build logs alone.
Most of the action is in product skills
Nearly every issue traced back to a skill file rather than a doc page on our site. On this platform, the agent rarely reads the docs directly. The skills effectively are the docs.
That shifts where I think tech writers need to put more effort:
- Reviewing the example code in product skills, because agents copy it as if it were a spec.
- Keeping example files consistent with each other. Several problems came from two examples teaching different patterns for the same task.
- Knowing how often skills are regenerated, so that a doc fix actually reaches the agent.
- Rerunning the same prompts after a fix to see whether the problem went away.
The bugs are a different kind of bug
The issues weren’t the usual unclear paragraph or broken link. They were specific, technical, and tied to the toolchain:
- Project configuration steps that the agent discovered by trial and error.
- Credentials handled in ways you wouldn’t want in production code.
- Required attribution missing in some parts of the UI.
- Requests for more data than the app needed, which can cost more.
- Server-side code that works in a development preview but not in a deployed app.
- Shortcuts copied straight from example code.
Most of these don’t break the preview. The app looks like it works. But they matter once someone deploys the app. A handful of patterns showed up in many runs and traced back to a small number of files, so those are worth fixing first. Many of the other issues were one-offs that affected only one kind of app.
I also saw agents invent things when guidance ran out. A few called methods that don’t exist, and some filled in made-up data when an API call failed, without telling the user. Gaps like these don’t show up as errors. They show up as apps that look like they work.
It still doesn’t answer how agents read docs
This test told me a lot about how an agent reads product skills on one platform. It didn’t tell me much about how agents read doc pages, because in this setup they rarely did. To learn that, I’d need to look at other setups, such as IDE agents connected to a docs MCP server, or at analytics that show which pages agents fetch.
For now, my next steps are smaller than the original plan:
- Focus on the few patterns that showed up in most runs, instead of chasing every one-off bug.
- Keep the same prompts as a fixed set and rerun them after fixes land, comparing how often each problem appears.
- Automate the checks. Many of these patterns can be detected by searching the exported code, so the scoring doesn’t need a person reading every log.
Posts on agent readability
- Agent readability: Why agents can’t see content that JavaScript loads on click
- Looking at a site like an agent — identifying semantics in styling
- Chunking your content for agent readability
- Agent readability: Carrying over top-level page notes into independent chunks
Related posts
About Tom Johnson
I'm an API technical writer based in the Seattle area. On this blog, I write about topics related to technical writing and communication — such as software documentation, API documentation, AI, information architecture, content strategy, writing processes, agent skills, tech comm careers, and more. Check out my AI courses for more on the latest in AI and tech comm. If you're looking for info about documenting APIs, see my API documentation course.
If you're a technical writer and want to keep on top of the latest trends in tech comm, be sure to subscribe to email updates below. You can also learn more about me or contact me.
Finally, I want to note a couple of things. I usually write the first drafts of my posts myself and then use AI to improve accuracy, clarity, and detail in places, so my writing is a mix of human and AI. (Hopefully that's not too off-putting.) Also, the opinions I express on my blog are my own perspectives, not those of my employer.