Documentation theater and the acceleration paradox -- podcast episode 3 with Fabrizio Ferri Benedetti

by Tom Johnson on Nov 30, 2025
categories: ai • podcasts

• Audio only
• Articles discussed
• Takeaways
• Shorts
• Chapters
• Transcript

Audio only

If you just prefer the audio version:

Articles discussed

These are some articles we discussed in this episode.

• Code wikis are documentation theater as a service (Fabrizio Ferri Benedetti): AI tools claiming to fully automate docs produce “documentation theater”—inferior byproducts that check a box but lack content integrity and the “why,” and should never be shipped to users without human curation.

• AI hype (Reddit Thread): The author argues that AI can’t replace jobs because it only performs discrete tasks, not holistic roles, and that the real threat is C-suite executives believing the hype rather than the technology itself.

• Changing the AI narrative from liberation to acceleration (Tom Johnson): The “liberation” narrative is misleading; AI actually accelerates the pace of work (rather than freeing up time), increasing expectations for volume while creating a high-risk bottleneck in content validation and review.

• You need an AI policy for your docs (Fabrizio Ferri Benedetti): Instead of banning AI, technical writers should create policies that manage its inevitable use, emphasizing human accountability, rigorous testing (linters), and voluntary disclosure to foster learning.

• Developing an AI strategy for documentation (Sarah Moir): Writers should partner with product teams to feed authoritative docs into native AI agents, optimize content for “machine readers” (AEO), and use AI for internal chores rather than just generating text.

• The difficulty of tracking and interpreting AI usage labels (Tom Johnson): Labeling content as “AI-generated” risks devaluing human work and oversimplifying the collaborative “Centaur” workflow, potentially leading leadership to conclude that writers are unnecessary button-pushers.

• Why long-running tasks autonomously carried out by agentic AI aren’t the future of doc work (Tom Johnson): Autonomous AI agents struggle with the “winding conversation” of documentation work; instead, AI is best used as a collaborative “thought partner.”

• Slack thread from Fabrizio about using AI (Fabrizio Ferri Benedetti): The discussion highlights a shift from writing to “directing” LLMs, noting that success depends heavily on having high-quality foundational documentation to seed the model; otherwise, writers must still draft from scratch to establish this foundation.

Takeaways

Here are some takeaways from the conversation:

• Visual IDEs vs CLIs: For documentation, visual IDEs (like Antigravity or Cursor) are preferred over CLI tools because writers need to see the diffs and iterate conversationally, whereas CLI tools are often better for pure code generation.
• The “good base” requirement: AI tools like Claude can generate 70% complete drafts when provided with rich context, but this relies heavily on having a strong, existing foundation of documentation to “seed” the model or RAG system.
• Documentation theater: Auto-generated “code wikis” can create a dangerous illusion of documentation (“documentation theater”)—they capture the what and how from code but miss the critical why and product appraisal that users need.
• Transparency dilemmas: While disclosing AI usage promotes a culture of learning and honesty, it risks being misinterpreted by leadership as “push-button” work, potentially devaluing the human effort involved in prompting, refining, and verifying.
• The review bottleneck: AI accelerates content generation but exacerbates the review bottleneck. To manage this, use “dumb robots” (linters) for basics and “smart robots” (AI) for initial reviews, and maintain strict PR hygiene (small, manageable changes).
• Redefining productivity: Traditional metrics like PR counts are misleading in an AI world where volume is cheap. True productivity should be measured by the value delivered—unblocking teams, enabling contributions, and focusing on strategic architecture rather than just “plumbing” fixes.
• Swinging for home runs: Instead of just doing “more of the same, faster,” technical writers should use AI to clear low-level blockers and focus on “home runs”—cross-organizational initiatives, knowledge verification systems, and strategic information architecture.
• Editorial intuition: As writers shift to editing AI-generated content, they must actively sharpen their “editorial instinct” to catch subtle inaccuracies and “red flags” that might otherwise slip through in plausible-sounding AI text.

Shorts

Here are some shorts pulled from the longer video.

Chapters

Here are some timecodes for the podcast:

• 00:00 Introduction to AI in tech writing
• 02:05 Exploring AI tools and their impact
• 08:32 AI-assisted writing: A new approach
• 08:58 The importance of a strong documentation foundation
• 14:38 Evaluating automated documentation tools
• 17:40 The dangers of automated documentation
• 20:31 Communicating the value of tech writing
• 21:54 Transparency in AI usage
• 26:34 The role of intuition in AI-generated content
• 28:58 Managing change lists effectively
• 30:16 Overcoming review bottlenecks
• 34:31 Rethinking productivity metrics
• 38:10 Focusing on high-value work
• 40:00 Leveraging AI for documentation
• 42:27 The role of AI in technical writing
• 46:01 Swinging for home runs in documentation
• 53:57 Navigating AI hype and reality

Transcript

Tom: Welcome to another episode of this joint podcast between Fabrizio Ferri Benedetti and me, Tom Johnson. Today we will be discussing a number of articles and topics, just kind of going over the latest happenings, news goings on, going on in the tech comm space, especially related to AI documentation. That is the focus here, AI and tech writing. So Fabrizio, I know you’re under the weather a bit there, but I appreciate you joining and powering through. Do you have anything you want to say as opening words to people?

Fabrizio: Yeah, well, no, you know, just sometimes life happens. I kind of envy the LLMs I work with because they don’t go through this or DL every time, but it really makes you feel like very organic, but otherwise I’m well.

Tom: Yeah, we’re celebrating Thanksgiving here. So I’ve had a long weekend of not doing a whole lot, except we hosted like 20 gazillion people at our house. So it was a little bit exhausting, but I forget that Thanksgiving is not a global holiday. It’s very US centric.

Fabrizio: Nice. Yeah, even though I really like the concept, I think is one of those celebrations that could be exported worldwide. The idea of it is nice.

Tom: I like Thanksgiving because it’s one of those non-commercial holidays where, you just have people over, you go visit people and you have good food and you sit around and maybe even watch football. It’s not like Christmas where you’re really expected to spend a lot of money or Halloween where you dress up and then eat loads of candy. Not that I do that, but, it is, it is a nice, enjoyable, weekend. So, but I,

Fabrizio: Right. Indeed.

Tom: I’ve been kind of resting a lot and I’ve got lots of ideas. Was able to read through a lot of these articles and I’ve got thoughts and I was experimenting. Yeah.

Fabrizio: So you were able to stop the acceleration thing. We’re going to come to that, right?

Tom: Yeah, you know, it’s yes, okay, so you’re referring to this my idea that acceleration is like a defining quality of AI. And yeah, I think I was able to temporarily stop the acceleration. So that’s nice.

Fabrizio: Right, yeah, yeah, yeah. Indeed, indeed. It’s nice to see that we are capable of stopping and pausing not just when we run out of context or tokens, you know.

Tom: Well, I did I did experiment with something that I was pretty excited about this release of anti-gravity this new like visual studio coding app with Gemini integrated Gemini 3 and it is phenomenal It’s actually very close to the sort of setup at work that I have so I was excited to see that and Yeah, I was doing some stuff. I was updating my book club site or page, you know, I wanted to

Fabrizio: yeah. Hmm.

Tom: change the structure, add a bunch of stuff, was able to really do just put AI to work there. So it’s nice. Yeah.

Fabrizio: Yeah. Yeah, so right now I’m an enthusiast user of cursor. So if you could like write a review of anti-gravity, know, Doc Soriano, maybe you can get me to anti-gravity because I was a bit taken aback from the fact that it felt like, yet another Visual Studio Code fork.

Tom: Hmm.

Fabrizio: And I’m sure there’s depth to it and there are features that I haven’t explored yet, but I felt like I kind of lazy in front of it.

Tom: Well, I’m biased on this whole perspective because of just like, you know, working at Google and so on and being so familiar with Gemini and the way these tools work. But I tried Cursor, I tried Windsurf, not heavy, not as a heavy user. I mostly use AI during my day job at work, but…

Fabrizio: Ha ha. Of course. Right.

Tom: I do think anti-gravity is hands down a great… Coding editor for writing and doing other stuff. And Gemini 3 is pretty great too. So like, I don’t really know what the token limits are. I was trying Claude code again because I always read online on Reddit. People seem to love Claude code. And I was like, man, what am I missing? It’s been a while since I bounced around across different AI platforms. And so I was just checking it out to compare. But yeah, I think, I think.

Fabrizio: Yeah, yeah. Yeah. Yeah. It’s funny because the CLI experience like cloud codes, I think is great for code for some reason. Think it’s more responsive, more down to the code. But for docs, I actually prefer to have an IDE where I can see the doc evolving under my eyes. I can’t really explain why, but it’s like that for me.

Tom: I’m 100 % on the same page there. I’m not a huge fan of the CLI tools in part because I feel that the writing process is so iterative and I want to see changes and have a conversation and go through many turns with it. And it’s just so much easier with more of a visual side pain than the CLI.

Fabrizio: Yeah, yeah. Yeah, also because I don’t know for documentation at least I sometimes I feel a bit wary of the small changes, know, the subtle changes and I really want to see like the whole diff in front of my eyes and it’s more difficult with CLI tools.

Tom: Yeah. Definitely like that experience of seeing what the changes are, the file diff preview and being able to accept reject is superior in the IDE instead of the command line area. I honestly, unless you’re really using the command line a lot and you’re more comfortable there, I would stick with just the, I don’t even know what you call it, just like the side pain sort of integration. Anyway.

Fabrizio: Yeah. Yep, agreed. Yeah, what, you know, Gemini 3 aside, what would you say is the killer feature of anti-gravity? The thing that is most useful for you writing docs like…

Tom: It it does a lot of planning and thinking like it will go through and make a plan first before it does updates and I usually just hit yeah go for it. I accept the plan.

Fabrizio: Yeah.

Tom: And I think that makes it so much smarter. I’ve just become so, I’ve become very familiar, familiar with like scenarios where it excels and scenarios where I need to have more human input. So, I don’t know. You can also, it’s got like MCP integration so you can add things. I believe it makes that a lot easier. I haven’t really dug deep into that part of it yet.

Fabrizio: Yeah. Yeah. Yeah. Yeah, yeah.

Tom: You can, you can add rules and other things more easily there, but, I’m still, I’m still kind of coming up to speed on it. It’s not a tool that despite the fact that it’s externally available and it’s everywhere, it’s not the same IDE that we use internally. So.

Fabrizio: Yeah, it’s interesting that there are like, can like identify different styles of using these tools. And for example, do you use the auto-complete, the iPower auto-complete much? Yeah.

Tom: Sure, sometimes. Actually, not really. It mostly gets in the way. I mostly want to turn that off. I need to figure out how to turn it off because it’s like I don’t need a tool. Auto-guessing how to complete every time I’m writing is a word.

Fabrizio: You Yeah, yeah, yeah, yeah. That’s interesting because I know writers that despise the autocomplete, you know, in cursor, and I taught them how to snooze it because you can snooze it. But I do like it. You know, I like the small, the way it learns from the patterns of what you want to do is kind of a game. But I think it really suits the different personalities, you know, of how you use approach to tool or like this.

Tom: Yeah. Yeah. Well, hey, maybe that’s a good starting point for one of these articles we want to discuss. And this isn’t really an article, but a thread you were starting on, on the Slack channel about how,, you’re using AI to do more automatic sort of writing, or I don’t know what you call it AI based writing. And was kind of a funny feeling. And I don’t know, you, you were talking a little bit about this transition. You want to share more about.

Fabrizio: Yeah. Yeah. Yeah. Yeah, Hmm.

Tom: this direction that you were hinting at there about.

Fabrizio: Yeah, it kind of started experimentally. I had these tasks of this document that we had to create from scratch. But it’s one of those documents where you really have to put together multiple sources. And I was a bit short of time, which again is the key thing in these situations. And what I did was I provided a prompt with lots of contacts, URLs, and I just let Claude do its thing. And the draft surprisingly was 70 % there, was quite good already. It’s happening like increasingly, it’s increasingly happening that I start like that and then I refine, I steer the LLM to do something different, but it’s very good to kind of break the ice sometimes when you’re stuck. Sorry, my voice is like, my gosh. So yeah, but there were very good comments in the thread and I think one key aspect is you can do this when you start from a very good base, when you have… Already like a very good corpus of documentation. And then you can afford to do that. Otherwise, it’s more difficult.

Tom: Yeah. Well, starting from a good base is an interesting comment because I do see a lot of people make the argument that, yeah, if you’ve already got a good foundation of documentation that you can feed in as context to the AI, the LLM, whatever we’re calling it, then it’s able to leverage that and produce pretty good drafts.

Fabrizio: Hmm. Hmm.

Tom: But many tech writers might not have that good foundation. And I was just trying to imagine what that even would be like. Usually in my experience, let’s say that a team is creating a new API, which might just be a method added to an existing API. Well, before they start coding this, there’s going to be some kind of product requirements document, some kind of

Fabrizio: Yeah. Yeah.

Tom: two pager that pitches the need for this. There’s going to be the engineers usually have written the code. So there’s going to be a code artifact with comments and output. You know, by the time it gets to me, I’ve already got probably 50 pages of source material, engineering design, security reviews. I will vet the content and maybe remove stuff that didn’t get coded. But if I’m just writing an API overview,

Fabrizio: Hopefully.

Tom: I can draw upon a rich like foundation of material. Just, do you, are you in scenarios where you don’t have anything and you’re just like, making your way.

Fabrizio: no, it’s almost never. Like there’s always, we are always building upon an existing foundation. So it’s very rare, but I can see scenarios like a startups, for example, where you have to maybe write or contribute those foundational documents. And, and I think it’s, it’s like seeding like a rag or a knowledge base or any, you know, because in the end, like cursor, for example, uses its own local cloud-based rag also. And it learns from your edits. It learns from your reposts.

Tom: Yeah.

Fabrizio: and it becomes increasingly aware and more efficient at doing the edits you want to do. So it’s like feeding the machine a bit and there’s always the need for that I think but maybe less so when you have like a very rich documentation base from where you’re starting from which I case you know it’s in my case it’s like that we have thousands of documents we can rely upon.

Tom: Yeah, maybe I’m taking for granted some… Extra like rag intelligence because the models we use at my work. They’re trained on Google’s internal code base And there’s like this giant mono repo of code You could just search code across the company and all that is used to train it So when I’m using the same AI tools to write documentation It’s really got a good foundation of all that other code so

Fabrizio: Right. Yeah, that doesn’t model also include like the monorepos include docs or it’s a separate system or repo.

Tom: Yeah, the docs are literally treated as code, both policy-wise, tool-wise, for the most part. Obviously you don’t have to pass a security review to release a new document. But yeah, that’s actually been one of the frustrations is that because docs are treated the same as code, the security and other policies are a bit more stringent than we would like. But…

Fabrizio: okay. Right.

Tom: Yeah, it’s now to the point where most everything is open and accessible and you can use everything that you want. But yeah, it blends the same environment. It makes it really easy for engineers to write because it’s like you just use the same editor, the same workflow.

Fabrizio: Yeah. But you see, think something very similar happens with these code wikis thing where you get better results if you start from a repo that already contains docs, of course. You know, it’s way different when you just provide like, is this tool, you know, automatically generates or so it claims to generate like an entire doc set or a wiki as they call it from code.

Tom: Yeah, yeah, I was experimenting with that yesterday, actually. So there’s two tools that we were talking about earlier. One is CodeWiki.Google, and the other is DeepWiki. Both of these tools allow you to feed it a GitHub repo. It will index it and then write you some docs for it, right?

Fabrizio: Yep. LiveWiki, yeah, yeah. That’s right, yeah.

Tom: Sounds like just like a recipe for garbage, right? However, so I fed it my, my own blog, my, my, my site, Tom, joh t dot get hub dot something. And I fed this to deep Wiki because I don’t think I could do it on the code Wiki one. And after about 15, 20 minutes after it finished indexing my massive site, it produced documentation.

Fabrizio: Yeah.

Tom: And I was suddenly realizing that, yeah, there’s actually a lot of internal know-how about how my Jekyll theme works and how I produce PDFs, for example, of my API documentation course, or how my collections work and my different like…

Fabrizio: Yeah. So it just focus on the code bits, not the docs bits.

Tom: Yeah, it kind of ignored the posts, but then I was like, well, did you index that? And I asked it a question. What does Tom Johnson think about Zumba and technical writing, which is refers to a post I wrote like a decade ago comparing the two. He didn’t find it. But then when I paste it in the title, it did find it and it told me about it. And then I said, Hey, what does Tom Johnson think about Wikis? And it seemed to look through all my posts and give me some kind of interesting response. So.

Fabrizio: Yeah. Yeah, yeah, yeah. Well, I think your blog is probably part of most training sets anyway.

Tom: I don’t know, interesting. Well, there’s a lot to a blog. If I were to inherit your, is it Hugo based site, Passo Uno? And let’s say I wanted to, I don’t know, work with it. Is it documented? Is it, have you described how it’s set up? Like, say I wanna.

Fabrizio: Yeah. Yep. Right, yeah, yeah, yeah, it’s… No redmi file, no nothing,

Tom: And I think a lot of GitHub repos are the same. Who has time to sit there and document how their little hobby site works? So it’s better than nothing. But of course it’s like not really that close to actual docs, but anyway.

Fabrizio: Yeah. Yeah. Yeah, what really made me frown is the is the, you know, is the pedagogical aspect of it, like, you know, that they have to understand what Docs is before saying you’re going to create Docs. And I’m not always sure they know, you know, you know, that meme from the Princess Bride of, know, you keep using that word, but I don’t think you know what it means. So it’s like that, you know, for Docs. And I’m like,

Tom: Yeah.

Fabrizio: We really, know, we tech writers should do better at promoting what we’re doing and explaining that those are not really good docs, you know.

Tom: Yeah, and this gets into your post about documentation as theater, which is an analogy I really like. The idea that something can look like docs, like these Wiki outputs that just automatically index and generate something that feels kind of like docs, and maybe to a C-suite group, they’re like, yeah, this is good enough. We don’t need to have perfect docs. Nobody cares about.

Fabrizio: Yeah. Right. Yeah.

Tom: perfect grammar, you know, let’s just go ahead and run with this. And that is a dangerous idea. How do we avoid the C-suite from reaching that conclusion that automated docs are good enough and that these tools have gotten to the maturity point where we don’t need tech writers? Like how do you change that narrative?

Fabrizio: Yep. Yep. Well, it really depends on the situation, but probably the first thing I would do if I’m like, say, at small startup is just ask them to follow those docs to do something with the product. And they probably won’t be able to. But also, Interestingly, CodeWiki and DeepWiki are very much focused on the internal docs side of things, like architecture, how to compile the thing, how to run the thing, how to develop the thing, but not really about why you should be using this. The whole side of appraisal or selling or the why. You should be using this as there was this comment on LinkedIn by Manny Silva was excellent because really this is about the what and maybe the how, but not about the why. And I think that’s something we can get to explain to the C-Suites. The other idea is a bit more crazy, but I was actually thinking about writing a business novel, like the Phoenix Project, but for tech writing. Know, all the ideas that come to my mind are very lame, but I think it’s the way to put something in front of the eyes is to write a business novel. Despite these are genre that I generally despise, but it’s something maybe that we have to do. Don’t know.

Tom: Okay. Wow. Business a business novel. What do you mean by this?

Fabrizio: Yeah, you the genre, you know, like, is like a fictionalized account of, of a business story that is really like just a vehicle for promoting an idea or framework or whatever. Phoenix project is a very famous example of that in the IT world. You know, you know, it’s kind of in between like a novel and self-help or business. But yeah, I don’t know. I’m being very strategic right now about things.

Tom: Yeah, okay. Yeah, it’s an interesting conundrum because like, Techcom for the last 60 years has wrestled with this question of how do we communicate the complexity of our job upwards so that people don’t think we just fix grammar and make language pretty. And so now, suddenly we have to communicate that yeah, we do more than just generate content at the same level as an automated tool.

Fabrizio: Right. Yeah, yeah.

Tom: But are our C-suites suddenly going to stop and think, yeah, look at all the stuff this tech writer is doing. I just don’t see that. But here’s one thing where I was pushing back on. We want to quantify how much AI we’re using as we write docs so that we can measure velocity and impact and show that, we’re integrating. So I started adding some labels to my.

Fabrizio: Yeah. Yeah. Yeah.

Tom: pull requests, change lists, kind of indicating that I’d used AI, and we had a structured tag and so on. But then I started to get nervous and think, wait a minute, how is the C-suite going to interpret this label that AI has been used? Are they gonna see that it still took me five hours to write this update or whatever it is? Like maybe it was pretty severe or pretty in-depth.

Fabrizio: Right. Yeah, yeah.

Tom: they’re probably just going to think that I pushed a button and that there was no human really, human input. Then I decided to get rid of the label and stop using the label. What do you, what are your thoughts on how to identify AI usage? Because you were writing about this in your policy post and you did advocate for disclosing AI usage. So you’ve clearly thought about this.

Fabrizio: Yeah. So for now, it’s mostly about honesty in the sense that there’s no way or no sure way of identifying AI contributions. It’s very difficult. Of course, you have to queue some types of. But also they accuse themselves like, a hyphen. Well, dash. Well, to the point that this kind of demonized right now, but they’re not 100%. They’re not fireproof. So to me, it’s about the honesty and learning together of saying, and also a company level of saying, I’ve used the I, and I’ve used it this way. And it’s perfectly fine, you know, there’s no way I shaming, there’s no concern, but describing the whole, you know, eventually you could pace the whole artifact of the conversation you had with the AI, but I think that’s probably going too far, you know, in terms of auditing, et cetera, to me is more about, yeah, just being transparent about it.

Tom: Yeah, I’m still wrestling with this question because when I try to get more specific, for example, let’s say I want to quantify how AI was used or maybe even a rubric was AI used in. Proofreading versus content generation versus something else. It gets to be way too much detail. Nobody seems to care. Think really mostly engineers just like the heads up if something’s AI generated so that they look at it a little bit more carefully. So I usually just put a little note on my change list saying, hey, AI assisted, please review carefully. But at the end of the day,

Fabrizio: Hmm.

Tom: The user really shouldn’t matter if AI was used. It’s the final output. And you could bias the reader one way or the other. Yes, AI was used. No, it wasn’t. It’s the, yeah.

Fabrizio: Yeah. Yeah, yeah. Yeah, think at some point I believe this will become more irrelevant. The reason, for example, why I’m doing the disclosure thing voluntarily is because for two reasons. One is I want to sort of promote a culture where there’s no shame in admitting that you use the eye. So there’s showing that you use the eye. And also, there’s like the sheer concern of, see, you know, I’m doing something new and I want people to think together about whether this is a good usage, good scenario or not. And this is especially important. Like, for example, if I only use AI auto-complete, I probably won’t disclose it because it’s, you know, it’s commas, maybe it’s period at the end of the list items, is irrelevant. But if I use the eye, for example, to create like a first draft, then I want to explain like, you know, how I did it. And I think people can learn from it. And at the same time, they can raise valid concerns on like maybe, you know, maybe you should have used a different prompt or you could have done things differently or do like another iteration. So there’s a bit of learning together process, in my opinion.

Tom: Well, it’s interesting. I keep thinking at some point engineers will flood me with all kinds of CLs that have a lot of automatic or AI written documentation. But I really haven’t seen that. I’ve seen a few where I’m like, it’s obvious that AI wrote that because I’ve seen previous writing from this person and it’s not that level, but honestly, I’m happy to get AI written stuff. It’s usually like really pretty good. And if it’s coming from.

Fabrizio: Yeah. Yeah.

Tom: an engineer I assume they reviewed it for accuracy so it’s now just a matter of style and I even have a button I can click to just apply all our style updates and it most it mostly works you know but here’s one thing I have to have to learn to do more

Fabrizio: Yeah. Right. Yeah, yeah.

Tom: In this world where we’re using AI to generate content, we really have to pay attention to our own intuition about things. If something feels like maybe that’s not right or maybe that’s still confusing to an engineer, I have to like really listen to that and play up a really, really like focus on that. Be like, you know what? That’s like a little red flag. Don’t just ignore it, probably really is confusing, or it probably is wrong, or there probably is something. So that intuition can easily become muted if we’re just cranking stuff out and copying and pasting it.

Fabrizio: Yeah. Yeah. Yeah. Yeah, yeah. I think it’s, I think these, these, well, you know, these, these changes that we’re going through are really calling for like a stronger editorial instinct, you know, like training our editorial skills. As if it were like, proof reading something or editing something as editors. I feel like an editor some days of, yeah. And which I think is fine. It’s providing lots of value. And with developers, I think they usually have already a very strong pull request hygiene in general. So for example, you will seldom see a humongous PR. Of code or docs coming from a developer because they despise very big PRs, know, usually, Yeah, I’ve just seen this example the other day in the in the OCaml programming language repository. There was this guy who created like added like 10,000 lines of code and he was claiming well, the AI knows about this stuff. And my gosh, the maintainers were so patient, you know, but There’s no way that something like this could go through in a corporate setting. Think it’s more sometimes external contributors pass us by. And eventually that will probably disappear. I don’t know.

Tom: I agree. Engineers definitely have good pull request hygiene. Like that the way you phrase that I was thinking about a project I was, I was working on a few months ago where I was moving content and I, it was like a 10,000 word pull requests.

Fabrizio: Yeah. Whoa. Well, sometimes, you know, you have to do it, but.

Tom: Cause I was not only moving things, I was kind of renaming things and I was fixing out of date things and I just shoved it into one big, we call them changeless. And the engineer immediately was like, can you break this up? So I broke it up into like five or six and yeah, it was a lot more manageable. I don’t know why I didn’t do that in the first place, but yeah.

Fabrizio: Hahaha. Yeah, yeah, yeah, but you know, it’s almost like a learned response from them. Know, like they see it, a big PR is like, please break it up.

Tom: Yeah, I think you’re right like immediately they’re like, yeah, that’s that’s not a good thing but that hygiene of small changes and reviewing that is a good way to get around this like AI Errors and hallucinations because if you can put more focus and attention on it. I kind of thought in the back of my mind that I was

Fabrizio: Yeah, yeah, yeah. Yeah. Yeah.

Tom: Inflating my number of change lists by like, you know dividing it I mean what if you took and every single page change was its own change lists it seems like you’re just trying to increase your numbers Tom, but not really. I mean To a degree it just makes the review easier I’ve also okay. So this gets to this larger question about how we get around the bottleneck

Fabrizio: Yeah. Yeah.

Tom: of the review that is where things slow down because you can generate 10,000 words pretty fast, right? But how do you get it through the review cycle? That’s where everything slows down because you have to you have to know who to review it. You have to send them the request. You have to follow up on it. You have to implement any kind of comments and changes. You apply other linters and error checkers and all that takes time.

Fabrizio: Hmm.

Tom: And this is something that people are not realizing when they want to replace writers with AI. It’s like you, still going to need that review. How are you breaking the bottleneck of the review process? How are you accelerating that? Are you doing? Yeah.

Fabrizio: Yeah. Well, yeah, so for now it hasn’t become a problem yet for me, too much of problem in the sense that I haven’t seen yet like very, like a big increase in documentation contributions yet. The strategy I would follow if things become unmanageable, it would probably be a combination of… Like dumb robots and smart robots, what I call dumb and smart robots in sense of a combination of the dumb robots are the linters, for example. So you do like the low level checking of, you know, it’s following this tie guide, the links are correct. It’s not hallucinating on basic FATS or sources. And then you could have load some of the review work to an AI that follows repo instructions, for example. And Well, it feels a bit like, you know, when I was a kid playing with Transformers, you put a robot in front of another robot and they fight. And it’s a bit like this, I think, the sense that you are pitting an LLM against another LLM. And the result is not guaranteed to be great, but at the very least, there will be like an initial filter. So that’s something I would consider doing.

Tom: Yeah, I’ve got kind of that already set up. We have this nice little like Gemini checkpoint that reviews all change lists for like errors right there. I always use that and a lot of times it finds things that I’m like, yeah. So I really want to like present the change list to the engineer in a perfect state so they don’t really have a bunch of issues.

Fabrizio: Hmm. Right.

Tom: Yeah, I’m getting more to the points where, this is becoming more of a challenge for me. Sometimes I’ve had like 10 change lists queued up and people are not reviewing them. And I’m like, why aren’t they reviewing them? I’ve kind of whittled down some strategies that I think are helpful, like having a single owner for a change list so that it’s like clear what the responsibility is and who’s who we’re waiting on. I, I don’t spend a lot of emotional energy pinging them immediately. Give them a day or

Fabrizio: Yeah. Right.

Tom: to, to just see the incoming change list in their queue before I start reaching out on chat and being like, Hey, what’s up? So there’s that, but, but also just in general, it’s the slow part of everything. And especially when they’re, when they have changes that require updates, it’s like, it, it slows everything down. I want to go super fast, but yeah, it doesn’t always happen.

Fabrizio: Yeah, yeah. I guess that, like, for example, when you try to implement those labels… I suspect this is also tied to the metric system you use at work for measuring performance or whatever. I don’t know if that’s your case, but we have to be careful, I think. We have to rethink maybe productivity metrics in the sense of what is really productivity now is not the amount of PRs, it’s not the lines of code you’re turning. It’s probably never been, but Before AI, at least you had a guarantee that those were all manually made, except they copy pasted or whatever other trick they pulled. So now I think we have to rethink a bit what productivity means and looks like. And I think back to the acceleration topic.

Tom: Yeah, I am really waiting for somebody to come out and definitively show productivity metrics around AI. I feel like most of us are just saying, yeah, this would have taken me a long time and I did it in a fraction of the time.

Fabrizio: Yeah. Yeah.

Tom: But how do you really know? Did look at my number of change lists that I had like submitted for the last five plus years because I’ve stayed in the same org for now six years. Actually today is my six year Googleversary. It’s kind of a silly thing. But yeah, actually this is the longest I’ve been at one company. I’ve been at a couple of others for five years each, but never six years.

Fabrizio: Yeah. Hey, congrats.

Tom: So why did I bring that up? Talking about…

Fabrizio: Sorry, I was picturing you with the nougat cup, beanie cup with the spinning, yeah.

Tom: You I still have that cap. Don’t really know when it’s appropriate to wear or, but yes, it’s kind of a fun thing. What were we even talking about here? I’m completely lost. Productivity. Thank you. Okay. So, all right, because I’ve been in the same spot for so long, I’m able to sort of look and see, am I more productive? And yeah, when AI came out,

Fabrizio: Yeah. Well, productivity. Yeah, yeah, yeah. Hmm.

Tom: over the last couple of years, I think I definitely have submitted a lot more change lists. But it’s very hard to really compare because I’ve also been working on more APIs and generating the reference docs for more APIs. I’ve migrated a bunch of content from one site to another. All this messes with the code deltas. And so if you look at bugs, I’ve also like… Done more bugs, but maybe I’m chopping them up smaller. Like it’s very hard to measure.

Fabrizio: Yeah, you know, I think the way I would go about it is first of all, you are more productive if you can do the type of work that provides more value instead of the type of work. Like if you do more of the business as usual maintenance work, but you’re not doing the works that really matter. That really can make a difference, then you are not more productive, in my opinion. Like, you know, if you close lots of small bugs or fix lots of typos, that’s good maintenance work. But in the end, you want AI to take care of that so you can write the tutorials, you can write the high value pieces that you’re really trained to do better than other people. And then I will also measure this in terms of, for example, issues. You know, growing stale in the backlog. And I probably allow you to unblock those things earlier than just leaving them forever. And related to this, the third thing that comes to my mind is opening new avenues that weren’t possible before. So for example, contributing to the tooling is something that makes the team faster because you can add those features to the tools that you had to wait developers for before and now you don’t have to wait. So for me, it’s more about not having to wait so much as before when you want to do something. So it’s not work coming at you, but it’s more like doing the work you want to do. Doing the high value work more than the low value work. That’s my picture there.

Tom: I really like your point about how, like when you look at productivity, you have to look at the value of what you’re outputting. Cause you’re right. If somebody just goes and fixes, let’s say a hundred like typos in docs and they submit a hundred PRs or change lists and they, a bug for each of them. Yeah. They can easily inflate the numbers and make it look like they’re hyperproductive.

Fabrizio: Yep. Right.

Tom: but what value is really being delivered at the end of the day.

Fabrizio: That’s the thing, yeah.

Tom: I do want to get to the point where I’m focusing on these high value tasks. I feel like I’m still mired in so many low level fixes like, you got to fix this part. We didn’t mention this. This needs a little more clarity or this comment is, is this link is bad in the reference section. And I’m just doing all these small little things, but I really, honestly, I feel most comfortable when I’m working on some meaty task. Like want something, you know, challenging.

Fabrizio: Yeah. Right? Yeah. Yeah. Yeah. Yeah, yeah, we it’s fine. Sometimes you have period of like doing Docs plumbing as I would call it, you know, like where you patch stuff like there’s a pipe leaking, whatever some metadata glitch. But in the end, you know, no matter the volume of those fixes, they’re not going to move, you know, tip the scale towards like benefits or whatever.

Tom: intellectually. Yeah. So are, are there any kind of larger efforts you’re tackling? Like, because we, constantly hear this line about how AI will free us up from these, the plumbing tasks and allow us to do more strategic work. But if we’re also just getting 10 times as many plumbing tasks, it’s like, we never get that time to think big. And as, as we approach, yeah.

Fabrizio: Yeah. So we have these, yeah, we have this big project at work where we want to restructure a sizable part of our documentation. And that will require a human, team of humans being there and we’re thinking information architecture and so on. So at the same time, what I’m working on very much is automating and making it easier for others to contribute to those bits of the docs that are resource strategic, but they’re still important, of course, like reference documentation, for example, we want to make it very easy for developers internally to contribute to those docs and have the checks in place. Like we are implementing the bail interchecks, for example, we are thinking about doing like a library of docs prompts that folks can use. So if we manage to offload lots of that workload to developers, for example, using AI tools and we do like a review, the review part of it, then we are free to focus more on the big architectural changes, for example.

Tom: Well, it reminds me of the wiki days when everybody was orienting things for crowdsourcing and developers were going to write docs. I still haven’t seen much takeoff there. The fastest way for a developer to write docs is to file a bug for a tech writer to write the docs. It’s like, maybe not the fastest, the easiest, the lowest effort is for them not to do anything at all. But.

Fabrizio: yeah. Yeah. Yeah.

Tom: Yeah, anyway, sorry. I’m a little bit jaded about the whole crowd sourcing thing because it never worked out for me

Fabrizio: You know, yeah, but you know, yeah, but the good thing, if there’s a positive side to your code wiki or deep wiki is that developers will generate these stock sites out of nothing, out of the code, and they will look at them and they will feel proud and then they will crash land on the reality of customers saying this is rubbish. And then maybe then they will realize, you know, through those failed artifacts. They will realize that there’s a craft behind it. And maybe they will look at our profession in a different way. That’s my hope. You know, similar to what happened to design in the 90s, you know, when everybody thought that they could do web design with tables and Photoshop, and then they realized, there’s there’s more to design than that. Well, yes, you know.

Tom: Yeah, any. Bye. Yeah, I can definitely see the logic there, right? You have somebody actually do the job of writing docs and they’ll realize that it is hard. It’s a lot more involved. Do you ever fear that you’re going to become an editor of AI slop? Maybe people are just gonna…

Fabrizio: Yep. Hmm.

Tom: use AI to crank something out, they make sure it’s accurate but they know stylistically it’s full of problems and they just throw it over to Fabrizio to fix and to publish.

Fabrizio: Well, That’s a good question. It really is. This is really tied to the volume we’re talking about, you know, if it’s a huge volume or not. But I don’t know if something like that is happening, I think is because there is either a system of incentives that is wrong somewhere, you know, or yeah, something is wrong in the system that leads to that, you know.

Tom: Yeah. Yeah.

Fabrizio: because otherwise there’s no reason for producing so many changes using AI. They must be, it could be like, I don’t know, a hackathon or week of docs or whatever. Maybe they got like the incentive of let’s create a docs PR or it’s an ego metric somewhere. And some folks are doing this, but it’s usually it’s temporary. In the end, want to do good work in general. At least that’s my… Perhaps I’m a bit naive, but my belief is that people will want to be remembered because of good work. So they will not create lots of rubbish just because. But yeah, perhaps I’m a bit innocent, I’m not, you know, to your question, I’m not too concerned about being, becoming an editor. I think it’s part of our job is to do editing. We just need perhaps to be, but that’s wider than that. I think that these things happen, especially in a DoxSense code environment. And in a DoxSense code environment, we need to maybe take up and learn from developers some of those good repo, PRIG practices, for example, that maybe they were unknown to us until very recently, but we need to act more like those developers, I think.

Tom: Well, I really am sort of.

Fabrizio: you

Tom: Enamored of this idea of swinging for the fences and not getting mired down into editing small I don’t want to have I don’t want to become an AI slop editor and I don’t want to become the band-aid plumber Person where I just have I’m supporting 40 teams and they just throw me little fixes here and there and that’s all I’m doing all day that’s a that’s our recipe for Lack of for no full no fulfillment, right? Really a boring kind of job. So I’m thinking like what

Fabrizio: Yeah. Right. Yeah, yeah. Yeah.

Tom: What are the home runs that I’m swinging for?

Fabrizio: Hmm.

Tom: And this ties into this larger frustration. Think many of us have about AI tools. We’re, we’re told that AI tools AI is going to cure cancer. It’s going to bring about acceleration of science. It’s going to address climate change. It’s going to solve a class issues and, I don’t know, abundance, you know, by bringing it economic prosperity. And yet what are we all working on? Who’s working on all that stuff? Maybe there’s tons of people focused on.

Fabrizio: Yeah. Heh. Heh. Right,

Tom: that but it seems like we’re all just focused on doing more of the same but faster and maybe better and so we’re not really swinging the for the home runs and in Docs what are the home runs that we should be focused on so that we don’t get stuck in all that the trivial work

Fabrizio: Right. Yeah. Yeah. Yeah.

Tom: I’ve got some ideas of what I want to do at my job. But there are things that also nobody’s asking me to do. There, there are things that are cross organizational. Like nobody would own them. And if I did it, it would be like, hard to, see, would be hard for me to see like a specific leader of a particular group getting excited about it. So I don’t know. It’s a tough question.

Fabrizio: Right. Wait. So, you know, for me, it’s really about one thing is like I’m blocking. Like I think tech writers are usually living in frustration because they’re blocked by something because we work so much across organizations and teams. And maybe we’re waiting for design to churn out like a diagram or we’re waiting for developers to add a feature to the doc site. And I think… LLMs can help us to unblock those things. And in that sense, they can grant us freedom to go forward with our real goals. And then the other side, well, that’s more difficult, but anything that can provide us with reliable knowledge. And there are tools like there, I’m trying out this tool, Falkorer, for example. There are many tools that do like knowledge base construction using Rack from across multiple sources. And I think that’s promising. We have built this experimental MCP server that calls upon a semantic search engine at work. And it’s fantastic because I can just ask the MCP server to do like an accuracy check based on our docs. Of the first draft, for example, that I got from an AI. And that’s already like a kind of knowledge verification and tapping into knowledge that really speeds up my work. It’s not really about the writing substitution. It’s more about everything that can help us summarize, break down concepts. Yeah, I have this mental picture of a washing machine of, you take different sources here and there, Slack conversations, which are very opinionated, and you throw them into the LLM and what comes out are like basic, you know, elements you can use to build that documentation, you know.

Tom: Yeah, I that ties in with one of my goals, to really understand the flywheel of just like testing content against an LLM and evaluating the accuracy and then improving the docs based on those scores. This is.

Fabrizio: Right.

Tom: Something that I think is a workflow that I hear talked about a lot where you constantly test your docs against like an LLM, you identify the gaps and then you plug them, maybe you throw in analytics from user queries, I don’t know how you get those, and you just keep improving and improving your content. I hear that talked about and I’ve never met anybody, I think I’ve met one person who’s actually like in an executable kind of loop where they’re doing that.

Fabrizio: Hmm. There are some interesting conversations happening about like a Docs benchmark for AI. And I think it would be great to have something there with the caveat that we have to be very careful about. Because the moment you have a benchmark, two things will happen. Somebody will try to game it because it’s a score. And then depending on what you’re measuring, that could steer.

Tom: called. Yeah.

Fabrizio: the development of tools towards a direction that maybe you didn’t want to. Like, you have to be careful there. But you know, the Pelican drawing test by Simon Wilson, like he tests these DLM’s by drawing a Pelican riding a bike. Well, maybe we need something like that for docs.

Tom: Yeah. Actually, I don’t know. Thought you were going to say something else. There’s a, there’s a test where you ask it an L O Y C. I got it. The test is to ask the L to do something that’s not going to be in its training data or something unfamiliar. Yeah. Got it. Okay. Pelican riding a bike. Yeah. Yeah. Well, I think this would be a noble kind of goal to really dive into some of these tools. There’s so much like, I mean,

Fabrizio: Right. Right.

Tom: Let’s see, one of the other posts we were reading or I had in this list to discuss was Sarah Moir, Moir, M-O-I-R, Moir’s on strategy for docs or policies for docs. And she was talking a lot about like your LLMS file and this other, I don’t know if she talked about benchmarks and so on, but it seems like she was getting into the tool side and focusing on the user experience and how we plug into shaping that whole output. And that is certainly like,

Fabrizio: very good, very good post, yeah.

Tom: tough to figure out how do you shape the accuracy of even third party AI tools? That would be a home run if you could figure that out. If you could, yeah, anyway, but there’s probably plenty, plenty to focus on there if we only had endless amounts of time to figure it out. But yeah, I definitely want to, I want to get that benchmark thing going.

Fabrizio: Yeah. Yeah. Right.

Tom: I’m thinking about like a different way of approaching it. And I don’t know if this will actually work, but I want to just take all my docs are behind like. ACLs and so I’m really in a different category than most of the documentation at my work, but I want to feed docs into a notebook LM as a rag and then use that as a quick way to iterate and see if it’s going to find the answers. But the problem is I don’t know if like notebook LM would match how other AI tools ingest and deliver content, how they index it. So it’s a lot of unknowns, but.

Fabrizio: Yeah, yeah, but I think it’s something worth exploring because there are tools out there like, you know, the so-called AI detectors like GPT-0, et cetera. And I understand where they come from. I understand their goal, which I think is a, you know, is a lost battle from the beginning, but they’re trying, they’re trying. But I think it’s just silly to focus on patterns like you’re using too many dashes or I had a text I wrote when I pasted these tools and it told me I am an AI, which might as well be true, nobody will know. But the thing is there’s so many false positives also with these tools. And I don’t think is the best approach right now, that one. Plus, we don’t really want to demonize CI usage.

Tom: Yeah, yeah. I haven’t really heard of the AI detector that you’re talking about, but I’m actually trying to connect that AI detection to just this larger goal of benchmarking. How do those two connect? This idea.

Fabrizio: Well, we would probably have to reverse the intent, you know, like method might be similar, but we might want to measure or focus on other things. But yeah, yeah.

Tom: Ah, I see, okay. Yeah, well, all right, so we’ve talked about a lot of different things. We actually had a whole bunch of articles, and we’ve touched upon a lot of them. I’ll paste in a lot of these articles into our show notes, but is there anything we haven’t really gotten to that you think is worth discussing or highlighting here?

Fabrizio: Yeah. Well, there’s these… I couldn’t read it entirely, there’s the AI hype ready thread. What are your thoughts about this one? Because I still continue to lurk on Reddit from time to time to see the doom and gloom thing, but yeah, what are your thoughts there?

Tom: So somebody pasted this just like really well written essay about AI hype, strange underscore show 9015, whoever that is. And I thought it was really well written. One of the main points that the person explained is that.

Fabrizio: Yeah.

Tom: AI can’t do the holistic part of a job. Can have AI do a specific task that you explain it to do, tell it to do, but it’s not going to read your email and communicate with your project leads and check your calendar for timelines and you know, like.

Fabrizio: Yeah. Yeah. Right.

Tom: I don’t know, do all the parts of a job that would be required to actually replace a human. And that’s what a lot of people don’t do. And the writer argues that these companies that have tried to replace writers with AI are not succeeding. They’re, quiet rehiring and, Yeah. So the, the author was trying to downplay the AI hype and to make tech writers feel a little bit more confident that their role will exist in, five years or something. Nobody can project very far into the future anymore, but, this is definitely a ton of anxiety that we have in this field about whether we’re going to be here in five years or not.

Fabrizio: Right. Right. Yeah. Yeah.

Tom: and the writer was trying to just reassure us.

Fabrizio: Yeah, yeah, it’s as you say, it’s it’s hard to predict. I’m cautiously optimistic about it. Of course, you never know like about the big motions of, you know, market or whatever. But I think if we, like Sarah Moore’s blog post, if we continue being more strategic about how we orchestrate the tools in our work, you know, to get something meaningful out and really seek the impact. So it’s interesting because this whole change is forcing us to be, to change our mindset a bit. Like the technical writer working in isolation in a cubicle. It’s a fantasy. It’s not going to happen anymore. The tech writer now has to be more goal-oriented, impact-oriented, and it has to leverage AI in that sense. And at the same time, I think, and this is something that developers do very well, And I think tech writers sometimes struggle with is that we have to get used to be a little more imperative in the sense of we have LLMs and there are Armenians and we can issue commands to LLMs. We don’t have to feel maybe we’re not used to because sometimes we are long riders. We never manage people, but there are assets, there are resources and we shouldn’t feel like uncomfortable with, you know, asking an LLM, you know, in a very imperative way to do something. And I still feel that sometimes because of our background, because of whatever reason, writers struggle with issuing commands, you know.

Tom: Yeah, I think we should try to push the limits more of what we can do with AI tools as well. I think we underestimate. So many people just start out with such a low bar. They’re like, let me just fix the style of this one paragraph. It’s like,

Fabrizio: Yeah. Yeah, yeah.

Tom: You got to think bigger. If you want to have impact, I don’t know who knows what these tools are even capable of where that, that, that top threshold is, but I wouldn’t have definitely explore it. And I want to look for ways to have more visible impact. I don’t want to be that invisible writer in the corner that nobody even knows what they do. And, just the person who’s quietly working on who knows what, like I want to, I want to hit the home runs and, and

Fabrizio: Right. Yeah,

Tom: Really, really?

Fabrizio: unfortunately we can no longer afford to be silent. We never did really, but now it’s more pressing than ever.

Tom: Yeah. Yeah. Well, thanks Fabrizio for this chat and this conversation. I always feel like I want to learn from you and from others about how you’re working with AI. What are your strategies? What are your main concerns? And how can I incorporate that kind of learning? So I really appreciate your input on everything.

Fabrizio: Sure. Mm-hmm. Sure. Yeah, let’s see if we can get somebody in the next episode to tell us how they use the AI at work as well and get more perspective.

Tom: Yeah. Yeah. Yes, we were talking before the show about a potential guest, so we will see if that turns out. All right. For all the listeners, you can get the show notes on our sites. I’dratherberiting.com, Passo Uno as well from Fabrizio. And write us back with feedback if you have it for topics you’re interested in or questions, and we’ll maybe tackle them in an upcoming episode.

Fabrizio: Yeah. For sure. Thank you, Tom.

Tom: Thanks for listening.

---

Note that these shownotes are AI-assisted.