Documentation failures, Bestiaries, AI Explain post mortem, Inside TechComm podcast (Oct 3, 2023)
Articles listed here include the following:
- Thoughts on the manual, by Seth Godin
- Reflections on AI Explain: A postmortem, by the MDN team
- WTD Atlantic Conference 2023 highlights
- Step into the future of technical writing with AI insights from Caity Cronkhite — podcast by Zohra Mutabanna
- Bestiary: On Doc Types and Other Animals, by Fabrizio Ferri Benedetti
Thoughts on the manual, by Seth Godin
Seth’s Blog • September 28, 2023 • LINK
- The author provides a host of feedback about the failure of user guides to be helpful. (The feedback is an unorganized list.)
- He points out challenges with docs, such as lack of empathy, inability to know unknown unknowns, endless warnings required by lawyers, non-linear formats, the lack of video and tediousness of video, wordless docs, lack of context, failed feedback loops for iterative improvements, and more.
What makes this post noteworthy is that it’s coming from Seth Godin, a well-known thought leader who has authored many books and is recognized for his leadership in marketing and entrepreneurship.
Putting aside the author and why he’s writing about documentation, what strikes me about his post is that user documentation has been broken for a long time, with no easy fixes. Creating perfect learning experiences is nearly impossible under the circumstances (limited time, limited knowledge, limited input).
This is why I’m excited by the possibility of AI chatbots: they can transform the experience of documentation. Learning can be self-directed, downleveled or upleveled depending on your information needs, dynamically arranged in different orders, expanded at any point, clarified with countless examples, and more. Seth doesn’t mention AI as a potential fix to address the learning challenges with help, but it does seem like the problems he describes are insurmountable without a major shift in a tech writer’s approach.
Reflections on AI Explain: A postmortem, by the MDN team
Mozilla blog • July 10, 2023 • LINK
- MDN recently launched two AI-related features: AI Help and AI Explain. The purpose of AI Explain was to help users understand code examples, but they disabled AI Explain after 2 days due to community feedback that it was inaccurate and misleading.
- Users were upset because incorrect technical explanations misled users and also undermined MDN’s reputation as an authoritative documentation source.
- This postmortem explains some of their motivations for AI Explain. They wanted to make content more discoverable for users who didn’t know the right terms to search, for example. The code examples also combined features that were explained across different pages, making explanations on any single page problematic. AI Explain was intended to help in these instances, and also to provide code coverage where it was lacking.
Note that the AI Explain feature not only provided explanations for existing code samples, but also dynamically generated code. The article says the team saw use cases for “generating examples for pages lacking them, and generating unique code examples that combined different technologies according to user queries.” (This is a much bolder attempt than a technical writer generating code examples using AI tools and pasting it into docs.)
In their lessons learned, they said there were two problems: lack of context and evolving LLM models.
- “Prompting GPT-3.5 to explain code samples without context is not sufficient given its 2021 limitation. We should have used a similar approach for AI Explain as we did with AI Help, using relevant MDN content as context.”
- “Going forward, we need to thoroughly validate any differences between models, as these variations can significantly affect output.”
The AI Help feature is still available (to logged-in users) and is impressive even with the 5 query limit per session. I didn’t get a chance to try AI Explain before it was disabled, but dynamically generating code seems like a bold and risky attempt.
Dynamically generated code samples do seem like the holy grail of developer documentation. Had they better labeled it as experimental and prone to error, perhaps users would have been more tolerant. I can’t help but think novice users would prefer a mostly right code example to none at all, though perhaps the errors created more problems than they solved.
Their experiment also raises another point: users hold the AI chats on individual sites to a higher standard than the general AI chats like ChatGPT, Claude, or Bard that provide one-stop shopping for all information.
WTD Atlantic Conference 2023 highlights
Write the Docs • LINK
- The Write the Docs Atlantic conference was a success with almost 300 global attendees. It featured main talks, Q&A, lightning talks, unconferences, sprints, and a sponsor expo. Video recordings of all talks are available on YouTube.
I find it amazing that WTD has always posted videos of their conference sessions, without charge, yet still has hundreds of conference attendees. In contrast, other technical writing conferences continue to either prevent free recording and sharing of sessions, or they charge for their recordings.
People don’t come to conferences simply to learn during sessions but for the community and hallway conversations and other interactions. Were other conferences to make their recordings freely available, it might not lessen attendance but rather increase the visibility and conference community.
Step into the future of technical writing with AI insights from Caity Cronkhite — podcast by Zohra Mutabanna
Inside Tech Comm with Zohra Mutabanna • September 21, 2023 • LINK
In this season of Zohra Mutabanna’s Inside Tech Comm podcast, Zohra explores AI topics through a variety of guest interviews. For this episode, she talks with Caity Cronkhite, founder of Good Words, about using AI tools for technical writing, including the risks and how to mitigate them. She explores whether AI could replace technical writers, the skills that will remain important, and how writers can prepare for the changes ahead.
I recently participated in a “Blogs, vlogs, and podcasts” panel as part of STC Spectrum 2023 with Zohra and Ed Marsh. The conversation inspired me to rethink the value of podcasts and maybe rekindle my fizzled podcast. Zohra was just as pleasant, insightful, and articulate in person as she is on her podcast.
Bestiary: On Doc Types and Other Animals, by Fabrizio Ferri Benedetti
Passo Uno • September 23, 2023 • LINK
In this post, for reasons unexplained, Fabrizio describes common types of documentation (release notes, tutorials, troubleshooting, etc.) in the style of a medieval bestiary patterned after the Physiologus, which dates back to 140 BC in Alexandria. A bestiary is a book describing a compendium of beasts, both real and imagined. Each entry in a bestiary typically describes a creature, often accompanied by an illustration, and might provide a moral lesson drawn from the animal’s supposed behavior. Fabrizio includes illustrations created through Dalle of the beasts in his documentation bestiary.
Fascinatingly creative, this post prompts the question, why do this? Is Fabrizio experimenting with a convergence of two unlikely forms merged together? Did he generate the text with AI? For fun, I tried to reverse engineer the text, translating the bestiary to plain English using AI, then prompting AI to translate the plain English version into bestiary style like the Physiologus, but the results weren’t great. They were much less readable and interesting, to the point of being trashcan-worthy.
Perhaps Fabrizio found himself deep in the Physiologus and thought, what if I could write a similar bestiary for the modern age, with beasts the technical writer wrestles with? Or maybe he was playing around with stylistic filters on generative AI tools. Regardless of how it was created, this post demonstrates the ability to look at commonplace objects with new perspectives. To see our modern world through the eyes of another lens such as a medieval bestiary is surely not only fun but also curious and rewarding. This shift in perspectives is one of the primary goals of artists.
One outcome from his piece might be to get us to see our inanimate documentation deliverables with a newfound zoomorphism, which might change our relationship about commonplace forms to instead perceive them as magical, shapeshifting, and chimerical. Or it might just be a party trick. At any rate, I like how he transposes genres and ideas in creative ways that help us change perspectives.
By the way, I wrote up some notes about a similar technique from a workshop at Amazon by Carla Johnson years ago in a post titled Brainstorming by transposing patterns from one category to another. In the workshop, Carla had us pose questions about how to improve the experience in one setting (e.g., getting coffee in a coffee shop to another category) and then transpose those same questions to another scenario (such as improving our software products). The results did not disappoint.
About Tom Johnson
I'm an API technical writer based in the Seattle area. On this blog, I write about topics related to technical writing and communication — such as software documentation, API documentation, AI, information architecture, content strategy, writing processes, plain language, tech comm careers, and more. Check out my API documentation course if you're looking for more info about documenting APIs. Or see my posts on AI and AI course section for more on the latest in AI and tech comm.
If you're a technical writer and want to keep on top of the latest trends in the tech comm, be sure to subscribe to email updates below. You can also learn more about me or contact me. Finally, note that the opinions I express on my blog are my own points of view, not that of my employer.