'Putting together things': Articulating a thesis about the effects of hyper-specialization on documentation

by Tom Johnson on Sep 22, 2022
categories: technical-writing

• The conclusion I’m leaning towards
• An intriguing quote debunking rationalism
• Avoiding irrelevance
• Opportunities forward
• Why horizontal writing is so hard
• Conclusion
• References
• Next post

The conclusion I’m leaning towards

In this series, I’ve more or less arrived at the following conclusion: As technology becomes more specialized, the systems view becomes lost. Almost every knowledge worker understands only their specialized sliver of the whole. The ability to document the big picture becomes its own specialized skill since no one has such horizontal understanding to fully comprehend all the pieces of the product.

When the horizontal view isn’t available, the developer portal becomes full of specialized documentation that resembles someone’s personal technical notes, explaining how to implement this or that service but with little context about why, or what dependencies it has, or how the system works as a whole. The larger developer journey becomes lost. Documentation becomes something that’s only intelligible to the specific partner it was created for and the team that made it.

An intriguing quote debunking rationalism

Reading Thinking in Systems by Donella Meadows yesterday, I came across a quote that articulates what I’m trying to say even better. Meadows starts out many chapters with quotes, one of them from John Ralston Saul, who writes:

Rational elites … know everything there is to know about their self-contained technical or scientific worlds, but lack a broader perspective. They range from Marxist cadres to Jesuits, from Harvard MBAs to army staff officers…. They have a common underlying concern: how to get their particular system to function Meanwhile … civilization becomes increasingly directionless and incomprehensible.

– John Ralston Saul, political scientist (qtd. in Meadows, 111)

This quote intrigued me, so I looked for more context from Saul’s writing beyond the source above, which is a paraphrase from a 1992 interview in the International Herald Tribune with Barry James. The interview was prompted by Saul’s book, Voltaire’s Bastards: The Dictatorship of Reason, published in 1993.

I haven’t read the original book, but apparently, it’s a 600-page diatribe against the over-elevation of reason, which began with Enlightenment thinkers and evolved toward jargon-filled technocrats. Saul argues that reason has unfairly taken priority over so many other human qualities, such as intuition, memory, common sense, creativity, and ethics. One casual reviewer, Sherwood Smith, explains: “His central point, that the ‘reason’ of the Enlightenment age, and to which we modern westerners pay lip service, has run amok, that our world is run by soulless technocrats” (Goodreads). Reason, when heralded above other human qualities, moves us into isolated communities of expertise that fragment and destabilize a more holistic understanding of the world.

In an interview with Scott London, Saul elaborates more on the deteriorating effects of over-elevating reason and expertise. Here’s an excerpt from their exchange:

London: Our modern fixation with reason seems to be exemplified in the cult of expertise.

Saul: What people like Voltaire and his friends thought was that if people became experts in areas, you would gradually uncover all that was unknown. In a sense this expert knowledge would become part of a great field. We would all cooperate with our expertise. We would turn to each other and say, “Well, you know about sewers, tell us all about sewers,” or “You know all about nuclear bombs, tell us all about nuclear bombs.” In that sense, we would all be a part not of a fractured society, but of a single society that had many, many elements in it, and we would be much the richer for it.

Well, we’ve ended up with what they wanted, which is, we have our thousands and thousands of areas of expertise and we do know more than we have ever known and that is fabulous. But the result has been the exact opposite of what we expected because it’s led to a fracturing of society. In other words, each bit of knowledge, each area of knowledge — the sewer experts — have gone away into their corporation of sewer experts and held that knowledge within.

In other words, specialized knowledge fragments us into smaller communities. What’s fascinating is that Saul published Voltaire’s Bastards in 1993, so his assertions here mostly predate the Internet. The idea about community fragmentation echoes a point I made earlier in Technical diversity/pluralism/fragmentation in tech comm. In that post, I summarized John Davis’ research about how specialization in the field of economics fragments the discipline into many independent and siloed niches, which “runs the risk of less disciplinary coherence.” With less disciplinary coherence, is there even a common conversation about the big questions in economics to bind the community together? Or has it split into a dozen smaller communities, each now independent and distanced from these discipline-cohering interests? (See “Specialization, fragmentation, and pluralism in economics” for more details.)

Let’s continue with the London and Saul interview. Saul says:

One of the effects of the rational revolution has been that knowledge has become the currency of our society — much more important than money, if one were to return to the idea of the marketplace. The control of knowledge, in the full sense of that term, is the real currency of our society. It’s power. Of course, because it’s power, because it has a major effect on your career and what you are going to be able to do in your life, you hold it back. It’s something you negotiate with.

No doubt you’ve experienced this in any job interview related to a technology role. Interviewers want to know what programming languages, technical frameworks, tools, or other specialized knowledge you possess. Writing skills tend to take a backseat, more or less put in the same category as soft skills like project management, problem-solving, leadership, people skills, and more. The conversation becomes all about your technical prowess — and trying to probe the depths/limits of that technical knowledge. Can you solve this algorithm? Can you read this code? How technical are you?

Never mind that once you assume the role, your technical acumen becomes far less important than larger analytical skills related to being a technical writer, such as learning how systems work (which might involve a lot of reading), information synthesis and distillation (across thousands of pages of documentation on a developer portal), user-focused alignment (understanding how to find the user’s voice and perspective), project organization (in the midst of supporting multiple engineering groups, each with their own roadmaps and methodologies), and more. This doesn’t so much matter, but if you understand asynchronous callbacks, well, then, the job is yours!

Saul continues:

So instead of knowledge coming together into a great whole, knowledge has been broken up into tens of thousands of isolated corporations or specialist groups. It’s meant two things. It’s meant, first of all, that society loses all sense of direction, because if everything is separated into little groups that don’t really talk to each other in an honest manner, except to negotiate between each another for power, then there is no possibility to have any kind of directed conversation about society. The second thing is, of course, that it has been very, very bad for each of those areas. The fact of the matter is that sewers run next to autoroutes and hearts lie next to lungs.

In technology groups, engineers are quick to draw boundaries around what they own. They sort of have to — the degree of knowledge needed to operate in their existing role is immense and overwhelming. Any way they can limit that endless amount of information, even if product related, can help reduce the suffocating avalanche of information.

But what happens as a result? With developer portals, when the documentation is produced by so many specialized groups, the developer journey as a whole becomes murky. There’s no larger thread to follow. Documentation lives in separate little spaces, walled off from one project to the next by a mere heading level in a sidebar navigation or tab. Groups keep pushing documentation onto the portal without fully understanding what the other groups are publishing.

In my role at work, there are two different groups publishing updates in the same SDK, with little coordination or understanding of what each group is doing. One team works on one API in the SDK, another works on another API in the SDK, and sometimes I understand more about each team than they do.

As an analogy, imagine a large room full of people, with everyone separated into small groups, talking amongst themselves, with no interaction across the groups. With these dozens of separate conversations, the room is a loud babble of noise. In walks a customer who wants an end result that will require all groups in the room to work together. How are each of these groups, which barely understand what each group does, going to come together in some harmonious orchestration to deliver a result?

Saul continues:

…When I talk about expertise, what I’m talking about is the way in which we approach education. We’re clearly not following the humanist approach, which is a sort of integral view of human intelligence — putting together things. Our education system is a) based on the taking apart of things and the isolating of smaller and smaller elements of knowledge ….

“Putting together things” is the central thesis I’m advocating in this series. This should be a primary focus as technical writers who want to still function as authors. That is, if you want to write, you should focus on the larger view of the product and system. The details will be owned by the engineers on the hyper-specialized product teams, but few can articulate the big picture.

Avoiding irrelevance

In fact, if technical writers try to author the nitty-gritty technical details (the parameters of an API, the code samples for how to initialize the API, generating the Javadoc, and such), I think tech writers will struggle to remain relevant. Technical writers will instead shift into roles as editors, curators, and publishers — but not authors. Technical “writers” will no longer be writing but rather managing engineers who write. This is because the technical information is so specialized, it requires more immersion than a generalist writer can learn in the allotted time.

I have to admit, I think editor/curator/publisher roles are secondary in skill to original content development. The hard part about writing is creating something from nothing, filling a blank page. Fixing commas in poorly written content will also soon be a skill no one needs. AI tools are getting better and better. You’ve already seen how AI tools can automatically generate images (Midjourney, DALL-E, Imagen, and more). Story-writing AI tools like WordCraft are also impressive in the ability to generate coherent, readable prose. Using other tools like Copymatic, you can automatically generate lightweight, on-message copy from just a few subheadings. These tools seem to have appeared overnight, and there’s a long way to go before services like Grammarly can do the same. But how long before engineers can take something written poorly, hit a magic “language transform” button, and suddenly get rid of all that awkwardness and broken grammar? Maybe not too long.

Of course, editing is about a lot more than making docs pretty, or fixing grammar and spelling. It’s about clarity, organization, messaging, and flow. But the emergence of more sophisticated AI language tools will lessen the importance of editors.

In Deep Work, Cal Newport (drawing upon research in Race Against the Machine) says,

As intelligent machines improve, and the gap between machine and human abilities shrinks, employers are becoming increasingly likely to hire “new machines” instead of “new people.” And when only a human will do, improvements in communications and collaboration technology are making remote work easier than ever before, motivating companies to outsource key roles to stars — leaving the local talent pool underemployed. (23)

In other words, as machines become more sophisticated, they will begin to replace workers whose skills can be automated (like identifying broken grammar and fixing it). Newport continues: “… an increasing number of people will lose in this new economy as their skill becomes automatable or easily outsourced” (23). The only exception to being automated or outsourced, he says, are those who are highly skilled, superstars (uber productive workers), and owners with capital. Newport argues that those who can focus on deep work are those who can elevate into the highly skilled, superstar role.

The simplification of publishing tools will also reduce the relevance of technical writers acting as publishers. Most developer groups aren’t writing in XML or managing content in a robust component content management system (CCMS). They’re writing in Markdown, following the same publishing tools and techniques for documentation as engineers do for code. Almost every big tech company with APIs manages its documentation in a docs-as-code framework.

For example, I send out changelists for product teams to comment on, in a line-by-line diff that shows only what has changed, using the same workflow that engineers use for code changes. Product teams don’t need technical writers to operate the publishing tools and pipelines anymore. (Those responsibilities are owned by a specialized tool group, who are more engineers than writers.)

And as for curation, a good search engine takes care of much of that. Readers rarely navigate content through the sidebar anymore. Search is the entry point, and then inline links. You can spend weeks figuring out the right hierarchies in the table of contents only to find that users aren’t using it at all.

Without editing, publishing, or curation tasks, the technical writer’s role will diminish in importance. Where does this leave technical writers if they aren’t writing either? It doesn’t leave them on a critically important path, in my opinion. As such, technical writers might move gradually into more and more irrelevance when it comes to developer documentation.

Opportunities forward

What can tech writers do to avoid this gradual slide into irrelevance? One area technical writers can exploit is the big-picture view, which individual engineering teams often fail to understand. Writing more horizontal than vertical, technical writers can develop overviews of systems that describe all constituent parts, how they interrelate, and other big-picture thinking and lifecycles in this system. They can specialize in describing the developer’s journey.

Systems view writing encompasses the components that make up a system and traces the lifecycle of common journeys through the system. For example, a system overview might be a “Life of a ____” narrative, with a typical example being the life of a web query. These narratives trace the journey of an action through an entire system.

I can guarantee that almost no engineer will be writing this kind of overarching information, nor articulating the product and system in a broad overview that contextualizes information across the corporation and industry domain. At least, they won’t be creating written content, just slides or diagrams. Engineers typically write only about their sliver of expertise.

Why horizontal writing is so hard

Most people agree about the value of articulating the big picture. However, this type of horizontal writing (or big picture views, systems views, holistic writing, etc.) is hard and not frequently done. There are several reasons for this:

• Technical writers depend on engineering teams for information. If engineering teams themselves are specialized and lack the big picture, technical writers will struggle to locate the needed information. Tech writers will have to piece together the story through interviews across multiple teams, gathering pieces of a puzzle that they then assemble together themselves.

• The systems view becomes most relevant during onboarding, when engineers try to get a sense of the whole before moving into their specialized role. After a number of years, the engineer learns that he or she doesn’t need to know anything beyond a limited set of topics to perform their role. Unless tech writers interact with onboarding groups, they may not see the need for larger information.

• Organizational boundaries reinforce the specialized knowledge within a technology company. Division Acme is responsible for component 1, Division Beta is responsible for component 2, and so on. The product consists of components 1-10, but the organizational divisions reinforce the idea that each division need only concern itself with what the divisions “owns.” As long as it delivers its output, the product’s overcall outcome doesn’t matter.

• In a world where people are drowning in information, it’s natural to carve out boundaries around what you need to know. Engineers quickly decide what they’re responsible for and the information necessary for their roles. Many other parts of the system become irrelevant because they exceed what the engineer can feasibly understand and master. If tech writers support these more hyper-focused engineering teams, they won’t naturally move beyond the engineering team’s scope.

• Few tickets that come into a system ask for the larger picture. For example, a user rarely files a ticket explaining that he or she doesn’t understand the purpose of the product or what it does. The tickets usually focus on some small technical detail, such as missing parameter formats or a bug related to the data returned by an API. As such, no one drives technical writers to create this larger view.

Conclusion

I’m still ramping up on my approach and refining my thoughts on what it means to tackle these larger system-level topics. Here are a few questions that such writing might address:

• What are all the components involved in the system, and how do they relate?
• What’s the general workflow for some action through the entire system? what path does a “life of a _____” take?
• What are the points of interface across teams, where one team’s services are rendered by another team, such as the API they make available?
• What dependencies exist for this service/framework/component?
• How does information flow across the system? What are the inflows, outflows, and what controls the rate of inflow or outflow?
• How does this product relate to other company products in the same category?
• How does this product relate to products at competitor companies?

These questions aren’t the ones we’re accustomed to documenting. As such, the whole direction of horizontal writing goes against the natural flows of typical requests and documentation projects. Through more experimentation, I plan to test whether these larger narratives have the merit that I suspect they might.

References

London, Scott. “The End of Rationalism: An Interview with John Ralston Saul.” Ottawa Citizen, December 2001.

Meadows, Donella H. Thinking in systems: A primer. Chelsea Green Publishing, 2008.

Newport, Cal. Deep work: Rules for focused success in a distracted world. Hachette UK, 2016.

Next post

Continue to the next post in this series: Attempting to write a Life of a [something] narrative.

About Tom Johnson

I'm an API technical writer based in the Seattle area. On this blog, I write about topics related to technical writing and communication — such as software documentation, API documentation, AI, information architecture, content strategy, writing processes, plain language, tech comm careers, and more. Check out my API documentation course if you're looking for more info about documenting APIs. Or see my posts on AI and AI course section for more on the latest in AI and tech comm.

If you're a technical writer and want to keep on top of the latest trends in the tech comm, be sure to subscribe to email updates below. You can also learn more about me or contact me. Finally, note that the opinions I express on my blog are my own points of view, not that of my employer.