1. I'd Rather Be Writing
2. Docs

Quantifying your progress

Download PDF

This section continues from the previous page, Quality checklist for API documentation.

• Analysis and quantitative metrics
• Moving towards quantification

Analysis and quantitative metrics

It’s hard to imagine that documentation that checks all of the boxes in the quality checklist wouldn’t also score highly with user satisfaction surveys. Can you honestly see any documentation that legitimately satisfies all of these criteria as falling short with users?

And yet, to achieve the level of information quality, we didn’t have to rely on constant user surveys to gather feedback. By identifying best practices for content design (specifically for API/developer documentation), we’re able to increase the documentation quality in more self-sufficient, self-directed ways.

Moving towards quantification

In my initial go-around with the quality checklist, I tried to move towards quantification by including scores and weights for each criteria, and then dividing the achieved number of points by the total points. From this scoring, I tried to move from qualitative to quantitative measurement.

However, in practice, I found that assigning scores for each section felt arbitrary and subject to personal whims. I don’t think others found the scores meaningful either. Instead, just having a checklist of criteria to consider was valuable enough. That’s why i stripped out the section on quantification in later revisions to the content — because I want the advice I give here to be helpful in practice, not just theory.

I’m not saying that some approach to quantifying documentation wouldn’t work, just that my approach did not. Also, recognize that the quality checklist has no official data to support it — instead, these best practices come from experience in the industry and from best practices that I and others have observed within the realm of developer documentation.

This is likely the problem with my approach: who’s to say that documentation needs each of these criteria to succeed? It’s possible that documentation might still be findable, accurate, relevant, and clear without many of these more concrete components. I don’t have any user-based research to say that docs should be this way, that they should have an overview, that reference material should follow a consistent structure, that tasks should be detailed in steps, or that error messages should be documented, etc. And honestly, I doubt any checklist can prove its objective value.

Remember that user surveys, which I criticized earlier as problematic, should both complement and confirm the criteria in the checklist. User surveys specifically for docs that rate highly with the quality checklist should also rate higher in satisfaction surveys than surveys for docs that score more poorly. But again, to establish a kind of correlation through surveys relies on a host of factors (objective, unbiased, unambiguous survey questions from a large sample of a representative audience across domains), which is likely difficult to pull off on a regular basis.

Overall, I am confident that few would object to most of the criteria in the checklist. Most of the checklist’s criteria would be agreed upon by both readers and writers with enough common ground as to be a practical guide for improving documentation quality. Also, the criteria should be seen as a first draft, a starting point that can be refined and improved, checked against industry standards, confirmed against docs that are loved by users, refined through constant feedback, and more.