Oh, let’s have some fun. “The most important tool(s) of my professional life is/are: (a) my HATT tool; (b) my drawer full of red pens; (c) my caffeine container of choice; (d) my Web browser; (e) my STC or UPA membership card.”

monkeyPis last blog post..BREAKING NEWS!!! (i think)

“There’s enough research out there that proves that you don’t need a grasp of the total, just an understanding of where the point of diminishing returns are.”

Overall more than 2100 people participated in all 3 surveys. I noticed that with my survey, even as more people participated in the survey, it didn’t seem to alter the percentages much anymore. I guess I’m reaching the point of diminishing returns.

Overall, I should probably do more research in creating surveys. There’s a lot more to it than I originally thought.

Any thoughts on topics for the next tech comm survey?

It’s not entirely accurate to say that “to find a representative sampling, you would need to somehow determine the whole, so you could tell if your sample represented it.”

Most statisticians would argue that this is not the case… I don’t speak as a statistician myself, but working in the scientific R&D field, what you describe is a common problem that my team encounters, and we solve it by having one of our resident statisticians on the team provide a power analysis. Takes your typical PhD stats guy a few hours to work out. There’s enough research out there that proves that you don’t need a grasp of the total, just an understanding of where the point of diminishing returns are.

Furthermore, these types of questions do not require a high fidelity of quantification, anyway. It’s like using a sledgehammer to drive in a finishing nail. It’s overkill, and unnecessary. If you’re Adobe, you won’t care if Madcap’s share is 25% or 27.8%, all you care is that those folks aren’t using Robohelp. It’s easy to get caught in a forest-vs-trees discussion with stats.

Anyway, again, me being a bonehead and speaking only from experience, the sample sizes in the surveys mentioned were more than appropriate, (assuming the demographics involved were representative of the demographics of the whole – which seems on the surface a reasonable assumption).

I agree that the challenge is not in the sample size, but in the questions asked. IMO, none of the surveys mentioned were worded very well. E.g., as you mentioned, “which tool is best for you” is inadequate for determining popularity or market share, as it mixes too many things: what the respondent cares for, what they have to use, what their client’s needs are, what the user wants/expects, etc. Ambiguity is the enemy of defensible quantification.

A healthy, well written survey uses a combination of yes/no, Likert scale, multiple choice, and open ended questions…. AND… is facilitated by a trained “debriefer” who is the traffic cop for the questions (preventing respondents from deviating from the question or misunderstanding it). The latter is a nice-to-have, but the others are definitely achievable (with enough time to research, develop, and test them out).

I guess I’m saying that it could definitely be done. You’d have to invest time and $$, but it could definitely be done well and accurate, whether or not you knew the total # of tech-commers in the world.

I do love these surveys, though; if for no other reason than they give us something to talk about.