I'd Rather Be Writing » aggregation

The Importance of Chunking for Sorting

Tom Johnson — Mon, 18 Apr 2011 14:39:25 +0000

If you want to be able to sort information by various classification schemes, such as by most popular, or by role, or by problem, your content has to be chunked in a granular enough way to facilitate the various means of sorting.

Consider a work that is one large book, with no chunks at all. In that case, it would be impossible to sort anything, because you have just one object. With one object, the only pattern you can configure is itself. But if you have a handful of objects, you can arrange those objects into as many patterns as you want.

To use an analogy, let’s say you have a pile of rocks. If you have 1,000 small rocks, the potential number of patterns you can configure with the rocks is infinitely greater than the patterns you can configure with just a few rocks.

I noticed this in a recent trip to Arches in Moab. While walking along trails, we saw a lot of rock piles called cairns that act as guide points. The cairns can be stacked and arranged in myraid ways, because they consist of little rocks:

But the big rocks are much more pattern-limited. They mostly just sit there, alone:

Thus if your goal is to enable a variety of patterns or classification schemes, so your users can choose from myriad classifications, according to their individual needs, you must chunk your content in a granular enough way to facilitate the classifications.

Granular chunking poses some difficulties for help content, because if you chunk things too small, the help system becomes arduous to navigate. If each page contains just one topic, you end up with so many pages, navigating the pages will give users a headache.

To avoid this, on my calendar help wiki, the Viewing Calendars page has the following topics on the same page:

All of these topics appear on the Viewing Calendars page.

Now, suppose I want to manipulate this content on a more granular level. Suppose the “View Calendars of Other Wards” topic is a popular topic; the “FAQ” issues would be appropriate in a problems-based classification. The “About Subscribed Calendars and Subscribed Locations” belongs to a conceptual table of contents. The “View Churchwide Calendars” belongs to a “Coming Features” type of organization, and so on.

In short, let’s say I want to add metadata to each of these sub-topics so that they can be sorted, rearranged, recompiled, or otherwise organized in different classification schemes. If they are compiled in one giant topic, they can’t be manipulated at all except on a more macro-level. This is why chunking is such a fundamental principle to technical writing, because without small chunks of content, you don’t have many options for manipulating it.

Whether you use a wiki or not, deciding how granular to chunk your content is a challenge. For example, on Microsoft Word’s Help, this is the topic for Changing or Setting Page Margins.

This topic on working with margins really contains five separate topics.

By combining these five topics into one topic, it becomes more difficult to manipulate the individual sub-topics as their own topics. The metadata you add to this topic must account for all the sub-topics within this topic.

Now consider the opposite strategy. Let’s make each subtopic its own topic. You can see the effects of this approach in the following Office help search:

When you chunk things in a granular way, it becomes harder to find the chunks, and you lose some context.

Here the topics on formatting are all chunked into their own topics, so you end up with Clear all text formatting, Show or hide formatting marks, Apply strikethrough formatting, and so on. When a user clicks on a topic, the topic is short, such as the following:

This is a short topic. This is all that's there.

This short topic either answers the user’s question or it doesn’t. There’s not much room for error, since no similar topics are grouped together. If it’s not the right topic, the user must return to the list of results and click another, and another, and another until he or she locate the right topic.

In contrast, if you combine a larger number of topics together on the same page, you give more context to the user. He or she can read conceptual introductions followed by a handful of sub-topics that all deal with the general topic. The user can easily scan down the subheadings to find the right sort of task for this topic. But it’s harder to manipulate each individual sub-topic separate from the larger topic. And your metadata can’t describe each of the individual sub-topics but must cover the larger topic generally.

Chunks that Consist of Chunks

I’ve been contrasting big chunks versus little chunks without acknowledging that big chunks can consist of combinations of little chunks. So in each of the examples above, the topics can exist separately but be grouped together into the larger topics that you see.

With Mediawiki, this method of reuse is called transclusion. Last week, convinced that I needed to chunk each topic more, I separated all the topics that you see in that first calendar screenshot onto their own individual pages. I then “transcluded” these chunks to form a longer page.

Currently, from the user’s point of view, it looks exactly the same. But really, I can now arrange and manipulate these chunks however I want because I can apply unique metadata to each one of the topics.

However, this poses a new problem: searches will find the individual chunks and the larger pages that combine these chunks, which means content will be in multiple places rather than one place.

The Collage and the Painting

Don Day’s post on The Collage and the Painting describes how search becomes problematic with little chunks. Day is writing in the context of DITA, but the challenge of working with small chunks is the same. Day writes,

A common talking point about DITA is how the topic-referencing architecture makes it easy to reuse topics in new maps of information. By extension, searching on a facet of interest should bring up a collection of topics that you can read as a focused subset of a larger whole. Print it as a PDF, or output it in eBook format, and you’ve got some good reading for the commute or for the weekend. But how practical is this vision?

The flaw in the theory comes from loss of context when you pull a set of topics by query. Imagine doing a web search on a subject of interest and then printing the whole list of hits, as is, into a single PDF for later reading. Obviously you will have the problem of duplicated content, possibly some older and less reliable content, a good deal of discussion by people who are not experts on the subject, organizing the hits in a reasonable manner (by timeline, by author, in a hierarchy) and so forth. Metadata might help in preserving bits of a former organization or rationale, but the new use might be totally different from how any of that content was originated. Bringing order out of disarray is the whole drive behind the growing trend of Content Curation.

Don Day uses the metaphor of the collage and painting to distinguish between small topics pulled together without order and a larger chapter that provides context and sequence for each of the topics.

In other words, if you pull together all topics that have specific metadata, such as all topics related to scheduling events, you may get an unordered collage of topics. The order of the topics may not reflect any kind of sequenced or arranged reading. The list of topics no longer forms a larger, well-written chapter that contextualizes each topic, but rather may seem like little scattered objects here and there.

The effect might be compared to taking an entire book and ripping out all the pages and throwing them on the ground, mixing them up, and then reading the randomly arranged orders. That reading experience is dizzying and un-fun.

In sum, when you run searches on all the topics together that have similar metadata, you end up with assortments of small chunks that lack the continuity and context of a larger chapter or book. This simply seems to be the tradeoff of chunking your content. Your search results become more like a collage, but you have more flexibility in how you arrange your topics.

Blog Sponsors

Paper.li as an Alternative to Google Reader [Screencast]

Tom Johnson — Fri, 08 Oct 2010 14:40:20 +0000

To find good content online, I find that I’m going less and less to Google Reader and more to sources like Paper.li, an automated content curation tool that filters out some of the content noise. The problem with Google Reader is lack of content curation. You get a ton of noise, regardless of how fine-tuned your list of feeds are. With tools such as paper.li, which rank the most shared links on Twitter (based on a user’s followers, a hashtag, or a list), you can filter out the first layer of noise and more quickly find relevant content.

http://www.youtube.com/watch?v=uwXeWqaFP9A

Blog Sponsors

I Need Your Human Aggregated Content

Tom Johnson — Mon, 29 Jun 2009 13:34:37 +0000

If you have a way of tagging or marking the good content you read online — such as adding it to a specific category on your blog, bookmarking it through Delicious, or putting the link on some other online site — send me the RSS feed for it, and I’ll add it to the Yahoo Pipes aggregated feed that I have going with Writer River.

Here’s what the Yahoo Pipes feed looks like at the moment.

Writer River Yahoo Pipes feed

It’s simple compared to other Yahoo Pipes feeds. Basically the pipe takes RSS feeds from as many sources as I add here, sorts the posts by the date published, filters out any duplicate titles, and then merges all the information into one RSS feed. Writer River then displays this RSS feed on its home page. When you subscribe to the Writer River RSS feed (or when you subscribe to Writer River’s email delivery or Twitter updates), you’re also subscribing to this same Yahoo Pipes feed.

I’m convinced that human-assisted aggregation and filtering, with the help of such tools as Yahoo Pipes, is the trend for managing the deluge of information online. Since everyone is an author, publishing on separate sites, RSS is the only way to keep up. And people are publishing like mad, pushing out about a million posts a day.

Post titles are often hit and miss in terms of quality, so some human filtering is necessary. We need people to pick and choose the good content from the poor. People are naturally doing this all the time. I’m just trying to leverage those efforts in an effortless way to pull all of this good information into one running feed. This is what Writer River is all about. It attempts to gather all of this worthwhile content and help you find better information more quickly. If enough people participate, the quality of content flowing through Writer River could easily surpass the quality of any print publication.

Here’s an example. I like UXMatters, but I missed the latest articles published on it because I have hundreds of feeds in my feedreader and I don’t sit there watching feeds all day. However, Alistair Christie saw an interesting UX Matters article by Mike Hughes and posted briefly about it in his What I’m Reading category. I saw it on Writer River because Alistair told me about his What I’m Reading feed, and I added it to the Yahoo Pipe that’s feeding Writer River. I checked out the article tonight and immediately felt it was a valuable post. Without this human filtering and aggregation, I would have missed the post.

Now imagine if not just one or two people submitted similar What I’m Reading or What I’m Bookmarking feeds to Writer River, but dozens, even 100 people. It would be like having 100 researchers scouring the Internet for you, looking for the best posts available.

Here’s a little more math. Let’s say on average, the 100 researchers post one article a day to their What I’m Reading feed — one article a day they feel is worthwhile. In one month, that would be 3,000 articles.

Now of course not everyone has the same interests and tastes as you, so let’s say that only about 10% of these “worthwhile” articles are actually interesting to you. That still means that in one month, you’ll have 300 worthwhile articles to read.

Compare that to static print publications like the Tech Comm Journal, Intercom, the Communicator, or other print publications, which only have about 10 articles per issue, and you begin to see how valuable and powerful human aggregated content can be. This is the rationale behind Writer River. We now need more people to add feeds to it.

The manual method of going to the Writer River site and publishing a link to your post is somewhat archaic. It takes time and is slow. It takes effort. But the RSS feed doesn’t take effort. It only asks that you share your what-I’m-reading RSS feed with the Yahoo Pipe (by sending it to me, so I can add it), and then you don’t ever have to return to the site again. Content will just flow through the feed, however you choose to subscribe to it.

It makes sense to somehow mark or tag or bookmark or post or share or tweet good content that you read, right? You want to hang on to that article somehow so that you can find it later. That’s the nature of reading. But for online content, you need a method for keeping track of it, because the World Wide Web is too deep and wide and slippery to find something again after letting it go.

For those people who don’t have a blog or Delicious account, or Identi.ca or some other way of posting or marking content, I recommend starting one. One of the easiest ways to keep track of your good reads is through a WordPress.com blog, which is free, requires no maintenance, and provides you with an easy-posting bookmarklet that allows you to quickly add a link in two clicks from any page you’re reading.

Let’s pull together these efforts. Rather than having everyone run in their own direction, which accomplishes little, let’s harness all these individual efforts (which people are already doing) and turn them into a massive collective effort that dwarfs anything one simple person can do alone. Send me your category-specific RSS feed or links page and we’ll build an information machine that churns out the best content of the web without requiring you to do much at all to find it.

Blog Sponsors

“What I’m Reading”: A New Feature on My Site and a Tweak of Writer River

Tom Johnson — Wed, 24 Jun 2009 13:43:18 +0000

I’m trying something a little new on my blog. Previously, every time I read a cool post, I submitted the link to Writer River. The problem with that, however, is that posting to another site isn’t such a smart search engine optimization (SEO) strategy. Using the Writer River method, people who follow trackbacks don’t follow them back to my site (idratherbewriting.com), but rather go to another site (writerriver.com).

Additionally, it’s more beneficial for me to link to others from my idratherbewriting.com site, because it has a higher authority than writerriver.com. Links from higher authority sites are more beneficial in transferring search engine visibility than links from lower authority sites. For example, a link from NYTimes.com will push you to the top of Google results while a link from Sam’s vacation blog probably won’t have much influence.

So here’s what I did to better search engine optimize my site. I created a new section on my site called What I’m Reading. The page shows all the posts I’m reading (which I want to share), with short commentaries or summaries about the content. This way I keep the keywords and links on my site. I’m hoping that this strategy will create more pull back to my own site and will increase the rank of those I link to, more so than links from Writer River.

I didn’t want to limit my reading page to blogs only, because I read books too. So I used Shelfari, an online bookshelf site, to embed a few bookcases of books I’m reading, books I plan to read, and books I’ve read. If you buy a book through one of my Shelfari bookshelves, I will someday get Amazon affiliate revenue. It just made sense to consolidate everything I’m reading on a single page.

Technically, setting up this What I’m Reading page wasn’t that easy to do. The WordPress geeks can read on for the details, because this post is moving from conceptual to technical information. “What I’m Reading” is a category on my site hidden from the main page and RSS feed. I also excluded the posts from appearing in the Next and Previous links at the bottom of the home page (index.php).

I then used a custom category template, naming it category-1246.php, so that when users click the What I’m Reading category (the category ID is 1246), it opens category-1246.php rather than category-php or archives.php, which is the generic template for all categories. With this custom category template, I customized the sidebar, added some intro text at the top and inserted the javascript code from Shelfari to display the bookshelf widgets.

In the custom category template (category-1246.php), I also hid the post title and manipulated the styles a bit. I hid the title tag because I’m using the WordPress Press It bookmarklet to quickly and easily post links from the articles I’m reading while viewing the articles (rather than logging in to my WordPress dashboard to post them). The Press It bookmarklet automatically creates a link to the article I’m reading, so I didn’t want this link to be redundant with the title of the post—hence I removed the title from the category-1246.php template.

However, here’s where it got tricky. Trackbacks are one of my main SEO strategies, because most people are curious to know what you’re writing about them, even more so than what you add in the comments below their posts. A trackback sends a notification to the original blog authors that someone has linked to them. It’s like tapping them on the shoulder and saying hey, this is what I’m writing about you.

The problem is that a trackback’s link opens the single post template (single.php) rather than the category-1246.php template I customized. This leads to a major shortcoming of WordPress: you can customize category.php, but not single.php.

With a little research, I found a script that I could insert into my functions.php file that gives me the same functionality with single.php as category.php, so I then created a custom single-1246.php that matched category-1246.php, and I added a note at the top letting people know a bit about the page, because I didn’t want people thinking I was scraping their feed (according to Feedburner, about 100 people are scraping [stealing and reposting] my RSS feed). My short commentary and summary next to their links also helps avoid the appearance of scraping.

I also created a custom RSS feed and email delivery option for content specifically on my What I’m Reading page. However, I wanted the posts to update my regular Twitter feed (where I have most of my followers). Using Twitterfeed, I pointed the secondary RSS feed to my main Twitter account, so now I have two feeds pointing to my Twitter account. When my main feed updates Twitter, the tweet is prefaced with New Post. When the What I’m Reading feed updates Twitter, it’s prefaced by Recommended Read.

Now, there’s another complication. I’m not abandoning Writer River (a community link blog I started) by any means. I’m trying to move to a model that allows more flexibility and automated submissions. To accommodate this, I had to change Writer River a bit. First, because of all the spam that keeps seeping through, I changed the default registered user role to Contributor (which means I’ll have to approve their drafts first). For people I recognize, I’ll keep their role as Author.

Additionally, the Writer River home page now shows an aggregated RSS feed rather than links to the direct Writer River posts. The aggregated RSS feed displays results from my What I’m Reading category feed, the Writer River feed, and anyone other “good-reads” type feeds that people want to submit to me. The items in the feed are sorted by date.

To aggregate the feeds, I used Yahoo Pipes, which allows you to create and filter and apply rules to large numbers of feeds, and then spits out a single RSS feed from those multiple feeds.

I created a page on Writer River called Latest Posts, and I set this as the home page of the site rather than a reverse chronological list of only the latest posts from Writer River. (You can do this with WordPress through the Settings > Reading options.) On this new Writer River home page, I used the Simplepie plugin for WordPress to parse and display the Yahoo Pipes feed.

The benefit of displaying an aggregated Yahoo Pipes feed on Writer River rather than just content posted to Writer River is that it allows every blogger to do what I’ve done with my What I’m Reading page. Bloggers can simply designate a category that says “Recommended Reads” and select it when they post links to something worth reading.

I guess this assumes the bloggers would also be on WordPress, because WordPress has feeds for each category by default. For example, the feed for my What I’m Reading category is http://idratherbewriting.com/category/whatimreading/feed. I’m not sure if the same is true for Blogger and other platforms.

So if you’re a blogger and you have a category for good-reads, or something similar, let me know what the RSS feed is and I’ll aggregate it with the Writer River Pipes feed.

Finally, to encourage people to share links on Writer River, I created a Spring Widget, which is a little RSS reader that anyone can embed in the sidebar of their blog. To get the code for the Spring Widget, just click the Get this widget link below the Spring Widget.

I want to personally thank Alistair Christie, a tech comm. podcaster and blogger in the UK, for providing feedback and advice on how to handle the Writer River setup. If anyone else has suggestions for me, please let me know.

Blog Sponsors

Yahoo Pipes and the Mashed Up World of Aggregated, Filtered, Blended Information

Tom Johnson — Sat, 24 Mar 2007 05:11:47 +0000

Yahoo Pipes is a new online tool that allows you to blend, manipulate, and combine feeds from various data sources to create a streamlined, single feed of information. Essentially Yahoo Pipes allows you to create feed mashups of different data sources without having a knowledge of programming.

Yahoo Pipes has received a lot of praise. In its debut, Tim O’Reilly said:

Yahoo!’s new Pipes service is a milestone in the history of the internet. It’s a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output. Yahoo! describes it as “an interactive feed aggregator and manipulator” that allows you to “create feeds that are more powerful, useful and relevant.” While it’s still a bit rough around the edges, it has enormous promise in turning the web into a programmable environment for everyone.

A few examples will bring this tool’s capability into focus. Suppose you want to combine your favorite feeds into one feed. Yahoo Pipes allows you to aggregate the feeds, choose a sorting order, define filters, truncations, keywords, and so on. The pipe then generates a single RSS feed containing this information.

In another example, let’s say you want to monitor the web for keyword searches on “technical writing.” Traditionally, you could perform individual searches on various search engines, even subscribing to RSS feeds for each engine’s search results. But Yahoo Pipes allows you to define different data sources and then run your keyword search across all the data sources at once.

The Aggregated News Alerts Pipe (shown visually by the image on the left) takes the keywords you input and searches for the across the following search engines: www.blogpulse.com, news.search.yahoo.com, technorati.com, blogsearch.google.com, icerocket.com, findory.com, yahoo.com, www.bloglines.com, search.yahoo.com, api.technorati.com, www.icerocket.com, bloglines.com, search.live.com, news.google.com, rss.findory.com, blogpulse.com, google.com

After you type a keyword and click “Run this pipe,” the pipe retrieves the data and allows you to subscribe to the results as an RSS feed. This is an excellent way to keep your eye on the blogosphere for particular topics.

The BI Jobs Pipe, another piping application, allows you to search various sources for jobs, after inputting a location and job title.

Yahoo Pipes is not immediately intuitive (but here’s a good tutorial for doing the simplest mashup). Essentially, the pipe-building is a bit like creating an advanced search with different criteria. You can define operators, filters, number of returns, dates, locations, and other parameters for the search returns. You also define the data sources. The pipes can be robust and intricate, or they can be simple. What’s totally new is the visual drag-and-drop pipe interface.

Yahoo Pipes also allows you to clone well-built pipes and tweak them to suit your own purposes. Because of this, Yahoo Pipes is very web 2.0. You can browse the most commonly run and cloned pipes by clicking “Browse Pipes” from the Pipes home page.

Although I think the product is unique, innovative, and powerful, I was little disappointed in Yahoo’s documentation. I may be mistaken, but it appears Yahoo launched its innovative product with little instruction. Their home page says,

How do I use pipes?

Please check back, in a few days we’ll have online tutorials demoing how to get the most out of Pipes.

The product was launched more than a month ago. I guess if the product is cool enough, users will write the help.

The drag-and-drop interface, with connecting pipes all over the place, make it visually appealing and fun. In the near future, I’d like to create technical writing pipes that would be useful to our community. I created a simple pipe that aggregates 10 different tech writing feeds, but for some reason when I subscribe to the pipe’s RSS feed in IE or FeedDemon, it strips the feed of its formatting. Another limitation is that you can’t see the source each feed is coming from (although Lifehacker apparently has a workaround).

In using Yahoo Pipes, it’s probably easiest to find successful and interesting pipes, clone them, and then tweak them to suit your purpose.

Update: I found some excellent video tutorials for using Yahoo Pipes at http://usefulvideo.blogspot.com. There’s also a written tutorial here.