Yahoo Pipes and the Mashed Up World of Aggregated, Filtered, Blended Information
Yahoo Pipes is a new online tool that allows you to blend, manipulate, and combine feeds from various data sources to create a streamlined, single feed of information. Essentially Yahoo Pipes allows you to create feed mashups of different data sources without having a knowledge of programming.
Yahoo Pipes has received a lot of praise. In its debut, Tim O'Reilly said:
Yahoo!'s new Pipes service is a milestone in the history of the internet. It's a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output. Yahoo! describes it as "an interactive feed aggregator and manipulator" that allows you to "create feeds that are more powerful, useful and relevant." While it's still a bit rough around the edges, it has enormous promise in turning the web into a programmable environment for everyone.
A few examples will bring this tool's capability into focus. Suppose you want to combine your favorite feeds into one feed. Yahoo Pipes allows you to aggregate the feeds, choose a sorting order, define filters, truncations, keywords, and so on. The pipe then generates a single RSS feed containing this information.
In another example, let's say you want to monitor the web for keyword searches on "technical writing." Traditionally, you could perform individual searches on various search engines, even subscribing to RSS feeds for each engine's search results. But Yahoo Pipes allows you to define different data sources and then run your keyword search across all the data sources at once.
The Aggregated News Alerts Pipe (shown visually by the image on the left) takes the keywords you input and searches for the across the following search engines: www.blogpulse.com, news.search.yahoo.com, technorati.com, blogsearch.google.com, icerocket.com, findory.com, yahoo.com, www.bloglines.com, search.yahoo.com, api.technorati.com, www.icerocket.com, bloglines.com, search.live.com, news.google.com, rss.findory.com, blogpulse.com, google.com
After you type a keyword and click "Run this pipe," the pipe retrieves the data and allows you to subscribe to the results as an RSS feed. This is an excellent way to keep your eye on the blogosphere for particular topics.
The BI Jobs Pipe, another piping application, allows you to search various sources for jobs, after inputting a location and job title.
Yahoo Pipes is not immediately intuitive (but here's a good tutorial for doing the simplest mashup). Essentially, the pipe-building is a bit like creating an advanced search with different criteria. You can define operators, filters, number of returns, dates, locations, and other parameters for the search returns. You also define the data sources. The pipes can be robust and intricate, or they can be simple. What's totally new is the visual drag-and-drop pipe interface.
Yahoo Pipes also allows you to clone well-built pipes and tweak them to suit your own purposes. Because of this, Yahoo Pipes is very web 2.0. You can browse the most commonly run and cloned pipes by clicking "Browse Pipes" from the Pipes home page.
Although I think the product is unique, innovative, and powerful, I was little disappointed in Yahoo's documentation. I may be mistaken, but it appears Yahoo launched its innovative product with little instruction. Their home page says,
How do I use pipes?
Please check back, in a few days we'll have online tutorials demoing how to get the most out of Pipes.
The product was launched more than a month ago. I guess if the product is cool enough, users will write the help.
The drag-and-drop interface, with connecting pipes all over the place, make it visually appealing and fun. In the near future, I'd like to create technical writing pipes that would be useful to our community. I created a simple pipe that aggregates 10 different tech writing feeds, but for some reason when I subscribe to the pipe's RSS feed in IE or FeedDemon, it strips the feed of its formatting. Another limitation is that you can't see the source each feed is coming from (although Lifehacker apparently has a workaround).
In using Yahoo Pipes, it's probably easiest to find successful and interesting pipes, clone them, and then tweak them to suit your purpose.
Update: I found some excellent video tutorials for using Yahoo Pipes at http://usefulvideo.blogspot.com. There's also a written tutorial here.
About Tom Johnson
I'm an API technical writer based in the Seattle area. On this blog, I write about topics related to technical writing and communication — such as software documentation, API documentation, AI, information architecture, content strategy, writing processes, plain language, tech comm careers, and more. Check out my API documentation course if you're looking for more info about documenting APIs. Or see my posts on AI and AI course section for more on the latest in AI and tech comm.
If you're a technical writer and want to keep on top of the latest trends in the tech comm, be sure to subscribe to email updates below. You can also learn more about me or contact me. Finally, note that the opinions I express on my blog are my own points of view, not that of my employer.