Till Nagel


Processing RSS Feeds

by Till Nagel.

This tutorial gives an overview on how to load RSS feeds into Processing.org sketches, how to read and use the available data, and how to display it. Processing’s internal XML methods are used, and various examples visualizing the contents of the news feed are shown.

Note: This tutorial is for Processing 1.5 (using XMLElement). For Processing 2.0 it’s similar but you need to use XML instead of XMLElement, and some other names for the methods (see Processing’s new XML reference). Below, the FeedVisRadialThumbnails example also comes with a sketch for 2.0.

Read this general introduction or jump directly to the tutorial kick off or to the first examples with explanations.

RSS

RSS (Real Simple syndication) is a format for delivering regularly changing web content. An RSS file consists of both static information about the feed, as well as dynamic items. These items normally are news items or blog entries, but they can be about any web-accessible content. Each item is inside an <item> element and contains among others a title, a url, and a description.

<item>
	<title>New Tutorial</title>
	<link>http://www.example.com/creative-coding.html</link>
	<description>
		A new tutorial is online. It is about lorem ipsum dolores
	</description>
</item>

Depending on the specific feed, other elements can appear. They can contain further metadata, or link to media files, such as images, and flash movies, but any data could be described in RSS (feed readers simply ignore elements which they don’t recognize).

  • What Is RSS? – RSS Primer: One Page Quick Introduction to RSS

XML

RSS formatted files are specified using XML (as are many other web formats, such as HTML, or SVG). XML (Extensible Markup Language) is a markup language for documents containing structured information. These are structured as elements, attributes, values, and others. For instance, in

<guid isPermaLink="true">http://www.tillnagel.com/2007/01/01</guid>

the guid is the element, the isPermaLink is an attribute (with an attribute value of true), while http://www.tillnagel.com/2007/01/01 is the content value.

The elements can be structured hierarchically. If you look in the above RSS example you see that <item> has three sub-elements (title, link, description). These are called children elements.

To read and write XML files in Processing there are different possibilites. Besides Processing XML libraries such as proXML, there are also full-blown frameworks such as dom4j. But for now, we stick to the Processing core library.

(For further information on how to read RSS feeds with more complex libraries visit this Code & form article “Read RSS feeds in Processing”)

Finding data

There are myriads of RSS feeds in the wild wild web. Besides your favourite blog, there are a lots of web feeds with a huge range of diverse non-news data. For instance, weather forecasts, last uploaded photographs, or personal link recommendations.

The general path to find good data is to

  1. Choose an RSS feed
  2. Analyze the XML structure
  3. Identify interesting data
  4. Select XML elements and values
  5. Do creative stuff

Load and use feeds in Processing

So, let’s say you have chosen the RSS feed about Processing updates (http://processing.org/updates.xml). Now load that feed and start utilizing the information therein.

Analyze feed

First, take a look which information the chosen feed has. Therfore, open the feed URL in your browser, and view the page source. Then you can see the XML structure and decide which data to use, and how to access it.

Clipping of an RSS XML

RSS feed shown in Firefox (clipping)

Using XML in Processing

To load an RSS feed (or any other XML file) create a new XMLElement and provide a URL in the constructor.

String url = "http://rss.news.yahoo.com/rss/topstories";
XMLElement rss = new XMLElement(this, url);

Afterwards, walk through the XML structure to the elements to use. To begin, simply get the title of each item with rss.getChildren("channel/item/title"). The paramater specifies which elements to return as array. The XPath channel/item/title selects the elements hierarchically – compare to the XML structure above.

// Load RSS feed
String url = "http://processing.org/updates.xml";
XMLElement rss = new XMLElement(this, url);

// Get title of each element
XMLElement[] titleXMLElements = rss.getChildren("channel/item/title");
for (int i = 0; i < titleXMLElements.length; i++) {
	String title = titleXMLElements[i].getContent();
	println(i + ": " + title);
}

Inside the loop get the content of every title element, and store it for later use. Besides getContent() the other important method is getStringAttribute(attrName) to get the value of an attribute of the element.

Example: SimpleFeedReader

Loads RSS feed and stores the titles of each item in an array.

Example: SimpleFeedVisualizer

Loads RSS feed and visualizes the titles in a very simple manner. Color and bar depend on the length of the corresponding title.

Use media files

Besides reading text data like title, or description you can also get media files from a feed. Given that the feed contains these data.
BBC News for instance provides a thumbnail image for each of their news items.

size(400, 400);

String url = "http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/world/rss.xml";
XMLElement rss = new XMLElement(this, url);
// Get thumbnail of each element
XMLElement[] imageURLElements = rss.getChildren("channel/item/media:thumbnail");
for (int i = 0; i < imageURLElements.length; i++) {
	String imageURL = imageURLElements[i].getStringAttribute("url");
	PImage img = loadImage(imageURL);
	// do something, e.g.
	image(img, random(width), random(height));
}

First, all <media:thumbnail> elements are extracted from the RSS file (line 7) with the getChildren() method. Then, in line 9 the URL of the image is read (from an attribute). Next, the image is loaded from that web address and stored in an PImage variable (line 10). As you can see the loadImage() function not only is able to load image files from the data folder but also directly from an internet address.

Keep in mind that the way how to access media files (or any other element) can differ from feed to feed. Always look in the specific RSS XML structure first (see Analyze Feed paragraph).

Example: FeedVisRadialThumbnails

Displays the image thumbnail of each item in a radial layout. The user can rotate and scale the image wheel with the mouse, interactively.

Example: FeedVisColoredTitles

Displays titles in a color picked from the center of the thumbnail. Images and headlines scroll over the screen.

Item class

To simplify using the item information, and to ease storing, updating, and displaying each item’s data the following class can be used as a basis:

class Item {
	String title;
	PImage img;
	float x;
	float y;

	// Creates a new Item with title, image, and position
	Item(String title, PImage img, float x, float y) {
		this.title = title;
		this.img = img;
		this.x = x;
		this.y = y;
	}

	// Displays this item
	void display() {
		image(img, x, y);
	}
}

This class (or an extended version of it) is used in the two examples above.

Download data from within an applet

“For security reasons, a Processing sketch found online can only download files from the same server from which it came.” (from Processing Reference – loadImage())
Thus, all examples from this tutorial do not work correctly if run on the web. Whether loading RSS feeds, or external images, the processing sketch is permitted to do this only if run offline.

So to get RSS feeds from another server you need a special page on your server, which forwards the original data.

(Explanations need to be written.) For now, visit this thread at the Processing discussion board: Topic with how-to create a PHP file

11 comments on ‘Processing RSS Feeds’

  1. Dylan Weening says:

    Hey Mr. Nagel,

    I am trying to create your feedvisradial design for RSS Feeds using your code. So far that seems to be one of the only things that has caught my eye as something interesting and useful. However, I am either downloading your processing sketches wrong, or it just seems to not work. My error always says that processing.xml no longer exists as there is no library found for it. Is there an up-to-date code that I can plug into processing so I can see your creation? By the way, this is one of my first times using this software, and thats why it would be amazing if I could get this to work so I can have a starting point to play around with.

    Thank you!
    Dylan

    • Till Nagel says:

      It should work anyway. That error only means you don’t need to import processing.xml.*; anymore (in Processing 1.5) If you wait just for a bit the feed and the images are loaded and displayed. You also could simply delete that one import line.

      If you are using Processing 2.0 you need to download the updated sketch (above). Processing 2.0 has changed how it handles XML.

  2. Dylan Weening says:

    is there a reason that the class Item cannot be found?

  3. Dylan Weening says:

    I figured it out. Thank you Nagel! Also, do you know of any other rss feeds besides bbc that have the similar characteristic of creating a pictured thumbnail for each news article? And, is there a way to actually access each thumbnails contents using processing?

    • Till Nagel says:

      There are plenty of RSS feeds with thumbnails.

      And yes, you can access the image directly in Processing. Take a look at the FeedVisColoredTitles above for a simple example.

  4. Matt H. says:

    So, I’ve tried the updated version of FeedVisRadialThumbnails-2.0.zip and it absolutely will not run. It gives me an error about the constructor being undefined every single time. I’m running the newest version of Processing.

    • Till Nagel says:

      You need to update the XML loading. Change it from
      new XML(this, "http://examples.com/rss.xml");
      to
      loadXML("http://examples.com/rss.xml")
      This has been changed in Processing 2.0a7
      Thanks for the note.

  5. Lee Erickson says:

    Thanks for your article. I am having great fun with Proccessing and reading the current weather and trying to display it on an IEE serial port pole display.
    I am pulling the weather from http://rss.weather.com/weather/rss/local/37801?cm_ven=LWO&cm_cat=rss&par=LWO_rss

    The item I want is:

    0.8005517227804482
    Fri, 17 Oct 2014 15:57:37 EDT

    <![CDATA[Sunny, and 73 ° F. For more details?]]>

    I am stuck for an idea how to read the content of the tag with out the image link.
    Here is some of my code:

    //Find Current conditions item
    String title = titleXML[i].getContent();
    String date = dateXML[i].getContent();
    String description = descriptionXML[i].getContent();
    // String description = descriptionXML[i].getChild();
    // if (title == “Current Weather Conditions In Maryville, TN (37801)”) {
    if (title.equals(“Current Weather Conditions In Maryville, TN (37801)”) == true) {

    //Clear to Home
    println(0x0C);
    myPort.write(0x0C);
    //IEE Cusomter Display Clear Home
    myPort.write(0×15);

    println(title + “\n”+ date);
    // myPort.write(title + “\n”+ date +”\n”);

    println(description + “\n”);
    myPort.write(description + “\n”);

    }

  6. Lee Erickson says:

    The submission above does not look like the code I submitted. I assume the XML got interpreted.

    What is want is just this part, “Sunny, and 73 ° F” of the item description.

Leave a Reply