NPR deploys Splunk for web analytics

In a novel use of the software, National Public Radio is using the Splunk log search engine to analyze Web traffic for its audio streams and downloads.

NPR metrics analyst Sondra Russell described the setup at GigaOm’s Big Data conference, held Wednesday in New York.

Splunk offers what it calls a search engine for machine data. It was originally built to parse log files, or the files programs and hardware generate to document their transactions, errors and other operational information. By coordinating the timestamps of messages from different applications and hardware, Splunk allows system administrators to pinpoint difficult-to-locate system problems.

In recent years, however, customers have been expanding their uses of Splunk to other duties, explained Splunk Chief Technology Officer Erik Swan, also speaking at the event. Web traffic analysis and business intelligence are two such ancillary uses.

For much of its Web traffic monitoring, NPR uses standard Web traffic analytic software, which can deliver reports on how many people visit each Web page. Such software usually generates these counts by using cookies or by embedding each page with a small script that alerts the software when the page is rendered in a browser.

The media organization, however, found it difficult to get reliable usage summaries for a number of aspects of its service. For instance, the organization needed to get an accurate count of how many listeners tuned into their streamed audio and video programs.

To get this data, NPR had prepared a PHP script that would parse the server log files and translate the results into a form that could be digested by Adobe’s Omniture, a Web analytic tool. Getting information back, however, could take up to 24 hours, and Russell still didn’t trust that the results provided an accurate count.

In the cases of streaming usage, many users might start a stream, then pause it, and restart it. Or perhaps a user would restart a stream after a failed Internet connection. In the server log files, all these events were logged as separate events, not a linear sequence of actions by a single user. As a result, there was no way of determining how many connections were from different listeners, and how many were multiple streams to a single user.

“With our PHP scripts, we could not get that level of sophistication. So our numbers could be off,” Russell said.”It was hard to make rational decisions based on this.”

By working with Splunk, NPR could derive listener numbers and information directly from its servers’ log files. The software allows users to script search results and then graph the results, or show them on a dashboard.

In NPR’s case, for instance, it provided the exact number of listeners for each program that was streamed or downloaded. “With Splunk you can wrap multiple [log entries] into a single distinct visit, so I feel my numbers are actually accurate,” she said.

Splunk helped identify users’ mobile platforms as well. An increasing amount of traffic to the NPR site comes from mobile clients, such as iPhones, iPads and Android smartphones. In one case, a manager wanted to know which version of the iPhone operating system was most often used, as the results would direct the company’s design work for its iPhone app.

“I could not have answered that without Splunk,” Russell said.

Splunk also solved a seemingly unsolvable problem for the organization: determining how to pay royalties for streamed songs. NPR offers a streaming service for songs, called SoundExchange. It must pay out royalties for each song played, based on the number of listeners that stream had at the moment. “It can get complicated to reconcile the two,” Russell said.

Using Splunk, Russell was able to merge two files — a list of when each song was played, and the number of listeners that stream had when the song was played. “Splunk put them together,” Russell said.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab’s e-mail address is Joab_Jackson@idg.com