Sonibyte Podcasting Blog

Battling Bots

Hope the title of this post doesn't raise fears that we are working on Terminator 5 (or that we are anti-cyborg!).  Instead, we thought we'd offer a couple comments on the ongoing struggle we face with what are known as "bots."

In case you are not familiar with them, bots are programs that "harvest" files from sites that have "interesting" content. They run off servers in places like China and Russia.  Podcast files like those our clients offer perfectly fit the profile of the type of material they are looking for.  Our clients' posts have exciting topics, lots of tags and keywords, and hefty enough file sizes (2MB to 20MB) to seem worth snatching.

Once a file has been found and collected, the bot site will turn around and feed it to other users. Some bots load the files they distribute with viruses and other nasty things, but most seem to use them as a way to attract traffic.  By exposing tons of interesting files, bots bring in visitors who can then be shown ads and monetized in other ways.  Their activity is both illegal and sleazy--but it is not especially dangerous for either the bot owner (who is out of the reach of our law) or for the file owner (who is not directly associated with the bot site).

Like spam emails, bots are annoying and increase the cost of serving legitimate users.  We try to block the IP addresses of known bots.  However, there are more than 3,000 bot types tracked by our favorite bot tracker ("Bots vs Browsers").  These bots use more than 300,000 different IP addresses--so it is virtually impossible to block them all.

In case you wondered, I have not found a scientific way to spot bot traffic.  My current approach ugly and quite manual and has these three steps:

1. Grab our log file and summarize it by IP address (down the side) and month (across the top).
2. Scan the resulting matrix, looking for instances where there are a number of requests from IP addresses that are contiguous.
3. Check one or two of the IP addresses to see if they are known to be used by bots.

We are not the only site afflicted by bots.  I suspect that all major podcasting sites serve files to bots.  Our entire industry's download statistics are inflated, as a result.  Of course, the same is true for a lot of other media businesses that count TVs that are on but not listened to, newspapers that are delivered but not read, etc.

I had time in the past few weeks to hand-check traffic from several of our clients, to see how much of their traffic came from bots.  I was pleased to see accounts with less than 5% of downloads going to bots and disappointed that some of our accounts showed 40% or more going to them.  The percentage of downloads going to bots varies a lot.  Here is about twenty months of data on a typical client:

As you can see this podcast saw as much as 20% of its downloads going to bots and as little as 1%.  This makes it hard to suggest any across the board adjustment for this problem.  Instead, until we can weed out these pests entirely, we must just recognize that all of our download counts are somewhat inflated.  Users of our statistics (such as advertisers) may adjust the value they attribute to each download down a bit, as a result.  They might also ask podcast distributors for proof that they have made reasonable efforts to block bots.  Otherwise, there could be an incentive for podcasters to open up their servers and use bots to increase their download traffic and their "ad inventory."

Useful Sources of Podcasting Information--Part 1

We thought it might be helpful to offer a set of links to useful sources of information on Podcasting.  We have found similar lists to be helpful, when we find them on other sites.

 

This type of list always seems “random,” and is necessarily incomplete.  We invite our readers to add their suggestions.  We hope to offer updates and additions to the list, every few months.

 

Podcast Free America was created by Ryan Irelan to provide information on Podcasting for all levels of users.  It includes a guide on Podcast marketing, news and information on companies in the area, and a bunch of “how to” stuff.  All well written and thoughtful.

 

As the head of a successful Podcast production firm, Scott Whitney is well-positioned to give advice to business and corporate Podcasters.  His Podcasting for Business blog has a lot of great tips for the growing ranks of business Podcasters.  The only thing I don’t like is that he allows annoying self-promo pop overs on his site.

 

Awakened Voice from Robert Safuto, covers all types of technology and new media.  However, he has done a lot of posts about Podcasting, and does a great job of fitting audio feeds into the broader context of social and streaming media.

 

Podcast411 carries interesting interviews with the creators of a number of the most popular Podcasts.  The hosts of this site have added a bunch of useful tools and links, to make their site a hub of information for those interested in the area.

 

Rawvoice offers the Blubrry Podcasting hosting service (yes I spelled it right).  They also have a pretty useful blog.  They have some of the best info I’ve seen on Podcast statistics.

 

Feedburner’s blog is pretty much a must-read for anyone interested in feeds (and Podcasts are a type of RSS feed!).  Feeburner is now owned by Google…so it is natural to see this blog also as a way of monitoring Google’s activities in the feed area.

 

NMP (New Media Pro) has a whole set of Podcasts that discuss various aspects of creating Podcasts.  Interviews, commentary, polls and advice.

 

The Hopkinson Report has done two episodes on Podcasting that are pretty handy.  The first talks about setting up to do a Podcast and the second talks about Podcast marketing.  The blog is associated with Wired, one of our Conde Nast clients.

 

Kevin Davis has published three posts in his blog that describe how Podcasting can be used to boost the visibility of smaller businesses.  I was glad to see someone talking about this aspect of Podcasting.

 

The Podpiper blog has a lot of information on using Podcasting for educators.  Ted Lai makes a good case for Podcasting as an educational tool.  The Around Podcasting blog recently offered a related post on using Podcasting in Elearning.  This blog also has a lot of other useful info on Podcasting for businesses.

 

I thought I’d close by sharing two nice examples of people talking about our in-house Podcasts.  The first is from a site on Islam that links to our Daily Koran reading.  The second is from Gerald England, one of the artists who we featured on Poem of the Day.  Unfortunately, we neglected to get his permission before we read his poem (we apologized subsequently—and apologize again, here).  I thought his post on this would be interesting to those of you who either like to track the use of your work or those who are thinking about using the work of others without permission (don’t do it!).

 

Understanding Podcast traffic statistics--Part IV, Counting Subscribers

How can we tell how many people subscribe to our podcast feeds?  Unfortunately, there is no central registry of subscription information.  Few feed aggregators and directories announce how many people have subscribed to each of the feeds they index.  One example I could find was the honest and open people at Bloglines, always one of my favorite feed organizing tools.  A second is Google--which sticks the number of Feedfetcher subscribers it represents into its agent request string.  It is possible to get more info by using Feedburner statistics.  Unfortunately, with our dynamic approach to feed creation, this is tricky to implement.

So, what can we do, on our own?  We know that 844 distinct IP addresses downloaded our RSS documents, on January 16th.  57 of them requested more than one of our documents, for a total of 901 IP address/document combinations.  Of these 901 requests, 530 were made only once within twenty four hours.  It turns out that most feed readers automatically check a subscribed feed more than once per day.  (The default for iTunes I believe, is to check a subscribed feed every six hours.)  So, we can guess that the 530 single requests out our total of 3,113 RSS document requests were NOT from subscribers.  In fact, we can probably exclude as a subscriber anyone who asked for our document fewer than every six hours or five times in a day.  This allows us to eliminate 760 of our IP address/document combinations and 1,109 of our RSS document requests.

We now know that our maximum number of subscribers would be 3,113 - 1,109 = 2,004 RSS document requests / 4 = 501 subscribers.  Remember that we had one example from the feed reader side--the place where the subscription records are kept.  We had 40 requests for our RSS documents from IP address 65.214.44.29, an IP address that reverse DNS shows is Bloglines.  They claim they handle three subscribers for our podcasts.  Bloglines check feeds for their subscribers, once every hour.  That means that our three subscribers ought to have generated 3 x 24 = 72 requests for our RSS document.  We found 40 of these.  I suspect that there is another IP address for Bloglines (I didn't reverse DNS them all) that has the other 32 requests.

We can also get an estimate of subscribers from our file download log.  We happened to post new episodes on each of our three house feeds on January 16.  In theory, the podcatchers of those users who subscribe to our feeds should see these new episodes and download them within a relatively short time.  It turns out that 395 of our 2314 downloads occurred within twelve hours of their time of release and 231 occurred within six hours.  These numbers are in the same general ball park as the 501 number above.  I feel comfortable saying that between 10% and 15% of our downloads were from subscribers to our podcasts.

Around 1,454 listeners downloaded our files.

554 users came from podcast directories and other sources.  Some of these may have subscribed to our feed, after trying it out.

About 900 users learned about us by browsing our RSS document.  Many of these downloaded our podcast files almost immediately afterwards.

Between 230 and 345 of these immediate downloaders were probably subscribers, who were using a podcatcher to monitor our feed.

This creates a picture of a core group of dedicated listeners (300?), a group of regular visitors who "check out" our RSS feed on a directory and pick up a file when it interests them (600?), and a group of random visitors who pick up a file when they hear about it (550?).  You can see that it takes some work to accumulate subscribers and that "casual" visitors are important to overall file download numbers.  The best way to boost subscribers is probably to regularly add new (and interesting) episoides.  The best way to boost casual visits is probably to improve your search engine optimization and to upload to multiple podcast directories.  We'll talk about both of these issues in future posts.

In commercial terms, we can claim to reach between 10,000 listeners and 25,000 listeners per month.  If someone paid $10 for a "CPM" (Cost Per Thousand) worth of advertising on our site, and we described our site as having 2,000 downloads per day x 30 days (how it is normally done) = 60,000 downloads a month, they would pay us 2,000/1000 x 30 days x $30 = 1,800 for one month's advertising.  Our media would cost $1,800 / 25,000 distinct listeners = $0.072 per ad listened to.  Don't shudder--radio and TV can be much more costly than this, when you measure actual cost to reach a listener!  Since we download new podcasts each week, our 300+ core listeners would get up to four impressions during that one month period--a useful fact that makes our feed a bit more valuable in advertising terms.

I hope these posts have been helpful.  Let me know if you have ideas or tricks for understanding podcast statistics that I have missed.

Understanding Podcast traffic statistics--Part III, RSS Document Requests

In the second post in this series, we decided that on January 16th, 2008, we had probably provided podcast files to around 1,454 listeners.  These people either downloaded our files directly or found them via the link inside an RSS document (a feed).  How do we know which was which?

Just as we track downloads of our sound files, we track how many people download our RSS documents.  On January 16th, the RSS documents describing our podcasts were downloaded 3,113 times.  This RSS download number seems similar to our original file download number of 2,334--but that is only by chance.  A more careful inspection of the RSS logs will reveal that there are big differences between these two numbers.

For one thing, an RSS document describes a set of podcast episodes (our feed is also called a "channel").  So, a request for the RSS document describing our Daily Poem feed is different from a request to deliver our Langston Hughes poem file.  A person who gets our RSS document could request all of the files the document describes (we generally limit the number described in a feed to the last twenty podcast episodes), the most recent file, one or more older files, or none of the files that are listed in the RSS document.

Did our 3,113 RSS document requests have anything to do with our 2,334 file download requests?  There were only 844 distinct IP addresses that asked for our RSS documents.  611 of these show up in the list of 695 IP addresses that downloaded sound files from our site.  These users (or groups of users, if they shared a common internet provider) may have looked both at our RSS documents and downloaded files.

Notice the words "may have looked."  Thanks to the anonymity of the Web, we don't really know who is behind most of the visits to our site and exactly what they are doing with the information we give them.  It is hard to get that information from our Web traffic tracking system (we use Google Analytics), because our feed pages are created dynamically.

We can compare the time when the request was made for our RSS document and the time when our file downloads occurred.  We can then match the IP addresses on both sides AND match the channel for the RSS document with the channel from which our file came.  We had 901 downloads where an RSS request was followed within five minutes by a file download from the same IP address and for the correct channel.  These download are likely to be real requests for our podcast, by potential listeners who have seen and reviewed our RSS document.

We can pull something else from the RSS log.  We can look at what agent our visitors used, to request our RSS document.  Of the 3,113 requests, 284 were made directly (without any agent information in the http request).  I suspect that these were update requests from "podcast directories."  The folks at Podcast411.com are kind enough to keep a list of more than 100 of these directories.  Each time we update our RSS documents, we send our updated document to a number of several of the major directories.  They store our RSS document, index the information in it, and make it easy for podcast listeners to search for and find podcasts they want to listen to.

The requests for information from podcast directories look like "bot" inquiries--they cloak their requests and provide as little information as possible on where they are coming from.  As you'd expect, reverse DNS tells me nothing about these addresses.  Through some poking around, I figured out that the IP (67.192.44.33) with the biggest number of requests (180 of the 284) came from Podcast.com, which hosts a 60,000 entry podcast directory.

We think that 901 of our downloads came from people who first requested our RSS document and then downloaded our file.  The other 554 downloads probably came from people who looked at copies of our RSS file on a podcast directory.  These used this information to make their file requests--which is why we did not see a match between the IP behind their file download and the IPs in our RSS document requests.

The requests for our RSS documents came from various feed agents and feed readers.  Here is a quick summary of the agent breakdown for our in house feeds:

 

                                       

Agent	% of RSS Requests
Bloglines	1.4%
Google Feedfetcher	4.2%
iTunes	70.9%
Microsoft Zune	0.5%
MagpieRSS	1.6%
Totorosa-Toto	1.8%
Yahoo	9.0%
38 Other Agents	10.4%
Total	100.0%

Those who love Apple (and hate Microsoft) will be happy to see that iTunes dominates the list (and Microsoft's Zune reader is near the bottom).  I found it interesting see how many users were taking advantage of MyYahoo!'s nice ability to grab RSS feeds, and that Google's Feedfetcher seems to be making headway.  This pattern is different for different types of podcast.  I will look more at these variations, in a later blog post.

We now know roughly how many people got our sound file and we have a vague idea how they learned about it.  The only thing left is to see if we can tell how many of our listeners come back regularly.  In other words, how many of our downloads were due to people who had subscribed to our podcasts?

Understanding Podcast traffic statistics--Part II, File Downloads

In our last post, we gave a brief overview of how podcasts worked and defined some terms.  Now, it is time to get into the dirty details of looking at podcast traffic logs.

The first place to look is our file download logs.  The podcast sound files for our modest in-house blogs were downloaded 2,334 times on January 16th.  That seems pretty impressive, as it suggests we had an audience of more than 60,000 people per month.  If we group our results by IP address, we can see that  1,949 distinct IP addressed downloaded from our site.  This is because a number of IP addresses downloaded files from us, more than once.  311 of our 1949 IP addresses download more than one copy of the same file.  Their 696 downloads are about 30% of our 2,334 total downloads.

Why were particular users (IP addresses) downloading the same file over and over?  For instance, IP address 68.46.125.236 turns out to be Comcast--a major internet provider.  (It is easy to look up IP addresses using reverse DNS lookup from sites such as this.)  The 7 downloads of our reading of the poem "Dreams By Langston Hughes" could have been from a single user hitting the download button 7 times.  However, it is more likely that seven of our listeners are using computers served by Comcast.  When these users think they are connecting to us, they are actually connecting to a Comcast server that feeds their request through a different server and to us.

Note that Comcast also shows up also as IP address 76.27.59.224 and several other addresses.  Major Internet Service Providers (ISPs) such as Comcast, Verizon, and Speakeasy feed their traffic through many different IP addresses--as do portals such as Yahoo!, AOL, and MSN.

We also see multiple downloads from IP addresses such as 202.108.23.56, which was responsible for a total of 80 downloads, spread over 21 different files.  A reverse DNS on this address tells us only that the source is in China.  We can assume that these requests are coming from a Chinese internet provider similar to Comcast--one who has not fully disclosed its ownership of this server.  Since our "house" podcasts include English readings of the Bible, English readings of the Koran, and English readings of both classic and contemporary poetry, I'd guess that a lot of Chinese people are listening to them as part learning English.  It is nice to know that we are doing some good for the world!  By the way, lest you think that only Chinese people want to learn English, we also had at least 3 downloads from 83.202.100.211, which is Wanadoo, in France.

Of our 2,334 downloads, all but 41 were done by "agents."  These agents are also called "podcatchers."  There are twenty podcatchers listed on PodcatcherMatrix--and this nicely organized site helps you compare their features and functions.  The main function of a podcatcher is to watch the RSS feed of a podcast for new episodes.  When a new episode appears, the podcatcher grabs it and downloads it to the user's computer.  They can also provide a license or password authorization (when it is required), allow searching for new podcast feeds, and manage sychronizing the list of episodes on a user's computer with the list of episodes on her or his iPod or other portable player.

iTunes was responsible for more than 50% of the downloads on our site that were handled by agents.  Mozilla (the agent name used by Firefox browsers) added another 13%, followed by another 3% from download agents such as Doppler, Juice, and NS Player.  Unfortunately, about 34% of our downloads come from what are commonly referred to as "bots."  These are programs that scan the Internet looking for useful and interesting information.  When they find "content" such as our sound file, they greedily download it and store it away for later review and digestion by their owners.  Some of these bot downloads probably are from legitimate search firms or podcast directories.  The rest...well no one really knows where all the data absorbed by bots, really goes.

(By the way, remember that the RSS document was 64% iTunes?  Once you subtract the bots from the file download log, we get 77% of downloads via iTunes.  Our RSS log also gets bot requests--about 10% of our requests come from them.  Take these out and we get an adjusted percentage of iTunes RSS request of about 70%.  This is close enough to make us comfortable.)

If we subtract the 803 downloads from bots, we are left with a net of 1,531 downloads to listeners.  Of course, some of the multiple downloads may be due to retries (when a download fails and a user tries a second time).  I have seen an estimate on Jason VanOrden's blog that retries account for between 5% and 20% of all downloads.  I think I'll use the 5% number in our situation, and claim that we had 1,531 x 95% = 1,454 downloads on January 16th.

Short of getting users to install some kind of tracking software (possible--Volomedia has a pretty cool one), there is no way to know how many of these downloads were actually listened to.  But, no one knows how many people actually read newspapers or listen to programs that are broadcast on radio or TV.  Still, we'd like to know a bit more about our listeners, including how they found our podcast and whether or not they have subscribed to it.  We'll try to get more data on this from looking at our RSS request log, in my next post.

Understanding Podcast traffic--Part I, Some Basics

It is hard to make sense of the data we see on podcast traffic.  This is the first of a series of four posts, that will offer as much help as we can, on understanding and interpreting podcast traffic statistics.

Like most other podcast publishers, we look routinely at our file download logs and the records of how many people request our RSS feeds and those of our major publisher customers.  However, we often end up unable to answer fairly simple questions such as, "How many people listen to our podcasts" and "How many people subscribe to our podcasts?"

Our research is hampered partly by the way podcasts are delivered and partly because of the general anonymity and obscurity of Web interactions.  In the past, podcasting was a sideline and the numbers didn't matter that much.  Nowadays, many podcasters are starting to monetize their feeds by inserting ads into them.  The prices paid for these ads will be driven by how many people listen to these messages, and that increases the value of well-documented podcast traffic statistics.

We host feeds for a number of major magazines.  We can't/don't disclose their statistics.  Fortunately, we also publish several podcasts of our own, "Poem of the Day," "Daily Bible Verse," and "Daily Koran."  Therefore, we can and will share these numbers.

To keep things as simple as possible, I decided to focus on a single day in January.  I chose January 16th of this year, because it is not a holiday, it is not part of a holiday week, it is not a weekend day, and no huge national disaster or international event occurred that day.  (BTW, it also happened to be my sister's birthday!)  If you are analyzing your own logs, be sure to watch out for factors like the day of the week or the time of the year, as they will definitely affect your traffic patterns.

Before I start, I'm going to give an overview of how podcasts are distributed--just in case you forgot (or never bothered to find out!).  A podcast is a series of sound file (we are ignoring video, for the moment) on a particular topic.  Examples are radio shows (one of the first things distributed via podcast), comedy sketches, discussions of new gadgets, articles on news or politics, poetry readings, serial novels, Spanish lessons, and so on.  Most podcast listeners download podcast sound files to their computers.  They then either listen to the podcast episode on their computer or transfer it to their personal listening device (e.g., their iPod).  (For a great short description of how podcasts work, please look at this article from the University of Leeds.  For an even shorter version, see this post on podcast411.com--which is a great all around site for info on podcasting.)

When a user downloads our file, his or her computer has to establish a connection with our file server and request the file.  As a result, we get to see the user's IP address and what type of downloading software he or she is using.  (For instance, about 50% of those downloading our files on January 16th were using iTunes and about 13% were using Mozilla.  About 64% of those requesting our RSS documents were using iTunes--a difference we will talk about, later.)

How does a potential listener find out that we have a juicy sound file waiting to be listened to?  There are two paths to this knowledge:

1.  Some listeners could remember where we keep our files or know we have a Web page that links to our files.   Despite hard looking, I can't find much evidence that these people exist.
2.  Most (or all) listeners discover our files via our RSS document or "feed."  Our RSS document describes our podcast in both general and technical terms.  It says what category our podcast belongs in, supplies a text summary of our product, and shows where a "cover" photo resides.  It also lists the most recent episodes in our series of podcasts.  Most people examine RSS documents with an RSS "reader."  The best known of these for podcasting is Apple's iTunes reader.  (We will talk more about readers and podcast directories in a later post.)

After a user looks at our feed, he or she can choose to download one or more of the files listed in the feed.  If the user likes our content, he or she can "subscribe" to the RSS feed.  Once the user tells his or her reader that he or she wants to subscribe to a podcast, the reader will faithfully and regularly check our RSS feed document to see if we have posted any new episodes for our podcast.  When we update our feed by adding a new sound file, the reader will notice this and automatically download the new file to a subscribing user's computer.

You can see a page of podcast feeds for both our podcasts and those of our customers on our site at http://www.sonibyte.com/rssfeed and inspect a page that holds our Daily Koran feeds at http://www.sonibyte.com/rssfeed/35.xml.  For techie readers I'll mention that these pages are actually a transformation of our RSS document into an HTML page using some Java script and an XSL translation template.

Does this sound simple, so far?  Just wait.  You will see in our next three posts that it gets pretty difficult to decipher the relationships between listeners, subscribers, file downloads, and RSS document requests.