Reach Customers Online

Visitors and Conversions and Page Views, Oh My: Website Activity Reporting Tools

“Dirt bag soap” surely is an odd phrase. So is “My dog Mundi.” What do they both have in common? They showed up in my website log files recently. At least six visitors have used “dirt bag soap” to find this site. It turns out that a Google search on both terms results in this site being listed high in the results page. This column describes how that happens, what no-cost and low-cost tools exist to extract this data, and what it means.

The mystery of dirt bag soap can’t be solved through Google: the results link and the cached page link don’t show the entry on my weblog that Google found. On July 28, 2003 I blogged a story from the New York Times about how people use the internet to barter to start their businesses. I had seen the article in print, in the Fashion section no less. Dirt Bag Soap is one of the companies featured in the article. The owner traded his soap for soapmaking equipment, a logo design for his business, and other business necessities, dramatically reducing his costs and increasing his chances for success.

I found both search terms through a routine check of awstats, the free website activity tool provided by my webhosting vendor. Before we dive into tools, however, let’s talk quickly about where awstats got these phrases.

Every web server (and ftp server and mail server and …) generates a log of its activity so that it can be managed over time. You can use the log to determine how much data has been transferred over a period of time and other useful data. With websites, log files have a business purpose in addition to the technical purpose of finding out what happened. The log files can be used to determine how many people have visited, what pages they have seen, and other data.

Here is a line from a recent log file from this website:

168.226.183.249 – - [13/Aug/2003:11:31:38 -0400] “GET /images/rco_logo.gif HTTP/1.1″ 200 2481 “http://www.reachcustomersonline.com/how-to/articles/article.php?id=174_0_25_0″ “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”

Gibberish certainly, but here is what this data means:

• 168.226.183.249 is the IP address of the visitor to this website. Some IP addresses are specific to one computer while other addresses are shared. So a repeat of this IP address in the log files could be one computer one person or many people many computers. DNSStuff.com says this IP address means the visitor used an internet connection from Buenos Aires, Argentina.
• [13/Aug/2003:11:31:38 -0400] is the date and time of their visit.
• “GET /images/rco_logo.gif HTTP/1.1″ is the specific file requested. In this case, the server has recorded just one of the graphics and other files needed to display one page in the visitor’s web browser. By the way, to really educate and confuse you, GET is one of two ways to retrieve information from a web server. POST is the other method.
• 200 is the response back from the server as a result of the request. 200 means okay. 404 would mean the file requested was not found.
• 2481 is the file size (in kilobytes) of the item requested and sent.
• “http://www.reachcustomersonline.com/how-to/articles/article.php?id=174_0_25_0″ is the page the visitor requested. This line of the log file, therefore, records sending one element of this page.
• “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)” tells what browser (IE 6.0) and operating system (Windows NT 5.1) the visitor used to make the page request.

See, that didn’t hurt. While this information is esoteric, it can be valuable. Knowing what browsers visitors use can tell designers and programmers what sort of code to use to display images and content in a visitor’s web browser. And calculating the start and end time for a specific IP address can determine how long the visitor spent on the website (and which pages they visited and in what order).

There are two formats for website log files, common and extended. The extended format has to be configured but it is the only way to get useful data from a webserver. The common format yields a bare minimum of data that does not include referrer (where the visitor came from) or information about browsers and operating systems. Yet the extended format cannot tell you screen resolution (how wide and high the visitor’s computer screen is), geographic location, and other important data.

To get that data, you need scripting code embedded in your web page that captures and records that information. I should state here that none of the data collected by scripts or website log files is inherently identifiable to a unique individual or address. Nor does the scripting delete files or compromise the visitor’s computer. Worst case, scripting code would place a text file (called a “cookie”) on a visitor’s computer with a flag or number that tells the web server the visitor has been to the site before. (Of course, unscrupulous people could present you with a form which if you filled it out with your personal information could be tied to the cookie text file to identify you. That’s why people are and should be suspicious about filling out forms online.)

But I digress. My point is that there are two distinct classes of tools used to analyze website traffic. One set of tools works solely with web server log files that (mostly) use the extended log file format to analyze visitor activity data. Another set of tools use log files (in some cases) plus scripting, unique web addresses (URLs), cookies, and other techniques to present an even more complete picture of website activity.

The website log file-only tools include free tools like Analog, Webalizer, and awstats. The second set of tools includes WebTrends, LiveStats, Urchin, and NetTracker as well as hosted solutions such as SiteMeter, EcommStats, ClickTracks, and Conversion Ruler.

So which tool works best for a small or medium sized business on a budget? It depends upon the effectiveness of your current website. If you have little or no idea who visits your website, you should start with a combination of tools such as awstats, SiteMeter, and EcommStats. If you have a good handle on your site visitors but want to know more, then you should consider more sophisticated tools such as WebTrends, LiveStats, Urchin, or NetTracker plus services such as ClickTracks, ClickLab, Conversion Ruler, and IndexTools.

If you have no idea who visits your website, your website activity reporting should be designed to provide information about number of visitors, how they arrive at your site, what search terms they use to find you (and which search engines). No-cost and low-cost services like SiteMeter and EcommStats can provide a rich set of data to get started. SiteMeter is more stream of conciousness, showing you each visitor and letting you drill down to see details. EcommStats is more structured with the ability to see a list of frequently visited pages, search terms, referrers, and other useful data based on user defined date ranges. Neither displays unique visitors. However, if your goal is to get a basic understanding about how your website is used, that may not be a problem.

Both SiteMeter and EcommStats use javascript or html code to collect data. You place their code in each web page you want to track (easier to do than it sounds). This approach eliminates any site activity from search engine robots and, therefore, is more valuable. If you do not know, search engines like Google send out software on the internet that crawls websites, calling up page after page to scan and categorize content to display when people use that search engine. Using scripting to record website activity means you do not have to guess how much your visitor numbers and other data has been inflated by search engines.

For businesses with little insight into their current website activity, a combination of tools works best. Analog, Webalizer, or awstats are common to most hosting environments and they can tell you how much bandwidth you’re using, what search engines visit, and other data. However, use of services like EcommStats and SiteMeter will show you how human visitors work with your website, where they come from, what search engines and search terms they use, and other important data.

If your business has a good handle on this basic information over the past three to six months, then the next step up is to use more complex tools and services. You might, for example, wonder what makes visitors ignore an important link and click instead on a link of less importance to you. Or you might wonder exactly how many people click through to a sale page.

Because these questions can range widely, the tools to use vary widely. If you want to track your Google AdWords campaigns and which links people click on your website, for example, ClickTracks is worth investigation because their solution uses custom URLs to track activity so that you can see your website interface with clickthrough percentages next to each link. ClickTracks also includes the ability to incorporate your Google AdWords campaigns so that you can measure performance of text ads you buy for specific search keywords. However, the hosted version of ClickTracks costs $50-90/month USD with a three month minimum. So a small business on a budget might use ClickTracks once or twice a year for three months. For example, before you redesign your website it might help to know exactly what links visitors click on in your current design.

Other more complex tools worth investigation are non-hosted options like WebTrends, Urchin, LiveStats, and NetTracker. These tools use log files (configured in extended log file format) as well as cookies to track activity. Hosted options include ClickTracks, ClickLab, Conversion Ruler, HitsLink, Index Tools, among others.

The best way to investigate these tools is to first write down the questions you have about how visitors use your website. Then visit the vendor sites, sign up to try software at sites that look promising, and engage their sales people as you try out the service. Be brash and ask to talk with customers and not just the ones that like the product.

You also should search several websites to see what results other people have had with specific tools. For example, a search for “LiveStats” at Webmasterworld.com shows that, among other things, the latest version 6.0 does not display complete search terms, an important deficiency. Other sites to check are SitePoint and the newsgroups at Google. And you should search ClickZ, MarketingProfs, MarketingSherpa, and other marketing sites to see what they have to say about specific tools or services.

When you go through this process and become familiar with activity on your website, you will discover a mystery (or dirty secret, depending upon your point of view): statistics lie. Every analytics tool uses different methods to extract and calculate data. The result is small or large variances between information reported by one tool and another. The best way to protect yourself is to ask vendors how they calculate key metrics such as unique visitors and other metrics you need to answer your questions about visitor activity.

The bottom line for website activity reporting is that you need a foundation to start then tools to help you understand customer behavior in more depth. How your prospects and customers behave in the real world in your store or office might be different than how they behave on your website. Good analysis over time hinges on what tools your visitors use, what tasks they need or want to accomplish, what sort of people they are in terms of personal history and motivations and ambitions, and not least the context of the day and time they show up at your website.

So now you know all about “dirt bag soap” and “my dog mundi” and how they showed up at this website. The real conundrum for me is the search phrase, “my dog mundi.” I searched the top twenty links at Google in an effort to discover what the visitor had in mind that day. Maybe they were checking to see if they could use that name for a book title? One could imagine all sorts of scenarios. It does highlight the best part of website analytics, and perhaps life in general, exploring mysteries leads to insights and more mysteries to explore.

Disclaimer: While certainly no animals were harmed in writing this article, I should mention that I receive no money from any vendor mentioned here. You can trust the information here represents only my research and experience.

Resources Mentioned In This Article

Analog: http://www.analog.cx/
AWStats: http://awstats.sourceforge.net
ClickLab: http://www.clicklab.com
ClickTracks: http://www.clicktracks.com/hosted/
Conversion Ruler: http://www.conversionruler.com
EcommStats: http://www.ecommstats.com
HitsLink: http://www.hitslink.com
IndexTools: http://www.indextools.com
LiveStats: http://www.deepmetrix.com/log_analyzer/corporate/
NetTracker: http://www.sane.com/products/NetTracker/
SiteMeter: http://www.sitemeter.com/
Urchin: http://www.urchin.com/
Webalizer: http://www.mrunix.net/webalizer/
WebStat: http://www.webstat.com
WebTrendsLive: http://www.webtrendslive.com

ClickZ: http://www.clickz.com
Google Newsgroups: http://groups.google.com
MarketingProfs: http://www.marketingprofs.com
MarketingSherpa: http://www.marketingsherpa.com
SitePoint Forums: http://www.sitepointforums.com
Webmasterworld: http://www.webmasterworld.com

W3C Extended Log File Format: http://www.w3.org/TR/WD-logfile.html
Log File Formats Explained: http://www.bacuslabs.com/WsvlCLF.html
HTTP Response Codes: http://www.bacuslabs.com/WsvlHtResp.html

Dirt Bag Soap website: http://www.dirtbagsoap.com
Dirt Bag Soap weblog entry: http://www.reachcustomersonline.com/weblog/entries.php?id=P504_0_1_0_C

Related Stories

HitLinks & other hosted website statistics tracking and analysis tools

EcommStats Adds Click Paths

AWStats 5.4 web activity reporting software is available

IAnything Goes: How did you get here? Designing for visitors who don’t enter through the home page

How to Protect Your Website Images

One Hit Wonders: How To Manage Your Search Engine Traffic

How to turn a Static HTML page into a PHP page