Squid Analyzer – A Parser for Squid proxy access.log File

Squid is one of the most popular proxy software in Linux world. It’s powerful, it has a lot of features and of course, it’s free. Squid has been used by ISP’s since the early 1990 to provide faster download speeds and reduce latency. For common proxy use, I believe squid is implemented in most companies around the world.

Squid provides access.log to record all user activities which through it. IT Administrator can parse the file to see what happens there. But access.log is a raw file. You really to read it carefully to get valuable information. Since access.log file is a raw file, a third party software is needed to process it into a human readable information.

There are a lot of squid-parser software out there. On the Squid-Cache website, there are more than 20 software for squid log-analysis. If you are confused of which is suitable for your needs, we will show you one of squid log-analysis software called Squid-Analyzer.

What is Squid-Analyzer

Squid Analyzer parses Squid proxy access log and reports general statistics about hits, bytes, users, networks, top URLs, and top second level domains. Statistic reports are oriented toward user and bandwidth control.

Installation

As an Analyzer, this application has a great graphical reports. You will be surprised if you compare Squid Analyzer with other applications. Let’s start to install it.

Download the source

The latest version of Squid Analyzer now is version 5.2. You can download it from here. After that, you can extract the source file.

tar zxcf squidanalyzer-5.2.tar.gz

Install the software

Squid Analyzer is depends on Perl software. You need to make sure that you have it on your system. To check it whether your system have Perl or not, type :

perl -v

You will see Perl version as the output. If you don’t see it, you need to install Perl first.
On Debian / Ubuntu based, you can use this command :

sudo apt-get install perl

On RedHat / CentOS based, you can use this command :

sudo yum install perl

Once the Perl is installed, then you can continue to install Squid Analyzer. Go to the folder where the Squid Analyzer file is stored. Assume that the Squid Analyzer is located in folder /home/squidanalyzer-5.2/ , then type this command to install it (you may need to be root):

/home/squidanalyzer-5.2/install_all.sh

Configuring the web server

Squid Analyzer is a web based tool. So we need to configure our web server to recognize it. If you are using Apache web server, here’s the minimal configuration to make Squid Analyzer run.

Alias /squidreport /var/www/squidanalyzer

Options -Indexes FollowSymLinks MultiViews
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.1

Don’t forget to restart the Apache web server to take effect.

Configuring Squid Analyzer

Squid Analyzer has a it’s own configuration file. By default, you don’t have to change anything. But if you want to custom it, the file is located in /etc/squidanalyzer/squidanalyzer.conf

Test it Running

If you found no error at installation stage, browse to http://myhost.com/squidreports
You will be taken to the front page of Squid Analyzer

Squid Analyzer Front page

Creating Cron Job

Squid log files are almost never stopping running. The log content will grow from time to time. To parse the log files, we need to run Squid Analyzer periodically. Cron is best tool to do it. Here’s a sample to run it for everyday at 02:00 AM

Crontab

Inside Squid Analyzer

Squid Analyzer front page is quite simple. You will see Global Statistics which grouped by Years. Click the Years you want to see. Then you will be taken to the Cache Statistics of that Year. Assume we will use year 2013.

There will be a calendar 2013 at the top right area. The months which are covered by Squid Analyzer can be clicked.

Calender month view

Let say we click on Oct month then you will see beautiful graphs about Daily Requests statistics and Daily Mega Bytes statistics on October. The calender at top right is also changed into a daily view.

Calendar daily view

Daily request statistique

Daily Megabyte cache

To view graphs in day-to-day view, click on the date on the calender.

The Menus

Squid Analyzer have 5 menus. There are Mime Types, Networks, Users, Top URLs, and Top Domains.

Squid Analyzer menu

Here are the details :

MIME-Types

This menu will only show you a list of MIME-Types which is recognized by Squid Analyzer. Just for information, from MIME is an Internet media type, a standard identifier used on Internet to indicate the type of data that a file contains.

MIME Types

Networks

As you can guess from it’s name, Networks will give you information about network statistics.

Network Statistics

It will give you detail information from IP Addresses to the accessed URL. The interesting part is you can select the IP Address there to view information more detail related to that IP Address. When you click it, Squid Analyzer will show you information about this :

  • A graph shows daily requests based on amount of hit and date
  • A graph shows daily megabyte transferred based on date
  • A list of IP Address in the same subnet who used the proxy
  • Daily request

    Daily  network Megabyte

    List IP Network

    Once again, you can click the list of IP Addresses there to view more details.

    Users

    On Users menu, you will see a list of users that accessed the proxy. If you see line such as “User Statistis on 2013-10” it’s mean that you are seeing a list of Users on October 2013.

    List of Users

    If your Squid proxy is using proxy-authentication such as LDAP, you will see some of users are displayed by name. You can also click at the IP Addresses or names on that list to view information more detail.

    Detail network request per IP

    Detail network megabyte per IP

    List of Users URL

    Top URLs

    You will see the most accessed URLs in a year or a specific month. If you hover your mouse cursor on top the URL, you will see a box pop-up. It contains the information about who access that URL and how many times the users access it.

    Top URL

    Quick view top URL

    If you click on the URL, you will be taken to it’s website.

    Top Domains

    Top Domains is similar with Top URLs. The difference is it just show you the domains. If you access http://www.youtube.com URL, Top Domains just display *.youtube.com

    Top Domain

    But in Top Domains, you will see pie-chart graphs.

    Top domain statistiques

    Top Domain hits

    Below those graphs, you will see a list of Top Domains. Similar with Top URL, you can also hover you mouse cursor on top the domains to see a quick view of Users and Count.

    Top Domain hover

    Conclusion

    Squid Analyzer may not the best Squid access.log parser. But it gives you a useful information about what happen in your proxy. And it’s Open Source and free. It got 4,9 stars of 5 stars from Sourceforge.net. You may explore Squid Analyzer in more detail from SquidAnalyzer website.

    Category: MONITORING, OPEN SOURCE TOOLS, OPENSOURCE

    Leave a Reply

    Commenting Policy:
    Promotion of your products ? Comment gets deleted.
    offtopic ? Comment gets deleted.
    All comments are subject to moderation.