Everything about Web and Network Monitoring

Home > Server Management > Sysadmin Tools > Apache & Monitis’ M3: The Perfect Match

Apache & Monitis’ M3: The Perfect Match

Apache is the most popular webserver, and M3 (Monitis Monitor Manager) is one very powerful Monitis tool. It’s a no-brainer to bring them together.

In this short document, first we will look briefly at how Apache presents its logs. Next we will define the way to measure the speed of a webserver, and finally we will learn how we can present results with Monitis (using M3).

ErrorLog

By default, an Apache HTTP server creates two logs, and usually both are written to different files. ErrorLog is considered more important. The server stores there information about all problems and irregularities, not only directly related to Apache, but also from web applications (i.e. written in PHP, Django) served by it. Below are a few lines taken from the log of a webpage written in Django (Technical Note:  The actual log lines are too long to be readable in the article  they were split by “\”)  :

[Wed Aug 11 22:40:13 2010] [error] [client 78.105.203.220] mod_wsgi (pid=6863): \
  Exception occurred within WSGI script '/opt/camtechnet/demo/apache/site.wsgi'., \
  referer: http://demo.camtechnet.info/news/tags/125/
[Wed Aug 11 22:40:13 2010] [error] [client 78.105.203.220] Traceback \
  (most recent call last):, referer: http://demo.camtechnet.info/news/tags/125/
[Wed Aug 11 22:40:13 2010] [error] [client 78.105.203.220]   File \
  "/opt/camtechnet/third_party/django/core/handlers/wsgi.py", \
  line 241, in __call__, referer: http://demo.camtechnet.info/news/tags/125/
[Wed Aug 11 22:40:13 2010] [error] [client 78.105.203.220]   \
  response = self.get_response(request), \
  referer: http://demo.camtechnet.info/news/tags/125/

From the above, you can see that the error log is relatively free-formatted, and there are no ways to control its format. You cannot control the format of output, but thanks to the LogLevel directive, you can define how much information we want webservers to pass to the log.

The log destination is defined by the ErrorLog directive. It can be saved to a file (on Debian /var/log/apache2/error_log or on RedHat /var/log/httpd/error_log), piped to an external program or passed to syslog-ng. (About ways to use syslog-ng check our blog).

AccessLog

AccessLog, second of the Apache standard logs, records all HTTP(s) requests processed by a web server. Its location is control by the CustomLog and it’s format by the  FormatLog directive.  In the Apache config file, you can specify more than one format, but each needs to have a distinguishing nickname. For example, in Debian, there are four predefined formats and, of course, we can specify even more. The format arguments which can be used to define a format are described in mod_log_config document. For example Ubuntu 10.04 LTS has the following format define:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Below an example of output defined by such a directive:

86.161.193.207 - - [29/Sep/2011:23:07:57 +0100] " \
  GET /static/companies/CambridgeUniversity______.gif HTTP/1.1" 200 3335 \
  "http://www.camtechnet.info/jobs/view/216099" "Mozilla/4.0 \
  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; \
  .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; \
  HPNTDF; .NET4.0C)"

There is an IP address that the request has come from (%h), but the logname (%l) and remote user (%u) are missing. Next, there are the time of request (%t), the first line of request in quotation (\”%r\”), status of the last request (%s>), size of response (%b), the content of Referer (\”%Referer}i\”), and User-Agent (\”%{User-Agent}i\”).

Designing Your Own Log

Now that you understand the way to control a format of the access log, we can design our own. Let say that we have a server with static content (images, css & js files) and we would like to learn how fast it sends that content. Of course we cannot make any analyses without data. To get them first somewhere in Apache config files, we add a new log format definition and later call out an example in one of a vhost’s configuration directive. In our example we want to have: information about the remote host, the request, time and date of operation, size and length of response and finally the response status code:

LogFormat “%h, %t, %B, %D, %r, %>s“  speed

It’s worth to note that in the above line, variables are separate by “,” characters. It makes data easier to process. For example we create a normal csv file which can later be imported to a spreadsheet. Another interesting thing: we use %B rather than %b. Both print the same variable (size of response), but differ in the way “0” is handle. The first one will throw ‘0’, the second ‘-’. The former in the column consists of integer numbers and are easier to process.

And a final reminder: Remember to define the file in which you want to keep your new log, for example:

CustomLog /var/log/apache2/filename.log speed

Server speed

So, we’ve learned something about the Apache logs. But how can we measure the performance of our webserver that features static content? A good indicator might be serving speed [byte/microsecond], defined as ratio of size and time (length of response) of each file. Obtaining such value from your newly defined file is relatively easy. For example, we can use awk:

awk -F’,’ ‘{printf “%d, %f, %d”,  $1, $4/$5, $6}’ filename.log

So this new variable can tell us something about the speed of our webserver. However, it probably won’t tell us too much straight away, because we will have too much data. A more meaningful number is the average serving speed. It won’t be so useful if we figure the average over the whole time that we gather data. During the course of the webserver life span, the type and purpose of data might as well be different. Such variation won’t be visible in one number. Moreover, you might notice a different pattern on a different weekday, or even at different times of the day or night. Therefore, the best tactic will be to make an average over a given period of time. It might be 1 hours, 5 minutes or even 1 minute. However, if we don’t, we’ll end up with many numbers that might be difficult to analyse.

On the other hand, it is well known that one good image is worth a thousand words; so we should somehow visualize our results. The good news is that this is possible with Monitis — thanks to its flexible Open API.  It will probably be easier to use M3. To start, of course we need to put M3 on our server. M3 is based on Monitis’ Perl module, so you need to have it installed as well. Another prerequisite is the secret API key, which you can get from your Monitis account.

Example

Finally, we can start to prepare the actual example. First we need to configure the XML config file. Available at github, ‘example.xml’ is a very good starting point. We need to change the agent, metric and monitor names, type and uom variables. But the most important changes are exectemplate and regex as well as API secret and key values. To keep our example simple, let’s  assume that we want to have an average speed of over one minute and that the log rotates at least once per day. Using those assumptions, we can use the following gawk one-liner:

gawk -F’,’ ‘BEGIN {ts=systime();  lastmin=strftime(“%R”,ts-60)} \
   $0 ~ lastmin {avg+=($3/$4); nr++} END {print avg/nr}’ \
   /var/log/apache2/filename.log

First, in the BEGIN part, we check current time (ts=systime()). Next, create a string with time in the following format: HH:MM (such a format is used in the Apache log) for previous minute (lastmin = strftime(“%R”,ts-60)). In the main part of the program, we use that string to obtain serving speed for files requested during the last minute — as well as the number of served files ($0 ~ lastmin {avg+=($3/$4); nr++}). Finally, in the END part, we print average speed (sum of all files speed divided by number of files {print avg/nr}).

Notice that we have addressed “exectemplate,” but what about “regex.” Our script prints only one value; so we need the whole string. Therefore our regex is extremely easy ^(\S+), so the whole xml file looks like:

<?xml version="1.0"?>
 <config>
  <apicredentials apikey="XXX" secretkey="XXX"/>
   <agent name="average serving speed monitor" interval="5">
    <monitor name="Average serving speed at %HOSTNAME%">
     <exectemplate>
       gawk -F’,’ ‘BEGIN {ts=systime();lastmin=strftime(“%R”,ts-60)} \
         $0 ~ lastmin {avg+=$3/$4} END {print $avg/$FNR}’\
         /var/log/apache2/filename.log
     </exectemplate>
     <metric name="Average serving speed [Bytes per Second]">
      <type>float</type>
      <uom>Bytes/sec</uom>
      <regex>^(\S+)</regex>
     </metric>
    </monitor>
   </agent>
</config>

Summary

Below is the graph you’ll see making a test of the above script.

The result graph obtained thanks to Monitis M3.

This is only a very simplified example, but its another reminder that there are many ways to use Monitis’ monitoring services. You can quite easily measure the total number of server files, the average size of files etc.

About Wawrzyniec