Everything about Web and Network Monitoring

Home > Uptime Monitoring > Website Monitoring > HTTP extraction with Monitis

HTTP extraction with Monitis

HTTP what?


Yes, HTTP extraction. Imagine you have a web page you would like to probe for parameters. Such as the number of your twitter subscribers, or the temperature somewhere – and profile it in Monitis.
Or another scenario could be to probe the responsiveness of your website – how fast is it?
Was it compromised and defaced?
I think you get the point – it is needed and important.

Infrastructure

For a while I’m promoting M3 – Monitis Monitor Manager monitoring framework as the infrastructure for adding monitors and counters to Monitis.

M3 monitoring framework is a Perl based infrastructure for parameter extraction with template support.
In order to implement HTTP parameter extraction, just slight modifications were required to get M3 do also this job.

Different protocols

M3 currently supports three ways of parameter extraction:

  1. Regular Expressions
  2. JSON
  3. XML (providing an xpath)

Together with that, M3 monitoring framework can also provide you with various statistics regarding the responsiveness of the polled web page. Statistics such as delay, page size and more.
Authentication of web pages is also supported – in case you need to provide a user and password to access the page.

M3 Usage

The quick checkout snippet is here to help you, as usual:

 # mkdir -p /usr/share/monitisexchange && cd /usr/share/monitisexchange # git clone git://github.com/monitisexchange/Monitis-Linux-Scripts.git # cd Monitis-Linux-Scripts/M3 

For the self sufficient – the following files are sample files, intended to show you how M3 works with HTTP extraction:

For those of you who thrive for a live example, we’ll analyze the json extraction example:

 <?xml version="1.0"?> <config> <apicredentials apikey="XXX" secretkey="XXX"/> <agent name="Weather monitor" interval="5"> <monitor name="Weather in Sunnyvale"> <url>http://weather.yahooapis.com/forecastjson?w=2502265</url> <http_statistics>1</http_statistics> <metric name="Temperature"> <type>integer</type> <uom>F</uom> <json>{'forecast'}[0]->{'low_temperature'}</json> </metric> </monitor> </agent> </config> 

You’ll notice that we instruct M3 to fetch the URL  http://weather.yahooapis.com/forecastjson?w=2502265
Fetching it ourselves will show:

 # curl -s http://weather.yahooapis.com/forecastjson?w=2502265 { ... "forecast": [ { "day": "Today", "condition": "Partly Cloudy", "high_temperature": "75", "low_temperature": "57" }, { "day": "Tomorrow", "condition": "Partly Cloudy", "high_temperature": "79", "low_temperature": "59" } ] } 

Yes – I cheated a bit, I’ve used a JSON beautifier to beautify the output – because otherwise it’s a one ugly non-readable one-liner. I’ve also omitted some of the uninteresting output from the JSON call.
If you’ll follow the JSON path in the sample file – {‘forecast’}[0]->{‘low_temperature’} – you’ll see that it’ll bring us straight to the low_temperature of today’s in the JSON output.
Quite easy, isn’t it?

XML and regular expressions

Regular expressions were core functionality in M3 and there are also other examples showing the usage of them with M3.
XML xpath extraction is as simple as the JSON extraction. I’ll tell you even another secret – the implementation of xpath and JSON matching is the same!

HTTP statistics

In addition to that – M3 can also load some HTTP statistics like latency, page size and HTTP code. In order to enable it, simply have a look at config_sample_json_extraction.xml. You’ll see the <http_statistics> attributes over there – and this is what will add the statistics.

So… What now?

The snapshot provided in the top is graphing the temperature and humidity in Wanaka, New Zealand. Pretty neat, isn’t it?


What now? – Now you should look for parameters you’d like to extract from web pages, use M3 to load them to Monitis – and set the proper alerts for them.

Liked it? Sign up for Monitis and try it now! Don’t forget to check also monitisexchange – our git-hub open source repository for some more surprises.

About Dan Fruehauf

Web & Cloud
Monitoring