Unleashing the power of M3 & timers

During the lifespan of M3 (Monitis Monitor Manager) there has always been something lacking – timers.

M3 execution procedure was outlined in this previous article.

The execution mentioned in the latter was a one-time-execution, whereas server monitoring requires periodic invocation of monitors in order to actually provide counters over time, graphing performance.

The periodic invocation method suggested up until today was to integrate M3 with crontab.

Crontab, in a nutshell, is a Linux/Unix service for periodic invocation of executables. Implementing M3 with crontab properly meant M3 would run every X minutes, producing a Monitis counter update every X minutes, should everything run properly.

In the following article I’ll outline the changes done to support timers in M3.

Design & implementation

When assessing a programming task, I usually stick to the KISS simple – “Keep it simple, stupid!”.

I had a few ideas of how to implement timers in M3, eventually I went with the simplest one, yes, for sake of simplicity.

If we’ll go back to M3′s XML hierarchy, we’ll discover again that the top most level is an agent. Every agent can have a few monitors and every monitor can have a few counters (metrics).

The simplest idea I had in mind was to simply spawn every agent in a different thread.

Simply put, what M3 would do when invoked in timer mode is to spawn a thread for every agent defined in the XML. Every one of these new spawned threads will:

1. Run all the monitors of the agent and update their data in Monitis
2. Sleep for X seconds (where X is the agent interval defined)
3. Loop back

It’s true that if M3 is configured with dozens of agents, then dozens of threads would be spawned. If you are such a person using M3 with dozens of agents, please feel free to contact me and we’ll find a solution also for that.

Going to the beginning of the paragraph and mentioning dozens of spawned threads, I have to admit I was tempted to go with a single-threaded asynchronous design, but realized that implementing such a thing might take too long and cause too many bugs.

We at Monitis believe that monitoring your servers should be easy and never hamper performance. If you believe that the mentioned design of M3 might harm your performance, I’ll be more than happy to go forward with a single-threaded asynchronous implementation which would definitely yield better performance and a smaller memory footprint.

Usage

M3’s usage is still very simple. If you have M3 checked out, simply update it and you’ll find:

• TimerRun.pl
• TimerDryRun.pl

The former will invoke M3 in a loop, running monitors and updating counters. The latter will do exactly the same without updating counters.

Both executions can be terminated using ^C, which would terminate all threads cleanly. Be patient while terminating.

Invocation stayed the same. For the sake of simplicity we can use the etc_file_monitor.xml configuration file which is configured with an interval of 300 seconds (5 minutes). Invocation will be as follows:

 # TimerRun.pl etc_file_monitor.xml 

That’s it, go have lunch. M3 will do the rest for you!

Backward Compatibility

In case you wonder about backward compatibility or if you are just happy to run M3 via crontab – then worry no more. M3 functionality as a stand-alone single-execution behavior is still functional. Timers came as an addition and not as a replacement.

Enjoy it and remember that feedback is always more than welcome!

Integrating software into the powerful Monitis has never been easier, now that you have M3!

Monitis can monitor anything! Follow us also on Twitter and GitHub.

Pluggable M3 (Monitis Monitor Manager) Framework

Who needs an introduction about M3? – Perhaps no one!
After gaining some reputation with M3, providing extra-easy integration of any monitor into Monitis it was time to take it to the next level.

Generally speaking, the work flow of M3 was described in detail in this article.

After some thought and design, we’ve decided it’d be best if M3 was pluggable. Pluggable in terms of being able to easily add execution and parsing plugins.
The interface and behavior of M3 stayed exactly the same, however now it is much easier to obtain data from any source and parse the data the way you want it.
Saying that, it was time to put the new design for a test. We tried to integrate the DBI support into M3.
Guess what – it was much easier than expected!

The general idea of DB integration with Monitis

DB (as in Database) integration with Monitis has shown to be invaluable. Being able to extract application related data from a DB and load it to Monitis means you can graph anything going on in your database related application. Monitoring your DB is an integral part of server monitoring in today’s world.

Be it maximum number of users at peak hours, gathering statistics about record usage or just checking if your DB is serving requests quickly enough – graphing performance and uptime. Your imagination is the limit – considering your application is DB related of course.

How it’s done – the technicalities beyond

The sharp-eyed among you would notice that M3 got beefed up with a directory named “Execution” and a directory named “Parsing”.
These directories hold the plugins M3 uses for each.

All that I had to do to add DBI support is to create a new execution plugin called “DBI.pm” and fill it up with code from the Monitis DBI module which already existed.

And that my friends, that, was easy. Mission accomplished – Pluggable M3 eases the job for us.

Another way to integrate MySQL

MySQL could have been integrated previously into M3 by executing mysql and feeding it with a query, having something similar to:

 <exectemplate>echo "select count(*) from mysql.user;" | mysql</exectemplate> 

This form of execution will invoke the ‘mysql’ executable, feeding it the above query. Execution of this will result in the response of the query being returned with some other useless output.

Please avoid this form of execution, as now DBI support is fully integrated into M3. In the next paragraph I’ll show how it should be done.

Lets fetch

While conveying ideas about M3, I decided to prefer to provide very simple examples of operation, however, I encourage you, the end user to let your imagination go wild, adding interesting counters which show how your application really perform and behave.

In this example we’ll just count the number of users in an installed MySQL database. An example anyone should be able to imitate and then scale to his own need.

We’ll examine the following example. I suggest opening the mentioned link in a new window while continuing the read.
Looking at this example it is quite straight-forward to understand. The query to execute is over there, together with some other parameters which are mandatory to connect to a database, such as the database name, user name, password, etc.

An explanation should be provided however for the odd db_statistics parameter. Those of you who have read the HTTP extraction article about M3 would have probably remembered the http_statistics parameter. Then, the db_statistics one does more or less the same, providing some meta counters for the query:

1. How long did the query take to process (in seconds)
2. The success/failure of the query (1 for success, 0 for failure)

I encourage to do the parsing of query results, or actually the lack-of-parsing of query results in a way that takes the whole output returns from the query, hence the regular expression of (.*).

If your query returns more than what you need, please tune your query better.
Test your execution with:

 # ./DryRun.pl dbi_sample.xml DB: 'root@DBI:mysql:mysql:localhost', Query: 'select count(*) from mysql.user;' at MonitisMonitorManager.pm line 223 Matched '(.*)'=>'6' in '6' at MonitisMonitorManager.pm line 237 OK This is a dry run, data for monitor 'Sample MySQL Monitor' was not really updated. at ./DryRun.pl line 18 

It should work. Then you can work your query to fit your own needs.

Next in line

We’re looking for some more new ideas to add as plugins for M3. If you have any idea about any plugin (execution or parsing plugin) that you would like to see in M3 – we want to hear you!

Integrating software into the powerful Monitis has never been easier, now that you have M3!

Monitis can monitor anything! Follow us also on Twitter and GitHub.

The Internet is a great resource, but it is sometimes too much to wade through hundreds of web pages looking for tips about using Tomcat with a database. So we at Monitis have done it for you! Below are some of the more useful tips found in various places on the Internet.

Tip #1 – Don’t Ignore Database Performance

Webapps typically use databases. So, you have to tune the webapp along with the database if you want optimal performance. Take our advice. Performance test the database and define an appropriate database connection pool (maxActive, maxIdle, and maxWait in the Resource element).

Tip #2 – Determine the Cause of Database Connection Timeouts

Simply increasing maxWait in the resource element may not be the best way to deal with database connection timeouts. The timeouts may be the result of garbage collection.

Garbage collection requires that all running processes in the Java virtual machine be temporarily suspended. If garbage collection takes too long, the database connections will time out. The timeout here is not because of some problem in the application. It is simply because garbage collection is taking too long.

The more frequently garbage is collected, the less garbage there will be to collect each time. Configure garbage collection so it does not take more than one second.

Tip #3 – Share Database Connections

If every application opens and closes its own database connections, all applications run slower. Instead, if applications share connections by returning them to a common pool rather than closing them, all applications run faster.

Tomcat can be configured for this type of connection pooling. Simply set maxActive and maxIdle to appropriate values in the resource element.

Tip #4 – Return Database Connections to the Pool Properly

Do not close database connections twice. When using connection pooling, the first time you close you return the connection to the pool, and the second time you close, you could inadvertently close the connection after another application has started using it. If you are experiencing “Connection Closed” exceptions, this may be the cause. So, don’t do that!

Tip #5 – Configure Your Database as a Global Naming Resource or For a Single Context

Identify the Java Database Connectivity (JDBC) database in a resource element in the context.xml file if you want the resource to be available to one application only. Identify the JDBC resource in a resource element in the server.xml file if you want the resource to be available to all applications.

If multiple applications need access to a database, the database resource can be created globally in the server.xml file or each application can create its own resource in its own context.xml file. You need to decide which is best.

Making databases available to one application only (in context.xml) creates the resource when the application starts. Making databases available globally (in server.xml) creates the resource when the server starts, which means restarting the server every time you make changes to the resource configuration.

Configuring the database resource in the context.xml file once for each application is recommended because it makes the application more portable.

If different applications have different access levels and those access levels are determined by the database user id, then the database resource must be configured in each application’s context.xml file.

Tip #6 – Do Not Use the JDBC-ODBC Bridge

The JDBC-ODBC (Open Database Connectivity) Bridge bundled with Tomcat’s Java Development Kit (JDK) was never intended for a production environment. Use a third-party driver instead.

Using Tomcat with a database is easy. All you have to know are some good tips to keep your database performing trouble-free. Monitis is happy to provide you with this knowledge and advice. Read more about Tomcat here!

More about Apache Tomcat performance optimization tips here.

Active connections: 291
server accepts handled requests
16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106

Figure 1: Monitis shows a nice graph for NGINX Stub Staus.

Figure 2: Monitis provides a cleaner output for NGINX Server Stats.

In a nutshell, the example here simply fetch your NGINX stats, and send them to your Monitis account periodically:

1. Install NGINX stub_status module.

2. Edit your nginx.conf.

   Here we use port 80, and listen on localhost with address of nginx_status.
   server {
          listen               80;
          server_name   localhost;
          location /nginx_status {
          stub_status         on;
          access_log         off;
          allow        127.0.0.1;
          deny               all;
          }
   }
   That is, with the configuration above, you can view the original
   NGINX stats @ http://localhost/nginx_status

3. Start your NGINX Server.

4. Download the code and run the program.

   On your windows command line, run the program like this:

   monitis_nigix_stats.exe "[apikey]" "[secretkey]"
   "[monitorname]" "[minitortag]" "[nginx stub status url]" 

   For example, assume you already have a Monitis account:

   monitis_nginx_stats.exe "14NAC40PIMSUEEBQFJOQL18T5U"  
   "74H6U7A2DG71JU80QR48FEOPAL"  "Nginx  Monitor"
   "Nginx+Stub+Status" "http://localhost/nginx_status"

5. Log in to your Monitis account and start monitoring.

   You can see a new custom monitor ready to be added.
   (Monitors->Manage Monitors->Custom Monitors).
   Check and add your new monitor to your dashboard.

  

Conclusion:

In this article, we demonstrate an enterprise solution in NGINX monitoring. In five steps, Monitis is able to organize the statistics into columns; you can also track data continuously with a nice graph. Furthermore, you can add as many custom monitors as you like, all in your Monitis dashboard. Get your free registration here, and monitor your NGINX servers effectively.

Happy NGINX monitoring!

The Monitis Team
—

External Scripts:

NGINX Monitoring in C#
https://github.com/monitisexchange/Monitis-Linux-Scripts/tree/master/nginx/csharp

NGINX Monitoring in Python
https://github.com/monitisexchange/Monitis-Linux-Scripts/tree/master/nginx/python

External Links for further information:
http://monitis.com/api/api.html#addCustomMonitor
https://www.monitis.com/free_signup.jsp

Monitis is pleased to present the following tips for configuring Apache Tomcat for maximum performance. Apache Tomcat is an open source software implementation of the Java Servlet and JavaServer Pages technologies. Apache Tomcat powers numerous large-scale, mission-critical web applications across a diverse range of industries and organizations.

Tip #1 – Watch Out for White Space

When installing Tomcat under Windows, the path to the installation directory should not contain white space. The default (c:/Program Files) is not acceptable, because it contains a white space character. Installing under “C:/ProgramFiles” is acceptable because this path does not contain any white space characters.

Tip #2 – Check The System Variables

Regardless of the operating system that Tomcat is installed under, the installer should check the system variables to make sure they are set correctly. In particular, CATALINA_HOME should be set to Tomcat’s installation directory.

Tip #3 – Set Username and Password for Tomcat Manager

If you are using the Tomcat Manager, set a username and password. Set this username’s role to “manager,tomcat”.

Tip #4 – Read the Log Files

By default, Tomcat’s log files are stored in the CATALINA_HOME/log directory. Use them to solve a myriad of problems.

Tip #5 – Configuring the Admin Web Application

If you are using the admin webapp that comes with Tomcat, make sure the path specified in the docBase attribute of the Context element is absolute, not relative. The admin context can be specified in admin.xml (for auto deployment) or server.xml.

If you are not using the admin webapp, delete admin.xml.

Tip #6 – The Admin Webapp vs. the Manager Webapp

The manager webapp that comes with Tomcat is a scaled-down, simplified version of the admin webapp. The Manager webapp lets you install new webapps on a non-persistent basis for testing, and also allows you to stop, reload, remove, or undeploy webapps. For better security, use manager rather than admin wherever possible.

Tip #7 – Two Options For Deploying a WAR File

If you have a WAR file, you can deploy it by simply copying it into CATALINA_BASE/webapps. When Tomcat notices the new WAR file sitting there, it will unpack the file into the appropriate subdirectory and create a context for it. The context it creates is defined in the DefaultContext element in server.xml.

If it is not appropriate to use the default context, create a context fragment for the webapp, put it in a file named x.xml (where x is the name of your webapp), and put the file into CATALINA_BASE/webapps. The webapp itself can then be stored anywhere on the filesystem.

In either case, you can deploy your webapp without editing server.xml and without restarting Tomcat. One caveat: Do not turn off liveDeploy.

Tip #8 – Configure Virtual Hosts in server.xml

Your test environment can be set up as a virtual host by adding a host element to the engine element in server.xml. Don’t forget to create the appropriate DNS entry, too.

Tip #9 – When Usage Increases, Balance the Load

Tomcat provides several load balancing techniques that allow requests to be spread across multiple servers. See also the following link: http://people.apache.org/~markt/presentations/2009-04-01-TomcatTuning.pdf

Tip #10 – Implement Custom Error Pages

Default error pages do not give the end users useful information. Create your own error pages and give the end users meaningful messages.

Tip #11 – Use a Logging Toolkit

Eliminate System.err and System.out from application code. Use a logging toolkit (e.g., Log4J) instead.

Tip #12 – Create Users to Access the Manager Webapp

If you plan to use the manager webapp, the default configuration won’t help you much because there are no users with the manager role. Add the manager role and at least one user who has the manager role in CATALINA_HOME/conf/tomcat-users.xml.

Tip #13 – Filter by IP or Host

If only certain machines are allowed to access your webapp, create a valve to let only those machines in. Specify the machines by IP address or hostname.

Tip #14 – Strip Down server.xml

Remove comments, connectors, and any other components you don’t need from server.xml to make the file more readable. However, since comments can be important, keep a copy of the original file around for future reference.

Remove

<Connector port=”8009″ enableLookups=”false” redirectPort=”8443″ protocol=”AJP/1.3″></Connector>

from server.xml if you are using standalone Tomcat (i.e., without the Apache server).

Tip #15 – Thread Pool

If using multiple CPUs, raising the thread pool above the default 250 may prove beneficial. On the other hand, reducing the thread count may be beneficial if your server is running slowly.

Tip #16 – Compress Network Traffic

Use the compression and compressableMimeType attributes of the connector element to configure GZIP compression and to specify the MIME types that should be compressed.

Tip #17 – Production Logging

Tomcat’s default configuration creates duplicate log files. Remove duplicate logging in .handlers (in logging.properties).

Tip #18 – Rotate The Tomcat Log Files

This example rotates log files. Adjust the values to suit your environment.

1catalina.java.util.logging.FileHandler.pattern = ${catalina.base}/logs/catalina.%g.log

1catalina.java.util.logging.FileHandler.limit = 20000000

1catalina.java.util.logging.FileHandler.count = 5

Tip #19 – Use Clustering With Load Balancing

When balancing the load between multiple Tomcat instances, if an instance fails, another instance can seamlessly take over its sessions if clustering is configured. Without clustering, a failing instance’s sessions will crash.

Tip #20 – Disable the Shutdown Port

Turn off Tomcat’s default shutdown port by setting the port attribute to -1 in the server element. This makes it impossible to shut down Tomcat via IP:port. Shutdown will only be possible by a kill command issued by the user who owns the Tomcat instance.

Tip #21 – Ignore Interfaces That Are Not Required

Connectors listen to all interfaces by default. Use the address attribute of the connector element to force the connector to ignore all interfaces that are not used by your webapp.

See also 18 Java Tomcat Application Optimization Tips blog post.

Monitis offers real browser load testing and synthetic end-user monitoring services to ensure the website are operable 24/7. Without such monitoring, the investment in implementing an online application will not only bring little revenue but may also affect the overall company’s brand image and reputation.

For more information on Monitis please visit our website: www.monitis.com or signup for 15-days free trial at https://www.monitis.com/free_signup.jsp.

 In this article we’ll discuss creating a command line interface to query the Monitis Agents and Monitors you have activated for your systems. We’ll provide the source code for the utility in VBScript and we will also be working on a C# version.  We’ll keep things simple for now and start with VBScript and provide a link to the complete source at the end of this article.

To get started we need to identify what information we want to display from our Monitors, what can be easily supported, and what is not supported.  The main parts are the different types of agents that Monitis offers, which are the Internal,
External, and custom monitors.

In this article we’ll focus on the basic ‘framework’ of the utility that deals with the definition of some classes to make working with the various agents and monitors easier, accepting and processing command-line arguments, and a small function library to keep our main script very clean. We’ll introduce a neat little feature of VBScript showing you how to work with include files.

Let’s take a look at the functionality and output we like to get from Monitis-Top.

Querying Internal Monitors

We want to be able to display the latest performance counters from our internal, external, and custom monitors by entering commands like:

MonitisTop /cmd:listagents /type:internal /monitor:all

Which would return the performance counters for all
supported monitors of the internal agents, or

MonitisTop /cmd:listagents /type:internal /monitor:cpu,memory,drive

This returns the Cpu, Memory, and Drive monitors for the internal agents.

Querying Process Monitors

We can enter query the process monitors that we’ve configured in our Monitis dashboard like so:

MonitisTop /cmd:listagents /type:int  /process:outlook,search

This will return any process monitors containing the phrases ‘outlook’ and ‘search’. MonitisTop will automatically wildcard any processes listed on the command line.

Querying External Monitors

Monitis Top can also be used to query external monitors. You can specify the parameter /type:ext or /type:all, where ext will only return the external monitors and all returns both internal and external monitor results. For example, the command:

MonitisTop /cmd:listagents /type:all /monitor:cpu

will return both the internal CPU monitor result and the external monitors.

The results will show by each location that you have specified for a monitor.

Managing API and Secret Keys

We want similar commands for the external and custom agents.  In addition, it would also be nice if there was some functionality to maintain our APIKeys, so we’ll add a function that allows you to set and display these keys by entering:

MonitisTop /cmd:setkeys  /apikey:<yourapikey> /secretkey:<yoursecretkey>

Similar, you can display the keys with the command:

 MonitisTop /cmd:getkeys

VBScript Include Files

We’ll use VBScript to program the code for MonitisTop  initially. In the next phase of this project we’ll use C#. If you’re not  familiar with the possibility of using include files in VBScript, take a look  at the code snippet below.

Sub Include(strFilename)
	On Error Resume Next
	Dim oFSO, f, s

	Set oFSO = CreateObject("Scripting.FileSystemObject")
	If oFSO.FileExists(strFilename) Then
		Set f = oFSO.OpenTextFile(strFilename)
		s = f.ReadAll
		f.Close
		ExecuteGlobal s
	End If

	Set oFSO = Nothing
	Set f = Nothing
	On Error Goto 0
End Sub

Class Definitions

To make working with the various agents and monitors more manageable, we’ll create a few classes that we use to store the performance counter results we get back from the Monitis server. To start we’ll create a few classes; one for the agents, one for the monitors belonging to an agent, and one for the various metrics and values in a monitor. We want the Agent class to maintain a list of Monitors, and the Monitor class in its turn maintains a list of Metrics.

Class class_Metric
	Public Name
	Public Metric
	Public Result
	Public Suffix
	Public Width

	Private Sub Class_Initialize
		Name = ""
		Metric = ""
		Result = ""
		Suffix = ""
		Width = 0
	End Sub
End Class


Class class_Monitor
	Public Id
	Public Name

	Public MetricList

	Private Sub Class_Initialize
		Id = ""
		Name = ""
		Set MetricList = CreateObject("Scripting.Dictionary")
	End Sub

	Private Sub Class_Terminate
		Set MetricList = Nothing
	End Sub
End Class


Class class_InternalAgent
	Public Id
	Public Name
	Public MonitorList

	Private Sub Class_Initialize
		Set MonitorList = CreateObject("Scripting.Dictionary")
	End Sub

	Private Sub Class_Terminate
		Set MonitorList = Nothing
	End Sub
End Class

In the class_InternalAgent we create a list of monitor objects that is initialized in the constructor of the class. Notice that in VBScript the methods Class_Initialize and Class_Terminate are automatically called if they are defined and these methods compare to your C# constructor and Dispose methods.

Retrieving the Internal Agents

Once we have parsed the list of monitors that we want to show (specified on the command line), we can call the function GetInternalAgents. In the function below aObjHttp, aShowMonitors, and aShowProcesses  are input values, while aObjAgents is an output value. It returns the list of internal agents retrieved from the API.

Function GetInternalAgents(aObjHttp, aObjAgents, aShowMonitors, aShowProcesses)
	Dim objAgents, xmlAgents
	Set objAgents = CreateObject("Scripting.Dictionary")

	'Retrieve the list of agents
	url = "http://www.monitis.com/api?action=agents&apikey=" + apiKey + "&output=xml"
	aObjHttp.open "GET", url, False
	aObjHttp.send

	'Parse response
	Set xmlAgents = CreateObject("Microsoft.XMLDOM")
	xmlAgents.async = False
	xmlAgents.LoadXML(aObjHttp.responseText)
	If xmlAgents.parseError.errorCode <> 0 Then    
		wscript.Echo xmlAgents.parseError.errorCode    
		wscript.Echo xmlAgents.parseError.reason    
		wscript.Echo xmlAgents.parseError.line
	End If  	


	'Retrieve the agent information for each agent
	For Each agent in xmlAgents.documentElement.childnodes

		'Create an InternalAgent Object	
		Set IntAgent = New class_InternalAgent
		IntAgent.Id = agent.selectSingleNode("id").text
		IntAgent.Name = agent.selectSingleNode("key").text

		'Retrieve the agent information 
		url = "http://www.monitis.com/api?action=agentInfo&apikey=" + apiKey + "&output=xml&agentId=" + IntAgent.Id + "&loadTests=true"
		aObjHttp.open "GET", url, False
		aObjHttp.send

		Set xmlAgentInfo = CreateObject("Microsoft.XMLDOM")
		xmlAgentInfo.async = False
		xmlAgentInfo.LoadXML(aObjHttp.responseText)

		For Each info in xmlAgentInfo.documentElement.childnodes

			If aShowProcesses.Count = 0 Then
				GetGlobalMonitors aObjHttp, info.childnodes, IntAgent, aShowMonitors
			Else
				If LCase(info.nodename) = "processes" Then
					GetProcessMonitors aObjHttp, info.childnodes, IntAgent, aShowProcesses
				End If
			End If

		Next

		'Add the agent object to the list of agents
		aObjAgents.Add IntAgent.Id, IntAgent
	Next

End Function

The main part of this function is the creation of the Agent class, retrieving the ‘loadTests’ from the API and cycling through the list of returned Monitors for each Agent.

Because we want to build in support for querying process monitors as well as global drive, memory, and cpu monitors, we’ve included the list ‘aShowProcesses’. This list contains the processes entered on the command line that we want to query. If
this list contains zero elements, the program defaults to collecting and returning the global monitors.

After GetGlobalMonitors (or GetAgentProcessMonitors) have returned, we’ll add the Internal Agent object to aObjAgents; the list of Internal Agents.

In the GetAgentMonitors function we’ll collect all the monitors of an agent and make a call to GetResults to retrieve all the metric
values.

Function GetGlobalMonitors(aObjHttp, oNames, aObjAgent, aShowMonitors)
	Dim oName
  
	For Each oName in oNames

    	If oName.NodeName <> "#text" And oName.NodeName <> "process" And _
		  SupportedMonitors.Exists(LCase(oName.NodeName)) And _
		  aShowMonitors.Exists(LCase(oName.NodeName)) Then
    
    		' Create new monitor object
			Set Monitor = New class_Monitor
			Monitor.Name = GetMonitorName(oName.selectSingleNode("name").text)
		    strMonitorName = LCase(GetMonitorBaseName(Monitor.Name))
			Monitor.Id = UCase(oName.selectSingleNode("id").text)
			Monitor.DisplayName = UCase(Monitor.Name)

			url = "http://www.monitis.com/api?version=2&apikey=" & apikey & "&output=xml&action=top" & strMonitorName & "&limit=50&detailedResults=true"

			aObjHttp.open "GET", url, False
			aObjHttp.send
			Set oRes = CreateObject("Microsoft.XMLDOM")
			oRes.async = False
			oRes.LoadXML(aObjHttp.responseText)

			Set oNode = oRes.selectSingleNode("data/tests")
			If Not oNode Is Nothing Then
				For Each oCell In oNode.childnodes
					GetResult oCell, Monitor, SupportedMonitors(strMonitorName)
				Next
				aObjAgent.MonitorList.Add Monitor.Name, Monitor
			End If
		End If 

	Next  
End Function

In GetGlobalMonitors we create a Monitor object for each required monitor we want to return and call the function GetResult to retrieve the metric values for each monitor.

Function GetProcessMonitors(aObjHttp, oNames, aObjAgent, aShowProcesses)

	url = "http://www.monitis.com/api?version=2&apikey=" & apikey & "&output=xml&action=topProcessByCPUUsage&limit=50&detailedResults=true"

	aObjHttp.open "GET", url, False
	aObjHttp.send
	Set oRes = CreateObject("Microsoft.XMLDOM")
	oRes.async = False
	oRes.LoadXML(aObjHttp.responseText)

	Set oNode = oRes.selectSingleNode("data/tests")
	For Each t In oNode.childnodes

		Set oTest = t.selectSingleNode("testName")
		If Not oTest Is Nothing Then

			If ShowThisProcess(oTest.text, aShowProcesses) Then

				'Create a new monitor object
				Set Monitor = New class_Monitor
				Monitor.Id = t.selectSingleNode("id").text
				Monitor.Name = t.selectSingleNode("testName").text
				Monitor.DisplayName = GetMonitorName(t.selectSingleNode("testName").text)

				'Retrieve the results for this monitor
				GetResult t, Monitor, SupportedMonitors.Item("process")

				'Add the monitor to the list of agents
				aObjAgent.MonitorList.Add Monitor.Id, Monitor

			End If

		End If

	Next

End Function

'-------------------------------------------------------------------

Function ShowThisProcess(aProcess, aShowProcesses)
	ShowThisProcess = False
	For Each sp In aShowProcesses
		If (InStr(LCase(aProcess), LCase(sp)) > 0) Or (sp = "all") Then
			ShowThisProcess = True
		End If
	Next
End Function

'-------------------------------------------------------------------

Function GetResult(aNode, aMonitor, aFields)
	arrValues = Split(aFields, "|")
	For Each value In arrValues 

		'Split each value in the API field name and the suffix string 
		arrDetails = Split(value,",")
		strValue = arrDetails(0)
		strSuffix = arrDetails(1)

		'Retrieve the result for the given counter
		Set t = aNode.selectSingleNode(strValue)
		If Not t Is Nothing Then

			'Create a new metric object
			Set Metric = New class_Metric
			Metric.Name = strValue
			Metric.Result = t.text & strSuffix

			'Add the metric object to the current monitor object
		 	aMonitor.MetricList.Add Metric.Name, Metric
		End If

	Next
End Function

The GetResult function loops through all the result values of a monitor, creates a Metric object for each monitor and adds it to the Monitor object, and finally the function ShowInternalAgents writes all results to the screen.

Sub ShowInternalAgentsMonitors	
	'Write the list of agents to the screen	
	For Each agent In InternalAgents.Items
		WScript.Echo "" 
		WScript.Echo "-------------------------------------------------------------------------------"
		WScript.Echo " AGENT: " & agent.Name
		WScript.Echo "-------------------------------------------------------------------------------"

		strHeader = ""
		strRow = ""

		For Each monitor In agent.MonitorList.Items

			For Each objMetric In Monitor.MetricList.Items
				strHeader = strHeader & format(objMetric.Name, objMetric.Width)
				strRow = strRow & format(objMetric.Result & objMetric.Suffix, 
				objMetric.Width)
			Next

		Next

		WScript.Echo strHeader
		WScript.Echo strRow
	Next
End Sub

The function GetExternalMonitors gathers all the external monitors, the locations, and the monitor results.

Function GetExternalMonitors(aObjHttp, aObjAgents, aShowMonitors)
	Dim objAgents, xmlAgents
	Set objAgents = CreateObject("Scripting.Dictionary")

	'Create the dummy Agent object
	Set Agent = New class_InternalAgent
	Agent.Id = ""
	Agent.Name = "EXTERNAL"

	WScript.Echo "Acquiring external monitors..."

	'Retrieve the list of agents
	url = "http://www.monitis.com/api?apikey=" + apiKey + "&output=xml&version=2&action=tests"
	aObjHttp.open "GET", url, False
	aObjHttp.send

	'Parse response
	Set xmlMonitors = CreateObject("Microsoft.XMLDOM")
	xmlMonitors.async = False
	xmlMonitors.LoadXML(aObjHttp.responseText)
	If xmlMonitors.parseError.errorCode <> 0 Then    
		wscript.Echo xmlMonitors.parseError.errorCode    
		wscript.Echo xmlMonitors.parseError.reason    
		wscript.Echo xmlMonitors.parseError.line
	End If  	


	'Retrieve the agent information for each agent
	For Each mon in xmlMonitors.documentElement.childnodes

		'Create a Monitor Object	
		Set Monitor = New class_Monitor
		Monitor.Id = mon.Attributes.getNamedItem("id").text
		Monitor.Name = mon.text
		Monitor.DisplayName = mon.text

		dt = DateSerial(year(now), month(now), day(now))
		url = "http://www.monitis.com/api?action=testresult&apikey=" + apiKey + "&output=xml&testId=" & Monitor.Id + "&timezone=" & timezone & "&day=" & day(dt) & "&month=" & month(dt) & "&year=" & year(dt)
		aObjHttp.open "GET", url, False
		aObjHttp.send

		Set oRes = CreateObject("Microsoft.XMLDOM")
		oRes.async = False
		oRes.LoadXML(aObjHttp.responseText)

		'Loop throught the locations
		For Each t In oRes.firstchild.childnodes
			If t.nodeName = "location" Then

				Set Metric = New class_Metric
				Metric.Name = t.Attributes.GetNamedItem("name").text

				'Loop through the results for the current location
				For Each cell In t.firstChild.childnodes
					If LCase(cell.nodename) = "cell" Then
						Metric.Result = cell.text	
					End If
				Next

				'Add the Metric object to the Monitor List
				Monitor.MetricList.Add Metric.Name, Metric
			End If
		Next

		'Add the agent object to the list of agents
		Agent.MonitorList.Add Monitor.Id, Monitor
	Next


	aObjAgents.Add Agent.Id, Agent
End Function

The complete code of this article is available for download at GitHub.

Testing Web Content with Monitis, Excel, and PowerShell

In the last few articles, we’ve shown you how easy it is to monitor anything with the open source Monitis PowerShell Module.  Last time, we covered how to interact with web applications, and before that, you got a small taste of how working in a real scripting language like PowerShell can make it miles easier to add batches of monitors.

Today, we’re going to cover two more parameters of Add-MonitisExternalMonitor that makes it easy to build web content, and then we’re going to build a really simple way to make use an excel spreadsheet to create all of your web monitors in PowerShell.

The two new parameters you’ll want to learn are:

• ExpectedContentPattern – A regular expression you expect to find in the results
• RequiredResponseTime – the maximum amount of time it should take to get a response

To start out with, create an excel spreadsheet, and make a few columns:

• Url
• MonitorType
• OperationType
• Postdata
• ExpectedContentPattern
• RequiredResponseTime

Now, row by row, fill in things you’d like to test.  These are going to be parameters of Add-MonitisExternalMonitor.  Only Url, MonitorType, and OperationType are required.  OperationType can be “Get”, “Post”, “Put” or “Delete”.   MonitorType can be any number of internet protocols, but you’ll probably use ping, http, and https.

Once you’ve filled in the spreadsheet, save it as a CSV.

At this point, you can use this CSV as a way to quickly recreate all of your monitors in two lines:

Connect-Monitis -ApiKey $apiKey -SecretKey $secretKey

Import-Csv .\MyWebTest.csv | Add-MonitisExternalMonitor

The PowerShell trick that makes this work is the object pipeline.  We talked some about the pipeline a couple of days ago, when we introduced how to manage your monitors.  Almost all of the commands in the Monitis PowerShell module are aware of the object pipeline, and that means that when Import-CSV returns back a bunch of objects, those objects just work with Monitis.

This simple example should start to really give you a taste of what Monitis and PowerShell can do together, but we’re really just getting started.   Next time, we’re going to start looking at the coolest part of Monitis, custom monitors, and show you how you can use PowerShell to monitor absolutely anything with Monitis.

See also

Monitor Everything with Monitis – And do it easily with PowerShell – Part 1

Monitor Everything with Monitis – And do it easily with PowerShell – Part 2

Monitor Everything with Monitis – And do it easily with PowerShell – Part 3

Monitor Everything with Monitis – And do it easily with PowerShell – Part 4

Monitoring Web Applications with Monitis

We’ve been learning about a lot of cool things we can do with the Monitis PowerShell Module, which is built on top of the Monitis REST API.  Yesterday, we saw how we can use a simple script to monitor as many domains as we want.  The day before that, we saw how we can seamlessly manage all of our monitors.  Today, we’re going to cover monitoring web applications.

As a reminder, start out by running PowerShell, and running the command:  Import-Module Monitis.  Then use the Connect-Monitis command and your APIKey and SecretKey to connect.  From then on in, you’ll reuse your keep for every operation.

To see how you can use Monitis to help you monitor your web applications, run:

Get-Help Add-MonitisExternalMonitor –parameter PostData

In PowerShell, every command and parameter has built in help and examples.  You can use this to explore the possibilities of what you can do with the command right on your desktop.

PostData sends additional data for the web request, and from this, you can test your web applications.  For instance, if you have a search page, you can use –PostData to send the appropriate information to the page.   Another parameter, -Credential allows you to store a credential for the request, and another –OverSsl, will force the request to occur over SSL.

By making the monitor use postdata, instead of just ping your site, you can ensure that your web sites  and services are properly functioning.   Here’s a quick example of a monitor for a custom search engine:

Add-MonitisExternalMonitor -Name "SearchEngineMonitor" -Tag "MySites" -MonitorType http -PostData "q=hello+world" -url "a.searchengine.com"

If it was a user update page, you might try something like:

Add-MonitisExternalMonitor -Name "UserUpdateMonitor" -Tag "MySites" -MonitorType http -OperationType Post –Credential (Get-Credential) -OverSsl -Url "users.mydomain.com/update.aspx" -PostData "newPhoneNumber=8675309"

Monitis keeps track of how long it takes to complete the request from several locations, so you can keep track of how your web application responds from anywhere within the world.

Today, we’ve learned how easy it is to monitor web applications with Monitis.  Next, we’ll talk about testing web content, and how you can use a nifty little command in PowerShell called Import-CSV to easily create lists of tests for your web application, and then keep track of how they perform in Monitis.

See also

Monitor Everything with Monitis – And do it easily with PowerShell – Part 1

Monitor Everything with Monitis – And do it easily with PowerShell – Part 2

Monitor Everything with Monitis – And do it easily with PowerShell – Part 3

Mining External Monitor Results with Monitis and PowerShell

In the last article, we walked thru some of the other commands related to external monitors.  Today, we’ll focus on the one you’ll probably use the most:  Get-MonitisMonitorResult.

Before we get started, please remember to download the Monitis module from Monitis’ github open source repository.  Download it to Documents\WindowsPowerShell\modules, and import it in PowerShell by running Import-Module Monitis.

Get-MonitisMonitorResult gives you results from any type of Monitor.  You use it by piping in the monitors you want.

For instance, this will show todays results from the current monitor:

Get-MonitisExternalMonitor | Select-Object -First 1 | Get-MonitisMonitorResult

for example:

Like everything else in the Monitis module, you have a few  different options when you get results.  Instead of just getting results for today, you can use the –Last parameter to get results for the last number 3 days:

Get-MonitisExternalMonitor |

    Select-Object -First 1 |

    Get-MonitisMonitorResult -Last (New-TimeSpan -Days 3)

You can also get results since a date

Get-MonitisExternalMonitor |

    Select-Object -First 1 |

    Get-MonitisMonitorResult -Since 9/1/2011

Or for a range of dates

Get-MonitisExternalMonitor |

    Select-Object -First 1 |

    Get-MonitisMonitorResult -Start 9/1/2011 –End 9/10/2011

By default you see a summary object from each location.  This tells you the location, and a number of different subproperties tell you what’s going on.

• Data contains each result
• Trend contains trends from the day.
• LocationName is where it was run.

You might notice that Select-Object in the middle of the pipeline.  We’re using it to pick the first of your monitors, but you can use it for a lot more.  For instance, if you just want to see a summary of how things are on each day, you could use:

Get-MonitisExternalMonitor |

    Select-Object -First 1 |

    Get-MonitisMonitorResult -Since 9/1/2011 |

    Select-Object -ExpandProperty Trend

like: Trend gives you an OK or not ok count.  If you can’t ping the box, or if a complex external monitor fails, you can use another built in PowerShell command to figure it out:  Where-Object.  Where-Object takes a filter, and, if the item in that filter returns an object which isn’t zero or a blank string, then Where-Object will output the original item.  This lets you filter anything in any way you’d like.  To find only the things in the last month that had trouble, use:

Get-MonitisExternalMonitor |

    Select-Object -First 1 |

    Get-MonitisMonitorResult -Since 9/1/2011 |

    Where-Object {

        $_.Trend.NotOkCount

    }

If that filter works, and your servers are working well, you should see nothing.    To prove it to yourself, go ahead and change NotOkCount to OkCount, and see only the locations with working results.  This is a good thing to be aware of.  In PowerShell, unless you’re getting data, no results are good results.  Meaning that if you see no output and no errors and it’s not a command that should return information to you, then your command just worked.

Now you know how make mincemeat of your results.  Next time, we’ll show you how generate a lot more results to manage, by monitoring web applications with Monitis.

See also

Monitor Everything with Monitis – And do it easily with PowerShell – Part 1

Monitor Everything with Monitis – And do it easily with PowerShell – Part 2

HTTP what?

Yes, HTTP extraction. Imagine you have a web page you would like to probe for parameters. Such as the number of your twitter subscribers, or the temperature somewhere – and profile it in Monitis.
Or another scenario could be to probe the responsiveness of your website – how fast is it?
Was it compromised and defaced?
I think you get the point – it is needed and important.

Infrastructure

For a while I’m promoting M3 – Monitis Monitor Manager monitoring framework as the infrastructure for adding monitors and counters to Monitis.

M3 monitoring framework is a Perl based infrastructure for parameter extraction with template support.
In order to implement HTTP parameter extraction, just slight modifications were required to get M3 do also this job.

Different protocols

M3 currently supports three ways of parameter extraction:

1. Regular Expressions
2. JSON
3. XML (providing an xpath)

Together with that, M3 monitoring framework can also provide you with various statistics regarding the responsiveness of the polled web page. Statistics such as delay, page size and more.
Authentication of web pages is also supported – in case you need to provide a user and password to access the page.

M3 Usage

The quick checkout snippet is here to help you, as usual:

 # mkdir -p /usr/share/monitisexchange && cd /usr/share/monitisexchange # git clone git://github.com/monitisexchange/Monitis-Linux-Scripts.git # cd Monitis-Linux-Scripts/M3 

For the self sufficient – the following files are sample files, intended to show you how M3 works with HTTP extraction:

• config_sample_json_extraction.xml
• config_sample_regex_extraction.xml
• config_sample_xpath_extraction.xml

For those of you who thrive for a live example, we’ll analyze the json extraction example:

 <?xml version="1.0"?> <config> <apicredentials apikey="XXX" secretkey="XXX"/> <agent name="Weather monitor" interval="5"> <monitor name="Weather in Sunnyvale"> <url>http://weather.yahooapis.com/forecastjson?w=2502265</url> <http_statistics>1</http_statistics> <metric name="Temperature"> <type>integer</type> <uom>F</uom> <json>{'forecast'}[0]->{'low_temperature'}</json> </metric> </monitor> </agent> </config> 

You’ll notice that we instruct M3 to fetch the URL  http://weather.yahooapis.com/forecastjson?w=2502265
Fetching it ourselves will show:

 # curl -s http://weather.yahooapis.com/forecastjson?w=2502265 { ... "forecast": [ { "day": "Today", "condition": "Partly Cloudy", "high_temperature": "75", "low_temperature": "57" }, { "day": "Tomorrow", "condition": "Partly Cloudy", "high_temperature": "79", "low_temperature": "59" } ] } 

Yes – I cheated a bit, I’ve used a JSON beautifier to beautify the output – because otherwise it’s a one ugly non-readable one-liner. I’ve also omitted some of the uninteresting output from the JSON call.
If you’ll follow the JSON path in the sample file – {‘forecast’}[0]->{‘low_temperature’} – you’ll see that it’ll bring us straight to the low_temperature of today’s in the JSON output.
Quite easy, isn’t it?

XML and regular expressions

Regular expressions were core functionality in M3 and there are also other examples showing the usage of them with M3.
XML xpath extraction is as simple as the JSON extraction. I’ll tell you even another secret – the implementation of xpath and JSON matching is the same!

HTTP statistics

In addition to that – M3 can also load some HTTP statistics like latency, page size and HTTP code. In order to enable it, simply have a look at config_sample_json_extraction.xml. You’ll see the <http_statistics> attributes over there – and this is what will add the statistics.

So… What now?

The snapshot provided in the top is graphing the temperature and humidity in Wanaka, New Zealand. Pretty neat, isn’t it?

What now? – Now you should look for parameters you’d like to extract from web pages, use M3 to load them to Monitis – and set the proper alerts for them.

Liked it? Sign up for Monitis and try it now! Don’t forget to check also monitisexchange – our git-hub open source repository for some more surprises.