EventID

Now event id seems to me like something which is not really used as intended or was badly designed. The general idea is that each message (read text) has a unique identifier provided by the implementing application. Thus far it is even pretty smart as it means that the event log does not contain any actual messages only an id and some metadata. The oddities if you ask me is that they decided to use the event-id for a lot of other things as well. The field contained in this number (and since it is a number many of these are just bits) are:

• Severity. The severity is defined as follows:
  • 00 – Success
  • 01 – Informational
  • 10 – Warning
  • 11 – Error
• Customer bit. This bit is defined as follows:
  • 0 - System code
  • 1 - Customer code
• Facility Facility code (I think of this as a component within the application).
• Code Number identifying the message

The first thing to notice is that if you look at modern windows the event viewer will not show you any of this information unless you “click advanced” and then only as a number next to the name Qualifiers.

The second thing is that some of these fields are a bit odd. Severity sounds a lot like “type” or level as it is called now. Facility sounds a lot like source. None of these fields will help provide any information for the end user and is mainly intended for the developer to help them organize message files or so it seems to me.

Filters

Filters was introduced in 0.3.8 of NSClient++ and consists of something similar to SQL statements (or WQL statements for that matter) where you can filter information you want. As of 0.4.0 we have the following keywords on which we can filter:

• Numbers:
  • id The event id
  • category The event category
  • qualifier Qualifier part of event id
  • facility Facility part of event id
  • customer Customer bit part of event id
  • rawid The entire event id including any qualifier/facility/customer/* parts This is obviously faster than separating them into multiple statements.
• Strings
  • source Event source
  • file / log Log file
  • message The renderd message
  • strings The string data (used to render the message) This is much much faster and yields the same result as message.
  • computer
• Dates
  • written
  • generated
• Keywords
  • type (level)
  • severity

So essentially we can filter on anything except sid which for some strange reason I missed . Writing filters is pretty straight forward as well and if you know some basic SQL you should not have too much trouble. This article is not about writing event log queries so I will only give you an example here (let me know if you want me to write an article about writing filters):

generated gt -2d AND severity NOT IN ('success', 'informational')

Real time event log

Real time event log monitoring is almost simpler to set up then using server-side checks. The drawback though is that it requires up-front and client-side configuration which is obviously not for everyone.

[/modules]
CheckEventLog=1

[/settings/eventlog/real-time]
enabled=true
filter=id = 1000 and category = 0

The above configuration will do two things, first enable CheckEventLog module and then enable real-time checking. We also configure a random filter I was using when testing something. So configuring this is pretty straight forward and simple. Apart from this there are a hand full of options you can tweak as well. The only problem here is what happens when you hit a message?

Well, since we have not configured a destination nothing will happen so thus far this is a bit useless. Unfortunately this blog post is not about configuring NSCA so the next section will only skim through it a bit. But first lets look at the other options we can tweak:

• enabled Spawns a background thread which detects issues and reports them back instantly.
• destination The destination for intercepted messages
• maximum age How long before reporting "ok" (if this is set to off no ok will be reported only errors)
• filter The initial age to scan when starting NSClient++
• syntax The initial age to scan when starting NSClient++
• log Coma separated list of logs to check
• debug Log missed records (useful to detect issues with filters) not useful in production as it is a bit of a resource hog.
• enable active This will store all matches so you can use real-time filters from active monitoring (use CheckEventlogCache).
• ok message This is the message sent periodically when no error is discovered.
• alias The alias to use for this event (in NSCA this is the service name).

NSCA

If we are planning to use this with NSCA (and we are) there are a few things to extra to configure. First we have a concept called service_name which we need to define. A service name is the way Nagios (or Icinga) associates the result with a check. So if you have multiple checks you need to be able to configure multiple service names for different filters. The way this is handled is using aliases. Aliases can be configured in two ways:

1. Locally by settings the key for the filter query under […/filters].
2. Globally in the […/real-time] section All filters without alias will use this alias.

So if you in Nagios have two different service_checks (eventlog_1 and eventlog_2) you need to set this using the […/filters] section like so:

[/settings/eventlog/real-time/filters]
eventlog\_1=id = 1000 and category = 1
eventlog\_1=id = 1000 and category = 0

The other thing we need to configure is the NSCA client itself which for simple scenarios is pretty straight forward to configure as well:

[/modules]
; ...
NSCAClient = 1

[/settings/NSCA/client/targets/default]
address=nsca://127.0.0.1:5667
encryption=aes256
password=YL04nBb14stIgCjZxcudGtMqz4E6NN3W

Finally we need to tell CheckEventlog to send messages to NSCA which is done by specifying the destination:

[/settings/eventlog/real-time]
; ...
destination=NSCA

Causing problems

Now, how can we test this (apart from sitting around waiting for something to happen?

Well, NSClient++ provides a rather nifty (and dangerous) command which allow you to inject messages into the event log.

nscp eventlog
CheckEventLog Command line syntax:
Allowed options:
-h [ --help ] Show help screen
-s [ --source ] arg (=Application Error)
source to use
-t [ --type ] arg Event type
-l [ --level ] arg Event level (type)
-f [ --facility ] arg Facility/Qualifier
-q [ --qualifier ] arg Facility/Qualifier
--severity arg Event severity
-c [ --category ] arg Event category
--customer arg Customer bit 0,1
-a [ --arguments ] arg Message arguments (strings)
--eventlog-arguments arg Message arguments (strings)
--event-arguments arg Message arguments (strings)
-i [ --id ] arg Event ID

In our case since we filter on event id 1000 we can use the following command to insert an application error.

nscp eventlog --exec insert-eventlog --source "Application Error" --id 1000 --level error --category 0

Which will trigger the following message to be sent to NSCA: *“Felet uppstod i programmet med namn: %1, version %2, tidsstämpel 0x%3…”* all the %1, %2 represent insert points which is where your event log message strings would normally end up. To add this we can add a series of --eventlog-argument options to insert some strings into these markers like so:

nscp eventlog --exec insert-eventlog --source "Application Error" --id 1000 --level error --category 0 --eventlog-argument a --eventlog-argument b ...