Category Archives: sysadmin

research on monitoring and the modern sysadmin

This morning I’m working through a chapter in my book (the Operations Primer, tentatively) on monitoring and logging.  Something that is important to me to communicate in this chapter is how different a monitoring ‘solution’ is from what many people think they’re doing, but I want to do this without sounding snarky.  It’s tough.  

As a preview to what I’m writing, here’s a breakdown of what I think needs to be included in any comprehensive monitoring solution:

“A monitoring system should have the following characteristics:

Delivered from a system-neutral platform

100% available (or as close as you can get it)

Available based on sensible access controls and least-privilege security

Able to deliver information in a flexible manner

No system should be the exclusive source of monitoring information about itself


The actions performed by the system should include the following functionality:


Generating alerts based on conditions

Generating alerts based on heuristics

Resolving alerts manually

Resolving alerts based on conditions

Displaying performance metrics

Recording event history

Enabling a ‘maintenance mode’ manually or on a schedule

Launching utilities to perform common maintenance tasks


Other valuable characteristics of monitoring systems:


The admins who run the systems configure the monitors themselves . . .”

And that’s where I left off.  The next thing I planned to discuss was the building blocks of monitoring, such as SNMP, ICMP, and data sources, but the last thing I wrote in that list above made me stop and think.  The idea that admins should be in charge of configuring their own monitors is a lesson I learned myself while I was a SAN administrator trying to get help with using Zenoss.  Which I was not allowed to configure on my own.  This kind of lesson that was learned through pain and annoyance is exactly the kind of material I want to include in my book, so what other lessons have people learned out there?


This led me to stop writing and to start doing some research.  At 6:20am I’m not going to be able to bend the ears of too many of my colleagues, but the inter-webs have provided me with two videos from sysadmins who have a lot to say:


“The evolution of the SysAdmin & holistic monitoring for apps and servers” by Matt Simmons.  Provided by Solarwinds and requires registration.


“Monitoring Maturity: a 16 year journey and lessons learned” by Simon Finch at Nagios Con 2014



Mobile device security policies

I recently ran into a situation at work where a colleague of mine was traveling overseas and lost her iPhone. After the initial ‘oh crap what would I have done?’ reaction to this scenario I got thinking about the implications of mobile devices and information security. This doesn’t require a very high level of training in IT security to think through. Someone who has your phone in hand probably has access to:

  • Your contact information and the contact information of everyone you call or text
  • Your photos and personal experiences
  • Some browsing history
  • Your music
  • Your ability to purchase things through either the iTunes store or Google Play
  • Saved credential -based access to<
  • websites you frequent from your phone

How much of this would you be willing to give away?

Many people scoff at this loss since they have already wisely configured a passcode to prevent unauthorized use of their phone. This is great for keeping your nephew away from Angry Birds, but several methods exist for bypassing passcodes, depending on your model and operating system version. Dedicated phone intruders could skip these junior high approaches, however, and jump right in with tools designed for digital forensics like enCase or Sleuthkit.

By the way, if you think in terms of a hardcore attack of your iPhone data, keep in mind the possibility of someone attacking your iTunes backup of your phone, stored on the local disk of your computer. Even if you chose to encrypt that backup, it’s subject to brute force attacks. And it contains pretty much every single thing your phone holds. I happened to find someone’s paper on attacking mobile device backups and mobile devices themselves pretty easily on the web. Check it out at SMU.


Protecting a mobile device once it has been left behind in a taxi is pretty tough, so how should we protect ourselves in advance of this? Obviously using a passcode and encrypting things where we can is the bare minimum, but a broad mobile device policy seems to be the smart thing. This policy ought to include the following components:

To whom does the policy apply and under what criteria? (Are iPads included? Surface tablets?)
How is the mobile device provisioned under the policy? (Are technical policies pushed from a central resource? Is the device documented in inventory?)
What are acceptable uses of the device?
Under what conditions will the device be excluded from the policy?
What actions need to be taken in the event of a lost device?
What actions need to be taken when the device is de-commissioned?

The SANS Reading Room is one of my favorite places to go for academic discussions of stuff like this, and I was able to quickly find a paper there on the subject of mobile device policies in corporate environments. This is a very practical discussion of all the moving parts of such a policy, and does a great job of outlining the vocabulary and the process of getting something of this rolling. Nice work, Nicholas.