Simon's Musings

March 27, 2009

UKUUG Spring Conference

Filed under: Uncategorized — sxw @ 12:16 pm
Tags: , , , , ,

I’ve just returned from the spending 3 days in London at the UKUUG Spring Conference. I presented a Kerberos tutorial on the first day, and spent the following 2 as a conference delegate. The tutorial was well attended, with over 50 people there on the day, and seemed to go really well with a lot of good feedback from the attendees.

The second and third days were taken up with the conference proper. There seemed to be more delegates than in previous years, although the number of talks was smaller, with only one conference track. Whilst holding the conference in London obviously served to increase its appeal to those living locally, the venue wasn’t entirely ideal. Whilst the space for the talks was fine, there was a lack of break out and foyer space, making lunch and coffee breaks a scramble for space, and in depth conversations out of the conference hall harder.

The talks themselves covered a good mixture of topics, with security, LDAP and monitoring being particularly prevalent. The conference started with a presentation from Barry Scott of Centrify about integrating Unix boxes with Active Directory.  This gave a good overview of the situation (and said some nice things about the Kerberos tutorial), but talked more about their commercial product than what was possible with the available open source tools. From my perspective, this was a slightly missed opportunity, although the overview would have been of use to anyone contemplating that integration.

Later in the day,  Andrew Findlay gave a very strong and well presented talk on LDAP access control policies. (there is also a pdf paper) Whilst this continued the logical progression from what Andrew’s said about LDAP ACLs at previous conferences, it wrapped all of his current thinking up into a single, easily digestible block. It reconfirmed some of my design choices with prometheus, and challenged others. 

After lunch, there was a “Systems Monitoring Shootout“, comparing the features of various different systems monitoring packages. There were some really interesting ideas in here, including the use of NagiosGraph to produce rrd files which can then be used for trend and capacity analaysis. Following this, Jane Curry presented on ZenOss, a Zope based network monitoring tool. This appeared to be more network focussed than the service focus of Nagios, with lots of features like automatic device discovery and a very pretty looking interface. However, nothing that convinced me we should drop Nagios and use it instead. Finally in this session we had a very well presented skip through the … interesting … things you could do the the SCSI bus with sysfs, and the power of lvm in terms of disk management. 

In the final session of this day, Darren Moffat from Sun ran through some of the security features in Open Solaris. As well as a name check for my OpenSSH work, Darren talked about the new concept of role users, the move towards privileges in the kernel, and the additional RBAC work that’s in OpenSolaris. He also trailed the encryption features which will shortly be appearing in ZFS. All in all, a fascinating talk.

After Gavin Henry had talked about the replication strategies currently available in OpenLDAP, Howard Chu gave a great talk about its new MySQL NDB backend. Primarily developed with telco grade customers in mind, this allows you to share your database between MySQL and OpenLDAP, and take advantage of NDB’s clustering properties to linearly scale your load by simply adding more servers. The downside is that there are fixed constraints on attribute set size and tree depth. So, not a new general purpose backend, but a real insight into the large scale deployments that Symas is doing with OpenLDAP. I took the opportunity to quiz Howard about API stability for overlays – his answer unfortunately confirmed my view that the API isn’t stable enough to let us use them for prometheus.

Continuing the telco theme, Craig Gellen spoke about OpenNMS, a network management system which was designed from the ground up for large scale enterprise and telecommunications customers. Again, this system seems more network than systems monitoring focussed, and probably far too complex for our needs, but it was really interesting to see a piece of Open Source software which is specifically targeted at this market.

The final session started with a couple of virtualisation talks. Kris Buytaert talked about the current, and ever shifting, state of the Open Source virtualisation world, including a discussion of the current allegiances of the major vendors. Following this openQRM, an open source, virtual datacentre management tool, was presented. Matthias Rechenburg’s talk focussed in particular on cloud computing. OpenQRM has an automated provisioning model, where a user can use a web interface to request (and pay for!) a certain amount of time on a certain number of auto built virtual machines. The talk concluded with a demo that both worked, and held the audiences attention – no mean feat!

Alex Howells from Gradwell gave the final talk of the day – a tour of the major external security threats he’s become aware of during his time managing systems for Bytemark and Gradwell. This was a detailed look at the common security issues on today’s internet, as well as giving helpful advice on how to counter them. Whilst some things (for example using fail2ban on external facing services) would be easy to put into practice here, others (requiring code review for everything that runs on a web server) wouldn’t be appropriate to our environment. All in all though, this was a good talk, containing a lot of things to ponder, and a great way to end the conference.

Despite having a smaller set of talks than in the past, the technical content of the conference seemed stronger than it has been in the last couple of years. Having a single track did help to improve its focus, although the reduction in moving around, coupled with the lack of break out space did reduce the opportunities to interact with other delegates. The UKUUG are changing the focus of their Summer Conference (which has typically been Linux based) to encompass a very wide scope, some of which overlaps with the LISA focus of this event. I suspect its long term future remains to be seen.

All in all, though, I think the UKUUG Spring Conference is a very useful event to attend.

February 3, 2009

Opting out of Nagios Notifications

Filed under: Uncategorized — sxw @ 11:34 am
Tags: ,

If you are going to be away for a long amount of time, you can opt out of all Nagios notifications by changing some entries in your LDAP record. Unfortunately the UI for this is currently pretty non-existent, so here’s some low level LDAP hackery that should acheive the desired results…

First things first, you need to have the nagiosUser objectClass. You can get that, by running the following ldapmodify command (The lines in black are what you type, lines in grey are examples of return from the command)

[boogaloo]sxw: ldapmodify -h ldap.inf.ed.ac.uk
SASL/GSSAPI authentication started
SASL username: sxw@INF.ED.AC.UK
SASL SSF: 56
SASL installing layers

dn: uid=sxw, ou=People,dc=inf,dc=ed,dc=ac,dc=uk
changetype: modify
add: objectClass
objectClass: nagiosUser
modifying entry “uid=sxw, ou=People,dc=inf,dc=ed,dc=ac,dc=uk”

Type CTRL-D to exit the ldapmodify command.

Now that you’ve got the relevant objectClass, you need to configure your Nagios settings so that you aren’t bothered. There are a number of ways of doing this, but the easiest is to set the notification period (the times of the day which Nagios will tell you of problems) to be none, which is a predefined period meaning ‘never tell me’.

[boogaloo]sxw: ldapmodify -h ldap.inf.ed.ac.uk
SASL/GSSAPI authentication started
SASL username: sxw@INF.ED.AC.UK
SASL SSF: 56
SASL installing layers

dn: uid=sxw,ou=People,dc=inf,dc=ed,dc=ac,dc=uk
changetype: modify
add: nagiosHostNotificationPeriod
nagiosHostNotificationPeriod: none

add: nagiosServiceNotificationPeriod
nagiosServiceNotificationPeriod: none

modifying entry “uid=sxw,ou=People,dc=inf,dc=ed,dc=ac,dc=uk”

As before, type CTRL-D to exit the ldapmodify command

After the usual propagation dance has occurred, you will find you’ll stop getting Nagios notifications. Just remember to turn them back on (by deleting these two attributes) when you get back!

Update: Graham just asked in the chatroom what the required incarnation to disable this is. Just so you don’t have to wait until I get back, here it is…

[boogaloo]sxw: ldapmodify -h ldap.inf.ed.ac.uk
SASL/GSSAPI authentication started
SASL username: sxw@INF.ED.AC.UK
SASL SSF: 56
SASL installing layers

dn: uid=sxw,ou=People,dc=inf,dc=ed,dc=ac,dc=uk
changetype: modify
delete: nagiosHostNotificationPeriod

delete: nagiosServiceNotificationPeriod

modifying entry “uid=sxw,ou=People,dc=inf,dc=ed,dc=ac,dc=uk”

April 3, 2008

UKUUG spring conference

Filed under: Uncategorized — sxw @ 11:04 am
Tags: , , , ,

I’ve just got back from the UKUUG Spring Conference, where a group of us from Informatics (myself, Stephen, Paul and Gihan from Flexiscale) were giving talks. I talked on two subjects – the LCFG based monitoring system framework I developed last year, and the new account management system I’m currently writing. Slides from both of these talks are available on the DICE publications page, which also has Stephen’s slides from his “An end to hacky scripts” talk about the LCFG system.

Despite gaining a scripting language track, and the addition of a parallel one-day PostgreSQL conference, the event seemed smaller this year, with many of the familiar faces missing. Some unfortunate scheduling meant that switching between tracks wasn’t as easy as it could have been, with 45 minute sessions in one room scheduled against 30 minute sessions in another one. However, the event was still productive, useful and stimulating, with a number of interesting talks – slides and audio from which should hopefully be up on the conference website shortly.

Some highlights were the talk from Mark Gledhill from the BBC on “Feeding the BBC Homepage“, which provided a fascinating insight into perl and Catalyst usage at a large organisation, as well as giving a useful background on their project management techniques, and test and deployment issues. Gavin Henry’s talk on OpenLDAP 2.4 provided a valuable summary of the changes in the latest version of OpenLDAP, as well as giving some examples of practical uses for these new features. Randy Appleton’s “Today’s Software … Is It Really Bloated?” talk took a very humorous tour through a number of code size and performance statistics he and his students have been collecting over the years – a perfect start to the day after the conference dinner!

The Transitive (which I ended up seeing because it was swapped with the talk I wanted to hear – one peril of last minute schedule changes!), and ZFS talks pretty much repeated material I’d heard at other conferences, but the ZFS one, in particular, was a helpful reminder of a system I’d really like to have time to look at in more detail. Whilst I wasn’t specifically interested in the scripting language talks, I did manage to catch “USENET Gems” which provided details of a number of interesting perl quirks, which are now firmly filed as things to watch out for.

Paul and Stephen arranged a well attended LCFG BOF on the Tuesday afternoon, and Paul, Stephen and I took some time to chat on Wednesday about possible designs for the new LCFG compiler. As with all UKUUG conferences, it tends to be these unscheduled events, and impromptu corridor conversations where the real value lies. There was a large amount of interest in prometheus, both from people in the commercial sector who have deployed similar systems, and had insights to share, and those who are interested in similar systems for their own sites. Hopefully we’ll be able to build some kind of a community around this technology.

There was continued interest in OpenAFS and Kerberos, with a number of people asking questions both about the technology, and our deployment experiences. Access to the source code for the monitoring system was also in demand – I really should arrange to publish this somewhere less adhoc.

March 15, 2008

New apacheconf and monitoring thoughts

Filed under: Uncategorized — sxw @ 5:05 pm
Tags: , ,

Yesterday, I shipped a new apacheconf component, with some significant changes to its monitoring support.

Apache is a complicated beast, with many different mechanisms for configuring it. Apacheconf doesn’t necessarily handle all of these different options, and sometimes work arounds are necessary. For example, apache supports providing multiple ip:port combinations to a VirtualHost directive. Apacheconf only supports providing one. For this reason, Neil had configured a service with two VirtualHosts, both with the same server name. Unfortunately, apacheconf assumed that all of the server names would be unique on a given hosts, and so builds its Nagios service descriptions (which must be unique) based on these server names. Upshot of this is that we end up with a monitoring configuration that won’t load.

I’ve made two changes to help mitigate this. Firstly, every apacheconf virtualhost now has a
vhostnagiosmonitor directive, which can be set to false to disable monitoring for that virtual host. Secondly, the apacheconf translator now keeps a list of all of the service descriptions it has created, and adds uniquifiers to any duplicates (initially the IP address and, if that isn’t sufficient, a number).

In addition to this, a new lcfg-monitor has shipped containing a number of bug fixes.

In the long run, we need to give lcfg-monitor the ability to take a list of machines and components for which monitoring is disabled – so that, if this happens again, we don’t end up having to rush to fix broken configurations, or components, just to keep monitoring running for everyone else.

February 15, 2008

Talking

Filed under: Uncategorized — sxw @ 6:14 pm
Tags: , , , , , ,

I’m giving a few talks over the next couple of months

  • UKUUG Files and Backup Seminar I’m giving a general overview of AFS from a users and administrators perspective, particularly focusing on features that will be of interest to new deployments
  • FOSDEM I’m giving a developers overview of OpenAFS as a lightning talk
  • UKUUG Spring Conference I’m currently scheduled to give two talks. The first is an overview of our monitoring system, talking in particular about the benefits (and challenges) of integrating it with LCFG. The second is about our in-development account management system, prometheus, and some of its unique features.
  • AFS & Kerberos Best Practices Workshop

I’m going to FOSDEM, the Free and Open Source Software Developers’ European Meeting

Theme: Rubric.