LCFG profile handling

May 13, 2016

Over the last few months the new libraries for handling LCFG profiles have been shaping up nicely. They are finally reaching a point where they match up with my original aims so I thought I’d give a taste of how it all works. Here’s an example of processing an LCFG XML profile into the Berkeley DB and rpmcfg files required by the client:

use LCFG::Profile;

my $xml    = '/var/lcfg/conf/profile/xml/example.lcfg.org.xml';
my $dbm    = '/tmp/example.lcfg.org.DB2.db';
my $dbm_ns = 'example';
my $rpmcfg = '/tmp/example.lcfg.org.rpmcfg';

my $new_profile = LCFG::Profile->new_from_xml($xml);

my $update_dbm = 0;
if ( -f $dbm ) {
    my $cur_profile = LCFG::Profile->new_from_bdb($dbm);

    my $diff = $cur_profile->diff($new_profile);

    if ( $diff->size > 0 ) {
        $update_dbm = 1;
    }
} else {
    $update_dbm = 1;
}

if ( $update_dbm ) {
    $new_profile->to_bdb( $dbm, $dbm_ns );
    say 'Updated DBM';
}

my $pkgs_changed = $new_profile->to_rpmcfg($rpmcfg);
if ( $pkgs_changed ) {
    say 'Updated packages';
}

This is basically what the LCFG client does whenever it processes a new profile but is a lot nicer than the current rdxprof code!


LCFG::Component environment plugins

January 5, 2015

Version 1.13.0 the Perl version of the ngeneric framework (LCFG::Component) provides an all-new environment initialisation system for component methods. This has support for plugins which mean it is fully extensible.

There is a new InitializeEnvironment method which is called for most standard methods which are accessible via om (including configure, start, restart, stop, run, and logrotate). The method can also be called from any additional methods you have added to your own components, the method needs access to the resources so it must be called after a call to LoadProfile or LoadStatus.

There are currently two plugins – a very simple one which can be used to set values for environment variables before the method is called and a more complex one that can do the equivalent of kinit and aklog to acquire Kerberos credentials and AFS tokens.

For full details see the LCFG wiki.


LCFG::Build::Skeleton changes

December 8, 2014

At the LCFG Annual Review meeting held last week one topic which was discussed was the idea of all Perl based LCFG components being implemented as modules with the component script just being a very thin wrapper which loads the module and calls the dispatch method. This has been our recommended coding style for quite a while now and we use this approach for many of the core components.

During the discussion I realised that the lcfg-skeleton tool which is used to create the outline directory structure for new projects does not support this way of working. To make it really easy to create new Perl-based components which follow recommended best-practice I have consequently updated LCFG-Build-Skeleton. The new version 0.4.1 creates a module file (e.g. lib/LCFG/Component/Foo.pm), the necessary CMake file and also tweaks the specfile appropriately. This will be in the stable release on Thursday 18th December or you can grab it from CPAN now.


LCFG authorization

December 3, 2014

The authorization of LCFG component methods (which are called using the om command) is typically done using the LCFG::Authorize module. This is limited to checking usernames and membership of groups managed in LCFG.

In Informatics we have for a long-time used a different module – DICE::Authorize – which extends this to also checking membership of a netgroup. Recently we discovered some problems with our implementation of this functionality which make it very inflexible. We have been connecting directly to the LDAP server and doing the lookup based on hardcoded information in the module. As this really just boils down to checking membership of a netgroup this can clearly be done more simply by calling the innetgr function. This will work via the standard NS framework so will handle LDAP, NIS or whatever is required. The necessary details are then only stored in the standard location and not embedded into the code.

Rather than just rewrite the DICE::Authorize module I took the chance to move the functionality to the LCFG layer, so we now have LCFG::Authorize::NetGroups. This nicely sub-classes the standard module so that if the user is not a member of a netgroup the other checks are then carried out. This is much better code reuse, previously we had two distinct implementations of the basic checks.

Having a new implementation of the authorization module is also handy for dealing with the transition stage. We can keep the old one around so that if a problem is discovered with the new code we can quickly switch back to the old code.

I also took the chance to improve the documentation of the authorization framework so if you’re still running om as root now is a good time to improve things!


Sub-classing LCFG components

December 3, 2014

One topic that often comes up in discussions about how to make things easier for LCFG component authors is the idea of sub-classing.

Although I had never tried it I had always assumed this was possible. Recently whilst looking through the LCFG::Component code I noticed that the list of methods are looked up in the symbol table for the module:

    my $mtable = {};
    for my $method ( ( keys %LCFG::Component:: ),
        ( eval 'keys %' . ref($self) . q{::} ) )
    {
        if ( $method =~ m/^method_(.*)/i ) {
            $mtable->{ lc $1 } = $method;
        }
    }
    $self->{_METHOD} = lc $_METHOD;
    my $_FUNCTION = $mtable->{ $self->{_METHOD} };

So, this will work if your method comes from LCFG::Component or LCFG::Component::Foo but it wouldn’t work if you have a sub-class of Foo. You would potentially miss out on methods which are only in Foo (or have to copy/paste them into your sub-class).

Not only does this make sub-classing tricky it also involves a horrid string eval. There had to be a better way. Thankfully I was already aware of the Class::Inspector module which can do the necessary. This module is widely used by projects such as DBIx::Class and Catalyst so is likely to be reliable. It has a handy methods method which does what we need:

    my $_FUNCTION;
    my $public_methods = Class::Inspector->methods( ref($self), 'public' );
    for my $method (@{$public_methods}) {
        if ( $method =~ m/^Method_(\Q$_METHOD\E)$/i ) {
            $_FUNCTION = $method;
            $_METHOD = $1;
            last;
        }
    }

Much nicer code and a tad more efficient. Now the LCFG component Perl modules are properly sub-classable.


LCFG Client Guide

March 31, 2014

As part of my work on updating the LCFG client I’ve written a guide to the inner workings of the LCFG client. This is intended to be fairly high-level so it doesn’t go into the details of which subroutine calls which subroutine. The aim is that this should cover all the main functionality and provide the information necessary to get started with altering and extending the client code base.


LCFG Client Refactor: New profile parser

January 24, 2014

Recently I’ve been working on developing a new framework which encapsulates all aspects of handling the LCFG profiles on the client-side. This framework is written in Perl and is named, appropriately enough, LCFG::Profile, I plan to blog about the various details in due course. The coding phase is almost complete and I’ve moved onto adding documentation for all the module APIs. I’ve found the documentation phase to be a very useful review process. It has helped me spot various dark corners of the code and methods which were added earlier in the development process which are no longer required. Removing this dead code now is a good idea as we may otherwise end up being committed to supporting the code if it forms part of a public API. I’ve also found it to be a very good way to spot inconsistencies between similar APIs implemented in the various modules. It’s definitely a good idea to follow the principle of least surprise whenever possible. If methods are named similarly and take a similar group of arguments they probably ought to return similar types of results.


LCFG Client Refactor: Phase Two

December 5, 2013

As the results of Phase One of the LCFG Client refactoring project are now in the beta-testing stage and approaching a roll-out date we have commenced work on Phase Two. The primary aim of this new work is to remove all dependencies on the W3C::SAX Perl modules which have been unmaintained for a very long time. We’re probably the last place in the world still using those modules so it’s definitely time to be moving on to something more modern. The project plan for this new work is available for anyone interested.

As a first step I’ve been prototyping some new XML parsing code based on the popular and well-maintained XML::LibXML module. I’ve also been thinking about ideas for an API for storing/accessing the information regarding components and resources. I’ve put together some useful notes on the LCFG XML profile structure to help me get my head around it all.


LCFG V3 Client – beta release

October 17, 2013

I am pleased to announce that the v3 update for the LCFG client has now reached the beta-release stage. As of stable release 2013101401a everything is in place to begin testing at your ownsite. Full details are available on the LCFG wiki.

If you come across any bugs or unexpected behaviour please file a bug at bugs.lcfg.org.


LCFG Annual Review Meeting

October 16, 2013

On Thursday 5th December instead of our normal Deployers Meeting we are going to be holding an Annual Review meeting.

All users of LCFG are encouraged to attend this meeting to hear about what has been happening over the last year and what developments they can look forwards to in the next year. This is also an excellent opportunity to raise issues that are important to you, put forward ideas for future developments you would like to see and chat about all things LCFG!

This will start at 2pm and we aim to be finished by 5pm. It will be held in room 2.33 of the Informatics Forum (note that this is NOT the usual room).

Full details are available on the LCFG wiki.

Afterwards there will be an informal gathering in a local pub followed by some of us going somewhere for food.

I hope to see lots of people there!