LCFG Core: resource types

November 21, 2017

The recent round of LCFG client testing using real LCFG profiles from both Informatics and the wider community has shown that the code is now in very good shape and we’re close to being able to deploy to a larger group of machines. One issue that this testing has uncovered is related to how the type of a resource is specified in a schema. A type in the LCFG world really just controls what regular expression is used to validate the resource value. Various type annotations can be used (e.g. %integer, %boolean or %string) to limit the permitted values, if there is no annotation it is assumed to be a tag list and this has clearly caught out a few component authors. For example:

@foo %integer
foo

@bar %boolean
bar

@baz
baz

@quux sub1_$ sub2_$
quux
sub1_$
sub2_$

Both of the last two examples (baz and quux) are tag lists, the first just does not have any associated sub-resources.

The compiler should not allow anything but valid tag names (which match /^[a-zA-Z0-9_]+$/) in a tag list resource but due to some inadequacies it currently permits pretty much anything. The new core code is a lot stricter and thus the v4 client will refuse to accept a profile if it contains invalid tag lists. Bugs have been filed against a few components (bug#1016 and bug#1017). It’s very satisfying to see the new code helping us improve the quality of our configurations.


LCFG Core: Resource import and export

November 7, 2017

As part of porting the LCFG client to the new core libraries the qxprof and sxprof utilities have been updated. This has led to the development of a new high-level LCFG::Client::Resources Perl library which can be used to import, merge and export resources in all the various required forms. The intention is that eventually all code which uses the LCFG::Resources Perl library (in particular the LCFG::Component framework) will be updated to use this new library. The new library provides a very similar set of functionality and will appear familiar but I’ve taken the opportunity to improve some of the more awkward parts. Here’s a simple example taken from the perldoc:

# Load client resources from DB
my $res1 = LCFG::Client::Resources::LoadProfile("mynode","client");

# Import client resources from environment variables
my $res2 = LCFG::Client::Resources::Import("client");

# Merge two sets of resources
my $res3 = LCFG::Client::Resources::Merge( $res1, $res2 );

# Save the result as a status file
LCFG::Client::Resources::SaveState( "client", $res3 );

The library can import resources from: Berkeley DB, status files, override files, shell environment and explicit resource specification strings. It can export resources as status files, in a form that can be evaluated in the shell environment and also in various terse and verbose forms (e.g. the output styles for qxprof).

The LCFG::Resources library provides access to resources via a reference to a hash which is structured something like:

{
   'sysinfo' => {
                 'os_id_full' => {
                                  'DERIVE' => '/var/lcfg/conf/server/releases/develop/core/include/lcfg/defaults/sysinfo.h:42',
                                  'VALUE' => 'sl74',
                                  'TYPE' => undef,
                                  'CONTEXT' => undef
                                 },
                 'path_lcfgconf' => {
                                  'DERIVE' => '/var/lcfg/conf/server/releases/develop/core/include/lcfg/defaults/sysinfo.h:100',
                                  'VALUE' => '/var/lcfg/conf',
                                  'TYPE' => undef,
                                  'CONTEXT' => undef
                                 },
                }
}

The top level key is the component name, the second level is the resource name and the third level is the name of the resource attribute (e.g. VALUE or TYPE ).

The new LCFG::Client::Resources library takes a similar approach with the top level key being the component name but the value for that key is a reference to a LCFG::Profile::Component object. Resource objects can then be accessed by using the find_resource method which returns a reference to a LCFG::Resource object. For example:

my $res = LCFG::Client::Resources::LoadProfile("mynode","sysinfo");

my $sysinfo = $res->{sysinfo};

my $os_id_full = $sysinfo->find_resource('os_id_full');

say $os_id_full->value;

Users of the qxprof and sxprof utilities should not notice any differences but hopefully the changes will be appreciated by those developing new code.


Testing the new LCFG core : Part 2

May 18, 2017

Following on from the basic tests for the new XML parser the next step is to check if the new core libs can be used to correctly store the profile state into a Berkeley DB file. This process is particularly interesting because it involves evaluating any context information and selecting the correct resource values based on the contexts. Effectively the XML profile represents all possible configuration states whereas only a single state is stored in the DB.

The aim was to compare the contents of the old and new DBs for each Informatics LCFG profile. Firstly I used rdxprof to generate DB files using the current libs:

cd /disk/scratch/profiles/inf.ed.ac.uk/
for i in $(find -maxdepth 1 -type d -printf '%f\n' | grep -v '^\.');\
do \
 echo $i; \
 /usr/sbin/rdxprof  -v -u file:///disk/scratch/profiles/ $i; \
done

This creates a DB file for each profile in the /var/lcfg/conf/profile/dbm directory. For 1500-ish profiles this takes a long time…

The next step is to do the same with the new libs:

find /disk/scratch/profiles/ -name '*.xml' | xargs \
perl -MLCFG::Profile -wE \
'for (@ARGV) { eval { $p = LCFG::Profile->new_from_xml($_); \
$n = $p->nodename; \
$p->to_bdb( "/disk/scratch/results/dbm/$n.DB2.db" ) }; \
print $@ if $@ }'

This creates a DB file for each profile in the /disk/scratch/results/dbm directory. This is much faster than using rdxprof.

The final step was to compare each DB. This was done simply using the perl DB_File module to tie each DB to a hash and then comparing the keys and values. Pleasingly this has shown that the new code is generating identical DBs for all the Informatics profiles.

Now I need to hack this together into a test script which other sites can use to similarly verify the code on their sets of profiles.


Testing the new LCFG core : Part 1

May 17, 2017

The project to rework the core LCFG code is rattling along and has reached the point where some full scale testing is needed. The first step is to check whether the new XML parser can actually just parse all of our LCFG profiles. At this stage I’m not interested in whether it can do anything useful with the data once loaded, I just want to see how it handles a large number of different profiles.

Firstly a source of XML profiles is needed, I grabbed a complete local copy from our lcfg server:


rsync -av -e ssh lcfg:/var/lcfg/conf/server/web/profiles/ /disk/scratch/profiles/

I then ran the XML parser on every profile I could find:


find /disk/scratch/profiles/ -name ‘*.xml’ | xargs \
perl -MLCFG::Profile -wE \
‘for (@ARGV) { eval { LCFG::Profile->new_from_xml($_) }; print $@ if $@ }’

Initially I hit upon bug#971 which is a genuine bug in the schema for the gridengine component. As noted previously, this was found because the new libraries are much stricter about what is considered to be valid data. With that bug resolved I can now parse all 1525 LCFG XML profiles for Informatics.


LCFG Core Project

May 2, 2017

Over the last few years I have been working on (and off) creating a new set of “core” libraries for LCFG. This is now finally edging towards the point of completion with most of the remaining work being related to polishing, testing and documentation.

This project originated from the need to remove dependencies on obsolete Perl XML libraries. The other main aims were to create a new OO API for resources/components and packages which would provide new opportunities for code reuse between client, ngeneric and server.

Over time several other aims have been added:

  • Simplify platform upgrades.
  • Platform independence / portability.
  • Make it possible to support new languages.
  • Ensure resource usage remains low.

Originally this was to be a rewrite just in Perl but the heavy resource usage of early prototypes showed it was necessary to move at least some of the functionality into C libraries. Since that point the chance to enhance portability was also identified and included in the aims for the project. As well as making it possible to target other platforms (other Linux or Unix, e.g. MacOSX), the enhanced portability should make it much simpler and quicker to port to new Redhat based platforms.

The intention is that the new core libraries will be totally platform-independent and portable, for example, no hardwired paths or assumptions that platform is Redhat/RPM (or even Linux) based. The new core is split two parts: C and Perl libraries with the aim that as much functionality as possible is in the C libraries to aid reuse from other languages (e.g. Python).

The aim is that these libraries should be able to co-exist alongside current libraries to ease the transition.

I have spent a lot of time on documenting the entire C API. The documentation is formatted into html and pdf using doxygen, I had not used this tool before but I am very pleased with the results and will definitely be using it more in the future. Although a slow task, documenting the functions has proved to be a very useful review process. It has helped me find many inconsistencies between functions with similar purposes and has led to numerous small improvements.

LCFG Client

The client has been reworked to use new Core libraries. This is where the platform-specific knowledge of paths, package manager, etc, is held.

Resource Support

Format Read Write
XML YES NO
DB YES YES
Status YES YES
Environment YES YES

There is currently no support for reading header files or source profiles but this could be added later.

There is new support for finding the “diffs” between resources, components and profiles.

Package Support

Format Read Write
XML YES YES
rpmcfg YES YES
rpmlist YES YES

There is currently no support for reading package list files but this could be added later.

Remaining Work

There is still work to be done on the top-level profile handling code and the code for finding the differences between resources, components and profiles needs reworking. Also the libraries for reading/writing XML files and Berkeley DB need documentation.

That is all the remaining work required on the “core” libraries. After that there will be some work to do on finishing the port of the client to the new libraries. I’ve had that working before but function APIs have changed, I don’t expect it to require a huge amount of work.


LCFG Client: Hasn’t died yet…

August 2, 2016

Coming back from holiday I was pleased to see that I have a v4 client instance which has now been running continuously for nearly 3 weeks without crashing. It hasn’t done a massive amount in that time but it has correctly applied some updates to both resources and packages.

In the time I’ve not been on holiday I’ve been working hard on documenting the code. For the C code I’ve chosen to use doxygen, it does a nice job of summarizing all the functions in each library and it makes it very simple to write the documentation using simple markup right next to the code for each function. I’ve also been working through some of the Perl modules and adding POD where necessary. It might soon be at the stage where others can pick it up and use it without needing to consult me for the details…


LCFG Client: It lives!

July 15, 2016

Cue forked lightning and crashes of thunder…

After much effort I finally have the first functional installation of the v4 LCFG client. This sees all the XML parsing and profile handling moved over to the new LCFG::Profile Perl modules which are wrappers around the new lcfg-core suite of libraries. There is still a bit of work required to properly handle LCFG contexts but otherwise it can handle everything we need. There are probably lots of small bugs to be resolved, there is also an almost total lack of documentation and the tests needs lot of attention but hey, at least it runs!


LCFG Profile – Secure mode

May 19, 2016

The LCFG client has a, slightly weird, feature called “secure mode“. This makes the client hold off applying any resource changes until they have been manually reviewed. The manual checking is done by examining the contents of a “hold file” which shows the differences in values for each modified resource in a simple text form. The file also contains a “signature” which is the MD5 digest (in hex) of the changes. A change set is applied manually by passing that signature to the client which then regenerates the hold file and compares that signature with the one supplied. This is not a heavily used feature of the client but it is something we want to support in the new LCFG profile framework. The new framework has built-in support for diffing the data structures which represent LCFG profiles, components and resources. This makes it relatively straightforward to add a feature which generates the secure-mode hold file when required, the only awkward part was finding some code to do the MD5 digest in a nice way.

Here’s an example using the C API, error checking and suchlike has been dropped to keep it simple.


#include <lcfg/profile.h>
#include <lcfg/bdb.h>
#include <lcfg/differences.h>

int main(void) {

char * msg = NULL;

LCFGProfile * p1 = NULL;
lcfgprofile_from_status_dir( “/run/lcfg/status”,
&p1, NULL, &msg );

LCFGProfile * p2 = NULL;
lcfgprofile_from_bdb( “/var/lcfg/conf/profile/dbm/example.lcfg.org.DB2.db”,
&p2, NULL, 0, &msg );

LCFGDiffProfile * diff = NULL;
lcfgprofile_diff( p1, p2, &diff, &msg );

char * signature = NULL;
lcfgdiffprofile_to_holdfile( diff, “/tmp/holdfile”, &signature, &msg );

lcfgprofile_destroy(p1);
lcfgprofile_destroy(p2);
lcfgdiffprofile_destroy(diff);

free(msg);

return 0;
}


LCFG profile querying

May 13, 2016

The new LCFG profile framework makes it simple to retrieve component and resource information from profiles stored in the various standard formats (XML, Berkeley DB and status files).

Loading a profile from XML, DB or status directory:


my $p = LCFG::Profile->new_from_xml(“example.xml”);


my $p = LCFG::Profile->new_from_bdb(“example.db”);


my $p = LCFG::Profile->new_from_status_dir(“/run/lcfg/status”);

Loading a component from a DB or status file:


my $c = LCFG::Profile::Component->new_from_bdb( “example.bdb”, “client” );


my $c = LCFG::Profile::Component->new_from_statusfile( “/run/lcfg/status/client” );

Retrieving a component (e.g. client) from the profile:


my $c = $p->find_component(“client”);

Retrieving a resource (e.g. client.components) from a component:


my $r = $c->find_resource(“components”);

Getting the resource value:


say $r->value;

For convenience, if the resource is a tag list then you can get the value as a perl list:


@comps = $r->value;
for my $comp (@comps) {

}


LCFG profile handling

May 13, 2016

Over the last few months the new libraries for handling LCFG profiles have been shaping up nicely. They are finally reaching a point where they match up with my original aims so I thought I’d give a taste of how it all works. Here’s an example of processing an LCFG XML profile into the Berkeley DB and rpmcfg files required by the client:

use LCFG::Profile;

my $xml    = '/var/lcfg/conf/profile/xml/example.lcfg.org.xml';
my $dbm    = '/tmp/example.lcfg.org.DB2.db';
my $dbm_ns = 'example';
my $rpmcfg = '/tmp/example.lcfg.org.rpmcfg';

my $new_profile = LCFG::Profile->new_from_xml($xml);

my $update_dbm = 0;
if ( -f $dbm ) {
    my $cur_profile = LCFG::Profile->new_from_bdb($dbm);

    my $diff = $cur_profile->diff($new_profile);

    if ( $diff->size > 0 ) {
        $update_dbm = 1;
    }
} else {
    $update_dbm = 1;
}

if ( $update_dbm ) {
    $new_profile->to_bdb( $dbm, $dbm_ns );
    say 'Updated DBM';
}

my $pkgs_changed = $new_profile->to_rpmcfg($rpmcfg);
if ( $pkgs_changed ) {
    say 'Updated packages';
}

This is basically what the LCFG client does whenever it processes a new profile but is a lot nicer than the current rdxprof code!