LCFG Profile Security Project

June 12, 2018

The various changes necessary to add support for GSSAPI authenticated profile fetching have now been moved from our local dice configuration into the general lcfg area. The new support can be enabled by specifying the LCFG_AUTH_GSSAPI macro at the start of an LCFG source profile. Most of the resources necessary are configured in the lcfg/options/lcfg-client-gssapi.h header file but sites wanting to use this will still need a little additional configuration.

For the installer, support is enabled by setting the lcfg.kauth=1 option on the kernel command line. Those sites using the ed/options/pxeserver.h header will automatically get a menu option which supports this mode. The installer can be be enabled for a client with something like this:

#ifdef LINUX_EL7
!pxeclient.platforms            mREPLACE(sl7,sl7kauth)
!pxeclient.default_label        mSUBST(sl7,sl7kauth)
#endif /* GSSAPI support enabled */
#endif /* LINUX_EL7 */

It is important to note that the lcfg-client-gssapi.h header does not remove http access for the XML profile. That could either be turned off completely for the apache virtual host or a per-client deny all rule could be put into the associated .htaccess file. Also, the default access control for the XML profiles is based on an apache group named lcfgadmins which must be populated on the LCFG servers, for example:

!apacheconf.groups   mADD(lcfgadmins)
!apacheconf.members_lcfgadmins mSET( 

Note that this is a list of admin principals, this is done so the credentials can also be used to register machines in the KDC using the kdcregister tool.

As part of this work I’ve also added the necessary methods to support using kdcregister in the LCFG installer. Most sites use this and have up to now been duplicating the configuration. It can be enabled by specifying the LCFG_OPTIONS_KERBEROS_CLIENT_REGISTER macro prior to including the lcfg/options/kerberos-client.h header file.

LCFG apacheconf improvements

June 1, 2018

I know that the LCFG apacheconf component is one of the more widely used of the non-default components so I thought it worth summarising some recent changes.

  1. Groups support

    It’s now possible to manage simple apache groups through apacheconf resources like this:

    !apacheconf.groups mADD(admins)
    !apacheconf.members_admins mSET(user1 user2 user3)

    By default these are stored in the file /etc/httpd/lcfg.groups.d/general but that can be changed on a per-group basis. Within your apache configuration this can then be used like this:

    AuthGroupFile "/etc/httpd/lcfg.groups.d/general"
    Require group "admins"

    Of course, the authz_groupfile module needs to be enabled, that’s the default on EL7.

  2. sysconfig support

    Occasionally it is necessary to set extra environment variables before starting apache. This is best done through the /etc/sysconfig/httpd file. Previously the name of the template for this file was hardwired in the component code so that it was not possible to generate truly site-local config. There is now an apacheconf.sysconfig_tmpl resource which can be used to switch the template name.

  3. virtualhosts with multiple addresses

    The support for virtualhosts which listen on multiple explicit addresses has been improved. This is where the virtualhost block
    starts like:


    Previously, due to a bug in the template, the default port (from the apacheconf.vhostaddr_tag resource) was only applied to the final address in the list. Further to this, if any of the addresses have an explicit port which differs from the default for the vhost it will now be honoured.

  4. template paths

    For convenience, all template paths specified through resources may now be expressed as relative paths (e.g. sysconfig.tmpl), there’s no need to know where they are stored in the local filesystem.

  5. nagios translator

    I believe this is only used by Informatics. I’ve improved the way the IP address for monitoring each vhost is selected. This fixes a couple of peculiar problems we’ve been seeing with nagios that meant the host problems were permanently acked. I’ve also added IPv6 support although it’s not clear what the nagios checks would do with such addresses. There’s still room for improvement here but that will require a more extensive refactoring of the code.

LCFG Profile Security Project

May 17, 2018

In the LCFG installer we have previously only needed to prompt the administrator for a Kerberos principle and password when using the kdcregister tool. With a switch to GSSAPI authentication for LCFG profile fetching we now need the administrator to also authenticate for that purpose. Everyone knows that system administrators are busy people who really don’t want to be prompted twice for the same principle/password so we need to avoid that if at all possible. For most services once the user has authenticated and acquired a ticket granting ticket (TGT) they don’t need to authenticate again but kdcregister needs a kadmin/admin ticket which causes us a slight problem as it typically (depending on site policy) cannot be acquired using the TGT. This is a deliberate security measure to make it harder to become an admin user with enhanced privileges to modify entries in the KDC. The simple solution taken for the LCFG installer is to prompt the user for principle and password and then use them behind the scenes to authenticate twice using kinit in slightly different ways using two different credential cache files. As we have to do a number of LCFG profile fetches the KRB5CCNAME environment variable gets set to the path to the credentials cache which holds the TGT so all profile fetches (and anything else which might need it) will just work. For this to work with the kdcregister tool the code had to be modified (see bug#1068 for details) to support a new command line option so that the path to the credentials cache can be specified. Previously kdcregister has always used the default credentials cache which was either the path in KRB5CCNAME or whatever the krb5.conf had specified. With those modifications in place the LCFG installer is now fully ready for fetching LCFG profiles in a secure way which doesn’t require any extra authentication steps for administrators. For this part of the project to be totally completed there are still quite a few resource changes which need to be pushed out over the next few weeks but those can only happen once sites have deployed new installer images.

LCFG ngeneric kinit plugin

April 26, 2018

I’ve recently put a bit of work into making the LCFG ngeneric kinit environment initialisation plugin more generally useful. Prior to Kenny’s helpful tip on getting the gssapi library to do most of the work I had planned to use this plugin within the LCFG client for Kerberos authentication when fetching profiles. It can now manage the credentials cache in a similar way to k5start with renewing the credentials whenever they expire or are within a certain time before expiry. So, it’s now possible to do this from any Perl code:

use LCFG::Component::Plugin::Kinit;

my $plugin = LCFG::Component::Plugin::Kinit->new(
    params    => {
        keytab    => "/etc/lcfg/client.keytab",
        principal => 'lcfg/',
        group     => 'lcfg',
        mode      => '0640',

I’ve also added support for specifying the owner, group and mode of the credentials cache file (bug#1061) and improved support for controlling whether the cache file is removed automatically when the calling process ends. At the same time I have resolved a longstanding issue with calling the run method multiple times (bug#1060) which is hit for, at least, when an LCFG component restart method is used.

LCFG Profile Security Project

April 25, 2018

Having added support to the LCFG installer for prompting the user for a Kerberos principal at the very beginning of the process it would be nice to be able to use that for all other steps which need authentication. In particular, the call to kdcregister to create the host keytab file should not need to also prompt the user for the principal and password. I quickly discovered this is a slightly awkward problem because the kdcregister command is issued via the baseinstall component which is called using the om command. When calling any component method om will firstly clear the environment to make it safe. This means that the path to the credentials cache, which is usually found from the KRB5CCNAME environment variable, is not available to kdcregister. I considered various approaches to solving this problem and eventually concluded that om needed a new feature which would be similar to the env_keep option support by sudo. This can then be used to specify a list of environment variables which will not be cleared. Using this feature is straightforward, the LCFG schema for om, which is inherited by most components, now includes an om_env_keep resource. The feature is supported in version 0.13.1 of the lcfg-om package which will be in the next weekly stable release.

LCFG Profile Security Project

April 18, 2018

Having completed the work to add support for GSSAPI auth to the client for fetching profiles I’ve now moved on to the LCFG installer. Currently the installer attempts to fetch the LCFG profile for the machine just prior to the (I)nstall, (D)ebug, (S)hell, (P)atchup, (R)eboot prompt. That fetching is done by calling the client component install method which in turn calls rdxprof in one shot mode. Having previously ported the client component to the Perl LCFG::Component framework I had hoped this would “just work” but it turned out that a number of bootstrapping issues were only being avoided previously due to many hardwired paths in the shell ngeneric code. The Perl framework takes a different approach and prefers to use the LCFG sysinfo resources wherever possible, this improves platform independence and maintainability but presents a bootstrapping problem at the first stage of the install when we have not yet downloaded any profile and thus have no sysinfo resources… I wasn’t keen on performing major surgery on the Perl component framework so I decided that the simplest solution to this problem was to get the installer to call rdxprof directly. With this change the installer worked again but still required support for Kerberos authentication.

Adding support for Kerberos authentication has been done in a fairly simple way. I’ve added support for two new install kernel command line options: lcfg.kauth and lcfg.realm. When the lcfg.kauth option is specified the user is prompted to enter their principal name and the kinit program is run to do the authentication. The user may specify the full principal name, if the realm is not specified then either the lcfg.realm option or the upper-cased domain name is used (e.g. @LCFG.ORG). If the authentication fails then the user is prompted to re-enter the principal name (which defaults to the previously entered string) and password. Once the Kerberos authentication has succeeded the credentials will be automatically used by rdxprof when required for fetching the LCFG profile.

LCFG Profile Security Project

March 28, 2018

This week I have been working on providing a way to configure the LCFG client profile fetcher via client component resources. In particular some sites will need to be able to specify SSL options (e.g. ca_path, verify_hostname) and also be able to set parameters for the authentication modules (e.g. gssapi might need the keytab file path to be specified). By default profile fetching will work for most sites without any additional configuration, furthermore as this is most easily expressed in terms of list and hash data structures I’ve decided to only support setting these parameters via a configuration file. Although it is currently configured entirely through the command line, rdxprof daemon supports loading configuration from a YAML file. I’ve altered the SetOptions method so that when it encounters a fetch entry in the configuration data hash it will pass this through to the LCFG::Client::ProfileFetcher instance via a configure method which knows how to handle the various options.

The current LCFG client component is written in bash which makes generating a config file in YAML more tricky than I would like. As we have a longstanding plan to rewrite all the core LCFG components into Perl this seemed like a good opportunity to get on with that job. I’ve previously been putting off this particular rewrite since the component is rather old and very complex. It manages the starting, stopping and signalling of the rdxprof daemon and as such it has a lot of code for handling PID files and checking for the liveness of processes. Given that we no longer support platforms such as SL6 and older this situation can be massively improved by switching to systemd for the management of rdxprof. I’ve introduced /usr/lib/systemd/system/rdxprof.service and /etc/sysconfig/rdxprof files which can be used by the component to control the daemon. To properly verify that the rdxprof daemon has successfully started the component creates a null callback and waits for it to be processed. I’ve moved the implementation of that into the LCFG::Client module itself so that the details are nicely hidden behind an API.

This is all implemented in perl-LCFG-Client version 4.3.4 and lcfg-client version 4.0.3. To make it easier to test I’ve added a dice/options/lcfg-client.h header. If the DICE_OPTIONS_LCFG_CLIENT_GSSAPI macro is defined then a new keytab will be created and the LCFG client will use it for authentication. The LCFG server is not yet quite ready for me to enable the use of gssapi but hopefully will be in the next couple of days.

Enabling gssapi for an LCFG client will be done something like this:

!kerberos.keys mADD(lcfg)
kerberos.keytab_lcfg /etc/lcfg/client.keytab
kerberos.keytabuid_lcfg root
kerberos.keytabgid_lcfg lcfg

!client.url mSET(

!client.fetch_auth mSET(gssapi)
!client.fetch_params_gssapi mSET(keytab)
!client.fetch_param_gssapi_keytab mSET(<%kerberos.keytab_lcfg%>)

LCFG Profile Security Project

March 21, 2018

After improving support for Apache authentication in the LCFG server I have moved onto the client this week. The bulk of the work has been focused on the creation of a new LCFG::Client::Fetcher module which encapsulates all the details associated with fetching XML profiles from various sources. As well as improving the authentication support I am taking the chance to overhaul a chunk of code which has not seen much love in either of the v3 or v4 projects. One particular issue is that currently the handling of the list of profile sources is spread around the client libraries, this means that even a small change can involve locating and altering many separate small pieces of code. This general work also includes adding support for IPv6, enhancing SSL security as well as making the code much more maintainable.

One big change in approach I’ve made is that the lists of local file and remote web server sources are now handled in a unified way where previously they were dealt with completely separately. The new Fetcher module has a single list of source objects (either LCFG::Client::Fetch::Source::File or LCFG::Client::Fetch::Source::Remote) which come from the value of the client.url resource. One advantage here is that it is now trivial to add an entirely new type of source (e.g. rsync or ldap) anything with an LWP::Protocol module is a possibility. When configured to use both local files and remote sources the client has always preferred local files where possible, this behaviour is retained by using a priority system with file sources being guaranteed to have a higher default priority than any remote source.

The other part of recent development work is the addition of support for different authentication mechanisms. This is supported via modules in the LCFG::Client::Fetch::Auth namespace, currently we have modules for basic (username/password) and gssapi authentication. As with the new source modules this approach means it is easy to support alternative mechanisms, including site-specific needs which might not be appropriate for merging into the upstream code base. Before making a request the Fetcher will call the relevant authentication module to initialise the environment. I am also working on supporting multiple mechanisms so that if one fails the next will be tried until one succeeds.

Most of the code for the client is now in place and I am working on documentation for the various new modules. Once that is done I need to consider how the necessary authentication information can make it from LCFG resources into the rdxprof application via the LCFG client component. Although I would rather not make such a big change it might be that I finally need to bite the bullet and rewrite the client component from bash into Perl.

LCFG Profile Security Project

March 13, 2018

I have recently begun work on the Review Security of LCFG Profile Access project. So far I have mostly been considering the various aspects of the project with the aim being to produce a list of ideas which can be discussed at some future Development Meeting.

The first aspect of the project I have looked at in more depth is the LCFG server which has support for generating Apache .htaccess files. These can be used to limit access to each individual LCFG profile when fetched over http/https. We have traditionally supported both http and https protocols and relied on IP addresses to limit access but would like to move over to https-only along with using GSSAPI authentication, the LCFG client would then use a keytab to get the necessary credentials. To help with this change I have introduced a new schema (4) for the profile component and made some modifications to the LCFG server code which makes it easier to use the Apache mod_auth_gssapi module. In particular there is new auth_tmpl_$ resource which allows the selection of a different template (e.g. the template which is provided in the package) which more closely meets local requirements. There are also auth_vars_$ and auth_val_$_$ resources which can be used to specify any additional information that is required. For example:

!profile.version_profile mSET(4) /* not yet the default */
!profile.auth          mADD(ssl)
!profile.auth_tmpl_ssl mSET(
!profile.acl_ssl       mADD(@admin)
!profile.auth_vars_ssl mADD(groupfile)
!profile.auth_val_ssl_groupfile mSET(/etc/httpd/conf.d/

which results in the the LCFG server generating the following .htaccess file:

AuthName ""
GssapiBasicAuth Off
GssapiBasicAuthMech krb5
GssapiSSLonly On
GssapiCredStore "keytab:/etc/httpd.keytab"
AuthGroupFile "/etc/httpd/conf.d/"
  Require user "host/"
  Require group "admin"

The profile.acl_ssl resource holds a list of users and groups (which have an ‘@’ prefix). In a real deployment it might make more sense to use an lcfg/ principal rather host/. The groupfile support is provided by the mod_authz_groupfile module which needs to be loaded.

I have tested this with curl and it works as required. The LCFG client doesn’t currently have support for doing a kinit (or launching something like k5start in the background) prior to fetching the profile so it isn’t yet possible to actively use this authentication method.

User management improvements

November 23, 2017

Management of local users and groups (i.e. those in /etc/passwd and /etc/group) is done using the LCFG auth component. One feature that has always been lacking is the ability to create a home directory where necessary and populate it from a skeleton directory (typically this is /etc/skel). The result of this feature being missing is that it is necessary to add a whole bunch of additional file component resources to create the home directory and that still doesn’t provide support for a skeleton directory.

Recently I needed something along those lines so I’ve taken the chance to add a couple of new resources – create_home_$ and skel_dir_$. When the create_home resource is set to true for a user the home directory will be created by the component and the permissions set appropriately. By default the directory will be populated from /etc/skel but it could be anything. This means it is now possible to setup a machine with a set of identically initialised local users.

For example:

auth.pw_name_cephadmin           cephadmin
auth.pw_uid_cephadmin            755
auth.pw_gid_cephadmin            755
auth.pw_gecos_cephadmin          Ceph Admin User
auth.pw_dir_cephadmin            /var/lib/cephadmin
auth.pw_shell_cephadmin          /bin/bash
auth.create_home_cephadmin       yes /* Ensure home directory exists */

auth.gr_name_cephadmin           cephadmin
auth.gr_gid_cephadmin            755