Trusting a local CA

For some of our services we use SSL certificates which are signed using the local university CA. On Redhat systems to get applications such as firefox to trust this CA we have always had to hackily patch and rebuild the nss packages. This has always been a rather undesirable situation since it means we need to ensure we keep up-to-date with security releases. Also, every time there is a new release the package tends to change just enough to mean the patches need reworking, this all means that our solution to this problem has been rather fragile.

Thankfully Redhat have now provided a nice solution which makes it trivial to solve this problem. They have enhanced the ca-certificates package to provide an update-ca-trust script, see the Redhat announcement for full details. What it now boils down to is the simple process of placing your pem or DER file into the /etc/pki/ca-trust/source/anchors/ directory and then running the update-ca-trust script.

To make it even easier the local eucs-sslcerts package will be updated to store the CA file into the correct location and run the script from its post-install script.

As an added bonus, the eucs-sslcerts package will now be included in the LCFG installer so that it is possible to trust LCFG servers using https with a locally signed certificate.

Posted in Uncategorized | Leave a comment

Describing features of a system

Recently I have added a new concept to the sysinfo component which allows admins to describe general features of a system. These are simple arbitrary flags which indicate the presence (or possibly lack) of some feature on a system. The aim of the sysinfo component has always been to provide a place in which all general information can be gathered. Previously we had support for an extensible list of standard paths along with a static set of parameters which describe things such as the operating system, the architecture and various inventory information. Notably there was a lack of a locally extensible way of describing other general features.

To understand why this is useful it helps to see where this requirement came from in the SL7 project. A while back we became aware of a problem with the way we generate logrotate configuration files for LCFG components on SL7 (see bug#812). Whenever a component configure method is called the ngeneric framework attempts to build an associated logrotate configuration file using a template. Annoyingly, due to the way we have the permissions set on our log directories (which grant write access to the lcfg group) the latest version of logrotate requires the use of a new su directive. There are various ways we could have approached this problem. We could have made this optional based on the version of the OS (e.g. on for SL7 but off for SL6) but this runs the risk of the newer version of logrotate being added to the next minor release of SL6 (it also reduces platform-independence if not done carefully). We could also have made this optional based on the version of logrotate, this seems quite sensible but we’ve learnt the hard way that Redhat like to backport software features without changing the major version (just incrementing the package release string instead). This makes it very hard to spot that some new feature is required. So, we introduced the new sysinfo concept of features and the first feature is logrotate_needs_su. The ngeneric framework queries sysinfo to check for that feature and the template is then processed appropriately to include (or not) that configuration directive. A nice bonus here is that if we ever decide to stop granting write access for the lcfg group on that directory we can just remove the feature and everything will work correctly.

Further information is provided on the LCFG wiki describing the sysinfo resources and how to query sysinfo from LCFG components and scripts.

Posted in Uncategorized | Leave a comment

Systemd and buffered output

Kenny MacDonald submitted LCFG bug #799 reporting that the output of updaterpms appeared to be buffered at boot time. He is correct, and this behaviour is particularly noticeable when installing a large-ish number of substantial RPMs – it is not at all obvious that the system is doing anything productive. There is a real danger that a desktop user may decide to power-cycle/reboot his/her desktop should they encounter this behaviour.

By default, systemd units are configured to output to ‘journal+console‘. This means that output is sent to both journald and to /dev/console. For some reason, with this configuration, output to /dev/console is buffered (multiple lines). If a unit is configured to output to ‘tty‘ (which is equivalent to ‘console‘), lines are only singly buffered. I can only guess that systemd is blocking on logging to journald, and the output to /dev/console is blocked behind this.

The only obvious solution is for lcfg-updaterpms to output just to ‘tty‘. This does mean that we lose the ability to log useful error messages that are produced during the updaterpms run (by install scripts), but we haven’t captured those error messages in the past.

The current solution (work-around, really) is to output to ‘tty’ by default, but allow buffered output to be enabled by use of

#define LCFG_UPDATERPMS_BUFFERED_OUTPUT

at the top of a machine’s profile. Buffered output could usefully be enabled on machines used for release testing.

Posted in systemd | Leave a comment

Which display manager? A decision… for now

An earlier post described the dilemma facing us with respect to the choice of display manager for EL7.

Well, the decision has been made much easier by the existence of a bug in GDM described at https://access.redhat.com/solutions/960583 and https://bugzilla.redhat.com/show_bug.cgi?id=1092274. Under certain circumstances, GDM will prompt twice for the user’s username. This can easily result in a user typing his password in clear text into a field expecting a username. This is such an obvious security flaw that it is quite depressing that Redhat have sat on this bug since July 2014.

So, we will default to LightDM for at least the School of Informatics – for now.

Posted in lightdm | Leave a comment

Sleep

The LCFG sleep component has now been adapted for EL7 platforms. Here’s a summary of the main changes.

The ever-growing init software systemd has its own sleep capabilities, replacing those of pm-utils. While pm-utils is still available in EL7, its use is deprecated and it will be removed in a future release.

For the sleep component the shift from pm-utils means two changes: the command to initiate sleep changes (from /usr/sbin/pm-suspend to /usr/bin/systemctl suspend) and a different mechanism is used to provide the sleep.suspendcommands and sleep.resumecommands hooks which run arbitrary commands at suspend-time or resume-time. Although systemd presents several possibilities for integrating the hooks, putting an LCFG-built script into the /usr/lib/systemd/system-sleep directory seems the best option, both because it’s the simplest (no need to concoct any systemd configuration files) and because it’s really very similar to the pm-utils method already in use (a shell script in /usr/lib64/pm-utils/sleep.d/). This solution also means that commands can be run one at a time, avoiding the usual aggressive parallelisation of systemd.

Since the sleep component still has to work on both SL6 and SL7 platforms it now creates whichever hook script is applicable on the platform on which it finds itself. It uses a templating system to make the script. The LCFG templating system is also now deprecated, so the component has been altered to use the recommended Template Toolkit.

The sleep component is affected by the advance of systemd in another way: the systemd tentacle logind has taken over management of user logins. This replaces ConsoleKit as the service used by the sleep component when making enquiries about machine idleness. ConsoleKit is queried using its DBus interface. Although logind also has a DBus API it proved more straightforward for the component to use its command line interface.

While systemd has brought a lot of change, it hasn’t changed everything (yet). The kernel wake alarm is still the same, and seems to work just as well. This is the facility which is used to wake machines at a certain time – usually in order to run a particularly important cron job. The cron jobs themselves are also to be found in the same places, and in the same formats, and the sleep component recognises and parses them as correctly on EL7 as on SL6.

Giving devices the ability to wake a sleeping machine is similar too. The new OS version still uses /proc/acpi/wakeup to list devices which may be wake-enabled. However on EL7 some devices cannot be wake-enabled at all, so the sleep component now merely does its best to enable as many devices as it can, rather than insisting on enabling them all.

The new lcfg-sleep – now grandly numbered 1.0.0 – is available for EL7 machines, and for SL6 machines on the develop release. After a period of testing it will be rolled out to other SL6 machines.

Posted in Uncategorized | Tagged | Leave a comment

Which display manager to use for EL7?

We chose to use KDM, rather than GDM, for Scientific Linux 6 because it allowed us to tweak the behaviour of the login screen in a number of ways :

  1. Change background image (used, for example, to distinguish between exam and non-exam mode in student labs)
  2. Disable/enable power/shutdown/hibernate options (to stop users shutting down PCs in AV podiums)
  3. Hooks to run scripts, as root, at login and logout (to change device permissions)
  4. Disable user login list (we have far too many users for this to be practical)

Unfortunately, development of KDM has ceased and there is no version available for EL7, so we had to find an alternative.

We decided to investigate whether the current EL7 version of GDM provides the above required functionality. We found that 4) was simple to disable using dconf and 2) is possible using polkit configuration. We expect that changing device permissions at user login/logout, 3), should be possible using udev configuration – we have yet to confirm this. As far as we can see, 1) is not possible. Whilst it is possible to change various elements of the login screen using dconf resources, it is not possible to change the background image. The GDM login (“greeter”) screen is built up using CSS elements – the background image is hard-wired to be noise-texture.png in /usr/share/gnome-shell/theme/gnome-shell.css. It appears that the only way to modify the the “greeter” screen is to modify this file,  owned by the gnome-shell RPM. GDM also has some “usability” issues. Many of the prompts are displayed in too small a font – impossible to change without making changes to the CSS as above. Tab completion for user names has been deprecated, which results too easily in one typing one’s password in the Username box.

The  LightDM display manager is available in EPEL for EL7. This provides all the required functionality listed above – an LCFG component has been produced to configure these. However, LightDM has its own problems. Gnome 3 has done away with a separate screen lock program – locking is now provided jointly by GDM, gnome-shell and gnome-session. LightDM does not itself support screen locking – later versions do, with LightLocker – so a separate screen lock program is required.  We have built xscreensaver (version from Fedora18) for use with LightDM – it’s not yet clear how we configure sensible defaults (eg blankscreen for screensaver) for this without modifying the RPM.  Another concern with LightDM is power-management – this needs further investigation.

Summary – it appears that we can do more with GDM, under EL7, than we could under EL6. Using GDM feels like a more sustainable approach as it is the official out-the-box solution, but can we live without the ability to change the background login screen and those usability issues?

 

 

Posted in lightdm | Leave a comment

Replacing boot.run

With the demise of the LCFG boot component we have lost the boot.run facility. This was a useful place to hang jobs which needed to be run on a daily basis, for example, the daily run of updaterpms to update the installed set of software packages. It provides a very simple and convenient way of running a list of LCFG component methods and shell commands in the specified order on a daily basis at a time which is appropriate for the machine.

Although we could just replace this with a simple cron job for each task, such as updaterpms, we see this as an opportunity to enhance the facility to provide other functionality, in particular, it would be nice if we could have an anacron-like service. The anacron tool ensures that jobs run with a particular frequency (e.g. daily, weekly or monthly), to do this it keeps a timestamp of when the job was last run and then compares that to the current time to see if the interval has been exceeded. We cannot easily just use anacron as we also wish to limit when certain jobs will run based on the time of day, for instance, we do not usually want to have updaterpms run during office hours as it can inconvenience users.

It is likely that most of our regular system-maintenance cron jobs could be grouped so that they only run at certain times of day. They rarely ever need to run at a particular, specific time, just stating something like “morning”, “afternoon” or “evening” might be sufficient. If the distribution of cron jobs could be reduced to something like 4 groups per day then there is the opportunity for lightly used machines to sleep for long periods rather than the current situation where they have to wake up for cron jobs quite a lot more frequently.

At the moment I’m just at the stage of thinking about what functionality we can provide so now is the time to chip in with your thoughts.

Posted in Uncategorized | Leave a comment

Repository changes and yum/mock config

I have finally finished restructuring our internal packages repository so that we now have paths to the SL6 and EL7 local packages directories which include the architecture as an element. So we now have:

  pkgs/rpms/os/sl6/i386/{devel,inf,uoe,world,lcfg}
  pkgs/rpms/os/sl6/x86_64/{devel,inf,uoe,world,lcfg}
  pkgs/rpms/os/el7/x86_64/{devel,inf,uoe,world,lcfg}

  pkgs/srpms/os/sl6/i386/{devel,inf,uoe,world,lcfg}
  pkgs/srpms/os/sl6/x86_64/{devel,inf,uoe,world,lcfg}
  pkgs/srpms/os/el7/x86_64/{devel,inf,uoe,world,lcfg}

It’s a bit weird having the architecture in the srpms directory path but there’s no better way to do this at the moment due to the limitations of the pkgsubmit tool. Ideally there would be one shared directory for all SRPMs for a distro.

This makes it possible to hugely simplify our yum and mock configurations, in particular we now have a single SL6 configuration for each “bucket” rather than one per-architecture, so the number of repository configurations required has halved. The new configuration also makes much greater use of the $basearch and $releasever yum variables rather than trying to achieve everything through LCFG resources, sometimes it’s better to use the support built-in to the standard tools…

I have also taken the opportunity to push the yum configuration for SL and EPEL repositories into the lcfg level headers so that everyone gets these automatically. In the short term this might break a few configurations but the fixes should be simple and the long-term benefit definitely outweighs the short-term costs.

Posted in Uncategorized | Leave a comment

Systemd – LCFG documentation

A start has been made on LCFG systemd documentation.

The Systemd Cookbook documents common queries / recipes.

The lcfg-systemd document is a more full description of LCFG systemd.

Posted in systemd | Leave a comment

Systemd – fun with targets dependencies

If you have a target B.target which wants/requires S.service and you want S.service to start after A.target – it isn’t sufficient to state B.target ‘requires’ and ‘after’ A.target. If you just do that, S.service will start based on its own requires/afters – usually much earlier than after A.target. You also need to state that S.service ‘requires’ and ‘after’ A.target.
Posted in systemd | Leave a comment