April 25, 2008

With some help from Simon we now have the koji service mostly up and running at http://koji.inf.ed.ac.uk/koji/. It doesn’t actually do anything useful yet but there is some hope that it can now be battered into working the way we need. The code makes lots of assumptions that the world works in a way that mirrors the redhat network. For instance, they assume that the kerberos realm can be found from the last two parts of the FQDN for a machine. That works fine for foo.redhat.com but rather less well for telford.inf.ed.ac.uk, thankfully it is all written in python and is fairly easy to modify.

After further investigation I think the koji service generates mock configurations on the fly and it also appears that it doesn’t use external yum repositories. It appears that all the packages need to be imported for each distribution. I guess that koji has been designed around the idea of building an entire distribution. I’m not sure how this fits with us wanting to build a smaller set of packages targetted at one of “world“, “uoe” or “inf” for different platforms (e.g. fc6 and sl5). I need to get the service fully functional and then start experimenting.

One other thing I’ve done is that I’ve reworked the headers into two separate parts:

  • dice/options/koji-hub.h
  • dice/options/koji-builder.h

There is only ever one hub but it is possible to use any number of builder machines, for now both parts are on telford.


April 25, 2008

I came across this amusing cartoon, now where did Alastair put that EeePC?…


April 18, 2008

I’ve now started work on getting the koji service running on telford. There is a huge amount of information to get my head around so that I can actually understand the architecture and how it is all intended to slot together. The main references are a couple of wiki pages:

In some parts they go into great detail but in other places there is a complete lack of information so this is going to take a while… I’ve put together a few headers to get things working:

  • lcfg/options/git.h – For some reason koji needs the whole suite of git tools so I thought it best to put these into a separate header to aid reuse.
  • lcfg/options/mock.h – This adds the necessary mock packages and the mock LCFG component. It doesn’t set up any chroots by default, you need to do that yourself.
  • dice/options/koji.h – This is a start at configuring the koji system. I am currently working on the “put it all on one machine” principle, if necessary the parts can be split out later to cope with having a set of build servers. Currently this gets a PostgreSQL DB server running and apache setup with mod_python.

mock and package building

April 17, 2008

Recently I’ve been working on getting mock installed and configured on our new build server. The idea is to provide a set of chroots which authorized users can use for package building in an automated fashion. There are big advantages to doing things this way, for a start it only requires one machine to allow building for any RPM based platform (either i386 or x86_64 as long as the machine is installed as x86_64). It also requires packagers to be much more aware of their build requirements as the chroot is pretty minimal so unless the dependencies are well-specified the package won’t build. This should result in a much better quality of packaging which can be more easily distributed to other users and sites.

For reference, most of the information I needed to get mock correctly configured was taken from the fedora MockTricks wiki page. The most crucial bit I missed on the first pass was that on x86_64 you need to be careful with the yum configuration to exclude pretty much all i386 packages. If you don’t do that you get a chroot full of i386 packages which can cause interest problems, in particular file conflicts, when it comes to building some packages.

For each platform we are going to provide a set of chroots which have access to different package repositories. Taking sl5 as an example there is a basic chroot (named sl5-i386) which can access {distro,updates,extras,lcfg,world} buckets, there is a uoe chroot (uoe-sl5-i386) which has all of those buckets and can also access the uoe package repository and thirdly there is an inf chroot (inf-sl5-i386) which adds the inf bucket as well as uoe. The hope is that most packagers will use the basic chroot to minimize dependencies on locally built packages. Anything built in those chroots will have to go into the associated bucket, the packages should never be submitted to the lcfg or world bucket, for instance.

During testing of the chroots Simon and I discovered one unfortunate problem with the LCFG component source packages. The intention has always been that by bundling the various buildtools makefiles (e.g. os.mk, buildtools.mk, lcfg.mk) that the package could be rebuilt entirely independently of buildtools. This is not the case, without already having the lcfg-buildtools RPM installed it is impossible to rebuild an LCFG component SRPM. The only way this can be fixed is as part of the buildtools rewrite project.

UKUUG Spring Conference 2008

April 4, 2008

I recently attended the UKUUG Spring Conference 2008 in Birmingham. Primarily this was to give a presentation titled "System Configuration: An end to hacky scripts?". I think the attendance to my talk was pretty good, I reckoned somewhere about 50 to 60 people in the room. This year I had tried to come up with a mildly provocative title and I think it produced good results. I strongly believe that a good presentation involves a degree of showmanship, many a good talk is let down by the presenter being dull. Although clearly we always want to get the main message across, a little bit of humour (particularly at the beginning to grab the attention of the audience) and some effort put into the standard of the slides and delivery makes a huge difference. Whenever I am in the audience for a good talk, particularly anything on a subject in which I was not already strongly interested, I try to remember what it was that particularly made me sit up and pay attention, hopefully that will help me get better at giving these talks.

I think a good indicator that the talk went quite well was that we generated enough interest to hold a BOF in the afternoon on System Configuration and LCFG. Along with Paul, Simon and myself we had a further 10 people who were interested in sharing their experiences with different systems and finding out more about LCFG. It went on for nearly 2 hours so there was clearly lots to discuss. There was interest from a couple of people in getting the core of LCFG packaged for Debian and I am hoping that leads somewhere. Being a Debian Developer myself I’ve always hoped that one day we could get LCFG into the Debian archive. From the BOF and other discussions I had with people, something that is now apparent to me is that we really need to work on having a “How to Get Started with LCFG” page on the LCFG website. I think this could be based almost entirely on the LCFG workshop we held in June 2007.

For me the “scripting languages” theme for this year was not that exciting and as such I didn’t attend as many talks as normal. One of the most interesting ones was "Feeding the BBC Homepage" where they talked about the use of the Catalyst web framework for Perl in the BBC and a bit about their software development process. I already knew that the BBC was a big Perl user but I was unaware of their use of Catalyst, that gives me a lot more confidence in the capabilities of the framework. The other one that I enjoyed was "Today’s Software … Is It Really Bloated?", this was a perfect opener for the morning after the conference dinner and it is a shame they didn’t make it a plenary session. The results were not particularly scientific in most cases and, particularly with the examination of the kernel, rather out-of-date but it did give a general feeling of how things are changing for the better or worse. The timetabling was, in general, a little odd this year which didn’t help with attendance to talks in different streams and the almost complete lack of plenary sessions was surprising, over the last couple of years we have had good talks from google employees in that slot.

Location-wise I thought the venue was excellent, it was within easy walking distance of the railway station and it was surrounded by a range of suitably priced hotels to cater for all tastes. The main lecture room was fine but sadly the venue was really let down by one of the talk rooms being in a classroom, with a huge pillar in the middle, rather than a proper lecture room with tiered seating.