using mock to test rpm packages

Our PkgForge service is a locally developed build farm. We use this for automatically building .src.rpm files for the multiple platforms and architectures we often have to support concurrently . The service uses Mock underneath. It works very well for known working packages and a “send and forget” approach. It is not so good however for debugging build issues (although that was never its intended purpose). Really hit this recently when trying to build a new locally developed package for wider distribution. Being used to only having a few issues to resolve normally (commonly just missing pre-requisite packages), started off using the build farm service. But the numerous iterations needed and the slow rebuild cycles were getting tedious. Eventually an issue that really needed direct access to the build root to debug was the final nail in the coffin. Previously been put off using Mock manually by assuming that it would be complicated. Turns out it is an absolute breeze and definitely the way to pre-test, or even final build if you want speed and controlled (re)submission, new packages (particularly when you only need to worry about distribution for one architecture and platform). So here is a brief howto (as root).

yum install mock

Mock comes with the necessary configuration file for our main current platform and architecture  already, so in principle you need do nothing more than:

mock -r epel-7-x86_64.cfg --rebuild PACKAGE.src.rpm

You can then follow the output live as it builds. If you are running this on your desktop you will also find it runs refreshingly fast. The build root and results are also kept around for subsequent analysis in:

/var/lib/mock/epel-7-x86_64/root
/var/lib/mock/epel-7-x86_64/result/{build,root,state}.log

Commonly you may need to include our own local repositories. This is also simple, for example to use our “inf” repository:

cp /etc/mock/epel-7-x86_64.cfg /tmp

Edit and add the following within the set of repositories :

[el7inf]
name=el7inf
baseurl=http://http.pkgs.inf.ed.ac.uk/rpms/os/el7/x86_64/inf
priority=1
enabled=1
gpgcheck=0

then:

mock -r /tmp/epel-7-x86_64.cfg --rebuild PACKAGE.src.rpm

If you get failures during build you can simply chroot and test the build environment, do further debugging and manual package/file installations and rebuilds:

chroot /var/lib/mock/epel-7-x86_64/root

Note to the wicked: if you are using Java you are likely to find that after doing the chroot you also need to do:

mount -t proc none /proc

PkgForge can automatically build multiple locally dependent packages in one go. If you need to do this with a local mock setup then you need to manually build each package one by one and submit them so they are accessible to mock (or you could investigate the feature of mock that means it can dynamically generate and update a yum repository from the results of previous builds).

Python Command Scripts

You would think trying to install and package a Python package and module with a command script would be easy. Turns out this is a twisty minefield on top of shifting quicksand just itching to catch the unwary. Found this article though which was most helpful.

Adopting Theon – Part 1

Theon is a Schema Version Management system for PostgreSQL databases. So the first question to ask yourself is why you would want to bother with such a thing at all.

Why use Theon?

  • your data model is in constant flux, needs to be adapted frequently and responsively to meet new requirements
  • your data model is large and complex
  • you need to manage different instances of your data model for production and development servers, for handling continuous release/testing cycles, for deployment at multiple organisations
  • many different people are concurrently working on making changes to your data model
  • your data model needs to be kept in alignment with other outside systems, you need it to be automatically maintained based on the requirements of those systems
  • you want to develop your data model at a higher abstracted layer than physical database tables, columns and relationships
  • you or others need to maintain/upgrade external applications so that they remain in alignment with specific versions of your data model
  • others maintain independent systems/applications using your data model and need a robust/automatic upgrade path

If you want/need to do any/all the above then Theon  may help you out. If not then adopting a Schema Version Management system might still be the right thing to do – after all it is considered sensible to version control source code so why not databases, see Database Refactoring) – but you will probably get less immediate benefit.

There are other Schema Version Management systems that support PostgreSQL: Liquibase, Flyway, sqlHawk, MIGRATEdb, schema-evolution-manager, Alembic, Sqitch, DBSteward. All these vary in functionality, complexity and price. Follow the links and make your evaluation of the pros and cons of each depending on your own specific needs. Architecturally Theon is probably closest to DBSteward.

Why choose Theon?

  • open source under GPLv3
  • unix command line scriptable tool chain
  • extensible
  • change sets are just SQL DDL
  • built in graphical schema editor
  • make changes by direct DDL manipulation on development database instances
  • import and go

Why not?

  • immature
  • evolving
  • only supports a subset of the PostrgreSQL feature set

The second part of this article examines what you need to get started using Theon. The final part will demonstrate managing a database in Theon.

make comments on criteria available

Another (feature creep) request for the new system. It looks likes the mechanism is there and this will be easy to bolt on subsequently in which case it is unlikely to be done with this particular project.

Returning feedback to students wasn’t part of the  original project description, rather a requirement that has arisen in the meantime. But the fact that the reports are now stored is surely a  big step in providing that feedback. When we agreed at Teaching Committee that information from the reports  should be returned to students as feedback, I pointed out that it would probably not be possible to do it in June 2015, but I hope we will have something in place for June 2016.

Achieving this with the new system is probably just a case of Webmark also producing a
reduced version of the form just including the relevant fields students can see and then a specific index page for access by students that shows them that form.

Comment: That would be perfect! The “relevant fields” are:

  • Individual marking form: Comments on the criteria
  • Agreed mark form: nothing
  • Moderator’s form, if any: Comments on the criteria

I guess that an ideal “specific index page for access by students” would  be to include it on student.inf.ed.ac.uk but with a time delay so that  it is only accessible once the examiners’ meetings have passed.

Modeling Part Completed

The documentation for TheonModel (managing the physical database schema and related configuration for TheonCoupler, TheonUI and TheonPortal) has been completed and including the illustrations. It is in principle a first draft, but its been through a number of revisions so its a fairly solid first draft. The command toolchain is clearly defined now so we can start implementing that – all the underlying technology has been done already, or at least proven in principle, so its just tidying and wrapping it up properly here to do. The documentation for TheonCoupler is practically done as well now, should be completed within the next few days. Still got architectural documentation to do on TheonUI and TheonPortal to do (bits and pieces on usage and configuration are there already under TheonModel). Need to write the release/package management documentation as well (although the toolchain for that is at least already defined and coded up). Finally the system installation/configuration documentation needs to be done – although that is a little bit in limbo to see whether we can get a viable standalone LCFG config up and running. The remaining documentation will proceed in parallel with getting the TheonModel toolchain implemented as that is really the primary goal of this project.

Started back on Documentation

So after a long break (I think May was the last time this was touched) got back into doing the documentation. It probably took a day just to orientate to where we were. Anyhow the main purpose of doing the documentation before the technical work is complete is to pin down the workflow and hence what the commands and options will need to be provided by the wrapper toolkit. This so far has been quite successful in identifying any gaps in the technical implementation (due to the attempt to reflect real use cases).