Petabyte Scale Storage

March 20, 2010

I’ve never come across the Ceph filesystem project before but it is Open Source and it has just been merged in for the 2.6.34 linux kernel. It claims “Ceph is an open source distributed file system capable of managing many petabytes of storage with ease.” It looks like there is still quite a bit of development work going on but it could be very interesting in the future if it manages to fulfill all its goals.

Configuration Languages

March 19, 2010

I spotted this blog post about using (or not) domain specific languages to customise programs. I can’t help feeling there is an interesting overlap here with the way we configure entire systems, we all face similar problems they are just at different levels. Just because a piece of software can be configured using the full power of Perl doesn’t make it a good thing (yes, I’m looking at you, RT…). LCFG deliberately has a minimal “language” for this very reason, it offers far fewer ways in which people can shoot themselves in the foot (as long as we ignore cpp).

F12 ntp

March 19, 2010

To keep kerberos happy you need your client machines to have their clocks fairly well synchronised with the KDCs. The easiest way to achieve this is to use ntp. I’ve added an LCFG header, inf/options/ntp.h which uses the file component to do a simple setup on F12. The file /etc/ntp.conf now just contains:

driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict -6 ::1
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys

As usual, one of the problems with the file component is that it cannot restart services after a configuration file has changed. So, once this is in place it is necessary to do /etc/init.d/ntpd restart.

If ntpd was not previously running (you can check first) then it is necessary to use chkconfig to activate the service:

# chkconfig --list ntpd
ntpd            0:off   1:off   2:off   3:off   4:off   5:off   6:off
# chkconfig --level 2345 ntpd on
chkconfig --list ntpd
ntpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

openafs on F12

March 10, 2010

Moving straight on from getting kerberized logins working it’s time to get openafs running. The packages for F12 are all pre-compiled and the official repository supports yum so that’s the easiest approach. Here is the yum repository config file (openafs.repo) for 1.4.11:

name=Openafs 1.4.11 for F12

Once that is in /etc/yum.repos.d, it is possible to do:

yum install openafs-authlibs openafs-client openafs-server openafs-krb5 openafs-docs

It is also necessary to grab a kmod-openafs package, for some reason I have experienced problems with the automatic support for this on F12 so it’s best to grab the correct version of the RPM for the running kernel from the openafs website and install it manually.

Once that is done:

echo > /usr/vice/etc/ThisCell
/etc/init.d/openafs-client start

it might be necessary to also edit /etc/sysconfig/openafs. I made it contain:

AFSD_ARGS="-dynroot -afsdb -fakestat -daemons 5 -volumes 200 -chunksize 20  -nosettime"

Network, Kerberos and openssh on F12

March 10, 2010

Firstly we need to deactivate the nastiness that is NetworkManager and switch to configuring the network interface so that it comes up at boot time and uses DHCP to get an address and DNS configuration. This is done by using the system-config-network tool as root and doing an “Edit” on the eth0 device. After finishing the alterations the networking needs restarting with /etc/init.d/network restart

To make sure this continues to work after a reboot, as root, do:

chkconfig --levels 2345 NetworkManager off
chkconfig --levels 2345 network on

As well as this, to get the machine to have the correct hostname and domain name, I had to edit /etc/hosts to look like:   localhost
::1         localhost bowmore

and set the domainname like:


The next step is to start using kerberos for authentication and LDAP for user info. That is done using the system-config-authentication tool, again as root. The LDAP base DN is dc=inf,dc=ed,dc=ac,dc=uk and I used ldap:// for the server.

For kerberos authentication the realm is INF.ED.AC.UK, I didn’t list any KDCs but rather ticked both options to use DNS. The admin server is

If you don’t have AFS available then on the “Options” tab you probably want to select “Create home directories on first login”.

I can never be bothered with typing in my password all the time so the next step is to get kerberos up and running and then configure openssh appropriately. Nicely Fedora finally includes all the patches provided by Simon which we have been applying locally for years so no rebuilding is necessary.

The next step is to grab the hostclient and host principals for the specific machine. If it is a new machine you will need to create it first, if it already exists then (as root) you can do something like:

kadmin -p squinney/admin \
            -q 'ktadd -k /etc/krb5.keytab host/'
kadmin -p squinney/admin \
            -q 'ktadd -k /etc/krb5.keytab hostclient/'

You can now configure openssh to work like a normal DICE machine. You will need to copy over /etc/ssh/ssh_config and /etc/ssh/sshd_config to your f12 machine. Note that the daemon config file is only visible by root. After reconfiguration restart the sshd.

It should now be possible to ssh in without a password!

Starting on F12/x86_64

March 10, 2010

I’ve made a start on the F12/x86_64 port. The first thing I did was to install from the F12 CD and make a base packages list:

 rpm -qa --queryformat '%{NAME}-%{VERSION}-%{RELEASE}/%{ARCH}\n' \ 
| perl -pe 's{/x86_64$}{}; > lcfg_f12_64_base.rpms

On this platform, at this stage, there are no packages with architectures other than x86_64 and noarch so I did not have to worry any more about getting the formatting correct.

The next stage was to get yum working with our local repositories:

su -
perl -pi -e 's/enabled=1/enabled=0/' /etc/yum.repos.d/* /etc/yum/pluginconf.d/presto.conf
cd /etc/yum.repos.d/
yum check-update

This deactivates any existing repositories in use and turns off the presto plugin which does the delta-rpm stuff which we do not need.

I have put together a yum configuration file for our Informatics F12 repository. Note that, by default, only the base directory is enabled. This makes it possible to easily install extra base packages with yum and know that the changes are directly applicable to the LCFG F12 base package lists. At a later point when updaterpms is installed and being run the updates can be applied.