A presentation on progress on the Inventory Project was given at the 5th April development meeting. The slides are available here.
Clientreport and PostgreSQL’s JSONB datatype
I’m a real convert to the JSONB data type. We’re using this to store the JSON clientreports submitted by each client. Using JSON allows us to trivially add more information (eg extra reports) and then make queries against both the data in the JSONB row and data in other rows (and tables).
The following SQL code demonstrates the power – it returns a list of mac addresses (with some descriptive info) from the clientreports that we haven’t already recorded in the macaddr table.
WITH hostmac AS (SELECT hostname,
jsonb_array_elements(data->'network'->'nics')->'name',
jsonb_array_elements(data->'network'->'nics')->>'macaddr' AS macaddr,
data->'dmi'->'system'->>'system-serial-number' as sno
FROM clientreportjson
)
SELECT hostmac.hostname,macaddr,sno,i.item_id,i.description FROM item i
JOIN hostmac ON (
hostmac.sno = i.serial
)
WHERE macaddr::MACADDR NOT IN (SELECT macaddr from macaddr);
Experimentation with different JSON representations revealed that where you have data on multiple items of the same type (for example several NICs on the same machine), it is better to group these together in a list rather than in a hash. This makes for easier SQL processing of the data (eg using the jsonb_array_elements() operator).
Inventory – REST api
As reported in the previous blog item, an important element of the new inventory system will be the RESTful API. Currently, the invquery and invedit commands use SQL directly to perform queries and updates. This has integrity “issues”, means that changing behaviour can be laggy (code needs updated on clients), and means that other code that wants to do queries or updates must duplicate the code. A RESTful API will overcome these problems.
After reading numerous web articles and blogs on REST API design, and being confused by the differing views on what constitutes good API design, I have produced a draft API design. The API will be available in two forms – one which is completely un-authenticated but allowing only queries, the other being GSSAPI authenticated for both queries and updates. (The original intention had been for the GSSAPI authenticated tree to provide a login method issuing a short-lived token to be used by the client for subsequent operations (which wouldn’t be GSSAPI authenticated). However it turns out that GSSAPI for every operation isn’t that expensive and has the benefit of significantly simplifying the code.
The intention is to use perl-Catalyst-Action-REST to implement the API.
Inventory status – May 2016
The Inventory project was effectively stalled for January to March, to release effort to work on the SL7 Server base project. In an attempt to improve focus on this project, a deadline of end of 2016 has been set for introduction into service.
Service in parallel – incrementally adding features
The aim is to run the new inventory system (Tartarus) in parallel with the existing system and incrementally add features over the coming year. Order files are now sync’d across (currently nightly) to Tartarus and updating appropriately. The new system has been ported to SL7 and upgraded to PostgreSQL 9.5 (on host ‘bandama’). Thanks to Toby for building perl-DBIX-Class, and deps, to SL7.
Inbound supplier reports (via email) – used to populate mac address tables
Information on the kit purchased off the Select PC/Laptop scheme is available, via periodic reports, from the suppliers. This includes information on purchase order, serial number, primary mac address. The intention is to master mac address information in Tartarus, and not LCFG profiles – feeding these supplier reports into Tartarus will assist in automating this process.
A generic kerberised curl script (krbcurl) has been written which can be run as part of a mail server alias rule – this submits the incoming email to a CGI script (running on the Tartarus server) over https using GSSAPI authentication. The CGI script feeds the incoming mail report, based on the From address, to a manufacturer specific module which decodes (ie takes apart the MIME structure) the report and updates the Tartarus supplierreport table appropriately. This is now running live, with scheduled weekly reports from HP and manually invoked reports from Dell.
Logging
A simple centralised logging library has been written so that all code running on the Tartarus server (including CGIs) logs into one SQL event log – much like syslog (log facility, log level, message, time) . This has proven invaluable in debugging problems and should assist in debugging problems when in service.
Clientreport – more work
Chris has written some more clientreport modules – reporting on firmware, PSUs, BMC info.
The upgrade to PostgreSQL 9.5 introduces the JSONB data type. Storing the clientreport JSON in a JSONB column means that we can combine queries on data in SQL tables with data held within the JSON data structure.
Next work
An important element of the new system will be the RESTful API. Currently, the invquery and invedit commands use SQL directly to perform queries and updates. This has integrity “issues”, means that changing behaviour can be laggy (code needs updated on clients), and means that other code that wants to do queries or updates must duplicate the code. A RESTful API will overcome these problems. The next significant task is to define an API and circulate for comment.
Inventory project – status (November 2015)
In the last few months, the following progress has been made on the Inventory project :-
Orders file processing
- The format of order files has been “formalised” and documented
- A new parser library has been produced, code reviewed and tested against test data.
- The existing order files have been cleaned so that they confirm to the documented format (by validating against the new parser)
- The new parser has been integrated into the existing system so that order files are parsed post edit thus ensuring that the quality of the data is maintained
Order sync
- Code to synchronise order files with the Inventory database has been produced
- A test suite of sequenced order data snapshots has been produced to represent the various transitions of an order (order, delivery, correction etc)
Clientreport
- A new modular clientreport mechanism has been implemented
- new client probing modules can be added to harvest data, without modifying the receiving server code
- the client submits a JSON data blob (combining the results of the probing modules) and submits to a kerberos authenticated CGI on the server
- The received JSON blobs are currently stored in a VARCHAR as the version of perl-DBIX we currently have installed does not support JSON datatypes
Feeds from School database
- There has been some discussion on how we can feed information from the School Database (eg people and location) to the Inventory.
Detecting monitors (for the clientreport script)
The current clientreport
script uses a rather crude method to determine the monitor connected to a DICE desktop. It greps through /var/log/Xorg.0.log
to pick out the monitor model name, serial number and year of manufacture.
Whilst this has worked for many, many years, it is obviously rather risky. That log file is not designed to be processed and the format could change at any time. Moreover, the current code only reports on one monitor and doesn’t work for monitors attached via an nVidia graphics card.
The obvious solution was to detect monitors using the “proper” mechanism – that is, to read and decode the EDID data for each monitor. A monitor’s EDID data is usually available via /sys/class/drm/{port}/edid
. On machines with nVidia cards (at least using the nVidia drivers), the EDID data is not available via /sys/class
and must instead be obtained using the VBE interface. This is done using the monitor-get-edid
script which is part of the monitor-edid
package. However, only one monitor can be detected using this method.
Having obtained the EDID data, it can be parsed either by edid-decode,
part of the xorg-x11-utils
package, or monitor-parse-edid
which is part of the monitor-edid
package. Unfortunately, neither of these scripts display the serial number of the monitor, so it was decided to ship a patched version of the monitor-parse-edid
script with clientreport
which does display the serial number and other useful information.
The new clientreport
script first attempts to locate the EDID data using /sys/class
. If that fails it falls back to using the VBE method.
Harvesting MAC addresses
The clientreport
script of the current inventory system only reports on one MAC address, and even then simply derives this from the dhclient.mac
resource.
As we intend, in the new inventory system, for the clientreport
script to be the principal route for determining the MAC address of a machine, it must somehow obtain the MAC address directly from the Linux kernel.
There are numerous WWW postings documenting a variety of different ways to enumerate the physical network interfaces of a machine. The quickest route would appear to be to make use of the fact that only physical network interfaces have a file /sys/class/net/{iface}/device
. Each of these interfaces will have a file /sys/class/net/{iface}/address
the contents of which are the MAC address of that interface.
For most machines, that’s the end of the story – but not so on machines with bonded interfaces. When two interfaces are bonded, by necessity one of the pair takes on the MAC address of the other interface – so you end up with two interfaces with the same MAC address. As far as I could see, there is no way to determine the permanent MAC address of the interface whose MAC address has been altered – at least not by reading files in /sys
nor by using the majority of network tools. Even the ioctl()
call SIOCGIFADDR
returns the modified MAC address. Once I discovered the last fact, I almost gave up with finding a solution, assuming that all tools would use either this ioctl()
call, or read /sys
, to determine the MAC address.
However, persistence paid off. I wondered how OCSinventory
solves this problem and took at look at the source code for the OCSinventory
agent. A quick peruse of this code showed that it makes use of ethtool
to determine things like connection speed etc. A peek at the ethtool
man page suggested ethtool --showpermaddr
might do what we wanted. Fully expecting this to return the wrong address, I was pleased to find that it does indeed return the correct permanent address of an interface even if that interface is a slave in a bonded pair. So, how does it do that if ioctl(SIOCGIFADDR)
and /sys
return the wrong answer? It turns out that ethtool
has its own private ioctl()
call called SIOCETHTOOL
.
Inventory clientreport via kerberised CGI using host key principal
For many, many, years, DICE desktops and servers have submitted a daily report to the Informatics inventory system. This report, known as ‘clientreport’, includes details such as the model type, the running kernel, the amount of physical memory, the time of the last successful updaterpms run and the details of any attached monitor.
In the current inventory system, clients submit their report directly to the database using an SQL UPDATE, secured by a ‘secret’ shared password. This is not exactly considered “best practice” and it was decided that, for the new Inventory system, we should update the ‘clientreport’ script to use 21st century technologies!
The proposed system is to serialise the report data using JSON and submit the JSON using an HTTPS POST to a CGI script running on the Inventory system. The POST will be made over a kerberos authenticated connection, using the client’s host principal. The CGI script can then deserialise the JSON and update the database as necessary.
You might expect such an architecture to be in common use in DICE, but surprisingly this turns out not to be the case. Certainly I failed to find any example of a Perl implementation. Fortunately Stephen had some proof-of-code snippets to start from, and a prototype was (reasonably) soon produced.
On the client side this uses Authen::Krb5 to effectively ‘kinit’ using the client’s host principal. LWP::UserAgent is then used to make the HTTPS POST, with LWP::Authen::Negotiate providing the GSSAPI connection.
On the server side is a standard lcfg-apacheconf managed web server, using mod_auth_kerb for authentication. (Obviously, a service keytab needs to be created using the lcfg-kerberos component – and this keytab needs to be owned by the apache user).
A couple of problems were encountered. Firstly, the documentation for LWP::Authen::Negotiate fails to mention that it expects the kerberos service name to be ‘HTTP@hostname’. On consideration, it’s obvious that it must make such an assumption, but it would have been nice for the documentation to state it. Secondly the documentation for mod_auth_kerb states incorrect default values for some resources. The following resources were found to work.
AuthType Kerberos KrbServiceName HTTP Krb5KeyTab /etc/httpd.keytab KrbMethodK5Passwd On KrbVerifyKDC On require valid-user
How to boot under F12
Fedora made a big noise about moving to Upstart from the old SysVinit system. But, in reality, they haven’t really. OK, Upstart’s init daemon is being used, and this starts Upstart “jobs” in /etc/event.d, but only a handful of services are started using Upstart “jobs”. The vast majority of services are still started using init.d scripts, called from Upstart “jobs” which simply emulate the old SysVinit rc.d mechanism.
So.. for F12 we have to consider how we’re going to control how services and LCFG components start at boot time. There appear to be three choices :-
- We don’t bother with upstart at all and replace it with the old SysVinit and start the LCFG boot component as we have done with previous platforms. We would need to find some way to start those things that ARE started using upstart (ie have /etc/event.d/ files) and there’s a possible problem if services migrate to Upstart during the life of FC12 (unlikely).
- We could write an upstart component to manage /etc/event.d, but it’s not easy to see how to configure this from LCFG resources, given that the configuration info for Upstart “jobs” are embedded within the “jobs”‘ code. Given that the majority of services are still controlled using an init.d mechanism, we would still need a mechanism to control these.
- We could replace the Upstart “jobs” which emulate the old SysVinit rc.d mechanism with “jobs” which call the existing LCFG boot component. This would involve shipping a patched version of the initscripts RPM, but would be the easiest option.
After some deliberation and discussion we decided that, given that Fedora is still in a state of transition to Upstart, we should chose the least cost option (3) for F12.
LCFG installroot – groan, no Union FS in Fedora
I had been hoping to use a combination of a union filesystem (UnionFS or AUFS) with something like SquashFS for the new LCFG installroot, with writes going to the initramfs.
However the stock Fedora 12 kernel doesn’t include any kind of union filesystem (because there is no such filesystem in the mainline kernel). It doesn’t look like mainline kernel support will come any time soon – see http://lwn.net/Articles/327738/. So, given that we want to use stock kernels, think we’ll stay clear of this route for now…
Shrinking the LCFG installroot
Have recently started to look at re-engineering the LCFG installroot. One of the main aims of the project is to remove the need to run with ROOT_NFS. The current PXE installroot system mounts the whole LCFG installroot over NFS and switches root to this NFS mount. Redhat (and Fedora?) kernels don’t ship with ROOT_NFS, so using ROOT_NFS means that we have to roll our own kernels: a time consuming manual process.
One possibility is to forget NFS and fetch across (using http or tftp?) a squashfs image of the installroot into tmpfs.For this to work efficiently, we would want the LCFG installroot to be as small as possible. The current LCFG installroot is around 549Mb (187 Mb when squashfs’d using /sbin/mksquashfs). The current LCFG installroot package list was hand-crafted quite a while ago, and contains quite a few packages which aren’t needed. Using yum (on lcfg_sl5_lcfg_installroot.rpms) we can produce a more accurate list of those packages that are really required – a quick experiment indicates that the installroot would shrink to 524Mb (squashfs 178Mb).
Another option is to create a root in tmpfs and NFS mount top-level directories in this root (eg /bin, /usr etc.). This would allow us to drop ROOT_NFS whilst still storing the installroot on NFS.