Following this mornings unplanned restart of one of the Forum SAN storage machines, various AFS volumes (where your files are stored) have been affected. We are bringing these back on-line a partition at a time. Unfortunately there is a delay as we check each partition for file system consistency, to make sure there’s been no corruption.
We are doing this manually so we can bring each partition and associated volumes back as they pass their checks. This means volumes will come back gradually for users, rather than having everyone wait until the last partition is checked.
However, the process of reattaching the checked partition, does mean a shortish break in access to the already recovered volumes, so please bear with us if your home directory has been recovered, but you find that every hour or so it seems to freeze for a couple of minutes while we reattach another recovered partition.
We are giving priority to user volumes first, followed by group volumes. Missing group volumes will affect web pages served from those areas.
Apologies for the prolonged recovery, but we’re going as fast as we dare.
Update: A report explaining the actions that caused the unplanned break in service, and the steps to restore the data is available at https://wiki.inf.ed.ac.uk/twiki/pub/DICE/ServicesUnit/incident-report-10-11-11.pdf