On Tuesday 4th of March there was a period of a few minutes, from 11:15am to about 11:25am, when most peoples’ home directory quota was incorrectly shrunk to 2MB. Anything trying to write to home directories during that time will have failed.
This happened because the script, to calculate peoples’ quota, had not been updated to take in to account a change in our account management system. This change had been flagged by our colleagues weeks in advance, but the dependency of this script on the change had not been spotted. When the change happened on the Tuesday morning, the quotas script was no longer able to determine the roles a user had, and so could not allocate the correct quota, eg 10GB for staff, 2GB for UG1 and UG2. So it defaulted to the minimal quota of 2MB.
The quick fix was to change that minimal quota to 20GB for everyone, and then later update the quotas script to use the new location for the user role information. Then the previous, correct role based quota was applied, eg 10GB for staff, 2GB for UG1 etc.
As this user data (like roles) is now retrieved from a central source, it will be easier in future to see what queries are being made for that data, and so what would be affected by any changes to that central source.
Sorry for this break in operation.