make comments on criteria available

Another (feature creep) request for the new system. It looks likes the mechanism is there and this will be easy to bolt on subsequently in which case it is unlikely to be done with this particular project.

Returning feedback to students wasn’t part of the  original project description, rather a requirement that has arisen in the meantime. But the fact that the reports are now stored is surely a  big step in providing that feedback. When we agreed at Teaching Committee that information from the reports  should be returned to students as feedback, I pointed out that it would probably not be possible to do it in June 2015, but I hope we will have something in place for June 2016.

Achieving this with the new system is probably just a case of Webmark also producing a
reduced version of the form just including the relevant fields students can see and then a specific index page for access by students that shows them that form.

Comment: That would be perfect! The “relevant fields” are:

  • Individual marking form: Comments on the criteria
  • Agreed mark form: nothing
  • Moderator’s form, if any: Comments on the criteria

I guess that an ideal “specific index page for access by students” would  be to include it on student.inf.ed.ac.uk but with a time delay so that  it is only accessible once the examiners’ meetings have passed.

Modeling Part Completed

The documentation for TheonModel (managing the physical database schema and related configuration for TheonCoupler, TheonUI and TheonPortal) has been completed and including the illustrations. It is in principle a first draft, but its been through a number of revisions so its a fairly solid first draft. The command toolchain is clearly defined now so we can start implementing that – all the underlying technology has been done already, or at least proven in principle, so its just tidying and wrapping it up properly here to do. The documentation for TheonCoupler is practically done as well now, should be completed within the next few days. Still got architectural documentation to do on TheonUI and TheonPortal to do (bits and pieces on usage and configuration are there already under TheonModel). Need to write the release/package management documentation as well (although the toolchain for that is at least already defined and coded up). Finally the system installation/configuration documentation needs to be done – although that is a little bit in limbo to see whether we can get a viable standalone LCFG config up and running. The remaining documentation will proceed in parallel with getting the TheonModel toolchain implemented as that is really the primary goal of this project.

Started back on Documentation

So after a long break (I think May was the last time this was touched) got back into doing the documentation. It probably took a day just to orientate to where we were. Anyhow the main purpose of doing the documentation before the technical work is complete is to pin down the workflow and hence what the commands and options will need to be provided by the wrapper toolkit. This so far has been quite successful in identifying any gaps in the technical implementation (due to the attempt to reflect real use cases).

Mostly Done

The bulk of this has now been done (the first four goals).

The “Proposal for Submissions” was followed pretty much as-is. The MSc submissions link will now jump to an MSc variant of the UG4 form but using the same underlying mechanism. Both handling flags can be set and materials uploaded at submission time.

The “Proposal for Access to Submissions” was modified slightly. In the end the existing “projsubs/ug4” and “projsubs/msc” folders were retained (rather than new URLs and folders being created) with the modified indexing code. Instead of passing arguments the indexing behaviour is automatically changed based on whether the current year is the same as the year being processed. If not then the old behaviour is retained. If it is then the new behaviour is followed. This new index format for the current (assessment) year is as described, except that to achieve sorting a variation of the “dotable” cgi script is used and a data file is produced instead (mapped to the cgi script by a rule in .htaccess). Some other minor changes were made also to the existing setup (such as indexing in year descending order). The indexing scripts now generate the liveroot files directly (since they have to change the name and extension) rather than via stdout. They also generate a data file into the upload directory which maps the matriculation number to the submitted folder number – this is used by Webmark (see below).

The “Proposal for Webmark Returns” was also modified slightly. The end result is the same though – a PDF for each filled in form and a data file for each mark is copied into the submission upload directory. Instead of using a cc mail alias and remctl a feature was added to Webmark to allow per-output subdirectories in the final file output path, components of which could be literal or set by the value of a form field. An additional source was added which is the data file generated by the indexing scripts above. This allows a “dynamic” output directory (which is the students upload submission directory) to be set against the additional outputs. By this means Webmark can write the files for the indexing scripts directly into their final location (and safely as it is part of the user submission process, which also ensures that the students upload directory is in place before a Webmark submission can be made). A consequence of this though is that it is no longer possible to fill in a blank form for the mark returns – a student must be selected from the drop-down, in practice it would be an error condition if not all the students are listed in the drop-down.

The “Proposal for Public Access” has not been done yet – but will be another modification to the indexing scripts. In this context they will simply produce a list of projects for all years (including the year in the index) with just the student name and project title (which is a link to the copied PDF as at the moment). This list will only include students with distinctions though (as previously described).

To clean everything up the older msc project submissions were moved from their original “infthesis” upload location (where they were mixed up with PhD and MScRes submissions) into the new “mscprojects” location and the indexing scripts re-run for the default three years and then individually all the way back to 2003. The indexing scripts were also manually re run for all the existing UG4 years.

Goals and Proposal

Goals

See also the original project descriptor.

  • MSc submission to be the same as UG4 is now, hence to include project materials archive and “handling” flags.
  • Webmark mark return PDFs to be accessible alongside the project PDF and materials.
  • Provide a simple master index page of the current sessions projects with links to the report PDF, materials and showing the project marks (1st marker mark, 2nd marker mark, agreed mark and moderator mark) each with link to actual corresponding Webmark return PDF. The index page should include student name, UUN and project title. It should be sortable by any column, but particularly mark columns.
  • All to be directly accessible to external examiners, however authentication does not need to be constrained down to specific examiner access per-project, controlled access to any is sufficient.
  • Projects of MSc students awarded distinctions (where student has gained a distinction overall) to be published openly. Projects of UG4 students where the projects final mark (as appropriately calculated) is at a distinction grade level to be published openly. In both cases this is however only if the “handling” flags do not indicate that the project was submitted for assessment only.

Proposal for Access to Submissions for Markers

We will create two new project websites. Access will be limited by default to current staff (as is the case for the current project area: www.inf.ed.ac.uk/teaching/projsubs) and external examiners, e.g.

www.inf.ed.ac.uk/teaching/ug4projects
www.inf.ed.ac.uk/teaching/mscprojects

External examiners will need to obtain an iFriend account which can then be given access to either or both of these areas. Given the small numbers of accounts likely to be involved this process will not be automated – each new iFriend account will require frontline support to make a small configuration change to the ACLs for these URLs for each session.

We already have two indexing scripts which work well (provide access to each students projects PDF, materials and “handling” flags for UG4, and could also do the same for MSC, see below). We propose modifying these to also act as the index page generator for each area above:

  1. Modify “submission_links_common.cgi” so that by optional argument only the most recent submission by each student (by reference to the submission date and time held in the “record” file in the “upload” directory) is listed.
  2. Modify “submission_links_common.cgi” so that by optional argument it will also display per student submission the 1st marker mark, 2nd marker mark, agreed mark and moderator mark. These will have been made available in the “upload” directory (see Webmark Returns proposal below). This option should also include a “COALESCE(moderated,agreed)” and “ABS(1stmark-2ndmark)” calculated column. The four raw marks will be links to each of the four possible Webmark PDF returns from markers (1st, 2nd, Agreed and Moderator) for that students project. These will have been copied into the “upload” directory (see Webmark Returns proposal below).
  3. Modify “submission_links_common.cgi” so that by optional argument the “handling” flags are not shown (these are not relevant to the marking process itself nor do external examiners need to see them).
  4. Modify “submission_links_common.cgi” so that the generated index page includes per-column sort functionality.
  5. Make “submission_links_ug4.cgi” act as “index.html” for “teaching/ug4projects”.
  6. Make “submission_links_msc.cgi” act as “index.html” for “teaching/mscprojects”.

Proposal for Submissions

We already have a submission form for UG4 which works well and does everything we need. The MSc submission form while using similar technology does not collect project materials or the “handling” flags. We propose dropping the current MSc submission form and using a straight clone of the UG4 process.

  1. Copy “submission_form_ug4.pl” to “submission_form_msc.pl”.
  2. Factor out common code in “submission_form_ug4.pl” and “submission_form_msc.pl” into “submission_form_common.pl”. The only effective differences between them are the final “upload” directory path (which will be either “ROOT/ug4projects” as at the moment or “ROOT/mscprojects” for the new MSc version, in place of “ROOT/infthesis” where the MSc submissions are currently held) and a few tags which need to be either UG4 or MSC respectively.  Action script will be either the existing UG4 one or the new MSc one (see next item).
  3. Copy “accept-ug4.pl” (which is the “submit” action of the current “submission_form_ug4.pl”) to “accept_msc.pl”.
  4. Factor out common code in “accept_ug4.pl” and “accept_msc.pl” into “accept_common.pl”. Again the only effective differences are the final “upload” directory path and a few tags.

As before these will create a unique directory per-submission. This will contain the project report PDF, the project materials archive file and a “record” file which has relevant metadata – the students name, matriculation number, “handling” flags, date and time of submission. To put the new mechanism in place:

  1.  Update existing references to UG4 and MSc project submission to use the new scripts above.
  2.  Make sure that “submission_form_thesis.pl” (previously used for MSc project submission) and links to it are marked for PhD Thesis submission only (I am not sure if this mechanism is still used for PhD submission, if not this could just be deleted).

Proposal for Webmark Returns

For various reasons it would be tricky to get Webmark to write its internal copy of the submitted 1st, 2nd, Agreed and Moderator mark forms directly into the “upload” directory. So to achieve this instead we propose the following.

  1. Modify the Webmark config for the project marking forms so a “copy” of the generated PDF (as sent to the marker) is sent into the corresponding students project submission “upload” directory. This will be done by adding a “Cc” to the Webmark email that is to an alias address, e.g. “wm-projmark@inf.ed.ac.uk”. The alias will be defined as a pipe to a “remctl” script on the Webmark server itself (which already has access to the student submission “upload” directories via NFS under an AMD mountpoint). The “remctl” script will strip the PDF attachment from the email and copy it into the most recent submission directory for the corresponding student. The directory matching will be done by reference to the students matriculation number which will be included in the Webmark generated PDF attachment filename and the students matriculation number held in the “record” file in the submission “upload” directory. This will require changing the generated attachment name in Webmark so this additional data is put into the filename. The Webmark PDF attachments will be stored with fixed/known names in the relevant “upload” directory (e.g. 1st.pdf, 2nd.pdf, Agreed.pdf and Moderator.pdf) so that the indexing script needs no additional intelligence to provide links to them.
  2. Also modify the Webmark config so that an additional (hidden) output is produced from the entered form data containing the actual mark awarded. As in item 1 above this will be pushed out via “remctl” script which will identify the appropriate “upload” directory to copy the data into and will be saved into a file with fixed/known name so that the indexing script can include the value as the title text for the link to the actual submitted form PDF.

Proposal for Public Access to Archive of Projects

Part of the submission technology we are continuing to use above includes a notification to the (legacy) School database to collect the submitted papers for further publication (if the “handling” flags give permission and in the case of the MSc that the student in question had been awarded a Distinction). This is currently broken for a few reasons:

  • The “upload” directory path being used by the database scripts seems to be wrong so it is currently looking in the wrong place.
  • The Distinction award is being looked up in the old legacy assessment structures where it is no longer maintained – in fact it was not being maintained in the new system (Theon) either until this session.

It is probably trivial to fix the above – however it is not clear what the long term status of these processes is (being largely tied to the legacy database service) nor what the plan will be with respect to PURE (and integration of reports onto personal pages).

So for now we propose (only for MSc students that got a Distinction Overall and UG4 projects where the final project mark was at a distinction grade level) that there will be a separate public access page, much like the existing “teaching/projsubs” tree, for the published project PDF reports.

The “projsubs” mechanism is basically a cron driven forced processing of the “submission_links_[ug4|msc].cgi” script for multiple sessions to produce a static index page per session of student projects with link (via the cgi) to the actual PDF report and materials, just like the live current session cgi script.

  1. Make a crontab entry derived from that used for “/projsubs” for a new URL, e.g. www.inf.ed.ac.uk/teaching/projects/[ug4|msc]/YYYY.html, where YYYY.html is the relevant session snapshot, 2013.html for this session. Access to this URL tree will be public. This modified crontab entry will use additional arguments to “submission_links_common.cgi” to suitably modify the behaviour (see item 2 and 3 below) and also use the argument to hide “handling” flags (see item 3 in Proposal for Access to Submissions for Markers above).
  2. Modify the “submission_links_common.cgi” script so that by optional argument (given in the crontab entry for this URL) only students that were awarded a distinction and have not indicated in “handling” flags that the project was submitted for assessment only are listed (see item 4 below).
  3. Modify the “submission_links_common.cgi” script so that by optional argument (given in the crontab entry for this URL) each relevant students project PDF file is physically copied into the “teaching/projects” directory tree and that the index points to this rather than via the link script. This means that the public projects contain a separate copy from that held in the “upload” directory (which should be being deleted after the marking process has been completed, or as according to the students preference in the “handling” flags). This also fences access to just the project report PDF and means exposure to any project materials or marking reports will be prevented.
  4. Add a report to Theon that generates a list of MSc student matriculation numbers in each session that were awarded a distinction overall and a list of UG4 student matriculation numbers in each session that achieved a distinction grade for their final project mark. This report will be written into some area that the “submission_links_common.cgi” script can access and will then be used by that script to filter the directory tree of “upload” submissions to make the index a suitable subset (see item 2 above).
  5. Update any links to published projects to refer to the new URL (see item 1).

Function Conversion – Test Case 2

Next function looked at was one that runs its processing through the event queue. There were some core structural changes here because of limitations in PostgreSQL triggers. The old method was based on statement data set processing which added multiple entries for an event to the queue. The event daemon deleted events from the queue on notification which triggered the processing functions again via a statement data set. The event notification system can be replaced almost like for like in PostgreSQL with NOTIFY/LISTEN. The new structure involves the event daemon updating records in the queue to flag a particular set of records for an event to be processed. The processing function can then query the queue for these updated records associated with its event handling as a virtual data set in much the same way as the original virtual statement data set. The function must then explicitly delete the entries from the event queue after processing is finished. This is all working now and there is a PostgreSQL equivalent event daemon (still in testing) and a re-implemented function carrying out queue entry and queue removal for unit work marks. Some new issues to be aware of in conversion are below.

  • PostgreSQL does not do short circuit evaluation, so IF OLD IS NOT NULL AND OLD.X = S will not work when OLD is NULL as the remainder of the condition will still be evaluated causing an error. Conditions must be written like, IF OLD IS NOT NULL THEN IF OLD.X = S THEN ELSE … END IF; ELSE … END IF; which tends to be a lot more clunky.
  • Most of the unit processing functions are multi-talented and cover different sets of update types with the same code base. Handling of this is controlled by trigger updates based on changes to specific named columns. Of course you can’t do this in PostgreSQL, so all the separate trigger conditions must be amalgamated into one and folded into the start of the function itself which of course doesn’t do short circuit evaluation … This logic is generally attached to the function pushing the event into the queue so it is at least contained in a reasonably clear and consistent place.
  • Again due to the lack of a CASE statement in OpenIngres a table called “er_relop_map” was used to provide a query based mechanism to do IF A > B THEN 1 ELSE 0 type mappings by using normal joins. This could probably be replaced by simply using CASE statements in the expressions directly and remove the need for the additional table and joins. This hasn’t been done yet (waiting until a the core unit processing functions have been re-implemented).

Function Conversion – Test Case 1

Some monologue for converting the first of the unit assessment functions. This was the us function, standing for unit score. It recalculates statistical values for any given “unit” (built from templates for each of a module, course and year). It is triggered via a touch process from any changes in associated unit marks. We aren’t looking at that specific trigger process here, although the re-implementation of the touch process is included. It is also triggered when unit marks are deleted or the ignore status changes. We aren’t going to cover any of the general substitution conversion parts of the process here, just interesting things specific to this function.

  • Using the old nomenclature, the function is triggered on U(s=s)cs; D,U(i)cm. So an UPDATE on class_score when the update affects the session@ column and the value is unchanged, a DELETE on the class_mark table or an UPDATE to the ignore column on the class_mark table. These have been replaced by Ucs; D,Ucm in PostgreSQL. These are separate triggers. In the latter case logic was added to the function so that the function will drop out when TG_OP is UPDATE and TG_TABLE_NAME is class_mark and OLD.ignore = NEW.ignore. In the former case logic was added to the function so that the function will drop out when TG_OP is UPDATE and the update changes any non-derived column or the update does not change a derived column to NULL (this latter is critical to avoid loops as the function itself updates the class_score table thus re-triggering itself).
  • Logic was added so that the Vsession and Vunit variables always have values (by assigning from NEW.session or OLD.session depending on TG_OP).

Lets consider an alternative approach.

  • We could make the function an ordinary non-trigger function. It is called directly by the function that normally forces the touch and it is called indirectly via a trigger wrapper function to cover the triggers from class_mark. The wrapper function in this cases handles the ignore column check logic. The problem with this is that the function causing the touch is using statement based set processing to update batches of rows, this could be done with a select loop, however it is as yet unclear how to deal with statement based processing in PostgreSQL – that should be the next function to convert!
  • Alternatively the function that normally forces the touch could do so by deleting the corresponding row(s). The function would have to then re-insert, but it could do this using a single SELECT. However this approach may not work for other cases.

Finally although the new function behaves the same way there are some improvements.

  • The original OpenIngres function executes 14 separate SELECT statements to collate numbers of different grades. In PostgreSQL with the use of the CASE expression function these could all be reduced to a single SELECT with 14 column results instead.
  • The original OpenIngres function executes two SELECT statements – one to collate MIN, MAX and AVG aggregates and one to calculate the standard deviation as a further aggregation using the previously calculated AVG values. However, PostgreSQL has a STDDEV aggregate function that actually produces identical results so that could be used instead to reduce this to a single SELECT. It would also remove the need to ensure that done counts of less than 2 do not need to be nullified and coalesced back to 0 (this was done to avoid a division by zero error when the standard deviation was aggregated over one row).
  • The two new PostgreSQL SELECT statements could be rolled into one now, so the original 16 are done as one.

None of the refinements above have been made yet pending alterations to other key functions to see how everything will work together.