Incomplete failed runs

[Donaim]

A container run can be marked as failed (state = "F") without Kive recording any useful explanation of why it failed.

Example:

• Run: 1796472
• Slurm job id: 455052
• State: F
• Return code: None
• Stopped by: -
• stderr log exists but is 0 bytes
• stdout log does not exist
• sandbox has already been purged

This appears to happen when the Slurm job ends without runcontainer updating the run. ContainerRun.check_slurm_state() eventually marks the run as failed.

In that path, Kive currently:

• queries sacct only for jobid,end
• marks the run as failed after the timeout
• imports the first matching Slurm stderr file, if one exists
• does not import Slurm stdout
• does not record Slurm's job state or exit code
• does not generate any fallback error message when the stderr file is empty

As a result, a run can be shown as failed while giving the user no indication of what actually happened.

This is closely related to #786, which was intended to make Slurm-side failures visible. The fix for #786 added loading of the Slurm stderr file, but that does not help when the file is empty.

It would be useful for check_slurm_state() to always record some diagnostic information when it is the code that changes a run to FAILED.

For example, it could query additional fields from sacct, such as:

State
ExitCode
Reason
FailedNode

and create a stderr/error log describing the Slurm-side failure, even if the Slurm stderr file itself is empty.

It would also make sense to import the Slurm stdout file when available, as originally suggested in #786.