Skip to navigation
ReferenceAPI Gateway HTTP referenceHeimdall Service

Record a recovery milestone

View as Markdown

Idempotently persist one scheduler-action completion occurrence for recovery tracing and latency measurement.

Request

RecordRecoveryMilestoneRequest records one scheduler-action completion occurrence.

sequenceIdstringRequired
Durable recovery sequence UUID.
idempotencyKeystringRequired
Stable key that deduplicates reports of this action occurrence.
operationenumRequiredDefaults to RECOVERY_OPERATION_UNSPECIFIED

RecoveryOperation identifies the scheduler-neutral recovery intent.

  • RECOVERY_OPERATION_UNSPECIFIED: No recovery operation was specified.
  • RECOVERY_OPERATION_RESTART: Restart the failed source attempt.
  • RECOVERY_OPERATION_TERMINATE: Terminate the source attempt.
  • RECOVERY_OPERATION_DAG_RELEASE: Release a held job after its DAG dependencies are satisfied.
Allowed values:
actionenumRequiredDefaults to RECOVERY_SCHEDULER_ACTION_UNSPECIFIED

RecoverySchedulerAction identifies one scheduler command occurrence.

  • RECOVERY_SCHEDULER_ACTION_UNSPECIFIED: No scheduler action was specified.
  • RECOVERY_SCHEDULER_ACTION_SLURM_REQUEUE_HOLD: Ask Slurm to requeue the job and hold the target attempt.
  • RECOVERY_SCHEDULER_ACTION_SLURM_UPDATE_CONSTRAINTS: Ask Slurm to update the target attempt’s node constraints.
  • RECOVERY_SCHEDULER_ACTION_SLURM_RELEASE: Ask Slurm to release the target attempt from its hold.
  • RECOVERY_SCHEDULER_ACTION_SLURM_CANCEL: Ask Slurm to cancel the job.
Allowed values:
outcomeenumRequiredDefaults to RECOVERY_SCHEDULER_ACTION_OUTCOME_UNSPECIFIED

RecoverySchedulerActionOutcome classifies one scheduler action occurrence.

  • RECOVERY_SCHEDULER_ACTION_OUTCOME_UNSPECIFIED: No scheduler action outcome was specified.
  • RECOVERY_SCHEDULER_ACTION_OUTCOME_PLANNED: The action was planned in observe mode and no command ran.
  • RECOVERY_SCHEDULER_ACTION_OUTCOME_SUCCESS: The scheduler command completed successfully.
  • RECOVERY_SCHEDULER_ACTION_OUTCOME_ERROR: The scheduler command returned an error.
  • RECOVERY_SCHEDULER_ACTION_OUTCOME_ALREADY_SATISFIED: Fresh scheduler state showed that the action was already satisfied.
Allowed values:
completedAtdatetimeRequired
When the scheduler action returned or its logical outcome was determined.
completesSequencebooleanRequired

Whether this occurrence completes the owning recovery sequence. The scheduler adapter sets this from its workflow semantics; Job Manager does not derive it from operation, action, or outcome.

Response

A successful response.
duplicatebooleanOptional
Whether this idempotency key had already been recorded.
sequenceCompletedbooleanOptional
Whether the sequence is completed after accepting this report.
endToEndDurationSecondsdoubleOptional

Trigger-to-terminal-milestone duration when this report newly completed its sequence.