Cancel Job#

Cancel a running NeMo Safe Synthesizer job to stop processing and free up resources.

Prerequisites#

Before you can cancel a NeMo Safe Synthesizer job, make sure that you have:

  • Obtained the base URL of your NeMo Safe Synthesizer service

  • Set the SAFE_SYN_BASE_URL environment variable to your NeMo Safe Synthesizer service endpoint

  • A running job that you want to cancel

export SAFE_SYN_BASE_URL="https://your-safe-synthesizer-service-url"

To Cancel a NeMo Safe Synthesizer Job#

Choose one of the following options to cancel a running NeMo Safe Synthesizer job.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['SAFE_SYN_BASE_URL']
)

# Cancel a specific job
job_id = "job-abc123def456"

try:
    # Check job status first
    current_job = client.beta.safe_synthesizer.jobs.retrieve(job_id)
    job_status = current_job.status
    print(f"Current job status: {job_status}")
    
    if job_status in ("active", "created", "pending"):
        # Cancel the job
        cancelled_job = client.beta.safe_synthesizer.jobs.cancel(job_id)
        print(f"Successfully cancelled job: {job_id}")
        print(f"New status: {cancelled_job.status}")
    else:
        print(f"Cannot cancel job in status: {job_status}")
        
except Exception as e:
    print(f"Failed to cancel job {job_id}: {e}")

JOB_ID="job-abc123def456"

# Cancel the job
curl -X POST \
  "${SAFE_SYN_BASE_URL}/v1beta1/safe-synthesizer/jobs/${JOB_ID}/cancel" \
  -H 'Accept: application/json' \
  | jq

echo "Job cancellation request sent"
Example Response
{
  "id": "job-abc123def456",
  "name": "pii-redaction-example",
  "project": "default",
  "status": "cancelled",
  "created_at": "2024-01-15T10:30:00.000Z",
  "updated_at": "2024-01-15T10:45:00.000Z"
}

When to Cancel Jobs#

Consider cancelling jobs in these scenarios:

  • Incorrect configuration: Job was started with wrong parameters

  • Resource constraints: Need to free up compute resources for higher priority tasks

  • Long-running jobs: Jobs taking longer than expected

  • Testing: Development jobs that are no longer needed

Job Cancellation Behavior#

When a job is cancelled:

  1. Immediate response: The cancel request returns immediately with updated job status

  2. Graceful shutdown: The system attempts to stop processing gracefully

  3. Resource cleanup: Compute resources are freed for other jobs

  4. Partial results: Any completed processing steps may still be available

  5. Final status: Job status changes to “cancelled”

Important Considerations#

Note

Cancellation Timing: Depending on the current processing stage, it may take a few moments for the job to fully stop. The job status will update to “cancelled” once the cancellation is complete.

Warning

Partial Results: Cancelled jobs may have partial results available depending on how much processing was completed before cancellation.

Error Handling#

Common scenarios when cancelling jobs:

def safe_cancel_job(client, job_id):
    try:
        # Check current status
        current_job = client.beta.safe_synthesizer.jobs.retrieve(job_id)
        job_status = current_job.status
        
        if job_status == "completed":
            print(f"Job {job_id} is already completed")
            return False
            
        elif job_status == "error":
            print(f"Job {job_id} has already failed")
            return False
            
        elif job_status == "cancelled":
            print(f"Job {job_id} is already cancelled")
            return False
            
        elif job_status in ("active", "created", "pending"):
            # Cancel the job
            cancelled_job = client.beta.safe_synthesizer.jobs.cancel(job_id)
            print(f"Successfully cancelled job: {job_id}")
            return True
            
        else:
            print(f"Unknown job status: {job_status}")
            return False
            
    except Exception as e:
        print(f"Error cancelling job {job_id}: {e}")
        return False

Next Steps#

After cancelling a job:

  • Check the job logs to understand what processing was completed

  • Delete the job if it’s no longer needed

  • Create a new job with corrected configuration if needed