Pipeline Driver

Application code looking to make use of Clara’s pipeline orchestration must create an instance of clara.Driver. It is the context by which all other interactions with the Clara Pipeline API are made.

The clara.Driver class constructor accepts a number of lambda methods, or “callback handlers”, as well as a reference to a block of application managed data. The application data reference is the primary method by which the application can pass state to itself as part of the Clara Pipeline Driver operator life-cycle.

As the operator life-cycle progresses, each event callback will provide a reference to the instance of clara.Driver orchestrating the life-cycle. Using the clara.Driver reference, application code can access its application data; thus providing state to itself during the operator life-cycle.

see also: Clara Pipeline Driver Callbacks

Clara Pipeline Operator Life-Cycle

  1. Prepare

    Called once, before stage inputs are ready.

    lambda driver: True
    
  2. Execute

    Called once, after prepare once stage inputs are ready.

    lambda driver, payload: True
    
  3. Cleanup

    Called once, after execute before the operator life-cycle terminates.

    lambda driver: True
    

Additionally, a callback handler for notifications can be supplied. The driver will pass notifications to application code whenever a significant event or error occurs. This is a useful way for application code to know what is happening “inside” the driver, and a useful tool when trying to debug issues involving Clara Pipeline Driver interaction.

see also: Clara Pipeline Driver Callbacks

Application Data

Returns the application data reference provided to the clara.Driver constructor; or None if one was not provided.

Job Identity

Gets the unique identity of the currently executing pipeline job.

Return

Returns a uuid.UUID.

@property
def job_id(self):

Job Name

Gets the name of the currently executing pipeline job.

Return

Returns a string.

@property
def job_name(self):

Stage Name

Gets the name of the currently executing pipeline stage.

@property
def stage_name(self):

Return

Returns a string.

Stage Timeout

Gets the maximum amount of time the currently executing pipeline stage is alloted.

Return

Returns a datetime.timedelta.

@property
def stage_timeout(self):

Start

Starts the operator life-cycle.

.start() should only be called once per instance of clara.Driver, additional calls will raise an error.

Once called the Clara Pipeline Driver will begin work immediately using its own internal thread-pool. Any event callback handlers will be called using the driver’s thread-pool. This means that once .start() is called, the application should not terminate before the driver has completed its work. The easiest way to ensure the driver has completed all work is to use .wait_for_completion().

def start(self):

Wait for Completion

Calling .wait_for_completion() blocks the calling thread until the operator life-cycle has completed. Requires the operator life-cycle has been started using .start().

Calling this method before calling .start() will result in an error being raised; and the calling thread will not be blocked.

Calling this method after the operator life-cycle has complete is safe, and will return immediately.

def wait_for_completion(self):