nv_ingest_api.util.multi_processing package#

Submodules#

nv_ingest_api.util.multi_processing.mp_pool_singleton module#

class nv_ingest_api.util.multi_processing.mp_pool_singleton.ProcessWorkerPoolSingleton[source]#

Bases: object

A singleton process worker pool using a dual-queue implementation.

Instead of a global result queue, each submitted task gets its own Pipe. The submit_task() method returns a SimpleFuture, whose result() call blocks until the task completes.

close() → None[source]#

Closes the worker pool and terminates all worker processes.

Sends a stop signal to each worker and waits for them to terminate.

submit_task(
process_fn: Callable,
*args: Any,
) → SimpleFuture[source]#

Submits a task to the worker pool for asynchronous execution.

If a single tuple is passed as the only argument, it is unpacked.

Parameters:

  • process_fn (Callable) – The function to be executed asynchronously.

  • *args (Any) – The arguments to pass to the process function. If a single argument is a tuple, it will be unpacked as the function arguments.

Returns:

A future object that can be used to retrieve the result of the task.

Return type:

SimpleFuture

class nv_ingest_api.util.multi_processing.mp_pool_singleton.SimpleFuture(parent_conn: Connection)[source]#

Bases: object

A simplified future object that uses a multiprocessing Pipe to receive its result.

When the result() method is called, it blocks until the worker sends a tuple (result, error) over the pipe.

result() → Any[source]#

Retrieve the result from the future, blocking until it is available.

Returns:

The result returned by the worker function.

Return type:

Any

Raises:

Exception – If the worker function raised an exception, it is re-raised here.

Module contents#

class nv_ingest_api.util.multi_processing.ProcessWorkerPoolSingleton[source]#

Bases: object

A singleton process worker pool using a dual-queue implementation.

Instead of a global result queue, each submitted task gets its own Pipe. The submit_task() method returns a SimpleFuture, whose result() call blocks until the task completes.

close() → None[source]#

Closes the worker pool and terminates all worker processes.

Sends a stop signal to each worker and waits for them to terminate.

submit_task(
process_fn: Callable,
*args: Any,
) → SimpleFuture[source]#

Submits a task to the worker pool for asynchronous execution.

If a single tuple is passed as the only argument, it is unpacked.

Parameters:

  • process_fn (Callable) – The function to be executed asynchronously.

  • *args (Any) – The arguments to pass to the process function. If a single argument is a tuple, it will be unpacked as the function arguments.

Returns:

A future object that can be used to retrieve the result of the task.

Return type:

SimpleFuture