morpheus.utils.shared_process_pool.SharedProcessPool

class SharedProcessPool

Bases: object

Attributes:

status

total_max_workers

Methods

join([timeout])	Wait until the SharedProcessPool is terminated.
reset()	Clear all the previous settings and restart the SharedProcessPool.
set_usage(stage_name, percentage)	Set the usage of the SharedProcessPool for a specific stage.
start()	Start the SharedProcessPool that is currently stopped and keep the settings before last shutdown.
stop()	Stop receiving any new tasks.
submit_task(stage_name, process_fn, *args, ...)	Submit a task to the SharedProcessPool.
terminate()	Terminate all processes and shutdown the SharedProcessPool immediately.
wait_until_ready([timeout])	Wait until the SharedProcessPool is running and ready to accept tasks.

join(timeout=None)

Wait until the SharedProcessPool is terminated.

Parameters:

timeout_type_, optional

timeout in seconds to wait for the SharedProcessPool to be terminated, by default None. If None, it will wait indefinitely.

Raises:

RuntimeError

If is called on a SharedProcessPool that is not stopped.

TimeoutError

If has been waiting more than the timeout.

reset()

Clear all the previous settings and restart the SharedProcessPool.

Raises:

RuntimeError

If the SharedProcessPool is not already shut down.

set_usage(stage_name, percentage)

Set the usage of the SharedProcessPool for a specific stage.

Parameters:

stage_namestr

The unique name of the stage.

percentagefloat

The percentage of the total workers that will be allocated to the stage, should be between 0 and 1.

Raises:

RuntimeError

If the SharedProcessPool is not running.

ValueError

If the percentage is not between 0 and 1 or the total usage is greater than 1.

start()

Start the SharedProcessPool that is currently stopped and keep the settings before last shutdown.

Raises:

RuntimeError

If the SharedProcessPool is not shutdown.

stop()

Stop receiving any new tasks.

submit_task(stage_name, process_fn, *args, **kwargs)

Submit a task to the SharedProcessPool.

Parameters:

stage_namestr

The unique name of the stage.

process_fnCallable

The function to be executed in the process pool.

argsAny

Arbitrary arguments for the process_fn.

kwargsAny

Arbitrary keyword arguments for the process_fn.

Returns:

Task

The task object that includes the result of the process_fn.

Raises:

RuntimeError

If the SharedProcessPool is not running.

ValueError

If the stage_name has not been set in the SharedProcessPool.

terminate()

Terminate all processes and shutdown the SharedProcessPool immediately.

wait_until_ready(timeout=None)

Wait until the SharedProcessPool is running and ready to accept tasks.

Parameters:

timeout_type_, optional

timeout in seconds to wait for the SharedProcessPool to be ready, by default None. If None, it will wait indefinitely.

Raises:

RuntimeError

If the SharedProcessPool is not initializing or running.

TimeoutError

If has been waiting more than the timeout.