blueetl_core.parallel

Utilities for parallelization.

Functions

isolated(func)

Isolate a function to be executed in a separate process.

run_parallel(tasks, *[, jobs, backend, ...])

Run tasks in parallel.

Classes

Task(func)

Task class.

TaskContext(task_id, loglevel[, seed, ppid])

TaskContext class containing information to be passed to the tasks in subprocesses.
class blueetl_core.parallel.Task(func: Callable)

Bases: object

Task class.

Initialize the Task object.

Parameters:

func – function to be executed whe the Task is called in a subprocess.

class blueetl_core.parallel.TaskContext(task_id: int, loglevel: int, seed: int | None = None, ppid: int | None = None)

Bases: object

TaskContext class containing information to be passed to the tasks in subprocesses.

blueetl_core.parallel.isolated(func)

Isolate a function to be executed in a separate process.

  • It uses loky instead of multiprocessing to be able to use joblib inside the subprocess.

  • It can work as a decorator, if desired.

Parameters:

func (function) – function to isolate.

Returns:

the isolated function.

blueetl_core.parallel.run_parallel(tasks: Iterable[Callable[[TaskContext], Any]], *, jobs: int | None = None, backend: str | None = None, verbose: int | None = None, base_seed: int | None = None, shutdown_executor: bool = True) → list[Any]

Run tasks in parallel.

Parameters:

  • tasks – iterable of callable objects that will be called in separate threads or processes. The callable must accept a single parameter ctx, that will contain a TaskContext.

  • jobs – number of jobs. If not specified, use the BLUEETL_JOBLIB_JOBS env variable, or use half of the available cpus. Set to 1 to disable parallelization.

  • backend – backend passed to joblib. If not specified, use the BLUEETL_JOBLIB_BACKEND env variable, or use the joblib default (loky). Possible values: loky, multiprocessing, threading.

  • verbose – verbosity of joblib. If not specified, use the BLUEETL_JOBLIB_VERBOSE.

  • base_seed – initial base seed. If specified, a different seed is added to the task context, and passed to each callable object.

  • shutdown_executor – if True and using loky, shutdown the subprocesses before returning.

Returns:

list of objects returned by the callable objects, in the same order.