Skip to content

Joblib executor

Documentation in joblib executor docstrings.

Module executors.joblib_executor

Overview

An executor that runs simulations in parallel using joblib.Parallel. This executor integrates with an Enchanted Surrogates sampler to generate parameter configurations and execute tasks concurrently on the local machine.

JoblibExecutor 

JoblibExecutor(runner_config=None, output_dir=None, *args, **kwargs)

Bases: Executor

Features

  • Uses joblib to parallelize simulation tasks across available CPU cores.
  • Integrates with Enchanted Surrogates sampler for parameter exploration.
  • Generates a unique run directory for each sample.
  • Automatically registers completed futures with the sampler.

Notes

  • This executor does not manage clusters; it runs everything locally.
  • No dynamic scaling or distributed execution is supported.
  • Cleanup is minimal since joblib runs in-process and does not leave persistent resources.
Source code in src/enchanted_surrogates/executors/base_executor.py 
 8
 9
10
def __init__(self, runner_config=None, output_dir=None, *args, **kwargs):
    self.runner_config = runner_config
    self.output_dir = output_dir  # TODO rename

execute 

execute(input, sampler)

Execute simulation tasks in parallel using joblib.

Parameters:

Name Type Description Default
input list[str, dict]

A list of simulation tasks to execute. Each element is a tuple consisting of path to the directory where the simulation run should be executed and dictionary of simulation parameters.

 required
sampler object

Sampler instance responsible for tracking submitted simulation tasks

 required
Source code in src/enchanted_surrogates/executors/joblib_executor.py 
42
43
44
45
46
47
48
49
50
51
52
53
54
55
def execute(self, input: list[(str, dict)], sampler):
    """
    Execute simulation tasks in parallel using joblib.

    Params:
        input (list[(str, dict)]): A list of simulation tasks to execute. Each element is a tuple consisting of path to the directory where the simulation run should be executed and dictionary of simulation parameters.
        sampler (object): Sampler instance responsible for tracking submitted simulation tasks
    """
    joblib.Parallel(n_jobs=-1, verbose=10)(
        joblib.delayed(run_simulation_task)(
            self.runner_config, sample_run_dir, params=sample
        )
        for sample_run_dir, sample in input
    )