Skip to content

Joblib Executor

Overview

An executor that runs simulations in parallel using joblib.Parallel. This executor integrates with an Enchanted Surrogates sampler to generate parameter configurations and execute tasks concurrently on the local machine.

JoblibExecutor 

JoblibExecutor(output_dir=None, *args, **kwargs)

Bases: Executor

Features

  • Uses joblib to parallelize simulation tasks across available CPU cores.
  • Integrates with Enchanted Surrogates sampler for parameter exploration.
  • Generates a unique run directory for each sample.
  • Automatically registers completed futures with the sampler.

Notes

  • This executor does not manage clusters; it runs everything locally.
  • No dynamic scaling or distributed execution is supported.
  • Cleanup is minimal since joblib runs in-process and does not leave persistent resources.
Source code in src/enchanted_surrogates/executors/base_executor.py 
8
9
def __init__(self, output_dir=None, *args, **kwargs):
    self.output_dir = output_dir  # TODO rename

execute 

execute(input, runner_config)

Execute simulation tasks in parallel using joblib.

Parameters:

Name Type Description Default
input list[str, dict]

A list of simulation tasks to execute. Each element is a tuple consisting of path to the directory where the simulation run should be executed and dictionary of simulation parameters.

 required
Source code in src/enchanted_surrogates/executors/joblib_executor.py 
42
43
44
45
46
47
48
49
50
51
52
53
54
def execute(self, input: list[(str, dict)], runner_config):
    """
    Execute simulation tasks in parallel using joblib.

    Params:
        input (list[(str, dict)]): A list of simulation tasks to execute. Each element is a tuple consisting of path to the directory where the simulation run should be executed and dictionary of simulation parameters.
    """
    joblib.Parallel(n_jobs=-1, verbose=10)(
        joblib.delayed(run_simulation_task)(
            runner_config, sample_run_dir, params=sample
        )
        for sample_run_dir, sample in input
    )