Skip to content

tutorial.parallelism.rst

Latest commit

 

History

History
114 lines (73 loc) · 3.15 KB

tutorial.parallelism.rst

File metadata and controls

114 lines (73 loc) · 3.15 KB

Parallelism - using multiple cores

Often you will want to evaluate the function on some remote computing resources. adaptive works out of the box with any framework that implements a PEP 3148 compliant executor that returns concurrent.futures.Future objects.

concurrent.futures

On Unix-like systems by default adaptive.Runner creates a ~concurrent.futures.ProcessPoolExecutor, but you can also pass one explicitly e.g. to limit the number of workers:

from concurrent.futures import ProcessPoolExecutor

executor = ProcessPoolExecutor(max_workers=4)

learner = adaptive.Learner1D(f, bounds=(-1, 1))
runner = adaptive.Runner(learner, executor=executor, goal=lambda l: l.loss() < 0.05)
runner.live_info()
runner.live_plot(update_interval=0.1)

ipyparallel.Client

import ipyparallel

client = ipyparallel.Client()  # You will need to start an `ipcluster` to make this work

learner = adaptive.Learner1D(f, bounds=(-1, 1))
runner = adaptive.Runner(learner, executor=client, goal=lambda l: l.loss() < 0.01)
runner.live_info()
runner.live_plot()

distributed.Client

On Windows by default adaptive.Runner uses a distributed.Client.

import distributed

client = distributed.Client()

learner = adaptive.Learner1D(f, bounds=(-1, 1))
runner = adaptive.Runner(learner, executor=client, goal=lambda l: l.loss() < 0.01)
runner.live_info()
runner.live_plot(update_interval=0.1)

mpi4py.futures.MPIPoolExecutor

This makes sense if you want to run a Learner on a cluster non-interactively using a job script.

For example, you create the following file called run_learner.py:

import mpi4py.futures

learner = adaptive.Learner1D(f, bounds=(-1, 1))

# load the data
learner.load(fname)

# run until `goal` is reached with an `MPIPoolExecutor`
runner = adaptive.Runner(
    learner,
    executor=MPIPoolExecutor(),
    shutdown_executor=True,
    goal=lambda l: l.loss() < 0.01,
)

# periodically save the data (in case the job dies)
runner.start_periodic_saving(dict(fname=fname), interval=600)

# block until runner goal reached
runner.ioloop.run_until_complete(runner.task)

On your laptop/desktop you can run this script like:

export MPI4PY_MAX_WORKERS=15
mpiexec -n 1 python run_learner.py

Or you can pass max_workers=15 programmatically when creating the executor instance.

Inside the job script using a job queuing system use:

export MPI4PY_MAX_WORKERS=15
mpiexec -n 16 python -m mpi4py.futures run_learner.py

How you call MPI might depend on your specific queuing system, with SLURM for example it's:

#!/bin/bash
#SBATCH --job-name adaptive-example
#SBATCH --ntasks 100

export MPI4PY_MAX_WORKERS=$SLURM_NTASKS
srun -n $SLURM_NTASKS --mpi=pmi2 ~/miniconda3/envs/py37_min/bin/python -m mpi4py.futures run_learner.py