cubed.Spec

class cubed.Spec(work_dir=None, *, intermediate_store=None, allowed_mem=None, reserved_mem=0, executor=None, executor_name=None, executor_options=None, storage_options=None, zarr_compressor='default')

Specification of resources available to run a computation.

__init__(work_dir=None, *, intermediate_store=None, allowed_mem=None, reserved_mem=0, executor=None, executor_name=None, executor_options=None, storage_options=None, zarr_compressor='default')

Specify resources available to run a computation.

Parameters:

• work_dir (str or None) – The directory path (specified as an fsspec or obstore URL) used for storing intermediate data.

• intermediate_store (store, optional) – The Zarr store for intermediate data. Takes precedence over work_dir.

• allowed_mem (int or str, optional) –

  The total memory available to a worker for running a task, in bytes.

  If int it should be >=0. If str it should be of form <value><unit> where unit can be kB, MB, GB, TB etc. This includes any reserved_mem that has been set.

• reserved_mem (int or str, optional) –

  The memory reserved on a worker for non-data use when running a task, in bytes.

  If int it should be >=0. If str it should be of form <value><unit> where unit can be kB, MB, GB, TB etc.

• executor (Executor, optional) – The default executor for running computations.

• storage_options (dict, optional) – Storage options to be passed to fsspec.

• zarr_compressor (dict or str, optional) –

  The compressor used by Zarr for intermediate data.

  If not specified, or set to "default", Zarr will use the default Blosc compressor. If set to None, compression is disabled, which can be a good option when using local storage. Use a dictionary to configure arbitrary compression using Numcodecs. The following example specifies Blosc compression: zarr_compressor={"id": "blosc", "cname": "lz4", "clevel": 2, "shuffle": -1}.

Methods

__init__([work_dir, intermediate_store, ...])

Specify resources available to run a computation.

Attributes

allowed_mem	The total memory available to a worker for running a task, in bytes.
executor	The default executor for running computations.
executor_name
executor_options
intermediate_store	The Zarr store for intermediate data.
reserved_mem	The memory reserved on a worker for non-data use when running a task, in bytes.
storage_options	Storage options to be passed to fsspec.
work_dir	The directory path (specified as an fsspec or obstore URL) used for storing intermediate data.
zarr_compressor	The compressor used by Zarr for intermediate data.