cubed.measure_reserved_mem

cubed.measure_reserved_mem(executor, work_dir=None, **kwargs)

Measures the reserved memory use for a given executor runtime.

This is the memory used by the Python process for running a task, excluding any memory used for data for the computation. It can vary by operating system, Python version, executor runtime, and installed package versions.

It can be used as a guide to set reserved_mem when creating a Spec object.

This implementation works by running a trivial computation on a tiny amount of data and measuring the peak memory use.

Parameters:

• executor (cubed.runtime.types.Executor) – The executor to use to run the computation. It must be an executor that reports peak memory, such as Lithops or Modal.

• work_dir (str or None, optional) – The directory path (specified as an fsspec URL) used for storing intermediate data. This is required when using a cloud runtime.

• kwargs – Keyword arguments to pass to the compute function.

Returns:

The memory used by the runtime in bytes.

Return type:

int