cubed.measure_reserved_mem

cubed.measure_reserved_mem#

cubed.measure_reserved_mem(executor, work_dir=None, **kwargs)#

Measures the reserved memory use for a given executor runtime.

This is the memory used by the Python process for running a task, excluding any memory used for data for the computation. It can vary by operating system, Python version, executor runtime, and installed package versions.

It can be used as a guide to set reserved_mem when creating a Spec object.

This implementation works by running a trivial computation on a tiny amount of data and measuring the peak memory use.

Parameters:
  • executor (cubed.runtime.types.Executor) – The executor to use to run the computation. It must be an executor that reports peak memory, such as Lithops or Modal.

  • work_dir (str or None, optional) – The directory path (specified as an fsspec URL) used for storing intermediate data. This is required when using a cloud runtime.

  • kwargs – Keyword arguments to pass to the compute function.

Returns:

The memory used by the runtime in bytes.

Return type:

int