Cubed

Bounded-memory serverless distributed N-dimensional array processing

Cubed is a distributed N-dimensional array library implemented in Python using bounded-memory serverless processing and Zarr for storage.

• Implements the Python Array API standard

• Guaranteed maximum memory usage for standard array functions

• Follows Dask Array’s chunked array API (map_blocks, rechunk, etc)

• Zarr for storage

• Multiple serverless runtimes: Python (in-process), Lithops, Modal, Apache Beam

• Integration with Xarray via cubed-xarray

Documentation

For users

• Getting Started
  • Why Cubed?
  • Installation
  • Demo
• User Guide
  • Executors
  • Storage
  • Memory
  • Reliability
  • Scaling
• Intro slides
• Examples
• API Reference
  • Array
  • IO
  • Chunk-specific functions
  • Non-standardised functions
  • Random number generation
  • Runtime
  • Executors
• Python Array API
  • Missing from Cubed
  • Differences between Cubed and the standard
• Related Projects
  • Dask
  • Xarray
  • Previous work

For developers

• Design
  • Storage
  • Runtime
  • Primitive operations
  • Core operations
  • Array API
• Operations
  • Dependency Tree
  • elemwise
  • map_blocks
  • map_direct
  • blockwise
  • rechunk
  • reduction and arg_reduction
• Computation
  • Plan
  • Memory
  • Execution
• Contributing
  • Development

Articles

Cubed: Bounded-memory serverless array processing in xarray

Optimizing Cubed