Summary

Matt Rocklin revived two mothballed Rust projects—frisky, a Rust-based Dask scheduler, and dask-array, a query optimizer for Dask arrays—after consulting work exposed severe performance bottlenecks in a finance group's 10 TiB Dask/Xarray workload. Three weeks of AI-assisted development produced PyPI-ready packages that, when combined with bleeding-edge Dask and Xarray, dramatically improve performance for large task graphs, slow scheduling, and rechunking pain. The post is part announcement, part tutorial, offering minimal installation steps to get users running. Rocklin takes a moment to push back on the 'just rewrite it in Rust' narrative, arguing real performance comes from hundreds of careful iterations rather than language choice.

Key Insight

Real distributed computing performance comes from hundreds of careful iterations and attention to detail—language choice is just the headline.

Spicy Quotes (click to share)

• 7

  I use 'Rust!' above as a catch-all that people today seem to latch onto, but real performance has much more to do with thoroughness and diligent attention to detail over hundreds of iterations.

• 2

  It goes well beyond a choice of language.

• 4

  It was slow. I decided to make it faster.

• 5

  form a system for Xarray computations that is high performance, and usually doesn't break :)

• 2

  I don't know how to capture that in words, so instead I'll just show you, and hope it comes across.

Tone

pragmatic, quietly opinionated