Traditional compilers offer little assistance in ensuring the correctness of distributed programs. The Hydro project at Berkeley is an effort to build a compiler stack to address these issues. The lowest layer of the Hydro stack is Hydroflow, a Rust-based dataflow runtime with an IR based on a semilattice formalism. Hydroflow enables correct program transformations that are natural in the context of distributed systems. Transformations include:

• Refactoring: Given an arbitrary block of code, refactor it into smaller blocks that can be launched on independent machines
• Replication: Given an arbitrary block of code, determine whether it can be safely replicated in deployment
• Partitioning: Given an arbitrary block of code, determine how its inputs can be safely partitioned (``sharded'') to multiple machines in deployment

These transformations in turn allow distributed programs to be optimized for various goals, including parallelism (both pipelines and partitioning), memory scaling, performance isolation, geoproximity and physical security.

We present case studies showing correctness, latency and scaling results when optimizing programs ranging from infrastructure like key-value stores, applications like shopping carts and messaging systems, and tricky consensus protocols.

Joseph M. Hellerstein’s work focuses on data-centric systems and the way they drive computing. He is the Jim Gray Professor of Computer Science at the University of California, Berkeley. Recognitions of his research contributions include the ACM SIGMOD Codd Innovations Award, ACM Fellow and Sloan Fellowship, and six “Test of Time” awards. Fortune Magazine has included him in their list of 50 smartest people in technology, and MIT’s Technology Review magazine included his work on cloud programming in their TR10 list of the 10 technologies “most likely to change our world”. Hellerstein is also a longtime participant in the computing industry: a co-founder of startups including Aqueduct and Trifacta, a Faculty Fellow at Sutter Hill Ventures, former Director of Intel Research, Berkeley, and technical advisor to startups and established tech firms.