Experience report from rolling out Local-First in factories

At Actyx we have been implementing local-first software before the seminal paper from Ink & Switch came out, but for different reasons: our hand was forced by needing to minimise dependence on IT infrastructure (which is problematic to grow in factories) and maximising the speed of roll-out — this was our start-up advantage and a key selling point. Going for coordination-free swarm systems with only peer-to-peer communication and distributed data storage and compute satisfied our constraints, but it came with a new challenge: how to write programs for this environment that actually do what production managers want?

After implementing a few systems by ourself and concurrently developing our corresponding middleware and libraries we turned the latter into a product and started selling that to partner companies — IT providers who wanted to move down to the shop floor and OT (classical automation) providers who wanted to move up from single machine control to shop floor orchestration. We are now collaborating with machine and plant builders who develop their next generation machines and production lines to allow for flexible IT-based process management directly on the shop-floor, in a local-first fashion. The reasons span all of the above motivations, including data ownership, agency, and minimal infrastructure as well as latency, bandwidth, and resilience.

The difference in programming model shows foremost in the lack of strong consistency, but there are other aspects that are less expected and equally impactful, like the UX problem of managing connectivity expectations or showing which data are available where. This presentation highlights the main such learnings from seven years of Actyx.

Beyond data — evolving behaviours in swarm systems

The Actyx model allows nodes to work in a local-first manner, to that end, each node runs their own state machine and merely reacts to incoming events. These events can be compared to RPCs, they even share similarities! Unfortunately, one of the shared similarities is the difficulty evolving the data format — how do we add an extra field? Akin to the data format, behaviours must also evolve, otherwise, as goals evolve, the system risks diverging from said goals (arguably becoming useless).

When discussing P2P systems, it is unclear how to upgrade the system as a whole as the nature of each peer varies wildly — peers can belong to different people, have different hardware, etc.

Evolving these P2P state machines requires flexibility on two axis — data, and behavioural — Cambria by Litt et al. targets the data axis, providing a mechanism which we can use to transform the events between different versions. As for transitions, it’s clear they belong on the behavioural axis, what is less clear is how to enable a system to support several versions of transitions.

The presentation discusses on-going research into how versioning could be applied to a state machine, enabling it to handle the evolution of behaviour and data the system must process.

The driving idea behind this approach can be summed up as “version everything”. More concretely, it proposes recognising the state machine for the graph it is; splitting it into it’s components — state, events and transitions — and versioning each of them separately; each version then becomes a new, separate “path” in the state machine. This approach allows incremental modifications (addition, modification and deletion) to the state machine as new and old versions of components cohabit the same state machine.