A growing percentage of outages are caused by simple configuration changes as the amount of code controlled by configuration continues to increase. We need to understand the risk of these changes before they are rolled out to a live environment and configuration rarely gets the full suite of testing and reviews that a similar code change would receive. Why should changing a feature flag receive less scrutiny than even a simple code change introducing the conditional that will guard the new feature.

While an individual line of configuration is often simple, a boolean guarding a new feature or the address of a server, the code using these values might hide the true complexity. By looking at the intermediate and terminal usages of a configuration value, we can determine the true significance of what appears to be a small change.

We’re used to tracing values using tools that move forward one level at a time through a branching disarray of function calls and reassignments. However, manually tracing a widely used configuration and understanding everywhere it’s used can be infeasible. When looking at a configuration value and changing its current state, I want to understand where this will impact the production system.

Instead, we can propagate existing code complexity and coverage metrics up the call stack and aggregate them at a configuration level. Hear about our experiences computing test coverage for configuration based on the aggregate coverage where the configuration values are meaningfully used.

Beyond test coverage, we can consider what existing and new code metrics are informative when applied to configuration. Number of usages, complexity at a terminal usage, distance to terminal usage and other metrics can elucidate the risk of changing an otherwise simple configuration value, highlight troubling areas in your code base and, by knowing precisely where a configuration value makes a difference in production code, can quicken the recovery from an issue caused by the configuration change.