Why Feedback Implementations Fail: The Importance of Systematic Testing

Over the last decade, there has been great progress in using formal methods from control theory to design closed loops in software systems. Despite this progress, formal methods are rarely used by software practitioners. One reason is the substantial risk of making changes to closed loops in software products, code that is typically complex and performance sensitive. We argue that broad adoption of formal methods for controller design require addressing how to reduce the risk of making changes in controller implementations. To this end, we propose a framework for testing controller implementations that focuses on scenario coverage, scenario evaluation, and runtime efficiencies. We give examples of applying this framework to the Microsoft .NET Thread Pool, the Google Cluster Manager, and a Google stream processing system.

• Software Systems