Fault injection Testing:

Stability and Resiliency of software is critical for smooth running of an application. Fault injection testing is the deliberate introduction of errors and faults to a system to observe its behavior. The goal is for the software to work correctly despite errors encountered from calls made to dependencies such as other APIs, system calls and so on. By introducing intermittent failure conditions over time, the application behaves as realistically as in production where hardware and software faults can occur randomly, but the services must remain available, and the business continuity must be maintained.

System needs to be resilient to the conditions that cause production disruptions.  The dependencies might include infrastructure, platform, network, 3^rd party software, or APIs. The risk of impact from dependency failure may be direct or cascading. Fault injection methods are a way to increase coverage and validate software robustness and error handling, either at build time or at run-time with the intention of embracing failure as a part of development lifecycle. These methods assist service teams in designing and continuously validating for failure, accounting for known and unknown failure conditions, architect for redundancy and employ retry and back-off mechanisms. Together with the introduction of intermittent failures and continuous monitoring in the stage environment of service deployments, these methods promote near total coverage of known and unknown faults that can impact the service in production. The purpose of the monitoring aspect during these experiments is the observation of fault and its recovery time, overview of symptoms in related components and the determination of the threshold and values with which alerts can be set.

Fault engineering is equally applicable to software, protocol, and infrastructure.  Software faults include error-handling code paths and in-process memory management for which edge-case unit-tests, integration tests and stress and soak load tests are written. Protocol faults include the vulnerabilities in communication interfaces such as command line parameters or APIs. Examples of tests that mitigate this includes fuzzing which provides invalid, unexpected, or random data as input and we can access the level of protocol stability of a component.  Infrastructure faults include outages, networking, and hardware failures. The tests that mitigate these cause fault in the underlying infrastructure such as shutting down virtual machines, crashing processes, expiring certificates and others.

One of the challenges with these methods is the signal to noise ratio from the errors. A fault is a hypothesis of an error. An error is a failure in the system and can lead to other errors. Since they occur in a cycle, the fault-error-failure cycle can lead to many errors from which the ones that must be fixed to improve system resilience and reliability need to be discerned. When these experiments are run for short durations, the number of errors to investigate is usually low. The leveraging of automation to continuously validate what matters during the experiment allows the detection of even errors that are hard to find manually.

Such automation can even be introduced into the pipeline to release software. This promotes a shift-left approach where the testing occurs as early in the development and project timeline as when the code is written. It follows the test early and often principle and the benefit is in the possibility to troubleshoot the issues encountered via debugging.

The outcomes of the fault injection testing are the measurement and definitions of a steady healthy state for the system’s interoperability, finding the difference between the baseline state and the anomalous state and documenting the processes and observations to identify and act on the result.