It is always better to be safe than sorry! And when human life is at stake, the commitment to safety requires to be of highest order.
We are talking about safety of the automobiles – passenger vehicles, commercial vehicles, off-road vehicles, Electric Vehicles and more. To be more specific, we are referring to the Functional Safety in Automotive.
As per the ISO 26262 standard, functional safety corresponds to the fool-proof design and development of electric and electronic components of a vehicle.
“There are an exploding number of electronic and electric components in a modern-day automobile. A lot of them including the anti-lock braking system, airbags, electronic power steering and more have a direct impact on the safety of the driver and the passengers. Rigorous and standardized software and hardware testing process is one of the ways to ensure robust and fool-proof systems.”
In this blog, we will talk about the unit testing aspect of the functional safety. The need for such a guideline will also be discussed.
The transition from ISO 9001 based Testing to ISO 26262 Compliant Unit Testing
As already mentioned, the safety criticality of several automotive components is quite high. In such scenarios, the functional safety at both unit level and system level needs to be ensured.
A few years back when ISO 26262 was uninitiated, the unit testing standards were in compliance with ISO 9001. The need for the concepts of ‘functional safety’ arose when certain failures in safety critical components started getting reported on a regular basis.
After Though testing has always remained a part of automotive product development process, but after these recalls, the testing process was in dire need of a revamp.
On top of this, the numerous car recalls by many global automotive OEMs over the years also added fuel to the fire. In these recalls, sometimes, the fault was found to be with the airbag and at times, with the braking system.
Though testing has always remained a part of automotive product development process, but after these recalls, the testing process was in dire need of a revamp.
“One of the major aspects where ISO 26262 compliant Unit Testing differs with ISO 9001 is related to the writing of the test cases. For ISO 26262 compliance, the test cases are always written in accordance with the safety requirements.”
New concepts and methods of unit testing have been included in the ISO 26262 that make it ideal for achieving functional safety in automobiles. We will touch upon these methods further in the blog.
ISO 26262 Compliant Unit Testing: What Makes the Process Different?
If we look from a broad perspective, ISO 26262 guidelines are focused on making the software codes more comprehensible, reliable and easy to test. One of the example is the restriction of the software component and their interface sizes. By doing so, the code is rendered less prone to errors.
The ISO 26262 testing guidelines aim at demonstrating that only the unit design specifications are fulfilled and any unrequired feature is done away with.
Such requirement based unit testing lets the automotive testing team input the values and check the output. Depending on the output, the bugs are identified and processed.
Understanding the Unit Testing Recommendation Table, Defined by ISO 26262
The section 6 of ISO 26262 standard has the recommendations for unit and integration testing. The recommendation table 10 has all the methods for software unit testing. Whereas, the table 11 recommends the methods for derivation of the test cases.
We will go through each of the methods to have a better understanding:
ISO 26262 Table 10 – Methods for unit testing of the software
Requirement-based test: This method helps in systematic identification of implementation failures. The failures occurring due to incomplete and inconsistent requirement are also identified.
In this method, the input values are derived from the behavioral requirements which are usually developed previously. If there are no documented requirements, they need to be deduced from the functional model as it has the executable specifications.
It has to be made sure that for all the requirements that have to be implemented by a software unit, one test case must be defined.
After the requirements are finalized, requirement-based testing is applied. As we can see in the table, it is mandatory for all ASIL ratings (A to D).
PS: The ‘+’ symbol in the table implies Recommended and ‘++’ implies strongly recommended
Interface Testing as per ISO 26262 standard: Correct implementation of interfaces is very strongly recommended by the ISO 26262 unit test recommendation table. This is because, every software unit may have been tested individually but their communication with other units is also very critical.
During the interface testing, the value range of the incoming signals from the interface is defined and divided in classes. These classes have a boundary value which is used to define the test cases. After the test cases are defined interface testing can be performed.
Interface testing is performed to detect early failures that could go unnoticed till the integration testing process. At that stage, it would be even difficult to localize the issue.
Fault Injection Testing: Quite a self-explanatory term, fault injection implies the insertion of some arbitrary faults. This may include corrupting the values of CPU registers, code mutation, corrupting the variable value and more. Faults can be injected either at compile time or run-time.
This testing is done to ensure that the software unit behaves in an expected manner at the occurrence of any error.
Fault Injection Testing is strongly recommended, but only for automotive products that require ASIL D compliance.
Resource Usage Test: This test is performed to ensure that the unit being tested does not eat up a lot of CPU memory. Resource Usage testing is usually done on the target platform or the simulator. This test is strongly recommended only when the software product is targeting to be ASIL D compliant.
Back-to-back Comparison Test: The software unit under test must behave like the model based on which the unit was designed. Back-to-back comparison test achieves exactly that.
For this test method, a model is required that can simulate the unit being tested. The test suites from the source code are applied to the model and vice versa. The results of both the scenarios are compared and output is recorded.
ISO 26262 Recommendation Table 11- Derivation of Test Cases
The entire process of requirement based testing depends on the test cases you have defined. ISO 26262 underlines some methods to derive these test cases to ensure the software is assigned the required ASIL level
We look at some of the test case derivation methods at a glance:
Requirement Analysis: Every software unit will have some behavioral requirements. For instance, if a function is supposed to return ‘0’ as the output for a negative input value, we will keep -1 as a test case value.
If you are using some unit testing tools, these values will be generated based on the requirements. It is strongly recommended for all ASIL levels.
Analysis of the Equivalence Classes: There is a range of input values for which the behavior of the software unit should be similar. This range is called the equivalence class.
These classes can be identified based on the input and output division. The input range thus derived is categorized under a subset of the unit. Once the equivalence classes are determined, a value that represents the class is chosen and test class is generated based on it.
Equivalence class test generation is strongly recommended by ISO 26262 recommendations for ASIL B to D compliance.
Boundary Values Analysis: Every input can have a minimum and maximum possible value, called the boundary values. For software being tested for ASIL B, C or D, test generation based on boundary values is strongly recommended.
The Metrics for ISO 2626 Compliant Unit Test Coverage
Statement Coverage: A part of unit testing, statement coverage covers those lines of codes that have been tested at least once. Only one test case is required. Statement coverage is strongly recommended only for ASIL level A and B.
Branch Coverage: Branch coverage is applicable at a decision point, for eg. an ‘if…else’ condition. In such a scenario, both true and false condition need to be executed. Naturally, two test cases are required here. It is strongly recommended for ASIL B, C and D functional safety compliance.
Modified Condition/Decision Coverage (MCDC): This is where Functional Testing becomes a very serious business. MCDC is highly recommended for ASIL level D. Here, every condition in the decision in the ‘if.. else’ condition or the loops, can independently influence the outcome.
Role of Testing Tools in ISO 26262 compliant Unit Test Process
The Unit Testing tools can be quite helpful in helping you achieve the right ASIL level for your software. Running the unit test methods manually can be an extensive task and prone to errors.
Tools like Reactis and CANTATA etc. ease the process considerably. They can help the developers automate the testing process by
- Test case derivation
- Framework generation
- Execution of the test cases
- Test report generation etc.
Many of these tools are built around the ISO 26262 guidelines and offer complete structural coverage metrics at the software unit level.
Concluding Thoughts
ISO 26262 has become a de facto standard for automotive industry in terms of functional safety. The unit testing recommendations by ISO 26262 should be followed in letter and spirit to build a really safe automotive system.
If you are new to ISO 26262 and functional safety, we recommend these blogs & ISO 26262 Webinar for you to have a clear understanding of concepts like safety management, ASIL level etc.
