Automotive electronics has been undergoing a rapid transformation, much like consumer tech such as smartphones or tablets. More often than not, automotive tech and consumer tech are based on similar chip architectures and technologies.
However, they are still a world apart.
Let us consider the case of your automotive infotainment system.
Unlike your smartphone, the IVI system is subjected to constant G forces and vibrations during its lifecycle. It also must withstand extreme temperatures and humidity. There are several legal and regulatory issues associated with infotainment systems.
Additionally, they are integrated with various other automotive subsystems. So, in case of failures, the safety of the passengers should be ensured.
It is, hence, of great importance that infotainment system development teams perform rigorous testing of these devices before they are integrated with the vehicle.
In this article, we analyse the Orthogonal Array Testing technique, commonly referred to as OAT or OATS. It is a reliable software validation methodology for automotive infotainment testing.
The Need for Orthogonal Array Testing
In today’s automotive software development lifecycles, it is common for the development phase to extend beyond the pre-planned timelines. In such scenarios, the timelines for the testing phase may have to be compromised by a week or so.
One of the most desirable qualities in a tester is his/her ability to be flexible. They should understand that they may have planned for 3-4 weeks of testing, but at the end of an extended development phase, they are only given 1.5 – 2 weeks’ time for testing and validation.
The tester should be able to work within these timelines by considering areas that can be automated, onboarding of additional resources, putting in extra effort, etc.
The tester should also be able to optimize the test suite and come up with effective testing combinations that can uncover the bugs quickly. But this doesn’t mean that the tester can perform random testing (as that will not ensure complete coverage).
This is where the concept of Orthogonal Array Testing comes into the picture.
Using the orthogonal array testing methodology, the tester can optimise the test suite so that it is capable of detecting all types of faults using pairwise combinations of input parameters. This will help in minimizing testing efforts, while also identifying the bugs efficiently.
In this software testing methodology, orthogonal arrays are used to create the test cases. It is a statistical testing concept that is particularly useful when there is a large number of input combinations to be validated for the system under test.
By pairing and combining inputs, the total number of test cases are significantly reduced. And if the right inputs are chosen to be paired together by the tester, the testing coverage and efficiency is also enhanced.
Importance of OAT Automotive Infotainment Testing
An infotainment system receives information from a GPS receiver, GSM receivers, RF receivers, and more. So, it is a multi-engineering topic. It receives various types of inputs and multiple combinations of inputs.
In the input test data for validation of infotainment/entertainment applications like CD drives, the number of file formats itself will be huge. Hence, the validation cannot be done within a timeframe of just 2-3 weeks. Also, it will be very expensive to keep testing all different functionalities, one by one.
One more aspect to consider is that after the development cycle, usually the functional testing phase will be on aggressive timelines. So, in a limited timeframe, it is important to test maximum features and uncover the bugs efficiently.
Orthogonal Array Testing During Unit Testing Phase
At Embitel, we perform orthogonal array testing for infotainment systems, right from the unit testing stage.
In unit testing, it is referred to as MCDC (Multiple Condition Decision Coverage). If we do not perform MCDC, we will end up writing a lot of test cases.
In unit testing, as per MCDC, the developer checks the different conditions or input combinations for making a decision. In automotive software validation, it is crucial to check a lot of combinations before we make any decision.
Let’s explore this testing methodology with an example.
OAT can be used in all types of industries, any application, during any phase of testing, any levels or types of testing.
In unit testing, consider that we are validating different combinations of 5 variables. Each variable has 5 different states.
| Combination No | A | B | C | D | E |
| 1 | 1 | 1 | 1 | 1 | 1 |
| 2 | 1 | 1 | 1 | 1 | 2 |
| 3 | 1 | 1 | 1 | 1 | 3 |
| 4 | 1 | 1 | 1 | 1 | 4 |
| 5 | 1 | 1 | 1 | 1 | 5 |
| 6 | 1 | 1 | 1 | 2 | 1 |
| 7 | 1 | 1 | 1 | 2 | 2 |
| 8 | 1 | 1 | 1 | 2 | 3 |
| 9 | 1 | 1 | 1 | 2 | 4 |
| 10 | 1 | 1 | 1 | 2 | 5 |
And so on….
Total no of combinations = 5^5 = 3125 combinations.
After applying Orthogonal Array Testing methodology, the total number of test cases can be reduced to 33.
OATS will remove all the duplicated pairings and provide an efficient list of test combinations as shown below:
| Combination No | A | B | C | D | E |
| 1 | 1 | 1 | 1 | 1 | 1 |
| 2 | 1 | 2 | 2 | 2 | 2 |
| 3 | 1 | 3 | 3 | 3 | 3 |
| 4 | 1 | 4 | 4 | 4 | 4 |
| 5 | 1 | 5 | 5 | 5 | 5 |
| 6 | 2 | 1 | 2 | 3 | 4 |
| 7 | 2 | 2 | 1 | 4 | 3 |
| 8 | 2 | 3 | 4 | 1 | 2 |
| 9 | 2 | 4 | 3 | 2 | 1 |
| 10 | 2 | 5 | 1 | 2 | 4 |
| 11 | 3 | 1 | 3 | 4 | 2 |
| 12 | 3 | 2 | 4 | 3 | 1 |
| 13 | 3 | 3 | 1 | 5 | 4 |
| 14 | 3 | 4 | 2 | 1 | 3 |
| 15 | 3 | 5 | 2 | 4 | 1 |
| 16 | 4 | 1 | 4 | 2 | 3 |
| 17 | 4 | 2 | 3 | 1 | 4 |
| 18 | 4 | 3 | 2 | 4 | 5 |
| 19 | 4 | 4 | 1 | 3 | 2 |
| 20 | 4 | 5 | 3 | 5 | 1 |
| 21 | 5 | 1 | 5 | 1 | 5 |
| 22 | 5 | 2 | 4 | 5 | 2 |
| 23 | 5 | 3 | 5 | 2 | 1 |
| 24 | 5 | 4 | 5 | 3 | 3 |
| 25 | 5 | 5 | 1 | 3 | 5 |
| 26 | 2 | 5 | 5 | 5 | 2 |
| 27 | 3 | 5 | 5 | 2 | 5 |
| 28 | 5 | 5 | 2 | 5 | 3 |
| 29 | 5 | 2 | 5 | 4 | 4 |
| 30 | 2 | 4 | 3 | 5 | 5 |
| 31 | 4 | 5 | 4 | 1 | 5 |
| 32 | 4 | 1 | 5 | 5 | 1 |
| 33 | 5 | 2 | 3 | 1 | 5 |
This tutorial on orthogonal array testing helps you to understand how to create the orthogonal array for a test scenario.
OATS for System Testing of Automotive Infotainment Software
Let us consider the testing process of infotainment system to check whether there is a sudden increase in volume when the audio parameters change.
Audio Parameters:
- Bass
- Treble
- Loudness
- 7 bands of equalizers
Total number of combinations without OATS:
(Bass=2)*(Treble=2)*(Loudness=2)*( 7 Equalizers=2 ^7) = 2048.
After applying orthogonal array testing methodology, the following 10 test cases are found to be sufficient to ensure good testing coverage:
| Case | Band 1 | Band 2 | Band 3 | Band 4 | Band 5 | Band 6 | Band 7 | Bass | Treble | LD |
| 1 | Vallo | Vallo | Vallo | Vallo | Vallo | Vallo | Vallo | Vallo | Vallo | On |
| 2 | Vallo | Valhi | Valhi | Valhi | Valhi | Valhi | Valhi | Valhi | Valhi | Off |
| 3 | Valhi | Vallo | Valhi | Vallo | Valhi | Vallo | Valhi | Vallo | Valhi | On |
| 4 | Valhi | Valhi | Vallo | Valhi | Vallo | Valhi | Vallo | Valhi | Vallo | Off |
| 5 | Vallo | Vallo | Vallo | Valhi | Valhi | Vallo | Vallo | Valhi | Valhi | On |
| 6 | Vallo | Valhi | Valhi | Vallo | Vallo | Valhi | Valhi | Valhi | Vallo | Off |
| 7 | Valhi | Vallo | Valhi | Valhi | Vallo | Valhi | Vallo | Vallo | Valhi | On |
| 8 | Valhi | Valhi | Vallo | Vallo | Valhi | Vallo | Valhi | Vallo | Vallo | Off |
| 9 | Vallo | Vallo | Vallo | Vallo | Vallo | Valhi | Valhi | Valhi | Valhi | Off |
| 10 | Vallo | Valhi | Valhi | Valhi | Valhi | Vallo | Vallo | Vallo | Vallo | On |
Advantages of Orthogonal Array Testing
- OAT method can be used in the earlier testing stages itself. This implies that right from unit testing, this method can be used for evaluation.
- OATS reduces time with increased coverage of test cases (due to combinations/pairings).
- It is best suited for testing applications in which there is a large amount of test data/requirements to validate. For eg., Automotive infotainment testing.
- OATS is an efficient and effective testing process.
Summary
As part of the role of a software tester, especially when performing black box testing, the motive is to try and break the code in every possible way. The system is usually tested from the end-user’s point of view.
Even if the testing is done for a small component of a larger system, it is observed that defects/bugs are found easily when the test data is combined/paired. This is the basis for the orthogonal array testing methodology.
While testing using OATS, it is important to create test cases by pairing the right test inputs and combinations. This will help in uncovering bugs as early as possible, with minimal effort and time investment.
Author
Yogesha Lakkanna, Associate Director – Embitel
Yogesha Lakkanna is responsible for Verification and Validation at Embitel. He is an Electronics and Communication Engineer and Certified Testing Professional. He has 20+ years of experience in Release Management, Quality Assurance and Test Lifecycle management in the field of Realtime Embedded Applications in Avionics, Consumer, Medical and Automotive Electronics.
Yogesha is also interested in Farming, Literature, Music, and Social Service. He is an active member and volunteer in various cultural and social organisations in Bangalore.
