AVUnit – a tesing framework for autonomous vehicles.
Autonomous vehicle (AV) systems must be comprehensively tested and evaluated before they can be deployed. High-fidelity simulators such as CARLA or LGSVL allow this to be done safely in very realistic and highly customizable environments. Existing testing approaches, however, fail to test simulated AVs systematically, as they focus on weak oracles (e.g. `always no collisions’) and lack any coverage criteria measures.
In this line of work, we propose a framework (called AVUnit) for systematically testing AV systems against rich correctness specifications. Designed modularly to support different simulators, AVUnit consists of multiple languages for specifying dynamic properties of scenes (e.g. changing pedestrian behaviour after waypoints) and fine-grained assertions about the AV’s journey. AVUnit further supports multiple fuzzing algorithms that automatically search for test cases that violate these assertions, using robustness and coverage measures as fitness metrics. AVUnit further supports the modeling and testing AV against traffic laws (e.g. the completet set of Chinese traffic laws is built in as a library).
AVUnit is implemented for the LGSVL+Apollo simulation environment as well as the Autoware + CARLA simulation environment. It has been used to finding dozens of issues in Apollo, which indicate that the open-source Apollo does not perform well in complex intersections and lane changing related scenarios and is vulnerable to attacks.
Details on AVUnit can be found at: https://avunit.readthedocs.io/en/latest/index.html