Runtime and Behavioural Assurance - Can we make AI behave like a SIL 4 system?

Recently, we at SAIF Autonomy have been working with Critical National Infrastructure organisations, UK MoD, the US DoD and a number of defence primes on the development and verification of control systems for autonomous, multi-domain, uncrewed systems (e.g. drones, tanks, submarines, boats). In all cases, a crucial constraint is the unit cost of the autonomous system - where commercial constraints necessitate that the unit price of a drone is significantly less than that of say the equivalent crewed counterpart (e.g. a helicopter).

However, in programmes such as The Loyal Wingman Programme, customers still require upper assurance levels of standards such as DO-178C (typically DAL-A / DAL-B). Therefore, following work done by NASA and partners, standards such as ASTM F3269-21 have been proposed whereby lower assurance AI controllers (or complex functions) are "wrapped" with SIL3/4 safety guards. As one of the first developers of this approach, we have faced a number of challenges - that even touch on broader ethical concerns of AI. Can AI ever be tamed to be safe? Maybe but can it do it on time and to budget?

Speaker

Matt Banham

CTO, SAIF Autonomy

Matt Banham is CTO of SAIF Autonomy, leading technology development for assurance of autonomous systems and physical AI. He has a varied background predominantly within safety-critical systems but also in quantum optics, aerospace, defence, self-drive cars, air traffic management systems and has worked on digital infrastructure for drones and advanced air mobility and leading the OpenAir programme for UTM/ATM integration within the National Air Traffic Service (NATS).