The many ways human-AI systems can fail dangerously; how important ones are neglected; and how to do better

Abstract:

Human-AI systems - consisting of people supported by AI, supervising AI, etc. - increasingly operate in high-stakes applications like medicine, weapon control, or industrial automation. Hence a growing demand for proper assessment and mitigation of the risk in their operation. Media stories abound about the risk of people trusting AI too much: AI recommends a wrong therapy for a patient, route for a driver, etc; they follow the advice and come to harm. Many studies document "overreliance" on AI. It seems people just need to learn to trust AI less; but the problem is more complex. This talk is meant to contribute to discussion among  colleagues concerned with human-AI systems, across communities that include AI, human factors, safety, security, and application domains like health and robotics. The main points addressed will be: human-AI systems can cause harm for more complex reasons than people's "overreliance" on the AI (I'll discuss examples); developers often miss some of these, so the remedies they adopt may be useless or harmful; assessing such systems requires analysing many interacting factors that affect risk; to organise this complex analysis, it is useful to apply two interrelated viewpoints, that we call "diversity" (to which extent people can remedy errors of the AI, or vice versa) and "adaptation" (how people's behaviour changes when operating with AI).

The Speaker

Lorenzo has worked on how to assess and mitigate risks from use of computers for 40-odd years, of which 30-odd years in the Centre for Software Reliability (CSR) at City; he was director of CSR in 2012-2024. He previously worked for the Italian National Research Council, and has also been a visiting researcher at UCLA, Bell Communication Research and SRI International. His work has involved interaction with industry and regulatory agencies, and diverse application areas, including aerospace, nuclear power, autonomous vehicles, medical automation, election systems. Much of this work is about decisions on whether a system will be reliable, safe and secure enough for its intended use, or how best to make it so; often proposing formal probabilistic arguments to allow scrutiny of decisions that are usually justified in informal, hard-to-audit language. Currently his main interests are (1) risk in human-AI systems and (2) helping safety regulators to take into account undetected errors in the safety arguments they receive.

Microsoft Teams meeting

Join: https://teams.microsoft.com/meet/3942989568805748?p=r8gB6nhExVURqPw0jn

Meeting ID: 394 298 956 880 574 8

Passcode: uw7Zd37j