A leading industry data science expert has been awarded a Simon Industrial and Professional Fellowship at AMBS to research on the role of trust in AI systems.

Dr. Darminder Singh Ghataoura is Director of AI & Data Science within the Defence & National Security division of Fujitsu UK, and was brought in by the company two years ago to build and lead its offerings and capabilities in AI, data science and data strategy.

In January he will start the six-month Fellowship looking at the important areas of trustworthy AI and AI decision making in the defence sector and elsewhere, for example at how simulation techniques and gaming technologies could be used by the defence industry to help address supply chain logistics in specific scenarios.

As he explained: “As the defence industry integrates AI into its systems and missions there are questions about the role of trust in human-machine teams. To address these concerns there have been continuing technological advancements that look to build trust into AI systems by enhancing system functions and features like transparency, explainability, auditability, reliability, robustness, and responsiveness.

“However little effort has been made to address solutions which exploit research on human attitudes towards ‘machine-teaming’, accounting for the differences in people’s perceptions and experiences, especially within dynamic and changing environments such as a battlefield scenario.”

Human-machine teaming

He added that understanding human-machine teaming requires us to pay attention to three sets of factors - those focused on the human (cognitive workload), the machine, and the interactions - all of which are inherently intertwined, affecting each other and shaping trust.

“This is all about trying to develop trustworthy AI solutions based on user experience. Battlegrounds are inevitably high-paced and stressful environments so we want to find out how machines can help take decisions away from humans in a safe and secure way while ensuring that the correct balance between humans and machines is always achieved in terms of teaming.

“It might sound surprising but the defence industry is very open-minded about the potential use of AI. In particular it appreciates that where there are human resources gaps then these could potentially be plugged with autonomous solutions. These concepts are not actually new as drones have been used for several years now, but where we are lacking is research into how this can be applied to live operational scenarios.”

Critical

Yu-Wang Chen, Professor of Decision Sciences and Business Analytics at AMBS, added: “As we become increasingly dependent on digital technology, it is critical to understand whether we can trust the security and resilience of the AI systems we use and interact with in an industry such as defence. The ability of using AI systems to provide trustable and responsible outcomes for the different stakeholders also becomes increasingly critical, from a business, legal and ethical perspective.”

The Fellowship will develop expertise that addresses the nature of human trust in cognitive workloads/user experiences within fast and uncertain environments. It will also leverage the academic community to test out ideas in a real-life situation, thereby influencing Fujitsu business direction, policymakers and practitioners by testing different ways of sharing expertise around driving trust in AI systems.

Knowledge exchange

The Fellowship also hopes to identify opportunities for collaborative research projects, knowledge exchange, and teaching drawing on the full range of academic capabilities across AMBS and The University of Manchester.

It will specifically develop a strategy to identify key academics and knowledge exchange opportunities for research impact in this area, drawing in expertise around human perception for human-machine teaming and trustworthy AI.

It is hoped that it will also lead to future Knowledge Transfer Partnerships (KTPs) between Fujitsu and The University of Manchester, while the new Data Visualisation Observatory at AMBS could also be used as part of the research project in terms of creating simulated environments.