A recent study by researchers at UC Merced has highlighted a troubling trend: people are overly trusting of artificial intelligence (AI), even in simulated life-or-death scenarios.
The study found that nearly two-thirds of participants changed their decisions based on advice from a robot, even though they were warned that the AI had limited capabilities and its guidance could be wrong.
In reality, the AI’s advice was completely random, raising concerns about how easily people can be influenced by technology, especially when the stakes are high.
Professor Colin Holbrook, the study’s lead investigator from UC Merced’s Department of Cognitive and Information Sciences, voiced his concerns about the growing overreliance on AI. “As AI advances rapidly, we need to be aware of the risks of overtrust,” he said.
The study shows that people tend to trust AI more than they should, even when it comes to crucial decisions where mistakes could be disastrous.
Holbrook emphasizes the importance of maintaining a balanced approach to AI, especially in high-risk situations. “We should have a healthy skepticism about AI, particularly when dealing with life-or-death decisions,” he added.
The study, published in Scientific Reports, involved two experiments where participants controlled a simulated armed drone tasked with deciding whether to fire a missile at a target.
During the simulation, participants were shown eight images of potential targets, marked as either allies or enemies, for a brief moment. They then had to rely on their memory to make the critical decision: is the target an ally or an enemy? Should they fire or hold back?
After making their initial decision, participants received input from a robot, which offered its own judgment. The robot either agreed with the participant’s choice or disagreed, providing additional comments like “I hope you are right” or “Thank you for changing your mind.”
Despite the randomness of the robot’s advice and the participants’ awareness of the robot’s limitations, two-thirds of participants allowed the robot to influence their final decision.
The study used different types of robots to see if their appearance affected participants’ willingness to trust them. In some scenarios, a human-like android was physically present in the room, while in others, a less human-looking robot appeared on a screen.
The results showed that people were slightly more likely to trust the human-like robots, but overall, participants were swayed by the AI regardless of the robot’s appearance.
One key finding was that when the robot agreed with the participant’s initial decision, they were far more confident in their choice.
However, when the robot disagreed, many participants changed their decision, dropping their accuracy from around 70% initially to about 50% after the robot’s unreliable advice. This indicates that participants trusted the AI’s input, even when it led them away from the correct answer.
To add emotional weight to the simulation, researchers showed participants images of innocent civilians, including children, and the aftermath of drone strikes before the experiment began.
Participants were urged to treat the simulation as though it were real and to avoid mistakenly harming innocent people. Interviews and surveys afterward confirmed that participants took the task seriously, which makes their overtrust in AI even more concerning.
Holbrook explained that while the study focused on military-style decisions, its implications extend to other areas of life where AI could influence high-stakes decisions.
For example, police might rely on AI to decide when to use force, or medical professionals could be swayed by AI when choosing which patients to treat in an emergency.
The findings even apply to big personal decisions, such as buying a house, where AI could have undue influence over our choices.
“Our project was about high-risk decisions made under uncertainty when the AI is unreliable,” Holbrook said, emphasizing that the study’s goal was to explore how people respond to AI guidance in uncertain and risky situations.
This research adds to the ongoing debate about how much trust we should place in AI as it becomes increasingly integrated into our lives.
While AI has made remarkable advancements, Holbrook warns that we shouldn’t assume it is capable of ethical judgment or fully understanding the complexities of the real world.
AI’s intelligence, while impressive in certain areas, may not translate to other domains, especially those involving ethical considerations.
“We see AI doing extraordinary things, and we think that because it’s amazing in one area, it will be amazing in another,” Holbrook said. “We can’t assume that. These are still devices with limited abilities.”
The study serves as a reminder that while AI has great potential, we must be cautious about giving it too much control in situations where human judgment and ethical considerations are critical.
Maintaining a healthy level of skepticism and being aware of AI’s limitations is crucial as we navigate the increasing presence of these technologies in our daily lives.
The research findings can be found in Scientific Reports.
Copyright © 2024 Knowridge Science Report. All rights reserved.
