If we deploy automation without thinking strategically about intelligence, too, isn’t AI likely to backfire on us?
Airplane manufacturer, Boeing, made headlines in 2019 for all the wrong reasons. Its 737 Max aircraft was indefinitely grounded after two fatal crashes in the space of just six months had claimed the lives of 346 people. Investigation into the accidents revealed that updates to an automated system – the Maneuvering Characteristics Augmentation System, known as MACS – had failed to integrate one of two intelligent sensors, meaning the system lacked a critical security backstop. As the aircrafts switched into autopilot mode shortly after takeoff, the error sent them both into fatal nosedives within minutes.
These tragedies highlight an issue with automation that needs more focused attention says Goizueta’s Ruomeng Cui, assistant professor of information systems and operation management. And it’s this: if we deploy automation without thinking strategically about intelligence too, is AI likely to backfire on us?
Cui is an expert in operations strategies in digital retail and platform markets. To better understand the challenges surrounding automation and intelligence in operational processes, she teamed with Shichen Zhang of Tianjin University and Rutgers’ Meng Li to explore how AI brings value in the procurement space.
With Deloitte reporting that almost 45% of Chief Procurement Officers globally are now using, piloting, or planning to integrate AI into their operations, these insights should provide interesting food for thought, says Cui.
“AI isn’t just about being quicker, it’s also about being smarter. It can deliver automation but can also deliver predictive intelligence; and while these two dimensions might be correlated, one doesn’t necessarily imply the other – as the Boeing example demonstrates,” says Cui.
From the tech perspective, there’s a lot of buzz about how AI is helping to drive decision-making, she adds. But there is still plenty that we don’t know about the operational dimensions to using artificial intelligence.
“With international procurement, you’re basically talking about big retailers going in and requesting prices for goods or products from wholesale suppliers. And that’s a process that could, in theory, lend itself very well to AI, since it can automate simple (and tedious) tasks over and over again. So there’s a significant potential gain in companies outsourcing this kind of task to the machine.”
But although the potential might be clear, Cui and her colleagues believe that simply automating these processes might not in fact yield optimal results; and could in fact work against buyers by encouraging suppliers to quote higher prices than they might in personalized, human transactions.
Who Comes Out Ahead on Price? Humans or AI Chat Bots?
“We speculated about the possibility of wholesalers discriminating against the AI,” says Cui. “Specifically, we wanted to know if the sellers would quote higher prices to AI bots than they would to human buyers, because at the end of the day these bots are just machines; they don’t bring the authenticity or sincerity of human beings.”
Cui was also keen to understand whether AI bots equipped with predictive intelligence would fare better: whether the “smartness dimension” would offset any potential discrimination on the part of wholesalers.
“We wondered whether signaling intelligence in some way would make suppliers trust the bots more and how that might impact price quotes.”
To put all of this to test, Cui, Zhang and Li created a landmark field experiment – the first of its kind to explore AI in the procurement setting – using Alibaba 1688, the largest B2B platform in China.
The platform connects 30 million enterprise buyers and suppliers in 49 major categories, and uses a built-in instant chat system called Aliwangwang that enables buyers to contact suppliers for product specifics and prices. Buying companies are allowed to embed autonomous chatbot features in Aliwangwang in order to automate communications.
Cui et. al collaborated with a large, Chinese retail company using the 1688 platform to buy car accessories.
“The company sends out requests for prices via human procurement managers and an automated chat bot,” says Cui. “So we could see pretty clearly how price quotes vary when it’s AI or actual human beings making the contact with the suppliers.”
In the first part of the experiment, Cui et. al programmed the AI bot to simply identify suppliers and request price quotes automatically. These quotes were then compared to those received by the human procurement managers. In total, the researchers parsed just under 4000 product price quotes from just under 4000 wholesale suppliers.
What they found was consistent price discrimination against the chat bots.
“We found that when there is just automation, suppliers give a significantly higher quote to the AI bots than they do to human agents,” says Cui. “The effect is consistent: chatbot, female, and male buyers receive an average price discount of 18.01%, 19.15%, and 20.96%, respectively. In other words, simply automating the process using AI doesn’t actually help they buyer, and actually seems to backfire in this setting.”
Cui puts this down to something called algorithm aversion: human distrust or dislike of machines and robots.
When Machines are Smart, Discounts Rise
“When wholesalers are just asked over and over for their prices, they know that they are dealing with a machine and the intuition is that the machine is not intelligent, that it doesn’t have market expertise, and that it isn’t capable of decision-making. There’s no incentive to build relationships or to engage in any kind of negotiating dynamic here.”
However, this kind of price discrimination effectively disappears when the AI bots are programmed to evince some form of intelligence.
“In the second part of the experiment, we added the dimension of smartness,” says Cui.
“We used an algorithm to analyze different suppliers, and to make selections based on the lowest market prices for goods. When each supplier received the request from the AI bot, they also got a message telling them about this process. In other words, the bot was programmed to signal the capability of being smart and selective. And here we saw significantly different results.”
Looking at the data, they found that “smart” chat bot was getting higher price discounts than human peers. Male buyers, female buyers and smart bots were discounted 21.04%, 18.76% and 22.57% respectively.
“There’s a long tradition of asking for prices in person and building human relationships – especially when companies are working with smaller sets of suppliers. But this is changing. As AI ramps up we’re seeing more and more applications in industry and automation is on the up especially in those first few rounds of screening and asking around.”
The takeaway for procurement executives, she says, is to get smarter about the use of AI. “Automation is the future because of the huge gains it offers in productivity. But to get the better results, you really need to be thinking about the smartness element in tandem with the automation piece, and you need to align the development of both as you roll out AI in your operations.”
That smartness can be simple. It can take the form of nuanced messages that signal intelligence. The key, says Cui, is to be strategic. “AI is a strategic tool, not just a timesaver, and you can’t let the automation part get way ahead of its intelligence. You need it to be smart or it might just backfire.”
