Tech Target: Enterprises chasing AI confront a harsh reality

by Antone Gonsalves, Tech Target News Director

Enterprises trying to tap generative AI for higher productivity and revenue growth are battling an expensive, immature technology with an elusive return on investment.

That's according to recent reports from KPMG, McKinsey & Co. and Goldman Sachs that reflect the GenAI experiences of many large organizations. A critical challenge with the technology is generating a return that justifies the high cost of deploying GenAI infrastructure at scale.

A recent AI survey from management consultant McKinsey shows that only 11% of companies polled have deployed GenAI broadly, Aamer Baig, a senior partner at the firm, told attendees of the MIT Sloan CIO Symposium in May. Also, only 15% of the respondents reported earnings improvements from GenAI.

Justifying cost is critical because AI infrastructure is expensive. For example, NVIDIA's previous-generation H100 AI GPU costs roughly $30,000, and the number needed can range from a couple of hundred to thousands, depending on the model and its size.

"The technology is very new; it barely works, and it's very costly," said Anshul Chaturvedi, managing director at IT services provider World Wide Technology (WWT).

A large language model (LLM) today is a "black box" that's difficult to make accurate and ensure consistent responses, said Rob Mason, CTO of Applause. The company that tests the results of AI in software from the user's perspective.

"Inside of the companies that make GenAI, people are studying what they built to figure out what it will do, as opposed to what they want it to do," Mason said.

A Fortune 100 company that hired AI application specialist Trustbit, acquired this year by IT consultancy Timetoact Group, had to deploy three different models internally to ensure at least 95% response accuracy, said Trustbit technical consultant Rinat Abdullin.

If the three models provided the same answer, it was considered accurate. Less than that, and a human would decide which answer was correct. Accuracy was critical because customers could hold the business liable for the wrong answer.

"Companies don't always need 100% accuracy, but they want to be certain that when the model provides an answer, if it's not sure, then they will know about that," Abdullin said.