CIOs weigh generative AI costs as ROI comes into focus
GenAI deployment expenses defy conventional wisdom. IT leaders must consider the potential for higher-than-expected change management and model run costs when calculating ROI.
CIOs increasingly are scrutinizing generative AI projects with a sharper business-value focus, paying particular attention to the cost of GenAI technology as well as its potential benefits.
That's a departure from the early days of generative AI, when enterprises were mostly concerned with exploring the technology's possibilities and cultivating myriad ideas for use cases. The business case for the technology is now becoming more important as organizations look to expand generative AI beyond initial pilots. Two related imperatives are emerging among enterprise adopters: identifying use cases with the best prospects for ROI and spotting generative AI costs that could erode financial gains.
In that context, GenAI follows the course of conventional IT deployments, which, ideally, hinge on a financial rationale and built-in cost controls.
"Last year, what we saw was a lot of experimentation," said Juan Orlandini, CTO for North America at Insight Enterprises, a solutions integrator based in Chandler, Ariz. "This year, we're finally looking at GenAI as just another capability. We still have to have a traditional enterprise application's justification and ROI."
That's particularly critical for companies with small GenAI teams, limited budgets and little margin for error.
Danielle Conklin, CIO at Quility, an online insurance -- or insurtech -- company based in Swannanoa, N.C., said it has a two-person data science team, including herself. Instead of strictly using off-the-shelf large language models (LLMs) for GenAI, Quility aims to create its own advanced models, Conklin said. Initial use cases include customer engagement and CRM. But she added that cost and ROI are key considerations.
"To get to a sophisticated level requires time and people and resources," Conklin said. "With two people, we can only focus on one or two things. We have to make sure the one we are choosing is the one thing that is going to have a high return on investment."
She said cost involves much more than the initial investment in two people's time: "Do we need to use other vendors? Or third-party data? Do we need data cleansing tools and data quality tools? And there's a long-term cost of maintaining the model [and] refreshing the model."
Uncovering generative AI costs: Managing change and preparing data
IT leaders are likely to find higher-than-expected outlays as they examine the economics of generative AI in greater detail. Aamer Baig, a senior partner at McKinsey & Company, said enterprises could be taken in by GenAI's relatively low startup costs. McKinsey research found GenAI models drive only about 15% of a typical project's cost.
We all grew up with certain orthodoxies around how cost is estimated. And we're finding that a lot of those orthodoxies are not turning out to be true with generative AI.
Aamer BaigSenior partner, McKinsey & Company
But other, less obvious costs can boost a project's price tag compared with conventional IT initiatives.
"We all grew up with certain orthodoxies around how cost is estimated," Baig said, speaking earlier this year at the 2024 MIT Sloan CIO Symposium. "And we're finding that a lot of those orthodoxies are not turning out to be true with generative AI."
"A few years ago, we made quite a splash by saying you need to budget as much for change management as you do for development," he said, referring to digital transformation efforts. "Now, [with GenAI], we're finding as much as three times [the development cost] needed for investment in change management."
Generative AI, like digital transformation, requires change across workflows, business processes, policies and KPIs, Baig said. But GenAI also involves new change management considerations such as prompt engineering and specialized AI training.
Mike Mason, chief AI officer at Thoughtworks, a technology consultancy in Chicago, also cited the importance of change management in GenAI projects.
"Change management is something we see organizations not paying enough attention to," he said. "You're talking about changing the way humans do their jobs -- you can't discount the change management aspect of that."
Mason also cited AI readiness as a cost that organizations should include in their calculations. That includes the readiness of data to support AI applications. Data must be available, as opposed to confined in storage silos, and cleansed before it's fed into a GenAI system, he said. An IT department might need to upgrade infrastructure to make that happen. Steps could include cloud migration and the adoption of a modern data platform, Mason added.
He shared the example of a Thoughtworks life sciences client that pursued data modernization to make data more available and support the use of GenAI in drug discovery. The company had preclinical trial data scattered across numerous data stores, Mason explained. As a result, drug researchers struggled to find information on the company's previous experiments and incurred unnecessary costs rerunning tests. The life sciences company deployed a data mesh, which provides a unified platform for accessing data on experiments and trials.
"Visibility into existing data can build a very strong ROI case," Mason said.
Quility, meanwhile, also focuses on data as part of its GenAI efforts. The insurer uses Snowflake as its enterprise data warehouse and Apache Kafka, an open source distributed event streaming platform that supports data pipelines and data integration in organizations.
"We want to be a data-driven company," Conklin said. "We want to give [employees] information at the time of decision."
Looking beyond conventional run costs vs. build costs
The ongoing expense of operating a generative AI application could prove another unexpected cost that hampers ROI. With digital transformation, a commonly accepted projection is that the cost of running applications ranges from 15% to 30% of build costs, Baig said.
"With GenAI, I humbly suggest you throw that out the window," he said, noting that run costs for generative AI models might equal build costs depending on the use case.
CIOs should factor in hidden costs when deploying GenAI at scale.
Mason said the costs of running a model and the inference process, in which the model interprets new data, will usually dwarf the cost of training a model. Some of those costs, moreover, can prove hard to predict. For example, the way GenAI vendors price API calls to their LLMs complicates pricing and cost projections. Vendors use a system of tokens to price those calls, with longer textual responses eating up more tokens.
"Token-based pricing is new for organizations, and I think it is less predictable," Mason said.
When a user gives an LLM an input, the result could be a short or long answer, he said. Accordingly, token-based pricing makes it difficult for organizations to determine the actual cost of running an application until it is in production, he added.
Vamsi Duvvuri, Americas technology, media and entertainment, and telecommunications AI leader at consultancy EY, cited cost uncertainty as one of the takeaways from the first wave of generative AI projects.
"Companies are still struggling with managing and predicting costs around GenAI," he observed.
Duvvuri said most of the current cost models don't deliver economies of scale within pay-as-you-go scenarios. PAYG is the approach many businesses take when they start using generative AI, he added.
One positive cost development: The price of generative AI models such as ChatGPT 4o or Claude 3.5 has declined recently due to competitive pricing and efficient architectures, Duvvuri said. Technology adopters, however, should still focus on controlling GenAI expenses.
"Enterprises must do the hard work of optimizing the unit cost of work done by AI models," Duvvuri said.
That task for IT leaders is scaling an AI system's underlying technical and functional patterns, he noted. Technical patterns include retrieval-augmented generation and multimodel chaining. RAG is used to boost the accuracy of LLMs, while model chaining aims to improve the quality of model output. Functional patterns include summarization/classification, translation and composition, Duvvuri said.
Technical choices early on in a generative AI deployment can dramatically influence cost and ROI. Cost variance might range from 10x to 20x with GenAI, compared with 1x to 5x with digital transformation, Baig noted.
"I personally have been a very strong believer in the power of great technical choices," he said. "A very strong business case, making the right decisions technologically and from a business model standpoint -- that can drive huge swings in your cost. So, the big decisions upfront really matter."
John Moore is a writer for TechTarget Editorial covering the CIO role, economic trends and the IT services industry.
