GenAI strategy dictates ROI challenges for IT leaders

Watching SaaS GenAI costs

A multitude of SaaS vendors are bundling generative AI features into their products in a bid to monetize the technology. Orlandini categorized businesses that purchase such ready-made, off-the-shelf capabilities as GenAI "consumers." For example, consumer organizations might purchase Copilot through the Microsoft 365 product suite or Adobe's Firefly image generator through its applications, he noted.

The consumption model might be the easiest path for ramping up GenAI, but it's not without financial challenges.

Prasad Ramakrishnan, who last month retired as CIO of Freshworks but continues to advise the company, said CIOs must carefully consider vendor pricing and be prepared to negotiate contract terms. Freshworks, based in San Mateo, Calif., provides SaaS-based CRM and IT service management tools.

Those discussions should revolve around how much IT leaders are willing to pay for generative AI and its net benefits, above and beyond a vendor's core product, Ramakrishnan said. It's also important to avoid overcommitting to a new and largely untested technology, he added.

"Don't go for a long contract," he said. "No more than a year."

But there are savings to be had beyond the license price. John Buccola, CTO at E78, a service provider based in Oak Brook, Ill., acknowledged that companies can negotiate discounts on software licenses as part of their GenAI business strategy. But he places a greater emphasis on the cost-cutting potential of license management. E78 provides advisory and managed services to private equity firms and their portfolio companies.

Enterprises are overprovisioning their SaaS-based GenAI licenses and, thereby, overpaying for the technology, Buccola contended. The task: Determine who is using the licenses, and reconcile that against what the organization spends on them.

"There's a lot of opportunity in SaaS to make sure that the amount of licensing in the estate matches the utilization of those licenses," he said. "The savings are not going to be so much on the buy rates as they will be on the stewardship."

Assessing the business benefits of SaaS GenAI

License optimization is only part of the ROI equation, however. There's also the matter of the business benefit, which, in the case of SaaS-based GenAI, revolves around productivity gains.

Here, the employee utilization rate helps document the technology's business value. But organizations must also quantify whether the tool saves employees' time compared with previous technologies, Orlandini said. Measuring the before-and-after effects of a vendor's GenAI offering typically calls for an employee survey, he said.

Such surveys become particularly critical when generative AI vendors don't offer ways to track tool use.

Mike Mason, chief AI officer at Thoughtworks, a Chicago-based technology consultancy, said AI assistants, such as GitHub Copilot, don't provide per-developer utilization statistics. He believes that's because the vendor doesn't want to create the appearance of providing spyware for developers.

But, ultimately, an organization must decide whether the advantages of GenAI justify the additional fee, which can tack on $25 to $40 per user each month depending on the offering.

Mason said he talks with executives who are doing the math on $30 a month times the number of users and concluding the cost "couldn't possibly be worth it." But that assessment fails to consider what employees could end up achieving with GenAI, he added.

"Personally, I get way more than $30 per month of value out of the AI tools that I use for work," Mason said.

To support that point, Mason used ChatGPT to conduct a quick, back-of-an-envelope calculation of how much time a Fortune 500 employee needs to save during a month to at least break even on a $30-per-month GenAI charge. His assessment: An average employee on a $54-per-hour wage needs to save about 34 minutes per month to recoup the monthly fee.

SaaS GenAI utilization a moving target

While some industry executives view such savings as readily obtainable, the business value of GenAI can vary markedly from product to product. That's the situation with the Copilot tools embedded within various Microsoft products, according to Buccola. Value, when measured as utilization, becomes a moving target, he said, noting that employees can start working with Copilot in one product but change six weeks later to another product's instance of the technology.

Copilot's accessibility within products plays an important role in sustained tool use, Buccola said. It's very much front and center in Outlook, offering users the ability to quickly draft messages, he said. Copilot in Excel, on the other hand, offers a greater accessibility challenge. Excel users need to save spreadsheets in a Microsoft OneDrive location and format them in a particular way to use Copilot, he added.

"It remains to be seen which tools make the best use of AI," Buccola said. "We see a huge variation in the Microsoft suite in terms of how much Copilot is used."

Rising infrastructure costs with GenAI sophistication

Enterprise GenAI consumers might focus on avoiding the pitfalls of overprovisioning licenses or miscalculating business value. Organizations taking a more bespoke approach to GenAI have other ROI considerations, however.

Companies that Orlandini termed GenAI "adapters" take GenAI foundation models and modify them for specific use cases. Most organizations at the midmarket level and above end up doing some form of adaptation as they build AI-based products, he noted. Such models might be hosted in public or private clouds. GenAI's demand on such resources -- cloud compute and storage -- can heavily contribute to the cost of scaling the technology beyond the pilot stage.

Accrete, an enterprise AI company based in New York, runs its Nebula platform on AWS. The company provides AI services through Nebula, which is built on modified transformers and large language models. Accrete uses GenAI services from OpenAI in addition to its proprietary technology. Peter Bierfeldt, Accrete's CISO, said the company's cloud infrastructure costs had begun to escalate, prompting a cost control regimen.

"We had a pretty substantial burn rate, where we were probably looking at, annualized, $2 million a year," he said.

Cost contributors included Amazon Elastic Compute Cloud (EC2) instances, Amazon SageMaker and Amazon Elastic Block Store, among other AWS services. Accrete's initial cost-cutting venture involved moving a "tremendous amount" of EC2 instances to containers and the Kubernetes management platform, Bierfeldt said. That migration took place in 2023. This year, the company deployed Kubecost, a Kubernetes cost monitoring and management application.

The combination of Kubernetes and Kubecost has reduced Accrete's AWS spend by 40% to 45%, Bierfeldt said. That works out to a bit over $1 million in cost savings this year, he estimated.

Costs, while they still exist on the container side, are easier to track in that setting compared with EC2, he added.

"It's basically burning money 24 hours today," he said of an EC2 instance. "Now, with Kubernetes and Kubecost, we're going to monitor this a little bit more closely."

Optimizing cloud resources for cost efficiency

The ability to optimize resource use through containers also contributes to cost efficiency. Accrete can add containers to accommodate a spike in demand and scale them down again within 10 to 15 minutes when the load drops off, Bierfeldt said. This flexibility lets the company quickly switch off unused resources.

Model run costs can quickly go up. We are seeing a whole bunch of architecture patterns and new deployment strategies.

Accrete also has trimmed the cost of running SageMaker, which companies use to customize and deploy AI models. Some processing workloads don't require the compute resources SageMaker provides, Bierfeldt noted. Those workloads have been moved to containerized instances instead.

As a result, "we have been able to rely less on SageMaker," he said.

Bierfeldt said Accrete has repurposed a portion of its cloud cost savings to hire more engineers, researchers and salespeople. The spend reduction will also help the company hold the prices of its AI services down as it adds more customers, he added.

Accenture's Tay said companies providing GenAI services are rethinking infrastructure as costs accumulate. He's seen growing interest among customers in exploring deployment options along a continuum of cloud infrastructure: private, public and hybrid. IT leaders must decide which models run best in which environments, considering cost and accuracy, among other factors.

"Model run costs can quickly go up," Tay said. "We are seeing a whole bunch of architecture patterns and new deployment strategies."

John Moore is a writer for TechTarget Editorial covering the CIO role, economic trends and the IT services industry.