Hear from CIOs, CTOs, and other senior executives and leaders on data and AI strategies at the Future of Work Summit on January 12, 2022. Learn more

Artificial intelligence (AI), machine learning (ML), and other emerging technologies have the potential to solve complex problems for organizations. Yet despite increased adoption over the past two years, only a small percentage of companies believe they are deriving significant value from their AI initiatives. Where are their efforts going wrong? Simple missteps can derail any AI initiative, but there are ways to avoid those missteps and be successful.

Here are four mistakes that can lead to failed AI implementation and what you need to do to avoid or fix these issues for a successful AI deployment.

Don’t solve the wrong problem

When determining where to apply AI to solve problems, look at the situation from the right angle and engage both sides of your organization in design brainstorming sessions, because neither business nor IT has all of it. the answers. Business leaders know what levers can be leveraged to gain competitive advantage, while technology leaders know how to use technology to achieve those goals. Design thinking can help create a complete picture of the problem, requirements, and desired outcome, and can prioritize changes that will have the greatest operational and financial impact.

A consumer products retail company with a 36-hour bill processing schedule recently encountered this issue when she asked for help to speed up their process. A proof of concept found that applying an AI / ML solution could reduce processing time to 30 minutes – a speed increase of 720%. On paper, the improvement looked great. But the company’s weekly checkout process meant improving processing time didn’t matter. The solution never went into production.

When looking at the problem at hand, it is important to relate it to one of three essential business drivers: increasing revenue, increasing profitability, or reducing risk. Saving time does not necessarily translate into increased revenue or reduced costs. What business impact will the change bring?

Data quality is critical to success

Data can have a decisive impact on AI programs. Clean, reliable and accessible data is essential for obtaining accurate results. The algorithm may be good and the model efficient, but if the data is of poor quality or is not easy and feasible to collect, there will be no clear answer. Organizations need to determine what data they need to collect, whether they can actually collect it, how difficult or expensive it will be to collect it, and whether they will provide the necessary information.

A financial institution wanted to use AI / ML to automate loan processing, but missing data elements in the source records created a high error rate, causing the solution to fail. A second ML model was created to examine each record. Those who have reached the required confidence interval have been moved forward in the automated process; those that did not have been removed for human intervention to resolve data quality issues. This multi-step process significantly reduced the human interaction required and allowed the institution to increase its efficiency by 85%. Without the additional ML model to manage data quality, the automation solution would never have enabled the organization to achieve meaningful results.

In-house or in third parties? Everyone has their own challenges

Each type of AI solution brings its own challenges. In-house developed solutions provide more control as you develop the algorithm, clean the data, test and validate the model. But building your own AI solution is complicated, and unless you use open source, you will face costs related to licensing the tools used and the costs associated with the initial development and maintenance of the software. the solution.

Third-party solutions present their own challenges, including:

  • No access to the model or its operation
  • Inability to know if the model is doing what it is supposed to do
  • No access to data if the solution is in SaaS mode
  • Inability to perform regression testing or know false acceptance or error rates.

In highly regulated industries, these issues become more difficult as regulators will ask questions on these topics.

A financial services company was looking to validate a SaaS solution that used AI to identify suspicious activity. The company had no access to the underlying model or data and no details of how the model determined which activity was suspicious. How was the company able to perform due diligence and verify that the tool was effective?

In this case, the company discovered that its only option was to perform simulations of suspicious or harmful activities that it was trying to detect. Even this method of validation presented challenges, such as ensuring that testing wouldn’t impact negatively, create denial-of-service conditions, or affect service availability. The company decided to run simulations in a test environment to minimize the risk of impact on production. If companies choose to take advantage of this validation method, they should review service agreements to verify that they have the authority to perform this type of testing and should consider the need to obtain authorization from other third parties. potentially affected.

Invite all the right people to the party

When considering developing an AI solution, it’s important to include all relevant decision makers from the start, including business, IT, compliance, and internal audit stakeholders. This ensures that all critical requirements information is gathered before planning and work begins.

A hotel company wanted to automate its process for responding to data subjects access requests (DSAR) as required by the General Data Protection Regulation (GDPR), the strict European data protection law. A DSAR requires organizations to provide, upon request, a copy of all personal data the company holds for the requester and the purpose for which it is used. The company hired an external vendor to develop an AI solution to automate elements of the DSAR process, but did not involve IT in the process. The resulting definition of requirements did not align with the technological solutions supported by the company. While the proof of concept confirmed that the solution would result in a more than 200% increase in speed and efficiency, the solution was not put into production because IT was concerned that the long-term cost of the maintenance of this new solution does not exceed the savings.

In a similar example, a financial services organization did not involve its compliance team in developing requirement definitions. The AI ​​solution under development did not meet the organization’s compliance standards, the provability process was not documented, and the solution did not use the same Identity and Access Management (IAM) standards. ) than those required by the company. Compliance stalled the solution when it was only partially in the proof-of-concept stage.

It is important that all relevant voices are present from the start when developing or implementing an AI / ML solution. This will ensure that the definition of requirements is correct and complete and that the solution meets the required standards and achieves the desired business goals.

When considering AI or other emerging technologies, organizations need to take the right steps early in the process to ensure success. Above all else, they need to make sure that 1) the solution they are looking for meets one of three key objectives – to increase revenue, improve profitability, or reduce risk, 2) they have processes in place to obtain the data needed, 3) their build purchasing decision is well founded, and 4) they have all the right stakeholders involved from the start.

Scott Laliberte is Managing Director of the Emerging Technologies Group at Protiviti.


VentureBeat’s mission is to be a digital public place for technical decision-makers to learn about transformative technology and conduct transactions. Our site provides essential information on data technologies and strategies to guide you in managing your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the topics that interest you
  • our newsletters
  • Closed thought leader content and discounted access to our popular events, such as Transform 2021: Learn more
  • networking features, and more

Become a member