Why AI Projects Fail and How to Fix Them (Based on Research)
The Critical Role of AI Product Managers and Good News for Aspiring Ones
#beyondAI
Have you ever wondered why the AI revolution hasn't transformed companies as promised? Despite substantial investments, many companies find that the anticipated efficiency and effectiveness gains are elusive or, at best, modest compared to the effort invested.
No wonder many companies are still cautious despite the new AI hype. Since OpenAI announced ChatGPT in late 2022, ushering in the era of Generative AI, we've been swamped with new AI products. These AI companies promise quick integration into business processes, easy incorporation into employee workflows, and significant returns on investment from day one. These promises sound familiar to many businesses. Early adopters of the first AI wave have realized that many crucial factors limit AI's effectiveness. Sometimes, it becomes ineffective altogether.
But now, the applications for Generative AI have expanded significantly, technical barriers are more manageable, and AI providers are covering many of these issues. Does this change the game?
No.
Even if we no longer need our own AI infrastructure, don’t have data quality issues, and aren’t lacking experts, there are still plenty of challenges.
Despite having many solutions available today, it’s rare to find someone willing to tackle these challenges, often saying: "That's not my job!"
With this article, I delve into research papers answering the question of why AI projects fail and argue why there must be an explicit role to address these challenges: the AI Product Manager.
I will cover:
Analysis of the current landscape of AI adoption
Key insights from two research papers on AI project failures
Comparative analysis of the research findings
The mandatory role of AI Product Managers
Practical steps to become an AI Product Manager
Happy reading 🛋️
Current Landscape of AI Adoption
AI in businesses was already a hot topic before 2023 and the introduction of large language models and Generative AI. By 2023 in Germany, about one in three large companies (35%) had invested in AI, compared to only one in six (16%) medium-sized companies with 50 to 249 employees and one in ten (10%) small businesses with 10 to 49 employees. These numbers reflect the landscape even after significant advancements in AI (GenAI) are introduced. Source: Destatis
And if we assume that Germany is not one of the early adopters, we can expect slightly larger numbers in AI adoption for each company size in other countries
But even though some companies have already introduced AI, they still need to be successful with it. Companies that haven't yet invested in AI have a good idea of where the implementation might fail, citing a lack of know-how (72%) and the challenge of integrating AI into existing devices, software, and systems (54%). Additionally, many companies view data availability and quality as significant challenges (53%), and others have concerns about data protection and privacy (48%).
These are all valid concerns, confirmed by companies that tackled these challenges years ago. Many of these issues are also reasons why numerous AI initiatives fail.
However, scientific research has also identified other primary causes of these failures.
Analysis of "Beyond the Hype: Why Do Data-Driven Projects Fail?"
The primary objective of the research is to identify and analyze the specific challenges that lead to the failure of data-driven projects in delivering significant business value. Despite substantial investments, many companies find their data science initiatives falling short of expectations. The study aims to systematically explore these shortcomings and provide recommendations to overcome them.
The research approach involves multiple qualitative semi-structured interviews and a questionnaire survey. Initially, 13 domain experts from various roles in data-driven projects were interviewed to identify potential challenges. This was followed by designing, validating, and refining a survey, which was then distributed to 112 experts across eleven industries. The survey data was analyzed to identify the main challenges contributing to the failure of data-driven projects.
The study identified three primary reasons for the failure of data-driven projects:
Lack of understanding of the business context and user needs.
Low data quality.
Data access problems.
Additionally, 54% of respondents noted a conceptual gap between business strategies and the implementation of analytics solutions. The survey also highlighted the following critical and significant impacts on the non-success of data-driven projects: data quality, data access, budget/time constraints, cultural resistance, and lack of both soft and hard skills.
The discussions emphasized the alignment between the study's findings and previous literature. Common challenges include data access, volume, and quality, as well as legal and security issues. The lack of technical expertise, organizational alignment, and a systematic process was also noted. The study suggests that addressing business understanding and user needs is crucial for the success of data-driven projects. Tools like Empathy Maps and the involvement of Design Thinking Leaders can significantly improve project outcomes.
The study concludes that understanding the business goal and user needs is pivotal to the success of data-driven projects. The conceptual distance between business strategies and analytics implementation needs to be bridged. Addressing challenges related to data quality, access, and skills is essential. The study's findings provide a foundation for future research and practical guidelines for organizations to enhance the success rate of their data-driven projects.
Analysis of "Failure of AI Projects: Understanding the Critical Factors"
The primary objective of the research is to identify and understand the distinct factors that are critical for the failure of AI projects. While the adoption of AI has increased significantly, many firms do not achieve the expected benefits or terminate projects prematurely. This study aims to uncover both organizational and technological issues that contribute to AI project failures by analyzing empirical data collected through expert interviews.
The research employs a qualitative approach based on semi-structured interviews with AI experts from various industries. The study utilized inductive coding methods to analyze the interview data and identify critical factors leading to the failure of AI projects. The experts were selected through purposeful sampling, ensuring a diverse range of professional backgrounds and industry experiences. The interviews were recorded, transcribed, and then coded to extract relevant themes and factors.
The study identified 12 factors contributing to AI project failures, which were categorized into five main groups:
Unrealistic Expectations:
Misunderstanding of AI capabilities
Overambitious project scopes
Use Case Related Issues:
Lack of value or poor cost-benefit ratio
High complexity
Low error tolerance
Organizational Constraints:
Insufficient budget
Regulatory hurdles
Lack of Key Resources:
Inadequate expertise
Data availability issues
Technological Issues:
Model instability
Lack of transparency (black box problem)
Potential for result manipulation
The discussion highlighted that while technological issues can cause project failures, non-technical factors such as unrealistic expectations and lack of resources are often more critical. The study revealed that some challenges, like expertise and data availability, are manageable with proper planning, while others, like model instability and result manipulation, are harder to anticipate and control. The findings confirm that previously known challenges like know-how, business impact, and result validation are indeed critical for AI projects. Conversely, factors like communication and infrastructure, while important, do not directly lead to project failure if managed properly.
The study concludes that both technological and non-technological factors can lead to AI project failures. Key recommendations for organizations include:
Evaluating their AI capabilities and resources honestly.
Setting realistic expectations for AI projects.
Addressing critical risks before project initiation.
Ensuring sufficient expertise and data availability.
Organizations should focus on understanding and mitigating the risks associated with AI projects to increase their chances of success. The study suggests further quantitative research to validate these findings and provide more comprehensive guidance for AI project management.
Comparative Analysis
Similarities:
Both papers identify non-technical factors (such as understanding the business context and setting realistic expectations) as critical to project success.
Emphasis on the importance of organizational readiness and proper resource allocation.
Both studies highlight the need for systematic processes and better alignment between project goals and implementation.
Differences:
The first paper focuses more on general data-driven projects, while the second specifically addresses AI projects.
The second paper identifies technological issues such as model instability and result manipulation, which are not explicitly mentioned in the first paper.
The first paper discusses the conceptual gap between business strategies and analytics, while the second paper focuses more on unrealistic expectations and the complexity of use cases.
Why AI Product Management is Not Just Helpful, But Mandatory
The comprehensive analyses of the two research papers provide a clear picture of why many data-driven and AI projects fail.
The insights from these studies reveal that understanding business needs, setting realistic expectations, ensuring resource availability, and implementing systematic processes are pivotal for success.
These might also be the reason why, on average, AI initiatives only deliver a 5% return on investment, while the best-in-class players achieve 13%. Considering the average capital cost of 10%, AI initiatives don't seem like good investments if efficiency and effectiveness gains are the primary goals.
This paints a bleak picture and makes it very understandable why so many business leaders are still hesitant about AI.
But what is the alternative?
No company can afford to skip investing in AI today or in the next few years. If we manage to overcome the challenges, AI has the potential to significantly improve efficiency and effectiveness. Companies that excel in AI first will outpace their direct competitors.
To better start effectively addressing and overcoming these challenges, AI Product Management emerges as not only helpful but mandatory:
Understanding the Business and User Needs
A recurring theme in both studies is the lack of understanding of the business context and user needs. AI Product Managers act as navigators, ensuring AI initiatives are aligned with business goals and tailored to user needs. They bridge the gap between business objectives and technological implementation, keeping projects on course and preventing the misalignment that often leads to failure.
Setting Realistic Expectations
Both papers highlight the issue of unrealistic expectations. AI is often perceived as a magical solution to all problems, leading to overambitious projects. AI Product Managers are the realists in the room, grounding these lofty expectations with practical insights. They educate stakeholders on AI’s capabilities and limitations, ensuring everyone is on the same page. They set achievable goals and prevent projects from becoming overly complex and unmanageable, a critical factor identified in the studies.
Ensuring Adequate Resources
AI projects need the right mix of talent, data, and technology. These are necessary for the best ideas to succeed. AI Product Managers are like skilled chefs who understand the importance of quality ingredients. They advocate for sufficient budgets, hire the right expertise, and ensure necessary data and tools are available. Otherwise, they won’t start the initiative at all. This proactive resource management directly addresses the organizational constraints and lack of key resources identified as major failure factors.
Implementing Systematic Processes
Successful AI projects result from systematic processes, not ad-hoc efforts. AI Product Managers bring in proven frameworks and methodologies, ensuring projects follow a structured path. They act as conductors, coordinating between teams and ensuring harmony in project execution. This structured approach minimizes risks and aligns with the need for systematic processes emphasized in the studies.
Facilitating Communication and Alignment
In the chaotic world of AI, where technical jargon can alienate business stakeholders, AI Product Managers play the role of translators. They ensure clear and constant communication between all parties involved. They foster a culture of collaboration, reduce misunderstandings, and ensure alignment towards common goals, addressing another critical factor highlighted in the research.
Addressing Technological AND Non-Technological Challenges
AI projects face both technological and non-technological challenges. From model instability to cultural resistance, these challenges can derail projects. AI Product Managers are problem solvers who anticipate these issues and devise strategies to overcome them. They ensure projects are technically sound and culturally and legally compliant. Their role is crucial in navigating the complexities of AI projects, a need clearly identified in both papers.
The Verdict: AI Product Management is Mandatory
Stakes are high and the margin for error is slim in the world of AI. Viewing AI Product Management as merely a nice-to-have indicates a lack of understanding of the complexity of AI initiatives.
Without it, projects drift aimlessly, facing the very real risk of failure. With AI Product Management, however, projects are equipped to navigate challenges, meet expectations, and deliver real business value. And if they don’t, they at least prevent significant financial losses by transparently communicating the results and recommending the next best actions—whether that means building a strong data foundation first, hiring the right experts, or stopping the initiative altogether.
It addresses the critical issues identified in the research, issues that many experts recognized even before but often felt no one was responsible for.
If we look at this trend, we see that the interest over time for both Data and AI Product Managers is increasing, and quite steeply at that.
So, companies are waking up, realizing that AI is not only about technology but requires a broad approach - one that goes beyond AI.
A Pathway for Aspiring AI Product Managers
These are actually quite good news for aspiring AI Product Managers. Companies will hunt for you; they will need you. You better prepare today than tomorrow.
To help you with that, here’s how you can get started:
Develop a Strong Foundation in AI and Data Science - Understand the basics of AI, machine learning, and data science. Online courses, certifications, and degrees in these fields can provide you with the necessary technical knowledge.
Gain Experience in Product Management - Learn the fundamentals of product management. This includes understanding customer needs, market analysis, product lifecycle management, and agile methodologies. Hands-on experience, whether through internships or roles in product teams, is invaluable.
Build Cross-Functional Skills - AI Product Managers need to communicate effectively with both technical and non-technical teams. Develop your skills in areas such as leadership, communication, and project management. These will help you bridge the gap between business objectives and technical implementation.
Stay Updated with AI Trends - The field of AI is rapidly evolving. Keep up with the latest trends, tools, and technologies. Follow industry news, attend conferences, and join AI communities to stay informed.
Network with Professionals - Connect with other AI professionals and product managers. Networking can provide you with insights, mentorship, and opportunities. Join relevant LinkedIn groups, attend meetups, and participate in forums.
Practice Problem-Solving - AI projects often face unique challenges. Sharpen your problem-solving skills by working on real-world AI projects or case studies. This will prepare you to anticipate issues and develop effective strategies to overcome them.
Seek Mentorship - Find mentors who are experienced AI Product Managers. They can offer guidance, share their experiences, and help you navigate your career path.
You can build a solid foundation and position yourself for success as an AI Product Manager, but you must be willing to start your journey. And if you have fears and self-doubts, remind yourself of this quote from Ratatouille, famously spoken by Chef Gusteau: “Anyone can cook.” 🍽️ - here is my LinkedIn post on that.
JBK 🕊
P.S. If you’ve found my posts valuable, consider supporting my work. While I’m not accepting payments right now, you can help by sharing, liking, and commenting here or on my LinkedIn posts. This helps me reach more people on this journey, and your feedback is invaluable for improving the content. Thank you for being part of this community ❤️.