Embracing the power of AI/ML in product management - your top questions answered by product from Intuit, Gartner, and more

Q: How did you get into AI/ML product management? What’s your origin story?

Liliya McLean: While doing my CS PhD in 2014, I got curious about ML because I was into statistics, and somehow ventured into doing my dissertation on a topic related to massive scale data featurization. That put me on a path to AI-related work soon after that, and eventually led to me stumbling into Recommendation Systems (even though that wasn’t truly a thing in the industry yet). I fell in love with RecSys and Personalization work because it’s one of the most transferable, contextually aware of the entire business and interconnected technical problems that has a direct view of both the consumer and the underlying infra and data a business operates on. 

Through continuous exposure to more and more applications in this space, I started owning entire portfolios of applied AI problem domains and the nature of problems I began solving evolved as well. AI is a broad range of techniques and correspondingly applications that keeps evolving at a rapid rate, so that keeps me energized and excited to continue learning and scaling myself up. 

Fiona Zhou: Following the breakthrough success of AlexNet, I co-founded AI+ Club at Stanford with a few CS PHDs, hosting firechats featuring academic figures and startups founders in the space. To build technical acumen and speak the same language with AI researchers, I took several classes in machine learning and participated in Kaggle competitions. All this experience helped me land my first PM job at a Stanford startup in the medical AI field as a product manager.  

Bo Mei: About 7 years ago, I began my journey with AI/ML-driven product initiatives, primarily centered around player engagement and monetization. We worked with technologies that were considered cutting-edge at the time, such as big data analytics, recommendation models, and reinforcement learning. I quickly recognized the immense value AI could bring to the gaming industry. This realization finally led me to join a startup focused on creating a metaverse where meta beings and humans coexist. From the outset, my current game studio has been dedicated to crafting an AI-native gaming experience.

Shengyu Chen: Around 2015, I began hearing an increasing amount of intense interest on AI/ML focused product features from our enterprise clients. Part of those interests were driven by the hype on the future promise of AI. Outside of the hype, the intuition has always been clear to me. The more data you have, the more information you have. The more information you have, the more leverage you have in making decisions. I thought this was almost universally true across all domains. The amount of data created and captured only accelerated year over year. Someone will figure out how to use these data effectively and all of our lives across all domains would be changed. For us, we just needed to figure out which problem is better solved with the newer techniques. On a personal level, I also simply wanted to be more involved in this domain.

My product leader also agreed to this line of reasoning. On top of that, because of what we observed in the market, we began very aggressively experimenting with different pilots that leverage ML models for a variety of workflow features. Some of the problems were much better suited for ML and saw much better successes while others became repackaging & repositioning exercises. The aha moment for me was observing the joy where users are enabled with something that they weren’t able to before (e.g. taking hours to manually sort and find the right targets with complex formulas that someone else wrote in Excel. Silly example but left a lasting impression). Granted it didn’t always work but for a brief moment, it felt like magic. And for that brief moment, I could kind of see the future. The tech just had to get much much better. It was definitely overhyped then but the underlying argument isn’t wrong. So it is definitely expected that we are experiencing an even stronger and refreshed interest in AI/ML at the moment. 

Q: What are some biggest challenges you are facing right now and how do you resolve them?

Liliya McLean: There are some evergreen challenges in the space of AI, and others that come as your role grows. In terms of pretty consistent challenges, a common one I’ve seen in every role I’ve ever held has been the polarizing nature of applied AI. For many, what we do is black magic or a black box that they don’t understand the inner workings of but certainly have opinions about. Threading the fine line between hearing and appreciating diverse feedback, and knowing how to stay focused on the true north of what you are trying to achieve, even if it doesn’t serve equally everyone’s objectives, is an art form. 

In terms of challenges coming with seniority and scale, these days GenAI is top of mind for everyone, so as a senior leader I need to effectively guide my teams to making smart investments into solutions that both solve some of our immediate needs but have the potential to create future runway for our business. In a market environment of economic distress, but once in a generation technological evolution, picking between safe bets and big bets is a tricky job that makes or breaks businesses in the long run. 

Bo Mei: Interestingly, in game settings, what's termed as a 'challenge' in other contexts might be an advantage if we use genAI correctly. For instance, "hallucinations" can be reframed as "creativity". The pressing challenge for us is that while there's excitement around AI's potential, there isn't a universally accepted definition of an optimal AI-native gaming experience. Our guiding principle is to constantly ask, “Is this enjoyable for players?”. We then incorporate AI into game design and promptly validate our concepts with real players.

Shengyu Chen: In one of our teams, we have been building a prototype-chatGPT like experience (like almost everyone nowadays) as a way to discover and consume our contents so that users can get to value much much faster. There are many challenges. The biggest one right now is to figure out how we can take this experience to market quickly. The user need is very real and intense but from the business standpoint, key stakeholders’ expectations are hard to align (and many many stakeholders to align). There are also some very real risks that can be introduced from inaccurate & unuseful responses. Frankly, I don’t have a good solution for this challenge. I think it can be done but it just takes time. 

Q: How do you keep up-to-date with AI/ML technologies? There’s always something new popping up daily.

Liliya McLean: I tend to connect with various practitioners from top AI companies on a regular basis and keep myself close to implementation details. I’m also continuously reading new research, following various communities and newsletters and frankly trying to think creatively on how something can be utilized in a unique way in the context of the specific business I work for at the time. For instance, I’m currently part of the FinTech giant Intuit and the applications there have some of the strictest data utilization requirements of any industry due to the nature of the available data. Inventing solutions within these limitations is especially challenging but frankly quite intellectually stimulating and thrilling as well. 

Fiona Zhou: I spent time absorbing a lot of information to stay ahead of rapidly evolving AI/ML technologies, which can be categorized into the following three:

1. Leverage Social Media and Online Courses: I follow lists of people in AI on Twitter, LinkedIn and Youtube. I also love AI courses taught by Andrew Ng, which should be beneficial for all aspiring AI/ML product managers.

2. Experiment with Consumer-Grade AI Tools: I like trying all kinds of AI products out there and do design critique.

3. Go to AI Events: I go to AI talks and hackathons where entrepreneurs share the latest project using AI and venture capitalists share their insights.

Bo Mei: First off, I believe that not every new tech out there is going to be a good fit for our game. But that doesn’t mean we bury our heads in the sand. I organize in-house hackathons every so often, allowing the team to experiment with the latest AI/ML technologies and see how they might fit into our game setting. It's a fun, hands-on way to keep everyone in the loop without losing sight of our primary game vision.

Shengyu Chen: I am a podcast power user. Whenever I am doing housework, driving somewhere, I am always listening to podcasts. Most of the podcasts are focused on tech, product, startups, VCs and specific ones on MLs. This is primarily how I source the latest conversations, discussions around AI/ML. And on Youtube, there are a few channels that take the latest AI/ML papers and offer some quick insights on these papers. LinkedIn feed has also been very good in the last few months in surfacing new feature announcements, new startups and new research published in AI/ML. All of these sources do a sufficient job in helping me catch the latest conversations. If I decide that some of these topics require deeper dive, I’d typically run tear down exercises to go deeper and potentially create a presentation to share my leadership and team or occasionally write a blog. Not all are directly applicable but they are genuinely very interesting. 

Q: What do you think are the biggest differences between AI/ML Product Managers and leaders v.s. More traditional PM domains?

Liliya McLean: First, I’d like to begin with the distinction that even in the AI/ML product world there are different categories of product management - those who focus on platform and infrastructure work, those who focus on certain foundational capabilities (e.g., content understand, fraud, etc.) and those who use applied AI for efficiency growth (e.g., ranking, recommendations or other smart discovery services). With that in mind, what’s true at varying degrees across all categories of AI product management is that we have to be highly holistic practitioners who are strategically, technically and analytically skilled to an extent not consistently present, or even necessary, in other types of product roles. Additionally, our roles are often highly horizontal giving us a really unrivaled understanding of the entire product ecosystem. 

As an AI product practitioner and leader, you have to know how to communicate with the combined perspective of empathy for the customer/benefit to the business and feasibility/velocity of the technical solution. I often joke that I am whatever I need to be at the moment - a product strategist, a tech lead, a data scientist or even a systems architect depending on the audience and horizon of the conversation. In order to do all of these things effectively, one needs to invest a significant amount of effort into keeping themselves aware of both the market and science landscapes.

Fiona Zhou: I think the biggest difference is that AI product managers have the most powerful technology tools - AI - in their pocket for unsolved challenges. On top of all the skills required for good product management in traditional domains, there maybe two differences worth calling out: 

1. Technical expertise, in particular deep understanding of: a. what makes a great problem space for AI, b. what it takes to build the AI product, c. top trends in AI that are ready for production. 

2. First principle mindset. AI/ML product managers often have the opportunity to reshape a complex domain dramatically where conventional assumptions and past trends may not hold. Therefore, AI/ML product managers are often practitioners of first principles who break down the problems into fundamental elements and focus on underlying truth. This allows them to design disruptive solutions using AI. 

Bo Mei: You’ve got to be more open-minded and super adaptable, always ready for a curveball. Sometimes, it’s not just about tweaking a few things; you might find yourself rethinking the entire pipeline. You have to embrace the unknown, and honestly, that's where the magic happens.

Shengyu Chen: At an IC level, I think the core competencies are similar i.e understanding users and their problems, understanding how solving problems can create value that aligns to the business strategy & vision. To me the AI/ML part requires the PM to have additional skills around understanding data, AI/ML techniques, production systems etc. These additional skills do take more time to build and depending on the focus area, the levels required may differ quite a bit. Some of these skills do require actual hands-on experience so quite a few AI/ML PMs I have seen do come from the technical side or have a deep technical foundation. 

Now as for why these skills are important:

1. These skills would help understand and articulate specific problems where not having these skills would be difficult to discover

2. These skills would enable discovery of novel/feasible solution approaches with the team

3. These are required in order for the PM to work effectively with the team of data scientists, machine learning engineers, data engineers. 

4. Taking a step back, these skills are also important for understanding how the future can change and thereby impacting the current state. No one can reliably predict the future but it helps to go through the process in a rigorous way. 

Not sure if this would be a hot take or not, part of me always feels that the AI/ML designation may be transient. I remember in the 2016-2017 period, everyone was talking about AI/ML features but then that just kind of died down after sometime among folks outside of the industry. Now AI/ML features are ever more ubiquitous and it will get increasingly so. When technology gets sufficiently ubiquitous and well understood, the technology itself ceases to be the focus and everyone only ends up caring about the outcome as it should have been always. 

Now it doesn’t mean the space of AI/ML PM or products will be marginalized after the hype dies down. Quite the contrary, these products/features and people building them will be even more ubiquitous and commonplace, so much so that the AI/ML designation kind of loses its meaning. On one hand, PMs in this space will probably have even more structured specializations. On the other hand, some of these required skills may be further abstracted away so that a normal PM can function quite adequately. Not sure how long it would take to get there though. 

Q: What advice do you have for aspiring AI/ML Product folks?

Liliya McLean: Before getting into any product roles be sure that’s what you really want to do by reflecting on your motivation for it. I’ve been approached by numerous aspiring PMs, who upon further discussions of what drives them, discover they don’t actually want to be product managers per se but have some adjacent ideas and aspirations. To become a great product manager you have to be daring, dedicated and open-minded, and that’s not something one should take lightly. Alternatively, you can lack genuine passion for product and become a mediocre product manager, which is hands down one of the most bleak and uninspiring jobs you could do in tech. So, either do it for the right reasons or don’t do it at all. 

In terms of specifically AI product roles, I would strongly recommend speaking with at least several experienced AI product managers and understanding the nuances of different AI PM roles. They are quite different in nature and will not appeal equally to everyone. In fact, finding the odd birds who love working on AI infra and on applied AI with equal zeal would be quite the task. At the same time, if you don’t know what you like exactly, intentionally exposing yourself to the full spectrum of AI related product roles could prove to be supremely beneficial in the long run. It would give you a really thorough appreciation and understanding of how certain parts of the ecosystem come together, which would make you a more competent communicator and leader. 

Finally, in terms of how to break into AI product management, I could give you an array of strategies depending on where your starting point sits.

For experienced product manager without AI expertise, I’d suggest setting your sights on applied AI roles (as they sit the closest to the consumer experience and would take advantage of your prior product expertise) and leaning on your technical partners to bring you up to speed on the unique nature of AI applications (e.g., longer development process, non-deterministic nature of solutions, continuous iteration, etc.) Lean into your strength by clearly translating how the technological capabilities developed by your engineering teams unlock a variety of customer experiences. 

For experienced technical and analytical professionals, understanding exactly where your past expertise benefits AI services, and choosing a role close to that space, so through that lens you can bring valuable domain expertise into your new product role, is the way to go. For instance, for former data engineers, you’d have your pick of the litter because you could go both in the direction of AI infra, working on something like feature management services, or go down the applied AI route and use your knowledge of data to inform prediction quality improvements or to unlock new customer experiences.  

Finally, for early career individuals, picking AI product management as your first gig is a daunting task, so be mentally prepared. The learning curve is steep and unless you are willing to dive deeply into a variety of hairy problems, you’d suffer a lot. One way to soften that blow is to avoid owning too much. Pick something focused and stick to it until you learn the basics really well. Scope comes easy in the AI world, so there’s no need to be greedy when you are not ready for it. 

Fiona Zhou: For aspiring AI/ML product managers, first of all, do not get cold feet given the enormous advancement in the past few years. It’s never too late to get in the field. I suggest the following to kick start the journey: 1. Develop your own way to learn and stay up-to-date with AI technology. 

2. Focus on a problem space you are passionate about and explore how AI/ML can transform it. This targeted approach helps in applying AI solutions meaningfully and effectively.

Bo Mei: AI will change the world, and the gaming industry is no exception. It’s not just about making things more efficient; it’s about redefining how we design and play games. If you’re considering diving into this world, now’s the time.

For beginners:

Dive into gaming communities. Understand what players love and where they face challenges. While AI/ML knowledge is crucial, understanding player motivations is equally vital. Begin with grasping AI fundamentals, but always link them back to gaming applications.

For professionals considering a shift:

Leverage your expertise and identify how AI can enhance that domain. Familiarize yourself with AI-driven gaming innovations and envision how your current skills can merge with them. Push the gaming frontier. Explore niche areas like AI-driven game narratives or adaptive gameplay mechanics. While the tech is essential, player experience is the endgame. Always innovate with the player in mind.

The playing field is wide open, and there’s no playbook yet. Dive in, get your hands dirty, and learn on the fly. Stay flexible and keep pushing boundaries. Be brave, be water, my friend.

Shengyu Chen: In a much more idealistic sense, deep reflection is probably the most important yet underrated exercise. I think the foundational advice is to be genuine in thinking about your interest in this domain. This may sound bland. But the question of why you are interested in this is very important for the long term. Only genuine interest can help you uncover those unique insights and discover unique opportunities. I really like Paul Graham’s take on doing great work in general (mostly centered around doing something that’s both of your genuine interest and what you are good at). That applies to AI/ML PM as well. PG offers a very good set of questions and exercises that help figure out whether you are genuinely interested in something. I won’t iterate it here. 

Now on a much more tactical level for fresh grads, I’d recommend the following (as you can see this does take time and if the goal is something that’s much more pragmatic that focuses on job offer conversion, the following probably isn’t the most efficient route):

1. Consume contents: Talk to people in the domain, talk to a lot of them (other PMs, MLEs/DEs, Data scientists)Follow the latest topics, trends on X, youtube, podcasts, substack etc.Take few courses in this area

2. Apply: Use and analyze other AI/ML products that you like (how do they work, why are they made this way etc.)Try building something that solves an urgent problem that you may have that leverage AI/ML techniques potentially

3. Share: Record & Publish your progress somewhere. This will help you find more people who share similar interests and help you consume more industry related content. 

If the goal is about landing AI/ML PM jobs, the tactics would be quite different and highly contextualized based on your current experience and skill level. I think there’s a ton of literature & approaches already written on this. Thinking back, the success case of people landing in the industry most of the time is about being at the right places at the right time or knowing the right people. So the tactic here is to again talk to people and opportunities will emerge as long as you are genuine.