AI for Product Managers

Overview

Drive better results with AI for Product Managers. In this talk, Jordan overviews the key PM workflows and how AI can accelerate or improve results for each. For each section, Jordan provides a demo of how to build a custom agent/chat to achieve the desired result.

Key Points:

• Map AI to the full PM lifecycle—discover, ideate, create, and synthesize—rather than treating it as a one-off productivity hack.
• Build custom agents with curated knowledge (e.g., RAG over reports and docs) to accelerate industry learning, competitive monitoring, and persona simulation.
• Use AI to brainstorm across stakeholder perspectives, draft PRDs, spin up prototypes quickly, and generate user-testing question banks—including edge cases you might miss on your own.
• At scale, lean on AI to cluster customer feedback, surface patterns in large datasets, and sharpen executive narratives and presentations.
• Treat AI as a partner, not a substitute: get to 80% fast, then refine manually—provide strong context, challenge the output, and avoid shipping "AI slop."

Transcript

[00:00] Intro

Thanks for having me back again this week. Thanks for making the time, Neema. This is the follow-up to last week's talk. So this one's titled AI in the product workflow, and I'll kind of show this slide here.

So last week we talked about parts one and two: what is AI, what are LLMs, and then we dove into some of the core building blocks around taking those core concepts and how they come together to actually inform a product. Today we're going to discuss AI in the PM workflow. So after our session, you'll be empowered through our demos to start integrating AI into your workflows — ranging from simple prompts that can still propel you forward pretty quickly, to how to create your own very basic agent as well. So without further ado, let's just dive in here.

[00:43] Key PM Workflows

I'm going to start with this slide called key PM workflows because I think these frameworks are super helpful for contextualizing the rest of the topic here.

So when I think about my role as a PM, there are two frameworks that I think sum it up really nicely. The first is Jackie Bavaro's six Ds of product management. She is the famous author of Cracking the PM Interview and Cracking the PM Career. She's awesome.

And the second is the PM workflow by Britt Jameson, which is one of the people that I listened to during my focus week. And so I've mapped these two actually together — the first line and the second line in this table — to kind of showcase both at a higher level, which would be maybe Britt's framework, and then a little bit more granular detail for Jackie's.

So typically as PMs, I think we are kind of shepherding a product from concept through development, launch, and eventually reaching a maturity stage too. And so if we think about it in that way, then we can kind of break it down into these buckets.

So starting at the concept phase, which is discover and research: we're doing research on industry trends, competitive landscape, and who our target customers are. We might conduct interviews, read reports, look at data trends — but the challenge here is really figuring out what the landscape looks like, what the needs of the market and customers are, and where our product kind of fits in there.

Moving to the ideation stage, this is where we have a good grasp of the landscape. We've discovered a bunch of problems, we've narrowed in on a single problem to go solve, and now we are exploring solutions, picking one, writing requirements, and evaluating the impact to the business and customers. So here I think the challenge tends to be brainstorming a diversity of potential solutions and products. Oftentimes, because we're also very technical, we tend to jump to a solution or we build something that a customer told us to without doing the true evaluation.

Moving on to the creation stage, this is where the actual building of the product is going to take place. So as PMs in this phase, we are going to be involved in designing the solution from a UX perspective, building some prototypes, and doing user testing with a minimum viable product as well. And I think the challenge here is getting a prototype to market quickly so that you can start learning and iterating from that too.

And then lastly, we reach the synthesize stage, and this is where we're actually going to be delivering the product and debriefing the team. So after we've built the successful prototype, we've iterated with pilot customers, we know it's good — now we're starting to scale up. We're starting to collect customer feedback at scale, starting to incorporate it as the product matures, iterating, and along that journey really doing a lot of executive reporting on the impact that we're driving. So here the challenge is really working at scale and presenting those ideas in a compelling way.

[03:29] Discover / Research

Okay, so discover/research is the first phase here, right? In these tables, I'm going to represent PM tasks, challenges, and potential AI solutions that can help solve those, right?

So as we discussed, the main tasks in this area are knowing your customers and knowing your competition. I think with each of these there's a few challenges that tend to come up.

So when it comes to knowing your customers, oftentimes you're actually going to have a lot of different personas of customers that have different needs, right? So for an enterprise B2B product, that could be small businesses, it could be large businesses, it could be businesses with very niche requirements on security or transparency. For us, it's things like: we build for developers, we build for service engineers, customer success — we also build for VPs sometimes.

And some of the things that AI can help with here is persona simulation and customer segmentation. So based on going out and collecting a lot of information, you can use an AI model to help group different customers into different buckets, right? And it can go beyond just their title in the business — you can bucket them by the different requirements that they've given to you.

When it comes to knowing your competition, there's lots of alternatives out there usually, right? There are tons and tons of solutions, and oftentimes there are frequent announcements being made, and it can actually be hard to stay on top of what all the different things your competitors are doing. So even within the company — working on sometimes internal tools — we have customers that go and they create their own tools and they start evangelizing around them. There are lots of features that get rolled out from our partner teams within the company too.

So staying on top of that can be a challenge. A potential solution here is to build an AI agent for yourself that monitors a specific list of competitors in the marketplace, or reads through a bunch of newsletters for you and gives you summaries of what's going on and how the competition is positioning themselves.

And then lastly, I think when you're trying to learn an industry, especially when you're new to something, the industry is a very complex landscape, right? There are a lot of foundational concepts you have to grasp, regulation depending on the industry, and you need to know where things are headed from a trend perspective too. And this is where you can create your own AI agents where you can put in industry reports and executive briefings, use it as a knowledge curation tool, and also a teacher that can create a learning course for you and help you understand deeper concepts. So that's actually what the demo for this one is going to be. We're going to build a really simple RAG model that can help us with industry learning, and we'll use Copilot to do this.

[06:00] Copilot custom agent demo

So I'm going to switch to my Edge browser here where I have Copilot and the demo living, and I think it's this one.

Perfect. So this might actually be the most interesting demo that we give. So what I'm thinking about here is: one of my passion areas is sustainability. And so if I was trying to get up to speed on what all of Microsoft's sustainability efforts were, one of the first things I would do is go read the company reports.

But the company's now been publishing reports for five years. And so how do I get to learn about the history and the trends and the new things that are up and coming and what we've accomplished?

So for that I think what's really cool is we can create an agent to help us do this. So over here in the sidebar, it comes with some built-in default agents. But if we actually go here to create agent, this is what's going to allow us to create an agent that we can define to do whatever we need it to be. So the first way you can do this is you can describe what you want the agent to do, which is interesting — but we're actually going to go over to configure because we have a little bit more that we want to dive into.

So I've already built this demo here since it takes a little bit of time for Copilot Studio to actually build in the background, but I'll walk you through the steps that I've followed and kind of what each area has to do with it. So obviously name and description — very straightforward. We're not going to be leveraging a template, but they have a bunch of different things here that you can build off of and customize.

So the first thing here is instructions, right? And if we open this little thing, this is basically telling our agent how we want it to interact. So this is not fine-tuning — this is defining the system prompt that we want to give to our agent. Basically, the prompt that tells the agent how we want it to behave. So for these instructions, I'm going to go over to this Loop page here where I actually last week spent some time writing this set of instructions for the agent. And we'll go into kind of what each paragraph means here. But I copied this over and I pasted it into this instruction field to inform how we want the AI to behave.

We'll come back to the instructions in just a second. And then they also have an area for knowledge, right? So you can select and upload custom knowledge to this agent that it will have access to. So for instance, if I click upload from device here, I have gathered sustainability data from Microsoft for the last five years, excluding 2023 because it's in a different reporting style for that year.

But these are the PDFs that we publish every year. So what I did was I just selected all of these and I uploaded them to the model. And this is kind of getting into retrieval augmented generation. This is not a super advanced version of that, but you can upload up to 20 files I think for this, which for most purposes is going to be more than enough.

And the reason why we upload them to the knowledge center here, as opposed to just doing it in a single prompt in a chat, is you can only do one file at a time in chat. And with Copilot, at least, it only exists for the length of that chat. So if we want to come back to it and use it for multiple chats, we want it to live in knowledge so it actually becomes part of the knowledge base.

The other interesting thing you can actually do with Copilot Studio is you can give it access to your other internal things. So for instance, I can actually give it access to a Teams chat. So we have a sync with one of our partners every Monday — I could actually click on this and then it can read the history every single week that this meeting happens. And so as things are happening, we can then go back to this agent and get new context and all this stuff. It can also read the meeting recaps and notes from this too, just so you're aware. And then at the end, you can also give it the capability to do code interpretation, image generator — we don't need that for the sake of this model.

So basically, I would come over here, I would copy all this, I would paste it, I'd upload the files, boom, it's ready to go. Now I have something I can work with. But I want to dive into the actual instructions here first. So again, this is where we're giving context to the AI to tell it how to behave.

So the first thing that I'm telling the AI assistant is: you are an agent, right? So it's already going to have a persona that it's taking on here. I'm also giving it a very specific context for it to work with — that it helps people find and analyze sustainability from the set of sustainability reports. And I explicitly tell it to reference the reports that I've added to its knowledge source. Okay.

So then I thought, okay, what else do I kind of need to give to the model to make sure that it behaves in the way that I'm used to? So I did some digging and I found two things.

The first is this thing from OpenAI called the OpenAI Cookbook. And what this is: it's an open source collection of examples and guides for building with AI. So there's a lot that's specific to working with OpenAI APIs, but there's a lot of stuff that's really good and general as well.

So in this particular one, I found this article from April called the prompting guide. And this went really in depth around working on system prompts for your AI models. And this section here really stood out to me because it was new information. But these three prompts here that you add — which if we go back to here are these three here — were really interesting.

So the first one instructs the agent to always go until the user's query is completely resolved before it ends its own turn and gives control back to the user. And the reason why this one's important is because if you've asked a multi-part question or multiple questions in your prompt, or a question that requires a multi-part answer, if you don't have something in the system prompt, there is a chance that the model can terminate its turn when it thinks it's done — which could be just answering the first question or the first part of that question. So this one ensures that everything that you give it will be completed.

The second one is saying: if you're not sure about some of the information given, use your tools to read the files and gather the information, and do not guess or make up an answer. So this is saying, hey, make sure you use the files that I'm giving you, and don't pretend to know if you don't actually know.

And then the last one is asking the AI model to actually plan before it's going to do every function call — so before it's going to search the web, before it's going to read one, two, three PDFs, right? And it's asking it to reflect on the outcomes of the previous calls as well.

So basically this is trying to get it to actually create a plan and be fully complete and have some value there. So then lastly I added a couple things to it that I thought were really great based on interacting with previous chat agents. So I said: before you craft your response, consider whether or not the question requires a simple or a complex answer. When a response requires a complex answer, please structure it in this exact following way.

First, answer the question simply with no more than two sentences. Then break down the answer into understandable parts. First identify how many parts to break it into. And then for each part, provide a more detailed answer.

And then finally, consider — so it's giving optionality to the model to add a metaphor or narrative way of answering the question at the end to help promote deeper learning. And then this last one I actually borrowed from Microsoft's Copilot Studio overview guides — specifically this page on prompt modification to provide custom instructions for agents.

And if you go down here, it's talking about how do you get stylistic tone to be included? And so I basically copied parts of this prompt and added it in as well to the last one here.

So going back to here, what does this yield us? So if we load up this chat, this will take a second. Oh, actually if I go back one page here, the back button doesn't work. You can also give it some suggested prompts that will come up when the user starts interacting with it for the first time.

Which I don't know if they'll pop up for us here. But anyways, I clicked on one of the first prompts and I said, "How has Microsoft's carbon emissions changed over the past five years? What role has carbon removal played in their strategy?" This is a two-part question. This is also a question asking it to analyze data across all of the reports.

So I felt that this was a good prompt to start off with because it's specifically targeting some of the things that I put into the prompt. So we can see that it wrote a two-sentence simple description for answering this question, but it recognizes that this is a complex question and requires a complex answer. So we get the two-sentence one first, and then it has identified that it needs to break this down into carbon emission trends followed by carbon removal. For each of those, it then has further scoped this down to much more detailed information, and then at the end it did actually have the narrative too.

Right? So if we actually look at some of this data, we can also see where it is referenced from the first part of this answer. I think I'm back. Neema, do you know where I left off before it cut out real quick? It's only cutting off for a couple of seconds, but it's happened a few times — but we're following. We haven't missed anything.

Okay, great. Yeah. So you can see here that it's also providing the citation for where it's getting the information from.

So clearly at some point in this report from 2025, it's noting that the baseline was in 2020 and the emissions have come down 30%. Anything else on these? Looks like for this response it mostly pulled from 2025. And then it came up with a narrative.

Awesome. So then I asked it to generate highlight moments from the sustainability reports from each of the past five years — look for something that stands out from the standard content that a human would find novel or interesting. Again, it recognizes this is a complex answer because it needs to look through all of the years, and it does that right.

So 2020: the year of bold commitments. 2021: circularity — going up to last year revisiting circular economy but now introducing this idea of sustainable AI. So anyways, that's the end of the first demo, but I wanted to show how you can really customize the models to work exactly how you want them to and give them the context that you need by providing additional documentation to it. I'll also mention that through Azure AI Foundry, you can create much more complex models. You can customize which actual open-source or proprietary model you want to use behind the scenes. And you can also do much more complex retrieval-augmented generation architectures that can go well beyond just 20 documents.

[16:30] Define / Ideate

Okay, moving on. Now we're at the ideation phase. So the tasks here are around brainstorming ideas, picking the right business model, and drafting a product requirements document.

So I think one of the hardest problems around brainstorming is actually being able to put ourselves into the mindset of the different stakeholders in a project, right? So whether you're the actual designer, the engineer, the customer, the customer support agent, or even the VP — it's hard to pretend to be somebody else. And at the same time, we can't always rely on having a teammate available to join a meeting and brainstorm with us. So AI can take on different personas if you customize how it's going to behave to help you with brainstorming.

When it comes to the business model, there's also a lot of options around how we charge a customer or how we handle our costs. So even with internal tools, we have to handle the budgeting and the planning, the cross-charging model — there are plenty of options available to us. And then furthermore, building a cost and revenue model isn't always a core skill of a product manager.

So AI can also help with that. And then lastly, when you're drafting a PRD, we spend a lot of time writing. Some of our stuff is 10 pages plus long because of how technical our space is, and that needs to be captured appropriately. So AI can leverage templates and help extend bullets into full-fledged ideas, or translate from really good note-taking over into more of a narrative format. I'm not going to do a demo for this one, and instead we'll move into the creation phase.

[17:55] Design / Develop / Create

I think this right now is probably the coolest and potentially the space where there's actually the most hype going on right now. But I think there are three things that PMs do here, right? We create UX mockups.

We create actual working prototypes, and then we also do user testing with the mockups and the prototypes alike. So in the UX space, not all of us have great design skills, right? It can be hard to know what the right user experience should look like, but tools like Figma Make can help us translate instructions into pretty mid-fidelity concepts that you could put in front of a customer.

When it comes to creating prototypes, AI tools like Copilot for Visual Studio, Cursor, Bolt — they can get you a working prototype in record time. So coding prototypes that used to take days to get basic functionality spun up because it's hundreds and hundreds of lines of code — that can all now be generated, and you can get that initial window down to mere hours.

And then lastly, for user testing, it can be hard to think of all the things that need to be verified, right? So AI can help you in this case build off of the base questions that you came up with, and it can help you find edge cases that

[19:11] User testing framework demo

So I'm actually going to demo that one as well. And now we're going to go down to demo two: user testing. And I'll scroll back up here.

So shout out to Diego and Tom for letting me use their space right now because they are driving a lot of new features when it comes to recovery for BuildXL. And so what I've done here with this prompt is I chose the Idea Coach agent and I've given it a custom prompt. So I've told the model that I've attached a PowerPoint deck that outlines the specification for a feature called recovery. And based on that, I've asked it to generate a user testing question bank that is specifically tuned to the feature's goals.

And then it is listing out the PowerPoint here. So it ingested that, and based off of that it came up with a bunch of different categories for which we should be asking questions of our users if the product is meeting their expectations and requirements. So it thought about discovery and entry points, like how the workflows get kicked off. It understood the differences between restarting and new builds, which is a new paradigm that's getting introduced.

It's also thinking about build selection and navigation specifically in parts of the product. It even came up and thought about accessibility too, which is a really big push for us at Microsoft, which is awesome. And then it also started to think about some edge cases or advanced use — right, how would you expect the system to handle restart if the FQBN is already taken? That's a pretty detailed question that our customers would potentially have a thought about. So yeah, quick demo here on that — but it just shows you that it really can think for you, and it can think of edge cases as well.

[20:49] Deliver / Debrief / Synthesize

Last one is called synthesize, right? So I think one of the main things we do, especially working at a company where there are a lot of mature products, is incorporating ongoing feedback to iteratively make the product better. So often what this means for us is collecting information at scale — whether it's product usage data or qualitative verbatims — and when you have thousands of verbatims coming back to you, it can be very hard to sort through those, categorize them, and find patterns.

I think AI is really awesome with that, and well, that's what the demo I'll show will be for this section. AI can also help find patterns that would be too hard for humans to recognize themselves just because of the volume.

So first off, it can help with merging complex data sets by helping you to write SQL queries, or potentially just join the data for you if you upload it into the model — something that typically might require a specialist to help with. And then lastly, when you think about executive presence and presentations, AI can act as a writing coach for you to review and make suggestions on your outlines, help shape your narrative by choosing a story structure, and make sure that the key points in the content that you've written are salient throughout the presentation as well. So last demo I'll show is using Copilot for how we can analyze quantitative data.

Back over to Edge one last time. So, summary page here. This is the money slide that you can screenshot and reference on how you can start integrating AI into your workflow. I happen to think that some of these ideas are pretty frequent use cases for myself.

[22:23] Summary

But really what I want to end on is advice. So, two pieces. I'm actually going to say the second one first in this case. I think AI tools are really great for getting to good enough quickly and then refining manually. I don't think right now that they are great for doing the entire thing for you all up front.

You know, 80/20 rule, right? Use it to get the 80% in 30 minutes, then spend the 20% for the next few hours or days refining. Lastly, you really want to avoid AI slop. And this is something that's really pervasive on the internet right now. You don't want to be using AI tools to just get the job done when you're not even doing the job well.

And the three things that I think can really help you here: one, by providing better context. So the model's only going to be as smart as you tell it. It's only going to be able to execute instructions as well as you provide them.

The second is challenge the answers that you get from it. Did it produce something good? Because you can also prompt back to it: can you think about the answers you gave to me? So the more you force it to actually think — and that can be done through the system prompt or by going back and forth with it — generally that's how you refine what you're going for to a better state. And hopefully the example I showed with the custom instructions demonstrated that that is possible by providing more context and challenging it.

And lastly, don't use it to think for yourself. I think also, kind of in the vein of the second one here, give it something to start with — which in the industry is called few-shot examples. So give it a couple base examples, or if you're using it for user testing, give it five questions that you've come up with first, then go to the AI model to get to 80%, then refine down that last 20%.

So don't use it as a substitute — use it as a peer and as a partner. I hope it inspires you to come back and start using AI in your workflows appropriately. Happy to take any questions if folks have some. And I know we've got about five minutes left.