Some teams might also include a customer-facing person who helps with certain aspects, such as finding the right users or handling communication with those users. These team members might come from customer success or support.<\/p>\n\n\n\n
In addition, product marketing managers might be involved to help with user sentiment and messaging.<\/p>\n\n\n\n
But the PM is a responsible and indispensable team member when it comes to beta tests. Now let\u2019s take a look at the 10 steps for running a successful beta program.<\/p>\n\n\n\n
10 steps for running a beta program<\/h2>\n\n\n\n\n- Define what you\u2019re testing<\/li>\n\n\n\n
- Define who you\u2019re testing with<\/li>\n\n\n\n
- Define how long you\u2019re testing<\/li>\n\n\n\n
- Define the target outcomes<\/li>\n\n\n\n
- Choose between public vs. private beta<\/li>\n\n\n\n
- Recruit testers<\/li>\n\n\n\n
- Conduct the beta test<\/li>\n\n\n\n
- Track and analyze the beta results<\/li>\n\n\n\n
- Report the results<\/li>\n\n\n\n
- Follow through and iterate<\/li>\n<\/ol>\n\n\n\n
1. Define what you\u2019re testing<\/h2>\n\n\n\n
The first step to running a beta program is to define what exactly you are testing. Is it one feature or multiple? Are there several different functionalities that should be observed? Know what it is that\u2019s new and why you need to test it.<\/p>\n\n\n\n
When and what should you beta test?<\/h3>\n\n\n\n
Beta testing can \u2013 and probably should! \u2013 happen at any stage of the product, from initial launch to one-off feature releases. I think features should go through beta as much as possible.<\/p>\n\n\n\n
As for what <\/em>you should beta test: anything that needs to have risk reduced.<\/p>\n\n\n\nSome changes can go straight out, particularly if they\u2019re bug fixes or things that you’ve de-risked enough in the planning and design stages. De-risking in the planning stage means prototyping and checking with real users that this change is small enough, inconsequential enough, and obvious enough. If it\u2019s exceedingly clear that it\u2019s the right thing to do, then you don’t necessarily need to put it through beta.<\/p>\n\n\n\n
However, if the change is likely to disrupt user workflow and cause confusion, or you’re unsure what the results will be, then beta test it.<\/p>\n\n\n\n
2. Define the target outcomes of your beta program<\/h2>\n\n\n\n
Defining the target outcomes is crucial to the success of your beta program and, indeed, any larger product launch checklist<\/a>. This is actually why we built target outcomes directly into ProdPad! We capture targets and outcomes, at both the initiative level<\/a> (which is a higher level) as well as at the individual idea level.<\/p>\n\n\n\nRegardless of what you’re releasing, whether it’s a micro-experiment or a bigger chunk of your project, you should clearly outline your target outcomes ahead of time. Know the metrics that you need to see.<\/p>\n\n\n\n
These should be SMART goals, meaning they should be:<\/p>\n\n\n\n
\n- Specific <\/li>\n\n\n\n
- Measurable<\/li>\n\n\n\n
- Attainable<\/li>\n\n\n\n
- Realistic<\/li>\n\n\n\n
- Time-bound<\/li>\n<\/ul>\n\n\n\n
Here\u2019s an example of SMART goals embedded in target outcomes for a beta program: <\/p>\n\n\n\n
\n- \u201cWe expect this to go live on [DATE], and by this date we want [N] number of users using it.\u201d <\/li>\n\n\n\n
- \u201cOf those users, we want [N] to have successfully completed [X] job with it.\u201d<\/li>\n\n\n\n
- \u201cWe want to have heard from at least 5% of users that they found it useful.\u201d (Or whatever metric you need to make it successful.)<\/li>\n<\/ul>\n\n\n\n
3. Choose between public beta vs. private beta vs. feature flags<\/h2>\n\n\n\n
A public beta<\/strong> is, as the name suggests, public-facing. You not only label the beta feature in the app but also on your marketing site. It shows the world, and potential customers, what you\u2019re working on \u2013 which adds transparency and value. Once the feature is solid, you remove the \u201cbeta\u201d label from marketing materials.<\/p>\n\n\n\nA private beta<\/strong> might be revealed only to your user base, via a label in the app or some other outreach like an email announcement. Private betas can also be small subsets of your user base that you\u2019ve selected specifically and communicated with privately.<\/p>\n\n\n\nUsing feature flags<\/strong><\/a> is a common practice when you just want to observe user behavior without orchestrating a beta program. These users won\u2019t necessarily know they\u2019re using a beta feature, while you collect data and insights on their behavior. You can continue iterating on the beta feature until it\u2019s ready to launch \u2013 at which point you simply remove the flag.<\/p>\n\n\n\nHow do you know which to choose? Generally, you can publicize stuff that\u2019s pretty stable and will be iterated upon. If the change is still iffy or experimental, you can keep it private. That said, here at ProdPad, we try to weed out any unstable experiments before we even get to the point of building in production.<\/p>\n\n\n\n
4. Define who is in the beta program<\/h2>\n\n\n\n
The next step to running a beta program is to decide which users will participate in the test. Here are a few guidelines for defining who is in the program.<\/p>\n\n\n\n
Who do you beta test with?<\/h2>\n\n\n\n
You should beta test with users who are okay with instability in the product. If you’re working in enterprise software and your product is mission-critical, then you probably have a subset of users who only want the stuff that is rock solid. The beta period is just too risky for them.<\/p>\n\n\n\n
On the other hand, some users will enthusiastically try new, less stable versions of your product. They love to test stuff first! These users are interested in seeing your product improve as quickly as possible.<\/p>\n\n\n\n
You can also beta test with users who asked for the feature in the first place. Follow up with people who\u2019ve requested certain things, such as an integration with a certain tool. This is an obvious set of users who would not only use the feature but are actively interested in giving you feedback on it. <\/p>\n\n\n\n
Whether or not you actually tell these users they\u2019re in a beta program \u2013 that\u2019s a different question. We\u2019ll address the topic of recruitment vs. feature flags, and conducting a public vs. private beta test.<\/p>\n\n\n\n
How many testers should you include in the beta program?<\/h2>\n\n\n\n
The number of beta testers depends on how many people will use the feature once it\u2019s launched. How wide is the rollout? That answer will help you determine how large the beta pool should be.<\/p>\n\n\n\n
In my opinion, high-quality feedback from a few people can be enough. Identify 10-20 people, interviewing as many as you can. (See our best practices for customer feedback sessions<\/a>.)<\/p>\n\n\n\nIf you\u2019re beta testing a feature with hundreds or thousands of users, that\u2019s fine, too! You\u2019ll just likely use software to track their in-app actions rather than ask for anecdotal feedback. More on that later!<\/p>\n\n\n\n
5. Define how long your beta test will last<\/h2>\n\n\n\n
Another important part of designing a beta test is defining how long the test will last. The length of beta programs varies depending on your product and your users.<\/p>\n\n\n\n
Google kept Gmail in beta<\/a> for five years! Other companies may test for a single cycle, or a single sprint, just to see how it works for the first couple of weeks until a decent number of users are on it.<\/p>\n\n\n\nHow to decide the duration of a beta test<\/h2>\n\n\n\n
The duration of a beta test depends on two main factors: what kind of traffic you’re getting on the feature, and what level of assurance you need that the feature is sufficiently de-risked and ready for the wider world. How sure do you want to be that this is, in fact, the right thing?<\/p>\n\n\n\n
Obviously, if traffic is low, then the testing period will be longer so that your team collects enough user data.<\/p>\n\n\n\n
If the feature is more complex than a one-and-done functionality, then the testing period will be longer, too. Sometimes you need to observe how a cohort of users interacts with the feature over time. Then you might compare cohorts together and determine when user behavior or performance has reached your target. <\/p>\n\n\n\n
6. Recruit your testers: How to recruit to a beta program<\/h2>\n\n\n\n
If you\u2019ve decided to select specific users for a private or public beta program, the key to recruiting them is \u2013 just to ask. A lot of people like to participate in stuff such as beta tests or feedback sessions.<\/p>\n\n\n\n
If need be, coordinate with the customer success or customer support teams on how best to communicate with these users and loop them in.<\/p>\n\n\n\n![\"Asking](\"https:\/\/www.prodpad.com\/wp-content\/uploads\/2022\/03\/podcasts-blog-1024x445.png\")
<\/figure>\n\n\n\n7. Conduct the beta test<\/h2>\n\n\n\n
Let \u2018er rip! With everything defined and clearly outlined, you\u2019re ready to give users access to the beta feature. Keep in mind that the test will still be underway as you begin to track and analyze the results.<\/p>\n\n\n\n
8. Track and analyze beta test results<\/h2>\n\n\n\n
You should track beta test results the same way you track anything else in your product. Most modern products have in-app tracking of user behavior. Make sure you’ve embedded appropriate tracking in this new feature so that you can get these insights.<\/p>\n\n\n\n
Some teams want to have just the session logs, where they can read about the number of clicks on different elements of the app. Other teams like to have session video recordings, like Hotjar<\/a> or FullStory<\/a>, where they can literally watch how people interact with it.<\/p>\n\n\n\nBoth types of tracking software are valuable, depending on how much time you’re willing to give to it and how much you\u2019ll use the data.<\/p>\n\n\n\n
How to analyze beta test insights<\/h2>\n\n\n\n
First, you should check that people are finding the new feature! After a day or two, just check that people can find it. If no one can find it, you’re not going to get any data on it.<\/p>\n\n\n\n
From there, you can look at other issues. Maybe everyone finds it, but no one’s able to complete their job in it. Maybe everyone is able to complete their job, but no one actually liked <\/em>using it. On this last point, you\u2019ll need to run some customer interviews to gather some qualitative feedback.<\/p>\n\n\n\nOf course, compare your target outcomes to your actual outcomes as you start measuring. This is an ongoing process until you hit your targets. <\/p>\n\n\n\n
9. Report on the beta outcome<\/h2>\n\n\n\n
The product manager should report back to their immediate team about the beta test results. This could be in the form of a retrospective or just a regular weekly report that says, \u201cHere’s an update on recent releases, what\u2019s working and not working.\u201d<\/p>\n\n\n\n
The team should have a chance to comment on the results and discuss the next steps. Then this team feedback can be reviewed, and any necessary changes can be prioritized into the next sprint.<\/p>\n\n\n\n
10. How to make decisions based on your beta<\/h2>\n\n\n\n
Remember to prioritize iterations on your beta features and continue pushing those through development! This can be tricky.<\/p>\n\n\n\n
In product management, every decision must be considered in light of all the other decisions that need to be made. When you’ve launched something, you need to understand the impact it’s had on the app. And when you’re planning the next sprint, you need to look holistically at the product roadmap<\/a> and business objectives. <\/p>\n\n\n\nMany teams like to move on to build new stuff because it\u2019s more exciting. But I recommend prioritizing things that are already live but not working quite right.<\/p>\n\n\n\n
1. Define what you\u2019re testing<\/h2>\n\n\n\n
The first step to running a beta program is to define what exactly you are testing. Is it one feature or multiple? Are there several different functionalities that should be observed? Know what it is that\u2019s new and why you need to test it.<\/p>\n\n\n\n
When and what should you beta test?<\/h3>\n\n\n\n
Beta testing can \u2013 and probably should! \u2013 happen at any stage of the product, from initial launch to one-off feature releases. I think features should go through beta as much as possible.<\/p>\n\n\n\n
As for what <\/em>you should beta test: anything that needs to have risk reduced.<\/p>\n\n\n\n Some changes can go straight out, particularly if they\u2019re bug fixes or things that you’ve de-risked enough in the planning and design stages. De-risking in the planning stage means prototyping and checking with real users that this change is small enough, inconsequential enough, and obvious enough. If it\u2019s exceedingly clear that it\u2019s the right thing to do, then you don’t necessarily need to put it through beta.<\/p>\n\n\n\n However, if the change is likely to disrupt user workflow and cause confusion, or you’re unsure what the results will be, then beta test it.<\/p>\n\n\n\n Defining the target outcomes is crucial to the success of your beta program and, indeed, any larger product launch checklist<\/a>. This is actually why we built target outcomes directly into ProdPad! We capture targets and outcomes, at both the initiative level<\/a> (which is a higher level) as well as at the individual idea level.<\/p>\n\n\n\n Regardless of what you’re releasing, whether it’s a micro-experiment or a bigger chunk of your project, you should clearly outline your target outcomes ahead of time. Know the metrics that you need to see.<\/p>\n\n\n\n These should be SMART goals, meaning they should be:<\/p>\n\n\n\n Here\u2019s an example of SMART goals embedded in target outcomes for a beta program: <\/p>\n\n\n\n A public beta<\/strong> is, as the name suggests, public-facing. You not only label the beta feature in the app but also on your marketing site. It shows the world, and potential customers, what you\u2019re working on \u2013 which adds transparency and value. Once the feature is solid, you remove the \u201cbeta\u201d label from marketing materials.<\/p>\n\n\n\n A private beta<\/strong> might be revealed only to your user base, via a label in the app or some other outreach like an email announcement. Private betas can also be small subsets of your user base that you\u2019ve selected specifically and communicated with privately.<\/p>\n\n\n\n Using feature flags<\/strong><\/a> is a common practice when you just want to observe user behavior without orchestrating a beta program. These users won\u2019t necessarily know they\u2019re using a beta feature, while you collect data and insights on their behavior. You can continue iterating on the beta feature until it\u2019s ready to launch \u2013 at which point you simply remove the flag.<\/p>\n\n\n\n How do you know which to choose? Generally, you can publicize stuff that\u2019s pretty stable and will be iterated upon. If the change is still iffy or experimental, you can keep it private. That said, here at ProdPad, we try to weed out any unstable experiments before we even get to the point of building in production.<\/p>\n\n\n\n The next step to running a beta program is to decide which users will participate in the test. Here are a few guidelines for defining who is in the program.<\/p>\n\n\n\n You should beta test with users who are okay with instability in the product. If you’re working in enterprise software and your product is mission-critical, then you probably have a subset of users who only want the stuff that is rock solid. The beta period is just too risky for them.<\/p>\n\n\n\n On the other hand, some users will enthusiastically try new, less stable versions of your product. They love to test stuff first! These users are interested in seeing your product improve as quickly as possible.<\/p>\n\n\n\n You can also beta test with users who asked for the feature in the first place. Follow up with people who\u2019ve requested certain things, such as an integration with a certain tool. This is an obvious set of users who would not only use the feature but are actively interested in giving you feedback on it. <\/p>\n\n\n\n Whether or not you actually tell these users they\u2019re in a beta program \u2013 that\u2019s a different question. We\u2019ll address the topic of recruitment vs. feature flags, and conducting a public vs. private beta test.<\/p>\n\n\n\n The number of beta testers depends on how many people will use the feature once it\u2019s launched. How wide is the rollout? That answer will help you determine how large the beta pool should be.<\/p>\n\n\n\n In my opinion, high-quality feedback from a few people can be enough. Identify 10-20 people, interviewing as many as you can. (See our best practices for customer feedback sessions<\/a>.)<\/p>\n\n\n\n If you\u2019re beta testing a feature with hundreds or thousands of users, that\u2019s fine, too! You\u2019ll just likely use software to track their in-app actions rather than ask for anecdotal feedback. More on that later!<\/p>\n\n\n\n Another important part of designing a beta test is defining how long the test will last. The length of beta programs varies depending on your product and your users.<\/p>\n\n\n\n Google kept Gmail in beta<\/a> for five years! Other companies may test for a single cycle, or a single sprint, just to see how it works for the first couple of weeks until a decent number of users are on it.<\/p>\n\n\n\n The duration of a beta test depends on two main factors: what kind of traffic you’re getting on the feature, and what level of assurance you need that the feature is sufficiently de-risked and ready for the wider world. How sure do you want to be that this is, in fact, the right thing?<\/p>\n\n\n\n Obviously, if traffic is low, then the testing period will be longer so that your team collects enough user data.<\/p>\n\n\n\n If the feature is more complex than a one-and-done functionality, then the testing period will be longer, too. Sometimes you need to observe how a cohort of users interacts with the feature over time. Then you might compare cohorts together and determine when user behavior or performance has reached your target. <\/p>\n\n\n\n If you\u2019ve decided to select specific users for a private or public beta program, the key to recruiting them is \u2013 just to ask. A lot of people like to participate in stuff such as beta tests or feedback sessions.<\/p>\n\n\n\n If need be, coordinate with the customer success or customer support teams on how best to communicate with these users and loop them in.<\/p>\n\n\n\n Let \u2018er rip! With everything defined and clearly outlined, you\u2019re ready to give users access to the beta feature. Keep in mind that the test will still be underway as you begin to track and analyze the results.<\/p>\n\n\n\n You should track beta test results the same way you track anything else in your product. Most modern products have in-app tracking of user behavior. Make sure you’ve embedded appropriate tracking in this new feature so that you can get these insights.<\/p>\n\n\n\n Some teams want to have just the session logs, where they can read about the number of clicks on different elements of the app. Other teams like to have session video recordings, like Hotjar<\/a> or FullStory<\/a>, where they can literally watch how people interact with it.<\/p>\n\n\n\n Both types of tracking software are valuable, depending on how much time you’re willing to give to it and how much you\u2019ll use the data.<\/p>\n\n\n\n First, you should check that people are finding the new feature! After a day or two, just check that people can find it. If no one can find it, you’re not going to get any data on it.<\/p>\n\n\n\n From there, you can look at other issues. Maybe everyone finds it, but no one’s able to complete their job in it. Maybe everyone is able to complete their job, but no one actually liked <\/em>using it. On this last point, you\u2019ll need to run some customer interviews to gather some qualitative feedback.<\/p>\n\n\n\n Of course, compare your target outcomes to your actual outcomes as you start measuring. This is an ongoing process until you hit your targets. <\/p>\n\n\n\n The product manager should report back to their immediate team about the beta test results. This could be in the form of a retrospective or just a regular weekly report that says, \u201cHere’s an update on recent releases, what\u2019s working and not working.\u201d<\/p>\n\n\n\n The team should have a chance to comment on the results and discuss the next steps. Then this team feedback can be reviewed, and any necessary changes can be prioritized into the next sprint.<\/p>\n\n\n\n Remember to prioritize iterations on your beta features and continue pushing those through development! This can be tricky.<\/p>\n\n\n\n In product management, every decision must be considered in light of all the other decisions that need to be made. When you’ve launched something, you need to understand the impact it’s had on the app. And when you’re planning the next sprint, you need to look holistically at the product roadmap<\/a> and business objectives. <\/p>\n\n\n\n Many teams like to move on to build new stuff because it\u2019s more exciting. But I recommend prioritizing things that are already live but not working quite right.<\/p>\n\n\n\n2. Define the target outcomes of your beta program<\/h2>\n\n\n\n
\n
\n
3. Choose between public beta vs. private beta vs. feature flags<\/h2>\n\n\n\n
4. Define who is in the beta program<\/h2>\n\n\n\n
Who do you beta test with?<\/h2>\n\n\n\n
How many testers should you include in the beta program?<\/h2>\n\n\n\n
5. Define how long your beta test will last<\/h2>\n\n\n\n
How to decide the duration of a beta test<\/h2>\n\n\n\n
6. Recruit your testers: How to recruit to a beta program<\/h2>\n\n\n\n
<\/figure>\n\n\n\n7. Conduct the beta test<\/h2>\n\n\n\n
8. Track and analyze beta test results<\/h2>\n\n\n\n
How to analyze beta test insights<\/h2>\n\n\n\n
9. Report on the beta outcome<\/h2>\n\n\n\n
10. How to make decisions based on your beta<\/h2>\n\n\n\n