By Greg DeMichillie [bio]
Posted: Sep. 6, 2004

• Chart: Software Prerelease At a Glance

The following is the full text of an article published by Directions on Microsoft, an independent research firm focused exclusively on Microsoft strategy & technology. Each month we make one or more key articles available to non-subscribers.

However, customers and partners should also be aware that Microsoft’s decentralized product development means there will be variations in how each product group manages its prerelease activities. This article only describes the general principles that are applied across all groups. (For a chart summarizing the prerelease stages of Microsoft software development, see "Software Prerelease At a Glance".)

Milestones: Progress vs. Schedule

Because of the size and the complexity of the software it develops, Microsoft (like most professional developers) breaks development into a series of checkpoints known as milestones. Each milestone has a definition document that describes which features will and will not be implemented by that milestone and what the acceptable quality level for those features are.

Microsoft typically uses what is called a "zero defect" milestone. This doesn’t mean that the product has no bugs or missing features, but only that it has been tested to ensure it meets a specified quality criteria. Ideally, the quality level was determined when the milestone was established and hasn’t changed. This helps teams assess their progress against their schedules. If a team planned to complete milestone 1 (typically referred to as "M1") by June 1, for example, but was unable to get all the necessary features implemented to the required quality level until July 1, the team has concrete data showing that it is one month behind schedule. Until that time, the team could only give schedule predictions based on the best guesses of its developers and program managers. But with specific information in hand, the team can identify areas of the product that are in trouble, balance loads among teams, and adjust its schedule to better reflect reality.

Because they are not "feature complete" and contain many (often severe) bugs, milestone builds are not usually distributed outside of Microsoft. Extremely close partners of Microsoft may be able to obtain these builds, but they must be careful not to draw conclusions about the final features, stability, or performance based upon such a preliminary build.

Alpha: Opening the Kimono

Napoleon is reported to have said, "No battle plan survives contact with the enemy." Similarly, no software product, no matter how well designed and conceived, survives first customer contact without requiring changes. As a result, at some point during the development process, product teams decide their product is far enough along to begin getting external feedback to help validate the features and architecture. To help get that feedback while there is still enough time to act, most product groups release alpha versions of their products.

Sometimes (as in the case of the Office products) these alpha (or preview) releases are given to only select customers who have demonstrated a willingness to examine prerelease products and are known to give quality feedback. Other times (as with Visual Studio and Windows "Longhorn"), they are made available for download. Customers who are participating in a formal alpha program typically have dedicated Microsoft support resources available to them. Those in the download program must make do with public newsgroups.

In addition to providing Microsoft with feedback, alpha releases give customers an early look at the architecture of a product and help them make decisions on how and when to deploy the product. Customers using alpha releases in this fashion, however, must accept that the final product will differ, sometimes significantly, from the alpha, and any plans will need to be reevaluated as the product nears completion.

Finally, an alpha release can serve a marketing purpose for Microsoft, particularly for products that require significant third-party developer support (such as Windows and developer tools) or which have long lead times for adoption (such as server products like SQL and Exchange). Publicizing the availability of alpha builds helps Microsoft competitively by slowing the adoption of competing products and can draw developers and IT managers' attention toward its own offerings, even if they are not yet commercially available. Using prerelease builds to draw attention in this manner is by no means unique to Microsoft.

Beta: The End Is in Sight

After one or more alpha releases, a product reaches a state where it is ready for widespread testing outside of Microsoft. Unlike alphas, which are designed primarily to gain feedback about the design and features of a product, the goal of beta releases is to shake out bugs. Not all bugs will be found this way, nor will all of the bugs discovered be fixed. Nevertheless, the beta cycle is important for shipping high-quality products because beta testing exposes the product to the enormous number of combinations of hardware and existing software in use across millions of PCs.

As with alpha releases, Microsoft selects some customers to participate in its formal beta program and sometimes makes the beta available for download as well. Those participating in the formal program can expect dedicated support resources while those in the free program must make do with newsgroups. But unlike alpha releases, where Microsoft is able to add new features or significantly alter existing ones based upon customer feedback, changes made in response to the beta release are typically restricted to cutting features that are a source of too many bugs. Therefore, for customers and partners interested in having a significant impact on the design of a product, the beta stage is too late.

The other significant difference between alpha and beta releases is that a beta release is expected to be feature complete. Having all product features completed by the first beta release was once a cardinal rule of software development at Microsoft, but in recent years some groups have taken a more lax attitude and begun to release so-called betas that have incomplete features. For example, the initial beta release of Windows XP Service Pack 2 did not have all of its features implemented and new features were added in subsequent betas. Any announced schedule for a product with an incomplete beta should be viewed with great skepticism because introducing new features late in the product development cycle has unintended, and usually destabilizing, consequences. Accurate estimates of a product’s ship date can only be made once all new feature development has stopped and the developers have fully transitioned to bug-fixing mode.

Release Candidates: Are We There Yet?

The last steps toward the final release of a product are Release Candidates (RCs). As its name implies, an RC is a build of the product that Microsoft believes is good enough to be shipped and wants to verify. Verifying an RC involves final testing by the product’s quality assurance (testing) team as well as distribution to key customers.

By the time a product reaches the RC stage, no further changes are contemplated, except those required to fix extremely serious "show-stopping" bugs. Other changes, including fixes for less severe bugs, are likely to be deferred until a later release or the first service pack. Customers evaluating an RC release can therefore expect very few changes between it and the final build. This makes RC builds appropriate for final performance and scalability evaluations.

As with beta releases, product groups occasionally release RC builds that were obviously not ready for shipment. The RC builds of BizTalk 2004, for example, lacked product documentation. Many customers assumed this was an oversight and were dismayed to find out that the final version of the product contained no documentation and that documentation would be gradually posted to Microsoft’s Web site after the product shipped.

Resources

For an explanation of how Microsoft categorizes and prioritizes bugs, see "A Bug’s Life" on page 19 of the Aug. 2003 Update.

Jim McCarthy, one-time Product Unit Manager of the Visual C++ team, describes many of Microsoft’s development practices in his book Dynamics of Software Development, (ISBN 1-556158238) and in a presentation called "21 Rules of Thumb for Shipping Great Software on Time," a summary of which can be found at blogs.msdn.com/David_Gristwood/archive/2004/06/24/164849.aspx.