The following material is Chapter 1 from The Build Master: Microsoft’s Software Configuration Management Best Practices by Vincent Maraia, ISBN: 0321332059.
CHAPTER 1: Defining a Build
Philosophy: The build is a piece of software and should be treated as such.The build is among the most heavily used and complex pieces of software in the development group and should be treated as such.
—Danny Glasser, Microsoft developer in the Systems Group
March 9, 1991
The first thing we should do is define what a build is. What Danny describes in the previous quotation is important.The purpose of a build is to transform code written in any computer language into an executable binary. The end result of a software build is a collection of files that produce a product in a distributable package. In this case, package can mean a standalone application, Web service, compact disc, hot fix,or bug fix.
If you do not think it is worthwhile to spend resources on a good build process, your product will not be successful.I have been on a couple of product teams at Microsoft that have failed, and I have seen many others fail because they were not able to consistently build and test all of the product’s code. I also see this at customer sites when I am reviewing their build process. The companies that have clean, crisp, reliable build and release processes are more successful than the ones with ad hoc, insufficient processes.
The Two Types of Builds: Developers and Project
I like to say that there are really only two types of builds: ones that work and ones that don’t. Seriously, though, when you’re shipping a product, you should consider these two different types of builds:
- Developers’ (local machine builds)—These types of builds often happen within an editor such as Visual Studio, Emaqs, Slick, or VI. Usually, this is a fast compile/link of code that the developer is currently working on.
- Project (central build process)—This type of build typically involves several components of an application, product, or a large project, such as Windows, or in some cases several projects included in a product, such as Microsoft Office.
The developer’s build process should be optimized for speed, but the project build process should be optimized for debugging and releases. I am talking about optimizing the process, not compiler or linker optimization switches. Although speed and debugging are important to everyone who is writing code, you must design a project build process to track build breaks and the offender(s)as quickly as possible because numerous people are waiting for a build to be released. For a developer, what seems to be most important is clicking some type of Build and Run button to make sure the code compiles without errors and then checking it in. For the build team, building without errors and having the ability to track down the person who broke the build is the most important thing.
Note: In some simple scenarios, these two build cases can use the same process. If this is the case, the team—what I refer to as the Central Build Team—should dictate the build process. This team—not the developers—should design the project build process. All too often, the developers design the project build process, which causes problems. Because developers usually build just the code modules that they work on and not the whole project on a regular basis, they look for shortcuts that are not necessarily in the best interest of building the entire project. For example, they might use file references instead of project references.
If a developer specifically references a file in Visual Studio and the sources of that file change, they are not automatically picked up because a specific version of the file was referenced instead of the project that builds the referenced file. In the interest of saving time, developers use file references.They are not interested in picking up the latest sources of the specified file, but it is not recommended to use file references in a project build.
The Central Build Team should never be at the mercy of mandatory build environment settings for building a specific component. If such a setting is necessary to build a component, it should be proposed to the Central Build Team for inclusion. Then the CBT can determine the impact of the addition or change to the entire project and approve or disapprove the proposal.
Building from the Inside Out
One of my favorite questions to ask a customer’s development or build manager when I go onsite is how often they release a new build process. I usually get long pauses or funny looks and then Finally get the answer “Every day.” Of course, as you might suspect, I am not talking about releasing a daily build, but a new build process. The fact that so many companies do not release new build processes on a regular basis does not surprise me. This is because traditionally creating a build process is an afterthought when all of the specifications of a project have been written. Many project and program managers think that the actual building of a project is pretty trivial. Their attitude is th7at they can simply have the developer throw his code over the wall and hire someone to press a Build button, and everything will be fine. At Microsoft, we understand that whether you’re building the smallest application or something huge and complicated like Windows, you should plan and think through the process thoroughly in advance.
Again, I recommend that you consider the build process a piece of software that you regularly revise and deploy throughout your product team. You should also add to your project schedule some “cushion time” to allow for unforeseen build breaks or delays, I would at least pad the milestone dates one week for build issues.
The concept of “building from the inside out “tends to confuse customers who are not familiar with a centralized build process. The idea is that the Central Build Team determines what the build process is for a product and then publishes the policies to an internal build site. All development teams in the project must comply with the Central Build Team process; otherwise, their code check-in is not accepted and built. Unfortunately, this concept is usually the complete opposite of how a build system for a project actually evolves over time. The Central Build Team for a project usually goes out of its way to accommodate the way developers build their code. “Building from the inside out “means that the Central Build Team figures out the best way to get daily builds released, and everyone uses that process independently or in parallel with the way his specific development team builds. This total change in development philosophy or religion can be a culture shock to some groups. I talk more about changing a company’s culture or philosophy in Chapter 18, “Future Build Tools from Microsoft.” For now, let’s stay on the topic of builds.
What we did in the past in the Windows group—and what they still do today—is to deploy new releases of the build process at major milestones in the project life cycle. Sometimes the new releases involve tool changes such as compilers, linkers, and libraries. At other times, there are major changes such as a new source code control tool or a bug tracker.
Because a build lab tends to have some downtime while the build team waits for compiles, links, and tests to finish, it should take advantage of these slow times to work on improvements to the build process. After the lab tests the improvements and confirms they are ready for primetime, it rolls out the changes. One way to deploy a new build process after a shipping cycle is to send a memo to the whole team pointing to an internal Web site that has directions on the new process that the Central Build Team will be using in future product builds.
| Microsoft Sidenote: Developers in a Build Lab |
|---|
|
Today, the Windows build lab has its own development team working on writing and maintaining new and old project tools. The development team also works on deploying new build processes. Conversely, of the more than 200 customers I’ve spoken to, only one or two of them have developers working in a build team. Remember Danny’s quote at the beginning of this chapter and notice the date—1991. In 1991, Windows NT had only a few hundred thousand lines of code, unlike the more than 40 million lines of code that Windows XP has today. Even in the early stages of developing Windows NT, Microsoft recognized the importance of a good build process. |
Chapter 3, “Daily, Not Nightly, Builds,” covers in more detail the importance of the build team being the driving force to successfully ship a product.
More Important Build Definitions
I need to define some common build terms that are used throughout this book. It is also important for groups or teams to define these terms on a project-wide basis so that everyone is clear on what he is getting when a build is released.
- Pre-build—Steps taken or tools run on code before the build is run to ensure zero build errors.Also involved are necessary steps to prepare the build and release machines for the daily build, such as checking for appropriate disk space.
- Post-build—Includes scripts that are run to ensure that the proper build verification tests (BVTs)are run. This also includes security tests to make sure the correct code was built and nothing was fused into the build.
- Clean build—Deleting all obj files, resource files, precompiled headers, generated import libraries, or other byproducts of the build process. I like to call this cleaning up the “build turds.” This is the first part of a clean build definition. Most of the time, build tools such as NMake.exe or DevEnv.exe handle this procedure automatically, but sometimes you have to specify the file extensions that need to be cleaned up. The second part of a clean build definition is rebuilding every component and every piece of code in a project. Basically the perfect clean build would be building on a build machine with the operating system and all build tools freshly installed.
| Microsoft Sidenote: Clean Build Every Night |
|---|
|
While working in the Windows NT build lab on NT 3.51, I remember reading in a trade magazine that the Windows NT group ran clean builds every night. The other builders and I laughed at this and wondered where this writer got his facts. We would take a certain number of check-ins (usually between 60 and 150 per day) and build only those files and projects that depended on those changes. Then one of us would come in over the weekend and do a clean build of the whole Windows NT tree, which took about 12 hours. We did the clean builds on the weekend because it took so long, and there were usually not as many check-ins or people waiting on the daily build to be released. Today,with the virtual build lab model that I talk about in Chapter 2, “Source Tree Configuration for Multiple Sites and Parallel (Multi-Version) Development Work,” the Windows NT team can perform clean builds every night in about 5 or 6 hours. |
- Incremental build—The secret to getting out a daily build to the test team, regardless of circumstances, is to perform incremental builds instead of daily clean builds. This is also the best way that you can maintain quality and a known state of a build. An incremental build includes only the code of the source tree that has changed since the previous build. As you can guess, the build time needed for an incremental build is just a fraction of what a clean build takes.
- Continuous integration build—This term is borrowed from the extreme programming (XP) practice. It means that software is built and tested several times per day as opposed to the more traditional daily builds. A typical setup is to perform a build every time a code check-in occurs.
- Build break—In the simplest definition, a build break is when a compiler,linker,or other software development tool (such as a help file generator) outputs an error caused by the source code it was run against.
- Build defect—This type of problem does not generate an error during the build process; however, something is checked into the source tree that breaks another component when the application is run. A build break is sometimes referred to or subclassed as a build defect.
- Last known good (LKG)or internal developers workstation (IDW) builds—These terms are used as markers to indicate that the build has reached a certain quality assurance criterion and that it contains new high-priority fixes that are critical to the next baseline of the shipping code. The term LKG originated in the Visual Studio team, and IDW came from the Windows NT organization. LKG seems to be the more popular term at Microsoft.
| Microsoft Sidenote: Test Chart Example |
|---|
|
The best way to show how Microsoft tracks the quality of the product is through an example of the way the Windows team would release its version of a high-quality build. Again, the Windows team uses the term internal developers workstation (IDW), and other teams use last known good (LKG). In the early days of the Windows NT group, we had a chart similar to the one in Figure 1.1 on the home page of the build intranet site. Most people on the project kept our build page as their default home page so that whenever they opened Internet Explorer (IE), the first thing they would see was the status of the project; then they would check the Microsoft (MSFT) stock price. |
FIGURE 1.1: Sample quality chart.
The way to read Figure 1.1 is that any build we released that passed more than 90 percent of the basic product functionality tests—what we called regressions tests—and did not introduce new bugs was considered an IDW build. This quality bar was set high so that when someone retrieved a build that was stamped IDW, he knew he had a good, trustworthy build of the product. As you can imagine, when the shipping date got closer, every build was of IDW quality.
Furthermore, when a new IDW build was released to the Windows team, it was everyone’s responsibility to load the IDW build on the machine in his office and run automated stress tests in the evening. Managers used to walk to their employees’ offices and ask them to type winver to verify that they had the latest IDW build installed before they went home for the evening. Today, managers have automated ways to make sure that everyone is complying with the common test goal. This is also where the term “eating our own dog food” originated. Paul Maritz, general manager of the Windows team at that time, coined that phrase. It simply means that we test our software in-house on our primary servers and development machines before we ship it to our customers. Dogfooding is a cornerstone philosophy at Microsoft that will never go away.
The build team would get the data for the quality chart from the test teams and publish it as soon as it was available. This is how we controlled the flow of the product. In a “looser ” use of the word build, the quality became part of the definition of a build number. For example, someone might say, “Build 2000 was an excellent build” or “Build 2000 was a crappy build,” depending on the test results and personal experience using the build.
