DevLog: Speeding up Visual Studio build

Posted on Nov 15, 2016

The problem

If you are working on a large solution with hundreds of thousands lines of code it can take dozens of minutes to build, wasting time and productivity. Enabling a feature like incremental build (/Gm flag) can save you a considerable amount of time in this case - luckily VS enables it for us by default when building in Debug configuration.

Incremental build helps, a lot - it usually allows you to test changes in a few dozen seconds rather than minutes. However it only works as long as you don't modify some commonly used header, require a configuration change, or trigger a full rebuild for some other reason. And eventually this will happen, wasting your time once again.

Wouldn't it be nice if we could somehow speed up the full rebuild as well? Well there's the multi-processor compilation (/MP) option that will completely parallelize the compilation process over all available CPU cores. It does a good job, speeding up the build process theoretically by 2-4x (or more, depending on your CPU). But it has a huge caveat - it cannot be enabled together with incremental rebuild. Therefore it is not very helpful since /Gm usually saves you much more time during normal development.

However there is a way to get around this limitation and still parallelize your build, getting nearly the same speedup. The only requirement is that your solution consists of multiple projects. This is the case for projects like Banshee that are built in a modular and layered way, where libraries are built on top of libraries, plugins on top of that, and executables on the very top.

Although this guide is aimed at existing multi-project solutions, if your solution is not structured this way, you can consider a refactor by separating logical units into separate modules (projects). If your projects are not dependent on each other, you have just parallelized your build as Visual Studio will automatically build non-dependent projects in parallel.

But in a real world scenario many projects will end up being dependent, so lets see how we can parallelize such builds.

The solution

Imagine we have a solution with project layout similar to this:

BansheeFastBuild.png

Where each entry represents a VS project, and arrows represent dependencies (Core depends on Utility, etc.).

When VS starts your build it will proceed by compiling & linking each project above in a sequential manner (e.g. first Utility, followed by Core, etc.). Because the projects are compiled sequentially it will be utilizing only a single CPU core at a time.

If each of the projects takes 3 minutes to build, it will take 15 minutes for a full rebuild. But if we could build everything in parallel theoretically we could then build the entire solution in just three minutes. That's quite an improvement!

There doesn't seem to be a technical reason all of the projects couldn't be compiled at the same time, even if they depend on each other (at least I'm not aware of one). This is because the dependency is only relevant to the linker, and actual compilation part of the build can be parallelized.

Other build systems seem to recognize this and will perform compilation in parallel, but Visual Studio always performs compilation and link on a single project before moving on to the next (dependent) project.

The implementation

We can separate out the compilation & link steps by calling the msbuild tool manually, instead of using the default VS build. Luckily msbuild accepts VS projects and solutions as input, so calling it is fairly trivial. To compile a project (without linking) call:

msbuild "myProject.vcxproj" /t:ClCompile /p:Configuration=Debug;Platform=x64

Substitute your project name, optionally change the "Configuration" and "Platform" parameters as needed. You will also have to either add msbuild to your PATH variable, or run the code in VS Command Prompt, otherwise it won't be able to find msbuild executable.

But we want to call msbuild in parallel for multiple projects. If you're using Python you can do this easily using the multiprocessing module:

import multiprocessing
import os

def compileWorker(project):
    projectPath = project + ".vcxproj"
    os.system("msbuild {0} /t:ClCompile /p:Configuration=Debug;Platform=x64".format(projectPath))
    return

if __name__ == '__main__':
    # List of all your projects to compile (normally you'd want to generate this from the .sln file)
    projects = ["P0", "P1", "P2", "P3",
                "P4", "P5", "P6", "P7",
                "P8", "P9", "P10", "P11"]

    # Start msbuild for every one of your projects, and wait until they all finish
    pool = multiprocessing.Pool(multiprocessing.cpu_count())
    pool.map(compileWorker, projects)

This will trigger a separate msbuild compile process for every logical CPU core, and queue new builds as old ones finish, as long as there are projects to process.

After all projects have compiled, you now just need to link them. Either call msbuild directly on the .sln file, or just build as normal within VS. It should automatically pick up the compiled files from the previous step and just perform the link step.

msbuild "mySolution.sln" /p:Configuration=Debug;Platform=x64 /m

Note: When compiling a large solution with many projects in parallel your storage could end up being a bottleneck as gigabytes of data could be generated. Using an SSD will help.

Conclusion

Voila, that's it! This has been tested in VS2015, but earlier versions should work in a similar way.

In my case the build time went down from ~14 minutes to ~6 minutes using a quad-core CPU, your results may vary.

Working code

For convenience I've also provided a full python script that parallelizes builds for all C++ projects within a VS solution: https://github.com/BearishSun/BansheeEngine/blob/master/Scripts/fast_build.py

The script is expected to be triggered from within VS as a pre-build step. Set up the pre-build step in project properties for the first project in your build order (this script needs to run before anything else), as shown below.

VSBuildSetup.png

Share

Facebook
Twitter
Google+
Pinterest
Reddit
LinkedIn
Email