Build Process Metrics - What Should I Capture?

The short answer is everything!

When I talk about build process metrics I'm not talking about code coverage, or lines of code - there's a plethora of tools that will extract that information for you such as cobertura, findbugs etc. What I mean is information about the build process itself.

Here's just an example of a few metrics I find useful:

Current time is Dec 28, 2013 10:41:38 AM
System.getProperty('os.name') == 'Windows 7'
System.getProperty('os.version') == '6.1'
System.getProperty('os.arch') == 'amd64'
System.getProperty('java.version') == '1.7.0_45'
System.getProperty('java.vendor') == 'Oracle Corporation'
System.getProperty('sun.arch.data.model') == '64'

You might be asking why this is important, or why bother when you're using a build automation tool such as Jenkins. Unfortunately for me not every project is fully automated. It should be, but we're not quite there yet. I like to use Gradle as it provides the tools to extract a lot of information about your build process.  As this is performed at a lower level than CI this means for those odd tasks that are run manually we don't miss out on that juicy data. Data is valuable, just ask Google.....

So, what do we hope to find? 

The interesting thing about data is you don't really know what's there until you start digging. One such example presented itself recently ... at my company we split the task of releasing software into 2 stages. First, we have the volatile build phase, followed by the stable release phase which is performed by a different team. During one projects build phase I was repeatedly running deployments of our web application in approximately 3 minutes to our QA environments. However, when the release manager was deploying into production it was taking 45 minutes..... what the deuce??

Before even looking at the data I knew we both used the same work issued laptop, with the same version of Java, and the Gradle wrapper ensured we were using the same version of the build tool. So, what was causing this massive increase in deployment time? Well, I later found out from the data that the release manager had failed to mention he was running the deployment from his bedroom across a VPN ..... Bingo! the increased deployment time was due to the network, and was easy to fix. (we quickly put this project in Jenkins with a manual start).

The moral of the story is to collect all the information you can. Gradle is an extremely powerful DSL, and its groovy support means your options for pushing and processing the data are endless.