Never compile your widgetset again with Gradle build-cache

Widgetset compile is one of the most time-consuming things in any Vaadin build. On a fast machine it will usually take from just under one minute up to several minutes every time we launch a build . If we are using a smart build system like Gradle, along with a smart plugin that can monitor the inputs of the compilation we can get away with only compiling when something changes, but still in reality we will have to re-compile it from time to time.

While long build times usually are not a problem when building on a CI server like Jenkins, it does get very tedious in development.

Vaadin CDN

Vaadin CDN is Vaadin's (the company) solution to the problem by providing a service (http://cdn.virit.in/) that will remotely compile the widgetset and serve it later whenever the widgetset is needed.

But it has a few drawbacks.

First of all we can only use addons that are available in Vaadin Directory. This is very limiting as most likely we will have company internal addon libraries we want to share among our projects but do not want to upload to the Vaadin Directory.

The second issue is that it does not support remotely building GWT sources we have in the project. This is also very limiting as most likely if we have a widgetset, we also have client sources in the project.

The Gradle Vaadin Plugin supports the Vaadin CDN fully if we still want to use it. To take it into use one can follow the few simple steps in the Using Vaadin CDN wiki article. But before doing that, read on, we might instead want to use the build cache provided by Gradle.

Gradle Build Cache

In Gradle 3.5 the Gradle team introduced an interesting incubating feature in this regard. A build artifact cache.

While previously Gradle had been able to monitor inputs/outputs of the task and not re-compile if not needed, now they also introduced a cache that contains the artifacts of the tasks. This fits perfectly in the problem domain we have with widgetsets in Vaadin projects!

Using the build cache locally

The first thing we need to do is upgrade our Gradle version we use in our project to Gradle 4.0 to get the feature.

After that we need to upgrade to the latest alpha build of the Gradle Vaadin plugin.

plugins {
  id "com.devsoap.plugin.vaadin" version "1.2.1"
}

Since the build cache is still an incubating feature we will need to use a command line parameter --build-cache to turn the build cache on when running the build.

Here is what it will look like when running the widgetset compilation for the first time with the build cache turned on.

$ gradle --build-cache vaadinCompile
Using Gradle Vaadin Plugin 1.2.1
Build cache is an incubating feature.
Using directory (/home/john/.gradle/caches/build-cache-1) as local build cache, push is enabled.
:vaadinPluginVersionCheck SKIPPED
:compileJava UP-TO-DATE
:vaadinUpdateWidgetset
:processResources UP-TO-DATE
:classes UP-TO-DATE
:vaadinClassPathJar SKIPPED
:vaadinCompile

BUILD SUCCESSFUL

Total time: 43.742 secs

7 tasks in build, out of which 2 (29%) were executed
2  (29%) skipped
3  (43%) up-to-date
1  (14%) cache miss
1  (14%) not cacheable

As can be seen, the build took 44 seconds to execute where most of the time went compiling the widgetset. Also notice we didn't have any cached content yet.

Now, lets make it interesting. Now we first run gradle clean on the project to remove any pre-built classes or artifacts from the build directory from the previous build. The project should now be in the shape which it was before any compilation had been done.

And now lets run the vaadin widgetset compile again for the project.

$ gradle --build-cache vaadinCompile
Using Gradle Vaadin Plugin 1.2.1
Build cache is an incubating feature.
Using directory (/home/john/.gradle/caches/build-cache-1) as local build cache, push is enabled.
:vaadinPluginVersionCheck SKIPPED
:compileJava FROM-CACHE
:vaadinUpdateWidgetset
:processResources
:classes
:vaadinClassPathJar SKIPPED
:vaadinCompile FROM-CACHE

BUILD SUCCESSFUL

Total time: 1.4 secs

7 tasks in build, out of which 4 (57%) were executed
1  (14%) skipped
2  (29%) loaded from cache
4  (57%) not cacheable

Now a pristine new build took only 1.4 seconds!

This is because Gradle has determined that the artifacts that previously was generated in the previous build still are viable for this build as the inputs (source files/resources/etc) had not changed. We can see that roughly 30% of all artifacts which consists of all the compiled classes as well as the compiled widgetset has been loaded from the cache. The way Gradle does this is examining the checksum of the inputs and outputs of the build tasks and if nothing has changed then we can use a cached version.

But while having a local cache is nice for a single developer we can make it even more interesting by sharing the cache with multiple developers!

Using the build cache remotely

Just like we could use the Vaadin CDN to host our widgetset we can also host our own Gradle build cache remotely and share it with all our developers making our fellow developers builds much faster!

Gradle provides a simple Docker container to host build cache so setting it up is a breeze.

Building the container is easy by following the instructions at https://github.com/gradle/task-output-cache-demos/tree/master/samples/03-use-http-backend.

But to even make it easier, I have already pre-built it for you so one can just run an instance of the cache by running:

docker run --name gradle-task-cache -d -p 9000:80 johndevs/gradle-task-cache:latest

Once that is run we have a remote Gradle build cache running on port 9000!

Now we just need to point our Vaadin project Gradle build to the build cache. We can do that by opening up our settings.gradle in the project and adding the following:

buildCache {
    local {
        enabled = false
    }
    remote(HttpBuildCache) {
        url = "http://192.168.0.4:9000/"
        push = true
    }
}

That will turn off the local build cache and add our remote cache that works over HTTP. The example is running on 192.168.0.4:9000, you most likely will need to point that to the correct ip address. You can also leave on the local cache if you want a two-tier cache :)

Now when we run our build again for the first time it will look like something like this:

$ gradle --build-cache vaadinCompile
Using Gradle Vaadin Plugin 1.2.1
Build cache is an incubating feature.
Using HTTP build cache (http://192.168.0.4:9000/) as remote build cache, push is enabled.
:vaadinPluginVersionCheck SKIPPED
:compileJava
:vaadinUpdateWidgetset
:processResources
:classes
:vaadinClassPathJar SKIPPED
:vaadinCompile

BUILD SUCCESSFUL

Total time: 49.256 secs

7 tasks in build, out of which 6 (86%) were executed
1  (14%) skipped
2  (29%) cache miss
4  (57%) not cacheable

Since we turned the local cache off, Gradle could not find the cached artifacts so it re-compiled the widgetset. But what it doesn't show is that it also uploaded the build artifacts to the remote build cache.

Now, if we would ask the developer sitting next to us to checkout the project and compile it on their machine (assuming they have no new changes and we have committed the changes to build.gradle and settings.gradle) then it would look like this:

$ gradle --build-cache vaadinCompile
Using Gradle Vaadin Plugin 1.2.1
Build cache is an incubating feature.
Using HTTP build cache (http://192.168.0.4:9000/) as remote build cache, push is enabled.
:vaadinPluginVersionCheck SKIPPED
:compileJava FROM-CACHE
:vaadinUpdateWidgetset
:processResources
:classes
:vaadinClassPathJar SKIPPED
:vaadinCompile FROM-CACHE

BUILD SUCCESSFUL

Total time: 1.546 secs

7 tasks in build, out of which 4 (57%) were executed
1  (14%) skipped
2  (29%) loaded from cache
4  (57%) not cacheable

Now, since the project widgetset is already build for him, his first build would only take 1.5 seconds!

Automate it!

If we now take the build and put it on a server like Jenkins and build the project every night, then every morning the build cache would contain the latest pre-built widgetset and our developers would never need to wait for a widgetset to build again unless they specifically change the client side code. But even if the do change the client side code, the widgetset would only be re-built once by either the CI or the developer himself which after everyone would get it directly from the cache.

Taking the cache into use is very easy for existing projects, we don't need to split out the widgetset compile into a separate module, we don't even need to set up any proxy rules to pass the company network as we can run it inside our network. So as we can see, the build cache from Gradle is a God send to any Vaadin developer out there who has been waiting for the widgetset to compile for so many years.