Spring Cloud Config Server with Camel

Spring Cloud Config Server with Camel

First of all, a disclaimer: this post will outline an interesting solution to an interesting problem that only exists due to the infrastructure being as it is and choices what they are. Things can be setup easier and more efficient. But where is the fun in that?

A bit of background to start things off with. At a current project we're pretty deep in everything Microsoft. So, we're running Spring Boot with Camel on Windows servers. I'll repeat: we're doing the whole microservice-y thing, on Windows, with our Java applications wrapped in Windows Services. We're using Azure DevOps for CICD, and using a broad assortment of Windows based tooling for monitoring and the likes. I'm actually pretty impressed in the way it works, even if it rubs me in the wrong way a bit.

Given our current CICD setup and tendency to write quite a lot of unit- and integration tests, it takes a relatively long time for a single application to go through the full pipeline and be running on the target machine. Deploying to any other machine then DEV is also scheduled and will only take place once a week at most.

So, let's say a password changes on Production. We're not informed, but we need to deploy a fix yesterday if possible. This'll be a manual action in our current setup, as we need to circumvent the scheduled deployment and trigger the damn thing by hand. Not ideal. So we started looking at way to speed up the deployment of small configuration changes, without rebuilding the entire application.

Enter Spring Cloud Config Server

SPRING CLOUD CONFIG SERVER

We were having some discussion in our team on how we can do a quiet reload or redeploy when we're faced with 'simple' changes like configuration changes. We're already running with Spring Boot and Spring Cloud. After some digging, Spring's Config Server seemed a nice fit.

The Config Server allows you to pull your configuration from a remote location. This can be a database, git repository, file share, whatever. With some Spring Boot magic, all you need to do is create an application with the spring-cloud-starter-config dependency, the @EnabledConfigServer annotation, add some magic Spring Boot properties and the server is all setup.

For clients to connect to it, all you need to do is add the spring-cloud-starter-config dependency and add a bootstrap.yml file, with some properties to point to the server.

I'll not go into full detail on how to set this up. Documentation is pretty straight forward: https://spring.io/guides/gs/centralized-configuration/#_reading_configuration_from_the_config_server_by_using_the_config_client

What "Spring Cloud Configuration Server" can do for Microservices
https://www.linkedin.com/pulse/what-spring-cloud-configuration-server-can-do-ashish-verma

The short version is this:

The Config Server will pull configuration from a remote repository. Clients can call the Config Server in the Bootstrap Phase of Spring Boot to request properties from the server. The properties on the remote are just application.yaml files, but with the application name as part of the fileName. The Config Server will match fileName to the calling application and spring.profile and serve the correct properties. Any client will have a refresh endpoint that will trigger the client to call the config server again and perform a runtime refresh. Clients can even listen to an message broker and publish refresh events for other clients.

I've set this up for our project, even added a Kafka bus with Spring Cloud Bus to trigger configuration reloads via Kafka events, and all seemed well. But I forgot one thing: we use Apache Camel...

THE CONFIG SERVER AND A CAMEL

The nice thing with the Config Server is that it plays really nice with Spring Boot. If you keep withing Pivotal's Spring ecosystem, you can actually refresh properties at runtime with zero downtime by just calling an endpoint or sending a Kafka event. There are some limitations however. Only @Value annotated and @PropertySource annotated fields an classes are reloaded. Fields set in a constructor will be 'stuck' as they are set.

If you're using any classes or frameworks that use constructor injection, builder patterns or anything else that sets fields without Springs @Value annotation, you'll be running into some issues.

Enter Camel. Camel routes are configured by the configure() method in the RouteBuilder. This method is called once, and only once, when the Camel Context is constructed. Even when using @Value'ed fields, the configure() method is never called again, even when we refresh the Spring configuration. There is actually a bug listed with Camel in this regard: https://issues.apache.org/jira/browse/CAMEL-13892

But as you can also read, the bug listing has been closed. The Camel devs have no intention to implement it, and the guys at Pivotal (Spring) haven't either. So, that leaves it up to us to figure out how to get those new properties to stick.

THE SOLUTION

First of all, we need to know which application to actually refresh. We've got all our config files in a single repository, so a commit to a single file will push an update for the entire repository. Be it via webhook or Kafka event, this will restart all of our applications. You could argue that we can split configuration in multiple repositories and you'd be right. With decisions being are what they are, however...

We cannot easily add the application of file names to the webhook, as the DevOps webhook doesn't really know what files were updated. We could add this to a pipeline or our config repository of course, but that would mean we further enlarge our dependency on Azure for all our applications and have an even tighter coupling to the Microsoft landscape. We therefore decided to keep the refresh logic in our applications and decouple from any CI/CD implementation.

Our solution is as ugly as it is elegant:
1. We're going to embed the config server in every single application
2. We're going to enable all embedded config servers to configure themselves
3. We're removing the Kafka events as we don't need to trigger other applications anymore
4. We're going to poll the repository for changes from the application instead of relying on webhooks from DevOps.
5. We're going to reload Camel ourselves in case of configuration changes.

Embedding the Config Server and Configuring Itself

This is quite easy. We'll just add the configuration from the Server's bootstrap.yml to every client:

spring.profiles.active: dev

spring:
  cloud:
    config:
      server:
        bootstrap: true
        git:
          basedir: temp/config-repo/
          uri: https://our-glorious-repo/git
          username: myUser
      retry:
        max-attempts: 1
  application:
    name: myApp

You'll also see a flag:

spring.cloud.config.server.boostrap: true

This covers point 2: Enabling the application to configure itself from the remote repository. Without it it will only serve the configuration to other applications.

Honestly, lots of Spring Boot magic going on here. Just assume it works and don't question it any further. This really is all you need.

Poll the remote for changes

We were facing a number of issues here. All config files are located in a single repository. Refreshing all applications on a push is a no-go, so we need a way to know which files are updated. Secondly, Spring Cloud only supports a webhook on the application. This would work, but would require Azure to be able to reach all our application and for us to setup a webhook for each new application in DevOps, using a message broker to propagate events or coming up with something else.

After some back and forth, we decided on the following solution:
- We'll poll git ourselves to check for changes and check which files are updated. Based on changed files we will only reload the application if needed.

The Spring Config Server already uses git as a source for configuration. It uses a JGitRepository under the hood, which we can also @Autowire in our application. This will be the entry point to the correct git repository. By using this, we can use the existing configuration and prevent the need for separate configuration just for the polling.

We'll be using Spring Boot Scheduling Support to create a cron scheduler that will do our polling. Configuration for that can be in the remote configuration so we can update it on the fly if needed.

In our polling, we'll be matching our spring.application.name against the list of changed files. When the file matching our applicationName is in the changelist, we perform the restart. This uses the same simple applicationName based matching as the ConfigServer does in order to keep as close to Spring as possible.

Putting it all together, we get the following 'AutoRefreshConfig' class:

@Slf4j
@Configuration
public class AutoRefreshConfig {

    @Value("${spring.cloud.config.server.git.password}")
    public String gitPassword;

    @Value("${spring.cloud.config.server.git.username}")
    public String gitUserName;

    @Value("${spring.application.name}")
    public String applicationName;

    public AutoRefreshConfig(RestartEndpoint restartEndpoint, JGitEnvironmentRepository gitEnvironmentRepository, TaskScheduler scheduler, CamelConfigClientExampleConfiguration config) {
        scheduler.schedule(scheduledRefresh(gitEnvironmentRepository, restartEndpoint), new CronTrigger(config.getRefreshCron()));
    }

    public Runnable scheduledRefresh(JGitEnvironmentRepository gitEnvironmentRepository, RestartEndpoint restartEndpoint) {
        return () -> {
            try (var git = gitEnvironmentRepository.getGitFactory().getGitByOpen(gitEnvironmentRepository.getBasedir())) {

                var repository = git.getRepository();
                var reader = repository.newObjectReader();
                // 1
                var fetchCommand = git.fetch();
                fetchCommand.setCredentialsProvider(new UsernamePasswordCredentialsProvider(gitUserName, gitPassword));
                fetchCommand.call();
                // 2
                var localTreeParser = new CanonicalTreeParser();
                localTreeParser.reset(reader, repository.resolve("refs/heads/" + repository.getBranch() + "^{tree}"));
                var remoteTreeParser = new CanonicalTreeParser();
                remoteTreeParser.reset(reader, repository.resolve("FETCH_HEAD^{tree}"));
                // 3
                var diffs = git.diff().setShowNameAndStatusOnly(true)
                        .setNewTree(remoteTreeParser)
                        .setOldTree(localTreeParser)
                        .call();

                if (diffs.stream().anyMatch(diff -> diff.getOldPath().contains(applicationName) || diff.getNewPath().contains(applicationName))) {
                    log.warn("Configuration out of date. Reloading context from remote");
                    restartEndpoint.restart();
                } else {
                    log.info("Local config up to date. Skipping reload.");
                }
            } catch (Exception e) {
                log.error("Unable to refresh properties: " + e.getMessage());
            }
        };
    }

}

Personally, I have some trouble wrapping my head around JGit. I'm not a Git guru and mostly know a guy, who knows a guy, that knows what I need to type in the terminal.

So for the sake of clarity, a quick rundown of what is happening  here:
1. We first perform a git fetch. The jGitRepository does not keep track of credentials, so we need to set those again here, using our Spring Config Server credentials we're already using.
2. Then we resolve both the local head ("refs/heads/{branch}/^{tree}) and remote head from our fetch (FETCH_HEAD^{tree}) and create treeParsers for both. We use these parsers to walk the file trees and find changes in our git.diff() call.
3. The result is a simple List<DiffEntry>, that contains file names and change status (like new, deleted or changed). We'll stream this list and look for anyMatch on fileName with our application name. When found, we restart. If not, we just log and wait for the next cron-trigger.

Last but not least, the call to restartEndpoint.restart(). I've been hunting on the interwebs for far to long for a way to restart Spring Boot without rebooting the service from Windows. It seems like Spring Cloud already had something on the shelves for just this: the RestartEndpoint. This will create a new thread with a new application context, start that one and stop the current context. This will keep the JVM running and will nicely create a new Spring Application Context.

The newly created context will use the embedded Config Server to get the latest configuration from Git and create a new cron-scheduler to poll for future changes. That closes the loop. We're running on the latest configuration and polling for any new changes.

CLOSING THOUGHTS

I'm quite aware that things could have been worked out a bit easier. We could've optimized our pipeline and trigger a refresh or restart from there. We could've ditched Windows and go full-on containers with Kubernetes or OpenShift, and saved ourselves the entire hassle (read: replace the old hassle with new and shiny hassle). We could''ve placed our config files in a separate file share and refresh on timestamp. Lots of options, but we had to work with the tools we got.

The overhead of an embedded ConfigServer is minimal. We're not running hundreds upon hundreds of services, so the load on Azure DevOps will be low anyways. Yes, the full restart of the application will take more time, but stability and easy of updating is more important than high performance and up time.

The full Config Server setup with Spring Cloud Bus over Kafka is a great tool to have. If you can split up the config files over their own repositories and label them for environments (Dev, Prod, whatever), it can really speed up your deployments in case of a simple configuration change. It's a shame that Camel doesn't fit well out-of-the-box and requires some slightly hacky workarounds to work with the reload.

In a landscape with only a few applications, the full Config Server and Cloud Bus setup is overkill. You'll be looking at adding a Kafka Broker and Config Server, and maybe a service registry, all for the sake of being able to do a hot-refresh. Embedding the Server in this way provides the best of both worlds: almost-runtime refresh and no need to maintain more infrastructure.