Practical Hashicorp Nomad and Consul — Part 4: Consul KV Store

Co-authored by Navin Nair, Jawahar Suresh Babu S, Manikandan S and ARUN KUMAR

This is in continuation of previous three articles Part1, Part2 and Part3, If you came here directly, I will recommend to give a read for the previous 3 articles as well.

Practical Hashicorp Nomad and Consul — A little more than Hello World (Part 1)

Practical Hashicorp Nomad and Consul — CI/CD Pipeline to deploy the API and WebApp Part 2

Practical Hashicorp Nomad and Consul - Monitoring & Autoscaling using Prometheus & Grafana (Part 3)

But incase you completed the previous one and came here, I will give a brief about last three article and what we will do as part of this article, Let’s recall!! 😎😎

• In the first part, we discussed the steps to set up a 3-Tier application and deploying it in a Nomad/Consul cluster as part of a service mesh.
• In the second part, we discussed about setting up the continuous deployment(CD) pipeline to get the code from the application, build it and deploy it to nomad cluster.
• In the third part, we discussed about how we can add observability to our Nomad cluster & service mesh and also setup auto-scaling based on load.
• In this article, we will be discussing 2 ways to fetch configuration data from consul key-value store instead of hard coding it in the application.

So, let’s get started!!

For keeping it simple, We will just remove port, context-path and greeting configurations from application.properties file and try to fetch these information from consul key value store.

1.) Using spring cloud dependency in the application

Step1: This is the common step for both the ways, where we need to create key-value pairs in consul. You can create key value pair in consul using consul UI, which is running on port 8500.

• Click key/value option in the left navigation menu, you should be able to see below screen:

• Click on Create button present on the top right in the screen.
• Now add the key value pair, and click on Save button. Please add the key in the same format config/<application_name>/<key_name>: <value>, just like how we did in the below screen.

Thats it!! you should be able to create your key value pair using consul UI.

There are lot of ways to create key-value pairs in consul kv store which can be checked here.

Optional: If you have too many key value that needs to be created then you can create key value using json file. Below are the steps for the same.

Step1: This is the json file which have key-value in the format of config/<application_name>/<key_name>: <value>

[
  {
    "key": "config/consul_kv_demo/server.port",
    "value": "OTk2Ng=="
  },
  {
    "key": "config/consul_kv_demo/server.servlet.context-path",
    "value": "L3BldGNsaW5pY2FwaS8="
  },
  {
    "key": "config/consul_kv_demo/greeting",
    "value": "IkphY2tBbmRKb2UxMjMi"
  }
]

After creating this, run the below command to create consul kv pairs

consul kv import @<path-to-the-json-file>/properties.json

In our case, the command looks like this:

consul kv import @consul-kv-store/secrets/properties.json

where consul-kv-store/secrets/properties.json is the path of the file.

You should be able to see the below key-value pairs in your consul

Note:

• The value present in the file needs to base64 encrypted value.
• This key-value format is applicable only for this spring-cloud way of fetching config from consul kv store.
• Your application_name need to be same what you will give in the application.properties file (spring.application.name=consul_kv_demo)

Step 2: Add the spring-cloud-dependencies and spring-cloud-starter-consul-config dependency in the pom.xml file.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>2020.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-consul-config</artifactId>
</dependency>

Note: Few versions of spring boot is not compatible with spring-cloud-dependencies, we used spring boot version 2.4.2 and spring-cloud 2020.0.0, which is compatible to each other. You can check the compatibility here. Believe me, it took lot of time to figure out this 🤯🤯

Step 3: Add the below configuration in application.properties file

spring.application.name=<application_name>
spring.config.import=consul:<consul_host>:<consul_port>
spring.cloud.consul.discovery.health-check-interval=<any_time_to_check_health>
spring.cloud.consul.discovery.instance-id=<any_instance_id>:<consul_port>

In our case, application_name is consul_kv_demo consul_host will 172.16.1.101 and consul_port is 8500(default). So our configuration look like this:

spring.application.name=consul_kv_demo
spring.config.import=consul:172.16.1.101:8500
spring.cloud.consul.discovery.health-check-interval=15s
spring.cloud.consul.discovery.instance-id=random:8500

Step 4: Add @EnableDiscoveryClient annotation in the main java file., PetClinicApplication.java (in our case).

Step 5: Add @RefreshScope annotation in all the java file where @Value annotation is used so that on any key-value change the application refresh the value automatically.

Step 6: Now, we need to package the JAR, which can be done by 2 ways

a.) We can manually package the JAR file using the below command and use github to host the JAR file then directly use the path(raw) of the hosted jar file in the source parameter of artifact stanza in the nomad job file.

In case, you want to check the spring boot changes, you can check here.

mvn package

Note: Incase, you want to check the entire job file for setting up manually packaged jar, you can check here.

b.) Otherwise, we can make the changes in java source code as shown in example and use the pipeline shown in part 2 to build the jar and deploy it.

Thats it!!. This time your application should run using values you added in consul kv store. If yes, Congratulations 🎉🎉, our application is able to communicate and fetch config data from consul kv store.

You can try changing any value in the consul kv store and hit the api to reflect the changed value.This way you don’t need to package your JAR file or build it every time on any configuration changes because consul kv will detect the changes and provide the new value.

Though this way, we achieve the separation of config from the application code but this is more spring way of doing thing, where we need to use spring cloud dependencies to get it work and it involves code changes also. Let’s see the other way which is not very coupled with spring way of doing things and less code changes in spring boot project.

2. ) Using Envconsul Binary

In this way, we will use envconsul binary to fetch data from consul kv store.

Step 1: First we need to create key value pairs in consul kv store, similar to how we did it previously in Step1.

Note: When we are using envconsul, key value pair can be in any format not necessarily to be in the below format(which was the case when using spring cloud dependencies):

config/<application_name>/<key_name>: <value>

But for simplicity, we will use the same format i.e., our key value pair look like this:

config/consul_kv_demo/greeting: "Hello consul"
config/consul_kv_demo/server.port: 9966
config/consul_kv_demo/server.servlet.context-path: /petclinicapi/

Step 2: Download a pre-compiled, released version from the envconsul releases page.

Step 3: Extract the binary file and move the binary to your PATH.

In the Github example, we have implemented the step2 and step3 on vagrant machine boot up, but if you are not using vagrant you can do it manually in your local.

Step 4: This step is just to check whether envconsul set up is done and its listening to consul kv store properly or not. For this, run the below command which should give you all the key value pairs for a directory(config2).

envconsul -prefix <directory-name> env

In our case, the command looks like:

envconsul -prefix config/consul_kv_demo env

Step 5: This step is optional incase you need to execute the JAR file manually using envconsul(where nomad jobs is not in picture) so it fetch the data from kv store. For this, we need to run the below command.

envconsul --prefix <directory_name> java -jar <jar-file-path>

In our case, the command looks like:

envconsul --prefix config/consul_kv_demo java -jar envconsul_jar/spring-petclinic-rest-2.4.2.jar

This way it will pick the config properties from consul kv store(ofcourse your consul needs to be running 😉) when you want to run the application manually(without nomad job).

Step 6: Now the main step, we need to run the JAR file by using nomad job file, our task should look like this:

task "petclinic-api" {
  driver = "exec"
  config {
    command = "envconsul"
    args    = ["--config", "local/envconsul.hcl", "java", "-jar", "/tmp/spring-petclinic-rest-2.4.2.jar", "-Xms256M", "-Xmx256M", "--nogui;"]
  }
  artifact {
    source = "https://github.com/sankita15/spring-petclinic-rest/raw/master/envconsul_jar/spring-petclinic-rest-2.4.2.jar"
    destination = "/tmp"
  }
  template {
    data = <<EOH
      consul {
        address = "172.16.1.101:8500"
      }
      prefix {
        path = "config/consul_kv_demo"
      }
    EOH
    destination = "local/envconsul.hcl"
  }
  resources {
    cpu    = 500
    memory = 300
  }
  env {
    API_PORT = "9966"
  }
}

Explanation:

driver = "exec"

• We are using exec instead of Java driver here, as we need to run the envconsul as part of nomad job and then envconsul will run the JAVA application (name: petclinic-api).

template {
    data = <<EOH
      consul {
        address = "172.16.1.101:8500"
      }
      prefix {
        path = "config/consul_kv_demo"
      }
    EOH
    destination = "local/envconsul.hcl"
  }

• In the template stanza, we use data parameter to write all the configuration inline which we want to apply for envconsul. e.g.,

1. We gave consul-address because we want envconsul to check consul on 172.16.1.101:8500 because our consul is running here.
2. We gave prefix-path as config/consul_kv_demo because we want envconsul to check the JAR file configuration in the config/consul_kv_demo directory.

Note: If you have lot of such configuration for envconsul, then instead of using data parameter, you can use source parameter also.

• All the envconsul’s configuration that we got from data will be stored in a file envconsul.hcl and that’s what our destination tells us.

config {
    command = "envconsul"
    args    = ["--config", "local/envconsul.hcl", "java", "-jar", "/tmp/spring-petclinic-rest-2.4.2.jar", "-Xms256M", "-Xmx256M", "--nogui;"]
  }

• In the config stanza, we used “envconsul” for command parameter as we want to run envconsul, straight and simple 😁.
• For the args parameter, we passed the following values:

1. - -config: “local/envconsul.hcl”

This is to tell the job to pick up envconsul’s configuration from this location. If you remember, we created a config file using data parameter in the template stanza. we are using the same file here.

2. And then we gave the command to run the JAR file.

In case you want to check the entire Job file, you can check here.

Note:

• This approach doesn’t need to do any changes in spring boot application.
• In the first approach, if you change any value in consul key value pair, then it will take few seconds to reflect the new changed value in the application but your application will always be available without any downtime.
• In second approach, if you change any value in the consul key value pair, then envconsul will restart the application, which make our application unavailable for a short duration(till the application gets boot up again). We can disable this feature and ask envconsul not to listen for any change in the application by passing — once flag.
• envconsul also provide one more config as a workaround for the above downtime issue i.e., splay which can be set any time.e.g., splay = “30s”, then all the instances of your application will not reboot simultaneously, instead envconsul will reboot all the instances one by one by taking any random amount of time between 0 to 30s. Generally, high splay and multiple instance can reduce the downtime experience a lot but it still random so no downtime is not guaranteed.
• There are lot to configure for envconsul which you can check here.

That’s it folks. We discussed the 2 ways by which we can separate the configuration from our application source code. I tried to make it as simple as possible, hope it helps you all. 🙏🙏