Deploy Elastic Stack to Kubernetes

09-03-2019



Imgur



This post will discuss how to test and deploy an Elastic Stack application to a Kubernetes cluster on Azure cloud service. All codes shown in this post can be found in this GitHub repo.



Application Components

The purpose of this application is to simulate data being published from IoT devices and visualised those data using Kibana.

The application is made up of Elastic Stack which consists of Logstash, Elastic Search, and Kibana. RabbitMQ is use as a message broker from NodeRED to Logstash. NodeRED is use to publish message to RabbitMQ simulating IoT device publishing data. Logstash will consume the messages published and output it to Elastic Search and the data will be visualised using Kibana.


Elastic Stack

Elastic Stack is a stack of open source applications that can be use for collecting, storing, analysing, and visualising data. The stack is also known as the ELK stack (ElasticSearch Logstash Kibana). We're using the stack in this application for logging the data, storing it in Elastic Search, and visualising it in Kibana.


RabbitMQ

RabbitMQ is an open source message broker. Message brokers are use for sending messages between application and services, an example of its use case is where one process want to share data with another process. Using RabbitMQ the 1st process can publish the data to RabbitMQ and the 2nd process can consume the data from RabbitMQ.

We're using RabbitMQ in this application to transfer data from a publisher (NodeRED) to a consumer (Logstash). This is so that we don't have to directly send messages from NodeRED to Logstash. Also RabbitMQ can be implemented as a cluster providing the application stack scalibility.


NodeRED

NodeRED is a visual editor that allow user to create program flows. It enables users to easility create program flows use for load testing services, and simulating data input.

We're using it here to simulate the data being outputted from IoT devices. These data could be temperature readings using sensors, or motion detector data.



Development (Testing) Environment

One methodology to test a application stack consisting of multiple components is to dockerised each components and connect them together. Connecting and making each components work together can be a hassle, therefore a container orchestration tool is use. Come, Docker Compose.

"Compose is a tool for defining and running multi-container Docker applications."
~ Docker Compose documentation

Docker Compose can be use to easility deploy a stack of Docker containers (after reading some documentations, and some practice). The application stack can be set up and deploy quickly which makes it attractive for setting up a development environment for testing the stack.

The YAML file below show the configuration for a Docker Compose network:

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
# docker-compose.yaml

version: '3'
networks:
  # The docker network all the docker containers are in.
  # Network is use for ensuring that all containers can find and
  # connect to each other.
  elasticstack-rabbitmq-network:

# List all the docker containers that are going to be running as services.
services:

  rabbitmq:
    container_name: rabbitmq
    hostname: rabbitmq
    image: rabbitmq-mqtt:v1   # We'll create this custom image below.
    environment: 
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
    ports:
      - 5672:5672     # AMQP port. Rich messaging for server.
      - 15672:15672   # Management port, if plugin is enabled.
      - 1883:1883     # MQTT port. Lightweight messaging for IoT.
    networks: 
      - elasticstack-rabbitmq-network

  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
    environment: 
      - discovery.type=single-node
    ports:
      - 9300:9300    # REST
      - 9200:9200    # Node Communication
    networks: 
      - elasticstack-rabbitmq-network

  logstash:
    container_name: logstash
    image: docker.elastic.co/logstash/logstash:6.6.1
    volumes: 
      - ./logstash-config:/usr/share/logstash/config-rabbitmq
    command: logstash -f ./config-rabbitmq/logstash-rabbitmq.conf
    ports:
      - 9600:9600
    networks: 
      - elasticstack-rabbitmq-network

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:6.6.1
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    ports:
      - 5601:5601
    networks: 
      - elasticstack-rabbitmq-network

  nodered:
    container_name: nodered
    image: nodered/node-red-docker:v8
    ports: 
      - 1880:1880
    networks: 
      - elasticstack-rabbitmq-network

You might have noticed that the RabbitMQ docker image is a custom image. We need to create a custom image because it need to use a MQTT RabbitMQ plugin, and we need to install the plugin on start up. You might also be asking, why not just put the install command inside the docker-compose.yaml file above like this:

0
1
2
3
4
5
6
7
8
# extracts from docker-compose.yaml
environment: 
  - RABBITMQ_DEFAULT_USER=user
  - RABBITMQ_DEFAULT_PASS=password
command: "rabbitmq-plugin enable rabbitmq_mqtt" # <---
ports:
  - 5672:5672     # AMQP port. Rich messaging for server.
  - 15672:15672   # Management port, if plugin is enabled.
  - 1883:1883     # MQTT port. Lightweight messaging for IoT.

While creating this project I also though of the same thing, but for some unknown reason RabbitMQ kills itself (it's docker service) after installing the plugin. So to go around this problem, I create a Dockerfile extending the official RabbitMQ docker image and set the command there instead. The file is shown below:

0
1
2
3
# Dockerfile.RabbitMQ

FROM rabbitmq:3.7.12
RUN rabbitmq-plugins enable rabbitmq_mqtt

To create the custom RabbitMQ docker image, run the command below in your terminal:

0
$ docker build -t rabbitmq-mqtt:v1 -f Dockerfile.RabbitMQ .

The above command will create a docker image called rabbitmq-mqtt:v1. We tagged the v1 at the end to stand for version 1. This allow us to version our docker images, where when change happended we can create a new docker image with incrementing version number.

Now, the docker services can be run using docker compose. To run the services execute the command below:

0
$ docker-compose -f docker-compose.yaml up -d
  • We use to f tag to specify the docker-compose file,
  • up argument to bring up the docker services,
  • and d tag to detach from the docker service log.

After docker compose brought up the services successfully, we've got a:

  • RabbitMQ service listening on port 5672, 15672, and 1883,
  • LogStash service listening on port 9600,
  • ElasticSeach service listening on port 9200, and 9300,
  • Kibana service listening on port 5601,
  • and NodeRED service listening on port 1880.

Therefore, Kibana web browser UI can be access at localhost:5601 (assuming that you're running the docker compose on your local machine), and NodeRED web browser UI can be access at localhost:1880.


Testing NodeRED

To test that the application stack is working as intended, in this scenario we're going to use NodeRED to publish a message to RabbitMQ let the ElasticStack consume the message and use it.

Go to the NodeRED web browser UI at http://localhost:1880, and click on the "hamburger" menu on the top right corner, then select Import, and then Clipboard.

Imgur

This will pop up a window with a blank entry field in the middle.

Imgur

Paste the follow code below into it and click the Import button on the bottom right hand corner of the pop up window:

0
# nodered-flow-publisher.json
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
[{
  "id": "29c39c05.b82344",
  "type": "tab",
  "label": "RabbitMQ Publisher Docker Compose",
  "disabled": false,
  "info": ""
}, {
  "id": "280b20c0.6de3",
  "type": "inject",
  "z": "29c39c05.b82344",
  "name": "",
  "topic": "mqtt_topic",
  "payload": "{\"data\": \"hello world\"}",
  "payloadType": "json",
  "repeat": "",
  "crontab": "",
  "once": false,
  "onceDelay": 0.1,
  "x": 200,
  "y": 120,
  "wires": [
    ["d12e4b7e.e609b8"]
  ]
}, {
  "id": "d12e4b7e.e609b8",
  "type": "mqtt out",
  "z": "29c39c05.b82344",
  "name": "",
  "topic": "",
  "qos": "",
  "retain": "",
  "broker": "1b14ebe2.5d2de4",
  "x": 430,
  "y": 120,
  "wires": []
}, {
  "id": "1b14ebe2.5d2de4",
  "type": "mqtt-broker",
  "z": "",
  "name": "",
  "broker": "rabbitmq",
  "port": "1883",
  "clientid": "",
  "usetls": false,
  "compatmode": true,
  "keepalive": "60",
  "cleansession": true,
  "birthTopic": "",
  "birthQos": "0",
  "birthPayload": "",
  "closeTopic": "",
  "closeQos": "0",
  "closePayload": "",
  "willTopic": "",
  "willQos": "0",
  "willPayload": ""
}]

The code you imported is a NodeRED flow. It's a configuration of a program flow. This flow will allow you to publish a message using MQTT protocol to the RabbitMQ docker service. Before you start publishing messages, you have one more thing to config: the RabbitMQ user and password.

You'll see these 2 nodes (each block in the flow is call a node), double click on the mqtt node to bring up the node's config window.

Imgur

A config window will slide from the right side of your screen looking like the image below. Notice that we're not using localhost, but rather we're using the docker service name of the RabbitMQ stated in docker-compose.yaml. This is because we've configured all the docker services to be in the same docker compose network and therefore they can connect with each other using their docker service container name. This concept will translate well when the application is deployed to Kubernetes (more on this later).

Imgur

Click on the pencil icon on the right side of the Server input field. Once clicked, it will bring up another window looking like the image below.

Imgur

Click on the Security tab.

Imgur

Here, we will input the username and password we've set in the docker-compose.yaml file for the RabbitMQ docker service. The username is user, and the password is password.

Imgur

Once, you've entered the username and password click the red Update button, and then click the red Done button. There's one more thing we need to do to commit the changes. Click on the red Deploy button on the top right hand corner. This will deploy any changes you've previously made, and also in this case, it'll also connect our MQTT publisher to RabbitMQ. After deploying you should see the green connected label below our MQTT node.

Imgur

You can now publish messages to RabbitMQ and LogStash will consume it and store it to ElasticSearch, where Kibana can query for the data.

We've set the message to be published as the JSON format below:

0
1
2
{
  "data": "hello world"
}

If you wish to change the data being published you can do so by going into the config of the Inject node (the blue coloured one).


Testing Kibana

Now, go to http://localhost:5601. This is the Kibana web browser UI. Here we're going to query for the data we've published from NodeRED.

Imgur

Click on Dev Tools.

Imgur

Here is a interface for developer to query for data from ElasticSearch. We're going to use it to query our data published from NodeRED. Click on the green play button to execute the query.

You'll see some data being output on the right side of the sceen. If you've already published some data from NodeRED you'll see it here.

Imgue

If you haven't published any data, we can do so by clicking on the square tab on the left of the inject node.

Imgur

When querying ElasticSearch using the Kibana Dev Tools, you should now see the NodeRED published data.

Imgur

This proves that the application stack is working as intended locally using docker compose. The next step we're going to take is to deploy the application stack onto a Kubernetes cluster on Microsoft Azure cloud service.



Production Environment

Before we can get going with the deployment, we've got to set up the Azure side of things first.

What we have to do:

  1. Create a resource group
  2. Create a container registry for storing docker images
  3. Tag our docker images and push it to the Azure container registry
  4. Create a service principal so that our Kubernetes cluster can pull the docker images from the container registry
  5. Assign the role permission for the service principal to allow a image pulling
  6. Create the Kubernetes cluster
  7. Get its credential for our local kubectl config, so that we can access the cluster
  8. Apply the Kubernetes deployment to the cluster

Run the following command to create a resource group with the name elasticStackRabbitMQ, and the location of south east asia.

0
$ az group create --name elasticStackRabbitMQ --location southeastasia

Now, we're going to create an Azure Container Registry (ACR) in our recently created resource group and a SKU of Basic size. Note that choosing a higher SKU will provide more performance and scale. Also, note that the name of the container registry must be unique all thoughout Azure.

0
$ az acr create --resource-group elasticStackRabbitMQ --name <uniqueContainerRegistryName> --sku Basic

Before we can use the registry we must login to it. Run the command below to get the ACR login server name.

0
$ az acr list --resource-group elasticStackRabbitMQ --query "[].{acrLoginServer:loginServer}" --output table

Then login using the server name as shown below.

0
$ az acr login --name <acrName>

Next, we've got to tag the our docker images before we push them up to the registry.

0
$ docker tag rabbitmq-mqtt:v1 <acrName>/rabbitmq-mqtt:v1

Now, we can push the image to the registry.

0
$ docker push <acrName>/rabbitmq-mqtt:v1

Currently in our development environment, our LogStash docker container depends on a local config file. When we deploy the LogStash docker image up to Kubernetes, it won't be able to access the local config file anymore because it's not there. So, what we've got to do is to create our own custom LogStash docker image by extending the base LogStash image and adding the config file as part of the image.

0
1
2
# Dockerfile.Logstash
FROM docker.elastic.co/logstash/logstash:6.6.1
ADD /logstash-config /usr/share/logstash/config-rabbitmq

Run the following command to use our Dockerfile to create a custom LogStash docker image.

0
$ docker build -t logstash-rabbitmq:v1 -f Dockerfile.Logstash .

As with the RabbitMQ image, we've got to tag this image and push it as well.

0
1
$ docker tag logstash-rabbitmq:v1 <acrName>/logstash-rabbitmq:v1
$ docker push <acrName>/logstash-rabbitmq:v1

To check that the docker images have been pushed to our registry, run the command below.

0
$ az acr repository list --name <containerRegistryName> --output table

Now, we're going to create a service principal. This command will output a JSON. Take note of the appId and password, we'll be using those later.

0
$ az ad sp create-for-rbac --skip-assignment

To assign the role for the service principal to be able to pull images from the registry we need the the ACR id. Run the following command to get the id.

0
$ az acr show --resource-group elasticStackRabbitMQ --name <containerRegistryName> --query "id" --output tsv

Then run the following command to assign the image pull role. In this command we'll be using our appId from before when we created the service principal. If you didn't note it down, you can create another service principal.

0
$ az role assignment create --assignee <appId> --scope <acrId> --role acrpull

Now to the fun bit, where we actually create the Kubernetes cluster. Here we'll create a cluster name AKSCluster with 2 nodes. You'll again use your service principal appId here and also your service principal password. The --no-wait tag is to tell the CLI to just execute the command and return the terminal process to us (to not wait for the whole cluster creation process to finished, which can take a long time, and then return the terminal to us).

0
$ az aks create --resource-group elasticStackRabbitMQ --name AKSCluster --node-count 2 --service-principal <appId> --client-secret <password> --generate-ssh-keys --no-wait

Now to get our kubectl CLI to connect with our Azure Kubernetes cluster we must provide it with the credentials to the cluster. Run the following command to give kubectl our cluser credentials. The --overwrite-existing tag is needed if you've created the cluster with the same name before.

0
$ az aks get-credentials --resource-group elasticStackRabbitMQ --name AKSCluster --overwrite-existing

Congratulation, you've just set up your Kubernetes cluster on Azure. There's no deployments in it yet though, it's just an empty cluster with 2 nodes at the moment. Below is the Kubernetes yaml deployment file we're going to use to deploy our application stack. Note that you're going to have to change the <acrName> value to your own <acrName>

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
# elastic-stack-rabbitmq.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
      - name: rabbitmq
        image: <acrName>/rabbitmq-mqtt:v1
        resources:
        ports:
        - containerPort: 5672     # AMQP port. Rich messaging for server.
          name: rb-amqp
        - containerPort: 15672    # Management port, if plugin is enabled.
          name: rb-mngt
        - containerPort: 1883     # MQTT port. Lightweight messaging for IoT.
          name: rb-mqtt
        env:
          - name: RABBITMQ_DEFAULT_USER
            value: "user"
          - name: RABBITMQ_DEFAULT_PASS
            value: "password"
---        
apiVersion: v1
kind: Service
metadata:
  name: rb-amqp
spec:
  ports:
  - port: 5672
  selector:
    app: rabbitmq
---        
apiVersion: v1
kind: Service
metadata:
  name: rb-mngt
spec:
  ports:
  - port: 15672
  selector:
    app: rabbitmq
---
apiVersion: v1
kind: Service
metadata:
  name: rb-mqtt
spec:
  ports:
  - port: 1883
  selector:
    app: rabbitmq
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: logstash
spec:
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: <acrName>/logstash-rabbitmq:v1
        resources:
        command: ["logstash", "-f", "./config-rabbitmq/logstash-rabbitmq-k8s.conf"]
        ports:
        - containerPort: 9600
          name: logstash
        env:
          - name: XPACK_MONITORING_ELASTICSEARCH_URL
            value: "http://elsrch-nd-comm:9200"
---
apiVersion: v1
kind: Service
metadata:
  name: logstash
spec:
  ports:
  - port: 9600
  selector:
    app: logstash
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
spec:
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
        resources:
        ports:
        - containerPort: 9300       # REST 
          name: elsrch-rest
        - containerPort: 9200       # Node Communication
          name: elsrch-nd-comm
        env:
          - name: discovery.type
            value: "single-node"
---
apiVersion: v1
kind: Service
metadata:
  name: elsrch-rest
spec:
  ports:
  - port: 9300
  selector:
    app: elasticsearch
---
apiVersion: v1
kind: Service
metadata:
  name: elsrch-nd-comm
spec:
  ports:
  - port: 9200
  selector:
    app: elasticsearch
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kibana
spec:
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:6.6.1
        resources:
        ports:
        - containerPort: 5601
          name: kibana
        env:
        - name: ELASTICSEARCH_URL
          value: "http://elsrch-nd-comm:9200"
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 5601
  selector:
    app: kibana
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nodered
spec:
  selector:
    matchLabels:
      app: nodered
  template:
    metadata:
      labels:
        app: nodered
    spec:
      containers:
      - name: nodered
        image: nodered/node-red-docker:v8
        resources:
        ports:
        - containerPort: 1880
          name: nodered
---
apiVersion: v1
kind: Service
metadata:
  name: nodered
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 1880
  selector:
    app: nodered

What we're doing here with the yaml file is to tell Kubernetes to create a deployment for each of our docker services. For each docker service it listens on 1 or more ports, we create a Kubernetes service for each of those port. Note that the name of the service is the name that will be use for connecting to the docker service (the same concept as in docker compose). Each service must also have a selector with an app key with the value being the name of the deployment for which the service port comes from.

Additional note, we're mapping port 80 for both the NodeRED and Kibana service so that we can directly use the IP address of the service to access the UI. Also, note that the type of service for NodeRED and Kibana is a LoadBalancer which provides us with an external IP to connect to. These IP and with the use of port 80 allow us to also redirect a domain name to it, for example, mysupercoolcluster.com (note, I'm using this domain name as an example, it may or may not be a real website).

To deploy the application stack to the cluster run the following command. This will apply our Kubernetes config to the cluster.

0
$ kubectl apply -f elastic-stack-rabbitmq.yaml

To see which IP address you can access your NodeRED and Kibana UI run the following command. It will display a list of services you currently have in your cluster. Look to the EXTERNAL-IP column for your IP addresses.

0
$ kubectl get services

Now, since we've assign service names that are different from the container name, we're going to have to change the config of the MQTT node in NodeRED. We've only actually needed to change the Server in the MQTT node config from rabbitmq to rb-mqtt, but below is the new node config anyway.

0
# nodered-flow-publisher-k8s.json
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
[{
  "id": "89242c63.c7e82",
  "type": "tab",
  "label": "RabbitMQ Publisher Kubernetes",
  "disabled": false,
  "info": ""
}, {
  "id": "2a41ba10.b32956",
  "type": "inject",
  "z": "89242c63.c7e82",
  "name": "",
  "topic": "mqtt_topic",
  "payload": "{\"data\": \"hello world\"}",
  "payloadType": "json",
  "repeat": "",
  "crontab": "",
  "once": false,
  "onceDelay": 0.1,
  "x": 200,
  "y": 120,
  "wires": [
    ["7dfea135.aa565"]
  ]
}, {
  "id": "7dfea135.aa565",
  "type": "mqtt out",
  "z": "89242c63.c7e82",
  "name": "",
  "topic": "",
  "qos": "",
  "retain": "",
  "broker": "42939a80.1fadd4",
  "x": 430,
  "y": 120,
  "wires": []
}, {
  "id": "42939a80.1fadd4",
  "type": "mqtt-broker",
  "z": "",
  "name": "",
  "broker": "rb-mqtt",
  "port": "1883",
  "clientid": "",
  "usetls": false,
  "compatmode": true,
  "keepalive": "60",
  "cleansession": true,
  "birthTopic": "",
  "birthQos": "0",
  "birthPayload": "",
  "closeTopic": "",
  "closeQos": "0",
  "closePayload": "",
  "willTopic": "",
  "willQos": "0",
  "willPayload": ""
}]

After you're done with the application stack and want to remove it from the cluster, you can run the following below.

0
kubectl delete -f elastic-stack-rabbitmq.yaml

You can also run the command below to delete the Azure Kubernetes Cluster.

0
$ az aks delete --name AKSCluster --resource-group elasticStackRabbitMQ --no-wait

And also to delete the whole Azure Resource Group.

0
$ az group delete --name elasticStackRabbitMQ --yes --no-wait


Conclusion

In conclusion, I've mainly just read the Azure Kubernetes documentation, and then try to implement it to my own application stack. This post is the result of my experience with using Azure and Kubernetes to deploy the stack. What I've found useful for undertaking this task of using new tools to deploy my stack is to start with a small stack and testing your way up and scaling it. What I meant is to have a stack consisting of 1 container, then if everything went well use 2 containers and try to connect those 2 together.

In all, I hope that you've found this post useful.