Getting Started with Logging Using EFK on Kubernetes


After creating a and deploying the apps, the question that rises is: How can we handle the logs?

One option to view the logs is using the command: kubectl logs POD_NAME. That is useful for debugging. But there is a better option suited for production systems. That is using EFK. The rest of the article will introduce EFK, install it on Kubernetes and configure it to view the logs.

What is EFK

EFK is a suite of tools combining Elasticsearch, Fluentd and Kibana to manage logs. Fluentd will collect the logs and send it to Elasticsearch. This latter will receive the logs and save it on its database. Kibana will fetch the logs from Elasticsearch and display it on a nice web app. All three components are available as binaries or as Docker containers.

Info: ELK is an alternative to EFK replacing Fluentd with Logstash.

For more details on the EFK architecture, follow this video:


Installing EFK on Kubernetes

Because EFK components are available as docker , it is easy to install it on k8s. For that, we'll need the following:

  • Kubernetes (Minikube or AKS…)
  • Kubectl CLI
  • Helm CLI

1. Installing Elasticsearch using Helm

We'll start with deploying Elasticsearch into Kubernetes using the Helm chart available here on Github. The chart will create all the required objects:

  • Pods to run the master and client and manage data .
  • Services to expose Elasticsearch client to Fluentd.
  • Persistent Volumes to store data (logs).
$ helm install elasticsearch stable/elasticsearch 

Let's wait for a few (10-15) minutes to create all the required components. After that, we can check the created pods using the command:

$ kubectl get pods


Then we can check for the created services with the command:

$ kubectl get services


2. Installing Fluentd as DaemonSet

Fluentd should be installed on each node on the Kubernetes . To achieve that, we use the DaemonSet. Fluentd development team provided a simple configuration file available here:

apiVersion: apps/v1
kind: DaemonSet
  name: fluentd
  # namespace: kube-system
    k8s-app: fluentd-logging
    version: v1
      k8s-app: fluentd-logging
      version: v1
        k8s-app: fluentd-logging
        version: v1
      - key:
        effect: NoSchedule
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
            value: "elasticsearch-client"
            value: "9200"
            value: "http"
          # Option to configure elasticsearch plugin with self signed certs
          # ================================================================
            value: "false" # changed by me
          # Option to configure elasticsearch plugin with tls
          # ================================================================
            value: "TLSv1_2"
          # X-Pack Authentication
          # =====================
            value: "elastic"
            value: "changeme"
          # Authentication
          # ======================
          - name: LOGZIO_TOKEN
            value: "ThisIsASuperLongToken"
          - name: LOGZIO_LOGTYPE
            value: "kubernetes"
            memory: 200Mi
            cpu: 100m
            memory: 200Mi
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      - name: varlog
          path: /var/log
      - name: varlibdockercontainers
          path: /var/lib/docker/containers

Fluentd should be able to send the logs to Elasticsearch. It needs to know its service name, port number and the schema. These configurations will be passed to the Fluentd pods through environment variables. Thus, in the yaml file, we notice the following configurations:

            value: "elasticsearch-client"
            value: "9200"
            value: "http"

The value “elasticsearch-client” is the name of the Elasticsearch service that routes traffic into the client pod.

Note: A DaemonSet is a Kubernetes object used to deploy a Pod on each Node.

Note: Fluentd could be deployed also using a Helm chart available on this link

Now let's deploy Fluentd using the command:

$ kubectl apply -f .fluentd-daemonset-elasticsearch.yaml

We can verify the install by checking the 3 new pods, 3 because we have 3 nodes in the cluster:


3. Installing Kibana using Helm

The last component to install in EFK is Kibana. Kibana is available as a Helm chart that could be found here:

The chart will deploy a single Pod, a Service and a ConfigMap. The ConfigMap get its key values from the values.yaml file. This config will be loaded by the Kibana container running inside the Pod. This configuration is specific to Kibana, to get the Elasticsearch host or service name, for example. The default value for the Elasticsearch host is http://elasticsearch:9200. While in our example it should be http://elasticsearch-client:9200. We need to change that.

The Service that will route traffic to Kibana Pod is using type ClusterIP by default. As we want to access the dashboard easily, we'll override the type to use LoadBalancer. That will create a public IP address.

In Helm, we can override some of the config in values.yaml using another yaml file. We'll call it kibana-values.yaml. Lets create that file with the following content:

    ## Default Kibana configuration from kibana-docker. kibana "0"
    ## For kibana < 6.6, use elasticsearch.url instead
    elasticsearch.hosts: http://elasticsearch-client:9200
  type: LoadBalancer # ClusterIP

Now, we are ready to deploy Kibana using Helm with the overridden configuration:

$ helm install kibana stable/kibana -f kibana-values.yaml

In a few seconds, when the deployment is complete, we can check whether the created Pod is running using kubectl get pods command.


Then we check for the created service with LoadBalancer type.


From here, we can copy the external IP address ( here) and open it in a web browser. We should not forget to add the port number which is 443.


Then click on “Discover”. And there we'll find some logs !


4. Deploying and viewing application logs

In this section, we'll deploy a sample container that outputs log files in an infinite loop. Then we'll try to filter these logs in Kibana.

## counter.yaml
apiVersion: v1
kind: Pod
  name: counter
  - name: count
    image: busybox
    args: [/bin/sh, -c, 'i=0; while true; do echo "Demo log $i: $(date)"; i=$((i+1)); sleep 1; done']

Let's first deploy this sample Pod:

$ kubectl apply -f .counter.yaml

We make sure it is created successfully:


Now, if we switch to Kibana dashboard and refresh it, we'll be able to see logs collected from the counter Pod:


We can also filter the logs using queries like: kubernetes.pod_name: counter.

The content of this article is also available as a video on youtube on this link:


It was easy to get started with EFK stack on Kubernetes. From here we can create custom dashboards with nice graphs to be used by the developers.

Additional notes

Note: We installed the ELK stack in the default namespace for simplicity. But it is recommended to install it on either kube-system or a dedicated namespace.

Note: Elasticsearch have its own repo built to support v7, it is still in preview today

The sample are not supported under any Microsoft standard support program or service. The sample are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.


This article was originally published by Microsoft's Azure SQL Database Blog. You can find the original article here.