Deploying the OpenTelemetry Collector to Kubernetes with Helm

The OpenTelemetry Collector is a useful application to have in your stack. However, deploying it has always felt a little time consuming: working out how to host the config, building the deployments, etc. The good news is the OpenTelemetry team also produces Helm charts for the Collector, and I’ve started leveraging them. There are a few things to think about when using them though, so I thought I’d go through them here.

I’m going to assume that if you’ve gotten this far, you know what the OpenTelemetry Collector is, and you have a general understanding of what Helm is. Let’s proceed. 

Structure of the Collector

We’re going to be using a few concepts of the Collector that are important to understand.

1. Config file

It might seem obvious, but the Collector has a config file. It controls a lot of things, such as which endpoints are available (HTTP, gRPC, etc.), and where the telemetry data is sent. It’s a YAML file, however, the interesting part here is that the Helm chart defines a ton of defaults so the config you supply can be a lot smaller.

2. Presets

As part of the config file, the Collector Helm chart defines some “presets” which allow for some of the more advanced concepts to be configured automatically. Presets are useful when the functionality you want needs more than configuration, like permissions. We’ll use the k8sattributes preset in this example, but we’ll get into that later. 

3. Environment variable config replacement

This capability allows you to supply placeholders in the config file that are replaced at runtime with environment variables. I like this because it means you don’t have to perform the replacement on the config file itself as part of your deployment, thus making your scripts cleaner.

4. Deployment modes

There are lots of deployment modes in Kubernetes. The simplest seems to be called Deployment, and it allows you to easily manage the creation of pods and scale them. The Helm template supplied by the OpenTelemetry Collector team allows you to override this. We’ll use the DaemonSet type, as this means that every node in your cluster will get its own instance of the Collector (and therefore reduce cross-node bandwidth).

The environment

The environment that we’ll deploy the Collector into is a two node Azure Kubernetes Cluster (AKS), the important part being that we have two nodes, so using a DaemonSet makes more sense. Beyond that, there is nothing special about the Kubernetes cluster.

The config

As I said above, there is a default configuration the helm chart uses, therefore what we’re supplying is in addition to that.

mode: daemonset
resources:
  limits:
    cpu: 1
    memory: 1Gi
image:
  tag: "0.81.0"
presets:
  kubernetesAttributes:
    enabled: true
config:
  receivers:
    otlp:
      protocols:
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*

  exporters:
    otlp:
      endpoint: api.honeycomb.io:443
      headers:
        x-honeycomb-team: ${HONEYCOMB_API_KEY}

    otlp/metrics:
      endpoint: api.honeycomb.io:443
      headers:
        x-honeycomb-team: ${HONEYCOMB_API_KEY}
        x-honeycomb-dataset: service-metrics

    otlp/logging:
      endpoint: api.honeycomb.io:443
      headers:
        x-honeycomb-team: ${HONEYCOMB_API_KEY}
        x-honeycomb-dataset: service-logs

  service:
    pipelines:
      traces:
        exporters: [otlp]
      metrics: 
        exporters: [otlp/metrics]
      logs: 
        exporters: [otlp/logging]

Let’s break this down a little.

mode: daemonset
resources:
  limits:
    cpu: 1
    memory: 1Gi
image:
  tag: "0.81.0"

This is the configuration around how the Kubernetes deployment is set up and generally how Kubernetes will treat our Collector containers. We’re telling the Helm chart to use the DaemonSet mode we talked about above. We’re also telling it to give the pods 1GB of RAM and a single CPU. Finally, we’re overriding the default latest tag for the Collector image with a specific version. You can also use repository to use a different image (like your own custom-built Collector image). Do be careful as the command to run the Collector can be different when you build your own.

Then, we get to the presets:

presets:
  kubernetesAttributes:
    enabled: true

This will enable the k8sattributesprocessoras a processor in our pipeline. This is important because:  

• It adds the k8sattribute processor, and configures it for you. 
• There’s some config that is non-trivial here, such as the permissions the Collector will then need. 

These presets are a gamechanger, in my opinion, when it comes to getting off the ground quickly.

The next part is the config of the Collector itself:

  receivers:
    otlp:
      protocols:
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*

exporters:
    otlp:
      endpoint: api.honeycomb.io:443
      headers:
        x-honeycomb-team: ${HONEYCOMB_API_KEY}

    otlp/metrics:
      endpoint: api.honeycomb.io:443
      headers:
        x-honeycomb-team: ${HONEYCOMB_API_KEY}
        x-honeycomb-dataset: service-metrics

    otlp/logging:
      endpoint: api.honeycomb.io:443
      headers:
        x-honeycomb-team: ${HONEYCOMB_API_KEY}
        x-honeycomb-dataset: service-logs

  service:
    pipelines:
      traces:
        exporters: [otlp]
      metrics: 
        exporters: [otlp/metrics]
      logs: 
        exporters: [otlp/logging]

Receivers

You’ll notice here that I’ve added the CORS attributes to the HTTP receiver, and that I added the gRPC receiver. It’s going to merge this with the default config from the Collector, so we don’t have to worry about setting things like listen addresses. Once again, sensible defaults are a superpower of developer experience!

Exporters

Honeycomb requires default datasets for logs and metrics data as the service.name attribute isn’t a mandatory attribute/label. Therefore, we have to set up three different exporters. Please note ${HONEYCOMB_API_KEY} isn’t a placeholder. This tells the Collector to replace that item with the value of an environment variable with that when it runs.

Pipelines

We’re taking the defaults of each of the pipelines (named metrics, logs, traces) and adding our new exporters as additional exporters. We don’t have to specify the receivers or processors here.

Adding the Honeycomb API Key

Now that we have our YAML file, we can test to see if it will work using the --dry-run parameter on the Helm command.

Add the open-telemetry Helm repository to the cluster.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

Then, run the template install, but with the --dry-run parameter.

helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector --values <path to my values file>  --set extraEnvs[0].value={my-honeycomb-api-key}--set extraEnvs[0].name=HONEYCOMB_API_KEY --dry-run

I’d highly recommend that you pipe that into a file to review with > temp.yaml at the end. You can now search for the string Kind: ConfigMap to see what the rendered config map looks like.

Conclusion

The Collector is a powerful thing, and getting started isn’t difficult at all if you’re in Kubernetes due to the Helm charts the OpenTelemetry team provides. Thanks to sensible defaults and presets, getting Kubernetes observability into Honeycomb is really easy to do. 

Give it a go! We offer a free account with the ability to send up to 20 million events without even providing a credit card.

If you’re not quite ready to try Honeycomb yet, continue your reading journey with more Collector info.