Fluentd is a widely-used data router. If you’re using Fluentd to aggregate structured logs, Honeycomb’s Fluentd output plugin makes it easy to forward data to Honeycomb.
Install the Honeycomb plugin by running:
fluent-gem install fluent-plugin-honeycomb
If you’re using the stable distribution package of Fluentd,
td-agent, run the following instead:
td-agent-gem install fluent-plugin-honeycomb
To set up the plugin, first grab your team write key from your account page, then update your Fluentd configuration file (usually found in
/etc/td-agent/td-agent.conf). A basic configuration to forward events with the
my.logs tag to Honeycomb looks like this:
<match my.logs> @type honeycomb writekey "YOUR_WRITEKEY" dataset "fluentd_test_dataset" </match>
Fluentd and Honeycomb support a wide variety of logging setups. Here are some things you might want to do.
If you want to combine multiple related log sources into one dataset, you can add the option
include_tag_key true to augment each event with its Fluentd tag before sending it along.
On the other hand, if you have completely unrelated log sources, you can forward them to separate datasets by using two separate configuration blocks:
<match a.*> @type honeycomb writekey "YOUR_WRITEKEY" dataset "dataset_b" </match> <match b.*> @type honeycomb writekey "YOUR_WRITEKEY" dataset "dataset_a" </match>
To send output to both Honeycomb and another destination simultaneously, we recommend using the
fluent-plugin-copy_ex plugin. For example, a configuration to send nginx access logs to both Honeycomb and Elasticsearch might look like:
<match myapp.*> @type copy_ex <store ignore_error> @type honeycomb writekey "YOUR_WRITEKEY" dataset "fluentd_test_dataset" <store ignore_error> @type elasticsearch ... </store> </match>
If you have a high volume of traffic, you can use the
sample_rate option to send a sampled fraction of your traffic to Honeycomb. For example, to forward one out of every five events:
<match your_match> @type honeycomb writekey "YOUR_WRITEKEY" dataset "fluentd_test_dataset" sample_rate 5 </match>
fluent-plugin-honeycomb supports the following options, in addition to Fluentd’s standard options for controlling output buffering:
|writekey||string||yes||Your Honeycomb write key.|
|dataset||string||yes||The name of the destination dataset in your Honeycomb account.|
|sample_rate||integer||no||Sample your event stream by sending 1 out of every N events.|
|include_tag_key||bool||no||Whether to include the Fluentd tag in the submitted event.|
|tag_key||string||no||If include_tag_key is true, the tag key name in the event (default: fluentd_tag).|
fluent-plugin-honeycomb is open source, Apache 2.0 licensed. Its source is on GitHub.