Properties

Last built 1 year ago using go version go1.5.1 linux/amd64 and 67 seconds of time

Downloads
Filename
File Size
 
go-log2kafka_master_darwin-386.zip
1.8MiB
go-log2kafka_master_darwin-amd64.zip
1.9MiB
go-log2kafka_master_linux-386.zip
1.7MiB
go-log2kafka_master_linux-amd64.zip
1.9MiB
go-log2kafka_master_linux-arm.zip
1.7MiB
Project-Readme for this version

go-log2kafka

A simple daemon that reads a file (tail -f style) and sends every line to a Kafka topic.

Uses https://github.com/ActiveState/tail and https://github.com/Shopify/sarama

Intended for nginx and apache access logs – so it does some special character encoding/escaping for that format (because \xXX is not valid JSON).

Usage of ./go-log2kafka:
  -comp=0: Kafka compression codec (default: 0 for no compression)
  -file="/var/log/syslog": filename to watch
  -n=false: Quit after file is read, do not wait for more data, do not read/write state
  -server="localhost:9092": Kafka endpoints, comma separated (default: localhost:9092)
  -topic="logs": Kafka topic name (default: logs)
  -v=false: Verbose output

Package example

The ubuntu directory contains an example how to package this tool: build.sh will fetch all sources, compile them, and use fpm to build a Debian package containing the binary, the configuration, and the upstart config.

Config examples

The example directory contains configuration snippets for Apache, nginx, and varnishncsa.