Loggie is a lightweight, high-performance, cloud-native log collection agent and aggregator based on Golang.
English | 中文
Loggie is a lightweight, high-performance, cloud-native agent and aggregator based on Golang.
Based on Loggie, we can build a cloud-native scalable log data platform.
Features Next-generation cloud-native log collection and transmission Building pipelines based on CRD
Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRDs, allowing for the creation of data collection, transfer, processing, and sending pipelines through simple YAML file creation.
eg:
apiVersion: loggie.io/v1beta1 kind: LogConfig metadata: name: tomcat namespace: default spec: selector: type: pod labelSelector: app: tomcat pipeline: sources: | - type: file name: common paths: - stdout - /usr/local/tomcat/logs/*.log sinkRef: default interceptorRef: default
Multiple architectures
Agent: Deployed via DaemonSet, Loggie can collect log files without the need for containers to mount volumes.
Sidecar: Supports non-intrusive auto-injection of Loggie sidecars, without the need to manually add them to the Deployment/StatefulSet templates.
Aggregator: Supports deployment as an independent intermediate machine, which can receive aggregated data sent by Loggie Agent and can also be used to consume and process various data sources.
But regardless of the deployment architecture, Loggie still maintains a simple and intuitive internal design.
High Performance Benchmark
Configure Filebeat and Loggie to collect logs, and send them to a Kafka topic without using client compression, with the Kafka topic partition configured as 3.
With sufficient resources for the Agent specification, modify the number of files collected, the concurrency of the sending client (configure Filebeat worker and Loggie parallelism), and observe their respective CPU, memory, and pod network card transmission rates.
Agent File Size File Count Sink Concurrency CPU MEM (rss) Transmission Rates
Filebeat 3.2G 1 3 7.5~8.5c 63.8MiB 75.9MiB/s
Filebeat 3.2G 1 8 10c 65MiB 70MiB/s
Filebeat 3.2G 10 8 11c 65MiB 80MiB/s
Loggie 3.2G 1 3 2.1c 60MiB 120MiB/s
Loggie 3.2G 1 8 2.4c 68.7MiB 120MiB/s
Loggie 3.2G 10 8 3.5c 70MiB 210MiB/s
Adaptive Sink Concurrency
With sink concurrency configuration enabled, Loggie can:
Lightweight Streaming Data Analysis and Monitoring
Logs are a universal data type and are not related to platforms or systems. How to better utilize this data is the core capability that Loggie focuses on and develops.
Real-time parsing and transformation
With the configuration of transformer interceptors and the configuration of functional actions, Loggie can achieve:
eg:
interceptors:
Helps you quickly detect potential problems and anomalies in the data and issue timely alerts. Support custom webhooks to connect to various alert channels.
Supports matching methods such as:
Often, metric data is not only exposed through prometheus exporters, but log data itself can also provide a source of metrics. For example, by counting the access logs of a gateway, you can calculate the number of 5xx or 4xx status codes within a certain time interval, aggregate the qps of a certain interface, and calculate the total amount of body data, etc.
eg:
- type: aggregator interval: 1m select: # operator:COUNT/COUNT-DISTINCT/SUM/AVG/MAX/MIN
Loggie Filebeat Fluentd Logstash Flume
Language Golang Golang Ruby JRuby Java
Multiple Pipelines ✓ single queue single queue ✓ ✓
Multiple output ✓ one output copy ✓ ✓
Aggregator ✓ ✓ ✓ ✓ ✓
Log Alarm ✓
Kubernetes container log collection support container stdout and logs files in container stdout stdout
Configuration delivery through CRD manual manual manual manual
Monitoring support Prometheus metrics,and can be configured to output indicator log files separately, sending metrics, etc.
prometheus metrics need exporter need exporter
Resource Usage low low average high high
License
Contributions
Pull requests, comments and suggestions are welcome.
Please see CONTRIBUTING.md for more information.
Twice a month we will interview people behind open source businesses. We will talk about how they are building a business on top of open source projects.
We'll never share your email with anyone else.