Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
This legacy version of ElastiFlow™ is now deprecated. Try the new ElastiFlow™!!! PLEASE NOTE: This repository remains for historical purposes, however no part of this legacy solution is shared with the next generation of ElastiFlow™. To install the new solution please refer to the documentation at docs.elastiflow.com. For assistance with the new solution, please join the ElastiFlow Community Slack.
Now that the new ElastiFlow™ Unified Flow Collector is generally available, the legacy Logstash-based solution in this repository is deprecated. It will remain here for historical purposes, but will not receive updates. You are encouraged to try out the new collector. See more details below.
We have released the next generation of ElastiFlow™ which introduces the new ElastiFlow Unified Flow Collector for Netflow, IPFIX and sFlow. Some of the enhancements included in the all-new collector include:
Get Started with the next generation of ElastiFlow... HERE
You can also join the ElastiFlow community Slack... HERE
ElastiFlow™ provides network flow data collection and visualization using the Elastic Stack (Elasticsearch, Logstash and Kibana). It supports Netflow v5/v9, sFlow and IPFIX flow types (1.x versions support only Netflow v5/v9).
I was inspired to create ElastiFlow™ following the overwhelmingly positive feedback received to an article I posted on Linkedin... WTFlow?! Are you really still paying for commercial solutions to collect and analyze network flow data?
User Testimonials
Organization Feedback
"ElastiFlow has significantly reduced our network flow monitoring costs. The flexibility and extensibility inherent to ElastiFlow was essential for an easy integration with our infrastructure monitoring suite."
“Right now this is my personal favorite analytics tool. I use it extensively and am always finding a new way to leverage it."
"We're using it since two months in our new datacenter and our network admins are very happy and impressed."
"Of all the netflow tools I’ve tested it has, by far, the best visualizations."
"We absolutely love ElastiFlow and recently stood it up in production. Looking forward to new functionality and dashboards."
Getting Started
NOTE - Please refer to
INSTALL.md
for installation instructions.
ElastiFlow™ is built using the Elastic Stack, including Elasticsearch, Logstash and Kibana. Please refer to INSTALL.md for instructions on how to install and configure ElastiFlow™.
If you are new to the Elastic Stack, this video goes beyond a simple default installation of Elasticsearch and Kibana. It discusses real-world best practices for hardware sizing and configuration, providing production-level performance and reliability.
Additionally local SSD storage should be considered as mandatory! For an in-depth look at how different storage options compare, and in particular how bad HDD-based storage is for Elasticsearch (even in multi-drive RAID0 configurations) you should watch this video...
NOTE: Please make sure that have reviewed KNOWN_ISSUES.md prior to getting started.
Kibana App Compatibility SIEM
Logs
Provided Dashboards
The following dashboards are provided.
NOTE: The dashboards are optimized for a monitor resolution of 1920x1080.
Overview
Top-N
There are separate Top-N dashboards for Top Talkers, Services, Conversations and Applications.
Top Talkers
Top Services
Top Conversations
Top Applications
Threats
ElastiFlow™ includes a dictionary of public IP addresses that are known to have a poor reputation. This dictionary is built from many OSINT data sources, normalized to a common taxonomy. The Threats dashboard uses this IP reputation information to highlight three threat/risk types.
Flows
There are separate Sankey dashboards for Client/Server, Source/Destination and Autonomous System perspectives. The sankey visualizations are built using the new Vega visualization plugin.
Client/Server Flows
Source/Destination Flows
Autonomous System Flows
Geo IP
There are separate Geo Location dashboards for Client/Server and Source/Destination perspectives.
Client/Server Geo IP
Source/Destination Geo IP
AS Traffic
Provides a view of traffic to and from Autonomous Systems (public IP ranges)
Flow Exporters
Traffic Details
Provides more detailed breakdown of various network traffic characteristics.
Traffic Types
Traffic Attributes
Traffic Locality
Flow Records Client/Server
Source/Destination
Ziften ZFlow
ElastiFlow™ v3.4.0 added support for IPFIX records from Ziften's ZFlow agent. In addition to being fully integrated with the standard dashboards, a stand-alone ZFlow dashboards displays network traffic based on user and command data provided by ZFlow.
Attribution
This product includes GeoLite2 data created by MaxMind, available from (http://www.maxmind.com)
Twice a month we will interview people behind open source businesses. We will talk about how they are building a business on top of open source projects.
We'll never share your email with anyone else.