Open source cloud-native security lake platform (SIEM alternative) for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
Open source security data lake for AWS
Matano Open Source Security data lake is an open source cloud-native security data lake, built for security teams on AWS.
Note
Matano offers a commercial managed Cloud SIEM for a complete enterprise Security Operations platform. Learn more.
<a href="https://www.matano.dev/docs" rel="nofollow">Docs</a>
|
<a href="https://www.matano.dev" rel="nofollow">Website</a>
|
<a href="https://discord.gg/YSYfHMbfZQ" rel="nofollow">Community</a>
Features
Architecture
👀 Use cases
✨ Integrations Managed log sources
Alert destinations
Query engines
Quick start
View the complete installation instructions
Installation
Install the matano CLI to deploy Matano into your AWS account, and manage your deployment.
Linux
curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-linux-x64.sh chmod +x matano-linux-x64.sh sudo ./matano-linux-x64.sh
macOS
curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-macos-x64.sh chmod +x matano-macos-x64.sh sudo ./matano-macos-x64.shDeployment
Read the complete docs on getting started
To get started, run the matano init
command.
Directory structure
Once initialized, your Matano directory is used to control & manage all resources in your project e.g. log sources, detections, and other configuration. It is structured as follows:
➜ example-matano-dir git:(main) tree ├── detections │ └── aws_root_credentials │ ├── detect.py │ └── detection.yml ├── log_sources │ ├── cloudtrail │ │ ├── log_source.yml │ │ └── tables │ │ └── default.yml │ └── zeek │ ├── log_source.yml │ └── tables │ └── dns.yml ├── matano.config.yml └── matano.context.json
When onboarding a new log source or authoring a detection, run matano deploy
from anywhere in your project to deploy the changes to your account.
🔧 Log Transformation & Data Normalization
Read the complete docs on configuring custom log sources
Vector Remap Language (VRL), allows you to easily onboard custom log sources and encourages you to normalize fields according to the Elastic Common Schema (ECS) to enable enhanced pivoting and bulk search for IOCs across your security data lake.
Users can define custom VRL programs to parse and transform unstructured logs as they are being ingested through one of the supported mechanisms for a log source (e.g. S3, SQS).
VRL is an expression-oriented language designed for transforming observability data (e.g. logs) in a safe and performant manner. It features a simple syntax and a rich set of built-in functions tailored specifically to observability use cases.
Example: parsing JSON
Let's have a look at a simple example. Imagine that you're working with HTTP log events that look like this:
{ "line": "{\"status\":200,\"srcIpAddress\":\"1.1.1.1\",\"message\":\"SUCCESS\",\"username\":\"ub40fan4life\"}" }
You want to apply these changes to each event:
line
string into JSON, and explode the fields to the top levelsrcIpAddress
to the source.ip
ECS fieldusername
fieldmessage
to lowercaseAdding this VRL program to your log source as a transform
step would accomplish all of that:
log_source.yml
transform: | . = object!(parse_json!(string!(.json.line))) .source.ip = del(.srcIpAddress) del(.username) .message = downcase(string!(.message))schema: ecs_field_names: - source.ip - http.status
The resulting event 🎉:
{ "message": "success", "status": 200, "source": { "ip": "1.1.1.1" } }📝 Writing Detections
Read the complete docs on detections
Use detections to define rules that can alert on threats in your security logs. A detection is a Python program that is invoked with data from a log source in realtime and can create an alert.
Examples Detect failed attempts to export AWS EC2 instance in AWS CloudTrail logs.
def detect(record): return ( record.deepget("event.action") == "CreateInstanceExportTask" and record.deepget("event.provider") == "ec2.amazonaws.com" and record.deepget("event.outcome") == "failure" )Detect Brute Force Logins by IP across all configured log sources (e.g. Okta, AWS, GWorkspace) detect.py
def detect(r): return ( "authentication" in r.deepget("event.category", []) and r.deepget("event.outcome") == "failure" )detection.ymldef title(r): return f"Multiple failed logins from {r.deepget('user.full_name')} - {r.deepget('source.ip')}"
def dedupe(r): return r.deepget("source.ip")
--- tables:
from detection import remotecache
user_to_ips = remotecache("user_ip")
def detect(record): if ( record.deepget("event.action") == "ConsoleLogin" and record.deepget("event.outcome") == "success" ): # A unique key on the user name user = record.deepget("user.name")
existing_ips = user_to_ips[user] or []
updated_ips = user_to_ips.add_to_string_set(
user,
record.deepget("source.ip")
)
# Alert on new IPs
new_ips = set(updated_ips) - set(existing_ips)
if existing_ips and new_ips:
return True</pre>
🚨 Alerting
Read the complete docs on alerting
Alerts table
All alerts are automatically stored in a Matano table named matano_alerts
. The alerts and rule matches are normalized to ECS and contain context about the original event that triggered the rule match, along with the alert and rule data.
Example Queries
Summarize alerts in the last week that are activated (exceeded the threshold)
select matano.alert.id as alert_id, matano.alert.rule.name as rule_name, max(matano.alert.title) as title, count(*) as match_count, min(matano.alert.first_matched_at) as first_matched_at, max(ts) as last_matched_at, array_distinct(flatten(array_agg(related.ip))) as related_ip, array_distinct(flatten(array_agg(related.user))) as related_user, array_distinct(flatten(array_agg(related.hosts))) as related_hosts, array_distinct(flatten(array_agg(related.hash))) as related_hash from matano_alerts where matano.alert.first_matched_at > (current_timestamp - interval '7' day) and matano.alert.activated = true group by matano.alert.rule.name, matano.alert.id order by last_matched_at descDelivering alerts
You can deliver alerts to external systems. You can use the alerting SNS topic to deliver alerts to Email, Slack, and other services.
A medium severity alert delivered to Slack
❤️ Community support
For general help on usage, please refer to the official documentation. For additional help, feel free to use one of these channels to ask a question:
👷 Contributors
Thanks go to these wonderful people (emoji key):
Kai Herrera
💻 🤔 🚇
Zach Mowrey
🤔 🐛 📓
Tim O'Guin
🤔 🐛 💻
This project follows the all-contributors specification. Contributions of any kind are welcome!
License
Twice a month we will interview people behind open source businesses. We will talk about how they are building a business on top of open source projects.
We'll never share your email with anyone else.