Kubernetes provides additional layers of abstraction, which help create business flexibility for deploying applications in a rapid and agile manner. However, at the same time, these layers of abstraction create additional complexity for security monitoring and incident response. Large production Kubernetes environments can feel like a Star Wars megacity with the size of Coruscant, where the movement of data, compute workloads, and authentications move fast to the lower levels and higher levels of a megacity.
In this blog post, we will talk about how to connect Falco run-time security alerts with Microsoft Sentinel SIEM (Security Information and Event Management) to get Near-Real-Time (NRT) security alerts within your EKS (Amazon Elastic Kubernetes Service) or AKS (Azure Kubernetes Service) environment. This blog post will not cover how to secure your Kubernetes (k8s) environment. Instead, we will explain how to detect real-time threats inside container workloads and Kubernetes nodes.
How Falco works
In the past, EDR (Endpoint Detection and Response) solutions were not compatible with container workloads. This is why Falco from the makers of Sysdig got created as a container run-time detection solution. Falco is not an EDR solution because it relies on a relatively "static" community-based rule-set (1039 commits) referred to as falco_rules.yaml. Instead, Falco is comparable to a Host-based Intrusion Detection System (HIDS). Falco can monitor active security threats (attempts) inside container workloads based on Indicators of Attack (IoA) and Indicators of Compromise (IoC).
Falco can monitor all file accesses, spawned processes, network activity, and system call events on the kernel level or eBPF, which is excellent because system calls never lie. This makes Falco hard to evade without using custom binaries or binary aliases, and bash shell command-line encoding. A malicious attack would leave traces that Falco can capture. Falco should be deployed as a Kubernetes pod on all your Kubernetes nodes within your cluster. This means Falco can possibly detect the following:
- Linux privilege escalation (attempts) from a non-root user to root
- Data exfiltration (data theft)
- Malicious altering of sensitive files
- Unwanted access to sensitive files
- Malicious ownership and mode changes to files
- Post exploitation of (0-day) vulnerabilities detection
- Malicious access creation hacking attempt
- Compromised container workload making a malicious connection (example source: infected image)
- Compromised container workload being exploited for mining cryptocurrency
- Malicious privileged pod creation
- Signs of a compromised Kubernetes node
- Malicious entity removed traces of a container compromiseFalco also has a plugin available to ingest and analyse Kubernetes audit logs. This way, we can monitor and alert for anomalies within the Kubernetes nodes, next to the Kubernetes container workloads. This is shown in Figure 1:
Figure 1: Falco container security monitoring with Microsoft Sentinel SIEM
The Falco engine compares the monitored container host and Kubernetes node activities to the Falco rule-set, and generates an alert log message when a match is found. This way, detection alerts are created close to the source, which makes this methodology cost-effective and scalable in large Kubernetes environments. Sentinel SIEM charges per ingested Gigabyte. Therefore, analysing your ingestion costs based on a Falco rule-set makes sense.
End-to-end alerting
When a Falco alert is triggered based on the Falco rule set within the Falco engine, an alert with all its meta-data will be forwarded via Falcosidekick to a central remote Syslog server in JSON Syslog format. Falcosidekick supports JSON Syslog CEF output since version 2.27. This means we do not need to configure a custom Azure Log App, Event Hub, or rely on AWS Security Hub forward delays when connecting Falco data to a SIEM solution such as Sentinel.
Next, the Syslog forwarding server in AWS or Azure cloud forwards the Falco data in CEF (Common Event Format) to Sentinel SIEM. In Sentinel, we created a Sentinel KQL parser Function to map each Falco JSON field to a Log Analytics variable. Next, we call the parser and make per use-case KQL Sentinel Analytics Incidents based on Falco alerts. Next, we will re-use the MITRE ATT&CK tactic tag from the Falco rule set to get a holistic view of which stage of malicious attack alerts is triggered. This way, we can follow a lateral attack in Sentinel SIEM based on configured IoAs and IoCs use-cases.
Note: the Kubernetes cluster name is not included in the default Falco JSON output fields. To identify from which cluster the event is coming, we created a custom Sentinel KQL parser to recognise the Kubernetes cluster name. This is important if you have a scalable environment when SOC analysts need to contact the right Kubernetes cluster owner.
Falco alerts data is forwarded via remote Syslog to Sentinel. You can create custom alerts in Sentinel and perform threat hunting based on KQL queries. However, a mature Falco rule-set can reduce false positives and save Sentinel ingestion costs, together with a more pragmatic follow-up process for notable critical alerts.
Finally, the SOC analyst team gets an alert with all the necessary data to investigate any detected anomalies further. Figure 2 shows a Sentinel Incident alert based on reverse-shell activity. You can see here the malicious command, the affected container, the image source, and Kubernetes cluster name:
Figure 2: KQL query results in Sentinel based on the below use-case 1 - alert on a reverse shell connection
Falco run-time detection use-cases
For this blog post, we configure five use-cases which are also available on our GitHub page.
Use-case 1: Reverse-shell connection detected based on the following Falco alert:
//Falco run-time Container Threat Detection: Netcat Reverse Shell connection detected
//cryptsus.com - We craft Cyber Security Solutions
FalcoContainerSecurity
| where Activity == "Netcat Remote Code Execution in Container"
| project-away TenantId, DeviceVendor, DeviceProduct, DeviceVersion, DeviceEventClassID, Message, AdditionalExtensions, SourceSystem, Type, outputfields, parent, cmdline, Computer
| sort by TimeGeneratedUse-case 2: Read /etc/shadow file based on the following Falco alert:
//Falco run-time Container Threat Detection: read /etc/shadow Linux password file
//cryptsus.com - We craft Cyber Security Solution
FalcoContainerSecurity
| where Activity == "Read sensitive file untrusted"
| where file contains_cs "/etc/shadow"
| project-away TenantId, DeviceVendor, DeviceProduct, DeviceVersion, DeviceEventClassID, Message, AdditionalExtensions, SourceSystem, Type, outputfields, parent, cmdline, Computer
| sort by TimeGeneratedUse-case 3: Create new SSH keys with ssh-keygen to authenticate to the container based on the following Falco alert:
//Falco run-time Container Threat Detection: new SSH keys created with ssh-keygen
//cryptsus.com - We craft Cyber Security Solutions
FalcoContainerSecurity
| where Activity == "Write below monitored dir"
| where command contains_cs "ssh-keygen"
| project-away TenantId, Activity, LogSeverity, AdditionalExtensions, Message, parent, DeviceVendor, DeviceProduct, DeviceVersion, DeviceEventClassID, SourceSystem, Type, outputfields, Computer
| where command contains_cs "ssh-keygen"Use-case 4: Secrets extracted with kubctl (read Kubernets secrets) based on the following Falco alert:
//Falco run-time Container Threat Detection: Secrets extracted with kubctl (read Kubernets secrets)
//cryptsus.com - We craft Cyber Security Solutions
FalcoContainerSecurity
| where Activity == "The docker client is executed in a container"
| where cmdline contains_cs "kubectl get secrets"
| project-away TenantId, DeviceVendor, DeviceProduct, DeviceVersion, DeviceEventClassID, Message, AdditionalExtensions, SourceSystem, Type, outputfields, Computer
| sort by TimeGeneratedUse-case 5: Remove evidence of compromise by erasing the Bash shell commands history based on the following Falco alert:
//Falco run-time Container Threat Detection: bash history deleted
//cryptsus.com - We craft Cyber Security Solution
FalcoContainerSecurity
| where Activity == "Delete or rename shell history"
| project-away TenantId, DeviceVendor, DeviceProduct, DeviceVersion, DeviceEventClassID, Message, AdditionalExtensions, SourceSystem, Type, outputfields, Computer
| sort by TimeGenerated
| where command contains_cs "bash_history"In order to cover more detection opportunities, you can compare your use-cases with the Container run-time MITRE Kubernetes threat framework in Figure 3:
Figure 3: Falco container security monitoring with Microsoft Sentinel SIEM. Source: AquaSec Blog
Feel free to contact us to configure Falco use-cases based on your business risks. We specialise in creating detection opportunities for your business environment.
