Falco extended output and custom rules
Summary of Falco and Falcosidekick output types
By default, Falco's events have five outputs: stdout, file, GRPC, shell, and http. As shown in the figure below:
Even if they are convenient, we may soon be limited to integrating Falco with other components. falcosidekick can help. This is a small daemon that extends the possible output.
The currently available falcosidekick output list (version 2.24.0) is:
Chat
Metrics / Observability
- Datadog
- Influxdb
- StatsD (for monitoring of falcosidekick)
- DogStatsD (for monitoring of falcosidekick)
- Prometheus (for both events and monitoring of falcosidekick)
- Wavefront
Alerting
Logs
- Elasticsearch
- Loki
- AWS CloudWatchLogs
- Grafana (annotations)
- Syslog
Object Storage
FaaS / Serverless
Message queue / Streaming
- NATS
- STAN (NATS Streaming)
- AWS SQS
- AWS SNS
- GCP PubSub
- Apache Kafka
- Kafka Rest Proxy
- RabbitMQ
- Azure Event Hubs
- SMTP
Web
- Webhook
- WebUI (a Web UI for displaying latest events in real time)
Deploy Falco
In this article, we will send events on the Slack channel and to the WebUI, so first get your WebHook URL (https: / /...). Before that, install falco (using helm3)
1. Add falcosecurity warehouse in Helm**
$ helm repo add falcosecurity https://falcosecurity.github.io/charts $ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "falcosecurity" chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈Happy Helming!⎈
2. Install chart
$ helm install falco falcosecurity/falco --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falcosidekick:2801 -f howto/falcoCustomRule.yaml -n falco NAME: falco LAST DEPLOYED: Mon Nov 9 22:09:28 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Falco agents are spinning up on each node in your cluster. After a few seconds, they are going to start monitoring your containers looking for security issues. No further action should be required. #Falcocustomrule Yaml is a custom rule file
3. Falco should start running in a few seconds. Verify it and you will see the pod you created right away.
$ helm ls $ kubectl get pods
Deploy Falcosidekick
1.helm install falcosidekick
$helm install falcosidekick falcosecurity/falcosidekick --set config.slack.webhookurl=https://hooks.slack.com/services/xxxxxxxx --set webui.enabled=true -n falco
#output
NAME: falcosidekick
LAST DEPLOYED: Thu Jan 14 23:55:12 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace falco -l "app.kubernetes.io/name=falcosidekick,app.kubernetes.io/instance=falcosidekick" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 2801:2801
echo "Visit http://127.0.0.1:2801 to use your application"
#About hooks, the connection needs to be created independently https://slack.com/intl/zh-cn/
View pod
$ kubectl get pod -n falco NAME READY STATUS RESTARTS AGE falco-9hftg 1/1 Running 0 89m falco-gr5q7 1/1 Running 1 90m falco-lpczj 1/1 Running 0 90m falcosidekick-586bdb8c6b-j2wdf 1/1 Running 0 5h39m falcosidekick-586bdb8c6b-zncvq 1/1 Running 0 5h39m falcosidekick-ui-68f9444cbc-8x9b6 1/1 Running 0 40m
view log
$kubectl logs deployment/falcosidekick -n falco Found 2 pods, using pod/falcosidekick-586bdb8c6b-j2wdf 2021/12/30 03:42:06 [INFO] : Enabled Outputs : [Slack WebUI] 2021/12/30 03:42:06 [INFO] : Falco Sidekick is up and listening on :2801 2021/12/30 03:48:12 [INFO] : WebUI - Post OK (200) 2021/12/30 03:48:15 [INFO] : Slack - Post OK (200)
Modify the svc type of falcosidekick
#View svc
$kubectl get svc -n falco
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
falcosidekick ClusterIP 10.106.191.244 <none> 2801/TCP 5h46m
falcosidekick-ui ClusterIP 10.99.191.73 <none> 2802/TCP 5h46m
#Modify svc type
$kubectl edit svc -n falco falcosidekick-ui
.......
spec:
clusterIP: 10.99.191.73
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 30802 #Add NodePort port, port range 30000-32767
port: 2802
protocol: TCP
targetPort: http
selector:
app.kubernetes.io/instance: falcosidekick-ui
app.kubernetes.io/name: falcosidekick-ui
sessionAffinity: None
type: NodePort #Change to NodePort type
.......
#View svc
$ kubectl get svc -n falco
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
falcosidekick NodePort 10.106.191.244 <none> 2801:30801/TCP 5h46m
falcosidekick-ui NodePort 10.99.191.73 <none> 2802:30802/TCP 5h46m
Log in to WebUI to view
Log in to slack to view the output
Custom rules
1, Introduction to rules
Falco rule files are YAML files that contain three types of elements:
| element | describe |
|---|---|
| Rules | Generate trigger conditions for alerts. Rules and alerts with descriptive information are sent together |
| Macros | Reusable rule condition fragments in rules and even other macros. Macros provide a way to name common patterns and eliminate redundancy in rules. |
| Lists | A collection of items that can be included in a rule, macro, or other list. Unlike rules and macros, lists cannot be resolved to filter expressions |
2, Rules rules
The Falco rule contains the following keys:
| Key | Required | Description | Default |
|---|---|---|---|
| rule | yes | Short unique rule name | |
| condition | yes | Filter expression that detects whether the event matches the rule | |
| desc | yes | Detailed description of rule detection content | |
| output | yes | The message output when the event matches follows the Sysdig output format syntax | |
| priority | yes | A representation of the severity of the event (case insensitive). It should be one of the following: emergency, alert, critical, error, warning, notice, informational, debug. | |
| exceptions | no | A set of exceptions that cause the rule not to generate alerts | |
| enabled | no | If set to false, the rule neither loads nor matches any events. | true |
| tags | no | List of labels applied to the rule | |
| warn_evttypes | no | If set to false, Falco suppresses warnings related to rules unrelated to events | true |
| skip-if-unknown-filter | no | If set to true, if the rule condition contains filter check items, such as FD some_ new_ Field, and this version of Falco does not recognize this item. Falco silently accepts the rule but does not execute it; If set to false, Falco reports an error when it encounters an unknown filter check item. | false |
2.1 Conditions
The core of the rule is the condition field. Any Sysdig filter is a valid Falco condition. In addition, Falco conditions can contain Macros.
Example of a condition that alerts whenever the bash shell is running inside a container:
container.id != host and proc.name = bash
The first clause checks whether the event occurs in the container (if the event occurs on a regular host, the container field of Sysdig event is equal to "host").
The second clause checks whether the process name is bash.
Note that this condition does not include clauses with system calls! It only checks the event metadata.
Therefore, if the bash shell does start in the container, Falco outputs events for each system call executed by the shell.
The complete rule for using the above conditions may be:
- rule: shell_in_container #Rule name desc: notice shell activity within a container #Rule comments condition: container.id != host and proc.name = bash #Conditional filter expression (bash is used for unconventional hosts) output: shell in a container (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline) #Output information when matching priority: WARNING #Event level
2.2 Macros
For a very simple example, if we have many rules related to events in the container, we may define an in_container macro:
- macro: in_container #Macro name condition: container.id != host and proc.name = bash #Conditional filter expression
After defining this macro, we can rewrite the condition of the above rule to in_container and proc name = bash.
For more examples of rules and macros, see the documentation on default macros or rules/falco_rules.yaml file.
Complete rules for using Macros:
- rule: shell_in_container #Rule name desc: notice shell activity within a container #Rule comments condition: (in_container) #Conditional filter expression (bash is used for unconventional hosts) output: shell in a container (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline) #Output information when matching priority: WARNING #Event level
2.3 Lists
A list is a collection of named items that can be included in rules, macros, and even other lists. Note that the list cannot be resolved to a filter expression. Each list node has the following keys:
| Key | Description |
|---|---|
| list | Unique name of the list |
| items | Value list |
List examples and macros that use it:
- list: shell_binaries #List name items: [bash, csh, ksh, sh, tcsh, zsh, dash] #Include value - list: userexec_binaries items: [sudo, su] - list: known_binaries items: [shell_binaries, userexec_binaries] #Include other lists - macro: safe_procs condition: proc.name in (known_binaries) #Condition (process name list contains values)
A reference list inserts a list item into a macro, rule, or Lists list.
3, Attach to lists, rules, and macros
If you use multiple Falco rule files, you may want to append a new item to an existing list, rule, or macro. To do this, define an item with the same name as an existing item and add the append: true attribute to the list.
When you attach a Lists List, item is added to the end of the List.
When you attach a rule / macro, additional text is attached to the field of the condition: rule / macro.
Additional rules can also add configuration rule files when deploying with Helm Chart
$ helm install falco falcosecurity/falco -f falcoCustomRule.yaml -n falco
falcoCustomRule.yaml
customRules:
falco_custom_network_rule.yaml: |-
- list: website_ips
items: ['"197.255.20.45"']
- rule: Connection to test or releaes
desc: Detect attempts to connect to yjp website
condition: outbound and fd.sip in (website_ips)
output: Outbound connection to yjp website http://www.release.yijiupidev.com or http://www.test.yijiupidev.com/(command=%proc.cmdline connection=%fd.name container_id=%container.id container_name=%container.name %container.info image=%container.image)
priority: WARNING
tags: [network]
Note that the order of rule profiles is important when attaching to lists, rules, or macros!
For example, if you attach to an existing default rule, you must ensure that your custom configuration file (for example, / etc/falco/rules.d/custom-rules.yaml) is loaded after the default configuration file (/ etc/falco/falco_rules.yaml).
There are three ways to realize the loading sequence of configuration files:
- It can be configured by using multiple - r parameters in the correct order (local installation)
- Through the rules in the falco configuration file (falco.yaml)_ file
- Use the official Helm Chart's Falco Rulesfile value
3.0 example (local installation)
falco -r /etc/falco/falco_rules.yaml -r /etc/falco/falco_rules.local.yaml
3.1 attach to list
Examples of attaching to a list:
/etc/falco/falco_rules.yaml
- list: my_programs items: [ls, cat, pwd] - rule: my_programs_opened_file desc: track whenever a set of programs opens a file condition: proc.name in (my_programs) and evt.type=open output: a tracked program opened a file (user=%user.name command=%proc.cmdline file=%fd.name) priority: INFO
/etc/falco/falco_rules.local.yaml
- list: my_programs #Attach to my_programs list name append: true items: [cp]
When ls, cat, pwd, or cp opens a file, my_programs_opened_file rules are triggered.
3.2 attach to macro
Attach to macro example:
/etc/falco/falco_rules.yaml
- macro: access_file condition: evt.type=open - rule: program_accesses_file desc: track whenever a set of programs opens a file condition: proc.name in (cat, ls) and (access_file) output: a tracked program opened a file (user=%user.name command=%proc.cmdline file=%fd.name) priority: INFO
/etc/falco/falco_rules.local.yaml
- macro: access_file #Macro name append: true condition: or evt.type=openat #Additional rule conditions
Rule program_accesses_file will be triggered when ls/cat uses open/openat for the file
3.3 attach to rule
Attach to rule example:
/etc/falco/falco_rules.yaml
- rule: program_accesses_file desc: track whenever a set of programs opens a file condition: proc.name in (cat, ls) and evt.type=open output: a tracked program opened a file (user=%user.name command=%proc.cmdline file=%fd.name) priority: INFO
/etc/falco/falco_rules.local.yaml
- rule: program_accesses_file #Rule name append: true condition: and not user.name=root #Additional conditions (not triggered when user.name is root)
Rule program_accesses_file will be triggered when ls/cat uses open for the file, but not if the user is root.
3.4 rule / macro attachment and logical operator issues
Remember that when attaching rules and macros, the text of the second rule / macro is only added to the conditions of the first rule / macro.
If the original rule / macro has fuzzy and ambiguous logical operators, this may lead to unexpected results. Here is an example:
- rule: my_rule desc: ... condition: evt.type=open and proc.name=apache output: ... - rule: my_rule append: true condition: or proc.name=nginx
There are two explanations
Explanation 1: open a file or evt.exe in apache/nginx Type = open operation?
Explanation 2: does apache open files or allow nginx to do anything?
In this case, parentheses can be used to limit the scope of the logical operator of the original condition, or to avoid additional conditions if they cannot be added.
4, Disable default rule
Although Falco provides a very powerful default rule set, you sometimes need to disable some of these default rules because they don't work properly in your environment. Fortunately, Falco offers you a choice:
4.1 through existing macros
Most default rules provide some consideration_ Macro. These considers_ Macros are usually set to (never_true) or (always_true) to enable or disable related rules.
Now, if you want to enable a rule that is disabled by default (such as outbound connection destination), you just need to override the rule's consider in the custom Falco configuration_* Macro (in this case, consider_all_outbound_conns).
Example of your custom Falco configuration (note (always_true) condition):
- macro: consider_all_outbound_conns condition: (always_true)
Note again that it is important to specify the order of configuration files! The last defined macro with the same name wins.
4.2 passing Falco parameters
Falco provides the following parameters to limit which default rules can be enabled or used and which cannot:
-D <substring> Disable any rules with names having the substring <substring>. Can be specified multiple times.
-T <tag> Disable any rules with a tag=<tag>. Can be specified multiple times.
Can not be specified with -t.
-t <tag> Only run those rules with a tag=<tag>. Can be specified multiple times.
Can not be specified with -T/-D.
If you deploy Falco through the official Helm Chart, you can also specify these parameters as Helm Chart values (extraArgs).
4.3 through custom rules
You can use a combination of the append: true and enabled: false attributes to disable a rule that was originally enabled by default. This is important for not providing a consider by default_* Macro rules are particularly useful.
Ensure that the custom profile is loaded after the default profile. Multiple - r parameters can be used to ensure the correct order (local installation), or in the falco configuration file falco Rules in yaml_ Ensure the correct order. If you are using the official Helm Chart, please use falco The rulesfile values are configured in the correct order.
For example, to disable / etc / Falco / Falco_ rules. For the User mgmt binaries default rule in yaml, see / etc / Falco / rules d/custom-rules. Define custom rules in yaml:
- rule: User mgmt binaries append: true enabled: false
remarks:
#We are investigating this feature. There seems to be an error. If enabling: false does not work, you can use the following workaround as an alternative: - rule: User mgmt binaries condition: and (never_true) append: true
5, Rule priority
Each Falco rule has a priority indicating the severity of the violation. Priority is contained in message/JSON output, etc. The following are the available priorities:
- EMERGENCY
- ALERT
- CRITICAL
- ERROR
- WARNING
- NOTICE
- INFORMATIONAL
- DEBUG
The allocation criteria of rule priority are as follows:
- If the rule is related to the write status (i.e. file system, etc.), its priority is ERROR.
- If the rule is related to unauthorized read status (i.e. reading sensitive files, etc.), its priority is WARNING.
- If the rule is related to unexpected behavior (generating an unexpected shell in the container, opening an unexpected network connection, etc.), its priority is NOTICE.
- If the rule is related to a violation of best practices (unexpected privileged containers, containers with sensitive mounts, running interactive commands as root), its priority is INFO.
- An exception is the rule "Run shell untrusted", which has FP prone and priority of DEBUG
6, Rule label
Starting with 0.6.0, rules have an optional set of labels for classifying rule sets into related rule groups. Here is an example:
- rule: File Open by Privileged Container desc: Any open by a privileged container. Exceptions are made for known trusted images. condition: (open_read or open_write) and container and container.privileged=true and not trusted_containers output: File opened for read/write by privileged container (user=%user.name command=%proc.cmdline %container.info file=%fd.name) priority: WARNING tags: [container, cis]
In this case, the rule "File Open by Privileged Container" is given the labels "container" and "cis". If the label key for the given rule does not exist or the list is empty, the rule has no label.
The following is how to use the label: (local installation)
- You can disable a rule with a given label using the - t parameter- T can be specified more than once.
For example, to skip all rules labeled "filesystem" and "cis", you can use falco -T filesystem -T cis- T cannot be specified with - T.
- You can use the - t parameter to run only rules with a given tag- T can be specified more than once.
For example, to run only rules with "filesystem" and "cis" tags, you can use falco -t filesystem -t cis- T cannot be specified with - t or - D (rules are disabled by rule name regular expressions).
6.1 label of the current Falco rule set
We also checked the default rule set and marked all the rules with an initial set of tags. Here are the labels we have used:
| Tag | Description |
|---|---|
| filesystem | This rule is related to reading / writing files |
| software_mgmt | This rule is related to any software / package management tool, such as rpm, dpkg, etc. |
| process | This rule is related to starting a new process or changing the state of the current process |
| database | The rule is database related |
| host | This rule applies only outside the container |
| shell | This rule is specifically related to starting the shell |
| container | This rule applies only within containers |
| cis | This rule is related to the CIS Docker benchmark |
| users | This rule relates to the user managing or changing the identity of a running process |
| network | This rule is related to network activity |
If the rule is related to more than one of the above, the rule can have multiple labels. Currently, each rule in the falco rule set has at least one label
7, Rule condition best practices
In order to allow rules to be grouped by event type to improve performance, Falco prefers to have at least one EVT at the beginning of the condition condition before any negative operator (i.e. not or! =) Type = operator. If the condition condition does not have any EVT Type = operator, Falco will record the following warning:
Rule no_evttype: warning (no-evttype):
proc.name=foo
did not contain any evt.type restriction, meaning that it will run for all event types.
This has a significant performance penalty. Consider adding an evt.type restriction if possible.
If the rule has EVT in the second half of the condition Type operator, Falco will record the following warning:
Rule evttype_not_equals: warning (trailing-evttype):
evt.type!=execve
does not have all evt.type restrictions at the beginning of the condition,
or uses a negative match (i.e. "not"/"!=") for some evt.type restriction.
This has a performance penalty, as the rule can not be limited to specific event types.
Consider moving all evt.type restrictions to the beginning of the rule and/or
replacing negative matches with positive matches if possible.
8, Escape special characters
In some cases, rules may need to contain special characters, such as (, spaces, etc. for example, you may need to look up proc. For (systemd) Name, including the surrounding parentheses.
Falco, like Sysdig, allows you to capture these special characters using "" to capture them. This is an example
- rule: Any Open Activity by Systemd desc: Detects all open events by systemd. condition: evt.type=open and proc.name="(systemd)" or proc.name=systemd output: "File opened by systemd (user=%user.name command=%proc.cmdline file=%fd.name)" priority: WARNING
When you include an item item in the list, make sure that the double quotation marks are not interpreted from the YAML file by enclosing the quoted string in single quotation marks. Here is an example:
- list: systemd_procs items: [systemd, '"(systemd)"'] - rule: Any Open Activity by Systemd desc: Detects all open events by systemd. condition: evt.type=open and proc.name in (systemd_procs) output: "File opened by systemd (user=%user.name command=%proc.cmdline file=%fd.name)" priority: WARNING
Take a closer look at the second line above. Items: [system D, "" (system d) ""], where "(system d)" "is quoted in single quotation marks.
9, Ignored system call
For performance reasons, some system calls are currently discarded before Falco processes them. This is the list:
access alarm brk capget clock_getres clock_gettime clock_nanosleep clock_settime close container cpu_hotplug drop epoll_create epoll_create1 epoll_ctl epoll_pwait epoll_wait eventfd eventfd2 exit_group fcntl fcntl64 fdatasync fgetxattr flistxattr fstat fstat64 fstatat64 fstatfs fstatfs64 fsync futex get_robust_list get_thread_area getcpu getcwd getdents getdents64 getegid geteuid getgid getgroups getitimer getpeername getpgid getpgrp getpid getppid getpriority getresgid getresuid getrlimit getrusage getsid getsockname getsockopt gettid gettimeofday getuid getxattr infra io_cancel io_destroy io_getevents io_setup io_submit ioprio_get ioprio_set k8s lgetxattr listxattr llistxattr llseek lseek lstat lstat64 madvise mesos mincore mlock mlockall mmap mmap2 mprotect mq_getsetattr mq_notify mq_timedreceive mq_timedsend mremap msgget msgrcv msgsnd munlock munlockall munmap nanosleep newfstatat newselect notification olduname page_fault pause poll ppoll pread pread64 preadv procinfo pselect6 pwrite pwrite64 pwritev read readv recv recvmmsg remap_file_pages rt_sigaction rt_sigpending rt_sigprocmask rt_sigsuspend rt_sigtimedwait sched_get_priority_max sched_get_priority_min sched_getaffinity sched_getparam sched_getscheduler sched_yield select semctl semget semop send sendfile sendfile64 sendmmsg setitimer setresgid setrlimit settimeofday sgetmask shutdown signaldeliver signalfd signalfd4 sigpending sigprocmask sigreturn splice stat stat64 statfs statfs64 switch sysdigevent tee time timer_create timer_delete timer_getoverrun timer_gettime timer_settime timerfd_create timerfd_gettime timerfd_settime times ugetrlimit umask uname ustat vmsplice wait4 waitid waitpid write writev
When running with - i, Falco prints the collection of ignored events / system calls and exits.
If you want to run Falco for all events, including the system calls in the list above, you can use the - A flag to run Falco.
Reference documents:
https://github.com/falcosecurity/falco
https://github.com/falcosecurity/falcosidekick