Filter bot

[sebix]

During a workshop we collected some feedback on needed filtering possibilities. We now came up with this list of useful comparisons. Could be implemented by an abstract base class and derived specific classes.

I used pseudo code for easier reading.

Generic filter

The filter bot shall support these generic filtering comparisons:

Match any key in the event against a value with these comparison operators:

• ==
• <,>
• <=,>=
• !=
• in, e.g. source.asn in [1234, 1235]
• ~=, matching: extended regex, with re.search (ENH: Filterexpert can now use RegEx #676)
• exists/is not None
• not exists/is None

Ip specific filtering

In particular, the filter bot shall have a derived class "IPAddressFilter" which can compare:
source.ip, destination.ip, source.network, destination.network against a value with these comparators:

• ip = '192.0.2.1'
• ip in '192.0.2.1-192.0.2.10'
• ip in ['192.0.2.1', '192.0.2.2']
• ip in '192.0.2.0/24'
• ip in ['192.0.2.1-192.0.2.10', '192.0.2.100/28']
• for source.ip and destination.ip

Time specific filtering:

Again this is a derived class.
Compare time.source, time.observation via:

• <,>
• <=,>=
• ==
• !=
• absolute
• relative

comments appreciated

EDIT 2016-07-18: Added exists
EDIT 2016-09-07: ip-filtering: different entries in lists

[ondj]

Because not every message has every field, "field exists" and "field does not exist" filters are necessary (or handle not exists fields as null).

And then at least and, or and not operators are useful for advanced filtering.

[aaronkaplan]

On 18 Jul 2016, at 14:28, ondj [email protected] wrote:

Because not every message has every field, "field exists" and "field does not exist" filters are necessary (or handle not exists fields as null).

And then at least and, or and not operators are useful for advanced filtering.

Agreed.

EDIT 2018-06-04 @wagner-certat: fix citation syntax

[dmth]

An enhancement of the filters would be great. Do these ideas also include a "choice of output queues", for example: if ip in 192.0.2.1/28 then my-special-expert-queue else normal-expert-queue?

Can this be realised?

[sebix]

Can this be realised?

Currently not, but definitely on the wishlist.

[dmth]

FYI: I'm going to start working on the RegEx thing, later it is going to be extended, in order to be capable of understanding the extras-JSON.

[sebix]

The extras-Field should actually be a native dict (or similar) and its JSON-representation should just be used for outputs and in the message queue. The usage of json for extra in intelmq itself is a shortcoming in the current version.

[dmth]

See #676 for RegEx in the Filter Expert.
The interpreting of JSON ist currently paused.

[dmth]

Idea: What about using Conditioned Pipelines additional to filters?
This might solve my requirement stated in #569 (comment)

Each destination pipeline has an entry-condition which has to be met before an event is inserted into the pipeline. Maybe you can imagine it like a bouncer in front of a club. The default condition for each pipeline is true, so every event can get into the pipeline.

How might this look in a pipeline.conf file?

    "parser": {
        "source-queue": "parser-queue",
        "destination-queues": {
            "expert1-queue": "if event.get(\"source.cc\") = \"DE\""
            "expert2-queue": "if event.get(\"source.cc\") != \"DE\""
        }
    },
    "expert2": {
        "source-queue": "expert2-queue",
        "destination-queues": {
            "another-expert-queue": true
            "yet-another-expert-queue": true
        }
    },

Problems:

1. This allows the execution of arbitrary code from the *.conf files, if a programming language like python is used instead of a rule-language.

Questions:

1. What happens to bounced events?

[ghost]

What happens to bounced events?

What are bounced events?

[dmth]

What are bounced events?

Bounced events are those events which were not fit to get into the destination pipeline.

[ghost]

An enhancement of the filters would be great. Do these ideas also include a "choice of output queues", for example: if ip in 192.0.2.1/28 then my-special-expert-queue else normal-expert-queue?

Can this be realised?

Named destination queues are in develop branch, for support in filter expert see #1208

[ghost]

I created a new issue for the Conditioned Pipelines, so this issue can focus on the filter bot's capabilities itself as existing solution

[ghost]

the sieve bot can cover all of them. != can be done with paths -> Closing here. If you think that something should still be implemented, please reopen here or another issue.