Blacklists

This configuration can be found in Settings → Blacklists.

This page allows you to manage the blacklists that are used by the detection method BLACKLIST. There are two types of blacklists. These are described in the following sections.

Flowmon blacklists

This type of blacklist is directly maintained by the Progress Software company. Blacklists of this type are periodically (every 6 hours) updated from the Flowmon services portal if the Flowmon device is connected to the Internet. The particular Flowmon blacklists are designed for the detection of the following malicious activities:

• Communication with IP addresses of known attackers (AttackerActivities blacklist)

• Communication with IP addresses of known botnet command and control servers (BotnetActivities blacklist)

• Communication with IP addresses of known SPAM sources (SpammerActivities blacklist)

• Communication with known malware domains (MalwareDomains blacklist)

• Communication with known botnet domains (BotnetDomains blacklist)

• Communication with known phishing domains (PhishingDomains blacklists)

Custom blacklists

It is also possible to add custom blacklists whose content is then used by the BLACKLIST detection method to detect communication with specified IP addresses, domains, or services at the specified IP addresses. The source of custom blacklists may be a local file (Local custom blacklists - possible to add by clicking New local blacklist) or a URL to the file at a remote server (Remote custom blacklists - possible to add by clicking New remote blacklist). In both cases, the files must be in CSV format and their content must correspond with exactly one of the following types.

Host (IP address)

When communication with any of the provided IP addresses is detected, the BLACKLIST event is generated. Individual entries in this blacklist type consist of two items:

• IP address: Only the IPv4 addresses are supported.

• Comment: Consists only of ASCII characters. This item is optional and its maximum length is limited to 256 characters.

Web or domain (hostname, path, comment)

This type of custom blacklist is designed to detect communication with a specified domain and a path within this domain. The BLACKLIST event is generated when the domain and path detected in a flow match any entry in the specified blacklist. One entry in this type of blacklist consists of a triplet with the following items, separated by a comma:

• Domain name (hostname): Consists only of ASCII characters. The maximum length is limited to 63 characters for detection from DNS queries and to 31 characters for detection from the HTTP host field.

• Path within this domain (path): Consists only of ASCII characters. The maximum length is limited to 63 characters. When there is a slash symbol is inserted instead of a valid path, any path within the domain is considered to be a positive match.

• Comment: Consists only of ASCII characters. This item is optional and its maximum length is limited to 256 characters.

Service (IP address, port, protocol, comment)

This type of custom blacklist is designed to detect the usage of specified services at a specific IP address. One entry in this type of blacklist consists of four items, separated by a comma:

• IP address: Only the IPv4 address type is supported.

• Port: An integer number in the range 0 - 65535.

• Protocol: String with value TCP, UDP, or ANY.

• Comment: Consists only of the ASCII characters. This item is optional and its maximal length is limited to 256 characters.

Fingerprint (JA3 fingerprint, comment)

This type of custom blacklist is designed to detect network applications with specific JA3 fingerprints. These fingerprints are used to identify network communication of particular applications (web browsers, remote access software, and so on). JA3 fingerprints may be also used for the detection of malicious activity (for example, malware operating in the network). One entry in this type of blacklist consists of two items, separated by a comma:

• JA3 fingerprint: Text string with a length equal to 32 ASCII characters.

• Comment: Consists only of the ASCII characters. This item is optional and its maximal length is limited to 256 characters.

Application

This type of custom blacklist is designed to detect communication with IP addresses where a blacklisted application is hosted. Blacklisted applications are specified directly using the "New local blacklist" form, without the need for a blacklist file. You can export/import the application blacklist using the configuration file.

Adding a new custom blacklist

It is possible to add a new custom blacklist using the New remote blacklist or New local blacklist button based on the source of a blacklist.

After a new custom blacklist is added, you must assign it to some BLACKLIST method instance so it can be properly evaluated by the detection method. You can do this by clicking Assign in the row with a custom blacklist and choosing a method instance that the blacklist should be assigned to. This is also possible in the configuration of the BLACKLIST method using the ActiveBlacklists parameter which is described in the section BLACKLIST – Communication with blacklisted hosts.

The content of a custom blacklist can be displayed by expanding a row with a particular blacklist and clicking Blacklist content.

Local blacklist

To add a local blacklist you must fill in the following fields:

• Blacklist name: Unique name of a custom blacklist.

• Description: Description of a custom blacklist. This description is used in the detail of the BLACKLIST detection method. This parameter is optional. If a description is not provided, the Blacklist name is used in the event detail.

• Assigned instances: Instances of the BLACKLIST method to which the blacklist is assigned.

• File to upload: Local file that is a source of the blacklisted content.

• Data format: Type of a custom blacklist (these types are mentioned above).

It is possible to view the content of a local blacklist in the user interface immediately after the blacklist is added.

Remote blacklist

The remote blacklist is divided into two types:

• CSV: Blacklist entries specified by a CSV file. The correct format of CSV files is described at the beginning of this chapter.

• MISP: Blacklist entries generated from the MISP (Open source threat intelligence platform and open standards for threat information sharing).

To add a remote blacklist it is necessary to fill in the following fields:

• Blacklist name: Unique name of a custom blacklist.

• Description: Description of a custom blacklist. This description is used in the detail of BLACKLIST detection method. This parameter is optional. If there is no description, the Blacklist name is used in the event detail.

• Assigned instances: Instances of the BLACKLIST method to which the blacklist is assigned.

• Type of blacklist: Type of the remote blacklist.

The options are further expanded based on the type of the blacklist you have chosen.

CSV

Configuration fields:

• Remote URL: Remote URL that is a source of the blacklisted content.

• Data format: Type of the custom blacklist (these types are mentioned above).

It is possible to view the content of a remote csv blacklist in the user interface immediately after the blacklist is added. If the content is empty, it may be caused by an error during the download process which will be reported in the notification area (bell icon).

MISP

It is possible to use a remote MISP server as a blacklist. The conversion process and blacklist life cycle are described below.

Configuration Fields:

• Remote URL: Remote URL of the MISP server.

• API key: API key used to authenticate to the provided MISP server.

• Maximum days of valid records: Integer number between 1 and 365. This number specifies the maximum age of the records in the MISP feed that should be downloaded.

• Include records only for Intrusion Detection System: Only download records marked as "only for Intrusion Detection System" on the MISP server.

The processing of the remote MISP blacklist should start immediately after being added. If some MISP blacklists are currently being updated when adding a new one, the blacklist will be processed during the next update process (up to 30 minutes). The initial processing may take a long time depending on the server speed and number of records. The content of the blacklist should be visible immediately after the completion of the processing. If the content is empty, it may be caused by an error during the download process which will be reported in the notification area (bell icon).

The conversion process

The data downloaded from the MISP server is converted to the Flowmon supported blacklist formats - Host, Web or domain, Service, and Fingerprint.

For the conversion we only need to fetch MISP attributes with following types:

• domain

• domain|ip

• hostname

• hostname|port

• ip-dst

• ip-dst|port

• ip-src

• port

• url

• uri

• ja3-fingerprint-md5

Types ip-dst and port with matching object id are aggregated, creating ip-dst|port pairs.

Attributes are later used to create four different blacklist formats:

• Web or domain

  • Attribute types used to generate this blacklist format:

    • domain

    • domain|ip (only the domain is used)

    • hostname

    • hostname|port (only the hostname is used)

    • url

    • uri

  • A comment is created from the attribute id and corresponding event id

• Service

  • Attribute types used to generate this blacklist format:

    • ip-dst|port

  • A comment is created from the attribute id and corresponding event id

• Host

  • Attribute types used to generate this blacklist format:

    • ip-src

    • ip-dst

• Fingerprint

  • Attribute types used to generate this blacklist format:

    • ja3-fingerprint-md5

  • A comment is created from the attribute id and corresponding event id

Blacklist life cycle

Upon creation of the blacklist, the system will periodically (every 30 minutes) download updates from the server, ensuring up-to-date records without any need for user interaction.

The download process of the records from the MISP API may take a long time depending on the server speed and number of records. Due to this behavior, we implemented a caching system, which allows us to only download new attributes, speeding up the fetching process. However, during these updates, we don't receive information about deleted attributes. To avoid issues caused by this behavior, we delete the cache after 24 hours resulting in a re-download of all records within the specified age limit. This also ensures that we only hold records for the time period specified by the parameter Maximum days of valid records entered when adding a new MISP blacklist.

The cache is also removed when the configuration of the blacklist changes.

Management of blacklists

It is possible to filter blacklists according to their name or description at the top left corner of the blacklist settings page. It is also possible to choose which blacklists should be displayed (based on their origin) with the checkboxes in the top-right corner. To edit the already existing blacklists it is possible to use the Edit button. It is also possible to delete any custom blacklist with the context menu that is activated by the three dots icon at the end of a custom blacklist row. When there is an orange icon with a symbol of gear with an exclamation mark displayed on the left of the blacklist name, the blacklist is not assigned to any instance of the BLACKLIST method. This means that the blacklist is not processed.