Data Feeds

This part of the configuration can be found in Settings → Processing → Data feeds.

Description

Flow data sources represent individual monitored points of the network and are one of the licensing restrictions (the number of simultaneously active Flow data sources). For each monitored point of the network, a Flow data source must be created in the plug-in.

Configuration

• Name: Unique data source name.

• State: The current state of the data feed.

• Location: Node of the Distributed Architecture to run the data feed. Each node must have a unique name. There is localhost only if the Distributed Architecture is turned off.

• Profile: Name of the profile which is used as input.

• FPS limit: Flows per second limitation for a given data feed. Flows that are received over this limit are not processed. Note that the real value of the FPS limit may be actually lower than the value set for the respective data feed due to license restrictions or technical limits (not enough resources, and so on). The maximal amount of FPS that can be processed on all active data feeds is restricted by the limit specified by the license being used. The license limits and the required appliance resources can be seen in the Flowmon ADS specification. The license limits are treated differently based on the subject of processing (for better understanding, see the example at the end of this page):

  • BPATTERNS signatures processing: The license limit is applied as a hard limit in 5-minute data batches. When the processing is started, the hard license limit is distributed to each active data feed equally. User-defined FPS limit allows adjusting this distribution

  • Stream processing of the other detection methods: The license limit is applied when the one-hour average value of flows per second exceeds the threshold. Therefore short flow bursts and traffic spikes will be processed, even if over the limit. User-defined FPS limit allows you to set a custom limit to avoid exceeding one hour average in the sum.

• Channels: Selection of channels that are used as input data for the application.

• Split channels into standalone data feeds: When creating a new data feed or editing the existing one it is possible to activate splitting of the channels in the selected profile into standalone data feeds. The data from each data feed is processed separately. Splitting channels into standalone data feeds allows the channels to be assigned to the instances of detection methods and have separate priorities. The maximum number of active data feeds is limited by license. If a data feed with enabled splitting into standalone data feeds is activated and the license does not allow data feeds to be activated for all selected channels, the data feed is activated partially. Information with a count of activated data feeds is shown on the data feed status page. The usage example of the standalone data feeds may be the following: An ISP can create data feeds per ASN group and analyze traffic anomalies for every group separately. Just create a new profile with channels for every ASN group, assign the profile to the ADS data feed, select all channels, and activate splitting into standalone data feeds.

• Deduplicate: If active, duplicate flows are skipped. The flows that are considered to be duplicates are those with the same 5-tuple (src IP, dst IP, src port, dst port, protocol) and overlapping time intervals (flow start, duration).

• SIP traffic processing: The switch between the Flow data processing and processing of Flow data enhanced with the SIP entries. It is impossible to process both (Flow data with SIP entries and Flow data without SIP entries) on the single data feed together. Only the detection methods with the "SIP" prefix are used if the SIP processing is active.

• Report wrong timestamp: Enables checking of incorrect time settings on the flow exporters. The timestamp of the incoming flows is compared with the current time and a warning is raised if there is a significant difference (see the Min delta parameter).

• Min delta: Specifies the minimal difference of time (in seconds) in the incoming flows and the current time to raise a warning.

• Sampling rate: Rate for sampling the input data.

• Active proxy: Enables the correlation that performs the replacement of two flows: inner flow (client → proxy) and outer flow (proxy → server) by one flow (client → server). Monitoring of the inner and outer flows is required for proxy correlation to work properly. Usually, the network needs to be monitored at two points: inside the local network (client side of the proxy server) and inside the external network (public side of the proxy server). This correlation allows some methods to perform detection that otherwise would not be possible in the network with a proxy server. However, the correlation is only applied on paired flows (biflows) and the processing of all correlated flows may be delayed for a few seconds (up to 70 seconds in edge cases). Within the data feed configuration, it is possible to set up the tolerated data amount difference between the two particular flows that have to be correlated (Maximum data difference) and the count of milliseconds that could take the outer flows longer (Request prolongation, Response prolongation). It is necessary to set up the IP addresses of the outer (External IP) interface, inner (Internal IP) interface, the proxy server’s listening port (Internal Port), and the proxy clients (Clients Filter). It is possible to define multiple proxy servers for each data feed. The maximum count is limited to 20 per data feed.

Example of the FPS limit distribution

Flowmon ADS Ultimate license limit is 100.000 flow/s (one hour average) for stream data processing and 15.000 flow/s (hard limit) for behavior patterns processing.

Let's define two data feeds. First with an FPS limit of 1.000, and second without any FPS limit. Stream data processing will process up to 1.000 FPS from data feed 1. Data feed 2 is unlimited by the user but the sum of FPS (one hour average) from both data feeds cannot exceed the license limit. One hour average FPS on data feed 2 will be up to 99.000 FPS. Behavior patterns processing will process up to 1.000 FPS on data feed 1 and up to 14.000 FPS on data feed 2 due to the hard license limit.

Let's define three data feeds with FPS limits: 1.000, 20.000, and 90.000. Stream data processing will process up to 1.000 FPS from data feed 1. up to 20.000 from data feed 2 and up to 90.000 from data feed 3. Sum of FPS (one hour average) from all data feeds cannot exceed the license limit. Behavior patterns processing will process up to 1.000 FPS from data feed 1 and up to 7.000 FPS from data feeds two and three due to equal distribution of the rest of the limit.

To provide the best results possible, the Flowmon ADS module requires the flow exporters to set the proper values of the active and inactive timeout. Details on configuring the exporters can be found in the Flowmon appliance documentation. The granularity of flows impacts the accuracy of detection methods. To reduce the number of flows that are generated by the exporter, the following values are appropriate (Flowmon appliances have these values set by default):

• active timeout – 300 s

• inactive timeout – 30 s

Actions

• Assign to detection methods: Assigns data feed to all detection methods. Assigning to a specific method can be also done manually in Methods configuration.