FMC Configuration

This page contains details about the Flowmon Monitoring Center (FMC) settings and is further divided into six sections, each for particular part of the FMC.

Built-In Collector

You can set up the built-in collector on the FMC Configuration - Built-In Collector page.

Use this page to perform configuration changes to the built-in collector. Click Start/Stop to start/stop the built-in collector. You can see the collector status on this button (Running/Stopped). You will not be able to access the Flowmon Monitoring Center if the built-in collector is stopped.

If there are some queries running in the Flowmon Monitoring Center, a button showing their count will appear. In some cases, very complicated queries on a large amount of data may take a very long time and slow down the device. It may be useful to kill these queries by pressing Kill running X FMC queries.

When the Enable IP indexing option is enabled, Flowmon starts building an index of IPv4 addresses present in incoming flows. The index can be then used to accelerate filter queries for IPv4 addresses. The index is only available for flows from the moment the indexing option was enabled. If you need to recalculate data from the past, contact our Support team. See Types of Analysis for details on how to use the IP index to accelerate your filter queries.

The Clear data storage button is used to clear the built-in collector database. This operation irreversibly removes all stored NetFlow data. Depending on the size of stored NetFlow data this operation can take several minutes. During this time the Monitoring center will not be accessible.

Built-in Collector - Flow database fields

On the Flow database fields page you can select which values are to be stored to the flow database and processed in the Flowmon Monitoring Center. Selected values must be present in the exported flows from the probe or router. If not, they are filled with zeroes. Keep in mind that enabling a new values increases the disk space necessary to store new flows. A description of the fields can be found on the following page: Flow Database Fields.

Built-in Collector - Sources settings

On the Sources settings page, the limit for the number of profiled sources and their interfaces can be configured. See the Sources chapter for more information.

Built-in Collector - Listening ports (Collector only)

In this page you can configure the listening ports for NetFlow, IPFIX, sFlow, and other supported flow protocols and their forwarding. The listening port is defined by its name, port, network protocol, and flow protocol. Select the flow protocol used by your flow exporting device (router, probe). There are two options: NetFlow/IPFIX or sFlow. The option NetFlow/IPFIX applies also for all NetFlow clones like jFlow, NetStream, and so on. Contact support at https://support.flowmon.com for more information about the supported protocols.

There is no need to define different listening ports for individual flow sources (Probes, routers, and so on) because Flowmon automatically recognizes and configures the individual flow sources. It is recommend to keep the default settings of listening ports unless there is a specific reason for defining an additional listening port.

A new listening port can be added by clicking New listening port. A new form appears.

Enter the name of listening port, port number, network protocol, and flow protocol. If NetFlow/IPFIX is selected as flow protocol, TCP or UDP can be selected as network protocol. If TCP is selected, only IPFIX protocol is supported. For sFlow, only UDP is supported.

If TCP is selected as network protocol, the encryption TCP/TLS can be enabled. For TCP/TLS, the set of keys and certificates have to be generated for the flow exporting device (monitoring port) and for collector. All certificates must be signed by the same certification authority (CA). Its certificate (CA certificate) must be provided together with the collector key and certificate to each listening port using TCP/TLS protocol.

The sampling rate of received flow data is determined from the flow protocol. It can be also defined statically (available for NetFlow only). For this purpose, check the Define source sampling rate and enter the number. If the entered value is positive, it is used only if the flow monitoring port does not provide the sampling rate information. If it provides the sampling rate, then this value is used. If you want to enforce your sampling rate, enter it as negative value.

Normally the start time and end time of each flow is generated by the flow source and included in the flow data. However, some flow sources are not able to generate flow times and the flows are exported with no information about start time and end time. In this case, the Flowmon Collector can generate the times itself based on the time of flow reception and active timeout defined on the flow source. The times are generated as follows:

t start = reception_time − active_*timeout

t* end = reception_time

The generated times are only indicative. For long term flows, where active timeout applies, the flow duration is correct. The start time and end times are delayed due to the time between ending the flow on the flow source and its reception on the Flowmon collector. For short flows where active timeout does not apply, the flow duration will be wrong. To enable this feature, enable the Generate missing timestamps switch and provide the Active timeout of the flow source sending data to this listening port.

Modify flow timestamps by flow receive time - this feature fixes incorrect flow timestamps by replacing the flow end-time with the flow receive time and computing the flow start-time as the flow receive time minus the flow duration.

Received flow data can be forwarded to multiple different targets. For this purpose, use the Forwarding targets selector to choose forwarding targets for the Listening ports. The Forwarding targets must be configured in the Forwarding targets page.

Built-in Collector - Forwarding targets (Collector only)

This section enables the configuration of targets of forwarding of the listening ports. The configured forwarding targets are shown in the table below. Click New target or the Edit icon in the Action column to create a new forwarding target or to edit an existing one. This forwarding target will be applied to all listening ports selected in the Listening ports selector at the bottom of the page. Forwarding can be performed in two modes: Compatible mode and Advanced mode. These are available in separate tabs.

Forwarding mode - compatible

This mode allows flow forwarding using the UDP protocol with a spoofed IP address of the flow source. This mode is compatible with all Flowmon collectors and third party collectors. In compatible mode, the original IP address of flow source is preserved (that is, IP spoof mode), so the target collector assigns the flows to the IP address of the original flow source. Keep this in mind when configuring firewall rules, and so on.

In compatible mode, enter the IP address of the collector and the UDP port.

Forwarding mode - advanced

This mode allows flow forwarding using advanced capabilities such as TCP or TCP/TLS export, flow protocol conversion, flow sampling, and flow filtering. This mode is compatible with Flowmon collectors v9.01.00 and higher.

In advanced mode there are two tabs - Export target and Export protocol.

In Export target tab, enter the IP address of target collector, port, flow sampling rate and choose the transport protocol. TCP protocol is allowed only when IPFIX is used as an export protocol (see the Export protocol tab). Moreover, the export filter can be added to define what flows will be forwarded to this target. For the filter syntax, see the Syntax of Filter of Monitoring port section. If the TCP protocol is selected, the flow data can be forwarded encrypted using the TCP/TLS protocol if the option Enable encryption is enabled. Then the collector private key, collector certificate and CA certificate must be provided.

In Export protocol tab, the flow export protocol can be selected out of the options NetFlow v5, NetFlow v9, and IPFIX. For NetFlow v9 and IPFIX there is an option to change default template re-sending intervals.

Click Save to apply changes. The entered values are checked for loop presence which can be fatal for the collector. This operation can be more time consuming.

Reports' settings

Reports settings consists of Basic settings, Remote storage, Working hours, and Branding.

Basic Settings

In basic settings, you can disable or enable reporting functionality. If you disable reporting, schedules from Dashboards and Reports will stop being sent (email and Samba). You are also allowed to recompute all chapters at once. Pick the desired time interval and then click Recompute. Progress of jobs computing shows, how many tasks are computed and how many tasks are waiting. The Reserved CPU value says how much CPU performance can be used to compute chapters statistics (done every hour). The Allow sampling for large amounts of data option is enabled in the default configuration and allows the system to sample flow data during computation of reports if the amount of data is very big. So it speeds up the computation significantly and saves a lot of resources on heavily loaded collectors. The precision of computed statistics is decreased only a little because for large amounts of data the sampled data is statistically unimportant. To save new value, click Save.

Working Hours

Here, you can set your company working hours. Then you will be able to reflect these settings in reports where statistics are computed over these values.

• Name - enter name for this entry.

• For interval - pick times "from" and "to" up to four times. Usually only two intervals are used, with a gap for lunch time.

• On days - select which days these time slots are active on.

Remote Storage

In the Remote storage section, parameters for storing reports to remote storage can be configured. Enter the Report directory, where the reports will be copied. The Copy timeout is used for specifying the maximum time for copying of a single report. If the copy transaction takes longer, it is interrupted as unsuccessful. Use the value zero for setting unlimited time. The Delete not copied files after option is used for configuration of the maximum interval in days, when the older reports are removed from the queue and the system will not attempt to try to to copy them again. Use the value zero for setting unlimited time.

Branding

In Branding you can specify the look of generated PDF reports. You can select the main color, report name, email report subject, and body. You can use macros here (described on the panel).

Active Devices

This page is used to configure the Active Devices monitoring functionality. The function is enabled by default, but it is not available for Distrubuted Architecture configurations. To disable/enable it, you must use the Enable active devices logging toggle switch and click Save.

Pick monitored flow sources from the selection menu. Only data from these sources are collected. You can also specify a filter if you want to monitor only specific traffic. Click Save to save changes.

The Identify by field specifies the default device identifier in FMC and is used to aggregate your query results. The identifier can be temporary switched to a different one when building a query at any time, using the aggregation options.

Database remote connection settings

A remote connection to a PostgreSQL database can be configured by clicking Remote connection settings. If you do not need to connect remotely to the database, we recommend rejecting all external connections to it. This can be done in the Remote access page in the Active firewall rules panel - look for Postgres rule (port 5432).

Below, there are two inputs that allow you to change the password for the remote user of the PostgreSQL database. Enter the original password in the Current password field and enter the new password in the New password field. Click Save to perform the change. A message appears if the change was successful.

For remote access to a database, use the server address and port 5432. The user login is ipmac_cache_ro and the default password is inv3a-t3ch. Tables are stored in the ipmac_cache schema. This user is allowed and has read-only permissions.

Active devices - IP ranges

The IP ranges table is used to configure all subnets where the active devices are to be monitored. It makes sense to collect these in the local network and therefore there are preset values for all private and local networks: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fe80::/10. To add a new subnet, simply enter this value in the form: IP address/mask.

Active devices - Routers

The Routers table is used to manage MAC addresses of routers that are hidden in reports by default because they have normally assigned a large number of IP addresses. You can change your preference to show them in the FMC in the search form.

AWS Flow Logs Converter

What is the AWS Flow Logs Converter?

The AWS Flow Logs Converter is a configurable module of the Flowmon Monitoring Center (FMC).

It enables the user to collect, process, and visualize AWS VPC Flow Logs (further referred to as flow logs) which contain information about the traffic captured in Amazon Virtual Private Cloud.

Brief Implementation Description

The flow logs are periodically acquired from Amazon CloudWatch, processed, converted to IPFIX format, and subsequently sent to the Flowmon Collector to a defined UDP port.

The Flowmon Collector treats data from this port the same as regular flows recovered from any other port.

Setting Up Flow Logs for a VPC

To set up the flow logs in your cloud and forward them to AWS CloudWatch, follow the instructions specified here: Publish flow logs to CloudWatch Logs. It is important that every flow log stream contains flow logs from one interface only.

The AWS Flow Logs Converter can process TCP flags which are not enabled in the AWS VPC Flow Logs by default. To enable processing of TCP flags, you must specify a custom format of the Log Record when creating a new Flow Log.

The custom format must contain the following fields in the following order:

The AWS Flow Logs Converter can process only the default Flow Record format and the custom format specified above.

Setting Up Flow Logs in Flowmon Configuration Center

To start receiving flow logs in the Flowmon Monitoring Center, follow these instructions:

Step 1: Create a new listening port in Configuration Center -> FMC Configuration -> Listening Ports

The name and port number of the new listening port can be chosen as needed. However, the network protocol must be UDP and the format of the transferred data must be NetFlow/IPFIX.

Step 2: Configure the access information, regions and log groups from which the flow logs will be retrieved.

Configuration Center -> FMC Configuration -> AWS Flow Logs

The access key ID and the secret access key are mandatory credentials provided by Amazon.

Select the previously configured listening port.

Click Add Region to configure the endpoints where the flow logs should be retrieved.

Insert the name of the region (without the availability zone) where your flow logs are physically stored. A list of all possible regions can be found here: Regions, Availability Zones, and Local Zones. Note that the region Name field is expected to contain values like eu-central-1 rather than EU (Frankfurt). It is also possible to define a short description of the region.

Lastly, it is necessary to provide at least one log group (by clicking Add group and filling in the name). All flow log streams in the provided group are processed and every stream is shown as a unique interface of the log group in the Monitoring Center.

The provided configuration can be optionally verified by clicking Verify. This checks if the FMC is able to connect to the specified log groups using the provided AWS credentials.

Note that the provided configuration undergoes the verification process every time you click Save.

Newly created configurations must be saved (by clicking Save). This starts the process of retrieving the Flow logs. To stop the process of retrieving, disable it, and click Save.

Viewing VPC Flow Logs in the Monitoring Center

It can take up to 20 minutes (see the limitations) before the first flow logs can be visualized.

Every log group has internally assigned a unique IP address (from the subnet 127.128.0.0/16) and is treated as a unique flow source.

All sources can be found in Flowmon Monitoring Center -> Sources.

Click Profile to see traffic of the individual streams.

Select all available streams and click Save.

Switch to: Flowmon Monitoring Center -> Profiles -> Sources -> Your Log Group

It is possible to view and analyze flows from flow logs as if they were flows from regular data sources.

Limitations of Flow Logs

There are some limitations which stem from the flow logs themselves that need to be taken into account:

• If your network interface has multiple IPv4 addresses and traffic is sent to a secondary private IPv4 address, the flow log displays the primary private IPv4 address in the destination IP address field.

• If traffic is sent to an ENI and the destination is not any of the ENI IP addresses, the flow log displays the primary private IPv4 address in the destination IP address field.

• If traffic is sent from an ENI and the source is not any of the ENI IP addresses, the flow log displays the primary private IPv4 address in the source IP address field.

• If traffic is sent to or sent by a network interface, the flow log always displays the primary private IPv4 address, regardless of the packet source or destination, in the interface IP address field.

Flow logs do not capture all IP traffic. The following types of traffic are not logged:

• Traffic generated by instances when they contact the Amazon DNS server. If you use your own DNS server, then all traffic heading to that DNS server is logged.

• Traffic generated by a Windows instance for activation of the Amazon Windows license.

• Traffic to and from 169.254.169.254 for the instance metadata.

• Traffic to and from 169.254.169.123 for the Amazon Time Sync service.

• DHCP traffic.

• Traffic to the reserved IP address for the default VPC router. For more information, see VPC and Subnet Sizing.

• Traffic between an endpoint network interface and a Network Load Balancer network interface. For more information, see VPC Endpoint Services (AWS PrivateLink).

• Some flow log records might get skipped during the capture window. This may be because of an internal capacity constraint, or an internal error.

Furthermore, the delay between the time when the traffic actually occurred and the time it can be seen in the Monitoring Center can reach up to 20 minutes in the worst case scenario. However, the delay will get smaller with a higher amount of traffic volume present in the monitored cloud.

This is caused by the 10-15 minute capture window in which the packets are aggregated to the flow logs before being published, and by the subsequent 5 minute delay before the Flowmon Collector closes the current profile and shows the traffic in the GUI.

The Flowmon Collector stores incoming flows to a currently opened profile, and therefore it is advised to select multiple adjacent profiles when searching for flows in a particular time.

Google Cloud Flow Logs

Flowmon Collector is capable of processing and visualizing Google Cloud VPC Flow Logs. Google Cloud VPC Flow Logs (further referred to as flow logs) are records of network connections between VM instances in VPC networks. The Flowmon Collector acquires flow logs by polling on the Google Cloud Pub/Sub subscription.

Setting Up Google Cloud VPC Flow Logs

Follow the official instructions to enable generating flow logs for certain subnets in your VPC.

It is important to mention several configurable options during the configuration of flow logs:

• Aggregation Interval: 5 minutes - recommended (standard configuration of Flowmon probes also use the 5-minute aggregation interval)

• Include metadata: On - mandatory (necessary to display information about the VPC and subnets in FMC)

• Sample Rate: 100 - recommend in order to obtain all flow logs

Configuring Google Logs Router Sink

Follow the official instructions to configure logs router sink and Pub/Sub topic.

These configuration options might be helpful:

• Message retention duration should be opted out when creating Pub/Sub topic, because some form of flow logs retention will be done in the Pub/Sub subscription.

• It is preferable to provide an inclusion filter. In the Choose logs to include in sink panel, specify that you want to include only flow logs. This will increase the performance of processing of flow logs and decrease the overall price:

  • logName=~"/logs/compute.googleapis.com%2Fvpc_flows" - to see all flow logs

  • logName="projects/<project_name>/logs/compute.googleapis.com%2Fvpc_flows" - to see flow logs only from a specific project

Configuring the Google Cloud Pub/Sub Subscription

The Google Cloud Pub/Sub subscription must follow certain criteria so it can be utilized efficiently by Flowmon Collector.

The recommended configuration of a subscription to maximize the performance and minimize the cost:

• Delivery type: Pull - mandatory

• Message retention duration: 1 hour

• Retain acknowledged messages: No

• Acknowledgement deadline: 10 seconds

• Message ordering: No

• Dead lettering: No

• Retry policy: Retry immediately

The Flowmon Collector uses the Google Cloud Service Account Key (in JSON format) for authentication when acquiring flow logs from the Google Cloud Pub/Sub subscription. The service account used for acquiring flow logs must include the Pub/Sub Subscriber role in Google Cloud IAM. Note that such service account can access any Pub/Sub subscriptions with a Google Cloud project. For more information about setting up permissions, please refer to the official guide.

Setting Up Google Cloud VPC Flow Logs Processing

To start receiving flow logs in the Flowmon Monitoring Center, follow the following instructions:

Step 1: Create a new listening port in Configuration Center -> FMC Configuration -> Listening Ports

The name and port number of the new listening port can be chosen as needed. However, the network protocol must be UDP and the format of the transferred data must be NetFlow/IPFIX.

Optionally, you can define the source sampling rate of this listening port, because Google Cloud already samples packets that leave and enter a VM to generate flow logs. Not every packet is captured into its own log record. About 1 out of every 10 packets is captured, but this sampling rate might be lower depending on the VM's load. You cannot adjust this rate.

Step 2: Enable processing of the Google Cloud Flow Logs and configure individual subscriptions.

Navigate to: Configuration Center -> FMC Configuration -> Google Cloud Flow Logs

Toggle the Enable button and select the previously created Listening port from the drop-down menu.

Click New Subscription which allows you to configure a list of Google Cloud Pub/Sub subscriptions from which flow logs will be obtained and processed. The following parts of a subscription can be configured:

• Subscription ID - ID of the Google Cloud Pub/Sub subscription

• Project ID - ID of the Google Cloud project to which the subscription belongs

• Service account credentials - Google Cloud Service Account Key in JSON format, with permissions to subscribe to the Pub/Sub subscription. Follow the official instructions to create the key.

• Description - custom description of the subscription

• Advanced Configuration - several options which can affect performance of the subscription process at the cost of increased resources consumption

  • Max. messages in backlog - the maximal number of Pub/Sub messages which can be in queue for processing (not recommended to set below 1000 messages).

  • Max. megabytes in backlog - the maximal number of bytes which can be in queue for processing (it is recommended to respect the size of messages containing flow logs - not more than several KB per message)

  • Max. messages processed simultaneously - number of parallel background workers for polling flow logs from the Pub/Sub subscription. It is recommended to set this value as low as possible based on the expected number of the processed Pub/Sub messages per second. The range is limited to 2 - 16 possible workers (it is recommended to use a power of 2). Two workers can handle processing around 100,000 Pub/Sub messages per second (tested on a c2-standard-16 computing instance). Keep in mind that configuring several subscriptions on the same appliance lowers the performance in general. It is not recommended to use more than 32 background workers in total across all configured subscriptions.

The provided configuration can be optionally verified by clicking the Verify button. This will check whether the FMC is able to connect to the specified Pub/Sub subscriptions using the provided Service account credentials.

Note that the provided configuration undergoes the verification process every time the Save button is clicked.

Viewing VPC Flow Logs in Monitoring Center

Multiple flow sources are created when using Google Cloud VPC Flow Logs. Each flow source is internally assigned a unique IP address (from subnet 127.129.0.0/16) and its name corresponds to a VPC inside a Google Cloud project in a format: vpc-name.project-id.

All sources can be found in Flowmon Monitoring Center -> Sources.

Click Profile if you want to divide the flow source into a separate channel. Each channel corresponds to a subnet inside the VPC and is uniquely distinguishable by the subnet name.

Select all available subnets and click Save.

It is possible to view and analyze the flows from the flow logs as if they were flows from regular data sources.

Azure Flow Logs

The Flowmon Collector is capable of processing and visualizing Azure NSG Flow Logs. Azure NSG Flow Logs (further referred to as flow logs) are sampled records of the network flow sent from and received by VM instances. Flow logs is a feature provided by the Network Watcher service and is dependent on the Microsoft Insights resource provider. The Flowmon Collector periodically connects to the configured Azure Blob Storage containers and downloads newly added flow logs. The flow logs are subsequently converted to the IPFIX format and can be viewed in the Flowmon Monitoring Center (FMC).

Setting Up Azure NSG Flow Logs

Follow the official instructions to enable collecting of flow logs in Azure Blob Storage for your virtual machines.

Setting Up Azure NSG Flow Logs Processing

To start receiving flow logs in FMC, follow these instructions:

Step 1: Create a new listening port in Configuration Center -> FMC Configuration -> Listening Ports

The name and port number of the new listening port can be chosen as needed. However, the network protocol must be UDP and the format of the transferred data must be NetFlow/IPFIX.

Step 2: Enable processing of the Azure NSG Flow Logs and configure individual subscriptions.

Navigate to: Configuration Center -> FMC Configuration -> Azure Flow Logs

Toggle the Enable button and select the previously created Listening port from the drop-down menu.

Click New Subscription which allows you to configure a list of subscriptions. This list specifies which flow logs will be obtained and processed. For the Flowmon Collector to access the flow logs, it requires the URL of the Shared Access Signature (SAS) created for the Azure Blob Storage container where the flow logs are stored. The SAS URL can be easily obtained using Storage Explorer. The SAS must provide permissions to Read and List blobs.

Flow logs inside a single Azure Blob Storage container may originate from several Azure Account Subscriptions. Therefore, you must also specify the Subscription ID that determines which flow logs should be processed by the Flowmon Collector. You can process flow logs from multiple Azure Account Subscriptions by adding another subscription in the Azure Flow Logs FMC configuration page.

The provided configuration can be optionally verified by clicking the Verify button. This checks if the Flowmon Collector is able to connect to all Azure Blob Storage containers using the provided SAS URLs and will also attempt to find the correct directory with the flow logs (using the provided subscription ID).

Note that the provided configuration undergoes the verification process every time you click Save.

Newly created configurations must be saved (by clicking Save). This starts the process of retrieving of the flow logs. To stop the processing the flow logs, toggle the Enable button and click Save. Note that your configuration is stored even when the flow log processing is disabled, so that it can be easily enabled again.

Viewing Azure NSG Flow Logs in Monitoring Center

Multiple flow sources are created when using Azure NSG Flow Logs. Each flow source is internally assigned a unique IP address (from the subnet 127.130.0.0/16) and corresponds to a single resource group inside in the Azure Account Subscription. The name of the source has the following format: resource_group.subscription_id.

All sources can be found in Flowmon Monitoring Center -> Sources.

Click Profile if you want to divide the flow source into separate channel. Each channel contains flows from a particular Network Security Group and is uniquely identified by its name.

Select all available subnets and click Save.

It is possible to view and analyze the flows from the flow logs as if they were flows from regular data sources.