Monitoring Center Configuration

This page contains details about the Monitoring Center settings and is further divided into six sections, each for particular part of the Monitoring Center.

Built-in Collector - Basic Settings

You can set up the built-in collector on the Configuration Center > Monitoring Center > Collector page.

Use this page to perform configuration changes to the built-in collector. Click Start/Stop to start/stop the built-in collector. You can see the collector status on this button (Running/Stopped). You will not be able to access the Monitoring Center if the built-in collector is stopped.

If there are some queries running in the Monitoring Center, a button showing their count will appear. In some cases, very complicated queries on a large amount of data may take a very long time and slow down the device. It may be useful to kill these queries by clicking Terminate running queries.

When you enable the Enable IP indexing option, Flowmon starts building an index of IPv4 addresses present in incoming flows. You can then use the index to accelerate filter queries for IPv4 addresses. The index is only available for flows from the moment you enabled the indexing option. If you need to recalculate data from the past, contact our Support team.

You can enable two levels of IP indexing:

• Enable for the 'All Sources' profile - The index is only available for the 'All Sources' profile.

• Enable for all profiles - The index is available for any real profile on the device.

See Types of Analysis for details on how to use the IP index to accelerate your filter queries.

The Clear data storage button is used to clear the built-in collector database. This operation irreversibly removes all stored NetFlow data. Depending on the size of stored NetFlow data this operation can take several minutes. During this time the Monitoring center will not be accessible.

Built-in Collector - Processing Modules

On this page you can enable post processing modules in the Built-in Collector. These modules can enrich the flow data with additional information. For a complete list of fields added by the Collector, see Flowmon Collector - Extra Fields.

You can enable these modules:

• Use autonomous system list - Enriches flow records with Autonomous System (AS) numbers based on IP addresses to identify traffic origin. You can select a custom or built-in AS list. This module adds the following fields: src autonomous-systems, dst autonomous-systems.

• Normalize flow records - Unifies flow records from different vendors (for example, Cisco, Gigamon) by converting proprietary field values into standardized Flowmon fields. This module adds the following fields: npm-round-trip-time, npm-server-response-time, npm-tcp-retransmission, dns-flags-codes, http-host, http-url, tls-signature-algorithm, tls-public-key-algorithm, tls-public-key-length, npm-server-response-time-min, npm-server-response-time-max, dns-response-info.

• Add DNS information - Adds domain names to flows by correlating them with recent DNS responses, improving visibility into accessed services. This module adds the following fields: src hostname, dst hostname. To make the feature operational, the collector must receive flow data from probes configured with DNS protocol metadata extraction and export. Enable DNS protocol monitoring on your Flowmon Probes in Configuration Center > Monitoring Ports > Advanced Settings.

• Add GeoIP information - Tags flows with geographic location data derived from IP addresses to support regional analysis. This module adds the following fields: src country, dst country.

• Add user identity - Associates flow records with usernames received from external systems such as DHCP servers, VPN servers, and Active Directory. This information is sent from those external sources to Flowmon through Syslog messages. This module adds the following fields: src user-id, dst user-id. For more information, see User Identity in Flowmon Solution.

Built-in Collector - Listening ports (Collector only)

On this page you can configure the listening ports for NetFlow, IPFIX, sFlow, and other supported flow protocols and their forwarding. The listening port is defined by its name, port, network protocol, and flow protocol. Select the flow protocol used by your flow exporting device (router, probe). There are two options: NetFlow/IPFIX or sFlow. The option NetFlow/IPFIX also applies to all NetFlow clones like jFlow, NetStream, and so on. Contact Support at the Flowmon Support and Learning Hub for more information about the supported protocols.

There is no need to define different listening ports for individual flow sources (Probes, routers, and so on) because Flowmon automatically recognizes and configures the individual flow sources. It is recommend to keep the default settings of listening ports unless there is a specific reason for defining an additional listening port. A new listening port can be added by clicking New listening port. A new form appears.

Enter the name of the listening port, port number, and network protocol. If TCP is selected as the network protocol, the encryption TCP/TLS can be enabled. For TCP/TLS, the set of keys and certificates have to be generated for the flow exporting device (monitoring port) and for the collector. All certificates must be signed by the same Certificate Authority (CA). Its certificate (CA certificate) must be provided together with the collector key and certificate to each listening port using the TCP/TLS protocol.

If TCP is selected as the network protocol, the encryption TCP/TLS can be enabled. For TCP/TLS, the set of keys and certificates have to be generated for the flow exporting device (monitoring port) and for the collector. All certificates must be signed by the same Certificate Authority (CA). Its certificate (CA certificate) must be provided together with the collector key and certificate to each listening port using the TCP/TLS protocol.

The sampling rate of received flow data is determined from the flow protocol. For cases when the sampling rate is not provided or is reported incorrectly, it is possible to define it statically for the purpose of calculating metrics and statistics. For these purposes, check the Define source sampling rate and enter the number. The entered value is only used if the flow monitoring port does not provide the sampling rate information. If the flow monitoring port provides the sampling rate, then its value is used. Check the Force sampling rate option and the value will be used in all cases.

t start = reception_time − active_timeout

t end = reception_time

The generated times are only indicative. For long term flows (where active timeout applies) the flow duration is correct. The start time and end times are delayed due to the time between ending the flow on the flow source and its reception on the Flowmon Collector. For short flows where active timeout does not apply, the flow duration will be wrong. To enable this feature, enable the Custom active timeout switch and provide the Active timeout of the flow source sending data to this listening port.

Modify flow timestamps by flow receive time - this feature fixes incorrect flow timestamps by replacing the flow end-time with the flow receive time and computing the flow start-time as the flow receive time minus the flow duration.

Received flow data can be forwarded to multiple different targets. For this purpose, use the Forwarding targets selector to choose forwarding targets for the Listening ports. The Forwarding targets must be configured in the Forwarding Targets page.

Built-in Collector - Forwarding targets (Collector only)

This section enables the configuration of targets of forwarding of the listening ports. The configured forwarding targets are shown in the table below. Click New target or the Edit icon in the Action column to create a new forwarding target or to edit an existing one. This forwarding target will be applied to all listening ports selected in the Listening ports selector at the bottom of the page. Forwarding can be performed in two modes: Compatible mode and Advanced mode. These are available in separate tabs.

Forwarding mode - compatible

This mode allows flow forwarding using the UDP protocol with a spoofed IP address of the flow source. This mode is compatible with all Flowmon Collectors and third-party collectors. In compatible mode, the original IP address of the flow source is preserved (that is, IP spoof mode), so the target collector assigns the flows to the IP address of the original flow source. Keep this in mind when configuring firewall rules, and so on.

In compatible mode, enter the IP address of the collector and the UDP port.

Forwarding mode - advanced

This mode allows flow forwarding using advanced capabilities such as TCP or TCP/TLS export, flow protocol conversion, flow sampling, and flow filtering. This mode is compatible with Flowmon Collectors v9.01.00 and higher.

In advanced mode there are two tabs - Export target and Export protocol.

In the Export target tab, enter the IP address of target collector, port, flow sampling rate, and choose the transport protocol. TCP protocol is only allowed when IPFIX is used as an export protocol (see the Export protocol tab). Moreover, the export filter can be added to define what flows will be forwarded to this target. For the filter syntax, see the Filter Syntax section. If the TCP protocol is selected, the flow data can be forwarded encrypted using the TCP/TLS protocol if the option Enable encryption is enabled. Then the collector private key, collector certificate, and CA certificate must be provided.

In the Export protocol tab, the flow export protocol can be selected (from the options NetFlow v5, NetFlow v9, and IPFIX). For NetFlow v9 and IPFIX there is an option to change the default template re-sending intervals.

Click Save to apply changes. The entered values are checked for loop presence, which can be fatal for the Collector. This operation can be more time-consuming.

Built-in Collector - Sources settings

On the Sources settings page, you can configure the limit for the number of profiled sources and their interfaces. Refer to the Sources section for further information.

Reports settings

The Reports settings section consists of Basic Settings, Remote Storage, and Branding.

Basic Settings

In basic settings, you can disable or enable reporting functionality. If you disable reporting, schedules from Dashboards and Reports will stop being sent (email and Samba). You are also allowed to recompute all chapters at once. Pick the desired time interval and then click Recompute. Progress of jobs computing shows how many tasks are computed and how many tasks are waiting. The Reserved CPU value displays how much CPU performance can be used to compute chapters statistics (done every hour). The Allow sampling for large amounts of data option is enabled in the default configuration and allows the system to sample flow data during computation of reports if the amount of data is very large. So, it speeds up the computation significantly and saves a lot of resources on heavily-loaded Collectors. The precision of computed statistics is decreased only a little because for large amounts of data, the sampled data is statistically unimportant. To save a new value, click Save.

Remote Storage

In the Remote storage section, you can configure parameters for storing reports to remote storage. Enter the Report directory (where the reports will be copied). The Copy timeout is used for specifying the maximum time for copying of a single report. If the copy transaction takes longer, it is interrupted as unsuccessful. Use the value zero to set unlimited time. The Delete unsuccessfully copied files after option is used to configure of the maximum interval in days, when the older reports are removed from the queue and the system will not attempt to try to copy them again. Use the value zero to set unlimited time.

Branding

In Branding, you can specify the look of generated PDF reports. You can select the main color, report name, email report subject, and body. You can use macros here (described on the panel).

Active Devices

This page is used to configure the Active Devices monitoring functionality. The function is enabled by default, but it is not available for Distrubuted Architecture configurations. To disable/enable it, you must use the Enable active devices logging toggle switch and click Save.

Pick monitored flow sources from the selection menu. Only data from these sources are collected. You can also specify a filter if you want to monitor only specific traffic. Click Save to save changes.

The Identify by field specifies the default device identifier in the Monitoring Center and is used to aggregate your query results. You can temporarily switch the identifier to a different one when building a query at any time, using the aggregation options.

Active devices - IP ranges

The IP ranges table is used to configure all subnets where the active devices are to be monitored. It makes sense to collect these in the local network and therefore there are preset values for all private and local networks: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, and fe80::/10. To add a new subnet, simply enter this value in the form: IP address/mask.

Active devices - Routers

The Routers table is used to manage MAC addresses of routers that are hidden in reports by default because they have normally assigned a large number of IP addresses. You can change your preference to show them in the Monitoring Center in the search form.

AWS Flow Logs Converter

What is the AWS Flow Logs Converter?

The AWS Flow Logs Converter is a configurable module of the Monitoring Center.

It enables the user to collect, process, and visualize AWS VPC Flow Logs (further referred to as flow logs) which contain information about the traffic captured in Amazon Virtual Private Cloud.

Brief Implementation Description

The flow logs are periodically acquired from Amazon CloudWatch, processed, converted to IPFIX format, and subsequently sent to the Flowmon Collector to a defined UDP port.

The Flowmon Collector treats data from this port the same as regular flows recovered from any other port.

Setting Up Flow Logs for a VPC

To set up the flow logs in your cloud and forward them to AWS CloudWatch, follow the instructions specified here: Publish flow logs to CloudWatch Logs. It is important that every flow log stream contains flow logs from one interface only.

The AWS Flow Logs Converter can process TCP flags which are not enabled in the AWS VPC Flow Logs by default. To enable processing of TCP flags, you must specify a custom format of the Log Record when creating a new Flow Log.

The custom format must contain the following fields in the following order:

The AWS Flow Logs Converter can process only the default Flow Record format and the custom format specified above.

Setting Up Flow Logs in Flowmon Configuration Center

To start receiving flow logs in the Monitoring Center, follow these instructions:

Step 1: Create a new listening port in Configuration Center > Monitoring Center > Listening Ports.

You can choose the name and port number of the new listening port as needed. However, the network protocol must be UDP and the port must only be used for retrieving AWS VPC Flow Logs.

Step 2: Configure the access information, regions and log groups from which the flow logs will be retrieved.

Go to Configuration Center > Monitoring Center > Flow Logs > Amazon Web Services.

The access key ID and the secret access key are mandatory credentials provided by Amazon.

Select the previously configured listening port.

Click Add Region to configure the endpoints where the flow logs should be retrieved.

Insert the name of the region (without the availability zone) where your flow logs are physically stored. A list of all possible regions can be found here: Regions, Availability Zones, and Local Zones. Note that the region Name field is expected to contain values like eu-central-1 rather than EU (Frankfurt). It is also possible to define a short description of the region.

Lastly, it is necessary to provide at least one log group (by clicking Add group and filling in the name). All flow log streams in the provided group are processed and every stream is shown as a unique interface of the log group in the Monitoring Center.

You can optionally verify the provided configuration by clicking Verify. This checks if the Monitoring Center is able to connect to the specified log groups using the provided AWS credentials.

Note that the provided configuration undergoes the verification process every time you click Save.

Newly created configurations must be saved (by clicking Save). This starts the process of retrieving the Flow logs. To stop the process of retrieving, disable it, and click Save.

Viewing VPC Flow Logs in the Monitoring Center

It can take up to 20 minutes (see the limitations) before the first flow logs can be visualized.

Every log group has internally assigned a unique IP address (from the subnet 127.128.0.0/16) and is treated as a unique flow source.

All sources can be found in Monitoring Center > Sources.

Click Create Profile to see traffic of the individual streams.

Select all available streams and click Save.

Switch to: Monitoring Center > Profiles > Sources > Your Log Group

It is possible to view and analyze flows from flow logs as if they were flows from regular data sources.

Limitations of Flow Logs

There are some limitations which stem from the flow logs themselves that need to be taken into account:

• If your network interface has multiple IPv4 addresses and traffic is sent to a secondary private IPv4 address, the flow log displays the primary private IPv4 address in the destination IP address field.

• If traffic is sent to an ENI and the destination is not any of the ENI IP addresses, the flow log displays the primary private IPv4 address in the destination IP address field.

• If traffic is sent from an ENI and the source is not any of the ENI IP addresses, the flow log displays the primary private IPv4 address in the source IP address field.

• If traffic is sent to or sent by a network interface, the flow log always displays the primary private IPv4 address, regardless of the packet source or destination, in the interface IP address field.

Flow logs do not capture all IP traffic. The following types of traffic are not logged:

• Traffic generated by instances when they contact the Amazon DNS server. If you use your own DNS server, then all traffic heading to that DNS server is logged.

• Traffic generated by a Windows instance for activation of the Amazon Windows license.

• Traffic to and from 169.254.169.254 for the instance metadata.

• Traffic to and from 169.254.169.123 for the Amazon Time Sync service.

• DHCP traffic.

• Traffic to the reserved IP address for the default VPC router. For more information, see VPC and Subnet Sizing.

• Traffic between an endpoint network interface and a Network Load Balancer network interface. For more information, see VPC Endpoint Services (AWS PrivateLink).

• Some flow log records might get skipped during the capture window. This may be because of an internal capacity constraint, or an internal error.

Furthermore, the delay between the time when the traffic actually occurred and the time it can be seen in the Monitoring Center can reach up to 20 minutes in the worst case scenario. However, the delay will get smaller with a higher amount of traffic volume present in the monitored cloud.

This is caused by the 10-15 minute capture window in which the packets are aggregated to the flow logs before being published, and by the subsequent 5-minute delay before the Flowmon Collector closes the current profile and shows the traffic in the GUI.

The Flowmon Collector stores incoming flows to a currently opened profile, and therefore it is advised to select multiple adjacent profiles when searching for flows in a particular time.

Google Cloud Flow Logs

Flowmon Collector is capable of processing and visualizing Google Cloud VPC Flow Logs. Google Cloud VPC Flow Logs (further referred to as flow logs) are records of network connections between VM instances in VPC networks. The Flowmon Collector acquires flow logs by polling on the Google Cloud Pub/Sub subscription.

Setting Up Google Cloud VPC Flow Logs

Follow the official instructions to enable generating flow logs for certain subnets in your VPC.

It is important to mention several configurable options during the configuration of flow logs:

• Aggregation Interval: 5 minutes - recommended (standard configuration of Flowmon probes also use the 5-minute aggregation interval)

• Include metadata: On - mandatory (necessary to display information about the VPC and subnets in the Monitoring Center)

• Sample Rate: 100 - recommend in order to obtain all flow logs

Configuring Google Logs Router Sink

Follow the official instructions to configure logs router sink and Pub/Sub topic.

These configuration options might be helpful:

• Message retention duration should be opted out when creating Pub/Sub topic, because some form of flow logs retention will be done in the Pub/Sub subscription.

• It is preferable to provide an inclusion filter. In the Choose logs to include in sink panel, specify that you want to include only flow logs. This will increase the performance of processing of flow logs and decrease the overall price:

  • logName=~"/logs/compute.googleapis.com%2Fvpc_flows" - to see all flow logs

  • logName="projects/<project_name>/logs/compute.googleapis.com%2Fvpc_flows" - to see flow logs only from a specific project

Configuring the Google Cloud Pub/Sub Subscription

The Google Cloud Pub/Sub subscription must follow certain criteria, so it can be efficiently used by the Flowmon Collector.

The recommended configuration of a subscription to maximize the performance and minimize the cost:

• Delivery type: Pull - mandatory

• Message retention duration: 1 hour

• Retain acknowledged messages: No

• Acknowledgement deadline: 10 seconds

• Message ordering: No

• Dead lettering: No

• Retry policy: Retry immediately

The Flowmon Collector uses the Google Cloud Service Account Key (in JSON format) for authentication when acquiring flow logs from the Google Cloud Pub/Sub subscription. The service account used for acquiring flow logs must include the Pub/Sub Subscriber role in Google Cloud IAM. Note that such service account can access any Pub/Sub subscriptions with a Google Cloud project. For more information about setting up permissions, please refer to the official guide.

Setting Up Google Cloud VPC Flow Logs Processing

To start receiving flow logs in the Monitoring Center, follow these instructions:

Step 1: Create a new listening port in Configuration Center > Monitoring Center > Listening Ports

You can choose the name and port number of the new listening port as needed. However, the network protocol must be UDP and the port must only be used for retrieving Google Cloud VPC Flow Logs. Optionally, you can define the source sampling rate of this listening port, because Google Cloud already samples packets that leave and enter a VM to generate flow logs. Not every packet is captured into its own log record. About 1 out of every 10 packets is captured, but this sampling rate might be lower depending on the VM's load. You cannot adjust this rate.

Step 2: Enable processing of the Google Cloud Flow Logs and configure individual subscriptions.

Go to: Configuration Center > Monitoring Center > Flow Logs > Google Cloud.

Toggle the Enable button and select the previously created Listening port from the drop-down menu.

Click New Subscription which allows you to configure a list of Google Cloud Pub/Sub subscriptions from which flow logs will be obtained and processed. The following parts of a subscription can be configured:

• Subscription ID - ID of the Google Cloud Pub/Sub subscription

• Project ID - ID of the Google Cloud project to which the subscription belongs

• Service account credentials - Google Cloud Service Account Key in JSON format, with permissions to subscribe to the Pub/Sub subscription. Follow the official instructions to create the key.

• Description - custom description of the subscription

• Advanced Configuration - several options which can affect performance of the subscription process at the cost of increased resources consumption

  • Max. messages in backlog - the maximal number of Pub/Sub messages which can be in queue for processing (not recommended to set below 1000 messages).

  • Max. megabytes in backlog - the maximal number of bytes which can be in queue for processing (it is recommended to respect the size of messages containing flow logs - not more than several KB per message)

  • Max. messages processed simultaneously - number of parallel background workers for polling flow logs from the Pub/Sub subscription. It is recommended to set this value as low as possible based on the expected number of the processed Pub/Sub messages per second. The range is limited to 2 - 16 possible workers (it is recommended to use a power of 2). Two workers can handle processing around 100,000 Pub/Sub messages per second (tested on a c2-standard-16 computing instance). Keep in mind that configuring several subscriptions on the same appliance lowers the performance in general. It is not recommended to use more than 32 background workers in total across all configured subscriptions.

You can (optionally) verify the provided configuration by clicking Verify. This checks whether the Monitoring Center is able to connect to the specified Pub/Sub subscriptions using the provided Service account credentials.

Note that the provided configuration undergoes the verification process every time the Save button is clicked.

Viewing VPC Flow Logs in Monitoring Center

Multiple flow sources are created when using Google Cloud VPC Flow Logs. Each flow source is internally assigned a unique IP address (from subnet 127.129.0.0/16) and its name corresponds to a VPC inside a Google Cloud project in a format: vpc-name.project-id.

All sources can be found in Monitoring Center > Sources.

Click Create Profile if you want to divide the flow source into a separate channel. Each channel corresponds to a subnet inside the VPC and is uniquely distinguishable by the subnet name.

Select all available subnets and click Save.

It is possible to view and analyze the flows from the flow logs as if they were flows from regular data sources.

Azure Flow Logs

The Flowmon Collector is capable of processing and visualizing Azure VNET Flow Logs. Azure VNET Flow Logs (further referred to as flow logs) are sampled records of the network flow sent from and received by VM instances. Flow logs is a feature provided by the Network Watcher service and is dependent on the Microsoft Insights resource provider. The Flowmon Collector periodically connects to the configured Azure Blob Storage containers and downloads newly added flow logs. The flow logs are subsequently converted to the IPFIX format and can be viewed in the Monitoring Center.

Setting Up Azure VNET Flow Logs

Follow the official instructions to enable collecting of flow logs in Azure Blob Storage for your virtual machines.

Setting Up Azure VNET Flow Logs Processing

To start receiving flow logs in the Monitoring Center, follow these instructions:

Step 1: Create a new listening port in Configuration Center > Monitoring Center > Listening Ports.

You can choose the name and port number of the new listening port as needed. However, the network protocol must be UDP and the port must only be used for retrieving Azure VNET Flow Logs.

Step 2: Enable processing of the Azure VNET Flow Logs and configure individual subscriptions.

Go to: Configuration Center > Monitoring Center > Flow Logs > Microsoft Azure.

Toggle the Enable button and select the previously created Listening port from the drop-down menu.

Click New Subscription which allows you to configure a list of subscriptions. This list specifies which flow logs will be obtained and processed. For the Flowmon Collector to access the flow logs, it requires the URL of the Shared Access Signature (SAS) created for the Azure Blob Storage container where the flow logs are stored. The SAS URL can be easily obtained using Storage Explorer. The SAS must provide permissions to Read and List blobs.

Flow logs inside a single Azure Blob Storage container may originate from several Azure Account Subscriptions. Therefore, you must also specify the Subscription ID that determines which flow logs should be processed by the Flowmon Collector. You can process flow logs from multiple Azure Account Subscriptions by adding another subscription in the Configuration Center > Monitoring Center > Microsoft Azure page.

You can optionally verify the provided configuration by clicking Verify. This checks if the Flowmon Collector is able to connect to all Azure Blob Storage containers using the provided SAS URLs and will also attempt to find the correct directory with the flow logs (using the provided subscription ID).

Note that the provided configuration undergoes the verification process every time you click Save.

Newly created configurations must be saved (by clicking Save). This starts the process of retrieving of the flow logs. To stop the processing the flow logs, toggle the Enable button and click Save. Note that your configuration is stored even when the flow log processing is disabled, so that it can be easily enabled again.

Viewing Azure VNET Flow Logs in Monitoring Center

Multiple flow sources are created when using Azure VNET Flow Logs. Each flow source is internally assigned a unique IP address (from the subnet 127.130.0.0/16) and corresponds to a single resource group inside in the Azure Account Subscription. The name of the source has the following format: resource_group.subscription_id.

All sources can be found in Monitoring Center > Sources.

Click Create Profile if you want to divide the flow source into separate channel. Each channel contains flows from a particular Virtual Network and is uniquely identified by its name.

Select all available subnets and click Save.

It is possible to view and analyze the flows from the flow logs as if they were flows from regular data sources.