Google Cloud VPC Flow Logs

The Flowmon Collector is capable of processing and visualizing Google Cloud VPC Flow Logs. Google Cloud VPC Flow Logs (referred to as flow logs for the remainder of this section) are records of network connections between Virtual Machine (VM) instances in Virtual Private Cloud (VPC) networks. The Flowmon Collector acquires flow logs by polling the Google Cloud Pub/Sub subscription.

Set Up Google Cloud VPC Flow Logs

Follow the Google Cloud VPC flow logs setup guide to enable log generation for your VPC subnets.

Important configuration options:

• Aggregation Interval: 5 minutes - recommended (standard configuration of Flowmon probes also use the 5-minute aggregation interval)
• Include metadata: On - mandatory (necessary to display information about the VPC and subnets in the FMC)
• Sample Rate: 100 - recommend to obtain all flow logs

Configure the Google Cloud Pub/Sub Subscription

The Google Cloud Pub/Sub subscription must follow certain criteria so it can be used efficiently by the Flowmon Collector.

The recommended configuration of a subscription to maximize the performance and minimize the cost is as follows:

• Delivery type: Pull - mandatory
• Message retention duration: 1 hour
• Retain acknowledged messages: No
• Acknowledgement deadline: 10 seconds
• Message ordering: No
• Dead lettering: No
• Retry policy: Retry immediately

The Flowmon Collector uses the Google Cloud Service Account Key (in JSON format) for authentication when acquiring flow logs from the Google Cloud Pub/Sub subscription. The service account used for acquiring flow logs must include the Pub/Sub Subscriber role in Google Cloud Identity and Access Management (IAM). Note that such a service account can access any Pub/Sub subscriptions with a Google Cloud project. For further information about setting up permissions, refer to the official guide.

Set Up Google Cloud VPC Flow Logs Processing

To start receiving flow logs in the Flowmon Monitoring Center, follow these instructions:

Step 1: Create a new listening port in Configuration Center > Monitoring Center > Listening Ports.

You can choose the name and port number of the new listening port as needed. However, the network protocol must be UDP and the port must only be used for retrieving Google Cloud VPC Flow Logs.

You can optionally define the source sampling rate of this listening port, because Google Cloud already samples packets that leave and enter a VM to generate flow logs. Not every packet is captured into its own log record. About 1 out of every 10 packets is captured, but this sampling rate might be lower depending on the VM's load. You cannot adjust this rate.

Step 2: Enable processing of the Google Cloud Flow Logs and configure individual subscriptions.

Go to: Configuration Center > Monitoring Center > Flow Logs > Google Cloud.

Toggle the Enable option and select the previously created Listening port from the drop-down menu.

Click New Subscription, which allows you to configure a list of Google Cloud Pub/Sub subscriptions from which flow logs will be obtained and processed. The following parts of a subscription can be configured:

• Subscription ID - the ID of the Google Cloud Pub/Sub subscription.
• Project ID - the ID of the Google Cloud project to which the subscription belongs.
• Service account credentials - the Google Cloud Service Account Key in JSON format, with permissions to subscribe to the Pub/Sub subscription. Follow the official instructions to create the key.
• Description - a custom description of the subscription.
• Advanced Configuration - several options that can affect performance of the subscription process at the cost of increased resources consumption.
• Max. messages in backlog - the maximum number of Pub/Sub messages that can be in the queue for processing (It is not recommended to set this below 1000 messages).
• Max. megabytes in backlog - the maximum number of bytes that can be in the queue for processing (it is recommended to respect the size of messages containing flow logs - not more than several KB per message).
• Max. messages processed simultaneously - the number of parallel background workers for polling flow logs from the Pub/Sub subscription. It is recommended to set this value as low as possible based on the expected number of the processed Pub/Sub messages per second. The range is limited to 2 - 16 possible workers (it is recommended to use a power of 2). Two workers can handle processing approximately 100,000 Pub/Sub messages per second (tested on a c2-standard-16 computing instance). Keep in mind that configuring several subscriptions on the same appliance lowers the performance in general. It is not recommended to use more than 32 background workers in total across all configured subscriptions.

You can optionally verify the provided configuration by clicking Verify. This checks if the FMC is able to connect to the specified Pub/Sub subscriptions using the provided Service account credentials.

Note that the provided configuration undergoes the verification process every time the Save button is clicked.

View VPC Flow Logs in Monitoring Center

Multiple flow sources are created when using Google Cloud VPC Flow Logs. Each flow source is internally assigned a unique IP address (from subnet 127.129.0.0/16) and its name corresponds to a VPC inside a Google Cloud project in the format: vpc-name.project-id.

All sources can be found in Monitoring Center > Sources.

Click Create Profile if you want to divide the flow source into a separate channel. Each channel corresponds to a subnet inside the VPC and is uniquely distinguishable by the subnet name.

Select all available subnets and click Save.

It is possible to view and analyze the flows from the flow logs as if they were flows from regular data sources.