Differences between Flowmon 12 and 13 Collector Engines

This document provides a detailed comparison between the Flowmon 12 collector engine and the Flowmon 13 collector engine, focusing on their data query capabilities, filtering options, collector behaviors, and general functionalities. It highlights differences in aggregation methods, output formats, field handling, and filter syntax, offering insights into how each tool processes and represents network flow data.

Main differences

Flow collector

• Used disk space savings: Flowmon 13 uses a new compression algorithm and achieves up to 30% reduction in disk space usage for flow data compared to Flowmon 12 (compared when storing the same flow data).
• Charts by start-time vs received-time:
  • In Flowmon 12, the times in the charts correspond to the received-time, meaning the timestamp when the collector received the flow record. This could cause discrepancies if flows arrived late or out of order in high-latency or jittery environments.
  • In Flowmon 13, the charts use the start-time of the flow as the reference point.
• Protocol auto detection:
  • In Flowmon 12, the protocol type (IPFIX, NetFlow, sFlow) had to be explicitly set during collector configuration.
  • The collector in Flowmon 13 automatically detects the flow protocol (IPFIX, NetFlow, or sFlow) without requiring manual configuration.
• Timestamp fix module:
  • In Flowmon 12 there is no dedicated mechanism for fixing inaccurate timestamps from exporters.
  • Flowmon 13 detects and corrects anomalies in flow start/end times and normalizes timestamps from exporters using relative or unsynchronized clocks. The collector ensures that all flows contain start-time and end-time fields. The end-time must not be older than start-time. The duration must be less than or equal to the 'active timeout'. The start-time must be within the bounds defined by the current time and the 'active timeout'. It contains overflow detection for the 32-bit uptime flow fields. Invalid times are automatically fixed.
• Statistics granularity:
  • In Flowmon 12, profiles have configurable channel granularity (5 minutes, 1 minute, or 30 seconds).
  • In Flowmon 13 there is uniform granularity of 30 seconds for all channels for the last 3 months; beyond that, 5-minute slices for up to 5 years.
• Flow metrics distribution:
  • In Flowmon 12, flow metrics (packets, bytes, and retransmissions) are aggregated per profile interval without proportional distribution across time slices.
  • In Flowmon 13, flow metrics are proportionally distributed into 30-second slots based on flow duration.
• NetFlow v5/v9 forwarding changes:
  • In Flowmon 12, NetFlow v5 and v9 forwarding is supported but tied to static configuration. Changes require manual intervention and sometimes a restart of the collector service, which leads to flow data loss during the restart.
  • Flowmon 13 fully supports NetFlow v5 and v9 forwarding with dynamic template handling for v9. The collector dynamically adapts to NetFlow v9 templates. No restart is needed for forwarding configuration changes.
• Applying listening port changes:
  • Flowmon 12 requires full collector engine restart when listening ports are modified. Flows from all ports are interrupted during restart, causing potential data loss.
  • In Flowmon 13 no restart is needed for changing listening ports. Flows from unchanged listening ports remain active, ensuring data continuity.
• Stream-based alerting:
  • In Flowmon 12, alerting is static rule-based - alerts are tied to fixed time windows (for example, '3 connections in 5 minutes'). If activity spanned across windows, anomalies could be missed.
  • Flowmon 13 uses a sliding window evaluation method, continuously analyzing data streams rather than rigid intervals. Dynamic evaluation reduces missed anomalies and minimizes false positives.

Data queries

• Multithreading: Data queries now leverage multithreaded processing for concurrent query execution, improving CPU utilization and responsiveness under heavy workloads.
• Memory allocation: The engine introduces smarter allocation strategies to prevent memory fragmentation and ensure smooth operation during parallel queries.
• Aggregation and filtering: The Collector introduces approximately 150 new aggregation keys and 70 new sorting keys, along with enhanced filtering options. For detailed guidance on filtering, refer to the User Guide.
• IP address indexing for all profiles:
  • In Flowmon 12, IP address indexing is only available for the 'All Sources' profile. Other profiles were not indexed, limiting search acceleration.
  • In Flowmon 13, IP address indexing is extended to all profiles - every real profile now benefits from IP indexing for faster queries.
• NPM metric aggregation:
  • Flowmon 12 uses a normal average function for the NPM metric aggregation.
  • Flowmon 13 applies a weighted average based on the number of packets.
• New in-out-byte field: Bytes field %byt was the sum of the in-bytes and out-bytes fields in Flowmon 12. It now returns the value of the in-bytes field. If the in-bytes field is not in the flow, it returns the value of the out-bytes field. A new field named in-out-bytes was added to preserve the old behavior. The same applies to the %pkt field.
• Efficient data query execution: If a query requires data from a channel for a time period before the channel existed, the missing data is retrieved from the parent channel.
  • In Flowmon 12, the entire period is read from the parent channel.
  • In Flowmon 13, only the missing portion is read, resulting in faster query execution.

Other changes

Flow collector

• Flows are saved to files in the /data/flows directory based on the flow start-time instead of the received-time field.
• IPFIX fields for total-bytes and total-packets (85, 86, 171, 172) are no longer saved as bytes and packets fields.
• Enhanced sFlow - not adding empty fields that are not in the received flow.
• Flow timestamps are in nanosecond precision.
• Support for the deltaFlowCount (IPFIX id=3) flow field was added.
• Support for QinQ (double VLAN tags) was added (IPFIX fields 243, 58).
• Support for MODBUS, MQTT, DNP3, and S7 fields.
• Enhanced handling of the ICMP fields (version based on protocol instead of IP, split v4/v6 codes and types).
• The collector now handles zero MAC addresses as N/A.
• All flows with sampling enabled contain a new %sampling field with the value of the sampling used.
• A new %uuid field was added as the unique identifier of the flow.
• SHA256 is now default for secure SNMP v3 authentication. It is used when the collector contacts flow sources to retrieve information through SNMP.
• The collector now supports the 'option templates' withdrawal according to the RFC 7011 section 8.1.
• Juniper's jFlows exported because of the active timeout are now correctly interpreted. Timestamp adjustment depends on whether the flow ended because of an active timeout or another reason, ensuring accurate flow duration calculations. This adjustment occurs before flow forwarding, and the forwarding target receives flows with timestamps mapped to the appropriate fields as required by the selected flow format (NetFlow v5, NetFlow v9, or IPFIX).
• The collector internally prefers using variable length string fields, instead of fixed length, which allows it to receive longer string values and save space when storing short values.
• The collector now detects and handles 32-bit time values (NetFlow) overflows, so the start-time and end-time of the flows are always correct.
• When using TCP with TLS, certificate validity is checked.

Data queries (nfdump tool)

• nfdump now supports output fields selection (-o) together with the top-stats parameter (-S/-s).
• nfdump outputs the number of resulting flows when called with the sampling (-G) parameter, instead of the number of source flows.
• nfdump now uses a newer version of protocol names assignment (/etc/services), countries, autonomous systems, and other constants.
• The field fwd (IPFIX 89) is now named fwd-status (alias fwds), because fwd is a reserved word for future bi-flow support.
• Text headers for -o fmt outputs of -S/-s/-A/-a were unified.
• CSV output (-o csv) now supports lists of fields (-o csv:%field1,%field2).
• A new JSON output was added (-o json).
• Output of N/A values was fixed. Flowmon 13 outputs N/A for all fields that were not present in the original flow.
• nfdump does not output the aggregation key fields in the summary footer because key values cannot be aggregated.
• nfdump now outputs the original number for the %hos and %happ fields, when the mapping from number to string does not exist.
• All multi-character constant names (like protocol names, http-method names, and so on) are now case insensitive.
• The %direction field now contains N/A, if it was missing in the received flow, value 0=I is ingress and value 1=E is egress. Other values are output as numbers.
• Unprintable characters are removed from string values.
• Output of the IP address when printing together with the network address was fixed.
• The %sysid field was removed. This value was not consistent across the files and collector restarts.
• Output of -v was adjusted to the new file format.
• Parameter -G (sampling) does not automatically add the %flow field to the output.
• If a field was not in the received flow, then it is not saved to the file. In the previous version, it was saved with a special N/A value in many cases.
• It is no longer supported to have more -s/-S parameters producing multiple outputs in one nfdump call.
• Filter syntax was unified:
  • Substring/exact match filtering: hurl "abc" - search for substring, hurl = "abc" - search for exact match.
  • tls-cont/tls-hshk is no longer masked - exact match filtering is used.
  • coap-code is compared as a number, not a string.
  • The tds-sql and tds-isql fields are both case insensitive.
  • The inet filter outputs only IPv4 flows, excluding flows without an IP address.
  • Filtering of many fields was unified. For the detailed grammar, refer to the User Guide.
• Command line arguments are no longer supported for nfdump:
  • -Z for filter check. The check_filter standalone binary is used instead.
  • -i for changing the file 'ident' string.
  • -z to compress flows. Compression is enabled by default in Flowmon 13.
  • -j for file compress/uncompress. Files are always compressed in Flowmon 13.
  • -X to compile the filter syntax and dump the filter engine table to stdout. The standalone check_filter binary can be used for that.
  • -R expr, -M expr for channel/files selection. -C is used instead of that in Flowmon 13.
  • -D for DNS resolving.
• Flow file names - fsfile.* in /data/flows/... instead of nffile.* in /data/nfsen/profile-data/.
• When reading old v12 files, the start-time and end-time timestamps are checked and fixed if they are out of bounds of the current file.