Data Retention

Introduction

The Data Retention plugin enables Internet Service Providers (ISPs) to fulfill all requirements of local law in their particular country based on the European Union Directive 2006/24/EC. The directive requires Member States to ensure that communications service providers must retain data generated or processed as a consequence of a communication or a communication service for a specific period (from 6 months to 2 years).

Web interface

The module is supplied in the form of an installation package for the Flowmon device (probe and collector). Install it using the Configuration Center (refer to the Configuration Center documentation). After installation, there a new link to the Data Retention module appears in the web interface of the Flowmon device. After successfully logging in, the default Search tab appears. The main menu consists of the following pages: Search, Statistics, and Settings.

Search page

On this page, you can create new requests to be processed. Completed requests are displayed here.

Requests table

The Request results table displays finished requests. In the Action column, you can download or delete the results. Each finished request is automatically displayed in this list.

Each line corresponds to one request. The first part is the name of the request which consists of the three-letter organization acronym (this is set in the Settings page), the date of the request (in the format YYMMDD), the number of the request, and the extension .txt.

To save or open a request file, click Download. You can delete the request by clicking Delete. You need to confirm the delete operation.

Running requests table

This table displays running requests. The request number and start time are displayed.

Create request

To create a new request, click Create new request. The window shown below appears. Enter the required information into the fields. If the Create new request button is not displayed, then the plug-in is not licensed properly and you must enter a valid license key.

Request number is the four-digit number of the request. It should be unique for the actual day and logged-in user. If the same request number is used on the same day, the older request is rewritten by the newer one.

The following fields Type of connection - Wi-Fi AP identification can be either entered manually, or they can be obtained from external database using a custom script (refer to the Settings section for further details). The value can be obtained from an external source if the External data binding feature is enabled in the Settings page and the option Retrieve by a script for external data binding is enabled in this form.

Type of connection: Enter the type of connection of the actual user (for example, GPRS, ADSL, and so on) or select the Retrieve by a script for external data binding check box.

User label: Enter the label of the user or users (for example, name) that you are looking for or select the Retrieve by a script for external data binding check box.

User account ID: Enter the identifier of the user or users (for example, name) that you are looking for or select the Retrieve by a script for external data binding check box.

MAC address: Enter the MAC address of the device that is used for connection or select the Retrieve by a script for external data binding check box.

Wi-Fi AP identification: Only fill in this field when the user is connected through a wireless network to a specific access point or select the Retrieve by a script for external data binding check box.

Search by: Select the kind of IP address to use for the search. The options are public IP or private IP.

Output: Select the output format. The options are default (3.3.1), default with NAT (3.3.1 + 3.3.7), or NAT only (3.3.7).

Interest identifier: Enter the identifier to be searched for. The identifier can either be an IP address (for example, 127.0.0.1 or 2001:db8:1f70::999:de8:7648:6e8), an IP address with a port (for example, 127.0.0.1:443 or [2001:db8:1f70::999:de8:7648:6e8]:443), or a network entered in the format IP/CIDR (for example, 10.0.0.0/16 or 2001:db8:1f70::/64). If the switch Search by public IP is enabled, then a comma-separated list of IP addresses can be entered.

Public NAT IPL This option is only available if the switch Search by private IP and the switch Output with NAT is selected. Here, enter the public NAT IP address that was used for address translation of a private IP address or IP addresses in the Interest identifier.

Interval: Determines the interval in which the search is performed. The first selection menu contains preset intervals. You can use the next two fields (start and end time) to configure your own interval. When you select a preset interval, these two fields show the real time of the selected interval.

Click Save to start processing your query. If all arguments are entered correctly, the request will start to be processed by the server and the information message is shown.

Statistics page

The number of processed requests is displayed in this page. If a user with the admin role is logged in, the statistics are displayed for all users. Admins can select a username from the User drop-down menu and see the number of requests and the last ten requests in the table. Other users can only see the number of their own requests.

Settings page

This page is used to set the parameters of the application and the user presets.

Admin settings

Admin settings are only displayed to users with the admin role. Other users do not have access to it.

It allows admins to set the three-letter Company abbreviation that is added to the beginning of the name of the request. The Company abbreviation should be written using uppercase letters. If the user enters lowercase letters, they are automatically changed to uppercase.

Maximal number of requests: Sets the maximum limit of requests per user that are saved. If a new request is made, the oldest one is deleted. Valid values for this field range from 1 to 50.

The Listening port, Maximal size, Watermark, and Data expiration time fields are available for installations with disk array only. The Listening port is the UDP port used to receive retention NetFlow data.

Maximal size: A limit for the data size on the disk array. If it is reached, the data is removed to reach the Watermark value.

Data expiration time: Indicates the lifetime of files since he creation on the disk array. After the defined time, old files are erased from the disk array.

Ignore timestamps in flows: This should only be enabled if the source of flows does not provide correct timestamps. If enabled, then the time of reception on the collector is used for filtering data into the selected interval. If Ignore timestamps in flows is disabled, flows are filtered by flow start and flow end timestamps in the flow, which is better and provides more precise results.

Click Save to apply the settings.

User settings

This form allows you to set user presets for the currently logged-in user.

Delimiter: Set the delimiter sign to use to separate the single values in the output file. There are two options: semicolon (code 0059 of the character set) or tabulator (code 0009 of the character set).

Requests per page: Sets the number of rows in the requests table. If there are more requests, the table is split into more pages.

Click Save to apply the settings.

Segments

For NAT correlation engine to function correctly, you must configure NAT segments properly. A segment defines the mapping of a group of private addresses to a group of public addresses. To add a new segment, click Add new segment.

In the New segment form, enter the name of the segment, a list of public IP addresses (or subnets), and a list of sources containing these IP addresses. Also, enter the list of private IP addresses (or subnets) that are translated to these public IPs and data sources. The NAT correlation engine only pairs private addresses assigned to the same segment as the entered public IP address.

External data bindings

These settings are only displayed to users with an admin role. Other users do not have access to it.

The system is able to automatically add records that are not part of flow data but are required by the law. For this purpose, you can create your own script that can find the required records in an external database according to IP addresses and time stamps in flows. You can enable this feature can by checking the field Use external data binding. This shows new settings for uploading a user-defined script and downloading the last version of this script that was uploaded to the device. In the list Select items for binding, you can select items whose values will be bound from the external database.

The description of the user-defined script is as follows. This script has the following inputs:

• Path to the input file

• Path to the output file (needs to be created)

• Comma-separated list of fields for replacement

• Delimiter type

The output of the script be saved to the output path specified. This file must match the final format required by law.

In the output file, the following fields are replaced by required values. Replacement of each field is allowed/denied by the user and the list of allowed fields is provided as a comma-separated list of codes. The list of such codes is as follows:

• Connection type - code conntype

• User label - code userlabel

• User ID - code userid

• Mac Address - code macaddr

• WiFi Access Point - code wifiap

The above fields (which will be replaced) are in the input file and are represented by special macros. These macros are replaced by the required value in the output file. The list of macros is as follows:

• Connection type - CONNTYPE

• User label - USERLABEL

• User ID - USERID

• Mac Address - MACADDR

• WiFi Access Point - WIFIAP

Data in the input and output files can have three format types. A user-defined script performs macros replacement for types 1 and 2 only. In files of type 3, there is nothing to be replaced. Examples of all three file types are provided below. The user script must be able to recognize the correct file type and parse it accordingly. In the following examples, semicolon is used as a delimiter. You can use a tabulator as the delimiter too. The type of delimiter is defined by the Delimiter field (see above). A description of all three file types is as follows:

Type 1

Type 1 is an output file for requests without NAT traffic. Besides replaced values, each flow record contains information about the start and end time of communication, IP address, and port. The user script must copy the content of the input file into the output file and replace the required fields represented by macros.

INV_151119_6666.txt, all, 192.168.3.107, 18.11.2015 22:25:00, 19.11.2015 10:25:00, UTF8 
TYPE OF CONNECTION;USER LABEL;USER ID;MAC;START;STOP;ACCESS POINT;IP:PORT
CONNTYPE;USERLABEL;USERID;MACADDR;18.11.2015 22:25:37;19.11.2015
10:23:37;WIFIAP; 192.168.3.107:0 
CONNTYPE;USERLABEL;USERID;MACADDR;18.11.2015 22:26:36;19.11.2015 10:21:54;WIFIAP; 
192.168.3.107:56374 
Konec.

Type 2

Type 2 is an output file for requests with NAT traffic. Besides replaced values, each flow record contains information about the start and end time of communication, private IP address and port, and public IP address and port. The user script must copy the content of the input file into the output file and replace the required fields represented by macros.

INV_130815_2222.txt, all, 192.168.3.54, 15.08.2013 15:00:00, 15.08.2013 15:10:00, UTF8 
TYPE OF CONNECTION;USER LABEL;USER ID;MAC;START;STOP;ACCESS POINT;IP:PORT;PUBLIC IP:PORT
CONNTYPE;USERLABEL;USERID;MACADDR;15.08.2013 15:00:00;15.08.2013 15:00:00;WIFIAP; 192.168.3.54:48004;195.113.224.147:48004
CONNTYPE;USERLABEL;USERID;MACADDR;15.08.2013 15:00:00;15.08.2013 15:00:00;WIFIAP;
192.168.3.54:48004;195.113.224.147:48004
Konec.

Type 3

Type 3 is an output file for requests with NAT traffic. This contains only private IP address, public IP address, port and information about the start and end time of communication. No fields for replacement are present. The user script must copy the content of the input file into the output file. No replacement is performed.

INV_140429_1111.txt, all, 192.168.3.0/24,
28.04.2014 23:40:00, 29.04.2014 11:40:00, UTF8 
PRIVATE IP;PUBLIC IP:PORT;START;STOP 
192.168.3.54;195.113.224.147:48004;15.08.2013 15:00:00;15.08.2013 15:00:00
192.168.3.54;195.113.224.147:48004;15.08.2013 15:00:00;15.08.2013 15:00:00 
Konec.

Other types (if none of the above types is recognized)

The user script must copy the content of the input file into the output file. No replacement is performed.

A user-defined script can be written in any scripting language supported by the Flowmon platform or it can be provided in a Flowmon-compatible binary format.