Custom Patterns

The Flowmon ADS application allows you to create simple custom behavior patterns. They can be defined in the Settings → Processing → Custom patterns. These can be enabled for processing by the BPATTERNS method. A pattern is defined as a simplified SQL query from biflow records (see the Fields of the biflow entries).

Biflow records are formed by paired flows (the request and reply). The flow items that are different in each direction are stored in the biflow record with a prefix according to the original orientation. For example amount of transferred data is stored in the req_transferred and rep_transferred fields. The flow items that are the same in both original flows are stored only once according to the request flow. For example, the source IP address of the biflow record corresponds to the source IP address of the request flow, although in the reply the IP address is specified as the destination. In the case of unpaired flow, only the items corresponding to the request flow are filled in the biflow record.

The editable parameters of the custom pattern include the Pattern code, Pattern description, Pattern detail, and the pattern expression parameters:

• The Pattern code identifies the pattern (it is used as a parameter in the BPATTERNS method to (de)activate given pattern).

• The Pattern description parameter describes the events detected by the pattern.

• The Pattern detail is used as a substring in the event details.

• The pattern expression is described by Where clause, Having clause, and Event source.

The event detail string generated for the custom behavior pattern contains:

• Short description of the pattern (from Pattern detail parameter)

• Number of outgoing flows from the event source to event targets

• Number of incoming flows to the event source from event targets

• Number of the data sent by the event source to event targets

• Number of the data received by the event source from event targets

• Number of packets sent by the event source to event targets

• Number of packets received by the event source from event targets

• Number of event targets

There is a biflow table that contains the source and destination IP address. You can use the Event source option to specify whether the source (default option) or destination IP address is used as the Event source in events. If the destination IP address is chosen as the event source, the event is detected from records that are grouped by destination IP address. The destination IP address is used as the event source and the event targets are all relevant source IP addresses. Depending on the Event source, all the destination or source IP addresses that were communicating with the target IP address are then listed as event targets. Grouping by destination IP address is used, for example, by the SRVNA detection method. This method marks an unavailable target (IP address of service) as the event source and all source IP addresses (client’s attempts to connect) as the event targets.

Where and Having clauses

The pattern itself is defined using the where and having expressions which are written in SQL form. All the constructions of SQLite language that are valid for the Where and Having expressions are supported, including operators and functions stated below. Single quotes (apostrophes) are required for a string enclosing - this applies to both expressions. Parts of logical expressions can be inserted into parentheses.

The Where expression is intended to filter the records based on defined conditions and using logical operators and functions. The Having expression allows you to use the aggregate functions that use these results. Therefore, the difference between the Having and the Where expression is that the Where works with table rows and allows you to filter individual rows. The Having then adds the option of working with a group of records using the aggregation functions. The following aggregation functions are supported:

• SUM: sum of values

• AVG: average of values

• COUNT: number of values

• MAX: maximal value

• MIN: minimal value

• VARIANCE: variance of values

• STDDEV: standard deviation

The complete list of aggregation functions and their descriptions can be found in the SQLite documentation (see the Aggregate functions). The complete documentation for the Where and Having expressions can be found here or here respectively.

See the Custom pattern examples.

Fields of the biflow entries

The following fields can be used for filtering:

• source_ip_address: Source IP address in the internal form. See the Functions for the translation functionality.

• destination_ip_address: Destination IP address in the internal form. See the Functions for the translation functionality.

• protocol: Protocol number.

• source_port: Source port number.

• destination_port: Destination port number.

• req_duration: Duration of the request flow in milliseconds (integer).

• rep_duration: Duration of the response flow in milliseconds (integer).

• req_transferred: Total transmitted bytes in the request flow (integer).

• rep_transferred: Total transmitted bytes in the response flow (integer).

• req_packets: Total count of packets in the request flow (integer).

• rep_packets: Total count of packets in the response flow (integer).

• req_flags: TCP flags of request flow, it is recommended to use a suitable filtering function (see the Functions for the translation functionality).

• rep_flags: TCP flags of response flow.

• src_mac: Source MAC address in decadic form. See the Functions for the translation functionality.

• dst_mac: Destination MAC address in decadic form. See the Functions for the translation functionality.

• src_as: The source autonomous system number.

• dst_as: The destination autonomous system number.

• src_country: The country number, which is assigned to the source IP address (see List of countries).

• dst_country: The country number, which is assigned to the destination IP address.

• http_host: Last 32 characters of the HTTP hostname. It can contain the queried DNS name if it is a DNS query.

• http_url: First 64 characters of the URL from the HTTP GET request.

• src_user_id: Identification of the user which is using the source IP address (obtained from Active Directory in case of the configuration in Flowmon OS).

• dst_user_id: Identification of the user which is using the destination IP address.

• tcp_ttl: TCP TTL, set only if the TCP SYN flag is set.

• tcp_window: TCP window, set only if the TCP SYN flag is set.

• syn_size: Size of TCP SYN packet in request flow.

• ua_os: The number of the operating system (obtained from the HTTP user agent, see the List of operating systems), 65535 means that the OS is unknown.

• ua_os_version: The number of the version of the operating system (obtained from the HTTP user agent), decimal – its integer part is the major version and the decimal part is the minor version. A number greater than or equal to 65535 means the version is unknown.

• paired: Specifies whether the biflow contains the whole flow-pair or if there is only a single unpaired flow. The number is 0 or 1.

• ja3_fingerprint: Value of the TLS JA3 fingerprint of a client. This is a string with a length of 32 characters which consists of hexadecimal digits.

Variables that can be used in the custom patterns definitions:

• #src_filter#: The assigned filter that is applied to source IP addresses.

• #dst_filter#: The assigned filter that is applied to destination IP addresses.

• #neg_src_filter#: The negative version of the assigned filter that is applied to source IP addresses (IP address has to be outside the given filter).

• #neg_dst_filter#: The negative version of the assigned filter that is applied to the destination IP addresses.

• #empty_filter#: Used to check if the assigned filter is empty.

Operators that can be used in the custom pattern definitions:

• = Equality test (source_port = 80)

• <> Inequality test (protocol <> 6)

• > Is greater than (req_transferred > 102400)

• < Is lower than (req_duration < 10)

• >= Greater than or equal to

• <= Lower than or equal to

• IN Belongs to the given set (destination_port IN (80, 443))

• BETWEEN . . . AND Belongs to the given range (req_duration BETWEEN 10 AND 60, including the given borders)

• LIKE The string that contains given substring (http_host LIKE '%facebook%', the percent signs represent any string).

• NOT Negation of the given constraint (NOT (req_duration < 10 AND source_port = 80))

• + Addition (req_transferred + rep_transferred >= 1048576)

• − Subtraction (req_duration - rep_duration > 3)

• * Multiplication (req_transferred * paired > 0)

• * Division (req_transferred / req_packets = 40)

Functions for patterns

Functions that can be used in the custom patterns definitions:

• matchflags(flags, required, forbidden): Function for matching TCP flags. flags should be replaced by req_flags or rep_flags. The function returns 0 if (and only if) flags contains all required flags and does not contain any forbidden flag. The function has to be used as an equality test with 0.

• ip_to_int(readable_ip) Function for translating IP addresses into internal form.

• mac_to_int(readable_mac) Function for translating MAC address into internal form.

Custom pattern examples

List of countries

LAN	Local Area Network	0
AF	Afghanistan	4
AX	Aland Islands	248
AL	Albania	8
DZ	Algeria	12
AS	American Samoa	16
AD	Andorra	20
AO	Angola	24
AI	Anguilla	660
AQ	Antarctica	10
AG	Antigua and Barbuda	28
AR	Argentina	32
AM	Armenia	51
AW	Aruba	533
AU	Australia	36
AT	Austria	40
AZ	Azerbaijan	31
BS	Bahamas	44
BH	Bahrain	48
BD	Bangladesh	50
BB	Barbados	52
BY	Belarus	112
BE	Belgium	56
BZ	Belize	84
BJ	Benin	204
BM	Bermuda	60
BT	Bhutan	64
BO	Bolivia, Plurinational State of	68
BQ	Bonaire, Sint Eustatius and Saba	535
BA	Bosnia and Herzegovina	70
BW	Botswana	72
BV	Bouvet Island	74
BR	Brazil	76
IO	British Indian Ocean Territory	86
BN	Brunei Darussalam	96
BG	Bulgaria	100
BF	Burkina Faso	854
BI	Burundi	108
KH	Cambodia	116
CM	Cameroon	120
CA	Canada	124
CV	Cape Verde	132
KY	Cayman Islands	136
CF	Central African Republic	140
TD	Chad	148
CL	Chile	152
CN	China	156
CX	Christmas Island	162
CC	Cocos (Keeling) Islands	166
CO	Colombia	170
KM	Comoros	174
CG	Congo	178
CD	Congo, the Democratic Republic of the	180
CK	Cook Islands	184
CR	Costa Rica	188
CI	Côte d’Ivoire	384
HR	Croatia	191
CU	Cuba	192
CW	Curaçao	531
CY	Cyprus	196
CZ	Czech Republic	203
DK	Denmark	208
DJ	Djibouti	262
DM	Dominica	212
DO	Dominican Republic	214
EC	Ecuador	218
EG	Egypt	818
SV	El Salvador	222
GQ	Equatorial Guinea	226
ER	Eritrea	232
EE	Estonia	233
ET	Ethiopia	231
FK	Falkland Islands (Malvinas)	238
FO	Faroe Islands	234
FJ	Fiji	242
FI	Finland	246
FR	France	250
GF	French Guiana	254
PF	French Polynesia	258
TF	French Southern Territories	260
GA	Gabon	266
GM	Gambia	270
GE	Georgia	268
DE	Germany	276
GH	Ghana	288
GI	Gibraltar	292
GR	Greece	300
GL	Greenland	304
GD	Grenada	308
GP	Guadeloupe	312
GU	Guam	316
GT	Guatemala	320
GG	Guernsey	831
GN	Guinea	324
GW	Guinea-Bissau	624
GY	Guyana	328
HT	Haiti	332
HM	Heard Island and McDonald Islands	334
VA	Holy See (Vatican City State)	336
HN	Honduras	340
HK	Hong Kong	344
HU	Hungary	348
IN	India	356
ID	Indonesia	360
IR	Iran, Islamic Republic of	364
IQ	Iraq	368
IE	Ireland	372
IM	Isle of Man	833
IL	Israel	376
IT	Italy	380
JM	Jamaica	388
JP	Japan	392
JE	Jersey	832
JO	Jordan	400
KZ	Kazakhstan	398
KE	Kenya	404
KI	Kiribati	296
KP	Korea, Democratic People’s Republic of	408
KR	Korea, Republic of	410
KW	Kuwait	414
KG	Kyrgyzstan	417
LA	Lao People’s Democratic Republic	418
LV	Latvia	428
LB	Lebanon	422
LS	Lesotho	426
LR	Liberia	430
LY	Libya	434
LI	Liechtenstein	438
LT	Lithuania	440
LU	Luxembourg	442
MO	Macao	446
MK	North Macedonia, Republic of	807
MG	Madagascar	450
MW	Malawi	454
MY	Malaysia	458
MV	Maldives	462
ML	Mali	466
MT	Malta	470
MH	Marshall Islands	584
MQ	Martinique	474
MR	Mauritania	478
MU	Mauritius	480
YT	Mayotte	175
MX	Mexico	484
FM	Micronesia, Federated States of	583
MD	Moldova, Republic of	498
MC	Monaco	492
MN	Mongolia	496
ME	Montenegro	499
MS	Montserrat	500
MA	Morocco	504
MZ	Mozambique	508
MM	Myanmar	104
NA	Namibia	516
NR	Nauru	520
NP	Nepal	524
NL	Netherlands	528
NC	New Caledonia	540
NZ	New Zealand	554
NI	Nicaragua	558
NE	Niger	562
NG	Nigeria	566
NU	Niue	570
NF	Norfolk Island	574
MP	Northern Mariana Islands	580
NO	Norway	578
OM	Oman	512
PK	Pakistan	586
PW	Palau	585
PS	Palestine, State of	275
PA	Panama	591
PG	Papua New Guinea	598
PY	Paraguay	600
PE	Peru	604
PH	Philippines	608
PN	Pitcairn	612
PL	Poland	616
PT	Portugal	620
PR	Puerto Rico	630
QA	Qatar	634
RE	Réunion	638
RO	Romania	642
RU	Russian Federation	643
RW	Rwanda	646
BL	Saint Barthélemy	652
SH	Saint Helena, Ascension and Tristan da Cunha	654
KN	Saint Kitts and Nevis	659
LC	Saint Lucia	662
MF	Saint Martin (French part)	663
PM	Saint Pierre and Miquelon	666
VC	Saint Vincent and the Grenadines	670
WS	Samoa	882
SM	San Marino	674
ST	Sao Tome and Principe	678
SA	Saudi Arabia	682
SN	Senegal	686
RS	Serbia	688
SC	Seychelles	690
SL	Sierra Leone	694
SG	Singapore	702
SX	Sint Maarten (Dutch part)	534
SK	Slovakia	703
SI	Slovenia	705
SB	Solomon Islands	90
SO	Somalia	706
ZA	South Africa	710
GS	South Georgia and the South Sandwich Islands	239
SS	South Sudan	728
ES	Spain	724
LK	Sri Lanka	144
SD	Sudan	729
SR	Suriname	740
SJ	Svalbard and Jan Mayen	744
SZ	Swaziland	748
SE	Sweden	752
CH	Switzerland	756
SY	Syrian Arab Republic	760
TW	Taiwan, Province of China	158
TJ	Tajikistan	762
TZ	Tanzania, United Republic of	834
TH	Thailand	764
TL	Timor-Leste	626
TG	Togo	768
TK	Tokelau	772
TO	Tonga	776
TT	Trinidad and Tobago	780
TN	Tunisia	788
TR	Turkey	792
TM	Turkmenistan	795
TC	Turks and Caicos Islands	796
TV	Tuvalu	798
UG	Uganda	800
UA	Ukraine	804
AE	United Arab Emirates	784
GB	United Kingdom	826
US	United States	840
UM	United States Minor Outlying Islands	581
UY	Uruguay	858
UZ	Uzbekistan	860
VU	Vanuatu	548
VE	Venezuela, Bolivarian Republic of	862
VN	Viet Nam	704
VG	Virgin Islands, British	92
VI	Virgin Islands, U.S.	850
WF	Wallis and Futuna	876
EH	Western Sahara	732
YE	Yemen	887
ZM	Zambia	894
ZW	Zimbabwe	716

List of operating systems

ID	OS name		ID	OS name		ID	OS name
1	AIX		26	Arch Linux		51	Nintento WII U
2	Amiga OS		27	CentOS		52	OpenBSD
3	Android		28	Debian		53	OpenVMS
4	iOS		29	Fedora		54	OS/2
5	Mac OS		30	Gentoo		55	OS/2 Warp
6	Mac OS X		31	Knoppix		56	Palm OS
7	AROS		32	Kanotix		57	OPClinuxOS
8	Bada		33	Lindspire		58	QNX
9	BeOS		34	Mageia		59	RISC OS
10	BlackBerry		35	Mandriva		60	Sailfish
11	BlackBerry Tablet		36	Mint		61	SkyOS
12	Brew		37	Red Hat		62	Solaris
13	Chrome OS		38	Slackware		63	Syllable
14	Danger Hiptop		39	SUSE		64	Symbian
15	Darwin		40	Ubuntu		65	Tizen
16	DragonFly BSD		41	VectorLinux		66	Ubuntu Touch
17	Firefox OS		42	LiveArea		67	webOS
18	FreeBSD		43	MeeGo		68	Windows
19	GNU OS		44	MINIX		69	Windows 3.x
20	Haiku OS		45	MorphOS		70	Windows CE
21	HP-UX		46	MSN TV (WebTV)		71	Windows Phone
22	Inferno		47	NetBSD		72	Windows Xbox
23	IRIX		48	Nintento 3DS		73	XrossMediaBar
24	Joli OS		49	Nintento DS
25	Linux		50	Nintento WII