The LoadMaster generates syslog messages using the following format:

YYYY-MM-DDThh:mm:ss+hh:mm Hostname Subsystem: Message Data
An explanation of the format is as follows:

  • Date/Time Stamp: YYYY-MM-DDThh:mm:ss+hh:mm

    This includes the full date and time along with the timezone offset.

  • Hostname: The name of the LoadMaster host that generated the message.

  • Subsystem: The component or module that originated the message (for example, kernel, wafd, ssomgr, and so on).

  • Message Data: The actual content of the message.

All syslogs are rotated every seven days (older logs are removed).

Disk Usage - This section provides an indication of the percentage used/free of the log partition. Color-coding is used to highlight different usage levels:

  • 0% to 50%: green
  • 50% to 90%: orange
  • 90% to 100%: red

Boot.msg File - contains information, including the current version, during the initial starting of LoadMaster.

Warning Message File - contains warnings logged during the operation of LoadMaster.

System Message File - contains system events logged during the operation of LoadMaster. This includes both operating system-level and LoadMaster internal events.

Nameserver Log File - show the DNS name server log.

Nameserver Statistics - show the latest name server statistics.

IPsec IKE Log - show the IPsec IKE log.

Note: The IPsec IKE Log button does not appear if VPN Management is not configured under System Configuration > Network Setup > VPN Management.

WAF Debug Log File - contains debug traces useful for debugging WAF issues. Only enable this option when requested to do so by Progress Kemp Technical Support.

Note: The WAF Debug Log File button does not appear if there are no WAF debug logs.

WAF Event Log File- contains logs for most recently triggered WAF rules.

Note: The WAF Event Log File button does not appear if there are no WAF event logs.

Audit LogFile - contains a log for each action which is performed by a user; either using the API or the WUI. This will only function if session management is enabled. For further information on session management, refer to the Admin WUI Access section.

Clear Logs - clears the warning and system message logfiles. You can either click Clear All to clear all the system log files or click the arrow to select specific log files to clear.

Save Logs - you can use this option if you need to send logs to Progress Kemp support as part of a support effort. Click Save All to save all system log files to your computer and forward them to Progress Kemp support. Click the arrow to select specific log files to save. The logs are compressed. You can view/read them using a text editor after you uncompress the file.

The LoadMaster automatically rotates logs so the hard disk does not fill up with log files.

Debug Options

The LoadMaster has a range of features that will help you and Progress Kemp Support staff with diagnosing connectivity issues. Clicking Debug Options brings up the screen shown below.

Note: WARNING – we do not recommend using debug commands during normal operation. They should ideally only be used in conjunction with a Progress Kemp Support Technician’s recommendations.
Note: Debug commands have performance impacts on the LoadMaster and may expose your system to additional security vulnerabilities during the time they are running.

Disable All Transparency

Disables transparency on every Virtual Service and forces them to use Layer 7. Use with caution.

Note: This option is only for debugging and does not replace the normal controls to enable and disable transparency on a per-Virtual Service basis.
Note: Using this option to disable transparency saves a copy of the configuration file before disabling transparency. When transparency is turned back on (not all Virtual Services may have had transparency turned on before the change), the original configuration is restored. Therefore, any changes to the configuration during this time are lost. This includes creating new Virtual Services.

Enable L7 Debug Traces

This option enables debugging on all Layer 7 (L7) connections.

Generates log traffic in the message files. Due to the large amount of files being logged it slows down L7 processing.

CAUTION: Enabling this option can consume more resources and it is possible that some authorization parameters may be exposed. Only enable this option if recommended by Progress Kemp Support.

Enable Extended L7 Debug

Click Enable Extended Debug to enable L7 extended debug options.

In LoadMaster firmware version 7.2.53, the Enable Extended Debug option was enhanced. Enabling this option may be needed when performing extensive testing.

Once the Enable Extended L7 Debug option is enabled, a Process Debug button appears. Clicking Process Debug displays the list of processes and the debug level.

By default, the generated logs cannot be exported from the device. The user has to sanitize the logs before providing them externally. Logs should only be enabled for debug purposes and disabled immediately after. Logs should be removed from the LoadMaster as soon as possible when the debugging is completed.

CAUTION: Enabling the Extended L7 Debug option can consume more resources and it is possible that some authorization parameters may be exposed. Only enable this option if recommended by Progress Kemp Support.

When extended debugging is enabled, an additional Extended Debug configuration section becomes available in the Virtual Service modify screen (Virtual Services > View/Modify Services > Modify) for all VSs. When using Sub-Virtual Services (SubVSs), the Extended Debug settings are also inherited by the SubVS, so that a single call can be logged in its entirety. It is also possible to enable debug on a single SubVS if required and a further option to limit the logging by specifying the client IP address. All logs associated with this feature are recorded in the system messages file messages.txt.

There are two fields that you can configure in this section:

  • L7 Debug Level: Set the level of Layer7 debugging for this Virtual Service. Possible values are as follows:

    • No Debug

    • Call Tracing

    • Full Debug

    • Full Debug + HTTP Headers

    CAUTION: Setting the L7 Debug Level to Full Debug + HTTP Headers may expose sensitive information.

    Call Tracing is a basic level log that displays most relevant operations, while Full Debug displays all available debug logs, which is the same as the global setting of Enable L7 Debug Traces but on a per-VS level.

    Note: By default, the L7 Debug Level is set to No Debug for all Virtual Services and SubVSs. To enable logging for a particular Virtual Service or SubVS, you must set the L7 Debug Level to Call Tracing or Full Debug in the Extended Debug section of the Virtual Service or SubVS modify screen.

  • Client to Trace: It is also possible to limit the debug information even further by specifying a client IP address (you can specify an IPv4 or IPv6 address). If an address is specified, only connections coming from that specific client IP are logged/traced. This allows debugging capability from a single address.

Enable IRQ Pinning

Click the button to enable Interrupt Request Line (IRQ) pinning. This is disabled by default.

CAUTION: Only enable this option in consultation with Progress Kemp Support.
Note: When you change the IRQ pinning option from off to on, IRQ pinning is enabled on all network interfaces that are assigned an IP address. When IRQ pinning is enabled and you add an IP address to an unconfigured interface, that interface will not have IRQ pinning enabled until you either toggle the IRQ pinning off and back on again, or the system is rebooted.

Perform an l7adm

Note: Displays raw statistics about the L7 subsystem.

Enable WAF Debug Logging

You can enable logging to get detailed information about traffic that is analyzed by the Web Application Firewall (WAF). Information that is contained in the logs includes the time that LoadMaster WAF received the request from your LoadMaster resource, detailed information about the request, and the action for the rule that each request matched.

Enable WAF debug traces.

CAUTION: Please be aware that enabling this option will generate logs that may include Personally Identifiable Information as defined under the General Data Protection Agreement (EU GDPR). You should follow your organization's best practice to protect this information which may include anonymizing, deleting, or encrypting the data within the logs.
Note: This generates a lot of log traffic. It also slows down WAF processing. Only enable this option when requested to do so by Progress Kemp Technical Support. Kemp does not recommend enabling this option in a production environment.
The WAF debug logs are never closed and they are rotated if they get too large. WAF needs to be disabled and re-enabled in all WAF-enabled Virtual Service settings to re-enable the debug logs. Alternatively, perform a rule update, with rules that are relevant for the Virtual Service(s).

Enable IRQ Balance

Enable this option only after consulting with Progress Kemp support staff.

Enable TSO

Enable TCP Segmentation Offload (TSO).

Note: Only modify this option after consultation with Progress Kemp Technical Support. Changes to this option will only take affect after a reboot. For Intel 100G interfaces, TSO is always enabled and these interfaces are not affected by changing this option.

Enable TCP SACK

Click this button to enable TCP SACK (Selective ACKnowledgement) processing. This is a global setting that affects all Layer 7 Virtual Services. It only works if TCP SACK is enabled on a Virtual Service client and the LoadMaster.

Enable Layer 4 IPv6 Forwarding

This option, when enabled (the default), supports IPv6 forwarding behaviour as supported on LoadMaster versions previous to LoadMaster Operating System (LMOS) 7.2.50. When this option is disabled, it supports the IPv6 forwarding behavior required by the IPv6 standards. If IPv6 forwarding behavior that conforms to the IPv6 standards is required in your deployment, you should disable this option.

Enable/Disable CLI VS Management

Enable or disable the Command Line Interface (CLI) Service Management function.

GEO Debugging

Clicking GEO Debug displays a GEO Debug page. The GEO Debug page displays the GEO configuration related information and various GEO Debug Options. For more details on GEO Debugging, refer to the GEO System Info / Debug section.

Perform a PS

Performs a ps on the system.

Perform a Top

Performing a top command displays memory, CPU, and I/O usage for the LoadMaster. You can specify the number of samples and an interval between them (the default is 10 samples and a 1 second interval). You can also show threads and/or sort by memory usage by selecting the appropriate check boxes. By default the results are sorted by CPU usage.

Include Top in Backups

By default, the LoadMaster does not include a top output in backups. This can be enabled by selecting this check box. When included in backups, top is run using the default parameters (regardless of what is configured in the WUI) and is sorted by memory usage.

Display Meminfo

Displays raw memory statistics.

Display RAID Information

Note: The Display RAID Information and Display RAID Disks Information buttons only appear if a RAID controller is installed on the LoadMaster.

Display the Redundant Array of Independent Disks (RAID) controller details. Some example information is below:

-------------------------------------------------------------------
Controller details
-------------------------------------------------------------------
- Chip ID................: 10
- Parent Controller Index: 255
- OS Physical Name.......: /dev/sda
- Serial Number..........: 427491329
- AES Power on State.....: 0
- Sata Ports.............: 2
-------------------------------------------------------------------
Raid Port 0 details
-------------------------------------------------------------------
- Raid Model Name..............: H/W RAID1
- Raid Serial Number...........: OUEYEXCXTQ53GE1BSOSN
- EZBackup Disk Support........: 0
- Port Multiplier port.........: 0
- Raid Capacity................: 953 (29 GB)
- Raid Capacity low word.......: 0
- Raid State...................: 1 (Active)
- Raid Status..................: 3 (Normal)
- Raid Level...................: 1 (Raid 1 (mirror))
- Mark Type....................: 0
- Active Member................: 15
- Active Level.................: 0
- Rebuild Priority.............: 3
- Standby Timer................: 0
- Total members in the RAID....: 2
Member disk 0
- Ready....................: 1
- Lba 48 Bit Support.......: 1
- SATA Page................: 0
- SATA Port................: 0
- SATA Base................: 0
- SATA Size................: 953
----------------------------------------------------------
Member disk 1
- Ready....................: 1
- Lba 48 Bit Support.......: 1
- SATA Page................: 0
- SATA Port................: 1
- SATA Base................: 0
- SATA Size................: 953
----------------------------------------------------------

Display RAID Disks Information

Display details about the RAID disks. Some example information is below:

-------------------------------------------------------------------
Sata Port 0 details
-------------------------------------------------------------------
- Disk Model Name..............: 32GB SATA Flash Drive
- Disk Serial Number...........: C0122916B01000000074
- Disk Firmware Version........: SFDC001D
- EZBackup Disk Support........: 1
- Port Multiplier port.........: 15
- Disk Capacity................: 954 (29 GB)
- Port Type....................: 2 (RAID)
- Port Speed...................: 2 (GB)
- Page 0 State.................: 2
- Page 0 Raid Index............: 0
- Page 0 Member Index..........: 0
- Page 0 Raid Name.............:
- Page 0 Raid Serial Number....:
- Page 0 Raid Segment Base.....: 0
- Page 0 Raid Size.............: 953
- Page 0 Raid EZ Backup Support: 0
- Page 1 State.................: 0
- Page 1 Raid Index............: 0
- Page 1 MemberIndex...........: 0
- Page 1 Raid Name.............:
- Page 1 Raid Serial Number....:
- Page 1 Raid Segment Base.....: 0
- Page 1 Raid Size.............: 0
- Page 1 Raid EZ Backup Support: 0
- PortErrorStatus..............: 0
-------------------------------------------------------------------
Sata Port 1 details
-------------------------------------------------------------------
- Disk Model Name..............: 32GB SATA Flash Drive
- Disk Serial Number...........: E011321290100000005A
- Disk Firmware Version........: SFDC001D
- EZBackup Disk Support........: 1
- Port Multiplier port.........: 15
- Disk Capacity................: 954 (29 GB)
- Port Type....................: 2 (RAID)
- Port Speed...................: 2 (GB)
- Page 0 State.................: 2
- Page 0 Raid Index............: 0
- Page 0 Member Index..........: 1
- Page 0 Raid Name.............:
- Page 0 Raid Serial Number....:
- Page 0 Raid Segment Base.....: 0
- Page 0 Raid Size.............: 953
- Page 0 Raid EZ Backup Support: 0
- Page 1 State.................: 0
- Page 1 Raid Index............: 0
- Page 1 MemberIndex...........: 0
- Page 1 Raid Name.............:
- Page 1 Raid Serial Number....:
- Page 1 Raid Segment Base.....: 0
- Page 1 Raid Size.............: 0
- Page 1 Raid EZ Backup Support: 0
- PortErrorStatus..............: 0

Display Slabinfo

Displays raw slab statistics.

Perform an Ifconfig

Displays raw Ifconfig output.

Perform a Netstat

Displays Netstat output.

Include Netstat in Backups

By default, the LoadMaster includes a Netstat output in backups taken. When this is included, backups take longer to complete. You can stop the Netstat output from being included by disabling this option.

Reset Statistic Counters

Reset all statistics counters to zero and delete any old graphs. This also deletes the Round Robin Database (RRD) files but these files are automatically recreated when needed.

Flush OCSPD Cache

When using OCSP to verify client certificates, OCSPD caches the responses it gets from the OCSP server. This cache can be flushed by pressing this button. Flushing the OCSPD cache can be useful when testing, or when the Certificate Revocation List (CRL) has been updated.

Stop IPsec IKE Daemon

Stop the IPsec IKE daemon on the LoadMaster.

Note: If this button is clicked, the connection for all tunnels will go down.

Perform an IPsec Status

Display the raw IPsec status output.

Enable IKE Debug Level Logs

Control the IPsec IKE log level.

Flush SSO Authentication Cache

Clicking the Flush SSO Cache button flushes all Single Sign-On (SSO) records, resets all authentication server statuses, resets the KCD domain (if relevant) and re-reads the configuration. This has the effect of logging off all clients using Single Sign-On to connect to the LoadMaster.

SSO LDAP server timeout

Set the SSO LDAP server timeout value in seconds (default value is 5 seconds).

SSO Internal State

Click SSO State to show the internal state for the SSO Manager.

Linear SSO Logfiles

By default, older log files are deleted to make room for newer log files, so that the filesystem does not become full. Selecting the Linear SSO Logfiles check box prevents older files from being deleted.

Note: When using Linear SSO Logging, if the log files are not periodically removed and the file system becomes full, access to ESP-enabled Virtual Services will be blocked, preventing unlogged access to the virtual service. Access to non-ESP enabled Virtual Services are unaffected by the Linear SSO Logfile feature.

Netconsole Host

If directed by a Progress Kemp support engineer, you can use this feature to send critical kernel logs to a syslog server in the event of the LoadMaster failing or crashing. A syslog server configured as the Netconsole Host receives all critical kernel messages.

You can select which interface the Netconsole Host is set to using the Interface dropdown.

Note: Please ensure that the Netconsole Host IP specified is on the selected interface as errors may occur if it is not.
Note: The Netconsole Host parameter can be set to either an IPv4 or IPv6 address, the only restriction being that the Netconsole IP address must be in the same address family as the LoadMaster’s IP address on the selected Interface. In other words, if the LoadMaster’s IP address on the selected Interface is an IPv4 address, then the Netconsole Host IP address must be an IPv4 address as well. If you want to use an IPv6 address for Netconsole Host, then you must select an Interface on which the LoadMaster has an IPv6 address.
Note: Netconsole is not configurable on a bonded interface.

Ping Host

Performs a ping on the specified host. The interface which the ping should be sent from can be specified in the Interface drop-down list. The Automatic option selects the correct interface to ping an address on a particular network.

The interface tries to determine if the address to ping is an IPv4 or IPv6 address and selects the correct command to perform the ping. For an address in numeric form this is simple, however this is not possible for non-numeric addresses so they will always be treated as an IPv4 address.

Ping6 Host

Perform a ping6 of a specific IPv6 host.

Traceroute Host

Perform a traceroute of a specific host.

Kill LoadMaster

Permanently disables all LoadMaster functions. The LoadMaster can be re-enabled by being relicensed.

Note: Please do not kill your LoadMaster without consulting Progress Kemp Technical Support.
Note: The Kill LoadMaster option will not be available in LoadMasters which are tenants of the Multi-Tenant LoadMaster.

Enable DHCPv6 Client

When this option is enabled, the DHCPv6 client will run on the primary interface. This provides the capability to obtain an IPv6 address on boot. If you want DHCPv6 to be run on every boot, keep this option enabled. However, this is a long running process and it keeps running in the background when it is enabled so if you only need an IPv6 address to be assigned and you do not need to renew and release the IPv6 address you should disable this option after the IPv6 address is assigned.

TCP dump

A TCP dump can be captured either by one or all Ethernet ports. You can specify the address and port parameters in the WUI. To specify additional, optional parameters - use the Application Programming Interface (API) to perform a TCP dump.

You can stop and start the dump. You can also download it to a particular location. The results of the TCP dump can then be analysed in a packet trace analyser tool such as Wireshark.

For more information, refer to the Packet Trace Guide Technical Note.