Self-health Check Script

The self-health check script is available since version 8.03 and its purpose is to provide a quick way of assessing the health of the appliance. The script detects the most common known issues and does not perform any automatic recovery.

If issues are detected, you can resolve them using the CLI and Available commands described in the previous section or by requesting help at support.flowmon.com.

The script checks:

• The status of the PostgreSQL database essential for GUI and configuration storage

• For deadlocks

• For waiting processes

• For disk and memory space exhaustion

• For locked profiles

• Whether all required RRD files are present.

To run the script, use the following command:

    sudo /usr/sbin/SHC/self-health-checker.sh

All functions and outputs have assigned priorities. Based on these priorities, status, and found issues, the script provides a health score of the appliance. The output of this command is also available in the support logs after they are downloaded to be provided to Flowmon Support. The output looks like this:

Testing possible problem causes for plugin "ads"
    Deadlocks:
        STATUS 0 - There weren't found any deadlocks
    Waiting processes:
        STATUS 9 - In database was found some waiting processes
            Process with PID: 3375 (ClientRead | idle | select 
InputTools.updateDatafeeds('1638948000',arr) waiting
            Process with PID: 3377 (ClientRead | idle | SET extra_float_digits = 
3) waiting
            Process with PID: 3403 (ClientRead | idle | DEALLOCATE 
pdo_stmt_000c9412) waiting
Testing possible problem causes for plugin "flowmon"
    Apache:
        STATUS 0 - No Apache fatal errors found in log
    Distributed architecture check:
        Checking RRD files...
        Source directory: /data/nfsen/profiles-stat/live
        Time interval: 2021/12/08 00:00:00 to 2021/12/08 08:15:21
        STATUS 0 - 127-0-0-1_p3000-npm.rrd: checked for empty time slots.
        STATUS 0 - 35-241-192-99_p2055-npm.rrd: checked for empty time slots.
        STATUS 0 - 127-0-0-1_p3000.rrd: checked for empty time slots.
        STATUS 0 - 35-241-192-99_p2055.rrd: checked for empty time slots.
        Checking DaCli php test scripts...
        STATUS 0 - DaCli:GetAllConfigsHash ended successfuly
        STATUS 0 - DaCli:GetFlowProxyInfo ended successfuly
        STATUS 0 - DaCli:GetSourcesProfilesAlertsInfo ended successfuly
        Checking php coredumps...
        STATUS 0 - No coredumps found.
    DA - Verify dist_managerd daemon:
        Checking if DA is enabled
            DA is disabled
    Psql - Deadlocks:
        STATUS 0 - There weren't found any deadlocks
    Psql - Database unavailability CentOS 7:
        STATUS 0 - Service "postgresql" for CentOS 7 running since Mon 2021-11-29 09:59:57 CET; 1 weeks 1 days ago
    Psql - Waiting processes:
        STATUS 5 - In database were found some waiting processes
            Process with PID: 3326 (ClientRead | idle | DEALLOCATE
pdo_stmt_00000016) waiting
    DA - Verify logs:
        Exhausted disk:
            STATUS 0 - Disk "system" usage is under quota 75% (61%)
            STATUS 0 - Disk "data" usage is under quota 75% (27%)
            STATUS 0 - Disk "boot" usage is under quota 75% (60%)
        Locked profiles:
            STATUS 0 - All profiles are unlocked
        Myricom driver:
            STATUS 0 - This device does not have a 10Gb ethernet card
        Drops on the network interfaces:
            STATUS 0 - Drops on the network interfaces were not found.
        RRD file:
            STATUS 0 - RRD check completed successfully.
        Check Samba connection:
            STATUS 0 - External Data Storage Settings is turned off
        Higher IOWAIT checker:
            STATUS 0 - No higher IOWAIT found
        System time compare:
            STATUS 0 - System and Database time are the same.
    Verify log from dmesg:
        Search occurrence "({O|o}ut [O|o]f [M|m]emory)" in dmesg
            Nothing found
        Search occurrence "([S|s]egfault)" in dmesg
            Nothing found
        Search occurrence "Link[0-9]?\ is\ [U|u]\[P|p]" in dmesg
            Nothing found
        Search occurrence "0Link[0-9]?\ is\ [D|d][O|o][W|w][N|n]" in dmesg
            Nothing found
        Search occurrence "Parity\ [E|e]rror\ detected" in dmesg
            Nothing found
        Search occurrence "[D|d]etected\ conn\ error" in dmesg
            Nothing found
        Search occurrence "[C|c]ould\ [N|n]ot\ [G|g]et\ [M|m]yricom\ [L|l]icensing\ [I|i]nformation" in dmesg
            Nothing found
    Verify logs:
        Search occurrence "({D|d}ropped CTM: [1-9][0-9]*)" in /var/log/daemon
            Nothing found
        Search occurrence "([S|s]equence [E|e]rrors: [1-9][0-9]*)" in /var/log/daemon
            Jul 1 10:50:01 tora nfcapd[31548]: stats - LogFlogSourcesStats - Ident: 127-0-0-1_p3000' Flows: 118, Packets: 1410492, Bytes: 1769640743, Sequence Errors: 45, Bad Packets: 0 Exporters count: 2
        Search occurrence "([R|r]estarting flowmonexp5)" in /var/log/messages
            Nothing found
    Verify nfsen.conf:
        STATUS 0 - File "/data/nfsen/etc/nfsen.conf" is correct
    Verify .nfstat:
        STATUS 6 - Problems found inside .nfstat files in profile channels
    Profile channel: "DNSQue-1aab30"
        - Size of profile channel in .nfstat is not set.
    Profile data collecting:
        STATUS 0 - All profiles collecting data correctly
    Visible channels in profiles:
        STATUS 0 - All profiles has number visible channels are under 33
Testing possible problem causes for plugin "idr"
    iSCSI connection to disk array:
        STATUS 0 - iSCSI Extension is not installed

The output of the command has changed over time so it would depend which version of the Flowmon system you are using. A status other than zero means there is some trouble. In this example, there are some waiting commands in the database, sequence errors which are because we had some flows lost, and a problem with one profile. Detailed information can be found in a log file that is in the same directory as the script. The file can be read by using the following command:

    less /usr/sbin/SHC/info.log