IPWorks Manual Health Check

1	Introduction
1.1	Prerequisites
1.1.1	Documents
1.1.2	Conditions
1.2	Related Information
2	Health Check Procedure
2.1	Checking Alarms
2.2	Checking UUID for Alarms
2.3	Checking LDE Dump Files
2.4	Checking Core Middleware Services
2.5	Checking CPU Load
2.6	Checking Disk Use
2.7	Checking Memory Use
2.8	Checking Internal Communication
2.9	Checking Controller Status
2.10	Checking Virtual IP Status
2.11	Checking Latest Available Backup
2.12	Checking IPWork Counters
2.13	Checking Nodes Restart
2.14	Checking IPWorks Application Logs
2.14.1	Checking DNS Logs
2.14.2	Checking ASDNS Logs
2.14.3	Checking SS Logs
2.14.4	Checking ENUM Logs
2.14.5	Checking ENUM-FE Logs
2.14.6	Checking Radius AAA Logs
2.14.7	Checking EPC AAA Logs
2.14.8	Checking DHCPv4 Logs
2.15	Checking MySQL Nodes Status
2.16	Checking Number of MySQL Tables
2.17	Checking Software Release Version
2.18	Checking License
2.19	Checking LDAP Connections
2.20	Checking SOAP Listening Status
2.21	Checking Backup Files
2.21.1	Checking the Backup Files
2.21.2	SS Backup Files
2.21.3	DNS Backup Files
2.21.4	MySQL NDB Cluster Backup Files
2.21.5	ENUM Backup Files
2.21.6	ENUM-FE Backup Files
2.21.7	Radius AAA Backup Files
2.21.8	EPC AAA Backup Files
2.21.9	DHCPv4 Backup Files
2.22	Checking AAA Service Port Listening Status
2.22.1	Checking Radius AAA Service Port Listening Status
2.22.2	Checking EPC AAA Service Port Listening Status
2.23	Checking DHCPv4 Service Port Listening Status
3	Problem Reporting
Reference List

1 Introduction

This document describes how to perform the health check procedure on the IPWorks. The health check procedures are recommended to be performed before and after a system update/upgrade, a normal backup, or during periodic maintenance.

If you want to perform the auto health check via ECLI, refer to the document IPWorks Auto Health Check.

1.1 Prerequisites

This section states the prerequisites for performing the health check procedure.

1.1.1 Documents

Before starting this procedure, ensure that the following information or documents are available:

For how to use Ericsson Command-Line Interface (ECLI), refer to Ericsson Command-Line Interface User Guide.
The current IPWorks version in the system.
The network plan and the System Controller (SC) address of the cluster.

1.1.2 Conditions

By default all actions are performed on the SC, unless otherwise specified.

1.2 Related Information

For the trademark information, typographic conventions, definition, and explanation of acronyms and terminology, see the following documents:

2 Health Check Procedure

2.1 Checking Alarms

To check active alarms reported by the system:

Log on to ECLI.
#ssh <user>@<OAM IP Address> -p <port> -t -s cli

Note:
The <user> can be root or IPWadministrator depending if Local Authorization is enabled or not.
Verify that no active alarms are shown.
>show ManagedElement=<Node Name>,SystemFunctions=1, Fm=1,totalActive

The output must be zero. The string <node_name> is specific for the ME.

If active alarms are present, check them.

>show ManagedElement=<Node Name>,SystemFunctions=1,
Fm=1 -m FmAlarm
show ManagedElement=<Node Name>,SystemFunctions=1,Fm=1 -m FmAlarm | filter Problem

For specific information on alarms, refer to IPWorks Alarm List.

Log out.
#exit

2.2 Checking UUID for Alarms

To check UUID on SC or PL:

# cat /sys/devices/virtual/dmi/id/product_uuid

2.3 Checking LDE Dump Files

For core dumps at Linux Distribution Extensions (LDE) level, check that the directory /cluster/dumps is empty.
If the directory is not empty, gather the information and report it to the next level of support.

2.4 Checking Core Middleware Services

Check that the state of the following system items at Core Middleware (Core MW) level is Status OK:

#cmw-status node app csiass comp node sg si siass su pm

2.5 Checking CPU Load

On the SC, enter the following:

>for VAR in /etc/cluster/nodes/all/*/hostname; do echo $(<$VAR); ssh $(<$VAR) "mpstat -P ALL"; done;

SC-1
Linux 3.12.55-52.42-default (SC-1)      08/29/16        _x86_64_        (2 CPU)

03:46:00     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
03:46:00     all    2.18    0.13    2.41    0.37    0.00    0.28    0.06    0.00    0.00   94.56
03:46:00       0    2.18    0.13    2.42    0.31    0.00    0.26    0.00    0.00    0.00   94.69
03:46:00       1    2.18    0.13    2.40    0.43    0.00    0.30    0.12    0.00    0.00   94.44
SC-2
Linux 3.12.55-52.42-default (SC-2)      08/29/16        _x86_64_        (2 CPU)

03:46:01     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
03:46:01     all    1.00    0.05    1.48    0.22    0.00    0.16    0.07    0.00    0.00   97.03
03:46:01       0    0.99    0.05    1.54    0.16    0.00    0.15    0.14    0.00    0.00   96.97
03:46:01       1    1.00    0.05    1.43    0.28    0.00    0.16    0.00    0.00    0.00   97.09
PL-3
Linux 3.12.55-52.42-default (PL-3)      08/29/16        _x86_64_        (2 CPU)

03:46:01     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
03:46:01     all    2.77    0.00    4.79    0.07    0.00    0.47    0.01    0.00    0.00   91.89
03:46:01       0    2.81    0.00    4.85    0.07    0.00    0.44    0.01    0.00    0.00   91.82
03:46:01       1    2.74    0.00    4.74    0.07    0.00    0.50    0.00    0.00    0.00   91.95
PL-4
Linux 3.12.55-52.42-default (PL-4)      08/29/16        _x86_64_        (2 CPU)

03:46:01     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
03:46:01     all    2.57    0.00    4.43    0.03    0.00    0.38    0.01    0.00    0.00   92.57
03:46:01       0    2.60    0.00    4.51    0.03    0.00    0.35    0.02    0.00    0.00   92.48
03:46:01       1    2.54    0.00    4.36    0.03    0.00    0.41    0.00    0.00    0.00   92.66

Verify that the %idle column shows values higher than 20%.

2.6 Checking Disk Use

On both SCs, enter the command:

#df -h -x tmpfs -x devtmpfs

Filesystem                                     Size  Used Avail Use% Mounted on
/dev/vda2                                       20G  2.4G   17G  13% /
/dev/vda4                                      9.8G  634M  8.6G   7% /var/log
/dev/vda1                                      3.9G  136M  3.5G   4% /boot
/dev/mapper/lde--cluster--vg-lde--cluster--lv   50G  3.6G   44G   8% /.cluster
169.254.100.101:/.cluster                       50G  3.6G   44G   8% /cluster
/dev/vda5                                       99G  2.6G   91G   3% /local/ipworks
com_fuse_module                                 50G  3.6G   44G   8% /var/filem/nbi_root

Verify that the used disk space does not exceed 80%, otherwise, clean up this partition.

2.7 Checking Memory Use

On every SC and PL, enter the following, enter the following:
> /opt/ipworks/common/scripts/ipworks_memory_health_check
```
SC-1
3.02633
SC-2
3.09388
PL-3
17.2859
PL-4
30.867
```
Verify that the free memory represents more than 10% of the total memory in all nodes.

2.8 Checking Internal Communication

All the links must be up. On the SC, enter the following:

>for VAR in /etc/cluster/nodes/all/*/hostname; do echo $(<$VAR); ssh $(<$VAR) /sbin/tipc-config -n -l; done;

SC-1
Neighbors:
<1.1.2>: up
<1.1.3>: up
<1.1.4>: up
Links:
broadcast-link: up
1.1.1:eth0-1.1.2:eth0: up
1.1.1:eth0-1.1.3:eth0: up
1.1.1:eth0-1.1.4:eth0: up
SC-2
Neighbors:
<1.1.1>: up
<1.1.3>: up
<1.1.4>: up
Links:
broadcast-link: up
1.1.2:eth0-1.1.1:eth0: up
1.1.2:eth0-1.1.3:eth0: up
1.1.2:eth0-1.1.4:eth0: up
PL-3
Neighbors:
<1.1.1>: up
<1.1.2>: up
<1.1.4>: up
Links:
broadcast-link: up
1.1.3:eth0-1.1.1:eth0: up
1.1.3:eth0-1.1.2:eth0: up
1.1.3:eth0-1.1.4:eth0: up
PL-4
Neighbors:
<1.1.1>: up
<1.1.2>: up
<1.1.3>: up
Links:
broadcast-link: up
1.1.4:eth0-1.1.1:eth0: up
1.1.4:eth0-1.1.2:eth0: up
1.1.4:eth0-1.1.3:eth0: up

2.9 Checking Controller Status

To check or verify the controller status:

Log on to the SC.
# ssh <username>@<OAM IP Address>

Retrieve the Distributed Replicated Block Device (DRBD) state:

# cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 3c1f46cb19993f98b22fdf7e18958c21ad75176d build by SuSE Build Service
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:25765636 nr:0 dw:25761540 dr:893245 al:23571 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Make sure that the SC is connected. That is, the value of the connection state field (cs) is connected.
Make sure that the disk state is normal. That is, the value of the ds field is UpToDate.
Determine if the SC is primary or secondary by checking the ro field:
- On the primary SC: Primary/Secondary
- On the secondary SC: Secondary/Primary

2.10 Checking Virtual IP Status

To check virtual IP status:

Run script on SC1.
SC-1:~ #/opt/ipworks/common/scripts/ipworks_evip_health_check

Check the result.

If the result is STATUS OK, the output is shown as the example below:

Connection closed by foreign host.
Connection closed by foreign host.
Connection closed by foreign host.
STATUS OK

If the result is STATUS NOT OK, proceed to check Abstract Load Balancer (ALB) Status.

Connect to the eVIP CLI:
# telnet `/opt/vip/bin/getactivecontrol` 25190
See the configured Abstract Load Balancer (ALB), and check that every listed ALB has status ACTIVE:
> show albs
```
0 : 'ipw_sig_sp' : ACTIVE
1 : 'ipw_data_sp' : ACTIVE


OK  
```

Use show agents for each listed ALB, check the agents status, and verify that all eVIP links or agents are ACTIVE, ACTIVE UP , and ACTIVE RDY.

> show agents ipw_sig_sp |grep -v '\[0\]'

> show agents ipw_data_sp |grep -v '\[0\]'

Take show agents ipw_sig_sp for example:

+------------------------------------[ ALB ipw_sig_sp (ACTIVE) ]-----------------------------------+
+----------------------------------------------- PN -----------------------------------------------+
| pnagent (4)                                     | lbesel_pn (28)                                 |
|[4] fe80::ff:fe01:21 : ACTIVE                    |[4] fe80::ff:fe01:21 : ACTIVE                   |
|[3] fe80::ff:fe01:1f : ACTIVE                    |[3] fe80::ff:fe01:1f : ACTIVE                   |
+--------------------------------------------------------------------------------------------------+
| ersipc (0)                                      | repdb (28)                                     |
|[4] fe80::ff:fe01:21 : ACTIVE                    |[4] fe80::ff:fe01:21 : ACTIVE                   |
|[3] fe80::ff:fe01:1f : ACTIVE                    |[3] fe80::ff:fe01:1f : ACTIVE                   |
+----------------------------------------------- LBE ----------------------------------------------+
| lbeagent (44)                                   | sesel_lbe (12)                                 |
|[4] fe80::1:f4ff:fe01:4 : ACTIVE                 |[4] fe80::1:f4ff:fe01:4 : ACTIVE                |
|[3] fe80::1:f4ff:fe01:3 : ACTIVE                 |[3] fe80::1:f4ff:fe01:3 : ACTIVE                |
+----------------------------------------------- FEE ----------------------------------------------+
| feeagent (46)                                   | sesel_fe (12)                                  |
|[4] fe80::1:f6ff:fe01:19 : ACTIVE UP             |[4] fe80::1:f6ff:fe01:19 : ACTIVE               |
|[3] fe80::1:f6ff:fe01:17 : ACTIVE UP             |[3] fe80::1:f6ff:fe01:17 : ACTIVE               |
+----------------------------------------------- SE -----------------------------------------------+
| seagent (12)                                    | lbesel_se (34)                                 |
|[4] fe80::1:f5ff:fe01:e : ACTIVE RDY             |[4] fe80::1:f5ff:fe01:e : ACTIVE                |
|[3] fe80::1:f5ff:fe01:d : ACTIVE RDY             |[3] fe80::1:f5ff:fe01:d : ACTIVE                |
+--------------------------------------------------------------------------------------------------+
| sesel_se (10)                                   |                                                |
|[4] fe80::1:f5ff:fe01:e : ACTIVE                 |                                                |
|[3] fe80::1:f5ff:fe01:d : ACTIVE                 |                                                |
+---------------------------------------------- IPSEC ---------------------------------------------+
| ikeagent (0)                                    | ipsecuagent (6)                                |
|                                                 |[4] fe80::ff:fe01:22 : ACTIVE RDY               |
|                                                 |[3] fe80::ff:fe01:20 : ACTIVE RDY               |
+--------------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------------+
                eRSIP state: ACTIVE                            cIPSEC state: ACTIVE RDY

Exit the eVIP CLI:
# exit

2.11 Checking Latest Available Backup

Log on to ECLI.
#ssh <user>@<OAM IP Address> -p <port> -t -s cli

Note:
The <user> can be root or IPWadministrator depending if Local Authorization is enabled or not.
Get the name of the latest created backup.
> show ManagedElement=<Node Name>,SystemFunctions=1,BrM=1,BrmBackupManager=SYSTEM_DATA,BrmBackupLabelStore=SYSTEM_DATA,lastCreatedBackup

lastCreatedBackup="<latest Backup name>" <read-only>
Get the date of the latest created backup.
>show -v ManagedElement=<Node Name>,SystemFunctions=1,BrM=1,BrmBackupManager=SYSTEM_DATA,BrmBackup=<latest Backup name> | filter creationTime
Log out.
>exit
Check that the latest backup is not older than 48 hours. This figure is just a recommendation, it depends on the periodicity of the scheduled backup in the system.

2.12 Checking IPWork Counters

To verify that the PmJob instances are active for the different IPWorks activated modules, execute the following command checking the ones that start with IPWorks:
#cmw-pmjob-list
Note:
- Other counters can be also active in the node.
- Currently, independently of the module's activated, all IPWorks counters are active in the system by default.
Check that the IPWorks counters are stored under:
#ssh <user>@<OAM IP Address> -p <port> -t -s cli

>show ManagedElement=<Node Name>,SystemFunctions=1,FileM=1,LogicalFs=1,FileGroup=PerformanceManagementReportFiles

The counters are accessible also under /var/filem/nbi_root/PerformanceManagementReportFiles from one of the Service Controllers (SCs).
Note:
- The <user> can be root or IPWadministrator depending if Local Authorization is enabled or not.
- Because of a current bug already reported, Local Authorization cannot be enabled in the node to access these files.
Go to the file location of IPWroks coutners and decompress the files using gunzip if there are compression files.
#cd /var/filem/nbi_root/PerformanceManagementReportFiles
- If there is no compression files there, go to Step 4 directly.
- If CompressionType is configured as GZIP, the counters files will be compressed. Firstly decompress the files.
  Here is an example for DnsASDNSDefaultJob, in which compressionType is configured as GZIP.
```
>ManagedElement=IPWorkNode1,SystemFunctions=1,Pm=1,PmJob=DnsASDNSDefaultJob
(PmJob=DnsASDNSDefaultJob)>show
PmJob=DnsASDNSDefaultJob
   compressionType=GZIP
   currentJobState=ACTIVE
   granularityPeriod=ONE_MIN
   reportingPeriod=ONE_MIN
   MeasurementReader=mr_1
(PmJob=DnsASDNSDefaultJob)>
```
  Decompress the latest files using gunzip from the location in the SC.
  
  #gunzip A20160603.2035+0200-2040+0200_1.xml.gz
  
  Note:
  Although it complains about lack of permission, the operation is successful.
Look for IPWorks counters reflecting an undesired behavior.
The file specified here is just an example.

grep -i "userunknown unabletocomply rejected notallowed invalid toobusy dontmatch unsupported missing" A20160603.2030+0200-2040+0200_1.xml

2.13 Checking Nodes Restart

Enter the following:
>for VAR in /etc/cluster/nodes/all/*/hostname; do echo $(<$VAR); ssh $(<$VAR) who -b; done;
Check that the date corresponds to the latest backup restore, or the latest planned cluster, or node reboot.

2.14 Checking IPWorks Application Logs

This section describes how to check server logs.