Contents

1   Introduction

This document describes how to perform the health check on the MTAS running in virtualized environment. The health check tasks described in Section 2 Health Check Procedure are recommended to be performed before and after a system update or upgrade, a normal backup, or during the periodic maintenance.

1.1   Prerequisites

This section states the prerequisites for performing the health check procedure.

1.1.1   Documents

Before starting this procedure, ensure that the following information or documents are available:

• The release information for the MTAS software level that is intended to be run in the MTAS and MTAS RDP versions.

Note:  

The release information can, for example, be found in delivery reports, delivery specifications, delivery notes, release notes, or correction notes.

1.1.2   Knowledge

It is assumed that the user of this document is familiar with the Operation and Maintenance (O&M) area, in general. It is also assumed that the user is familiar with the concepts, terminology, and abbreviations within this area.

1.1.3   Tools

The following tool is required to check a summary of the health check:

• Any web browser supporting HTML 4.01.

2   Health Check Procedure

Health Check consists of a set of checks which verifies the status of the cluster, its fundamental functions, services, and external interfaces. These checks are called Health Check steps.

All steps are grouped into the following profiles:

• Basic
• Full
• PreUpgrade
• PostUpgrade

For a detailed description of the profiles, see Section 4 Health Check Profiles.

The Basic profile contains basic checks that determine decision of the MTAS node health status. The MTAS node can be considered healthy if all the checks are OK. By default, Health Check with Basic profile is performed periodically once per hour, but the periodicity is possible to change; see Section 2.1.2.2 Scheduled Periodic Health Check. In troubleshooting situations or when more information is desired, the checks can be performed manually, optionally with a broader profile.

When the execution of a profile is finished, a final verdict is produced by the Health Check. The result is written to the XML and HTML reports.

The result of the Health Check can be the following:

• OK (0) when the return of the steps is OK or INFO.
• VERIFY (2) when at least one of the steps return with VERIFY, and the others return with INFO, OK.
• FAIL (3) when at least one of the steps returns with FAIL and the others return with VERIFY, INFO, OK.
• ERROR (255) when at least one of the steps return with ERROR, and the others return with FAIL, VERIFY, INFO, OK.

2.1   Execution of Automatic Health Check

Health Check is integrated to the Crash Dump and Console Log Collection Service (CDCLS). Health Check can be performed by executing CDCLS packer objects. Refer to LEM Error Dump and Log User Guide.

It is possible through the Ericsson Command-Line Interface (ECLI) or directly from the System Controller.

2.1.1   Health Check Using ECLI

The Health Check profile packers are listed with the cdclsv-list command and executed with the cdclsv-invoke command. Execution status can be checked by cdclsv-status command.

Steps

1. Log on to the ECLI:

   ssh <username>@<oam-vip> -p 830 -t -s cli

2. Check which profiles are available:

   cdclsv-list

   The following is an example output:

   cdclsPk=DcMtasBasic,cdcls=CDCLSvSite cdclsPk=DcMtasFull,cdcls=CDCLSvSite cdclsPk=DcMtasLimited,cdcls=CDCLSvSite cdclsPk=DcMtasSla,cdcls=CDCLSvSite cdclsPk=HcMtasBasic,cdcls=CDCLSvSite cdclsPk=HcMtasFull,cdcls=CDCLSvSite cdclsPk=HcMtasPostUpgrade,cdcls=CDCLSvSite cdclsPk=HcMtasPreUpgrade,cdcls=CDCLSvSite

3. Select one of the profiles listed in the previous step:

   HcMtasBasic HcMtasFull HcMtasPostUpgrade HcMtasPreUpgrade

4. Start health check with the selected profile:

   cdclsv-invoke

   cdclsPk=<profile>,cdcls=CDCLSvSite

   For example, with the HcMtasBasic profile: cdclsv-invoke cdclsPk=HcMtasBasic,cdcls=CDCLSvSite

5. Check the status of the packing:

   cdclsv-status

   cdclsPk=<profile>,cdcls=CDCLSvSite

   Example with the HcMtasBasic profile:

   cdclsv-status cdclsPk=HcMtasBasic,cdcls=CDCLSvSite

   The results are found in:

   /cluster/storage/no-backup/hc/

2.1.2   Health Check Using CDCLS

2.1.2.1   Manually Started Health Check

The Health Check profile packers are listed with the cdclsv-list-packers command and executed with the cdclsv-pack command. Execution status can be checked with the cdclsv-pack-status command.

Steps

1. Check which profiles are available:

   cdclsv-list-packers | grep cdclsPk=HcMtas

   The following result is shown:

   cdclsPk=HcMtasFull,cdcls=CDCLSvSite cdclsPk=HcMtasBasic,cdcls=CDCLSvSite cdclsPk=HcMtasPostUpgrade,cdcls=CDCLSvSite cdclsPk=HcMtasPreUpgrade,cdcls=CDCLSvSite

2. Select one of the profiles listed in the previous step:

   HcMtasFull HcMtasBasic HcMtasMediumPriority HcMtasPostUpgrade HcMtasPreUpgrade

3. Start health check with the selected profile:

   cdclsv-pack cdclsPk=<profile>,cdcls=CDCLSvSite

   Example with the HcMtasBasic profile:

   cdclsv-pack cdclsPk=HcMtasBasic,cdcls=CDCLSvSite

4. Check the status of the packing:

   cdclsv-pack-status cdclsPk=<profile>,cdcls=CDCLSvSite

   Example with the HcMtasBasic profile:

   cdclsv-pack-status cdclsPk=HcMtasBasic,cdcls=CDCLSvSite

   The results are found in:

   /cluster/storage/no-backup/hc/

2.1.2.2   Scheduled Periodic Health Check

Health Check profiles are possible to schedule for periodic execution. By default, the Basic profile is executed in every hour, while other profiles are not scheduled for automatic execution.

The configured period can be read with the cdclsv-get-pack-period command, for example:

cdclsv-get-pack-period cdclsPk=HcMtasBasic,cdcls=CDCLSvSite

To set or change periodicity, use the cdclsv-set-pack-period command, for example:

cdclsv-set-pack-period cdclsPk=HcMtasBasic,cdcls=CDCLSvSite 3600

The value 0 means that the scheduled execution is switched off.

The scheduling periods are synchronized to entire hours. For example: if the period is set to 1200 (20min), Health Check is executed at the 0th, 20th, and 40th minutes of every hour.

2.2   Health Check Results

Health Check results are stored in directory /storage/no-backup/hc. Each Health Check run results in a separate package, a gzipped tar archive which contains the checkers status.

Furthermore, the Health Check report HTML files are copied into the directory /storage/no-backup/hc_reports.

2.2.1   Contents of the Health Check Result Package

The package contains the following items:

• summary.html: Contains general information about checkers status in the HTML format.
• summary.xml: Contains general information about checkers status in the XML format.
• <Checker name>.log: Contains data gathered by Data Collection. This log is not available if a particular Health Check step does not return data to the log, meaning that the result of the checker is OK. The Sla.log is an exception, since the log file is generated regardless of the SLA checker result.
• <Checker name>: Directory with data gathered by Data Collection step. The directory is not available if a particular Health Check step does not copy data or create specific structure. The Sla Directory is an exception, since this directory is generated regardless of the SLA checker result.

Data gathered by Data Collection is stored in result package only if any checker detects problems.

2.2.2   Housekeeping of the Results of Health Check

Housekeeping is required, as the results are collected cumulatively.

The housekeeping of directory /storage/no-backup/hc is configured in the ECLI. The configuration can be changed and checked using the ECLI, for example:

>dn ManagedElement=1,SystemFunctions=1,FileM=1,FileGroupPolicy=healthCheck

>show -v

The following is an example output:

fileGroupPolicyId="healthCheck"
fullFileGroupAction=DISCARD_OLDEST
maxFileGroupSize=1048576
maxNumberFiles=0
retentionTime=1440
userLabel=[]

The housekeeping of directory /storage/no-backup/hc_reports is configured in the ECLI. The configuration can be changed and checked using the ECLI, for example:

>dn ManagedElement=1,SystemFunctions=1,FileM=1,FileGroupPolicy=healthCheckReports

>show -v

The following is an example output:

fileGroupPolicyId="healthCheckReports"
fullFileGroupAction=DISCARD_OLDEST
maxFileGroupSize=10240
maxNumberFiles=24
retentionTime=1440
userLabel=[]

2.3   SLA Results

The result of Sla step is an exception, since it is generated and stored regardless of the Sla checker result.

The Sla result contains the following items:

• Vm_KPI_<Date_BeginTime_EndTime>.log Contains the following measurement for each VM:
  • Central Processing Unit (CPU) Total and CpuSteal
  • Total Memory, Used, and Free Memory
  • Free and Used Disk for System Controllers (SCs)
• PerCore_KPI_<Date_BeginTime_EndTime>.log Contains the following measurement for each CPU:
  • CPU Total and CpuSteal
• Network_KPI_<Date_BeginTime_EndTime>.log Contains the following measurement for transmit and receive side of each interface of PLs:
  • Throughput
  • Total number of packets
  • Dropped packets
  • Error packets
• Sla_Verdict_Details.log:
  Contains detailed information for each VERDICT.

Note:  

The Begin_time and End_Time in the KPI logs, shows the time interval for the KPI data collection.

2.4   Health Check Verdict

The result from the checks is stored in summary files. The verdict is a way to inform the user about status of the individual checks. The definitions of the different verdicts are shown in Table 1.

3   Health Check Steps

3.1   AlarmsAndNotifications

This step checks if there are any unresolved alarms or notification. If a non-OK verdict is given, an AlarmsAndNotifications directory is packed into the result package, where log files, containing the details of unresolved alarms and notifications, can be found for manual examination.

Verdict

3.2   AllMtasPortStatus

This step verifies if the MTAS ports are open.

Verdict

3.3   BackupList

This step checks if there is an active backup available to restore.

Verdict when step is executed from PreUpgrade profile:

Verdict when step is executed from any other profile:

3.4   ChargingBackupEvents

This step verifies if there are buffered charging events.

Verdict when step is executed from PreUpgrade profile:

Verdict when step is executed from any other profile:

3.5   CoreMWStatus

This step verifies if there are any AMF entities with questionable health.

Verdict

3.6   CpuLoadOnPLs and CpuLoadOnSCs

These steps verify the CPU load for each core of each node (PLs or SCs), and the average CPU load on each node (PLs or SCs).

The limits (max and average) for comparison depend on the profile this step has been called from.

For the PreUpgrade profile, both of the limits are set to 30%.

For any other profile, the limits for comparison are the values of the environment variables (in the following order):

1. LOAD_REG_CPU_MAX_LIMIT is the limit for the max load of each core.

   LOAD_REG_CPU_AVG_LIMIT is the limit for the average load of the cores of a CPU.

   If the variables are defined.

2. If LOAD_REG_CPU_MAX_LIMIT or LOAD_REG_CPU_AVG_LIMIT are not defined, LOAD_REG_LIMIT is used, if defined.
3. If LOAD_REG_LIMIT is not defined, both limits (max and average) are set to 85%.

Verdict when step is executed from PreUpgrade profile:

Verdict when step is executed from any other profile:

3.7   DiameterPortsStatus

This step verifies Diameter ports status. Data about diameter port configuration is gathered from COM management objects.

Verdict

3.8   DiskUsageOnSCs

This step verifies the level of available space on SCs disks.

The verdict depends on the profile this step has been called from.

Verdict when step is executed from PreUpgrade profile:

Verdict when step is executed from any other profile:

3.9   DrbdStatus

This step verifies whether the shared block device of the cluster is functioning correctly. Connection state, disk state, and out-of-sync blocks are verified.

Verdict

3.10   eVIP

This step verifies status of eVIP on all active ALBs.

Verdict

3.11   MemoryUsageOnPLs and MemoryUsageOnSCs

This step checks the memory use on the nodes (PLs or SCs).

The limit for comparison is the value of the environment variable (in the following order):

1. LOAD_REG_MEMORY_LIMIT, if defined.
2. If LOAD_REG_MEMORY_LIMIT is not defined, the limit is the value of LOAD_REG_LIMIT, if defined.
3. If none of these variables are defined, the limit is set to 85% of the available memory.

The verdict depends on the profile this step has been called from.

Verdict when step is executed from PreUpgrade profile:

Verdict when step is executed from any other profile:

3.12   Mmas

This step verifies whether MMAS traffic instances are operational on every payload node.

Verdict

3.13   NETCONFConnection

This step verifies if NETCONF is configured on only one controller.

Verdict

3.14   NeLSConnectivity

This step verifies the connectivity between the MTAS and the NeLS server.

Verdict

3.15   NetworkConnectivity

This step verifies the connectivity between each SC/PL node.

Verdict

3.16   NodeOutage

This step verifies SCs/PLs state and checks for recovery events in the last 24 hours of ISP logs.

Verdict

3.17   OngoingQueryPurge

This step verifies that a QueryPurge operation is ongoing.

The verdict depends on the profile this step has been called from.

Verdict when the step is executed from the PreUpgrade profile:

Verdict when the step is executed from any other profile:

3.18   OperationalState

This step checks MTAS operational state using COM interfaces.

Verdict

3.19   SIPPortsStatus

This step verifies if SIP ports are open.

Verdict

3.20   SS7Connections

This step verifies SS7 stack status.

Verdict

3.21   Sla

This step verifies the status of Service Level Agreement (SLA) and records the Key Performance Indicator (KPI) for the Virtual Machine (VM), Core and Network Interface under the Sla Directory for the last hour.

Verdict

For more details on the SLA Results, refer to Section 2.3 SLA Results.

For information on how to troubleshoot SLA, refer to Section 4.6 in MTAS Troubleshooting Guideline.

3.22   SecurityStatus

This step verifies if Core MW security package is installed on the SC/PL nodes.

Verdict

3.23   SoftwareInventory

This step collects a list of RPM/SDP files available on the SC/PL nodes.

Verdict

3.24   SoftwareVersions

This step collects the software components installed on the cluster and checks whether there are "not used" ones among them.

It also compares the installed non-MTAS components to the expected ones included in the software package.

The verdict depends on the profile this step has been called from.

Verdict when the step is executed from PreUpgrade profile:

Verdict when the step is executed from any other profile:

3.25   SystemEnvironmentVariables

This step checks whether the vDicos environment variables are set correspondingly to the reference values.

Verdict

3.26   SystemStatus

This step verifies the system services status. Data is gathered by cmw-status.

Verdict

3.27   VirtualDicosProcessOutage

This step checks status of vDicos Virtual Machines.

Verdict

3.28   VmLogs

This step inspects vDicos Virtual Machine Logs if severe error messages logged in the last 24 hours.

Verdict

3.29   XdmsCaiLicence

This step checks whether the XDMS server certificate is valid.

Verdict

3.30   XdmsInstance

This step verifies if the XDMS instance exists in MMAS and if its status is OK.

Verdict

3.31   XdmsRpm

This step verifies if every necessary XDMS-related package is installed on the system.

Verdict

3.32   XdmsTrafficApps

This step verifies traffic instance logs from the MMAS server.

Verdict

4   Health Check Profiles

This section describes health check profiles content. All health checks are grouped according to importance: Mandatory, Medium, and Low priority.

4.1   HcMtasBasic

This profile contains checkers only for the most crucial parts of the system. Health Check using this profile is automatically performed every hour by default.

HcMtasBasic profile includes the following steps:

• AlarmsAndNotifications
• CoreMWStatus
• DrbdStatus
• eVIP
• Mmas
• NETCONFConnection
• NeLSConnectivity
• NetworkConnectivity
• NodeOutage
• OperationalState
• SS7Connections
• Sla
• SystemStatus
• VirtualDicosProcessOutage
• XdmsInstance

4.2   HcMtasFull

This profile contains every checker available. By using this profile, a comprehensive set of information is constructed about system health.

Full profile contains all steps included in the Basic profile and the following steps:

• AllMtasPortsStatus
• BackupList
• ChargingBackupEvents
• CpuLoadOnPLs and CpuLoadOnSCs
• DiameterPortsStatus
• DiskUsageOnSCs
• MemoryUsageOnPLs and MemoryUsageOnSCs
• OngoingQueryPurge
• SIPPortsStatus
• SecurityStatus
• SoftwareInventory
• SoftwareVersions
• SystemEnvironmentVariables
• VmLogs
• XdmsCaiLicence
• XdmsRpm
• XdmsTrafficApps

Running Health Check using this profile can cause CPU load peaks and increase of memory use on primary SC.

4.3   HcMtasPreUpgrade

This profile is intended to be used before upgrade execution.

The PreUpgrade profile contains all steps from the Full profile. The following steps produce different verdicts when they are called from this profile:

• BackupList
• ChargingBackupEvents
• OngoingQueryPurge
• SoftwareVersions

4.4   HcMtasPostUpgrade

This profile is intended to be used after upgrade execution.

The PostUpgrade profile contains all steps from the Full profile. The following step produces different verdicts when it is called from this profile:

• SoftwareVersions

5   Problem Reporting

For any abnormal situation, refer to MTAS Troubleshooting Guideline.

If the problem still exists, the user can report it to the next level of support.

It is also important to collect the related data. For more information, refer to Data Collection Guideline for MTAS.