Contents

1   Introduction

This document describes how to perform the health check on the MTAS running in virtualized environment. The health check tasks described in Section 2 Health Check Procedure are recommended to be performed before and after a system update or upgrade, a normal backup, or during the periodic maintenance.

1.1   Prerequisites

This section states the prerequisites for performing the health check procedure.

1.1.1   Documents

Before starting this procedure, ensure that the following information or documents are available:

• The release information for the MTAS software level that is intended to be run in the MTAS and MTAS RDP versions.

Note:  

The release information can, for example, be found in delivery reports, delivery specifications, delivery notes, release notes, or correction notes.

1.1.2   Knowledge

It is assumed that the user of this document is familiar with the Operation and Maintenance (O&M) area, in general. It is also assumed that the user is familiar with the concepts, terminology, and abbreviations within this area.

1.1.3   Tools

The following tool is required to check a summary of the health check:

• Any web browser supporting HTML 4.01.

2   Health Check Procedure

Health Check consists of a set of checks which verifies the status of the cluster, its fundamental functions, services, and external interfaces. These checks are called Health Check steps. All steps are grouped into three profiles described in Section 4 Health Check Profiles.

The mandatory profile contains basic checks that determine decision of the MTAS node health status. The MTAS node can be considered healthy if all the checks are OK. By default, Health Check with mandatory profile is performed periodically once per hour. In troubleshooting situations or when more information is desired, the checks can be performed manually, optionally with a broader profile.

2.1   Packers List

To check which profiles are available use cdclsv-list-packers command. As a result the list of DNs is displayed:

cdclsPk=HcMtasLowPriority,cdcls=CDCLSvSite
cdclsPk=HcMtasMandatory,cdcls=CDCLSvSite
cdclsPk=HcMtasMediumPriority,cdcls=CDCLSvSite

Each DN denotes a reference to cdclsv pack object related to proper Health Check profile. Each cdclsv pack object has its own configuration and a list of checkers.

2.2   Manual Health Check

To perform selected profile use cdclsv-pack <DN>.

cdclsv-pack cdclsPk=HcMtasMandatory,cdcls=CDCLSvSite

Execution status of selected DN can be checked by cdclsv-pack-status <DN>.

cdclsv-pack-status cdclsPk=HcMtasMandatory,cdcls=CDCLSvSite

2.3   Health Check Results

Health Check results are stored in directory /storage/no-backup/hc. Each Health Check run results in a separate package, a gzipped tar archive which contains the checkers status.

The package contains the following items:

• summary.html : Contains general information about checkers status in the HTML format.
• summary.xml: Contains general information about checkers status in the XML format.
• <Checker name>.log: Contains data gathered by Data Collection. This log is not available if a particular Health Check step does not return data to the log, meaning that the result of the checker is OK. The Sla.log is an exception, since the log file is generated regardless of the Sla checker result.
• <Checker name>: Directory with data gathered by Data Collection step. The directory is not available if a particular Health Check step does not copy data or create specific structure. The Sla Directory is an exception, since this directory is generated regardless of the Sla checker result..

By default data gathered by Data Collection is stored in result package only if any checker detects problems.

2.4   SLA Results

The SLA Results are stored under the Sla Directory.

The directory contains the following items:

• Vm_KPI_<Date_BeginTime_EndTime>.log Contains the following measurement for each VM:
  • Central Processing Unit (CPU) Total and CpuSteal
  • Total Memory, Used, and Free Memory
  • Free and Used Disk for System Controllers (SCs)
• PerCore_KPI_<Date_BeginTime_EndTime>.log Contains the following measurement for each CPU:
  • CPU Total and CpuSteal
• Network_KPI_<Date_BeginTime_EndTime>.log Contains the following measurement for transmit and receive side of each interface of PLs:
  • Throughput
  • Total number of packets
  • Dropped packets
  • Error packets
• Sla_Verdict_Details.log:
  Contains detailed information for each VERDICT.

Note:  

The Begin_time and End_Time in the KPI logs, shows the time interval for the KPI data collection.

2.5   Health Check Verdict

The result from the checks is stored in summary files. The verdict is a way to inform the user about status of the individual checks. The definitions of the different verdicts are shown in Table 1.

3   Health Check Steps

3.1   AlarmsAndNotifications

This step checks if there are any unresolved alarms or notification. If a non-OK verdict is given, an AlarmsAndNotifications directory is packed into the result package, where log files, containing the details of unresolved alarms and notifications, can be found for manual examination.

Verdict

3.2   AllMtasPortStatus

This step verifies if the MTAS ports are open.

Verdict

3.3   BackupList

This step checks if there is an active backup available to restore.

Verdict

3.4   CoreMWStatus

This step verifies if there are any AMF entities with questionable health.

Verdict

3.5   CpuLoadOnPLs

This step verifies if the CPU load on each PL is below the overload protection limit (85%).

Verdict

3.6   CpuLoadOnSCs

This step verifies if the CPU load on each SC is below the overload protection limit (85%).

Verdict

3.7   DiameterPortsStatus

This step verifies Diameter ports status. Data about diameter port configuration is gathered from COM management objects.

Verdict

3.8   DiskUsageOnSCs

This step verifies the level of available space on SCs disks.

Verdict

3.9   DrbdStatus

This step verifies whether the shared block device of the cluster is functioning correctly. Connection state, disk state, and out-of-sync blocks are verified.

Verdict

3.10   eVIP

This step verifies status of eVIP.

Verdict

3.11   MemoryUsageOnPLs

This step checks whether memory usage is below the overload protection limit (95%) on each payload node.

Verdict

3.12   MemoryUsageOnSCs

This step checks whether memory use is below the overload protection limit (95%) on each system controller node.

Verdict

3.13   Mmas

This step verifies whether MMAS traffic instances are operational on every payload node.

Verdict

3.14   NETCONFConnection

This step verifies if NETCONF is configured on only one controller.

Verdict

3.15   NetworkConnectivity

This step verifies the connectivity between each SC/PL node.

Verdict

3.16   NodeOutage

This step verifies SCs/PLs state and checks for recovery events in the last 24 hours of ISP logs.

Verdict

3.17   OperationalState

This step checks MTAS operational state using COM interfaces.

Verdict

3.18   SIPPortsStatus

This step verifies if SIP ports are open.

Verdict

3.19   SS7Connections

This step verifies SS7 stack status.

Verdict

3.20   Sla

This step verifies the status of Service Level Agreement (SLA) and records the Key Performance Indicator (KPI) for the Virtual Machine (VM), Core and Network Interface under the Sla Directory for the last hour.

Verdict

For more details on the SLA Results, refer to Section 2.4 SLA Results.

For information on how to troubleshoot SLA, refer to Section 4.6 in MTAS Troubleshooting Guideline.

3.21   SecurityStatus

This step verifies if Core MW security package is installed on the SC/PL nodes.

Verdict

3.22   SoftwareInventory

This step collects a list of RPM/SDP files available on the SC/PL nodes.

Verdict

3.23   SoftwareVersionsInstalled

This step collects a list of software installed and running on the node.

Verdict

3.24   SoftwareVersionsRunning

This step collects a list of software installed and running on the node.

Verdict

3.25   SystemEnvironmentVariables

This step checks whether the vDicos environment variables are set correspondingly to the reference values.

Verdict

3.26   SystemStatus

This step verifies the system services status. Data is gathered by cmw-status.

Verdict

3.27   UpgradeList

This step based on Core MW repository list verifies if all campaigns are updated.

Verdict

3.28   VirtualDicosProcessOutage

This step checks status of vDicos Virtual Machines.

Verdict

3.29   VmLogs

This step inspects vDicos Virtual Machine Logs if severe error messages logged in the last 24 hours.

Verdict

3.30   XdmsCaiLicence

This step checks whether the XDMS server certificate is valid.

Verdict

3.31   XdmsInstance

This step verifies if the XDMS instance exists in MMAS and if its status is OK.

Verdict

3.32   XdmsRpm

This step verifies if every necessary XDMS-related package is installed on the system.

Verdict

3.33   XdmsTrafficApps

This step verifies traffic instance logs from the MMAS server.

Verdict

4   Health Check Profiles

This section describes health check profiles content. All health checks are grouped according to importance: Mandatory, Medium, and Low priority.

4.1   HcMtasMandatory

This profile contains checkers only for the most crucial parts of the system. Health Check using this profile is performed in every hour automatically.

HcMtasMandatory profile includes the following steps:

• AlarmsAndNotifications
• CoreMWStatus
• DrbdStatus
• eVIP
• Mmas
• NETCONFConnection
• NetworkConnectivity
• NodeOutage
• OperationalState
• SS7Connections
• Sla
• SystemStatus
• VirtualDicosProcessOutage
• XdmsInstance

4.2   HcMtasMediumPriority

This profile provides a broader insight to the health of the system, by checking memory and CPU use, and some of the external interfaces.

Medium priority profile contains all steps from mandatory profile and the following steps:

• AllMtasPortsStatus
• CpuLoadOnPLs
• CpuLoadOnSCs
• DiameterPortsStatus
• DiskUsageOnSCs
• MemoryUsageOnPLs
• MemoryUsageOnSCs
• SIPPortsStatus
• XdmsCaiLicence
• XdmsRpm
• XdmsTrafficApps

4.3   HcMtasLowPriority

This profile contains every checker available. By using this profile, a comprehensive set of information is constructed about system health.

Low priority profile contains all steps included in the medium profile and the following steps:

• BackupList
• SecurityStatus
• SoftwareInventory
• SoftwareVersionsInstalled
• SoftwareVersionsRunning
• SystemEnvironmentVariables
• UpgradeList
• VmLogs

5   Problem Reporting

For any abnormal situation, refer to MTAS Troubleshooting Guideline.

If the problem still exists, the user can report it to the next level of support.

It is also important to collect the related data. For more information, refer to Data Collection Guideline for MTAS.