Contents

1   Preventive Maintenance Introduction

This document describes maintenance procedures that must be performed regularly (daily, weekly, monthly, and so on) to check the correct behavior of the SAPC. The maintenance procedures allow system administrators to detect incidental problems.

1.1   Preventive Maintenance Tools

1.1.1   COM CLI

COM CLI (also known as ECLI) is a terminal-based command line interface which allows the operator to monitor and manage (for example check active alarms, manage the SAPC configuration data) the Managed Element (ME). It enables interaction with the Management Information Base (MIB) through common, generic-purpose commands.

To access the COM CLI for administration node operations, the system administrator must use:

ssh sapcadmin@<OAM VIP> -p <COM_port> -t -s cli

where <OAM VIP> is the SAPC VIP OAM and <COM_port> is the COM port, normally 830.

ssh sapcadmin@10.42.118.235 -p 830 -t -s cli

As an example of use of this tool is:

>show ManagedElement=1
ManagedElement=1
SystemFunctions=1
Transport=1
JavaCaf=1
PolicyControlFunction=1

For further information about this interface, refer to Ericsson Command-Line Interface.

1.1.2   SSH

SSH access to the system must be used for operation and maintenance purposes specified along this document.

To access the system, execute the following command:

ssh sapcadmin@<OAM VIP>

2   Maintenance Intervals

Ericsson recommends performing the following regular maintenance procedures:

Table 1    Regular Maintenance Procedures

Task	Daily	Weekly	Monthly
Check active alarms	X
Check the SAPC status	X
Check PMF Measures	X
Check Core Dumps existence	X
Check CPU and Memory Load	X(1)
Check IMM Persistent Back End		X
Check Disk Space		X
Make a System Data Backup when significant changes are done and the SAPC is working properly according to such configuration.			X
Make a User Data Backup when significant changes in subscriber provisioning are done and the SAPC is working according to such provisioning.	X

(1)  Check these values during the busy hour.

3   Maintenance Procedures

This section describes the maintenance procedures recommended to be executed in the SAPC.

3.1   Check Active Alarms

Perform a daily verification of the active alarms and notifications. There are two ways to do so:

• Through any external system configured to collect SNMP traps.
• Through the COM CLI tool provided inside the SAPC.

To verify the alarms and notifications through the COM CLI tool, follow these steps:

1. Access the COM CLI according to Section 1.1.1.
2. Execute the following command (more information in Ericsson Command-Line Interface):

   show-table ManagedElement=1,SystemFunctions=1,Fm=1 -m FmAlarm -p fmAlarmId, specificProblem

   ==================================================== 
   | fmAlarmId | specificProblem                      |
   ====================================================
   | 139       | Policy Control, Number of Gx Session | 
   |           | Rejections Reached                   |
   ====================================================

For information about procedures related to alarms and notifications, see the Fault Management folder in the library.

3.2   Check the SAPC Status

To verify the SAPC status, execute the following:

1. Access the SAPC according to the procedure described in Section 1.1.2.
2. Check the SAPC status:

   sapcadmin@SC-X> sudo sapcApplication -a status

   Status OK

   If the output is different to this one, contact Ericsson personnel.

3.3   Check High Availability State of a System Controller

System Controllers are configured as Active or Standby. To know the SAPC state, follow the next steps:

1. Access the SAPC according to the procedure described in Section 1.1.2.
2. Execute the following command:

   sapcadmin@SC-x>drbdadm status

3. The current state of System Controller is obtained from the attribute drbd0 role in the command output.

   Active state: drbd0 role: Primary

   Standby state: drbd0 role: Secondary

   If the output is different to this one, contact Ericsson personnel.

The following example shows the command output in order to get the state of SC-1:

sapcadmin@SC-1>drbdadm status

drbd0 role:Secondary
disk:UpToDate
SC-2 role:Primary
peer-disk:UpToDate

In that case the current System Controller (SC-1) is Secondary and SC-2 is Primary.

3.4   Check PMF Measures

Check the SAPC performance measurements according to List Performance Management Groups and Measurement Types and List Performance Management Jobs.

3.5   Check Core Files

Check the Core Dump existence by following these steps:

1. Access the SAPC according to the procedure described in Section 1.1.2.
2. Run sapcadmin@SC-X>ls -la /cluster/dumps

   If there are many core files, <filename>.core, contact Ericsson personnel.

3.6   Check CPU Load and Memory Usage

The SAPC is based on a Linux System. To check both the CPU Load and the Memory Usage real-time values, the recommendation is to use the top command.

To perform this task, do the following:

1. Access the SAPC according to the procedure described in Section 1.1.2.
2. Verify in the System Controller processors.

   sapcadmin@SC-X> top

   sapcadmin@SC-X> ssh sapcadmin@SC-Y

   sapcadmin@SC-Y> top

   sapcadmin@SC-Y> exit

3. Verify in the Pay Load processors.

   sapcadmin@SC-X> ssh sapcadmin@PL-3

   sapcadmin@PL-3> top

   sapcadmin@PL-3> exit

   Repeat it for all PLs

   sapcadmin@SC-X> ssh sapcadmin@PL-n

   sapcadmin@PL-n> top

   sapcadmin@PL-n> exit

4. To verify that the SAPC is correct and prevent any unexpected situation, check that in the result of the top command:
   • CPU idle, %id, is not less than 30%.
   • Used memory RAM does not exceed 80% of the total.
   • Used swap does not exceed 60% of the total.

5. Access the SAPC according to the procedure described in Section 1.1.2.
6. Run sapcadmin@SC-X>ls -la /storage/no-backup/com-apr9010443/PerformanceManagementReportFiles

   A list of log files are listed. The filename format is Ayyyymmdd.hhmm±hhmm-hhmm±hhmm_1.xml

7. To verify CPU Load and Memory Usage values for a specific time frame, search for the desired time range using the filenames.
8. To read the file information, run the following command: less /path/to/file

The log file format looks like:

<measInfo measInfoId="OSProcessingUnit">
<job jobId="ResourcesCountersJob"/>
<granPeriod duration="PT300S" endTime="yyyy-mm-ddThh:mm:ss±hhmm"/>
<repPeriod duration="PT300S"/>
<measType p="1">CPULoad.TotalmeasType p="1">CPULoad.Total</measType>
<granPeriod duration="PT300S" endTime="yyyy-mm-ddThh:mm:ss±hhmm"/>
<measType p="2">Mem.PercentUsedmeasType p="2">Mem.PercentUsed</measType>
<measValue measObjLdn="OSProcessingUnit=PL-X"
<r p="1">CPULoad.Total Value</r>
<r p="2">Mem.PercentUsed value</r>
</measValue>
.....
</measInfo>

To verify that the SAPC is correct and prevent any unexpected situation, the values for the CPULoad.Total must be less than 75% and Mem.PercentUsed must be less than 80%.

If any of these indicators are not correct, contact Ericsson personnel before freeing any memory or reducing the CPU load.

3.7   Check IMM Persistent Back End

To check that IMM Persistent Back End (PBE) is activated (that means that changes done in the configuration objects or attributes persist after a cluster restart), do:

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Execute the following command:

   sapcadmin@SC-X:~> immlist safRdn=immManagement,safApp=safImmService

3. Check that saImmRepositoryInit parameter is set to value 1:

   saImmRepositoryInit SA_UINT32_T 1 (0x1)

   If the parameter has a different value, contact Ericsson personnel.

3.8   Check Disk Space

To check the used disk space, use the Unix command df —Ph.

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Run sapcadmin@SC-X>df -Ph

   Where X is any controller active at that moment, it could be 1 or 2.

3. Verify in the traffic processors.

   sapcadmin@SC-X>ssh sapcadmin@PL-3

   sapcadmin@PL-3>df -Ph

   sapcadmin@SC-X>ssh sapcadmin@PL-4

   sapcadmin@PL-4>df -Ph

The printout of the command looks like:

Filesystem Size Used Avail Use% Mounted on

rootfs 2.0G 727M 1.3G 36% /

root 2.0G 727M 1.3G 36% /

tmpfs 1.9G 704K 1.9G 1% /dev/shm

shm 1.9G 704K 1.9G 1% /dev/shm

192.168.11.100:/.cluster/ 9.7G 4.5G 4.8G 49% /cluster

The use of any disk partition in the SAPC must be less than 80%, otherwise contact Ericsson personnel before removing any file on the system.

3.9   Check SAPC Backup

To check SAPC backup, list the existing backups.

1. Access the COM CLI according to Section 1.1.1.
2. Execute the following command (more information in Ericsson Command-Line Interface):

   > show ManagedElement=1,SystemFunctions=1,BrM=1,BrmBackupManager=SYSTEM_DATA

If a recent backup is not available, one should be created. For detailed information on how to create a backup, see Backup and Restore.