Contents

1   Introduction

This document describes maintenance procedures that must be performed regularly (daily, weekly, monthly, and so on) to check the correct behavior of the SAPC. The maintenance procedures allow system administrators to detect incidental problems.

1.1   Tools

1.1.1   COM CLI

COM CLI (also known as ECLI) is a terminal-based command line interface which allows the operator to monitor and manage (for example check active alarms, manage the SAPC configuration data) the Managed Element (ME). It enables interaction with the Management Information Base (MIB) through common, generic-purpose commands.

To access the COM CLI for administration node operations, the system administrator must use:

ssh sapcadmin@<OAM VIP> -p <COM_port> -t -s cli

where <OAM VIP> is the SAPC VIP OAM and <COM_port> is the COM port, normally 830.

ssh sapcadmin@10.42.118.235 -p 830 -t -s cli

As an example of use of this tool is:

>show ManagedElement=1
ManagedElement=1
SystemFunctions=1
Transport=1
JavaCaf=1
PolicyControlFunction=1

For further information about this interface, refer to Ericsson Command-Line Interface.

1.1.2   SSH

SSH access to the system must be used for operation and maintenance purposes specified along this document.

To access the system, execute the following command:

ssh sapcadmin@<OAM VIP>

2   Maintenance Intervals

Ericsson recommends performing the following regular maintenance procedures:

Table 1    Regular Maintenance Procedures

Task	Daily	Weekly	Monthly
Check active alarms	X
Check the SAPC status	X
Check PMF Measures	X
Check Core Dumps existence	X
Check CPU and Memory Load	X(1)
Check IMM Persistent Back End		X
Check Disk Space		X
Make a System Data Backup when significant changes are done and the SAPC is working properly according to such configuration.			X
Make a User Data Backup when significant changes in subscriber provisioning are done and the SAPC is working according to such provisioning.	X

(1)  Check these values during the busy hour.

3   Maintenance Procedures

This section describes the maintenance procedures recommended to be executed in the SAPC.

3.1   Check Active Alarms

Perform a daily verification of the active alarms and notifications. There are two ways to do so:

• Through any external system configured to collect SNMP traps.
• Through the COM CLI tool provided inside the SAPC.

To verify the alarms and notifications through the COM CLI tool, follow these steps:

1. Access to the COM CLI according to the Section 1.1.1
2. Execute the following command (more information in Ericsson Command-Line Interface):

   show-table ManagedElement=1,SystemFunctions=1,Fm=1 -m FmAlarm -p fmAlarmId, specificProblem

   ==================================================== 
   | fmAlarmId | specificProblem                      |
   ====================================================
   | 139       | Policy Control, Number of Gx Session | 
   |           | Rejections Reached                   |
   ====================================================

For information about procedures related to alarms and notifications, see the Fault Management folder in the library.

3.2   Check the SAPC Status

To verify the status of the node, execute the following:

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Check the SAPC status:

   sapcadmin@SC-X> sudo sapcApplication -a status

   Status OK

   If the output is different to this one, contact Ericsson personnel.

3.3   Check High Availability State of a System Controller

System Controllers are configured as Active or Standby. To know the state of the node, follow the next steps:

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Execute the following command:

   sapcadmin@SC-x>cat /proc/drbd

   version: 8.4.3 (api:1/proto:86-101)
   GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by phil@fat-tyre, 2013-02-05 15:35:49
   0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
   ns:5591280 nr:0 dw:5112836 dr:892706 al:491 bm:67 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

3. The current state of the node is obtained from the attribute ro.

   Active state: ro:Primary/Secondary

   Standby state: ro:Secondary/Primary

   If the output is different to this one, contact Ericsson personnel.

3.4   Check PMF Measures

Check the SAPC performance measurements according to the Measurements.

3.5   Check Core Files

Check the Core Dump existence by following these steps:

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Run sapcadmin@SC-X>ls -la /cluster/dumps

   If there are many core files, <filename>.core, contact Ericsson personnel.

3.6   Check CPU Load and Memory Usage

The SAPC is based on a Linux System. To check both the CPU Load and the Memory Usage real-time values, the recommendation is to use the top command.

To perform this task, do the following:

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Verify in the System Controller processors.

   sapcadmin@SC-X> top

   sapcadmin@SC-X> ssh sapcadmin@SC-Y

   sapcadmin@SC-Y> top

   sapcadmin@SC-Y> exit

3. Verify in the Pay Load processors.

   sapcadmin@SC-X> ssh sapcadmin@PL-3

   sapcadmin@PL-3> top

   sapcadmin@PL-3> exit

   Repeat it for all PLs

   sapcadmin@SC-X> ssh sapcadmin@PL-n

   sapcadmin@PL-n> top

   sapcadmin@PL-n> exit

The printout of the command looks like:

Cpu(s): 1.2%us, 0.3%sy, 0.0%ni, 98.2%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 24149M total, 17687M used, 6461M free, 834M buffers

Swap: 23551M total, 0M used, 23551M free, 1717M cached

To verify that the node is correct and prevent any unexpected situation, check that in the result of the top command:

1. CPU idle, %id, is not less than 30%.
2. Used memory RAM does not exceed 80% of the total.
3. Used swap does not exceed 60% of the total.

To check both the CPU Load and Memory Usage historical values, the recommendation is to check the measurement values dumped to log files.

To perform this task, do the following:

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Run sapcadmin@SC-X>ls -la /storage/no-backup/com-apr9010443/PerformanceManagementReportFiles

   A list of log files are listed. The filename format is Ayyyymmdd.hhmm±hhmm-hhmm±hhmm_1.xml

3. To verify CPU Load and Memory Usage values for a specific time frame, search for the desired time range using the filenames.
4. To read the file information, run the following command: less /path/to/file

The log file format looks like:

<measInfo measInfoId="OSProcessingUnit">
<job jobId="ResourcesCountersJob"/>
<granPeriod duration="PT300S" endTime="yyyy-mm-ddThh:mm:ss±hhmm"/>
<repPeriod duration="PT300S"/>
<measType p="1">CPULoad.TotalmeasType p="1">CPULoad.Total</measType>
<granPeriod duration="PT300S" endTime="yyyy-mm-ddThh:mm:ss±hhmm"/>
<measType p="2">Mem.PercentUsedmeasType p="2">Mem.PercentUsed</measType>
<measValue measObjLdn="OSProcessingUnit=PL-X"
<r p="1">CPULoad.Total Value</r>
<r p="2">Mem.PercentUsed value</r>
</measValue>
.....
</measInfo>

To verify that the node is correct and prevent any unexpected situation, the values for the CPULoad.Total must be less than 75% and Mem.PercentUsed must be less than 80%.

If any of these indicators are not correct, contact Ericsson personnel before freeing any memory or reducing the CPU load.

3.7   Check IMM Persistent Back End

To check that IMM Persistent Back End (PBE) is activated (that means that changes done in the configuration objects or attributes persist after a cluster restart), do:

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Execute the following command:

   sapcadmin@SC-X:~> immlist safRdn=immManagement,safApp=safImmService

3. Check that saImmRepositoryInit parameter is set to value 1:

   saImmRepositoryInit SA_UINT32_T 1 (0x1)

   If the parameter has a different value, contact Ericsson personnel.

3.8   Check Disk Space

To check the used disk space, use the Unix command df —Ph.

1. Access to the SAPC according to the procedure described in Section 1.1.2.
2. Run sapcadmin@SC-X>df -Ph

   Where X is any controller active at that moment, it could be 1 or 2.

3. Verify in the traffic processors.

   sapcadmin@SC-X>ssh sapcadmin@PL-3

   sapcadmin@PL-3>df -Ph

   sapcadmin@SC-X>ssh sapcadmin@PL-4

   sapcadmin@PL-4>df -Ph

The printout of the command looks like:

Filesystem Size Used Avail Use% Mounted on

rootfs 2.0G 727M 1.3G 36% /

root 2.0G 727M 1.3G 36% /

tmpfs 1.9G 704K 1.9G 1% /dev/shm

shm 1.9G 704K 1.9G 1% /dev/shm

192.168.11.100:/.cluster/ 9.7G 4.5G 4.8G 49% /cluster

The use of any disk partition in the SAPC must be less than 80%, otherwise contact Ericsson personnel before removing any file on the system.

3.9   Make System Data Backup

To create a system data backup, follow the steps described in Backup and Restore.