Contents

1   Introduction

This instruction concerns alarm handling for the Application Counters, Fault In Subscriber Statistic Application alarm.

1.1   Alarm Description

This alarm is raised when it is not possible to get application counters.

The alarm is issued in the following situations:

• Failing to find source of application counters information.
• Faulty or missing configuration.
• Impossible to store collected application counters values.
• Impossible to execute the stored procedure.

This alarm may also be raised in relation to internal capacity license counter groups.

The possible alarm causes and the corresponding fault reasons, fault locations and impacts are described in Table 1.

Table 1    Alarm Causes

Alarm Cause	Description	Fault Reason	Fault Location	Impact
Failing to find source of application counters information.	The system cannot determine from where to retrieve counter information or there is no source of counter information.	No information about where to extract/collect/gather counter information. [Couldn't get slave SDL list or master list.]	BC cluster	No application counters generation for affected applications counters.
		No available slave replica for this DSG. [Couldn't get slave SDL list or master list.]	DSG without slave replica
		No master replica for this DSG. [There are some nodes without master]	Masterless DSG
Faulty or missing configuration.	The configuration for application counters is missing or incorrect.	The application counters configuration file does not exist. [The application counter's configuration file does not exist.]	Application counters configuration
		The counters configuration file is present but the information there is incorrect. [The expected application counter's database/table/column is not found.]	Application counters configuration
Impossible to store collected application counters values.	Application values are calculated but can’t be stored because of communication problems or other kind of problems.	Impossible to store generated application counters in the PLDB. [Couldn't update PL counters table.]	PLDB master
		The communication with the master PLDB replica cannot be established. [Couldn't connect to PL tables.]	PLDB master or communications channel with PLDB master
Impossible to execute the stored procedure.	An error occurred in the PLDB or DSG involved in the application counters process. It is not possible to execute the stored procedure allocated in the access server.	It is impossible to execute the stored procedure in PLDB. [error application counters trap for PL.]	PLDB access server on master replica
		It is impossible to execute the stored procedure in a DSG. [error application counters trap for DS#{dsg}.]	DS access server on slave replica

Note:  

An alarm can appear as a result of maintenance activity.

The following are the consequences for the node if the alarm is not solved:
There is no application counters generation for the affected applications counters. Affected application counters are counters reported in alarm descriptions.

The alarm attributes are listed and explained in Table 2.

Table 2    Alarm Attributes

Attribute Name	Attribute Value
Auto Cease	Yes
Module	APPLICATION-COUNTERS
Error Code	1
Timestamp First	Date and time when the alarm was raised for the first time.
Repeated Counter	Number which indicates how many times the alarm was raised.
Timestamp Last	Date and time of the most recent alarm raise.
Resource ID	.1.3.6.1.4.1.193.169.8.1.<DG>.<CGL>.<CG>
Alarm Model Description	Fault retrieving subscriber statistics, Application Counters.
Alarm Active Description	Application Counters: fault retrieving subscriber statistics for <DS_group> in Group counter <FileName> (<cause>).
ITU Alarm Event Type	processingErrorAlarm (4)
ITU Alarm Probable Cause	applicationSubsystemFailure (158)
ITU Alarm Perceived Severity	(4) - Major
Originating source IP	Node IP where the alarm was raised.
Sequence Number	Number which indicates the order in which the alarms are raised.

In Table 2, the indicated variables are as follows:

• <DG> is the Data Store Unit Group (DSG) or the Processing Layer Database (PLDB) whose application counters could not be retrieved, starting from 0 (PLDB) to the maximum number of DSGs in the system.
• <CGL> is the number of characters in the counter group name.
• <CG> is the name of the counter group to process. The name is listed as dot separated letters in ASCII code.

  For example:

  71.82.80.95.65.76.76.66.83.71 stands for GRP_ALLBSG.

• <DS_group> is :
  • The number of the affected DSG indicated as DS-group # <DG>. The value of <DG> is between 1 and the maximum number of DSGs in the system.
  • PLDB
• <FileName> is the name of the configuration file that contains the counter group to process.
• <cause> is an optional additional description field and its value is one of the following:
  • Couldn't get slave SDL list or master list.
  • The application counter's configuration file does not exist.
  • The expected application counter's database/table/column is not found.
  • Couldn't update PL counters table.
  • Couldn't connect to PL tables.
  • There are some nodes without master.
  • error application counters trap for PL.
  • error application counters trap for DS#{dsg}.
• {dsg} is the number of the affected DSG.

Note:  

Even when the cause of the alarm is not located in the PLDB, nor in the DSG (such as "Failing to find source of application counters information" or "Faulty or missing configuration"), the Resource ID field contains <DG>=0, while the value of <DS_group> in the Alarm Active Description field is PLDB.

For further information about attribute descriptions, refer to CUDB Node Fault Management Configuration Guide, Reference [1].

1.2   Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1   Documents

Before starting this procedure, ensure that you have read the following documents, depending on the type of application counters installed:

• Home Location Register Front End (HLR-FE) 16A Installation Instruction ("Installation Instruction for HLR Application in UDC"), regarding Installation script and application counters package (the Container File package is sometimes called "Software Records") download and instructions on how to install application counters.
• Home Subscriber Server FE (HSS-FE) Software Installation Instruction, regarding Installation script and application counters package download and instructions on how to install and check that the application counters are correctly installed.
• Service-Aware Policy Controller (SAPC) Installation Instruction, section "Configuring SAPC Application Counters in External Database (CUDB)" regarding implementation of the application counters and checking that the application counters are correctly installed.

And the following CUDB documents:

• CUDB Node Fault Management Configuration Guide, Reference [1], regarding alarm configuration.
• CUDB Application Counters, Reference [2].
• System Safety Information, Reference [10].
• Personal Health and Safety Information, Reference [11].

1.2.2   Tools

Not applicable.

1.2.3   Conditions

Not applicable.

2   Procedure

This section describes the procedure to follow when this alarm is received.

Note:  

Publishing interval is usually 15 minutes (default value), except for internal capacity license counter groups that are checked every 24 hours.

2.1   Actions for "Couldn't Get Slave SDL List or Master List"

Do the following:

1. Check if the Control, Blackboard Coordination Cluster Down alarm is raised. In case it is raised, refer to Control, Blackboard Coordination Cluster Down, Reference [3] on how to proceed.

   After the above alarm is cleared, wait for another publishing interval. Then go to Step 3.

2. Check if the Storage Engine, DS Cluster Node Down alarm is raised. In case it is raised, refer to Storage Engine, DS Cluster Node Down on how to proceed.

   After the above alarm is cleared, wait for another publishing interval. Then go to Step 3.

3. If the alarm does not cease, consult the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

2.2   Actions for "Faulty or Missing Configuration"

For the following causes:

• The application counter's configuration file does not exist.
• The expected application counter's database/table/column is not found.

Do the following:

1. Perform re-installation steps as described in Section 3.1.

   Wait for another publishing interval.

   If the alarm does not cease, consult the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

2.3   Actions for "Impossible to Store Collected Application Counters Values"

For the following causes:

• Couldn't connect to PL tables.
• Couldn't update PL counters table.

Do the following:

1. Consult the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

2.4   Actions for "There Are Some Nodes without Master"

Do the following:

1. Check if the Storage Engine, No Available Master Replica for DS alarm or the Storage Engine, No Available Master Replica for PLDB alarm is raised. In case it is raised, refer to Storage Engine, No Available Master Replica for DS, Reference [4] or Storage Engine, No Available Master Replica for PLDB, Reference [5] on how to proceed for each alarm.

   After the above alarm is cleared, wait for another publishing interval.

   If the alarm does not cease, consult the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

2.5   Actions for "Error Application Counters Trap for PL"

Do the following:

1. Check if the Storage Engine, PLDB Cluster Node Down alarm is raised. In case it is raised, refer to Storage Engine, PLDB Cluster Node Down, Reference [6] on how to proceed.

   After the above alarm is cleared, wait for another publishing interval. Then go to Step 4.

2. Perform restoration steps as described in Section 3.2.

   Note:  

   It is possible that the cluster has been restored as part of a system data restore procedure or following a flawed unit or group data restore procedure at an earlier point in time.

   
   

   After stored procedures restoration, wait for another publishing interval. If the alarm is cleared, no further steps are needed.

3. Perform re-installation steps as described in Section 3.1.

   Wait for another publishing interval.

4. If the alarm does not cease, consult the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

2.6   Actions for "Error Application Counters Trap for DSG"

Do the following:

1. Check if the Storage Engine, DS Cluster Node Down alarm is raised. In case it is raised, refer to Storage Engine, DS Cluster Node Down, Reference [7] on how to proceed.

   After the above alarm is cleared, wait for another publishing interval. Then go to Step 4.

2. Perform restoration steps as described in Section 3.2.

   Note:  

   It is possible that the cluster has been restored as part of a system data restore procedure or following a flawed unit or group data restore procedure at an earlier point in time.

   
   

   After stored procedures restoration, wait for another publishing interval. If the alarm is cleared, no further steps are needed.

3. Perform re-installation steps as described in Section 3.1.

   Wait for another publishing interval.

4. If the alarm does not cease, consult the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

3   Generic Recovery Steps

3.1   Application Counters Re-Installation Steps

3.1.1   Applications Counters Installed Using Application FE Installation Scripts

Make sure the correct revisions of the installation script and the application counters package (the Container File package is sometimes called "Software Records") is available for re-installation.

Note:  

For some application FE and schema revision should be checked.

For correct revisions, refer to the "Product Revision Information" document in the CPI of the corresponding application FE.

Uninstall the current script and package, and use the correct revisions of the installer script and application counters package to install them again.

Note:  

• Application Counters to be re-installed are listed in alarm descriptions of received alarms
• Check application FE installation instructions for procedures on how to check if application counters are installed correctly. Refer to the CPI documents of the applicable application FE.

3.1.2   Manually Installed Application Counters

If the application counters are to be re-installed manually, consult the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

3.2   Stored Procedures Restoration Steps

After restoration (Data Restore), it is necessary to recreate stored procedures. If --restore-stored-procedures | -s options were not used during cudbManageStore --order restore, it is necessary to restore them now.

Note:  

The Self-Ordered Backup and Restore procedure automatically recreates stored procedures.

To restore the stored procedures and application counter procedures on the local PLDB, enter the following command:

# cudbManageStore -p -o restorestoredprocedures

To restore the stored procedures and application counter procedures on a DS, enter the following command:

# cudbManageStore -d <dsId> -o restorestoredprocedures

Note:  

To determine the <dsId>, check the value of the cudbLocalDsId configuration attribute for the CudbLocalDs object for which the dsGroupId attribute corresponds to the affected DSG. For more information about these configuration settings, refer to the "Class CudbLocalDs" section of CUDB Node Configuration Data Model Description, Reference [8]. For more information on how to connect to the CUDB configuration CLI and check configuration settings, refer to the "Configuration Modification Procedure" section of CUDB Node Configuration Data Model Description, Reference [8].

Glossary

For the terms, definitions, acronyms and abbreviations used in this document, refer to CUDB Glossary of Terms and Acronyms, Reference [9].

Reference List

CUDB Documents
[1] CUDB Node Fault Management Configuration Guide.
[2] CUDB Application Counters.
[3] Control, Blackboard Coordination Cluster Down.
[4] Storage Engine, No Available Master Replica for DS.
[5] Storage Engine, No Available Master Replica for PLDB.
[6] Storage Engine, PLDB Cluster Node Down.
[7] Storage Engine, DS Cluster Node Down.
[8] CUDB Node Configuration Data Model Description.
[9] CUDB Glossary of Terms and Acronyms.

Other Ericsson Documents
[10] System Safety Information.
[11] Personal Health and Safety Information.