Operating Instructions 32/1543-CNH 160 6539/10 Uen D

Storage Engine, Data Inconsistency between Replicas Found in DS, Major
Ericsson Centralized User Database

Contents


1 Introduction

This instruction concerns alarm handling for the Storage Engine, Data Inconsistency between Replicas Found in DS, Major alarm.

1.1 Alarm Description

This alarm is raised as a result of issues found when running a Consistency Check on a Data Store Unit Group (DSG) slave replica by means of the cudbConsistencyMgr command. For further information about the command, refer to CUDB Node Commands and Parameters. For more information about Consistency Check, refer to CUDB Consistency Check.

The alarm is issued in the following situations:

  • The number of divergences between the slave replica and its master replica is higher than the threshold (default or selected value).

  • A schema mismatch between the slave replica and its master replica is found.

The possible alarm causes and the corresponding fault reasons, fault locations, and impacts are described in Table 1.

Table 1   Alarm Causes

Alarm Cause

Description

Fault Reason

Fault Location

Impact

High number of divergences between the slave replica and its master replica.

The number of divergences between the slave replica and its master replica found when running a consistency check is higher than the threshold.

Wrong backup restored on the slave replica.

Affected DS cluster.

If this slave replica becomes the master replica, there might be a service impact for the subscribers affected by the data inconsistency.

Slave replica addressed by faulty maintenance operation.

Schema mismatch between the slave replica and its master replica.

A schema mismatch between the slave replica and its master replica is found when running a consistency check.

Wrong backup restored on the slave replica.

Affected DS cluster.

If this slave replica becomes the master replica, there might be a service impact for the subscribers affected by the data inconsistency.

Slave replica addressed by faulty maintenance operation.

Incomplete of faulty schema update procedure.

The alarm attributes are listed and explained in Table 2.

Table 2   Alarm Attributes

Attribute Name

Attribute Value

Auto Cease

No

Module

STORAGE-ENGINE

Error Code

22

Timestamp First

Date and time when the alarm was raised for the first time.

Repeated Counter

Number which indicates how many times the alarm was raised.

Timestamp Last

Date and time of the most recent alarm raised.

Resource ID

.1.3.6.1.4.1.193.169.1.2.22.<DG>

Alarm Model Description

Data inconsistency between replicas found, Storage Engine.

Alarm Active Description

Storage Engine (DS-group #<DG>): Data inconsistency between replicas found, major (task <TASKID>) .

ITU Alarm Event Type

processingErrorAlarm (4)

ITU Alarm Probable Cause

databaseInconsistency (160)

ITU Alarm Perceived Severity

(4) - Major

Originating Source IP

Node IP where the alarm was raised.

Sequence Number

Number which indicates the order in which alarms were raised.

In Table 2, the indicated variables are as follows:

  • <DG> is the DSG the DS cluster belongs to.

  • <TASKID> is the identifier of the check task.

For further information about attribute descriptions, refer to CUDB Node Fault Management Configuration Guide.

1.2 Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.2 Tools

Not applicable.

1.2.3 Conditions

Not applicable.

2 Procedure

When this alarm is raised, perform the following steps:

Steps

  1. Locate and identify the Lightweight Directory Access Protocol (LDAP) tree log based on the <DG> and <TASKID> parameters in the alarm as follows:
    Search for the /local/cudb_ddci/replica_check/cudbDsuDiff_tree_TASKID*.xml file on both System Controllers (SCs) of:

    • Either the CUDB node that contained the master replica of the DSG with ID <DG> at the time the check was executed.

    • Or, if the previous information is not known, all the CUDB nodes that hold a replica of the DSG with ID <DG> , except the current node.

      Note: In case of 1+1+1 redundancy, this may mean two CUDB nodes to check.
  2. Align the slave replica to its master in DSG with ID <DG> as follows:

    • If this node is still a slave node of the DSG with ID <DG> , and the node where the LDAP tree log was found is still the master node of the DSG with ID <DG> , perform a combined unit data backup and restore on the DSG against this CUDB node.

      Note: If the DSG has more than one slave, then consider ordering consistency check for the rest of slaves after restoring the backup.

    • If this node (which was the checked slave at the time of the check) has become a DSG master node since raising the alarm, perform a combined unit data backup and restore on the DSG against the CUDB node where the LDAP tree log was found.

      Note: Do not perform combined backup and restore if that has been done since the mastership change of the DSG.

      Also, if the DSG has more than one slave, then consider ordering consistency check for the rest of slaves after restoring the backup.

    • In any other case ignore the results. Consider that the DSG may not be consistent and further checks—involving the new master—are needed to verify it.

  3. Perform the following steps to find subscribers impacted by data inconsistency:

    • Analyze the LDAP tree log file to find the subscribers impacted by the data inconsistency.

      For User Data Consolidation (UDC) application data (such as HLR or HSS), build a list of the impacted Distribution Entries (DEs, mscId and assocId; refer to CUDB LDAP Interwork Description for further details). A DE is impacted if any of its child entries are included in the LDAP tree log file.

      For example, if an entry with the ImsShDynInfId=ImsShDynInf,IMPU=sip:262280000194171@ims.mnc280.mcc262.3gppnetwork.org,serv=IMS,assocId=194171,ou=associations,<rootDn> DN shows up in the LDAP tree log file, then the assocId=194171,ou=associations,<rootDn> DE is impacted.

    • Once the list of impacted DEs is ready, identify the related public identities for each impacted DE by running the following filtered LDAP query:

      ldapsearch -Y CUDB-CRYPTO -h <cudbNodeVIP> -p 389 -U <ldapRootUser> -W -b <impactedDistributionEntryDn> -s subtree -a always | grep -E "^MSISDN|^IMSI|^IMPU|^IMPI"

      Note: A prompt to enter LDAP root user password will be shown.

    • Validate the subscribers with those public identities, or consider reprovisioning them. Refer to the application Front End (FE) documentation for the procedure to validate or reprovision subscribers.

  4. If the LDAP tree log contains internal error, notify the next level of support. To interpret the contents of the LDAP tree log, refer to CUDB Consistency Check.
  5. Clear the alarm manually as described in CUDB Node Fault Management Configuration Guide.
  6. If the alarm does not cease, contact the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

After This Task

To find out where the master replicas are, refer to CUDB System Administrator Guide. For further information about the combined unit data backup and restore procedure, refer to CUDB Backup and Restore Procedures.

Legal |© Ericsson AB 2016-2018