Operating Instructions 2/1543-CNH 160 6135/10 Uen B

Control, Potential Split Brain Detected
Ericsson Centralized User Database

Contents


1 Introduction

This instruction concerns alarm handling for the Control, Potential Split Brain Detected alarm.

1.1 Alarm Description

The alarm is issued when a potential symmetrical split situation is detected in the Ericsson Centralized User Database (CUDB) system.

The possible alarm causes and the corresponding fault reasons, fault locations, and impacts are described in Table 1.

Table 1   Alarm Causes

Alarm Cause

Description

Fault Reason

Fault Location

Impact

Potential symmetrical split situation detected.

Half of the sites of the CUDB system have failed or are unreachable. In deployments with more than two sites, some sites could be considered auto-removed and are not taken into account when determining the number of sites in the system. For more information, refer to the Split Situations section of CUDB High Availability.

  • Network or site failure.

  • A CUDB node has been shut down (in case of a two-node, two-site deployment).

Affected site(s) or node(s).

The CUDB system is split in two equal halves.

The alarm attributes are listed and explained in Table 2.

Table 2   Alarm Attributes

Attribute Name

Attribute Value

Auto Cease

Yes

Module

CONTROL

Error Code

2

Timestamp First

Date and time when the alarm was raised for the first time.

Repeated Counter

Number which indicates how many times the alarm was raised.

Timestamp Last

Date and time of the most recent alarm raise.

Resource ID

.1.3.6.1.4.1.193.169.7.2

Alarm Model Description

CUDB system in potential split brain, Control.

Alarm Active Description

Control: Potential split brain detected.

ITU Alarm Event Type

communicationsAlarm (2)

ITU Alarm Probable Cause

communicationsSubsystemFailure (505)

ITU Alarm Perceived Severity

(3) – Critical

Originating source IP

Node IP where the alarm was raised.

Sequence Number

Number which indicates the order in which the alarms are raised.

For further information about attribute descriptions, refer to CUDB Node Fault Management Configuration Guide.

2 Procedure

In most cases, the CUDB system is able to recover from the symmetrical split situation automatically when the unreachable sites are available again. Then, the alarm is automatically cleared.

Refer to CUDB High Availability and CUDB System Split Partial Recovery Procedure for more information on the behavior of the CUDB system in different split situations and recovery paths.

If the alarm does not cease, do the following:

Steps

  1. Manually verify the status of the system in both partitions, master and slave assignments.
  2. Verify the cause of the split situation: network, node, or site failure, or a combination of these. No specific actions can be taken until the nature of the failure is determined.
  3. Take the specific repairing actions.
  4. In case of node/site failure, continue with Step 5. In other cases, continue with Step 6.
  5. Verify the following:

    • The failing elements in the system are fixed, that is, nodes in the formerly isolated site are operational again.

    • Site infrastructure is fully operational, and the site is connected to the rest of the sites in the system.

    • The failing nodes have not been restarted since they went down. The symmetrical split situation is still present.

    Then the affected nodes must be reintroduced in the system. Perform the procedure described in Failing Node Introduction.
  6. The failing nodes have been started up after failure or the network failure has been repaired. The symmetrical split situation has ended.
    Verify if one or more instances of the Storage Engine, Unable to Synchronize Cluster in DS, Major and Storage Engine, Unable to Synchronize Cluster in PLDB, Major alarms are raised either in the surviving sites or in the recovering sites. If any of these alarms are raised, then refer to their specific Operating Instruction (OPI).

2.1 Failing Node Introduction

The following procedure operates on a per recovered node basis. It is assumed that the symmetrical split situation is still present at this point. To reintroduce a failing node, do the following:

Steps

  1. Check whether the system configuration has changed or not. If system configuration has changed, the procedure cannot continue.
  2. Prevent spontaneous reconnection of the recovered node by disabling the external interfaces: vlan CUDB_SITE, CUDB_FE and PROVISIONING.
  3. Put the PLDB cluster in maintenance mode for the recovered node.
  4. Obtain the list of DSGs that are to be resynchronized manually. For each DSG in this list, perform manual replica synchronization (see Manual Replica Synchronization Procedure).
    Note: Step 4 is optional. However, if the need for the manual resynchronization is expected, then it is recommended to perform it, as it can avoid the simultaneous raising of multiple alarms due to replica synchronization failures.
  5. Activate the external interfaces for Inter-CUDB communication in the recovered node (enable external interfaces for vlan CUDB_SITE).
  6. Activate ready mode for the PLDB cluster in the recovered node.

The system abandons the symmetrical split situation at this point.

  1. Verify whether the recovered node is hosting slave PLDB and DSG replicas. If it does not, then the procedure cannot continue.
  2. If there were alarms related to master replica synchronization, perform a manual replica synchronization (see Manual Replica Synchronization Procedure for the exact steps).
  3. Verify whether the replication is fully operational. This is performed by entering the command cudbCheckReplication. For further information on the command, refer to CUDB Node Commands and Parameters. If replication is not fully operational, the process is unsuccessful.
  4. Activate the rest of the external interfaces in the recovered node (enable vlans: CUDB_FE and PROVISIONING).
  5. It is recommended to perform a software and data backup at this point. Refer to CUDB Backup and Restore Procedures, for information on backup and restore procedures.

2.2 Manual Replica Synchronization Procedure

In case manual replica synchronization is needed, do the following:

Steps

  1. Create a backup of the master DS Unit of the affected DSG that is already running. Refer to the "Unit Data Backup" section of CUDB Backup and Restore Procedures for more information.
  2. Restore the backup of the master DS Unit in the local DS Unit of the affected CUDB node. Refer to the "Unit Data Restore" section of CUDB Backup and Restore Procedures for more information.

    The command cudbUnitDataBackupAndRestore makes the above process automatic. Refer to CUDB Node Commands and Parameters for more information on the command.

Legal |© Ericsson AB 2017-2018