1 Introduction

This document provides the description and troubleshooting steps to take for the Storage Engine, Automatic Handling of Network Isolation not Completed for PDDB alarm.

1.1 Alarm Description

This alarm is raised when the Automatic Handling of Network Isolation process has failed to repair the Processing Layer Database (PLDB) cluster inconsistency between former and current master replica servers.

The alarm is issued in the following situations:

Selective Replica Check task was not completed.
Data Repair task was not completed.
Triggering Reconciliation task was not completed.

The possible alarm causes and the corresponding fault reasons, fault locations and impacts are described in Table 1.

Table 1 Alarm Causes
Alarm Cause	Description	Fault Reason	Fault Location	Impact
Selective Replica Check task was not completed.	Automatic Handling of Network Isolation process was unsuccessful in repairing PLDB cluster inconsistency between former and current master replica servers.	Any issue preventing LDAP queries to be performed on the invoked PLDB Cluster. Any other issue preventing access to the operational logs results in a failed execution.	Slave (Selective Replica Check) replica server.	Rescuing non-replicated data from former master has failed.
Data Repair task was not completed.	Automatic Handling of Network Isolation process was unsuccessful in repairing PLDB cluster inconsistency between former and current master replica servers.	Any issue preventing LDAP queries to be performed on the invoked PLDB Cluster. Any other issue resulting in a failed execution.	Master (Data Repair) replica server.	Rescuing non-replicated data from former master has failed.
Triggering Reconciliation task was not completed.	Automatic Handling of Network Isolation process was unsuccessful in adding local DS units that were elected master for their DSG to the Reconciliation Pending Task List.	Any issue preventing an update of the Reconciliation Pending Task List.	Master replica server.	No data reconciliation process.

Note:

An alarm can appear as a result of a maintenance activity.

The following are the consequences for the node if the alarm is not solved:

Non-replicated data transactions residing on former master (due to mastership change) are lost.
Possible unresolved inconsistencies between PLDB and DSG data.

The alarm attributes are listed and explained in Table 2.

Table 2 Alarm Attributes
Attribute Name	Attribute Value
Auto Cease	`No`
Module	`STORAGE-ENGINE`
Error Code	`29`
Timestamp First	Date and time when the alarm was raised for the first time.
Repeated Counter	Number which indicates how many times the alarm was raised.
Timestamp Last	Date and time of the most recent alarm raised.
Resource ID	`.1.3.6.1.4.1.193.169.1.1.29.<Timestamp>`
Alarm Model Description	`Automatic Handling of Network Isolation not Completed, Storage Engine.`
Alarm Active Description	`Storage Engine (PLDB): Automatic Handling of Network Isolation task <add_info> was not completed <add_info2> (task <taskid>, blade <Blade>), uuid: <uuid>`
ITU Alarm Event Type	`processingErrorAlarm (4)`
ITU Alarm Probable Cause	`softwareError (163)`
ITU Alarm Perceived Severity	`(4) – Major`
Originating source IP	Node IP where the alarm was raised.
Sequence Number	Number which indicates the order in which the alarms are raised.

In Table 2, the indicated variables are as follows:

<Timestamp> is the Unix time representing the time of the incident, that is, the time stamp used to determine which events from the operational logs of the former master were analyzed.
<add_info> is a variable description: "Selective Replica Check", "Data Repair", or "trigger Reconciliation".
<add_info2> is an optional additional description field showing up when the Automatic Handling of Network Isolation process has terminated a Selective Replica Check or a Data Repair task due to too long execution duration. Its value is: "due to time limit exceeded".
<taskid> is a Selective Replica Check or a Data Repair task identifier based on the Automatic Handling of Network Isolation activity start time.
<Blade> is the host name of the blade or Virtual Machine (VM) where repair was executed and the logs are stored.
<uuid> is the universally unique identifier of the computing resource (blade or VM). It is blank if it is not possible to figure out its value.

For further information about attribute descriptions, refer to CUDB Node Fault Management Configuration Guide.

1.2 Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1 Documents

This instruction references the following documents:

1.2.2 Tools

Not applicable.

1.2.3 Conditions

Not applicable.

2 Procedure

This section describes the procedure to follow when this alarm is received.

2.1 Actions for Selective Replica Check Task Was Not Completed

Perform the following steps:

Steps

If the Storage Engine, Unable to Synchronize Cluster in PLDB, Major alarm is raised when the Self-Ordered Backup and Restore function is not enabled or fails to restore the replication automatically, follow the procedure described in Storage Engine, Unable To Synchronize Cluster In PLDB, Major.
Cease the alarm manually.

Note: The procedure is to fix data inconsistency among master and slave replicas, but it cannot guarantee the full repair of system data.

2.2 Actions for Data Repair Task Was Not Completed

Do the following:

Cease the alarm manually.

Note:

Full repair of system data cannot be guaranteed in this case.

2.3 Actions for Triggering Reconciliation Task Was Not Completed

Perform the following steps:

Steps

Run the following command to establish an admin "CUDB CLI" session towards the CUDB node where the master for PLDB is:
ssh <admin_user>@<CUDB_Node_OAM_IP_Address>

Refer to CUDB System Administrator Guide on how to list all master DSG replicas.
Run the following command to check if there is any pending or ongoing reconciliation task for the specific DSG(s) from Step 1:
cudbReconciliationMgr -l

This command returns the DSG(s) in an affirmative case. Otherwise, it returns nothing. In an affirmative case, exit this procedure. In a negative case, follow with the next step.Refer to CUDB Node Commands and Parameters for further information about this command.
Schedule reconciliation for the specific DSG(s) from Step 1:
cudbReconciliationMgr -a <dsId>

If the task for DSG identified with <dsId> is added, the command has no output. Otherwise, the output provides the error(s) fetched from the database. Refer to CUDB Data Storage Handling for further information about the reconciliation process.
If the alarm does not cease, contact the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

Copyright	© Ericsson AB 2016-2018. All rights reserved. No part of this document may be reproduced in any form without the written permission of the copyright owner.
Disclaimer	The contents of this document are subject to revision without notice due to continued progress in methodology, design and manufacturing. Ericsson shall have no liability for any error or damage of any kind resulting from the use of this document.
Trademarks	All trademarks mentioned herein are the property of their respective owners. These are shown in the document Trademark Information.