Operating Instructions 10/1543-CNH 160 6539/10 Uen B

Storage Engine, DS Cluster Node Down
Ericsson Centralized User Database

Contents


1 Introduction

This document provides an overview for the Storage Engine, DS Cluster Node Down alarm.

1.1 Alarm Description

This alarm is raised when one of the nodes of the cluster database is down or unreachable.

The alarm is issued in the following situations:

  • The data node (NDB) of the cluster database is down or unreachable.

  • The management node (MGM) of the cluster database is down or unreachable.

  • The replication node (SQL) of the cluster database is down or unreachable.

  • The access node (SQL) of the cluster database is down or unreachable.

The possible alarm causes and the corresponding fault reasons, fault locations and impacts are described in Table 1.

Table 1   Alarm Causes

Alarm Cause

Description

Fault Reason

Fault Location

Impact

A management node of the database is down or unreachable.

One of the two management node processes cannot start up or is unreachable.

  • Network connection error.

  • Hardware error.

  • Disk is almost full.

Blade or Virtual Machine (VM).

No impact, as each cluster database has two management nodes.

A data node of the database cluster is down or unreachable.

The data node process cannot start up due to file system consistency errors, or is unreachable.

  • Non-graceful shutdown.

  • Uncontrolled crash.

  • Hardware error.

Blade or VM.

Lower database cluster performance while the data node is down.

A replication node of the database cluster is down or unreachable.

One of the replication node processes cannot start up or is unreachable.

  • Network connection error.

  • Corrupted binlog or relay log files.

  • Hardware error.

Blade or VM.

No impact, as each cluster database has two replication servers per replication type (master and slave).

An access node of the database cluster is down or unreachable.

One of the access node processes cannot start up or is unreachable.

  • Network connection error.

  • Hardware error.

Blade or VM.

No impact, as each cluster database has two access servers.
Note: An alarm can appear as a result of maintenance activity.

The following are the consequences for the node if the alarm is not solved:

  • The DS (Data Store) cluster can continue providing service, but depending on the affected node type, data redundancy is decreased.

The alarm attributes are listed and explained in Table 2.

Table 2   Alarm Attributes

Attribute Name

Attribute Value

Auto Cease

Yes

Module

STORAGE-ENGINE

Error Code

2

Timestamp First

Date and time when the alarm was raised for the first time.

Repeated Counter

Number which indicates how many times the alarm was raised.

Timestamp Last

Date and time of the most recent alarm raised.

Resource ID

.1.3.6.1.4.1.193.169.1.2.2.<ND>.<DG>.<IP>

Alarm Model Description

Cluster node down, Storage Engine.

Alarm Active Description

Storage Engine (DS-group #<DG>): <NT> node #<ND> down @ <IP>, uuid: <uuid>

ITU Alarm Event Type

processingErrorAlarm (4)

ITU Alarm Probable Cause

softwareProgramError (546)

ITU Alarm Perceived Severity

(4) – Major

Originating source IP

Node IP where the alarm was raised.

Sequence Number

Number which indicates the order in which the alarms are raised.

In Table 2, the indicated variables are as follows:

  • <DG> is the DSG (DS Unit Group) this cluster belongs to.

  • <NT> is the faulty node type (NDB, SQL, MGM).

  • <ND> is the node number within the database cluster.

  • <IP> is the IP address of the faulty node.

  • <uuid> is the universally unique identifier of the computing resource (blade or virtual machine). It is blank if it is not possible to figure out its value.

For further information about attribute descriptions, refer to CUDB Node Fault Management Configuration Guide.

The possible causes are as follows:

  • Data node went down or is unreachable.

  • Node type MGM cannot retrieve cluster status.

  • The data node may not be able to start up if the file system has consistency errors due to a non-graceful shutdown, uncontrolled crash, or HW errors.

1.2 Prerequisites

This section lists the prerequisites required for the procedure described in Procedure.

1.2.2 Tools

Not applicable.

1.2.3 Conditions

Not applicable.

2 Procedure

This section describes the procedure to follow when this alarm is received.

2.1 Actions for BSP Alarm or Alert Related to Hardware Identified by IP Address

If the alarm is not cleared automatically after a short period of time, perform the following steps:

Steps

  1. Check if there is an outstanding BSP alarm or alert related to the hardware identified by <IP> address.
    In case the blade is broken and cannot be fixed, replace the faulty blade. For more information on blade replacement, refer to Server Platform, Blade Replacement.
  2. Check if the Operating System, Disk Usage Too High alarm is raised. In case it is raised, refer to Operating System, Disk Usage Too High.
  3. Check if the data node can start up. To do so, run the following command on the System Controller blades to search for error code 2341 in the log:
    grep "error 2341" /local/cudb/mysql/mgmt/ds$(get_local_ds_id_for_dsg_id <dsg_id>)/ndb_1_cluster.log
    If any data node failed to start with this error, file system or disk errors are the probable cause. Please contact next level of support.
  4. If the alarm does not cease, contact the next level of maintenance support. Further actions are outside the scope of this Operating Instruction.

After This Task

If the faulty node is a data node (NDB), find out if the failed node in the cluster database belongs to the master replica of its DSG by following the instructions in CUDB System Administrator Guide. If this is the case, CUDB might not be able to process the nominal amount of traffic for that DSG. If the nominal traffic-processing capacity is likely to be needed before corrective actions are finished, do consider moving the mastership of the affected DSG to a healthy replica by following the master change procedure in CUDB System Administrator Guide.

Legal |© Ericsson AB 2016, 2018