Contents

1   Introduction

This instruction concerns alarm handling.

1.1   Alarm Description

The IPWorks Storage Server periodically monitors the status of connection between Storage Server and IPWorks database table, once it detects the mysql database is unreachable by using SQL statement query, it reports an alarm.

The possible alarm causes, the corresponding fault reasons, fault locations, and impacts are described in Table 1.

Table 1    Alarm Causes

Alarm Cause	Description	Fault Reason	Fault Location	Impact	Solution
SQL Node Issue	All SQL Nodes are down.	SQL Node is down for maintenance activity or other errors.	NDB cluster	The Storage Server cannot access to SQL Node. This issue affects provisioning in DNS or ENUM service.	See Section 2.2
Data Node Issue	All Data Nodes are down.	Data Node is down for maintenance activity, configuration issue, or memory issue.	NDB cluster	The Storage Server cannot access to database. When both Data Nodes are down, the service (such as, DNS or ENUM) is affected.	See Section 2.3

Note:  

An alarm can appear as a result of the maintenance activity.

The alarm attributes are listed and explained in Table 2.

Table 2    Alarm Attributes

Attribute Name	Attribute Value
Major Type	193
Minor Type	860162
Managed Object Class	IpworksEM
Source	ManagedElement=<Node Name>,SystemFunctions=1,Fm=1,FmAlarmModel=ipworksEM,FmAlarmType=ipworksEmSsDbUnreachable,Source=Storage Server
Specific Problem	Storage Server, MySQL Database Unreachable
Event Type	communicationsAlarm(2)
Probable Cause	x733CommunicationsSubsystemFailure(306)
Additional Text	"This alarm is issued when Storage Server lost communication with Database.";uuid:<Product_UUID>(1)
Perceived Severity	Critical

(1)  <Product_UUID> is the universally unique identifier (UUID) of machine that generates the alarm. The value can be fetched from /sys/devices/virtual/dmi/id/product_uuid on the PL node.

1.2   Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1   Documents

Before starting this procedure, ensure that you have read the following documents:

• System Safety Information
• Personal Health and Safety Information

1.2.2   Tools

No tools.

1.2.3   Conditions

No conditions.

2   Procedure

This section describes the procedure to follow when this alarm is received.

2.1   Analyzing the Alarm

Do the following at the maintenance center:

1. Check SQL Node
2. Check Data Node

2.2   Checking SQL Node

To clear the alarm, do the following:

1. Check whether the SQL Node is down.

   Example:

   SC-1:~#/etc/init.d/ipworks.mysql show-status

   Connected to Management Server at: SC-2:1186
   Cluster Configuration
   ---------------------
   [ndbd(NDB)]     2 node(s)
   id=27  @169.254.100.1  (mysql-5.6.27 ndb-7.4.8, Nodegroup: 0, *)
   id=28  @169.254.100.2  (mysql-5.6.27 ndb-7.4.8, Nodegroup: 0)
   
   [ndb_mgmd(MGM)] 2 node(s)
   id=1  @169.254.100.1  (mysql-5.6.27 ndb-7.4.8)
   id=2  @169.254.100.2  (mysql-5.6.27 ndb-7.4.8)
   
   [mysqld(API)]   24 node(s)
   id=3 (not connected, accepting connect from SC-1)
   id=4 (not connected, accepting connect from SC-2)
   ...

   This example output indicates that the SQL Nodes (id=3 and id=4) are down.

2. If the SQL Node is down, execute the following command to start it:

   Example:

   SC-1:~#/etc/init.d/ipworks.mysql start-sqlnode

3. After the all SQL Nodes are started successfully, confirm that the alarm has ceased. If the alarm remains, consult the next level of maintenance support. Further actions are outside the scope of this instruction.

2.3   Checking Data Node

To clear the alarm, do the following:

1. Check whether the Data Node is down.

   Example:

   SC-1:~#/etc/init.d/ipworks.mysql show-status Connected to Management Server at: localhost:1186 Cluster Configuration --------------------- [ndbd(NDB)] 2 node(s) id=27 (not connected, accepting connect from SC-1) id=28 @169.254.100.2 (mysql-5.6.27 ndb-7.4.8, Nodegroup: 0, *) [ndb_mgmd(MGM)] 2 node(s) id=1 @169.254.100.1 (mysql-5.6.27 ndb-7.4.8) id=2 @169.254.100.2 (mysql-5.6.27 ndb-7.4.8) [mysqld(API)] 24 node(s) id=3 @169.254.100.1 (mysql-5.6.27 ndb-7.4.8) id=4 (not connected, accepting connect from SC-2) ...

   This example output indicates that the Data Node (id=27) is down.

2. If the Data Node is down, execute the following command to start it:

   Example:

   SC-1:~#/etc/init.d/ipworks.mysql start-ndbd

   If the operator needs to initialize the Data Node, execute the following command instead:

   #/etc/init.d/ipworks.mysql start-ndbd-initial

   Note:  

   The initialization deletes all the data in the Data Node.

   
   
3. Check whether the mysql database is reachable.

   Example:

   #ssh SC-1

   #/usr/local/mysql/bin/mysql -P 3307 -h localhost --protocol=tcp

   Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 42 Server version: 5.6.27-ndb-7.4.8-cluster-commercial-advanced MySQL Cluster Server - Advanced Edition (Commercial) Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

   mysql> use ipworks;

   Reading table information for completion of table and column names
   You can turn off this feature to get a quicker startup with -A
   
   Database changed
   

   mysql> select username from user;

   +----------+
   | username |
   +----------+
   | admin    |
   | initial  |
   +----------+
   2 rows in set (0.00 sec)
   

   If the result of the select query is more than one record, that means the database is reachable.

4. Confirm that the alarm has ceased. If the alarm remains, consult the next level of maintenance support. Further actions are outside the scope of this instruction.