Diameter AAA, Server Lost Connections of DB
IPWorks

Contents

1Introduction
1.1Alarm Description
1.2Prerequisite
1.3Related Information

2
Procedure
2.1Configuring AAA server
2.2Starting Management Node and Data Node

Reference List

1   Introduction

This instruction concerns alarm handling.

1.1   Alarm Description

The alarm is issued when the connections between AAA server and MySQL NDB clusters are down.

The possible alarm causes and the corresponding fault reasons, fault locations, and impacts are described in Table 1.

Table 1    Alarm Causes

Alarm Cause

Description

Fault Reason

Fault Location

Impact

Solution

AAA configuration error occurs.

The alarm is raised because of the incorrect configuration of AAA.

The attributes ndbConnectString in the MO IpworksDiameterAAA are configured incorrectly.

AAA server

AAA server cannot provide service.

See Section 2.1

NDB cluster is under abnormal condition.

All Management Nodes or Data Nodes are down.

All the NDB connections are down.

NDB cluster

See Section 2.2

Note:  
An alarm can appear as a result of the maintenance activity.

The alarm attributes are listed and explained in Table 2.

Table 2    Alarm Attributes

Attribute Name

Attribute Value

Major Type

193

Minor Type

864257

Managed Object Class

IpworksDiameterAAA

Source

ManagedElement=<Node Name>, SystemFunctions=1,Fm=1,FmAlarmModel=ipworksDiameterAAA, FmAlarmType=ipworksDiameterAAAConnectionToDBError

Specific Problem

Diameter AAA, Server Lost Connections of DB

Event Type

communicationsAlarm(2)

Probable Cause

x733CommunicationsSubsystemFailure(306)

Additional Text

All the NDB connections are down. Diameter AAA requests that trigger NDB queries will fail with SERVFAIL response.;uuid:<Product_UUID>(1)

Perceived Severity

Major

(1)  <Product_UUID> is the universally unique identifier (UUID) of machine that generates the alarm. The value can be fetched from /sys/devices/virtual/dmi/id/product_uuid on the PL node.


1.2   Prerequisite

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1   Documents

Before starting this procedure, ensure that the following document has been read:

1.2.2   Tools

Not available

1.2.3   Conditions

Not applicable.

1.3   Related Information

Trademark information, typographic conventions, and definition and explanation of abbreviations and terminology can be found in the following documents:

2   Procedure

This section describes the procedure to follow to clear this alarm.

2.1   Configuring AAA server

To clear the alarm, do the following:

  1. Log on to the ECLI interface.

    # ssh <username>@<OAM IP Address> -t -s cli

  2. Check the configuration of MySQLInfo.

    >ManagedElement=<Node Name>,IpworksFunction=1,IpworksCommonRoot=1,DataBaseInfo=1,MySQLInfo=1

    (MySQLInfo=1)>show -v -r

    For example:

    MySQLInfo=1
   mySQLInfoId="1" <default>
   ndbConnectString <default>
      "SC-2:1186"
      "SC-1:1186"
   SQLNodeInfo=1
      host="SC-1" <default>
      password=[] <empty>
      port=3307 <default>
      sqlNodeInfoId="1" <default>
      user="root" <default>
   SQLNodeInfo=2
      host="SC-2"
      password=[] <empty>
      port=3307 <default>
      sqlNodeInfoId="2"
      user="root" <default>

  3. Check the following configuration parameters of MySQL:

    The configuration files are saved at SC-X: /etc/ipworks/mysql/confs.

    File Name

    Parameter

    ipworks_datanode_my.conf

    ndb-connectstring

    ipworks_mgm.conf

    HostName

    PortNumber

    ipworks_sqlnode.conf

    ndb-connectstring

    port

    Verify whether the configuration shown in Step 2 match with the configuration in this step. If not, continue to proceed with Step 4.

  4. Configure the MO MySQLInfo.

    >ManagedElement=<Node Name>,IpworksFunction=1,IpworksCommonRoot=1,DataBaseInfo=1,MySQLInfo=1

    (MySQLInfo=1)>configure

    (config-MySQLInfo=1)>ndbConnectString=["SC-1:1186","SC-2:1186"]

    (config-MySQLInfo=1)>commit

    (config-MySQLInfo=1)>exit

  5. Restart the AAA service on Payload (PL) node to make the change take effect.

    PL-X:~ # /etc/init.d/ipworks.aaa stop

    PL-X:~ # /etc/init.d/ipworks.aaa start

  6. If this alarm still exists, consult the next level of maintenance support. Further actions are outside the scope of this instruction.

2.2   Starting Management Node and Data Node

To clear the alarm, do the following:

  1. Log on to the SC-1.

    # ssh <Username>@<SC-1 IP Address>

  2. Check NDB status:

    SC-1:~ # ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=27   @169.254.100.1  (mysql-5.6.31 ndb-7.4.12, Nodegroup: 0, *)
id=28   @169.254.100.2  (mysql-5.6.31 ndb-7.4.12, Nodegroup: 0)

[ndb_mgmd(MGM)] 2 node(s)
id=1    @169.254.100.1  (mysql-5.6.31 ndb-7.4.12)
id=2    @169.254.100.2  (mysql-5.6.31 ndb-7.4.12)

[mysqld(API)]   24 node(s)
id=3    @169.254.100.1  (mysql-5.6.31 ndb-7.4.12)
id=4    @169.254.100.2  (mysql-5.6.31 ndb-7.4.12)
id=5    @169.254.100.3  (mysql-5.6.31 ndb-7.4.12))
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
id=10 (not connected, accepting connect from any host)
id=11 (not connected, accepting connect from any host)
id=12 (not connected, accepting connect from any host)
id=13 (not connected, accepting connect from any host)
id=14 (not connected, accepting connect from any host)
id=15 (not connected, accepting connect from any host)
id=16 (not connected, accepting connect from any host)
id=17 (not connected, accepting connect from any host)
id=18 (not connected, accepting connect from any host)
id=19 (not connected, accepting connect from any host)
id=20 (not connected, accepting connect from any host)
id=21 (not connected, accepting connect from any host)
id=22 (not connected, accepting connect from any host)
id=23 (not connected, accepting connect from any host)
id=24 (not connected, accepting connect from any host)
id=25 (not connected, accepting connect from any host)
id=26 (not connected, accepting connect from any host)

ndb_mgm>

    Above output is expected result, if result indicates that certain nodes are not started. Continue to proceed with Step 3.

  3. Start the Management Node, Data Node, and SQL Node:

    SC-1:~ # /etc/init.d/ipworks.mysql start-mgmd

    SC-1:~ # /etc/init.d/ipworks.mysql start-ndbd

    SC-1:~ # /etc/init.d/ipworks.mysql start-sqlnode

    For more information on how to manage MySQL NDB Cluster, refer to Configure MySQL NDB Cluster.

    Log on to the SC-2, then start the Management Node , Data Node and SQL Node. And ensure the NDB status is identical to the output of Step 2.

    If your all mysql nodes is down, you can start all nodes with one command :

    SC-X:~ # /etc/init.d/ipworks.mysql start-ndbcluster

  4. Start AAA server on PL:

    PL-X:~ # /etc/init.d/ipworks.aaa start

    Starting the AAA Serve

    Also, you can restart AAA server on SC, for example:

    SC-X:~ # ipw-ctr restart aaa_diameter PL-3

    SC-X:~ # ipw-ctr restart aaa_diameter PL-4

  5. Confirm that the alarm has ceased. If the alarm remains, consult the next level of maintenance support. Further actions are outside the scope of this instruction.

Reference List

Ericsson Documents
[1] Trademark Information.
[2] Typographic Conventions.
[3] Glossary of Terms and Acronyms.
[4] Fault Management.
[5] Configure MySQL NDB Cluster.


Copyright

© Ericsson AB 2017, 2018. All rights reserved. No part of this document may be reproduced in any form without the written permission of the copyright owner.

Disclaimer

The contents of this document are subject to revision without notice due to continued progress in methodology, design and manufacturing. Ericsson shall have no liability for any error or damage of any kind resulting from the use of this document.

Trademark List
All trademarks mentioned herein are the property of their respective owners. These are shown in the document Trademark Information.

    Diameter AAA, Server Lost Connections of DB         IPWorks