LOTC Disk Replication Communication

1	Alarm Description
2	Procedure
2.1	Handle Alarm LOTC Disk Replication Communication

1 Alarm Description

The alarm is raised when the control nodes have lost connection to each other for more than 20 minutes, and are no longer in redundant mode. The control node pair is in a non-redundant mode when the control nodes have no connection with each other.

Table 1 LOTC Disk Replication Communication Alarm Causes
Alarm Cause	Description	Fault Reason	Fault Location	Impact
Loss of connection between control nodes for more than 20 minutes	The control nodes have lost connection to each other for more than 20 minutes. The Linux® service Distributed Replicated Block Device (DRBD) is not in connected mode.	Network failure leading to communication problems between the control nodes	Network	Both controllers take the primary role and no data is transferred between the nodes
		Hardware failure on the secondary control node	Secondary control node	If one of the controller nodes is down, the cluster does not have a controller node to which it can fail over

Note:: This alarm can appear as a result of a maintenance activity.

2 Procedure

2.1 Handle Alarm LOTC Disk Replication Communication

Prerequisites

This instruction references the following documents:
- Data Collection Guideline
- LOTC Ethernet Bonding
No tools are required.
The following condition must apply:
- The alarm is raised.

Steps

Log on to the host to access a Linux shell, for example:
ssh <user>@<hostname> -p 7022

The hostname is part of alarm attribute Source.
Is the alarm raised during initial installation or replacement of a control node?
Yes: Continue with the next step.

No: Proceed with Step 8.

Check which drbd version you are running:

cat /proc/drbd

version: 8.4.2 (api:1/proto:86-101) 
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@lixia, 2012-09-19 16:40:30  
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/UpToDate C r---n-

Is drbd version: 8.* ?

Yes: continue with next step.

No: proceed with Step 6.

Wait for DRBD connection to be established. Check if the following command results in output cs:Connected:

cat /proc/drbd

The following is an example output in a normal situation. The connection state (cs) is Connected. The alarm is cleared within 5 seconds.

version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@lixia, 2012-09-19 16:40:30
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:438816 nr:0 dw:372 dr:440669 al:11 bm:40 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oo

The following is an example output in a faulty situation. The connection state (cs) is WFConnection (Waiting For Connection).

version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@lixia, 2012-09-19 16:40:30
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
    ns:143396 nr:0 dw:448 dr:147057 al:17 bm:28 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:84

Does the output contain cs:Connected and is the alarm cleared?
Yes: Proceed with Step 11.

No: Perform data collection, refer to Data Collection Guideline. Contact the deployment organization. Proceed with Step 8.

Wait for DRBD connection to be established. Check if the following command results in output connection:Connected:

drbdsetup events2 --statistics --now

The following is an example output in a normal situation. The connection state (connection) is Connected. The alarm is cleared within 5 seconds.

exists resource name:drbd0 role:Primary suspended:no write-ordering:flush
exists connection name:drbd0 peer-node-id:1 conn-name:node2-vc11 connection:Connected role:Secondary congested:no
exists device name:drbd0 volume:0 minor:0 disk:UpToDate size:10485760 read:10774713 written:4200 al-writes:6 ⇒
bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no
exists peer-device name:drbd0 peer-node-id:1 conn-name:node2-vc11 volume:0 replication:Established ⇒
peer-disk:UpToDate resync-suspended:no received:0 sent:10486244 out-of-sync:0 pending:0 unacked:0
exists -

The following is an example output in a faulty situation. The connection state (connection) is Connecting.

exists resource name:drbd0 role:Primary suspended:no write-ordering:flush
exists connection name:drbd0 peer-node-id:1 conn-name:node2-vc11 connection:Connecting role:Unknown congested:no
exists device name:drbd0 volume:0 minor:0 disk:UpToDate size:10485760 read:289085 written:3420 al-writes:5 ⇒
bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no
exists peer-device name:drbd0 peer-node-id:1 conn-name:node2-vc11 volume:0 replication:Off peer-disk:DUnknown ⇒
resync-suspended:no received:0 sent:0 out-of-sync:974888 pending:0 unacked:0
exists -

Does the output contain conection:Connected and is the alarm cleared?
Yes: Proceed with Step 11.

No: Perform data collection, refer to Data Collection Guideline. Contact the deployment organization. Proceed with Step 11.
Is the LOTC Ethernet Bonding alarm raised?
Yes: Clear the LOTC Ethernet Bonding alarm, refer to LOTC Ethernet Bonding. Further actions are outside the scope of this instruction. Proceed with Step 11.

No: Continue with the next step.
Perform data collection, refer to Data Collection Guideline.
Consult the next level of maintenance support. Further actions are outside the scope of this instruction.
Job is completed.