Contents

1   Introduction

This instruction concerns alarm handling.

1.1   Alarm Description

The Remote File System is Not Accessible alarm is issued by the Managed Object (MO) Host, when the remote file system, /var/lib/glance is inaccessible in a vCIC.

The severity of the alarm is MAJOR.

The possible alarm causes and fault locations are explained in Table 1.

Table 1    Alarm Causes

Alarm Cause	Description	Fault Reason	Fault Location	Impact
The file system on a remote storage device is inaccessible.	The file system is desynchronized with its remote storage device.	Connection to a remote storage device that is backing the file system is interrupted.	vCIC	Glance services are unable to access the remote storage device.

The following is the consequence for the node if the alarm is not solved:

• Glance services are degraded or lost on the vCIC.
• The level of redundancy is decreased.

The alarm attributes are listed in Table 2.

Table 2    Alarm Attributes

Attribute Name	Attribute Value
Major Type	193
Minor Type	2031818
Managed Object Class	Host
Managed Object Instance	Region=<name_of_the_region>, Equipment=1, Host=<hostname>
Specific Problem	Remote file system is not accessible
Event Type	operationalViolation
Probable Cause	underlayingResourceUnavailable
Additional Text	On node <hostname> <file_system> is not accessible; <hw_uuid>
Severity	MAJOR (4)

1.2   Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1   Documents

Not applicable.

1.2.2   Tools

No tools are required.

1.2.3   Conditions

Before starting this procedure, ensure that the following conditions are met:

• There are no errors reported for the distributed storage system.
• No maintenance activities are taking place.

2   Procedure

This section describes the procedure to follow when the alarm is received.

2.1   Actions

Perform the following:

1. Stop Cinder, Glance, and Swift services:
   1. Stop Cinder services:

      service cinder-api stop 
      service cinder-scheduler stop

   2. Migrate cinder-volume to another vCIC if it runs on the vCIC with the active alarm:

      crm resource migrate p_cinder-volume <cic_address>

      Note:  

      To check where the cinder-volume runs, execute the crm resource status p_cinder-volume on any vCIC.

      
      
   3. Stop Glance services:

      service glance-api stop 
      service glance-registry stop 
      service glance-glare stop

   4. Stop Swift services:

      service swift-account-auditor stop 
      service swift-account stop 
      service swift-account-reaper stop 
      service swift-account-replicator stop 
      service swift-container-auditor stop 
      service swift-container stop 
      service swift-container-replicator stop 
      service swift-container-sync stop 
      service swift-container-updater stop 
      service swift-object-auditor stop 
      service swift-object stop 
      service swift-object-replicator stop 
      service swift-object-updater stop 
      service swift-proxy stop 

2. Check that no process is listed as using /var/lib/glance:

   lsof /var/lib/glance

   The command output should be empty.

3. Unmount the external device:

   umount /dev/mapper/image-glance

4. Verify that no external device is mounted to /var/lib/glance:

   findmnt /var/lib/glance

   An example of the printout is:

   root@cic-1:~# findmnt /var/lib/glance 
   TARGET          SOURCE                     FSTYPE OPTIONS 
   /var/lib/glance /dev/dm-3[/var/lib/glance] ext4   rw,relatime,errors=panic,stripe=32411,data=ordered 
   

   Check that /dev/mapper/image-glance is not listed in the printout.

5. Remount the external device:

   mount /dev/mapper/image-glance

6. Start the services back in reverse order:
   1. Start Swift services:

      service swift-proxy start 
      service swift-object-updater start 
      service swift-object-replicator start 
      service swift-object start 
      service swift-object-auditor start 
      service swift-container-updater start 
      service swift-container-sync start 
      service swift-container-replicator start 
      service swift-container start 
      service swift-container-auditor start 
      service swift-account-replicator start 
      service swift-account-reaper start 
      service swift-account start 
      service swift-account-auditor start 

   2. Start Glance services:

      service glance-glare start 
      service glance-registry start 
      service glance-api start

   3. Start Cinder services:

      service cinder-scheduler start 
      service cinder-api start 

7. Verify that the external device is remounted to /var/lib/glance:

   findmnt /var/lib/glance

   An example of the printout is:

   root@cic-1:~# findmnt /var/lib/glance 
   TARGET          SOURCE                     FSTYPE OPTIONS 
   /var/lib/glance /dev/dm-3[/var/lib/glance] ext4   rw,relatime,errors=panic,stripe=32411,data=ordered 
   /var/lib/glance /dev/mapper/image-glance   xfs    rw,relatime,attr2,inode64,noquota 
   

   Check that /dev/mapper/image-glance is in the printout.

   If the alarm ceases, exit this procedure. If the alarm persists, continue with Step 8.

8. Collect troubleshooting data as described in the Data Collection Guideline.
9. Contact the next level of maintenance support.

   Further actions are outside the scope of this instruction.

10. The job is complete.