Contents

1   Introduction

This instruction concerns alert handling.

1.1   Alert Description

The Service Stopped alert is issued in the following cases:

• A service operating at a vCIC or compute node is stopped.

  The possible alarm cause and the corresponding fault reasons, fault locations and impacts are described in Table 1. For the list of services monitored by the Service Supervision plugin, see Section 3.

• The alarm can also appear as a result of maintenance activity.

The alert attributes are listed in Table 2.

1.2   Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1   Documents

Not applicable.

1.2.2   Tools

No tools are required.

1.2.3   Conditions

Before starting this procedure, ensure that the alert was not issued due to ongoing planned maintenance. If the alert was issued due to ongoing planned maintenance, no further actions are required.

2   Procedure

This section describes the procedure to follow when this alert is received.

2.1   Analysis

Do the following to analyze the alert:

1. Check if the Service Permanently Stopped alarm is issued for the same service.
   • If the Service Permanently Stopped alarm is issued, refer to Service Permanently Stopped, and exit this procedure.
   • If the Service Permanently Stopped alarm is not issued, continue with Step 2.
2. Count the number of alert occurrences in a 10 minute period and perform the relevant action:
   • If the alert occurs less than five times in 10 minutes, no actions are needed, the job is completed as the service has been recovered by Service Supervision.
   • If the alert occurs five or more times in 10 minutes, continue with Section 2.2.

2.2   Actions

Do the following:

1. Depending on the node type, perform the relevant action:
   • If the affected node is not a compute node, continue with Step 3.
   • If the fault is detected at a compute node, try to move the virtual machines (VMs) by using the following command with the <hostname_of_the_node> reported in the alert:

for VM in $(nova list --host <hostname_of_the_node>); do nova forcemove $VM; done

2. If the alert does not reoccur in the next 10 minutes after moving the VMs, the job is completed. Else, continue with Step 3.
3. Collect troubleshooting data as described in the Data Collection Guideline.
4. Consult the next level of maintenance support. Further actions are outside the scope of this instruction.
5. The job is completed.

3   Additional Information

The Service Supervision plugin monitors the following services: