Health Check Procedure

Cloud Execution Environment

1	Introduction
1.1	Scope
1.2	Target Groups
1.3	Prerequisites
2	Overview
2.1	Execution of Commands for Several Hosts
2.2	Execution of Commands for vCICs
2.3	Execution of Commands for Compute Hosts
3	Daily Health Check Procedure
3.1	Check Alarms And Alarm History
3.2	Check Presence of Crash Dumps
3.3	Verify Date and Time
3.4	Check Pacemaker - CIC State/Cluster Resource State
3.5	Check CIC Maintenance Mode
3.6	Check Neutron Agents
3.7	Check Nova Services
3.8	Verify Disk Space Utilization
3.9	Verify RAM Utilization
3.10	Check iSCSI Multipath Connection to VNX
3.11	Check ScaleIO Cluster Status
4	Pre- and Post-Activity Health Check Procedure
4.1	Check OpenStack Components
4.2	State of Extreme Switches from Perspective of Neutron
4.3	Check the State of Ethernet Interfaces
4.4	Check Service Status
4.5	Check Ethernet Statistics
4.6	Check RabbitMQ Cluster Status
4.7	Check Zombie Processes
4.8	Check Fuel Status
4.9	Check Fuel Services
4.10	Check Swift Store on VNX / ScaleIO
5	Report Problems
Reference List

1 Introduction

This document is to help support engineers check that the Cloud Execution Environment (CEE) operates in a fault-free state, and to detect issues that can affect normal operation.

1.1 Scope

This document has been verified on the CEE certified configuration, as specified in the BOM for Certified HW Configurations, Reference [1]. The process is applicable to other CEE configurations.

1.2 Target Groups

This document is intended for both internal and external customers monitoring system health:

Support organization personnel
Customer O&M personnel

1.3 Prerequisites

This section describes the prerequisites for performing the health check procedure.

1.3.1 Documents

Before starting the procedure, ensure that the following documents are available:

1.3.2 Conditions

Before performing a health check, ensure that the following conditions are met:

Users of this document must be familiar with commands and tools within CEE and OpenStack.
Access to deployment-specific credentials must be available.
There are no ongoing CEE maintenance activities.

2 Overview

This document covers the procedures for checking the health of CEE and detecting issues before they become threats to the system.

"Health" in the context of this document means that CEE is running, provides the required functionality, and is available for the users.

Health condition is evaluated by executing several checks. These checks are based on the information collected from printouts. If problems are encountered during any of the checks, the user is provided with a recommendation.

The time needed to execute health checks depends on factors such as the complexity of the checks or the system performance. If a check takes a long time, it is possible that CEE is not functioning correctly. The checks in this document are classified as:

Daily Health Check: Checks for CEE to be executed on a daily basis
Pre- and Post-Activity Health Check: Checks to be executed before and after an update of CEE or other maintenance activities are performed. It is recommended to execute these checks on a monthly basis.

Note:: CEE collects a large amount of ISP and Fault Management (FM) data. Alarms must be available in the management system (Atlas).

2.1 Execution of Commands for Several Hosts

It is possible to execute commands for several hosts by using the below syntax, replacing <command> with the specific command to be executed. The following examples show the execution of commands from Fuel, as root user.

Example syntax for execution of commands for Virtual Cloud Infrastructure Controller (vCIC) nodes:

for n in $(fuel node | awk -F '|' '$7 ~ ⇒
/controller/ {print $3}'); do echo ${n}; ssh -o ⇒
LogLevel=quiet ${n} '<command>'; done

for n in $(fuel node | awk -F '|' '$7 ~ /controller/ {print $3}'); do echo ${n}; ssh -o LogLevel=quiet ${n} '<command>'; done

for example

for n in $(fuel node | awk -F '|' '$7 ~ ⇒
/controller/ {print $3}'); do echo ${n}; ssh -o ⇒
LogLevel=quiet ${n} 'date'; done

for n in $(fuel node | awk -F '|' '$7 ~ /controller/ {print $3}'); do echo ${n}; ssh -o LogLevel=quiet ${n} 'date'; done

Example syntax for execution of commands for Compute Hosts:

for n in $(fuel node | awk -F '|' '$7 ~ ⇒
/compute/ {print $3}'); do echo ${n}; ssh -o ⇒
LogLevel=quiet ${n} '<command>'; done

for n in $(fuel node | awk -F '|' '$7 ~ /compute/ {print $3}'); do echo ${n}; ssh -o LogLevel=quiet ${n} '<command>'; done

for example

for n in $(fuel node | awk -F '|' '$7 ~ ⇒
/compute/ {print $3}'); do echo ${n}; ssh -o ⇒
LogLevel=quiet ${n} 'date'; done

for n in $(fuel node | awk -F '|' '$7 ~ /compute/ {print $3}'); do echo ${n}; ssh -o LogLevel=quiet ${n} 'date'; done

Example syntax for execution of commands for Fuel, vCIC nodes and Compute Hosts:

for n in fuel $(fuel node | awk -F '|' '$7 ~ ⇒
/controller|compute/ {print $3}'); do echo ${n}; ⇒
ssh -o LogLevel=quiet ${n} '<command>'; done

for n in fuel $(fuel node | awk -F '|' '$7 ~ /controller|compute/ {print $3}'); do echo ${n}; ssh -o LogLevel=quiet ${n} '<command>'; done

2.2 Execution of Commands for vCICs

In order to populate the environment variables needed for execution of OpenStack commands, and to execute following commands as root, the following commands are recommended:

ssh <CEE_Administrator>@<CIC_Hostname/IP_Address>
sudo -i
source openrc

2.3 Execution of Commands for Compute Hosts

In order to execute commands as root—which is needed to get command output—it is recommended to use the sudo command:

ssh <CEE_Administrator>@<Compute_Host_IP_Address>
sudo -i

3 Daily Health Check Procedure

This section describes the procedures for checking the health of the CEE system on a daily basis.

3.1 Check Alarms And Alarm History

Performance and Fault Management alarms are reported by Watchmen to Atlas or the Ericsson Cloud Manager (ECM). Check the active alarms and act according to the relevant Operating Instructions (OPIs) in each case.

Required tools

Atlas or ECM Graphical User Interface (GUI)

Conditions

There are no conditions.

Procedure

To check the alarms and alarm history using Atlas, do the following:

Log on to Atlas through the GUI.
Check for alarms.

Expected result

One of the following results is expected:

No alarms are found.
Alarms are found. The cause of the alarms is known. In case of any active alarms, refer to the corresponding OPI and act accordingly.

3.2 Check Presence of Crash Dumps

Required tools

Command-Line Interface (CLI)

Conditions

There are no conditions.

Procedure

To check for the presence of crash dumps, do the following:

Execute the below commands on all Cloud Infrastructure Controller (CIC) hosts and compute hosts:
```
ls -al /var/log/crash/cores
ls -al /var/log/crash/kernelcrashes/
```

Expected result

No crash dumps are present.

3.3 Verify Date and Time

Note:: Execute the command on all CIC hosts and compute hosts.

Required tools

1 Introduction

1.1 Scope

1.2 Target Groups

1.3 Prerequisites

1.3.1 Documents

1.3.2 Conditions

2 Overview

2.1 Execution of Commands for Several Hosts

2.2 Execution of Commands for vCICs

2.3 Execution of Commands for Compute Hosts

3 Daily Health Check Procedure

3.1 Check Alarms And Alarm History

3.2 Check Presence of Crash Dumps

3.3 Verify Date and Time

3.4 Check Pacemaker - CIC State/Cluster Resource State

3.5 Check CIC Maintenance Mode

3.6 Check Neutron Agents

3.7 Check Nova Services

3.8 Verify Disk Space Utilization

3.9 Verify RAM Utilization

3.10 Check iSCSI Multipath Connection to VNX

3.11 Check ScaleIO Cluster Status

4 Pre- and Post-Activity Health Check Procedure

4.1 Check OpenStack Components

4.2 State of Extreme Switches from Perspective of Neutron

4.3 Check the State of Ethernet Interfaces

4.4 Check Service Status

4.5 Check Ethernet Statistics

4.6 Check RabbitMQ Cluster Status

4.7 Check Zombie Processes

4.8 Check Fuel Status

4.9 Check Fuel Services

4.10 Check Swift Store on VNX / ScaleIO

5 Report Problems

Reference List