Region Scale-in

Cloud Execution Environment

1	Introduction
1.1	Scope
1.2	Prerequisites
1.2.1	Documents
1.2.2	Tools
1.2.3	Data
2	Procedure
2.1	Create Backups
2.2	Disable nova-compute
2.3	Handling VMs
2.4	Scale in the CEE Region
2.5	Concluding Routine

1 Introduction

This document provides instructions on how to perform region scale-in in the Cloud Execution Environment (CEE).

1.1 Scope

This document describes how to remove one or more compute hosts from the CEE region. When a compute host is removed from the CEE region, it is decommissioned from the CEE cluster, and corresponding Nova compute and Neutron agent resources (if present) are removed. Previously recorded data for the removed host or hosts is not removed from CEE databases.

The described procedure is applicable for scale-in of compute hosts not hosting a Virtual Cloud Infrastructure Controller (vCIC) or vFuel. Scale-in of compute hosts hosting a virtual Cloud Infrastructure Controller (vCIC) or the active or passive virtual Fuel (vFuel) is not available in the current release of CEE.

Multiple hosts can be scaled in with the same procedure. The upper limit for the number of removable compute hosts with one procedure is only determined by the number of computes in the region to which the procedure is applicable. Compute hosts hosting vCIC or vFuel or both, and additional physical servers, for example ScaleIO hosts, cannot be removed using the region scale-in procedure.

The procedure is not applicable to Dell single server platform.

The possible physical removal of any server from the data center is out of scope of this document. Refer to manufacturer documentation or instructions, provided by the owner of the data center.

Note:: The CEE region scale-in procedure does not handle the removal of infrastructure and tenant data from the scaled-in server or servers. Handle the scaled-in server or servers according to company procedures regarding data security. Sensitive data may be present on the server or servers.

1.2 Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1 Documents

Not applicable.

1.2.2 Tools

A computer able to do a Secure Shell (SSH) logon to vCICs and vFuel is required.

root access to the vFuel node is required. The procedures below can only be executed as root.

1.2.3 Data

The IP addresses and credentials are required for SSH connections to the following devices:

vCIC
vFuel

For more information, refer to the CEE Connectivity User Guide.

2 Procedure

This procedure describes how to remove one or more compute hosts from the CEE.

The procedure consists of the following stages:

Creating backups, see Section 2.1.
Disabling nova-compute, see Section 2.2
VM migration, see Section 2.3
CEE region scale-in, see Section 2.4
Concluding steps, see Section 2.5

Start the procedure with Section 2.1.

2.1 Create Backups

Perform the procedures described in the following documents:

CIC Domain Data Backup, including exporting the backup to an external location
Fuel Synchronization

For a full list of backup and restore documents in CEE and their use cases, refer to the Backup and Restore Overview.

2.2 Disable nova-compute

To avoid any VMs being scheduled to the compute hosts to be scaled in, nova-compute must be disabled on all compute hosts to be scaled in. Do the following:

Note:: If this procedure fails, exit the procedure and contact next level of support. Corrective actions are outside the scope of this document.

Log on to any of the vCICs using SSH. For more information, refer to the CEE Connectivity User Guide.

Disable nova-compute for each of the compute hosts to be scaled in, one by one:

source openrc; nova service-disable <hostname>.domain.tld nova-compute --reason scaledin

source openrc; nova⇒
 service-disable <hostname>.domain.tld nova-compute⇒
 --reason scaledin

where <hostname> corresponds to the name of the compute host to be scaled in, in the format compute-<shelf_id>-<blade_id>.

An example of the command is the following:

source openrc; nova service-disable compute-1-2.domain.tld nova-compute --reason scaledin

source openrc; nova⇒
 service-disable compute-1-2.domain.tld nova-compute⇒
 --reason scaledin

Exit the vCIC:
exit

2.3 Handling VMs

Before performing scale-in, VMs hosted on the compute hosts to be scaled in must be migrated or deleted, depending on user requirements.

VMs running on a compute host can be identified by logging on to the compute host as described in the CEE Connectivity User Guide and executing the command virsh list --all.

Note:: If this procedure fails, exit the procedure and contact next level of support. Corrective actions are outside the scope of this document.

Log on to any of the vCICs using SSH. For more information, refer to the CEE Connectivity User Guide.
To migrate VMs, do the following:
1. If the system is configured with tightly integrated SDN, execute the following script before migrating the VMs:
```
/opt/sdnc/tools/scalein/sdn_tombstone_computes.sh <hostname>
```
```
/opt/sdnc/tools/scalein/sdn_tombstone_computes.sh⇒
 <hostname>
```
  where <hostname> refers to a space-separated list of the names of the compute hosts to be scaled in, in the format compute-<shelf_id>-<blade_id>.
  
  An example of the command is the following:
```
/opt/sdnc/tools/scalein/sdn_tombstone_computes.sh compute-1-2 compute-1-3
```
```
/opt/sdnc/tools/scalein/sdn_tombstone_computes.sh⇒
 compute-1-2 compute-1-3
```
2. Migrate the VMs one by one using the following command:
```
source openrc; nova migrate <vm_name>
```
  Note:
  If the VM is launched with affinity/anti-affinity scheduler hint, migration with nova migrate can fail. In this case, use nova forcemove with --ignore-hints option for migration. If nova forcemove --ignore-hints is used, affinity groups are not considered any more for the instance.
  For more information about affinity groups, refer to the section on hints for VM affinity and anti-affinity in the OpenStack Compute API in CEE document.
3. Confirm the migration:
```
nova resize-confirm <vm_name>
```
To delete VMs, execute the following command:
```
source openrc; nova delete <instance>
```
where <instance> corresponds to the name or ID of the VM to be deleted.
If all required VMs are migrated or deleted, exit the vCIC:
exit

2.4 Scale in the CEE Region

Do the following:

Note:: If this procedure fails, exit the procedure and contact next level of support. Corrective actions are outside the scope of this document.

Scale in the region using the following command on vFuel:
```
ceescalein --names <hostname>
```
where <hostname> refers to a space-separated list of the compute hosts to be scaled in, in the format compute-<shelf_id>-<blade_id>.

An example of the command is the following:
```
ceescalein --names compute-1-2 compute-1-3
```
Note:
If scale-in fails for any of the compute hosts when multiple hosts are scaled in, the scale-in command continues to execute for the remaining set of compute hosts. The details of the compute host for which scale-in has failed is listed in the printout.

After the scale-in operation is completed, alarms related to the scaled-in computes are triggered and remain in the active alarm list after the compute is permanently removed. The respective alarms are not cleared during Watchmen synchronization as the state of the event remains unchanged. These non-active "stale" alarms are cleared by Watchmen after the next regular periodic check. Periodic checks are performed every 1800 seconds. If the alarms do not clear after the next regular periodic check, refer to the relevant alarm Operating Instruction (OPI).
Update /mnt/cee_config/config.yaml by removing the configuration data for the scaled-in servers. The server configuration information is located at the following section:
```
ericsson:
  ...
  shelf:
    -
      id: <shelf_id>
      blade:
        -
          id: <blade_id>
          ...
```
The section for the relevant servers can be identified using the name of the host, which has the following structure: compute-<shelf_id>-<blade_id>.

For more information, refer to the section on server configuration in the Configuration File Guide.
Synchronize the active Fuel VM to the cold standby Fuel VM as described in Fuel Synchronization.

2.5 Concluding Routine

Do the following:

Perform the backup procedures described in the following documents:
- CIC Domain Data Backup
- Disaster Recovery
For more information, refer to Backup and Restore Overview.
Handle the scaled-in server or servers according to company procedures regarding data security.
Note:
Sensitive data, including passwords in cleartext may be present on the scaled-in server or servers.