1 Graceful Scale-in Introduction

This instruction describes how to perform graceful scale-in operation, to finally remove one or several PayLoad (PL) Virtual Machines (VM) from the SAPC.

2 Graceful Scale-in Prerequisites

The following conditions must apply:

The number of Payload (PL) Virtual Machines to be scaled-in is known: PL-3 is the first PL of the SAPC, PL-4 is the second one, PL-5 is the third one, and so on.

Note: The first two PLs (PL-3 and PL-4) cannot be scaled-in as they belong to the initial deployment. Therefore, the number of the PL to be scaled-in is always PL-5 onwards.
No other upgrade operations are ongoing, for example, software upgrade.
An Ericsson Command-Line Interface (ECLI) session in Exec mode is in progress.
A secure shell (SSH) access to the SAPC OAM VIP.
To run the scale-in script for removing VMs in CEE using ECM, it is required to have a host acting as "preparation server". This preparation server must have connectivity towards the ECM server.

3 Graceful Scale-in Procedure

To scale-in a PL VM in the SAPC, basically there are two main phases:

The PL to be scaled-in is deleted internally from the SAPC.
Once the PL is not internally recognized and everything keeps fine in the SAPC, the PL VM can be deleted from the cloud manager.

3.1 Internal Graceful Scale-In

To scale-in a PL VM internally in the SAPC:

Steps

From a System Controller (SC) Virtual Machine of the SAPC Virtual Application, execute the following command to perform a graceful scale-in of the PL indicated. At the end of the process, the PL Virtual Machine is automatically shut down:
sapcadmin@SC-X:~> sapcScaleIn <PL-Y>

Note: PL-Y terminology is used in the CBA cluster.
Navigate to the CrM Managed Object (MO), for example:
> dn ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1
Verify that the scaling-in process has started, for example:
(CrM=1)> show -r

Note: The scale-in operation can take 1–2 minutes.

The following is an example output:

Example
```
CrM=1
[...]
   ComputeResourceRole=PL-8
      adminState=SHUTTINGDOWN
      instantiationState=UNINSTANTIATING
[...]
```
The attributes adminState=SHUTTINGDOWN and instantiationState=UNINSTANTIATING show that the graceful scale-in has started for the PL-8 VM.

Verify that the PL VM is scaled-in:

(CrM=1)> show -r

The following is an example output showing that the VM ComputeResourceRole=PL-8 is no longer running:

Example

CrM=1
   autoRoleAssignment=ENABLED
   ComputeResourceRole=PL-3
      adminState=UNLOCKED
      instantiationState=INSTANTIATED
      operationalState=ENABLED
      provides="ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,Role=SYSTEM"
      uses="ManagedElement=1,Equipment=1,ComputeResource=PL-3"
   ComputeResourceRole=PL-4
      adminState=UNLOCKED
      instantiationState=INSTANTIATED
      operationalState=ENABLED
      provides="ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,Role=Default-Role"
      uses="ManagedElement=1,Equipment=1,ComputeResource=PL-4"
   ComputeResourceRole=SC-1
      adminState=UNLOCKED
      instantiationState=INSTANTIATED
      operationalState=ENABLED
      provides="ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,Role=SYSTEM"
      uses="ManagedElement=1,Equipment=1,ComputeResource=SC-1"
   ComputeResourceRole=SC-2
      adminState=UNLOCKED
      instantiationState=INSTANTIATED
      operationalState=ENABLED
      provides="ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,Role=SYSTEM"
      uses="ManagedElement=1,Equipment=1,ComputeResource=SC-2"
   ComputeResourceRole=PL-5
      adminState=UNLOCKED
      instantiationState=INSTANTIATED
      operationalState=ENABLED
      provides="ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,Role=Default-Role"
      uses="ManagedElement=1,Equipment=1,ComputeResource=PL-5"
   ComputeResourceRole=PL-6
      adminState=UNLOCKED
      instantiationState=INSTANTIATED
      operationalState=ENABLED
      provides="ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,Role=Default-Role"
      uses="ManagedElement=1,Equipment=1,ComputeResource=PL-6"
   ComputeResourceRole=PL-7
      adminState=UNLOCKED
      instantiationState=INSTANTIATED
      operationalState=ENABLED
      provides="ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,Role=Default-Role"
      uses="ManagedElement=1,Equipment=1,ComputeResource=PL-7"
   Role=Default-Role
      isProvidedBy
         "ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,ComputeResourceRole=PL-3"
         "ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,ComputeResourceRole=PL-4"
         "ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,ComputeResourceRole=PL-5"
         "ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,ComputeResourceRole=PL-6"
         "ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,ComputeResourceRole=PL-7"
      scalability=SCALABLE
   Role=SYSTEM
      isProvidedBy
         "ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,ComputeResourceRole=SC-1"
         "ManagedElement=1,SystemFunctions=1,SysM=1,CrM=1,ComputeResourceRole=SC-2"
      scalability=NON_SCALABLE

Perform a SAPC health check, refer to SAPC Advanced Troubleshooting Guideline document.
Repeat the procedure as many times as the number of PLs to be scaled-in.
Perform a system backup following the produce described in Create Backup.

3.2 VM Deletion from the Cloud Manager

Once the PL VM has been internally deleted from the SAPC, it can be also deleted from the cloud manager to release resources and for a proper scale-out operation in the future.

Depending on the cloud infrastructure and manager used, refer to VM Deletion using ECM (CEE using ECM), VM Deletion using Atlas/Director (OpenStack using Atlas/Director), VM Deletion using VMware vSphere Web Client (VMware vCenter using vSphere Client) or VM Deletion using VMware vCloud Director (VMware vCenter using vCloud Director).

4 VM Deletion using ECM

Steps

From the preparation server, the script for scale-in, delivered as part of the Virtual Delivery Package (VDP), must be executed. Before the execution, the following information must be provided in the "ecm" and "scaling" sections of the configuration file SAPC.cfg:

Note:

Detailed description of the parameters contained on the scaling section of SAPC.cfg can be found on the SAPC VNF Descriptor Generator Tool.

The name of the SAPC Virtual Application (vApp).

Note:

In case, more than one SAPC vAPP are deployed on the same Virtual Data Center (VDC), a prefix is added to the name of the additional instances. This prefix must be also provided as part of the name of the vApp to be scaled-in.

VDC where the vApp is deployed.
Tenant of the VDC where the vApp is deployed.
ECM server hostname or IP Address.
User to connect with the ECM server. Password is optional, in case it is not provided in the configuration file, the user is prompted.
Number of PLs to be scaled-in.

The SAPC_orchestrator, located in the bin/ directory of the VDP, is the tool which removes VMs in ECM. To execute it, the following arguments are available:

Note:

For the execution of the script from the bin directory of the VDP, only the presence of the configuration file SAPC.cfg and the --option argument with the operation to perform is needed.

--help or -h: if this argument is present, an explanation for every argument is shown and the execution finishes.
--version: if this argument is present, the version is shown and the execution finishes.
--option: mandatory argument. It indicates the operation to perform. Possible values are: onboarding, instantiation, scaleout, scalein, termination.
--cfg <CFG> or -c <CFG>: mandatory argument. It indicates where the SAPC.cfg configuration file is. The default SAPC.cfg configuration file is placed in the root directory of the Virtual Delivery Package explained in SAPC VNF Descriptor Generator Tool.
--debug: optional argument for debugging and troubleshooting purposes.
--log <LOG>: optional argument to log the messages to a file instead of showing them into the standard output.

Then, an example of the SAPC_orchestrator execution is:

<PreparationServer># SAPC_orchestrator --option scalein --cfg SAPC.cfg

Once the ECM has received the order from the previous script, the Virtual Machine is removed from the SAPC Virtual Application.
Repeat the procedure as many times as the number of PLs to be deleted.

5 VM Deletion using Atlas/Director

Steps

Connect by SSH to the machine running Atlas/Director using the provided credentials.
Execute the following command to obtain the current value of the parameter pls.
heat stack-show <stack_name> | grep -i '"pls": "'

Note: For all the commands to be executed, it is needed to set up the following environment variables:
- OS_AUTH_URL points to Keystone service.
- OS_CACERT for the verification of the server certificate.
- OS_TENANT_NAME contains the name of the tenant where the stack was created.
- OS_USERNAME: user with delete rights.
- OS_PASSWORD: password for OS_USERNAME.
Execute the following command to remove the latest <PL-N> PL from the desired stack. The value of N is the number of pls obtained in previous command:
heat stack-update -f <stack_yaml_file> -e <stack_params_file> -r <stack_name> -P 'pls=<N-1>'

Note: See SAPC VNF Deployment Instruction for OpenStack for description of other parameters.
Run the command "heat stack-list"; the status for the stack changes to UPDATE_IN_PROGRESS, until operation is finished and the status changes to UPDATE_COMPLETED.
Repeat the procedure as many times as the number of PLs to be deleted.

6 VM Deletion using VMware vSphere Web Client

Steps

Select the PL VM to be deleted from the SAPC Virtual Application, click the PL-X Virtual Machine, and select Delete from Disk item.
Once the vCenter has received the order, the Virtual Machine is removed from the SAPC Virtual Application.
Repeat the procedure as many times as the number of PLs to be deleted.

7 VM Deletion using VMware vCloud Director

Steps

Select the PL VM to be deleted from the SAPC Virtual Application. It is shown in Partially Powered Off status. Click the PL-X VM, and select Power Off.
Once the PL VM status changes to Powered Off, click the PL-X VM, and select Delete.
A window pop-up is displayed. Confirm the PL VM deletion by clicking Yes.
Once the vCloud Director has received the order, the Virtual Machine is removed from the SAPC Virtual Application.
Repeat the procedure as many times as the number of PLs to be deleted.

Copyright	© Ericsson AB 2018. All rights reserved. No part of this document may be reproduced in any form without the written permission of the copyright owner.
Disclaimer	The contents of this document are subject to revision without notice due to continued progress in methodology, design and manufacturing. Ericsson shall have no liability for any error or damage of any kind resulting from the use of this document.
Trademarks	All trademarks mentioned herein are the property of their respective owners. These are shown in the document Trademark Information.