Contents

1   Introduction

This document is part of the installation flow for the Cloud Execution Environment (CEE) multi-server deployment and describes how to install CEE software in a CEE region. Complete this procedure when directed here from CEE Installation:

1. Start the procedure in CEE Installation.
2. Continue with this document when directed here from CEE Installation.
3. Return to CEE Installation and carry out the remaining steps.

For the complete installation flow, refer to section Installation Flow in CEE Installation.

This instruction assumes that a kickstart server is used. For the installation and testing of the kickstart server, refer to Preparation of Kickstart Server.

1.1   Prerequisites

This section describes the prerequisites that must be fulfilled before CEE software can be installed.

1.1.1   Documents

Activities in the following documents must be performed before the steps in this instruction are performed:

• Relevant sections of CEE Installation
• Preparation of Kickstart Server
• Configuration File Guide

1.1.2   Hardware and Software Required

The required software can be downloaded from the SW Gateway. If you have problems with the download procedure, contact the next level of support.

The following software is always required:

• CEE software release tarball, for more information refer to the Configuration File Guide.

The recommended installation method described in this document is using a kickstart server with Linux OS. For more information, refer to Preparation of Kickstart Server.

1.1.3   Tools

The following hardware tools are required:

• Kickstart server, refer to Preparation of Kickstart Server
• In case of BSP hardware: two Ethernet cables (DensiShield-RJ45, TSR 491 603/x) to connect the kickstart server to the system to be installed. In case of HP hardware: one RJ45-RJ45 cat6 Ethernet cable to connect the kickstart server to the system to be installed.

1.1.4   Installation Data

The following data is needed:

1.2   Time Required

The expected execution time for the installation procedure is around four hours, in case all prerequisites are available and depending on the hardware used.

2   Temporary pre-Installation Steps

This section describes limitations and workarounds to be performed before the installation.

2.1   Auto-Generated Password Not Accepted When LDAP Server Settings Are Configured for VNX

Note:  

This workaround is applicable for HP and Dell hardware platforms.

Ansible playbook eri-emc-vnx can fail at the LDAP configuration of EMC VNX part if the config.yaml file is specified with the field anonymous_bindpwd empty. In this case a password is generated and if the length is a multiple of 16, the password is rejected by VNX.

Associated TR: HV12958.

Workaround: Before installation, manually specify a password in the anonymous_bindpwd field in the ldap section of the config.yaml file with a length, which is not a multiple of 16. For more information on the credentials of the ldap users, refer to Configuration File Guide and Security User Guide.

2.2   CEE Deployment Fails If No DNS Server and Gateway IP Is Defined on fuel_ctrl_sp (Pre-Installation)

Note:  

This workaround is applicable for HP C7000 BL 460c Gen8/Gen9, Dell R620/R630, BSP, single server (Dell R630), and Ericsson Hyperscale Datacenter System (HDS) hardware platforms.

If the DNS server IP is set to a dummy IP or a server IP that does not provide actual DNS service (for example, Internet accessibility), Fuel takes an extremely long time to reach ready state. This time exceeds 2 hours, causing installation through installcee.sh to hang.

Associated trouble report: HV22423

Workaround: Perform the below procedure.

1. Use dummy IP for DNS server in config.yaml on Fuel. For example: 100.100.100.100
2. Remove the following files:
   • /etc/yum.repos.d/mos.update
   • /etc/yum.repos.d/mos.security
3. Continue with CEE installation.

After the installation, remove the dummy IP by following the instructions in Section 4.1.

2.3   Installation Fails When Non-Integrated NICs Are Used for Control Network

Note:  

This workaround is applicable for Dell R620/R630 hardware platform.

CEE installation does not work with non-integrated NICs, as racadm getsysinfo is used to fetch the MAC address for the control network and this command does not work when non-integrated NICs are used.

Associated trouble report: HV29997

Workaround: To fetch the MAC address for the control network for non-integrated NICs, perform the below steps.

1. Open macfetcher.py.
2. Add a new method, get_non_integrated_mac:

3. Replace the find_mac_dell method with the following:

Note:  

There must be two PXE devices placed consecutively.

3   Install CEE Software in Server System

This section describes how to install CEE in the server system.

1. Log on to the kickstart server.
2. In case of using BSP hardware, connect to BSP from the kickstart server and check the BSP software release and BSP backups.

   COM CLI interface is available via E-DBG Ethernet interface using the IP address defined as lct_ip in config.yaml.

   Example:

   ssh advanced@10.0.10.2 -p 2024
   (password for advanced user)
   show ManagedElement=1,SystemFunctions=1,SwInventory=1,active
   show ManagedElement=1,SystemFunctions=1,BrM=1,⇒
   BrmBackupManager=1,BrmBackup=1
   exit

   The value of SWVERSION cannot be lower than the software version indicated in CEE on BSP, Reference [1]. The name of backup (backupname) must contain the suffix specified in cmx_switch.yaml so backup name is suggested to be <cee_region_name>_YYYYMMDD_preCEETenant. Mismatch between the suffix in the backup name and in cmx_switch.yaml causes the CEE installation script to exit with error.

3. Check that vFuel is running in the kickstart server:

   virsh list --all

   Examples:

      
   root@fuelhost:~# virsh list --all
   
    Id    Name                           State
   
   ----------------------------------------------------
   
    2     fuel_master                     running
   

   root@fuelhost:~# virsh list --all
   
    Id    Name                           State
   
   ----------------------------------------------------
   
    -     fuel_master               shut off

   In case vFuel is in shut off state, start vFuel and wait until booting is complete:

   virsh start fuel_master

4. Log on to vFuel using SSH. For changing the password, refer to the System Hardening Guideline.

   ssh root@<Fuel IP address in network fuel_ctrl_sp>

5. Verify that the correct timezone, time and date have been set by using the below command:

   date

6. Change the working directory to /opt/ecs-fuel-utils with following command:

   cd /opt/ecs-fuel-utils

7. To ensure that the installation process is not interrupted, you may use a Screen session:

   # screen -S installcee -L

   If the connection to vFuel is lost, log on to vFuel again and reattach the screen session with the below command:

   # screen -r installcee

   Note:  

   The nohup option can cause installation failure and must not be used.

   
   
8. Execute the following:

   ./installcee.sh

   The time required for command execution is approximately four hours.

   Check that the printout is the following:

   Ericsson CEE installed successfully

9. In R6 on BSP hardware the Jumbo frame feature is not supported on the traffic switching domain.

   Note:  

   This issue affects BSP systems.

   
   

   Perform the below procedure:

   1. Connect to the BSP management interface:

      ssh advanced@<scx_mgmt_ip> -p<BSP8100_mgmt_port>

      For example, from vFuel running inside the blade system:

      ssh advanced@192.168.2.2 -p2024

      For example, from vFuel running outside the blade system:

      ssh advanced@10.0.10.2 -p2024

   2. Issue the below commands:

      ManagedElement=1,DmxcFunction=1,Trm=1,VirtualGateway=CEE
      configure
      mtu=2140
      commit
      exit

4   Temporary post-Installation Steps

This section describes limitations and workarounds to be performed after the installation.

4.1   Fix DNS Resolution (Post-Installation)

Note:  

This workaround is applicable for HP C7000 BL 460c Gen8/Gen9, Dell R620/R630, BSP, single server (Dell R630), and HDS hardware platforms.

If the DNS server IP is set to a dummy IP or a server IP that does not provide actual DNS service (Internet accessibility), Fuel takes an extremely long time to be in ready state. This time exceeds 2 hours, causing installation through installcee.sh to hang.

Associated trouble report: HV22423

Workaround: After the installation, remove the dummy IP from all the vCICs and Fuel from /etc/resolv.conf.

4.2   cinder-volume Service Is down after Deployment

Note:  

This workaround is applicable for Dell R620/R630 and HP C7000 BL 460c Gen8/Gen9 hardware platforms.

After the deployment of a system with EMC VNX or EMC ScaleIO (in an automatic way), it is possible that the cinder-volume service does not have connection to MySQL. As a result, no volume operations are possible and the service appears as DOWN in the output of the cinder service-list command.

Associated trouble report: HV36144

Workaround: Restart the cinder-volume service on one of the CICs by giving the following command:

crm resource restart p_cinder-volume

4.3   Volumes Created for "Swift on VNX" or "Swift on ScaleIO" Are in Error Status after Deployment

Note:  

This workaround is applicable for Dell R620/R630 and HP C7000 BL 460c Gen8/Gen9 hardware platforms.

As a result of the limitation cinder-volume Service Is down after Deployment in Section 4.2, the volumes for Swift on VNX or Swift on ScaleIO cannot be created and end up in Error status. This results in installation failure if the activation of feature Swift on VNX or Swift on ScaleIO is set to automatic in the configuration file.

Associated trouble report: HV36144

Workaround: After the cinder-volume service has been successfully restarted on one CIC, remove the volumes which are in Error status by following these steps:

1. Set the activation of feature Swift on Backend Storage to manual in the configuration file.
2. Activate the feature manually after deployment according to the instructions in Swift Store on VNX Activation or Swift Store on ScaleIO Activation.

   An example is:

       swift:
         swift_on_backend_storage:
           type: centralized
           activation_mode: manual
           lun_size: 0GiB

4.4   Core Dumps Are Created for a Process during Deployment

Note:  

This workaround is applicable for HP C7000 BL 460c Gen8/Gen9, Dell R620/R630, Blade Server Platform (BSP), single server (Dell R630), and HDS hardware platforms.

Due to an internal software error, certain processes create core dumps during deployment. The affected processes are rsyslog, autolog, ovsdb-client, and multipathd.

Associated trouble reports: HV42182, HV36121

Workaround: The core dumps can be removed after installation. To remove the core dumps, follow these steps:

1. Try to access the node with the core dump using SSH:

   ssh <user_id>@<name_of_the_node IP address>

   The name of the node can be found in the alarm. If the personal user ID does not work, use the ceeadm user ID.

2. If the node can be accessed, proceed to Step 4 and carry out all subsequent steps.
3. If the node cannot be accessed, follow the instructions of the alarm OPI Compute Host Failed or CIC Failed.
4. Issue the following command on the node:

   rm <path_to_the_core_dump>

   The full path to the core dump file can be found in the alarm.

4.5   Configure CPU Cores Reserved for OVS PMD Threads

Note:  

This workaround is applicable for all hardware platforms.

The idle CPU or CPUs allocated for OVS PMD threads need to be added to the argument list in the configure-interrupts upstart task to prevent the CPUs from serving interrupt requests. For more information on the limitation, refer to the documents Configuration File Guide and the relevant System Dimensioning Guide.

Associated trouble report: HV45059

Do the following on all Compute hosts:

1. Add the idle CPUs to the noirqs argument list in /etc/default/configure-interrupts.conf

   Note:  

   The idle CPU or CPUs are allocated to the owner: idle under reservedCPUs, in the config.yaml; refer to the Configuration File Guide.

   
   

   The following is an example of the edited file, with the added CPUs marked in bold:

   root@compute-0-4:~# cat /etc/default/configure-interrupts
   # Ansible managed: /usr/share/ericsson-orchestration/playbooks/roles/infra_eri_environment/templates/configure-interrupts.j2 modified on 2016-11-22 22:47:33 by root on fuel.domain.tld
   ENABLED="yes"
   ARGS="--noirqs=2,23,26,47"
   

   root@compute-0-4:~# cat /etc/default/configure-interrupts
   # Ansible managed: /usr/share/ericsson-orchestration/playbooks/⇒
   roles/infra_eri_environment/templates/configure-interrupts.j2 ⇒
   modified on 2016-11-22 22:47:33 by root on fuel.domain.tld
   ENABLED="yes"
   ARGS="--noirqs=2,23,26,47"
   

2. Re-run the configure-interrupts upstart job by executing the following command:
   start configure-interrupts

   The following is an example of the command and the expected output when the task is finished:

   root@compute-0-4:~# start configure-interrupts
   configure-interrupts stop/waiting
   

   Note:  

   The expected output status is stop/waiting, as this is a one-time task.

   
   

5   Migrate vFuel to CEE Region

This section describes how to migrate vFuel to a CEE region.

5.1   Migrate vFuel in Linux OS

To migrate vFuel in a Linux OS, perform the below steps:

1. Log on to the kickstart server.
2. Execute the below script:

   CEE_RELEASE/scripts/migrate_fuel.sh

   An example of the output is:

./migrate_fuel.sh

migrate_fuel.sh.info: Checking current Fuel state

migrate_fuel.sh.info: Preparing to migrate Fuel

migrate_fuel.sh.info: Fuel will be migrated to compute-0-4 (192.168.0.23)

migrate_fuel.sh.info: The vFuel image and the Domin XML will also be prepared ⇒
on compute-0-6 (192.168.0.25)

migrate_fuel.sh.info: Shutting down current Fuel

migrate_fuel.sh.info: Waiting for Fuel to complete shutdown

migrate_fuel.sh.info: Copying Fuel disk image to compute-0-4 (172.30.160.1)

sending incremental file list

fuel_br3160.qcow2

 68,730,224,640 100%  107.69MB/s    0:10:08 (xfr#1, to-chk=0/1)

migrate_fuel.sh.info: Copying Fuel disk image to compute-0-6 (172.30.160.2)

sending incremental file list

fuel_br3160.qcow2

 68,730,224,640 100%  103.17MB/s    0:10:35 (xfr#1, to-chk=0/1)

migrate_fuel.sh.info: Starting new vFuel inside CEE region on compute-0-4 (192.168.0.23)

migrate_fuel.sh.info: Waiting for new vFuel to start up

migrate_fuel.sh.info: Waiting for new vFuel to be ready

migrate_fuel.sh.info: New vFuel ready

migrate_fuel.sh.info: Performing post migrate actions

migrate_fuel.sh.info: Post migrate actions done

migrate_fuel.sh.info: Fuel is successfully migrated to compute-0-4 (192.168.0.23)

6   Post-Installation Activities

Execute the following steps after the installation:

1. Disconnect the kickstart server.
2. Verify the version of CEE by executing the command cat /etc/cee_version.txt on the Fuel master node.

   The output has the following format:

   RELEASE=CEE CXC1737883_4-<build_number>
   NAME=Mitaka on Ubuntu 14.04
   VERSION=R6-<R-state>-<specific_build_number>-9.0

   An example of the output is the following:

   [root@fuel ~]# cat /etc/cee_version.txt RELEASE=CEE CXC1737883_4-4280
   NAME=Mitaka on Ubuntu 14.04
   VERSION=R6-R4A02-35547a3-9.0
   
   [root@fuel ~]#

   Verify the CEE version by comparing build_number and R-state to Product Revision Information.

3. Continue with the relevant sections of CEE Installation.

7   Error Handling

In case of any errors during the installation procedure, follow the below steps:

1. Check the console for failure messages or reference to any logs that possibly contain failure messages. Refer to the Configuration File Guide for the location of logs.
2. Fix possible problems.
3. Copy the original network templates to the /mnt/cee_config directory.

   Note:  

   If this step is missed, VLANs and interfaces from the previous run will be used, which causes the newer configuration options to be skipped.

   
   

   On the vFuel node issue the following command:

   cp CEE_RELEASE/host_net_templates/host_nw_*.yaml /mnt/cee_config/

4. Rerun installcee.sh and collect logs:

   ./installcee.sh 2>&1 | tee <file_name>.log

   Note:  

   The installcee command does not automatically delete an existing CEE Region (Fuel environment), so installation attempts with an existing Fuel environment will fail. In this case reinstall CEE with the below command:

   /installcee.sh --force

   
   
5. The following scenarios are possible:
   • The cause of failure is identified, fixed, or the install succeeds.

     In this case, exit this procedure.

   • Or the cause of failure is not identified, fixed, or the install still fails for presumably the same reason.

     In this case, proceed to Step 6.

6. Perform data collection according to the Data Collection Guideline.
7. Contact the next level of support.

Appendix

8   CA and NBI Certificates for Secure HTTPS Access

Certification Authority (CA) and Northbound Interface (NBI) certificates are required for secure HTTPS access to CEE.

Make sure to perform the following tasks before starting the installation process:

1. Choose a unique hostname for the vCIC NBI.
2. Choose a unique hostname for the Atlas NBI.
3. Obtain certificates for the NBIs from an authorized Certification Authority (CA).

   The following certificate files are needed:

   • CA certificate (or chain of certificates) of the organization issuing the Atlas NBI
   • CA certificate (or chain of certificates) of the organization issuing the vCIC NBI
   • Atlas NBI certificate
   • vCIC NBI certificate

   Note:  

   Atlas and vCIC certificates can be issued by the same CA, or by two separate CAs.

   
   

   The Common Name (CN) and at least one DNS entry in the Subject Alternate Name (SAN) attribute must contain the publicly known hostname chosen for the NBI, so that the certificate refers to this publicly known hostname. The private key belonging to the certificate cannot be encrypted.

4. Concatenate the vCIC NBI certificate and private key into a single PEM format under /mnt/cee_config on vFuel. Perform the same for the Atlas NBI.

   ASCII format is preferred for the individual certificates.

   Note:  

   The pkcs12 binary format is commonly used. This output format contains multiple entities in a single binary file and uses encryption. Issue the below command to convert it to PEM format:

   openssl pkcs12 -in <inputfile> -out <outputfile> -nodes

   -nodes is needed to save the private key in unencrypted format, as encrypted private keys are not supported.

   In case other binary formats need to be converted, refer to Reference [2] or Reference [3].

   
   
5. Update the config.yaml file with the necessary information. Refer to the Configuration File Guide for updating the publicly known hostname and other relevant options in the config.yaml file.
6. Update the DNS resolver to contain the hostname and IP address pairs for the NBI.

Reference List