Blade Replacement Instruction for Native Deployment
Ericsson Dynamic Activation 1

Contents

1Introduction
1.1Purpose and Scope
1.2Target Group
1.3Typographic Conventions

2
Prerequisites
2.1Tools
2.2GEP Blade Requirement

3
Blade Replacement
3.1GEP5 Blade Replacement
3.2GEP3 Blade Replacement

Reference List

1   Introduction

This document provides instructions for replacing hardware (HW) parts, Payload (PL) or Control (SC) blades, on an Ericsson™ Dynamic Activation (EDA) system, using GEP3 or GEP5 blades.

1.1   Purpose and Scope

The scope of this document is to provide instructions for how to replace and configure a PL or SC blade in a Dynamic Activation cabinet.

1.2   Target Group

The target group for this document is as follows:

The target groups are described in more detail in the Library Overview, Reference [2], document.

1.3   Typographic Conventions

Typographic conventions are described in the document Library Overview, Reference [2].

For information about abbreviations used throughout this document refer to Glossary of Terms and Acronyms, Reference [1].

2   Prerequisites

Replacement of a Dynamic Activation GEP3 or GEP5 blade system requires that the user of this document has:

2.1   Tools

The following tools are required:

2.2   GEP Blade Requirement

To get the cluster to fully work, before replacing the faulty SC blade, a new license needs to be ordered.

Caution!

The instructions throughout this document require that the GEP blades that will be used for replacing the old ones is a new GEP blade. That is, not previously used in a cluster setup.

Make sure that the new blade has the same ROJ number as the one that is going to be replaced.

3   Blade Replacement

This section contains information on how to replace a Dynamic Activation PL or SC blade using either a GEP5 or GEP3 blades. The procedures in this section can be performed in runtime, for both PL and SC blades.

Note:  
When a node has been down for more than three hours, the Cassandra consistency will be impacted.

To avoid loosing log consolidation data during blade replacement, if the disk size is more than 50% of the available disk space in the /var/cassandra/data directory, follow the instructions in section Exporting Logs in System Administrators Guide for Native Deployment, Reference [3] to export the oldest days stored, until the disk size is decreased to below 50%.


The following chart depicts the replacement flow for both PL and SC blades, see Figure 1.

Figure 1   Blade Replacement Flow Chart

3.1   GEP5 Blade Replacement

This section contains information on how to replace SC and PL blades (GEP5).

Attention!

All commands throughout these sections are run as user root.

3.1.1   PL Blade Replacement

This section contains information on how to replace a PL blade.

3.1.1.1   Blade Replacement

Perform a controlled shutdown on the faulty PL blade and manually replace it according to instructions in the step-list below:

  1. Log on to an SC node, and if possible, stop all applications, Cassandra and Zookeeper that are running on the faulty PL blade.

    Stop all applications:

    # bootloader.py node stop --host <hostname>

    Stop Cassandra:

    # 3ppmon stopcassandra --host <hostname>

    Check and stop ZooKeeper if it is running:

    # 3ppmon status --host <hostname>

    # 3ppmon stopzookeeper --host <hostname>

    Note:  
    <hostname> is the host name of the PL node that is to be stopped.

  2. Remove the faulty PL node from the Cassandra ring.
    1. Check the Cassandra nodes status.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
DN  169.254.100.4  76.77 GB   256     ?     5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2  74.9 GB    256     ?     de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256     ?     f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.1  74.44 GB   256     ?     004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

    2. Find the Cassandra node whose status is down (DN), and copy its Host ID.

      This Cassandra node is running on the faulty blade.

    3. Remove the node from the Cassandra ring.

      # nodetool removenode <Host ID>

      In this example:

      # nodetool removenode 5100b6ba-a90f-4ca2-ac40-eef26915a56b

    4. After prompt is returned, check whether the removal command is finished.

      # nodetool removenode status
      RemovalStatus: No token removals in process.

    5. Check whether the node has been removed.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  169.254.100.2  74.9 GB    256     ?     de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256     ?     f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.1  74.44 GB   256     ?     004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Turn off the power for the GEP blade to be replaced.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, PL4 needs to be replaced which in this case corresponds to slot position 7, circle marked, see Figure 2.

Figure 2   PL GEP Blade Replacement

  1. Remove the faulty blade and remember the positions of the cables when replacing the blade.
    Caution!

    Always wear an Electrostatic Discharge (ESD) strap when touching the hardware in the rack. Sensitive components, such as Integrated Circuits (ICs), can be damaged by discharges of static electricity.

    Note:  
    Use the screwdriver of type TORX T5, or torque screwdriver with TORX T5 bit for the cables.

    Use the screwdriver of type TORX T8, or torque screwdriver with TORX T8 bit for the blade.


  2. Insert the new blade and connect the cables in the same positions as before.
  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Prepare for a serial connection towards the new PL blade:

    # telnet <Console Server IP-address> <port number connected to PL Blade>

  6. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, PL4 has been replaced, which in this case corresponds to slot position 7, circle marked, see Figure 3.

    Note:  
    Do not boot up the blade without a Console Server connected. This in order to monitor the process, change BIOS settings and more.

Figure 3   New PL GEP Blade

  1. Disable the Power Technology in the GEP BIOS.

    During the BIOS startup sequence, wait for the following console printout:

    Press <DEL> or <F4> to enter set up.

    Press F4

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  2. In the Main window, use the arrow keys to navigate to Advanced > CPU Configuration, tab down to CPU Power Management Configuration >, and tab down to the Power Technology menu, press Enter, choose Disable and press Enter again.
  3. To save and exit:

    Press F4

    Use the arrow keys to choose Yes and press:

    Enter

  4. Turn off the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP Blade slot position. As an example, PL4 has been replaced, which in this case corresponds to slot position 7, circle marked, see Figure 3.

3.1.1.2   Modify LDEwS Configuration

Get the MAC address of the new PL GEP blade.

The MAC address is used as input to the cluster.conf file used by LDEwS. The MAC address is fetched through the DMXC Management System CLI.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Enter the command to show the MAC addresses.

    > ManagedElement=1,DmxcFunction=1,Eqm=1,VirtualEquipment=pg

    > show-table -m Blade -p bladeId,firstMacAddr

  4. A printout shows all MAC addresses of all blades in the Applications/PG group. Figure 4 is just an example.

Figure 4   Example of MAC Address Printout

  1. Log in to SC1 and edit the cluster.conf file. Replace all the MAC addresses from the faulty blade to the new blade:

    # vi /cluster/etc/cluster.conf

  2. Validate and reload the cluster configuration:

    # cluster config --validate

    # cluster config --reload --all

3.1.1.3   BIOS Settings, eVIP and LDEwS Installation

Install LDEwS by booting from network according to instructions in this section.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Prepare for a serial connection towards the still powered off PL blade:

    # telnet <Console Server IP-address> <port number connected to PL Blade>

  4. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP Blade slot position. As an example, PL4 has been replaced, which in this case corresponds to slot position 7, circle marked, see Figure 3.

  5. During the BIOS startup sequence, wait for the console printout Press F3 for GEP PopUp and then press:

    F3 to login to PBIST mode.

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  6. Choose the option 40 - UEFI Shell (PBIST) by entering the value:

    > 40

    In the next screen shown, press any key within a few seconds to proceed to the UEFI shell.

  7. Clear the present boot device order with command:

    > ipmi bo erase

  8. Check the result by command:

    > ipmi bo display

    The following printout should be displayed:

    BootList is empty

  9. Set boot device order to Backplane left device by command:

    > ipmi bo insert 1 00

    (1=priority, 00=Backplane left device).

  10. Set boot device order to Backplane right device by command:

    > ipmi bo insert 2 01

    (2=priority 2, 01=Backplane right device).

  11. Check the result by command:

    > ipmi bo display

    Should display the newly added boot device settings.

  12. Reset the blade:

    > pbist -r

    The blade is booted up and installed through the network automatically.

    After LDEwS has been booted, LDEwS, eVIP, and the application RPMs are loaded.

    Wait until the blade has booted. That is, when the prompt reappears in the Console Server.

    Note:  
    The time duration to complete the boot is approximately 10 minutes.

    When the blade has booted up, log in to the replaced node.

  1. Perform the following procedure to manually create partitions for the GEP5 PL node.
    1. Identify which disks to be raided by using the following four commands. An example output is given under each command.

      # ls -la /dev/disk/by-path/pci-0000:03:00.0-sas-phy0-0x4433221100000000-lun-0
      lrwxrwxrwx 1 root root 9 Jan 24 15:37 /dev/disk/by-path/pci-0000:03:00.0-sas-phy0-0x4433221100000000-lun-0 -> ../../sda

      # ls -la /dev/disk/by-path/pci-0000:03:00.0-sas-phy1-0x4433221101000000-lun-0
      lrwxrwxrwx 1 root root 9 Jan 24 15:37 /dev/disk/by-path/pci-0000:03:00.0-sas-phy1-0x4433221101000000-lun-0 -> ../../sdb

      # ls -la /dev/disk/by-path/pci-0000:03:00.0-sas-phy2-0x4433221102000000-lun-0
      lrwxrwxrwx 1 root root 9 Jan 24 15:37 /dev/disk/by-path/pci-0000:03:00.0-sas-phy2-0x4433221102000000-lun-0 -> ../../sdc

      # ls -la /dev/disk/by-path/pci-0000:03:00.0-sas-phy3-0x4433221103000000-lun-0
      ls: cannot access /dev/disk/by-path/pci-0000:03:00.0-sas-phy3-0x4433221103000000-lun-0: No such file or directory

      Note:  
      The four commands always output three disk identities and a No such file or directory notification.

      In this example, the three disks are identified as sda, sdb, and sdc.


    2. Create a md0 raid based on the three identified disks. For example:

      # mdadm --create --run /dev/md0 --chunk=4 --level=0 --raid-devices=3 /dev/sda /dev/sdb /dev/sdc

    3. Define partitions table name and sizes.

      # parted /dev/md0 mklabel gpt

      # parted -s /dev/md0 mkpart primary ext3 1GB 10.7GB

      # parted -s /dev/md0 mkpart primary ext3 10.7GB 21.4GB

      # parted -s /dev/md0 mkpart primary ext3 21.4GB 1095.7GB

    4. Create partition labels.

      # mkfs.ext3 -L DVE_LOGS /dev/md0p1

      # mkfs.ext3 -L CAS_COMLOG /dev/md0p2

      # mkfs.ext3 -L CAS_DATA /dev/md0p3

    5. From the replaced node, reboot the node to install Cassandra.

      # reboot

      When the node is up again, check that all disk partitions are mounted correctly.

      # df -h

      ..
/dev/md0p1      8.8G   49M  8.3G   1% /var/dve
/dev/md0p2      9.7G  577M  8.7G   7% /var/cassandra/commitlog
/dev/md0p3      985G  176M  935G   1% /var/cassandra/data

  2. Run the following command to make sure Cassandra has joined the ring:

    # nodetool status

    Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens    Owns    Host ID                               Rack
UJ  169.254.100.4  254.97 MB  256       ?       58ecf956-1196-4875-8f31-fcd51d4d61ff  rack1
UN  169.254.100.2  74.9 GB    256       ?       de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256       ?       f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.1  74.44 GB   256       ?       004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

    The replaced PL node status is UJ.

  3. Wait until the Cassandra status on the replaced node has changed to UN.
  4. Check the deployment status of all 3PP processes in all nodes.

    # 3ppmon status --host all

  5. From an SC node, run the following command to start all the modules on the replaced PL node:

    # bootloader.py node start --host <hostname of replaced blade>

  6. Check the the status of Dynamic Activation application.

    # bootloader.py node status --host all

  7. On all nodes in the cluster, one by one, run the following command:

    # nodetool cleanup

3.1.2   SC Blade Replacement

This section contains information on how to replace an SC blade.

3.1.2.1   Blade Replacement

Perform a controlled shutdown on the faulty SC blade and manually replace it according to the instructions in the step-list below:

  1. Log in to a working SC node, and if possible, stop all applications, Cassandra and Zookeeper that are running on the faulty SC node.

    Stop all applications:

    # bootloader.py node stop --host <hostname>

    Stop Cassandra:
    # 3ppmon stopcassandra --host <hostname>

    Check and stop ZooKeeper if it is running:
    # 3ppmon status --host <hostname>
    # 3ppmon stopzookeeper --host <hostname>

    Note:  
    <hostname> is the host name of the SC node that is to be stopped.

  2. Remove the faulty SC node from the Cassandra ring.
    1. Check the Cassandra nodes status.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens    Owns   Host ID                               Rack
UN  169.254.100.4  76.77 GB   256       ?      5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2  74.9 GB    256       ?      de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256       ?      f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
DN  169.254.100.1  74.44 GB   256       ?      004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

    2. Find the Cassandra node whose status is down (DN), and copy its Host ID.

      This Cassandra node is running on the faulty blade.

    3. Remove the node from the Cassandra ring.

      # nodetool removenode <Host ID>

      In this example:

      # nodetool removenode 004eb495-da8e-47eb-b6d0-0750ed5f5a50

    4. After prompt is returned, check whether the removal command is finished.

      # nodetool removenode status
      RemovalStatus: No token removals in process.

    5. Check whether the node has been removed.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  169.254.100.4  76.77 GB   256       ?      5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.3  72.34 GB   256       ?      f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.2  74.44 GB   256       ?      de177bc5-3142-436d-b1c2-0bbf13410267  rack1

  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Turn off the power for the GEP Blade to be replaced.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, SC1 needs to be replaced which in this case corresponds to slot position 1, circle marked, see Figure 5.

Figure 5   SC GEP Blade Replacement

  1. Remove the faulty blade and remember the positions of the cables when replacing the blade.
    Caution!

    Always wear an Electrostatic Discharge (ESD) strap when touching the hardware in the rack. Sensitive components, such as Integrated Circuits (ICs), can be damaged by discharges of static electricity.

    Note:  
    Use the screwdriver of type TORX T5, or torque screwdriver with TORX T5 bit for the cables.

    Use the screwdriver of type TORX T8, or torque screwdriver with TORX T8 bit for the blade.


  2. Insert the new blade and insert the cables in the same positions as in the faulty blade.
  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Prepare for a serial connection towards the still powered off SC blade:

    # telnet <Console Server IP-address> <port number connected to SC Blade>

  6. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, SC1 has been replaced, which in this case corresponds to slot position 1, circle marked, see Figure 6.

    Note:  
    Do not boot up the blade without a Console Server connected. This in order to monitor the process, change BIOS settings and more.

Figure 6   New SC GEP Blade

  1. Disable the Power Technology in the GEP BIOS.

    During the BIOS startup sequence, wait for the following console printout:

    Press <DEL> or <F4> to enter set up.

    Press F4

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  2. In the Main window, use the arrow keys to navigate to Advanced > CPU Configuration, tab down to CPU Power Management Configuration >, and tab down to the Power Technology menu, press Enter, choose Disable and press Enter again.
  3. To save and exit:

    Press F4

    Use the arrow keys to choose Yes and press:

    Enter

  4. Turn off the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, SC1 has been replaced, which in this case corresponds to slot position 1, circle marked, see Figure 6.

3.1.2.2   Modify LDEwS Configuration

Get the MAC address of the new SC GEP blade.

The MAC address is used as input to the cluster.conf file used by LDEwS. The MAC address is fetched through the DMXC Management System CLI.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Enter the command to show the MAC addresses.

    > ManagedElement=1,DmxcFunction=1,Eqm=1,VirtualEquipment=pg

    > show-table -m Blade -p bladeId,firstMacAddr

  4. A printout shows all MAC addresses of all blades in the PG group. Figure 7 is just an example.

Figure 7   Example of MAC Address Printout

  1. Log in to SC2 and edit the cluster.conf file. Replace all the MAC addresses from the faulty blade to the new blade:

    # vi /cluster/etc/cluster.conf

  2. Validate and reload the cluster configuration:

    # cluster config --validate

    # cluster config --reload --all

3.1.2.3   LDEwS Installation from the Other SC Blade

Install LDEwS by booting from the other SC blade and prepare the disks.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Prepare for a serial connection towards the still powered off SC1 blade:

    # telnet <Console Server IP-address> <port number connected to SC1 Blade>

  4. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position.

  5. During the BIOS startup sequence, wait for the console printout Press F3 for GEP PopUp and then press:

    F3 to login to PBIST mode.

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  6. Choose the option 40 - UEFI Shell (PBIST) by entering the value:

    > 40

    In the next screen shown, press any key within a few seconds to proceed to the UEFI shell.

  7. Clear the present boot device order with command:

    > ipmi bo erase

  8. Check the result by command:

    > ipmi bo display

    The following printout should be displayed:

    BootList is empty

  9. Set boot device order to Backplane left device by command:

    > ipmi bo insert 1 00

    (1=priority, 00=Backplane left device).

  10. Check the result by command:

    > ipmi bo display

    Should display the newly added boot device settings.

  11. Reset the blade:

    > pbist -r

    Note:  
    Wait until LDEwS has been installed.

    Once LDEwS installation has finished, the GEP blade will automatically reboot.


  12. During the BIOS startup sequence wait for the console printout Press F3 for GEP PopUp and then press:

    F3 to login to PBIST mode.

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  13. Choose the option 40 - UEFI Shell (PBIST) by entering the value:

    > 40

    In the next screen shown, press any key within a few seconds to proceed to the UEFI shell.

  14. Clear the present boot device order with command:

    > ipmi bo erase

  15. Check the result by command:

    > ipmi bo display

    The following printout should be displayed:

    BootList is empty

  16. Edit the boot device setting to boot from Internal SAS Disk:

    > ipmi bo insert 1 10

    (1=priority, 10=SAS Disk device).

  17. Check the result by command:

    > ipmi bo display

  18. Reset the new SC blade:

    > pbist -r

  19. Run the following command to see the synchronization progress:

    # drbd-overview

  20. After the disk synchronization has finished, LDEwS, eVIP and the Application RPMs are loaded.

3.1.2.4   Licenses

Note:  
Make sure that the new license has been ordered.

Caution!

Do not reboot the other SC blade which is not replaced, before the new license is installed successfully.

To install the licenses for Dynamic Activation follow the instructions in section License Administration in System Administrators Guide for Native Deployment, Reference [3].

3.1.2.5   Dynamic Activation Software Status

  1. Log in to the replaced node. Perform the following procedure to start data transfer from other Cassandra nodes.
    1. In cassandra.yaml file, located in the /cluster/home/casadm/nodes/<nodeId>/cassandra/conf/ directory, remove the IP address of the replaced node from seeds parameter. For example: 
      #- seeds: "169.254.100.1,169.254.100.2"
 - seeds: "169.254.100.2"
    2. Stop Cassandra.

      # 3ppmon stopcassandra

    3. From other SC node, remove the replaced node from Cassandra ring.

      Check the Cassandra nodes status.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load     Tokens Owns Host ID                               Rack
UN  169.254.100.4 74.44 GB   256  ?    5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2 74.90 GB   256  ?    de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3 72.34 GB   256  ?    f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
DN  169.254.100.1 74.44 GB   256  ?    004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

      Find the replaced Cassandra node and copy its Host ID.

      Remove the replaced node from the Cassandra ring.

      # nodetool removenode <Host ID>

      In this example:

      # nodetool removenode 004eb495-da8e-47eb-b6d0-0750ed5f5a50

      After prompt is returned, check whether the removal command is finished.

      # nodetool removenode status

      Expected result: RemovalStatus: No token removals in process.

      Check whether the node has been removed.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load     Tokens Owns Host ID                               Rack
UN  169.254.100.4 74.44 GB   256  ?    5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.3 72.34 GB   256  ?    f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.2 74.44 GB   256  ?    de177bc5-3142-436d-b1c2-0bbf13410267  rack1

    4. On the replaced node, remove all data from commitlog and data directory.

      # cd /var/cassandra/commitlog
      # rm -rf *

      # cd /var/cassandra/data
      # rm -rf *

    5. Start Cassandra.

      # 3ppmon startcassandra

      When Cassandra is started, the replaced node is joining in to the ring.

  2. Run the following command to make sure Cassandra has joined the ring:

    # nodetool status

    Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens    Owns   Host ID                               Rack
UN  169.254.100.4  76.77 GB   256       ?      5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2  74.44 GB   256       ?      de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256       ?      f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UJ  169.254.100.1  254.9 MB   256       ?      7b71585e-778f-424b-9ed2-813ef5867b01  rack1

    The replaced SC node status is UJ.

  3. Wait until the Cassandra status on the replaced node has changed to UN.
  4. In cassandra.yaml file, located in the /cluster/home/casadm/nodes/<nodeId>/cassandra/conf/ directory, write back the previously removed IP addresses of the replaced node in seeds parameter. For example: 
    - seeds: "169.254.100.1,169.254.100.2"
#-seeds: "169.254.100.2"
  5. To check the deployment status of all 3PPs, in all nodes, run the following command:

    # 3ppmon status --host all

  6. Start the Dynamic Activation application on the new SC blade:

    # bootloader.py node start --host <hostname of replaced blade>

  7. Execute the following command to check the status of the Dynamic Activation application:

    # bootloader.py node status --host all

  8. On all nodes in the cluster, one by one, run the following command:

    # nodetool cleanup

3.2   GEP3 Blade Replacement

This section contains information on how to replace SC and PL blades (GEP3).

3.2.1   PL Blade Replacement

This section contains information on how to replace a PL blade.

Attention!

All commands throughout these sections are run as user root.

3.2.1.1   Blade Replacement

Perform a controlled shutdown on the faulty PL blade and manually replace it according to the instructions in the step-list below:

  1. Log in to an SC node, and if possible, stop all applications, Cassandra and Zookeeper that are running on the faulty PL blade.

    Stop all applications:

    # bootloader.py node stop --host <hostname>

    Stop Cassandra:
    # 3ppmon stopcassandra --host <hostname>

    Check and stop ZooKeeper if it is running:
    # 3ppmon status --host <hostname>
    # 3ppmon stopzookeeper --host <hostname>

    Note:  
    <hostname> is the host name of the PL node that is to be stopped.

  2. Remove the faulty PL node from the Cassandra ring.
    1. Check the Cassandra nodes status.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
DN  169.254.100.4  76.77 GB   256     ?     5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2  74.9 GB    256     ?     de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256     ?     f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.1  74.44 GB   256     ?     004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

    2. Find the Cassandra node whose status is down (DN), and copy its Host ID.

      This Cassandra node is running on the faulty blade.

    3. Remove the node from the Cassandra ring.

      # nodetool removenode <Host ID>

      In this example:

      # nodetool removenode 5100b6ba-a90f-4ca2-ac40-eef26915a56b

    4. After prompt is returned, check whether the removal command is finished.

      # nodetool removenode status
      RemovalStatus: No token removals in process.

    5. Check whether the node has been removed.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  169.254.100.2  74.9 GB    256     ?     de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256     ?     f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.1  74.44 GB   256     ?     004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Turn off the power for the GEP blade to be replaced.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, PL4 needs to be replaced which in this case corresponds to slot position 7, circle marked, see Figure 8.

Figure 8   PL GEP Blade Replacement

  1. Remove the faulty blade and remember the positions of the cables when replacing the blade.
    Caution!

    Always wear an Electrostatic Discharge (ESD) strap when touching the hardware in the rack. Sensitive components, such as Integrated Circuits (ICs), can be damaged by discharges of static electricity.

    Note:  
    Use the screwdriver of type TORX T5, or torque screwdriver with TORX T5 bit for the cables.

    Use the screwdriver of type TORX T8, or torque screwdriver with TORX T8 bit for the blade.


  2. Insert the new blade and connect the cables in the same positions as before.
  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Prepare for a serial connection towards the new PL blade:

    # telnet <Console Server IP-address> <port number connected to PL Blade>

  6. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, PL4 has been replaced, which in this case corresponds to slot position 7, circle marked, see Figure 9.

    Note:  
    Do not boot up the blade without a Console Server connected. This in order to monitor the process, change BIOS settings and more.

Figure 9   New PL GEP Blade

  1. Disable the Power Technology in the GEP BIOS.

    During the BIOS startup sequence, wait for the following console printout:

    Press <DEL> or <F4> to enter set up.

    Press F4

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  2. In the Main window, use the arrow keys to navigate to Advanced, > tab down to CPU Power Management Configuration, and > tab down to the Power Technology menu, press Enter, choose Disable and press Enter again.
  3. To save and exit:

    Press F4

    Use the arrow keys to choose Yes and press:

    Enter

  4. Turn off the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP Blade slot position. As an example, PL4 has been replaced, which in this case corresponds to slot position 7, circle marked, see Figure 9.

3.2.1.2   Modify LDEwS Configuration

Get the MAC address of the new PL GEP blade.

The MAC address is used as input to the cluster.conf file used by LDEwS. The MAC address is fetched through the DMXC Management System CLI.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Enter the command to show the MAC addresses.

    > ManagedElement=1,DmxcFunction=1,Eqm=1,VirtualEquipment=pg

    > show-table -m Blade -p bladeId,firstMacAddr

  4. The printout shows all MAC addresses of all blades in the PG group. Figure 10 presents the MAC addresses for each interface.

Figure 10   Example of MAC Address Printout

  1. Log in to SC1 and edit the cluster.conf file. Replace all the MAC addresses from the faulty blade to the new blade:

    # vi /cluster/etc/cluster.conf

  2. Validate and reload the cluster configuration:

    # cluster config --validate

    # cluster config --reload --all

3.2.1.3   BIOS Settings, eVIP and LDEwS Installation

Install LDEwS by booting from network according to instructions in this section.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Prepare for a serial connection towards the still powered off PL blade:

    # telnet <Console Server IP-address> <port number connected to PL Blade>

  4. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP Blade slot position. As an example, PL4 has been replaced, which in this case corresponds to slot position 7, circle marked, see Figure 9.

  5. During the BIOS startup sequence, wait for the console printout Press F3 for GEP PopUp and then press:

    F3 to login to PBIST mode.

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  6. Choose the option 40 - UEFI Shell (PBIST) by entering the value:

    > 40

    In the next screen shown, press any key within a few seconds to proceed to the UEFI shell.

  7. Clear the present boot device order with command:

    > ipmi -o erase

  8. Check the result by command:

    > ipmi -o display

    The following printout should be displayed:

    BootList is empty

  9. Set boot device order to Backplane left device by command:

    > ipmi -o insert 1 00

    (1=priority, 00=Backplane left device).

  10. Set boot device order to Backplane right device by command:

    > ipmi -o insert 2 01

    (2=priority 2, 01=Backplane right device).

  11. Check the result by command:

    > ipmi -o display

    Should display the newly added boot device settings.

  12. Reset the blade:

    > pbist -r

    LDEwS is booted through the network automatically.

    After LDEwS has been booted, eVIP will be installed but the Dynamic Activation application RPMs will fail to install due to lack of disk labels. This will be handled in the next coming steps.

Wait until the blade has booted. That is, when the prompt reappears in the Console Server.

Note:  
The time duration to complete the boot is approximately 10 minutes.

  1. Partition the disks and label them according to instructions in section Partitioning on PL Nodes in Hardware Installation and IP Infrastructure Setup for Native Deployment GEP3, Reference [4].
    Note:  
    Depending on the revision of the new GEP3 blade, the 300 GB disk can be either on the sda or sdb device.

  2. Perform a controlled reboot of the replaced blade:

    # cluster reboot -n <nodeId of replaced blade>

  3. Check the deployment status for all 3PPs in all nodes:

    Execute the following command on any node, to check the deployment status for all 3PPs in all nodes:

    # 3ppmon status --host all

  4. Run the following command to make sure Cassandra has joined the ring:

    # nodetool status

    Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens    Owns    Host ID                               Rack
UJ  169.254.100.4  254.97 MB  256       ?       58ecf956-1196-4875-8f31-fcd51d4d61ff  rack1
UN  169.254.100.2  74.9 GB    256       ?       de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256       ?       f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.1  74.44 GB   256       ?       004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

    The replaced PL node status is UJ.

  5. Wait until the Cassandra status on the replaced node has changed to UN.
  6. Check the deployment status of all 3PPs, in all nodes.

    # 3ppmon status --host all

  7. From an SC node, run the following command to start all modules on the replaced PL node:

    # bootloader.py node start --host <hostname of replaced blade>

  8. Check the status of the Dynamic Activation applications:

    # bootloader.py node status --host all

  9. On all nodes in the cluster, one by one, run the following command:

    # nodetool cleanup

3.2.2   SC Blade Replacement

This section contains information on how to replace an SC blade.

3.2.2.1   Blade Replacement

Perform a controlled shutdown on the faulty SC blade and manually replace it according to the instructions in the step-list below:

  1. Log on to a working SC node, and if possible, stop all applications, Cassandra and Zookeeper that are running on the faulty SC node.

    Stop all applications:

    # bootloader.py node stop --host <hostname>

    Stop Cassandra:
    # 3ppmon stopcassandra --host <hostname>

    Check and stop ZooKeeper if it is running:
    # 3ppmon status --host <hostname>
    # 3ppmon stopzookeeper --host <hostname>

    Note:  
    <hostname> is the host name of the SC node that is to be stopped.

  2. Remove the faulty SC node from the Cassandra ring.
    1. Check the Cassandra nodes status.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens    Owns   Host ID                               Rack
UN  169.254.100.4  76.77 GB   256       ?      5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2  74.9 GB    256       ?      de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256       ?      f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
DN  169.254.100.1  74.44 GB   256       ?      004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

    2. Find the Cassandra node whose status is down (DN), and copy its Host ID.

      This Cassandra node is running on the faulty blade.

    3. Remove the node from the Cassandra ring.

      # nodetool removenode <Host ID>

      In this example:

      # nodetool removenode 004eb495-da8e-47eb-b6d0-0750ed5f5a50

    4. After prompt is returned, check whether the removal command is finished.

      # nodetool removenode status
      RemovalStatus: No token removals in process.

    5. Check whether the node has been removed.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  169.254.100.4  76.77 GB   256       ?      5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.3  72.34 GB   256       ?      f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.2  74.44 GB   256       ?      de177bc5-3142-436d-b1c2-0bbf13410267  rack1

  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Turn off the power for the GEP Blade to be replaced.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, SC1 needs to be replaced which in this case corresponds to slot position 1, circle marked, see Figure 11.

Figure 11   SC GEP Blade Replacement

  1. Remove the faulty blade and remember the positions of the cables when replacing the blade.
    Caution!

    Always wear an Electrostatic Discharge (ESD) strap when touching the hardware in the rack. Sensitive components, such as Integrated Circuits (ICs), can be damaged by discharges of static electricity.

    Note:  
    Use the screwdriver of type TORX T5, or torque screwdriver with TORX T5 bit for the cables.

    Use the screwdriver of type TORX T8, or torque screwdriver with TORX T8 bit for the blade.


  2. Insert the new blade and insert the cables in the same positions as in the faulty blade.
  3. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  4. Enter password for user advanced
  5. Check the productRevisionState of the replaced GEP3 blade:

    > show ManagedElement=1,DmxcFunction=1,Eqm=1,VirtualEquipment=pg,Blade=0-<slot position for SC1 or SC2>

    <slot position for SC1 or SC2> is the slot position for the SC blade that was exchanged.

    Example printout:

    Note:  
    Look for the productRevisionState revision, see emphasized text in the printout below.

    .
.
.
  productName="GEP3-HD300"
  productNumber="ROJ 208 840/3"
  productRevisionState="R9A"
.
.
.

    If the hardware revision of the new blade is R9A or higher, edit the installation.conf file with the following value option bd-sdb path=/dev/sda according to the sub-step below:

    1. Log in to the working SC blade and edit the installation.conf file.

      Change option bd-sdb path=/dev/sdb to option bd-sdb path=/dev/sda:

      # vi /cluster/etc/installation.conf

  6. Prepare for a serial connection towards the still powered off SC blade:

    # telnet <Console Server IP-address> <port number connected to SC Blade>

  7. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, SC1 has been replaced, which in this case corresponds to slot position 1, circle marked, see Figure 12.

    Note:  
    Do not boot up the blade without a Console Server connected. This in order to monitor the process, change BIOS settings and more.

Figure 12   New SC GEP Blade

  1. Disable the Power Technology in the GEP BIOS.

    During the BIOS startup sequence, wait for the following console printout:

    Press <DEL> or <F4> to enter set up.

    Press F4

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  2. In the Main window, use the arrow keys to navigate to Advanced, > tab down to CPU Power Management Configuration, and > tab down to the Power Technology menu, press Enter, choose Disable and press Enter again.
  3. To save and exit:

    Press F4

    Use the arrow keys to choose Yes and press:

    Enter

  4. Turn off the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=LOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position. As an example, SC1 has been replaced, which in this case corresponds to slot position 1, circle marked, see Figure 12.

3.2.2.2   Modify LDEwS Configuration

Get the MAC address of the new SC GEP blade.

The MAC address is used as input to the cluster.conf file used by LDEwS. The MAC address is fetched through the DMXC Management System CLI.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Enter the command to show the MAC addresses.

    > ManagedElement=1,DmxcFunction=1,Eqm=1,VirtualEquipment=pg

    > show-table -m Blade -p bladeId,firstMacAddr

  4. The printout shows all MAC addresses of all blades in the PG group. Figure 13 presents the MAC addresses for each interface.

Figure 13   Example of MAC Address Printout

  1. Log in to SC2 and edit the cluster.conf file. Replace all the MAC addresses from the faulty blade to the new blade:

    # vi /cluster/etc/cluster.conf

  2. Validate and reload the cluster configuration:

    # cluster config --validate

    # cluster config --reload --all

3.2.2.3   LDEwS Installation from the Other SC Blade

Install LDEwS by booting from the other SC blade and prepare the disks.

  1. Log on to the CLI:

    # ssh -p 2024 advanced@<BSP_NBI_IP>

  2. Enter password for user advanced
  3. Prepare for a serial connection towards the still powered off SC1 blade:

    # telnet <Console Server IP-address> <port number connected to SC1 Blade>

  4. Turn on the power for the new GEP blade.

    > configure

    (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position>,Blade=1,administrativeState=UNLOCKED

    > commit -s

    The <slot position> variable corresponds to the GEP blade slot position.

  5. During the BIOS startup sequence, wait for the console printout Press F3 for GEP PopUp and then press:

    F3 to login to PBIST mode.

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  6. Choose the option 40 - UEFI Shell (PBIST) by entering the value:

    > 40

    In the next screen shown, press any key within a few seconds to proceed to the UEFI shell.

  7. Clear the present boot device order with command:

    > ipmi -o erase

  8. Check the result by command:

    > ipmi -o display

    The following printout should be displayed:

    BootList is empty

  9. Set boot device order to Backplane left device by command:

    > ipmi -o insert 1 00

    (1=priority, 00=Backplane left device).

  10. Check the result by command:

    > ipmi -o display

    Should display the newly added boot device settings.

  11. Reset the blade:

    > pbist -r

    Note:  
    This step will initiate a system reboot.

  12. The software installation will now start and the following text is shown:

    Installing, please wait...

    The software installation is completed once the following text is shown:

    Installation completed successfully

    If anything went wrong during the installation, the following message is shown instead:

    Installation failed (see /root/install.log)

  13. During the BIOS startup sequence wait for the console printout Press F3 for GEP PopUp and then press:

    F3 to login to PBIST mode.

    Note:  
    If Putty is used as terminal, the keyboard should be set to Xterm R6 to enable the F<x> keys.

  14. Choose the option 40 - UEFI Shell (PBIST) by entering the value:

    > 40

    In the next screen shown, press any key within a few seconds to proceed to the UEFI shell.

  15. Clear the present boot device order with command:

    > ipmi -o erase

  16. Check the result by command:

    > ipmi -o display

    The following printout should be displayed:

    BootList is empty

  17. Edit the boot device setting to boot from Internal SAS Disk:

    > ipmi -o insert 1 10

    (1=priority, 10=SAS Disk device).

  18. Check the result by command:

    > ipmi -o display

  19. Reset the new SC blade:

    > pbist -r

    Note:  
    This step will initiate a system reboot.

    The boot sequence of the new SC blade will take a long time (usually somewhere between half an hour and several hours) due to the need for a full disk synchronization at first boot after installation. The actual time it takes to complete the disk synchronization depends on hardware properties (disk size, disk speed and network speed). Avoid rebooting either SC blade until the disk synchronization has completed, as that will only prolong the time it takes to complete the synchronization.


  20. Run the following command to see the synchronization progress:

    # drbd-overview

  21. After the disk synchronization has finished, LDEwS, eVIP and the Application RPMs are loaded.

3.2.2.4   Licenses

Note:  
Make sure that the new license has been ordered.

Caution!

Do not reboot the other SC blade which is not replaced, before the new license is installed successfully.

To install the licenses for Dynamic Activation follow the instructions in section License Administration in System Administrators Guide for Native Deployment, Reference [3].

3.2.2.5   Dynamic Activation Software Setup

  1. Log on to the replaced node. Perform the following procedure to start data transfer from other Cassandra nodes.
    1. In cassandra.yaml file, located in the /cluster/home/casadm/nodes/<nodeId>/cassandra/conf/ directory, remove the IP address of the replaced node from seeds parameter. For example: 
      #- seeds: "169.254.100.1,169.254.100.2"
- seeds: "169.254.100.2"
    2. Stop Cassandra.

      # 3ppmon stopcassandra

    3. From other SC node, remove the replaced node from Cassandra ring.

      Check the Cassandra nodes status.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load     Tokens Owns Host ID                               Rack
UN  169.254.100.4 74.44 GB   256  ?    5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2 74.90 GB   256  ?    de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3 72.34 GB   256  ?    f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
DN  169.254.100.1 74.44 GB   256  ?    004eb495-da8e-47eb-b6d0-0750ed5f5a50  rack1

      Find the replaced Cassandra node and copy its Host ID.

      Remove the replaced node from the Cassandra ring.

      # nodetool removenode <Host ID>

      In this example:

      # nodetool removenode 004eb495-da8e-47eb-b6d0-0750ed5f5a50

      After prompt is returned, check whether the removal command is finished.

      # nodetool removenode status

      Expected result: RemovalStatus: No token removals in process.

      Check whether the node has been removed.

      # nodetool status

      Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load     Tokens Owns Host ID                               Rack
UN  169.254.100.4 74.44 GB   256  ?    5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.3 72.34 GB   256  ?    f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UN  169.254.100.2 74.44 GB   256  ?    de177bc5-3142-436d-b1c2-0bbf13410267  rack1

    4. On the replaced node, remove all data from commitlog and data directory.

      # cd /var/cassandra/commitlog
      # rm -rf *

      # cd /var/cassandra/data
      # rm -rf *

    5. Start Cassandra.

      # 3ppmon startcassandra

      When Cassandra is started, the replaced node is joining in to the ring.

  2. Run the following command to make sure Cassandra has joined the ring:

    # nodetool status

    Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens    Owns   Host ID                               Rack
UN  169.254.100.4  76.77 GB   256       ?      5100b6ba-a90f-4ca2-ac40-eef26915a56b  rack1
UN  169.254.100.2  74.44 GB   256       ?      de177bc5-3142-436d-b1c2-0bbf13410267  rack1
UN  169.254.100.3  72.34 GB   256       ?      f760ea56-71f5-4b55-8e79-9e7821a3e785  rack1
UJ  169.254.100.1  254.9 MB   256       ?      7b71585e-778f-424b-9ed2-813ef5867b01  rack1

    The replaced SC node status is UJ.

  3. Wait until the Cassandra status on the replaced node has changed to UN.
  4. In cassandra.yaml file, write back the previously removed IP addresses of the replaced node in seeds parameter. For example: 
     -seeds: "169.254.100.1,169.254.100.2"
 #-seeds: "169.254.100.2"
  5. To check the deployment status of all 3PPs, in all nodes, run the following command:

    # 3ppmon status --host all

  6. Start the Dynamic Activation application on the new SC blade:

    # bootloader.py node start --host <hostname of replaced blade>

  7. Check the Dynamic Activation status:

    On an SC node, execute the following command to check the status of the Dynamic Activation applications:

    # bootloader.py node status --host all

  8. On all nodes in the cluster, one by one, run the following command:

    # nodetool cleanup


Reference List

Ericsson Documents
[1] Glossary of Terms and Acronyms, 0033-CSH 109 628 Uen
[2] Library Overview, 18/1553-CSH 109 628 Uen
[3] System Administrators Guide for Native Deployment, 1/1543-CSH 109 628 Uen
[4] Hardware Installation and IP Infrastructure Setup for Native Deployment GEP3, 2/1531-CSH 109 628 Uen
[5] Hardware Specification, 1/2135-CSH 109 628 Uen
Other Documents
[6] BSP Hardware Description. 1/1551-APP 111 01 Uen


Copyright

© Ericsson AB 2017. All rights reserved. No part of this document may be reproduced in any form without the written permission of the copyright owner.

Disclaimer

The contents of this document are subject to revision without notice due to continued progress in methodology, design and manufacturing. Ericsson shall have no liability for any error or damage of any kind resulting from the use of this document.

Trademark List
All trademarks mentioned herein are the property of their respective owners. These are shown in the document Trademark Information.

    Blade Replacement Instruction for Native Deployment         Ericsson Dynamic Activation 1