Contents1 Introduction
This document describes how to replace a blade in an Ericsson Centralized User Data Base (CUDB) node deployed on native BSP 8100.
1.1 Description
1.2 Revision Information
Other than editorial changes, this document has been revised as follows:
Prerequisites: Updated section.
Replacing a Blade: Updated note.
Identifying Blade Hardware Type and Board Revision: Added section.
Preparing the Blade Replacement: Added example to Step 3.
CUDB Node Configuration Changes: Added note.
Editing the LDE installation.conf File: Added section.
Editing the LDE installation.conf File in Case of GEP3 Hardware: Updated section title.
Editing the LDE installation.conf File in Case of GEP7L Hardware: Added section.
System Controller Replacement Steps: Added notes. Updated description.
1.3 Typographic Conventions
Typographic Conventions can be found in the following document:
2 Prerequisites
Before replacing any blades, make sure to check the hardware description of the node. Refer to CUDB Node Hardware Description for more information.
It is recommended to perform a software backup before starting the blade replacement procedure. Refer to CUDB System Administrator Guide for more information on using cudbSwBackup command.
Before starting this procedure, ensure that the following documents are available:
3 Replacing a Blade
This section describes how to identify a faulty blade, how to perform a blade replacement in a CUDB node, and how to prepare the replacement blade for operation.
Also, further actions after physical board replacement are described.
| Note: |
In the case of GEP5 SC blade replacement with Generic Ericsson Processor version 7, Low
Power (GEP7L) boards, both controllers must be GEP7L. Mixed GEP5/GEP7L scenarios are not
allowed on SCs. In the
case of replacing a single GEP7L blade with another GEP7L blade, it can be done individually
on a single SC. |
3.1 Identifying the Faulty Blade
3.1.1 Identifying the Blade Name
Perform the following steps to identify a faulty blade in a CUDB node:
Steps
3.1.2 Identifying Blade Rack and Subrack Position
To identify the physical blade that has to be replaced, do the following:
Steps
Results
The output must be similar to the following example:
======================= | bladeId | userLabel | ======================= | 0-1 | SC-1 | | 0-11 | PL-6 | | 0-13 | PL-7 | | 0-15 | PL-8 | | 0-17 | PL-9 | | 0-19 | PL-10 | | 0-21 | PL-11 | | 0-23 | PL-12 | | 0-3 | SC-2 | | 0-5 | PL-3 | | 0-7 | PL-4 | | 0-9 | PL-5 | | 1-1 | PL-13 | | 1-11 | PL-18 | | 1-13 | PL-19 | | 1-15 | PL-20 | | 1-17 | PL-21 | | 1-19 | PL-22 | | 1-21 | PL-23 | | 1-23 | PL-24 | | 1-3 | PL-14 | | 1-5 | PL-15 | | 1-7 | PL-16 | | 1-9 | PL-17 | | 2-1 | PL-25 | | 2-11 | PL-30 | | 2-13 | PL-31 | | 2-15 | PL-32 | | 2-17 | PL-33 | | 2-19 | PL-34 | | 2-21 | PL-35 | | 2-23 | PL-36 | | 2-3 | PL-26 | | 2-5 | PL-27 | | 2-7 | PL-28 | | 2-9 | PL-29 | ======================= |
| Note: |
LDE and BSP 8100 naming conventions are slightly different, so SC_2_1 on LDE level equals
to SC-1 on BSP 8100 and so on. |
The bladeId identifies the blade position in the rack, the first number meaning the subrack and the second meaning the slot within the subrack. For example, PL-14 is in the third slot of subrack 1.
3.1.3 Identifying Blade Hardware Type and Board Revision
To identify the blade hardware type and revision, do the following:
Steps
Results
The output must be similar to the following example:
productIdentity="ROJ 208 840/3"
productDesignation="GEP3-HD300"
productRevision="R4B"
productIdentity="ROJ 208 868/5"
productDesignation="GEP5-64-400"
productRevision="R2A"
GEP7
productIdentity="ROJ208864/7"
productDesignation="GEP7L-64-X16"
productRevision="R1B"
3.2 Preparing the Blade Replacement
Perform the following steps to prepare the blade replacement:
| Note: |
In the below commands,
<name>
and
<blade>
are used to identify blades, where: |
Steps
3.4 CUDB Node Configuration Changes
This section describes the configuration changes to perform in a CUDB node in case blade replacement is needed.
| Note: |
It is recommended to make a backup of the file to be modified. |
3.4.1 Obtaining MAC Addresses for the New Blade
The MAC addresses are used as input to create the cluster.conf file, which is used by LDE. The MAC addresses are also required to configure the Jumpstart server before installing LDE on the SCs, as well for the blade replacement procedure.
The MAC addresses are fetched through the BSP CLI. This MAC is the MAC base, used to obtain the MAC addresses necessary to complete the cluster.conf file generation.
To obtain the MAC addresses, do the following:
Steps
3.4.1.1 Obtaining All MAC Addresses
The MAC shown for each shelf slot in Obtaining MAC Addresses for the New Blade is the base MAC. All the MACs can be obtained by adding a number to the <base mac> , in accordance to the following tables. tbl-mac-addresses-relation applies to BSP 8100 (GEP3) boards, tbl-mac-address-relation-gep5 applies to BSP 8100 (GEP5) boards and Table 3 applies to BSP 8100 (GEP7L) boards.
|
Address |
Resulting MAC(1) |
|
|---|---|---|
|
<BASE MAC> + 1 |
eth3 |
Left SCX Backplane Port |
|
<BASE MAC> + 2 |
eth4 |
Right SCX Backplane Port |
|
<BASE MAC> + 3 |
eth2 |
ETH-Debug Front Port |
|
<BASE MAC> + 5 |
eth0 |
ETH-0 Front Port |
|
<BASE MAC> + 6 |
eth1 |
ETH-1 Front Port |
|
<BASE MAC> + 8 |
eth5 |
Left SCX 10GbE Backplane Port |
|
<BASE MAC> + 9 |
eth6 |
Right SCX 10GbE Backplane Port |
|
Address |
Resulting MAC(2) |
|
|---|---|---|
|
<BASE MAC> + 1 |
eth3 |
Left SCX 1GbE Backplane Port |
|
<BASE MAC> + 2 |
eth4 |
Right SCX 1GbE Backplane Port |
|
<BASE MAC> + 3 |
eth2 |
ETH-Debug Front Port |
|
<BASE MAC> + 5 |
eth5 |
Left SCX 10GbE Backplane Port |
|
<BASE MAC> + 6 |
eth6 |
Right SCX 10GbE Backplane Port |
|
<BASE MAC> + 8 |
eth0 |
ETH-0 Front Port |
|
<BASE MAC> + 9 |
eth1 |
ETH-1 Front Port |
|
Address |
Resulting MAC(3) |
|
|---|---|---|
|
<BASE MAC> + 1 |
eth3 |
Left SCX Backplane Port |
|
<BASE MAC> + 2 |
eth4 |
Right SCX Backplane Port |
|
<BASE MAC> + 7 |
eth5 |
Left SCX 10GbE Backplane Port |
|
<BASE MAC> + 8 |
eth6 |
Right SCX 10GbE Backplane Port |
| Note: |
Ports ETH-0 and ETH-1 are enabled only during the initial software installation phase
from the Jumpstart server. After the LDE is installed on the blade, they remain
disabled and cannot be used. |
3.4.3 Editing the LDE installation.conf File
3.4.3.2 Editing the LDE installation.conf File in Case of GEP7L Hardware
This procedure is required only after the replacement of the first SC, regardless if it is SC_1 or SC_2. After that no additional changes are required.
| Note: |
This procedure must be skipped for replacement of GEP7L blade with GEP7L
blade. |
Steps
3.4.4 Editing the LDE cluster.conf File
Perform the following steps to edit the cluster.conf file.
Steps
3.5 System Controller Replacement Steps
This section describes the procedure to finalize the SC blade replacement.
If GEP5 blades are replaced with GEP7L blades, replace
installation.conf accordingly.
The new blade is by default set to boot from network, the following procedure describes how to set it to boot from hard disk.
During this procedure, the new SC also synchronizes its replicated storage disk partition with another SC. This process can take up to one hour, depending on storage disk partition size and available network bandwidth. Use the following command on another SC to check the synchronization status and drdb Primary with the following command (first listed in command output is drbd status of the current SC):
cat /proc/drbd
If GEP5 blades are replaced with GEP7L blades, to use the whole disk partition, execute
the pvresize /dev/drbd0 and the lvresize -r -L 320G
/dev/lde-cluster-vg/lde-cluster-lv commands on SC with drbd process
Primary.
Perform the following steps to finalize the SC replacement:
Steps
3.7 Finalizing Replacement
To finish blade replacement, perform the following steps which apply to replacing every blade type (SC, DSG, and PLDB).
| Note: |
In case of SC replacement, crontab jobs and their definitions,
or similar tasks, which are not deployed by default in CUDB, or scheduled
with data or software backup scripts, will be lost. If necessary,
redeploy them after the procedure is completed. |
Steps
After This Task
Refer to the CUDB System Administrator Guide for more information.
3.8 Changing the Boot Device Order
3.8.1 Changing the Boot Device Order on GEP3 Boards
To change the boot device order on the GEP3 boards, connect to the SCXB RS232 connector and to the GEP3 console port at the same time. For these connections, two VT100 Terminals and two serial cables are required.
| Note: |
To ensure correct blade operation, the configured boot device
type must be Hard disk for SC boards,
and Backplane port for payload blades. |
Steps
-
Enter PBIST mode by pressing F3 during boot up. The screen must be
similar to Figure 1.
Figure 1 GEP3 BIOS Pop-Up -
Wait for the prompt to appear, then type 40 to invoke the
Unified Extensible Firmware Interface (UEFI) prompt. A screen
similar to the below example must appear.
Note: After typing 40 above, press any key in 10 seconds. If no keys are pressed in 10 seconds, the GEP3 blade starts booting, and the procedure must be restarted from Step 3.
Figure 2 PBIST Menu
Figure 3 Press Key -
Execute the ipmi -o display command to check the boot
configuration. Then use the ipmi -o pop command to erase the boot
devices for GEP3. Repeat this step until the list is empty.
See Figure 4.
Figure 4 Boot Devices -
Execute the ipmi -o display command to check the boot device
list. If the list is empty, execute the ipmi -o push 10 command to
add the hard disk to the list.
See Figure 5.
Figure 5 Hard Disk Device Input -
Execute pbist -r command to reboot the blade.
See Figure 6.
Figure 6 PBIST Reboot
3.8.2 Changing the Boot Device Order on GEP5 Boards
To change the boot device order on the GEP5 boards, connect to the SCXB RS232 connector and to the GEP5 console port at the same time. For these connections, two VT100 Terminals and two serial cables are required.
| Note: |
To ensure correct blade operation, the configured boot device type must be
for SC boards, and Ethernet Backplane port for payload blades.Hard
drive The instructions below apply to SC boards only. If a payload board is configured, the devices pushed to the IMPI boot table in Step 9 must be numbered as 00 and 01. |
Steps
-
Enter PBIST mode by pressing F3 during boot up. The screen must be
similar to Figure 7.
Figure 7 GEP5 PBIST Menu -
Wait for the prompt to appear, then type 40 to invoke the
UEFI prompt. A screen similar to the below example in Figure 8 must appear.
Note: After typing 40 above, press any key in 10 seconds. If no keys are pressed in 10 seconds, the GEP5 blade starts booting, and the procedure must be restarted from Step 3.
Figure 8 Press Any Key in the UEFI Shell -
Erase the boot configuration with the following steps:
-
Type the following command to check the boot configuration:
ipmi bo display
-
Erase the boot devices of the GEP5 board with the following command:
ipmi bo eraseIf you want to delete only one boot device, use the following command:ipmi bo pop
- Use the ipmi bo display command again to check the boot configuration after the erase.
See Figure 9 for an example of the above steps.
Figure 9 Erasing the List of Boot Devices -
Type the following command to check the boot configuration:
-
If the boot device list is empty, run the following commands to add all hard disks to
the list:
ipmi bo push 10ipmi bo push 11ipmi bo push 12See Figure 10 for an example.
Figure 10 Adding Hard Disks to the Boot Order -
Reboot the blade with the following command:
pbist -rAn example output is shown in Figure 11.
Figure 11 Rebooting the GEP5 Board
3.8.3 Changing the Boot Device Order on GEP7L Boards
To change boot device order on GEP7L boards, connect to the SCXB RS232 connector and to the GEP7L console port at the same time. For these connections, two VT100 terminals and two serial cables are required.
To ensure correct blade operation, the configured boot device type must be "Hard drive" for SC boards, and "Ethernet Backplane port" for payload blades.
Steps
-
Enter PBIST mode by pressing F3 during boot up. The screen must be similar to Figure 12.
Figure 12 GEP7L PBIST Menu -
Wait for the prompt to appear, then type 40 to invoke the
Unified Extensible Firmware Interface prompt.
Note: After typing 40 above, press any key in 3 seconds to proceed to the Unified Extensible Firmware Interface shell. If no keys are pressed in 3 seconds, the GEP7L blade starts booting, and the procedure must be restarted fromStep 3.
See Figure 13 as an example.
Figure 13 Unified Extensible Firmware Interface Shell -
Erase the boot configuration with the following steps:
-
Clear the present boot device order with command:
ipmi oem bo erase
-
Check the result with command:
ipmi oem bo displayIt must be empty.
See Figure 14 as an example for the above steps.
Figure 14 Erasing the List of Boot Devices -
Clear the present boot device order with command:
-
Execute the following commands to add two Internal SAS disks as
first and second priority, and Ethernet Backplane Left as third
priority to the list:
ipmi oem bo insert 1 10ipmi oem bo insert 2 11ipmi oem bo insert 3 00See Figure 15 as an example.
Figure 15 Adding the List of Boot Devices -
Reset the board using command:
resetSee Figure 16 as an example.
Figure 16 Rebooting the GEP7L Board
3.9 Replacing Multiple Blades in Parallel
This section provides instructions required to replace multiple blades in parallel on CUDB nodes.
3.9.1 Parallel Blade Replacement Procedure
Only the same group of blades can be replaced in parallel at once. In the CUDB system, blades can be grouped into three distinct groups: SC blades, PLDB blades, and DSG blades. These groups can be further divided into groups of even-numbered and odd-numbered blades, resulting six distinct groups of blades in total:
Do not replace blades in parallel if they belong to different blade groups. Replacing blades belonging to different groups in parallel at the same time can cause major node outage.
Perform the following steps to replace multiple blades in parallel:
| Note: |
To ensure that there is enough traffic handling capacity
during replacement execution, it is recommended that the maximum number
of payload blades to be replaced in parallel must not be larger than
the configured value of the redundancyLevel attribute of the CudbLdapAccess class. If there are more blades to be replaced, it must be done iteratively, in a way that in each iteration, replacement is done for maximum of N blades from the same group in parallel, where N is the value of the redundancyLevel attribute. However, if replacement is done in low traffic period or in a maintenance window, when the degraded traffic handling capacity could still be sufficient, it can be decided to execute replacement for more than N blades in parallel. |
Steps