Replacing nodes nondisruptively

You can replace SAN Volume Controller 2145-8F2, SAN Volume Controller 2145-8F4, or SAN Volume Controller 2145-8G4 nodes with SAN Volume Controller 2145-8A4 or SAN Volume Controller 2145-CF8 nodes. You can also use these procedures if you are replacing a SAN Volume Controller 2145-4F2 node with a SAN Volume Controller 2145-8A4 or earlier node.

These procedures do not include replacing a SAN Volume Controller 2145-4F2 node with a SAN Volume Controller 2145-CF8 node.

For information about replacing a SAN Volume Controller 2145-4F2 node with a SAN Volume Controller 2145-CF8 node, see Procedures for Replacing SAN Volume Controller 2145-4F2 Nodes with SAN Volume Controller 2145-CF8 Nodes on the Support for SAN Volume Controller (2145) Web site:

Support for SAN Volume Controller (2145)
Web site

The link to the procedures is located under Related Reading on the page that contains the V5.1.x IBM System Storage SAN Volume Controller Software Installation and Configuration Guide.

The following procedures are nondisruptive, because changes to your SAN environment are not required. This is because the replacement (new) node uses the same worldwide node name (WWNN) as the node you are replacing.

This task assumes that the following conditions have been met:

The existing cluster software must be at a version that supports the new node. If a node is being replaced by a SAN Volume Controller 2145-CF8 node, the cluster software version must be 5.1.0. If a node is being replaced by a SAN Volume Controller 2145-8A4 node, the cluster software version must be 4.3.1 or later.
All nodes that are configured in the cluster are present and online.
All errors in the cluster error log are addressed and marked as fixed.
There are no virtual disks (VDisks), managed disks (MDisks), or controllers with a status of degraded or offline.
The replacement node is not powered on.
The replacement node is not connected to the SAN.
You have a 2145 UPS-1U unit (feature code 8115) for each new SAN Volume Controller 2145-CF8 or 2145-8A4 node.
You have backed up the cluster configuration and saved the svc.config.backup.xml file.
The replacement node must be able to operate at the fibre-channel or Ethernet connection speed of the node it is replacing.
If the node being replaced contains solid-state drives (SSDs), all SSDs and SAS adapters should be transferred to the new node if it supports the drives. If the new node does not support the existing SSDs, you must transfer the data off of the SSDs before replacing the node to avoid losing access to the data.

Important:

Do not continue this task if any of the conditions listed above are not met unless you are instructed to do so by the IBM® Support Center.
Review all of the steps listed below before you perform this task.
Do not perform this task if you are not familiar with SAN Volume Controller environments or the procedures described in this task.
If you plan to reuse the node that you are replacing, ensure that the WWNN of the node is set to a unique number on your SAN. If you do not ensure that the WWNN is unique, the WWNN and WWPN are duplicated in the SAN environment and can cause issues.
Tip: You can change the WWNN of the node you are replacing to the factory default WWNN of the replacement node to ensure that the number is unique.
The node ID and possibly the node name change during this task. After the cluster assigns the node ID, the ID cannot be changed. However, you can change the node name after this task is complete.

Perform the following steps to replace active nodes in a cluster:

(If the cluster software version is at 5.1, complete this step.)
Confirm that no hosts have dependencies on the node.

When shutting down a node that is part of a cluster, or when deleting the node from a cluster, use the Show Dependent VDisks menu option on the Viewing Nodes panel in the SAN Volume Controller Console to display all the VDisks that are dependent on a node, or use the svcinfo lsnodedependentvdisk command to view dependent VDisks.

If dependent VDisks exist, determine if the VDisks are being used. If the VDisks are being used, either restore the redundant configuration or suspend the host application. If a dependent quorum disk is reported, repair the access to the quorum disk or modify the quorum disk configuration.
Perform the following steps to determine the cluster configuration node, and the ID, name, I/O group ID, and I/O group name for the node that you want to replace. If you already know the physical location of the node that you want to replace, you can skip this step and proceed to step 3.
Tip: If any of the nodes you want to replace are the cluster configuration node, replace it last.
1. Issue the following command from the command-line interface (CLI):
```
svcinfo lsnode -delim : 
```
  The following is an example of the output that is displayed for this command:
```
id:name:UPS_serial_number:WWNN:status:IO_group_id:IO_group_name:
config_node:UPS_unique_id:hardware:iscsi_name:iscsi_alias
3:dvt113294:100089J137:5005076801005A07:online:0:io_grp0:yes:
20400002096810C7:8A4:iqn.1986-03.com.ibm:2145.ldcluster-80.dvt113294:
14:des113004:10006BR010:5005076801004F0F:online:0:io_grp0:no:
2040000192880040:8G4:iqn.1986-03.com.ibm:2145.ldcluster-80.des113004:
```
2. In the config_node column, find the value yes and record the values in the id and name columns.
3. Record the values in the id and the name columns for each node in the cluster.
4. Record the values in the IO_group_id and the IO_group_name columns for each node in the cluster.
5. Issue the following command from the CLI for each node in the cluster to determine the front panel ID:
```
svcinfo lsnodevpd node_name or node_id
```
  where node_name or node_id is the name or ID of the node for which you want to determine the front panel ID.
6. Record the value in the front_panel_id column. The front panel ID is displayed on the front of each node. You can use this ID to determine the physical location of the node that matches the node ID or node name that you want replace.
Perform the following steps to record the WWNN or iSCSI name of the node that you want to replace:
1. Issue the following command from the CLI:
```
svcinfo lsnode -delim : node_name or node_id
```
  where node_name or node_id is the name or ID of the node for which you want to determine the WWNN or iSCSI name.
2. Record the WWNN or iSCSI name of the node that you want to replace. Also record the order of the fibre-channel and Ethernet ports.
Issue the following command from the CLI to power off the node:
```
svctask stopcluster -node node_name
```
Important:
1. Record and mark the order of the fibre-channel or Ethernet cables with the node port number (port 1 to 4 for fibre-channel, or port 1 to 2 for ethernet) before you remove the cables from the back of the node. The fibre-channel ports on the back of the node are numbered 1 to 4 from left to right. You must reconnect the cables in the exact order on the replacement node to avoid issues when the replacement node is added to the cluster. If the cables are not connected in the same order, the port IDs can change, which impacts the ability of the host to access VDisks. See the hardware documentation specific to your model to determine how the ports are numbered.
2. Do not connect the replacement node to different ports on the switch or director. The SAN Volume Controller can have 4 or 8 Gbps HBAs; however, do not move them to faster switch or director ports at this time to avoid issues when the replacement node is added to the cluster.
3. Do not move the fibre-channel cables of the node to faster or different ports on the switch or director at this time. This is a separate task that must be planned independently of replacing nodes in a cluster.
Issue the following CLI command to delete this node from the cluster and I/O group:
```
svctask rmnode node_name or node_id
```
Where node_name or node_id is the name or ID of the node that you want to delete. You can use the CLI to verify that the deletion process has completed.
Issue the following CLI command to ensure that the node is no longer a member of the cluster:
```
svcinfo lsnode 
```
A list of nodes is displayed. Wait until the removed node is not listed in the command output.
Perform the following steps to change the WWNN or iSCSI name of the node that you just deleted from the cluster to FFFFF:
For SAN Volume Controller V4.3 or later:
1. Power on the node.
2. With the Node WWNN: panel displayed, press and hold the down button, press and release the select button, and then release the down button. The display switches into edit mode. Edit WWNN is displayed on line 1. Line 2 of the display contains the last five numbers of the WWNN.
3. Change the displayed number to FFFFF. To edit the highlighted number, use the up and down buttons to increase or decrease the numbers. The numbers wrap F to 0 or 0 to F. Use the left and right buttons to move between the numbers.
4. Press the select button to save your changes and apply FFFFF as the new WWNN for the node.
For SAN Volume Controller versions prior to V4.3:
1. Power on the node.
2. Press and release the right button until the Status: panel is displayed.
3. With the node status displayed on the front panel, press and hold the down button; press and release the select button; release the down button. WWNN is displayed on line 1 of the display. Line 2 of the display contains the last five numbers of the WWNN.
4. With the WWNN displayed on the front panel; press and hold the down button; press and release the select button; release the down button. The display switches into edit mode.
5. Change the displayed number to FFFFF. To edit the highlighted number, use the up and down buttons to increase or decrease the numbers. The numbers wrap F to 0 or 0 to F. Use the left and right buttons to move between the numbers.
6. Press the select button to retain the numbers that you have updated and return to the WWNN panel.
7. Press the select button to apply the numbers as the new WWNN for the node.
8. Remove the node to be replaced and optionally the uninterruptible power supply from the rack.
Install the replacement node and the uninterruptible power supply in the rack and connect the uninterruptible power supply cables. See the IBM System Storage SAN Volume Controller Model 2145-XXX Hardware Installation Guide to determine how to connect the node and the uninterruptible power supply.
Important: Do not connect the fibre-channel or Ethernet cables during this step.
Power on the replacement node.
Record the WWNN or iSCSI nameof the replacement node. You can use this name if you plan to reuse the node that you are replacing.
Perform the following steps to change the WWNN or iSCSI nameof the replacement node to match the name that you recorded in step 3:
For SAN Volume Controller V4.3 or later:
1. With the Node WWNN: panel displayed, press and hold the down button, press and release the select button, and then release the down button. The display switches into edit mode. Edit WWNN is displayed on line 1. Line 2 of the display contains the last five numbers of the WWNN.
2. Change the WWNN that is displayed to match the last five numbers of the WWNN that you recorded in step 3. To edit the highlighted number, use the up and down buttons to increase or decrease the numbers. The numbers wrap F to 0 or 0 to F. Use the left and right buttons to move between the numbers.
3. Press the select button to apply the numbers as the new WWNN for the node.
For SAN Volume Controller versions prior to V4.3:
1. Press and release the right button until the Status: panel is displayed.
2. With the node status displayed on the front panel, press and hold the down button; press and release the select button; release the down button. WWNN is displayed on line 1 of the display. Line 2 of the display contains the last five numbers of the WWNN.
3. With the WWNN displayed on the front panel; press and hold the down button; press and release the select button; release the down button. The display switches into edit mode.
  To edit the highlighted number, use the up and down buttons to increase or decrease the numbers. The numbers wrap F to 0 or 0 to F. Use the left and right buttons to move between the numbers.
4. When the five numbers match the last five numbers of the WWNN that you recorded in step 3, press the select button to retain the numbers that you have updated and return to the WWNN panel.
5. Press the select button to apply the numbers as the new WWNN for the node.
Wait one minute. If Cluster: is displayed on the front panel, this indicates that the node is ready to be added to the cluster. If Cluster: does not display, see the IBM System Storage SAN Volume Controller Troubleshooting Guide to determine how to address this problem or contact the IBM Support Center before you continue with the next step.
Connect the fibre-channel or Ethernet cables to the same port numbers that you recorded for the original node in step 4.
Issue the following CLI command to verify that the last five characters of the WWNN are correct:
```
svcinfo lsnodecandidate
```
Important: If the WWNN is not what you recorded in step 3, you must repeat step 11.
Issue the following CLI command to add the node to the cluster and ensure that the node has the same name as the original node and is in the same I/O group as the original node. See the svctask addnode CLI command documentation for more information.
```
svctask addnode -wwnodename WWNN -iogrp iogroupname/id 
```
where WWNN and iogroupname/id are the values that you recorded for the original node.

The SAN Volume Controller V5.1 automatically reassigns the node with the name that was used originally. For versions prior to V5.1, use the name parameter with the svctask addnode command to assign a name. If the original node's name was automatically assigned by SAN Volume Controller, it is not possible to reuse the same name. It was automatically assigned if its name starts with node. In this case, either specify a different name that does not start with node or do not use the name parameter so that SAN Volume Controller automatically assigns a new name to the node.

If necessary, the new node is updated to the same SAN Volume Controller software version as the cluster. This update can take up to 20 minutes.
Important:
1. Both nodes in the I/O group cache data; however, the cache sizes are asymmetric. The replacement node is limited by the cache size of the partner node in the I/O group. Therefore, it is possible that the replacement node does not utilize the full cache size until you replace the other node in the I/O group.
2. You do not have to reconfigure the host multipathing device drivers because the replacement node uses the same WWNN and WWPN as the previous node. The multipathing device drivers should detect the recovery of paths that are available to the replacement node.
3. The host multipathing device drivers take approximately 30 minutes to recover the paths. Do not upgrade the other node in the I/O group until for at least 30 minutes after you have successfully upgraded the first node in the I/O group. If you have other nodes in different I/O groups to upgrade, you can perform those upgrades while you wait.
See the documentation that is provided with your multipathing device driver for information on how to query paths to ensure that all paths have been recovered before proceeding to the next step. If you are using the IBM System Storage® Multipath Subsystem Device Driver (SDD), the command to query paths is datapath query device.
Repeat steps 3 to 15 for each node that you want to replace.