------------------------------------------------------------------------------- Software name Lenovo Storage V3700 V2 Controller Firmware Update Bundle Supported Model System name Lenovo Storage V3700 V2 Lenovo Storage V3700 V2 XP Version 8.2.1.17 Issue date March 1st, 2023 Prerequisites: None ------------------------------------------------------------------------------- WHAT THIS PACKAGE DOES ------------------------------------------------------------------------------- This firmware update enables you to update the controller firmware via; 1. Lenovo Storage V3700 V2 - Web User Interface. 2. Lenovo Storage V3700 V2 - Command Line Interface (CLI). ------------------------------------------------------------------------------- Version history ------------------------------------------------------------------------------- The following versions of Controller Firmware that been released to date. Summary of Changes Where: < > Package version [Important] Important update (New) New function or enhancement (Fix) Correction to existing function Security Issues Resolved: Security issues are documented using a reference number provided by "Common Vulnerabilities and Exposures" (CVE). CVE-2022-21626 and CVE-2022-43873 Resolved in v8.2.1.17. Security Issues Resolved: Security issues are documented using a reference number provided by "Common Vulnerabilities and Exposures" (CVE). CVE-2018-25032, CVE-2021-35550, CVE-2021-35603 and CVE-2022-0778 Resolved in v8.2.1.16. - [Important] The following issues have been resolved with this release: Using addvdiskcopy in conjunction with expandvdisk with format may result in the original being overwritten, by the new copy, producing blank copies. ( HU02327 ) - Volume Mirroring A problem in the virtualization component of the system can cause a migration IO to be submitted in an incorrect context resulting in a node warmstart. In some cases it is possible that this IO has been submitted to an incorrect location on the backend, which can cause data corruption of an isolated small area. ( HU02400 ) - Storage Virtualisation Occasionally when an offline drive returns to online state later than its peers in the same RAID array there can be multiple node warmstarts that send nodes into a service state. ( HU02342 ) - RAID Hosts with Emulex 16Gbps HBAs may become unable to communicate with a system with 8Gbps Fibre Channel ports, after the host HBA is upgraded to firmware version 12.8.364.11. This does not apply to systems with 16Gb or 32Gb Fibre Channel ports. ( HU02374 ) - Hosts Automatic resize of compressed/thin volumes may fail causing warmstarts on both nodes in an I/O group. ( HU02393 ) - Storage Virtualisation EasyTier can move extents between identical mdisks until one runs out of space. ( HU02401 ) - EasyTier An interoperability issue between Cisco NX-OS firmware and the Spectrum Virtualize Fibre Channel driver can cause a node warmstart on NPIV failback (for example during an upgrade) with the potential for a loss of access. ( HU02406 ) - Interoperability During RAID rebuild or copyback on systems with 16gb or less of memory, cache handling can lead to a deadlock which results in timeouts. ( IT38015 ) - RAID - (Fix) The following issues have been resolved with this release: Slow internal resource reclamation by the RAID component can cause a node warmstart. ( HU02366 ) - RAID When a BIOS upgrade occurs excessive tracefile entries can be generated. ( HU02433 ) - System Update - [Important] The following issues have been resolved with this release: An upgrade may fail due to corrupt hardened data in a node. This can affect an IO group. ( HU01968 & HU02215 ) - System Update Changing a remote copy relationship from GMCV to MM or GM can result in a Tier 2 recovery. ( HU02058 ) - Global Mirror, Global Mirror with Change Volumes, Metro Mirror A Hot Spare Node (HSN) timing window issue can, during an HSN activation or deactivation, cause the cluster to broadcast an invalid VPD update to other clusters on the SAN. This may trigger a Tier 2 recovery on the other cluster. ( HU02213 ) - Hot Spare Node An issue in auto-expand can cause expansion to fail and the volume to be taken offline. ( HU02266 ) - Thin Provisioning RAID parity scrubbing can become stalled causing an accumulation of media errors leading to multiple drive failures with the possibility of data integrity loss. ( HU02277 ) - RAID When upgrading from v8.2.1 to v8.3.1, or later, an issue with the handling of node metadata may cause a Tier 2 recovery. ( HU02295 ) - System Update Global Mirror environments may experience more frequent 1920 events due to writedone message queuing. ( HU02156 ) - Global Mirror An issue in Remote Copy may cause a loss of hardened data when a node is warmstarted. ( HU02164 ) - Global Mirror, Global Mirror with Change Volumes, Metro Mirror During upgrade a node may limit the number of target ports it reports causing a failover contradiction on hosts. ( HU02176 ) - Hosts When a 3PAR controller experiences a fault that prevents normal IO processing it may issue a SCSI TARGET RESET command. This command is not supported and may cause multiple node asserts, possibly cluster-wide. ( HU02184 ) - Backend Storage When upgrading from v8.1 or earlier to v8.2.1 or later a remote copy issue may cause a node warmstart, stalling the upgrade. ( HU02200 ) - System Update A node might fail to come online after a reboot or warmstart such as during an upgrade. ( HU02288 ) - Reliability Availability Serviceability - (Fix) The following issues have been resolved with this release: An issue in the handling of ATS commands from VMware hosts can cause a single node warmstart. ( HU02048 ) - Hosts It is possible for a backend unmap process to become stalled, preventing system configuration changes from completing. ( HU02142 ) - Distributed RAID False positive node error 766 (depleted CMOS battery) messages may appear in the Event Log. ( HU02244 ) - System Monitoring The use of maximum replication delay within Global Mirror may occasionally cause a node warmstart. ( HU02292 & HU02308 ) - Global Mirror An issue in the handling of read transfers may cause hung host IOs leading to a node warmstart. ( HU02354 ) - Hosts An issue in Remote Copy, that stalls a switch of direction, can cause IO timeouts leading to a node warmstart. ( HU02358 ) - Global Mirror, Global Mirror with Change Volumes, Metro Mirror - [Important] The following issues have been resolved with this release: After node reboot, or warmstart, some volumes accessed by AIX, VIO or VMware hosts may experience stuck SCSI2 reservations on the NPIV failover ports of the partner node. This can cause a loss of access to data (HU01894) - Hosts Where FlashCopy mapping targets are also in remote copy relationships there may be node warmstarts with a temporary loss of access to data (HU01921) - FlashCopy, Global Mirror, Metro Mirror Migrating extents to an MDisk, that is not a member of an MDisk group, may result in a Tier 2 recovery (HU01924) - Thin Provisioning When a GMCV relationship is stopped with the -access option, and the secondary volume is immediately deleted with -force, then all nodes may repeatedly warmstart (HU01970) - Global Mirror With Change Volumes An issue in the background copy process prevents grains, above a 128TB limit, from being cleaned properly. As a consequence there may be multiple node warmstarts with the potential for a loss of access to data (HU02005) - Global Mirror, Global Mirror With Change Volumes, Metro Mirror The event log handler maintains a second list of events. On rare occasions, for log full events, these lists can get out of step, resulting in a Tier 2 recovery (HU02054) - System Monitoring Mishandling of DRP allocation request rejections can lead to node warmstarts that can take an MDisk group offline (HU02065) - Data Reduction Pools When a SCSI command, containing an invalid byte, is received there may be a node warmstart. This can affect both nodes, in an IO group, at the same time (HU02069) - Hosts Workloads, with data that is highly suited to deduplication, can provoke high CPU utilisation, as multiple destinations try to dedupe to one source. This adversely impacts performance with the possibility of offline MDisk groups (HU02097) - Data Reduction Pools Multiple node warmstarts, in quick succession, can cause the partner node to lease expire (HU02106) - Quorum, IP Quorum Deleting a managed disk group, with -force, may cause multiple warmstarts with the possibility of a loss of access to data (HU02108) - Data Reduction Pools Free extents may not be unmapped after volume deletion, or migration, resulting in out-of-space conditions on backend controllers (HU02109) - Backend Storage, SCSI Unmap Removing multiple IQNs for an iSCSI host can result in a Tier 2 recovery (HU02135) - iSCSI An issue in DRP garbage collection can cause IO timeouts leading to an offline pool (HU02138) - Data Reduction Pools An issue in the max replication delay function may trigger a Tier 2 recovery, after posting multiple 1920 errors in the Event Log (HU02141) - Global Mirror If a node is rebooted, when remote support is enabled, then all other nodes will warmstart (HU02154) - Support Remote Assist Upgrading to v8.2.1 may result in offline managed disk groups and OOS events (1685/1687) appearing in the Event Log (HU02155) - Data Reduction Pools Bulk volume removals can adversely impact related FlashCopy mappings leading to a Tier 2 recovery (HU02197) - FlashCopy Incremental FlashCopy targets can be corrupted when the FlashCopy source is a target of a remote copy relationship (HU02205) - FlashCopy, Global Mirror, Global Mirror with Change Volumes, Metro Mirror Remote Copy secondary may have inconsistent data following a stop with -access due to a missing bitmap merge from FlashCopy to Remote Copy (HU02212) - Global Mirror with Change Volumes, HyperSwap A T2 recovery may occur when an attempt is made to upgrade, or downgrade, the firmware for an unsupported drive type (IT25367) - Drives After a manual power off and on, of a system, both nodes, in an IO group, may repeatedly assert into a service state (IT31113) - RAID An issue in the way Global Mirror handles write sequence numbers >512 may cause multiple node warmstarts (HU01923) - Global Mirror When a DRP is running low on free space, the credit allocation algorithm, for garbage collection, can be exposed to a race condition, adversely affecting performance (HU02080) - Data Reduction Pools Upgrading FCM firmware on multiple IO group systems can cause a drive to become stuck at 0% sync with the corresponding array in a 'syncing' state (HU02114) - Drives For direct-attached hosts, a race condition between the FLOGI and Link UP processes can result in FC ports not coming online (HU02123) - Hosts In the event of unexpected power loss a node may not save system data (HU02168) - Reliability Availability Serviceability - (Fix) The following issues have been resolved with this release: Error 1046 not triggering a Call Home even though it is a hardware fault (HU02190) - System Monitoring The mishandling of performance stats may occasionally result in some entries being overwritten (HU01238) - System Monitoring After deleting an encrypted external MDisk, it is possible for the 'encrypted' status of volumes to change to 'no', even though all remaining MDisks are encrypted (HU01868) - Encryption Chrome browser support requires a self-signed certificate to include subject alternate name (HU01917) - Graphical User Interface lsdrive will show a blank for the write endurance value instead of 0% for new read-intensive SSDs (HU01956) - Drives Some read-intensive SSDs are incorrectly reporting wear rate thresholds generating unnecessary errors in the Event Log (HU02015) - Drives Upgrading to v8.2.1.8, or later, may result in a licensing error in the Event Log (HU02091) - Licensing The system management firmware may, incorrectly, attempt to obtain an IP address, using DHCP, making it accessible via Etherne (HU02103) An issue with how DRP handles data, at the sub-extent level, may result in a node warmstart (HU02111) - Data Reduction Pools Due to an issue with FCM thin provisioning calculations the GUI may incorrectly display volume capacity and capacity savings as zero (HU02124) - System Monitoring An issue with support for target resets in Nimble Storage controllers may cause a node warmstart (HU02137) - Backend Storage An issue in the way inter-node comunication is handled can lead to a node warmstart (HU02183) - Reliability Availability Serviceability - [Important] The following issues have been resolved with this release: The performance profile, for some enterprise tier drives, may not correctly match the drives capabilities leading to that tier being overdriven ( HU02143 ) - EasyTier - [Important] The following issues have been resolved with this release: An issue in the RAID component, in the presence of very high IO workload and the exhaustion of cache resources, can see a deadlock condition occurring which prevents further I/O processing. The system detects this issue and takes the storage pool offline for a six minute period, to clear the problem. The pool is then brought online automatically, and normal operation resumes ( HU02104 ) - RAID - (Fix) The following issues have been resolved with this release: Excessive processing time required for FlashCopy bitmap operations, associated with large (>20TB) Global Mirror change volumes, may lead to a node warmstart ( HU02102 ) - Global Mirror With Change Volumes Excessive SSH connections may trigger a singe node warmstart on the configuration node ( HU02126 ) - Command Line Interface - [Important] The following issues have been resolved with this release: When IO, in remote copy relationships, experiences delays (1720 and/or 1920 errors are logged) an IO group may warmstart ( HU01967 ) - Global Mirror, Global Mirror with Change Volumes, Metro Mirror Active EasyTier migrations can block CLI initiated migrations resulting in a T2 recovery ( HU02036 ) - EasyTier Multiple DRAID arrays can, where one is performing a rebuild, be exposed to a RAID deadlock condition resulting in multiple node warmstarts and a loss of access to data ( HU02044 ) - Distributed RAID HyperSwap clusters with only two surviving nodes may experience warmstarts on both of those nodes where rcbuffersize is set to 512MB ( HU02063 ) - HyperSwap Upgrading to 8.2.1.6 or 8.3.0.0 can cause a loss of access to direct-attached Fibre Channel controllers ( HU02077 ) - Backend Storage Due to changes to quorum management, during an upgrade to v8.2.x, or later, there may be multiple warmstarts, with the possibility of a loss of access to data ( HU02089 ) - System Update Starting a relationship, when the remote volume is offline, may result in a T2 recovery ( IT2625 ) - HyperSwap A resource shortage in the RAID component can cause mdisks to be taken offline ( IT30595 ) - RAID When an auxiliary volume is moved an issue with pausing the master volume can lead to node warmstarts  ( HU01836 ) - HyperSwap GUI session handling has an issue that can generate many exceptions, adversely impacting GUI performance ( HU02049 ) - Graphical User Interface - (Fix) The following issues have been resolved with this release: When a write, to a secondary volume, becomes stalled, a node at the primary site may warmstart ( HU01880 ) - Global Mirror, Global Mirror with Change Volumes, Metro Mirror When shrinking a volume, that has host mappings, there may be recurring node warmstarts ( HU01936 ) - Cache Disabling garbage collection may cause a node warmstart ( HU02021 ) - Data Reduction Pools Freeze time of Global Mirror remote copy consistency groups may not be updated correctly in certain scenarios ( HU02085 ) - Global Mirror If an IP Quorum app is killed, during the commit phase of a code upgrade, then that offline IP Quorum device cannot be removed, post upgrade ( IT30448 ) - IP Quorum Attempting to activate USB encryption on a new V5030E will fail with a CMMVCU6054E error ( IT30449 ) - Encryption - [Important] The following issues have been resolved with this release: An issue with restore mappings, in the FlashCopy component, can cause an IO group to warmstart ( HU01888 & HU01997 ) - FlashCopy Under rare circumstances the DRP deduplication rehoming process can become truncated. Subsequent detection of inconsistent metadata can lead to offline Data Reduction Pools  ( HU01933 ) - Data Reduction Pools, Deduplication As a consequence of a DRP recovery bad metadata may be created. When the region of disk associated with the bad metadata is accessed there may be an IO group warmstarts  ( HU01985 ) - Data Reduction Pools For large drives, bitmap scanning, during a rebuild, can timeout resulting in multiple node asserts, possibly leading to offline IO groups  ( HU01989 ) - Distributed RAID Fabric congestion can cause internal resource constraints, in 16Gb HBAs, leading to lease expiries ( HU02027 ) - Reliability Availability Serviceability Collecting a snap can cause nodes to run out of boot drive space and go offline with node error 565 ( HU02043 ) - Support Data Collection When a node is removed from the cluster, using CLI, it may still be shown as online in the GUI. If an attempt is made to shutdown this node, from the GUI, whilst it appears to be online, then the whole cluster will shutdown  ( HU02045 ) - Graphical User Interface FlashCopy mappings, from master volume to primary change volume, may become stalled when a T2 recovery occurs whilst the mappings are in a 'copying' state  ( HU01890 ) - Global Mirror with Change Volumes Under certain IO workloads the garbage collection process can adversely impact volume write response times (show details) ( HU02012 ) - Data Reduction Pools A FlashCopy consistency group, with a mix of mappings in different states, cannot be stopped(show details) ( HU02037 ) - FlashCopy Creating a FlashCopy snapshot, in the GUI, does not set the same preferred node for both source and target volumes. This may adversely impact performance (show details) ( HU02055 ) - FlashCopy - (Fix) The following issues have been resolved with this release: A node hardware issue can cause a CLI command to timeout resulting in a node warmstart  ( HU01843 ) - Command Line Interface LUNs of greater than 2TB, presented by HP XP7 storage controllers, are not supported  ( HU01892 ) - Backend Storage With all Remote Support Assistant connections closed, the GUI may show that a connection is still in progress  ( HU01974 ) - System Monitoring Unable to create HyperSwap volumes. The mkvolume command fails with CMMVC7050E error  ( HU01978 ) - HyperSwap The figure for used_virtualization, in the output of a lslicense command, may be unexpectedly large ( HU01979 ) - Command Line Interface In an environment, with multiple IP Quorum servers, if the quorum component encounters a duplicate UID then a node may warmstart  ( HU01982 ) - IP Quorum Improve debug data capture to assist in determining the reason for a Data Reduction Pool to be taken offline  ( HU01983 ) - Data Reduction Pools An accounting issue in the FlashCopy component may cause node warmstarts  ( HU01986 ) - FlashCopy An issue in the handling of extent allocation, in the DRP component, can cause a node warmstart  ( HU01991 ) - Data Reduction Pools All SCSI command types can set volumes as busy resulting in IO timeouts and a node warmstart  ( HU01998 ) - Hosts An internal hardware bus, running at the incorrect speed, may give rise to spurious DIMM over-temperature errors  ( HU02020 ) - Reliability Availability Serviceability An issue with the SSMTP process may result in failed callhome, inventory reporting and user notifications. A testemail command will fail with a CMMVC9051E error  ( HU02029 ) - System Monitoring An issue in the management steps of DRP recovery may lead to a node warmstart  ( HU02039 ) - Data Reduction Pools - [Important] The following issues have been resolved with this release: During volume migration an issue, in the handling of old to new extents transfer, can lead to cluster-wide warmstarts (HU02007) - Storage Virtualisation Systems which are using DRP, with the maximum possible extent size of 8GB, and which experience a very specific IO workload, may experience an issue due to garbage collection. This can cause repeated node warmstarts and loss of access to data (HU02009) - Data Reduction Pools When a node warmstart occurs on a system using Data Reduction Pools, there is a small possibility that the node will not automatically return online. If the partner node is also offline, this can cause temporary loss of access to data (HU02011) - Data Reduction Pools Under certain IO workloads the garbage collection process can adversely impact volume write response times (HU02012) - Data Reduction Pools - [Important] The following issues have been resolved with this release: When creating a HyperSwap relationship, using addvolumecopy (or similar methods), the system should perform a synchronisation operation to copy the data from the original copy to the new copy. In some rare cases this synchronisation is skipped, leaving the new copy with bad data (all zeros) (HU01865) - HyperSwap Executing a command, that can result in a shared mapping being created or destroyed, for an individual host, in a host cluster, without that command applying to all hosts in the host cluster, may lead to multiple node warmstarts with the possibility of a T2 recovery (HU01900) - Host Cluster When FlashCopy mappings are created, with a grain size of 64KB, it is possible for an overflow condition in the bitmap to occur. This can resulting in multiple node warmstarts with a possible loss of access to data (HU01910) - FlashCopy When two IOs attempt to access the same address, the state of the data may be incorrectly set to invalid causing offline volumes and, possibly, offline pools (HU01928) - Data Reduction Pools Data Reduction Pools may go offline due to a timing issue in metadata handling (HU02000) - Data Reduction Pools The Unmap function can leave volume extents, that have not been freed, preventing managed disk and pool removal (HU01886) - SCSI Unmap After upgrading the system to v8.2, or later, when expanding a mirrored volume, the formatting of additional space may become stalled (HU01941) - Volume Mirroring When an array is in a quiescing state, for example where a member has been deleted, IO may become pended leading to multiple warmstarts (HU01972) - RAID, Distributed RAID - (Fix) The following issues have been resolved with this release: Single node warmstart due to an accounting issue within the cache component (HU00744) - Cache During garbage collection the flushing of extents may become stuck leading to a timeout and a single node warmstart (HU01860) - Data Reduction Pools Volume copy deletion, in a DRP, triggered by rmvdiskcopy rmvolumecopy or addvdiskcopy -autodelete (or similar) may become stalled with the copy being left in "deleting" status (HU01869) - Data Reduction Pools Systems with iSCSI-attached controllers may see node warmstarts due to I/O request timeouts (HU01912) - Backend Storage Systems, with encryption enabled, that are using key servers to manage encryption keys, may fail to connect to the key servers if the servers' SSL certificates are part of a chain of trust (HU01915) - Encryption An timing window issue in the Thin Provisioning component can cause a node warmstart (HU01959) - FlashCopy, Thin Provisioning A hardware issue can provoke the system to repeatedly try to collect a statesave, from the enclosure management firmware, causing 1048 errors in the Event Log (HU01961) - System Monitoring When Call Home servers return an invalid message it can be incorrectly reported as an error 3201 in the Event Log (HU01962) - System Monitoring A new mdisk array may not be encrypted even though encryption is enabled on the system (HU01976) - Encryption During a system upgrade an issue in callhome may cause a node warmstart stalling the upgrade (HU02001) - System Monitoring Timing window issue in the DRP rehoming component can cause a single node warmstart (IT28433) - Data Reduction Pools Email alerts will not work where the mail server does not allow unqualified client host names (IT28728) - System Monitoring - [Important] The following issues have been resolved with this release: When a rmvdisk command initiates a DRP rehoming process any IO to the removed volume may cause multiple warmstarts leading to a loss of access. (HU01932) - Data Reduction Pools, Deduplication - [Important] The following issues have been resolved with this release: Where Data Reduction Pools have been created on earlier code levels, upgrading the system, to an affected release, can cause an increase in the level of concurrent flushing to disk. This may result in a loss of access to data (HU01918) - Data Reduction Pools - (New) The following new features have been introduced in 8.2.1.0 release: NVMe over Fibre Channel support on 16Gb Fibre Channel adapters Full IP-based quorum Increased maximum host mappings to 64K Support for Storwize V7000 Next Gen systems Single copy volume expansion with format iSER support for host attachment with 25 GbE adapters Clustering support over Ethernet using RDMA Support for Gemalto SafeNet KeySecure Write Cache optimization for Data Reduction Pools. (Note: This means that the write cache fullness for Data Reduction Pools will be lower than for standard pools) Write throughput enhancements for San Volume Controller SV1 nodes, FlashSystem 9100 and Storwize V7000 Next Gen - [Important] The following issues have been resolved with this release: Timing window issue can affect operation of the HyperSwap addvolumecopy command causing all nodes to warmstart (HU01799) - HyperSwap Invoking a chrcrelationship command when one of the relationships, in a consistency group, is running in the opposite direction to the others may cause a node warmstart followed by a T2 recovery (HU01825) - FlashCopy If both nodes, in an IO group, start up together a timing window issue may occur, that would prevent them running garbage collection, leading to a related DRP running out of space (HU01833) - Data Reduction Pools If the execution of a rmvdisk -force command, for the FlashCopy target volume in a GMCV relationship, coincides with the start of a GMCV cycle all nodes may warmstart (HU01845) - Global Mirror With Change Volumes Clusters using Data Reduction Pools can experience multiple warmstarts on all nodes putting them in a service state (HU01855) - Data Reduction Pools Where systems are connected to controllers, that have FC ports that are capable of acting as initiators and targets, when NPIV is enabled then node warmstarts can occur (HU01876) - Backend Storage During an upgrade, from v7.8.1 or earlier to v8.1.3 or later, if a mdisk goes offline then, at completion, all volumes may go offline (HU01878) - System Update Until the initial synchronisation process completes, high system latency may be experienced when a volume is created with two compressed copies or when space-efficient copy is added to a volume with an existing compressed copy (HU01507) - Volume Mirroring In systems, where a VVols metadata volume has been created, an upgrade to v8.1.3 or later will cause a node warmstart, stalling the upgrade (HU01837) - VVols Bursts of IO to Samsung Read-Intensive Drives can be interpreted as dropped frames against the resident slots leading to redundant drives being incorrectly failed (HU01842) - Drives Config node processes may consume all available memory, leading to node warmstarts. This can be caused, for example, by large numbers of concurrent SSH connections being opened (HU01883) - Deduplication - (Fix) The following issues have been resolved with this release: Increasing mirror memory may fail with error CMMVC6399E (HU01510) - Volume Mirroring Creation and distribution of the config file may cause an out-of-memory condition, leading to a node warmstart (HU01832) In rare circumstances, a drive replacement may result in a "ghost drive" (i.e. a drive with the same ID as the replaced drive stuck in a permanently offline state) (HU01863) - Drives An issue with bitmap synchronisation can lead to a node warmstart (HU01871) - Data Reduction Pools Latency induced by DWDM inter-site links may result in a node warmstart (HU01879) Attempting to remove a copy of a volume, which has at least one image mode copy and at least one thin/compressed copy, in a Data Reduction Pool will always fail with a CMMVC8971E error (IT25457) - Data Reduction Pools After a FlashCopy consistency group is started a node may warmstart (IT25970) - FlashCopy ------------------------------------------------------------------------------- INSTALLATION INSTRUCTIONS ------------------------------------------------------------------------------- Review the procedures on how to update the Lenovo Storage V3700 V2 / V5030 Series systems under the "Updating the system" section of the online documentation. Obtaining the software packages Each update requires that you run the update test utility and then download the correct software package. Specific steps for these two processes are described in the topics that follow. Update Test Utility The update test utility indicates whether your current system has issues that need to be resolved before you update to the next level. The test utility is run as part of the system update process or for drive firmware. The most current version of this tool or the system software packages can be downloaded from the following support website: http://datacentersupport.lenovo.com/tw/en/products/storage/lenovo-storage/v3700v2/6535 or http://datacentersupport.lenovo.com/tw/en/products/storage/lenovo-storage/v5030/6536 After you download the update test utility, you have the following options: 1. If you are using the, select Settings>System> Update system and click Update to run the test utility. Complete directions are included in Updating the system automatically. 2. If you are using the command-line interface (CLI), directions are included in Updating the system automatically using the CLI. 3. If you are using the manual update procedure, see Updating the system manually. 4. To check drive firmware levels either by using the or the CLI, follow the directions in drive firmware package. 1. Updating the System Automatically You can update the entire system in a coordinated process with no user intervention after the update is initiated. Before you update your system, review all of the topics in "Updating" section of the documentation to understand how the process works. Allow adequate time, such as up to a week in some cases to look for potential problems or known bugs. Additional information on updating the system is available at the following website: http://datacentersupport.lenovo.com/tw/en/products/storage/lenovo-storage/v3700v2/6535 or http://datacentersupport.lenovo.com/tw/en/products/storage/lenovo-storage/v5030/6536 When the system detects that the hardware is not running at the expected level, the system applies the correct firmware. If you want to update without host I/O, shut down all hosts before you start the update. Complete the following steps to update the system automatically: 1. In the management GUI, select Settings > System > Update System. 2. Click Update. 3. Select the test utility and the code package that you downloaded from the support site. The test utility verifies that the system is ready to be updated. 4. Click Update. As the canisters on the system are updated, the displays the progress for each canister. Monitor the update information in the to determine when the process is complete. If the process stalls during the update, click Resume to continue with the update or click Cancel to abandon the update and restore the previous level of code. You can also use the CLI command applysoftware -resume to resume the update. 2. Updating the System Automatically using the CLI You can use the command-line interface (CLI) to install updates. Start here to update to a later version. Important: Before you start an update, you must check for offline or degraded volumes. An offline volume can cause write data that was modified to be pinned in the system cache. This action prevents volume failover and causes a loss of input/output (I/O) access during the update. If the fast_write_state is empty, a volume can be offline and not cause errors during the update. To update the system, follow these steps. 1. Download, install, and run the latest version of the test utility to verify that there are no issues with the current system. You can download the most current version of this test utility tool and software package at the following website: http://datacentersupport.lenovo.com/tw/en/products/storage/lenovo-storage/v3700v2/6535 or http://datacentersupport.lenovo.com/tw/en/products/storage/lenovo-storage/v5030/6536 2. Use PuTTY scp (pscp) to copy the update files to the node. 3. Ensure that the update file was successfully copied. Before you begin the update, you must be aware of the following situations: * The installation process fails under the following conditions: - If the code that is installed on the remote system is not compatible with the new code or if an intersystem communication error does not allow the system to check that the code is compatible. - If any node in the system has a hardware type that is not supported by the new code. - If the system determines that one or more volumes in the system would be taken offline by rebooting the nodes as part of the update process. You can find details about which volumes would be affected by using the lsdependentvdisks command. If you are prepared to lose access to data during the update, you can use the force flag to override this restriction. * The update is distributed to all the nodes in the system by using internal connections between the nodes. * Nodes are updated one at a time. * Nodes run the new code concurrently with normal system activity. * While the node is updated, it does not participate in I/O activity in the I/O group. As a result, all I/O activity for the volumes in the I/O group is directed to the other node in the I/O group by the host multipathing software. * There is a thirty-minute delay between node updates. The delay allows time for the host multipathing software to rediscover paths to the nodes that are updated. There is no loss of access when another node in the I/O group is updated. * The update is not committed until all nodes in the system are successfully updated to the new code level. If all nodes successfully restart with the new code level, the new level is committed. When the new level is committed, the system vital product data (VPD) is updated to reflect the new code level. * Wait until all member nodes are updated and the update is committed before you invoke the new functions of the updated code. * Because the update process takes some time, the installation command completes as soon as the code level is verified by the system. To determine when the update is completed, you must either display the code level in the system VPD or look for the Software update complete event in the error/event log. If any node fails to restart with the new code level or fails at any other time during the process, the code level is backed off. * During an update, the version number of each node is updated when the code is installed and the node is restarted. The system code version number is updated when the new code level is committed. * When the update starts, an entry is made in the error or event log and another entry is made when the update completes or fails. 4. Issue this CLI command to start the update process: applysoftware -file software_update_file Where software_update_file is the name of the code update file in the directory you copied the file to in step 2. If the system identifies any volumes that would go offline as a result of rebooting the nodes as part of the system update, the code update does not start. An optional force parameter can be used to indicate that the update continues regardless of the problem identified. If you use the force parameter, you are prompted to confirm that you want to continue. The behavior of the force parameter changed, and it is no longer required when you apply an update to a system with errors in the event log. 5. Issue the following CLI command to check the status of the code update process: lsupdate This command displays success when the update is complete. Note: If a status of stalled_non_redundant is displayed, proceeding with the remaining set of node updates might result in offline volumes. Contact a service representative to complete the update. 6. To verify that the update successfully completed, issue the lsnodecanistervpd CLI command for each node that is in the system. The code version field displays the new code level. When a new code level is applied, it is automatically installed on all the nodes that are in the system. Note: An automatic system update can take up to 30 minutes per node. 3. Updating the System Manually During an automatic update procedure, the system updates each of the canisters systematically. The automatic method is the preferred procedure for updating the code on the canisters; however, to provide more flexibility in the update process, you can also update each canister manually. During this manual procedure, you prepare the update, remove a canister from the system, update the code on the canister, and return the canister to the system. You repeat this process for the remaining canisters until the last canister is removed from the system. Every canister must be updated to the same code level. You cannot interrupt the update and switch to installing a different level. After all the canisters are updated, you must confirm the update to complete the process. The confirmation restarts each canister in order and takes about 30 minutes to complete. Prerequisites Start here to update to a later version. Before you begin to update nodes manually, ensure that the following requirements are met: - The latest update test utility was downloaded to your . - The latest system update package was downloaded to your . - All node canisters are online. - Errors in the system event log are addressed and marked as fixed. - There are no volumes, MDisks, or storage systems with Degraded or Offline status. - The service assistant IP is configured to every node in the system. - The system superuser password is known. - The current system configuration was backed up and saved. - You have physical access to the hardware. The following actions are not required; they are suggestions. - Stop all MetroMirror, Global Mirror, or HyperSwap operations during the update procedure. - Avoid running FlashCopy operations during this procedure. - Avoid migrating or formatting volumes during this procedure. - Stop collecting performance data for the system. - Stop any automated jobs that access the system before you update. - Ensure that no other processes are running on the system before you update. - If you want to update without host I/O, shut down all hosts before you start the update. Preparing to update the system The procedure to prepare for an update is run once for each system. To prepare the system for an update, follow these steps: 1. In the managment GUI, select Settings > System > Update System. The system automatically checks for updates and lists the current level. 2. Click Update. 3. Select the test utility and update package that you downloaded. Enter the code level that you are updating to, such as 7.6.1.3. 4. Click Update. Wait for the files to upload. 5. Select the type of update and click Finish. The test utility runs automatically and identifies any issues that it finds. Fix all problems before you proceed to step 6. 6. When all issues are resolved, click Resume. The system is ready for a manual update when the status shows Prepared. Preparing to update individual nodes Before you update nodes individually, ensure that the system is ready for the update. Before you begin; Verify the prerequisites listed above. After you verify that the prerequisites for a manual update are met, follow these steps: 1. Use the management GUI to display the nodes in the system and record this information. For all the nodes in the system, verify the following information: - Confirm that both canisters are online. - Identify which canister is acting as the configuration node. - Record the service IP address for each canister. 2. If you are using the management GUI, view External Storage to ensure that everything is online and also verify that internal storage is present. 3. If you are using the command-line interface, submit this command for each storage system: lscontroller controller_name_or_controller_id where controller_name_or_controller_id is the name or ID of the storage system. Confirm that each storage system has degraded=no status. 4. Verify that all hosts have all paths available to all the volumes that are presented to them by the system. Ensure that the multipathing driver is fully redundant, with every path available and online. 5. If you did not do so previously, download the installation package for the level that you want to install. You can download the most current package from the following website: http://support.lenovo.com/us/en/products/servers/lenovo-storage/v7000 Updating all nodes except the configuration node When you are updating nodes individually, before you update the configuration node, you must update all of the non-configuration nodes in the clustered system. Before you update all the non-configuration nodes on the system, you must record each name for each node on the system. To view the node name for each node in the , select Setting > Network > iSCSI. To update a non-configuration node canister, follow these steps: 1. Ensure that all hosts have all paths available to volumes that are presented to them by the system. If there are any unavailable paths, wait up to 30 minutes, and check again. If any path is still not available, investigate and resolve the connection problem before you start the code update. Ensure that the multi-pathing driver is fully redundant, with every path available, and online. During the update, you might see multi-pathing driver errors that are related to paths that are going away, and to the increased multi-pathing driver error count. 2. In the management GUI, check that no incomplete volume synchronization tasks are in progress. In the status bars that are at the bottom of the page, expand Running Tasks to display the progress of actions. Ensure that all synchronization tasks are complete before you remove the node. 3. In the management GUI, use the dynamic image of the system to view the canisters. Right-click the canister that you want to remove and select Remove. 4. Open a web browser and type https://service_ip in the address field, where service IP is the service IP address for the node that you deleted. The service assistant login page displays. 5. Verify that the node is no longer a member of the system by checking that the node status, as shown in the display, is service. The node has an error code of 690. The removed node is no longer visible to the system. If the node status is active, you are probably connected to the wrong node. 6. On the service assistant home page, select Update manually from the left menu. Attention: Each node must run the same code version; nodes with different versions are not compatible. 7. Select the correct update package, and click Update. The node restarts and updates. Access to the service assistant is lost while the node restarts, but you can still access the service assistant from a different node. 8. Eventually the node canister that you removed and updated automatically rejoins the system. When the canister is online, go to step 10. 9. After the all the nodes except configuration node are updated and added back to the system, rename the node canister to the name it had before it was removed and updated. In the , select Setting > Network > iSCSI. 10. If you have any remaining nodes to update that are not configuration nodes, repeat this task for the next non-configuration node that is yet updated, starting at step 1. Updating the configuration node After all the other nodes are updated on the system, you can update the configuration node. To update the configuration node, follow these steps: 1. Ensure that all hosts have all paths available to volumes that are mapped to those hosts. If not, wait up to 30 minutes and repeat this check. If some paths are still unavailable, investigate and resolve these connection problems before you continue the system code update. 2. Before you update the configuration node on the system, you must record the name of node. To view the node name in the management GUI, select Setting > Network > iSCSI. 3. In the management GUI, check that there are no incomplete volume synchronization tasks in progress. Click Running Tasks. 4. Remove the configuration node from the system. Note: When the configuration node is removed from the system, the SSH connection to the system closes. 5. Open a web browser and type http://service_assistant_ip in the address field. The service assistant IP address is the IP address for the service assistant on the node that was deleted. 6. On the service assistant home page, click Exit service state and press Go. The node is automatically added to the system. Because the process of adding the node automatically updates the code on the node, it takes some time before the node is fully online. This action automatically updates the code on this last node, which was the configuration node. 7. After the configuration node is updated and added back to the system, rename the node canister to the name it had before it was removed and updated. In the , select Setting > Network > iSCSI. Completing the update After the configuration node is successfully rebooted and updated, verify the update and return the system to its original state by following these steps. 1. Confirm the update: a. Enter the lsupdate command to determine if the update requires a further completion step. b. If the lsupdate command shows that the status is system_completion_required, enter svctask applysoftware -complete in the command-line interface. Each canister is restarted in order. The update process takes approximately 30 minutes with 5 minutes per node. During the confirmation step, the system is operational but no other updates can be started until the current update is confirmed. 2. Verify that the system is running at the correct version and that no other errors in the system must be resolved. 3. Verify that all the nodes are online. 4. Verify that all volumes are online. In the , select Volumes > Volumes. 5. Verify that all managed disks (MDisks) are online. In the , select Pools > MDisks by Pools. 6. Restart any services, advanced functions, or scripts that were stopped before the update. You completed the manual update. ------------------------------------------------------------------------------- Limitations and considerations ------------------------------------------------------------------------------- Customers with more than 5 x non-NVMe over FC hosts (i.e FC SCSI or iSCSI) in an IO group must not attach any NVMe over FC hosts to that IO group. Customers with more than 20 x non-NVMe over FC hosts (i.e FC SCSI or iSCSI) in a cluster must not attach any NVMe over FC hosts to that cluster. For new clusters without any hosts please refer to the appropriate V8.2.1 Configuration Limits and Restrictions pages for details of the maximum number of hosts that can be attached. These limits will not be policed by the Spectrum Virtualize software. Any configurations that exceed these limits will experience significant adverse performance impact. These limits will be lifted in a future major release. ------------------------------------------------------------------------------- Customers using Transparent Cloud Tiering should not upgrade to v8.2.1.0. This is a restriction that may be lifted in a future PTF. ------------------------------------------------------------------------------- Spectrum Virtualize for Public Cloud v8.2.1 is not available. ----------------------------------------------------------------------------- With Gemalto SafeNet KeySecure, the chkeyserverkeysecure -username command is used to set the KeySecure username credential. If this is changed to a username that is not recognised by the key server to be the valid username, associated with the Spectrum Virtualize encryption key, then a subsequent re-key operation can cause key servers to appear offline. This is a an issue that will be resolved in a future PTF. ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- TRADEMARKS ----------------------------------------------------------------------------- * The following are registered trademarks of Lenovo. Lenovo The Lenovo Logo ThinkServer * Intel is a registered trademark of Intel Corporation. * Microsoft and Windows are registered trademarks of Microsoft Corporation. * IBM is a registered trademark of International Business Machines Other company, product, and service names may be registered trademarks, trademarks or service marks of others. LENOVO PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. Lenovo may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. BY FURNISHING THIS DOCUMENT, LENOVO GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. (C) Copyright Lenovo 2001-2022. All rights reserved.