Patch-ID# 102670-02 Keywords: cvm pdb sparccluster cluster vm volume manager Synopsis: Volume Manager 2.1: Cluster Volume Manager 1.0 Jumbo Patch Date: Feb/13/96 Solaris Release: 2.4 SunOS Release: 5.4 Unbundled Product: Volume Manager (Cluster Volume Manger 1.0) Unbundled Release: 2.1 Relevant Architectures: sparc BugId's fixed with this patch: 1196053 1228329 1230355 1224779 Changes incorporated in this version: 1228329 1230355 1224779 Patches accumulated and obsoleted by this patch: Patches which conflict with this patch: Patches required with this patch: 102432-08 Obsoleted by: Files included with this patch: /kernel/drv/vxio /kernel/drv/vxio.conf /kernel/drv/vxspec /kernel/drv/vxspec.conf /usr/lib/vxvm/bin/vxreattach /sbin/vxconfigd Problem Description: 1196053/1228329: An array is pulled offline (or a tray is pulled) and Oracle is running against the database, DBWR sometimes goes into an IO-wait hang. If another process trys to access that volume it will also io-wait hang. With this patch installed the pdb system can recover from a SSA being powered off, a fiber cable being removed, and although not supported by the SSA product a disk tray being pulled. During anyone of the above conditions the plexes that are associated with physical disks in the offline disk component will be deattached from their volume by the CVM. Assuming the volume is mirrored (a pdb requirement), no loss of data or database availability will occur. Recovery from this problem once the plexes have been deattached can be accomplished in two ways. We recommend the first procedure because it involves less instructions and since only the volumes that have been recently touched will need to be resynced, recovery time will be less. In the case of the second procedure all of the volumes will be marked as dirty and thus all volumes will need to be recovered: 1. Power on the SSA, reconnect the fiber cable, push the disk tray back in or otherwise repair the SSA/disk. 2. Wait for all the disks in the SSA to spin up and become highlighted in the SSA led. 3. Kill or otherwise stop all the workload process on one node. 4. Stop Oracle on the same node. 5. Stop PDB on the same node. 6. The surviving node will reconfigure, reattach the plexes, and resync the volumes automatically. 7. When resyncing has completed, restart pdb and then oracle on the other node. At this point both nodes should be up and running. A second way to recover would be to: 1. Kill or otherwise stop all the workload process on on node. 2. Stop Oracle on the same node. 3. Stop PDB on the same node. 4. Once the surviving node has reconfigured, power on the SSA, reconnect the fiber cable, push in the disk tray or otherwise repair the SSA. 5. Wait for all the disks in the SSA to spin up and become highlighted in the SSA led. 6. Make sure that all the disk devices in the affected SSA have an 'online' status displayed by the 'vxdisk list' command before proceeding. Use 'vxdisk online ' to reset any devices that are in a state of either 'offline' or 'error'. 7. On the surviving node issue the commands: # /etc/vx/bin/vxreattach (This may take a while!) # /usr/sbin/vxrecover (This may take a while!) 8. When resyncing has completed, restart pdb and then oracle on the other node. At this point both nodes should be up and running. 1230355: vxconfigd will no longer take any requests from other volume manager utilities for volume configuration changes or reports if the volume configuration daemon has been enable/disable/start/stop on a system more than 46 times. If the problem occurs, a user will see "cannot create portal" error message on the screen. The problem has been corrected in this patch. 1224779: This is the back port of problem fixed in the pdb 1.1 release. During a 'pdbadmin startnode' if a bad disk is found then that disk group is not imported and an error message reporting which disk is bad is reported to the user. The problem is that in many cases the disk being reported as bad is incorrect. This leads to much confusion and loss of system availability as the wrong disks are being replaced only to find the same problem appearing again this time on another disk. The problem is solved here by forcing the import of a disk group even if a bad disk is found. In this case the disk will be detached and marked as bad. The system will report that this happened but will not identify any disks. The user will need to use the 'vxdisk list' command to find out which disk is bad. Patch Installation Instructions: -------------------------------- Generic 'installpatch' and 'backoutpatch' scripts are provided within each patch package with instructions appended to this section. Other specific or unique installation instructions may also be necessary and should be described below. Special Install Instructions: ----------------------------- The system should be rebooted after the patch has been installed. Instructions to install patch using "installpatch" -------------------------------------------------- 1. Become super-user. 2. Apply the patch by typing: installpatch where is the directory containing installpatch, and is the directory containing the patch itself. Example: # cd /tmp/123456-01 # ./installpatch . 3. If any errors are reported, see "Patch Installation Errors" in the Command Descriptions section below. Rebooting the system or restarting the application after a successful patch installation is usually necessary to utilize patch. NOTE: On client server machines the patch package is NOT applied to existing clients or to the client root template space. Therefore, when appropriate, ALL CLIENT MACHINES WILL NEED THE PATCH APPLIED DIRECTLY USING THIS SAME INSTALLPATCH METHOD ON THE CLIENT. See the next section for instructions for installing a patch on a client. Instructions for installing a patch on a dataless client -------------------------------------------------------------------- 1. Before applying the patch, the following command must be executed on the server to give the client read-only, root access to the exported /usr file system so that the client can execute the pkgadd command: share -F nfs -o ro,anon=0 /export/exec//usr The command: share -F nfs -o ro,root= \ /export/exec//usr accomplishes the same goal, but only gives root access to the client specified in the command. 2. Login to the client system and become super-user. 3. Continue with step 2 in the "Instructions to install patch using installpatch" section above. Instructions for installing a patch on a diskless client -------------------------------------------------------------------- ** To install a patch on a diskless client, you may either follow the instructions for installing on a dataless client (that is, you may logon to the client and install the patch), or you may use the following instructions to install the patch while on the server. 1. Find the complete path for the root directory of the diskless client. 2. Install the patch normally, but add the command option -R to the command line. should be the completely specified. Example: # cd /tmp/123456-01 # ./installpatch -R /export/root/client1 . Instructions for backing out a patch using "backoutpatch" ----------------------------------------------------------- 1. Become super-user. 2. Change directory to /var/sadm/patch: cd /var/sadm/patch 3. Backout patch by typing: /backoutpatch where is the patch number. Example: # cd /var/sadm/patch # 123456-01/backoutpatch 123456-01 4. If any errors are reported, see "Patch Backout Errors" in the Command Descriptions section below. Instructions for backing out a patch on a dataless client ---------------------------------------------------------- 1. Give the client root access to /usr as specified in the installpatch section. 2. Logon to the client and follow backoutpatch instructions as specified above. Instructions for backing out a patch on a diskless client ----------------------------------------------------------- ** To backout a patch on a diskless client, you may either follow the instructions for backout on a dataless client (that is, you may logon to the client and backout the patch), or you may use the following instructions to backout the patch while on the server. 1. Find the complete path for the root directory of the diskless client. 2. Backout the patch normally, but add the command option -R to the command line. should be the completely specified. Example: # cd /export/root/client1/var/sadm/patch # ./123456-01/backoutpatch -R /export/root/client1 123456-01 Instructions for identifying patches installed on system: ---------------------------------------------------------- Patch packets that have been installed can be identified by using the -p option. To find out which patches are installed on a diskless client, use both the -R option and the -p option, where is the fully specified path to the client's root directory. #cd /tmp/123456-01 #./installpatch -p #./installpatch -R /export/root/client1 -p Also note that the command "showrev -p" will show the patches installed on the local machine, but will not show patches installed on clients. Command Descriptions -------------------- NAME installpatch - apply patch package to Solaris 2.x system backoutpatch - remove patch package, restore previously saved files SYNOPSIS installpatch [-udpV] [-S ] backoutpatch [-fV] [-S ] DESCRIPTION These installation and backout utilities apply only to Solaris 2.x associated patches. They do not apply to Solaris 1.x associated patches. These utilities are currently only provided with each patch package and are not included with the standard Solaris 2.x release software. OPTIONS installpatch: -u unconditional install, turns off file validation. Allows the patch to be applied even if some of the files to be patched have been modified since original installation. -d Don't back up the files to be patched. This means that the patch CANNOT BE BACKED OUT. -p Print a list of the patches currently applied -V Print script version number -S Specify an alternate service (e.g. Solaris_2.3) for patch package processing references. -R Specify an alternate package installation root. Most useful for installing patches on diskless clients while logged on to the server. backoutpatch: -f force the backout regardless of whether the patch was superseded -V print version number only -S Specify an alternate service (e.g. Solaris_2.3) for patch package processing references. -R Specify an alternate package installation root. Most useful for removing patches on diskless clients while logged on to the server. DIAGNOSTICS Patch Installation Errors: -------------------------- Error message: The prepatch script exited with return code . Installpatch is terminating. Explanation and recommended action: The prepatch script supplied with the patch exited with a return code other than 0. Run a script trace of the installpatch and find out why the prepatch had a bad return code. Fix the problem and re-run installpatch. To execute a script trace: # sh -x ./installpatch . > /tmp/patchout 2>&1 The file /tmp/patchout will list all commands executed by installpatch. You should be able to determine why your prepatch script failed by looking through the /tmp/patchout file. If you still can't determine the reason for failure, contact customer service. Error message: The postpatch script exited with return code . Backing out patch. Explanation and recommended action: The postpatch script provided with the patch exited with an error code other than 0, and the patch has not previously been applied. Installpatch will execute backoutpatch to return the system to its pre-patched state. Create a script trace of the installpatch (see above) and find out why the postpatch script failed. Correct and re-execute installpatch. If you are unable to determine why the postpatch script failed, contact customer service. Error message: The postpatch script exited with return code . Not backing out patch because this is a re-installation. The system may be in an unstable state! Installpatch is terminating. Explanation and recommended action: The postpatch script provided with the patch exited with an error code other than 0. Because this is a re-installation of a patch, installpatch will not automatically backout the patch. You may backout the patch manually using the backoutpatch command, then generate a script trace of the installpatch as described above. Find out why the postpatch failed, correct the problem, and re-install the patch. If you are unable to determine why the postpatch script failed, contact customer service. Error message: Patch has already been applied. Explanation and recommended action: This patch has already been applied to the system and no additional patch packages would be added due to a re-installation. If the patch has to be reapplied for some reason, backout the patch and then reapply it. Error message: Symbolic link in package Symbolic links can't be part of a patch. Installpatch is terminating. Explanation and recommended action: The patch was incorrectly built. Contact customer service to get a new patch. Error message: This patch is obsoleted by patch which has already been applied to this system. Patch installation is aborted. Explanation and recommended action: Occasionally, a patch is replaced by a new patch which incorporates the bug fixes in the old patch and supplies additional fixes also. At this time, the earlier patch is no longer made available to users. The second patch is said to "obsolete" the first patch. However, it is possible that some users may still have the earlier patch and try to apply it to a system on which the later patch is already applied. If the obsoleted patch were allowed to be applied, the additional fixes supplied by the later patch would no longer be available, and the system would be left in an inconsistent state. This error message indicates that the user attempted to install an obsoleted patch. There is no need to apply this patch because the later patch has already supplied the fix. Error Message: None of the packages to patch are installed on this system. Explanation and recommended action: The original packages for this patch have not been installed and therefore the patch cannot be applied. The original packages need to be installed before applying the patch. Error message: This patch is not applicable to client systems. Explanation and recommended action: The patch is only applicable to servers and standalone machines. Attempting to apply this patch to a client system will have no effect on the system. Error message: The -S and -R arguments are mutually exclusive. Explanation and recommended action: You have specified both a non-native service to patch, and a package installation root. These two arguments are mutually exclusive. If patching a non-native usr partition, the -S option should be used to patch all clients using that service. If patching a client's root partition (either native or non-native), the -R option should be used. Error message: The service cannot be found on this system. Explanation and recommended action: You have specified a non- native service to patch, but the specified service is not installed on your system. Correctly specify the service when applying the patch. Error message: The Package Install Root directory cannot be found on this system. Explanation and recommended action: You have specified a directory that is either not mounted, or does not exist on your system. Specify the directory correctly when applying the patch. Error message: The /usr/sbin/pkgadd command is not executable. Explanation and recommended action: The /usr/sbin/pkgadd command cannot be executed. The most likely cause of this is that installpatch is being run on a diskless or dataless client and the /usr file system was not exported with root access to the client. See the section above on "Instructions for installing a patch on a diskless or dataless client". Error message: packages are not proper patch packages. Explanation and recommended action: The patch directory supplied as an argument to installpatch did not contain the expected package format. Verify that the argument supplied to installpatch is correct. Error message: The following validation error was found: Explanation and recommended action: Before applying the patch, the patch application script verifies that the current versions of the files to be patched have the expected fcs checksums and attributes. If a file to be patched has been modified by the user, the user is notified of this fact. The user then has the opportunity to save the file and make a similar change to the patched version. For example, if the user has modified /etc/inet/inetd.conf and /etc/inet/inetd.conf is to be replaced by the patch, the user can save the locally modified /etc/inet/inetd.conf file and make the same modification to the new file after the patch is applied. After the user has noted all validation errors and taken the appropriate action for each one, the user should re-run installpatch using the "-u" (for "unconditional") option. This time, the patch installation will ignore validation errors and install the patch anyway. Error message: Insufficient space in /var/sadm/patch to save old files. Explanation and recommended action: There is insufficient space in the /var/sadm/patch directory to save old files. The user has two options for handling this problem: (1) generate additional disk space by deleting unneeded files, or (2) override the saving of the old files by using the "-d" (do not save) option when running installpatch. However if the user elects not to save the old versions of the files to be patched, backoutpatch CANNOT be used. One way to regain space on a system is to remove the save area for previously applied patches. Once the user has decided that it is unlikely that a patch will be backed out, the user can remove the files that were saved by installpatch. The following commands should be executed to remove the saved files for patch xxxxxx-yy: cd /var/sadm/patch/xxxxxx-yy rm -r save/* rm .oldfilessaved After these commands have been executed, patch xxxxxx-yy can no longer be backed out. Error message: Save of old files failed. Explanation and recommended action: Before applying the patch, the patch installation script uses cpio to save the old versions of the files to be patched. This error message means that the cpio failed. The output of the cpio would have been preceded this message. The user should take the appropriate action to correct the cpio failure. A common reason for failure will be insufficient disk space to save the old versions of the files. The user has two options for handling insufficient disk space: (1) generate additional disk space by deleting unneeded files, or (2) override the saving of the old files by using the "-d" option when running installpatch. However if the user elects not to save the old versions of the files to be patched, the patch CANNOT be backed out. Error message: Pkgadd of package failed with error code . See /tmp/log. for reason for failure. Explanation and recommended action: The installation of one of patch packages failed. Installpatch will backout the patch to leave the system in its pre-patched state. See the log file for the reason for failure. Correct the problem and re-apply the patch. Error message: Pkgadd of package failed with error code . Will not backout patch...patch re-installation. Warning: The system may be in an unstable state! See /tmp/log. for reason for failure. Explanation and recommended action: The installation of one of the patch packages failed. Installpatch will NOT backout the patch. You may manually backout the patch using backoutpatch, then re-apply the entire patch. Look in the log file for the reason pkgadd failed. Correct the problem and re-apply the patch. Patch Installation Messages: --------------------------- Note: the messages listed below are not necessarily considered errors as indicated in the explanations given. These messages are, however, recorded in the patch installation log for diagnostic reference. Message: Package not patched: PKG=SUNxxxx Original package not installed Explanation: One of the components of the patch would have patched a package that is not installed on your system. This is not necessarily an error. A Patch may fix a related bug for several packages. Example: suppose a patch fixes a bug in both the online-backup and fddi packages. If you had online-backup installed but didn't have fddi installed, you would get the message Package not patched: PKG=SUNWbf Original package not installed This message only indicates an error if you thought the package was installed on your system. If this is the case, take the necessary action to install the package, backout the patch (if it installed other packages) and re-install the patch. Message: Package not patched: PKG=SUNxxx ARCH=xxxxxxx VERSION=xxxxxxx Architecture mismatch Explanation: One of the components of the patch would have patched a package for an architecture different from your system. This is not necessarily an error. Any patch to one of the architecture specific packages may contain one element for each of the possible architectures. For example, Assume you are running on a sun4m. If you were to install a patch to package SUNWcar, you would see the following (or similar) messages: Package not patched: PKG=SUNWcar ARCH=sparc.sun4c VERSION=11.5.0,REV=2.0.18 Architecture mismatch Package not patched: PKG=SUNWcar ARCH=sparc.sun4d VERSION=11.5.0,REV=2.0.18 Architecture mismatch Package not patched: PKG=SUNWcar ARCH=sparc.sun4e VERSION=11.5.0,REV=2.0.18 Architecture mismatch Package not patched: PKG=SUNWcar ARCH=sparc.sun4 VERSION=11.5.0,REV=2.0.18 Architecture mismatch The only time these messages indicate an error condition is if installpatch does not correctly recognize your architecture. Message: Package not patched: PKG=SUNxxxx ARCH=xxxx VERSION=xxxxxxx Version mismatch Explanation: The version of software to which the patch is applied is not installed on your system. For example, if you were running Solaris 5.3, and you tried to install a patch against Solaris 5.2, you would see the following (or similar) message: Package not patched: PKG=SUNWcsu ARCH=sparc VERSION=10.0.2 Version mismatch This message does not necessarily indicate an error. If the version mismatch was for a package you needed patched, either get the correct patch version or install the correct package version. Then backout the patch (if necessary) and re-apply. Message: Re-installing Patch. Explanation: The patch has already been applied, but there is at least one package in the patch that could be added. For example, if you applied a patch that had both Openwindows and Answerbook components, but your system did not have Answerbook installed, the Answerbook parts of the patch would not have been applied. If, at a later time, you pkgadd Answerbook, you could re-apply the patch, and the Answerbook components of the patch would be applied to the system. Message: Installpatch Interrupted. Installpatch is terminating. Explanation: Installpatch was interrupted during execution (usually through pressing ^C). Installpatch will clean up its working files and exit. Message: Installpatch Interrupted. Backing out Patch... Explanation: Installpatch was interrupted during execution (usually through pressing ^C). Installpatch will clean up its working files, backout the patch, and exit. Patch Backout Errors: --------------------- Error message: prebackout patch exited with return code . Backoutpatch exiting. Explanation and corrective action: the prebackout script supplied with the patch exited with a return code other than 0. Generate a script trace of backoutpatch to determine why the prebackout script failed. Correct the reason for failure, and re-execute backoutpatch. Error message: postbackout patch exited with return code . Backoutpatch exiting." Explanation and corrective action: the postbackout script supplied with the patch exited with a return code other than 0. Look at the postbackout script to determine why it failed. Correct the failure and, if necessary, RE-EXECUTE THE POSTBACKOUT SCRIPT ONLY. Error message: Only one service may be defined. Explanation and corrective action: You have attempted to specify more than one service from which to backout a patch. Different services must have their patches backed out with different invocations of backoutpatch. Error message: The -S and -R arguments are mutually exclusive. Explanation and recommended action: You have specified both a non-native service to backout, and a package installation root. These two arguments are mutually exclusive. If backing out a patch from a non-native usr partition, the -S option should be used. If backing out a patch from a client's root partition (either native or non-native), the -R option should be used. Error message: The service cannot be found on this system. Explanation and recommended action: You have specified a non- native service from which to backout a patch, but the specified service is not installed on your system. Correctly specify the service when backing out the patch. Error message: Only one rootdir may be defined. Explanation and recommended action: You have specified more than one package install root using the -R option. The -R option may be used only once per invocation of backoutpatch. Error message: The directory cannot be found on this system. Explanation and recommended action: You have specified a directory using the -R option which is either not mounted, or does not exist on your system. Verify the directory name and re-backout the patch. Error message: Patch has not been successfully applied to this system. Explanation and recommended action: You have attempted to backout a patch that is not applied to this system. If you must restore previous versions of patched files, you may have to restore the original files from the initial installation CD. Error message: Patch has not been successfully applied to this system. Will remove directory Explanation and recommended action: You have attempted to back out a patch that is not applied to this system. While the patch has not been applied, a residual /var/sadm/patch/ (perhaps from an unsuccessful installpatch) directory still exists. The patch cannot be backed out. If you must restore old versions of the patched files, you may have to restore them from the initial installation CD. Error message: This patch was obsoleted by patch . Patches must be backed out in the order in which they were installed. Patch backout aborted. Explanation and recommended action: You are attempting to backout patches out of order. Patches should never be backed-out out of sequence. This could undermine the integrity of the more current patch. Error message: Patch was installed without backing up the original files. It cannot be backed out. Explanation and recommended action: Either the -d option of installpatch was set when the patch was applied, or the save area of the patch was deleted to regain space. As a result, the original files are not saved and backoutpatch cannot be used. The original files can only be recovered from the original installation CD. Error message: pkgrm of package failed return code . See /var/sadm/patch//log for reason for failure. Explanation and recommended action: The removal of one of patch packages failed. See the log file for the reason for failure. Correct the problem and run the backout script again. Error message: Restore of old files failed. Explanation and recommended action: The backout script uses the cpio command to restore the previous versions of the files that were patched. The output of the cpio command should have preceded this message. The user should take the appropriate action to correct the cpio failure. KNOWN PROBLEMS: On client server machines the patch package is NOT applied to existing clients or to the client root template space. Therefore, when appropriate, ALL CLIENT MACHINES WILL NEED THE PATCH APPLIED DIRECTLY USING THIS SAME INSTALLPATCH METHOD ON THE CLIENT. See instructions above for applying patches to a client. A bug affecting a package utility (eg. pkgadd, pkgrm, pkgchk) could affect the reliability of installpatch or backoutpatch which uses package utilities to install and backout the patch package. It is recommended that any patch that fixes package utility problems be reviewed and, if necessary, applied before other patches are applied. Such existing patches are: 100901 Solaris 2.1 101122 Solaris 2.2 101331 Solaris 2.3 SEE ALSO pkgadd, pkgchk, pkgrm, pkginfo, showrev, cpio