PFW1548: Memory and processor subsystem problem isolation procedure

This procedure is used to locate defective FRUs not found by normal diagnostics. For this procedure, diagnostics are run on a minimally configured system. If a failure is detected on the minimally configured system, the remaining FRUs are exchanged one at a time until the failing FRU is identified. If a failure is not detected, FRUs are added back until the failure occurs. The failure is then isolated to the failing FRU.

Perform the following procedure:

• PFW1548-1
  1. Insure that the diagnostics and the operating system are shut down.

     Is the system at "service processor standby", indicated by "01" in the control panel?

     NO

     Replace the system backplane, location: Un-P1. Return to step PFW1548-1.

     YES

     Continue with substep 2.

  2. Turn on the power using either the white button or the HMC.

     Does the managed system reach power on at hypervisor standby as indicated on the HMC?

     NO

     Go to PFW1548-3.

     YES

     Go to PFW1548-2.

  3. Insert the AIX diagnostic CD-ROM into the optical drive.
     Note:
     If you cannot insert the diagnostic CD-ROM, go to PFW1548-2.
  4. When the keyboard indicator is displayed on an ASCII terminal, a directly-attached keyboard, or hardware management console (HMC), press the number 5 key.
  5. If you are prompted to do so, enter the appropriate password.

     Is the Please define the System Console screen displayed?

     NO

     Go to PFW1548-2.

     YES

     Go to PFW1548-14.

• PFW1548-2

  Insert the AIX diagnostic CD-ROM into the optical drive.

  Note:
  If you cannot insert the diagnostic CD-ROM, go to step PFW1548-3.

  Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux^(R) or AIX partition by clicking on the "Advanced" button on the activation screen. On the "advanced" activation screen, select "boot in service mode using the default boot list" to boot the diagnostic CD-ROM.

  If you are prompted to do so, enter the appropriate password.

  Is the "Please define the System Console" screen displayed?

  NO

  Go to PFW1548-3.

  YES

  Go to PFW1548-20.

• PFW1548-3
  1. Turn off the power.
  2. If you have not already done so, configure the service processor (using the ASMI menus) with the instructions in note 6 at the beginning of this procedure, then return here and continue.
  3. Exit the service processor (ASMI) menus and remove the power cords.
  4. Disconnect all external cables (parallel, system port 1, system port 2, keyboard, mouse, USB devices, SPCN, Ethernet on the system planar, and so on). Also disconnect all of the external cables attached to the service processor except the Ethernet cable going to the HMC.

  Is there more than one processor drawer in the system?

  NO

  Go to PFW 1548-5.

  YES

  Go to PFW1548-4.

• PFW1548-4

  Disconnect the flex cables from the front and the back of all the processor drawers if not already disconnected.

  Does the processor drawer with the service processor card power on OK?

  NO

  Go to PFW 1548-5.

  YES

  Go to PFW1548-17.

• PFW1548-5
  1. Place the drawer into the service position and remove the service access cover.
  2. Record the slot numbers of the PCI adapters and I/O expansion cards if present. Label and record the locations of all cables attached to the adapters. Disconnect all cables attached to the adapters and remove all of the adapters.
  3. Remove the removable media enclosure assembly by pulling out the orange tabs and then sliding the media enclosure toward you approximately three centimeters.
  4. Remove processor card #2 (if installed). If processor card #2 is removed, insure that processor card #1 is installed and contains at least one quad of DIMMs.
  5. Record the slot numbers of the memory DIMMs on processor card #1. Remove all memory DIMMs except for one quad from processor card #1.
     Notes:
     1. Place the memory DIMM locking tabs in the locked (upright) position to prevent damage to the tabs.
     2. Memory DIMMs must be installed in quads and in the correct connectors. Refer to Physical Locations for information on memory DIMMs.
  6. Remove the disk drives from the disk drive enclosure assembly.
  7. Release and disconnect the disk drive enclosure assembly by using the cam levers at the lower front part of the enclosure.
  8. Plug in the power cords and wait for "01" in the upper-left-hand corner of the control panel display.
  9. Turn on the power using either the HMC or the white button.

  Does the managed system reach power on at hypervisor standby as indicated on the HMC?

  NO

  Go to PFW1548-9.

  YES

  Go to PFW1548-6.

• PFW1548-6

  Were any memory DIMMs removed from processor card #1?

  NO

  Go to PFW1548-11.

  YES

  Go to PFW1548-7.

• PFW1548-7
  1. Turn off the power, and remove the power cords.
  2. Replug the memory DIMMs that were removed from processor card #1 in PFW1548-2 in their original locations.
     Notes:
     1. Place the memory DIMM locking tabs into the locked (upright) position to prevent damage to the tabs.
     2. Memory DIMMs must be installed in quads in the correct connectors. Refer to Physical Locations for information on memory DIMMs.
  3. Plug in the power cords and wait for "01" in the upper-left corner of the control panel display.
  4. Turn on the power using either the HMC or the white button.

  Does the managed system reach power on at hypervisor standby as indicated on the HMC?

  NO

  A memory DIMM in the quad you just replaced in the system is defective. Turn off the power, remove the power cords, and exchange the memory DIMMs in that quad one at a time with new or previously removed memory DIMMs. Repeat this step until the defective memory DIMM is identified, or all memory DIMMs have been exchanged.

  If your symptom did not change and all the memory DIMMs have been exchanged, call your service support person for assistance.

  If the symptom changed, check for loose cards and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to PFW1548-8.

• PFW1548-8

  Was processor card #2 removed from the system?

  NO

  Go to PFW1548-10.

  YES

  Go to PFW1548-9.

• PFW1548-9

  One of the FRUs remaining in the system unit is defective.

  Note:
  If a memory DIMM is exchanged, insure that the new memory DIMM is the same size and speed as the original memory DIMM.
  1. Turn off the power, remove the power cords, and exchange the following FRUs in the order listed:
     1. Memory DIMMs. Exchange one at a time with new or previously removed DIMMs
     2. Processor card #1, location: Un-P2-C1.
     3. Processor backplane, location: Un-P2
     4. Power supplies, locations: Un-E1 and Un-E2.
     5. Processor regulators, locations: Un-P2-C3, Un-P2-C4, Un-P2-C5.
     6. Service processor, location: Un-P1-C11.
     7. I/O backplane, location: Un-P1.
  2. Plug in the power cords and wait for "01" in the upper-left corner of the control panel display.
  3. Turn on the power using either the HMC or the white button.

  Does the managed system reach power on at hypervisor standby as indicated on the HMC?

  NO

  Reinstall the original FRU.

  Repeat the FRU replacement steps until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to Verifying a repair.

• PFW1548-10

  No failure was detected with this configuration.

  1. Turn off the power and remove the power cords.
  2. Reinstall processor card #2.
  3. Plug in the power cords and wait for "01" in the upper-left corner of the control panel display.
  4. Turn on the power using either the HMC or the white button.

  Does the managed system reach power on at hypervisor standby as indicated on the HMC?

  NO

  One of the FRUs remaining in the system is defective.

  Exchange the FRUs (that have not already been changed) in the following order:

  1. Memory DIMMs (if present) on processor card #2. Exchange the DIMMs one at a time with new or previously removed DIMMs.
  2. Processor card #2
  3. Processor backplane, location: Un-P2.
  4. Power supplies, locations: Un-E1 and Un-E2.
  5. Processor regulators, locations: Un-P2-C3, Un-P2-C4, Un-P2-C5.
  6. Service processor, location: Un-P1-C11.
  7. I/O backplane, location: Un-P1.

  Repeat the FRU replacement steps until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to PFW1548-11.

• PFW1548-11
  1. Turn off the power.
  2. Reconnect the system console.
     Notes:
     1. If an ASCII terminal has been defined as the firmware console, attach the ASCII terminal cable to the S1 connector on the rear of the system unit.
     2. If a display attached to a display adapter has been defined as the firmware console, install the display adapter and connect the display to the adapter. Plug the keyboard and mouse into the keyboard connector on the rear of the system unit.
  3. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking on the "Advanced" button on the activation screen. On the "advanced" activation screen, select "boot in service mode using the default boot list" to boot the diagnostic CD-ROM.
  4. If the ASCII terminal or graphics display (including display adapter) is connected differently from the way it was previously, the console selection screen appears. Select a firmware console.
  5. When the word keyboard is displayed, press the number 1 key on the directly attached keyboard, an ASCII terminal or HMC. This activates the system management services (SMS).
  6. Enter the appropriate password if you are prompted to do so.

  Is the SMS screen displayed?

  NO

  One of the FRUs remaining in the system unit is defective.

  Exchange the FRUs that have not been exchanged, in the following order:

  1. If you are using an ASCII terminal, go to the problem determination procedures for the display. If you do not find a problem, do the following:
     1. Replace the service processor, location: Un-P1-C11.
     2. Replace the I/O backplane, location: Un-P1.
  2. If you are using a graphics display, go to the problem determination procedures for the display. If you do not find a problem, do the following:
     1. Replace the display adapter.
     2. Replace the backplane in which the graphics adapter is plugged.

        Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

        If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

        If the symptom changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to the next step.

• PFW1548-12
  1. Make sure the diagnostic CD-ROM is inserted into the optical drive.
  2. Turn off the power and remove the power cords.
  3. Use the cam levers to reconnect the disk drive enclosure assembly to the I/O backplane.
  4. Reconnect the removable media enclosure assembly by sliding the media enclosure toward the disk drive backplane and then pressing the orange tabs.
  5. Plug in the power cords and wait for "01" in the upper-left corner of the operator panel display.
  6. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking on the "Advanced" button on the activation screen. On the "advanced" activation screen, select "boot in service mode using the default boot list" to boot the diagnostic CD-ROM.
  7. After the word keyboard is displayed, press the number 5 key on either the directly attached keyboard or an ASCII terminal keyboard.
  8. Enter the appropriate password if you are prompted to do so.

  Is the Please define the System Console screen displayed?

  NO

  One of the FRUs remaining in the system unit is defective.

  Exchange the FRUs that have not been exchanged, in the following order:

  1. Optical drive
  2. Removable media enclosure.
  3. Disk drive enclosure (no disk drives at this time)
  4. I/O backplane, Un-P1.

  Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to the next step.

• PFW1548-13

  The system is working correctly with this configuration. One of the disk drives that you removed from the disk drive backplanes may be defective.

  1. Make sure the diagnostic CD-ROM is inserted into the optical drive.
  2. Turn off the power and remove the power cords.
  3. Install a disk drive in the disk drive enclosure assembly.
  4. Plug in the power cords and wait for the OK prompt to display on the operator panel display.
  5. Turn on the power.
  6. After the word keyboard is displayed, press the number 5 key on either the directly attached keyboard or an ASCII terminal keyboard.
  7. Enter the appropriate password if you are prompted to do so.

  Is the Please define the System Console screen displayed?

  NO

  Exchange the FRUs that have not been exchanged, in the following order:

  1. Last disk drive installed
  2. Disk drive enclosure assembly.

  Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Repeat this step with all disk drives that were installed in the disk drive backplane.

  After all of the disk drives have been reinstalled, go to the next step.

• PFW1548-14

  The system is working correctly with this configuration. One of the devices that was disconnected from the system backplane may be defective.

  1. Turn off the power and remove the power cords.
  2. Attach a system backplane device (for example: system port 1, system port 2, USB, keyboard, mouse, Ethernet) that had been removed.

     After all of the I/O backplane device cables have been reattached, reattached the cables to the service processor one at a time.

  3. Plug in the power cords and wait for "01" in the upper-left corner on the operator panel display.
  4. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking on the "Advanced" button on the activation screen. On the "advanced" activation screen, select "boot in service mode using the default boot list" to boot the diagnostic CD-ROM.
  5. If the Console Selection screen is displayed, choose the system console.
  6. After the word keyboard displays, press the number 5 key on either the directly attached keyboard or on an ASCII terminal keyboard.
  7. Enter the appropriate password if you are prompted to do so.

  Is the Please define the System Console screen displayed?

  NO

  The last device or cable that you attached is defective.

  To test each FRU, exchange the FRUs in the following order:

  1. Device and cable (last one attached)
  2. If the last cable in this step was reconnected to the service processor, replace the service processor.
  3. I/O backplane, location: Un-P1.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Repeat this step until all of the devices are attached. Go to the next step.

• PFW1548-15

  The system is working correctly with this configuration. One of the FRUs (adapters) that you removed may be defective.

  1. Turn off the power and remove the power cords.
  2. Install a FRU (adapter) and connect any cables and devices that were attached to the FRU.
  3. Plug in the power cords and wait for the OK prompt to display on the operator panel display.
  4. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking on the "Advanced" button on the activation screen. On the "advanced" activation screen, select "boot in service mode using the default boot list" to boot the diagnostic CD-ROM.
  5. If the Console Selection screen is displayed, choose the system console.
  6. After the word keyboard displays, press the number 5 key on either the directly attached keyboard or on an ASCII terminal keyboard.
  7. Enter the appropriate password if you are prompted to do so.

  Is the Please define the System Console screen displayed?

  NO

  Go to the next step.

  YES

  Repeat this step until all of the FRUs (adapters) are installed. Go to Verifying a repair.

• PFW1548-16

  The last FRU installed or one of its attached devices is probably defective.

  1. Make sure the diagnostic CD-ROM is inserted into the optical drive.
  2. Turn off the power and remove the power cords.
  3. Starting with the last installed adapter, disconnect one attached device and cable.
  4. Plug in the power cords and wait for the "01" in the upper-left corner on the operator panel display.
  5. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking on the "Advanced" button on the activation screen. On the "advanced" activation screen, select "boot in service mode using the default boot list" to boot the diagnostic CD-ROM.
  6. If the Console Selection screen is displayed, choose the system console.
  7. After the word keyboard displays, press the number 5 key on either the directly attached keyboard or on an ASCII terminal keyboard.
  8. Enter the appropriate password if you are prompted to do so.

  Is the Please define the System Console screen displayed?

  NO

  Repeat this step until the defective device or cable is identified or all devices and cables have been disconnected.

  If all the devices and cables have been removed, then one of the FRUs remaining in the system unit is defective.

  To test each FRU, exchange the FRUs in the following order:

  1. Adapter (last one installed)
  2. I/O backplane, location: Un-P1.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  The last device or cable that you disconnected is defective. Exchange the defective device or cable then go to the next step.

• PFW1548-17

  Reattach the flex cables, if present, both front and back. Return the system to its original configuration. Reinstall the control panel, the VPC card, and the service processor in the original primary processor drawer. Reattach the power cords and power on the system.

  Does the system come up properly?

  NO

  Go to the next step.

  YES

  Go to Verifying a repair.

• PFW1548-18

  Have all of the drawers been tested individually with the physical control panel (if present), service processor, and VPD card?

  NO

  1. Detach the flex cables from the front and back of the system if not already disconnected.
  2. Remove the service processor card, the VPD card, and the control panel from the processor drawer that was just tested.
  3. Install these parts in the next drawer in the rack, going top to bottom.
  4. Go to step PFW1548-3.

  YES

  Reinstall the control panel, the VPD card, and the service processor card in the original primary processor drawer. Return the system to its original configuration.

  Suspect a problem with the flex cables. If the error code indicates a problem with the SPCN or service processor communication between drawers, replace the flex cable in the rear. If the error code indicates a problem with inter-processor drawer communication, replace the flex cable on the front of the system.

  Did replacing the flex cable(s) resolve the problem?

  NO

  Go to step the next step.

  YES

  The problem is resolved. Go to Verifying a repair.

• PFW1548-19

  Replacing the flex cables did not resolve the problem. If the problem appears to be with the service processor or SPCN signals, suspect the I/O backplanes. If the problem appears to be with processor communication, suspect the processor cards.

  Replace the I/O backplanes, or processor cards, one at a time until the defective part is found.

  Did this resolve the problem?

  NO

  Contact your next level of support.

  YES

  The problem is resolved. Go to Verifying a repair.

• PFW1548-20
  1. Follow the instructions on the screen to select the system console.
  2. When the DIAGNOSTIC OPERATING INSTRUCTIONS screen is displayed, press Enter.
  3. Select Advanced Diagnostics Routines.
  4. If the terminal type has not been defined, you must use the initialize terminal option on the FUNCTION SELECTION menu to initialize the AIX diagnostic environment before you can continue with the diagnostics. This is a separate operation from selecting the console display.
  5. If the NEW RESOURCE screen is displayed, select an option from the bottom of the screen.
     Note:
     Adapters and devices that require supplemental media are not shown in the new resource list. If the system has adapters or devices that require supplemental media, select option 1.
  6. When the DIAGNOSTIC MODE SELECTION screen is displayed, press Enter.
  7. Select All Resources. (If you were sent here from step PFW1548-21, select the adapter or device that was loaded from the supplemental media).

  Did you get an SRN?

  NO

  Go to step PFW1548-22.

  YES

  Go to then next step.

• PFW1548-21

  Look at the FRU part numbers associated with the SRN.

  Have you exchanged all the FRUs that correspond to the failing function codes (FFCs)?

  NO

  Exchange the FRU with the highest failure percentage that has not been changed.

  Repeat this step until all the FRUs associated with the SRN have been exchanged or diagnostics run with no trouble found. Run diagnostics after each FRU is exchanged. Go to Verifying a repair.

  YES

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

• PFW1548-22

  Does the system have adapters or devices that require supplemental media?

  NO

  Go to step the next step.

  YES

  Go to step PFW1548-24.

• PFW1548-23

  Consult the PCI adapter configuration documentation for your operating system to verify that all adapters are configured correctly.

  Go to Verifying a repair.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

• PFW1548-24
  1. Select Task Selection.
  2. Select Process Supplemental Media and follow the on-screen instructions to process the media. Supplemental media must be loaded and processed one at a time.

  Did the system return to the TASKS SELECTION SCREEN after the supplemental media was processed?

  NO

  Go to the next step.

  YES

  Press F3 to return to the FUNCTION SELECTION screen. Go to step PFW1548-24, substep 4.

• PFW1548-25

  The adapter or device is probably defective.

  If the supplemental media is for an adapter, replace the FRUs in the following order:

  1. Adapter
  2. I/O backplane, location: Un-P1.

  If the supplemental media is for a device, replace the FRUs in the following order:

  1. Device and any associated cables
  2. The adapter to which the device is attached

  Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Starting a Service Call and follow the instructions for the new symptom.

  Go to Verifying a repair.

  End of procedure.