PFW1548: Memory and processor subsystem problem isolation procedure when an HMC is attached

This procedure is used to locate defective FRUs not found by normal diagnostics. For this procedure, diagnostics are run on a minimally configured system. If a failure is detected on the minimally configured system, the remaining FRUs are exchanged one at a time until the failing FRU is identified. If a failure is not detected, FRUs are added back until the failure occurs. The failure is then isolated to the failing FRU.

Perform the following procedure:

• PFW1548-1
  1. Ensure that the diagnostics and the operating system are shut down.

     Is the system at "service processor standby", indicated by 01 in the control panel?

     NO

     Replace the system backplane, location: Un-P1. Return to step PFW1548-1.

     YES

     Continue with substep 2.

  2. Turn on the power using either the white button or the ASMI menus.

     Does the system reach hypervisor standby as indicated on the HMC?

     NO

     Go to PFW1548-3.

     YES

     Go to PFW1548-2.

  3. Insert the AIX® diagnostic CD-ROM into the optical drive.
     Note: If you cannot insert the diagnostic CD-ROM, go to PFW1548-2.
  4. When the word keyboard is displayed on an ASCII terminal, a directly attached keyboard, or hardware management console (HMC), press the number 5 key.
  5. If you are prompted to do so, enter the appropriate password.

     Is the "Please define the System Console" screen displayed?

     NO

     Go to PFW1548-2.

     YES

     Go to PFW1548-14.

• PFW1548-2

  Insert the AIX diagnostic CD-ROM into the optical drive.

  Note: If you cannot insert the diagnostic CD-ROM, go to step PFW1548-3.

  Turn on the power using either the white button or the ASMI menus. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux® or AIX partition by clicking the Advanced button on the activation screen. On the Advanced activation screen, select Boot in service mode using the default boot list to boot the diagnostic CD-ROM.

  If you are prompted to do so, enter the appropriate password.

  Is the "Please define the System Console" screen displayed?

  NO

  Go to PFW1548-3.

  YES

  Go to PFW1548-14.

• PFW1548-3
  1. Turn off the power.
  2. If you have not already done so, configure the service processor (using the ASMI menus), follow the instructions in note 6 located in PFW1548: Memory and processor subsystem problem isolation procedure and then return here and continue.
  3. Exit the service processor (ASMI) menus and remove the power cords.
  4. Disconnect all external cables (parallel, system port 1, system port 2, keyboard, mouse, USB devices, SPCN, Ethernet on the system planar, and so on). Also disconnect all of the external cables attached to the service processor except the Ethernet cable going to the HMC.
  Go to the next step.
• PFW1548-4
  1. If this is a deskside system, remove the service access cover. If this is a rack-mounted system, place the drawer into the service position and remove the service access cover. Also remove the front cover.
  2. Record the slot numbers of the PCI adapters and I/O expansion cards if present. Label and record the locations of all cables attached to the adapters. Disconnect all cables attached to the adapters and remove all of the adapters.
  3. Remove the removable media/disk drive enclosure assembly by pulling out the blue tabs at the bottom of the enclosure, then sliding the enclosure out approximately three centimeters.
  4. Remove and label the disk drives from the media/disk drive enclosure assembly.
  5. Remove one of the two memory DIMMs quads.
     Notes:

     1. Place the memory DIMM locking tabs in the locked (upright) position to prevent damage to the tabs.
     2. Memory DIMMs must be installed in quads and in the correct connectors. Refer to System FRU locations for information on memory DIMMs locations.
  6. Plug in the power cords and wait for 01 in the upper-left corner of the control panel display.
  7. Turn on the power using either the HMC or the white button.

  Does the managed system reach power on at hypervisor standby as indicated on the HMC?

  NO

  Go to PFW1548-7.

  YES

  Go to the next step.

• PFW1548-5

  Were any memory DIMMs removed from system backplane?

  NO

  Go to PFW1548-8.

  YES

  Go to the next step.

• PFW1548-6
  1. Turn off the power, and remove the power cords.
  2. Replug the memory DIMMs that were removed from system backplane in PFW1548-2 in their original locations.
     Notes:

     1. Place the memory DIMM locking tabs into the locked (upright) position to prevent damage to the tabs.
     2. Memory DIMMs must be installed in quads in the correct connectors. Refer to System FRU locations for information on memory DIMM locations..
  3. Plug in the power cords and wait for 01 in the upper-left corner of the control panel display.
  4. Turn on the power using either the HMC or the white button.

  Does the managed system reach power on at hypervisor standby as indicated on the HMC?

  NO

  A memory DIMM in the quad you just replaced in the system is defective. Turn off the power, remove the power cords, and exchange the memory DIMM quad with new or previously removed memory DIMM quad. Repeat this step until the defective memory DIMM quad is identified, or both memory DIMM quads have been exchanged.

  Note: the system backplane has two memory DIMM quads: Un-P1-C1x-C1 to Un-P1-C1x-C4, and Un-P1-C1x-C6 to Un-P1-C1x-C9. See System FRU locations.

  If your symptom did not change and both the memory DIMM quads have been exchanged, call your service support person for assistance.

  If the symptom changed, check for loose cards and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to the next step.

• PFW1548-7

  One of the FRUs remaining in the system unit is defective.

  Note: If a memory DIMM is exchanged, ensure that the new memory DIMM is the same size and speed as the original memory DIMM.
  1. Turn off the power, remove the power cords, and exchange the following FRUs, one at a time, in the order listed:
     1. Memory DIMMs. Exchange one quad at a time with new or previously removed DIMM quads
     2. System backplane, location: Un-P1
     3. Power supplies, locations: Un-E1 and Un-E2.
  2. Plug in the power cords and wait for 01 in the upper-left corner of the control panel display.
  3. Turn on the power using either the HMC or the white button.

  Does the managed system reach power on at hypervisor standby as indicated on the HMC?

  NO

  Reinstall the original FRU.

  Repeat the FRU replacement steps until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to Verify a repair.

• PFW1548-8
  1. Turn off the power.
  2. Reconnect the system console.
     Notes:

     1. If an ASCII terminal has been defined as the firmware console, attach the ASCII terminal cable to the S1 connector on the rear of the system unit.
     2. If a display attached to a display adapter has been defined as the firmware console, install the display adapter and connect the display to the adapter. Plug the keyboard and mouse into the keyboard connector on the rear of the system unit.
  3. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking the Advanced button on the activation screen. On the Advanced activation screen, select Boot in service mode using the default boot list to boot the diagnostic CD-ROM.
  4. If the ASCII terminal or graphics display (including display adapter) is connected differently from the way it was previously, the console selection screen appears. Select a firmware console.
  5. Immediately after the word keyboard is displayed, press the number 1 key on the directly attached keyboard, an ASCII terminal or HMC. This activates the system management services (SMS).
  6. Enter the appropriate password if you are prompted to do so.

  Is the SMS screen displayed?

  NO

  One of the FRUs remaining in the system unit is defective.

  Exchange the FRUs that have not been exchanged, in the following order:

  1. If you are using an ASCII terminal, go to the problem determination procedures for the display. If you do not find a problem, do the following:
     1. Replace the system backplane, location: Un-P1.
  2. If you are using a graphics display, go to the problem determination procedures for the display. If you do not find a problem, do the following:
     1. Replace the display adapter.
     2. Replace the backplane in which the graphics adapter is plugged.

        Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

        If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

        If the symptom changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to the next step.

• PFW1548-9
  1. Make sure the diagnostic CD-ROM is inserted into the optical drive.
  2. Turn off the power and remove the power cords.
  3. Use the cam levers to reconnect the disk drive enclosure assembly to the I/O backplane.
  4. Reconnect the removable media/disk drive enclosure assembly by sliding the media enclosure toward the rear of the system, then pressing the blue tabs.
  5. Plug in the power cords and wait for 01 in the upper-left corner of the operator panel display.
  6. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking the Advanced button on the activation screen. On the Advanced activation screen, select Boot in service mode using the default boot list to boot the diagnostic CD-ROM.
  7. Immediately after the word keyboard is displayed, press the number 5 key on either the directly attached keyboard or an ASCII terminal keyboard.
  8. Enter the appropriate password if you are prompted to do so.

  Is the "Please define the System Console" screen displayed?

  NO

  One of the FRUs remaining in the system unit is defective.

  Exchange the FRUs that have not been exchanged, in the following order:

  1. Optical drive
  2. Removable media enclosure.
  3. System backplane, Un-P1.

  Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Go to the next step.

• PFW1548-10

  The system is working correctly with this configuration. One of the disk drives that you removed from the disk drive backplanes may be defective.

  1. Make sure the diagnostic CD-ROM is inserted into the optical drive.
  2. Turn off the power and remove the power cords.
  3. Install a disk drive in the media/disk drive enclosure assembly.
  4. Plug in the power cords and wait for the OK prompt to display on the operator panel display.
  5. Turn on the power.
  6. Immediately after the word keyboard is displayed, press the number 5 key on either the directly attached keyboard or an ASCII terminal keyboard.
  7. Enter the appropriate password if you are prompted to do so.

  Is the "Please define the System Console" screen displayed?

  NO

  Exchange the FRUs that have not been exchanged, in the following order:

  1. Last disk drive installed
  2. Disk drive backplane.

  Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Repeat this step with all disk drives that were installed in the disk drive backplane.

  After all of the disk drives have been reinstalled, go to the next step.

• PFW1548-11

  The system is working correctly with this configuration. One of the devices that was disconnected from the system backplane may be defective.

  1. Turn off the power and remove the power cords.
  2. Attach a system backplane device (for example: system port 1, system port 2, USB, keyboard, mouse, Ethernet) that had been removed.

     After all of the I/O backplane device cables have been reattached, reattached the cables to the service processor one at a time.

  3. Plug in the power cords and wait for 01 in the upper-left corner on the operator panel display.
  4. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking the Advanced button on the activation screen. On the Advanced activation screen, select Boot in service mode using the default boot list to boot the diagnostic CD-ROM.
  5. If the Console Selection screen is displayed, choose the system console.
  6. Immediately after the word keyboard is displayed, press the number 5 key on either the directly attached keyboard or on an ASCII terminal keyboard.
  7. Enter the appropriate password if you are prompted to do so.

  Is the "Please define the System Console" screen displayed?

  NO

  The last device or cable that you attached is defective.

  To test each FRU, exchange the FRUs in the following order:

  1. Device and cable (last one attached)
  2. System backplane, location: Un-P1.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  Repeat this step until all of the devices are attached. Go to the next step.

• PFW1548-12

  The system is working correctly with this configuration. One of the FRUs (adapters) that you removed may be defective.

  1. Turn off the power and remove the power cords.
  2. Install a FRU (adapter) and connect any cables and devices that were attached to the FRU.
  3. Plug in the power cords and wait for the OK prompt to display on the operator panel display.
  4. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking the Advanced button on the activation screen. On the Advanced activation screen, select Boot in service mode using the default boot list to boot the diagnostic CD-ROM.
  5. If the Console Selection screen is displayed, choose the system console.
  6. Immediately after the word keyboard is displayed, press the number 5 key on either the directly attached keyboard or on an ASCII terminal keyboard.
  7. Enter the appropriate password if you are prompted to do so.

  Is the "Please define the System Console" screen displayed?

  NO

  Go to the next step.

  YES

  Repeat this step until all of the FRUs (adapters) are installed. Go to Verify a repair.

• PFW1548-13

  The last FRU installed or one of its attached devices is probably defective.

  1. Make sure the diagnostic CD-ROM is inserted into the optical drive.
  2. Turn off the power and remove the power cords.
  3. Starting with the last installed adapter, disconnect one attached device and cable.
  4. Plug in the power cords and wait for the 01 in the upper-left corner on the operator panel display.
  5. Turn on the power using either the HMC or the white button. (If the diagnostic CD-ROM is not in the optical drive, insert it now.) After the system has reached hypervisor standby, activate a Linux or AIX partition by clicking the Advanced button on the Advanced activation screen. On the Advanced activation screen, select Boot in service mode using the default boot list to boot the diagnostic CD-ROM.
  6. If the Console Selection screen is displayed, choose the system console.
  7. Immediately after the word keyboard is displayed, press the number 5 key on either the directly attached keyboard or on an ASCII terminal keyboard.
  8. Enter the appropriate password if you are prompted to do so.

  Is the "Please define the System Console" screen displayed?

  NO

  Repeat this step until the defective device or cable is identified or all devices and cables have been disconnected.

  If all the devices and cables have been removed, then one of the FRUs remaining in the system unit is defective.

  To test each FRU, exchange the FRUs in the following order:

  1. Adapter (last one installed)
  2. System backplane, location: Un-P1.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  YES

  The last device or cable that you disconnected is defective. Exchange the defective device or cable then go to the next step.

• PFW1548-14
  1. Follow the instructions on the screen to select the system console.
  2. When the DIAGNOSTIC OPERATING INSTRUCTIONS screen is displayed, press Enter.
  3. Select Advanced Diagnostics Routines.
  4. If the terminal type has not been defined, you must use the option Initialize Terminal on the FUNCTION SELECTION menu to initialize the AIX diagnostic environment before you can continue with the diagnostics. This is a separate operation from selecting the console display.
  5. If the NEW RESOURCE screen is displayed, select an option from the bottom of the screen.
     Note: Adapters and devices that require supplemental media are not shown in the new resource list. If the system has adapters or devices that require supplemental media, select option 1.
  6. When the DIAGNOSTIC MODE SELECTION screen is displayed, press Enter.
  7. Select All Resources. (If you were sent here from step PFW1548-18, select the adapter or device that was loaded from the supplemental media).

  Did you get an SRN?

  NO

  Go to step PFW1548-16.

  YES

  Go to the next step.

• PFW1548-15

  Look at the FRU part numbers associated with the SRN.

  Have you exchanged all the FRUs that correspond to the failing function codes (FFCs)?

  NO

  Exchange the FRU with the highest failure percentage that has not been changed.

  Repeat this step until all the FRUs associated with the SRN have been exchanged or diagnostics run with no trouble found. Run diagnostics after each FRU is exchanged. Go to Verify a repair.

  YES

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

• PFW1548-16

  Does the system have adapters or devices that require supplemental media?

  NO

  Go to step the next step.

  YES

  Go to step PFW1548-18.

• PFW1548-17

  Consult the PCI adapter configuration documentation for your operating system to verify that all adapters are configured correctly.

  Go to Verify a repair.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

• PFW1548-18
  1. Select Task Selection.
  2. Select Process Supplemental Media and follow the on-screen instructions to process the media. Supplemental media must be loaded and processed one at a time.

  Did the system return to the TASKS SELECTION SCREEN after the supplemental media was processed?

  NO

  Go to the next step.

  YES

  Press F3 to return to the FUNCTION SELECTION screen. Go to step PFW1548-14, substep 4.

• PFW1548-19

  The adapter or device is probably defective.

  If the supplemental media is for an adapter, replace the FRUs in the following order:

  1. Adapter
  2. System backplane, location: Un-P1.

  If the supplemental media is for a device, replace the FRUs in the following order:

  1. Device and any associated cables
  2. The adapter to which the device is attached

  Repeat this step until the defective FRU is identified or all the FRUs have been exchanged.

  If the symptom did not change and all the FRUs have been exchanged, call service support for assistance.

  If the symptom has changed, check for loose cards, cables, and obvious problems. If you do not find a problem, go to the Problem Analysis procedures and follow the instructions for the new symptom.

  Go to Verify a repair.

  End of procedure.