Bus, high-speed link (HSL or RIO) isolation information

Please read all safety notices below before servicing the system and while performing a procedure.

• Related concepts
• Fiber optic cleaning procedures
  Use the fiber optic cleaning kit and the fiber optic cleaning procedures in "SY27-2604 Fiber Optic Cleaning Procedures" for all fiber channel connections such as those used in optical high speed link (HSL) connections or fibre channel attached devices.

• Related tasks
• PCI bus isolation using AIX, Linux, or the HMC
  Use this procedure if you are isolating a PCI bus problem from the HMC or while running AIX^(R) or Linux^(R).
• Verifying a high-speed link, system PCI bus, or a multi-adapter bridge repair
  Use this procedure to verify a repair for the high-speed link, a system PCI bus, or for a multi-adapter bridge.
• DSA translation
  The Direct Select Address (DSA) may be coded in word 7 of the reference code.
• RIO/HSL/12X connection problems
  The failing component is the RIO/HSL/12X connection

• Related reference
• Breaking down a RIO/HSL/12X or PCI bus reference code
  Word 7 of the reference code allows you to determine the correct bus number, bus type, multi-adapter bridge number, multi-adapter bridge function number, and logical card number from the Direct Select Address (DSA).
• Card positions
  The following information specifies card positions for the various server models.
• Converting the loop number to RIO/HSL/12X port location labels
  Use the following table to convert the RIO/HSL12X loop number to port location labels.
• Status indications for RIO/HSL
  A status indication for a RIO/HSL loop is identified in the reference code.
• HSL loop configuration and status form
  Use this HSL loop configuration and status form to record the status of the HSL ports in the loop.
• Installed features in a PCI bridge set form
  Use this form to record the "PCI Bridge Set" card positions, and multi-adapter bridge function numbers.
• RIO/HSL link status diagnosis form
  Use this form to record the status of the RIO/HSL links.

PCI bus isolation using AIX, Linux, or the HMC

Isolating a PCI bus problem while running AIX or Linux

Can the operating system be IPLed?

• No: Perform MABIP52. This ends the procedure.
• Yes: Choose from the following:
  • If you are running AIX, go to Running the online and eServer stand-alone diagnostics to isolate the PCI bus failure with online diagnostics in concurrent mode.
  • If you are running Linux, go to Running the online and eServer stand-alone diagnostics to isolate the PCI bus failure with stand-alone diagnostics.
  This ends the procedure.

Isolating a PCI bus problem from the HMC

If you have locations for the FRUs given in the serviceable event view of the error log, use those locations and exchange:

1. Did the serviceable event view provide the location(s) for the failing FRU(s)?
   • Yes: Use those locations to exchange the given FRUs one at a time until the problem is resolved. This ends the procedure.
   • No: Continue with the next step.
2. Go to DSA translation to determine the Direct Select Address (DSA).
3. Perform the following:
   1. Record the bus number value (BBBB) from the DSA and convert it to decimal format.
   2. Search for the decimal system bus number in the partition resources screens on the HMC.
   3. Record the frame or unit type and continue with the next step.
4. Record the Cc value from the DSA. Is the Cc value greater than 00?
   • Yes: Continue with the next step.
   • No: The multi-adapter bridge number and the multi-adapter function number have not been identified, and so the card slot cannot be identified using the DSA. Look in the HMC partition resources for non-reporting or non-operational hardware. That will indicate which cards in which positions need to be replaced. Refer to Finding part locations for the model you are working on for information about the multi-adapter bridge that controls those card slots. That multi-adapter bridge is also a FRU. This ends the procedure.
5. Is the right-most character (c) F?
   • No: Continue with the next step.
   • Yes: Only the multi-adapter bridge number has been identified. Record the multi-adapter bridge number (left-most character of Cc) for later use. Since the card slot cannot be identified with the DSA, refer to Finding part locations for the model you are working on for information about the multi-adapter bridge that controls the card slots. Consider all card slots controlled by the multi-adapter bridge to be FRUs. This ends the procedure.
6. Refer to Card positions and use the BBBB and Cc values that you recorded to identify the card position. Then return to the procedure that sent you here. This ends the procedure.

Verifying a high-speed link, system PCI bus, or a multi-adapter bridge repair

Within this procedure, the terms "system" and "logical partition" are interchangeable when used individually.

1. Perform this procedure from the logical partition you were in when you were sent to this procedure, or from the HMC if this error was worked from the HMC.
2. If you previously powered off a system or logical partition, or an expansion unit during this service action, then you need to power it off again.
3. Install all cards, cables, and hardware, ensuring that all connections are tight. You can use the system configuration list to verify that the cards are installed correctly.
4. Power on any expansion unit, logical partition or system unit that was powered off during the service action. Is one of the following true?
   • If the system or a logical partition was powered off during the service action, does the IPL complete successfully to the IPL or does Install the System display?
   • If an expansion tower was powered off during the service action, does the expansion tower power on complete successfully?
   • If any IOP or IOA card locations were powered off using concurrent maintenance during the service action, do the slots power on successfully?
   • If you exchanged a FRU that should appear as a resource or resources to the system, such as an IOA, or I/O bridge, does the new FRU's resource appear in HSM as operational?
     • Yes: Continue with the next step.
     • No: Verify that you have followed the power off, remove and replace, and power-on procedures correctly. When you are sure that you have followed the procedures correctly, then exchange the next FRU in the list. If there are no more FRUs to exchange, then contact your next level of support. This ends the procedure.
5. Does the system or logical partition have mirrored protection? Select Yes if you are not sure.
   • No: Continue with the next step.
   • Yes: From the Dedicated Service Tools (DST) display, select Work with disk units, and resume mirrored protection for all units that have a suspended status.
6. Choose from the following options:
   • If you are working from a partition, from the Start a Service Tool display, select Hardware service manager and look for the I/O processors that have a failed or missing status.
   • If you are working from a HMC, look at the CEC properties.
   1. Choose the I/O tab.
   2. Look for IOAs or IOPs that have a failed or missing status.

   Are all I/O processor cards operational?

   Note:
   Ignore any IOPs that are listed with a status of not connected.
   • Yes: Go to step 10.
   • No: Display the logical hardware resource information for the non-operational I/O processors. For all I/O processors and I/O adapters that are failing; record the bus number (BBBB), board (bb) and card information (Cc). Continue with the next step.
7. Perform the following:
   1. Return to the Dedicated Service Tools (DST) display.
   2. Display the Product Activity Log.
   3. Select All logs and search for an entry with the same bus, board, and card address information as the non-operational I/O processor. Do not include informational or statistical entries in your search. Use only entries that occurred during the last IPL.

      Did you find an entry for the SRC that sent you to this procedure?

      • No: Continue with the next step.
      • Yes: Ask your next level of support for assistance. This ends the procedure.
8. Did you find a B600 6944 SRC that occurred during the last IPL?
   • Yes: Continue with the next step.
   • No: A different SRC is associated with the non-operational I/O processor. Go to the Starting a Service Call procedure and look up the new SRC to correct the problem. This ends the procedure.
9. Is there a B600 xxxx SRC that occurred during the last IPL other than the B600 6944 and informational SRCs?
   • Yes: Use the other B600 xxxx SRC to determine the problem. Go to the Starting a Service Call and look up the new SRC to correct the problem. This ends the procedure.
   • No: You connected an I/O processor in the wrong card position. Use the system configuration list to compare the cards. When you have corrected the configuration, go to the start of this procedure to verify the bus repair. This ends the procedure.
10. If in a partition, use the hardware service manager function to print the system configuration list.

    Are there any configuration mismatches?

    • No: Continue with the next step.
    • Yes: Ask your next level of support for assistance. This ends the procedure.
11. You have verified the repair of the system bus.
    1. If for this service action only an expansion unit was powered off or only the concurrent maintenance function was used for an IOP or IOA, then continue with the next step.
    2. Otherwise, perform the following to return the system to the customer:
       1. Power off the system or logical partition. See Powering on and powering off for details.
       2. Select the operating mode with which the customer was originally running.
       3. Power on the system or logical partition.
12. If the system has logical partitions and the entry point SRC was B600 xxxx, then check for related problems in other logical partitions that could have been caused by the failing part. This ends the procedure.

Breaking down a RIO/HSL/12X or PCI bus reference code

Physical card slot labels and card positions for PCI buses are determined by using the DSA and the appropriate system unit or I/O unit card positions. See Card positions for details.

DSA translation

This is either a PCI system bus number or a RIO/HSL/12X loop number, depending on the type of error. With the information obtained here, and the information in either the card position table (for PCI bus numbers) or the information in the loop-number-to-NIC-port table (for RIO/HSL/12X loop numbers), you should be able to isolate a failing PCI bus or RIO/HSL/12X loop. Follow the instructions below to translate the DSA:

1. Break down the DSA into the bus number, multi-adapter bridge number, and multi-adapter bridge function number. The DSA is of the form BBBB Ccxx, and breaks down into the following parts:
   • BBBB = bus number
   • C = multi-adapter bridge number
   • c = multi-adapter bridge function number
   • xx = not used
2. Is the bus number less than 0684?
   • Yes: The bus number is a PCI bus number. Convert the number to decimal, and then continue with the next step.
   • No: The bus number is a RIO/HSL/12X loop number. Convert the number to decimal, and then go to step 4.
3. Use one of the following guides to determine the type of processor or I/O enclosure in which the bus is located:
   • If you are using a Hardware Management Console interface, view the managed system's properties on the Hardware Management Console. Then select the IO tab. The bus numbers will be displayed.
   • If you are using AIX or Linux, use the command line interface to determine the enclosure type. On the command line, type the following:

     lshwres -r io --rsubtype bus

     The result will be in the form of:

     unit_phys_loc=Uxxxx.yyy.zzzzzzz,bus_id=a, ......

     Find the bus id "a" entry that matches the decimal bus number you determined in step 2. Using the corresponding Uxxxx value, look up the unit model or enclosure type using the Unit Type and Locations table in Locating FRUs. Continue with the next step.

4. Perform one of the following:
   • If you are working with a PCI bus number, refer to Card positions to search for the bus number, the multi-adapter bridge number, and the multi-adapter bridge function number that matches the processor enclosure or I/O enclosure where the bus is located. This ends the procedure.
   • If you are working with a RIO/HSL/12X loop number, refer to Converting the loop number to RIO/HSL/12X port location labels to determine the starting ports for the RIOHSL/12X loop with the failed link. This ends the procedure.

Card positions

Use the table for the model you are working on:

• Card positions for model MMA L4
• Card positions for model MMA ML8 (L4+)
• Card positions for model MMA ML12 (ML8+)
• Card positions for model MMA ML16 (ML12+)
• Card positions for 5088, 0588, 5094, 5096, 5294, 5296, and 8094-002 expansion units
• Card positions for 5095, 0595, and 7311-D20 expansion units
• Card positions for 7040-61D expansion unit
• Card positions for 7311-D10, 7311-D11, and 5790 expansion units
• Card positions for 7314-G30 expansion unit

Converting the loop number to RIO/HSL/12X port location labels

Fiber optic cleaning procedures

This ends the procedure.

Status indications for RIO/HSL

Use the table below to determine if the status indication requires a service action. Record the rightmost four characters of word 4 of the reference code. These characters are the program return code (PRC), which indicates the RIO/HSL status. The leftmost four characters of word 7 indicate the RIO/HSL loop number (in hexadecimal format).

Indicated status for PRC 3209

Recoverable CRC (cyclical redundancy check) errors have occurred on the loop. This error requires service action. An RIO/HSL cable or connection must be exchanged. Exchange only the cable that appears in the FRU list of this reference code in this document. If you are working from the serviceable event view, then only the FRU(s) required for this error will be displayed.

Choose from the following scenarios:

• If there is a serviceable event view entry with a cable FRU listed with a loop number, enclosure/unit ID, and port label identified with the cable FRU, then perform one of the following:
  • If the cable is optical, it is possible the optical connections need cleaning. You can choose either to clean the cable connections at each end without exchanging the cable, or to exchange the cable. Use symbolic FRU Fiber optic cleaning procedures for information on cleaning the connections. See Locating FRUs to exchange the cable.
  • If the cable is copper, examine the screws that hold the connector at the end of the cable identified in the reference code or the first cable location listed in the serviceable event view entry. It is possible to get CRC errors when the connector screws are not tight. You can choose to tighten the cable connector screws without exchanging the cable only if they are loose. Otherwise, you must exchange the cable. Refer to Locating FRUs to exchange the cable.
  • If neither of these actions resolves the problem, replace the FRUs in the serviceable event view one at a time. See Locating FRUs.

  This ends the procedure.

• If the serviceable event view entry does not list any cable FRUs, then exchange the failing items listed in the serviceable event view entry by following the normal service procedures for those FRUs. This will be the case when the RIO/HSL connection is embedded. See Locating FRUs.

  This ends the procedure.

• If the serviceable event view entry has cable FRU(s) listed, but the loop number, enclosure/unit ID, and port label are not all listed with the cable FRU, go to Determining the RIO/HSL port label below.

Determining the RIO/HSL port label

Retrieve and record the following information:

• Loop number. The loop number is displayed in hexadecimal format as the four leftmost digits of word 7 in the reference code. Convert the loop number to decimal format using DSA translation. Record both the hexadecimal and decimal formats of the loop number. If the loop is an internal loop on a 515, 520, 525, or 570 (loops 0680, 0683, 0686, 0689 hexadecimal or 1664, 1667, 1670, 1673 decimal), then record "Internal".
• Frame ID. The frame ID is displayed in hexadecimal format as the four leftmost digits of word 5 of the reference code. You must convert the frame ID to decimal format to match what is displayed in user interfaces and problem views. Record both the hexadecimal and decimal formats of the frame ID. If the frame ID is not zero, then translate the frame ID into the correct machine type, model, and serial number by performing the following:
  1. Log on to SST/DST.
     Attention: Do not IPL to DST.
  2. Select Hardware Service Manager.
  3. Select Packaging resources.
  4. Selecting Display details for each unit listed until the frame ID matches the ID you are working with. Once you find the matching frame ID, record the unit's machine type, model, and serial number.
  Note:
  The frame has an ID of 0000 at this point in the procedure. A frame ID of zero is indicating the NIC/RIO controller in a system unit.
• Port number indicator. The port number indicator is the four rightmost digits of word 5 of the reference code.

Use the frame ID and port number indicator in the following table to determine the RIO/HSL port label. If you are referred to Converting the loop number to RIO/HSL/12X port location labels, the failing item is in a system unit. You will need the RIO/HSL loop number to determine the FRU.

The following tables provide hub, port, and location code information for each of the currently supported models. This data is used to determine which RIO loop to use for the B7006981. The asterisk in the "Port indicator number" column indicates the leading port of the port pair.

Determining if the cable is the cause of the problem

If there is a cable attached to the failing port:

• If the cable is optical, it is possible the optical connections need cleaning. You can choose either to clean the cable connections at each end without exchanging the cable, or to exchange the cable. Use symbolic FRU Fiber optic cleaning procedures for information on cleaning the connections. See Locating FRUs to exchange the cable.
• If the cable is copper, examine the screws that hold the connector at the end of the cable identified in the reference code or the first cable location listed in the serviceable event view entry. It is possible to get CRC errors when the connector screws are not tight. You can choose to tighten the cable connector screws without exchanging the cable only if they are loose. Otherwise, you must exchange the cable.
• If neither of these actions resolves the problem, replace the cable. Perform RIOIP08 to determine the frame ID and RIO/HSL port label of the other end of the cable you will be exchanging (if you do not already know). Does this correct the problem?
  • Yes: This ends the procedure.
  • No: Continue with replacing the FRUs that the cable is connected to, starting with the FRU listed first.

Replacing the link's failing end point FRUs

Use the following table to determine the end point FRUs on the "From Frame ID" port. Replace the FRUs one at a time.

If replacing the end point FRUs for the "From Frame ID" port does not resolve the error, use the table below to replace the end point FRUs at the other end of the cable. Perform RIOIP08 to determine the system or I/O unit frame ID and RIO/HSL port label of the other end of the cable (if you do not already know).

This ends the procedure.

RIO/HSL/12X connection problems

When a RIO/HSL cable is disconnected, it may result in a lost connection between the units even after the cable is reconnected.

1. Choose from the following:
   • If you were sent to this procedure from another symbolic FRU, locate that FRU in Table 18 to see the description of the RIO/HSL/12X FRU. Then continue with the next step for more information about the FRU.
   • If you are working with this symbolic FRU in the FRU list, the failing component is an RIO/HSL/12X connection. Diagnostic code could not determine what kind of hardware was involved. The RIO/HSL/12X hardware can be any of the following:
     • Cable
     • Embedded RIO/HSL/12X link in a FRU (a planar, for example)
     • RIO/HSL/12X interposer card
     The RIO/HSL/12X link is on or between the other FRU or FRUs listed for the reference code. Continue with the next step.

   Table 18. RIO/HSL symbolic FRUs

   RIO/HSL FRU	Description
   HSL_OPT	This is an optical RIO/HSL cable. When exchanging optical RIO/HSL cables, use the optical cleaning kit and procedures. See Fiber optic cleaning procedures for details. If an interposer card is called for, be sure the interposer type matches the cable type (optical or copper).
   HSL1	There is a standard copper RIO/HSL cable at both ends (yellow connectors).
   HSL1_UN	There is a standard copper RIO/HSL cable (yellow connector) at the detecting end, and an unknown connector type at the other end.
   HSL2	There is a copper HSL2/RIO-G cable at both ends (black connectors), but the length of the cable could not be sensed.
   HSL2_xx	There is a copper HSL2/RIO-G cable at both ends (black connectors). Use the xx value to determine the cable length from this list: HSL2_01 = 1 meter HSL2 cable HSL2_03 = 3 meter HSL2 cable HSL2_08 = 8 meter HSL2 cable HSL2_10 = 10 meter HSL2 cable HSL2_15 = 15 meter HSL2 cable HSL2_17 = 1.75 meter HSL2 cable HSL2_25 = 2.5 meter HSL2 cable
   HSLH	There is a hybrid RIO/HSL to HSL2/RIO-G cable (yellow connector at one end and black connector at the other end), but the length of the cable could not be sensed.
   HSL_IB	This is an 12X cable (green connectors).
   HSLH_xx	There is a hybrid RIO/HSL to HSL2/RIO-G cable (yellow connector at one end and black connector at the other end). Use the xx value to determine the cable length from this list: HSLH_06 = 6 meter HSL/RIO to HSL2/RIO-G cable HSLH_15 = 15 meter HSL/RIO to HSL2/RIO-G cable
   HSL_I	There is a RIO/HSL interposer card for HSL/RIO cables (yellow connectors), HSL2/RIO-G cables (black connectors), or optical cables. The interposer card type could not be sensed.
   HSL_I2 HSL_I3 HSL_I4	There is a RIO/HSL interposer card for HSL/RIO cables (yellow connectors), HSL2/RIO-G cables (black connectors), or optical cables. Use the x value to determine the interposer card type from this list: HSL_I2 = Copper HSL2/RIO-G interposer card for HSL2/RIO-G cable connectors (black) in the system unit backplane position C08 or C09 of machine type 9406 model 825. HSL_I3 = Optical HSL/RIO interposer card for optical RIO/HSL cables in system unit backplane position C08 or C09 of machine type 9406 model 825. HSL_I4 = HSL2/RIO-G interposer/riser card on a 7040-61D I/O unit in location -P1-Riser or -P2-(Riser).

2. Choose from the following options:
   • If you are working from the serviceable event view, the location code or FRU description in the view will help determine the actual RIO/HSL/12X hardware to exchange. Continue with the next step.
   • If you are not working from the serviceable event view, or the view does not have a location code or better FRU description, then determine the location code of other FRUs in the FRU list for the error. Then continue with the next step.
3. Use the location code and the information from the preceding table to determine the machine type, model, or unit feature involved in the error. If necessary, use the location code for other FRUs listed in the FRU list for this error to determine the failing RIO/HSL/12X connection and any related FRUs that are part of that connection. Use Locating FRUs to find links to the locations information. Refer to the Parts Information chapter of the Service Guide for the failing machine type/model for part number information.
   Note:
   If you exchange all of the FRUs in the FRU list, but the problem still exists, contact your next level of support. You may be directed to exchange additional RIO/HSL/12X FRUs. Additional RIO/HSL FRUs has more information about RIO/HSL/12X FRUs on specific models and I/O units.

Additional RIO/HSL FRUs

The following are RIO/HSL/12X FRUs by model and/or unit type. For the model or unit type you are working on, there may be additional RIO/HSL/12X FRUs which were not listed in the FRU list of the error. Under the direction of your next level of support, you can try exchanging the additional FRUs.

1. In the following table, locate the unit type(s) on which you are working. Exchange the indicated RIO/HSL/12X loop connections (external or embedded) or RIO/HSL/12X interposer card.
2. Did the exchange correct the error?
   • Yes: The FRU you just replaced was the failing item. Go to Verifying a repair.

     This ends the procedure.

   • No: Call your next level of support.

     This ends the procedure.

   Table 19. RIO/HSL/12X cable or connections To find locations information, refer to Locating FRUs or the service guide for the I/O enclosure.

   System model or unit type	Description	Location code
   5094, 5096, 5294, 5296	HSL (optical) I/O bridge adapter	Un-CB1-C10
   5094, 5096, 5294, 5296	HSL I/O bridge card	Un-CB1-C10
   5094, 5096, 5294, 5296	HSL-2 I/O bridge adapter	Un-CB1-C10
   7311-D11, 5790	RIO/HSL adapter	Un-P1-C7
   0595, 5095	HSL (optical) I/O bridge adapter	Un-P1-C5
   7311-D20, 0595, 5095	RIO/HSL I/O bridge adapter	Un-P1-C5
   7311-D20, 0595, 5095	RIO/HSL-2 I/O bridge adapter	Un-P1-C5
   7314-G30, 5796	Dual-port 12X adapter	Un-P1-C7
   1519	Integrated xSeries Adapter

Powering on and powering off an I/O expansion unit

Follow these steps if you disconnected an RIO/HSL cable from an I/O expansion unit, and the RIO/HSL link on the unit did not recover when you reconnected the cable:

1. From the Hardware Service Manager screen, select Packaging hardware resources.
2. Select the frame that has just accidentally lost its RIO/HSL connection and select Concurrent maintenance. Then press Enter.
3. Select Power off domain.
4. After reconnecting the unit into the RIO/HSL loop, select Power on domain.

HSL loop configuration and status form

HSL loop configuration and status worksheet for system _______________, Loop number ___________

Installed features in a PCI bridge set form

RIO/HSL link status diagnosis form