Planning for QLogic InfiniBand switches

Use this information for planning and installing InfiniBand® switches and for specification information about the switches.

Planning and installation

For planning and installation information about cluster fabric that incorporates InfiniBand switches, see the IBM System p® HPC Clusters Fabric Guide Using Infiniband Hardware at the IBM clusters with the Infiniband switch web site.

InfiniBand switches are an industry-standard high-performance interconnect for high performance clusters (HPC) and enterprise grids. This industry-standard fabric creates clusters that address many of the HPC requirements, such as those found in scientific, technical, and financial applications. InfiniBand solutions also deliver the scalability and high availability that is required by distributed database processing.

The IBM® Power Systems™ machine type 7874 enables the ordering and servicing of specific QLogic 4X DDR InfiniBand switches. These switches are available in four models of up to 24-port, 48-port, 144-port and 288-port capability. The four available models follow:
  • The IBM 7874-024 is a QLogic 9024CU Managed 24-port 4X DDR InfiniBand Edge Switch designed to provide a compact 1U solution for use in building small node count fabrics. The 7874-024 includes redundant power supplies, power cords, rack mount kit, and QLogic InfiniBand Fabric Suite.
  • The IBM 7874-040 is a QLogic 9040 48-port 4X DDR InfiniBand Fabric Director Switch designed as a workgroup or database cluster model solution that supports up to four 12-port 4X InfiniBand leaf modules. The 7874-040 includes a DDR managed spine module, redundant power supplies, redundant fans, power cords, rack mount kit, and QLogic InfiniBand Fabric Suite.
  • The IBM 7874-120 is a QLogic 9120 144-port 4X DDR InfiniBand Fabric Director Switch designed to support larger high performance clusters. This model supports up to twelve 12-port 4X InfiniBand leaf modules. The 7874-120 includes redundant DDR managed spine modules, a full complement of spine modules to provide a 100 percent nonblocking fabric for all 144 ports, redundant power supplies, redundant fans, power cords, rack mount kit, and QLogic InfiniBand Fabric Suite.
  • The IBM 7874-240 is a QLogic 9240 288-port 4X DDR InfiniBand Fabric Director Switch designed to support very large high performance clusters and supports up to 24 12-port 4X InfiniBand leaf modules. This model includes redundant DDR managed spine modules, a full complement of spine modules to provide a 100 percent nonblocking fabric for all 288 ports, redundant power supplies, redundant fans, power cords, rack mount kit, and QLogic InfiniBand Fabric Suite.
The IBM Power Systems servers, host channel adapters, and cables that are supported for attachment to the machine type and model numbers of the QLogic InfiniBand switches are shown in the following table.
Table 1. Supported server, host channel adapter, data rate, and cable features
System Machine type and model number Host channel adapter feature code Data rate Cable feature code

Power 520 Express™

8203-E4A

5616

SDR

1828 (1.5 m (5 ft)),
1841 (3 m (10 ft)),
1842 (10 m (33 ft))

5608

DDR

1828 (1.5 m (5 ft)),
1841 (3 m (10 ft)),
18421 (10 m (33 ft))

Power 550 Express

8204-E8A

5616

SDR

1828 (1.5 m (5 ft)),
1841 (3 m (10 ft)),
1842 (10 m (33 ft))

5608

DDR

1828 (1.5 m (5 ft)),
1841 (3 m (10 ft)),
18421 (10 m (33 ft))

Power 560 Express

8234-EMA

1802

SDR

1828 (1.5 m (5 ft)),
1841 (3 m (10 ft)),
1842 (10 m (33 ft))

Power 570

9117-MMA

1802

SDR

1828 (1.5 m (5 ft)),
1841 (3 m (10 ft)),
1842 (10 m (33 ft))

Power 575

9125-F2A

5612

DDR

32912 (10 m (33 ft)),
32922 (20 m (66 ft)),
32942 (40 m (132 ft))

JS22 Express

7998-61X

8258

DDR

  

BladeCenter® H Chassis

7989-BCH

3216 passthrough module

3246 (3m (10 ft)),
3247 (8m (26 ft))
Notes:
  1. Feature code 1842, when attached to host channel adapter feature code 5608, can only be attached to a base port of the QLogic switch. Base ports are the bottom row of connectors on the 7874-024, or the bottom row of connectors on the leaf modules.
  2. For more information on optical cables, see the following section titled Usage notes for optical cables (feature codes 3291, 3292, and 3294).

The switches are field mounted in an IBM 19-inch rack that has available rack space and power distribution unit (PDU) outlets. The switches have 1-meter power cords to connect to the rack PDUs. The power connections are on the rear of the switch assemblies, located on both the right side and on the left side. The power cords connect to the rack PDUs, which are required on both the left side and right side of the racks. The 7874-024 and 7874-040 can be mounted in any IBM 19-inch rack. To address the need for larger system cable management of the InfiniBand cables, the 7874-120 and 7874-240 must be installed in a 7014-T42 rack with the rack extender feature code 6257. The rack extender is 20-inches deep.

High performance cluster configurations require the use of an external host subnet manager (HSM). This runs on an IBM system x3550 or IBM system x3650, with InfiniBand host channel adapter and InfiniBand cable. Two host channel adapters within the HSM server are required for redundancy or fail over. The InfiniBand host channel adapters and the InfiniBand cables for the HSM server must be obtained directly from QLogic Corporation.

The QLogic number for the specific HCA is 7104-HCA-LPX2P-DDR, which has a description of PCI Express 2.5 Ghz x8 20-Gbps Dual CX4 0MB adapter. QLogic has a number of 4x-4x InfiniBand cables for the HSM host channel adapter. The specific numbers are in the form of 7104-#M-Cable (For example, a 3-meter cable would be number 7104-3M-Cable with a description of 4x - 4x Cable - 3 Meters. You must choose non-SDR-only cables from QLogic.

Commercial configurations can use the embedded subnet manager (ESM), which is monitored from the network. Configurations with greater than 144 nodes require the use of the HSM.

The HSM runs the InfiniBand Fabric Suite, which is downloadable from the QLogic IBM Supported Software Web site (http://support.qlogic.com/support/oem_ibm.asp).

An approval key is required to download the software and is included in the switch packaging along with other documentation.

The 4x to 4x DDR Optical Cables can be used as an interswitch link (ISL) cable to interconnect supported InfiniBand switches in large special bid configurations. The 4x to 4x DDR Optical Cables are feature code 3301 (10 m (33 ft)), feature code 3302 (20 m (66 ft)), and feature code 3300 (100 m (330 ft)).

Usage notes for optical cables (feature codes 3291, 3292, and 3294)

When optical cables (feature codes 3291, 3292, and 3294 of 9125-F2A) are connected to QLogic InfiniBand switches (IBM machine type 7874 or InfiniBand switches supplied by QLogic), the preemphasis port attribute must be turned off for the ports being used. To turn off the preemphasis port attribute, use the command-line interface (CLI) of the switch. The CLI is accessed by using Telnet, Secure Shell (SSH), or the switch RS-232 serial ports.

The commands in the following table should be issued in the context of the "InfiniBand Switch Installation and Configuration Procedure" when performed during initial switch setup. This procedure is found in IBM System p HPC Clusters Fabric Guide Using InfiniBand Hardware at the IBM clusters with the Infiniband switch web site.

Note: The switch must be restarted for the commands to take effect. This action is critical if the change is being made to a switch that is already installed and in production.

The preemphasis attribute can be changed per port, or for all external ports (7874-240), using commands provided on the switch CLI. For the 7874-024, ports are referred to as cables. The Portname port is "Cable x" (quotation marks required), where x is the port in question. For all models except the 7874-024, the Portname port for port y on leaf module z is LzPy. See the following table for command information.

Table 2. Command information
Attribute Command1, 2 Command result
Per port ismPortSetDdrPreemphasis Portname 0x00000000 Turns off preemphasis attribute for the Portname port.
Per switch ismChassisSetDdrPreemphasis 0x00000000 Turns off preemphasis attribute for all external ports3
Notes:
  1. If the preceding commands are issued without the parameter 0x00000000, then they act as queries, and the preemphasis status is returned for the port in question (in the case of ismPortSetDdrPreemphasis Portname) or for all external ports (in the case of ismChassisSetDdrPreemphasis (per hemisphere for the 7874-240 model)).
  2. The ismPortSetDdrPreemphasis and ismChassisSetDdrPreemphasis commands can be run remotely using the FastFabric cmdall command or CSM dsh command. If you must set individual ports using the ismPortSetDdrPreemphasis command, a script could be written to do this using a remote command.
  3. The 7874-240 has two hemispheres (upper and lower). Leaf modules 1 - 12 are located in the lower hemisphere; leaf modules 13 - 24 are in the upper hemisphere. Access to the CLI is on a per-hemisphere basis. To change the preemphasis setting for a port located on leaf modules 1 - 12, access the CLI on the lower hemisphere. For a port located on leaf modules 13 - 24, access the CLI on the upper hemisphere. To set preemphasis off for all external ports on the 7874-240, the ismChassisSetDdrPreemphasis 0x00000000 command must be run on both hemispheres.

InfiniBand switches specifications

For information about InfiniBand switches, including physical, environmental, cooling and thermal management, and power specifications, see to the Installation Guide at the QLogic Web site.

IBM and QLogic models

When referencing QLogic documentation, use the following table to determine which QLogic model is associated with your IBM model.
Table 3. IBM and QLogic models
IBM model QLogic model
7874-024 QLogic 9024CU Managed 24-port 4X DDR InfiniBand Edge Switch
7874-040 QLogic 9040 48-port 4X DDR InfiniBand Fabric Director Switch
7874-120 QLogic 9120 144-port 4X DDR InfiniBand Fabric Director Switch
7874-240 QLogic 9240 288-port 4X DDR InfiniBand Fabric Director Switch