next_group up previous Fibre Channel HOW TO Maintainer:[1]Erling Nygaard, [2]Binary Operations Research Group (BORG), [3]University of Minnesota This HowTo is intended to give people unfamiliar with Fibre Channel an introduction to the concepts and possibilities offered by this exiting technology. It will not answer all questions regarding Fibre Channel, but will hopefully be a starting point. 1. Introduction 1.1 This document This is the [4]BORG's attempt to assimilate you into the world of Fibre Channel. Hopefully you will find this document somewhat useful and maybe you will learn something about the wonderful world of Fibre Channel. If anyone cares to know why this is not using the standard HOWTO building tools, it is because they don't have the support for the amount of structure this document requires. (They do not go beyond two levels of headings, and we have four.) So we switched over to latex. And a very happy with the results. (and LATEX2HTML ) 1.2 Brief history Fibre Channel is an open standard as defined by ANSI (American National Standards Institute) and OSI (Open Systems Interconnect) standards. 1.3 Using existing protocols for mapping data One of the features of Fibre Channel is to use existing, well-established protocols for transportation of data. It does not impose any new format for data, hence all existing applications can continue to use the standard protocols. Supported protocols include SCSI, IP, HIPPI and more. 1.4 Flexible Topologies Fibre Channel offers 3 different topologies that can be combined to fit any needed configuration. The simplest topology is simply a point-to-point connection between two nodes. A more advances topology is available by using an Arbitrated Loop. This configuration allows up to 126 devices sharing one loop. Finally, Fibre Channel Switches are available that can form fabrics with thousands of nodes. 1.5 Cabling Fibre Channel can use both optical and copper cabling. When using single-mode optical fiber and nodes with 1300nm lasers, distances up to 10km between nodes are supported. With multi-mode fibre distances up to 500 meters are supported, and electrical cabling is available up to 13 meters. It is important to make sure all cabling used meet the specifications for FC-cables. 1.5.1 Connectors With optical cabling, the connectors are SC connectors. With electrical cabling, the connectors are ether DB-9 connectors or HSSDC (High Speed Serial Data Connection). 2. Fibre Channel Ports All equipment connected to Fibre Channel have one or more FC Ports. These ports are entities capable of sending and receiving data under the Fibre Channel Protocol. For a computer, the Host Bus Adapter is the port. Fibre Channel specifies different ports with different purposes. Although all the different ports physically can be connected together, only logical valid connections will provide a Fibre Channel connection. 2.1 N-Ports N-Ports (Node-Ports) are the simplest ports. A N-Port can only participate in a point-to-point connection. A node with a N-Port only connects to another node with a N-Port or to a Switch. Using a switch means that more nodes can be reached, even with just a N-Port. 2.2 NL-Ports NL-Ports (NodeLoop-Ports) are node-ports that contain the extra functionality to participate on Fibre Channel Arbitrated Loops. 2.3 F-Ports F-Ports (Fabric-Ports) are the ports used on a Switch. A F-Port can only participate in a point-to-point connection with a N-Port. 2.4 FL-Ports FL-Ports (FabricLoop-Ports) are used on a Switch. This port contains the extra functionality to participate on Fibre Channel Arbitrated Loops. When connection a Arbitrated Loop on a Switch, all devices on the loop will have access to all other devices connected to the switch. 2.5 E-Ports E-Ports (Expansion-Ports) are ports used for Switch to Switch communication. This allows several switches to be connected, forming larger Fabrics. 2.6 G-Ports G-Ports (Generic-Ports) are ports on a Switch that will behave as a E-Port, FL-Port or F-Port, depending on what it is connected to. 2.7 L-Ports L-Port (Loop-Port) is not a term in Fibre Channel, but often used as a generic term for NL-Ports and FL-Ports when one from the context can determine what port is meant. 3. Upper Layer Protocols 3.1 Introduction Fibre Channel do not specify any format for data, but allows other, well-established protocol to use FC as a transport layer. The advantage of this is obvious, applications do not need to be rewritten in order to use Fibre Channel. 3.2 Upper Layer Protocols Supported by Fibre Channel Protocols supported on Fibre Channel are: * SCSI (Small Computer System Interface) * IP (Internet Protocol) * IPI (Intelligent Peripheral Interface) * HIPPI (High Performance Parallel Interface) * Link Encapsulation (FC-LE) using International Standard (IS) IS8802.2 * Single Byte Command Code Set Mapping (SBCCS) to implement ESCON and block multiplex interfaces * Audio Video Fast File Transfer * Audio Video Real Time Transfer * Avionics 4. Classes of Service Fibre Channel offers the possibility to use different classes of service for the data. Each class has it's advantages and disadvantages. It is important to remember that nodes are not required to implement all of the classes, most implement just a sub-set. And when using a switch, the switch must also have the requested classes implemented. 4.1 Class 1, Acknowledged Connection Service Class 1 provides true connection service. The result is circuit switched, dedicated bandwidth connections. Fibre Channels advantages is connection setup and tear-down measured in microseconds. An end-to-end path between the connecting devices is established through the Fabric. Since only the overhead for Class 1 is connection setup and tear-down, it is an efficient service for large data exchanges. Fibre Channel Class 1 service provides an acknowledgement of receipt for guaranteed delivery and bandwidth for applications like image transfer and storage backup and recovery. Some applications use the guaranteed bandwidth delivery feature to move data reliably and quickly without the overhead of a network protocol stack. "Camp on" is a Class 1 feature that enables a Switch to monitor a busy port and queue that port for the next connection. As soon as the port is free, the Switch makes the connection. This switch service speeds connect time, rather than sending a "Busy" back to the originating N_Port and requiring the N_Port to retry to make the connection. "Stacked Connect" is a Class 1 feature that enables an originating N_Port to queue sequential connections with the Switch. Again, this feature reduces overhead and makes the Switch service more efficient. Class 1 has a variation called buffered service. It is used to connect two Fibre Channel ports that are operating at different speeds There is also a form of Class 1 called Dedicated Simplex Service. Normally, Class 1 connections are bi-directional; however, in this service communication is in one direction only. This service is used to separate transmit and receive switching. It permits one node to transfer to another node while s simultaneously receiving from a third node. 4.2 Class 2, Acknowledged Connectionless Service Class 2 is a connectionless service, independently switching each frame and providing guaranteed delivery with an acknowledgement of receipt. As with traditional packet-switched (connectionless) systems, the path between two interconnected devices is not dedicated. The fabric multiplexes traffic from N_Ports and NL_Ports without dedicating a path trough the Fabric. Class 2 credit-based flow control eliminates congestion that is found in many connectionless networks. If no buffer space is available, a "Busy" is sent to the originating N_Port. The N_Port will then re-send the message. This way no data is arbitrarily discarded just because the Switch is busy at the time. Typical Class 2 frame latency is less than one microsecond, making it ideal for shorter data transfers. 4.3 Class 3, Unacknowledged Connectionless Service Class 3 is a connectionless service, similar to Class 2, but no confirmation of receipt is given. This unacknowledged transfer is used for multicasts and broadcasts on networks and for storage interface on Fibre Channel loop. The loop establishes a logical point-to-point connection and reliably moved data to and from storage. Class 3 Arbitrated Loop transfers are also used for IP networks. Some applications use logical point-to-point connections without using a network layer protocol, taking advantage of Fibre Channels reliable data delivery 4.4 Class 4, Fractional Bandwidth Connection Oriented Service Class 4 is fractional bandwidth, connection-oriented service. Virtual connections are established with bandwidth reservations for a predictable quality of service (QoS). A Class 4 connection is bi-directional with one virtual circuit operational in each direction and supports a different set of QoS parameters for each virtual circuit. These QoS parameters include guaranteed bandwidth and latency. A quality of service facilitator function is provided within the Switch to manage and maintain the negotiated quality of service on each virtual circuit. When a Class 4 connection is active, the switch paces frames from the source node to the destination node. Pacing is the mechanism used by the Switch to regulate available bandwidth per virtual circuit. This level of control permits congestion management for a switch and guarantees access to the destination node. The Switch multiplexes frames belonging to different virtual circuits between the same or different node pairs. Class 4 service provides in-order delivery of frames. Class 4 flow control is end-to-end and provides guaranteed delivery. Class 4 is ideal for time-critical applications. 4.5 Class 6, Simplex Connection Service Class 6 is similar to Class 1, providing simplex connection services. However, Class 6 also provides multicast and preemption. Class 6 is ideal for video broadcast applications and real-time systems that move large quantities of data. 4.6 Intermix FC has an optional mode called Intermix. Intermix allows the reservation of full FC bandwidth for a dedicated (Class 1) connection but also allows connectionless traffic within the fabric to share the link during idle Class1 transmission. 5. Fibre Channel Topologies 5.1 Point to Point This was the first and most simple Fibre Channel topology. It connects to nodes, be it 2 host machines, one host machine and one storage device or one host and a switch. There are several benefits of this setup: * No need for arbitration * No addressing problems, the package you receive must come from the other side. * The devices have access to the full bandwidth of the link. However, there are several arguments against this topology: * Restricted to communication between two nodes However, constant access to 100Mbytes/s is usually not needed. This kind of topology may be useful for connecting two machines that are physically separated by some distance. 5.2 Fabrics Switches were introduced to be able to connect several devices together. The idea was to have several point-to-point connections, each point-to-point connection connecting one node to the switch. This way several nodes would be able to connect together in a larger topology. The topology that is made when using one or more switches is called a fabric. 5.3 Arbitrated Loops Arbitrated Loops were actually added on to the original FC-specifications. It was realized that there would be a need for a topology between the point-to-point and the switched topologies. The Arbitrated Loop combines some of the features from the Point-To-Point with some of the features from the fabric. One of the main arguments against the pure fabric is the price pr. fabric port is to high if each fabric-port is just connected to just one node. Although this offer each device full connection to the fabric, most nodes to not constantly need the full bandwidth of Fibre Channel. The Arbitrated Loop offers (As the name suggests) a loop topology. The loop can contain up to 126 NL-ports and one fabric-port with all the nodes arbitrating for usage of the loop. The arbitrated loop is made up by having all nodes connect their outgoing connector to the incoming connector of the downstream node. This way, all nodes on the loop as repeaters for all the frames running around the loop. At any given time, only one port can be sending frames. (one exception: if both of the ports that want to communicate support full-duplex, they may agree to set up such a connection and hence fully utilize the connection in both directions.) An arbitration procedure selects the Loop Master, who thereafter owns the loop. Since any the nodes on the loop must win an arbitration before being allowed to use the loop, performance will degenerate as more nodes are added. A loop with 127 active ports is likely to not operate with the performance one would like. 5.3.1 Physical Layout (Hubs / No Hubs) Since all ports on the loop act as repeaters for all the packets running around, all nodes must be present and active. If a node is removed or incapable of forward a packet, the loop will break. Given the limits on maximum distance between nodes, one might be in the situation where one can not just remove a node and somehow connect the wires in order to repair the loop. Actually, most Fibre Channel equipment is not intended for being put into a physical loop. For copper cables, the receiver and transmitter wires are inside of the same physical cable and use one physical connector to connect to the node. Splitting the cable and connector is absolutely not recommended. Only with optical cabling can a real physical loop be constructed. (With most optical cables one can split the two optical fibers and get two "one-way" optical cables.) Most people do not consider this to be a problem, since having the cables form a physical star usually is preferred over having a physical loop. In Fibre Channel this is done with the use of hubs. All devices connect directly to the hub. Although this still gives a loop, both logically and electrically / optically, the physical cabling will have the much more convenient star layout. An even more important reason for using a hub is reliability. When using a hub and one of the nodes is removed or goes down, the hub will bypass that node and the loop will still be operable. And as long as each connection between the hub and nodes are within the distance limits, bypassing one node will not increase the distance between any of the remaining nodes, hence the loop will still be within the distance limits. Hubs may be of different levels of complexes. The simplest hubs will just contain several Fibre Channel connectors and hardware to detect the presence of nodes on the different connections as well as hardware to bypass the ports with no nodes attached. These are "dump" hubs, usually cheap but reliable. There also exist more advanced hubs with management options. These hubs may let the administrator disconnect nodes from the loop without pulling the connectors, and might be configurable so that the one hub may be divided up in several logical hubs, giving several independent loops. 5.3.2 Private Loops A Loop that consist of up to 126 NL-ports is considered to be a private loop. These nodes can access all other nodes on this loop, but can not (by using Fibre Channel) access any other nodes. 5.3.3 Public Loops When the FL-port of a switch is connected to the loop the loop becomes a public loop. Assuming there are more nodes connected to the switch, the nodes on the loop may access and be accessed by these nodes. By connecting several public loops to a fabric a huge number of nodes may be interconnected. In each of the loop the nodes arbitrate for the full use of the bandwidth of that loop. When a FL-port is part of a loop, arbitration is no longer totally fair, the FL-port will have higher priority than other ports on the loop. In order to be able to address nodes outside the local loop, a node must be capable of supporting Public Loops. If there are nodes on the loop that do not support Public Loops, these nodes will alone be able to address nodes outside the local loop. However, some switches incorporate functionality to help these nodes communicate outside the local loop. See paragraph [5]7.2 for more info. 6. HBA's supported by Linux 6.1 Qlogic 2100 and 2200 6.1.1 Qlogic 2100 The Qlogic 2100 is a FC-HBA produced by [6]Qlogic Corporation This card is based on the ISP2100 chip. The only Upper Layer Protocol supported by this card is SCSI, and the card supports both Fibre Channel Service class 3. The card supports both FC-AL Private and Public loops. It does not support point-to-point connections. The card is available as a 64 bit PCI HBA. 6.1.2 Qlogic 2200 The Qlogic 2200 is a FC-HBA produced by [7]Qlogic Corporation This card is based on the ISP2200 chip. Upper Layer Protocol supported by this card includes both SCSI and IP, and the card supports both Fibre Channel Service classes 2 and 3. The card supports both FC-AL Private and Public loops. It does not support point-to-point connections. The card is available as a 64 bit PCI HBA. 6.1.3 Notes The card uses and on-board database for addressing purposes. This database can contain up to 255 entries, with entries reserved 00h - 7Dh for nodes on the local loop. (A loop may contain 125 nodes in addition to the card itself) When connecting a fabric port (F-Port) to the loop, the Private Loop becomes a Public Loop. Entries 7Eh, 7Fh and 80h are reserved for the FL-port, the fabric controller and the Name Server. This leaves entries 81h - FFh (a total of 126) available for nodes "out on the fabric", meaning that the card can "only" access 126 nodes outside it's local loop. We have yet not seen this as a problem, since the group have not had access to fabrics this large. We are not sure how the card reacts to more nodes being present, but assume that more nodes simply will be ignored. The above is based on the specifications for the 2100. We assume (but currently don't know) that a similar limit exists for the 2200. In the testing we have found that there are no problems intermixing these cards. 6.2 Linux Drivers for the Qlogic 2x00 cards 6.2.1 University of New Hampshire InterOperability Lab Driver written by [8]Chris Loveland [9]University of New Hampshire InterOperability Lab The qlogicfc driver has been included in the Linux kernel as of v2.2.3. (v2.2.11 for the 2200) The most recent version of the driver can be obtained [10]here. This distribution package is a gzipped tar archive which contains the three files which make up the qlogicfc driver: qlogicfc.c, qlogicfc.h, and qlogicfc_asm.c. (Firmware) To update to the current version of the driver just replace the corresponding files in the drivers/scsi directory of the Linux kernel source and recompile. This driver is working and stable for BOTH x86 boxes as well as as Alpha's. It also works for multiprocessor machines. It has been tested on dual processor Intel-boxes as well as an ES40 Alpha-Server (4 Alphaprosessors) This driver supports SCSI for both of the cards. The IP functionality for the 2200 is not supported and I am not aware of any attempt to support it. Since this driver comes with the standard kernel, enabling support is just like enabling any other SCSI adapter. When compiling the kernel, include "Qlogic ISP FC SCSI support" under SCSI low-level drives. Making the driver a module is a good thing. Make sure that the Qlogic card is correctly inserted in a PCI-slot. Yes, since the card is a 64 bit PCI card some pins on the card may not be used on your PCI-system. No problem. Connect whatever Fibre Channel setup you want to the card. It is best (although not required) to turn on disk etc before turning on the computer. If you are using a PC, you may hit "CTRL Q" during boot up. This enables you to go into the setup of the card, where you can do stuff like: * Set the Loop ID of the HBA. (In accordance with the FC-Specs this is the address the HBA now will TRY to get, but in the case of address-conflicts, the HBA may get another Loop ID * Frame size. How big FC-packets should be used? * Scan the Fibre Channel network. This allows you to check that the HBA actually can see all the devices it is supposed to see. (Testing this saves time when you forgot a terminator or properly connecting a cable or something like that...) * Enabling BIOS. This should let you boot from a Fibre Channel disk, although this is something we have never tried. (Nor do we know about anybody who have) After the computer is Up & Running, type "insmod qlogicfc". This will install the module. Remember that on a large configuration, this may take some time since the HBA has to go out and talk to ALL other devices on the FC-network. When the insmod is done, do a "dmesg" to verify that all drives you expected actually were detected. (Or do a "more" on "/proc/partitions" or "/proc/scsi/scsi") If some disks are missing, do a "rmmod qlogicfc" and repeat the process. If disks still are missing, you might want to check cabling, that disks are turned on etc. The Fibre Channel sub-system is now ready for usage. Notes: Fabric (Public Loop) support is disables by default. If you plan to use a fabric setup, you need to edit the qlogicfc.c file before compiling it. The file is in drivers/scsi directory of the Linux kernel. Here is the change: (Around line 80 of the file) /* Set the following to 1 to include fabric support, fabric support is * currently not as well tested as the other aspects of the driver */ #define ISP2x00_FABRIC 0 Change this to: /* Set the following to 1 to include fabric support, fabric support is * currently not as well tested as the other aspects of the driver */ #define ISP2x00_FABRIC 1 If you have large collection of Fibre Channel disk or plan to do large I/O transfers, you risk getting a combination of the following error-message: qlogicfc#.c: no handle slots, this should not happen qlogicfc#.c : got a null value out of handle_ptrs, this sucks A solution to this problem is to increase ``QLOGICFC_REQ_QUEUE_LEN'' This is defined in qlogicfc.h, around line 73. The current default is 127, we have successfully used 255 when running into this problem. Just remember that this number must be ((power of two) - 1) 6.2.2 Feral Software Driver written by [11]Matthew Jacob [12]Feral Software 6.3 Interphase 5526 The Interphase 5526 is a FC-HBA produced by [13]Interphase Corporation This card is based on the HP Tachyon(TM) Fibre Channel Protocol engine. Upper Layer Protocol supported by this card includes both SCSI and IP, and the card supports both Fibre Channel Service classes 1, 2 and 3. The card supports both FC-AL Private and Public loops as well as point-to-point connections. 6.3.1 Notes Interphase stated that the card can support 126 devices concurrently. Since we do not have technical specifications for this card, we do not know if this card has an issue with addressing nodes outside the local loop. 6.4 Linux Drivers for the Interphase 5526 6.4.1 University of New Hampshire InterOperability Lab The Interphase driver has been included in the Linux kernel as of v2.2.11. The most recent version of the driver can be obtained [14]here. This driver is intended to provide both IP and SCSI services. In our testing, we have been able to use the IP service successfully. During testing we have had two hosts connected over a Fibre Channel switch, thus showing that the driver correctly handles fabric setups. However, the we have never been able to use the SCSI part. Although the driver finds SCSI-disks and successfully makes them available to the host, simple tasks like making a file-system or writing to the devices crashes the driver (And in most cases the host) badly. 6.4.2 Interphase Interphase have also written their own driver. This driver is unfortunately not available unless you contact Interphase. However, they don't really seem to have a problem with this, so give it a try. This driver is more complex to get up and running than other drives. In addition to the driver itself, the package has a daemon that offers a web-interface for remotely configuring the behavior of the driver as well as reporting performance. 6.4.3 Notes / Problems There are currently some problems with this driver. 6.4.3.1 Re-initialization When the Fibre Channel configuration changes, the driver should go out and find any new nodes that may have appeared. The driver should do this: * On a loop, a device connects and causes a LIP (Loop Initialization Protocol) * On a fabric, a new device connects. The Name Server sends out a ``Fabric State Change Notification'' to inform everybody that a new node has joined the fabric. However, in most cases this driver will go out, find all attached devices and consider them ALL as new devices. It will then assign SCSI IDs under the assumption that the devices were not previously assigned an ID. So a device that used to be mapped to ``Host 2, Channel 0, ID 3, LUN 0'' might suddenly be ``Host 2, Channel 0, ID 17, LUN 0''. The SCSI ID changed! If this device ID used to be mapped to, let's say /dev/sde it will no longer be there! (Just imagine what happens when a device just ``disappear''.) 6.4.3.2 Reporting devices to SCSI-layer When the driver finds devices, they are not always reported. When comparing /proc/scsi/scsi and /proc/scsi/ip5526, there might be several devices reported in the later. Before the devices can be used, you need to perform echo ``scsi add-single-device a b c d'' > /proc/scsi/scsi where a is the HBA#, b is the Channel#, c is the ID# and d is the LUN# All of these numbers are reported under /proc/scsi/ip5526 7. Other FC hardware 7.1 Hubs Hubs are used to set up a Fibre Channel Arbitrated Loop (both private and public). A hub do not in any way increase the performance of a loop, all devices attached to the loop still must share the bandwidth of the loop. Hence, for a setup with more than just a few hosts and storage devices, just using a hub will most likely not give the performance one would like. However, creating a loop without a hub is almost impossible. In section [15]5.3 we mentioned that for most electrical Fibre Channel cables the incoming and outgoing wire are inside one physical cable, attached to one physical connector. This makes splitting the cable impossible. Only with optical cabling is there a hope of creating a loop without a hub. However, most Fibre Channel Storage Devices and storage enclosures comes with 2 connectors for the same loop. This allows such devices to be physically "daisy chained" (although still making a loop) or to have to hosts connected to the device. Hubs have lately become more intelligent. There exists managed hubs, witch allow changing configurations without doing any recalling. Usually, these hubs have a great number of connectors and are configured to make up several independent loops. An administrator changes the configurations as needed. This provides an easy way to manage a larger number of nodes, but it is important to remember that the bandwidth of each loop is not increased and still has to be shared among the nodes on the loop. There also exists more simple hubs that just contain FC-connectors and enough internal hardware to connect the external connectors, detect the presents / absence of nodes connected to a connector and bypass-circuitry for the connectors where nothing is connected. Although these hubs to not have a high "coolness" factor, they are usually exactly what is needed when creating a Fibre Channel Loop. 7.2 Switches Fibre Channel switches allows large Fibre Channel topologies. With hubs, all nodes have to share the bandwidth of one Fibre Channel hub, giving limited performance when more than a few nodes are connected. When a larger network is to be constructed, switches will be the logical choice. Several Fibre Channel switches are available from different vendors. One thing to note is that switches from different vendors may not be able to communicate, meaning they can not be part of the same fabric. We have had access to several switches from [16]Brocade, thus all our experience is based on these switches. Switches come with different number of ports, 8 or 16 being the usual. Each of these ports can then connect to an Arbitrated Loop (connecting up to 126 nodes to this port), directly to a N_Port (connecting just one node to the port) or another switch. By connecting several switches large numbers of nodes will be connected. For a switch to be so generally usable, one has to make sure that all the ports are G_Ports. For some switches all the ports are G_Ports, for others the port-type must be changed by swapping daughter-boards inside the switch. See chapter [17]2 for more details on different ports. The [18]BORGs have successfully managed to set up a configuration consisting of 4 interconnected Brocade Silkworm 2400's, 8 hosts (4-processors Alpha, single processor Alpha's and PC's) and 475 Gigabyte of storage. 7.3 Disks and Enclosures Fibre Channel disk are just modified SCSI disks. The disks have a Fibre Channel interface and accepts SCSI commands encapsulated in Fibre Channel frames. Most Fibre Channel disks are dual ported, meaning it can be connected to two different Fibre Channel networks. Although this is possible, one must remember that the two networks now must share the performance of the disk. This feature is most likely to be used for redundancy purposes, where a second network will take over if the primary network fails. We have never configured any such networks. The physical connector on the Fibre Channel disk are not intended to be connected directly on a Fibre Channel network. Instead the disk are intended to be connected to a backplane. The backplane contains a Port Bypass Circuit for each disk, which will bypass an open slot or a failed disk. In most cases a backplane is build into an enclosure (JBOD, Just a Bunch of Disks), able to contain a large number of disk. There exists special cards that acts like a backplane for a single disk. 7.4 GBIC's GBIC's (Gigabit Interface Converters) offer flexibility for cabling options. Most switches and hubs do not come with actual Fiber Channel connectors, but instead they have a standard slot in which the GBIC's are inserted. The user can then choose GBIC's depending on what cables are to be used. GBIC's are available as Optical, DB-9 and HSSDC. In our work with Fibre Channel we have found that switches and most hubs can use GBIC's. A few hubs come with build-in FC-connectors, (usually DB-9). HBA's usually can not use GBIC's, but have a connector build on. There are other HBA's that comes with a GLM's (Gigabaud Link Modules), a small daughter-board that allows you to change the FC-interface. None of these cards are currently supported under Linux. All of the storage devices we have used come with DB-9 connectors. We have still not seen or heard about storage devices being shipped with any other connectors. 7.5 Cables Just like GBIC's, cables come with different connectors as well as different length and quality. This is a list for the different cables we have used * Optical (mulitmode). Mostly for interconnection between switches. * DB-9 to HSSDC. Useful when connecting storage devices to HBA's having HSSDC connectors. * DB-9 to DB-9. Interconnecting storage devices, connecting storage devices to DB-9 HBA's. DB-9 to DB-9 cables are available in different length, so their usage is flexible. Note: You should treat your Fibre Channel Cables with respect. We have a couple of times experienced that cables have gone bad. The problem is that there is no way to tell that a cable is bad just by looking at it. If a connection is "broken" or performing badly, trying a new cable should be the first thing to do. 7.6 Terminators Storage devices usually come with 4 FC-connectors. This is somewhat confusing in the beginning. The reason is that disks usually are dual-ported and can be part of 2 different loops at the same time. For each of these loops, the disk can be connected a node "upstream" as well as "downstream", hence there are 2 connectors for each loop. If the storage devices is the last on a "daisy chain", there need to be a terminator on the unused connector otherwise the Fibre Channel loop will be broken. There is no need to have terminators on the 2 connectors for the "other" loop if this loop is unused. (Which it will be in most cases) 7.7 Analyzer An analyzer is definitely a cool thing to have, although you don't need one in any production environment. If, however, you are interested at looking at Fibre Channel packets as they fly by, I would recommend buying one. 8. Glossary Bandwidth Maximum effective transfer rate for a given set of physical variants such as communication model, Payload size, Fibre speed and overhead specified by FC-PH. Coaxial Cable An electrical transmission medium consisting of concentric conductors separated by a dielectric material with the spacings and material arranged to give a specified electrical impedance. Coffee A beverage made by percolation, infusion, or decoction from roasted and ground or pounded seeds of a coffee tree. Credit The maximum number of receive buffers allocated to a transmitting Port. It represents the maximum number of outstanding frames which can be transmitted by that Port without causing a buffer overrun condition at the receiver. D_ID Destination Identifier, the address used to indicate the targeted destination of the transmitted frame. Destination Port The Port to which a frame is targeted. E_D_TOV Error Detect Timeout value. Exchange The basic mechanism which transfers information consisting of one or more related non-concurrent sequences which may flow in the same or opposite directions. An exchange may span multiple Class1 dedicated connections. The exchange is identified by an Originator Exchange_Identifier (OX_ID) and a Responder Exchange_Identifier (RX_ID). Exchange_Identifier (X_ID) A generic reference to OX_ID and RX_ID (See Exchange). F_BSY Fabric Port Busy. F_BSY(DF) F_BSY response to a data frame. F_BSY(LC) F_BSY response to any link control except P-BSY. F_CTL Frame Control. F_Port Fabric Port. Fabric The entity which interconnects various N_Ports / NL_Ports attached to it and is capable of routing frames by only using the D_ID information in the frame header. FCP Fibre Channel Protocol. FC-PA ANSI X3.230-1994, Fibre Channel Physical and Signaling Interface. Fibre A general term used to cover all transmission media specified in FC-PH. FL_Port An F_Port that contains Arbitrated Loop functions associated with Arbitrated Loop topology. Frame An indivisible unit of information used by FC-2. Frame Content The information contained in a frame between the Start-of-Frame and End-of-Frame delimiters, excluding the delimiters. Hard Address The AL_PA which an NL_Port attempts to acquire in the LIHA loop initialization Sequence. Initiator A SCSI device containing application clients that originate device requests and task management functions to be processed by a target SCSI device. Interface connector An optical or electrical connector which connects the media to the Fibre Channel transmitter or receiver. L_Port A term used for describing a N_Port or F_Port that contains Arbitrated Loop functions associated with Arbitrated Loop topology. LAN Local Area Network. LED Light emitting diode. LOGI Log in. Login_BB_Credits On FC-AL, equal to the number of receive buffer that a receiving NL_Port must have available when a loop circuit is established. Login_BB_Credit is discovered in the PDISC or PLOGI protocol. Loop_ID Loop_IDs are 7-bit values numbered contiguously from 0 to 126 and represent the 127 legal hard addresses on a loop (not all of the 256 possible PA_ALs are used in FC-AL for reasons related to running disparity). Loop_IDs correspond to the 7-bit SEL word in SFF-8045 used for specifying hard addresses. Decimal 127 (7F hex) is not a valid Loop_ID, but is used to signify that no hard address is being assigned to an NL_Port. Meaningful A control field or bit shall be applicable and shall be interpreted by the receiver, whenever it is specified as meaningful. Whenever it is specified as ``not meaningful'' it shall be ignored. N_Port A hardware entity which include a Link_Control_Facility. It may act as an originator,a responder, or both. N_Port Node_Port. N_Port Identifier A fabric-unique address identifier by which a N_Port is uniquely known. The identifier may be assigned by the fabric during the initialization procedure. The identifier may also be assigned by other procedures not defined in FC-PH. The identifier is used in the S_ID and D_ID fields of a frame. NA Not applicable. Name_Identifier A 64-bit identifier, with a 60-bit value preceded with a 4-bit Network Address Authority Identifier, used to identify entities in Fibre Channel such as N_Port, Node, F_Port or Fabric. NL_Port A N_Port that contains arbitrated loop functions associated with the Fibre Channel Arbitrated Loop topology. Node A collection of one or more N_Ports or NL_Ports controlled by a level above FC-2. Optical Fibre Any filament or fibre, made of dielectric material, that guides light. Payload Contents of the data field of a frame, excluding optional headers and fill bytes, if present. PDISC Discover N_Port Service parameters. Plug The cable half of the interface connector which terminates an optical or electrical signal transmission cable. Preferred Address On FC-AL, the AL_PA which a NL_Port attempts to acquire first during loop initialization. Following power-on reset, the preferred address of a private NL_Port is its hard address (if any). Following receipt of a LIP other than LIP(AL_PD,AL_PS), the preferred address of a private NL_Port is its previously acquired address. Fabric-assigned or soft addresses are not considered to be preferred addresses. Responder Exchange_Identifier (RX_ID) An identifier assigned by a Responder to identify an Exchange and meaningful only to the Responder. Running Disparity A binary parameter indicating the cumulative disparity (positive or negative) of all previously issued transmission characters. SOF Start of Frame. Source_Identifier (S_ID) The address identifier used to indicate the source port of the transmitted frame. STP Shielded twisted pair. Target A SCSI device that receives SCSI commands and directs such commands to one or more logical units for execution. Well-known addresses A set of address identifiers defined in FC-PH to access global server functions such as a name server. Worldwide_Name A Name_Identifier which is worldwide unique, and represented by a 64-bit unsigned binary value. WWN World Wide Name 9. Useful Links 9.1 Links to useful Fibre Channel pages * [19]Fibre Channel Industry Assosiation * [20]Fibre Channel Community * [21]CERN Laboratory (Switzerland) Fibre Channel Homepage * [22]ANSI T11 Standards Comittee * [23]Fibre Channel Consortium (University of New Hampshire's Interoperability Lab * [24]Global File System (University of Minnesota Bineray Operations Rersearch Group (BORG)) In addition, many vendors offer white papers on their homepages, so it is worth doing some surfing on their homepages. About this document ... Fibre Channel HOW TO This document was generated using the [25]LaTeX2HTML translator Version 99.1 release (March 30, 1999) Copyright © 1993, 1994, 1995, 1996, [26]Nikos Drakos, Computer Based Learning Unit, University of Leeds. Copyright © 1997, 1998, 1999, [27]Ross Moore, Mathematics Department, Macquarie University, Sydney. The command line arguments were: latex2html -split 0 -show_section_numbers -dir fibrechannel_howto_flat fibrechannel_howto.tex The translation was initiated by Erling Nygaard on 1999-11-17 _________________________________________________________________ next_group up previous Erling Nygaard 1999-11-17 References 1. mailto:nygaard@borg.umn.edu 2. http://www.borg.umn.edu/ 3. http://www.tc.umn.edu/ 4. http://www.borg.umn.edu/ 5. file://localhost/home/brule/nygaard/papers/fibrechannelhowto/fibrechannel_howto_flat/fibrechannel_howto.html#Switches 6. http://www.qlc.com/ 7. http://www.qlc.com/ 8. mailto:cwl@mars.iol.unh.edu 9. http://www.iol.unh.edu/consortiums/index.html 10. http://www.iol.unh.edu/consortiums/index.html 11. mailto:mjacob@feral.com 12. http://www.feral.com/ 13. http://www.interphase.com/ 14. http://www.iol.unh.edu/consortiums/index.html 15. file://localhost/home/brule/nygaard/papers/fibrechannelhowto/fibrechannel_howto_flat/fibrechannel_howto.html#Loops 16. http://www.brocade.com/ 17. file://localhost/home/brule/nygaard/papers/fibrechannelhowto/fibrechannel_howto_flat/fibrechannel_howto.html#Ports 18. http://www.borg.umn.edu/ 19. http://www.fibrechannel.com/ 20. http://www.fccommunity.org/ 21. http://www.cern.HSI/fcs/ 22. http://www.11.org/ 23. http://www.iol.unh.edu/consortiums/fc/ 24. http://www.globalfilesystem.org/ 25. http://www-dsed.llnl.gov/files/programs/unix/latex2html/manual/ 26. http://cbl.leeds.ac.uk/nikos/personal.html 27. http://www.maths.mq.edu.au/~ross/