A PCI Express cluster requires a dishosts.conf configuration file for the interconnect topology and the SuperSockets acceleration of existing Ethernet networks, and a networkmanager.conf file that contains the basic options for the mandatory Network Manager. Both of these files should be created using the GUI tool dis_netconfig. Using this tool vastly reduces the risk of creating an incorrect configuration file.
The file dishosts.conf is used as a specification of the PCI Express interconnect (in a way just like /etc/hosts specifies Cluster Nodes on a plain IP based network). It is a system wide configuration file and should be located with its full path on all Cluster Nodes at /etc/dis/dishosts.conf.
Templates of this file can be found in /opt/DIS/etc/dis/.
A syntactical and semantic validation of dishosts.conf can be done with the tooltestdishosts.
The Dolphin Network Manager and diagnostic tools will always assume that the current file dishosts.conf is valid. If dynamic information read from the network contradicts the information read in the dishosts.conf file, Dolphin Network Manager and diagnostic tools will assume that components are mis configured, faulty or removed for repair.
dishosts.conf is by default automatically distributed to all Cluster Nodes when the Dolphin Network Manager software is started. Therefore, do edit and maintain this file on the Cluster Management Node only. You should create and maintain dishosts.conf by using the Dolphin Network Configurator, dis_netconfig GUI: /opt/DIS/sbin/dis_netconfig or the command line version of the Dolphin Network Configurator, dis_mkconf: /opt/DIS/sbin/dis_mkconf
Normally, there is no reason to edit this file manually. To make changes in dishosts.conf effective, the Network Manager needs to be restarted.
# service dis_networkmgr restart
In case that SuperSockets settings have been changed, dis_ssocks_cfg needs to be run on every Cluster Node as SuperSockets are not controlled by the Network Manager.
The following sections describe the keywords used.
- #GX-HARDWARE
The first line of this file should contain an entry specifying what hardware you are going to use. If the #GX-HARDWARE is specified, parsing of the file will assume a generic adapter. This is the default for new eXpressWare 5.5.0 and newer installations.
- #MX-HARDWARE
The first line of this file should contain an entry specifying what hardware you are going to use. If the #MX-HARDWARE is specified, then Dolphin PCI Express MX is expected and the parsing of this file will fail if the hardware that you intended to use is other than Dolphin PCI Express MX.
- DISHOSTVERSION [ 0 | 1 | 2 | 3 | 4 | 5 | 6]
The version number of the dishosts.conf is specified after the keyword DISHOSTVERSION. The DISHOSTVERSION should be put on the first line of the dishosts file that is not a comment. DISHOSTVERSION 5 or higher is required for the DIS 5.0 software.
- HOSTNAME: <hostname/IP>
Each cluster Cluster Node is assigned a unique dishostname, which has to be equal to its hostname. The hostname is typically the network name (as specified in /etc/hosts), or the Cluster Nodes IP-address. Examples:
HOSTNAME: host1.dolphinics.no HOSTNAME: 193.69.165.21
- ADAPTER: <physical adaptername> <nodeId> <adapterNo> <link width>
A Dolphin network Cluster Node may hold several physical adapters. Information about a Cluster Node's physical adapters is listed right below the hostname. All Cluster Nodes specified by a HOSTNAME need at least one physical adapter. This physical adapter has to be specified on the next line after the HOSTNAME. The physical adapters are associated with the keyword ADAPTER. The link width is PCI Express link width x4 or x8.
#Keyword name nodeid adapter link width ADAPTER: host1_a0 4 0 8 ADAPTER: host1_a1 4 1 8
- REDUNDANT: <virtual adaptername> <physical adaptername 1> <physical adaptername 2>
Defines a virtual redundant adapter comprising two physical adapters, which will be used for automatic fail over in case one of the fabrics fails.
REDUNDANT: host1_r host1_a0 host1_a1
- ADAPTER: <physical adaptername> <nodeId> <adapterNo> <link width> [<serialno>]
Persistent adapter numbers are supported as optional feature in conjunction with multiple adapters. If enabled, a given physical adapter will be assigned the same adapter number independent of the presence of the other adapters. Without persistent adapter numbers the detected devices will be enumerated starting from zero. To enable persistent adapter numbers, use the cluster editor.
# dis_mkconf -fabrics 2 -sctt 2 -nodes node1::101 node2:220:221
In the example above, node1 has no serial number assigned to adapter 0, 101 assigned to adapter 1 which makes it persistent. node2 has both adapters specified as persistent: serial number 220 is set for adapter 0, 221 for adapter 1.
Only the serial number must be specified in the cluster editor tools. The adapter type and card revision are not required.
The SuperSockets configuration is responsible for mapping certain IP addresses to the PCI Express network. It defines which network interfaces are enabled for Dolphin SuperSockets.
It is passed down to the SuperSockets kernel module via the dis_ssocks_cfg command. You need to run this command to make changes in dishosts.conf effective.
Note
dis_ssocks_cfg will not be successful if there are established SuperSockets connections. You need to close all existing connections to be able to change the setup.
- SOCKET: <host/IP> <physical or virtual adaptername>
Enables the given <host/IP> for SuperSockets using the specified adapter.
In the following example we assume host1 and host2 have two network interfaces each designated host1pub, host1prv, host2pub, host2prv, but only the 'private' interfaces hostXprv are enabled for SuperSockets using a striping adapter:
SOCKET: host1prv host1_s SOCKET: host2prv host2_s
Starting with DISHOSTVERSION 2 SuperSockets can handle dynamic IP-to-NodeId mappings. I.e. a certain IP address does not need to be bound to a fixed machine but can roam in a pool of machines. The address resolution is done at runtime. For such a configuration a new type of adapter must be specified:
- SOCKETADAPTER: <socket adaptername> [ SINGLE | REDUNDANT ] <adapterNo> [ <adapterNo> ... ]
This keyword basically only defines an adapter number, which is not associated to any NodeId.
Example:
SOCKETADAPTER: sockad_s SINGLE 0 1
Defines socket adapter "sockad_s" in standard mode using physical adapters 0.
Such socket adapters can now be used in order to define dynamic mappings, and, in extension to DISHOSTVERSION 1 whole networks can be specified for dynamic mappings:
- SOCKET: [ <IP> | <hostname> | <network/mask_bits> ] <socket adapter>
Enables the given address/network for SuperSockets and associates it with a socket adapter.
It is possible to mix dynamic and static mappings, but there must be no conflicting entries.
Example:
SOCKET: host1 sockad_s SOCKET: host2 sockad_s SOCKET: host3 host3_s SOCKET: 192.168.10.0/24 sockad_s
Using multiple NodeIds per Cluster Node is supported.This can be used for some advanced high-availability switch configurations.
A short version of
dishosts.confis supported for compatibility reasons, which corresponds to DISHOSTVERSION 0. Please note that neither virtual adapters nor dynamic socket mappings are supported by this format.Example:
#host/IP nodeid host1prv 4 host2prv 8
The networkmanager.conf specifies the startup parameters for the Dolphin Network Manager. It is created by the dis_netconfig utility.
This file must not be edited by the user. It is a status file of the Network Manager that consists of the user-specified settings from networkmanager.conf and derived settings of the Cluster Nodes. It is created by the Network Manager. cluster.conf is discontinued with eXpressWare 5.5.0 and newer versions and can safely be deleted.