The prefetch memory region of an NTB adapter is used for mapping remote segments and multicast memory groups. When a remote segment with size 32MB is mapped, 32MB of the adapters prefetch space is consumed until the segment is unmapped. Most of the prefetch memory address range is available for customer use, but is a shared resource for all concurrent use of PCIe. eXpressWare will typically use a small portion of this resource for internal communication and protocols.
The factory default prefetch memory size of the Dolphin adapters is 256MB.
Due to various hardware constrains such as alignment requirements the full prefetch memory space might not be usable. The prefetch memory space size can be increased if needed.
The currently configured PCIe prefetch memory size is typically displayed by dis_diag and can be queried by running dis_config:
[root@PXMXS-01 ~]# dis_config -gpms Card 1 - Prefetch memory space size: 512
The PCIe card typically support a very large address space, but your BIOS will in many cases limit this significantly. The BIOS will in many cases refuse to boot the system so you have to be very careful to test the desired size on one system before you do this on all systems.
If your system refuses to boot, you have to 1) modify the DIP switch to select another NTB setting, 2) perform a firmware upgrade to set back the factory default PCIe prefetch size, 3) re-enabling correct DIP switch setting again. More information on firmware upgrade can be found in Chapter 9, Firmware upgrade
If you intend to increase the PCIe Prefetch Memory size beyond the factory default, it is strongly recommended to ensure that 4G decoding is enabled in the system BIOS before proceeding. This is sometimes also referred to as “64-bit BAR support” “64-bit address decoding”. Consult your system or motherboard manual for details. Some systems may fail to boot if the prefetch memory size is increased beyond the factory default. To recover from this the prefetch memory size must be decreased.
The Prefetch Memory size is changed using dis_config. The new size is given in Megabytes. Prefetch size must be a power of two and larger than 64MB:
[root@PXMXS-01 ~]# dis_config -spms 512
Card 1 - Prefetch space memory size is set to 512 MB
A reboot of the machine is required to make the changes take effect.
[root@PXMXS-01 ~]#