References
Intel® Omni-Path Fabric Performance Tuning User Guide [Rev. 18.0, January 2020]
Section 4.3 “HFI1 Driver Parameters” contains the information surrounding initramfs.
‘dracut –f’ forces hfi1.conf parameters to be applied at boot.
Example:
Verify hfi1.conf and reboot: [RHEL7.4 hds1fnad261 20171116_1220 scripts]# cat /etc/modprobe.d/hfi1.conf options hfi1 pcie_caps=0x51 sge_copy_mode=2 krcvqs=5 num_sdma=13 wss_threshold=70 piothreshold=0 [RHEL7.4 hds1fnad261 20171116_1222 scripts]# reboot Check hfi1 parameters in use after reboot: [RHEL7.4 hds1fnad261 20171116_1225 ~]# cd /scratch/mlkirkco/scripts [RHEL7.4 hds1fnad261 20171116_1226 scripts]# ./get-hfi1-params-conf.sh krcvqs= sge_copy_mode=0 wss_threshold=80 (*NOT using hfi1.conf as desired) Run ‘dracut –f’ to update initramfs: [RHEL7.4 hds1fnad261 20171116_1226 scripts]# dracut -f [RHEL7.4 hds1fnad261 20171116_1226 scripts]# reboot Verify hfi1 parameters after reboot: [RHEL7.4 hds1fnad261 20171116_1233 ~]# cat /etc/modprobe.d/hfi1.conf options hfi1 pcie_caps=0x51 sge_copy_mode=2 krcvqs=5 num_sdma=13 wss_threshold=70 piothreshold=0 [RHEL7.4 hds1fnad261 20171116_1233 ~]# cd /scratch/mlkirkco/scripts [RHEL7.4 hds1fnad261 20171116_1233 scripts]# ./get-hfi1-params-conf.sh krcvqs=5 sge_copy_mode=2 wss_threshold=70
A brief description of the above mentioned hfi1 parameters -:
- pcie_caps = This param sets the
Max Payload size
andMax read req
, if those are not set in BIOS. Settingpcie_caps=0x51
, will setMaxPayLoad=256B
andMaxReadReq=4096B
- sge_copy_mode = This param sets the Verbs copy mode. The value can be 0( Use Memcpy), 1 (cacheless copy) or 2 (adapt based on WSS - Adaptive cache memcpy) .
- krcvqs = Number of kernel receive queues. In case large number of clients accesses one or more servers through IPoIB then it is recommended to set this param to an odd value (3, 5 or 7).
krcvqs=5
assumes there is 1 active virtual lane. If there are more than one active virtual lane thenkrcvqs
can be set for each virtual lane.