View Source

References

Intel® Omni-Path Fabric Performance Tuning User Guide [Rev. 18.0, January 2020]

Section 4.3 “HFI1 Driver Parameters” contains the information surrounding initramfs.

‘dracut –f’ forces hfi1.conf parameters to be applied at boot.

Example:

Verify hfi1.conf and reboot:
[RHEL7.4 hds1fnad261 20171116_1220 scripts]# cat /etc/modprobe.d/hfi1.conf
options hfi1 pcie_caps=0x51 sge_copy_mode=2 krcvqs=5 num_sdma=13 wss_threshold=70 piothreshold=0
[RHEL7.4 hds1fnad261 20171116_1222 scripts]# reboot
 
Check hfi1 parameters in use after reboot:
[RHEL7.4 hds1fnad261 20171116_1225 ~]# cd /scratch/mlkirkco/scripts
[RHEL7.4 hds1fnad261 20171116_1226 scripts]# ./get-hfi1-params-conf.sh
    krcvqs=
    sge_copy_mode=0
    wss_threshold=80
(*NOT using hfi1.conf as desired)
 
Run ‘dracut –f’ to update initramfs:
[RHEL7.4 hds1fnad261 20171116_1226 scripts]# dracut -f
[RHEL7.4 hds1fnad261 20171116_1226 scripts]# reboot
 
Verify hfi1 parameters after reboot:
[RHEL7.4 hds1fnad261 20171116_1233 ~]# cat /etc/modprobe.d/hfi1.conf
options hfi1 pcie_caps=0x51 sge_copy_mode=2 krcvqs=5 num_sdma=13 wss_threshold=70 piothreshold=0
[RHEL7.4 hds1fnad261 20171116_1233 ~]# cd /scratch/mlkirkco/scripts
[RHEL7.4 hds1fnad261 20171116_1233 scripts]# ./get-hfi1-params-conf.sh
    krcvqs=5
    sge_copy_mode=2
    wss_threshold=70

A brief description of the above mentioned hfi1 parameters -:

pcie_caps = This param sets the Max Payload size and Max read req, if those are not set in BIOS. Setting pcie_caps=0x51, will set MaxPayLoad=256B and MaxReadReq=4096B
sge_copy_mode = This param sets the Verbs copy mode. The value can be 0( Use Memcpy), 1 (cacheless copy) or 2 (adapt based on WSS - Adaptive cache memcpy) .
krcvqs = Number of kernel receive queues. In case large number of clients accesses one or more servers through IPoIB then it is recommended to set this param to an odd value (3, 5 or 7). krcvqs=5 assumes there is 1 active virtual lane. If there are more than one active virtual lane then krcvqs can be set for each virtual lane.