On a default installation the system wouldn't be configured for optimum performance. Below are some steps to consider when tuning for performance. (Thanks to Doug for the info)

1. Create a config file under /etc/modprobe.d/ for the HFI driver (hfi.conf)

   1. options hfi1 krcvqs=4 piothreshold=0 sge_copy_mode=2 wss_threshold=70

2. >> dracut -f
   # to pick up the parameters, although I've observed that the parameters don't get picked up except after a reboot so...
   >> reboot

3. Check that the parameters are picked up:
   
   

   for x in /sys/module/hfi1/parameters/*; do echo $(basename $x) $(cat $x); done

4. Configure the OPA network with the optimum tunable configurations

   alias ko2iblnd-opa ko2iblnd
   options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4

Notes

If the above tunables are used for both OPA and MLX, MLX will experience a 5-10% drop in performance. This is not significant, but for the sake of argument, some of these tunables can be configured per network via lnetctl

    - net type: o2ib
      local NI(s):
        - nid: 192.168.1.2@o2ib
          status: up
          interfaces:
              0: ib0
          tunables:
              peer_timeout: 180
              peer_credits: 8
              peer_buffer_credits: 0
              credits: 256
          lnd tunables:
              peercredits_hiw: 4
              map_on_demand: 0
              concurrent_sends: 8
              fmr_pool_size: 512
              fmr_flush_trigger: 384
              fmr_cache: 1
              conns_per_peer: 1
          tcp bonding: 0
          dev cpt: 0
          CPT: "[0,1]"