IB0 is the primary. IB1 is secondary in the bond. LNet network is configured on the Bond. IB0 is the only interface being used since it's the primary. A suggested configuration change is to split the bond to 2 o2ib networks.
Have one set of routers go through o2ib0 and the other set of routers go through o2ib1. And then split the clients evenly across both router sets. I'm thinking this will distribute the traffic across both OSS/MDS interfaces as well as the switches.
Both interfaces can be configured on the same network aggregating the performance.