In standard IB terms, VL stands for Virtual Lane and SL for Service Level.
The the way to specify VL for o2iblnd would be:
- use the ko2iblnd 'service' option, which indrectly sets the IB SL, i.e. service level
- configure OpenSM or whatever SM, to map the o2iblnd SL into VL
This has been working for years. LLNL did several talks in LUGs about how to do Lustre QoS on IB which involves using both SL->VL mappings and somethings just use the server IP to distinguish Lustre traffic
The o2iblnd SL is set by the OFED RDMA CM, indirectly based on the o2iblnd service port (set via ko2iblnd option 'service', 987 by default) and its port space (RDMA_PS_TCP).
- The service_id is calculated based on the port space and service port used as the destination port
- The service port when connecting to a peer:
Code Block 1240 static void 1241 kiblnd_connect_peer(struct kib_peer *peer) 1242 { ... 1269 »·······memset(&dstaddr, 0, sizeof(dstaddr)); 1270 »·······dstaddr.sin_family = AF_INET; 1271 »·······dstaddr.sin_port = htons(*kiblnd_tunables.kib_service); 1272 »·······dstaddr.sin_addr.s_addr = htonl(LNET_NIDADDR(peer->ibp_nid)); 1273 1274 »·······kiblnd_peer_addref(peer);»······ /* cmid's ref */ 1275 1276 »·······if (*kiblnd_tunables.kib_use_priv_port) { 1277 »·······»·······rc = kiblnd_resolve_addr(cmid, &srcaddr, &dstaddr, 1278 »·······»·······»·······»·······»······· *kiblnd_tunables.kib_timeout * 1000); 1279 »·······} else { 1280 »·······»·······rc = rdma_resolve_addr(cmid, 1281 »·······»·······»·······»······· (struct sockaddr *)&srcaddr, 1282 »·······»·······»·······»······· (struct sockaddr *)&dstaddr, 1283 »·······»·······»·······»······· *kiblnd_tunables.kib_timeout * 1000); 1284 »·······} ... 1306 }
- When the address is resolved successfully then rdma_resolve_route() is called
Code Block rdma_resolve_route() -> cma_resolve_ib_route() -> cma_query_ib_route()Code Block path_rec.service_id = rdma_get_service_id(&id_priv->id, »·······»·······»·······»·······»······· cma_dst_addr(id_priv)); 2031 __be64 rdma_get_service_id(struct rdma_cm_id *id, struct sockaddr *addr) 2032 { 2033 »·······if (addr->sa_family == AF_IB) 2034 »·······»·······return ((struct sockaddr_ib *) addr)->sib_sid; 2035 2036 »·······return cpu_to_be64(((u64)id->ps << 16) + be16_to_cpu(cma_port(addr))); 2037 } 2038 EXPORT_SYMBOL(rdma_get_service_id); /* * The port space is: RDMA_PS_TCP */ 67 enum rdma_port_space { 68 »·······RDMA_PS_SDP = 0x0001, 69 »·······RDMA_PS_IPOIB = 0x0002, 70 »·······RDMA_PS_IB = 0x013F, 71 »·······RDMA_PS_TCP = 0x0106, 72 »·······RDMA_PS_UDP = 0x0111, 73 }; /* The service ID == 0x10603DB */
- The service port when connecting to a peer:
- The service_id is calculated based on the port space and service port used as the destination port
- configure OpenSM to match the service ID with the a specific QoS policy. From the OpenSM tree: doc/QoS_management_in_OpenSM.txt
Code Block # in /etc/opensm/qos-policy.conf qos-levels # Having a QoS Level named "DEFAULT" is a must - it is applied to # PR/MPR requests that didn't match any of the matching rules. qos-level name: DEFAULT use: default QoS Level sl: 0 end-qos-level # the whole set: SL, MTU-Limit, Rate-Limit, PKey, Packet Lifetime qos-level name: LustreTraffic sl: 1 end-qos-level end-qos-levels # Match rules are scanned in order of their apperance in the policy file. # First matched rule takes precedence. qos-match-rules # show matching by destination group and service id qos-match-rule service-id: 0x10603DB qos-level-name: LustreTraffic end-qos-match-rule end-qos-match-rulesThe above qos-policy matches any connections with service-id 0x10603DB to service level 1.
The SL2VL and VLArb tables should be configured in: /etc/opensm/opensm.conf. Refer to the SL2VL Mapping and VL Arbitration section in doc/QoS_management_in_OpenSM.txt for more details.
QoS configuration on Infiniband