You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

In standard IB terms, VL stands for Virtual Lane and SL for Service Level.

The way to specify VL for o2iblnd would be:

  1. use the ko2iblnd 'service' option, which indrectly sets the IB SL, i.e. service level
    1. The service_id is calculated based on the port space and service port used as the destination port
      1. The service port when connecting to a peer:
        1. 1240 static void
1241 kiblnd_connect_peer(struct kib_peer *peer)
1242 {
...
1269 »·······memset(&dstaddr, 0, sizeof(dstaddr));
1270 »·······dstaddr.sin_family = AF_INET;
1271 »·······dstaddr.sin_port = htons(*kiblnd_tunables.kib_service);
1272 »·······dstaddr.sin_addr.s_addr = htonl(LNET_NIDADDR(peer->ibp_nid));
1273 
1274 »·······kiblnd_peer_addref(peer);»······       /* cmid's ref */
1275 
1276 »·······if (*kiblnd_tunables.kib_use_priv_port) {
1277 »·······»·······rc = kiblnd_resolve_addr(cmid, &srcaddr, &dstaddr,
1278 »·······»·······»·······»·······»······· *kiblnd_tunables.kib_timeout * 1000);
1279 »·······} else {
1280 »·······»·······rc = rdma_resolve_addr(cmid,
1281 »·······»·······»·······»·······       (struct sockaddr *)&srcaddr,
1282 »·······»·······»·······»·······       (struct sockaddr *)&dstaddr,
1283 »·······»·······»·······»·······       *kiblnd_tunables.kib_timeout * 1000);
1284 »·······}
...
1306 }
      2. When the address is resolved successfully then rdma_resolve_route() is called
        1. rdma_resolve_route() -> cma_resolve_ib_route() -> cma_query_ib_route()
        2. path_rec.service_id = rdma_get_service_id(&id_priv->id,
»·······»·······»·······»·······»·······  cma_dst_addr(id_priv));
 
2031 __be64 rdma_get_service_id(struct rdma_cm_id *id, struct sockaddr *addr)                                                       
2032 {                                                                                                                              
2033 »·······if (addr->sa_family == AF_IB)                                                                                          
2034 »·······»·······return ((struct sockaddr_ib *) addr)->sib_sid;                                                                 
2035                                                                                                                                
2036 »·······return cpu_to_be64(((u64)id->ps << 16) + be16_to_cpu(cma_port(addr)));                                                 
2037 }                                                                                                                              
2038 EXPORT_SYMBOL(rdma_get_service_id);
 
/*
 * The port space is: RDMA_PS_TCP
 */
 67 enum rdma_port_space {                                                                                                          
 68 »·······RDMA_PS_SDP   = 0x0001,                                                                                                 
 69 »·······RDMA_PS_IPOIB = 0x0002,                                                                                                 
 70 »·······RDMA_PS_IB    = 0x013F,                                                                                                 
 71 »·······RDMA_PS_TCP   = 0x0106,                                                                                                 
 72 »·······RDMA_PS_UDP   = 0x0111,                                                                                                 
 73 }; 
 
/* The service ID == 0x10603DB */
  2. configure OpenSM to match the service ID with the a specific QoS policy. From the OpenSM tree: doc/QoS_management_in_OpenSM.txt
    1. # in /etc/opensm/qos-policy.conf
 
    qos-levels

        # Having a QoS Level named "DEFAULT" is a must - it is applied to
        # PR/MPR requests that didn't match any of the matching rules.
        qos-level
            name: DEFAULT
            use: default QoS Level
            sl: 0
        end-qos-level

        # the whole set: SL, MTU-Limit, Rate-Limit, PKey, Packet Lifetime
        qos-level
            name: LustreTraffic
            sl: 1
        end-qos-level

    end-qos-levels


    # Match rules are scanned in order of their apperance in the policy file.
    # First matched rule takes precedence.
    qos-match-rules

        # show matching by destination group and service id
        qos-match-rule
            service-id: 0x10603DB
            qos-level-name: LustreTraffic
        end-qos-match-rule

    end-qos-match-rules

      The above qos-policy matches any connections with service-id 0x10603DB to service level 1.

  3. The SL2VL and VLArb tables should be configured in: /etc/opensm/opensm.conf. Refer to the SL2VL Mapping and VL Arbitration section in doc/QoS_management_in_OpenSM.txt for more details.

 

 

  • No labels