MD memory passed down to the LND in IOV or KIOV needs to be mapped in order for it to be DMAed.

Let's look at the GET case for IOV with FMR:

• kiblnd_send() is called for LNET_MSG_GET
  • tx = kiblnd_get_idle_tx(ni, target.nid);
  • ibmsg = tx→tx_msg; # ibmsg is of type kib_msg_t. That's the structure that goes over the wire
  • rd = &ibmsg→ibm_u.get.ibgm_rd; # rd is of type kib_rdma_desc_t, which also gets sent to the peer. This describes the memory in which the data will be RDMAd in
  • rc = kiblnd_setup_rd_iov(...); # this maps the iov to make it DMAable
    • go through the iov and collect all the pages (struct page) into a scatter gather list.
      • for each page there could be a page_offset.
      • The data does not necessarily fill the entire page. This is recorded by: sg_set_page(sg, page, fragnob, page_offset);
    • Once  we have collected all the pages in a scatter gather list, we need to map the transmit by calling: kiblnd_map_tx(ni, tx, rd, sg - tx→tx_frags);
    • kiblnd_map_tx() maps the pages given the scatter gather list into the device's dma addresses.
      • For good documentation take a look at: linux/Documentation/DMA-API-HOWTO.txt
      • kiblnd_dma_map_sg() is called to map the pages gathered in the scatter/gather list to the device's DMA addresses.
        • The dma address is the page physical address + the sg_offset stored. Look at the implementation of sg_phys()
      • When this function completes then we call sg_dma_address() and sg_dma_length() macros to return the dma_address and length.
      • These values are stored in the rd (kib_rdma_desc_t)
      • If we are using FMR or FastReg, determined by if (net->ibn_fmr_ps != NULL), then call kiblnd_fmr_map_tx()
      • kiblnd_fmr_map_tx()
        • kiblnd_fmr_pool_map() is called. The main work here is done when ib_fmr_pool_map_phys() is called. This basically maps the pages where the data will be written into the FMR pools.
        • The rd that gets sent over to the peer, in the FMR case, will need to describe a relative addresses within the pages, in order for the remote to know where to RDMA the data.
• on the receiving peer we call kiblnd_handle_rx() for IBLND_MSG_GET_REQ. This will call up in lnet_parse, which will match the MD pointing to the data to be RDMAed to the peer
  • kiblnd_reply() is called, which calls kiblnd_setup_rd_iov(). Same logic is done is described above.
  • kilbnd_init_rdma() is then called to set up the work requests (wr) which is use in the call to ib_post_send()
    • Key point in this function is that it sets the sge (tx→tx_sge[]) to have the address for each fragment. Look at kiblnd_rd_frag_addr()
    • The sge →addr will have a relative address to the local source MD. That's used in ib_post_send() to RDMA the data into the buffers which are also described via relative addresses by the destination rd.

One difference between FMR and FastReg is that in FMR the pages are mapped to the FMR pool, and therefore the rd will need to be set to relative addresses. In the FastReg case, wr→wr.wr.fast_reg.page_list, points to the physical pages, and therefore the source rd, must describe the actual page addresses, and not just a relative offset.