MD memory passed down to the LND in IOV or KIOV needs to be mapped in order for it to be DMAed.
Let's look at the GET case for IOV with FMR:
- kiblnd_send() is called for LNET_MSG_GET
- tx = kiblnd_get_idle_tx(ni, target.nid);
- ibmsg = tx→tx_msg; # ibmsg is of type kib_msg_t. That's the structure that goes over the wire
- rd = &ibmsg→ibm_u.get.ibgm_rd; # rd is of type kib_rdma_desc_t, which also gets sent to the peer. This describes the memory in which the data will be RDMAd in
- rc = kiblnd_setup_rd_iov(...); # this maps the iov to make it DMAable
- go through the iov and collect all the pages (struct page) into a scatter gather list.
- for each page there could be a page_offset.
- The data does not necessarily fill the entire page. This is recorded by: sg_set_page(sg, page, fragnob, page_offset);
- Once we have collected all the pages in a scatter gather list, we need to map the transmit by calling: kiblnd_map_tx(ni, tx, rd, sg - tx→tx_frags);
- kiblnd_map_tx() maps the pages given the scatter gather list into the device's dma addresses.
- For good documentation take a look at: linux/Documentation/DMA-API-HOWTO.txt
- kiblnd_dma_map_sg() is called to map the pages gathered in the scatter/gather list to the device's DMA addresses.
- The dma address is the page physical address + the sg_offset stored. Look at the implementation of sg_phys()
- When this function completes then we call sg_dma_address() and sg_dma_length() macros to return the dma_address and length.
- These values are stored in the rd (kib_rdma_desc_t)
- If we are using FMR or FastReg determined by if (net->ibn_fmr_ps != NULL), then call kiblnd_fmr_map_tx()
- kiblnd_fmr_map_tx()
- kiblnd_fmr_pool_map() is called. The main work here is done when ib_fmr_pool_map_phys() is called. This basically maps the pages where the data will be written into the FMR pools.
- The rd that gets sent over to the peer, in the FMR case, will need to describe a relative addresses within the pages
- go through the iov and collect all the pages (struct page) into a scatter gather list.