I think it's a little more complex than that. I think the issue in newcastle is that the kernel they are running supports global memory regions so map_on_demand setting changes the behaviour. During my debugging session map-on-demand was set to 0 which means use global memory regions and therefore can not negotiate different max fragment sizes. Which explains why James doesn't have a problem. His kernel doesn't support global memory regions so the map-on-demand is always set to 1 which uses FMR, which allows the negotiation of different max frags. Regarding Total, we need to understand their kernel configuration to be absolutely sure about the behavior they are likely to encounter on the upgrade.
2532 »·······if (reqmsg->ibm_u.connparams.ibcp_max_frags >
2533 »······· IBLND_MAX_RDMA_FRAGS) {
2534 »·······»·······CWARN("Can't accept conn from %s (version %x): "
2535 »·······»······· "max_frags %d too large (%d wanted)\n",
2536 »·······»······· libcfs_nid2str(nid), version,
2537 »·······»······· reqmsg->ibm_u.connparams.ibcp_max_frags,
2538 »·······»······· IBLND_MAX_RDMA_FRAGS);
2539
2540 »·······»·······if (version >= IBLND_MSG_VERSION)
2541 »·······»·······»·······rej.ibr_why = IBLND_REJECT_RDMA_FRAGS;
2542
2543 »·······»·······goto failed;
2544 »·······} else if (reqmsg->ibm_u.connparams.ibcp_max_frags <
2545 »·······»······· IBLND_MAX_RDMA_FRAGS &&
2546 »·······»······· net->ibn_fmr_ps == NULL) {
2547 »·······»·······CWARN("Can't accept conn from %s (version %x): "
2548 »·······»······· "max_frags %d incompatible without FMR pool "
2549 »·······»······· "(%d wanted)\n",
2550 »·······»······· libcfs_nid2str(nid), version,
2551 »·······»······· reqmsg->ibm_u.connparams.ibcp_max_frags,
2552 »·······»······· IBLND_MAX_RDMA_FRAGS);
2553
2554 »·······»·······if (version == IBLND_MSG_VERSION)
2555 »·······»·······»·······rej.ibr_why = IBLND_REJECT_RDMA_FRAGS;
2556
2557 »·······»·······goto failed;
2558 »·······}