[Twg] LNet channel bounding

[Alexey Lyashkov]

Hi All,

i write a simple document about creating a channel bounding in lnet some time ago and want to discus that is
before we will start and make design and code for that project.

LNet bounding conceptual design
 
LNet is network module designed to exchange messages between lustre nodes.
Channel bounding is need to extend network bandwidth between nodes, it’s allow to send many messages parallel, each in own network channel.
Exist some channel bounding modes
·      System bounding – some networks channels merged in single logical channel on system side and lustre see that as one channel with single address space.
·      LND based bounding – currently supported on socklnd (tcp network module). In that case nodes use extended HELO protocol to send all addresses from node to remote peer and peer a look to these addresses to send request in round robin order.
Both these modes don’t support a channels bounding between different network types such as IB and Ethernet networks.
To avoid that problem we need an implementing bounding inside LNet module.
Exist some possibilities to implementing it.
1.     Reuse LNet routing code.
In that case each destination nid mapped some gateway via list of routing rules. Problems in that area
·      We need an implementing destination address mapping inside LNet code.
·      We need change lnet_ni_t to hide interfaces joined in bounding group.
·      We need a change half of code in routing send codepath to kill packet attributes change.
·      Changes in incoming codepath – need extract bounding messages from routed.
·      Troubles with peer health detection (it need to be know – peer is router or not).
·      Don’t possible to resend message in case interface makes down in message transmit time.
·       
I think it task will be need ~1000-1500 LOC of changes and need hard testing on both modes (node and router).
 
2.     Implementing “Bounding” LND
It’s way much clear, because needs a all new code placed in new module.
We need implementing a simple LND (lustre network driver) with channel bounding algorithm (similar to socklnd has).
Bounding LND will responsible to:
·      Mapping from virtual address to real on one of bounding interfaces.
·      Finding best interface to send
·      Handle peer health detection.
 
Some background.
 
Send path: When LNet want to send a message to peer, it start from finding NI (lustre network interface aka LND instance) – to sending it and call .lnd_send method of founded NI to transfer message over wire. LND has responsible to address resolution and free to change source and destination address. Connection to remote peer handled inside LND when LNet want a send message to remote peer and don’t visible to upper (LNet) layer.
 
Receive path: When network driver found a incoming request on wire it read a header and call a lnet_parse() with self pointer to handle incoming message. Lnet_parse() call a .lnd_recv method to get data over wire into buffer. Single check exist on that codepath – is destination nid is assigned to local NI or not.
 
Peer health detection – LND responsible to get health info for peer submitted from LNet and LNet scan a peer list to find unhealthy peers in timed manner.
 
These preconditions use to create a design of bounding driver.
 
Bounding LND structures
·      Own list of networks from each driver assigned to bounding as similar as routing code does.
·      Fake peer list, which a mapped to real peer, that is need to make peer health detection happy (that is similar socklnd has).
·       
 
Implementation details.
 
We need a parse line similar “networks” to bounding create a new NI’s.  But these NI’s need to marked with special flag to prevent finding via lnet_net2ni_locked() (or we can use same interface for non bounding transfer ???).
 
->lnd_send method of that “bounding” LND has responsible found best NI via lnet_compare_routes() function.
Source and destination NID of message has changed to route based similar as route code does (I think we can use simple rule as $network_start + offset).
.lnd_send method of founded NI called after change source and destination address to transfer message over wire.
           
lnet_notify should be changed to use new method in lnd_t structure which responsible to send notify event to upper layer instead of direct call lnet_notify from LND’s. 
Bounding LND need to own handler to handle notify events from lower NI. Handler responsible to resend failed message via different links and will send notify event to lnet only if peer don’t accessible over all links (that situation adds delay if node is down, but bounding will try send message over all links). That handler also responsible to resend messages
Non-bounding LND’s need pointer to lnet_notify directly.
 
Peer health detection – LNet uses two ways to check peers alive. One is periodic asking via .lnd_query method, second is notify event from lnet_notify.
lnet_notify need be implemented in way pointed before.
.lnd_query interface can supported via own list of virtual (fake) peers.
 
 
Estimation size.
 
I think it need to have ~1000-1200 LOCs, with lots reuse code from lnet routing, like are create, destroy, find best route and parsing network line.