Purpose

The purpose of this document is to provide the description of the design verification procedure necessary to verify Lustre operation with IPv6 feature. The procedure is not isolated to testing specific Lustre modules but rather Lustre as a complete system, unless specified otherwise for individual test cases.

Scope

This document covers design verification procedures for an IPv6-only system and a mixed IPv6-IPv4 system with the focus on IPv6 functionality. Testing of IPv4-only system is considered to be regression testing with regard to the IPv6 feature and is not covered by this document. 

Requirements

The requirements for executing this test plan are same as for a regular Lustre system testing with the exception that every node involved in this testing is expected to be able to work with IPv6.

HW Requirements

A pool of server nodes and a pool of client nodes is expected to be available. Exact numbers are TBD

Network Requirements

Each node is expected to have at least two interfaces available for the purpose of LNet configuration. The connection between the nodes should be such that both IPv4 and IPv6 can be supported on every link.

While majority of the testing is projected to use TCP/Ethernet networking, IPv6 with o2ib shall also be tested.

SW Requirements

Exact Lustre SW version to be used is TBD. Linux kernel versions to be used are TBD. The test is intended to be rerun on subsequent releases, so the SW requirements are expected to change. It is recommended to create a separate "results" document for each run and specify the SW and kernel versions there.

Testing Procedure

Test Case Structure

The test is defined as the combination of the following:

Identifier

Label which uniquely identifies the testcase.

Objective

Description of what the test case is verifying. Ideally refers to a specific requirement from a system design document which is being verified.

Procedure

Describes the test setup and procedure. Level of detail should be sufficient for a test engineer to follow (should not rely on information available to developers only).

Expected Behaviour

Describes how the system is expected to behave when test procedure is executed.

Actual Results

Describes actual results observed when the test case is executed.

Test Cases

Positive

IDObjectiveProcedureExpected BehaviourActual ResultsNotes
IPv6_only.Config.SR.Client.1Configure Lustre Client using only IPv6. (SR, same subnet)
  1. Select an IPv6-enabled machine
  2. Install lustre client
  3. Configure the client to use a connected interface which has only IPv6 address configured
  4. Bring up lustre
  5. Execute "lnetctl net show"
  6. Execute "lnetctl ping" of own NID
  1. "lnetctl net show" output should indicate an active NID featuring IPv6 address of the interface configured for use with LNet.
  2. "lnetctl ping" of own NID should be successful

pass

bug:

IPv6 route is not created if NID is added dynamically (LU-18315 created)


All nodes are

2.16.0_RC1

Rocky8.8

IPv6_only.Config.SR.Client.2Configure second Lustre Client using only IPv6 and test connectivity between the two clients. (SR, same subnet)
  1. Select an IPv6-enabled machine
  2. Install lustre client
  3. Configure the client to use a connected interface which has only IPv6 address configured. 
  4. Execute "ping" to the IPv6 of the other client
  5. Bring up lustre
  6. Execute "lnetctl net show"
  7. Execute "lnetctl ping" of own NID
  8. Execute "lnetctl ping" of the other client's NID
  1. "ping" of the other client's IPv6 address should succeed
  2. "lnetctl net show" output should indicate an active NID featuring IPv6 address of the interface configured for use with LNet.
  3. "lnetctl ping" of own NID should be successful
  4. "lnetctl ping" of the other client's NID should be successful

pass

bug?: it is possible for a client with an only IPv6 NID to successfully ping a client with an only IPv4 NID (likely) if any of them is routed via a gateway

Assumes that Client.1 and Client.2 are on the same subnet and are able to reach each other over IPv6 network.
IPv6_only.Config.SR.Server.1Configure Lustre Server using only IPv6 (SR, same subnet)
  1. Select an IPv6-enabled machine
  2. Install lustre server (ok to collocate MDS and OSS)
  3. Configure the server to use a connected interface which has only IPv6 address configured.
  4. Execute "ping" to the IPv6 of a client
  5. Bring up lustre
  6. Execute "lnetctl net show"
  7. Execute "lnetctl ping" of own NID
  8. Execute "lnetctl ping" of the client's NID
  1. "ping" of the client's IPv6 address should succeed
  2. "lnetctl net show" output should indicate an active NID featuring IPv6 address of the interface configured for use with LNet.
  3. "lnetctl ping" of own NID should be successful
  4. "lnetctl ping" of the client NID should be successful

pass


Assumes that servers and clients are on the same subnet and are able to reach each other over IPv6 network.
IPv6_only.Basic.LNet.SR.1Test basic connectivity between Lustre Server and Client on LNet level. (SR, same subnet) 

Use the lustre server and clients setup in previous tests and

  1. Load "lnet_selftest" on each node
  2. Configure and run "lnet_selftest" between the nodes
  1. "lnet_selftest" should run without errors. 
  2. Throughput reported by "lnet_selftest" should be adequate (verify against link speed).

fail

lnet_selftest fails to parse IPv6 NIDs (LU-18316 created)


IPv6_only.Basic.Lustre.SR.1Test Lustre FS performance with the Client and Server setup to use IPv6. (SR, same subnet)

Use the lustre server and clients setup in previous tests and

  1. Mount lustre FS on the client(s)
  2. Run FIO
  1. Lustre FS mount should succeed
  2. FIO should run without errors. Throughput reported by FIO should be adequate (verify against link speed and "lnet_selftest" results).

pass:

static via conf file if the only/first listed address assigned to the interface is IPv6 global or if using "lnetctl --nid" to configure NIDs dynamically

fail:

conf file with ip2nets="tcp0(eth0) <IPV6>" or

ip2nets="tcp0 <IPv6>" fails.


IPv6_only.Config.MultiSubnet.SR.1Configure Lustre Client and Lustre server to be on different IPv6 subnets (SR)
  1. Use IPv6_only.Config.SR.Client.1 to create a lustre client and IPv6_only.Config.SR.Server.1 to create a lustre server, but make sure they are using IPv6 addresses which are on different subnets.
  2. Execute "ping" from the client to IPv6 address of the server
  3. Bring up lustre
  4. Execute "lnetctl ping" between the client and the server NIDs
  1. ping" of the server's IPv6 address from the client should succeed
  2. "lnetctl ping" of between the client and the server should be successful


IPv6_only.MultiSubnet.LNet.SR.1Test basic connectivity between Lustre Server and Client across IPv6 subnets on LNet level. (SR) 
  1. Use IPv6_only.Config.MultiSubnet.SR.1 for setup
  2. Load "lnet_selftest" on each node
  3. Configure and run "lnet_selftest" between the nodes
  1. "lnet_selftest" should run without errors. 
  2. Throughput reported by "lnet_selftest" should be adequate (verify against link speed).


IPv6_only.MultiSubnet.Lustre.SR.1Test Lustre FS performance with the Client and Server setup to use IPv6 across subnets. (SR)
  1. Use IPv6_only.Config.MultiSubnet.SR.1 for setup
  2. Mount lustre FS on the client(s)
  3. Run FIO
  1. Lustre FS mount should succeed
  2. FIO should run without errors. Throughput reported by FIO should be adequate (verify against link speed and "lnet_selftest" results).


IPv6_only.Config.MR.Client.1Configure Lustre Client using only IPv6. (MR, same subnet)
  1. Select an IPv6-enabled machine
  2. Install lustre client
  3. Configure the client to use 2 connected interfaces with only IPv6 address configured on each
  4. Bring up lustre
  5. Execute "lnetctl net show"
  6. Execute "lnetctl ping" of own NIDs
  1. "lnetctl net show" output should indicate 2 active NIDs featuring IPv6 addresses of the interfaces configured for use with LNet.
  2. "lnetctl ping" of own NIDs should be successful
pass
IPv6_only.Config.MR.Client.2Configure second Lustre Client using only IPv6 and test connectivity between the two clients. (MR, same subnet)
  1. Select an IPv6-enabled machine
  2. Install lustre client
  3. Configure the client to use 2 connected interfaces with only IPv6 address configured on each 
  4. Execute "ping" to the IPv6 addresses of the other client
  5. Bring up lustre
  6. Execute "lnetctl net show"
  7. Execute "lnetctl ping" of own NIDs
  8. Execute "lnetctl ping" of the other client's NIDs
  1. "ping" of the other client's IPv6 addresses should succeed
  2. "lnetctl net show" output should indicate 2 active NIDs featuring IPv6 addresses of the interfaces configured for use with LNet.
  3. "lnetctl ping" of own NIDs should be successful
  4. "lnetctl ping" of the other client's NIDs should be successful
passAssumes that Client.1 and Client.2 are on the same subnet and are able to reach each other over IPv6 network.
IPv6_only.Config.MR.Server.1Configure Lustre Server using only IPv6 (MR, same subnet)
  1. Select an IPv6-enabled machine
  2. Install lustre server (ok to collocate MDS and OSS)
  3. Configure the LNet on the server to use 2 IPv6-only interfaces.
  4. Execute "ping" to the IPv6 addresses of the client
  5. Bring up lustre
  6. Execute "lnetctl net show"
  7. Execute "lnetctl ping" of own NIDs
  8. Execute "lnetctl ping" of the client's NIDs
  1. "ping" of the client's IPv6 addresses should succeed
  2. "lnetctl net show" output should indicate 2 active NIDs featuring IPv6 address of the interface configured for use with LNet.
  3. "lnetctl ping" of own NIDs should be successful
  4. "lnetctl ping" of the client NIDs should be successful
passAssumes that servers and clients are on the same subnet and are able to reach each other over IPv6 network.
IPv6_only.Basic.LNet.MR.1Test basic connectivity between Lustre Server and Client on LNet level. (MR, same subnet)

Use the lustre server and clients setup in previous tests and

  1. Load "lnet_selftest" on each node
  2. Make sure that nodes have "discovered" each other
  3. Configure and run "lnet_selftest" between the nodes, make sure to use the "primary NIDs" of each node
  4. Capture "lnetctl net show -v 4" and "netstat -i" outputs before and after the "lnet_selftest"
  1. "lnet_selftest" should run without errors. 
  2. Throughput reported by "lnet_selftest" should be adequate (verify against link speed).
  3. Compare "lnetctl net show -v 4" outputs captured before and after the "lnet_selftest" run. Verify that  test traffic is distributed evenly across NIDs.
  4. Compare "netstat -i" outputs captured before and after the "lnet_selftest"run. Verify that test traffic is evenly distributed across interfaces used by LNet.
skip due to IPv6_only.Basic.LNet.SR.1
IPv6_only.Basic.Lustre.MR.1Test Lustre FS performance with the Client and Server setup to use IPv6. (MR, same subnet)

Use the lustre server and clients setup in previous tests and

  1. Mount lustre FS on the client(s)
  2. Run FIO
  1. Lustre FS mount should succeed
  2. FIO should run without errors. Throughput reported by FIO should be adequate (verify against link speed and "lnet_selftest" results).
pass
IPv6_IPv4_collocated.Config.MR.LNet.1Configure node to use one IPv6 and one IPv4 interface for LNet (MR, two different lnets)
  1. Select a machine which is able to use both IPv4 and IPv6 and has at least two interfaces configured such that one can be IPv6 and another IPv4.
  2. Configure lustre on the node such that both IPv4 and IPv6 are used each on its separate lnet
  3. Bring up lustre
  4. Execute "lnetctl net show"
  5. Execute "lnetctl ping" of own NID
  1. "lnetctl net show" output should list both IPv4 and IPv6 NIDs as active.
  2. "lnetctl ping" of own NIDs should be successful
passThis test is run at LNet level so it doesn't matter whether nodes are designated as servers or clients
IPv6_IPv4_collocated.Basic.MR.LNet.1Test basic connectivity between nodes on LNet level. (MR, same subnet)
  1. Configuration is the same as IPv6_IPv4_collocated.Config.MR.LNet.1
  2. Make sure that nodes have "discovered" each other
  3. Load "lnet_selftest" on each node
  4. Configure and run "lnet_selftest" between the nodes
  1. "lnet_selftest" should run without errors. 
  2. Throughput reported by "lnet_selftest" should be adequate (verify against link speed).
  3. Compare "lnetctl net show -v 4" outputs captured before and after the "lnet_selftest" run. Verify that  test traffic is distributed evenly across NIDs.
  4. Compare "netstat -i" outputs captured before and after the "lnet_selftest"run. Verify that test traffic is evenly distributed across interfaces used by LNet.
skip due to IPv6_only.Basic.LNet.SR.1This test is run at LNet level so it doesn't matter whether nodes are designated as servers or clients
IPv6_IPv4_routed.Config.SR.LNet.1Configure one node to use a single IPv6 NID and another to use IPv4 NID. Add lnet router between the two. (SR, lnet routing)
  1. Select 3 machines: A, B and R. A needs to have IPv6 interface configured. B needs to have IPv4 interface configured. R needs two interfaces, one IPv4 and another IPv6. 
  2. Configure LNet on A to use tcp0 and on B to use tcp1
  3. Configure LNet on  R to have IPv6 NID on tcp0 and IPv4 on tcp1. Enable lnet routing on R.
  4. Add route to tcp1 via R on A.
  5. Add route to tcp0 via R on B.
  6. Bring up A and B
  7. Bring up R
  8. Execute "lnetctl route show" on A and B
  9. Execute "lnetctl ping" from A to B
  10. Execute "lnetctl ping" from B to A
  1. "lnetctl route show" should indicate that routes are "up"
  2. "lnetctl ping" between A and B should succeed
skip due to observation made in IPv6_only.Config.SR.Client.2
IPv6_IPv4_routed.Basic.SR.LNet.1Test basic connectivity between IPv4 and IPv6 nodes via LNet gateway. (SR, lnet routing)
  1. Use configuration from IPv6_IPv4_routed.Config.SR.LNet.1
  2. Load "lnet_selftest" on nodes A and B.
  3. Configure and run "lnet_selftest" between the nodes A and B
  1. "lnet_selftest" should run without errors. 
  2. Throughput reported by "lnet_selftest" should be adequate (verify against link speed).

skip due to observation made in IPv6_only.Config.SR.Client.2

and IPv6_only.Basic.LNet.SR.1


IPv6_o2ib_routed.Config.SR.LNet.1Configure one node to use a single IPv6 NID and another to use an o2ib NID. Add lnet router between the two. (SR, lnet routing)
  1. Select 3 machines: A, B and R. A needs to have IPv6 interface configured. B needs to have o2ib interface configured. R needs two interfaces, one o2ib and another IPv6. 
  2. Configure LNet on A to use tcp0 and on B to use o2ib0
  3. Configure LNet on  R to have NIDs on o2ib and tcp0. Enable lnet routing on R.
  4. Add route to o2ib via R on A.
  5. Add route to tcp0 via R on B.
  6. Bring up A and B
  7. Bring up R
  8. Execute "lnetctl route show" on A and B
  9. Execute "lnetctl ping" from A to B
  10. Execute "lnetctl ping" from B to A
  1. "lnetctl route show" should indicate that routes are "up"
  2. "lnetctl ping" between A and B should succeed

fail:

configuring IPv6 NID on an IB interface using o2ib net fails with an error if "lnetctl net add --nid" is used 

configuring IPv6 NID on an IB interface using o2ib net results in a corrupt IPv4-style NID if "lnetctl net add --net o2ib --if ib0" is used and first address listed for the interface is IPv6 global (LU-18317 created)

"lnetctl ping" from o2ib client to tcp (IPv6) server across an lnet router fails. Debug logs indicate NID corruption. Both endpoints are able to lnetctl ping the router NIDs. Routes are shown as "up". (LU-18318 created)



IPv6_o2ib_routed.Basic.SR.LNet.1Test basic connectivity between o2ib and IPv6 nodes via LNet gateway. (SR, lnet routing)
  1. Use configuration fromIPv6_o2ib_routed.Config.SR.LNet.1
  2. Load "lnet_selftest" on nodes A and B.
  3. Configure and run "lnet_selftest" between the nodes A and B
  1. "lnet_selftest" should run without errors. 
  2. Throughput reported by "lnet_selftest" should be adequate (verify against link speed).
skip due to IPv6_only.Basic.LNet.SR.1












Negative

IDObjectiveProcedureExpected BehaviourActual ResultsNotes
IPv6_only.Neg.Reboot.Client.MRVerify that connectivity can be restored after unexpected client reboot
  1. Setup same as IPv6_only.Basic.Lustre.MR.1
  2. Mount lustre FS on the client(s)
  3. Reboot the client host
  4. When client host is ready, load lustre and mount FS again on the client
  1. Client is able to mount lustre FS after being rebooted
pass
IPv6_only.Neg.Reboot.Server.MRVerify that connectivity can be restored after unexpected server reboot
  1. Setup same as IPv6_only.Basic.Lustre.MR.1
  2. Mount lustre FS on the client(s)
  3. Reboot the server host
  4. When server host is ready, load lustre on the server and mount FS again on the client
  1. Client is able to mount lustre FS after the server  reboot
pass
IPv6_only.Neg.Local_link_down.Client.MRVerify that local link down events are handled gracefully
  1. Setup same as IPv6_only.Basic.Lustre.MR.1
  2. Mount lustre FS on the client(s)
  3. Try the following methods of disrupting the local link: a) cable pull b) change interface state to "down" with "ifdown" or "ip link set state" c) remove ip address with "ip -6 addr del"
  4. Use "lnetctl net show" to check the NID state
  5. Restore local link 
  6. Use "lnetctl net show" to check the NID state
  7. Attempt a FS operation
  1. "lnetctl net show" indicates that state of the NID corresponding to the downed link follows the link state.
  2. FS operations can be resumed after the link is restored
pass




  • No labels

1 Comment

  1. Frank Sehr

    Can we include error cases to see if the system handles errors correctly or at least gracefully and restores the system correctly. These could include

    1) Shutdown/disconnect server

    2) Shutdown/disconnect clients

    3) Simulated network problems like reused IP address/ routing issues, disconnect switches, .....

    4) ......