Viewing: README.lst-survey
Overview
--------
This survey script performs a series of LNet selftest (LST) benchmarks between
groups of LNet peers. It can be used to characterize the performance of the LNet
interface(s) on Lustre servers, Lustre clients, or LNet routers.
The LST client group is defined using the '-f' flag, and the LST server group
is defined using the '-t' flag. Both of these flags take a space-separated or
comma-separated list of LNet NIDs. The '-M' and '-N' options can be used to
divide the client or server group into multiple smaller groups.
For example, given 16 clients and 8 servers, '-M 8' and '-N 2' would create
two client groups with eight peers in each group, and four servers groups with
two peers in each group. Every server group is tested against every client
group, so this would result in 4*2=8 test iterations.
By default, each test iterations performs 4k read and write, 1m read and write,
and ping LST benchmarks.
A directory is created in the current working directory to store results.
The csv output is written to a results.<timestamp>.csv file and the full
lst.sh output is stored in an lst.<timestamp>.out file. An alternative output
directory can be specified with the '-O' argument.
Various options exist to customize the benchmarks that are run. See
'lst-survey -h' for more information.
A note on interpreting the results:
By default, lst-survey displays bandwidth and rate statistics for peers in the
server group as reported by the LST utility.
These statistics reported by LST can be confusing because a "read" test will
typically report read bandwidth that is lower than write bandwidth, and a
"write" test will typically report write bandwidth that is lower than read
bandwidth. This is because a "read" test involves peers in the client group
setting up a sink that is then written to by peers in the server group, and a
"write" test involves the clients setting up a source that is then read by the
servers. Thus, the read test is really measuring the write performance of the
servers and the write test is really measuing the read performance of the
servers.
The '-g clients' option can be used to instead report the client bandwidth and
rate statistics. In this case, the reported stats will align with the benchmarks
in the expected manner.
Example 1: Default options
# pdsh -w n0[0-3] lctl list_nids | dshbak -c
----------------
n00
----------------
172.18.2.5@tcp
----------------
n01
----------------
172.18.2.6@tcp
----------------
n02
----------------
172.18.2.7@tcp
----------------
n03
----------------
172.18.2.8@tcp
# ./lst-survey -t 172.18.2.5@tcp,172.18.2.6@tcp -f 172.18.2.7@tcp,172.18.2.8@tcp
CSV results: /tmp/lst_survey.1666207637/results.1666207637.csv
LST output: /tmp/lst-survey/lst_survey.1666207637/lst.1666207637.out
Commence lst-survey - Wed 19 Oct 2022 01:27:17 PM MDT
Server Group: 172.18.2.5@tcp 172.18.2.6@tcp
Client Group: 172.18.2.7@tcp 172.18.2.8@tcp
Mode Read MB/s Read RPC/s Write MB/S Write RPC/s
read 4k 22 149981 608 299961
read 1m 2 14405 14405 28808
write 4k 489 241229 18 241229
write 1m 11463 22924 1 22924
ping 25 167928 25 167928
Finished lst-survey - Wed 19 Oct 2022 01:28:08 PM MDT
# cat /tmp/lst_survey.1666207637/results.1666207637.csv
Servers,Clients,Mode,Read_BW,Read_Rate,Write_BW,Write_Rate,Server_Errors,Client_Errors
172.18.2.5@tcp 172.18.2.6@tcp,172.18.2.7@tcp 172.18.2.8@tcp,read_4k,22,149981,608,299961,0,0
172.18.2.5@tcp 172.18.2.6@tcp,172.18.2.7@tcp 172.18.2.8@tcp,read_1m,2,14405,14405,28808,0,0
172.18.2.5@tcp 172.18.2.6@tcp,172.18.2.7@tcp 172.18.2.8@tcp,write_4k,489,241229,18,241229,0,0
172.18.2.5@tcp 172.18.2.6@tcp,172.18.2.7@tcp 172.18.2.8@tcp,write_1m,11463,22924,1,22924,0,0
172.18.2.5@tcp 172.18.2.6@tcp,172.18.2.7@tcp 172.18.2.8@tcp,ping,25,167928,25,167928,0,0
#
Example 2: Divide the servers into groups of size 1
# ./lst-survey -t 172.18.2.5@tcp,172.18.2.6@tcp -f 172.18.2.7@tcp,172.18.2.8@tcp -N 1
CSV results: /tmp/lst_survey.1666207844/results.1666207844.csv
LST output: /tmp/lst_survey.1666207844/lst.1666207844.out
Commence lst-survey - Wed 19 Oct 2022 01:30:44 PM MDT
Server Group: 172.18.2.5@tcp
Client Group: 172.18.2.7@tcp 172.18.2.8@tcp
Mode Read MB/s Read RPC/s Write MB/S Write RPC/s
read 4k 25 167068 678 334135
read 1m 2 16186 16186 32366
write 4k 512 252613 19 252612
write 1m 11353 22706 1 22704
ping 29 192358 29 192358
Finished lst-survey - Wed 19 Oct 2022 01:31:34 PM MDT
Commence lst-survey - Wed 19 Oct 2022 01:31:34 PM MDT
Server Group: 172.18.2.6@tcp
Client Group: 172.18.2.7@tcp 172.18.2.8@tcp
Mode Read MB/s Read RPC/s Write MB/S Write RPC/s
read 4k 22 144821 587 289642
read 1m 2 16841 16843 33681
write 4k 498 245552 18 245552
write 1m 11611 23219 1 23217
ping 22 145374 22 145374
Finished lst-survey - Wed 19 Oct 2022 01:32:25 PM MDT
#
Example 3: Divide the servers and clients into groups of size 1
# ./lst-survey -t 172.18.2.5@tcp,172.18.2.6@tcp -f 172.18.2.7@tcp,172.18.2.8@tcp -N 1 -M 1
CSV results: /tmp/lst_survey.1666208473/results.1666208473.csv
LST output: /tmp/lst_survey.1666208473/lst.1666208473.out
Commence lst-survey - Wed 19 Oct 2022 01:41:13 PM MDT
Server Group: 172.18.2.5@tcp
Client Group: 172.18.2.7@tcp
Mode Read MB/s Read RPC/s Write MB/S Write RPC/s
read 4k 11 75112 304 150224
read 1m 1 8808 8809 17616
write 4k 240 118402 9 118402
write 1m 6561 13119 1 13118
ping 13 90402 13 90402
Finished lst-survey - Wed 19 Oct 2022 01:42:03 PM MDT
Commence lst-survey - Wed 19 Oct 2022 01:42:03 PM MDT
Server Group: 172.18.2.5@tcp
Client Group: 172.18.2.8@tcp
Mode Read MB/s Read RPC/s Write MB/S Write RPC/s
read 4k 13 90017 365 180034
read 1m 1 7333 7328 14655
write 4k 280 138173 10 138173
write 1m 8694 17388 1 17388
ping 15 98316 15 98316
Finished lst-survey - Wed 19 Oct 2022 01:42:53 PM MDT
Commence lst-survey - Wed 19 Oct 2022 01:42:53 PM MDT
Server Group: 172.18.2.6@tcp
Client Group: 172.18.2.7@tcp
Mode Read MB/s Read RPC/s Write MB/S Write RPC/s
read 4k 9 64613 262 129225
read 1m 1 9101 9101 18201
write 4k 212 104575 7 104573
write 1m 6769 13537 1 13539
ping 10 71612 10 71612
Finished lst-survey - Wed 19 Oct 2022 01:43:44 PM MDT
Commence lst-survey - Wed 19 Oct 2022 01:43:44 PM MDT
Server Group: 172.18.2.6@tcp
Client Group: 172.18.2.8@tcp
Mode Read MB/s Read RPC/s Write MB/S Write RPC/s
read 4k 12 83144 337 166287
read 1m 1 7582 7584 15166
write 4k 293 144601 11 144602
write 1m 8913 17824 1 17825
ping 11 78409 11 78409
Finished lst-survey - Wed 19 Oct 2022 01:44:35 PM MDT
#
