tests: adapt the test.py script to build project tests in offline mode and accept arbitrary arguments for ctest #2682
Conversation
there are some tests that require access to global resources, like google.com or seastar.io, which are not accessible from the restricted environment. the tests give a false negative. setting ENABLE_TESTS_ACCESSING_INTERNET to 0 maros allows us to exclude such tests. let's add a handle to test.py to pass the value for this macro through cmake to build and run tests in standalone mode.
…ional args sometimes you only need to run a subset of tests. test.py can already enumerate the desired ones, but it happens that you need to exclude a subset from the full list. ctest -E takes a regular expression of test names to exclude. ctest also provides more useful features that we may want to use in the future. let's not repeat the parsing of the ctest command line, but instead redirect all positional arguments of this script directly to ctest.
| @@ -37,6 +37,8 @@ | |||
| parser.add_argument('--smp', '-c', action="store",default='2',type=int,help="Number of threads for multi-core tests") | |||
| parser.add_argument('--verbose', '-v', action = 'store_true', default = False, | |||
| help = 'Verbose reporting') | |||
| parser.add_argument('--offline', action="store_true", default = False, | |||
There was a problem hiding this comment.
this change still makes sense even with #2683 .
tests/unit/dns_test.cc
Outdated
| @@ -36,6 +36,8 @@ using namespace seastar::net; | |||
|
|
|||
| static const sstring seastar_name = "seastar.io"; | |||
|
|
|||
| #if SEASTAR_TESTING_WITH_NETWORKING | |||
There was a problem hiding this comment.
@Comrade88 , hi Galas, thank you for this fix! You've correctly identified that several tests need proper handling with the SEASTAR_TESTING_WITH_NETWORKING macro.
While your approach works, I've taken the initiative to implement an alternative solution in PR #2683 that centralizes all the network-dependency flags in a single location. This should improve code readability and maintenance by avoiding scattered conditional checks throughout the codebase.
Could you please take a look at the alternative implementation when you have a chance. I believe it addresses the same issue you've identified while providing a more consolidated approach to managing network-dependent tests.
test.py
Outdated
| @@ -26,6 +27,7 @@ | |||
| parser = argparse.ArgumentParser(description="Seastar test runner") | |||
| parser.add_argument('--fast', action="store_true", help="Run only fast tests") | |||
| parser.add_argument('--name', action="store", help="Run only test whose name contains given string") | |||
| parser.add_argument('--skip', action="append", help="Skip tests whose name contains given string") | |||
There was a problem hiding this comment.
how about just exposing a flag allowing user to pass random parameters to customize the behavior of ctest? like
./test.py --mode debug -- --test-timeout 3600
There was a problem hiding this comment.
good point. done.
also i've dropped the dns test patch in favor of PR #2683
|
is there anything else that needs to be changed here? |
|
@Comrade88 , the provided PR description is too shallow, please write more profound one |
Comrade88
changed the title
|
@xemul , done |
|
@Comrade88 currently the description of this PR puts:
i guess Pavel suggested to make it more specific. |
this pr suggests adding to test.py the ability to build "offline" mode tests. i.e. exclude tests that require access to global resources. such tests always fail in a restricted environment and create a false impression of the project's performance.
also added the ability to feed ctest arbitrary parameters to gain more control over running tests.