Improve e2e framework - refactor to remove shell

[chrislovecnm]

We are currently working on getting the e2e more stable, thanks @smukherj1, but IMO the bash scripts are kinda brittle. With the loops and such they are quite complex for a newer bash coder. I am recommending moving to another language that is testable, and more flexible. In order to have people write more tests, it has to be easy to add an e2e test. For instance, I am struggling to find a good entrypoint to add a test to #198. I really believe in tests, but without the capability to easily add new tests, people won't write them.

Here are some ideas.

1. Refactor the tests into a smaller chunks that can run in || - this allows for faster testing, and reduces the changes of flakes
2. Move to a language that is testable that has a testing framework. The two that come to mind are go or python. I would rather have go, but that is just me.
3. Document how to add a test

Thoughts? Concerns??

/cc @nlopezgi @fejta @smukherj1

[pierreis]

One additional comment for my part. I found it to be incredibly complicated to run e2e tests outside of the build environment because of the strong dependency to gcloud. Actually, this also creates a dependency to an external service, which is not ideal, but I'm nor sure to see any more practical solutions here.

In any case, I would for sure isolate as much as possible this dependency, most likely in a separate script that only takes care of setting up the context, and passing the required context / cluster / namespace information to the test runner.

Both Go and Python are find IMO.

[nlopezgi]

thanks for starting this thread @chrislovecnm ,
@alex1545 is working on documenting how to set up the context / cluster / namespace stuff to make it feasible for anyone to be able to run the tests locally. The underlying issue of flakiness is the really big issue, and the opportunity we have is that I have seen many different rules have a need to do this type of e2e testing and there are no good solutions atm (though we should focus on the rules_k8s use case exclusively atm).

I do think the solution is to have a testing framework. My preference is that we do something in go as its better supported in bazel, and my preference is to have bazel itself drive the e2e tests themselves (i.e., bazel to start up a sandbox env in which bazel run can be executed and success/error/results can be validated).

I think someone with time to work on this should start a design doc ( @chrislovecnm want to volunteer? otherwise for my team it would be in the Q1 timeframe to work on this properly). If we do find a volunteer we should set up some time to gvc to talk high level goals and design.

[chrislovecnm]

@pierreis see #202 for improving running the tests locally, and I would add outside of gcloud

[chrislovecnm]

and my preference is to have bazel itself drive the e2e tests themselves

That would be nice. But it would still need to run like a go test or a python test. So I am not understanding completely.

should start a design doc

For bazel itself or for this repo??

[pierreis]

To be clear, @nlopezgi, by saying that Bazel should drive the tests, you mean creating rules that would control the compilation / execution of the test cases as well as the setup / teardown of the environment?

[nlopezgi]

sorry for the delayed response @pierreis , yes, at a high level I do think bazel rules should be used to do all of that