[DevTSAN][E2E] Run test with deflake script to make positive tests more robust

[zhaomaosu]

Also disable two unexpected flaky negative tests for further investigation

[zhaomaosu]

@intel/dpcpp-devops-reviewers, @intel/llvm-reviewers-runtime, could you please help review this PR? Thanks.

[aelovikov-intel]

https://llvm.org/docs/CommandGuide/lit.html#test-status-results:

FLAKYPASS

The test succeeded after being re-run more than once. This only applies to tests containing an ALLOW_RETRIES: annotation.

Can you please check if that would work for you? Unfortunately, this feature doesn't have any other documentation (at least I was unable to find out), so you'd need to do some work here.

[zhaomaosu]

Can you please check if that would work for you? Unfortunately, this feature doesn't have any other documentation (at least I was unable to find out), so you'd need to do some work here.

Hi @aelovikov-intel, thanks for your information. I tried 'FLAKYPASS' locally, it works here. But there is problem that 'FLAKYPASS' will retry whole testcase (including build and run) when meet fail, it will waste a lot of unnecessary time (we only need to retry run). So I think current solution to use deflake script is more suitable for sycl/test-e2e.

[aelovikov-intel]

In pre-commit CI we run build/run of e2e tests separately... Also, can you look into implementing something like these retries for "run"-only RUN lines in e2e's format.py instead?

[zhaomaosu]

In pre-commit CI we run build/run of e2e tests separately...

@aelovikov-intel, updated the PR to use "ALLOW_RETRIES".

Also, can you look into implementing something like these retries for "run"-only RUN lines in e2e's format.py instead?

The retry's work is controlled by lit.TestRunner.py https://github.com/intel/llvm/blob/sycl/llvm/utils/lit/lit/TestRunner.py#L2268. It's test level instead of command level. So if we want to implement it in format.py, the most obvious way is to substitute '%{run}' with something like deflake script do when ALLOW_RETRIES is specified. Not sure if this is what you are expecting.

[aelovikov-intel]

I was too late to spot that title needed an update... :(

[zhaomaosu]

I was too late to spot that title needed an update... :(

Sorry, I also forgot to update it...