Continuous Testing: add support for build system like test selection

[Ladicek]

This commit adds support for selecting tests to run during continuous testing that is very close to the native test filtering capability of the given build system. That is, in Maven, there's support for the -Dtest=... system property, while in Gradle, there's support for the --tests command line option. Format of these options follows the format used by the build system.

Fixes #20231

[github-actions]

🙈 The PR is closed and the preview is expired.

[Ladicek]

I tried looking at the Istio tests and I honestly have no idea how they are supposed to work. The kubernetes-client-api-server-url system property doesn't seem to be set to any meaningful value anywhere (as opposed to the Kubernetes invoker tests, which at least do set it). I'll continue digging.

[aloubyansky]

I was wondering whether we could keep build tool-specific parsing somewhere in that build tool integration code (Gradle/Maven plugin impl) and pass a value in an expected format here.
That way we would keep build tool-specific code in the build tool integration module.

That raises the question which format we should adopt here: either our own version or pick one of Maven or Gradle.

[Ladicek]

So the Gradle format is super dumb and it could be translated to the Maven format ... if it didn't use a period . as a separator of class name pattern and method name pattern. Figuring out what any given period means would require adding more heuristics to the parsing side which are currently not present (because the decision happens during evaluation).

Also, the place where we have access to the user-provided value is in a different process than the place where we have to apply it. So we have to "serialize" it and then "parse" again anyway. In which case, I figured it would be simplest to pass the Maven-style or Gradle-style string directly, with minimum changes, and parse it in the test runner. The other option would either require defining a custom format, for no good reason, or to creatively use regular expressions, which would be doable, but those regexps would be relatively complex, because they would have to express in one place what we currently can express with multiple relatively simple regexps.

[gsmet]

I rebased to check if I finally fixed all the CI issues. Fingers crossed.

[quarkus-bot]

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit 528ae6b.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

Warning

There are other workflow runs running, you probably need to wait for their status before merging.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 528ae6b.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

---

Flaky tests - Develocity

⚙️ JVM Tests - JDK 17

📦 extensions/hibernate-orm/deployment

✖ io.quarkus.hibernate.orm.applicationfieldaccess.PublicFieldAccessInheritanceTest.testFieldAccess - History

• Expecting actual not to be null - java.lang.AssertionError

java.lang.AssertionError: 

Expecting actual not to be null
	at io.quarkus.hibernate.orm.applicationfieldaccess.PublicFieldAccessInheritanceTest$FieldAccessEnhancedDelegate$2.assertValue(PublicFieldAccessInheritanceTest.java:152)
	at io.quarkus.hibernate.orm.applicationfieldaccess.PublicFieldAccessInheritanceTest.doTestFieldAccess(PublicFieldAccessInheritanceTest.java:100)
	at io.quarkus.hibernate.orm.applicationfieldaccess.PublicFieldAccessInheritanceTest.testFieldAccess(PublicFieldAccessInheritanceTest.java:61)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.QuarkusUnitTest.runExtensionMethod(QuarkusUnitTest.java:521)

---

⚙️ JVM Tests - JDK 17 Windows

📦 extensions/micrometer-opentelemetry/deployment

✖ io.quarkus.micrometer.opentelemetry.deployment.compatibility.MicrometerTimedInterceptorTest.testTimeMethod_Failed - History

• Stream has no elements - java.lang.IllegalArgumentException

java.lang.IllegalArgumentException: Stream has no elements
	at io.quarkus.micrometer.opentelemetry.deployment.common.MetricDataFilter.lastReadingDataPoint(MetricDataFilter.java:236)
	at io.quarkus.micrometer.opentelemetry.deployment.compatibility.MicrometerTimedInterceptorTest.testTimeMethod_Failed(MicrometerTimedInterceptorTest.java:100)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.QuarkusUnitTest.runExtensionMethod(QuarkusUnitTest.java:521)
	at io.quarkus.test.QuarkusUnitTest.interceptTestMethod(QuarkusUnitTest.java:435)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[Ladicek]

Thanks! I've attempted investigation in my fork, but couldn't find anything, and now it works :-)

[aloubyansky]

Great work @Ladicek , thanks!