Split tests into multiple executables

[levkropp]

This pull request reorganizes the test suite in the tests/CMakeLists.txt file by splitting the tests into multiple categories and creating separate executables for each category.

Before:
multipass_tests = 3857 tests from 233 test suites ran. (42261 ms total)

Now:
multipass_tests = 1312 tests from 111 test suites ran. (12894 ms total)
multipass_client_ssh_tests = 820 tests from 56 test suites ran. (20483 ms total)
multipass_daemon_tests = 322 tests from 30 test suites ran. (2441 ms total)
multipass_formatter_tests = 373 tests from 9 test suites ran. (120 ms total)
multipass_memory_tests = 823 tests from 4 test suites ran. (36 ms total)
multipass_network_tests = 89 tests from 8 test suites ran. (2622 ms total)
multipass_qemu_tests = 131 tests from 16 test suites ran. (2271 ms total)

Reorganization of test suite:

• Created separate test source sets for SSH, Memory, Formatter, Network, Client, and QEMU tests, and updated the add_executable commands to create distinct test executables for each category.
• Added common include directories and linked libraries for the new test executables, ensuring that all necessary dependencies are included.
• Updated the add_test commands to include the new test executables, enabling them to be run as part of the test suite.
• Added mock binaries for testing and set dependencies for the new test executables to ensure the mock binaries are built before running the tests.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.27%. Comparing base (f7ba74f) to head (bbb52e5).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3998   +/-   ##
=======================================
  Coverage   89.27%   89.27%           
=======================================
  Files         259      259           
  Lines       14653    14653           
=======================================
  Hits        13081    13081           
  Misses       1572     1572           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xmkg]

Hi @levkropp, that's looking great! I just have one topic to discuss while we're at it.

The different categories are already better than what we have; shall we go another step further and make each test source a separate executable? This way, each of the test suites is isolated at the process level, and we can also parallelize their execution if we want to. Also, we can have better control over what each test suite depends on.

ctest is capable of curating tests from different executables, so the number of executables won't matter. It's designed to work in this way anyway.

@ricab @georgeliao opinions?

[ricab]

I don't know, I fear that total link time may grow too much when changing something that affects many tests. I believe linking is where most time is spent when developing, more than test execution.

With N executables we'll probably end up paying with link time when changing something that lots of code depends on (say utils), because we now have to link in N executables. But we gain when iterating on tests or higher level stuff, because they would affect a smaller executable. And linking smaller executables totalling M bytes is probably faster than linking the same M bytes in a single executable.

I suspect there is a sweet spot somewhere. Ideally we'd be able to make a graph of total link time when changing different parts of the code and chose the configuration that minimizes that. In practice, we could shoot a little lower and try a few configurations to compare. Another option could be to try to move things from static to dynamic libraries... I would take any improvement we can get.