Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move download, SHA256 check and unzip of Windows buildenv from windows_buildenv.bat to CMakelists.txt #13306

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

JoergAtGithub
Copy link
Member

This move the VCPKG buildenv download, SHA256 verification and unzipping into CMakelists.txt. This removes the need for third party tools like 7zip and Powershell (see #11524 for reference).

For the developer everything should behave as before. No change in the build instructions is neccessary - just the tasks mentioned above are executed later.

Out of scope of this PR is:

  • macOS
  • Replacing windows_buildenv.bat and CMakeSettings.json by CMake Presets

@github-actions github-actions bot added the build label May 30, 2024
@JoergAtGithub JoergAtGithub force-pushed the cmake_download_buildenv_zip branch from 22b1df5 to ecc462a Compare May 30, 2024 16:52
Copy link
Member

@daschuer daschuer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank for adopting this.

@JoergAtGithub JoergAtGithub requested a review from daschuer June 28, 2024 18:23
@JoergAtGithub JoergAtGithub force-pushed the cmake_download_buildenv_zip branch from 9cea1ee to b09ace0 Compare June 29, 2024 07:09
@JoergAtGithub
Copy link
Member Author

@daschuer Friendly Ping

@JoergAtGithub
Copy link
Member Author

@daschuer Can we merge this?

@@ -47,6 +47,68 @@ if(POLICY CMP0135)
cmake_policy(SET CMP0135 NEW)
endif()


if(WIN32)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you mentioned it as out-of-scope, but I agree that sharing this logic with macOS would be great, given that the vcpkg envs are structured practically identically as far as I understand.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if you could create a follow-up PR for macOS.

@JoergAtGithub
Copy link
Member Author

@daschuer Can we merge this now?

Copy link
Member

@daschuer daschuer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only works if we make sure
"MIXXX_VCPKG_ROOT = BUILDENV_BASEPATH/BUILDENV_NAME"

What are the use cases if this is not the case?

If we tray to use an alternative build environment "MIXXX_VCPKG_ROOT" needs to be set.
If that exists, I think we can skip the whole evaluation of "BUILDENV_BASEPATH/BUILDENV_NAME"

In case it does not exists, I think there is no point in downloading to a diffrent folder than "MIXXX_VCPKG_ROOT".
So we may fail fatal in that case. However if we are in a shell where "BUILDENV_BASEPATH/BUILDENV_NAME" is set it will allways fail.

Conclusion we may only use "MIXXX_VCPKG_ROOT" and may dispose "BUILDENV_BASEPATH/BUILDENV_NAME"

What do you think?

@JoergAtGithub JoergAtGithub force-pushed the cmake_download_buildenv_zip branch from b09ace0 to efa7b67 Compare February 9, 2025 16:36
@acolombier
Copy link
Member

/softfix:squash

feat: move buildenv setup to CMakeFile

@acolombier acolombier mentioned this pull request Feb 18, 2025
@github-actions github-actions bot force-pushed the cmake_download_buildenv_zip branch from f1ac722 to 7db1d4f Compare February 18, 2025 01:25
@acolombier
Copy link
Member

/softfix

feat: move buildenv setup to CMakeFile

@github-actions github-actions bot force-pushed the cmake_download_buildenv_zip branch from 7db1d4f to 8779ff3 Compare February 18, 2025 01:43
@acolombier
Copy link
Member

@daschuer were your change requests addressed? Merge?

@acolombier
Copy link
Member

Sorry for the early fixup @JoergAtGithub , I missred that PR and thought it was ready to go

@JoergAtGithub JoergAtGithub force-pushed the cmake_download_buildenv_zip branch from 8779ff3 to f1ac722 Compare March 14, 2025 18:14
@github-actions github-actions bot added the developer experience Issues, bugs and PRs related to the development process, development environment & developer docs label Mar 14, 2025
@JoergAtGithub
Copy link
Member Author

Restored original commits

@JoergAtGithub
Copy link
Member Author

One concern is what will happen if a user switches between branches with this PR merged and without. Will it run flawlessly or is the user always prompted with errors?

If all settings inside windows_buidenv.bat and the buildenv remain the same, the user has not to rerun it. It will work as now. If the buildenv or settings differ the user has to rerun it - this is also the same behavior as now.

@daschuer
Copy link
Member

Did you test it? For my understanding building an old build directory after applying this patch will fail because BUILDENV_NAME and BUILDENV_URL are not set.

@JoergAtGithub JoergAtGithub force-pushed the cmake_download_buildenv_zip branch from 4acb7a6 to 163f0bc Compare March 16, 2025 09:38
…XX_VCPKG_ROOT from windows_buildenv.bat to the place of it's first usage in CMakeLists.txt
@JoergAtGithub JoergAtGithub force-pushed the cmake_download_buildenv_zip branch from 163f0bc to ccc613b Compare March 16, 2025 10:18
@Eve00000
Copy link
Contributor

If you want me to do some more test ... let me know

@JoergAtGithub
Copy link
Member Author

If you want me to do some more test ... let me know

Oh yes, now it's a good time to test! As all is addressed now!

@JoergAtGithub
Copy link
Member Author

Did you test it? For my understanding building an old build directory after applying this patch will fail because BUILDENV_NAME and BUILDENV_URL are not set.

It tested now with switching from this PR to main. Deleted the CMake cache and configured without calling windows_buildenv.bat again. As the buildenv was still there and the toolchain file remains the same everything worked. I just got these warnings at the end:

1> [CMake] -- Configuring done (22.8s)
1> [CMake] -- Generating done (1.2s)
1> [CMake] CMake Warning:
1> [CMake]   Manually-specified variables were not used by the project:
1> [CMake] 
1> [CMake]     BUILDENV_BASEPATH
1> [CMake]     BUILDENV_NAME
1> [CMake]     BUILDENV_SHA256
1> [CMake]     BUILDENV_URL
1> [CMake] 
1> [CMake] 
1> [CMake] -- Build files have been written to: D:/mixxx-2.5-worktree/build/x64__legacy

Going the other direction it fails now with the following clear message:

1> [CMake] -- CMAKE_VERSION: 3.30.5-msvc23
1> [CMake] -- BUILDENV_URL not specified
1> [CMake] CMake Error at D:\mixxx-2.5-worktree\CMakeLists.txt:86 (message):
1> [CMake]   Neither BUILDENV_URL nor BUILDENV_NAME specified!
1> [CMake] 
1> [CMake]   Did you run `D:/mixxx-2.5-worktree/tools/windows_buildenv.bat`?
1> [CMake] 
1> [CMake] 
1> [CMake] -- Configuring incomplete, errors occurred!

@Eve00000
Copy link
Contributor

I did the test
-> removed all dep - linked items + .vs . json + ...
-> started VS -> cmake -> error 'tools\windows_buildenv.bat'
-> closed VS
-> executed windows_buildenv.bat
-> started VS
-> download + extract
CMake generation finished.
All good, nice! Congrats.

@daschuer
Copy link
Member

Going the other direction it fails now with the following clear message.

But isn't this unnecessary? To make it work was my idea if using MIXXX_VCPKG_ROOT primary.
What is the benefit of the current situation that forces all contributors to run the batch again?

@JoergAtGithub
Copy link
Member Author

JoergAtGithub commented Mar 16, 2025

No it's not, as we have to set cmakeToolchain anyway. and this requires it to run the script anyway. MIXXX_VCPKG_ROOT is only relevant for one single purpose: Checking if the buildenv is installed inside CMakeLists.txt. It's not considered by any other tool or IDE.
Please note that also other settings of the build are set by this script. Unless they are experts like you and me, users should run it any time.
With the current state of the PR we have a very simple straightforward data flow. MIXXX_VCPKG_ROOT was always generated by concatenating BUILDENV_BASEPATH and BUILDENV_NAME and with moving this to CMakeLists.txt I also addressed your request above, to unify the variable names. Only the variables with prefix BUILDENV remains now visible. And that without adding any new variable. It just moves the logic from the batch to CMakeLists.txt.

And this addresses also your request, that multiple worktrees can share the same buildenv. This is possible, because you can overwrite BUILDENV_BASEPATH by env variable. This wouldn't be possible by MIXXX_VCPKG_ROOT.

@daschuer
Copy link
Member

I can confirm that disposing MIXXX_VCPKG_ROOT is a good idea. It has never been documented as a user variable.

However in a context of an IDE this PR still introduces an issue. cmakeToolchain goes as CMAKE_TOOLCHAIN_FILE to the CMakeCache.txt this is a one time task when configuring the project

Unless they are experts like you and me, users should run it any time.

I have not tested this because I am not on Windows, but the for my understanding the batch file is not re-run if you use an IDE like QtCreator. In case of Visual studio, the settings ares stored in CMakeSettings.json. Other IDEAs have other ways to configure the CMakeCache.txt.

In Eclipse (I use on Linux) for example I can imagine to configure the batch file one time, to configure the project form the command line and than use only Eclipse without touching the command line ever again.

So I think instead of MIXXX_VCPKG_ROOT, CMAKE_TOOLCHAIN_FILE is the real source of truth. This is also widely known because it is a CMake standard variable. Can we check this to skip the prompt for rerunning the batch?

I will propose the change inline.

@daschuer
Copy link
Member

Since MIXXX_VCPKG_ROOT is now internal, we need to adjust the error messages mentions that.

@daschuer
Copy link
Member

I have just noticed that there is still the logic of configuring VCPKG_OVERLAY_PORTS and VCPKG_OVERLAY_TRIPLETS.
are they part of the CMakeCache? If not we need to take care that they are always configured.

@JoergAtGithub
Copy link
Member Author

Since MIXXX_VCPKG_ROOT is now internal, we need to adjust the error messages mentions that.

I could, but this variable is still generated by macos_buildenv.sh, where it's also generated by concatenating BUILDENV_BASEPATH and BUILDENV_NAME.
Do you agree to modify this script in the same way as windows_buildenv.bat ? As requested above by @fwcd #13306 (comment)

I have just noticed that there is still the logic of configuring VCPKG_OVERLAY_PORTS and VCPKG_OVERLAY_TRIPLETS. are they part of the CMakeCache? If not we need to take care that they are always configured.

To my understanding these are not of any use with the prebuilt buildenv. These apply only if you link to a local working copy of mixxxdj/VCPKG where you build the dependencies locally.

@daschuer
Copy link
Member

Nikhil Bisht did some extra for us on windows and discovers some details / requirements:

  • Once the CMakeCache.txt is populated, the project can be build without running the batch.
  • It requires however the VS command prompt to find comctl32.lib
  • Significant cached variables are:
    • CMAKE_TOOLCHAIN_FILE
    • Z_VCPKG_ROOT_DIR
    • VCPKG_INSTALLED_DIR
    • VCPKG_TARGET_TRIPLET
  • All libraries path are cached absolute
  • All other script variables are temporary

He struggles to reconfigure a configured build directory.
After starting over with an empty one it works. I have not verified it, because I am in Linux but I can also remembered to suffer from these issues with the cmake cache when installing home build libraries.
So I consider updating the vcpkg environment without cleaning the CMakeCache.txt as supported. Can you confirm?

From this we can generate these requirements:

  • Download vcpkg if not found
  • Not change the vcpkg environment if already configured
  • Build without running windows_buildenv.bat if already configured
  • Fail if running without windows_buildenv.bat and not configured
  • Warn if running with windows_buildenv.bat and a different vcpkg is configured.
  • Allow to configure any previous downloaded vcpkg environment without script. Currently done via MIXXX_VCPKG_ROOT

Does that make sense?

@JoergAtGithub
Copy link
Member Author

* Once the CMakeCache.txt is populated, the project can be build without running the batch.

Yes, this is because we set CMAKE_TOOLCHAIN_FILE as CACHE_STRING:

   set(
     CMAKE_TOOLCHAIN_FILE
     "${MIXXX_VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"
     CACHE STRING
     ""
   )

Therefore adding CMAKE_TOOLCHAIN_FILE as additional condition, similar as suggested above, should work.

He struggles to reconfigure a configured build directory. After starting over with an empty one it works. I have not verified it, because I am in Linux but I can also remembered to suffer from these issues with the cmake cache when installing home build libraries. So I consider updating the vcpkg environment without cleaning the CMakeCache.txt as supported. Can you confirm?

Yes, but CMake detects these cases and tells you very clear, when you have to delete the cache first. But it doesn't do this automatic.

From this we can generate these requirements:

* Download vcpkg if not found

Sure

* Not change the vcpkg environment if already configured

If the correct one is specified yes. But the most important thing is, that a user never gets the wrong vcpkg environment unless he explicitly overides the one set in windows_buildenv.bat . Otherwise switching between branches would no longer work.

* Build without running windows_buildenv.bat if already configured

Sure, this is using only Ninja not CMake

* Fail if running without windows_buildenv.bat and not configured

Yes

* Warn if running with windows_buildenv.bat and a different vcpkg is configured.

This warning makes no sense to me, because if the user runs windows_buildenv.bat, he explicitly set the new buildenv by this action. This is the main purpose of this batch file.

* Allow to configure any previous downloaded vcpkg environment without script. Currently done via MIXXX_VCPKG_ROOT

Sure

…nv when correct CMAKE_TOOLCHAIN_FILE is already cached
@daschuer
Copy link
Member

Yes, this is because we set CMAKE_TOOLCHAIN_FILE as CACHE_STRING:

Ah, so we mirror (probably unnecessarily) the default bahviour of CMake without VCPKG. See:
https://cmake.org/cmake/help/latest/envvar/CMAKE_TOOLCHAIN_FILE.html#envvar:CMAKE_TOOLCHAIN_FILE

At least "Z_VCPKG_ROOT_DIR" is reliably used and updated without our extra adjustments.

Yes, but CMake detects these cases and tells you very clear, when you have to delete the cache first. But it doesn't do this automatic.

I Have just tested to change the CMAKE_TOOLCHAIN_FILE. I receive this unclear warning:

CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_TOOLCHAIN_FILE

When I try to use a diffrent generator I get such a waring:

CMake Error: Error: generator : Unix Makefiles
Does not match the generator used previously: Ninja
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.

But this is not our case if we update to a new environment. What did you do to get a better warning?

But the most important thing is, that a user never gets the wrong vcpkg environment unless he explicitly overides the one set in windows_buildenv.bat . Otherwise switching between branches would no longer work.

I am not sure if I get you right. Please verify. Let's asume we have updated the main branch to our new 2.6 environment:

  • After updateing to recent master the windows_buildenv.bat pount to the new environment.
  • The CMakeCache is still old.
  • Normal Rebuilding will work and still use the 2.5 environment
  • Update to 2.5 requires to delete CMakeCache.txt, run the batch and reconfigure cmake.
  • Switching to the 2.5 branch will now use the 2.6 environment listed in the CMakeCache.txt
    This would be OK for me, like the same situaten when updating the Linux distro.

I use a 2.5 and a "main" branch to avaoid to much rebuilds. This is also a good receipt to use the right environment.
Sometimes I accedentally merge a 2.6 PR into the 2.5 working copy. The penalty is a longer rebuild.
In the windows situation it will likely fail because some required 2.6 libraries are missing. Here we should fail eraly with a descriptive message.

For convenience we should always keep te 2.5 branch building with the 2.6 environment. (We need probably anyway because of Linux)

Do you agree or do I miss something?

  • Warn if running with windows_buildenv.bat and a different vcpkg is configured.

This warning makes no sense to me, because if the user runs windows_buildenv.bat, he explicitly set the new buildenv by this action. This is the main purpose of this batch file.

Ok, Then we should fail. Works for me as well.

@daschuer
Copy link
Member

It looks like with your latest changes we are already close. Thank you. I have not tested, but I think the only missng piece is the question around manual setting the environment. And outdated commenst around MIXXX_VCPKG_ROOT
Use case:

  • User has a working copy with the default vcpkg environent.
  • Wants to create a second working copy without reusing the original environment.

Thay may want to do this (or equivalent, I don't mind)

git worktree add ..\experiment main 
cd ..\experiment
mkdir build 
cd build 
cmake .. -G "Ninja" -DMIXXX_VCPKG_ROOT=..\buildenv\mixxx-deps-2.5-x64-windows-c15790e

Officially it is:

cmake ../my/project -DCMAKE_TOOLCHAIN_FILE=<vcpkg-root>/scripts/buildsystems/vcpkg.cmake

https://learn.microsoft.com/en-us/vcpkg/users/buildsystems/cmake-integration#cmake_toolchain_file

For my underanding both should skip a download if configured as well in the enviroment variables.

@daschuer
Copy link
Member

The CMAKE_TOOLCHAIN_FILE ends as
CMAKE_TOOLCHAIN_FILE:FILEPATH=/home/daniel/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake
even if it is specified relative to the source directory

Internally a similar command like this is used:

file(REAL_PATH "${CMAKE_TOOLCHAIN_FILE}" CLEAN_CMAKE_TOOLCHAIN_FILE BASE_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" EXPAND_TILDE)

Since we cannot reconfigure cmake safely, we need to fail if the BUILD_ENV variables are set but don't match.
I will propose changes inline.

@JoergAtGithub
Copy link
Member Author

No, the buildenv variables must always win. If they differ the toolchain file path is invalid.

@daschuer
Copy link
Member

If the project is configured and the BUILD_ENV, we can't safely reconfigure. That's why the only chance we have that the buildenv variables can win is to ask for deleting the cache.

@JoergAtGithub
Copy link
Member Author

Yes, that is the behavior we currently have and works very well for me. I will not change this in this PR!
Changenging the general behavior is out of scope of this PR!

Copy link
Member

@daschuer daschuer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works good on Linux as far as I can tell. I am pretty sure that clanging the environment without clearing the Cache does not work. But I can't prove it yet.

Comment on lines +51 to +54
if(WIN32)
string(REPLACE "\\" "/" BUILDENV_BASEPATH "${BUILDENV_BASEPATH}")
string(REPLACE "\\" "/" CMAKE_TOOLCHAIN_FILE "${CMAKE_TOOLCHAIN_FILE}")
endif()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mirrors the CMAKE_TOOLCHAIN_FILE path handling, but uses a new variable to not change any cached value.

Suggested change
if(WIN32)
string(REPLACE "\\" "/" BUILDENV_BASEPATH "${BUILDENV_BASEPATH}")
string(REPLACE "\\" "/" CMAKE_TOOLCHAIN_FILE "${CMAKE_TOOLCHAIN_FILE}")
endif()
file(
REAL_PATH
"${CMAKE_TOOLCHAIN_FILE}"
CLEAN_CMAKE_TOOLCHAIN_FILE
BASE_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}"
EXPAND_TILDE
)
if(DEFINED BUILDENV_NAME)
file(TO_CMAKE_PATH ${BUILDENV_BASEPATH} BUILDENV_BASEPATH)
set(MIXXX_VCPKG_ROOT "${BUILDENV_BASEPATH}/${BUILDENV_NAME}")
endif()

Comment on lines +56 to +72
if(
WIN32
AND
(
NOT DEFINED CMAKE_TOOLCHAIN_FILE
OR
(
DEFINED BUILDENV_BASEPATH
AND DEFINED BUILDENV_NAME
AND
NOT
CMAKE_TOOLCHAIN_FILE
MATCHES
"${BUILDENV_BASEPATH}/${BUILDENV_NAME}/scripts/buildsystems/vcpkg.cmake"
)
)
)
Copy link
Member

@daschuer daschuer Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if(
WIN32
AND
(
NOT DEFINED CMAKE_TOOLCHAIN_FILE
OR
(
DEFINED BUILDENV_BASEPATH
AND DEFINED BUILDENV_NAME
AND
NOT
CMAKE_TOOLCHAIN_FILE
MATCHES
"${BUILDENV_BASEPATH}/${BUILDENV_NAME}/scripts/buildsystems/vcpkg.cmake"
)
)
)
if(
DEFINED CMAKE_TOOLCHAIN_FILE
AND DEFINED MIXXX_VCPKG_ROOT
AND
NOT
CLEAN_CMAKE_TOOLCHAIN_FILE
MATCHES
"${MIXXX_VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"
)
message(
FATAL_ERROR
"Build environment: '${MIXXX_VCPKG_ROOT}' does not match the environment used previously: '${CMAKE_TOOLCHAIN_FILE}'. Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory."
)
endif()
if(WIN32 AND NOT DEFINED CMAKE_TOOLCHAIN_FILE)

@daschuer
Copy link
Member

Here is a CI run that demonstrates the issue
https://github.com/daschuer/mixxx/actions/runs/13958279397/job/39074483674
It configures the build directory as usual. Than It move the vcpkg package directory and re-configures the project with the new location and fails.

It will work, if the original environment still exists and you download the new in addition, However this leads to a dubious situation where some parts of the old and some parts of the new environment are used. I consider this as maximal confusing and therefore the early error message with a clear instruction is helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build developer experience Issues, bugs and PRs related to the development process, development environment & developer docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants