Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only install binary packages on the second install pass. #1492

Merged
merged 8 commits into from
Oct 13, 2023
1 change: 1 addition & 0 deletions changes/1482.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
macOS apps can now be configured to produce single platform binaries, or binaries that will work on both x86_64 and ARM64.
1 change: 1 addition & 0 deletions changes/1492.misc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The second binary wheel installation pass (for the other architecture) on macOS now requires binary wheels, disabling source compilation.
15 changes: 15 additions & 0 deletions docs/reference/platforms/macOS/app.rst
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,21 @@ Do not submit the application for notarization. By default, apps will be
submitted for notarization unless they have been signed with an ad-hoc
signing identity.

Application configuration
=========================

The following options can be added to the ``tool.briefcase.app.<appname>.macOS.app``
section of your ``pyproject.toml`` file.

``universal_build``
~~~~~~~~~~~~~~~~~~~

A Boolean, indicating whether Briefcase should build a universal app (i.e, an app that
can target both x86_64 and ARM64). Defaults to ``true``; if ``false``, the binary will
only be executable on the host platform on which it was built - i.e., if you build on
an x86_64 machine, you will produce an x86_65 binary; if you build on an ARM64 machine,
you will produce an ARM64 binary.

Platform quirks
===============

Expand Down
15 changes: 15 additions & 0 deletions docs/reference/platforms/macOS/xcode.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,21 @@ Do not submit the application for notarization. By default, apps will be
submitted for notarization unless they have been signed with an ad-hoc
signing identity.

Application configuration
=========================

The following options can be added to the ``tool.briefcase.app.<appname>.macOS.Xcode``
section of your ``pyproject.toml`` file.

``universal_build``
~~~~~~~~~~~~~~~~~~~

A Boolean, indicating whether Briefcase should build a universal app (i.e, an app that
can target both x86_64 and ARM64). Defaults to ``true``; if ``false``, the binary will
only be executable on the host platform on which it was built - i.e., if you build on
an x86_64 machine, you will produce an x86_65 binary; if you build on an ARM64 machine,
you will produce an ARM64 binary.

Platform quirks
===============

Expand Down
31 changes: 15 additions & 16 deletions src/briefcase/commands/create.py
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,8 @@ def generate_app_template(self, app: AppConfig):
# Properties of the generating environment
# The full Python version string, including minor and dev/a/b/c suffixes (e.g., 3.11.0rc2)
"python_version": platform.python_version(),
# The host architecture
"host_arch": self.tools.host_arch,
# The Briefcase version
"briefcase_version": briefcase.__version__,
# Transformations of explicit properties into useful forms
Expand Down Expand Up @@ -445,19 +447,23 @@ def _extra_pip_args(self, app: AppConfig):
def _pip_install(
self,
app: AppConfig,
requires: list[str],
app_packages_path: Path,
include_deps: bool = True,
**pip_kwargs: dict[str, str],
pip_args: list[str],
install_hint: str = "",
**pip_kwargs,
):
"""Invoke pip to install a set of requirements.

:param app: The app configuration
:param requires: The list of requirements to install
:param app_packages_path: The full path of the app_packages folder into which
requirements should be installed.
:param progress_message: The waitbar progress message to display to the user.
:param pip_kwargs: Any additional keyword arguments to pass to the subprocess
:param pip_args: The list of arguments (including the list of requirements to
install) to pass to pip. This is in addition to the default arguments that
disable pip version checks, forces upgrades, and installs into the nominated
``app_packages`` path.
:param install_hint: Additional hint information to provide in the exception
message if the pip install call fails.
:param pip_kwargs: Any additional keyword arguments to pass to ``subprocess.run``
when invoking pip.
"""
try:
Expand All @@ -476,21 +482,14 @@ def _pip_install(
"--no-user",
f"--target={app_packages_path}",
]
+ (
[
"--no-deps",
]
if not include_deps
else []
)
+ self._extra_pip_args(app)
+ requires,
+ pip_args,
check=True,
encoding="UTF-8",
**pip_kwargs,
)
except subprocess.CalledProcessError as e:
raise RequirementsInstallError() from e
raise RequirementsInstallError(install_hint=install_hint) from e

def _install_app_requirements(
self,
Expand Down Expand Up @@ -520,8 +519,8 @@ def _install_app_requirements(
with self.input.wait_bar(progress_message):
self._pip_install(
app,
requires=self._pip_requires(app, requires),
app_packages_path=app_packages_path,
pip_args=self._pip_requires(app, requires),
**(pip_kwargs if pip_kwargs else {}),
)
else:
Expand Down
6 changes: 3 additions & 3 deletions src/briefcase/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,12 @@ def __init__(self, python_version_tag, platform, host_arch, is_32bit):


class RequirementsInstallError(BriefcaseCommandError):
def __init__(self):
def __init__(self, install_hint=""):
super().__init__(
"""\
f"""\
Unable to install requirements. This may be because one of your
requirements is invalid, or because pip was unable to connect
to the PyPI server.
to the PyPI server.{install_hint}
"""
)

Expand Down
163 changes: 96 additions & 67 deletions src/briefcase/platforms/macOS/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,77 +46,106 @@ def _install_app_requirements(
requires: list[str],
app_packages_path: Path,
):
# Perform the initial install targeting the current platform
host_app_packages_path = (
self.bundle_path(app) / f"app_packages.{self.tools.host_arch}"
)
super()._install_app_requirements(
app,
requires=requires,
app_packages_path=host_app_packages_path,
)

# Find all the packages with binary components.
# We can ignore any -universal2 packages; they're already fat.
binary_packages = self.find_binary_packages(
host_app_packages_path,
universal_suffix="_universal2",
)
if getattr(app, "universal_build", True):
# Perform the initial install targeting the current platform
host_app_packages_path = (
self.bundle_path(app) / f"app_packages.{self.tools.host_arch}"
)
super()._install_app_requirements(
app,
requires=requires,
app_packages_path=host_app_packages_path,
)

# Now install dependencies for the architecture that isn't the host architecture.
other_arch = {
"arm64": "x86_64",
"x86_64": "arm64",
}[self.tools.host_arch]
# Find all the packages with binary components.
# We can ignore any -universal2 packages; they're already fat.
binary_packages = self.find_binary_packages(
host_app_packages_path,
universal_suffix="_universal2",
)

# Create a temporary folder targeting the other platform
other_app_packages_path = self.bundle_path(app) / f"app_packages.{other_arch}"
if other_app_packages_path.is_dir():
self.tools.shutil.rmtree(other_app_packages_path)
self.tools.os.mkdir(other_app_packages_path)
# Now install dependencies for the architecture that isn't the host architecture.
other_arch = {
"arm64": "x86_64",
"x86_64": "arm64",
}[self.tools.host_arch]

if binary_packages:
with self.input.wait_bar(
f"Installing binary app requirements for {other_arch}..."
):
self._pip_install(
app,
requires=[
f"{package}=={version}" for package, version in binary_packages
],
app_packages_path=other_app_packages_path,
include_deps=False,
env={
"PYTHONPATH": str(
self.support_path(app)
/ "platform-site"
/ f"macosx.{other_arch}"
)
},
)
# Create a temporary folder targeting the other platform
other_app_packages_path = (
self.bundle_path(app) / f"app_packages.{other_arch}"
)
if other_app_packages_path.is_dir():
self.tools.shutil.rmtree(other_app_packages_path)
self.tools.os.mkdir(other_app_packages_path)

if binary_packages:
with self.input.wait_bar(
f"Installing binary app requirements for {other_arch}..."
):
self._pip_install(
app,
app_packages_path=other_app_packages_path,
pip_args=[
"--no-deps",
"--only-binary",
":all:",
]
+ [
f"{package}=={version}"
for package, version in binary_packages
],
install_hint=f"""

If an {other_arch} wheel has not been published for one or more of your requirements,
you must compile those wheels yourself, or build a non-universal app by setting:

universal_build = False

in the macOS configuration section of your pyproject.toml.
""",
env={
"PYTHONPATH": str(
self.support_path(app)
/ "platform-site"
/ f"macosx.{other_arch}"
)
},
)
else:
self.logger.info("All packages are pure Python, or universal.")

# If given the option of a single architecture binary or a universal2 binary,
# pip will install the single platform binary. However, a common situation on
# macOS is for there to be an x86_64 binary and a universal2 binary. This means
# you only get a universal2 binary in the "other" install pass. This then causes
# problems with merging, because the "other" binary contains a copy of the
# architecture that the "host" platform provides.
#
# To avoid this - ensure that the libraries in the app packages for the "other"
# arch are all thin.
#
# This doesn't matter if it happens the other way around - if the "host" arch
# installs a universal binary, then the "other" arch won't be asked to install
# a binary at all.
self.thin_app_packages(other_app_packages_path, arch=other_arch)

# Merge the binaries
self.merge_app_packages(
target_app_packages=app_packages_path,
sources=[host_app_packages_path, other_app_packages_path],
)
else:
self.logger.info("All packages are pure Python, or universal.")

# If given the option of a single architecture binary or a universal2 binary,
# pip will install the single platform binary. However, a common situation on
# macOS is for there to be an x86_64 binary and a universal2 binary. This means
# you only get a universal2 binary in the "other" install pass. This then causes
# problems with merging, because the "other" binary contains a copy of the
# architecture that the "host" platform provides.
#
# To avoid this - ensure that the libraries in the app packages for the "other"
# arch are all thin.
#
# This doesn't matter if it happens the other way around - if the "host" arch
# installs a universal binary, then the "other" arch won't be asked to install
# a binary at all.
self.thin_app_packages(other_app_packages_path, arch=other_arch)

# Merge the binaries
self.merge_app_packages(
target_app_packages=app_packages_path,
sources=[host_app_packages_path, other_app_packages_path],
)
# If we're not building a universal binary, we can do a single install pass
# directly into the app_packages folder.
super()._install_app_requirements(
app,
requires=requires,
app_packages_path=app_packages_path,
)

# Since we're only targeting 1 architecture, we can strip any universal
# libraries down to just the host architecture.
self.thin_app_packages(app_packages_path, arch=self.tools.host_arch)


class macOSRunMixin:
Expand Down
9 changes: 9 additions & 0 deletions src/briefcase/platforms/macOS/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,15 @@ def install_app_support_package(self, app: AppConfig):
runtime_support_path / "python-stdlib",
)

if not getattr(app, "universal_build", True):
with self.input.wait_bar("Ensuring stub binary is thin..."):
# The stub binary is universal by default. If we're building a non-universal app,
# we can strip the binary to remove the unused slice.
self.ensure_thin_binary(
self.binary_path(app) / "Contents" / "MacOS" / app.formal_name,
arch=self.tools.host_arch,
)


class macOSAppUpdateCommand(macOSAppCreateCommand, UpdateCommand):
description = "Update an existing macOS app."
Expand Down
8 changes: 4 additions & 4 deletions src/briefcase/platforms/macOS/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ def find_binary_packages(

return binary_packages

def ensure_thin_dylib(self, path: Path, arch: str):
"""Ensure that a library is thin, targeting a given architecture.
def ensure_thin_binary(self, path: Path, arch: str):
"""Ensure that a binary is thin, targeting a given architecture.

If the library is already thin, it is left as-is.

Expand Down Expand Up @@ -107,7 +107,7 @@ def ensure_thin_dylib(self, path: Path, arch: str):
)
except subprocess.CalledProcessError as e:
raise BriefcaseCommandError(
f"Unable to create thin library from {path}"
f"Unable to create thin binary from {path}"
) from e
else:
# Having extracted the single architecture into a temporary
Expand Down Expand Up @@ -182,7 +182,7 @@ def thin_app_packages(
futures = []
for path in dylibs:
future = executor.submit(
self.ensure_thin_dylib,
self.ensure_thin_binary,
path=path,
arch=arch,
)
Expand Down
1 change: 1 addition & 0 deletions tests/commands/create/test_generate_app_template.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ def full_context():
"document_types": {},
# Properties of the generating environment
"python_version": platform.python_version(),
"host_arch": "gothic",
"briefcase_version": briefcase.__version__,
# Fields generated from other properties
"module_name": "my_app",
Expand Down
Loading