Using shared libraries

Python projects may build shared libraries as part of their project, or link with shared libraries from a dependency. This tends to be a common source of issues, hence this page aims to explain how to include shared libraries in wheels, any limitations and gotchas, and how support is implemented in meson-python under the hood.

We distinguish between internal shared libraries that are built as part of the project, and external shared libraries that are provided by project dependencies and that are linked with the project build artifacts. For internal shared libraries, we also distinguish whether the shared library is being installed to its default system location (typically /usr/local/lib on Unix-like systems, and C:\\lib on Windows - we call this libdir in this guide) or to a location in site-packages within the Python package install tree. All these scenarios are (or will be) supported, with some caveats:

shared library source

install location

Windows

macOS

Linux

internal

libdir

✓ 1

✓

✓

internal

site-packages

✓

✓

✓

external

—

✓ 2

✓

✓

  1. Support for internal shared libraries on Windows is enabled with the tool.meson-python.allow-windows-internal-shared-libs option and requires cooperation from the package to extend the DLL search path or preload the required libraries. See below for more details.

  2. External shared libraries require delvewheel usage on Windows (or some equivalent way, like amending the DLL search path to include the directory in which the external shared library is located). Due to the lack of RPATH support on Windows, there is no good way around this.

Internal shared libraries

A shared library produced by library() or shared_library() built like this

example_lib = shared_library(
    'example',
    'examplelib.c',
    install: true,
)

is installed to libdir by default. If the only reason the shared library exists is to be used inside the Python package being built, then it is best to modify the install location, via the install_dir argument, to be within the Python package itself:

example_lib = shared_library(
    'example',
    'examplelib.c',
    install: true,
    install_dir: py.get_install_dir() / 'mypkg/subdir',
)

Then an extension module in the same install directory can link against the shared library in a portable manner by using install_rpath:

py.extension_module('_extmodule',
    '_extmodule.c',
    link_with: example_lib,
    install: true,
    subdir: 'mypkg/subdir',
    install_rpath: '$ORIGIN'
)

The above method will work as advertised on macOS and Linux; meson-python does nothing special for this case. Windows needs some special handling though, due to the lack of RPATH support:

def _append_to_sharedlib_load_path():
    """Ensure the shared libraries in this package can be loaded on Windows.

    Windows lacks a concept equivalent to RPATH: Python extension modules
    cannot find DLLs installed outside the DLL search path. This function
    ensures that the location of the shared libraries distributed inside this
    Python package is in the DLL search path of the process.

    The Windows DLL search path includes the path to the object attempting
    to load the DLL: it needs to be augmented only when the Python extension
    modules and the DLLs they require are installed in separate directories.
    Cygwin does not have the same default library search path: all locations
    where the shared libraries are installed need to be added to the search
    path.

    This function is very similar to the snippet inserted into the main
    ``__init__.py`` of a package by ``delvewheel`` when it vendors external
    shared libraries.

    .. note::

        `os.add_dll_directory` is only available for Python 3.8 and later, and
        in the Conda ``python`` packages it works as advertised only for
        version 3.10 and later. For older Python versions, pre-loading the DLLs
        with `ctypes.WinDLL` may be preferred.
    """
    basedir = os.path.dirname(__file__)
    subdir = os.path.join(basedir, 'sub')
    if os.name == 'nt':
        os.add_dll_directory(subdir)
    elif sys.platform == 'cygwin':
        os.environ['PATH'] = os.pathsep.join((os.environ['PATH'], basedir, subdir))


_append_to_sharedlib_load_path()

If an internal shared library is not only used as part of a Python package, but for example also as a regular shared library then the method shown above won’t work. The library is then marked for installation into the system default libdir location. Actually installing into libdir isn’t possible with wheels, hence meson-python will instead do the following:

  1. Install the shared library to the .<project-name>.mesonpy.libs top-level directory in the wheel, which on install will end up in site-packages.

  2. On platforms other than Windows, rewrite RPATH entries for install targets that depend on the shared library to point to that new install location instead.

On platforms other than Windows, this will make the shared library work automatically, with no other action needed from the package author. On Windows, due to the lack of RPATH support, the .<project-name>.mesonpy.libs location search path needs to be added to the DLL search path, with code similar to the one presented above. For this reason, handling of internal shared libraries on Windows is conditional to setting the tool.meson-python.allow-windows-internal-shared-libs option.

External shared libraries

External shared libraries are installed somewhere on the build machine, and usually detected by a dependency() or compiler.find_library() call in a meson.build file. When a Python extension module or executable uses the dependency, the shared library will be linked against at build time.

If the shared library is located in a directory on the loader search path, the wheel created by meson-python will work locally when installed. If it’s in a non-standard location however, the shared library will go missing at runtime. The Python extension module linked against it needs an RPATH entry - and Meson will not automatically manage RPATH entries for you. Hence you’ll need to add the needed RPATH yourself, for example by adding -Wl,rpath=/path/to/dir/sharedlib/is/in to LDFLAGS before starting the build. In case you run into this problem after a wheel is built and installed, adding that same path to the LD_LIBRARY_PATH environment variable is a quick way of checking if that is indeed the problem.

On Windows, the solution is similar - the shared library can either be preloaded, or the directory that the library is located in added to the DLL search path with os.add_dll_directory, or vendored into the wheel with delvewheel in order to make the built Python package usable locally.

Publishing wheels which depend on external shared libraries

On all platforms, wheels which depend on external shared libraries usually need post-processing to make them usable on machines other than the one on which they were built. This is because the RPATH entry for an external shared library contains a path specific to the build machine. This post-processing is done by tools like auditwheel (Linux), delvewheel (Windows), delocate (macOS) or repair-wheel (any platform, wraps the other tools).

Running any of those tools on a wheel produced by meson-python will vendor the external shared library into the wheel and rewrite the RPATH entries (it may also do some other things, like symbol mangling).

On Windows, the package author may also have to add the preloading like shown above with _append_to_sharedlib_load_path() to the main __init__.py of the package, delvewheel may or may not take care of this (please check its documentation if your shared library goes missing at runtime).

Note that we don’t cover using shared libraries contained in another wheel and depending on such a wheel at runtime in this guide. This is inherently complex and not recommended (you need to be in control of both packages, or upgrades may be impossible/breaking).

Using libraries from a Meson subproject

It can often be useful to build a shared library in a Meson subproject, for example as a fallback in case an external dependency isn’t detected. There are two main strategies for folding a library built in a subproject into a wheel built with meson-python:

  1. Build the library as a static library instead of a shared library, and link it into a Python extension module that needs it.

  2. Build the library as a shared library, and either change its install path to be within the Python package’s tree, or rely on meson-python to fold it into the wheel when it’d otherwise be installed to libdir.

Static linking tends to be easier, and it is the recommended solution, unless the library of interest cannot be built as a static library or it would inflate the wheel size too much because it’s needed by multiple Python extension modules.

Static library from subproject

The major advantage of building a library target as static and folding it directly into an extension module is that the RPATH or the DLL search path do not need to be adjusted and no targets from the subproject need to be installed. To ensures that library() targets are built as static, and that no parts of the subprojects are installed, the following configuration can be added in pyproject.toml to ensure the relevant options are passed to Meson:

[tool.meson-python.args]
setup = ['--default-library=static']
install = ['--skip-subprojects']

To then link against the static library in the subproject, say for a subproject named bar with the main library target contained in a bar_dep dependency, add this to your meson.build file:

bar_proj = subproject('bar')
bar_dep = bar_proj.get_variable('bar_dep')

py.extension_module(
    '_example',
    '_examplemod.c',
    dependencies: bar_dep,
    install: true,
)

Shared library from subproject

Sometimes it may be necessary or preferable to use dynamic linking to a shared library provided in a subproject, for example to avoid inflating the wheel size having multiple copies of the same object code in different extension modules using the same library. In this case, the subproject needs to install the shared library in the usual location in libdir. meson-python will automatically include it into the wheel in .<project-name>.mesonpy.libs just like an internal shared library.

Most projects, however, install more than the shared library and the extra components, such as header files or documentation, should not be included in the Python wheel. Projects may have configuration options to disable building and installing additional components, in this case, these options can be passed to the subproject() call:

foo_subproj = subproject('foo',
    default_options: {
        'docs': 'disabled',
    })
foo_dep = foo_subproj.get_variable('foo_dep')

Install tags do not work per subproject, therefore to exclude other parts of the subproject from being included in the wheel, we need to resort to meson-python install location filters using the tool.meson-python.wheel.exclude build option:

[tool.meson-python.wheel]
exclude = ['{prefix}/include/*']