GH-36411: [Python] Use scikit-build-core as build backend for PyArrow and get rid of setup.py

[raulcd]

Rationale for this change

Move our PyArrow build backend from setuptools and a custom setup.py to scikit-build-core which is just build backend for CMake related projects.

What changes are included in this PR?

Move from setuptools to scikit-build-core and remove PyArrow setup.py. Update some of the build requirements and minor fixes.
A custom build backend has been also been created in order to wrap scikit-build-core in order to fix problems on License files for monorepos.
pyproject.toml metadata validation expects license files to exist before exercising the build backend that's why we create symlinks. Our thin build backend will just make those symlinks hard-links in order for license and notice files to contain the contents and be added as part of the sdist.

Remove flags that are not used anymore (were only part of setup.py) and documented and validated how the same flags have to be used now.

Are these changes tested?

Yes all Python CI tests, wheels and sdist are successful.

Are there any user-facing changes?

Yes, users building PyArrow will now require the new build dependencies to exercise the build and depending on the flags used they might require to use the new documented way of using those flags.

• GitHub Issue: [Python] Improve the pyarrow build system #36411

[raulcd]

@github-actions crossbow submit -g python

[raulcd]

@github-actions crossbow submit wheel-*-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 166ff63

Submitted crossbow builds: ursacomputing/crossbow @ actions-f1bbdf4eaa

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 36fefd4

Submitted crossbow builds: ursacomputing/crossbow @ actions-bd811d95ea

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit -g python -g wheel

[raulcd]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: c518c90

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea249e92ce

Task	Status
python-sdist

[raulcd]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: 131e2c0

Submitted crossbow builds: ursacomputing/crossbow @ actions-1c58e78aff

Task	Status
python-sdist

[raulcd]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: 30e04be

Submitted crossbow builds: ursacomputing/crossbow @ actions-3179f97956

Task	Status
python-sdist

[raulcd]

@github-actions crossbow submit -g python -g wheel

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 5606749

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c4293e220

Task	Status
wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 685fd06

Submitted crossbow builds: ursacomputing/crossbow @ actions-405c7c8fd5

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

EDIT: even if I manually specify eg -C cmake.build-type=RelWithDebInfo in the pip install command, I still see that warning. And I see that this warning also happens in our CI (eg here)

Thanks @jorisvandenbossche ! Let me investigate where the log is coming from. I can see on the job you pointed we are using: -C cmake.build-type=Debug . and even though the log appears scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_BUILD_TYPE=Release it is using the correct build type Build Type: Debug. Let me see why the warning log from scikit_build_core is appearing on that one.

[raulcd]

@jorisvandenbossche this seems to happen only on conda environments and is due to scikit-build-core ignoring the default CMAKE_ARGS that conda sets (see -DCMAKE_BUILD_TYPE=Release there), this is directly from the container:

$ PANDAS=nightly NUMPY=nightly PYTHON=3.13 docker compose run -it --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=23.0.0.dev328 conda-python-pandas /bin/bash
root@6ac69eff4ef5:/# echo $CMAKE_ARGS
-DCMAKE_AR=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release
root@6ac69eff4ef5:/# 

I am unsure why conda defines a default -DCMAKE_BUILD_TYPE=Release on CMAKE_ARGS, maybe @h-vetinari can help here.
In any case for our specific case, this is ok as we are just defining it ourselves on the pyproject.toml or via -C cmake.build-type=Debug and working as expected.

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: d5a61ab

Submitted crossbow builds: ursacomputing/crossbow @ actions-c69c0863fa

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 9c6cf8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-b68682bb32

Task	Status
wheel-windows-cp313-cp313-amd64

[jorisvandenbossche]

Suggested change

	On PyArrow 24.0.0 we migrated our Python build backend from setuptools to
	scikit-build-core, which is a CMake-based build system. Previous versions used
	``PYARROW_CMAKE_OPTIONS`` environment variable to pass CMake options in the form of
	``-D<OPTION>=<VALUE>``. This is no longer supported.
	Instead, use the ``-C cmake.args=-D<OPTION>=<VALUE>`` option as described above.
	On PyArrow 24.0.0 we migrated our Python build backend from setuptools to
	scikit-build-core, which is a CMake-based build system. Previous versions used
	``PYARROW_BUILD_TYPE`` and ``PYARROW_CMAKE_OPTIONS`` environment variables
	to customize the CMake invocation. This is no longer supported.
	Instead, use the ``-C cmake.build-type=<build_type>`` and ``-C cmake.args=-D<OPTION>=<VALUE>`` option as described above.

Small suggestion to include PYARROW_BUILD_TYPE as well (for my local setup, that is the main one I have to update)