Splayed layout support

[jakirkham]

Currently cuda-python relies on all binaries (like nvcc), all headers, and all libraries to live in a single directory (specified by $CUDA_HOME or similar).

However there are use cases (like cross-compilation, as with conda-build) where the build tools may live in one location (and perform builds on that architecture) whereas the headers and libraries may live in a different location (and target a different architecture). In this case not everything lives in $CUDA_HOME.

It would be helpful to have a way of specifying where these different components come from. Here are some options:

1. Check $NVCC for the nvcc location
2. Use $CUDA_BIN (if specified) to get build tool directory
3. Support a list of directories in $CUDA_HOME
4. ?

Maybe there are other reasonable options worth considering?

[robertmaynard]

Another option is to support $CUDA_HOME being a list of directories. So for header searching cuda-python would append include to each entry in $CUDA_HOME.

[jakirkham]

Thanks Rob! 🙏

Converted the ideas to an enumerated list and added that idea to the list

[jakirkham]

cc @bdice

[vyasr]

Would it make sense to standardize a tool that uses the path to nvcc to parse information out of nvcc.profile? Something that could be used by anyone, not just cuda-python?

[leofang]

Sorry for lack of response. Is this still an issue? It seems we have been able to build cuda-python on conda-forge?

[jakirkham]

Yep it is still an issue

We workaround it in conda-forge

Through improvements on the packaging side, we have minimized the workarounds needed ( for example: conda-forge/cuda-python-feedstock#75 ), but we are unable to eliminate them without this fix

[leofang]

@jakirkham I am sorry but I don't get it. In any case we need a way to specify where the CUDA headers are, and setting CUDA_HOME is the right solution that's been working fine. Are there anything specific to the splayed layout that we have not addressed (either in CUDA Python or in conda-forge)?

[jakirkham]

Think an ideal solution (from my perspective), would be moving to a CMake-based build system here as CMake already understands how to work with splayed layouts correctly. For example moving to scikit-build-core would very likely solve this issue. This may be desirable anyways to simplify the build process and make it less bespoke

[leofang]

I still don't understand what specific changes are needed to support splayed layout. @jakirkham this issue might have been outdated since the earlier (CTK 12.0) efforts?

Right now, all headers are expected by CUDA Python to locate in one place, for the purposes of both parsing and compiling. So, it doesn't really matter if headers are in $PREFIX/include or $PREFIX/targets/<platform>/include or /usr/local/cuda/include or any place in the host environment, as long as $CUDA_HOME is set to take a single directory path. Can we please identify which part of CUDA Python is supposed to find headers locating in two or more places (which we'd fail to do so)? I've eyeballed the recent cuda-python recipe and I was unable to find such traces anymore, most likely they were removed starting conda-forge/cuda-python-feedstock#58 and further cleaned up later.

Also, there is no C++ components in this project (at least not yet; there will be once we start working on memory management but it would not happen before this Fall the earliest) and we need to balance with the currently limited engineering resource we have. I don't see the immediate benefit of moving away from setuptools to CMake (+ something that understands CMake like scikit-build-core) when things are already working. That said, if I can enlist a RAPIDS build expert to help with the build system migration, I would not object the change to CMake 😉