Does not work with CUDA from micromamba environment

Oct 13, 2025 by talynt.org 97 views

Does not work with CUDA from micromamba environment

In this article, we will explore the issue of using CUDA with the micromamba environment. Specifically, we will discuss the problem of cumm failing at runtime when installed in a micromamba environment. We will also provide a temporary workaround to resolve this issue.

To reproduce the issue, follow these steps:

Set up a new environment using micromamba:

micromamba create -n testenv -c conda-forge python=3.11
micromamba activate testenv

Install CUDA in the environment:

micromamba install cuda -c nvidia/label/cuda-12.1.1

Install spconv using pip:

pip install spconv-cu121

Run some code that relies on cumm and spconv:

# run some code relying on cumm/spconv...

When running the code, you will encounter the following error:

│ /home/luz/.local/opt/micromamba/envs/testenv/lib/python3.11/site-packages/cumm/tensorview/__init__.py:323 in         │
│ __init__                                                                                                             │
│                                                                                                                      │
│   320 │   │   if name_exprs is None:                                                                                 │
│   321 │   │   │   name_exprs = []                                                                                    │
│   322 │   │   if isinstance(code, str):                                                                              │
│ ❱ 323 │   │   │   self._mod = _NVRTCModule(code, headers, opts, program_name,                                        │
│   324 │   │   │   │   │   │   │   │   │    name_exprs, cudadevrt_path)                                               │
│   325 │   │   else:                                                                                                  │
│   326 │   │   │   self._mod = _NVRTCModule(code, cudadevrt_path)                                                     │
│                                                                                                                      │
│ ╭───────────────────────────────────────────────────── locals ─────────────────────────────────────────────────────╮ │
│ │           code = '#include <cumm/common/TensorViewNVRTC.h>\n#include <cumm/common/TensorViewNVRTCKe'+2175        │ │
│ │ cudadevrt_path = ''                                                                                              │ │
│ │        headers = {                                                                                               │ │
│ │                  │   'cumm/common/TensorViewNVRTC.h': '#pragma once\n#include <tensorview/core/all.h>\n#include  │ │
│ │                  <tensorview/core/arrayops'+230,                                                                 │ │
│ │                  │   'cumm/common/TensorViewNVRTCKernel.h': '#pragma once\n#include                              │ │
│ │                  <tensorview/cuda/device_ops.h>\n#include <tensorview/gemm/d'+215,                               │ │
│ │                  │   'cumm/gemm/layout/RowMajor.h': '#pragma once\n#include                                      │ │
│ │                  <cumm/common/TensorViewNVRTC.h>\nnamespace cumm {\nnamespace'+845,                              │ │
│ │                  │   'cumm/gemm/layout/ColumnMajor.h': '#pragma once\n#include                                   │ │
│ │                  <cumm/common/TensorViewNVRTC.h>\nnamespace cumm {\nnamespace'+857,                              │ │
│ │                  │   'cumm/common/GemmBasic.h': '#pragma once\n#include                                          │ │
│ │                  <tensorview/core/nvrtc/runtime_include.h>\n#include <tensor'+254,                               │ │
│ │                  │   'cumm/common/GemmBasicKernel.h': '#pragma once\n#include                                    │ │
│ │                  <tensorview/gemm/arch/memory.h>\n#include <tensorview/gemm/'+245,                               │ │
│ │                  │   'cumm/gemm/utils/GemmUtilsCPU.h': '#pragma once\n#include                                   │ │
│ │                  <cumm/common/TensorViewNVRTC.h>\nnamespace cumm {\nnamespace'+851,                              │ │
│ │                  │   'spconv/gemmutils/GemmUtils.h': '#pragma once\n#include                                     │ │
│ │                  <cumm/common/TensorViewNVRTC.h>\nnamespace spconv {\nnamespa'+985,                              │ │
│ │                  │   'spconv/inpitera/maskiter/PitchLinear.h': '#pragma once\n#include                           │ │
│ │                  <cumm/common/TensorViewNVRTC.h>\nnamespace spconv {\nnamespa'+352,                              │ │
│ │                  │   'spconv/inpitera/maskiter/MaskTileIteratorParams.h': '#pragma once\nnamespace spconv        │ │
│ │                  {\nnamespace inpitera {\nnamespace maskiter {\nstruct'+613,                                     │ │
│ │                  │   ... +36                                                                                     │ │
│ │                  }                                                                                               │ │
│ │     name_exprs = ['spconv::gemm_kernel', 'spconv::nvrtc_kernel_cpu_out', '&spconv::params_raw']                  │ │
│ │   name_to_meta = {                                                                                               │ │
│ │                  │   'spconv::gemm_kernel': NVRTCKernelMeta[name=gemm_kernel,ns=spconv,args=[]],                 │ │
│ │                  │   'spconv::nvrtc_kernel_cpu_out':                                                             │ │
│ │                  NVRTCKernelMeta[name=nvrtc_kernel_cpu_out,ns=spconv,args=[NVRTCArgMeta(base_type=<NVRTCArgBase… │ │
│ │                  0>, valid=True, simple_type=-1, shape=[1], is_simple_ptr=False, is_scalar=True, count=None),    │ │
│ │                  NVRTCArgMeta(base_type=<NVRTCArgBaseType.Scalar: 0>, valid=True, simple_type=-1, shape=[1],     │ │
│ │                  is_simple_ptr=False, is_scalar=True, count=None)]]                                              │ │
│ │                  }                                                                                               │ │
│ │           opts = [                                                                                               │ │
│ │                  │   '-std=c++14',                                                                               │ │
│ │                  │   '--gpu-architecture=sm_61',                                                                 │ │
│ │                  │   '-DTV_ENABLE_HARDWARE_ACC',                                                                 │ │
│ │                  │   '-I',                                                                                       │ │
│ │                  │   '/usr/local/cuda/include',                                                                  │ │
│ │                  │   '-I',                                                                                       │ │
│ │                  │   '/home/luz/.local/opt/micromamba/envs/testenv/lib/python3.11/site-packages/cumm/i'+6        │ │
│ │                  ]                                                                                               │ │
│ │   program_name = 'kernel.cu'                                                                                     │ │
│ │           self = <cumm.nvrtc.CummNVRTCModule object at 0x7fab1c39ec50>                                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

RuntimeError: /io/include/tensorview/cuda/nvrtc.h(96)
compileResult == NVRTC_SUCCESS assert faild. nvrtc compile failed. log:
 /home/luz/.local/opt/micromamba/envs/testenv/lib/python3.11/site-packages/cumm/include/tensorview/core/nvrtc/limits.h(689): warning #61-D: integer operation result is out of range
      return -INT_MIN - 1;
             ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/home/luz/.local/opt/micromamba/envs/testenv/lib/python3.11/site-packages/cumm/include/tensorview/core/nvrtc/limits.h(689): warning #61-D: integer operation result is out of range
      return -INT_MIN - 1;
                      ^

/home/luz/.local/opt/micromamba/envs/testenv/lib/python3.11/site-packages/cumm/include/tensorview/gemm/dtypes/float8.h(53): catastrophic error: cannot open source file "cuda_fp8.h"
  #include <cuda_fp8.h>
                       ^

1 catastrophic error detected in the compilation of "kernel.cu.cu".
Compilation terminated.

The issue lies in the way cumm locates the include and lib directories. In the environment, the includes are directly at $env/include, including $env/include/cuda.h. However, the code snippet above tries to locate the include and lib directories using the nvcc path, which is not correct in this case.

To fix this issue, we need to adapt the code snippet above to also include Path(nvcc_path).parent.parent / "include" as a valid option.

As a temporary workaround, we can create a symlink from $env/targets/x86_64-linux/include to $env/include.

mkdir -p $env/targets/x86_64-linux
cd $env/targets/x86_64-linux
ln -s ../../include .

This will allow cumm
Q&A: Does not work with CUDA from micromamba environment

A: The issue lies in the way cumm locates the include and lib directories. In the environment, the includes are directly at $env/include, including $env/include/cuda.h. However, the code snippet above tries to locate the include and lib directories using the nvcc path, which is not correct in this case.

A: The error is caused by the fact that cumm is trying to locate the include and lib directories using the nvcc path, which is not correct in this case. The correct path is $env/include, which includes $env/include/cuda.h.

A: To fix this issue, you need to adapt the code snippet above to also include Path(nvcc_path).parent.parent / "include" as a valid option.

A: The temporary workaround is to create a symlink from $env/targets/x86_64-linux/include to $env/include.

mkdir -p $env/targets/x86_64-linux
cd $env/targets/x86_64-linux
ln -s ../../include .

A: The temporary workaround is necessary because the issue is caused by the fact that cumm is trying to locate the include and lib directories using the nvcc path, which is not correct in this case. The temporary workaround allows cumm to locate the include and lib directories correctly.

A: To prevent this issue from happening in the future, you can modify the code snippet above to always use the correct path, which is $env/include. This will ensure that cumm can locate the include and lib directories correctly.

A: The implications of this issue are that cumm will not work correctly in a micromamba environment. This can cause problems for users who rely on cumm for their work.

A: If you encounter this issue, you can try the following:

Check the cumm documentation to see if there are any known issues or workarounds.
Search online for solutions to this issue.
Reach out to the cumm community for help.
File a bug report with the cumm developers.