TDT ASR seg fault

[domklement]

Hi,

I've just pulled and installed the latest version of NeMo and am trying tu run asr TDT training. I'm getting a segmentation fault whenever the training is about to begin. I stepped through the code and it seems like a problem with TDT loss computation.

I've installed the toolkit using pip install -e '.[all]' and am running it the following way:

python ~/ASR/NeMo/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py \
    --config-path=~/ASR/NeMo/examples/asr/conf/fastconformer/hybrid_transducer_ctc \
    --config-name=fastconformer_hybrid_tdt_ctc_bpe.yaml \
    model.train_ds.manifest_filepath=~/data/librispeech/train.json \
    model.validation_ds.manifest_filepath=~/data/librispeech/dev.json \
    model.tokenizer.dir=~/tokenizers/ls960/tokenizer_spe_bpe_v500 \
    model.tokenizer.type=bpe \
    trainer.devices=-1 \
    trainer.accelerator="gpu" \
    trainer.strategy="ddp" \
    trainer.max_epochs=100 \
    exp_manager.create_wandb_logger=False \
    exp_manager.exp_dir=~/nemo_experiments/segfault_debug

I've been running the same code using an older version of NeMo (2.4.0rc0) and it worked fine, but I updated the dependencies and the same segfault is happening when I checkout to 2.4.0rc0 tag. Hence, it seems like a dependency issue as not all the requirements are properly tagged with a corresponding version.

Am I the only one who is it happening to? Do you have any solution/workaround?

Hardware:
CPU: AMD EPYC 9454
GPU: Nvidia H100 NVL 94GB (the same happens if I run it on multiple H100 SXM5 - DGX machine)
CUDA: 13.0
DRIVER: 580.65.06

Thank you!

Best,
Dominik Klement

[domklement]

I solved it in the meantime. Please, suggest the best way of incorporating this fix to the NeMo repo and I can do a PR then, or feel free to fix it right away.

It turned out that the cluster I'm using got new GPU drivers (the version I mentioned above) compared to the last time I ran NeMo there, and it doesn't go well with the numba package (for more info, see this Numba issue related to the exactly same driver version: numba/numba#10188).

Solution:

Numba with CUDA target is deprecated and moved to numba-cuda (https://github.com/NVIDIA/numba-cuda), so instead of running pip install numba (or letting NeMo requirements install it), run the following command before installing NeMo:

conda install -c conda-forge numba-cuda "cuda-version=13"

Also, make sure you have Cuda toolkit installed on your machine, if not, run:

conda install nvidia::cuda-toolkit -y

If your conda cannot find the package (happened to me), use libmamba solver instead (https://github.com/conda/conda-libmamba-solver).

Then, install NeMo as described in the README.

Note:

If, for some reason you encounter this error: nvvmAddNVVMContainerToProgram, may need newer version of nvJitLink library, a possible solution is to turn off the usage of Nvidia Bindings by setting:

export NUMBA_CUDA_USE_NVIDIA_BINDING="0"

[maanug-nv]

Hi @domklement , thanks for raising this and sharing your fix.
Sounds like we need to add numba-cuda to our requirements to be able to support CUDA 13.0 . Feel free to make a PR!

I believe we're planning to support CUDA 13.0 by our 25.11 release.
@ko3n1g @chtruong814 is there any issue with adding this package?

[diarray-hub]

You saved my life @domklement , thank you for sharing your fix!

[domklement]

Hi @domklement , thanks for raising this and sharing your fix. Sounds like we need to add numba-cuda to our requirements to be able to support CUDA 13.0 . Feel free to make a PR!

I believe we're planning to support CUDA 13.0 by our 25.11 release. @ko3n1g @chtruong814 is there any issue with adding this package?

Sure, I will. The thing I'd like to figure out is if there's a way of fixing it without setting NUMBA_CUDA_USE_NVIDIA_BINDING="0". It would complicate running examples and code and ideally, I don't want to add it to every example python script. If you have any suggestions, feel free to let me know. I will play around with it.

[parthchadha]

@chtruong814 can you please update if this issue can be closed (linked issue/PR #15018 was merged).

[chtruong814]

@domklement sorry, we should have closed this issue as resolved earlier. If you're still having issues, please let us know.