Lai, Weijian 4e4b2d5f6d ModelArts UMN 23.3.0 Version.

Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Lai, Weijian <laiweijian4@huawei.com>
Co-committed-by: Lai, Weijian <laiweijian4@huawei.com>

2024-06-26 07:03:02 +00:00

2.5 KiB

Raw Blame History

Error Message "CUDNN_STATUS_NOT_SUPPORTED" Displayed in Logs

Symptom

The following error message is displayed during PyTorch training:

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

Possible Causes

The input data is not of contiguous type, which is not supported by cuDNN.

Solution

Disable cuDNN before training.
```
torch.backends.cudnn.enabled = False
```

Convert the input data into contiguous data.

images = images.cuda()  
images = images.permute(0, 3, 1, 2).contigous()

Summary and Suggestions

Before creating a training job, use the ModelArts development environment to debug the training code to maximally eliminate errors in code migration.

Parent topic: Service Code Issues

2.5 KiB Raw Blame History