Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs

Symptom

[ERROR] RUNTIME(3002)model execute error, retCode=0x91, [the model stream execute failed]

Possible Causes

The speed of reading data cannot keep up with the model iteration speed.

Solution

1. Reduce shuffle operations during preprocessing.

   dataset = dataset.shuffle(buffer_size=x)

2. Disable data preprocessing, which may affect system performance.

   NPURunConfig(enable_data_pre_proc=Flase)

Summary and Suggestions

Before creating a training job, use the ModelArts development environment to debug the training code to maximally eliminate errors in code migration.