2022-10-28
+
Model Compression and Conversion
+ +-
+
- Compressing and Converting Models
+
+ - Model Input Path Specifications
+
+ - Model Output Path Description
+
+ - Conversion Templates
+
+
Compressing and Converting Models
+To obtain higher computing power, you can deploy the models created on ModelArts or a local PC on the Ascend chip. In this case, you need to compress or convert the models to the required formats before deploying them.
+ModelArts supports model conversion, allowing you to convert a model to a required format before deploying the model on a chip with higher computing power and performance.
+Model conversion applies to the following scenarios:
+- If you use the TensorFlow framework (in frozen_graph or saved_model format) to train a model, you can convert the model to the .om format. The converted model can be deployed and run on Ascend chips.
Constraints
- Only Ascend chips are supported for model conversion.
- Only TensorFlow models can be converted. For a TensorFlow model, the input data type is of the INT32, BOOL, UINT8, or FLOAT type.
- ModelArts provides conversion templates for you to choose. For details about the supported templates, see Conversion Templates.
- The .tflite and TensorRT formats support fewer operators and quantization operators. Therefore, some models may fail to be converted. If the conversion fails, view the log dialog box or check error logs in the conversion output directory.
- An OBS directory must be specified in compression/conversion tasks. Ensure that the OBS directory you use and ModelArts are in the same region.
- When importing the converted model to ModelArts, you need to use the model template.
- For a TensorFlow model, the FrozenGraphDef and SavedModel formats are supported. If a model is in the SavedModel format, the model is converted to the FrozenGraphDef format and then to the OM format.
- Inputs with dynamic shapes are not supported, for example, NHWC = [?,?,?,3]. A fixed value needs to be specified during model conversion.
- The input can be up to 4-dimensional. Operators involving dimension changes (such as reshape and expanddim) cannot output five dimensions.
- Except the const operator, the input and output at all layers in a model must meet the condition dim!=0.
- Model conversion does not support models that contain training operators.
- A UINT8 quantized model cannot be converted.
- Model operators support only 2D convolution but do not support 3D convolution. The batch_normalization_1 and FusedBatchNorm operators cannot be converted in batches.
Creating a Model Compression/Conversion Task
- Log in to the ModelArts management console, and choose Model Management > Compression/Conversion in the left navigation pane. The Compression/Conversion page is displayed.
- Click Create Task in the upper left corner to create a task.
- On the Create Task page that is displayed, set the required parameters based on Table 1.
++
+Table 1 Parameters Parameter
+Description
+Name
+Name of a model conversion task
+Description
+Description of a model conversion task
+Conversion Template
+ModelArts provides various templates to define model conversion and the parameters required during the conversion.
+Conversion Templates details the supported model conversion templates. You can select a template from the template list. Alternatively, you can enter a keyword in the search box to search for a template, or select a template based on the chip type, framework type, or model file format.
+- Chip type: ModelArts conversion templates support Ascend chips.
- Framework type: The conversion templates generate models in different formats based on different frameworks. The TensorFlow framework is supported.
- Model file format: The supported model file formats are listed in the drop-down list. Select a format from the drop-down list. The frozen_graph and tf_serving formats are supported.
Conversion Input Path
+Path to the model to be converted. The path must be an OBS path and comply with the ModelArts specifications. For details about the specifications, see Model Input Path Specifications.
+Conversion Output Path
+Path to the converted model. The path must comply with the ModelArts specifications. For details about the specifications, see Model Output Path Description.
+Advanced Settings
+ModelArts allows you to configure advanced settings for different conversion templates, for example, the precision.
+Different conversion templates support different advanced settings. For details about the parameters supported by each template, see Conversion Templates.
+Figure 1 Creating a model compression/conversion task+
- After entering the task information, click Create Now in the lower right corner.
After the task is created, the system automatically switches to the Compression/Conversion page. The created conversion task is displayed on the page and is in the Initializing status. The conversion task takes several minutes to complete. When the task status changes to Successful, the task is complete and the model has been converted.
+If the task status changes to Failed, click the task name to go to the task details page, view the log information, adjust task parameters based on the log information, and create another conversion task.
+
Deleting a Model Compression/Conversion Task
You can delete unnecessary conversion tasks. However, tasks in the Running or Initializing status cannot be deleted.
+
Deleted tasks cannot be recovered. Exercise caution when performing this operation.
+Model Input Path Specifications
+During model conversion, the model input directory must comply with certain specifications. This section describes how to upload your model package to OBS.
+Ascend Chip
The requirements for converting the models run on the Ascend chip are as follows:
+- For TensorFlow-based models (in frozen_graph or saved_model format), the input path must comply with the following specifications during model conversion:frozen_graph format+
| +|---xxxx.pb (Mandatory) Model network file. Only one model network file can exist in the input path. The model must be in frozen_graph or saved_model format. +|---insert_op_conf.cfg (Optional) Insertion operator configuration file. Only one insertion operator configuration file can exist in the input path. +|---plugin (Optional) Custom operator directory. The input directory can contain only one plugin folder. Only custom operators developed based on Tensor Engine (TE) are supported.
+saved_model format
+| +|---saved_model.pb (Mandatory) Model network file. Only one model network file can exist in the input path. The model must be in frozen_graph or saved_model format. +|---variables (Mandatory) Fixed subdirectory name, including the model weight deviation. + |---variables.index Mandatory + |---variables.data-00000-of-00001 Mandatory +|---insert_op_conf.cfg (Optional) Insertion operator configuration file. Only one insertion operator configuration file can exist in the input path. +|---plugin (Optional) Custom operator directory. The input directory can contain only one plugin folder. Only custom operators developed based on Tensor Engine (TE) are supported.
+
Model Output Path Description
+Ascend Chip
The following describes the output path of the model run on the Ascend chip after conversion:
+- For TensorFlow-based models, the output path must comply with the following specifications during model conversion:
| +|---xxxx.om Converted model to run on the Ascend chip. The model file name extension is .om. +|---job_log.txt Conversion log file
+
Conversion Templates
+ModelArts provides the following conversion templates based on different AI frameworks:
+ +TF-FrozenGraph-To-Ascend-C32
Convert the model trained by the TensorFlow framework and saved in frozen_graph format. The converted model can run on the Ascend. The custom operators (TBE operators) developed based on Tensor Based Engine (TBE) can be used for conversion.
+ +Parameter + |
+Description + |
+
|---|---|
input_shape + |
+Enter the shape of the input data of the model, for example, input_name1:n1,c1,h1,w1;input_name2:n2,c2,h2,w2. input_name must be the node name in the network model before model conversion. This parameter is mandatory when the model has dynamic shape input. For example, in input_name1:? ,h,w,c, the question mark (?) indicates the batch size, that is, the number of images processed at a time. It is used to convert the original model with a dynamic shape into an offline model with a fixed shape. The batch feature is not supported. The batch value of the input_shape can only be 1. During the conversion, the system parses the input model to obtain the input tensor and prints it in the log. If you do not know the input tensor of the used model, refer to the parsing result in the log. + |
+
input_format + |
+NCHW and NHWC are supported. The default format is NHWC. For the TensorFlow framework, the default value is NHWC. To use the NCHW format, you need to specify NCHW. For the Caffe framework, only the NCHW format is supported. + |
+
out_nodes + |
+Specifies the output node, for example, node_name1:0;node_name1:1;node_name2:0. node_name must be the node name in the network model before model conversion. The digit after each colon (:) indicates the sequence number of the output. For example, node_name1:0 indicates the 0th output of node_name1. If the output node is not specified, the output of the last operator layer serves as the model output by default. To check the parameters of a specific operator layer, specify the operator layer by using this parameter. During the conversion, the system parses the input model to obtain the output node and prints it in the log. If you do not know the input tensor of the used model, refer to the parsing result in the log. + |
+
net_format + |
+Specifies the preferred data format for network operators. Possible values are ND (N cannot be more than 4) and 5D. This parameter only takes effect if the input data of operators on the network supports both ND and 5D formats. ND indicates that operators in the model are converted into the NCHW format. 5D indicates that . 5D is the default value. + |
+
fp16_high_prec + |
+Specifies whether to generate a high-precision FP16 Davinci model. 0 is the default value, indicating that a common FP16 Da Vinci model with better inference performance is generated. The value 1 indicates that a high-precision FP16 Da Vinci model with better inference precision is generated. High-precision models support only Caffe operators (Convolution, Pooling, and FullConnection) and TensorFlow operators (tf.nn.conv2d and tf.nn.max_poo). + |
+
output_type + |
+FP32 is the default value and is recommended for classification and detection networks. For image super-resolution networks, UINT8 is recommended for better inference performance. + |
+
enable_l2dynamic + |
+L2 dynamic optimization switch. This parameter may affect the inference performance of the network model. If the performance does not meet the requirement, you can disable this function to verify the impact on the performance. true is the default value, indicating that L2 dynamic optimization is enabled. false indicates that L2 dynamic optimization is disabled. + |
+
TF-SavedModel-To-Ascend-C32
Convert the model trained by the TensorFlow framework and saved in saved_model format. The converted model can run on the Ascend. The custom operators (TE operators) developed based on TE can be used for conversion.
+ +Parameter + |
+Description + |
+
|---|---|
input_shape + |
+Enter the shape of the input data of the model, for example, input_name1:n1,c1,h1,w1;input_name2:n2,c2,h2,w2. input_name must be the node name in the network model before model conversion. This parameter is mandatory when the model has dynamic shape input. For example, in input_name1:? ,h,w,c, the question mark (?) indicates the batch size, that is, the number of images processed at a time. It is used to convert the original model with a dynamic shape into an offline model with a fixed shape. The batch feature is not supported. The batch value of the input_shape can only be 1. During the conversion, the system parses the input model to obtain the input tensor and prints it in the log. If you do not know the input tensor of the used model, refer to the parsing result in the log. + |
+
input_format + |
+NCHW and NHWC are supported. The default format is NHWC. For the TensorFlow framework, the default value is NHWC. To use the NCHW format, you need to specify NCHW. For the Caffe framework, only the NCHW format is supported. + |
+
out_nodes + |
+Specifies the output node, for example, node_name1:0;node_name1:1;node_name2:0. node_name must be the node name in the network model before model conversion. The digit after each colon (:) indicates the sequence number of the output. For example, node_name1:0 indicates the 0th output of node_name1. If the output node is not specified, the output of the last operator layer serves as the model output by default. To check the parameters of a specific operator layer, specify the operator layer by using this parameter. During the conversion, the system parses the input model to obtain the output node and prints it in the log. If you do not know the input tensor of the used model, refer to the parsing result in the log. + |
+
net_format + |
+Specifies the preferred data format for network operators. Possible values are ND (N cannot be more than 4) and 5D. This parameter only takes effect if the input data of operators on the network supports both ND and 5D formats. ND indicates that operators in the model are converted into the NCHW format. 5D indicates that . 5D is the default value. + |
+
fp16_high_prec + |
+Specifies whether to generate a high-precision FP16 Davinci model. 0 is the default value, indicating that a common FP16 Da Vinci model with better inference performance is generated. The value 1 indicates that a high-precision FP16 Da Vinci model with better inference precision is generated. High-precision models support only Caffe operators (Convolution, Pooling, and FullConnection) and TensorFlow operators (tf.nn.conv2d and tf.nn.max_poo). + |
+
output_type + |
+FP32 is the default value and is recommended for classification and detection networks. For image super-resolution networks, UINT8 is recommended for better inference performance. + |
+
enable_l2dynamic + |
+L2 dynamic optimization switch. This parameter may affect the inference performance of the network model. If the performance does not meet the requirement, you can disable this function to verify the impact on the performance. true is the default value, indicating that L2 dynamic optimization is enabled. false indicates that L2 dynamic optimization is disabled. + |
+