Preparing Data

Uploading Data to OBS

This operation uses the OBS console to upload data. For more information about how to create a bucket and upload files, see Creating a Bucket and Uploading a File.

Perform the following operations to import data to the dataset for model training and building.

1. Log in to OBS Console and create a bucket.
2. Upload the local data to the OBS bucket. If you have a large amount of data, use OBS Browser+ to upload data or folders. The uploaded data must meet the dataset requirements of the ExeML project.

Requirements on Datasets

• The name of files in a dataset cannot contain Chinese characters, plus signs (+), spaces, or tabs.
• Ensure that no damaged image exists. The supported image formats include JPG, JPEG, BMP, and PNG.
• Do not store data of different projects in the same dataset.
• To ensure the prediction accuracy of models, the training samples must be similar to the actual application scenarios.
• To ensure the generalization capability of models, datasets should cover all possible scenarios.
• In an object detection dataset, if the coordinates of the bounding box exceed the boundaries of an image, the image cannot be identified as a labeled image.

Requirements for Files Uploaded to OBS

• If you do not need to upload training data in advance, create an empty folder to store files generated in the future, for example, /bucketName/data-cat.
• If you need to upload images to be labeled in advance, create an empty folder and save the images in the folder. An example of the image directory structure is /bucketName/data-cat/cat.jpg.
• If you want to upload labeled images to the OBS bucket, upload them according to the following specifications:
  • The dataset for object detection requires storing labeled objects and their label files (in one-to-one relationship with the labeled objects) in the same directory. For example, if the name of the labeled object is IMG_20180919_114745.jpg, the name of the label file must be IMG_20180919_114745.xml.

    The label files for object detection must be in PASCAL VOC format. For details about the format, see Table 1.

    Example of data files:

    ├─<dataset-import-path> 
          │      IMG_20180919_114732.jpg 
          │      IMG_20180919_114732.xml 
          │      IMG_20180919_114745.jpg 
          │      IMG_20180919_114745.xml 
          │      IMG_20180919_114945.jpg 
          │      IMG_20180919_114945.xml

  • Images in JPG, JPEG, PNG, and BMP formats are supported. When uploading images on the ModelArts console, ensure that the size of an image does not exceed 5 MB and the total size of images to be uploaded in one attempt does not exceed 8 MB. If the data volume is large, use OBS Browser+ to upload images.
  • A label name can contain a maximum of 32 characters, including letters, digits, hyphens (-), and underscores (_).

    Table 1 PASCAL VOC format description

    Field	Mandatory	Description
    folder	Yes	Directory where the data source is located
    filename	Yes	Name of the file to be labeled
    size	Yes	Image pixel width: image width. This parameter is mandatory. height: image height. This parameter is mandatory. depth: number of image channels. This parameter is mandatory.
    segmented	Yes	Segmented or not
    object	Yes	Object detection information. Multiple object{} functions are generated for multiple objects. name: class of the labeled object. This parameter is mandatory. pose: shooting angle of the labeled object. This parameter is mandatory. truncated: whether the labeled object is truncated (0 indicates that the object is not truncated). This parameter is mandatory. occluded: whether the labeled object is occluded (0 indicates that the object is not occluded). This parameter is mandatory. difficult: whether the labeled object is difficult to identify (0 indicates that the object is easy to identify). This parameter is mandatory. confidence: confidence score of the labeled object. The value range is 0 to 1. This parameter is optional. bndbox: bounding box type. This parameter is mandatory. For details about the possible values, see Table 2.

    Table 2 Description of bounding box types

    type	Shape	Labeling Information
    bndbox	Rectangle	Coordinates of the upper left and lower right points <xmin>100<xmin> <ymin>100<ymin> <xmax>200<xmax> <ymax>200<ymax>

    Example of the label file in KITTI format:

    <annotation>
       <folder>test_data</folder>
       <filename>260730932.jpg</filename>
       <size>
           <width>767</width>
           <height>959</height>
           <depth>3</depth>
       </size>
       <segmented>0</segmented>
       <object>
           <name>bag</name>
           <pose>Unspecified</pose>
           <truncated>0</truncated>
           <occluded>0</occluded>
           <difficult>0</difficult>
           <bndbox>
               <xmin>108</xmin>
               <ymin>101</ymin>
               <xmax>251</xmax>
               <ymax>238</ymax>
           </bndbox>
       </object>
    </annotation>