modelarts-umn/api-ref/source/data_management/dataset_management/creating_a_dataset.rst

original_name

CreateDataset.html

Creating a Dataset

Function

This API is used to create a dataset.

URI

POST /v2/{project_id}/datasets

Table 1 Path Parameters

Parameter	Mandatory	Type	Description
project_id	Yes	String	Project ID. For details about how to obtain the project ID, see Obtaining a Project ID <modelarts_03_0147>.

Request Parameters

Table 2 Request body parameters

Parameter	Mandatory	Type	Description
data_format	No	String	Data format. The options are as follows: - Default: default format - CarbonData: CarbonData (supported only by table datasets)
data_sources	No	Array of DataSource <createdataset__request_datasource> objects	Input dataset path, which is used to synchronize source data (such as images, text files, and audio files) in the directory and its subdirectories to the dataset. For a table dataset, this parameter indicates the import directory. The work directory of a table dataset cannot be an OBS path in a KMS-encrypted bucket.
dataset_name	Yes	String	Dataset name. The value contains 1 to 100 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed, for example, dataset-9f3b.
dataset_type	No	Integer	Dataset type. The options are as follows: - 0: image classification - 1: object detection - 100: text classification - 101: named entity recognition - 102: text triplet - 200: sound classification - 201: speech content - 202: speech paragraph labeling - 400: table dataset - 600: video labeling - 900: custom format
description	No	String	Dataset description. The value is empty by default. The description contains 0 to 256 characters and does not support the following special characters: ^!<>=&"'
import_annotations	No	Boolean	Whether to automatically import the labeling information in the input directory, supporting detection, image classification, and text classification. The options are as follows: - true: Import labeling information in the input directory. (Default value) - false: Do not import labeling information in the input directory.
import_data	No	Boolean	Whether to import data. This parameter is used only for table datasets. The options are as follows: - true: Import data when creating a database. - false: Do not import data when creating a database. (Default value)
label_format	No	LabelFormat <createdataset__request_labelformat> object	Label format information. This parameter is used only for text datasets.
labels	No	Array of Label <createdataset__request_label> objects	Dataset label list.
managed	No	Boolean	Whether to host a dataset. The options are as follows: - true: Host a dataset. - false: Do not host a dataset. (Default value)
schema	No	Array of Field <createdataset__request_field> objects	Schema list.
work_path	Yes	String	Output dataset path, which is used to store output files such as label files. - The format is /Bucket name/File path, for example, /obs-bucket/flower/rose/. (The directory is used as the path.)- A bucket cannot be directly used as a path.- The output dataset path is different from the input dataset path or its subdirectory.- The value contains 3 to 700 characters.
work_path_type	Yes	Integer	Type of the dataset output path. The options are as follows: - 0: OBS bucket (default value)
workforce_information	No	WorkforceInformation <createdataset__request_workforceinformation> object	Team labeling information.
workspace_id	No	String	Workspace ID. If no workspace is created, the default value is 0. If a workspace is created and used, use the actual value.

Table 3 DataSource

Parameter	Mandatory	Type	Description
data_path	No	String	Data source path.
data_type	No	Integer	Data type. The options are as follows: - 0: OBS bucket (default value) - 1: GaussDB(DWS) - 2: DLI - 3: RDS - 4: MRS - 5: AI Gallery - 6: Inference service
schema_maps	No	Array of SchemaMap <createdataset__request_schemamap> objects	Schema mapping information corresponding to the table data.
source_info	No	SourceInfo <createdataset__request_sourceinfo> object	Information required for importing a table data source.
with_column_header	No	Boolean	Whether the first row in the file is a column name. This field is valid for the table dataset. The options are as follows: - true: The first row in the file is the column name. - false: The first row in the file is not the column name.

Table 4 SchemaMap

Parameter	Mandatory	Type	Description
dest_name	No	String	Name of the destination column.
src_name	No	String	Name of the source column.

Table 5 SourceInfo

Parameter	Mandatory	Type	Description
cluster_id	No	String	ID of an MRS cluster.
cluster_mode	No	String	Running mode of an MRS cluster. The options are as follows: - 0: normal cluster - 1: security cluster
cluster_name	No	String	Name of an MRS cluster.
database_name	No	String	Name of the database to which the table dataset is imported.
input	No	String	HDFS path of a table dataset.
ip	No	String	IP address of your GaussDB(DWS) cluster.
port	No	String	Port number of your GaussDB(DWS) cluster.
queue_name	No	String	DLI queue name of a table dataset.
subnet_id	No	String	Subnet ID of an MRS cluster.
table_name	No	String	Name of the table to which a table dataset is imported.
user_name	No	String	Username, which is mandatory for GaussDB(DWS) data.
user_password	No	String	User password, which is mandatory for GaussDB(DWS) data.
vpc_id	No	String	ID of the VPC where an MRS cluster resides.

Table 6 LabelFormat

Parameter	Mandatory	Type	Description
label_type	No	String	Label type of text classification. The options are as follows: - 0: The label is separated from the text, and they are distinguished by the fixed suffix _result. For example, the text file is abc.txt, and the label file is abc_result.txt. - 1: Default value. Labels and texts are stored in the same file and separated by separators. You can use text_sample_separator to specify the separator between the text and label and text_label_separator to specify the separator between labels.
text_label_separator	No	String	Separator between labels. By default, a comma (,) is used as the separator. The separator needs to be escaped. The separator can contain only one character, such as a letter, a digit, or any of the following special characters: !@#$%^&*_=|?/':.;,
text_sample_separator	No	String	Separator between the text and label. By default, the Tab key is used as the separator. The separator needs to be escaped. The separator can contain only one character, such as a letter, a digit, or any of the following special characters: !@#$%^&*_=|?/':.;,

Table 7 Label

Parameter	Mandatory	Type	Description
attributes	No	Array of LabelAttribute <createdataset__request_labelattribute> objects	Multi-dimensional attribute of a label. For example, if the label is music, attributes such as style and artist may be included.
name	No	String	Label name.
property	No	LabelProperty <createdataset__request_labelproperty> object	Basic attribute key-value pair of a label, such as color and shortcut keys.
type	No	Integer	Label type. The options are as follows: - 0: image classification - 1: object detection - 100: text classification - 101: named entity recognition - 102: text triplet relationship - 103: text triplet entity - 200: speech classification - 201: speech content - 202: speech paragraph labeling - 600: video classification

Table 8 LabelAttribute

Parameter	Mandatory	Type	Description
default_value	No	String	Default value of a label attribute.
id	No	String	Label attribute ID.
name	No	String	Label attribute name.
type	No	String	Label attribute type. The options are as follows: - text: text - select: single-choice drop-down list
values	No	Array of LabelAttributeValue <createdataset__request_labelattributevalue> objects	List of label attribute values.

Table 9 LabelAttributeValue

Parameter	Mandatory	Type	Description
id	No	String	Label attribute value ID.
value	No	String	Label attribute value.

Table 10 LabelProperty

Parameter	Mandatory	Type	Description
@modelarts:color	No	String	Default attribute: Label color, which is a hexadecimal code of the color. By default, this parameter is left blank. Example: #FFFFF0.
@modelarts:default_shape	No	String	Default attribute: Default shape of an object detection label (dedicated attribute). By default, this parameter is left blank. The options are as follows: - bndbox: rectangle - polygon: polygon - circle: circle - line: straight line - dashed: dotted line - point: point - polyline: polyline
@modelarts:from_type	No	String	Default attribute: Type of the head entity in the triplet relationship label. This attribute must be specified when a relationship label is created. This parameter is used only for the text triplet dataset.
@modelarts:rename_to	No	String	Default attribute: The new name of the label.
@modelarts:shortcut	No	String	Default attribute: Label shortcut key. By default, this parameter is left blank. For example: D.
@modelarts:to_type	No	String	Default attribute: Type of the tail entity in the triplet relationship label. This attribute must be specified when a relationship label is created. This parameter is used only for the text triplet dataset.

Table 11 Field

Parameter	Mandatory	Type	Description
description	No	String	Schema description.
name	No	String	Schema name.
schema_id	No	Integer	Schema ID.
type	No	String	Schema value type.

Table 12 WorkforceInformation

Parameter	Mandatory	Type	Description
data_sync_type	No	Integer	Synchronization type. The options are as follows: - 0: not to be synchronized - 1: data to be synchronized - 2: label to be synchronized - 3: data and label to be synchronized
repetition	No	Integer	Number of persons who label each sample. The minimum value is 1.
synchronize_auto_labeling_data	No	Boolean	Whether to synchronously update auto labeling data. The options are as follows: - true: Update auto labeling data synchronously. - false: Do not update auto labeling data synchronously.
synchronize_data	No	Boolean	Whether to synchronize updated data, such as uploading files, synchronizing data sources, and assigning imported unlabeled files to team members. The options are as follows: - true: Synchronize updated data to team members. - false: Do not synchronize updated data to team members.
task_id	No	String	ID of a team labeling task.
task_name	Yes	String	Name of a team labeling task. The value contains 1 to 64 characters, including only letters, digits, underscores (_), and hyphens (-).
workforces_config	No	WorkforcesConfig <createdataset__request_workforcesconfig> object	Manpower assignment of a team labeling task. You can delegate the administrator to assign the manpower or do it by yourself.

Table 13 WorkforcesConfig

Parameter	Mandatory	Type	Description
agency	No	String	Team administrator.
workforces	No	Array of WorkforceConfig <createdataset__request_workforceconfig> objects	List of teams that execute labeling tasks.

Table 14 WorkforceConfig

Parameter	Mandatory	Type	Description
workers	No	Array of Worker <createdataset__request_worker> objects	List of labeling team members.
workforce_id	No	String	ID of a labeling team.
workforce_name	No	String	Name of a labeling team. The value contains 0 to 1024 characters and does not support the following special characters: !<>=&"'

Table 15 Worker

Parameter	Mandatory	Type	Description
create_time	No	Long	Creation time.
description	No	String	Labeling team member description. The value contains 0 to 256 characters and does not support the following special characters: ^!<>=&"'
email	No	String	Email address of a labeling team member.
role	No	Integer	Role. The options are as follows: - 0: labeling personnel - 1: reviewer - 2: team administrator - 3: dataset owner
status	No	Integer	Current login status of a labeling team member. The options are as follows: - 0: The invitation email has not been sent. - 1: The invitation email has been sent but the user has not logged in. - 2: The user has logged in. - 3: The labeling team member has been deleted.
update_time	No	Long	Update time.
worker_id	No	String	ID of a labeling team member.
workforce_id	No	String	ID of a labeling team.

Response Parameters

Status code: 201

Table 16 Response body parameters

Parameter	Type	Description
dataset_id	String	Dataset ID.
error_code	String	Error code.
error_msg	String	Error message.
import_task_id	String	ID of an import task.

Example Requests

• Creating an Image Classification Dataset

  {
    "workspace_id" : "0",
    "dataset_name" : "dataset-457f",
    "dataset_type" : 0,
    "data_sources" : [ {
      "data_type" : 0,
      "data_path" : "/test-obs/classify/input/cat-dog/"
    } ],
    "description" : "",
    "work_path" : "/test-obs/classify/output/",
    "work_path_type" : 0,
    "labels" : [ {
      "name" : "Cat",
      "type" : 0,
      "property" : {
        "@modelarts:color" : "#3399ff"
      }
    }, {
      "name" : "Dog",
      "type" : 0,
      "property" : {
        "@modelarts:color" : "#3399ff"
      }
    } ]
  }

• Creating an Object Detection Dataset

  {
    "workspace_id" : "0",
    "dataset_name" : "dataset-95a6",
    "dataset_type" : 1,
    "data_sources" : [ {
      "data_type" : 0,
      "data_path" : "/test-obs/detect/input/cat-dog/"
    } ],
    "description" : "",
    "work_path" : "/test-obs/detect/output/",
    "work_path_type" : 0,
    "labels" : [ {
      "name" : "Cat",
      "type" : 1,
      "property" : {
        "@modelarts:color" : "#3399ff"
      }
    }, {
      "name" : "Dog",
      "type" : 1,
      "property" : {
        "@modelarts:color" : "#3399ff"
      }
    } ]
  }

• Creating a Table Dataset

  {
    "workspace_id" : "0",
    "dataset_name" : "dataset-de83",
    "dataset_type" : 400,
    "data_sources" : [ {
      "data_type" : 0,
      "data_path" : "/test-obs/table/input/",
      "with_column_header" : true
    } ],
    "description" : "",
    "work_path" : "/test-obs/table/output/",
    "work_path_type" : 0,
    "schema" : [ {
      "schema_id" : 1,
      "name" : "150",
      "type" : "STRING"
    }, {
      "schema_id" : 2,
      "name" : "4",
      "type" : "STRING"
    }, {
      "schema_id" : 3,
      "name" : "setosa",
      "type" : "STRING"
    }, {
      "schema_id" : 4,
      "name" : "versicolor",
      "type" : "STRING"
    }, {
      "schema_id" : 5,
      "name" : "virginica",
      "type" : "STRING"
    } ],
    "import_data" : true
  }

Example Responses

Status code: 201

Created

{
  "dataset_id" : "WxCREuCkBSAlQr9xrde"
}

Status Codes

Status Code	Description
201	Created
401	Unauthorized
403	Forbidden
404	Not Found

Error Codes

See Error Codes <modelarts_03_0095>.