# How to Prepare Training Data ## Directory - [How to Prepare Training Data](#how-to-prepare-training-data) - [Directory](#directory) - [Description of Object Detection Data](#description-of-object-detection-data) - [Prepare Training Data](#prepare-training-data) - [VOC Data](#voc-data) - [VOC Dataset Download](#voc-dataset-download) - [Introduction to VOC Data Annotation File](#introduction-to-voc-data-annotation-file) - [COCO Data](#coco-data) - [COCO Data Download](#coco-data-download) - [Description of COCO Data Annotation](#description-of-coco-data-annotation) - [User Data](#user-data) - [Convert User Data to VOC Data](#convert-user-data-to-voc-data) - [Convert User Data to COCO Data](#convert-user-data-to-coco-data) - [Reader of User Define Data](#reader-of-user-define-data) - [Example of User Data Conversion](#example-of-user-data-conversion) ### Description of Object Detection Data The data of object detection is more complex than classification. In an image, it is necessary to mark the position and category of each object. The general object position is represented by a rectangular box, which is generally expressed in the following three ways | Expression | Explanation | | :---------: | :----------------------------------------------------------------------------: | | x1,y1,x2,y2 | (x1,y1)is the top left coordinate, (x2,y2)is the bottom right coordonate | | x1,y1,w,h | (x1,y1)is the top left coordinate, w is width of object, h is height of object | | xc,yc,w,h | (xc,yc)is center of object, w is width of object, h is height of object | Common object detection datasets such as Pascal VOC, adopting `[x1,y1,x2,y2]` to express the bounding box of object. COCO uses `[x1,y1,w,h]`, [format](https://cocodataset.org/#format-data). ### Prepare Training Data PaddleDetection is supported [COCO](http://cocodataset.org) and [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) and [WIDER-FACE](http://shuoyang1213.me/WIDERFACE/) datasets by default. It also supports custom data sources including: (1) Convert custom data to VOC format; (2) Convert custom data to COOC format; (3) Customize a new data source, and add custom reader; firstly, enter `PaddleDetection` root directory ``` cd PaddleDetection/ ppdet_root=$(pwd) ``` #### VOC Data VOC data is used in [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) competition. Pascal VOC competition not only contains image classification task, but also contains object detection and object segmentation et al., the annotation file contains the ground truth of multiple tasks. VOC dataset denotes the data of PAscal VOC competition. when customizeing VOC data, For non mandatory fields in the XML file, please select whether to label or use the default value according to the actual situation. ##### VOC Dataset Download - Download VOC datasets through code automation. The datasets are large and take a long time to download ``` # Execute code to automatically download VOC dataset python dataset/voc/download_voc.py ``` After code execution, the VOC dataset file organization structure is： ``` >>cd dataset/voc/ >>tree ├── create_list.py ├── download_voc.py ├── generic_det_label_list.txt ├── generic_det_label_list_zh.txt ├── label_list.txt ├── VOCdevkit/VOC2007 │ ├── annotations │ ├── 001789.xml │ | ... │ ├── JPEGImages │ ├── 001789.jpg │ | ... │ ├── ImageSets │ | ... ├── VOCdevkit/VOC2012 │ ├── Annotations │ ├── 2011_003876.xml │ | ... │ ├── JPEGImages │ ├── 2011_003876.jpg │ | ... │ ├── ImageSets │ | ... | ... ``` Description of each document ``` # label_list.txt is list of classes name，filename must be label_list.txt. If using VOC dataset, when `use_default_label=true` in config file, this file is not required. >>cat label_list.txt aeroplane bicycle ... # trainval.txt is file list of trainset >>cat trainval.txt VOCdevkit/VOC2007/JPEGImages/007276.jpg VOCdevkit/VOC2007/Annotations/007276.xml VOCdevkit/VOC2012/JPEGImages/2011_002612.jpg VOCdevkit/VOC2012/Annotations/2011_002612.xml ... # test.txt is file list of testset >>cat test.txt VOCdevkit/VOC2007/JPEGImages/000001.jpg VOCdevkit/VOC2007/Annotations/000001.xml ... # label_list.txt voc list of classes name >>cat label_list.txt aeroplane bicycle ... ``` - If the VOC dataset has been downloaded You can organize files according to the above data file organization structure. ##### Introduction to VOC Data Annotation File In VOC dataset, Each image file corresponds to an XML file with the same name, the coordinates and categories of the marked object frame in the XML file, such as `2007_002055.jpg`: ![](../images/2007_002055.jpg) The XML file corresponding to the image contains the basic information of the corresponding image, such as file name, source, image size, object area information and category information contained in the image. The XML file contains the following fields： - filename, indicating the image name. - size, indicating the image size, including: image width, image height and image depth ``` 500 375 3 ``` - object field, indict each object, including: | Label | Explanation | | :--------------: | :------------------------------------------------------------------------------------------------------------------------: | | name | name of object class | | pose | attitude description of the target object (non required field) | | truncated | If the occlusion of the object exceeds 15-20% and is outside the bounding box，mark it as `truncated` (non required field) | | difficult | objects that are difficult to recognize are marked as`difficult` (non required field) | | bndbox son laebl | (xmin,ymin) top left coordinate, (xmax,ymax) bottom right coordinate | #### COCO Data COOC data is used in [COCO](http://cocodataset.org) competition. alike, Coco competition also contains multiple competition tasks, and its annotation file contains the annotation contents of multiple tasks. The coco dataset refers to the data used in the coco competition. Customizing coco data, some fields in JSON file, please select whether to label or use the default value according to the actual situation. ##### COCO Data Download - The coco dataset is downloaded automatically through the code. The dataset is large and takes a long time to download ``` # automatically download coco datasets by executing code python dataset/coco/download_coco.py ``` after code execution, the organization structure of coco dataset file is： ``` >>cd dataset/coco/ >>tree ├── annotations │ ├── instances_train2017.json │ ├── instances_val2017.json │ | ... ├── train2017 │ ├── 000000000009.jpg │ ├── 000000580008.jpg │ | ... ├── val2017 │ ├── 000000000139.jpg │ ├── 000000000285.jpg │ | ... | ... ``` - If the coco dataset has been downloaded The files can be organized according to the above data file organization structure. ##### Description of COCO Data Annotation Coco data annotation is to store the annotations of all training images in a JSON file. Data is stored in the form of nested dictionaries. The JSON file contains the following keys: - info，indicating the annotation file info。 - licenses, indicating the label file licenses。 - images, indicating the list of image information in the annotation file, and each element is the information of an image. The following is the information of one of the images: ``` { 'license': 3, # license 'file_name': '000000391895.jpg', # file_name # coco_url 'coco_url': 'http://images.cocodataset.org/train2017/000000391895.jpg', 'height': 360, # image height 'width': 640, # image width 'date_captured': '2013-11-14 11:18:45', # date_captured # flickr_url 'flickr_url': 'http://farm9.staticflickr.com/8186/8119368305_4e622c8349_z.jpg', 'id': 391895 # image id } ``` - annotations: indicating the annotation information list of the target object in the annotation file. Each element is the annotation information of a target object. The following is the annotation information of one of the target objects: ``` { 'segmentation': # object segmentation annotation 'area': 2765.1486500000005, # object area 'iscrowd': 0, # iscrowd 'image_id': 558840, # image id 'bbox': [199.84, 200.46, 77.71, 70.88], # bbox [x1,y1,w,h] 'category_id': 58, # category_id 'id': 156 # image id } ``` ``` # Viewing coco annotation files import json coco_anno = json.load(open('./annotations/instances_train2017.json')) # coco_anno.keys print('\nkeys:', coco_anno.keys()) # Viewing categories information print('\ncategories:', coco_anno['categories']) # Viewing the number of images print('\nthe number of images：', len(coco_anno['images'])) # Viewing the number of obejcts print('\nthe number of annotation：', len(coco_anno['annotations'])) # View object annotation information print('\nobject annotation information: ', coco_anno['annotations'][0]) ``` Coco data is prepared as follows. `dataset/coco/`Initial document organization ``` >>cd dataset/coco/ >>tree ├── download_coco.py ``` #### User Data There are three processing methods for user data: (1) Convert user data into VOC data (only include labels necessary for object detection as required) (2) Convert user data into coco data (only include labels necessary for object detection as required) (3) Customize a reader for user data (for complex data, you need to customize the reader) ##### Convert User Data to VOC Data After the user dataset is converted to VOC data, the directory structure is as follows (note that the path name and file name in the dataset should not use Chinese as far as possible to avoid errors caused by Chinese coding problems): ``` dataset/xxx/ ├── annotations │ ├── xxx1.xml │ ├── xxx2.xml │ ├── xxx3.xml │ | ... ├── images │ ├── xxx1.jpg │ ├── xxx2.jpg │ ├── xxx3.jpg │ | ... ├── label_list.txt (Must be provided and the file name must be label_list.txt ) ├── train.txt (list of trainset ./images/xxx1.jpg ./annotations/xxx1.xml) └── valid.txt (list of valid file) ``` Description of each document ``` # label_list.txt is a list of category names. The file name must be this >>cat label_list.txt classname1 classname2 ... # train.txt is list of trainset >>cat train.txt ./images/xxx1.jpg ./annotations/xxx1.xml ./images/xxx2.jpg ./annotations/xxx2.xml ... # valid.txt is list of validset >>cat valid.txt ./images/xxx3.jpg ./annotations/xxx3.xml ... ``` ##### Convert User Data to COCO Data `x2coco.py` is provided in `./tools/` to convert VOC dataset, labelme labeled dataset or cityscape dataset into coco data, for example: （1）Conversion of labelme data to coco data: ```bash python tools/x2coco.py \ --dataset_type labelme \ --json_input_dir ./labelme_annos/ \ --image_input_dir ./labelme_imgs/ \ --output_dir ./cocome/ \ --train_proportion 0.8 \ --val_proportion 0.2 \ --test_proportion 0.0 ``` （2）Convert VOC data to coco data: ```bash python tools/x2coco.py \ --dataset_type voc \ --voc_anno_dir path/to/VOCdevkit/VOC2007/Annotations/ \ --voc_anno_list path/to/VOCdevkit/VOC2007/ImageSets/Main/trainval.txt \ --voc_label_list dataset/voc/label_list.txt \ --voc_out_name voc_train.json ``` After the user dataset is converted to coco data, the directory structure is as follows (note that the path name and file name in the dataset should not use Chinese as far as possible to avoid errors caused by Chinese coding problems): ``` dataset/xxx/ ├── annotations │ ├── train.json # Annotation file of coco data │ ├── valid.json # Annotation file of coco data ├── images │ ├── xxx1.jpg │ ├── xxx2.jpg │ ├── xxx3.jpg │ | ... ... ``` ##### Reader of User Define Data If new data in the dataset needs to be added to paddedetection, you can refer to the [add new data source] (../advanced_tutorials/READER.md#2.3_Customizing_Dataset) document section in the data processing document to develop corresponding code to complete the new data source support. At the same time, you can read the [data processing document] (../advanced_tutorials/READER.md) for specific code analysis of data processing The configuration file for the Dataset exists in the `configs/datasets` folder. For example, the COCO dataset configuration file is as follows: ``` metric: COCO # Currently supports COCO, VOC, OID, Wider Face and other evaluation standards num_classes: 80 # num_classes: The number of classes in the dataset, excluding background classes TrainDataset: !COCODataSet image_dir: train2017 # The path where the training set image resides relative to the dataset_dir anno_path: annotations/instances_train2017.json # Path to the annotation file of the training set relative to the dataset_dir dataset_dir: dataset/coco #The path where the dataset is located relative to the PaddleDetection path data_fields: ['image', 'gt_bbox', 'gt_class', 'is_crowd'] # Controls the fields contained in the sample output of the dataset, note data_fields are unique to the trainreader and must be configured EvalDataset: !COCODataSet image_dir: val2017 # The path where the images of the validation set reside relative to the dataset_dir anno_path: annotations/instances_val2017.json # The path to the annotation file of the validation set relative to the dataset_dir dataset_dir: dataset/coco # The path where the dataset is located relative to the PaddleDetection path TestDataset: !ImageFolder anno_path: dataset/coco/annotations/instances_val2017.json # The path of the annotation file, it is only used to read the category information of the dataset. JSON and TXT formats are supported dataset_dir: dataset/coco # The path of the dataset, note if this row is added, `anno_path` will be 'dataset_dir/anno_path`, if not set or removed, `anno_path` is `anno_path` ``` In the YML profile for Paddle Detection, use `!`directly serializes module instances (functions, instances, etc.). The above configuration files are serialized using Dataset. **Note:** Please carefully check the configuration path of the dataset before running. During training or verification, if the path of TrainDataset or EvalDataset is wrong, it will download the dataset automatically. When using a user-defined dataset, if the TestDataset path is incorrectly configured during inference, the category of the default COCO dataset will be used. #### Example of User Data Conversion Take [Kaggle Dataset](https://www.kaggle.com/andrewmvd/road-sign-detection) competition data as an example to illustrate how to prepare custom data. The dataset of Kaggle [road-sign-detection](https://www.kaggle.com/andrewmvd/road-sign-detection) competition contains 877 images, four categories：crosswalk，speedlimit，stop，trafficlight. Available for download from kaggle, also available from [link](https://paddlemodels.bj.bcebos.com/object_detection/roadsign_voc.tar). Example diagram of road sign dataset: ![](../images/road554.png) ``` # Downing and unziping data >>cd $(ppdet_root)/dataset # Download and unzip the kaggle dataset. The current file organization is as follows ├── annotations │ ├── road0.xml │ ├── road1.xml │ ├── road10.xml │ | ... ├── images │ ├── road0.jpg │ ├── road1.jpg │ ├── road2.jpg │ | ... ``` The data is divided into training set and test set ``` # Generating label_list.txt >>echo -e "speedlimit\ncrosswalk\ntrafficlight\nstop" > label_list.txt # Generating train.txt, valid.txt and test.txt >>ls images/*.png | shuf > all_image_list.txt >>awk -F"/" '{print $2}' all_image_list.txt | awk -F".png" '{print $1}' | awk -F"\t" '{print "images/"$1".png annotations/"$1".xml"}' > all_list.txt # The proportion of training set, verification set and test set is about 80%, 10% and 10% respectively. >>head -n 88 all_list.txt > test.txt >>head -n 176 all_list.txt | tail -n 88 > valid.txt >>tail -n 701 all_list.txt > train.txt # Deleting unused files >>rm -rf all_image_list.txt all_list.txt The organization structure of the final dataset file is: ├── annotations │ ├── road0.xml │ ├── road1.xml │ ├── road10.xml │ | ... ├── images │ ├── road0.jpg │ ├── road1.jpg │ ├── road2.jpg │ | ... ├── label_list.txt ├── test.txt ├── train.txt └── valid.txt # label_list.txt is list of file name, file name must be label_list.txt >>cat label_list.txt crosswalk speedlimit stop trafficlight # train.txt is the list of training dataset files, and each line is an image path and the corresponding annotation file path, separated by spaces. Note that the path here is a relative path within the dataset folder. >>cat train.txt ./images/road839.png ./annotations/road839.xml ./images/road363.png ./annotations/road363.xml ... # valid.txt is the list of validation dataset files. Each line is an image path and the corresponding annotation file path, separated by spaces. Note that the path here is a relative path within the dataset folder. >>cat valid.txt ./images/road218.png ./annotations/road218.xml ./images/road681.png ./annotations/road681.xml ``` You can also download [the prepared data](https://paddlemodels.bj.bcebos.com/object_detection/roadsign_voc.tar), unzip to `dataset/roadsign_voc/` After preparing the data, we should generally understand the data, such as image quantity, image size, number of target areas of each type, target area size, etc. If necessary, clean the data. Roadsign dataset statistics: | data | number of images | | :---: | :--------------: | | train | 701 | | valid | 176 | **Explanation:** (1) For user data, it is recommended to carefully check the data before training to avoid crash during training due to wrong data annotation format or incomplete image data (2) If the image size is too large, it will occupy more memory without limiting the read data size, which will cause memory / video memory overflow. Please set batch reasonably_ Size, you can try from small to large