DocumentAIKit/README_en.md

简体中文 | English

**A High-Efficient Development Toolkit for Object Detection based on [PaddlePaddle](https://github.com/paddlepaddle/paddle)**

Product Update

• 🔥 2022.11.15：SOTA rotated object detector and small object detector based on PP-YOLOE

  • Rotated object detector PP-YOLOE-R
  • SOTA Anchor-free rotated object detection model with high accuracy and efficiency
  • A series of models, named s/m/l/x, for cloud and edge devices
  • Avoiding using special operators to be deployed friendly with TensorRT.
  • Small object detector PP-YOLOE-SOD
  • End-to-end detection pipeline based on sliced images
  • SOTA model on VisDrone based on original images.

• 2022.8.26：PaddleDetection releasesrelease/2.5 version

  • 🗳 Model features：

  • Release PP-YOLOE+: Increased accuracy by a maximum of 2.4% mAP to 54.9% mAP, 3.75 times faster model training convergence rate, and up to 2.3 times faster end-to-end inference speed; improved generalization for multiple downstream tasks

  • Release PicoDet-NPU model which supports full quantization deployment of models; add PicoDet layout analysis model

  • Release PP-TinyPose Plus. With 9.1% AP accuracy improvement in physical exercise, dance, and other scenarios, our PP-TinyPose Plus supports unconventional movements such as turning to one side, lying down, jumping, and high lifts

  • 🔮 Functions in different scenarios

  • Release the pedestrian analysis tool PP-Human v2. It introduces four new behavior recognition: fighting, telephoning, smoking, and trespassing. The underlying algorithm performance is optimized, covering three core algorithm capabilities: detection, tracking, and attributes of pedestrians. Our model provides end-to-end development and model optimization strategies for beginners and supports online video streaming input.

  • First release PP-Vehicle, which has four major functions: license plate recognition, vehicle attribute analysis (color, model), traffic flow statistics, and violation detection. It is compatible with input formats, including pictures, online video streaming, and video. And we also offer our users a comprehensive set of tutorials for customization.

  • 💡 Cutting-edge algorithms：

  • Release PaddleYOLO which overs classic and latest models of YOLO family: YOLOv3, PP-YOLOE (a real-time high-precision object detection model developed by Baidu PaddlePaddle), and cutting-edge detection algorithms such as YOLOv4, YOLOv5, YOLOX, YOLOv6, YOLOv7 and YOLOv8

  • Newly add high precision detection model based on ViT backbone network, with a 55.7% mAP accuracy on COCO dataset; newly add multi-object tracking model OC-SORT; newly add ConvNeXt backbone network.

  • 📋 Industrial applications: Newly add Smart Fitness, Fighting recognition, and Visitor Analysis.

• 2022.3.24：PaddleDetection releasedrelease/2.4 version

  • Release high-performanace SOTA object detection model PP-YOLOE. It integrates cloud and edge devices and provides S/M/L/X versions. In particular, Verson L has the accuracy as 51.4% on COCO test 2017 dataset, inference speed as 78.1 FPS on a single Test V100. It supports mixed precision training, 33% faster than PP-YOLOv2. Its full range of multi-sized models can meet different hardware arithmetic requirements, and adaptable to server, edge-device GPU and other AI accelerator cards on servers.
  • Release ultra-lightweight SOTA object detection model PP-PicoDet Plus with 2% improvement in accuracy and 63% improvement in CPU inference speed. Add PicoDet-XS model with a 0.7M parameter, providing model sparsification and quantization functions for model acceleration. No specific post processing module is required for all the hardware, simplifying the deployment.
    
  • Release the real-time pedestrian analysis tool PP-Human. It has four major functions: pedestrian tracking, visitor flow statistics, human attribute recognition and falling detection. For falling detection, it is optimized based on real-life data with accurate recognition of various types of falling posture. It can adapt to different environmental background, light and camera angle.
  • Add YOLOX object detection model with nano/tiny/S/M/L/X. X version has the accuracy as 51.8% on COCO Val2017 dataset.

• More releases

Brief Introduction

PaddleDetection is an end-to-end object detection development kit based on PaddlePaddle. Providing over 30 model algorithm and over 300 pre-trained models, it covers object detection, instance segmentation, keypoint detection, multi-object tracking. In particular, PaddleDetection offers high- performance & light-weight industrial SOTA models on servers and mobile devices, champion solution and cutting-edge algorithm. PaddleDetection provides various data augmentation methods, configurable network components, loss functions and other advanced optimization & deployment schemes. In addition to running through the whole process of data processing, model development, training, compression and deployment, PaddlePaddle also provides rich cases and tutorials to accelerate the industrial application of algorithm.

Features

• Rich model library: PaddleDetection provides over 250 pre-trained models including object detection, instance segmentation, face recognition, multi-object tracking. It covers a variety of global competition champion schemes.
• Simple to use: Modular design, decoupling each network component, easy for developers to build and try various detection models and optimization strategies, quick access to high-performance, customized algorithm.
• Getting Through End to End: PaddlePaddle gets through end to end from data augmentation, constructing models, training, compression, depolyment. It also supports multi-architecture, multi-device deployment for cloud and edge device.
• High Performance: Due to the high performance core, PaddlePaddle has clear advantages in training speed and memory occupation. It also supports FP16 training and multi-machine training.

Exchanges

• If you have any question or suggestion, please give us your valuable input via GitHub Issues

Welcome to join PaddleDetection user groups on WeChat (scan the QR code, add and reply "D" to the assistant)

Kit Structure

KeyPoint

• DarkPose

FPN

• BiFPN
• CSP-PAN
• Custom-PAN
• ES-PAN
• HRFPN

Loss

• Smooth-L1
• GIoU/DIoU/CIoU
• IoUAware
• Focal Loss
• CT Focal Loss
• VariFocal Loss

Architectures	Backbones	Components	Data Augmentation
Object Detection Faster RCNN FPN Cascade-RCNN PSS-Det RetinaNet YOLOv3 YOLOF YOLOX YOLOv5 YOLOv6 YOLOv7 YOLOv8 RTMDet PP-YOLO PP-YOLO-Tiny PP-PicoDet PP-YOLOv2 PP-YOLOE PP-YOLOE+ PP-YOLOE-SOD PP-YOLOE-R SSD CenterNet FCOS FCOSR TTFNet TOOD GFL GFLv2 DETR Deformable DETR Swin Transformer Sparse RCNN Instance Segmentation Mask RCNN Cascade Mask RCNN SOLOv2 Face Detection BlazeFace Multi-Object-Tracking JDE FairMOT DeepSORT ByteTrack OC-SORT BoT-SORT CenterTrack KeyPoint-Detection HRNet HigherHRNet Lite-HRNet PP-TinyPose	Details ResNet(&vd) Res2Net(&vd) CSPResNet SENet Res2Net HRNet Lite-HRNet DarkNet CSPDarkNet MobileNetv1/v3 ShuffleNet GhostNet BlazeNet DLA HardNet LCNet ESNet Swin-Transformer ConvNeXt Vision Transformer	Common Sync-BN Group Norm DCNv2 EMA

Post-processing

• SoftNMS
• MatrixNMS

Speed

• FP16 training
• Multi-machine training

Details

• Resize
• Lighting
• Flipping
• Expand
• Crop
• Color Distort
• Random Erasing
• Mixup
• AugmentHSV
• Mosaic
• Cutmix
• Grid Mask
• Auto Augment
• Random Perspective

Model Performance

Performance comparison of Cloud models

The comparison between COCO mAP and FPS on Tesla V100 of representative models of each architectures and backbones.

Clarification：

• ViT stands for ViT-Cascade-Faster-RCNN, which has highest mAP on COCO as 55.7%
• Cascade-Faster-RCNNstands for Cascade-Faster-RCNN-ResNet50vd-DCN, which has been optimized to 20 FPS inference speed when COCO mAP as 47.8% in PaddleDetection models
• PP-YOLOE are optimized PP-YOLO v2. It reached accuracy as 51.4% on COCO dataset, inference speed as 78.1 FPS on Tesla V100
• PP-YOLOE+ are optimized PP-YOLOE. It reached accuracy as 53.3% on COCO dataset, inference speed as 78.1 FPS on Tesla V100
• The models in the figure are available in the model library

Performance omparison on mobiles

The comparison between COCO mAP and FPS on Qualcomm Snapdragon 865 processor of models on mobile devices.

Clarification：

• Tests were conducted on Qualcomm Snapdragon 865 (4 *A77 + 4 *A55) batch_size=1, 4 thread, and NCNN inference library, test script see MobileDetBenchmark
• PP-PicoDet and PP-YOLO-Tiny are self-developed models of PaddleDetection, and other models are not tested yet.

Model libraries

1. General detection

PP-YOLOE series Recommended scenarios: Cloud GPU such as Nvidia V100, T4 and edge devices such as Jetson series

Model	COCO Accuracy（mAP）	V100 TensorRT FP16 Speed(FPS)	Configuration	Download
PP-YOLOE+_s	43.9	333.3	link	download
PP-YOLOE+_m	50.0	208.3	link	download
PP-YOLOE+_l	53.3	149.2	link	download
PP-YOLOE+_x	54.9	95.2	link	download

PP-PicoDet series Recommended scenarios: Mobile chips and x86 CPU devices, such as ARM CPU(RK3399, Raspberry Pi) and NPU(BITMAIN)

Model	COCO Accuracy（mAP）	Snapdragon 865 four-thread speed (ms)	Configuration	Download
PicoDet-XS	23.5	7.81	Link	Download
PicoDet-S	29.1	9.56	Link	Download
PicoDet-M	34.4	17.68	Link	Download
PicoDet-L	36.1	25.21	Link	Download

Frontier detection algorithm

Model	COCO Accuracy（mAP）	V100 TensorRT FP16 speed(FPS)	Configuration	Download
YOLOX-l	50.1	107.5	Link	Download
YOLOv5-l	48.6	136.0	Link	Download
YOLOv7-l	51.0	135.0	链接	下载地址

Other general purpose models doc

2. Instance segmentation

Model	Introduction	Recommended Scenarios	COCO Accuracy(mAP)	Configuration	Download
Mask RCNN	Two-stage instance segmentation algorithm	Edge-Cloud end	box AP: 41.4 mask AP: 37.5	Link	Download
Cascade Mask RCNN	Two-stage instance segmentation algorithm	Edge-Cloud end	box AP: 45.7 mask AP: 39.7	Link	Download
SOLOv2	Lightweight single-stage instance segmentation algorithm	Edge-Cloud end	mask AP: 38.0	Link	Download

3. Keypoint detection

Model	Introduction	Recommended scenarios	COCO Accuracy（AP）	Speed	Configuration	Download
HRNet-w32 + DarkPose	Top-down Keypoint detection algorithm Input size: 384x288	Edge-Cloud end	78.3	T4 TensorRT FP16 2.96ms	Link	Download
HRNet-w32 + DarkPose	Top-down Keypoint detection algorithm Input size: 256x192	Edge-Cloud end	78.0	T4 TensorRT FP16 1.75ms	Link	Download
PP-TinyPose	Light-weight keypoint algorithm Input size: 256x192	Mobile	68.8	Snapdragon 865 four-thread 6.30ms	Link	Download
PP-TinyPose	Light-weight keypoint algorithm Input size: 128x96	Mobile	58.1	Snapdragon 865 four-thread 2.37ms	Link	Download

Other keypoint detection models doc

4. Multi-object tracking PP-Tracking

Model	Introduction	Recommended scenarios	Accuracy	Configuration	Download
ByteTrack	SDE Multi-object tracking algorithm with detection model only	Edge-Cloud end	MOT-17 half val: 77.3	Link	Download
FairMOT	JDE multi-object tracking algorithm multi-task learning	Edge-Cloud end	MOT-16 test: 75.0	Link	Download
OC-SORT	SDE multi-object tracking algorithm with detection model only	Edge-Cloud end	MOT-16 half val: 75.5	Link	-

Other multi-object tracking models docs

5. Industrial real-time pedestrain analysis tool-PP Human

Task	End-to-End Speed（ms）	Model	Size
Pedestrian detection (high precision)	25.1ms	Multi-object tracking	182M
Pedestrian detection (lightweight)	16.2ms	Multi-object tracking	27M
Pedestrian tracking (high precision)	31.8ms	Multi-object tracking	182M
Pedestrian tracking (lightweight)	21.0ms	Multi-object tracking	27M
Attribute recognition (high precision)	Single person8.5ms	Object detection Attribute recognition	Object detection：182M Attribute recognition：86M
Attribute recognition (lightweight)	Single person 7.1ms	Object detection Attribute recognition	Object detection：182M Attribute recognition：86M
Falling detection	Single person 10ms	Multi-object tracking Keypoint detection Behavior detection based on key points	Multi-object tracking：182M Keypoint detection：101M Behavior detection based on key points: 21.8M
Intrusion detection	31.8ms	Multi-object tracking	182M
Fighting detection	19.7ms	Video classification	90M
Smoking detection	Single person 15.1ms	Object detection Object detection based on Human Id	Object detection：182M Object detection based on Human ID: 27M
Phoning detection	Single person ms	Object detection Image classification based on Human ID	Object detection：182M Image classification based on Human ID：45M

Please refer to docs for details.

6. Industrial real-time vehicle analysis tool-PP Vehicle

Task	End-to-End Speed（ms）	Model	Size
Vehicle detection (high precision)	25.7ms	object detection	182M
Vehicle detection (lightweight)	13.2ms	object detection	27M
Vehicle tracking (high precision)	40ms	multi-object tracking	182M
Vehicle tracking (lightweight)	25ms	multi-object tracking	27M
Plate Recognition	4.68ms	plate detection plate recognition	Plate detection：3.9M Plate recognition：12M
Vehicle attribute	7.31ms	attribute recognition	7.2M

Please refer to docs for details.

Document tutorials

Introductory tutorials

• Installation
• Quick start
• Data preparation
• Geting Started on PaddleDetection
• FAQ

Advanced tutorials

• Configuration

  • RCNN Configuration
  • PP-YOLO Configuration

• Compression based on PaddleSlim

  • Pruning/Quantization/Distillation Tutorial

• Inference deployment

  • Export model for inference

  • Paddle Inference deployment

  • Inference deployment with Python

  • Inference deployment with C++

  • Paddle-Lite deployment

  • Paddle Serving deployment

  • ONNX model export

  • Inference benchmark

• Advanced development

  • Data processing module
  • New object detection models
  • Custumization
  • Object detection
  • Keypoint detection
  • Multiple object tracking
  • Action recognition
  • Attribute recognition

Courses

• [Theoretical foundation] Object detection 7-day camp: Overview of object detection tasks, details of RCNN series object detection algorithm and YOLO series object detection algorithm, PP-YOLO optimization strategy and case sharing, introduction and practice of AnchorFree series algorithm

• [Industrial application] AI Fast Track industrial object detection technology and application: Super object detection algorithms, real-time pedestrian analysis system PP-Human, breakdown and practice of object detection industrial application

• [Industrial features] 2022.3.26 Smart City Industry Seven-Day Class : Urban planning, Urban governance, Smart governance service, Traffic management, community governance.

• [Academic exchange] 2022.9.27 YOLO Vision Event: As the first YOLO-themed event, PaddleDetection was invited to communicate with the experts in the field of Computer Vision around the world.

Industrial tutorial examples

• Rotated object detection based on PP-YOLOE-R

• Aerial image detection based on PP-YOLOE-SOD

• Fall down recognition based on PP-Human v2

• Intelligent fitness recognition based on PP-TinyPose Plus

• Road litter detection based on PP-PicoDet Plus

• Visitor flow statistics based on FairMOT

• Guest analysis based on PP-Human

• More examples

Applications

• Fitness app on android mobile
• PP-Tracking GUI Visualization Interface

Recommended third-party tutorials

• Deployment of PaddleDetection for Windows I
• Deployment of PaddleDetection for Windows II
• Deployment of PaddleDetection on Jestson Nano
• How to deploy YOLOv3 model on Raspberry Pi for Helmet detection
• Use SSD-MobileNetv1 for a project -- From dataset to deployment on Raspberry Pi

Version updates

Please refer to the Release note for more details about the updates

License

PaddlePaddle is provided under the Apache 2.0 license

Contribute your code

We appreciate your contributions and your feedback！

• Thank Mandroide for code cleanup and
• Thank FL77N for Sparse-RCNNmodel
• Thank Chen-Song for Swin Faster-RCNNmodel
• Thank yangyudong, hchhtc123 for developing PP-Tracking GUI interface
• Thank Shigure19 for developing PP-TinyPose fitness APP
• Thank manangoel99 for Wandb visualization methods

Quote

@misc{ppdet2019,
title={PaddleDetection, Object detection and instance segmentation toolkit based on PaddlePaddle.},
author={PaddlePaddle Authors},
howpublished = {\url{https://github.com/PaddlePaddle/PaddleDetection}},
year={2019}
}