From 83c6eba700997eb826952e217b069b2779d0d26e Mon Sep 17 00:00:00 2001 From: Glenn Jocher Date: Wed, 20 Mar 2019 13:26:46 +0200 Subject: [PATCH 1/4] Update README.md --- README.md | 43 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0fe97933..1e26b8ee 100755 --- a/README.md +++ b/README.md @@ -84,11 +84,48 @@ Run `detect.py` with `webcam=True` to show a live webcam feed. **PyTorch** format: - https://drive.google.com/drive/folders/1uxgUBemJVw9wZsdpboYbzUN4bcRhsuAI -# Validation mAP +# mAP -Run `test.py` to validate the official YOLOv3 weights `weights/yolov3.weights` against the 5000 validation images. You should obtain a .584 mAP at `--img-size 416`, or .586 at `--img-size 608` using this repo, compared to .579 at 608 x 608 reported in darknet (https://arxiv.org/abs/1804.02767). +Run `test.py --save-json --conf-thres 0.005` to test the official YOLOv3 weights `weights/yolov3.weights` against the 5000 validation images. Compare to .579 at 608 x 608 reported in darknet (https://arxiv.org/abs/1804.02767). -Run `test.py --weights weights/latest.pt` to validate against the latest training results. **Default training settings produce a 0.522 mAP at epoch 62.** Hyperparameter settings and loss equation changes affect these results significantly, and additional trade studies may be needed to further improve this. +Run `test.py --weights weights/latest.pt` to validate against the latest training results. Hyperparameter settings and loss equation changes affect these results significantly, and additional trade studies may be needed to further improve this. + +``` bash +sudo rm -rf yolov3 && git clone https://github.com/ultralytics/yolov3 +# bash yolov3/data/get_coco_dataset.sh +sudo rm -rf cocoapi && git clone https://github.com/cocodataset/cocoapi && cd cocoapi/PythonAPI && make && cd ../.. && cp -r cocoapi/PythonAPI/pycocotools yolov3 +cd yolov3 && python3 test.py --save-json --conf-thres 0.005 + +... + +Namespace(batch_size=32, cfg='cfg/yolov3.cfg', conf_thres=0.005, data_cfg='cfg/coco.data', img_size=416, iou_thres=0.5, nms_thres=0.45, save_json=True, weights='weights/yolov3.weights') + +loading annotations into memory... +Done (t=4.17s) +creating index... +index created! +Loading and preparing results... +DONE (t=1.75s) +creating index... +index created! +Running per image evaluation... +Evaluate annotation type *bbox* +DONE (t=39.30s). +Accumulating evaluation results... +DONE (t=4.63s). + Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.307 + Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.545 + Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.309 + Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.140 + Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.333 + Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.453 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.266 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.396 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.415 + Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.222 + Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.449 + Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.575 +``` # Contact From a5468acb5433535a244862825fef6023f833ed4f Mon Sep 17 00:00:00 2001 From: Glenn Jocher Date: Wed, 20 Mar 2019 13:35:03 +0200 Subject: [PATCH 2/4] Update README.md --- README.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/README.md b/README.md index 1e26b8ee..e39122ad 100755 --- a/README.md +++ b/README.md @@ -45,6 +45,21 @@ Each epoch trains on 120,000 images from the train and validate COCO sets, and t `from utils import utils; utils.plot_results()` ![Alt](https://user-images.githubusercontent.com/26833433/53494085-3251aa00-3a9d-11e9-8af7-8c08cf40d70b.png "train.py results") +# Speed + +https://cloud.google.com/deep-learning-vm/ +**Machine type:** n1-highmem-4 (4 vCPUs, 26 GB memory) +**CPU platform:** Intel Skylake +**GPUs:** 1-4 x NVIDIA Tesla P100 +**HDD:** 100 GB SSD + +GPUs | `batch_size` | speed | COCO epoch +--- |---| --- | --- +(P100) | (images) | (s/batch) | (min/epoch) +1 | 24 | 0.84s | 70min +2 | 48 | 1.27s | 53min +4 | 96 | 2.11s | 44min + ## Image Augmentation `datasets.py` applies random OpenCV-powered (https://opencv.org/) augmentation to the input images in accordance with the following specifications. Augmentation is applied **only** during training, not during inference. Bounding boxes are automatically tracked and updated with the images. 416 x 416 examples pictured below. From 7e8fc146e1d610b76d9becaaba6b9b2ba8ba95aa Mon Sep 17 00:00:00 2001 From: Glenn Jocher Date: Wed, 20 Mar 2019 13:35:39 +0200 Subject: [PATCH 3/4] Update README.md --- README.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index e39122ad..c5d7f059 100755 --- a/README.md +++ b/README.md @@ -45,21 +45,6 @@ Each epoch trains on 120,000 images from the train and validate COCO sets, and t `from utils import utils; utils.plot_results()` ![Alt](https://user-images.githubusercontent.com/26833433/53494085-3251aa00-3a9d-11e9-8af7-8c08cf40d70b.png "train.py results") -# Speed - -https://cloud.google.com/deep-learning-vm/ -**Machine type:** n1-highmem-4 (4 vCPUs, 26 GB memory) -**CPU platform:** Intel Skylake -**GPUs:** 1-4 x NVIDIA Tesla P100 -**HDD:** 100 GB SSD - -GPUs | `batch_size` | speed | COCO epoch ---- |---| --- | --- -(P100) | (images) | (s/batch) | (min/epoch) -1 | 24 | 0.84s | 70min -2 | 48 | 1.27s | 53min -4 | 96 | 2.11s | 44min - ## Image Augmentation `datasets.py` applies random OpenCV-powered (https://opencv.org/) augmentation to the input images in accordance with the following specifications. Augmentation is applied **only** during training, not during inference. Bounding boxes are automatically tracked and updated with the images. 416 x 416 examples pictured below. @@ -76,6 +61,21 @@ HS**V** Intensity | +/- 50% +## Speed + +https://cloud.google.com/deep-learning-vm/ +**Machine type:** n1-highmem-4 (4 vCPUs, 26 GB memory) +**CPU platform:** Intel Skylake +**GPUs:** 1-4 x NVIDIA Tesla P100 +**HDD:** 100 GB SSD + +GPUs | `batch_size` | speed | COCO epoch +--- |---| --- | --- +(P100) | (images) | (s/batch) | (min/epoch) +1 | 24 | 0.84s | 70min +2 | 48 | 1.27s | 53min +4 | 96 | 2.11s | 44min + # Inference Run `detect.py` to apply trained weights to an image, such as `zidane.jpg` from the `data/samples` folder: From 87cb8e661b8679730ed6ac9c4284d03266170c17 Mon Sep 17 00:00:00 2001 From: Glenn Jocher Date: Wed, 20 Mar 2019 14:08:24 +0200 Subject: [PATCH 4/4] Update README.md --- README.md | 78 ++++++++++++++++++++++++++++--------------------------- 1 file changed, 40 insertions(+), 38 deletions(-) diff --git a/README.md b/README.md index c5d7f059..6e0ca2e5 100755 --- a/README.md +++ b/README.md @@ -92,54 +92,56 @@ Run `detect.py` with `webcam=True` to show a live webcam feed. # Pretrained Weights -**Darknet** format: -- https://pjreddie.com/media/files/yolov3.weights -- https://pjreddie.com/media/files/yolov3-tiny.weights - -**PyTorch** format: -- https://drive.google.com/drive/folders/1uxgUBemJVw9wZsdpboYbzUN4bcRhsuAI +- Darknet `*.weights` format: https://pjreddie.com/media/files/yolov3.weights +- PyTorch `*.pt` format: https://drive.google.com/drive/folders/1uxgUBemJVw9wZsdpboYbzUN4bcRhsuAI # mAP -Run `test.py --save-json --conf-thres 0.005` to test the official YOLOv3 weights `weights/yolov3.weights` against the 5000 validation images. Compare to .579 at 608 x 608 reported in darknet (https://arxiv.org/abs/1804.02767). +- Use `test.py --weights weights/yolov3.weights` to test the official YOLOv3 weights. +- Use `test.py --weights weights/latest.pt` to test the latest training results. +- Compare to official darknet results from https://arxiv.org/abs/1804.02767. -Run `test.py --weights weights/latest.pt` to validate against the latest training results. Hyperparameter settings and loss equation changes affect these results significantly, and additional trade studies may be needed to further improve this. + | ultralytics/yolov3 | darknet +--- | ---| --- +YOLOv3-320 | 51.3 | 51.5 +YOLOv3-416 | 54.9 | 55.3 +YOLOv3-608 | 57.9 | 57.9 ``` bash sudo rm -rf yolov3 && git clone https://github.com/ultralytics/yolov3 # bash yolov3/data/get_coco_dataset.sh sudo rm -rf cocoapi && git clone https://github.com/cocodataset/cocoapi && cd cocoapi/PythonAPI && make && cd ../.. && cp -r cocoapi/PythonAPI/pycocotools yolov3 -cd yolov3 && python3 test.py --save-json --conf-thres 0.005 +cd yolov3 -... - -Namespace(batch_size=32, cfg='cfg/yolov3.cfg', conf_thres=0.005, data_cfg='cfg/coco.data', img_size=416, iou_thres=0.5, nms_thres=0.45, save_json=True, weights='weights/yolov3.weights') - -loading annotations into memory... -Done (t=4.17s) -creating index... -index created! -Loading and preparing results... -DONE (t=1.75s) -creating index... -index created! -Running per image evaluation... -Evaluate annotation type *bbox* -DONE (t=39.30s). -Accumulating evaluation results... -DONE (t=4.63s). - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.307 - Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.545 - Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.309 - Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.140 - Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.333 - Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.453 - Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.266 - Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.396 - Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.415 - Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.222 - Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.449 - Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.575 +python3 test.py --save-json --conf-thres 0.001 --img-size 416 +Namespace(batch_size=32, cfg='cfg/yolov3.cfg', conf_thres=0.001, data_cfg='cfg/coco.data', img_size=416, iou_thres=0.5, nms_thres=0.45, save_json=True, weights='weights/yolov3.weights') + Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.308 + Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.549 + Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.310 + Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.141 + Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.334 + Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.454 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.267 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.403 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.428 + Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.237 + Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.464 + Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.585 + +python3 test.py --save-json --conf-thres 0.001 --img-size 608 --batch-size 16 +Namespace(batch_size=16, cfg='cfg/yolov3.cfg', conf_thres=0.001, data_cfg='cfg/coco.data', img_size=608, iou_thres=0.5, nms_thres=0.45, save_json=True, weights='weights/yolov3.weights') + Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.328 + Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.579 + Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.335 + Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.190 + Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.357 + Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.428 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.279 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.429 + Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.456 + Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.299 + Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.483 + Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.572 ``` # Contact