Drone Detection - Haar Cascade vs Convolutional Neural Networks 2

This video was taken during a public event, but if anybody feels offended by their presence in it please mention it in the comment section with a specified timestamp and I will remove that section from the video. I tried to remove all frames with no drone present or with too many faces shown.

The video itself was taken by standard Samsung Galaxy S6 front camera, with no additional filters or features implemented (zoom is only digital). It was then divided into frames, processed through Haar/Tensorflow object detection pipeline with detection bounding boxes added on top of the frame (red for Haar cascade, green for CNN). Resultant frames were combined into one and stitched back together to make a video.

Camera movement, blur, rotation, zoom, shakiness was purposefully added so as to allow object detection model testing under different conditions. None of the frames processed were used neither for training nor testing thus this video constitutes something of a deployment set.

Training (51,446 images, 51,445 positives) and testing set (5,375 images, 2,625 positives) were created by extracting every 50-140 frame from publicly available videos showing drones in different conditions and then manually labeled to create full dataset.

Haar Cascade model was trained using typical Haar pipeline with 1500 positives and 1000 negatives (larger input image datasets and CPU only support makes training larger Haar Cascades extremely time consuming). Altogether 603 Haar Cascades have been created (with different number of positives and negatives), but the presented one shows the highest test set accuracy of 54.505% @IoU=0.5 (precision 0.877, recall 0.119, specificity 0.983, F1 score 21%).

Presented CNN based object detection model was built based on Tensorflow Object Detection API, using Faster RCNN architecture pretrained on ImageNet and fine turned using entire train dataset, 640x640 image resizer and batch size of 1 (due to memory restrictions). This particular model was obtained and frozen after 370065 iterations (approximately 7 epochs) and shows test set accuracy of 81.411% @IoU=0.5&conf=0.5 (precision 0.831, recall 0.783, specificity 0.845 and F1 score of 80.6%).