Unmanned Aerial Vehicles (UAVs) play vital role in a number of application domains including search and rescue, traffic monitoring, border control, to name a few. A robust computer vision system for detecting and tracking moving targets is essential to enable UAVs operate autonomously against challenges such as occlusions and abrupt camera motion. This paper presents a robust system that can handle these challenges and operate in real-time. Camera motion is decoupled from scene motion by performing motion compensation using multi-point-descriptor image registration while background subtraction is performed to compute regions of potential moving targets that are subsequently fed to a multi-classifier system where each classifier learns target appearance model. A ranking algorithm combines the results of the classifiers to estimate the final position of each target. The proposed system is tested on the DARPA VIVID dataset and demonstrates improved tracking accuracy over single classifier systems while incurring minimal computation overheads.