Robust adaptive subspace detection in impulsive noise

This paper addresses the design of the Adaptive Subspace Matched Filter (ASMF) detector in the presence of compound Gaussian clutters and a mismatch in the steering vector. In particular, we consider the case wherein the ASMF uses the regularized Tyler estimator (RTE) to estimate the clutter covariance matrix. Under this setting, a major question that needs to be addressed concerns the setting of the threshold and the regularization parameter. To answer this question, we consider the regime in which the number of observations used to estimate the RTE and their dimensions grow large together. Recent results from random matrix theory are then used in order to approximate the false alarm and detection probabilities by deterministic quantities. The latter are optimized in order to maximize an upper bound on the asymptotic detection probability while keeping the asymptotic false alarm probability at a fixed rate. © 2016 IEEE.