Robust motion recovery in tracking multiple targets using image features is affected by difficulties in obtaining good correspondences over long sequences. Difficulties are introduced by occlusions, scale changes, as well as disappearance of features with the rotation of targets. In this work, we describe an adaptive geometric template-based method for robust motion recovery from features. A geometric template consists of nodes containing salient features (e.g., corner features). The spatial configuration of the features is modeled using a spanning tree. This paper makes the following two contributions: (i) an adaptive geometric template to model the varying number of features on a target, and (ii) an iterative data association method for the features based on the uncertainties in the estimated template structure in conjunction with its individual features. We present experimental results for tracking multiple targets over long outdoor image sequences with multiple persistent occlusions. A comparison of the results of the data association method with a standard Mahalanobis distance gating applied to individual features is also presented.