Most of the speaker recognition systems use system features for speaker recognition which are mostly spectral in nature. Recently, there has been significant work on using source features, viz., prosodies and pitch dynamics, glottal flow derivative, Linear Prediction (LP) residual and its phase, wavelet-domain representation of LP residual, etc for speaker recognition. In this paper, a new source-like feature set, viz., Teager Energy Operator (TEO) Phase is developed for speaker recognition. Proposed TEO Phase has several salient features as compared to that of LP residual phase, viz., less number of spurious peaks in Hilbert envelope plot, and better perceptual and sample correlation with speech signal. Furthermore, this avoids use of voiced/unvoiced detection, preprocessing techniques such as windowing and pre-emphasis and LP residual computation. Experiments have been carried out for speaker recognition task using TEO Phase and LP residual phase with polynomial classifier of 2 nd order approximation. For speaker verification, a reduction in equal error rate (EER) by 2.62 % over LP residual phase is achieved for proposed feature set.