It is well known that vehicle tracking processes are very computationally intensive. Traditionally, vehicle tracking algorithms have been implemented using software approaches. The goal of this paper is to investigate the feasibility of developing a hardware-based tracking system. We propose a tracking algorithm with vehicle motion detection based on block matching algorithm, which is well suited to customized hardware implementation. The proposed tracking algorithm is first simulated in MATLAB and the CMOS circuit design was simulated in Cadence. When tested on traffic images captured from an intersection, it is found that vehicle movements can be accurately identified in spite of some noisy motion. Also, the estimated computational time for the hardware based tracking system is much reduced compared to traditional software-based tracking system.