Fast excitatory synaptic transmission in central synapses is mediated primarily by AMPA receptors (AMPARs), which are heteromeric assemblies of four subunits, GluA1-4. Among these subunits, rapidly gating GluA3/4 appears to be the most abundantly expressed to enable neurotransmission with submillisecond precision at fast rates in subsets of central synapses. However, neither definitive identification of the molecular substrate for native AMPARs in these neurons, nor their hypothesized functional roles in vivo has been unequivocally demonstrated, largely due to lack of specific antagonists. Using GluA3 or GluA4 knockout (KO) mice, we investigated these issues at the calyx of Held synapse, which is known as a high-fidelity synapse involved in sound localization. Patch-clamp recordings from postsynaptic neurons showed that deletion of GluA4 significantly slowed the time course of both evoked and miniature AMPAR-mediated excitatory postsynaptic currents (AMPAR-EPSCs), reduced their amplitude, and exacerbated AMPAR desensitization and short-term depression (STD). Surprisingly, presynaptic release probability was also elevated, contributing to severe STD at GluA4-KO synapses. In contrast, only marginal changes in AMPAR-EPSCs were found in GluA3-KO mice. Furthermore, independent of changes in intrinsic excitability of postsynaptic neurons, deletion of GluA4 markedly reduced synaptic drive and increased action potential failures during high-frequency activity, leading to profound deficits in specific components of the auditory brainstem responses associated with synchronized spiking in the calyx of Held synapse and other related neurons in vivo. These observations identify GluA4 as the main determinant for fast synaptic response, indispensable for driving high-fidelity neurotransmission and conveying precise temporal information.