Graphene's optical properties in the infrared and terahertz can be tailored and enhanced by patterning graphene into periodic metamaterials with sub-wavelength feature sizes. Here we demonstrate polarization-sensitive and gate-tunable photodetection in graphene nanoribbon arrays. The long-lived hybrid plasmon-phonon modes utilized are coupled excitations of electron density oscillations and substrate (SiO2) surface polar phonons. Their excitation by s-polarization leads to an in-resonance photocurrent, an order of magnitude larger than the photocurrent observed for p-polarization, which excites electron-hole pairs. The plasmonic detectors exhibit photo-induced temperature increases up to four times as large as comparable two-dimensional graphene detectors. Moreover, the photocurrent sign becomes polarization sensitive in the narrowest nanoribbon arrays owing to differences in decay channels for photoexcited hybrid plasmon-phonons and electrons. Our work provides a path to light-sensitive and frequency-selective photodetectors based on graphene's plasmonic excitations.