In many technological applications, excess solvent must be removed from liquid droplets to deposit solutes onto substrates. Often, the substrates on which the droplets rest may possess some roughness, either intended or unintended. Motivated by these observations, we present a lubrication-theory-based model to study the drying of droplets of colloidal suspensions on a substrate containing a topographical defect. The model consists of a system of one-dimensional partial differential equations accounting for the shape of the droplet and depth-averaged concentration of colloidal particles. A precursor film and disjoining pressure are used to describe the contact-line region, and evaporation is included using the well-known one-sided model. Finite-difference solutions reveal that when colloidal particles are absent, the droplet contact line can pin to a defect for a significant portion of the drying time due to a balance between capillary-pressure gradients and disjoining-pressure gradients. The time-evolution of the droplet radius and contact angle exhibits the constant-radius and constant-contact-angle stages that have been observed in prior experiments. When colloidal particles are present and the defect is absent, the model predicts that particles will be deposited near the center of the droplet in a cone-like pattern. However, when a defect is present, pinning of the contact-line accelerates droplet solidification, leading to particle deposition near the droplet edge in a coffee-ring pattern. These predictions are consistent with prior experimental observations, and illustrate the critical role contact-line pinning plays in controlling the dynamics of drying droplets.