Spalling near a free surface in highly stressed rock is an open topic in design of underground openings. Spalling is often considered as an indicator of a potential collapse mechanism such as a rock burst, and therefore, is of high importance. In this study, a hybrid discrete-finite element model is used to simulate the surface instability. The bonded particle model was calibrated to mimic the mechanical properties of a Berea sandstone. Numerical results are compared with experiments performed using the University of Minnesota surface instability apparatus, which reproduces the state of plane strain near the free surface in a semi-infinite medium subjected to far field compressive stresses. The damage in the rock was monitored by acoustic emission. It was observed that the friction between the rock and the platens, as well as material ductility, affects the fracture pattern and the failure phenomenon. In particular, the existence of friction is important in providing lateral stress between the rock and the platen in order to assist the initiation of fracture from the free surface.