We report the results of a parquet renormalization group (RG) study of competing instabilities in the full 2D four-pocket, three-orbital low-energy model for iron-based superconductors. We derive and analyze the RG flow of the couplings, which describes all symmetry-allowed interactions between low-energy fermions. Despite that the number of the couplings is large, we argue that there are only two stable fixed trajectories of the RG flow and one weakly unstable fixed trajectory with a single unstable direction. Each fixed trajectory has a finite basin of attraction in the space of initial system parameters. On the stable trajectories, either interactions involving only dxz and dyz or only dxy orbital components on electron pockets dominate, while on the weakly unstable trajectory interactions involving dxz (dyz) and dxy orbital states on electron pockets remain comparable. The behavior along the two stable fixed trajectories has been analyzed earlier [Chubukov, Khodas, and Fernandes, Phys. Rev. X 6, 041045 (2016)10.1103/PhysRevX.6.041045]. Here we focus on the system behavior along the weakly unstable trajectory and apply the results to FeSe. We find, based on the analysis of susceptibilities along this trajectory, that the leading instability upon lowering the temperature is towards a three-component d-wave orbital nematic order. Two components are the differences between fermionic densities on dxz and dyz orbitals on hole pockets and on electron pockets, and the third one is the difference between the densities of dxy orbitals on the two electron pockets. We argue that this order is consistent with the splitting of band degeneracies, observed in recent photoemission data on FeSe by Fedorov et al. [Sci. Rep. 6, 36834 (2016)10.1038/srep36834].
Bibliographical noteFunding Information:
R.X. and A.V.C. are supported by the Office of Basic Energy Sciences, US Department of Energy, under award DE-SC0014402. L.C. thanks the School of Physics and Astronomy of the University of Minnesota for hospitality during this work and acknowledges funding by the Studienstiftung des deutschen Volkes and the HGSFP at Heidelberg University. MK is supported by the Israel Science Foundation, Grant No. 1287/15 and NSF DMR-1506668.