This paper develops a convex optimization method to analyze the feasibility of a nonconvex bilinear matrix inequality (BMI), which is traditionally treated as a NP hard problem. First, a sufficient condition for the convexity of a quadratic matrix inequality (QMI), which is a more general semidefinite constraint than a BMI, is presented. It will be shown that the satisfaction of sufficient convexity condition implies that the QMI constraint can be transferred into an equivalent linear matrix inequality (LMI) constraint, which can be efficiently solved by well-developed interior-point algorithms. This result constitutes perhaps the first systematic methodology to verify the convexity of QMIs in the literature of semidefinite programming (SDP) in Control. For the BMI problem, a method to derive a convex inner approximation is discussed. The BMI feasibility analysis method is then applied to a nonlinear observer design problem where the estimation error dynamics is transformed into a Lure system with a sector condition constructed from the element-wise bounds on the Jacobian matrix of the nonlinearities. The developed numerical algorithm is used to design a nonlinear observer that satisfies multiple performance criteria simultaneously.