Solving the electronic structure problem for nanoscale systems remains a computationally challenging problem. The numerous degrees of freedom, both electronic and nuclear, make the problem impossible to solve without some effective approximations. Here we illustrate some advances in algorithm developments to solve the Kohn-Sham eigenvalue problem, i.e. we solve the electronic structure problem within density functional theory using pseudopotentials expressed in real space. Our algorithms are based on a nonlinear Chebyshev filtered subspace iteration method, which avoids computing explicit eigenvectors except at the first self-consistent-field iteration. Our method may be viewed as an approach to solve the original nonlinear Kohn-Sham equation by a nonlinear subspace iteration technique, without emphasizing the intermediate linearized Kohn-Sham eigenvalue problems. Replacing the standard iterative diagonalization at each self-consistent-field iteration by a Chebyshev subspace filtering step results in a significant speed-up, often an order of magnitude or more, over methods based on standard diagonalization. We illustrate this method by predicting the electronic and vibrational states for silicon nanocrystals.