Background: The Accreditation Council for Graduate Medical Education is shifting its focus from a structure and process system of graduate medical education to one that is outcomes based, requiring all residents in training to achieve competence in six broad domains and presenting educators with the challenge of evaluation. Summary: We searched the literature and reviewed all articles relating to portfolio assessment in medicine. In addition, we reviewed articles that addressed standard tools currently in use for resident performance assessment. The use of a portfolio allows one to incorporate a variety of assessment tools needed to evaluate the diverse domains of competence and also fosters reflective learning, which is key to professional development. From lessons learned through this review, we developed a Web-based evaluation portfolio for residency training that we describe here in the hope of facilitating this process for others. Conclusions: Web-based portfolio assessment provides an ideal venue for the evaluation of competence and has the ability to provide educators with a research infrastructure to practice evidence-based education.