The evaluation of project quality exhibits multivariable, VRS (variable return to scale) and decision maker’s preference properties. In this paper, we present a Data Envelopment Analysis (DEA) based evaluation approach. The DEA VRS model, which handles multivariable and VRSeffectively, is used to measure project quality. And the DEA cone ratio model, which utilizes Analytical Hierarchy Process (AHP) to constrain quality metrics with respect to decision maker’s preference, is also adopted to analyze the return to scaleof the projects. A case study, which assesses 10 projects from ITECHS and 20 “Top active” projects on sourceforge.net with the novel method, is demonstrated. The results indicate that our approach is effective for quality evaluation and can get accurate estimates of future possible improvements.