Changes in land use may alter land cover, which results in carbon stock changes in biomass as well as in the soil. In China’s loess plateau, vegetation restoration has been conducted since 1950s to control soil erosion and improve the ecosystem, with significant investment of money and manpower. Despite these efforts, soil erosion has still been severe. To reduce soil erosion and improve land quality, China initiated another state-funded project, Grain-for-Green, in 1999 in the loess plateau. However, it is not clear how effective this newly initiated project will be. In this study, we evaluated the effect of land-use conversion on soil organic carbon (SOC) and the potential effect of the current project on SOC sequestration in the Anjiapo catchment area of the loess hilly area of the loess plateau in China. This evaluation is based on SOC measurements in cropland versus in other converted land use types. We found that SOC sequestration mainly occurred in the surface soil after land use conversion took place. Land use conversion from cropland to shrubland or wild grassland (i.e. undisturbed land) was better for SOC sequestration than tree plantation in the semi-arid loess hilly area. By using the land use change in the study area as a scenario, the potential contribution of land use change on SOC sequestration due to the Grain-for-Green project was estimated. It was found that this project in the loess plateau of China would be helpful for SOC sequestration if successfully implemented.