The Grid middleware must be developed iteratively and incrementally, so Grid performance testing is critical for middleware developers of Grid system. Considering the special characters of Grid system, in order to gain meaningful and comprehensive results of performance testing, it is necessary to implement testing on real Grid environment with various types of workload. CROWNBench, as described in this paper, is a system for helping Grid middleware developers to evaluate middleware design and implement using customizable synthetic workload. Middleware developers can customize testing workload basing on the model of Grid workload derived from real workload traces, including its structure and parameters, and then workload is synthesized automatically and contained jobs will be submitted by CROWNBench in a distributed manner. CROWNBench defines several metrics for measuring Grid performance as automatic testing results. The experiment, which used CROWNBench to test the performance of Grid system with CROWN Grid middleware, shows that the system already finished have accomplished its prospective goal. It can implement Grid performance testing in an efficient, flexible, controllable, replayable and automatic way to help middleware developers evaluate and improve their products effectively.