This paper addresses the performance of a user simulator in producing interactions with a spoken dialog system which include usability problems. Simple and general models of user behavior and speech understanding performance, as they are known in early design stages, are shown to detect up to 85% of the usability problems found in a real user test. Data can be reordered to allow a quick manual inspection of the test result. Thus, in an iterative design process small iterations, each incorporating knowledge gained from simulations, are recommended to efficiently improve the system design.