This paper discusses new dedicated hardware architecture for hybrid optimization based on Genetic algorithm (GA) and Simulated Annealing (SA). The proposed hardware achieves high speed processing comparison with software processing. Moreover, the hybrid optimization of GA and SA in the proposed architecture realized searching not only globally but also locally. To keep general purpose, self-control processing by a handshake system is introduced. By introducing this type of handshake system, the proposed hardware can be applied to various combinatorial optimization problems by only changing encoder, decoder, and evaluation circuit. Furthermore, the proposed architecture is flexible for many genetic operations on GA and achieves high speed processing by adopting dedicated hardware. Simulation results prove the effectiveness of the proposed hardware architecture.