The purpose of this work is to explore the possibility of using (Multi-disciplinary Design and Optimization) MDO frameworks for complex system design of space experiment. Considering both launch and space environmental requirements, the multi-objective optimization of a novel nested flying vehicle for space science experiments is carried out by decreasing launch weight, frequency response, vacuum effect and etc. Due to the complexity of structure parameters modification, the CAX software, the design model and analysis process are integrated based on ModelCenter framework. The objective weight method with gradient optimization algorithm and genetic algorithm for multi-objective optimization are utilized for validation mutually. In view of the optimization results, the overall performance of exterior shell with fixed mount is improved significantly.