The demand for urban ecological environment is becoming more and more urgent. We use MR-Mixed Reality technology to reserch the restoration of urban ecological environment. First of all, we build a “gray box” model based on XML-Xtensible Markup Language and GIS-Group Innovation Space software and X3D virtual reality development platform. Then we using virtual reality development platformX3D and Kinect equipment to achieve the user and the system somatosensory interaction. Immediately following we propose the improvement of HMM-Hidden Markov Model gesture recognition algorithm based on both hands. This somatosensory interaction is more natural. Finally, we use ARToolKit to visualize the information of the virtual environment, Initial integration of Virtual Reality and Augmented Reality technologies. By giving a simple and three-dimensional scene, new and dynamic urban ecological multidimensional information, the multi-dimensional perception of the ecological environment information of the virtual city is enhanced.