In this paper we propose a Hidden Markov Model for modeling and extracting vine structure from images. We built up from previous research to infer connectivity of cane segments extracted from binary images. We use skeletonisation and polylines to model cane segments and we use simulated annealing to optimize an energy function defined in terms of attributes observed for each connection. We formulate our proposed solution in the context of MAP inference which is a state-of-the-art framework for inference in computer vision. We show comparative results of our method against state-of-the-art methods used for the same tasks, and our model generalizes and improves precision over prior research.