This paper presents an unsupervised segmentation method applicable to both 2D and 3D images. The segmentation is achieved by a bottom-up hierarchical analysis to progressively agglomerate pixels/voxels in the image into non-overlapped homogeneous regions characterised by a linear signal model. A hierarchy of adjacency graphs is used to describe agglomeration results from the hierarchical analysis, and is constructed by successively performing a clustering operation which produces an optimal classification by merging each region with its nearest neighbours. The nearest neighbour of a region is determined by a merge condition derived under the framework of a statistical inference and a dissimilarity function based on the error produced by fitting the region model to pixels/voxels in two adjacent regions. The top level of the hierarchy then describes the segmentation result.