Using a large set of human segmented natural images, we study the statistics of region boundaries. We observe several power law distributions which likely arise from both multi-scale structure within individual objects and from arbitrary viewing distance. Accordingly, we develop a scale-invariant representation of images from the bottom up, using a piecewise linear approximation of contours and constrained Delaunay triangulation to complete gaps. We model curvilinear grouping on top of this graphical/geometric structure using a conditional random field to capture the statistics of continuity and different junction types. Quantitative evaluations on several large datasets show that our contour grouping algorithm consistently dominates and significantly improves on local edge detection.