The use of robots in precision farming has the potential to reduce the reliance on herbicides and pesticides through selectively spraying individual plants or through manual weed removal. A prerequisite for that is the ability of the robot to separate and identify the value crops and the weeds in the field. Based on the output of the robot's perception system, it can trigger the actuators for spraying or removal. In this paper, we address the problem of detecting sugar beet plants as well as weeds using a camera installed on a mobile field robot. We propose a system that performs vegetation detection, local as well as object‐based feature extraction, random forest classification, and smoothing through a Markov random field to obtain an accurate estimate of crops and weeds. We implemented and thoroughly evaluated our system using a real farm robot in different sugar beet fields, and we illustrate that our approach allows for accurately identifying weeds in a field.