A microgrid can be a useful entity to support stable and efficient operation of power systems with large-scale penetration of distributed generators. Suppose that a microgrid should provide reserve operation in a decentralized environment; determining a suitable reserve capacity becomes an important problem for reliable and economic operation. To address this problem, in this paper we present a method of determining the optimal reserve capacity under the power exchange for frequency control (PXFC) market environment. After creating some penalty rules that take microgrid islanding into account, determining the optimal reserve capacity is formulated as an optimization problem. Microgrid islanding probability (MIP) is proposed as an index to describe the likelihood of microgrid islanding. The effectiveness of the method in terms of the operating cost is investigated using simulations, where our method and two further methods are applied to microgrids with different generation capabilities. The properties of MIP are also demonstrated by the simulations.