Combination of pre-selected nanoparticles with micro- and nanostructures is a crucial task in building hybrid quantum optical devices. Here a scanning probe based pick-and-place technique is introduced, capable of transferring nanoparticles to nearly arbitrarily shaped structures. This is demonstrated by building a variety of hybrid quantum optical and plasmonic systems using nitrogen vacancy centers in nanodiamonds as emitters.