X-ray imaging with spectral resolution, "Color X-ray imaging" is a new imaging technology that is currently attracting a lot of attention. It has however been observed that the quality of spectral response is degraded as the pixel size is reduced. This is an effect of charge sharing where the signal from a photon absorbed close to the border between two pixels is shared between pixels. This effect is caused by both diffusion during the charge transport and X-ray fluorescence in heavy detector materials. In order to understand the behavior of pixellated detectors with heavy detector materials operating in single photon processing mode, we have simulated the X-ray interaction with the sensor and the transport of the charge to the readout electrode using a Monte Carlo model for the X-ray interaction and a drift diffusion model for the charge transport. By combining these models, both signal and noise properties of the detector can be simulated.