We have designed a system to predict the sun occlusion due to clouds. Prediction of solar irradiance is an important function in order to reduce the cost of power management when integrating solar energy. The study is towards solar irradiance prediction. We further assume that the solar irradiance is highly dependent on the cloud coverage. Using our system, we are able to predict the cloud coverage as far as 20 minutes and up to 15 minutes with the accuracy better than the baseline algorithms. Our system includes all sky images for database acquisition, optical flow based cloud tracking, sun location back propagation methods and cloud segmentation modules. We perform systematic evaluation of our system on wide range of sky images collected by using ground based all sky imager (fisheye lens). The performance analysis shows that the prediction using the proposed method reduces the prediction error compared to random prediction and prediction using persistent model. With the proposed sun occlusion prediction using back propagation model, we are able to predict the occlusion percentage with the error of around 6% for 1 minute interval and around 30% for 20 minutes interval.