The increasing demand of global video traffic presents a grand challenge for Internet service providers. This is particularly important for Smart Cities, where the citizens will be surrounded by various multimedia services. The Information Centric Networking (ICN) paradigm has been proposed to handle the vast amount of data, mobility of users, and provide better scalability and efficiency in terms of bandwidth demand. However, the distributed caching mechanism in ICNs brings new challenges in maintaining satisfactory Quality of Experience (QoE) for video streaming applications. Because of the caching capability of ICN routers, streaming from multiple sources will be common, and switching between content sources might affect QoE by inducing delays and consequently stalls in video playback. This paper investigates the impact of content source switching, which is brought by distributed in-network caching, on the QoE for video streaming in ICNs. Human subjective tests are conducted using videos generated from an ICN emulation platform, in which the videos are streamed to human users under different caching strategies and available bandwidths. Test results indicate that, as the number of source switches increases, the satisfaction with stalling decreases and thus resulting in lower QoE scores. The impact of stalling on the overall QoE is further quantified under different video clarity levels through several statistical methods.