This paper considers the challenging problem of Cartesian tracking of multiple sources using multiple distributed microphone arrays when the number of sources is unknown and varies with time due to new sources appearing and existing sources disappearing or undergoing long silence periods. The problem is posed in a bearings-only tracking framework. Frequency-domain independent component analysis (ICA) in conjunction with state coherence transform (SCT) is used as a robust method to extract the bearing information of the speakers. Also, by exploiting the frequency sparsity of the sources, ICA/SCT has proven to be effective even when the number of simultaneous speakers is larger than the number of microphones in an array. Next, the bearing information for each array is fused using a sequential-corrector probability hypothesis density (PHD) filter with a limited field of view (FOV) for each microphone array. The limited FOV is essential for applications like speech in order to account for the more distant sources not registering detections with respect to a sensor array. The promising tracking capability of the proposed method is demonstrated using simulations of multiple speakers in a reverberant environment.