This paper presents three new ensembles of low density generator matrix (LDGM) codes, all of which are systematic codes. We prove that, under the maximum likelihood (ML) decoding, these codes can achieve the capacity of binary-input output-symmetric (BIOS) memoryless channels in terms of bit-error rate (BER). The proof technique is different from that for totally random block code ensembles. The recently proposed systematic block Markov superposition (BMST) code is treated as a variant of the presented ensembles, exhibiting capacity approaching performance with iterative belief propagation decoding algorithm.