We study the semi-simplicity of the second dual of the Banach algebra of operators on a Banach space, B(E)'', endowed with either Arens product. It was previously shown that if E is a Hilbert space, then B(E) is Arens regular and B(E)'' is semi-simple. We show that for a large class of Banach spaces E, including subspaces of L p spaces not isomorphic to a Hilbert space, B(E)'' is not semi-simple. This is achieved by deriving a new representation of B(l p )', and then constructing a member of the radical of B(l p )'', for p<>2.