In recent years, a growing number of cryptosystems based on chaos have been proposed, many of them fundamentally flawed by a lack of robustness and security. This Letter describes the security weaknesses of a recently proposed cryptographic algorithm with chaos at the physical level based on the baker map. It is shown that the security is trivially compromised for practical implementations of the cryptosystem with finite computing precision and for the use of the iteration number n as the secret key. Some possible countermeasures to enhance the security of the chaos-based cryptographic algorithm are also discussed.