To avoid the use of toxic building materials is one of the principles of sustainable construction. However, and contrary to general beliefs, current residential buildings frequently contain many toxic building materials some of which even comply with legal regulations. This paper discusses some cases of toxic building materials; it covers the materials responsible for the production of dioxins, furans, phthalates, the emission of volatile organic compounds from paints and varnishes, the toxicity of impregnating agents, materials that release toxic fumes during a fire, asbestos-based materials, radioactive materials and lead plumbing. Finally, considerations regarding leaching tests and how they can help to assess hazardous materials are made.