Intersections are known for their integral and complex nature due to a variety of the participants' behaviors and interactions. This paper presents a review of recent studies on the behavior at intersections and the safety analysis for three types of participants at intersections: vehicles, drivers, and pedestrians. This paper emphasizes on techniques which are strong candidates for automation with visual sensing technology. A new behavior and safety classification is presented based on key features used for intersection design, planning, and safety. In addition, performance metrics are introduced to evaluate different studies, and insights are provided regarding the state of the art, inputs, algorithms, challenges, and shortcomings.