Software-defined networking (SDN) simplifies the forwarding devices by introducing a centralized controller. The controller calculates routing rules for the whole network and the forwarding devices cache the routing rules. This working process leads to the new-flow attack. When malicious packets with different headers arrive at the network, they are treated as new flows. These useless flows consume lots of the resources in the controller and the forwarding devices. According to the current solution, suspicious flows are redirected to the security middleware. However, the security middleware can be a bottleneck when lots of flows are redirected to it in a short time. In this paper, we propose SmartSec to prevent the new-flow attack and optimize the security middleware at the same time. SmartSec uses the standard control link message to monitor the new-flow attack, and it achieves a low cost on the control link. Based on the monitoring results, SmartSec redirects the suspicious flows to the security middleware and monitors the workload of the security middleware. An optimization method is designed in SmartSec to reduce the workload of the security middleware. We evaluate our mechanism in both simulator and test bed. The simulation and experiment results verify the performance of SmartSec.