Whereas SDN (Software Defined Networks) provides the opportunity for the flexibility of network configuration, the introduction of controller systems raises new issues about developing firewall system design, such as controller attack, rule setup, and communication overhead for control messages. Especially, the delay and overload for dynamic control of stateful firewall are serious bottlenecks. This paper examines the current challenges and their origins, and presents a comprehensive solution to the key operational steps: topology-based selective filtering rules for setup and maintenance stage, three-layer rule structure for in-switch flow tables, and controller attack protection based on adaptive host connection tracking with multiple hashing queues named FlowTracker algorithm. The experiment results using multiple OVS switches and virtual hosts in GENI testbed demonstrate FlowTracker succeeds in monitoring all network connections and completely profiling host normal routine with acceptable latency increment (170ms). Moreover, by utilizing multiple request queues, the proposed DoS attack detection algorithm reduce the response time to DoS 5 to 20 times less than using a single queue.