Delay-sensitive cross-layer design has recently attracted increasing interests in providing delay performance guarantee and exploiting multi-user diversity throughput advantage for real-time traffic. Most of the existing solutions are centralized which are undesirable from the complexity scalability, signaling overhead scalability as well as the flexibility w.r.t. dynamic active sessions. In this work, we focus on delay-sensitive decentralized cross-layer design of OFDMA systems. Due to the stochastic system state, combinatorial nature of subcarrier allocation as well as the consideration for efficient feedback, conventional brute force decomposition technique failed to work. By using a novel two-level dual decomposition technique with comparison argument, we shall derive a feedback-efficient decentralized OFDMA cross-layer solution, with consideration of the stochastic system state, heterogeneous delay requirements, outdated CSIT as well as the combinatorial nature of subcarrier allocation. The proposed distributive design achieves optimal performance as the centralized solution scheduling with scalable complexity, signaling overhead as well as flexibility w.r.t. dynamic active sessions in the systems. In addition, using asymptotic analysis, we have derived the minimum feedback cost for ??-optimal system performance, wherein the average feedback cost per user is also shown to quickly approaches zero as systems scales.