Under shearing, joint asperities get sheared off and damaged. Moreover, shearing and sliding and the interaction between normal and shear stresses occur simultaneously. The nonlinearity of the shear strength envelope is closely related to the joint material, normal stress level, and morphological characteristics. This study analyzed the correlation of the friction angle with the normal stress level to overcome overestimation or underestimation at relatively low or high normal stress levels in the JRC–JCS empirical model. A hyperbolic function was adopted to describe the degradation of friction for different normal stress levels, and the modified non-linear shear criterion $$\tau = \sigma_{\text{n}} \tan (\phi_{\text{r}} + \Delta \phi /(1 + a\sigma_{\text{n}} /{\text{JCS}}))$$ τ=σntan(ϕr+Δϕ/(1+aσn/JCS)) was proposed. Furthermore, the proposed model avoids direct connection to the surface morphology, which is convenient for practical use. Statistical analysis of the experimental data and comparison with the JRC–JCS model verified the validity of the proposed model and present an excellent method for characterizing the non-linearity of the failure envelope for the rock joint.