In Reiter's Default Logic (DL), semi-normal defaults are needed to give priorities to defaults to eliminate anomalous extensions. However, Morris' example with the same pattern of the Yale Shooting Problem has demonstrated that the use in DL of McCarthy's simple abnormality formalism with semi-normal default rules (henceforth called a simple abnormality default theory) still gives rise to an anomalous extension. In this paper, we present a modified version of DL called Prioritized Contextual Default Logic (PCtDL). In this logic a context is used to give priority to a default deriving another default's exception condition and therefore the non-normal part of the justification of a semi-normal default rule is treated differently from its normal part. PCtDL eliminates the possibility of anomalous extensions in both problems without however sacrificing the use of the simple abnormality theory. We also give some basic properties of PCtDL (like joint consistency) and present a possible worlds semantics, with respect to which it is shown to be sound and complete. In some ways, our result is akin to McCarthy's Prioritized Circumscription (although in the YSP other forms of circumscription were used to solve it, like Lifschitz's Pointwise Circumscription). Likewise, the preference criterion used in Brewka's Cumulative Default Logic (with filters), CDLF, albeit similar to ours, still results in an anomalous extension when applied to the aforementioned problems.