A number of philosophers have recently argued that we should interpret the debate over moral responsibility as a debate over the conditions under which it would be “fair” to blame a person for her attitudes or conduct. What is distinctive about these accounts is that they begin with the stance of the moral judge, rather than that of the agent who is judged, and make attributions of responsibility dependent upon whether it would be fair or appropriate for a moral judge to react to the agent in various (negative) ways. This is problematic, I argue, because our intuitions about whether and when it would be fair to react negatively to another are sensitive to a host of considerations that appear to have little or nothing to do with an agent’s responsibility or culpability for her attitudes or behavior. If this is correct, then theories which make attributions of responsibility dependent upon the appropriateness of our reactions as moral judges will turn out to be fundamentally misguided.