Recently, the amount of personal medical information online is increasing exponentially, opening up new avenues for hackers to expose personal data that, unlike financial information, can result in a permanent violation of privacy. To protect the privacy of patient data, such as electronic health records (EHRs), access control was used before and attributed-based encryption is used recently. These techniques can effectively prevent from the outside attacks, but are hard to withstand the inside attacks, where the database administrator or the key manager is an attacker. In this paper, we provide a solution to protect the privacy of patient data (EHRs) under the multi-party framework where all EHRs are encrypted with the common public key and an encrypted EHR can be decrypted only by the cooperation of all parties. Based on the ElGamal threshold public key encryption scheme, we propose several EHR access control protocols where multiple parties cooperate to control clinicians' access to EHRs without actually knowing EHRs. Our solution can protect the patient data against the inside attacks as long as at least one party can be trusted. Because our solution is built on Public Key Infrastructure (PKI), it facilitates the clinician registration and revocation.