As with any medication, stopping taking it abruptly can lead to severe withdrawal symptoms. That is the case with the immunosuppressant called Prednisone. Doctors should prescribe a gradually lower dose of the medication to prevent you suffering steroid withdrawal symptoms such as fever, low blood pressure, muscle and joint pain, headaches, nausea and fatigue.