He had been standing alone in this very office, savoring the triumph that was his after so many years of dreaming and scheming, when he heard a cough behind him, just like tonight. and turned to find that ugly little portrait talking to him, announcing that the Minister of Magic was about to arrive and introduce himself.
Naturally, he had thought that the long campaign and the strain of the election had caused him to go mad. He had been utterly terrified to find a portrait talking to him, though this had been nothing to how he felt when a self-proclaimed wizard had bounced out of the fireplace and shaken his hand. He had remained speechless throughout Fudge's kindly explanation that there were witches and wizards still living in secret all over the world and his reassurances that he was not to bother his head about them as the Ministry of Magic took responsibility for the whole Wizarding community and prevented the non-magical population from getting wind of them. It was, said Fudge, a difficult job that encompassed everything from regulations on responsible use of broomsticks to keeping the dragon population under control (the Prime Minister remembered clutching the desk for support at this point).
On the return of Voldemort
The Prime Minister gazed hopelessly at the pair of them for a moment, then the words he had fought to suppress all evening burst from him at last. "But for heaven's sake - you're wizards! You can do magic! Surely you can sort out - well - anything!"
Fudge... really did manage a smile this time as he said kindly, "The trouble is, the other side can do magic too, Prime Minister."
Fra Harry Potter and the Half Blood Prince