The warning from Harbour House comes after Gwyn Prosser, speaking in the House of Commons last week, said: "Dover Harbour Board's plans to sell off the port of Dover to the highest bidder have been thrown into chaos and confusion this morning, following the announcement by P&O and the other major port operators that the board may have been misappropriating revenue funds, and that T2, the new terminal, might not go ahead."
Comments made by MPs in the Commons are protected by Parliamentary privilege, but the harbour board has said it will not be afraid to take legal action if such accusations are made outside Westminster.
A source at Harbour House said: "Misappropriation suggests a criminal offence, frankly we are so angry because he is using Parliamentary privilege to suggest criminal wrongdoing and that is certainly not the case.
"It is a very inflammatory remark to make and if he chooses to use that kind of language outside of Parliament, where he has privilege, Dover Harbour Board will have no alternative but to take legal action."
Harbour board chief executive Bob Goldfield was unavailable for comment.
Mr Prosser said he was not taking the threat seriously and that he would not be making accusations of misappropriation of funds outside Parliament.
He said: "I would think their lawyers would be busy enough with P&O, Norfolkline and SeaFrance."