Probably yes.  There is always a question of fiduciary duty if a charity gives away money (not pursuant to its mission) that it has already received and has no obligation to give back.  But in this type of case, there may be a legitimate argument that the man is so divisive that his gift and affiliation undercut the ability of the organization to pursue its mission.  You ought to have gift acceptance policies that give guidelines, or at least establish a set individual or group to make the decision, about when a gift should be refused or returned. 

Since you already have the funds, particularly if they are substantial in amount, you may want to try to mediate between the donor and the board member.  Giving the money back and telling the donor to get lost may have its own repercussions.