Th Shulchan Aruch begins the laws of Zimun by saying "Three that have eaten as one". This sentence captures the essence of Zimun in that the 3 people who make up the Zimun must be one unit. Thus if one of the 3 ate a dairy meal and the others meat, or one ate at a separate table or did not end the meal with the other 2, then they can not be considered one unit since they have not "eaten as one". In the case of women, it is not considered modest for men and women to eat together; therefore, there is nothing to bind them together as one unit. While this reason certainly does not apply to one's wife and children, the rabbis do not make that distinction and, once we disallow men and women from making a Zimun, that includes the situation where the men and women are close relatives.
NOTE: The woman’s obligation in Zimun is very real so in a case where she eats with 3 men she should not leave until she bentches with the Zimun. The case for 3 women who ate together is also a strong one but I think we don't again out of modesty although I am not sure.