Where can the line be drawn? Who ever said that people should be able to have sex with animals? Who ever said that men should have the right to marry animals?
That's different. An animal can't be consenting because it can't think like a human being. Two gay men or women are thinking properly and are aware of their situation. They're not animals. You've just compared them to animals...
As for multiple guys in a polygamy, that just doesn't affect anyone but the people concerned. Whether it be between heteros or homosexuals, it's none of your concern actually. Besides, if in certain country, you can have multiple wives, why can't there be multiple husbands(for one guy)?
Like it's been said earlier, even if most of them don't want marriage, it should still be allowed for the ones that want.