That's a fallacious -very fallacious argument. Indeed, I'd go so far as to say its uninformed. In no way does it follow that polygamy or child-marriage would be acceptable if consenting adults of the same sex were allowed to marry.
If you're opposed to same-sex marriage there is a simple fix: don't marry someone of the same-sex. If your neighbors choose to marry, it in absolutely no way has any real affect on you. Not in any reality that we exist in.
If religions want to marry, perhaps they should call it something else.