Three words: Duel game type.
What you describe is very hard to do, although it could be done.
The only reason you don't want the duel type is because then people can "watch it in a much more crowd setting". Why can't the spectators just be spectators (in duel)?
The duel game type is already used for that which you mention: allowing people to duel with each other in a controlled fashion (okay, not very controlled right now). All you need is an admin who could kick people if they broke the rules.