I've always worried about it. I'm not so much worried about them actually saying "this team is going to win no matter what," but I am worried that they give one team unfair advantages.
The thing that worries me the most is what happened in the NBA. The NBA would assign certain officiating crews to specific games, and they had outlined how "effective" these refs were. By effective I mean how likely they were to make/not make calls based on how they reacted to a hostile crowd. Some refs are more likely to make/not make calls based on the intimidation factor of the crowd. It doesn't necessarily mean the other team is for sure going to win, but it gives them plenty of help.
I sure hope it's not fixed...
If I die today, I'm happy how my life turned out