Okay, I've figured it out: It's Skype.
As soon as I shut it down, the problem disappears.
So the question becomes, kaelik: Do you have Skype installed and on aswell?
And if so, if you shut it down, will the problems disappear?
Back to the same this morning.