I take it you're hosting a dedicated server (not listen server) off of your home pc and connection, right? And using the latest server binaries?
What kind of hardware are you running it on (system specs)? Are you hosting on Linux or windows?
That might help us get an idea of what might be going on.