Dive straight in: The Outsider and Others
. IIRC, it has practically every important story he ever wrote, including the novellas (towards the end).
I'm not sure if the book is actually available for sale as is, but I used its contents as a guide to read the stories off the Internet (Lovecraft is in the public domain).
Edit: Welp, it seems the book is on Amazon
at a bargain price of $1,940. I'd snap it up if I were you.
There's also Wikipedia's bibliography list
you can consult if you're looking to read his works chronologically and comprehensively.
Lovecraft's theme and genre never really wavers a lot, so it doesn't matter where you start, the supreme cosmic gods that slumber in dimensions unfathomable will be sure to get you either way.