Well, to start with, there's no real "mythos". His friends, after his death, tried to create one so they could write stories in it, but he didn't really have one. His tales seem to share the same world, with ideas and concepts in common, but there's no real order (in any sense of the world).
As for where to start... any generic collection of his stories will probably have the best introductions: Call of Cthulhu, Rats in the Walls, The Dunwich Horror, The Colour Out of Space... he did almost all short stories, so it's easy to just sample here and there. One thing to keep in mind is how much influence he's had. As a teen, I rented audio versions of both The Rats in the Walls, and a Stephen King short story. To my shock, the King was almost identical to Rats in the Walls! Luckily I was already a Lovecraft fan, but it did turn me off of King for a bit 
Once you're somewhat used to his style, definitely go for his one novel, In the Mountains of Madness. My favorite work of his, it starts with a scene of unspeakable horror and just builds.
Last edited by Invid (2012-06-12 15:10:03)