Okay, read "I'm more worried about" as "I wonder what". You're right, if there's one thing I'm not worried about, it's the quality of the show. Season Five is going to be great.
Steven Moffat himself, in the Confidential special that introduced Matt Smith yesterday, said that there was far too many really young actors on their audition list that the BBC had prepared. But what swayed me towards Matt was Moffat's insistence on a 30-40 year old, which he instantly disregarded the moment Smith auditioned.
The only thing Smith doesn't have is that air of geekiness that David Tennant always did. Tennant's one of us. He was always a fan of the show, and indeed is one of only two actors to appear in Doctor Who before they were cast as the Doctor (the other being Colin Baker). Tennant's was only in audio though, so he probably got away with it.
The lack of geek-cred is a disappointment in Smith, but it's not a problem. He (and Season Five) I have full confidence in.