Yeah, Expertise is the stat that ONLY works in PvP scenarios. This means only in Warzones, when you're dueling another player or are fighting against a PvP tagged player on the opposite faction.
About the only time PvP gear becomes the least bit useful is when you get your Recruit mk-2 gear when you hit lv50 and they come with a couple of Battlemaster relics, these can be highly useful relics to have until you can get the better endgame relics later on.
@ Hassat: You can pick up recruit mk-2 gear from the PvP mission terminal.
I did 0 PvP for a long time until I suddenly started a month ago... now I have full War Hero gear with an Elite War Hero chest piece, a tonne of Expertise and I'm at Valor 55. Didn't take me all that long to get this far...