HOV lanes are big out here.......And when someone uses them, it's to pass congested traffic. Not very many people use it to "kick back" and enjoy some peace & quiet.
If traffic isn't congested, and you have no desire to go faster than them, stay in the regular lanes. Just because your vehicle may be HOV-eligible doesn't mean you have to USE the HOV lane.
Oh, and here in Cali, lane-splitting is legal, so bikes often pass other HOV users.