The Year of the Overserviced Junior Professor

Right between the year of the rat and the year of the dog...

OK, so those years aren't even near one another, and the academic year doesn't correspond with the Chinese year, so whatever, but...

It's my second year here and I'm already doing waaayyy too much service:

• The T&P committee,
  
• A search committee,
  
• The PhD qualifying exam committee,
  
• The College scholarship committee (which just had me holed up in a windowless room for 12 hours in the last 2 weeks, the same ones with 30 student conferences),
  
• A focus group on childcare at the university,
  
• The department faculty research prize committee
• PLUS a bunch of piddling little "being around the department" stuff that some colleagues think fulfills their obligation to service.

And I said no to one or two things, including advising the grad student association and serving on an undergrad curriculum task force, both of which I would've liked to be a part of very much.

I know I'm being whiny here, since I know many of my readers are also contributing more than their share to their department cultures. The thing is, I believe in the culture of the department, of supporting the work we do as a body of faculty. And so I will very very likely continue to do too much service, which, as long as my research productivity is solid and my teaching is sound, will earn lots of thanks you's and a few cursory "you're doing too much" nods.
So the question is: When I really really want to say yes, to be on that committee, to help shape and support the life of the department or the college, when do we say "no?" and how?