Work = misery

I think I must have a huge sign above my head which is invisible to me.
Yet again I am pointed at and told I am not working. Well ok so it was a kindly soul who decided to tell me there were "rumours" going around that I do no work... and then I happened to cross paths with someone who was having a rant about people who just sit around and they may as well just go home (I'm trying not to take it as being mememe, but coming after the last comment it's not easy). So I have been forced into changing group suddenly and by surprise last Monday and I had thought even though I pointedly pointed that I didn't want to, as I had thought things resolved.
I mean I could point to people working alongside who spend 2 hours chatting to friends on the work phone, those who sit and surf the net obviously not doing work, and others who have running conversations with friends by e-mail. Perhaps that's what I'm doing wrong. Perhaps because I am actually head down in a pile of invoices all day (no not sleeping no matter how tired I get) that I'm not conforming to what everyone else s doing...... ??
I don't really come to anger or hate easily (I hope) but I really am hating going to work, not quite with a passion these days but I can see that if these things keep happening I will loath it all there.
Mum reminded me that this things about not working has nearly always cropped up in my work life - I remember when I first started work and it was a colleague who pointed out to the manager (while I stood gape mouthed at his accusation) that I had run around the whole shop floor checking that each till operator had enough cash, till roll, vouchers etc and I had just come downstairs from running her and the front tills this errand. Well I always thought if you go for one you may as well check all and get everything needed in one go. I wish sometimes it wasn't in my nature to go out of my way for people and be helpful.
I really don't get it, I don't understand what I'm doing wrong. I've always thought that being busy with work makes the day go past quicker, and anyway I get really bored if there's nothing to do.