I agree that even if you don't stand you SHOULD take your cap off out of respect, but the fact he didn't intentionally NOT take it off and was harrassed I feel is totally out of line.
There is a famous picture of former Detroit Tiger Ron LeFlore in the outfield in 1975 watering a tomato plant. But the plant wasn't in a pot... it was growing in the outfield