Co-workers accusing me of rudeness?

the supervisor from the other department at my job complained to my boss that the other secretaries said that I'm rude and i snatch papers off of their hands on several occasions. I usually come to work I keep my distance and just do my job. They basically turned a little thing into a big issue. What can I do to protect myself and avoid this conflict in the future?