To rub someone up the wrong way

His mannerisms rub me up the wrong way.

She tends to rub people up the wrong way without realizing it.