To rub someone up the wrong way

His arrogant attitude rubbed me up the wrong way.

He tends to rub people up the wrong way without realizing it.