When someone calls you "difficult," or other criticisms, what they usually mean is "I can't control you as easily as I’d like."
Most labels are just tools people use to try and nudge you back into a shape that is convenient for them. Once you realize that most criticism is just a projection of the critic’s own frustrations, it stops hurting and starts being pathetic and/or sad.