What kind of holster? Cheap holsters just suck, if its a quality holster you may get it to work for you, see below
Do you have a proper gun belt? A stuff belt made for carrying a gun will support even a cheap holster well, while a standard flimsy belt wont properly support even the best made holster.
Where on your body are you wearing it? Most prefer strong side, just behind the hip, 4 o'clock. Appendix carry works better for others, 2-3 o'clock. Experiment with different positions.
Are you wearing a shirt under the holster? Obviously acts as a liner between skin and holster. You could also try adding some moleskin (found at CVS/Walgreens)
If that fails, then try another holster. Like guns, a particular holster may be great for one person, but may not work at all for another.