I think the best way to learn is to listen to good players, live or on recordings, and pay attention to their variations. If you listen long enough, you’ll get a sense for what works within the bounds of good taste, and eventually you’ll start hearing variations in your head, which you can work out on your instrument. And then there will come a time when variations start popping out unplanned while you’re playing.
There are lots of great players who excel at variations; John Carty is one who comes to mind, but listen to any great Irish musician and you’ll hear variations. The thing to avoid is extended melodic departures that are thrown in merely to show off one’s technical prowess or creativity. Working within the constraints of the tradition doesn’t stifle creativity at all, and the result is more satisfying.