I don’t think “at the edge of the tradition” is precise but bear with me.
I have realized that the players I enjoy the most are anchored in traditional music but experiment with techniques and different modern styles. I’m thinking of Erwan Menguy (“The road in the fork” is the kind of playing one can only dream of), Malo Carvou, Conor Crimmins, Alan Doherty, …
The common denominator seems to be a judicious use of jazz-y undertones, rich articulation, exploration of range and techniques, …
Who else could be in that list? Who should I be listening to?
Nicholas Williams, especially his (mostly) flute albums: The Whispering Woods and The Crooked River. There’s many videos of him on YT, including several tunes with his band, Genticorum, in which he plays the flute (he also plays the accordion and the piano). He has very interesting ideas about tone and articulation, and you can hear it in his playing.
Seamus Egan comes to mind. I’m not sure he plays at the edge of trad music in terms of how he plays the flute, but certainly in how he plays the tunes. He’s a great musician.
I agree with most of the names already mentioned - was surprised Mike McGoldrick wasn’t mentioned sooner! Niall Keegan would be another one to check out. Can’t say I’m a fan of his style, but it certainly fits the OP criteria and is technically impressive.