As I understand it, if you find Treasure Chests around Skull Island (either by themselves or in Dungeons - maybe bosses?), you can get a some very nice looking Pirate type clothes. Of course, you will have to stitch them with the particular items of clothing you have now for you to get the necessary attributes. Good Hunting!
I don't think it is a silly idea at all, 'mighty swashbuckler'. Avery's clothes are the pretty much the height of 'Age of Pirates' elegance, with his lovely cocked hat, gold braid embellished and embroidered waistcoat and frockcoat, silk stockings with fashionably buckled shoes, lace from his fine shirt showing beneath the false-buttoned cuffs of his frockcoat, snowy white neckstock fastened with a gold stickpin... who wouldn't love such an elegant and beautifully tailored ensemble, even a girl -- like me!
Happily, at the last update, similar 18th-century-ish ensembles are dropped now by Finn Dorsal in the Temple of Gloom, and I have heard other locations as well. Our swashbuckler frockcoats are cut a bit differently than Avery's -- his is of a style much like that in the earlier part of the 18th C. (and notice his well-curled gentleman's wig to match), which gives the sense of someone who is, after all, a lot older than you.
And I agree with the desire not to have the look of armor for swashbucklers. My privateer and buccaneer look fine and appropriate in theirs, but 'tis not to my swashbuckler's taste (which pretty much is to look as elegant as she possibly can) or my witchdoctor's (who likes to dress as fancifully and exotically as she can possibly manage.)