I have played a Witch Doctor, and completed the game. I then began a Swashbuckler; which is presented as a powerful hitting school. I but crown gear, and regularly update my weapons. I have done this for both schools. I dont see Swashbuckler as hitting near as hard as my Witchdoctor class; although that should be the opposite according to the class description. I feel the classes need tweaked so a class like Swashbuckler actually does the damage it is advertised to do.
Part of the issue i think is that the amount of damage done varies to much, and more is weak then is powerful. I think it would be nice to know what your gonna get with a class, but dont believe the classes live up to their billing.
I've noticed something weird like this too, my buccaneer will sometimes hit over my supposed damage or under. For your swashbuckler problem it's pretty basic as i've found out. It seems that most bosses have more armor than resistance. Witch doctors attacks are blocked by resistance, and swashbucklers are blocked by armor, so that may be the case. Maybe you don't have the best combination of stats?