There are a couple ways to track what sort of things you are able to tranq on a given target. What I've done on my user interface is moved the target frame to slightly below where my character stands on my screen using Xperl, and xperl's integrated enemy buff tracker highlights buffs that are able to be dispelled (seen below). Its not always a great idea to tranq a target to no buffs before you start dpsing them though, because it requires a ton of focus to do it. Instead, try to identify buffs that are very bad for you based on their spell effects, and make an effort to get rid of those instead.
|Kind of hard to see, but whatever!|
To a certain extent, you can anticipate when these buffs are coming and have the tranq already in the air when the enemy player is getting ready to cast it. For example, vs. ret paladins, if you have them in the middle of a freezing trap and you're building distance between you, he's almost certainly going to cast hand of freedom on himself. Look for the red circle at his feet, or if you're feeling cocky, wait about 2 seconds after the root from your trap has activated, then throw the tranq. The only real way to understand what sort of buffs are mandatory to get rid of is through experience, but as a general guide, damage shields (like priest/mage barriers), enrage effects (druid Savage roar, warrior damage cooldown), and hand of freedom type effects are the ones that need to be gotten rid of as soon as possible.
Of course, in the arena, its a bit of a different ballgame, but keep practicing and you'll figure it out.
Also, MSBT (miks scrolling battle text i think) gives a notification when you successfully dispel something, and theres a shaman purge mod that announces it in chat for what its worth.
Sorry for my absence, but hey, better late than never.
Good luck out there,