Yes, you should worry. If the transmission has ripped a tooth of a gear already, the gear is literally waiting to fail. If a gear fails it will slams the works to a stop (if your lucky) or force the shafts apart hard enough to break out your cases (if you're unlucky).
I have a set of transmission gears you can get cheap. I bought them off a member here for little to nothing and they're in usable shape.