You'll need a new lock ball and spring kit to stop the rattle. I bet awk has a couple rebuild kits laying around, and even if not, they are out there. You'll just have to do a little looking.
Or you can pick up a good used kicker on ebay, but rebuilding your existing one isnt hard to do, and its a lot cheaper than replacing the whole thing.
Just stay away from rigged fixes. You know, using silicone, etc. It won't last or work properly.