Yeah....my memory does work!! The spacers come out in the middle gap part.
Use either a long punch, or actually a drift...just a long cylindrical piece of metal, ideally brass so it doesn't mess up bearings or the surfaces where bearings sit. You could use a piece of metal conduit or a small pipe or something that fits inside the swinger pivot....but put it in from one side, get it on the inside lip of the bearing on the oposite side and pound it outwards. Don't pound it out with the drift/punch/pipe in the same spot, once it budges a bit, move it 180 and hit again. Repeat until it pops out, then do the other side.