The central bearing spacer is such a close fit there's no space to get a drift behind the bearing. Attempts just result in the punch slipping off the inner race's radiused edge.
The best way to avoid further damage is to weld something across the hole on the middle. An old nut will do, it needs to be strong enough to take hammer and punch blows.
Then the drift can punch out the wheel bearing. The spacer will come out too, making it easy to get the drift onto the inner race of the wheel bearing on the other side.