Meh, all four of these guys are, if not Hall of Famers, pretty close to it.
Pettitte has 60 bWAR (which I find a little high, but it's clear he's HOVG). Bernie has 50, with 60 offensively - he lost a lot by playing CF much too long. Dumbo has 42, which is probably in the top 20 all time for a catcher.
26 World Series + a policy that retires numbers for guys like Guidry, Mattingly, Billy Martin and The Canton Aviator? The Core Three That Aren't Jeter + Bernie deserves their numbers retired as much as those four. That'll get you to 20 retired numbers.
Put it like this: everyone else's favorite hockey team has 18 retired numbers, all Hall of Famers, and they only retired Guy Lapointe's number last year. Steve Shutt, Jacques Lemaire, Tom Johnson, Toe Blake, etc - none have their numbers retired.