It was probably Luis Tiant for me. He was absolutely lights out down the stretch in 1978 (4-2, 3.14 ERA in September), as well as 1975 (3-1, 1.47 ERA) and 1972 (6-2, 1.35 ERA). His departure was inexcusable. And, of course, he pitched a gem in Fenway in 1979 for the Yanks.
Another one that had fans annoyed at the time it happened was Bob Watson. The Sox picked him up in 1979 from Houston for Peter Ladd and, .... one Bobby Sprowl. He adjusted well to the new league (0.949 OPS and 13 HR's in 84 games). He was then signed as a free agent by the Yankees. Turned out the Sox got the better of that deal, as Watson had one decent year before falling off the cliff in his mid-30's. But it was symbolic of the Sox at the time, as Sullivan and LeRoux really had no idea how to run a baseball team in the era of free agency.
Lyle probably hurt most from a team impact point of view, although I'm sure Zimmer would have burnt out his arm like he did Campbell's. But Sox probably win one of Games 2 or 7 in 1975 with him coming out of the bullpen.
Damon didn't bother me that much; 2004 erased pretty much all of the Yankee envy, and the Sox went on to win again in 2007. The Boggs on the horse image was just silly; he was long past his prime. Clemens bothered me a lot more, although he was always irritating to have around here and I found him even more irritating after he left.