Balls and strikes alone is far harder than anything NFL refs have to do in my view, as it demands full concentration and focus for the entirety of the game.Baseball is, by far, the easiest. The game is static. NHL is not that tough, too. Refs miss calls or may not make calls that you want, but it's not nearly as subjective as hoops or football.
NBA is probably the toughest with the NFL slightly below that.