You can sort this list as you please: http://stats.nba.com/players/touches/#!?sort=TIME_OF_POSS&dir=1The problem is not just defense, but rather the fact that Kyrie wants to hold onto the ball for long periods without running the offense. He is a shot creator, but he only wants to create shots for himself rather than the team. That is the issue that needs to change.
Most of the scoring gurus in the NBA are shitty defenders. When a guy is expending so many calories per night on offense he will not have much energy left for defense. There's only so much adenosine triphosphate available in the body at any given time.
If Kyrie sets up other players in Boston and averages 10+ assists per game I can live with him being a poor defender. And if he is willing to give up the ball and contribute through scoring off ball and playing defense that works too. His problem thus far has been wanting to contribute very little to his team except individual scoring.
Kyrie Irving is basically behind Isaiah Thomas in any measure on this page in terms of ball stopping. I forget who it was, someone did post some good data that showed Irving being more ball dominant in some way, but on the whole the per touch metrics show that he's behind lots of other notable players in per touch, time of poss, number of dribbles kind of metrics. He's high for sure, but there are many worse offenders.