This is exactly why there is such a huge disconnect between the points per game allowed and more advanced metrics like DVOA, that attempt to specifically isolate the play of the defensive unit. In terms of winning and losing it might not matter as much, since the contributions of all three units count, but if we are talking just about the performance of the defense, points allowed is not necessarily a useful metric. One of the most useful things Football Outsiders does is provide per-drive stats that are at the down series level, which helps remove the offensive time of possession and field position considerations. While it hasn't been updated to include yesterday's game, as of yesterday morning the Pats were ranked 31st by yards allowed per drive (about 39 yards, compared to 23 for JAX at #1, and an average of 30), and dead last in opponent 'Drive Success Rate,' which measures the percentage of opponent down series that end with a first down or touchdown (about .733 opponent success rate, with JAX #1 at .601 and the average of .682). There are some better numbers, we are 19th in points per drive (so missed field goals, fumble luck, and red zone success help there), and obviously are still a top team in terms of average opponent starting position and time of possession.
But yeah, the defense has slightly improved from the beginning of the season, but it is still one of the worst in the league, and has mostly just been benefitting from better complimentary football and having the O and ST units cover for them. Wins are wins, and they definitely look a bit better by the eye test, but this is still a liability for the team, not a strength. Luckily, we have the best offense in football, so they don't need to make a ton plays to get it done come playoff time.