Definition of Pareto efficiency and prisoner's dilemma Pareto efficiency is defined in Wikipedia as: 


  Pareto efficiency or Pareto optimality is a state of allocation of
  resources from which it is impossible to reallocate so as to make any
  one individual or preference criterion better off without making at
  least one individual or preference criterion worse off.


And it has similar definitions in other sources.

Let's say in a Prisoner's dilemma game, A is confession, B is tie.

The strategies with results are:

(A, A) = -6, -6
(A, B) = 0, -10
(B, A) = -10, 0
(B, B) = -1, -1


Nash equilibrium is (A, A), also both confessing, and it's not Pareto efficient because by moving from (A, A) to (B, B), both could improve the result (form -6 to -1, also by 5).

But why is (A, B) and (B, A) not Pareto efficient? In both cases, no player can be made better off without making another player worse off.

Moving from (A, B) to (B, B) makes "player 2" better off (from -10 to -1) but makes "player 1" worse off (from 0 to -1).

Moving from (A, B) to (A, A) makes "player 2" better off (from -10 to -6) but makes "player 1" worse off (from 0 to -6).

Moving from (A, B) to (B, A) makes "player 2" better off (from -10 to 0) but makes "player 1" worse off (from 0 to -10).

So moving from (A, B) to any other combination makes one player worse off. That's the case with (B, A) too. 

So why is (A, B) and (B, A) not Pareto efficient?
(A, B) and (B, A) are in fact Pareto efficient.
I believe that your confusion may be because when discussing the Pareto inefficiency of the Prisoner's dilemma equilibrium, we always discuss (B,B) as the Pareto efficient alternative to (A,A) and (almost) never discuss (A,B) or (B,A).
Keep in mind that Pareto efficiency always requires a starting point to be evaluated. I.e. one way to determine if a case is Pareto efficient is to ask &quot;Starting from this point, is there any other allocation that is a Pareto improvement?&quot;. Here a Pareto improvement is a case making at least one person better off and no one worse off.
A natural starting point of discussion is the Nash equilibrium (A,A). Starting from there, only (B,B) is a Pareto improvement, which suffices to show that (A,A) is not Pareto efficient. That is what economists like to emphasize about the Prisoner's Dilemma and why textbook discussions focus on (B,B).
However, if your starting point is (A,B) or (B,A) then there are no Pareto improvements possible, as you argue. Hence, both those cases are Pareto efficient as well. However, note that these are not very interesting starting points anyway, because they are not Nash equilibria and therefore a bit irrelevant to discuss.