I understood that the line referred both to the fact that Ten had enjoyed being Five, "...dashing about and playing cricket" (Davison was less stodgy and authoritative than his precursors), as well as the fact that the Fifth Doctor was the one on the air when Tennant was a boy. He would have been about 9 or 10 years old at the time Davison began his run. What I didn't comprehend at the time was the emotion that Tennant displayed when he said that line. However when I watched Matt Smith say farewell as number Eleven in The Time of the Doctor earlier, something dawned on me.
Eleven just isn't my Doctor.