»To haunt a man of forty-two«

“No doubt,” said I, “they settled who
      Was fittest to be sent
Yet still to choose a brat like you,
To haunt a man of forty-two,
      Was no great compliment!”

 
In his 29th annotation (MG029) to The Hunting of the Snark, Martin Gardiner stated:

Curiously, Carroll refers to his age as 42 in his poem Phantasmagoria (Canto 1, Stanza 16) though at the time [1869 or earlier] the poem was written, he was still in his thirties. The number 42 certainly seems to have had some sort of special significance for Carroll.

To me that simply means that for Carroll the number 42 does not refer to his own age.