The pluperfect feels fine here to me, though I wonder whether “I may hurt you by accident,” John said should also be pluperfect. The use of plu for entry and exit of a flashback in normal past tense... seems to me to need a longer flashback. The nested flashback is OK here as it's only mentioned as a memory. Even if it gets de-nested into a parallel flashback in the reader's head, I don't see a problem with that.