Well, he is, like, 90 or so during LotR (and so is Denethor), so they aren't actually children anymore when the story takes place. In fact, one of the plot elements is Denethor wanting to marry Imrahil's sister, Finduilas (who'll eventually become the mother of Boromir and Faramir).