He may have unknowingly been there at the same time as Gandalf, who was imprisoned ... "Long have I wandered by roads forgotten, seeking the house of Elrond, of which many had heard, but few ...