Why is it easier to go BACK from a location rather than going to that location you've never gone?

Without a trace of memory there would likely be little difference except that the path backwards was possible and the path forward may not be equally possible (scale a cliff?). Short term memory will recall anything that caught your attention while en-route to where you are now. No help if you are blindfolded? A house of mirrors is difficult to navigate because there are no unique features to guide you (only your own multiple reflection).