Yup!
The village sits on the edge of the forest by the bank of the river. It stands to reason that there's a creek or stream that runs through the forest further upriver. So that's the detour you can take.
It's not as direct as village -> cut through forest -> guardian shrine ->
more forest -> torii gate -> temple steps -> temple,
but more like village -> hug the riverbank -> follow the stream ->
more forest -> get to ?? -> temple slope -> temple,
where you're currently at 'more forest'.
Borg-style maps are notoriously lacking in visualisation detail. Have a look at this (with the details blanked out):