Unmarked spoilers ahoy!
The Aging Hermit in the swamps is Saint Matheus
You meet the Hermit a stone throw away from St. Matheus' cave, he sports facial tattoos similar to the ones St. Matheus received from the natives and a shawl like the one St. Matheus wears on the cave frescoes. His face and head structure also make him look more like a continental than an islander. The only problem with this is that St. Matheus is supposed to have died on the island centuries ago... but no one has ever found his body, not even De Sardet when they explore his cave. We know that the Guardians, at least, are functionally immortal, so the island has the capacity to prolong the life of some creatures indefinitely, and it is possible that St. Matheus, having achieved some kind of enlightenment in communion with the God of Many Faces, had been granted the boon of a particularly long life span.The Nauts did not claim De Sardet because they were Yecht Fradí
A big point is made in the tutorial missions that the sea-born of all nations — i.e. anyone born on their ships — belong to the Nauts from birth. However, when it is revealed that De Sardet was born on one of the ships leaving Teer Fradee after one of the Congregation's raiding missions, not once does anyone bring up on the implication that they, by all customs and traditions, should have stayed on the ship to grow up as a Naut, rather than being handed over to Prince d'Orsay. The only plausible explanation (other than the devs overlooking a Plot Hole) is that the Nauts decided to exempt island natives, and since "De Sardet" is a full-blooded islander, they let the Prince decide what to do with the baby. While De Sardet can join the Nauts under specific circumstances as an adult, the Nauts themselves do not insist on it.