Answering this question depends on whether you value caste, or you are indifferent to it. If you value caste, then this thought can be depressing, even terrifying. If you are indifferent, it will not matter.
In pre-modern times, caste really mattered, as it identified you as part of the community. This community gave you a vocation and a wife, and demanded you follow the community rules, which included giving your daughter only to a member of the caste. It was an extended family. And so anything, which led to loss of caste, acquired great significance.
Baudhayana Dharma-sutra, composed about 2,000 years ago, maybe earlier, lists this “Samudrolanghana” or “Sagarollanghana” as the first of many reasons for loss of castes (II.1.2.2). This especially applied to Brahmins, as there was fear that travel abroad prevented a Brahmin from performing various rites and rituals in the prescribed manner at the prescribed time. The belief was that movement away from the sacred Vedic fire, made one vulnerable to pollution. The contemporary ritual of “aarti” or waving of lamps when one is leaving the house is meant to create a shield to protect against pollution; the same at the time of the return is meant to wipe out all pollutants, and ensure purification.
The irony is that India has a long history of sea travel. Yes, the major epics, Ramayana and Mahabharata, do not refer to sea travel (Ram builds a bridge to go to Lanka, and Ravana flies through the air in his Pushpak Viman), but the vrata-kathas of India like Satyanarayana Puja and the Topoye story of Odisha, and Sanskrit plays like Ratnavali by Harsha, refer to sea-travels and shipwrecks. We do know that sea-merchants travelled from India to Arabia in the West in Harappan times 5,000 years ago. There are Vedic verses that suggest (but not conclusively) awareness of the sea and sea-travel nearly 3000 years ago.
There was definitely a thriving sea-trade to South East Asia in the Gupta Age 1,500 years ago. Sages like Agastya and Kaundinya did travel to faraway lands like Malaysia and Cambodia. Chola kings travelled over the sea to Sri Lanka and Malaysia to expand their empire and to increase the wealth of the land through trade routes. Even today, in Odisha, and in the island of Bali, there are festivals related to the departure and arrival of ships, reminding us of ancient travel over sea. It is this sea travel that ensured epics such as Ramayana and Mahabharata, the art of shadow puppetry and weaving, reached as far as Indonesia and Thailand.
However, in medieval times, roughly 1,000 years ago, after the age of Adi Shankaracharya, the arrival of Muslims and the collapse of Buddhism, we find the rise of an orthodox form of Hinduism that forbade sea travel. Sea trade continued but was outsourced to the Arabs. The caste system became increasingly rigid and pollution was a constant fear. No one knows what caused this shift. Many theories have been proposed but none can be proved: maybe this was a knee-jerk reaction to the violence of the new warlords who came from Central Asia and were breaking temples; maybe it was a way to destroy the merchant class who valued Buddhism and Jainism but not Brahmins; maybe it was to protect Hinduism from being diluted. At one time, this rule was fairly widespread and so strict, that some communities even forbade their members from crossing certain rivers (some scholars argue that in Vedic Sanskrit ”samudra” means a large body of water, not necessarily the sea).
Some argue that sea-travel continued in medieval times, and while “upper” castes shunned sea-travel and outsourced it to the Arabs, the “lower” castes did continue to travel, and it is these “lower” castes who took Hinduism and Buddhism to South East Asia. While some banias in North India refused to cross even the river Indus, others like the Chettiars of Tamil Nadu, did travel abroad to Malaysia and Burma on ships, but they followed strict caste rules like celibacy and dietary restrictions, which included adoration of the celibate Murugan and patronage of Shiva temples in both ports of call.
It is interesting that even the Muslim rulers such as Mughals and Deccan sultans did not set up a navy, but to combat Portuguese might, the Hindu Maratha rulers did establish a navy. But this was about defending the borders of a newly established kingdom, not travelling and trading with another land.
When the Europeans finally wrested control of the sea-trade from the Arabs 500 years ago, a whole new way of thinking came to India. Suddenly, the powerful rulers of India were not men who came on horseback such as the Mughals, but men who came in great ships such as the Portuguese, the French and the English. They wanted Indians to work on their ships. They recruited Indians into their armies, which fought on sea. After slavery was outlawed, they wanted indentured labour from India to work in their farms in faraway colonies in the Caribbean.
The fear of “kalapani” or “black water” of the sea that wipes out caste was at its peak in the 19th century. The East India Company faced a lot of problems with the Brahmins they recruited in their army, who refused to cross the sea. So the worst punishment they came up with, after the 1857 Uprising, was to incarcerate political prisoners in the Cellular Jail at Andaman, across the Bay of Bengal sea, infamous as the kala-pani jail, most feared by Brahmin revolutionaries, as going there meant loss of caste and social excommunication.
People who travelled abroad in the 19th century faced a lot of problems, Raja Rammohan Roy, for example. Swami Vivekananda was criticised, but he took it in his characteristic stride as he spread ideas of Hinduism in America. In the film Man who Knew Infinity, based on the mathematician Ramanujan, we find references to this rule. But as education in England came to be valued, as China became the land for the lucrative opium trade, and job opportunities opened up in America, economic and political reality meant old Brahmin rules had to change.
In the 20th century, people have been largely relaxed about it, thought it does matter in certain circles, to the high priests of Tirupati temple and to the seers of Udupi Krishna temple, and in some Kerala temples, for example, leading to court battles. Like all things Hindu, there always was a way out. There are purification rites (shuddhi), such as chanting certain mantras and fasting, suggested for those who return home. This is acceptable to most orthodox Brahmin families, but not all.