Fxqoazh

The continent can be the sole remnant of a large impact with a less dense celestial body.

Being less dense, the impactor, or what remained on it on the surface of the planet after the impact, did not sank completely into the body, and remained afloat within the boundaries given by gravity to its maximum elevation.

Incidentally, this is one of the proposed explanation of how the continental crust on our own planet formed.

Where did this continent come from or how did it form on a planet that’s almost entirely encompassed by one massive ocean

Continental drift was the theory that the Earth's continents have moved over geologic time relative to each other, thus appearing to have "drifted" across the ocean bed. But before all that happened, all the landmass was one continent - Pangaea. And before the Pangea, there may have been landmasses that came together to form it.

It is fair to assume that your merpeople are living in one such world, where the land actually drifts to form larger masses.

how large a human or humanoid species population could inhabit such a small amount of land

Since your planet is earth-like but with smaller "visible" land mass of 10%, whereas Earth has 30% of visible land mass, you can easily have a third of the population of earth at any point of time on the land. Note that this will vary with advancement of technology in society.

The mer-people live in harmony with the leviathans

You could look for reasons such as domestication, symbiotic relationship, complementary lifestyles, mer-people taste rotten :D, etc to build this up.

You don't have to explain where the continent came from - it's just how the planet formed. Earth had a super-continent - Pangea. The size of the continent is directly proportional to the amount of water the planet has. There is nothing strange in that, it's just natural occurrence. Given the wide variety of exoplanets we've detected, it completely expected that such planets exist.

One concern I would have is how exactly the merpeople breath underwater. Intelligent animals that live in the our oceans such as cetaceans rise to the surface to breathe, and are well adapted to holding their breath for a long time. But actual undersea animals such as fish just don't get enough oxygen from their gill structures to support the energy hungry brains that provide a platform for intelligence.

So perhaps the merpeople have to surface regularly, which of course would mean they need to live near the surface? Perhaps there are naturally occurring air filled caverns replenished by geological processes in which they mostly live? Or perhaps these processes cause massive bubbling throughout the whole ocean and the merpeople's bodies are adapted to capture these bubbles? This could lead to a fun conflict narrative where the land dwellers are mining the ocean and causing a reduction in this bubbling though some sort of fracking process. Or perhaps there is a special plant material that grows all over the ocean surface that eventually sinks, and when they eat it it provides a huge amount of oxygen. Perhaps merpeople, to survive, need to have a backpack full of this food at all time. (Again there is a great conflict narrative there.)

As to my last point, perhaps there is some substance in this plant food that when eaten makes them extremely toxic to the leviathans, and so natural selection has caused the leviathans to not eat or attack merpeople.

As to where the land came from, I think the idea of an asteroid collision creating the land is excellent. And you could tie it together with the substances in the asteroid mixing with sea water to produce the bubble effect I described above. And the land people who need to mine that material for their own survival are put in conflict with the merpeople because their mining damages their air they breath.

It might be interesting too if the giant land mass was in fact composed of a great archipelago of islands so that using the land actually required the land people to use the sea, with, again, the generation of conflict.

Just a few thoughts. Sounds like an interesting world.

1. There was once more land.

Your merpeople were originally land dwellers, which is why they are humanoid. Your planet originally had considerably more land, and water tied up in ice caps. With the gradual warming of the planet (greenhouse gas? sun heating up?) there was melting of the icecaps. The humanoids adapted to wetlands, then shallows, then ultimately became aquatic. There is a theory that some human characterists are a result of partial adaptation to wetland / shallows - the aquatic ape hypothesis. Take that further.

2. Levitathians are adult merpeople
   Your merpeople are juveniles in a prolonged childhood. When they go through puberty they grow and become ravenous - the leviathans. These eat too much to live around the juveniles, who would be outcompeted for food, so they go off and live by themselves (like bull elephants or bull whales). They become hungry and huge, and they fight each other ferociously, but they do not become stupid. They do not eat their own kind.

This is something which would be revealed halfway through the book when your protagonist realizes 1: the leviathans are intelligent and later 2: they are also merpeople.

Actually, it's a minor tweak.

There is a certain, well, mindset that runs around collecting reasons the Earth "can't be" billions of years old. One of their pet claims is the erosion rate of Mt. Everest is some-value-they-quote and then multiply by 1 billion years and get a ridiculous number. They don't keep in mind that Mt. Everest is also growing. So the net height can rise or fall.

If the Earth's tectonic processes were only slightly slower then the amount of land pokiing above the oceans would be substantially smaller. Erosion would carry land to the sea, steadily, bit by bit. Islands would be worn down and disappear. Mountains also would get ground down bit by bit. As they washed into the ocean, the tendency would be for the deeper parts of the ocean to get filled in. This would also, marginally, raise the ocean level as soil disappeared under the waves.

The unusual planetary geography and the Leviathans/Merfolk are the result of a 3rd party, more advanced alien species. While perhaps a bit handwavy, this is a staple of some sci-fi.

Perhaps the island continent was a nasty penal colony at one point, with the Leviathans designed to stop people exploring by boat? Perhaps the entire planet was a research base?

Or this alien species were predominately an aquatic species who removed most of the land mass to increase living space? The leviathans and merfolk are left over security and maintenance biological drones, with the merfolk having increased in sentience subsequently.

Mere-folk being left alone by the leviathans. They could have a physical feature which is highly poisonous or distasteful. Evolution's taught the leviathans not to eat them, or simply disregard them as a natural obstacle.

There have been several valid answers to your mass of dry land. But if you require another, an impact from a large celestial body crushed its way into the crust, creating a cauldron with sides bending high up in the sky. The impact did its usual thing, melting ice caps and glaciers, ending all land locked life, perhaps leaving interesting artifacts of a long gone era spread across the bottom of a now deep sea.

Over the millennia, earthquakes and erotion crumbled the outer walls of the cauldron, making them fall inwards, creating a round valley in which non-nautical life could flourish.

Perhaps the bottom of this valley is actually below sea level, opening up the possibility in your story for submersion due to tunnels or cracks in the outer walls.