Some mythical animals in history have been found to exist- Saola, Mountain Gorillas, Komodo Dragons and Colossal Squids come to mind. The issue with a hypothetical like this is, once we find something, give it a name, and can physically see it- it loses its mythical status, it just becomes “another animal” that gets placed into the taxonomy of the world we all know. Saola aren’t unicorns- they are bovids, Mountain Gorillas are placed under Hominidae just like Chimps and humans, Komodo dragons are monitor lizards, etc.
Go back a century or two and all of these species, and many more would be considered mythical. All the mythical species with multiple characteristics from different orders is where things would get tricky.
I feel like mermaids wouldn’t have any humanoid/primate DNA at all, but would have developed the human “half” as a form of mimicry/camouflage in order to better ambush/defend themselves. Like how several reptiles, fish, and amphibians develope those fake-worm appendages to lure in prey, or like how certain animals camouflage themselves as deadlier animals to better survive, like cheetah cubs looking like honey badgers and those moths/butterflies with faces on their wings.
I don’t know which taxon I’d place them, mostly bc I am woefully ignorant in far too many, but they’d most likely belong in similar taxons as Angler Fish or maybe Frilled Sharks… Of course, they could be mammalian and so end up taxon-adjacesnt with Dolphins/Orcas for the intelligence needed to set their ambush traps.