Is it possible that so many nightshades, the witches herbs especially, evolved to produce psychedelic substances in order to immobilize, or confuse a mammal so as to make it an easier catch for a predator? A subsequent feast will yield plenty of high protein detritus as fertilizer.
An interesting idea, but I suspect that you’d be looking at a pretty small selective pressure, since that sort of thing is going to happen so rarely. (And because mammals, etc. would probably evolve to avoid them, so it would be a short term fix.) What I suspect is more likely is that those compounds evolved primarily to prevent other species (i.e. insects) from eating them, and what serves as a poison to insects works as a psychedelic in humans. (Caffeine and nicotine both act as stimulants in us, but can be used as insecticides in higher concentrations, for example.)
There should be somewhat of an exception to insects, because I’m seeing many of the nightshades being a sustenance for ticks and mealybugs, without being damaged.
Maybe larger insects, which I’ve never seen on solanum nigrum, daturas, nicotianas and petunias.
Insects are a vast group, with a lot of diversity. Even milkweeds are food for certain insects.
I have to echo @psweet here. Most plants have chemical or physical protection against insects. Primarily as a deterrent - if it smells, feels or tastes bad, insects won’t eat them. If it works against mammals, it’s a bonus. As mentioned, insects are a diverse group, and something unpleasant to a mammal may taste fine to a specific insect. And without being pedantic, ticks are not plant feeders - both sexes take blood meals, and use plants as a place to wait for a victim.
Then I need to understand the purpose of capsaicin which is further and deeper within pepper fruits. Or is this a case of humans having developed peppers which just turned out to have capsaicin not detectable within the outer millimeter of the skin?
Some insects actually do lay eggs inside fruits, and have their offspring feed on the seeds. (Fig Wasps, and some Weevils, for instance.) But capsaicin might be an exception – birds don’t react to it, only mammals, and the seeds are apparently dispersed by birds.
I can’t be of much help here. Peppers have been cultivated for a long time, and the capsaicin may have been enhanced or eliminated depending on culture and taste. A person familiar with that group - observer, breeder or historian - would have more answers. I find Wikipedia a good place to start - follow links that interest you. Alternatively, Google Scholar can often be helpful, if you know what you are looking for. Good luck!
Regarding peppers, I think I stumbled on something not obvious. Using some of questioning skills from my occupation, I’ve found that people who are proud to have been eating very hot peppers due to their ethnicity, give psycholinguistic response similar to addiction. They have chosen promotional justification for consuming peppers which borders on pseudoscience. For example, the hotness kills parasites and deseases, and is especially helpful in these times of pandemics.
When they were questioned, they were unaware that capsaicin has a targeted effect on lingual heat receptors, and no additional functions. Their response was also similar to individuals who chew tobacco regularly. It might mean that both peppers and tobacco contain biochemicals that humans subconsciously addicted to.
Birds don’t usually care about capsaicin, while mammals feel the effects. There may be a benefit to dispersing seeds by having birds eat the fruit, where smaller mammals are less useful that way. I would expect that you get some variation when you optimize for being tasty to one creature while unpalatable to another.
I’m checking this stuff on Google scholar, as mamestraconfigurata suggested above. The challenge is that it’s not the colloquial friendly googling, one needs to use pure academic English, and some legalese.
Google Scholar can be a bit overwhelming. That can be where Wikipedia is useful. If you can find a scientific paper in their references it may offer clues as to what terminology to use.