Well they spent most of their time at home. A huge portion of Origin of the Species is about domestic animals (pigeons, dogs, cows, etc.). My impression from reading it was that Darwin’s travels and correspondence just gave him more data for his theory as opposed to sparking it.
I’ve never heard this phrase, but I could see it being used problematically. But islands do tend to be really good places to study evolution, particularly in the tropics.