I think you're essentially dead on. The concern is the transitional phase. If population collapse leads to collapsing economies and mass civil unrest, there's a very real chance the still-reproductively-healthy cultures are dragged down with the rest. Not to extinction, but to the point if major civilizational regression. Luckily, and perhaps not coincidentally, those cultures you pointed to also happen to be distinctly insular.