If you've only read the first book (or only watched the show), I can sort of see how you'd make that statement, but in the lands outside of Westeros, there are dark-skinned people who are wise, spiritually enlightened, and better spoken than the bulk of the 'Northern European' folks presently on the show.

Further, it makes sense that a group of barbaric nomads who ride horseback on the plains would have darker pigment than those who spend their lives at court in a castle. That part isn't fantasy, it's evolution.