Diaspora: A scattered population whose origin lies within a smaller geographic locale. Diaspora can also refer specifically to Jewish historical exile. In our analogy, this would be other parties around town where others who share your music taste have gathered.
Refugees: People who have been forced out of their home country and cannot return for fear of persecution because of their race, religion, nationality, or political opinion. These are the people who have left the party due to the teasing.
Asylum Seekers: People who seek international protection but whose claim for refugee status has not yet been determined legally. They're like people waiting outside another party, hoping they'll be let in because they can't go back to the original one.