I wouldn't be so worried about non-native English speakers. 'Whitelist' and 'blacklist' are common enough terms that they've become loanwords in other languages. In Chinese, at least, the equivalent terms are 白名单 'white name list' and 黑名单 'black name list' and 拉黑 'pull (into) black(list)' is common vernacular for blocking someone on social media.