Actually,it was used based on appearance.
"High yellow, occasionally simply yellow (dialect: yaller, yeller), is a term for very light-skinned African-Americans and is a reference to the golden yellow skin tone of some mixed-race people."
Pigmentation gradations (along with facial characteristics like nose shape and lip size) have had an insidious and evil effect on and within Black culture which abides.
The wrongs and degradations which Blacks in America deal with throughout their lives, in every aspect of their lives, are not well understood (that is a gross understatement) by the rest of us.
