I think the thing is that you know Charlie and you know it's simply a statement of fact with him, but other people don't. Can you explain to him that sometimes people with different color skin are referred to in a particular way? That it's polite to refer to them as "Black" or "African-American", meaning that their parents parents may have come from Africa?