Dosen't having Confederate flags make you racist...?

So when I was back in school there was this guy who I used to see every once in a while. He had like a Confederate backpack and Confederate bumper stickers all over his truck. He used to stare at me all the time but I thought it was because I'm black/mixed and I thought he hated black people. But just now he's been trying to get with me or something, saying how beautiful I am, calling me, wanting to hang out. But I'm kinda weary of this guy because I thought that Confederate flags and whatnot ment that your racist. Or is it just somepeople? I live in the south so I'm used to this. Help? Sorry for spelling I'm on my phone. Thanks(: