Sanity check: traffic lights are not suggestions, right?

This has happened twice in the past week now, and it’s driving me bonkers. Here’s the situation:

– I, a pedestrian, am walking on the sidewalk.
– At a four way intersection with clearly visible traffic signals, I am told to stop (by a red hand).
– A driver on the street I will eventually cross has a green light.
– Driver with green light stops and waves for me to cross.
– I, now a confused pedestrian, point to their green light and my red hand.
– Driver becomes irritated with me, but begrudgingly drives in accordance with the law, albeit while yelling at me.

Am I missing something that makes me the asshole here? Why would I step in front of a car that should be moving?

A few disclaimers:
– The other end of the intersection is empty. This is key as a full/blocked intersection is an instance where green does NOT mean go.
– I am a good 6 feet back from the curb, patiently waiting for my red hand to turn into a white stick figure, and there is no one else around who desires to jaywalk. I cannot be more clear, I am NOT advocating for vehicular homicide.

If you are one of these drivers, I have questions for you. Thank you for your time.