You are dealing with "the Bernoulli Effect". My physicist dad taught me this when I was in scouts.
Basically, with a person standing near a fire, and an imperceptible breeze is blowing, an area of low pressure develops between the fire and person, so smoke heads to the vacuum.
I have fires in my fire pit on the patio often and the only way to avoid the smoke is to stand further back, or get really low to do your cooking. Even in high winds, smoke will head to the person. No matter where the person stands, or the direction of the wind, there will be a low pressure area somewhere around the person. Think about it.