Here's a picture of what the inside of the panel looks like from below. The pivot arm in the left of the picture is the air/fan control which is on the left of the heater/vent panel. The arm extending from its pivot point to the right is the wiper for the fan speed. Those black lines wrapped around the ceramic block (not sure how well the picture will show it) make up the variable resistor for the fan speed. Could be that the clip that holds the fan speed wiper in the same relative position to the air control is loose, allowing the wiper to remain on the variable resistor, thus powering the fan.

I acquired that piece separately from my car, so I don't know how to get it out. Merdad will know, I'm sure. In the meantime, suggest you keep the air control to the right or keep the vent open, so the fan isn't working against a closed air door.