High temperature (HT) decreases seed set percentage in sorghum (Sorghum bicolor [L.] Moench). The relative sensitivity of pollen and particularly pistil and the mechanistic response that induces tolerance or susceptibility to HT are not well known and hence are the major objectives of this research. The male sterile (ATx399) and fertile (RTx430) lines were exposed to 30/20 °C (optimum temperature), 36/26 °C (HT1 ), and 39/29 °C (HT2 ) from the start of booting to seed set in a controlled environment. Similarly, in the field, HT stress was imposed using heat tents. HT stress decreased pollen germination. Relatively high levels of reactive oxygen species and decreased antioxidant enzyme activity and phospholipid unsaturation were observed in pollen compared to pistil under HT. The severe cell organelle damage was observed in pollen and pistil at 36/26 and 39/29 °C, respectively. The seed set percentage was higher in HT-stressed pistil pollinated with optimum-temperature pollen. Direct and reciprocal crosses indicate that pollen was more sensitive with larger decreases in seed set percentage than pistil under HT stress. The negative impact was greater in pollen than pistil at lower temperatures. Overall, pollen was more sensitive than pistil to HT stress because it is more susceptible to oxidative damage than pistil.