Osteoporosis‐related fractures constitute a major health concern not only in women but also in men. To study the predictive role of serum sex steroids for fracture risk in men, serum sex steroids were analyzed by the specific gas chromatography‐mass spectrometry technique at baseline in older men (n = 2639; mean, 75 yr of age) of the prospective population‐based MrOS Sweden cohort. Fractures occurring after baseline were validated (average follow‐up of 3.3 yr). The incidence for having at least one validated fracture after baseline was 20.9/1000 person‐years. Estradiol (E2; hazard ratio [HR] per SD decrease, 1.34; 95% CI, 1.22–1.49), free estradiol (fE2; HR per SD decrease, 1.41; 95% CI, 1.28–1.55), testosterone (T; HR per SD decrease, 1.27; 95% CI, 1.16–1.39), and free testosterone (fT; HR per SD decrease, 1.32; 95% CI, 1.21–1.44) were all inversely, whereas sex hormone–binding globulin (SHBG; HR per SD increase, 1.41; 95% CI, 1.22–1.63) was directly related to fracture risk. Multivariable proportional hazards regression models, adjusted for age, suggested that fE2 and SHBG (p < 0.001), but not fT, were independently associated with fracture risk. Further subanalyses of fracture type showed that fE2 was inversely associated with clinical vertebral fractures (HR per SD decrease, 1.57; 95% CI, 1.36–1.80), nonvertebral osteoporosis fractures (HR per SD decrease, 1.42; 95% CI, 1.23–1.65), and hip fractures (HR per SD decrease, 1.44; 95% CI, 1.18–1.76). The inverse relation between serum E2 and fracture risk was nonlinear with a strong relation <16 pg/ml for E2 and 0.3 pg/ml for fE2. In conclusion, older Swedish men with low serum E2 and high SHBG levels have an increased risk of fractures.