FRAX is a tool that identifies individuals with high fracture risk who will benefit from pharmacological treatment of osteoporosis. However, a majority of fractures among elderly occur in people without osteoporosis and most occur after a fall. Our aim was to accurately identify men with a high future risk of fracture, independent of cause. In the population‐based Uppsala Longitudinal Study of Adult Men (ULSAM) and using survival analysis we studied different models' prognostic values (R²) for any fracture and hip fracture within 10 years from age 50 (n = 2322), 60 (n = 1852), 71 (n = 1221), and 82 (n = 526) years. During the total follow‐up period from age 50 years, 897 fractures occurred in 585 individuals. Of these, 281 were hip fractures occurring in 189 individuals. The rates of any fracture were 5.7/1000 person‐years at risk from age 50 years and 25.9/1000 person‐years at risk from age 82 years. Corresponding hip fractures rates were 2.9 and 11.7/1000 person‐years at risk. The FRAX model included all variables in FRAX except bone mineral density. The full model combining FRAX variables, comorbidity, medications, and behavioral factors explained 25% to 45% of all fractures and 80% to 92% of hip fractures, depending on age. The corresponding prognostic values of the FRAX model were 7% to 17% for all fractures and 41% to 60% for hip fractures. Net reclassification improvement (NRI) comparing the full model with the FRAX model ranged between 40% and 53% for any fracture and between 40% and 87% for hip fracture. Within the highest quintile of predicted fracture risk with the full model, one‐third of the men will have a fracture within 10 years after age 71 years and two‐thirds after age 82 years. We conclude that the addition of comorbidity, medication, and behavioral factors to the clinical components of FRAX can substantially improve the ability to identify men at high risk of fracture, especially hip fracture. © 2012 American Society for Bone and Mineral Research.