Body size relatively early in life may influence men’s later rate of prostate cancer. We searched for published, English-language studies of the association between prostate cancer incidence or mortality and body size between ages 5 and 29 years. We summarized analyses of childhood body size, early-adult waist circumference, and early-adult body mass index (BMI). Most study designs were case–control or retrospective cohort studies, in which body size was self-reported and recalled. The few studies of childhood body size and early-adult waist circumference indicated null or weak associations. The results from studies of early-adult BMI were heterogeneous (p = 0.04) and showed evidence of funnel plot asymmetry. The random-effects rate ratio (RR) was 1.06 (95% confidence interval [CI]: 0.99, 1.14) per five-unit increase in BMI. Studies using measured (as opposed to self-recalled) height and weight (n = 3) tended to produce stronger associations: fixed-effects summary RR = 1.22 (1.06, 1.39). The same was true for studies that did not adjust for later-life BMI (n = 13): fixed-effects RR = 1.13 (1.06, 1.21). Examining only analyses of advanced or high-grade prostate cancers, results were heterogeneous (p < 0.01). The random-effects summary RR per five units of BMI was 1.01 (95% CI: 0.89, 1.15). In all the reviewed studies, the vast majority of men were of normal weight in childhood and early adulthood. Few studies presented data describing the association between prostate cancer and obesity (e.g., early-adult BMI ≥ 30). The exact relationships between early-life body size and prostate cancer remain unclear but appear to be weak.