We recently provided evidence for a mitochondrial dysfunction in P301L tau transgenic mice, a strain modeling the tau pathology of Alzheimer’s disease (AD) and frontotemporal dementia (FTD). In addition to tau aggregates, the AD brain is further characterized by Aβ peptide-containing plaques. When we addressed the role of Aβ, this indicated a synergistic action of tau and Aβ pathology on the mitochondria. In the present study, we compared the toxicity of different Aβ42 conformations in light of recent studies suggesting that oligomeric rather than fibrillar Aβ might be the actual toxic species. Interestingly, both oligomeric and fibrillar, but not disaggregated (mainly monomeric) Aβ42 caused a decreased mitochondrial membrane potential in cortical brain cells obtained from FTD P301L tau transgenic mice. This was not observed with cerebellar preparations indicating selective vulnerability of cortical neurons. Furthermore, we found reductions in state 3 respiration, the respiratory control ratio, and uncoupled respiration when incubating P301L tau mitochondria either with oligomeric or fibrillar preparations of Aβ42. Finally, we found that aging specifically increased the sensitivity of mitochondria to oligomeric Aβ42 damage indicating that oligomeric and fibrillar Aβ42 are both toxic, but exert different degrees of toxicity.