It is argued that mathematics is unreasonably effective in fundamental physics, that this is genuinely mysterious, and that it is best explained by a version of Pythagorean metaphysics. It is shown how this can be reconciled with the fact that mathematics is not always effective in real world applications. The thesis is that physical structure approaches isomorphism with a highly symmetric mathematical structure at very high energy levels, such as would have existed in the early universe. As the universe cooled, its underlying symmetry was broken in a sequence of stages. At each stage, more forces and particles were differentiated, leading to the complexity of the observed world. Remnant structure makes mathematics effective in some real world applications, but not all.