Pairs of n×n matrices whose commutator differ from the identity by a matrix of rank r are used to construct bispectral differential operators with r×r matrix coefficients satisfying the Lax equations of the Matrix KP hierarchy. Moreover, the bispectral involution on these operators has dynamical significance for the spin Calogero particles system whose phase space such pairs represent. In the case r = 1, this reproduces well-known results of Wilson and others from the 1990’s relating (spinless) Calogero-Moser systems to the bispectrality of (scalar) differential operators.