This paper presents a method for estimating position and orientation of multiple robots from a set of azimuth angles of landmarks and other robots which are observed by multiple omnidirectional vision sensors. Our method simultaneously performs self-localization by each robot and reconstruction of a relative configuration between robots. Even if it is impossible to identify correspondence between each index of the observed azimuth angles and those of the robots, our method can reconstruct not only a relative configuration between robots using `triangle and enumeration constraints' but also an absolute one using the knowledge of landmarks in the environment. In order to show the validity of our method, this method is applied to multiple mobile robots each of which has an omnidirectional vision sensor in simulation and the real environment. The experimental results show that the result of our method is more precise and stabler than that of self-localization by each robot and our method can handle the combinatorial explosion problem.