Cognitive representations of spatial object relations are known to be affected by spatial as well as nonspatial characteristics of the stimulus configuration. We review findings from our lab suggesting that at least part of the effects of nonspatial factors originate already in perception and, hence, reflect principles of perceptual rather than memory organization. Moreover, we present evidence that action related factors can also affect the organization of spatial information in perception and memory. A theoretical account of these effects is proposed, which assumes that cognitive object representations integrate spatial and nonspatial stimulus information as well as information about object related actions.