The bottleneck for many parallel and distributed applications on networks of workstations is the high cost of communication on traditional network interfaces. Memory-mapped network interfaces provide latencies of a few microseconds and bandwidths close to the maximum of the local I/O bus. A major drawback with the current operating system support is that applications have to deal directly with data consistency and the allocation and placement of pinned physical memory. We propose a more elaborate interface, called SciOS, that provides a shared-memory abstraction where physical memory is treated like in a NUMA architecture. To lower the average memory access times, we use a relaxed memory model and dynamic page migration and replication combined with use of idle remote memory instead of disk swap. We describe the SciOS programming model and describe the issues for its implementation on an SCI cluster.