Web objects are stored in a local cache in order to reduce the cost of future accesses. The cache size is limited, so it is necessary to have a policy for replacing old objects with new ones. This paper examines the problem of constructing optimal or near-optimal replacement policies. Different objects are accessed with different frequencies, and have different retrieval costs when not in the cache. In addition, an object which remains in the cache for a long period may become out of date as its original version is modified. The optimization problem which takes these factors into account is formulated as a Markov decision process. For small cache sizes and small numbers of objects, an optimal replacement policy can be computed by solving the corresponding dynamic programming equation. That solution is compared with a number of heuristics, some of which achieve a near-optimal performance.