In this paper we discuss the problem of fitting ℓ₁ regularized prediction models in infinite (possibly non-countable) dimensional feature spaces. Our main contributions are: a. Deriving a generalization of ℓ₁ regularization based on measures which can be applied in non-countable feature spaces; b. Proving that the sparsity property of ℓ₁ regularization is maintained in infinite dimensions; c. Devising a path-following algorithm that can generate the set of regularized solutions in “nice” feature spaces; and d. Presenting an example of penalized spline models where this path following algorithm is computationally feasible, and gives encouraging empirical results.