Detection of near duplicate documents is an important problem in many data mining and information filtering applications.
When faced with massive quantities of data, traditional techniques relying on direct inter-document similarity computation
are often not feasible given the time and memory performance constraints. On the other hand, fingerprint-based methods, such
as I-Match, while very attractive computationally, can be unstable even to small perturbations of document content, which
causes signature fragmentation. We focus on I-Match and present a randomization-based technique of increasing its signature
stability, with the proposed method consistently outperforming traditional I-Match by as high as 40–60% in terms of the relative
improvement in near-duplicate recall. Importantly, the large gains in detection accuracy are offset by only small increases
in computational requirements. We also address the complimentary problem of spurious matches, which is particularly important
for I-Match when fingerprinting long documents. Our discussion is supported by experiments involving large web-page and email
datasets.
Keywords Information retrieval efficiency - Spam detection