DedupStats (semantic_dedup.py)

SemanticDedup — near-duplicate content block elimination FusionStage. Detects and eliminates repeated content blocks within a single text using 3-word shingle fingerprinting (no external dependencies). Blocks with Jaccard similarity > 0.8 are considered near-duplicates; only the

# Source: https://github.com/open-compress/claw-compactor | License: MIT | Copyright 2026 OpenClaw Contributors
# Provided via justclaw.it.com; keep this notice with the code.
"""SemanticDedup — near-duplicate content block elimination FusionStage.

Detects and eliminates repeated content blocks within a single text using
3-word shingle fingerprinting (no external dependencies).  Blocks with
Jaccard similarity > 0.8 are considered near-duplicates; only the first
occurrence is kept, later ones are replaced with a compact reference.

Also exposes ``dedup_across_messages`` for cross-message deduplication in
a chat message list.

Part of claw-compactor. License: MIT.
"""
from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import Sequence

from cla

... (truncated -- full source via MCP)