Monitor Post‑Processor (monitor/MonitorPostProcessor.ts)

AnyCrawl · scraping, Scraping / Crawling, monitor, postprocess, diff, notifications

Runs after a scheduled scrape, creates snapshots, diffs content, classifies changes, and fires webhook/email notifications.

// Source: https://github.com/mperkins0155/AnyCrawl | License: MIT | Copyright 2025 AnyForAI
// Provided via justclaw.it.com; keep this notice with the code.
/**
 * MonitorPostProcessor — runs after a scheduled scrape execution completes
 * successfully. It snapshots content, diffs against the previous run, and fires
 * notifications for meaningful changes.
 *
 * Called from ExecutionLifecycle.finalizeExecution() on the completed branch.
 * Must NEVER throw: wrap all errors and log warnings so the execution lifecycle
 * is never disrupted.
 */

import { randomUUID } from "crypto";
import {
    getDB,
    schemas,
    eq,
    sql,
    getMonitorByScheduledTask,
    getLatestSnapshot,
} from "@anycrawl/db";
import { getJobResults } from "@anycrawl/db";
import { log, config } from "@anycrawl/

... (truncated -- full source via MCP)

See the full source, get the GitHub permalink, and search 40K more like it.

Get a free API key