Abstract base and concrete async crawling strategies using Playwright or raw HTTP for retrieving web content.

Asynchronous web crawler managing browser sessions, crawling strategies, and crawl result aggregation.

High-level Node.js API to control headless Chrome/Firefox via DevTools Protocol or WebDriver BiDi, including page, browser, frame, and network interception primitives.

Defines extraction strategies: LLM, cosine similarity, topic modeling, CSS selectors for crawling content.

Interfaces with VirusTotal API to fetch threat intelligence and reputation data for files, domains, IPs, and URLs.

Models URL data for crawl results, supporting markdown generation via Pydantic.

Wraps Playwright page functionality for browser automation and web scraping with methods for page interaction, data extraction, and element handling.

Wraps Playwright's async element locator to perform browser interactions like clicks and text input with navigation tracking and outcome validation for web scraping.

Finds elements using CSS or XPath selectors with automatic type detection, optional timeout, and configurable error handling.

Generates a JavaScript injection script to spoof Web Worker fingerprinting surfaces like navigator and WebGL, ensuring consistent evasion in worker contexts.

Provides helper classes and functions for version management, robots parsing, HTML sanitization, chunking, concurrency, and system resource queries used across the library.

-*- coding: utf-8 -*- ------------------------------------------------------------------------------- Name: sfp_ripe Purpose: Queries Internet registryes like RIPE (incl. ARIN) to

Shodan API integration for network recon, exposing host, port, and vulnerability intel.

Web capability — `web.fetch` + `web.search`, stdlib-only, allowlist-gated. The first capability that lets an agent read the open web, so egress is default-deny twice over: 1. **Con

Typed configuration objects for browser, proxy, geolocation, deep crawling, and LLM settings with serialization helpers for async crawling.

-*- coding: utf-8 -*- ------------------------------------------------------------------------------- Name: sfp_spider Purpose: SpiderFoot plug-in for spidering sites and returning

Executes arbitrary JavaScript in the browser page context, optionally on a specific element, with extensive control over evaluation parameters. It enables dynamic page manipulation

Transfers URL and prompt data for checking selector cache status in element selection operations.

Central asynchronous crawler coordinating strategies, extraction, chunking, markdown generation, and storage for scalable web crawling.

Locates web elements by combining common HTML attributes (id, class, name, tag, text) into a single flexible method, supporting timeouts and multiple matches.

Factory that creates configured Playwright, Puppeteer, or Cheerio engines with launch context and concurrency settings for scraping tasks.

High‑performance asynchronous URL discovery using Common Crawl, robots.txt, and sitemap streaming with disk caching and global rate limiting.

Parses a User-Agent string into structured browser metadata, including platform, vendor, and version, to enable consistent browser emulation.

Strategy pattern implementation for detecting and extracting HTML tables, supporting default, none, and LLM‑based methods.

Generates IPv4/IPv6 address ranges from CIDR blocks via netaddr, returning expanded IP lists for network targeting.

Applies a browser fingerprint profile to a tab and its workers via CDP overrides and JS injection, ensuring consistent identity across pages, iframes, and workers.

Generates realistic, randomized HTTP headers (User-Agent, Accept-Language, sec-ch-ua, etc.) based on browser/OS/device constraints. Supports HTTP/1 and HTTP/2 header sets.

Sherlock Module This module contains the main logic to search for usernames at social networks.

Fetches a remote page or file and recursively follows links up to a configurable depth, applying optional file extractors to each resource.

Asynchronously crawls URLs listed in a sitemap.xml, optionally converting HTML to plain text, and yields Documents for each page.

108. Browser as required base abstraction to ensure multiple implementations.

Assembles a complete JavaScript injection script that overrides browser fingerprint surfaces CDP cannot control, using a shared native-toString hook and per-section builders.

Upstream Maigret project. 180 files total with comprehensive site database.

Command-line interface entry point for username search across social platforms.

Serializes open shadow roots and resolves slots to produce a complete HTML string of the composed page view for accurate content capture.

Generates random or online user‑agent strings and Sec‑CH‑UA client hints with fallback handling.

JavaScript library that parses HTML documents to extract clean, readable article text, removing clutter and ads.

Manages asynchronous HTTP requests and responses for web scanning tasks.

Cleans HTML by removing comments, scripts, styles, and other non-content tags while trimming attribute values to 100 characters for web scraping processing.

Exception raised when a DOM element cannot be located, signaling missing or stale elements during automation.

Exception raised when a required field cannot be extracted and no default value is provided, ensuring missing data is explicitly handled.

Wrapper around Crawlee's got‑scraping that adds header handling, optional proxy usage, retries, and timeout control for HTTP requests.

Runs after a scheduled scrape, creates snapshots, diffs content, classifies changes, and fires webhook/email notifications.

Core fingerprint generation engine producing browser fingerprints from various input signals.

Sherlock Sites Information Module This module supports storing information about websites. This is the raw data that will be used to search for usernames.

Fetches top scientific papers from PubMed based on a query and returns them as Document objects.

Comprehensive fault injection patterns for OS command injection, directory traversal, XSS, SQL injection, and more. Categorized by attack type and platform.

Helper functions for CSS parsing, heading detection, table handling, and markdown escaping when converting HTML to plain text.

Decorator that injects deep‑crawling capabilities into an async crawler's run method, managing recursion and streaming results.

Splits a CSS selector at iframe boundaries to enable cross-iframe element traversal, returning segment tuples for stepwise lookup.