Automating OSINT: 7 Steps to an Intelligence Workflow

According to the latest research published by IMARC Group, the global open- source intelligence market reached a valuation of approximately $16.7 billion in 2025. Forecast to grow at a CAGR of 26.7% through 2035, this growth rate reflects the expansion in public data and the challenges intelligence teams face in turning it into decisions.

By the time an analyst completes an exhaustive person-of-interest deep dive of social sources, court documents, breach dumps, dark web forums, certificate transparency logs and geolocation intelligence, the target’s digital activity may have changed. That disconnect between collection cadence and decision cadence is the challenge automating open source intelligence (OSINT) seeks to address.

However, automation doesn’t mean writing and running a Python script. An automated intelligence workflow is a process by which you take publicly available data “as-is” from its original sources and transform it into structured, decision-ready intelligence with minimal manual intervention that maintains legal review.

It must have requirements, sources, a tech stack, a collection layer, an enrichment layer, a monitoring layer and a governance layer. Leave any of them out and you don’t have a comprehensive process. You have a workflow that will sooner or later break, risk poor decision making, or get pulled into discovery.

Here is our seven step framework. Each step represents what an enterprise intelligence team must develop, in sequential order, to progress from ad-hoc collection to a repeatable process. Compare your present investigative process to this framework and the missing pieces will become clear.

1. Define your intelligence requirements (PIRs)
2. Map your OSINT data sources
3. Choose your automation stack (build vs buy)
4. Automate collection
5. Enrich, correlate and resolve identities
6. Set up continuous monitoring and alerting
7. Govern (ethics, compliance and audit)

Priority Intelligence Requirements are the first artifact you’ll develop when building a working OSINT workflow. They exist for one reason: automation leverages whatever you point it towards. Consider the stakes: In 2024, the U.S. Federal Trade Commission logged $12.5 billion in consumer fraud losses. That’s a 25% increase over 2023. U.S. lenders held an additional $3.3 billion in synthetic-identity exposure on new accounts through 2024 (Federal Reserve / FedPayments Improvement).

Train an automated system to “watch for fraud” and you generate noise against statistics like those above. Point it at tightly scoped PIRs and you will actually move the needle.

A PIR answers four questions for every line of collection:

• What decision is this intelligence supporting?
• Who is the customer?
• What entities and time horizons are relevant?
• What is explicitly out of scope?

Taken from military intelligence doctrine (U.S. DoD Joint Publication 2-0) and implemented by law enforcement, government and corporate security teams, PIRs transform leadership’s nebulous “watch for threats” into a task an automation system can perform.

Here are three examples from real-world use cases:

• Fraud team tracking synthetic identities. PIR: identify newly created accounts (30 days old) that share device fingerprint, infrastructure or behavioral pattern with five known fraud rings. Consumer: fraud operations manager. Out of scope: accounts older than 30 days. Context: The Federal Reserve Bank of Boston (April 2025) identified generative AI as the primary driver behind synthetic identity fraud.
• Corporate security monitoring threats against executives. PIR: Identify any direct or inferred threats of violence, doxxing attempts, or organized harassment campaigns directed at members of the C-suite from surface social, forums, and paste sites. Consumer: Director of Executive Protection. Out of scope: brand reputation or general sentiment.
• Law enforcement building case packages. PIR: list known accounts, aliases and associations for a named individual, with comprehensive source lineage. Consumer: investigating detective. Out of scope: generating risk scores.

There’s a hard line here. If a request doesn’t contain a named consumer, a stated decision and an out of scope statement, it isn’t a PIR.

It is a fishing expedition. When fishing expeditions are automated, they become costly undertakings.

A comprehensive OSINT source map has seven categories: social media, deep and dark web, public records, court and government filings, technical OSINT, media/news and proprietary feeds. This taxonomy was originally created by Justin Nordine for osintframework.com back in the mid-2010s. The framework ShadowDragon® teams use day-to-day, detailed in our OSINT Framework guide, expands upon the framework for enterprise sourcing.

The trap that too many teams succumb to is “collect everything.” Automation enables this obsession: disk storage is cheap, there are limitless sources, and every category seems useful on a slide deck.

However, every source you add means more maintenance, more legal exposure, and more alerts to manage.

Each source you take on commits you to four things: a refresh cadence, an access method (vendor API, partner feed, manual, etc.), a reliability tier, and the PIRs it supports.

The map should be treated as a living document. Sources decay (APIs get deprecated, sites institute paywalls, jurisdictions revise laws,) and a workflow that doesn’t constantly weed out obsolete sources gradually accumulates stale data you can’t trust.

Build-vs-buy in OSINT really only offers three potential paths: DIY scripting, commercial OSINT platform, or a hybrid of the two. The honest answer for most enterprise teams will be hybrid for one simple reason: operational economics.

According to the SANS 2025 SOC Survey, 66% of security operations teams are overwhelmed by alerts/signals. pure- DIY workflows exacerbate this exact point of failure. Source maintenance, ToS churn and orchestration breakage all fall to the same one or two engineers. This creates a perpetual maintenance burden that uses resources but does not yield operational leverage.

Working with intel teams in law enforcement, government and financial-services fraud, we find the hybrid model comes out ahead because it allows you to focus engineering effort on the things that actually compound: proprietary enrichment, casework integrations and edge sources. Vendor responsibility handles the static work (social platform API revs, ToS churn, dark-web crawler health) that tends to bog down DIY teams.

Leverage a commercial platform for categories where vendor’s scale of collection, compliance posture and source maintenance give them a definitive edge. Spend development effort on custom enrichment, correlation and casework integrations that are truly unique to your organization.

Don’t use free-tools lists or open-source repositories for your production layer. Reserve them for prototyping and non-primary sources.

An operational OSINT collection layer has scheduled and event-driven orchestration, observes ToS-friendly rate limiting, unattributed search infrastructure, and captures per-record lineage metadata. Some of these design decisions separate the hobbyist from the professional. The differences become apparent when your first finding is challenged in court or a compliance audit.

The same Joint Publication 2-0 (Department of Defense) doctrine that scopes PIRs in Step 1 also scopes collection ethics: lawful, attributable, traceable.

Orchestration. Automation of scheduled jobs is well-suited to sources that push updates on predictable schedules (think daily records dumps, certificate transparency logs). Event-driven collection (initiated by an alert, an analyst request or an upstream ping) is what scales person-of-interest investigations, because requests to “watch this target” do not conform to a cron schedule.

Rate limits and ethical access. Each collection job should honor platform ToS, source rate limiting and rules of jurisdiction(s) it operates in. This isn’t only about staying compliant or avoiding legal problems.

Large-scale automated collection leads to blocks, sabotages reliability and jeopardizes the whole operation. ShadowDragon®’s approach (no credential abuse, only public data, and no aggressive automated data collection) is the bare minimum an enterprise workflow should strive for if defensibility is a goal.

Anonymous searches. Searching a hostile forum or your target’s digital breadcrumbs will expose your intentions to that target. The search itself is an OPSEC incident.

Requests need to travel through managed identity infrastructure so the analyst’s identity or intent won’t be revealed to the target. That’s where engineering and proper OSINT practices intersect.

Nuances by source. Social APIs rate limits shift every few months. Tor-network crawling requires robust session handling.

These are examples of failure modes that are specific to each source category. The collection layer handles them so that your downstream analysts will never know they happened.

A working collection tier validates all of the following:

• Idempotent jobs (running it twice does not corrupt your dataset)
• Transient failure retry with exponential backoff
• Dead-letter handling so permanent failures are surfaced, not ignored silently
• Per-record lineage metadata: where it came from, when we collected it, what collector version retrieved it, what hash it was known by, etc.
• Alerting on collector health and not just intelligence content

Fail on any of these points, and you’ll be unable to answer when leadership inevitably asks about the data’s origin and collection time.

Identity resolution is by far the most important automation layer to build into an OSINT workflow. Without this crucial step, analysts find themselves bogged down in data integration and disambiguation for most of their day, rather than focusing on actual analysis.

Data preparation consumes the most time for analysts. The 2020 State of Data Science report from Anaconda found that analysts spend about 45% of their time on data prep. The 2024 State of Analytics Engineering report by dbt Labs showed that over half of practitioners consider organizing datasets to be their #1 time-consuming task. Poor data quality was reported to be a bottleneck by 57% of professionals, up from 41% just two years earlier in 2022. And yet, even as generative AI tools proliferate, the most recent large-scale survey of data analysts found that 76% rely on manual, spreadsheet-based workflows to prepare data. That report, Alteryx’s 2025 State of Data Analysts in the Age of AI, surveyed 1,400 analysts worldwide.

In working with identity-intelligence deployments we’ve found that teams embracing automation for enrichment reduce their per case cycle time by a factor of 10. Teams that fail to automate simply shift the bottleneck.

Entity resolution is the fundamental challenge. One individual may be registered as a username on one service, an email address on another, a phone number in a breach database, and a name associated with court filings in public records.

If your workflow cannot connect those dots, you are dropping your highest-value signal. Resolution fuses deterministic matching (precise hashes, precise identifiers) with probabilistic matching (behavioral fingerprints, linguistic patterns, network co-occurrence), and the resulting confidence score travels with the record for an analyst to decide how heavily to weigh it.

Enrichment is where context is applied. After resolution, enrichment links the associated evidence around an entity: related accounts, network contacts, posted content, technical attributes, activity history. The result is not just “a record.” It’s an individual, location or asset with a body of supporting evidence you can defend.

Link analysis is the visualization and graph layer on top: relationships rendered as a network the analyst can navigate, expand and prune. The graph is the artifact that enables human investigators to turn three thousand records into a decision in twenty minutes.

If a team automates collection but not enrichment, they’ve simply moved the bottleneck elsewhere. Analysts will spend less time gathering and more time stitching with the same end-to-end cycle time.

Continuous OSINT monitoring requires four foundational elements (watchlists, change detection, severity-scored alerts and routing), and alert fatigue is the toughest design constraint working against you in 2026. The recently published SANS 2025 SOC Survey reports 66% of SOCs are drowning in alerts, unable to keep pace. Cybersecurity Dive recently reported on an IDC/ FireEye survey that found when flooded, more than one third of analysts admit to ignoring alerts altogether.

A monitoring layer that creates those failure modes is a liability, not an asset. A point-in-time investigation answers the question ‘who is this person right now?’ A monitoring layer tells you when something about that person changes.

They are two different products, and the monitoring layer is what turns an analyst’s case from a one-time deliverable into an ongoing service.

The four core components of monitoring:

• Watchlists: what entities/keywords/signatures we’re watching.
• Change detection: differencing against a previously known state for each thing we watch so that new posts, new links, new infrastructure or new mentions bubble up for downstream consumption.
• Severity-scored alerts: every alert has a severity score so the consumer can triage, suppress or escalate.
• Routing: alerts appear where action happens (SOC consoles, casework queues, fraud ticket queues, exec-protection dashboards) instead of a shared inbox that no one pays attention to.

The solution is two-fold: severity scoring and alert-aware suppression. Dedupe identical alerts. Suppress alerts that don’t meet a tunable threshold.

Roll up correlated alerts into single incidents with underlying records attached. The result is fewer, better alerts that each warrant the analyst clicking through.

Governance is what makes an automated OSINT workflow either a justifiable enterprise capability or just an expensive liability with a dashboard. This is the stage where most workflows fail.

Governance certifications aren’t optional anymore in 2026: SOC 2 Type II (based on the AICPA 2017 framework with ongoing trust-services updates) is critical for any vendor in this stack. Regulated environments will layer on additional frameworks: CJIS (FBI, updated 2024), FedRAMP (GSA, Rev. 5 baseline) and ISO/IEC 27001:2022.

Failure presents itself when you least want it to, like in front of opposing counsel, an internal audit, or a regulator.

There are four pillars:

• Lawful access. Public sources only. No abuse of credentials. No automated data extraction that violates ToS or jurisdictional law. Document the legal rationale for each source on the source map.
• Privacy and minimization. Collect only what your PIRs need. Retain only as long as justified by the use case. Build deletion workflows into your systems. The default posture should be LESS data, not more.
• Attribution and OPSEC. Unattributed search infrastructure, identity rotation, and managed personas where justified by the use case. The collection layer should protect the analyst and the operation.
• Audit and defensibility. End-to-end lineage for every finding: Which source? Which collector? Which enrichment rule? Which analyst verified it? If you can’t recreate a finding six months later, it’s not defensible.

The important thing is the operational discipline that the certificate represents. The same operational discipline is necessary whether your workflow is fully built, fully bought or hybrid.

For teams that scale up an investigator workforce as they scale the workflow, the governance layer is also home to analyst training.

Top 5 anti-patterns that kill more OSINT workflows than any technology decision:

• Automating without PIRs. You end up collecting everything for nothing. The cost is spread to every layer downstream.
• Making automation the objective. Automation is how you get there. The objective is decision cadence. Teams that lose focus on that tend to build beautiful workflows that never materially decrease investigation time.
• Neglecting the governance layer. This is the easiest layer to omit, and the most expensive to add after the fact. Bake it into the process from day one, or risk paying 5x for it later.
• Failing to account for source decay. Data sources evolve silently. A workflow not actively watching source health will gradually become a warehouse of unusable records.
• Omitting the analyst in the loop. You cannot automate judgment. If you do full automation on sensitive topics, you accept legal and ethical risk. Build your workflow around the analysts’ judgment. Don’t expect it to replace their judgment.

The seven-step framework is a planning artifact. Virtually every team has components of it already built. Most teams have some collection. Most have some sources mapped. Most have had an analyst run scripts.

The smart first move is not to construct a complete workflow. It’s to audit where you are against the seven steps and figure out what your two weakest layers are. Those are where you should invest.

For most teams, the weakest layers are typically governance and enrichment. Governance because it’s easy to put off until tomorrow and costly to bolt on at the end.

Enrichment because it’s the layer that truly dictates whether automating intelligence actually reduces an analyst’s cycle time or merely moves the bottleneck further down the line. Resolve those two layers and watch the rest of your investigative workflow fall into place.

Want to see what an accelerated intelligence workflow looks like performing in production?  Chat with an OSINT expert or learn how the Horizon® Platform can improve your OSINT workflow.