2020 will be eventful and likely one of the most historical years containing change, balancing the tension between change, conflict, and hopefully peace in a long time. When reflecting on the changes of 2019, I can’t get away from some of the misnomers that have plagued our niche industry without coming off as a curmudgeon, self-righteous, or an owner of esoteric knowledge too much but I will try my best.
I can’t count how many folks I have encountered at least within the OSINT (Open Source Intelligence) / PAI (Publicly Accessible Information) space who have rallied the message that collection efforts are forever lost, and the landscape is changing far too rapidly. “The industry for collection and analysts will go dark within a matter of months of years.” We even had customers email us telling us APIs of major platforms have changed and things we should know about this.
From our perspective at ShadowDragon, we didn’t see the problems everyone else was having. Sure, a significant platform had changed how folks can engage their API, but this didn’t mean much to us nor impede collection in any form.
During the same period, Michael Bazzell announced Intel Techniques would be removing some of their free online tools from their website due to abuse and needing to keep things updated. The buzz on Twitter and almost every blog echoed the lemmings, and the message that forever was lost and things are changing; the industry will die.
Echo chambers pushed the lament of this downfall, while others asked philosophical questions of if any osint/pai data should be collected. Was it right? Was it wrong? Is the invasion of data privacy at hand?
We mostly sat silently, and sometimes I would nudge others like Nico at Bellingcat, asking if architecture, scale, planning, and collection at scale mattered.
As providence (and proper planning, architecture, etc.) would have it, the changes of [one] platform and the changes of [one] website (Intel Techniques) didn’t change anything for our customers. We increased the number of platforms we collect from (For SocialNet) 9%, 32% increase in collectable actions and increased collection on tailored monitoring and alerting (persistent monitoring in OIMonitor 43%), without losing things everyone else missed.
I empathize with folks that had stated they stayed up three straight days trying to fix things for collection (as if it was a badge of honor), but I have to pose the following questions.
- For the analyst, what are you focusing on? Are you focusing on the analysis, or are you focusing on collection? These two problems are distinctly different, and there is a reason the heavy lifting of collection is harder and will continue to become harder. Should you focus on building collection infrastructures, proprietary frameworks, or adding to an open-source repo because you are hip and cool? Or do you need to be a quiet professional focusing on verifying the information collected, writing better reports, and giving management insights they usually wouldn’t have?
- If you are stuck on collection, and you experienced a bump in the road this year, what questions are you asking? Are you asking the right questions? Is it cheaper for you to build or buy?
A second problem I kept seeing this year revolved around the message:
“Every investigation is different, and you can’t just follow a methodology, and to be a super pro, you just need to wing it…because every investigation is soooo different.”
I couldn’t disagree more; I would argue that this is the voice of someone who hasn’t had to teach a scalable methodology. Likely, this is the analyst that wants to always be in the spotlight and should be challenged dutifully by peers.
We all may use fancy terms like OSINT or PAI, but in the end, an investigation is an investigation. There is a scalable, repeatable process you should follow (and you can define) that helps you focus on the questions you haven’t asked yet.
As an analyst, your primary job is to ensure you are increasing quality assurance, objectivity while holding onto scientific deduction and reasoning skills as you are faced with the need to ask more questions. Just because a process is flexible and differs at intersections doesn’t mean you can’t have a process, and the analyst has always to be the hero in their own story.
We offer a training course around this and full disclosure; we use our tools. Still, our classes at ShadowDragon are about thinking, followed by tips, tricks, and methodologies we have discovered, practiced, and implemented on a vast scale for success. We’ve been doing this for ten years before it was hip, cool, and a github checking was the glory of a budding OSINT researcher.
Platforms will change, data collected will change, but changes in some elements don’t necessarily change your job as an investigator. That job being a chess master, a master of patterns, a master of asking questions, the elephant brain in the room, the quiet professional that can be wrong.
As you grow as an analyst, you will want to maintain focus, verification, and quality output. You are building a relationship with those who read your reports. You are selling trust.
You will need to choose, will you be a tool builder, or are you going to focus on correlation and analysis? The transition has tension. (Trust me, I don’t do code check-ins anymore, nor have I for years at ShadowDragon). Some parts of your ego have to die so you can emerge with more important questions, more patterns identified, and more profound confidence in the work product you produce.
As you look into this year, please pursue your adventures with a different perspective and enjoy the ride. Somethings may change, and some things may be misnomers because you are locked in an echo chamber.
