Build vs Buy. Deterministic vs Non-Deterministic. Validated vs Disposable.
With GenAI, such as Claude and ChatGPT, investigators now have more options about what tools they use during an investigation. Do they use the tools they’ve used for years, have GenAI write a disposable script, or reach for a new “AI-native” tool?
Organizations should take a systematic approach to making these decisions because GenAI is powerful for some things and a waste of time for others. This post gives you a consistent way to evaluate that choice, task by task, rather than picking a side once and sticking with it.
Changes Caused by GenAI
Investigators have always had tool options, including manual vs automated and commercial vs open source. But, GenAI has changed the options in four ways:
- More Open Source: There are far more free, open source tools on Github that have been built completely by AI with unknown validation. Any one can develop software now.
- Disposable Tools: GenAI will build a tool for you and you are responsible for testing. It may get used only once and it was not written to be maintained.
- AI Native Tools: There are new, commercial tools that are advertised as AI forward and “AI Native”. What does that actually mean for your results?
- AI Decision Making: GenAI can decide for you how to solve a problem, which may include building a brand new tool or using an existing, unvalidated tool. That could go great or terrible.
Investigators now have more decisions about how to approach investigations.
Key Underlying GenAI Concepts
Because everyone is adopting (or rejecting) AI at different levels, let’s cover some basic topics first that will help understand how to pick tools.
GenAI Needs Text. Not Binary Files
GenAI doesn’t know what to do with most binary files. The dominant GenAI models focus mostly on text and a little on images and videos.
They don’t have any built-in ability to process disk images, executables, or other binary files. They rely on external tools to convert the binary data to text. For example, it may download The Sleuth Kit to list and extract files from a disk image. Or, it could instead decide to download another Github project that you’ve never heard of and use that instead to parse NTFS.
Some GenAI clients can reverse engineer binary files by making hex dumps of the files or running other static analysis tools to get text. From there it could make a disposable parser that is intended for only one use. See the DFIR+AI challenge we hosted for an example where Claude reverse engineered a drone (LINK).
GenAI Will Write Programs That Need to be Validated
GenAI will happily make you a program to achieve a goal, such as parsing or analyzing a file. And it can be super gratifying because, within minutes, it will look like the task is near completion.
But, then starts the tedious process of testing. You are now responsible for validating that your custom tool output is correct.
For example, I asked Claude to add a disk imaging feature to our Cyber Triage Collector that would upload the E01 files to S3. Claude said it was complete within 30 minutes. But, we’ve since spent many days testing it and working out the bugs. We’ve delayed releasing the feature until we have enough time to fully test it.
GenAI is Non-Deterministic
By design, GenAI will give you different answers each time. Sometimes they are stylistically different, such as the colors in a report. And sometimes they come to different conclusions, such as the meaning of data.
You need to verify the output of GenAI each time because just because it was correct the day before, it doesn’t mean that it will be correct today. Compare that with deterministic parsers that will always give you the same answer when you give it the same data.
Your Tool Options
During your investigation, you’ll need tools to parse files, search data, and score artifacts. For each task, you now have four basic options:
- Manual: Use minimal technology and rely mostly on the investigator’s knowledge and skills.
- Example: Manually parsing a file system in a hex editor or reviewing every entry in an event log.
- Existing, Validated, Deterministic Tool: Use the “traditional” investigation tools that have years of testing and are used by hundreds in the community.
- Example: File system parsing by The Sleuth Kit or Magnet. Event log scoring from Cyber Triage or Hayabusa.
- Existing, Non-Validated, Deterministic Tool: Use an existing tool that you found on Github, a partner, or that GenAI previously created for you. It has little community use, but also wasn’t created just for this investigation.
- Example: You find a new repo on Github.
- Non-Deterministic Reasoning from GenAI: Use GenAI to do the task and let it decide how. It could write a custom script for you to run or download a validated, open source tool.
- Example: You ask Claude about events and if they are suspicious. Using an agentic AI system.
You now have more “non-validated, deterministic tools” to choose from and the ability to get non-deterministic answers from GenAI.
How to Decide When to Use AI
As you consider the previous AI vs non-AI options, the 5 things to think about are:
- Feasibility: Can the method solve the task? Has someone already created a validated, deterministic tool or is this task new?
- Time: How much total time will be required between when you start thinking about the task to completion? This includes time to:
- Prepare: The time required for building and testing a new tool, installing an existing tool, and training to do it manually.
- Perform: The time required to actually do the work and run the tool or do the manual work.
- Verify: The time required to verify the results are accurate and make sense. If you have a brand new tool or you are looking at GenAI output, you’ll spend more time here then if you are using a validated, deterministic tool.
- Cost: What does it cost to use this method? Does it require a commercial license? Does it require someone with specialized training? Does it require AI tokens?
- Creativity / Reasoning: Some tasks require creativity about how to approach a problem and how to solve it. You need to think about what levels of creativity and reasoning will get brought in by a “typical investigator” or the GenAI. Not all tasks require creativity though. Running a tool to parse a known data structure, such as NTFS, does not require creativity.
- Defensibility: While not relevant to all types of investigations, some will need to undergo a legal or regulatory review of how they reached conclusions. Once you diverge from established methods, you need to defend those decisions more. You need to decide if it is worth it. Presumably, if you spent the right time verifying results, this should not be a technical problem.
Some of these are a bit subjective and imprecise to measure, but they are the basic concepts to think through. Let’s look at some examples.
Examples
Let’s look at a couple of simple examples. We’ll dive into more of these in future posts.
NTFS File System Parsing
If you get a disk image, you need to parse it to get files. Let’s look at how to decide between your options for this well established process.
| Manual | Validated, Deterministic Tool | Non-Validated, Deterministic Tool | Direct GenAI Output | |
| Feasibility | Yes. Buy a copy of File System Forensic analysis and a hex editor. | Yes. Sleuth Kit, Magnet, Encase, etc. | Yes, there are Github parsers from school or side projects. | Yes. GenAI can decide to either download a validated tool, a non-validated tool, or make a custom parser for you. |
| Time: Prepare | High. You need to learn about file system structures. | Low | Low | Unknown: Could take a long time to reverse engineer NTFS. Or it could be quick to download The Sleuth Kit. |
| Time: Perform | High | Avg | Avg | Avg |
| Time: Verify | High | Low | High. You need to make sure it’s correct. | Medium to High: You need to understand what it did. Did it make you a parser that is untested that needs a lot of time to verify. Or did it download a validated tool? |
| Cost | High | Free to Avg | Free to Avg | Tokens |
| Creativity | Not needed | Not needed | Not needed | Not needed |
| Defensibility | Low – Hard to justify why you’d do this. | High | Low – Hard to justify why you’d not use a validated tool. | Low – Hard to justify why you’d not use a validated tool. |
The clear winner here for NTFS is a validated, deterministic tool. They are tested and reliable.
Unknown File Format Parsing
Now, let’s look at a file whose format is unknown and there is no existing parser.
| Manual | Validated, Deterministic Tool | Non-Validated, Deterministic Tool | Direct GenAI Output | |
| Feasibility | Yes. If the person can reverse engineer it. | None | None | Yes. GenAI can reverse engineer it and create a custom parser. |
| Time: Prepare | Super high if you need to manually reverse engineer. | N/A | N/A | High: Time required for GenAI to reverse engineer and test its script. |
| Time: Perform | High | N/A | N/A | Avg |
| Time: Verify | High | N/A | N/A | High. You need to make sure it’s correct. |
| Cost | High | N/A | N/A | Tokens |
| Creativity | Avg | N/A | N/A | Potentially high |
| Defensibility | Avg. | N/A | N/A | Avg |
For this situation, it is much easier to justify using GenAI because its creativity and knowledge may be far better than a typical investigator doing manual reverse engineering.
Hybrid Is the Solution
As promised, GenAI is powerful for some tasks and a waste of time for others. The difference comes down to whether creativity is genuinely required. NTFS parsing doesn’t need it, so a validated tool wins outright. Reverse-engineering an unknown format does, so GenAI provides value.
This means most investigations should use a mix of AI and non-AI methods rather than picking one for everything. Build vs. buy, deterministic vs. non-deterministic, validated vs. disposable are decisions that should be made per task. The five criteria above are how you can make them. We’ll apply this same framework to each investigative phase in follow up posts (planning, collection, scoring, etc.).
Our hybrid approach with Cyber Triage reflects this: we rely on deterministic tools for parsing and initial scoring and bring in AI on those results. Version 4, launching next month, automatically integrates GenAI to add knowledge about artifacts and their use. Current versions expose an MCP server so you can direct AI at the Cyber Triage findings. Try out the latest MCP version here.