Our DFIR+AI Challenge is complete, and our judges awarded the winners. This post provides a summary of the challenge.
Challenge Recap
The goal of this challenge was to see how people are using AI in DFIR and share the best and worst ways of using it. My goal was to keep it simple and make submissions with low effort.
People had to submit:
- Tools they used.
- Why they thought it was a success or failure.
- Screenshots.
The plan was to have public voting, and we’d award a best success and a best disaster.
Overview of the Submissions
We got 12 complete submissions. You can find them on the GitHub repo.
I want to first say Thank You to everyone who submitted! It can be hard to find the courage to submit something when the best practices of this technology are so undefined.
We learned as we reviewed them that we didn’t ask for enough information to fairly evaluate them!
- It was hard to know what AI did versus the underlying parsing tools.
- It was hard to know if the results in the screenshots were accurate.
- Sometimes we picked up on things in the output that the submitter didn’t.
To truly get precise answers on the value of AI, we needed more information. Some submissions were both successes and failures in different dimensions. So, we pivoted from our plan.
Instead of public voting and a single winner for success and failure, we made different categories.
Recap of Some Submissions
Here are some of the notable submissions and the discussions from the webinar we held (you can watch the webinar here)
- #2 from Naveen Hariharan Vijaya: This submission used a SANS memory image, Cyber Triage MCP server, and Claude. A very thorough prompt was used, and several AI-generated reports were generated.
- Judges like the prompt used.
- This was a classic example of where it was hard to know what Claude did versus Cyber Triage versus MemProcFs or Volatility (which Cyber Triage integrates) to know the value of AI alone.
- One of the judges mentioned that lots of the IOCs came from the same offset, but it wasn’t clear if that was OK, if AI caused that, or if that came out of the underlying tools.
- #6 from Matti Saarelma: This submission used the built-in Gemini within Timesketch. They validated the results and pointed out that Gemini assumed that “DC” referred to a domain controller instead of the XP recycle bin.
- This was a great example of the user validating the results.
- It’s also a great example of how, without much context, any AI will guess at the context and come to the wrong conclusion.
- This was a good example of how it was hard to judge because we don’t know how many other submissions had hallucinations and the submitter didn’t notice.
- #16 From nhantouli: This submission was from an undergraduate project, and they wrote their own MCP server for Autopsy before we made one, so it has different tools.
- This is an example of a thorough analysis of AI and its pros and cons. The judges appreciated the honesty and transparency.
- This provided an interesting example where the user asked about an OS time, and there was no Autopsy artifact for that. So, the AI proceeded to parse the registry to find it. This is a powerful example of how AI can find data even if another tool hasn’t found it.
- This data is probably in the Autopsy-generated RegRipper report, but that may not have been exposed via MCP.
- #11 from Maryam: This submission was one of the most interesting. They gave it four files from a NIST drone data set (there were over 80 relevant files on the drone). Claude proceeded to reverse-engineer the files and make a parser.
- This concept has super interesting potential because it gives capabilities to an analyst when they have a file format that can’t be parsed.
- This scenario, though, went bad because Claude got the parser wrong. It placed the drone over the Strait of Hormuz and explained away other errors from GPS jamming. The NIST data sets were not flown over the Strait of Hormuz.
- BUT, I loaded the full data set into Claude, and with the additional data, it got it correct!
- This was a great example of awesome potential, but it’s super important to verify the results.
- #15 from Aleem ladha and #8 from Serhiy B: These were “vibe coded” apps written to do investigations. The #15 submission has several screenshots of the numerous reports that its 50 agents created. The #8 submission focused more on the internals of the tool.
- Allowing more people to write programs is going to be a game changer in all industries.
- But, you still need to test and validate the programs. Writing is the easy part.
- We couldn’t tell from these submissions how much they relied on existing command-line tools that are well -ested and established versus how much they vibe-coded file system and file parsers. The more “new” code is written, the more verification is needed.
- We couldn’t tell if their results were any different from other approaches that have less untested software. Again, a limitation of what data we asked people to submit.
- #5 from Flip Forensics: They wrote their own tool and tried to see if they could trick the LLM by adding a file name into the prompt that didn’t exist.
- Ultimately, the response mentioned the file name but didn’t include it in a timeline.
- The approach of adding false data to prompts is an interesting approach to testing LLMs to see what they ignore versus what they focus on.
- #10 from Pier Noel Asuncion: They used the Cyber Triage MCP server and a NIST data set.
- One interesting part of this submission is how hard it is to know how to validate AI results.
- The authors did a good job of trying to validate the AI, and they gave it a JSON to confirm, but the JSON also came from Cyber Triage. So, ultimately, they were comparing the Cyber Triage MCP versus Cyber Triage JSON.
- Neat findings, but it shows it’s hard to validate.
Winners
- Best validation: Matti Saarelma (#6) (DC = Domain Controller)
- Best rigorous evaluation: nhantouli (#16) (Custom Autopsy MCP)
- Best vibe coded: Aleem ladha (#15) (50 AI Agents)
- Best enabler & cautionary tale: Maryam (#11) (Drone Reverse Engineering)
Congrats to them and thanks again to the judges and all who submitted. We’re thinking about doing this again in 6 months as everyone has played more with AI in their investigations.