TL;DR
- PenAI is a hackathon winning PoC for automated pen testing, built around a RAG pipeline of 224 HTB-style writeups and a separate findings agent.
- The idea was simple, if people are vibe coding apps at ridiculous speed, why not vibe pen test them too?
- Give it a target IP, a VPN file, a scope, and some run settings, and it works through the pen test workflow of recon, enumeration, exploitation, privilege escalation, findings, reporting.
- It still needs the judgement, experience, and business context of a professional tester. Its real potential lies in handling repetitive work, helping testers cover more ground, and providing an initial layer of testing where none currently exists.
Why I built PenAI
PenAI started as a project at a hackathon organised by Encode Club. It’s an AI agent that could work through Hack The Box-style lab machines on its own. Upload a VPN file, give it a target IP, pick a scope, set stealth mode and iteration limits, hit run, and let it get on with it.
I didn’t pick this idea because “AI pen testing” sounds cool (though, fine, it does). I picked it because of a problem.
I think vibe coding is becoming a major concern: Apps, internal tools, SaaS ideas, weekend projects – all getting shipped at a pace that would’ve been unthinkable a couple of years ago. Some of these are experienced developers moving faster with AI. Some are people who would never have built software at all before LLMs made it possible.
Honestly, that’s mostly a good thing. It’s one of the most exciting parts of AI coding tools. The barrier to building software has become a lot lower.
But it comes with an obvious security problem.
When the goal is “ship it,” security rarely gets much of an attention. The app works, the UI looks decent, the login page exists, the API responds, the demo’s ready, the founder’s happy.
But has anyone actually tested it at all? Usually, no. That’s the gap that PenAI was trying to poke at. The motto, only half joking became:
“If you’re vibe coding your app, why not vibe penetration test it too?”
It’s a slightly stupid line, but it’s basically the whole project in one sentence.
Where AI assisted testing could fit
PenAI is currently a hackathon proof of concept, but the experiment points towards several useful applications for this type of technology.
For individuals and small development teams, it could provide an initial security check before something is released. It can identify exposed services, weak configurations, missing hardening, and other common problems that may otherwise go unnoticed.
For professional penetration testers, the more interesting part is how a tool like this could help with the repetitive work.
If AI can take care of some of the repetitive work, testers get more time to focus on the parts that actually need experience and judgement. That includes understanding business logic, recognising unusual attack paths, adapting when the expected route leads nowhere, judging the real impact of a finding, and explaining risk in the context of the organisation being tested.
The first version was deliberately boring
My first instinct was to build the nice looking version. A web interface, a dashboard, live findings, reports, buttons, progress bars and all the stuff that makes a hackathon demo look finished.
Wrong place to start, though. Before making it look good, I needed to know if the core idea even worked. Could an LLM look at a target’s current state, decide on a useful next move, run a tool, understand what came back, update its state, and keep going without falling over?
So I started with something terminal based instead, no UI, no polish, just the loop.
The first job wasn’t really AI at all. It was environment setup. For PenAI, that meant keeping the environment deliberately limited in an authorised lab testing with Hack The Box-style targets, scoped runs, local usage, and controlled conditions. A WSL/Kali-style machine with the usual tooling such as Nmap, netexec, SMB/LDAP tooling, and the rest of the usual kit for working through a lab machine.
It’s tempting to just hand the model a terminal and say “go on then, hack it.” Great for a demo but very unreliable in practice, an LLM with raw shell access will happily invent commands, run the wrong thing, ignore scope, and bury you in messy output.
So early on I decided to avoid unrestricted terminal access wherever I could. PenAI exposes tools through wrappers instead. The model picks a tool and some parameters, but Python controls how the actual command gets built and run.
The agent isn’t working from “run any shell command you want”.
It’s working from something closer to “here’s the phase you’re in, here are the tools available, here’s the target and scope, pick the next useful action”.
When tool wrappers met reality
The wrapper idea worked well for predictable tools.
A basic Nmap scan, for example: expose a wrapper with target, ports, scan type, and timing as parameters, and it builds the command, runs it, captures the output, and hands back something structured.
That’s the dream version, but in reality, it got messy fast.
Sqlmap’s a good example of the problem – depending on what it finds, you can get pages of warnings, prompts, branches, and payload attempts, in a format that shifts depending on what happened. Plenty of other tools do the same. The output structure changes with the flags you passed, the target type, errors, version differences, whether anything interesting turned up at all.
Which makes parsing a genuine pain.
A wrapper’s only useful if it hands the agent something meaningful. Dump the raw output back as a wall of text and the agent’s still stuck interpreting messy terminal noise, the wrapper hasn’t solved anything, it’s just moved the problem one layer over.
Because this was a hackathon project, I didn’t have time to build a perfect parser for every tool.
So I compromised. Some tools kept their proper Python wrappers, others got demoted to shell access. It kept things moving, but it also chipped away at the safety model I’d just built.
It’s probably the clearest gap between a “hackathon project” and “something I’d trust in a serious environment.”
The real problem was context
One of the hardest parts was getting the AI to understand what stage of the test it was in, what it had already tried, what actually mattered, and when to stop chasing a bad idea.
At the start I went with the obvious approach, the same thing I do when I ask AI for pen test help myself, give it the current situation, ask what to do next.
Works fine for about five minutes, then it falls apart, because the model needs way more structure than that. It needs to know when recon is actually done, when to start web enumeration, how to avoid repeating the same scan, whether something counts as a real finding, when to move from enumeration into exploitation, and when a path is going nowhere.
That’s what pushed me towards phase-specific prompts.
Different stages of the workflow got their own prompt: network scanning, web enumeration, Active Directory enumeration, credential enumeration, exploitation, privilege escalation.
The early prompts were basic. The first exploitation prompt was maybe 50 lines. By the end it had grown to 391.
The more autonomous you want the agent to be, the more explicit you have to get. Eventually, the prompt stops being an instruction and turns into an operating manual.
It needs goals, constraints, examples, phase boundaries, stopping conditions, reminders not to repeat itself, rules on what not to do, and guidance on what kind of output is actually useful.
And even that wasn’t enough.
Get the memory wrong and the agent drifts. It forgets what it already tried, repeats scans, rediscovers the same thing twice, suggests a path it already proved useless ten minutes earlier.
So state became a core part of the system.
The run state tracks current target, scope, current phase, discovered hosts, services, credentials, shells, findings, action history, tool outputs, telemetry, and report data.
All of that gets serialised and fed back into the loop before the model picks its next move.
Which turned out to be one of the biggest lessons of the whole project. Autonomous security tooling isn’t really about which model you pick. It’s about what information you feed it, when, and how cleanly.
Poor state, poor behaviour. Noisy state, distracted model. Missing key facts, and the model just fills the gaps itself. Which is a polite way of saying it hallucinates.
RAG: giving the agent old writeups to learn from
One of the recurring problems early on was the agent getting stuck in loops. Chasing the same dead end long after it should’ve given up. Rather than just papering over that with iteration limits, I wanted something that actually addressed why it was happening. I figured, if I’ve seen enough writeups myself to have a rough sense of what to try next, why couldn’t the agent learn the same way? That’s where the RAG idea came from.
I didn’t want the agent relying purely on the model’s general knowledge; it can give generic pen testing advice, sure, but lab machines have patterns that click faster once you’ve seen something similar before. So I built PenAI around a local corpus of 224 HTB writeups, sourced publicly online rather than written by me.
The sourcing itself didn’t take too long, but everything after it did. Each writeup had to be pulled down individually, stripped of everything that wasn’t the actual writeup, then parsed, chunked, and ingested into a local ChromaDB vector database, with metadata like phase, service, tools used, commands, outcome, source, and filename. Repeat that 224 times and it becomes a genuinely labour-intensive job. It’s also worth remembering that having 224 files in a folder isn’t the same as having 224 retrievable documents. Each writeup becomes multiple chunks, so what actually matters is how many good, searchable chunks come back at runtime, and that depends entirely on the parser, the chunking, and whether the index got built properly. One thing still missing is that PenAI’s own reports don’t feed back into the RAG yet. That’s probably the next thing I’d build, using successful runs as new examples for future ones.
During a run, PenAI builds a query from the target’s current state, pulling on whatever’s most relevant to the phase it’s in. Discovered services in web enumeration, CVEs and vulnerable services in exploitation, usernames and loot in privilege escalation. The top results get folded into the agent’s context as a “relevant techniques” block, essentially the agent asking “has anything like this come up before?”
This genuinely helped once the agent moved past generic enumeration and needed more targeted ideas. RAG didn’t fix everything on its own though. Sometimes the retrieved writeup was spot on, sometimes only vaguely similar, and a vaguely similar example can pull the agent toward exactly the wrong path. More context helps, but only if it’s actually relevant, otherwise you’re not removing hallucination, you’re helping create it.
The real theme running under all of this is that the AI part is the easy bit to explain. The plumbing, sourcing, cleaning, chunking, indexing, is where everything actually succeeds or fails.
Two agents: one does the work, one takes notes
Another decision I liked was splitting the main exploitation workflow from the findings workflow.
Splitting the main exploitation workflow from the findings workflow was one decision I’m glad I made early. One agent trying to run tools, read output, choose next steps, take notes, spot findings, and write the report all at once got messy fast, that’s too much for a single loop to handle well.
So PenAI runs a second agent whose only job is watching outputs and pulling out findings: exposed services, credentials, possible vulnerabilities, evidence, affected assets, severity, remediation notes. The main agent just handles the active workflow, deciding what to run next, moving through phases, updating state, and pushing toward the target.
It’s a small architectural choice, but it mirrors how a manual test actually works. You’re never just attacking, you’re constantly taking notes and thinking about what will matter in the report. PenAI splits that same instinct in two, one part asking what to try next, the other asking what’s worth remembering.
Building the dashboard
The first time I reached a root flag, it stopped feeling like a pile of prompts and wrappers. It was still just a proof of concept with plenty of problems, but that screenshot was the moment I thought “Okay, there’s actually something here.” and that led me to fully commit to PenAI.
That was also the point where I stopped being satisfied with a terminal window. Finally, with everything wired in, I moved on to the dashboard. so poking a target with an AI agent didn’t require squinting at a terminal like it’s 2003.
It shows target setup, VPN upload, scope settings, stealth mode, model selection, iteration limits, current phase, live activity, tool actions, findings, telemetry, and report generation. Everything in one place instead of scrolling through logs.
If you want to see how the proof of concept was put together, I’ve shared the PenAI source code on our GitHub. It’s very much a hackathon build rather than production-ready tooling, but it shows the wrappers, phase prompts, RAG setup, dashboard, and findings workflow behind the project.
How it actually performed
Across testing, PenAI reached a root flag on most easy-rated HTB machines, and on a few medium-easy rated ones too. Anything harder than that and it started to struggle, which tracks, those boxes usually need creative pivots and judgment calls the current setup isn’t built for yet.
The repetition problem
The agent would sometimes find a path that looked promising and follow it for far too long. It was often obvious to a human that the path led nowhere, but the agent kept trying variations, re-checking similar ideas, and burning time on a dead end rather than giving up. A web path with zero evidence of being exploitable might get poked at for twenty minutes. A service might get tested repeatedly just because a previous writeup mentioned something similar, even when the current target was clearly different.
RAG made this worse in some cases. A retrieved writeup that looked similar enough could make the agent more confident in exactly the wrong path.
PenAI handles this with configurable iteration limits per phase. Go over the limit, and the phase terminates and moves on. That doesn’t fix the reasoning, but it stops the endless wandering. A smarter version would judge whether recent actions are actually producing new information rather than just counting attempts, but given the time I had, a blunt limit was good enough.
Report generation
A pen test is only useful if the findings actually get documented. You can pop every shell in the world, but if nobody can read what happened afterwards, none of it matters. So report generation isn’t really a bonus feature. Its a must.
PenAI handles this in two stages. While the agent’s actually working, a background findings agent polls the database every few seconds, reads whatever tool output has just come in, sends a truncated version to an LLM, and saves anything worth keeping as a structured finding. That part’s already covered in the two-agents section. This is what happens with that data afterwards.
The actual deliverable is a PDF, built with ReportLab once a run finishes (or on demand from the dashboard). At that point, PenAI pulls together three different sources, the findings the background agent saved during the run, a set of “static” findings computed straight from the run’s state (open ports, flags, credentials, known vulnerabilities), and a few hard-coded rows for things like root shells and dedupes them by title, and builds out a proper report with an executive summary, a findings table, a credentials section, a separate recommendations section (a second LLM call drafts these), a timeline, and a full action log as an appendix.
Severity comes from a mix of sources. LLM judges live findings directly, while certain things are set by rule: a root flag is always critical, an open service is always info-level, and so on. It’s not CVSS-scored, which is the next thing I’d add if I took this further.
There’s no human-in-the-loop step for the report itself, either. Once a run completes, the PDF gets generated automatically. You can delay it and generate manually later, but you still can’t edit anything before it locks in.
It produces something that looks like a pen test report. Treating it as client-ready without a human going through it first would still be an unwise move.
What worked better than expected
The general workflow worked better than I expected. Once the agent had tools, state, phase prompts, RAG context, and enough guardrails, it could move through and retrieve flags from the HTB labs.
Recon and enumeration are full of repetitive decisions, which is exactly where this kind of system starts to earn its keep. It can keep track of open ports, services, web paths, credentials, output snippets, and previous attempts, summarise tool results, suggest what to look at next, and keep a test moving without constantly copy-pasting output into a separate chatbot.
The RAG system helped too. It made the agent less generic by giving it examples from previously solved labs. Not perfect, but it often gave the model better ideas than reasoning from scratch would have.
What broke, or got weird
Repetition was the first big one. The agent could sit on a wrong path for way too long and keep going, find nothing, keep going anyway. That’s the whole reason the iteration limit exists.
Confidence was the second. LLMs are excellent at producing a plausible next step, which isn’t the same as a correct one. PenAI would sometimes over-prioritise noise, under-prioritise something that actually mattered, or suggest a path that sounded reasonable but had nothing to do with the actual target.
Parsing was the third, and probably the most tedious. Tool output is built for humans reading a terminal, not for LLM agents and some tools give short, predictable output, others dump huge, differently-structured output depending on what happened. Miss something in the parser and the state goes incomplete, which weakens the next model call, which weakens the next action. Small gaps compound fast.
And safety was the forth. Any system running security tools needs real care. Scope boundaries, command restrictions, logging, approval controls, even in a lab setting. PenAI has some of this, but it’s still a proof of concept, and I wouldn’t want to oversell it as anything more mature than that.
So, can you vibe pen test?
Kind of.
PenAI showed that an AI agent can do more than run a scanner and summarise the output.
With the right tools, state management, phase specific prompts, RAG, and guardrails, it could work through much of a familiar penetration testing process. But no amount of RAG and phase prompts replaces the judgement, scoping maturity, and business context a real engagement brings.
It also showed where experienced testers continue to add value. PenAI could get stuck, repeat itself, misjudge the significance of a result, and present uncertain conclusions with confidence. A professional tester can recognise when a path is going nowhere, adapt the approach, validate the evidence, and understand what a finding means within the wider business context.
AI assisted penetration testing is likely to become part of how security testing is delivered. I think the real value will come from combining that automation with the judgement and experience of professional testers.
So yes, if you’re going to vibe code it, maybe vibe pen test it too.
Just don’t vibe trust it.
If you want to see how it was put together, I’ve shared the PenAI source code on our GitHub.