Amazing and Terrifying: Securing an App Built By AI in a Weekend

A security review of an AI-built partner portal revealed significant gaps in the development process, particularly concerning security awareness among builders. A security engineer reviewed an application built quickly using AI tools and observed that while the AI successfully generated functionality like authentication, it often omitted necessary security considerations such as proper row-level security and least-privilege enforcement. The findings suggest that AI tools optimize for task completion rather than inherent security best practices, leading to unintentional exposure of data within the application.
The review highlighted a breakdown in traditional security checkpoints; speed, enabled by AI development workflows, allows software creation to move through numerous pull requests too quickly for manual human oversight. A key finding was that the platform's internal security scans suggested the application was secure, contrasting with an external expert’s findings that identified real vulnerabilities. This discrepancy prompted a cycle where human review and AI generation iterated rapidly to fix issues before external penetration testing was added as a final layer of verification.
Ultimately, the experience reshaped organizational security approaches, leading to the development of integrated solutions like AI AppGen Security designed to continuously discover and prioritize risks in applications built across various platforms. The ongoing reality involves shifting from episodic reviews to continuous, automated scanning integrated into the development pipeline, alongside recognizing the risk posed by unrecorded, personal access methods for building and deploying software.

* Dudi Peretz, Information Security Engineer, spent over ten years in security review.
* A security review was performed on Orca’s new partner portal built over a weekend using an AI development tool.
* The review involved syncing the platform configuration to the company's GitHub repository for AppSec tooling scans and manual checks.
* Checks included row-level security, least-privilege violations, external visibility, and internal visibility.
* AI-built software typically overlooks security during the development phase because it optimizes for requested tasks rather than anticipating unrequested security requirements.
* Traditional security procedures, like PR checks and separation of administrative roles, are insufficient when building with fast AI workflows involving thousands of pull requests.
* The AI platform's built-in security scan indicated the application was fine, contradicting external security checks performed by Peretz's tools.
* A process emerged where findings were fed back into the AI platform for automated fixes in a rapid cycle.
* Orca launched AI AppGen Security to discover and map risks across AI-generated applications on platforms like Lovable, Supabase, and Claude.
* The operational process now involves continuous scanning synchronized with GitHub for every change.
* A subtler risk identified was builders using personal email accounts, which leaves no organizational record of the tool's existence until a leak occurs.

The narrative demonstrates a fundamental tension between hyper-accelerated development facilitated by AI and the necessary rigor of security governance. The core implication is that when the speed of creation outpaces the capacity for human oversight—whether in traditional or AI-assisted workflows—security naturally degrades, not because intent is malicious, but because context and procedural checks are omitted by default. The system incentivizes doing *what* was asked rather than proactively ensuring security guardrails were *designed* for.
The pattern observed is a systemic failure of trust in automated assurances. When an AI output declares safety, it risks creating a false sense of security that bypasses necessary critical scrutiny. The shift from a single, retrospective audit to continuous, integrated scanning reflects an adaptive response: building security directly into the flow rather than bolting it on afterward. This suggests that resilience in this new landscape depends less on perfect tooling and more on embedding security consciousness into the very definition of the build process.
The focus on opaque access methods, such as personal accounts, highlights a broader challenge concerning identity and accountability in decentralized development environments. The proposed solution—a cross-platform discovery tool—attempts to impose centralized, high-level visibility onto disparate, ephemeral construction methods. The question for human agency is whether this centralization of oversight fundamentally shifts the power dynamic from the individual builder, who exploits the speed advantage, back toward verifiable organizational control. What structures must be established so that automated speed serves collective security rather than creating new vectors for hidden exposure?