In this Help Net Security interview, Erika Dean, CISO at Tricentis, talks about keeping production data out of test environments and why she thinks the alternatives are good enough now. She explains how her team caught a prompt injection gap in red-teaming and held a release for a week until it was fixed.
Dean describes what gets an AI vendor rejected, mostly vague answers about where data lives and how long it is kept. She also lists three things a small security team should build first, even with limited headcount.
You own enterprise security, compliance, and product security. Which of the three do you personally spend the least time on, and why is that probably the wrong allocation?
If I’m honest, governance and compliance is where I personally spend the least hands-on time day-to-day and I don’t think that’s the wrong allocation given where the threat landscape sits right now.
Compliance is still a fundamental part of our security program and always will be, especially since Tricentis is a customer-facing company and our customers carry their own compliance obligations. What’s changed is how we run it: we’ve applied automation to evidence gathering and audits and built strong self-service elements into the program, which reduces the day-to-day overhead without reducing its importance.
That’s what frees me to spend more time on enterprise and product security, where my team deploys and manages the technical controls that keep Tricentis and our customers safe. As threat actors use AI to exploit vulnerabilities faster and deploy new attack techniques, that’s where the highest-leverage time goes right now.
Test environments are notorious for holding copies of production data with none of the production controls. How do you handle that internally, and what do you tell a customer you can see doing it badly?
Internally, I have the same requirements as I recommend to our customers, don’t use prod data in lesser environments. From a career standpoint, I’ve seen multiple companies leverage production data in QA or non-production environments. For example, historically, it has been hard to perform load testing for things like check images and SSN validation processes, as these must be very accurate and have unique characteristics, so companies have defaulted to using production data. Even in high-stakes industries like financial services and healthcare, firms sometimes use production data.
The controls within a QA environment are sometimes not as strong as they are in production, and you want to make sure you keep that segmentation. As such, I always recommend keeping production data out of QA and testing environments as much as possible and find alternate ways to test. The technology has come a long way and there is almost always another way to solve for the challenges.
Every vendor’s agent story sounds alike right now. Tell me about an AI capability your team blocked or delayed, and the specific technical concern behind it.
There were certainly AI products and capabilities that I needed to delay or force a change in direction throughout my career. This was especially the case in the early days of AI, when companies were scrambling to implement it using agentic capabilities. I can’t name the specific product, but I can walk through the pattern: as companies rushed to make chatbots sound more human and engaging through agentic capabilities, my team flagged the risk of prompt injection, like an attacker crafting inputs designed to extract confidential information the system was never meant to expose.
We ran it through red-teaming before launch, found exactly that gap, and held the release until it was closed on the backend. It added a week to the timeline, but the alternative was a data leak. As such, delaying launches until all security testing is completed, and any bugs resolved prior to implementation is necessary today, just as much as it was in the early phases of Agentic AI. My stance is generally to help move engineering and product development quickly but not sacrifice security.
How do you evaluate the model providers and AI vendors embedded in your own stack? Describe one you rejected and what disqualified them.
We evaluate model providers and AI vendors through a security lens first to determine if we are comfortable using them internally or in our products. Some of the key things we evaluate is: How is our data being used and stored, what are the security controls they deploy, what does data retention look like, and how resilient is the vendor.
We also look at the architecture and how it will be implemented within our ecosystem including how we want our users to interact with the technology. Most companies are incorporating AI in some form and all of those must be evaluated against our standards as well before we can enable the use by our teams. The best AI vendors allow for enterprises to configure the use of the LLMs and restrict agents from an enterprise perspective. For example, enabling the use of a coding assistant, we would look to deploy on our own stack and utilize enterprise controls to ensure the prompts are running through our security checks before being executed by the AI vendor.
I can’t name a specific vendor we’ve rejected, but I can share what disqualifies one. The most common red flag is a vendor that can’t answer where our data lives, how long it’s retained, and whether it’s used to train their models, if we can’t get a straight answer on data residency and retention, that’s a hard stop regardless of how good the product is. The second is architecture: whether we’re going directly to a provider or the vendor is wrapping someone else’s model, because that changes who actually controls our data and what we can control. When a vendor can’t or won’t explain that chain, we walk away.
For a security leader at a smaller software vendor with a fraction of your headcount, what are the first three things you would do?
I’d start with three things, even if you have an extremely small security organization. First, I would recommend standing up a vulnerability management program. Companies need the ability to scan and identify security vulnerabilities.
Second, put monitoring in place or hire a company to monitor on your behalf. If you don’t have alerts or a way to determine if someone is doing something bad, you will never see an issue until your data or company has been compromised. AI is particularly good at helping a small team triage alerts and building new alerts, giving you leverage you wouldn’t have had a few years ago.
Third, have a focus on corporate security and have at least basic endpoint controls like encryption, AV/Malware detection, patching, and monitoring data loss. Even small companies will need to have the essentials to help protect them and their customers. If leveraged correctly, AI can help close the gap between a large team and a small team.
Download report: How security controls perform in practice