How OpenAI is using their own AI in house for cybersecurity
Wondering how OpenAI is using their LLMs to improve their security stance? The company's Head of Security Matthew Knight shares how they're applying it.
May 7, 2024 • 4 Minute Read
I’ve worked for a lot of tech companies, and one of the recurring questions customers ask is if you’re eating your own dog food (which makes you wonder what they ask if you actually work for a dog food company). Kudos to OpenAI, as they seem to be doing just that in the area of cybersecurity.
At RSA Conference '24, OpenAI’s Head of Security, Matthew Knight, shared some of the ways the company is using AI in house to automate tasks, help with incident reporting, get context on open tickets, and assist with bounty reporting.
Table of contents
4 ways OpenAI uses AI for security
OpenAI uses AI technology to augment and enhance their security processes in four key ways.
1. Inbound message filtering
“We use AI a lot in security at OpenAI," said Matthew. "We use it to deal with inbound messages and detection logistics. When employees reach out with questions, we need to make sure to get the right message to the right people.
“We used GPT-4 to use some classification with our internal front door. If you’re an employee who doesn’t work with the security team often . . . it gets your message to the right place. It also doesn’t expose our organization chart to our partners.
“In the worst case scenario, the message still goes to an engineer, so even though the process doesn’t involve a human, it has a human still looking at it. This application has only upsides.”
2. Facilitating and summarizing incident reporting
Report capture is a particular pain point for security professionals in 2024. This is an area where Matthew said LLMs can help.
“You can take a chat with two security engineers, put it in an LLM, and have it write the first draft of an incident report. Engineers can then edit it for accuracy. It takes the toll out of a first draft.”
3. Process automation
OpenAI uses AI to capture data about unsafe configurations and poor configuration settings within the business. It will then have a chatbot reach out to the employee in question to ask if these configurations were intentional.
“They can have the initial conversation with the chatbot, where they might say, ‘Oh, no, this was unintentional.' When the engineer catches up to the ticket, the context is already preloaded there. They don’t have to spend time and mental toil tracking these people down [and asking these questions]. You’ve still got a human getting to it at the end [but without the busywork]."
4. Bug bounty challenges
“We use LLMs for automating bug bounty review," explained Matthew. "You get a lot of spam with random internet stuff. We use the models to moderate an efficient review [by doing tasks like] reading tickets [and] classifying against policy.
“It’s not doing any real cybersecurity analysis. It’s asking, ‘Is this a cyber issue someone needs to look at, or is it out of scope?’ Security engineers then review [the filtered submissions]. . . . This saves time and helps them to get to the important things first.”
OpenAI says to consider known LLM flaws when applying to cybersecurity
“Large language models are not without flaws like hallucinations, context length limits, overreliance, and prompt engineering,” Matthew said.
“With hallucinations, models just make stuff up. They’re inclined to give an answer on a list even if it doesn't exist. For example, you can ask it to give you a list of CVEs, and it can make them up when they don’t exist.
“With context length, we currently use tokens, and these might run out. Some things tokenize better than others, like English text. However, PCAP data does not tokenize well. There are techniques to overcome this— LangChain is one, which lets you chunk up text that you want to give a model."
Prompt injection attacks can also degrade GPT’s security controls, despite the company’s best efforts. However, Matthew said OpenAI has been leveraging Reinforcement Learning for Human Feedback (RLHF) to combat that.
On a side note, for a great article explaining RLHF, check out Axel Sirota’s article “Ethical AI: How to make an AI with ethical principles using RLHF.”
“We’ve trained security . . . into our models. We’ve used RLHF to do that, which lets our models align more with our safety policies.”
But like with any system, someone can often find a way around it. That’s why OpenAI said it was working with enforcement agencies and investing in the security research community.
Want to learn more about what’s going on at RSA 2024?
Check out our live feed of all the latest updates! For deeper insights, check out the following blog posts:
Start a free trial of Pluralsight Skills to advance your security skills.
