Introduction
“A large language model is nothing but a next word prediction machine.”
You’ve probably heard that phrase. And while it’s technically true, it’s a bit like saying a jet engine is nothing but a machine that pumps a lot of air. Technically accurate – profoundly incomplete.
Large Language Models (LLMs) are among the most complex software systems ever built. They power ChatGPT, Gemini, Claude, Copilot, and dozens of enterprise AI tools being woven into critical business workflows right now. And yet, most security professionals have little understanding of what’s actually happening beneath the surface – which means they have little understanding of where these systems break.
This blog is the second in our AI and Cybersecurity series. In the first series, we explored hacking the application layer – agentic AI, RAG pipelines, chatbots, MCP servers. Now we go deeper. We look at hacking the large language model itself.
To understand how LLMs can be attacked, we first have to understand how they work. Not the PhD-level mathematics – but the key concepts a security researcher needs to reason about attack surfaces clearly.
Section 1: What Is a Model, Really?
The word model in LLM is the key to understanding everything. A model is a mathematical function that maps input to output in a way that reflects patterns found in real-world data.
Consider this simple example: imagine you have a dataset with two columns – the size of houses in a given area, and the price each house sold for. If you plot these data points and find a line that best fits them, you have a model. It takes in a house size (input) and predicts a price (output).
That line is expressed as: y = mx + b
- m is the slope
- b is the bias (where the line starts on the y-axis)
- x is the input
- y is the predicted output
Add more data points, and the values of m and b shift – the model learns. That’s training.
Not all problems are linear. Some data falls into clusters rather than along a line. For those cases, we use models like Support Vector Machines (SVMs), which learn decision boundaries. Effective for classification, but still limited for complex tasks like understanding language, interpreting images, or generating content.
Section 2: How Models Learn – Gradient Descent
The question at the heart of machine learning is: how does a model find the right parameters? The answer is an algorithm called gradient descent – a concept from differential calculus that is central to all modern AI.
Here’s the intuition:
- Start with random parameter values (random values for m and b).
- Run the model on a data point. Get a prediction.
- Compare the prediction to the real value. Calculate the loss (the error).
- Square those errors (to eliminate negative values and penalize larger errors more).
- Sum the squared errors to get a single loss value.
- Use calculus to determine which direction to nudge the parameters to reduce that loss.
- Repeat – thousands, millions, billions of times.
The metaphor that sticks: it’s like a hiker trying to find the fastest route down to base camp, navigating a landscape of mountains and valleys. Gradient descent is the process of always taking the steepest step downward until you reach a local minimum – a point where the model’s predictions are as close to the real values as possible.
When the loss is minimized and the model’s output closely mirrors the actual data, we say the model has been trained.
Security Insight: If the training data is poisoned, or if the training algorithm itself is compromised, the model will learn the wrong weights. This is not a theoretical vulnerability – it is a real attack vector.
Section 3: Neural Networks – Where Scale Changes Everything
Simple models like linear regression and SVMs can’t handle the complexity of human language. That’s where neural networks come in.
A neural network is made up of interconnected neurons. Each neuron takes inputs (x1, x2, x3…), multiplies each by a weight (w1, w2, w3…), adds a bias value, and passes the result through an activation function (like ReLU, sigmoid, or tanh).
ReLU is simple but powerful: if the value is negative, output zero. If positive, pass it through unchanged. This non-linearity is what allows networks to learn complex patterns.
Scale in numbers:
- A simple 3-input, 2-hidden-layer, 1-output network already has 41 parameters
- GPT-3: 175 billion parameters
- GPT-4: approximately 1.8 trillion parameters
At scale, something remarkable happens: emergent properties appear. The model begins to reason, translate, summarize, generate code – capabilities no one explicitly programmed. Scale was the catalyst.
Section 4: From RNNs to Transformers – The Architecture That Changed Everything
Recurrent Neural Networks (RNNs)
RNNs attempted to preserve context across words by passing information forward as the sequence was processed. The problem: vanishing gradient. By the time the network processed hundreds or thousands of words, context from early words had faded. Training on internet-scale data? Catastrophic failure.
Convolutional Neural Networks (CNNs)
CNNs were extraordinarily effective for images – sliding a filter across a bitmap to detect patterns. Applied to text, they fell short on the same core problem: long-range dependencies.
Consider this sentence: ‘The cat that sat on the mat which was near the window was sleeping.’ What was sleeping? The cat. Not the mat. Not the window. Unless you bring in all of this context, the model cannot learn the actual meaning.
The Transformer (2017) – Attention Is All You Need
In 2017, researchers at Google published a landmark paper: ‘Attention Is All You Need.’ Their proposal: throw out recurrence and convolution entirely. Use attention instead.
The key insight: let every word look at every other word in parallel, but not with equal importance.
How the attention mechanism works:
Each word is transformed into three vectors:
- Query (Q): What this word is looking for
- Key (K): What this word offers
- Value (V): The actual content this word carries
Think of it like a search engine: the query asks ‘Where is the cat sitting?’ The key responds ‘On the mat.’ Query-key scores determine relevance (attention scores). High scores lead to the value vector being weighted heavily. The result: each word’s representation also captures information from the words most relevant to it.
Multiple attention heads run in parallel, each learning different relationship types – syntactic, semantic, positional. Stack attention layers with feed-forward networks between them, add normalization, and you have the transformer architecture – the engine inside every major LLM today.
Section 5: Words to Numbers – The Embedding Layer
Before any of this works, there is one foundational requirement: machine learning only works with numbers. Text must be converted to numerical form.
This is done through a process called word embeddings (popularized by Word2Vec). This isn’t a simple word-to-number lookup – it’s geometrically meaningful:
- Words are placed in a high-dimensional vector space
- Words with similar meanings are placed closer together
- The famous example: King minus Man plus Woman is approximately equal to Queen
This works because the embedding process captures semantic meaning – not just spelling, but conceptual relationships. ‘King,’ ‘Queen,’ ‘Crown,’ and ‘Royal’ cluster together. Embeddings are trained, not hand-coded. And that means they are hackable.
Section 6: The LLM Attack Surface – Where Hackers Look
An LLM is not a black box – it’s a layered system with distinct, targetable components. Here are the four primary attack surfaces:
1 Training Data Poisoning
What it is: Injecting malicious, misleading, or biased data into the training corpus before or during model training.
Impact: The model learns permanently distorted patterns. Behaviors embedded in weights at training time are extremely hard to detect and remove post-deployment.
Real-world relevance: Open-source datasets used for LLM pre-training (e.g., CommonCrawl) are scraped from the web. An attacker who can manipulate web content that gets scraped can influence how the model responds to specific prompts at scale, for every user of that model.
2 Weight Manipulation
What it is: Directly modifying the trained parameters (weights) of a model.
Impact: Changes the fundamental nature of the model – its knowledge, reasoning patterns, behavior. This is the model’s intellectual property and core identity.
Real-world relevance: Model weights are increasingly shared via platforms like Hugging Face. A compromised model file looks identical to a legitimate one but behaves differently – analogous to a trojanized software binary. Detection requires targeted behavioral probing.
3 Embedding Attacks
What it is: Manipulating the vector representations of words or tokens in the embedding layer.
Impact: The model’s understanding of language relationships is distorted. Words that should be semantically close are moved apart; unrelated concepts are pushed together.
Real-world relevance: Fine-tuning on adversarially crafted datasets can shift embeddings. In RAG systems, the embedding model used for document retrieval is a specific attack target – poisoning embeddings can cause wrong documents to be retrieved and fed to the LLM as context.
4 Context Window Manipulation
What it is: Exploiting the input the model receives at inference time – the prompt and any injected context.
Impact: The model is confused, misled, or made to ignore its system instructions. This is the foundation of prompt injection attacks.
Real-world relevance: This is the most accessible attack vector – no model access required. An attacker crafts input that overrides system instructions, leaks sensitive context, or manipulates the model’s behavior for that session. In agentic AI systems taking real-world actions, the consequences can be severe.
Section 7: Why Security Professionals Must Understand LLM Internals
You don’t need to be an ML engineer to do AI security. But you do need to understand the threat model – what the components are, how they interact, and what happens when they’re tampered with.
The attacks above are not theoretical. They’re being demonstrated in research, and some are being exploited in the wild:
- Model poisoning has been demonstrated against open-source models
- Prompt injection is already being used against AI-integrated enterprise applications
- Supply chain attacks on model repositories (analogous to npm/PyPI package attacks) are an emerging threat
- Jailbreaks – structured prompts that bypass model safety guardrails – are being actively published and weaponized
The reason all of this is possible traces back to the fundamental architecture. The transformer’s strength – learning from massive data through adjustable parameters – is also its attack surface. Every component that can learn can be taught the wrong thing.
Understanding this architecture gives you the conceptual vocabulary to:
- Ask the right questions when evaluating AI vendor security
- Design meaningful controls at each layer
- Conduct AI security assessments with depth rather than surface testing
- Contribute to AI red team exercises
What’s Next in This Series
This video and blog are the foundation. In subsequent sessions, we’ll go deeper into each attack type:
- Training data poisoning – techniques, real case studies, defenses
- Model backdoors and trojanized weights – detection and analysis
- Embedding space attacks – adversarial examples and retrieval poisoning
- Advanced prompt injection – beyond jailbreaks, into agentic exploitation
Subscribe to our YouTube channel and follow our blog to get each episode as it drops.
