|
Catch AI cost spikes in real time (not months later) (Sponsored)
 |
Datadog’s free guide shows how to connect AI spend, infrastructure, and model performance into a single view, so you can correlate cost increases to the architecture changes that caused them before they show up on your cloud bill.
Learn how to:
Break down AI costs by token, model, provider, and team
Get alerted the instant inference volume spikes or API spend exceeds budget
Correlate cost increases directly to architecture changes so root-cause analysis takes minutes
A model with 70 billion parameters can take up to 140 GB of space. A good graphics card has 24 GB. A very good one might have 48 GB.
As you can see, the gap is quite significant. Disk space is pretty cheap, but fast memory is scarce and costly. Moreover, the models have grown roughly 100-fold in a few years, while consumer graphics memory has roughly doubled. It’s not just a matter of tightening things up to make them fit.
So how do you run such a model?
The simplest option is to purchase the hardware that is capable of running the model. But it is costly, and doesn’t work well with consumer hardware.
The other option is to shrink the model. But we don’t want to do so at the expense of the model’s intelligence. This is where certain techniques can help us make the model smaller in principle without a dip in the quality of its output.
In this article, we will cover:
What makes a language model intelligent?
Three techniques to shrink the model
How to shrink the model by packing fewer details?
How to shrink the model by trimming unused pathways?
How to shrink the model by mimicking behaviour?
Does shrinking damage the model’s intelligence?
 |
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
What Makes a Language Model Intelligent?
Large language models like ChatGPT are quite different from normal software programs. They don’t depend on typical if-else statements to decide what should happen next.
A large language model is essentially a very big pile of numbers called parameters or weights. These weights are the foundation of a language model’s intelligence. For reference, a model with 70 billion parameters means 70 billion numbers or weights. Each weight is normally stored in 16 bits, which is two bytes. Two bytes multiplied by 70 billion comes to 140 GB, which is basically considered the size of the model.
The weights are arranged into matrices. A single weight matrix is a grid, often something like 4096 by 4096. This comes to roughly 16.7 million numbers in a matrix. A 70 billion parameter model has hundreds of these matrices stacked across 80 or so layers.
Of course, weights in themselves aren’t the whole story behind a model’s capability. Multiple components work together to make a model intelligent. For example, the model needs an architecture like the Transformer to route data and perform attention. It also needs a context window that holds the prompt and everything generated during the conversation.
However, the architecture is made up of a few hundred lines of code. The prompt may be a few kilobytes. In contrast, the weights form the bulk of the language model. Without the proper weights, a language model cannot work as intended. The weights perform a bunch of tasks:
They store patterns such as grammar, facts, and reasoning shortcuts.
Once the training is finished, the weights freeze into an immutable network of numerical values.
The weights decide how strongly one simulated neuron influences the next.
Running the model means pushing the input through these weight matrices until the next word comes out of the other end.
One thing to keep in mind over here is that no single weight means anything on its own. If you opened a model file and looked at weight number N, you might see something like 0.0293. On either side of this number may be other numbers such as -0.0117, 0.004, -0.0862. On their own, each of these numbers hardly makes sense. The ability of the model is hidden in the relationships between the various weights. Think of it like a photograph where every pixel comes together to show something recognizable. Even if we modify every pixel’s brightness slightly, we can still make out the things in the picture. This is because the information isn’t sitting in one single place.
Following on from all this information, it is quite easy to figure out that to shrink a model, we’ve to somehow deal with these weights. And this is exactly where the techniques come into the picture. However, a couple of points can help us make better sense of the techniques to shrink a model:
Firstly, not all weights matter equally. Most of the weight values are close to zero and barely impact the final output. However, a small handful are large and produce a bigger impact.
Second, we can only judge a model based on its behaviour, not its internals.
Let us now look at the techniques.