|
Test Your Auth Flow Without Production (Sponsored)
 |
Authentication is often the least-tested part of an app. Live environments need network access and real credentials, while mocks miss the failures that break production.
@workos/emulate runs the WorkOS API locally for development and CI. Seed users, organizations, RBAC roles, and SSO connections, then test full AuthKit login flows, signed webhooks, token refresh, and error handling. Responses and event shapes come from the WorkOS OpenAPI spec, so tests exercise the same surface your app uses in production.
Imagine a scenario where a developer builds an AI-based coding assistant that runs on a desktop computer. The model is available to download, the application is quite straightforward, and the machine has plenty of storage. But when the program tries to load the model, it runs out of memory.
This is where local AI development becomes a hardware problem.
Merely downloading a large AI model to our computer doesn’t mean we can actually run that model. In fact, even if we successfully load it, we can’t guarantee a useful response time.
A large AI model can run on modest hardware only by reducing the memory it occupies, reducing the calculations it performs, or moving some work to slower hardware. Several techniques can help deal with these requirements. In this article, we’re going to look at these techniques. Here’s what we will cover:
What running a model actually means
What makes it difficult to fit a model on smaller hardware
Why make the effort to run locally
Quantization: Giving each weight a smaller representation
Layer-wise offloading: Move weights as they are needed
Mixture of experts: Use selected parts for each token
Distillation: Let a larger model teach a smaller model
Pruning: Remove work that contributes less
Speculative decoding: Propose several tokens before checking them
What Running a Model Actually Means
An AI model contains numerical values called parameters. These are also known as weights. They influence how input becomes output. During training, these values are adjusted so that the model becomes better at its task. An 8B model contains approximately eight billion parameters or weights.
These weights are organized into layers. Each layer performs calculations on incoming information and passes its results to the next layer.
 |
In a language model, the input text is first divided into tokens. These tokens can represent words, parts of words, or even punctuation. The model processes them and produces probabilities for the next token. Once a token is selected and added to the sequence, the generation of text continues.
Producing a complete answer is all about repeating this process.
A couple of points to keep in mind here are as follows:
Using an already trained model is called inference. The model’s weights normally remain unchanged while it answers a question.
Training requires additional calculations to update those weights, making the process quite demanding. It requires expensive infrastructure.
This difference helps explain why a model trained on expensive infrastructure can sometimes run on an ordinary computer. The main thing we are discussing here is not the training aspects, but the inference part of a model.
[Webinar] How to stop babysitting your agents (Sponsored)
 |
Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.
More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.
Join us for a FREE webinar on Sep 23 to see: