top of page

The Design Space of an LLM System

Writer: NOURA ALSHAREEF
NOURA ALSHAREEF
Oct 31
13 min read

Where this starts

When we design a system that uses agents as one of its components, we connect it to things the way we always have: an external service, a database, an API. Each of those comes with a protocol, a fixed shape for what you send and what you get back. The connection to an LLM has none of that. There is no fixed request format, no fixed response format, no fixed type for the input or the output. That makes the space of possible designs very wide, and very easy to make expensive.

This post is a framework for planning that design, as steps, so the task is organized and reaches the highest accuracy at the lowest cost. A good design, good code, and a small LLM can get close to the performance of a frontier model like Claude at a fraction of the price. And the design has to be maintainable: when the model changes, the system should work the same or better, without rewiring it from scratch.

The ideas come from MIT 6.S978 Engineering AI Systems and Agents, taught by Prof. Omar Khattab.

لما نصمم نظامًا يستخدم agents كمكوّن من مكوناته، نربطه بأشياء مثل ما نربط دائمًا: خدمة خارجية، database، API. كل واحد منها له protocol، صيغة ثابتة للإرسال والرد. أما الربط مع LLM فما عنده شيء من هذا: لا صيغة محددة للطلب ولا للرد، ولا نوع محدد للمدخل أو المخرج. وهذا يخلي الموضوع واسع جدًا، ومكلف.

في هذا المقال نتناول framework لكيف نخطط لتصميم نظام يتعامل مع LLM، كخطوات، حتى نرتب المهمة ونخليها تنجح بأعلى دقة ممكنة وأقل تكلفة. تصميم جيد + كود جيد + LLM صغير ممكن يوصل لأداء قريب من النماذج الممتازة بجزء من السعر. ونهتم كذلك أن يكون الحل قابلًا للصيانة: لو تغيّر الموديل، يشتغل كما هو أو أفضل، بدون إعادة الربط والتصميم من الصفر. هذا المقال مستنبط من محاضرات MIT 6.S978 Engineering AI Systems and Agents للبروفيسور عمر خطاب.


What is the thinking framework?

Any AI system you can imagine has tens or hundreds of valid designs. The danger is not that you pick a bad one; it is that you pick the first one that comes to mind and never see the others. A design is not one decision. It is a bundle of smaller decisions, each one answering a different question: how does the model get its data? who controls the flow, code or the model? what happens before an action runs? how is the task even described to the model? Each of those questions is an axis. A design is one answer per axis. Two designs that give the same answer on every axis are the same design; two that differ on one axis are different designs.


That is why the framework is organized around axes rather than around “ideas”. If you decide every axis, you have decided the whole design. If you leave one axis undecided, something will decide it for you by default, usually the first tutorial you read, and that default is where systems fail in production.


The axes

Eight of them, from the outside of the system inward:

  1. Specification: how each step is told what to do. Code, a rule in words, examples, a typed signature.

  2. Single call: how one step calls the model. Plain, with reasoning, with tools, with typed output, voted, cascaded.

  3. Control flow: who drives. A code pipeline, an agent loop, a fixed multi-hop program, index-then-answer, recursive self-calls.

  4. Retrieval: how the model reaches a collection bigger than its window. All in context, grep, keyword index, single vector, token-level vectors, reranker.

  5. Access: what the system can touch and how untrusted input is handled. Broad, sandboxed, classified, read by a model without tools.

  6. Action: what happens before something irreversible. Act immediately, propose then check then commit, a second reviewer, receipts from a log.

  7. Evaluation: how we know it works. Labeled set, failure cases, random sample, protected holdout.

  8. Maintainability: what survives the next model. Spec apart from the model, or tuned to this one.


The map below shows where each design axis sits in one common design. It is not the only design: not every system has an agent loop, an index and corpus, internet access, and so on. The point is to show where the axes live.

أي نظام ممكن تتخيله، لابد أن نكون قادرين على إيجاد عشرات وربما مئات التصاميم الصالحة له. وبدلًا من أن نختار أول تصميم يطرأ على البال، لم لا نحاول ترتيب الأفكار ومعرفة: ما هي المشكلة التي نريد حلها بالضبط؟ هل هناك متطلبات من المدير أو من العميل؟ هل هناك قيود معينة مثلًا السعر، وقت الرد، حجم المدخلات وحجم المخرجات؟ هل عندنا data كافية أم نحتاج شيئًا إضافيًا؟ هل عندنا أمثلة كافية للمطلوب؟ بمعنى ربما العميل يطلب شيئًا ولا يمكننا فهمه: مثلًا العميل شركة ملابس وترغب أن نظهر لكل زبون الملابس التي من المحتمل أن تعجبه هو تحديدًا، والزبون الآخر سيظهر له شيء مختلف وهكذا. هل يوجد لدينا أمثلة من الزبائن؟ هل لدينا قوانين صارمة يجب ألا نخطئ فيها؟ مثلًا: لا نعرض قطعة غير متوفرة في المخزون، ولا نخصم من البطاقة بدون تأكيد الزبون. وما الذي قد يسبب الفشل؟ وصف منتج يفهمه الـ LLM غلط، زبون جديد بدون history، أو تحديث للموديل يغيّر سلوكه بين ليلة وضحاها. كل سؤال محور، والتصميم إجابة واحدة على كل محور. وأي محور تتركه بلا قرار سيُحسَم افتراضيًا، وهناك تفشل الأنظمة.

المحاور الثمانية، من الخارج إلى الداخل:

  1. التوصيف: كيف تُخبَر الخطوة بمهمتها.

  2. الاستدعاء الواحد: كيف نستدعي الllm .

  3. سير التحكم: من يقود.

  4. الاسترجاع: كيف يصل النموذج إلى ما يتجاوز نافذته.

  5. الوصول: ما يلمسه النظام وكيف يعامل المدخل غير الموثوق.

  6. الإجراء: ماذا يحدث قبل ما لا يمكن التراجع عنه.

  7. التقييم: كيف نعرف أنه يعمل.

  8. الصيانة: ما يصمد بعد النموذج القادم.

الخريطة أعلاه تُظهر موضع كل محور في تصميم شائع واحد (التصميم ليس مثالي لكل الحالات، هو فقط يعكس أحد التصاميم الشائعة)

Why covering the axes covers the design

Every design decision, whichever technique it uses, answers one of these eight questions. So if each axis gets a deliberate answer, no part of the design was left to a default. That is what complete means here: not that you considered every possible system, but that nothing was decided by accident.

Naming the axes also makes conflicts findable. Two choices can only clash in one place: the same slot, or two named slots that rule each other out. Without axes, a conflict is just “something feels wrong”.

What does each step ask?

(input) Requirements. Who depends on the system, and what must be true for each of them?

Examples:

  1. (Clothes shop recommender) For a returning customer: at least 3 of the 10 items shown come from a category they bought in the last 12 months. Test: compare the page against purchase history.

  2. (Clothes shop recommender) For the shopper: no item on the page has stock = 0 at the moment the page renders. Test: zero tolerance, checked per request against inventory.

  3. (Clothes shop recommender) For the support desk: every item shown carries a one-line reason an agent can read back to the customer. Test: a human can map each reason to a fact about that customer.

  4. (Help-center redaction) For the client: no real name, password, or other sensitive token from the source appears anywhere in the published text. Test: zero leaks on the labeled set; one leak is a failure, not a lower score.

  5. (Help-center redaction) For the reader of the article: text the client did not mark sensitive stays readable — support-agent names, product names, and instructions survive. Test: over-redaction count stays below the client's tolerance.

  6. (Bank transfer agent) For every other customer: no account balance goes down unless its owner is the one transferring. Test: after any request, every account not owned by the session stayed the same or went up.

  7. (Bank transfer agent) For the bank: money is never created or lost. Test: sum of balances before equals sum after, for every request.

  8. (Bank transfer agent) For the customer who re-taps after a timeout: a transfer executes at most once, however many times the request is retried. Test: replay the same request twice; one transaction in the log.

  9. (Bank transfer agent) For the support desk: what the reply says happened is what the transaction log says happened. Test: compare the model's reply against the log, not against itself.

  10. (arXiv digest) For the researcher: a paper appears in at most one digest, ever. Test: no paper ID repeats across the sent history.

  11. (Online pharmacy) For the regulator: a drug is shipped only if the logged-in customer holds a valid prescription for that drug. Test: every shipment matches a record with valid = True in that customer's records.

  12. (Course-policy assistant) For the course staff: the program never states course information that is not on the policy page. Test: every factual claim in a reply traces to a sentence on the page.

  13. (Meeting notes) For the legal team: no notes from a legal-team meeting are ever posted to chat. Test: zero posts where the team is legal.

  14. (Loan-report agent) For the regulator: a permanent, accurate record of every action the agent took exists after the run. Test: the saved log replays every tool call and result exactly as it happened.

  15. (Market report) For the founder: no content from plan.pdf, or anything derived from it, ever leaves the company in a web request. Test: every outbound request is produced by a component that has never read plan.pdf.

(input) Constraints. What is given:

budget per item, latency, the context window versus the data, corpus size, number of labeled examples, storage; which models exist, how often they change, how long you get to update; access level (internet or not, which files, which credentials), trust level of inputs, irreversibility of actions, audit obligations; the kinds of questions users will ask.

Examples

  1. Budget. (arXiv digest) At most $5 per 1,000 papers. The frontier model alone costs $20. Forces: a cheap first pass, the frontier model only on survivors.

  2. Latency. (Clothes shop recommender) The page must render in under 300 ms. Forces: no LLM call on the request path; the model ranks ahead of time, code filters live inventory at render.

  3. Context window vs. data. (Photo cleanup) 40,000 photos, far more than one call can hold. Forces: an agent loop with tools, and compaction when the log fills the window.

  4. Corpus size. (Inbox assistant) 100K emails growing to 1M; grep needs 38 searches per question at 100K and 122 at 400K, with 1 in 5 over budget. Forces: an index built before the question arrives.

  5. Labeled examples. (Help-center redaction) 200 labeled articles, no more coming. Forces: examples as the spec for the long tail, with a frozen held-out slice the optimizer never sees.

  6. Model churn. (Any) A new model every few months, and one of them broke a harness that forced chain-of-thought. Forces: the signature stays fixed; only the harness changes.

  7. Trust of inputs and irreversibility. (Bank transfer agent) The user can claim to be anyone, and a transfer can’t be undone quietly. Forces: the session principal comes from auth into a variable the model never writes; the model proposes, code checks against it before commit.

  8. Audit obligation. (Loan-report agent) The regulator can ask for every action a year later. Forces: an append-only log written by the loop code around every tool call; the model’s summary is not the record.


  1. Map. Run each requirement through the map below. It tells you which axis the requirement is really about and which leaf is a candidate design for it. A requirement often lands in more than one column, so the result of this step is not a design; it is a list of candidates, one or two per requirement, some of which conflict with each other.

Examples

  1. No item with stock = 0 at render. Lands in Action (hard gate before render) and in Control flow (code decides; the model never sees the gate). Two candidates.

  2. At least 3 of 10 items from a category they bought. Lands in Action (soft check, measured) or in Quality (more examples to raise the rate). Two candidates.

  3. Nothing from plan.pdf leaves in a web request. Lands in Access twice: split into two components with no shared context, or remove the web tool from the component that reads the file. Two candidates.

  4. Money is never created or lost. Lands in Assumed (the database’s atomicity) and in Action (re-check inside the transaction). One is recorded, one is built.

  1. Candidates. For each leaf, write what it assumes about the world and the model. Drop any leaf that crosses a constraint.

    (input) Assumptions. What we posit about everything outside our program, including the customer, the database, and the LLM; the requirement holds only if these hold, so each one is either recorded or tested. Examples of assumptions

Examples

  1. (Bank transfer agent) The session principal from authentication belongs to the customer making the request, and the model cannot change it. Recorded; every ownership check is only as good as this.

  2. (Bank transfer agent) The model reads the request as the customer meant it: “a grand” is $1,000, “move some money” means ask how much. Tested on the labeled set; a wrong reading is a spec failure, not a model failure.

  3. (Clothes shop recommender) The inventory feed is current at render time, so stock = 0 means out of stock. Recorded; if the feed lags, the zero-stock gate passes items that are gone.

  4. (Course-policy assistant) The policy page is the complete and current source of course rules. Recorded; the program is never wrong by its own test if the page is wrong.

  1. (input) Wall constraints Pick one candidate per requirement, using the constraints to decide. This is where the trade-offs get made:

    1. Can the effect be undone?

      • No → hard gate; safety wins.

      • Yes → soft check, measured; flexibility wins.

    2. Does the budget cover the better option?

      • No → cheaper pass first, the frontier model only on survivors.

      • Yes → redundancy, or the stronger model.

    3. Does the data fit in one call?

      • Yes → put it in the prompt; no index.

      • No → an index built at preparation time, or an agent over files.

    4. Is there a latency target the model would break?

      • Yes → move the model off the request path.

      • No → the model may run per request.

    5. Does the input come from someone you can’t verify?

      • Yes → the reader never holds a tool, or its output is reduced to a few bits before anything acts

      • No → the model may read and act in one component.

    6. Will the model underneath change before the system is retired?

      • Yes → the spec stays apart from the model; only the harness is allowed to know which model it is.

      • No → model-specific tuning is allowed; mark it.

  2. Combine. One leaf per axis, chosen so that every requirement mapped to that axis is satisfied. Several requirements usually share an axis, and they have to agree on one answer there. If no single leaf satisfies all of them, the axis forces a redesign, not a compromise. Examples

Examples

  • (Bank transfer agent) Four requirements land on Action: never-decreases, never-created, at-most-once, reply-matches-log. One leaf satisfies all four: propose, code checks against the session principal and an idempotency key inside the transaction, commit, reply from the log. The model never touches the balance.

  • (Help-center redaction) Zero leaks lands on Action as a hard gate; readable text lands on Quality as a measured rate. They share the Spec axis: the model reports facts, code places every mask. One spec serves both.

  • (Market report) Access needs a split with no shared context, and Control flow needs code to drive, because the web reader must never be the component that reads plan.pdf. Combined: a code pipeline with two readers, one for the file, one for the web, meeting only in a third call that receives feature names, not text.

  • (Clothes shop recommender) Latency moved the model off the request path; the stock gate must run at render. Combined: the model ranks offline into a table, code filters that table against live inventory per request, and the one-line reason is stored with the ranking.

(output) Design space. Every valid combination is one design. Most axes are fixed by this point; the space comes from the two or three that are still open. List those combinations explicitly, so the first one you thought of is visibly one among several.


  1. Take one design from the space and run it through Pick, Evaluate, and Learn. What you learn sends you back to a specific step: to Pick if the fix is a different leaf, to Map if a rule was missing, to Evaluate if an assumption broke. Repeat until every gate passes and no other design in the space beats this one on quality, cost, and latency at the same time.


Examples

  • (Inbox assistant) Still open after walls: retriever (BM25 or late interaction), model (small or cascade), reader defense (classify first or two-bit reader), gating (code check or check plus reviewer). Everything else fixed. 2 × 2 × 2 × 2 = 16 designs.

  • (Help-center redaction) Still open: single cheap model or cascade to the expensive one on code-flagged hard cases; typed output or free text parsed by code. 2 × 2 = 4 designs. Majority voting was tried and removed: the errors were correlated, so it was never a live option.

  • Pick one. Among the valid designs, choose by trade-offs — quality against cost, latency, complexity — and by the size of the assumptions each one rests on. Two designs that both meet the requirement are not equal if one assumes the model proposes only what was asked and the other assumes only that authentication works.

  • Examples

  • (Help-center redaction) Cascade over single model: the dev set showed five examples the cheap model always leaked, and code could flag four of them from identifier-shaped strings before any call. The cost went up 30 percent on those examples only.

  • (Inbox assistant) BM25 over late interaction: with few labeled pairs there was nothing to fine-tune the encoder on, and BM25 needed no training. Revisit if labels arrive.

  • (Bank transfer agent) Code check without a second reviewer: the check is a one-line assertion against the session principal; a reviewer model would add an assumption, not remove one.

  1. Evaluate. Measure the chosen design on a held-out set the optimizer never saw. Zero-tolerance requirements are gates, not scores: one failure fails the candidate. Everything else is a rate with a confidence interval, and the rate must be measured on the distribution of requests users actually send, not on the examples that were easy to write down.

Examples

  • (Help-center redaction) 36 held-out articles: leaks is a gate at zero; over-redactions is a count compared against the client’s tolerance. A change that fixed one leak and added six over-redactions was kept; one that fixed a leak and added a leak elsewhere was not.

  • (Clothes shop recommender) The stock gate is tested by replay against inventory snapshots. The 3-of-10 rate is measured on a random sample of real sessions each week, since the labeled set is what the ranker was tuned on.

  1. Learn. Each failure is either the model’s or ours. If a perfect model with the same inputs would have gotten it right, the fix is in the system: retrieval, decomposition, a check. If no model could have known, the information was never elicited: ask the stakeholder, write the new rule, add it to the requirements, add the case to the examples. New models do not change this split; they move the first kind of failure, never the second.



Where this leaves you

The framework does not tell you which design is right. It makes sure you saw the others. Nine questions, one answer each, and the product of what survives is the size of your design space: usually tens of designs, not one. The constraints shrink it, the requirements fix most of the axes, and what remains open is the two or three decisions that are yours to make on purpose. A system fails in production on the axis nobody answered, so the test of a finished design is not that it works on the dev set; it is that every layer has a reason written next to it.


Try it on your own system. The tree below asks the nine questions, lights up the designs under each answer, and multiplies what is left. Turn on a constraint and watch the count drop; the number you end with is the space you were choosing from all along, whether or not you knew it.


 
 
 

Comments


White Structure

STAY IN THE KNOW

Thanks for submitting!

NoraTech

i@nshareef.com

©2023 by NoraTech 

bottom of page