top of page

Neural Network Search - NAS

Writer: NOURA ALSHAREEF
NOURA ALSHAREEF
Oct 26
24 min read

For years, neural network architectures were designed by hand. The space of possible designs is enormous: how many layers, how wide each one is, how many branches, which kernel sizes, what input resolution. Turning all of those knobs manually, and getting them right together, is famously difficult. Neural architecture search (NAS) hands the tuning to an algorithm, which explores those knobs under whatever accuracy and efficiency constraints we set. That saves a great deal of engineering effort, and it puts good architecture design within reach of teams who don't have years of experience to draw on. In this post we walk through NAS from the ground up.



لسنوات طويلة صُمّمت معماريات الشبكات العصبية يدويًا، مع أن فضاء التصاميم الممكنة هائل: عدد الطبقات، وعرض كل طبقة، وعدد الفروع، وأحجام الـ kernel، ودقة الـ input. ضبط هذه الإعدادات كلها يدويًا، وبشكل متناسق ، أكيد سيكون صعب لأن عدد التراكيب كبير جدا ( عدد التراكيب يعني لو أأخذ أول اعداد واغير فيه وأثبت باقي الاعدادات ، ثم أغير الي بعده وهكذا ، بتكون عندما مئات التصاميم المختلفة )، من هنا دعت الحاجة إلى آلية البحث عن التصاميم الجيدة وتقييمها واسمها هو : البحث عن المعمارية (NAS) ، هذه الخوارزمية تستكشف الإعدادات تحت ما نحدّده من قيود الدقة والكفاءة، فيوفّر جهدًا هندسيًا كبيرًا ويجعل التصميم الجيد في متناول من لا يملك سنوات من الخبرة. في هذا المقال نمشي في NAS من الأساس.


Early NAS methods trained every candidate network they sampled from scratch, and used a small recurrent controller trained with reinforcement learning to decide what to sample next. Reinforcement learning (RL) is learning from a reward rather than from a correct answer. A small agent makes a sequence of decisions (which kernel? how many filters? for each layer) and, at the end, receives a single score for the whole network: its accuracy, perhaps with a penalty for latency. Nobody tells the agent which decision was wrong; it tries thousands of times and drifts toward the decisions that came with higher scores. We use RL because the choices are discrete (3 or 5) and have no derivative, and it is the same approach as the AMC agent in the pruning posts.


الطرق المبكّرة في NAS كانت تدرّب كل شبكة مرشّحة من الصفر، وتستخدم متحكّمًا صغيرًا مدرّبًا بالتعلّم المعزّز ليقرّر ما يُجرّب تاليًا. التعلّم المعزّز (RL) هو تعلّم من مكافأة لا من إجابة صحيحة: وكيل صغير يتخذ سلسلة قرارات (أي kernel؟ كم فلتر؟ لكل طبقة)، وفي النهاية يحصل على درجة واحدة للشبكة كلها (الدقة، وربما خصم على الزمن). لا أحد يخبره أي قرار كان خطأ؛ يجرّب آلاف المرات ويميل إلى القرارات التي صاحبتها درجات أعلى. نستخدمه لأن الخيارات منفصلة (3 أو 5) فلا مشتقة لها، وهو نفس أسلوب وكيل AMC في الـ pruning.


From lighter networks to better shapes

In pruning we removed weights, and in quantization we shrank the bits each weight takes. Both start from a finished network and leave its shape alone. NAS asks an earlier question: why not design the shape itself to be light and accurate from the start?

NAS has three parts, and they form a loop:

  1. Search space — the shapes we allow. Two levels: the cell (one small block: which operations, how they connect) and the network (how many cells, how wide, where to shrink the image).

  2. Search strategy — how we propose the next shape: grid, random, reinforcement learning, gradient, evolutionary.

  3. Performance estimation — how we score a shape (accuracy, latency, memory, energy) without training every candidate from scratch, and how to make the search hardware-aware.

Propose a shape, score it, learn from the score, propose a better one.


This post covers the first two parts: the space and the strategy. Performance estimation and hardware-aware NAS come in the next post.

If you haven't read the previous post, read it first. It is the price list this whole loop reasons with: what each convolution, block and attention layer costs. → The Building Blocks NAS Chooses From


في الـ pruning حذفنا أوزانًا، وفي الـ quantization قلّلنا بتات كل وزن، وكلاهما يبدأ من شبكة جاهزة ولا يمسّ شكلها. أما NAS فيسأل سؤالًا أسبق: لماذا لا نصمّم الشكل نفسه ليكون خفيفًا ودقيقًا من البداية؟ له ثلاثة أجزاء تشكّل حلقة: فضاء البحث (الأشكال المسموح بها، على مستوى الـ cell ومستوى الشبكة)، واستراتيجية البحث (كيف نقترح الشكل التالي)، وتقدير الأداء (كيف نقيّم الشكل دون تدريب كل مرشّح). هذا المقال يغطّي الجزأين الأولين، والثالث في المقال القادم. وإن لم تقرأ المقال السابق فابدأ به: هو قائمة الأسعار التي تفكّر بها هذه الحلقة.


1. The search space

Start from the cell. In the previous post we met the recipes that make a block of layers smaller and cheaper: the bottleneck (1×1 squeeze → 3×3 → 1×1 expand), the inverted bottleneck (1×1 expand → depthwise 3×3 → 1×1 squeeze), and channel shuffle (grouped 1×1 with a reorder in between). Such a block is a cell: a drop-in replacement for one convolution layer plus its ReLU, with the same channels in and out and far fewer weights.

We also saw that a network needs two kinds of cell. A normal cell replaces conv + ReLU and keeps the picture size. A reduction cell is the same recipe with stride 2 in its 3×3, so it shrinks the picture and can replace the pooling. Those are different recipes too, and both can be chosen.


So how do we build a search space? Two steps.

Step 1 — the engineer writes the menu. A search space is not a list of networks. It is a menu: for every decision, the options we allow. The researcher writes it by hand, from experience; NAS searches inside it and never adds an option the menu does not contain. A small example:

decision

options

normal cell

A: 1×1 → 3×3 → 1×1 · B: 1×1 → depthwise 3×3 → 1×1 · C: 1×1 → depthwise 5×5 → 1×1 · D: plain 3×3

reduction cell

R1: the normal cell with stride 2 · R2: max-pool

repeats of the normal cell per stage

1, 2, 3

channels (width)

16, 32, 64

input resolution

96, 128, 160

Search space settings


4 × 2 × 3 × 3 × 3 = 216 possible designs. Real spaces have 10¹⁰ or more, which is why nobody tries them all. The first two rows are the cell level of the space (what is inside the box); the last three are the network level (how the boxes are arranged).


Step 2 — the loop picks from the menu. Each design is one way of filling in every row. The loop proposes a design, builds it, trains it, measures accuracy and cost, and uses the result to propose the next one. The goal is not the lightest network (an empty network is the lightest) but the best accuracy under a cost limit: latency, memory or energy. Eleven designs the loop might try, with a limit of 60 ms (numbers illustrative):

#

normal cell

reduction

repeats

width

res.

accuracy

latency

note

1

D plain 3×3

R2 pool

2

32

128

88%

95 ms

accurate, over budget

2

A

R2

2

32

128

87%

48 ms

ok

3

B depthwise

R2

2

32

128

85%

22 ms

cheap, lost accuracy

4

B

R2

3

32

128

87%

31 ms

deeper helped

5

B

R2

3

64

128

89%

58 ms

wider helped, just under budget

6

B

R1

3

64

128

90%

61 ms

better reduction cell, 1 ms over

7

C dw 5×5

R1

3

64

128

90%

70 ms

bigger kernel, no gain

8

B

R1

3

64

96

88%

40 ms

smaller image, cheaper

9

B

R1

2

64

160

90%

66 ms

bigger image, over

10

B

R1

3

48

128

89.5%

52 ms

width between options

11

A

R1

3

64

128

89%

74 ms

normal 3×3 in the middle: too slow

Winner: #5 or #10, the best accuracy that stays under 60 ms. #6 is more accurate but over budget, so it loses. Plotted, the designs become a map, and the search is a walk toward its top-left corner:


What about memory and energy? The map above compares accuracy with latency only, but a design really has four numbers: accuracy, latency, memory, energy. We cannot draw four axes, and the search never needs to. The engineer handles the extra costs in one of two ways:

  • Hard limits. Each cost gets a budget: latency ≤ 60 ms, memory ≤ 256 KB, energy ≤ 5 mJ. A design that breaks any one of them is thrown out, whatever its accuracy; among the survivors, the most accurate wins. On the map this is just more cut lines. For tiny devices this is the usual case: a model that does not fit in the microcontroller's memory is useless at any accuracy.

  • One combined score. Fold the costs into the number the loop optimises, for example score = accuracy − λ₁·latency − λ₂·memory − λ₃·energy. The λ's say how much accuracy we would trade for a millisecond or a kilobyte, and that is a product decision, not a math one.

In practice both are mixed: a hard limit on what truly cannot be exceeded, a soft penalty on what we would like lower. And often one cost stands in for the rest, since memory and energy both track the number of multiplies; that is why most plots show accuracy against MACs or latency alone.


الخريطة أعلاه تقارن الدقة بالزمن فقط، لكن للتصميم أربعة أرقام: دقة، زمن، ذاكرة، طاقة. لا نرسم أربعة محاور، والبحث لا يحتاج ذلك. يتعامل المهندس مع الكلف الإضافية بطريقتين: حدود صارمة (لكل كلفة ميزانية، وأي تصميم يتجاوز إحداها يُستبعد مهما كانت دقته، ثم يُختار الأدقّ بين الباقين، وهذا هو المعتاد في الأجهزة الصغيرة حيث نموذج لا يتّسع للذاكرة لا قيمة له)، أو درجة واحدة مركّبة تخصم من الدقة قدرًا من كل كلفة بأوزان λ يحدّدها المهندس بحسب ما يقبل التضحية به. وعمليًا يُمزج الاثنان، وغالبًا تُكتفى كلفة واحدة لأن الذاكرة والطاقة تتبعان عدد الضربات تقريبًا.


The order of the rows is the job of the search strategy (section 3). The cost of each row, a full training run just to read off one accuracy, is the problem the next post solves.


نبدأ من الـ cell: في المقال السابق تعرّفنا على الوصفات التي تجعل كتلة الطبقات أصغر وأرخص (bottleneck، inverted bottleneck، channel shuffle)، وهذه الكتلة هي الـ cell: بديل مباشر لطبقة convolution مع الـ ReLU بعدها. وعرفنا أن للشبكة نوعين: normal cell تستبدل conv + ReLU، وreduction cell تصغّر الصورة وتستطيع أن تحلّ محلّ الـ pooling. فكيف نبني فضاء البحث؟ أولًا يكتب المهندس القائمة (menu) بيده: لكل قرار خياراته المسموحة (نوع الـ cell، نوع الـ reduction، عدد التكرارات، العرض، دقة الـ input)، وNAS يبحث داخلها ولا يضيف خيارًا غير موجود فيها. ثانيًا تختار الحلقة من القائمة: تقترح تصميمًا، تبنيه وتدرّبه، تقيس دقّته وكلفته، ثم تقترح التالي. الهدف ليس الأخف بل أعلى دقة تحت حدّ من الكلفة، وفي الخريطة هو الزاوية العليا اليسرى داخل الميزانية.


2. A short history: how the price of one trial came down


The first NAS methods paid the full price for every row of that table. A small recurrent controller, trained with reinforcement learning, proposed an architecture; the architecture was trained from scratch; its accuracy became the controller's reward. NAS with RL, NASNet and MnasNet all work this way, and they cost thousands of GPU-days, because the controller cannot learn anything until a whole candidate has been trained.


Differentiable methods cut that cost. DARTS puts every candidate operation into one big network at once and makes each layer's output a weighted average of them; the weights of the average are trained by gradient descent like any other weight, and the strongest option wins. ProxylessNAS keeps only two of those paths in memory at a time instead of all of them, and FBNet adds the measured latency of each option to the loss. One-shot methods such as Single Path One Shot go further: train with a single random path per batch, then read any sub-network off the trained super-network.


Much cheaper, but still not free: differentiable and one-shot NAS run the whole train–search–fine-tune pipeline again for every new target, about 200–300 GPU-hours on ImageNet each time. With more than 20 billion IoT devices of different sizes, one search per device does not scale. That is the problem the next post picks up.

الطرق الأولى (NAS with RL 2017، وNASNet 2018، وMnasNet 2018) دفعت الثمن كاملًا عن كل تجربة: متحكّم صغير مدرّب بالـ RL يقترح معمارية (سأضع 3×3 في هذه الطبقة، ثم أدرّب وأقيس وأسجّل، ثم أضع 5×5 وأعيد التدريب والقياس، وهكذا إلى أن تنتهي الإعدادات المذكورة في القائمة)، كل مرشّح يُدرَّب من الصفر ودقّته هي المكافأة، فكلّفت آلاف أيام الـ GPU. ثم جاءت الطرق القابلة للاشتقاق: DARTS لا يختار بين 3×3 و5×5 بل يضع الخيارين معًا في الموضع نفسه ويجعل مُخرج الطبقة مزيجًا منهما بنسب α (مثلًا 0.7 من 3×3 و0.3 من 5×5)، ولأن α أرقام داخل الشبكة فهي تُدرَّب مع الأوزان بالتدرّج، وفي النهاية يُبقى صاحب النسبة الأعلى؛ تدريب واحد بدل تدريب لكل خيار. وProxylessNAS يُبقي مسارين فقط من هذه الخيارات في الذاكرة بدل جميعها، وFBNet يضيف الزمن المقاس لكل خيار إلى الخسارة، وSingle Path One Shot يدرّب مسارًا واحدًا عشوائيًا في كل دفعة. أرخص بكثير، لكنها ما زالت تعيد خطّ التدريب والبحث كاملًا لكل جهاز جديد (200–300 ساعة GPU على ImageNet)، ومع أكثر من 20 مليار جهاز IoT لا ينفع بحث لكل جهاز؛ وهذا موضوع المقال القادم.


Now try this so you can get the whole idea :

This is where Once-for-All (OFA) comes in. It greatly reduces the cost of specializing neural networks for different devices. OFA trains one large super network that contains every sub-network in the design space. A sub-network is extracted by choosing from it: we pick how many blocks to keep and drop the rest, use only part of the channels instead of all of them, choose a kernel size such as 3×3 or 5×5, and choose the input resolution. The sub-network reuses the large network’s weights without modification, so it reaches accuracy close to training it from scratch, and it can be deployed directly on the device with no retraining.

OFA also adds accuracy and efficiency predictors to reduce evaluation cost during search. Measuring the accuracy of a single sub-network means running it over the full validation set, which can take about a minute on ImageNet. Instead, OFA collects many (architecture, accuracy) pairs in advance and trains a regression model to predict accuracy during the search, cutting the time from about a minute to under a second per network. The same idea applies to latency, which is slow to measure because the candidate network has to be run many times. This opens the door to running the search for microcontrollers, which have very limited resources.


ومن هنا تأتي طريقة Once-for-All (OFA)، التي تخفض بشكل كبير تكلفة تخصيص الشبكات العصبية لأجهزة مختلفة. تدرب OFA شبكة عملاقة (super network) تحتوي على جميع الشبكات الفرعية ضمن فضاء التصميم. والشبكة الفرعية تُستخرج بالاختيار منها: نحدد عدد البلوكات التي نستخدمها ونتخلى عن الباقي، ونأخذ جزءاً من القنوات بدلاً من كلها، ونختار حجم النواة مثل 3×3 أو 5×5، ونختار دقة الصورة الداخلة. تُستخدم الأوزان نفسها من الشبكة الكبيرة دون تعديل، ولهذا تحقق الشبكة الفرعية دقة قريبة من دقة تدريبها من الصفر، ويمكن نشرها على الجهاز مباشرة دون إعادة تدريب.كما تضيف OFA مُتنبئين (predictors) للدقة والكفاءة لتقليل تكلفة التقييم أثناء البحث. فقياس دقة شبكة فرعية واحدة يعني تشغيلها على مجموعة التحقق كاملة، وقد يستغرق ذلك نحو دقيقة على ImageNet. لذلك تجمع OFA أزواجاً من (البنية، الدقة) مسبقاً، وتدرب نموذج انحدار يتنبأ بالدقة أثناء البحث، فينخفض الوقت من دقيقة إلى أقل من ثانية لكل شبكة. وتنطبق الفكرة نفسها على زمن التنفيذ، إذ يتطلب قياسه تشغيل الشبكة المرشحة مراراً، وهو بطيء. وهذا يفتح الباب لتطبيق البحث على المتحكمات الدقيقة (microcontrollers) ذات الموارد المحدودة جداً.


MCU

A microcontroller (MCU) is a tiny, cheap computer on a single chip: a slow processor, a small amount of memory, and some pins to read sensors and drive motors or LEDs, all together, costing anywhere from a few cents to a few dollars. It usually runs on a battery or a coin cell for months

What makes it different from a phone or a laptop:

  • Memory: a few hundred KB of RAM and about 1 MB of flash storage, versus gigabytes on a phone. That’s the 320 KB / 1 MB in the figure.

  • No operating system in the usual sense: one program runs directly on the chip.

  • Power: milliwatts, so it can live inside a device for years without charging.

Where you find them: thermostats, washing machines, car sensors, smartwatches, hearing aids, keyboards, toys, smart doorbells, industrial sensors. Common examples are the Arduino boards and the STM32 chips.


2. Designing the search space: big enough, small enough


Before searching, the engineer has to decide what the menu should contain. A menu that is too small may leave out the designs that work. A menu that is too big wastes the search on weak candidates. The question is how to tell the difference without training everything. The answer is a chain of decisions, each one following from the last. This is the framework to work through before any search starts.

Step 1 — Which device? Three targets, and they are not interchangeable:

  • Cloud: tens of gigabytes of memory, terabytes of storage.

  • Phone: 4 to 8 GB of memory, tens of gigabytes of storage.

  • Microcontroller (MCU): about 320 KB of memory and 1 MB of storage, roughly three orders of magnitude less than a phone.


Why start here? Because the device decides what "too big" means. Most models on the market at the time were a few megabytes: fine on a phone, already impossible on a microcontroller. And microcontrollers are not a niche: they sit in hearing aids, thermostats, industrial sensors, wearables and the billions of IoT devices that have no phone attached and no cloud connection. If we want AI there, it has to fit in 320 KB.

قبل البحث قراران: حجم القائمة والجهاز المستهدف. قائمة صغيرة جدًا تُفوّت التصاميم الجيدة، وقائمة كبيرة جدًا تُضيّع البحث على المرشّحين الضعفاء. والجهاز يحدّد الميزانية: السحابة بعشرات الجيجابايت من الذاكرة، والهاتف بـ 4–8 جيجابايت، أما المتحكّم الدقيق فبنحو 320 كيلوبايت فقط، أي أقل بثلاث مراتب عشرية من الهاتف، ونموذج بحجم بضعة ميجابايت لا يتّسع فيه أصلًا.

Step 2 — The device sets the constraint.

  • MCU → memory. There is only 320 KB. If the weights or activations do not fit in on-chip memory, fetching them from slower memory is so expensive that the model is unusable at any accuracy. The question changes from "how many operations can we afford?" to "how many operations fit in the memory we have?"

  • Phone → latency and energy. Memory is plentiful, but the user expects an answer within tens of milliseconds and the battery drains. The constraint is a time budget on that particular phone.

  • Cloud → cost and throughput. Memory and compute are almost unlimited, but every operation is paid for in electricity and GPU time, and the goal is as many requests per second as possible.


Step 3 — The constraint sets what we measure. This is where intuition goes wrong.

  • MCU: peak activation plus weight size, not parameter count. The instinct is "fewer parameters, smaller model". But remember that :

    • Parameters (weights) live in storage, then get loaded once.

    • Activations are made while the model runs, and live in RAM.

    For example : MobileNetV2 has 4.6× fewer parameters than ResNet-18 at the same accuracy, and its peak activation during inference is about 80% larger, because the expansion by six creates a big intermediate tensor. In training, a MobileNetV2 variant with about four times fewer parameters than ResNet-50 saved only about 10% of activation memory. The activations, the intermediate results between layers, are what take the space. A menu written around "fewest parameters" would pick models that do not fit.

  • Phone: latency measured on the phone, not FLOPs. Two operations with the same FLOPs can take very different times on a phone's processor; a depthwise convolution has few FLOPs but uses the hardware poorly, so it is slower than its count suggests. Each candidate operation is run once on the real phone and its time stored in a table; a design's latency is then the sum of its operations' entries.

  • Cloud: throughput (images or tokens per second) and energy per request.


Step 4 — What we measure shapes the menu. The constraint becomes the entry condition: every design on the menu must satisfy it. After that, the preference flips depending on the device.

  • MCU: under the same memory budget, prefer more FLOPs. Fetching a block of memory is the expensive part, so the more computation we do on what we fetched, the more capacity the model has, and more capacity usually means more accuracy. Memory is fixed, so the extra computation costs no space, only a little time, which the microcontroller can spare.

  • Phone and cloud: the opposite. Computation is what costs time and energy, so the goal is the best accuracy with the least computation. That is the accuracy-versus-latency map from section 1.


Step 5 — Score the menu before searching it. Counting FLOPs, or reading latency from the table, takes seconds per design and needs no training. So we can look at the whole menu at once.

Picture the large menu as 1,000 designs, all of which fit in 320 KB. We count the FLOPs of each one (seconds, no training) and sort them from most computation to least. The top half, 500 designs, we keep; the bottom half we drop from the menu. Why? Because the rule from step 4 says that under the same memory, more computation usually means higher accuracy, so the bottom half is the least promising, and we don't want to waste search and training on it. The search in step 6 then runs over the remaining 500 only, not the thousand.

Does the rule hold? Compare the FLOPs distribution of two candidate menus under the same memory limit: in the better one, 80% of the models sit above a median of 50.3 MFLOPs; in the weaker one, above 33 MFLOPs. The better menu also produced the better model: 78% accuracy against 74%. Across designs, FLOPs and accuracy rise together, but only because every design fits the same memory. Without that limit, FLOPs could grow without bound and the correlation would mean nothing.

The same effect shows up in end results. The ResNet-18 menu, searched without the memory condition, reaches 80.3%, but the model runs out of memory on the device. A random menu reaches 74.7%, a very large menu 77%, and the menu cut down with this rule 78.7%, all within the memory limit. This is TinyNAS: automated search-space optimization (cut the menu) followed by resource-constrained model specialization (search what is left for the model that fits this device). The boundary of the menu is still a human decision; the machine cuts it down and searches inside it.


Step 6 — Search inside the menu with one of the strategies in section 3.

Step 7 — Score each candidate (accuracy and latency) without a full training run; that is the next post.

One consequence follows from the whole chain. Each device has its own constraint, so each device has its own menu, and so its own search. A network found for a phone does not fit a microcontroller, and one found for a microcontroller wastes accuracy on a phone. One search per device does not scale to billions of devices, and that is the problem Once-for-All was built to solve.


In practice step 7 is two small models we build ourselves:

  • an efficiency predictor that counts a candidate's MACs and peak memory (the same counter that cut the menu in steps 3–5, now run on every candidate the search samples to check it fits the budget), and

  • an accuracy predictor, a small regression model trained on (architecture, accuracy) pairs that estimates accuracy from the design alone in about a millisecond instead of a minute of inference on the validation set.


One consequence follows from the whole chain. Each device has its own constraint, so each device has its own menu, and so its own search. A network found for a phone does not fit a microcontroller, and one found for a microcontroller wastes accuracy on a phone. One search per device does not scale to billions of devices, and that is the problem Once-for-All was built to solve.



قبل البدء ، تعرفة بسيطة بالـ( المتحكّم الدقيق) MCU : كمبيوتر صغير ورخيص على شريحة وحدة: معالج بطيء، وذاكرة صغيرة، ومنافذ تقرأ الحساسات وتشغّل المحرّكات، ويشتغل على بطارية لشهور. يختلف عن الجوال في ثلاث نقاط: ذاكرته كم مئة كيلوبايت و1 ميجابايت تخزين (مو جيجابايت)، وما فيه نظام تشغيل، برنامج واحد يشتغل مباشرة على الشريحة، وطاقته بالميلي واط.سبب ذكرنا له هو انه أصعب حالة ، الجوال والكلاود فيهم متسع ، هو لا.


قبل ما يبدأ أي بحث، المهندس يمشي على سلسلة قرارات، كل خطوة تطلع من اللي قبلها، والنتيجة هي القائمة اللي بيبحث فيها.

  1. الجهاز: سحابة، جوال، أو MCU (320 كيلوبايت ذاكرة بس، والنماذج المعتادة بالميجابايت ما تدخل فيه).

  2. القيد يطلع من الجهاز: السحابة → التكلفة والإنتاجية، الجوال → الزمن والطاقة، الـ MCU → الذاكرة.

  3. القياس يطلع من القيد: السحابة → الإنتاجية، الجوال → الزمن المقاس على الجوال نفسه (مو FLOPs)، الـ MCU → ذروة التنشيطات مع الأوزان (مو عدد البارامترات: الأوزان في التخزين، والتنشيطات تتولّد وقت التشغيل في الـ RAM وهي اللي تاخذ المكان).

  4. القياس يشكّل القائمة: القيد شرط دخول لكل تصميم، وبعده التفضيل ينقلب: على الـ MCU ثبّت الذاكرة وفضّل حساب أكثر (الجلب من الذاكرة هو الغالي)، وعلى الجوال والسحابة فضّل حساب أقل.

  5. قيّم القائمة قبل البحث: عدّ الـ FLOPs لكل تصميم (ثواني، بلا تدريب)، اقصّ النصف الأضعف وأبقِ الباقي؛ هذا TinyNAS (78.7% داخل الذاكرة مقابل 74.7% لقائمة عشوائية).

  6. ابحث داخل القائمة بإحدى استراتيجيات القسم 3.

  7. قيّم كل مرشّح بدون تدريب كامل بمتنبّئين نبنيهما بأنفسنا: متنبّئ كفاءة (MACs والذاكرة) ومتنبّئ دقة (انحدار يتنبّأ بالدقة من التصميم في ميلي ثانية).

النتيجة: لكل جهاز قيده وقائمته وبحثه، وهذا ما ينفع مع مليارات الأجهزة، وهنا تجي Once-for-All.


What if I have many devices? Say 50 different phones, or 100 different microcontrollers. The framework says: pick the device, and everything follows from it. So 50 devices means walking steps 1 to 7 fifty times. The expensive part is step 7 in its old form, "train each candidate to learn its accuracy": 50 devices × 200–300 GPU-hours each does not scale, and there are billions of devices, not 50.

Once-for-All (OFA) changes step 7, and with it step 5.

  • Before OFA: step 7 means training the candidate from scratch to read off its accuracy, repeated for every device.

  • With OFA: train one large super-network once. It contains every design on the menu as a sub-network. After that, a candidate is never trained; it is cut out of the super-network (how many blocks, how many channels, which kernel size, which input resolution) with its weights already in place, and it reaches accuracy close to training from scratch. Training happens once for all devices, not once per device.

Where is the menu (step 5)? In OFA the menu is the super-network itself: every sub-network that can be cut from it is a design on the menu. Instead of cutting the menu down ourselves as TinyNAS does, we receive the super-network ready-made (in the lab it was given, holding more than 10¹⁹ sub-networks).

And the two predictors (step 7)? Even with OFA, measuring one sub-network's accuracy means running the whole validation set, about a minute. So we build the accuracy predictor (a millisecond). The efficiency predictor does the job of step 3: it counts the candidate's MACs and memory and says whether it meets this device's constraint.

So for each new device: steps 1–3 set its constraint (memory or latency), step 6 searches the same super-network, step 7 scores candidates with the two predictors, and the winner is cut out and deployed with no retraining. What changed is that training left the loop: it happens once, before all the devices, instead of inside the loop for each one.

That is what the figure shows: one network on top, and under it a large, a small and a tiny sub-network, one per device, "direct deploy, no retrain".


طيب لو عندي 50 جوال أو 100 متحكّم؟ الإطار يقول لكل جهاز بحثه، والغالي فيه الخطوة 7: تدريب كل مرشّح من الصفر، 200–300 ساعة GPU لكل جهاز، وهذا ما ينفع مع مليارات الأجهزة.

هنا تجي Once-for-All (OFA): ندرّب شبكة عملاقة مرّة وحدة تحتوي كل تصاميم القائمة كشبكات فرعية، وبعدها أي مرشّح ما نُدرّبه بل نقصّهمنها (عدد البلوكات، القنوات، حجم النواة، الدقة) بأوزانه الجاهزة، ويعطي دقة قريبة من تدريبه من الصفر. فالقائمة (الخطوة 5) صارت هي الشبكة العملاقة نفسها، وتقييم المرشّح (الخطوة 7) صار بمتنبّئين: واحد للدقة في ميلي ثانية بدل دقيقة على مجموعة التحقق، وواحد للكفاءة يعدّ MACs والذاكرة ويقول هل يحقّق قيد الجهاز.

النتيجة: لكل جهاز جديد نحدّد قيده (خطوات 1–3)، نبحث في نفس الشبكة العملاقة (6)، نقيّم بالمتنبّئين (7)، ونقصّ الفائز وننشره بلا إعادة تدريب. اللي تغيّر إن التدريب طلع من الحلقة: مرّة وحدة قبل كل الأجهزة، بدل مرّة لكل جهاز، وهذا اللي تبيّنه الرسمة: شبكة وحدة فوق، وتحتها large وsmall وtiny لكل جهاز.


3. Search strategies: grid, random, RL, gradient, evolutionary


Search space is a set of candidate neural network architectures. 														Search strategy defines how to explore the search space.
Search space is a set of candidate neural network architectures. Search strategy defines how to explore the search space.

We now have everything but the search itself: a menu (the super-network), an efficiency predictor that says whether a candidate fits the budget, and an accuracy predictor that scores it in a millisecond. The search strategy is the loop that uses them. It samples a candidate from the super-network, asks the efficiency predictor whether it meets the constraint (MACs, peak memory), re-samples if not, and scores the survivors with the accuracy predictor. Its goal is the design with the best accuracy that still satisfies the constraint.


With the space fixed, the strategy decides which design to try next. Here are five common choices, from the simplest to the most efficient. The lab implements two of them, random search and evolutionary search, on top of the two predictors.


Grid search. it is the traditional way of doing hyperparameter optimization, carried over to architectures.

  • Each dimension is written as a short list of options, and the full design space is the Cartesian product of those lists: for example, resolution in [1.0×, 1.1×, 1.2×] and width in [1.0×, 1.1×, 1.2×].

  • Every resolution is paired with every width, which gives nine candidate networks; grid search visits every cell of that grid in order.

  • To obtain the accuracy of each candidate, we train it from scratch, so the cost grows with the number of cells.

  • The latency constraint is checked afterwards, ruling out the cells that break it; the best accuracy among the rest wins.


EfficientNet is an example: it searched the coefficients α, β and γ that scale depth, width and resolution, under the constraint that the new model costs about twice the FLOPs of the starting network.

Random search. Configurations are sampled at random. It is less efficient than a guided search, but it is a useful baseline. If a smarter algorithm cannot beat random sampling after a few iterations, something is wrong with the code. Researchers often run random search first, to debug the pipeline before the formal search.


Reinforcement learning. The controller is a sequential decision maker: it chooses the first node, then its operation, then the next node, and so on. It proposes an architecture with probability P, the architecture is trained as a child network, and its accuracy becomes the reward R. The gradient of P is scaled by R, so decisions that led to higher rewards become more likely. This is the expensive method from the history section, because every reward requires a trained child.



Gradient-based search. We explained DARTS at the start of this post, and if you have read this far you will see at once that RL belongs with the old methods: train each design one at a time and measure it. Every trial is a full training run from scratch, just to learn whether the better kernel is 3×3, 5×5 or 7×7. Very expensive.


DARTS asked: why choose at all? Put every option into the same layer at once and give each one a share α. The layer’s output becomes a weighted sum of its candidate operations:

layer output = 
α₁ × (output of 3×3) + α₂ × (output of 5×5) + α₃ × (output of 7×7)

A mix, in other words. With α = (0.7, 0.2, 0.1) the layer is “mostly 3×3 with a little 5×5”. The trick is that the α values are ordinary numbers inside the network, like any weight, so when we train with gradient descent the gradient reaches the α too and adjusts them alongside the weights in the same run: if 5×5 helps accuracy, α₂ rises on its own; if not, it falls. One training instead of one per option, and at the end we keep the option with the largest α.

Why is this impossible in RL? Because “pick 3×3 or 5×5” is a discrete decision with no derivative. The mix turns it into a continuous one, which has a derivative.


And latency? Each operation is measured once in advance: 3×3 takes 2 ms, 5×5 takes 5 ms, 7×7 takes 9 ms. The layer’s expected latency is α₁×2 + α₂×5 + α₃×9, and we add it to the loss:

loss = cross-entropy + regularization + expected latency

If α₃ rises, the loss rises (because 9 is large), so the gradient pulls it back down. The result is that slow options lose on their own, without anyone saying “don’t use 7×7”. That is the DARTS idea, and adding the latency term is what FBNet and ProxylessNAS did.


Evolutionary search. We start with a population: a set of designs, say 100 random networks from the menu. We score each one with a fitness function: a single number that combines accuracy and speed, for example accuracy on the condition that latency stays under the limit, or accuracy minus a penalty for latency.

Then we create the next generation from the best ones in two ways:

  • Mutation: take a good design and change one small thing in it. For example, a stage with three blocks becomes two, or a 3×3 convolution is replaced with a 5×5 or 7×7. In other words, “the same parent with a small edit”.

  • Crossover: take two good designs (two parents) and build a child whose every layer is taken from one of them: the first layer from parent one, the second from parent two, the third from parent one, and so on. In other words, “combine parts of two successful designs”.

We then score the children, keep the fittest, discard the rest, and repeat. Over the generations the whole population drifts toward designs that are both accurate and fast.

Why does it work so well with OFA? Because evaluation is cheap: every child is cut out of the super-network and scored by the two predictors in a millisecond, so we can afford to try thousands of children. This is the search I implemented in the lab.


All five in one line. They all answer the same question: which design should we try next? Grid and random search are simple. Reinforcement learning and evolutionary search need many evaluations, so they become practical only when evaluation is cheap. Gradient-based search is the cheapest per candidate, because one training run covers every option. But all five depend on a second question: how do we score a candidate? That is the subject of the next post.

The figure goes after the two bullets and before “We then score the children…”: the reader has just met mutation and crossover, and the flow shows where they sit in the loop (fitness → crossover → mutation → survivor selection → back to fitness), which is exactly what the next sentence describes.


صار عندنا القائمة (الشبكة العملاقة) والمتنبّئان، وبقي البحث نفسه: حلقة تسحب مرشّحًا، تتأكد بمتنبّئ الكفاءة إنه يدخل في الميزانية، وتقيّمه بمتنبّئ الدقة، وهدفها الأدقّ الذي ما زال يحقّق القيد. الاستراتيجية هي اللي تقرّر أي مرشّح نجرّب بعده، وهذي خمس طرق من الأبسط للأكفأ (اللاب يطبّق العشوائي والتطوّري).


١- الـGrid search. نكتب كل بُعد كقائمة خيارات، ونجرّب كل تركيبة: resolution في [1.0×, 1.1×, 1.2×] مع width في [1.0×, 1.1×, 1.2×] = تسع شبكات، كل واحدة تُدرَّب من الصفر، ثم نستبعد اللي تكسر قيد الزمن ونأخذ الأدقّ. سهلة، لكنها تنفجر مع كل بُعد جديد. EfficientNet مثال عليها.


٢- الـRandom search. نسحب عشوائيًا. أقل كفاءة، لكنها المرجع: لو خوارزمية أذكى ما غلبت العشوائي فالكود فيه خلل.

٣- الـReinforcement learning. فيه «متحكّم» صغير مهمته يقترح تصميمًا: لكل قرار (kernel؟ كم فلتر؟ stride؟) عنده احتمال لكل خيار، ويسحب من هذي الاحتمالات قرارًا قرارًا حتى يكتمل تصميم واحد. نبني هذا التصميم وندرّبه من الصفر، ونقيس دقته. هذي الدقة هي «المكافأة»: لو طلعت عالية، نرفع احتمال كل الخيارات اللي اختارها في هذا التصميم؛ ولو طلعت منخفضة، ننزّلها. ما أحد يقول للمتحكّم أي قرار بالضبط كان غلط؛ هو يشوف رقمًا واحدًا للشبكة كلها، ومع آلاف المحاولات تميل احتمالاته نحو الخيارات اللي ترافقت مع دقة أعلى. وهذا سبب غلائها: كل محاولة تدريب كامل.


٤-الـGradient-based search. هذي DARTS التي شرحناها في البداية. بدل اختيار kernel واحد، نضع 3×3 و5×5 و7×7 معًا ونعطي كل واحد نسبة α، فخرج الطبقة = α₁×(3×3) + α₂×(5×5) + α₃×(7×7). والحيلة أن α أرقام داخل الشبكة، فتُدرَّب بالتدرّج مع الأوزان في تدريب واحد، ونُبقي صاحب أعلى α. وهذا مستحيل في RL لأن الاختيار هناك discrete بلا مشتقة. ولإدخال الزمن: نقيس كل عملية مرّة (2، 5، 9 ms) ونضيف α₁×2 + α₂×5 + α₃×9 إلى الـ loss، فالخيارات البطيئة تخسر من نفسها. إضافة الزمن هي ما فعلته FBNet وProxylessNAS.


٥- الـEvolutionary search. نبدأ بـ population من 100 تصميم عشوائي ونقيّمها بـ fitness تجمع الدقة والسرعة. الجيل التالي من الأفضل بطريقتين: mutation (تعديل صغير على تصميم جيد: بلوك أقل، أو 3×3 تصير 5×5) وcrossover (ابن كل طبقة فيه من أحد أبوين جيدين). نبقي الأليق ونكرّر، فتنزاح المجموعة نحو الدقيق والسريع معًا. تنجح مع OFA لأن كل ابن يُقصّ ويُقيَّم في ميلي ثانية


الخمس في سطر. كلها تجاوب: أي تصميم بعده؟ الـ grid والعشوائي بسيطان، الـ RL والتطوّري يحتاجان تقييمًا رخيصًا، والتدرّجي الأرخص لكل مرشّح. لكنها كلها تعتمد على سؤال ثانٍ: كيف نقيّم المرشّح؟ وهذا المقال القادم.


What comes next

We now have a search space that is big enough to contain good designs and small enough to search, plus a strategy for choosing from it. What is still missing is the score. Every design so far needed a full training run to read off its accuracy, which is the cost the history section described. Ideally we would estimate accuracy and latency without training every candidate, and we would measure latency on the target hardware instead of inferring it from FLOPs.

The next post covers performance estimation and hardware-aware NAS: how candidates are scored, how predictors make that cheap, how a real device can guide the search, and how NAS is applied beyond image classification, including language models, GANs, point clouds and pose estimation.

Comments


White Structure

STAY IN THE KNOW

Thanks for submitting!

NoraTech

i@nshareef.com

©2023 by NoraTech 

bottom of page