A Deep Dive into Pruning Patterns
Updated: Oct 23
A simple guide to choosing the pattern and the weights to prune
In this article
Where we stopped last time
Granularity: fine vs coarse
Five patterns in a convolution layer
3.1 Fine-grained: any weight, anywhere
3.2 Pattern-based: zeros in allowed shapes
3.3 Vector-level: a whole row of a kernel
3.4 Kernel-level: a whole 3×3 kernel
3.5 Channel-level: a whole filter
Criterion: which weights go?
4.1 Magnitude
4.2 Wanda: weights × inputs
Test yourself: one panel, five patterns
1. Where we stopped last time
In the last post we said that pruning means deleting weights we do not need. We also saw that nobody can try every possible choice, so in practice we answer four simpler questions.
This post answers the first two:
Granularity. In what pattern do we prune?
Criterion. Which weights do we remove?
The pattern decides how much speed we get. The criterion decides how much accuracy we keep.
The other two questions, how much to prune in each layer and how to fine-tune, are for the next post.
في المقال السابق عرفنا أن الـ pruning هو حذف الأوزان اللي ما نحتاجها. في هذا المقال نجيب عن سؤالين: بأي نمط نحذف (granularity- مستوى الحرَش - أي هل تفاصيل الحذف فيها خشونة (حذف بالجملة) أم ملساء (حذف مجزّء))؟ وأيّ الأوزان نحذف (criterion- معيار التشذيب )؟ النمط يحدّد السرعة، والمعيار يحدّد الدقة.
2. Granularity: in what pattern do we prune?
Granularity is the size of the piece we remove at one time. A single weight is the finest piece. A whole row is a coarse piece.
Take one 8 × 8 weight matrix, 64 weights, and remove 24 of them in two ways.

Fine-grained (unstructured). We may remove any weight, anywhere.
Good: full freedom. If a weight is not needed, it goes.
Bad: the zeros are scattered. The matrix keeps its size, and the pattern is irregular, so it is hard to run in parallel.
Coarse-grained (structured). We remove whole rows.
Good: drop the empty rows and we get a 5 × 8 matrix. The GPU runs it with the same matrix multiplication it always uses.
Bad: less freedom. A row goes as a whole, so the useful weights inside it go too.
التشذيب المجزء (غير المنظّم):يحذف أوزانًا منفردة بحرية، لكن الأصفار تتناثر وتبقى أبعاد المصفوفة كما هي، مما يصعّب تسريع الحسابات.
التشذيب بالجملة (المنظّم):يحذف صفوفًا كاملة، فيقلّ حجم المصفوفة ويسهل تشغيلها بكفاءة على معالج الرسوميات، لكنه قد يحذف أوزانًا مفيدة معها.
Count the work the way we did in the first post. For one input vector, MACs = rows × columns.
Before pruning: 8 × 8 = 64 MACs
Structured: 5 × 8 = 40 MACs, on any hardware
Unstructured: still 64 MACs, unless the hardware knows how to skip the zeros
So the two sides pull against each other:
Fine-grained | Coarse-grained | |
Freedom to choose | High | Low |
Keeping accuracy | Easier | Harder |
Speedup on a normal GPU | Hard | Easy, it is just a smaller matrix |
خشونة/نعومة granularity تتحدد بطريقة الحذف ، حذف أوزان متفرقة (unstructured) يحافظ على الدقة بسهولة، لكن المصفوفة تبقى بنفس حجمها ويصعب تسريعها. وإذا حذفنا صفوف كاملة (structured) نحصل على مصفوفة أصغر يشغّلها الـ GPU مباشرة، لكن الحفاظ على الدقة أصعب.
3. Five patterns in a convolution layer
A convolution layer gives us more patterns to choose from, because its weights have four dimensions:
c_o: output channels, which is the number of filters
c_i: input channels
k_h and k_w: the height and width of each kernel
If kernels and channels are new to you, start with An Interactive Guide to CNNs.
Example: a layer with 3 filters, 2 input channels and 3 × 3 kernels has 3 × 2 × 3 × 3 = 54 weights. Here is that layer pruned in five patterns:

Don't Panic, we will study each of them:
1- Fine-grained (unstructured) pruning
What is pruned, and how is it chosen?Only the filter's weights, never the input. Any weight anywhere can go: there is no menu and no shape. After training we sort the layer's weights by |w| and set the smallest ones to zero until we reach the target ratio. Decided once, then the mask is frozen; the input changes at every window, the mask does not.
Before and after (one 3×3 kernel, prune 4 of 9)

The zeros land wherever the small weights were. The kernel is still 3×3, the layer is still 3 × 2 × 3×3. Nothing got smaller; cells got zeroed.
في الـ fine-grained نحذف من أوزان الفلتر فقط (مانحذف من المدخلات x)، بدون نمط محدد، فقط نختار الأقل حساسية والطريقة هي : نرتّب الأوزان حسب |w| ونصفّر الأصغر حتى نوصل إلي الحد الي حددناه مسبقاً : ٤ )، مرة وحدة بعد التدريب ثم يثبت الماسك (الفلتر) . الأصفار تقع وين ما كانت الأوزان الصغيرة، والـ kernel يبقى 3×3 بنفس حجمه، ما صغر شيء، بس خانات صارت صفر.

Is there something that tracks where the zeros are?In the normal (dense) format, no. A zero is stored as the number 0 and takes its 4 bytes like any other weight. To save memory you must switch to a sparse format that stores only the non-zeros plus their positions:
dense: 9 values × 4 B = 36 B
sparse: 5 values × 4 B + 5 positions × 1 B = 25 BDense (normal) storage: write all 9, zeros included
.57 .00 .61 .00 .90 .00 .44 .00 .76
4B 4B 4B 4B 4B 4B 4B 4B 4B = 36 BA zero costs the same 4 bytes as .90. Nothing is saved.
Sparse storage (✔) : write only the non-zeros, plus where they are
values: .57 .61 .90 .44 .76 5 × 4 B = 20 B
positions: 0 2 4 6 8 5 × 1 B = 5 B
------
25 BPositions fit in 1 byte because 9 < 256. To use it, the hardware reads a pair at a time: “position 4 = row 1, col 1, multiply .90 by that input pixel”, and never touches positions 1, 3, 5, 7.
The saving grows with sparsity: at 90% (AlexNet's FC layers) it is about 5× smaller. This is how AlexNet went from 61 M to 6.7 M parameters.
If the zeros stay, how does this reduce MACs?
On a plain GPU it does not. The dense kernel reads all 54 weights, multiplies by 0 and adds 0: same 54 MACs, same latency, same bytes moved. To skip a zero the hardware must read the sparse format and multiply only the listed non-zeros.
GPUs are bad at this: 32 threads in a warp move in lockstep, so a thread with a zero still waits; and finding each non-zero's input means a scattered "gather" read instead of one streaming read, which is slower than the multiply it saves. Custom chips like EIE are built for exactly this and get the speedup.

Why questions
Why is it the most accurate pattern? Full freedom. Only the smallest weights go; nothing is removed because of its neighbours.
Why does it compress the most? Same reason: it finds every redundant weight, so 80–90% can go.
Why no GPU speedup? The zeros are irregular. GPU speed comes from doing the same thing on long regular rows; holes in random places break that.
Why still useful then? Memory. From Part 1: one DRAM read costs about 200 multiplies. Fewer bytes to move (and to download, and to fit on a phone) is a win even when the MACs stay.
Hardware summary
Plain GPU | Hardware built for sparsity | |
MACs (our layer) | 54 | 27 |
Latency | same | lower |
Weight bytes moved | 216 B dense | ~135 B sparse at 50%, ~55 B at 90% |
Memory read pattern | streaming | gather (indexed) |
في التخزين العادي (dense) الصفر يُحفظ كرقم ويأخذ 4 بايت مثل أي وزن، فما حصل توفير . لكي نوفّر، نخزّن بصيغة sparse: القيم غير الصفرية فقط + مواقعها (9 أوزان: 36 بايت dense مقابل 25 بايت sparse)، والتوفير يكبر كل ما زادت نسبة الحذف،
لكن الحذف وحده ما يقلل الـ MACs: الـ GPU العادي يقرأ الأوزان كلها ويضرب في الصفر ويجمع صفر، نفس العدد ونفس الوقت. تخطّي الأصفار يحتاج hardware يقرأ صيغة الـ sparse ويضرب غير الصفري فقط (مثل EIE)، والـ GPU ما يناسبه لأن أصفاره متفرقة والـ threads فيه تمشي بالتوازي مع بعض (الصورة أعلاه) ، بالتالي الثريد الي فيه أصفار ينتظر الثريد الأخرى .. فما يحصل فائدة حقيقية في السرعة .
فالمكسب الحقيقي للـ fine-grained هو الدقة والذاكرة (ملف أصغر، تنزيل أسرع، يركب على الجوال)، مو السرعة.
2- Pattern-based (unstructured) pruning
The idea in one line. Fine-grained lets every kernel drop any weights it likes. Pattern-based says: you may drop weights, but only in one of a few allowed shapes. The shape is chosen from a fixed menu, not from the weights freely. That is the whole idea of this method, and its price.
What is pruned, and how is it chosen? Still only the filter’s weights, still decided once and frozen. But now each 3×3 kernel must keep exactly one shape from this menu, all “centre plus three arms”, 4 of 9:

What the panel does for a kernel. For each kernel it adds up the weights each shape would keep and picks the biggest sum. For the kernel in the panel:

So the choice is still by magnitude; that is why .42 beats .23 for the last arm. But .57 sits in a corner, and no shape in the menu has a corner, so it can never be kept, however big it is.
Same for .28, .32 and .15. This is what “a strong weight goes to keep the pattern” means, and why the pattern row in the comparison table loses 8 strong weights at 46% while fine-grained loses 0.
Why doesn’t the mask change with the input? Same answer as before: the mask is decided once from the trained weights and then frozen. The input changes at every window position; the mask does not.
Why a fixed menu at all? Hardware. Any 4-of-9 shape would mean 126 possible shapes for the GPU to handle; with 4 shapes you write 4 fast routines and reuse them. Real methods (PatDNN) pick the menu from the data: the shapes that keep the most magnitude across the whole network. Trained kernels usually put their energy on the centre cross, so corners are what gets dropped. Our made-up .57 in a corner is simply the unlucky case.
If the zeros stay, how does this reduce MACs? Only with a library that knows the 4 shapes. On a plain GPU, nothing changes except storage, exactly like fine-grained; the win is that storing “shape #3” costs 2 bits per kernel instead of a position per weight.
في التشذيب النمطي لا يحذف الكيرنل ما يشاء، بل يختار شكلًا واحدًا من قائمة ثابتة (┴)، وهو الشكل الذي يحتفظ بأكبر مجموع أوزان. لذلك قد يُحذف وزن قوي في الزاوية فقط لأن لا شكل في القائمة يحتويها، وهذا هو ثمن هذا النوع من التشذيب. الماسك يُحدَّد مرة واحدة ثم يثبت، والقائمة الثابتة تسهّل الأمر على العتاد: 4 أشكال بدل 126، لكن بدون مكتبة تعرف هذه الأشكال لا يتغير شيء سوى التخزين.
3- Vector-level (structured) pruning
Now we stop removing single weights and remove a whole row of a kernel at once. The row is chosen by its total magnitude, and the kernel keeps its 3×3 size with one empty row. The zeros are now in a regular line, which is easier to skip than scattered ones, but a strong weight in a weak row goes with it.
✓ The three zeros are side by side in memory, so the code skips that row as one block instead of testing each weight
✓ Cheap to store: one small number per kernel saying “row 2 is empty”. In fine-grained, every kept weight needed its own position (one byte each, so 5 bytes for a 5-of-9 kernel); here all 9 positions are known from that one number
✗ A strong weight in a weak row is lost
✗ Still needs a library that knows about the empty row; a plain GPU sees a 3×3 kernel with zeros in it
4- Kernel-level pruning
One step bigger: remove a whole 3×3 kernel. Since each kernel connects one filter to one input channel, this means that filter simply stops looking at that channel. Nine weights go at once, so accuracy drops faster than before, but the hardware now skips nine multiplies in one go.
✓ Nine multiplies skipped in one go
✓ Simple to store: just “this kernel is off”
✗ Accuracy drops faster, nine weights go together
✗ The filter still has to be run; the layer is not smaller
5- Channel-level pruning
The biggest block: remove a whole filter. The output channel it produced disappears, and the next layer has one fewer channel to read. This is the only level where the layer really gets smaller, so it runs faster on any hardware with no special library. The price is the largest, because 18 weights, strong and weak, go together.
✓ The layer really shrinks: fewer filters, fewer MACs, less memory
✓ Speedup on any hardware, no special library
✓ The next layer gets smaller too (fewer input channels)
✗ Largest accuracy loss; strong and weak weights go together
✗ Usually needs fine-tuning to recover

على مستوى المتجه (صف/عامود) نحذف صفًا كاملًا من الكيرنل، وعلى مستوى الكيرنل نحذف كيرنل 3×3 كاملًا فيتوقف الفلتر عن النظر إلى قناة مدخل واحدة، وعلى مستوى القناة نحذف فلترًا كاملًا فتختفي قناة من المخرج. كلما كبرت الكتلة المحذوفة سهُل الأمر على العتاد وزادت خسارة الدقة. وحده التشذيب بالقناة channel-based pruning يصغّر الطبقة فعلًا، لذلك هو الأكثر استخدامًا.
4. Criterion: which weights go?
Every pattern above answered the same question in the same way: look at the weight’s size and drop the smallest, whether that meant one weight, a row, a kernel or a filter. That is the magnitude criterion, and it is the simplest one. But the weight’s size is not the whole story: what matters to the output is weight × input. This section covers both.
4.1 Magnitude. Score = |weight|. Small weights go, the inputs play no part. This is what every pattern above used.
4.2 Wanda. Score = |weight| × typical size of its input. A small weight on a large input stays; a large weight on a tiny input can go.
Wanda (weights and activations)
Magnitude asks “how big is the weight?”. Wanda asks “how big is the weight × the input it usually multiplies?”. The input sizes come from one pass over a small sample of data, with no training.
Before and after (one neuron, prune 1 of 3)

Same network, same budget, a different weight is zeroed. The zero is still a hole in a full-size kernel.
Wanda needs one extra fact per weight: how big is the number this weight usually multiplies? That is not written anywhere in the model; it only appears when real data flows through it. So we take a few hundred real examples, run them through the trained model once, and note the typical size of each input, without touching any weight.
A few hundred examples are enough because we only need a rough idea, “usually small” or “usually huge”, and that does not change much from one example to the next.
If the zeros stay, how does this reduce MACs?It does not. Wanda is about keeping accuracy, not speed.
Why
Why multiply by the input? The output sees weight × input, not the weight alone. A small weight on a big input matters.
Why does magnitude fail on LLMs? Past a few billion parameters, a few hidden features are about 100× larger than the rest, and the model depends on them. Magnitude deletes their small weights. At 50% on LLaMA-7B: magnitude 17.3 perplexity, Wanda 7.3, dense 5.7.
Hardware summary
Magnitude | Wanda | |
Data needed | none | a small sample (128 texts) |
Work to decide | sort the weights | one forward pass, then sort |
Storage, MACs, speed after pruning | as in 1a | as in 1a |
يوجد معيارين للتشذيب المجزئ (Fine-grained) الأول magnitude وهو يعنى بالأوزان فقط ، والثاني wanda وهو يهتم بالأوزان والمدخلات كذلك
السبب : مرات يكون فيه وزن صغير ولكنه يضرب في مدخل كبير جدا ، فيظل هذا الوزن مؤثر حتى لو أنه صغير لأن حاصل الضرب رح يكون كبير نظرا لكبر المدخل ، بالتالي من غير الصحيح تشذيب هذا الوزن إلا إذا كان المدخل صغير أيضًا.
هذي فكرة الواندا ، كيف طيب نجيب المدخلات ؟ نلذلك نمرّر بضع مئات من الأمثلة مرة واحدة ونسجّل الحجم المعتاد لكل مدخل، دون أي تدريب ودون لمس الأوزان. بضع مئات تكفي لأننا نريد فكرة تقريبية فقط: «صغير عادةً» أو «ضخم عادةً».
To wrap up, try the panel below. It puts all five patterns side by side, so you can check what you took away from this article.
في الختام، جرّب اللوحة التفاعلية التالية: تجمع الأنماط الخمسة في مكان واحد، لتختبر ما خرجت به من هذا المقال.
I hope you enjoyed this article ♥




Comments