Two different numbers, two different questions
Parameters answer "how much does this model weigh in memory?" — the total count of learnable weights and biases, which directly determines storage size and (roughly) memory bandwidth needed to load the model. FLOPs (floating-point operations) answer "how much compute does one forward pass take?" — the number of arithmetic operations needed to actually run the model on one input. A model can have a huge parameter count and modest FLOPs (or vice versa) depending on its architecture — they're genuinely different measurements, not two ways of stating the same thing.
Counting parameters in a dense layer
A fully-connected layer going from n input units to m output units has one weight per input-output pair, plus one bias per output unit:
parameters = (n × m) + m
A layer taking 256 inputs to 128 outputs has 256 × 128 + 128 = 32,896 parameters — every one of those needs to be stored and updated during training.
Counting FLOPs: why the "2" matters
Computing one output unit in a dense layer requires, for each input, one multiplication (input × weight) and one addition (accumulating into the running sum) — a multiply-accumulate operation. By long-standing convention in model-efficiency literature, this counts as 2 FLOPs, not 1 — one for the multiply, one for the add. For the same 256→128 layer:
FLOPs = 2 × (n × m) + m = 2 × 256 × 128 + 128 = 65,664
The + m accounts for the bias addition, which happens once per output unit regardless of how many inputs there are.
Why FLOPs and parameters scale differently as networks grow
For a single dense layer, FLOPs are almost exactly double the parameter count (2nm vs. nm, with the bias term negligible at scale) — a fixed, predictable ratio. But this relationship breaks down across an entire network's architecture choices: a very deep, narrow network reuses the same modest per-layer computation many times (more FLOPs relative to parameters, since compute is spent applying the same-sized transformation repeatedly), while a wide, shallow network concentrates parameters into fewer, larger layers. This is exactly why comparing two models by parameter count alone can be misleading about actual compute cost — architecture shape matters, not just total parameter count.
Why this doesn't cover convolutional or attention layers
Convolutional layers share weights across spatial positions (a small kernel slides across the whole input), so their parameter count is independent of input size while their FLOPs scale with how many spatial positions the kernel is applied to — a fundamentally different relationship than a dense layer's fixed n × m formula. Attention layers (the core of transformer architectures) have yet another distinct FLOPs formula, scaling with sequence length squared due to how every token attends to every other token. Neither is covered by dense-layer arithmetic — they need their own dedicated formulas.
Common mistakes
- Assuming parameter count alone predicts inference speed. Two models with identical parameter counts but different architectures (deep-narrow vs. wide-shallow) can have very different FLOPs, and therefore very different real-world compute cost.
- Forgetting FLOPs scale with batch size linearly, while parameter count does not — parameters describe the model itself; FLOPs describe the compute needed to process however many samples you're running through it.
- Applying dense-layer FLOPs formulas to convolutional or attention layers. Both have structurally different, non-interchangeable FLOPs formulas.
FAQ
Why is a multiply-accumulate counted as 2 FLOPs instead of 1?
Convention — it involves one multiplication and one addition, and model-efficiency literature counts both operations rather than treating the combined step as a single unit.
Do more parameters always mean more FLOPs?
Generally yes within the same layer type, but the ratio between them depends on architecture — very deep networks can have high FLOPs relative to their parameter count, since the same modest-sized computation gets repeated across many layers.
Does this calculator work for CNNs or transformers?
No — it only models fully-connected (dense) layers; convolutional and attention layers have their own distinct FLOPs formulas based on spatial dimensions or sequence length, not covered here.
Estimate parameters and FLOPs for any fully-connected architecture with the Neural Network FLOPs Calculator — entirely client-side.