Backpropagation dominates deep learning, yet it uses a mechanism the brain likely cannot. Specifically, the backward pass needs exact transposes of forward weight matrices. This is the weight transport problem. Sakana AI’s new paper, Diffusing Blame, confronts this constraint directly. The research team trains networks that obey Dale’s principle while avoiding weight transport entirely.
What is Error Diffusion?
Error Diffusion (ED) is a local learning rule, first proposed by Kaneko (2000). Each weight update depends on three signals only. These are presynaptic activity, a postsynaptic activation derivative, and a single global error sign. Consequently, ED never transports transposed forward weights or uses random feedback matrices. That locality makes ED naturally compatible with Dale’s principle. However, prior work demonstrated ED only on binary classification and MNIST.
The Dual-Stream Architecture
To satisfy that constraint, the research team split each layer into two streams. One stream is excitatory (p), and the other is inhibitory (n). The forward pass computes excitatory-minus-inhibitory preactivations for each stream:
Copy CodeCopiedUse a different Browserp_i = φ_i( +p_{i-1} Wpp − n_{i-1} Wnp + bp )
n_i = φ_i( +n_{i-1} Wnn − p_{i-1} Wpn + bn )
Here, all four weight matrices stay non-negative element-wise. The biases bp and bn are the exception, since they need not be non-negative. Moreover, the negation signs before Wnp and Wpn are structural, not learned. Therefore cross-stream connections remain inhibitory while all learnable weights stay non-negative. This design needs four weight sub-matrices per layer. As a result, it uses roughly 4× more parameters than a single-stream network. For the same architecture, that is ∼32M versus ∼8M for DFA.
Modulo Error Routing
With that architecture in place, the main extension is modulo error routing. This lifts Error Diffusion (ED) beyond binary classification. For hidden unit i, the research team define the routing r(i) = i mod C. Here, C is the output dimension. That unit then learns from the routed error component. In short, each hidden unit is assigned one fixed output channel. Unlike DFA, whose feedback matrices are random, ED uses this structured correspondence.
Three Classification Innovations
Building on that routing, the research team adds three fixes for multi-class classification:
Layer-specific sigmoid widths use φi(z) = 1/(1 + e−2z/αi). Since the sigmoid derivative directly gates the error signal, attenuation is severe. In fact, post-hoc analysis reveals a 25× decay from the output to the first hidden layer. Wider sigmoids keep derivatives larger, preventing premature saturation. The team sets α = 3.0 for CIFAR-10 convolutional layers and α = 6.0 for fully connected layers.
Batch-centered class error subtracts the per-class mini-batch mean. This makes the one-vs-all error zero-mean across the batch for every class. It thereby reduces persistent suppression caused by the 9:1 target imbalance.
Asymmetric initialization scales excitatory weights by 1.5× and inhibitory weights by 0.5×. That gives an expected E/I scale ratio of 3:1, while the output layer stays symmetric.
(function(){
window.addEventListener(“message”, function(ev){
var d = ev.data || {};
if (d && d.type === “mtp-ed-resize” && d.height){
var f = document.getElementById(“mtp-ed-frame”);
if (f) f.style.height = d.height + “px”;
}
});
})();
Performance
With all three innovations, Error Diffusion (ED) reaches 96.7% on MNIST and 61.7% on CIFAR-10. In contrast, seed ED without them collapses to 50.4% and 11.6%. DFA scores higher on both tasks but violates Dale’s principle, using ∼2.84M negative weights. Notably, this is the first time ED has trained convolutional networks. Previously, Fujita (2026) reached ∼55.2% on CIFAR-10 using a flattened MLP. Even so, 61.7% remains far from standard gradient-based methods.
MethodMNISTCIFAR-10Dale-compliantNotesProposed ED96.7%61.7%YesAll weights non-negative; first ED on CNNsSeed ED50.4%11.6%YesNo innovations; α = 1.0, raw error, symmetric initDFA97.6%69.1%NoRandom feedback; ∼2.84M negative weights
The Ablation Reversal
Interestingly, the innovations’ importance flips between tasks. On MNIST, removing layer-specific widths is catastrophic (−71.4 pp), collapsing accuracy toward chance. Batch-centering barely matters there (−0.3 pp). On CIFAR-10, however, the order reverses. Removing batch-centered error becomes the largest drop (−47.9 pp), collapsing four of five seeds. This reversal exposes task-dependent credit-assignment bottlenecks invisible to single-benchmark evaluation.
Error Diffusion in Reinforcement Learning
Beyond classification, the research team integrate ED with Proximal Policy Optimization (PPO). They call the result ED-PPO and test it on Brax locomotion and Craftax. Here, policy-output error is routed to hidden units by output channel. For the scalar value network, the error is broadcast to all units. Importantly, ED-PPO drops the three classification innovations entirely. Across five seeds, ED-PPO beats BP-PPO on HalfCheetah (5494 vs 3520; p < 0.001) and matches DFA-PPO. On Ant, it stays on par with both PPO variants. On Craftax, meanwhile, DFA-PPO is the weakest method (19.8 vs BP-PPO 27.0). Thus random feedback that suffices for classification can fail on open-ended RL.
Use Cases and Examples
Three settings make this concrete:
Neuromorphic and photonic hardware often encodes non-negative synaptic magnitudes physically. ED’s fixed-sign routing maps cleanly onto such substrates, complementing prior photonic DFA work.
The non-negative floor drives 37.3% of weights to the floor (10⁻⁴) after training. Inhibitory cross-stream fully connected connections are pruned most, up to 68.8%. This implicit sparsity hints at model compression “for free.”
The dedicated inhibitory stream may help continual and open-ended learning. It provides a structural mechanism for dampening large gradient excursions.
Comparison
#mtp-ed-cmp{background:#0b0b0b!important;color:#e8e8e8!important;border:1px solid #1e1e1e!important;border-radius:12px!important;padding:20px!important;max-width:960px!important;margin:0 auto!important;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Helvetica,Arial,sans-serif!important;box-sizing:border-box!important}
#mtp-ed-cmp *{box-sizing:border-box!important}
#mtp-ed-cmp h3{margin:0 0 4px 0!important;font-size:18px!important;color:#ffffff!important;font-weight:700!important}
#mtp-ed-cmp .cap{margin:0 0 14px 0!important;font-size:13px!important;color:#9a9a9a!important;line-height:1.5!important}
#mtp-ed-cmp .scroll{overflow-x:auto!important;-webkit-overflow-scrolling:touch!important;border:1px solid #191919!important;border-radius:10px!important}
#mtp-ed-cmp table{border-collapse:collapse!important;width:100%!important;min-width:820px!important;background:#0e0e0e!important;font-size:12.5px!important}
#mtp-ed-cmp th,#mtp-ed-cmp td{border:1px solid #1c1c1c!important;padding:10px 12px!important;text-align:left!important;vertical-align:top!important;color:#d8d8d8!important;line-height:1.45!important}
#mtp-ed-cmp thead th{background:#141414!important;color:#FF9A3D!important;font-weight:700!important;font-size:11.5px!important;text-transform:uppercase!important;letter-spacing:.4px!important;position:sticky!important;top:0!important}
#mtp-ed-cmp tbody tr:nth-child(even){background:#101010!important}
#mtp-ed-cmp tr.mtp-hl td{background:#1a0f00!important;color:#ffe6cc!important;border-color:#3d2408!important;font-weight:600!important}
#mtp-ed-cmp tr.mtp-hl td:first-child{border-left:3px solid #FF7A18!important}
#mtp-ed-cmp td:first-child,#mtp-ed-cmp th:first-child{min-width:170px!important}
#mtp-ed-cmp a{color:#FF7A18!important;text-decoration:none!important;font-weight:600!important}
#mtp-ed-cmp a:hover{text-decoration:underline!important}
#mtp-ed-cmp code{background:#161616!important;color:#FF9A3D!important;padding:1px 5px!important;border-radius:4px!important;font-size:11.5px!important;border:1px solid #262626!important}
#mtp-ed-cmp .yes{color:#FF7A18!important;font-weight:700!important}
#mtp-ed-cmp .no{color:#8a8a8a!important;font-weight:700!important}
#mtp-ed-cmp .foot{margin-top:14px!important;padding-top:12px!important;border-top:1px solid #1c1c1c!important;font-size:11.5px!important;color:#6f6f6f!important;text-align:center!important}
#mtp-ed-cmp .foot b{color:#FF7A18!important}
#mtp-ed-cmp .take{margin-top:12px!important;padding:11px 14px!important;background:#1a0f00!important;border:1px solid #3d2408!important;border-radius:8px!important;font-size:12.5px!important;color:#ffd9b0!important;line-height:1.5!important}
#mtp-ed-cmp .take b{color:#FF9A3D!important}
@media (max-width:640px){#mtp-ed-cmp{padding:14px!important}#mtp-ed-cmp h3{font-size:16px!important}}
How Dale-Compliant Error Diffusion Compares
Proposed approach vs. other backpropagation-free and biologically motivated learning rules. “Dale-compliant” means separate excitatory/inhibitory populations with non-negative weights. Method names link to primary sources.
Method
Backprop-free (no weight transport)
How error reaches hidden layers
Dale-compliant (E/I, non-negative)
Shown on RL
Demonstrated reach / notes
Error Diffusion — ED / ED-PPO (proposed)
Yes
Global error sign routed directly to hidden units via modulo routing r(i) = i mod C
Yes — dual-stream E/I, non-negative weights
Yes (Brax, Craftax)
96.7% MNIST, 61.7% CIFAR-10; RL returns on par with DFA-PPO
Backpropagation
No — needs transposed forward weights
Exact gradient, layer by layer
No — arbitrary-sign weights
Yes (BP-PPO)
Reference baseline; state of the art across tasks
Feedback Alignment (FA)
Yes
Fixed random backward weights, layer by layer
No — arbitrary-sign feedback
Not shown
Learns deep and convolutional nets; limited on harder benchmarks
Direct Feedback Alignment (DFA)
Yes
Output error to each hidden layer via fixed random matrices
No — random signed feedback (~2.84M negative weights)
Yes (DFA-PPO)
Scales to convnets and transformers; 97.6% MNIST, 69.1% CIFAR-10; weakest on Craftax
Dale’s ANNs (DANNs)
No — trained with backprop
Backpropagation
Yes — separate E/I populations
Not shown
Matches standard ANNs on supervised tasks
Predictive coding
Yes
Local prediction-error units (Hebbian)
No — not enforced
Not shown
Approximates backpropagation on supervised tasks
Dendritic cortical microcircuits
Yes
Local dendritic prediction errors
No — not enforced
Not shown
Approximates backpropagation; small-scale tasks
Evolution Strategies (ES)
Yes
Black-box parameter perturbation and reward (no per-layer error)
No — not enforced
Yes
Gradient-free; scales poorly with parameter count
Original Error Diffusion
Yes
Global error sign, local update (binary)
Yes — naturally Dalean
No
Binary classification; ~55.2% CIFAR-10 with a flattened MLP
Takeaway: most backprop-free rules relax Dale’s principle, and most Dale-compliant networks still rely on backpropagation. Error Diffusion satisfies both at once, and is the first to carry that combination into reinforcement learning.
Comparison compiled by Marktechpost · Method from Sakana AI, “Diffusing Blame” (Yamada et al., 2026)
Minimal Code Sketch
The equations translate into a compact, illustrative update loop:
Copy CodeCopiedUse a different Browserimport torch
def dual_stream_forward(p, n, Wpp, Wnp, Wnn, Wpn, bp, bn, phi):
# All W >= 0; cross-stream signs are hardcoded inhibitory (Dale’s principle)
p_next = phi(p @ Wpp – n @ Wnp + bp) # excitatory stream
n_next = phi(n @ Wnn – p @ Wpn + bn) # inhibitory stream
return p_next, n_next
def routed_error(S, H, C): # S: output error, shape (B, C)
M = torch.zeros(H, C)
for i in range(H):
M[i, i % C] = 1.0 # r(i) = i mod C
return S @ M.T # R = S M^T, shape (B, H)
def ed_update(A_p, Z_p, R, phi_deriv):
U_p = phi_deriv(Z_p) * R # local postsynaptic drive
return A_p.T @ U_p # dWpp ∝ A_p^T U_p, shape (K, H)
Key Takeaways
Sakana AI’s Error Diffusion trains Dale-compliant dual-stream networks without weight transport or random feedback matrices.
Modulo error routing (r(i) = i mod C) scales the rule past binary classification to 96.7% MNIST and 61.7% CIFAR-10.
Three classification innovations reverse in importance between MNIST and CIFAR-10, exposing task-dependent credit-assignment bottlenecks.
ED-PPO brings the same architecture to reinforcement learning, matching DFA-PPO on Brax and beating it on Craftax.
Dale’s principle costs 0.9–7.4 points versus DFA on classification, quantifying the price of non-negative weights.
Check out the Paper. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Sakana AI’s Error Diffusion Trains Dale-Compliant Dual-Stream Networks, Reaching 96.7% MNIST and 61.7% CIFAR-10 Without Backpropagation appeared first on MarkTechPost.