Intro

Backprop computes the partials wrt to the weights but can sometimes be difficult to check it’s implementation. A numeric method is an easy way to check the analytical implementation. The numeric method is slow and needs to be disabled during actual training.

Calculate an estimate of $\small\frac{\partial J}{\partial \theta}$ where $\theta$ are the parameters of the model sometimes denoted as $w$ and $J$ is the cost computed using forward propagation

Math

$\frac{\partial J}{\partial \theta} \approx \lim_{\varepsilon \to 0} \frac{J(\theta + \varepsilon) - J(\theta - \varepsilon)}{2 \varepsilon} = \frac{\partial J}{\partial \theta_a}$

Computing the difference $diff$ between the analytic implementation $\partial \theta$ and the numeric implementation $\partial \theta_a$

$diff = \frac{\vert\vert\partial\theta - \partial\theta_a\vert\vert_2}{\vert\vert\partial\theta\vert\vert_2 + \vert\vert\partial \theta_a\vert\vert_2}$

If the difference is small, the numeric approximation and the analytic value suggest a common implementation

Code