### Lets talk about the following topics
1. Derivative
2. Derivative and partial derivative
3. Derivative and directional derivative
4. Derivative and gradient
5. Gradient descent algorithm

### Derivative


Definition of derivative:

$$
f'(x_0) = \lim_{\Delta x \to 0}\frac{\Delta y}{\Delta x} = \
\lim_{\Delta x \to 0}\frac{f(x_0 + \Delta x) - f(x_0)}{\Delta x}
$$

$f'(x_0)$ indicates the changing rate/trend of $f(x)$ along the positive x-axis direction, more intuitively to say, to some point of x:
  1. If $f'(x) > 0$, then the value of $f(x)$ will increase along the positive direction of x-axis;
  2. If $f'(x) < 0$, then the value of $f(x)$ will decrease along the positive direction of x-axis;
  
### Derivative and partial derivative
Definition of partial derivative:

$$
\frac{\partial}{\partial x_i}f(x_0, x_1, \dots, x_n) \
= \lim_{\Delta x \to 0}\frac{\Delta y}{\Delta x} \
= \lim_{\Delta x \to 0}\frac{f(x_0, x_1, \dots, x_i + \Delta x, \dots, x_n) - f(x_0, x_1, \dots, x_i, \dots, x_n)}{\Delta x}
$$

So as you can see, derivative and partial derivative are essentially same, just:
  1. Derivative indicates the changing rate/trend of unary function $f(x)$ along the positive x-axis direction;
  2. Partial derivative $\frac{\partial}{\partial x_i} (i = 0, 1, \dots, n)$ indicates the changing rate/trend of multivariable function $f(x_0, x_1, \dots, x_n)$ along the positive $x_i (i = 0, 1, \dots, n)$ axis direction;
  
### Derivative and directional derivative
Definition of directional derivative:

$$
\begin{align}
  \frac{\partial}{\partial v}f(x_0, x_1, \dots, x_n) \
  &= \lim_{v \to 0}\frac{\Delta y}{\Delta x} \
  = \lim_{v \to 0}\frac{f(x_0 + \Delta x_0, x_1 + \Delta x_1, \dots, x_i + \Delta x_i, \dots, x_n + \Delta x_n)}{v} \\
  \newline
  v &= \sqrt{(\Delta x_0)^2 + (\Delta x_1)^2 + \dots + (\Delta x_i)^2 + \dots + (\Delta x_n)^2}
\end{align}
$$

In previous derivative and partial derivative definitions, we were all talking about the changing rate/trend of $f(x)$ along the axes, if we want to calculate the changing rate/trend of $f(x)$ along any arbitrary direction(360-degree), then the directional derivative comes into picture.

That is: **directional derivative indicates the changing rate/trend of $f(x)$ along one arbitrary direction.**

### Derivative and gradient
Definition of gradient:

$$
grad\,f(x_0, x_1, \dots, x_n) = \
\Big(
  \frac{\partial f}{\partial x_0}, \
  \dots, \
  \frac{\partial f}{\partial x_i}, \
  \dots, \
  \frac{\partial f}{\partial x_n}
\Big)
$$

The notion of gradient is created only for answering one question: **in which direction of the function's independent variables space it has the maximum changing rate?**

That is: the function's gradient at one point is such a vector:
  1. It has the same direction with the directional derivative which has the maximum value at this point(at one point the directional derivative is 360-degree);
  2. And its norm is the value of above point 1 described directional derivative;
  
Here pay attentions to the below three points:
  1. Gradient is a vector, so it has both direction and magnitude;
  2. It has the same direction with the directional derivative which has the maximum value at that point;
  3. The value of the gradient is the value of above point 2 described directional derivative;
  
### Gradient descent algorithm
Since one function has the maximum changing rate along the gradient's direction at one point of the independent variables space, then if we move along the **reversed** direction of the gradient direction, we can reduce the value of the cost function most efficiently.

But how to move along the **reversed** direction of the gradient direction?
  
Since gradient and partial derivative are all vectors, then according to the rules of vector operating, we can just decrease correspondingly in each axis, that is:

$
\text{repeat until convergence }\Bigg\{
$

$$
x_0 = x_0 - \alpha\frac{\partial f}{\partial x_0} \\
x_1 = x_1 - \alpha\frac{\partial f}{\partial x_1} \\
\vdots \\
x_n = x_n - \alpha\frac{\partial f}{\partial x_n}
$$

$
\Bigg\}
$

### Credit to [[机器学习] ML重要概念：梯度（Gradient）与梯度下降法（Gradient Descent）](https://blog.csdn.net/walilk/article/details/50978864)

033d39d56d34ed2cec8377938f5dd505a02b9d76
ipynb
Jupyter Notebook
ml_basics/rdm002_gradient_and_gradient_descent/gradient_and_gradient_descent.ipynb
lnshi/ml-exercises
7482ca0f5bb599e6be0a02b50ca1db3c882e715f
MIT
2018-10-06T10:32:10.000Z
2019-07-04T14:55:55.000Z
ml_basics/rdm002_gradient_and_gradient_descent/gradient_and_gradient_descent.ipynb
lnshi/ml-exercises
7482ca0f5bb599e6be0a02b50ca1db3c882e715f
MIT
ml_basics/rdm002_gradient_and_gradient_descent/gradient_and_gradient_descent.ipynb
lnshi/ml-exercises
7482ca0f5bb599e6be0a02b50ca1db3c882e715f
MIT
2019-07-03T09:24:41.000Z
2019-07-03T09:24:41.000Z
Qwen/Qwen-72B
1. YES
2. YES