Question: Which of the following guidelines is applicable to initialization of the weight vector in a fully connected neural network.
A: Should not set it to zero since otherwise it will cause overfitting
B: Should not set it to zero since otherwise (stochastic) gradient descent will explore a very small space
C: Should set it to zero since otherwise it causes a bias
D: Should set it to zero in order to preserve symmetry across all neurons
Please eliminate two incorrect options first, then think it step by step and choose the most proper one option.