Basic RNNs struggle to remember information from far back due to which problem?

Basic RNNs struggle to remember information from far back due to which problem?

• A. Overfitting
• B. Exploding gradient problem
• C. Underfitting
• D. Vanishing gradient problem

Answer: (D)

Basic RNNs struggle to remember information from far back because of the vanishing gradient problem during training. When you backpropagate through time, you repeatedly multiply by derivatives of the activation function (like tanh or sigmoid), which are often less than one in magnitude. Those multiplications long ago time steps cause the gradients to shrink exponentially, so the error signal becomes too small to meaningfully adjust the early recurrent connections. As a result, information from distant inputs fades, and the network can’t learn long-range dependencies. Techniques like LSTMs or GRUs address this by providing paths that keep gradients from vanishing. Exploding gradients are a different issue where gradients blow up rather than vanish, and they’re handled with clipping or normalization.

Basic RNNs struggle to remember information from far back because of the vanishing gradient problem during training. When you backpropagate through time, you repeatedly multiply by derivatives of the activation function (like tanh or sigmoid), which are often less than one in magnitude. Those multiplications long ago time steps cause the gradients to shrink exponentially, so the error signal becomes too small to meaningfully adjust the early recurrent connections. As a result, information from distant inputs fades, and the network can’t learn long-range dependencies. Techniques like LSTMs or GRUs address this by providing paths that keep gradients from vanishing. Exploding gradients are a different issue where gradients blow up rather than vanish, and they’re handled with clipping or normalization.