Implement the Residual Gradient algorithm for policy evaluation with linear function approximation in reinforcement learning.
Standard semi-gradient TD methods only differentiate part of the temporal difference error with respect to the weights, ignoring the gradient through the bootstrapped target. The residual gradient algorithm instead minimizes the full mean squared Bellman error by computing the complete gradient, including the contribution from the next-state value estimate.
For linear function approximation, the value of state s is V(s; w) = phi(s)^T w, where phi(s) is the feature vector for state s.
Given:
episodes: a list of episodes, where each episode is a list of (state, reward, next_state) tuples. A next_state of -1 indicates a terminal transition (next-state value is 0, feature vector is the zero vector).features: a numpy array of shape (n_states, d) containing the feature vector for each state.gamma: the discount factor.alpha: the step size (learning rate).n_passes: the number of complete passes through all episodes.Your function should: