Implement off-policy n-step temporal difference (TD) prediction for estimating state-value functions. In off-policy learning, data is generated by a behavior policy but we wish to evaluate a different target policy. To correct for the mismatch between these two policies, importance sampling ratios must be applied to the n-step returns.
Your function receives:
episodes: A list of episodes. Each episode is a list of (state, action, reward) tuples, where the reward is received after taking the given action in the given state. Each episode is assumed to terminate after its last tuple.behavior_policy: A 2D list of shape (num_states, num_actions) giving b(a|s), the probability of selecting action a in state s under the behavior policy.target_policy: A 2D list of shape (num_states, num_actions) giving pi(a|s), the probability of selecting action a in state s under the target policy.num_states: Number of states (states are integers 0 to num_states-1).num_actions: Number of actions.n: The number of steps for the n-step return.alpha: Learning rate.gamma: Discount factor.Your implementation should:
t, compute the appropriate n-step return (bootstrapping from the current value estimate when the lookahead does not reach the end of the episode, and using the truncated return otherwise).Return the final value estimates as a numpy array of shape (num_states,).