Sutton & Barto let a learner say which states it cares about. A non-negative interest marks how much we care about valuing accurately, and an emphasis , built from the interest, scales each update:
with for . Here is the -step return (9.16) using the current weights for the bootstrap value, and it becomes the full return when .
Write emphatic_nstep_td(episode, n, gamma, alpha, n_episodes, use_emphasis=True, w0=None) for a linear value function :
episode is a list of (x_t, R_{t+1}, I_t) tuples for : the feature vector of , the reward that follows, and the interest. The episode ends after the last tuple (terminal value 0).n_episodes times. Within each pass, make the updates for in order (each one is the update the online algorithm makes at time , or at the end of the episode), each using the weights as they are after the previous update.use_emphasis=False use for every (ordinary -step semi-gradient TD).w0 (zeros if None).Return the final weight vector.