A café’s app guesses which customers will come back. Four models were trained, each on some of this month’s 20 customers: for those, it saw the answer. Write honest_score(predictions, answers, trained_on): one model’s score, higher is better, so that the best score goes to the model that will do best on customers it has never seen, like next week’s.
Four models guessed which of this month’s 20 customers come back; each was trained on some of them (shaded) and saw their answers. Scored on every customer, Parrot comes top, then flops on next week’s new customers: it learned its customers by heart. The goal is scores that rank the models the way next week does, like the bars it ends on. Your honest_score() scores one model at a time.
honest_score(predictions, answers, trained_on)| predictions | what the model guessed for each of this month's customers, 1 = comes back, 0 = doesn't (a numpy array) | e.g. [1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1,… |
| answers | what each customer really did, 1 = came back, 0 = didn't (a numpy array) | e.g. [1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0,… |
| trained_on | 1 where the model was trained on that customer (it saw the answer), else 0 (a numpy array) | e.g. [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,… |
| returns | the model's score: higher is better | |