Consider a finite-state finite-action Markovian decision process with unobservable costs in the sense that the total discounted cost is to be assessed at infinity. It is assumed that the initial probability distribution over the state space is known...