low variability while it is better to start from scratch when there is a high degree of variability. It is worth noting that MCTS does not consistently perform better than the greedy benchmark, likely because MCTS struggles with large and continuous state and action spaces. The specific solver choice is not important, the intent is more to show that formulating the problem as an MDP makes the problem amenable to a wide range of stochastic RL approaches. From a purely solver-tuning perspective, the performance of MCTS could be significantly improved by discretizing the state and action space (which also more accurately reflects blending and routing in practice). Other approaches such as particle swarm optimization or training a deep neural network with MCTS (as in Kumar et al. 2021) could be considered as well. Although the two scenarios presented are synthetic cases, they demonstrate the potential of an optimization-under-uncertainty approach to improve feedstock blending and routing. The blending network model and MDP formulation are sufficiently versatile to represent almost any mining operation’s blending network. Additionally, one of the advantages of using a reinforcement learning approach like solving an MDP with MCTS is that its decision-making is interpretable. The flow assignments depicted in Figure 5 are easy to implement and make logical sense. 6. CONCLUSIONS In this work, we structured the problem of feedstock blending and routing as a Markov Decision Process over a blending network. Formulating the problem in this way allows for clear identification and formalization of the sources of uncertainty: ore and process variability. Then, stochastic optimization techniques such as Monte Carlo tree search (MCTS) can be employed to solve the blending problem while accounting for uncertainty for better performance. An MCTS solver was shown to achieve higher profits more consistently with increased client contract fulfillment (reflecting improved product quality) in two synthetic scenarios that were chosen to demonstrate how solvers interact with uncertainty over time and represent realistic situations that a mining operation may face. However, there are clear limitations to an MCTS solver, especially when the problem space becomes larger and more parameters are included to, for example, represent ore deposits and their variability more comprehensively or model process variability with greater granularity. Instead of seeking other RL solvers, we believe that the most fruitful next step would be to extend the MDP to a partially observable MDP (POMDP) formulation. In the current formulation, as an MDP, there is only transition uncertainty: solvers do not know exactly how the feedstock or process conditions will vary between today and tomorrow. There is no state uncertainty: at every point in time, the solvers know the composition of feedstock at every node. In reality, however, the blending network has significant state uncertainty. Without real-time chemical analysis, as is still often the case in the mining industry, feedback from chemical analysis is generally not immediate, and may be delayed by 1-2 days. Even with real-time chemical analysis, it is wellestablished in academia and in industry that process outputs are heavily dependent not just on the chemical composition of feedstock, but the texture and mineralogy as well, which are much more difficult and costly to measure. At the current state of technology, it would be impractical and prohibitively expensive to monitor the texture and mineralogy of material across the whole blending network. As demonstrated by prior theoretical work, a POMDP formulation not only enables the consideration of state uncertainty, but perhaps more importantly, incorporates uncertainty
RkJQdWJsaXNoZXIy MTM0Mzk2