Disciplines
Artificial Intelligence and Robotics | Data Science
Abstract (300 words maximum)
Machine learning provides new methods of problem solving through applied pattern recognition. An interesting challenge is to utilize machine learning in the automation of tasks and behaviors in virtual environments. Minecraft is an open-world, sandbox style game giving players nearly limitless freedom to alter a procedurally generated world. In the survival game mode, the player must collect resources to craft tools and build structures. The collection of resources can be tedious, so this project seeks to automate the standard initial task of collecting wood. By combining a convolutional neural network with API, a bot can collect resources while remaining scalable to procedural environments.
This project utilizes the API Mineflayer for movement, and the Yolov8 neural network architecture from the Ultralytics package in Python for object detection. Data was initially collected in the form of 512x288 pixel images, uniformly sampled from the student researchers’ Minecraft gameplay. The data were then scaled and manually labeled into 7 distinct classes for each type of tree. After training the neural network for 15 epochs, the network could detect trees with an average precision of 88.5% at a recall threshold of 50%.
This project has several limitations. Currently, the project is designed only to work on locally hosted servers. Furthermore, the bot’s point of view is generated by a simplified render of the Minecraft environment without dynamic lighting. Lastly, the difficulty is restricted so that the bot only encounters environmental threats.
Future researchers may take interest in addressing any of these limitations or may create new datasets and scripts for collecting different resources. This project can also be expanded to include a state machine that switches between neural networks and scripts to carry out more complex behaviors as a sequence of discrete tasks.
Academic department under which the project should be listed
CCSE - Computer Science
Primary Investigator (PI) Name
Min Wang
Machine Learning in Minecraft: Proof of Concept for Object Detection Oriented Autonomous Bots in Minecraft
Machine learning provides new methods of problem solving through applied pattern recognition. An interesting challenge is to utilize machine learning in the automation of tasks and behaviors in virtual environments. Minecraft is an open-world, sandbox style game giving players nearly limitless freedom to alter a procedurally generated world. In the survival game mode, the player must collect resources to craft tools and build structures. The collection of resources can be tedious, so this project seeks to automate the standard initial task of collecting wood. By combining a convolutional neural network with API, a bot can collect resources while remaining scalable to procedural environments.
This project utilizes the API Mineflayer for movement, and the Yolov8 neural network architecture from the Ultralytics package in Python for object detection. Data was initially collected in the form of 512x288 pixel images, uniformly sampled from the student researchers’ Minecraft gameplay. The data were then scaled and manually labeled into 7 distinct classes for each type of tree. After training the neural network for 15 epochs, the network could detect trees with an average precision of 88.5% at a recall threshold of 50%.
This project has several limitations. Currently, the project is designed only to work on locally hosted servers. Furthermore, the bot’s point of view is generated by a simplified render of the Minecraft environment without dynamic lighting. Lastly, the difficulty is restricted so that the bot only encounters environmental threats.
Future researchers may take interest in addressing any of these limitations or may create new datasets and scripts for collecting different resources. This project can also be expanded to include a state machine that switches between neural networks and scripts to carry out more complex behaviors as a sequence of discrete tasks.