This week I was thrilled to find that all this time I had not been rewarding the cubes for bringing food back to their nest, which is why it took them a million years to learn something so simple. The only reason they learned it at all is because they ultimately found that it was useful to return to the nest at some point because this it would allow them to eat more food.
So I of course quickly fixed this, and then amibitiously added the ability for the cubes to steal from other cubes' nests; however, I found that this resulted in them just learning to run in circles. This is because the position of the nests wasn't randomized on each episode (they were always arranged in a circle), and at any given time there's probably at least one piece of food in a cube's nest, so a cube had a good shot of picking up food if it just went in a broad sweeping circle, and then returned to its nest.
Rather than randomizing the nest locations to resolve this, I just decided not to bother with it for now and just trained the agents to pursue food and bring it back to their nest. Now that they were actually being rewarded for doing so, they learned much faster and performed much better.
At this point I will unceremoniously reveal the inspiration for the direction of this game (which has since changed, but this is still a great video). Apparently Adelie penguins build their nests out of rocks, and will sometimes steal from other penguins' nests when they're not looking to furnish their own. It's pretty cute:
The ultimate goal was to create a sort of nest-building game in which players have to build the largest nest or some kind of safe haven before some sort of calamity strikes. They can steal from one another's structures to construct their own, and they can also scare off opposing players by going into aggressive mode. The current build of the game (if it can be called that) just allows the player to run around, collect food, bring it back to their nest, and interact with the AI penguins.
There are a lot of aesthetic issues with this build; for instance, the part of the penguin that changes color when the penguin becomes aggressive is its belly for some reason, which is barely visible to the player. There are also currently no sound effects and very few other indicators to tell the player when they have picked up a piece of food, been attacked by another penguin, or dropped food off at their nest. These are all forms of feedback I'll need to include in future builds.
I had a LOT of trouble creating this build, because I've been in "development" mode all this time, meaning I didn't really think about all the unpleasant details of builds until this week. Switching out the cubes for penguins was not easy, because the penguin model used for Joundoom's feeding adventure was configured in a very specific way to work with physics in an increased timescale.
Furthermore, Unity ML-Agents comes with its own starter project, the Unity SDK, which contains every single ML-Agents example and all of the associated scenes. There is no way to exclude files from a build right now, and even though I can exclude all of the extra scenes from the build, I only needed the prefabs and materials from a single example - not all of them. My version of Unity ML-Agents did not have a single centralized package that one could simply stick into a new project to create their own ML-Agents environment; there were scripts and dll's lying all over the place that I needed to track down, and figuring this stuff out wasted a lot of time. The latest version of Unity ML-Agents was released literally last week, so I downloaded that one just to see if it was easier to create an independent project - and it was! There was a single package.json to make installing easy, but all of the fundamental scripts (the Agent, Academy, and Area scripts) had been changed, which would have forced me to change all of my scripts that derived from them. Needless to say, I did not go with this option.
Although I tried to get a WebBuild up for this week, I encountered 138 errors in the process of building, and I have no idea how to resolve any of them. So instead, I have a downloadable Windows build linked here.
To play the game, move forward with w, turn left with a, turn right with d, and go into aggressive mode with the space bar (there is no feedback beyond a speed boost to indicate that you've entered aggressive mode, so if you suddenly slow down after holding down the spacebar for a while, it means you've run out of food and are no longer in aggressive mode). Your nest is indicated by a light blue slab of land with a collection of rocks on it. All other rock piles are enemy nests. Note that you can't steal from other nests in this build.
For these last few weeks, there are a lot of things I want to and need to do to make this game playable and fun:
These are all things I need to think about and try to implement in the coming week or two, as I am really running out of time :(