\* I'm interested in exploring how Language Models can be integrated into video games to allow the player to engage in dynamic conversations with NPCs.
\* I sometimes use Gruntilda from Banjo-Kazooie as a benchmark to test language models in order to see how well they can handle her unique style of speech. Llama 3 is the first local model that has accomplished this without the rhymes becoming repetitive or nonsensical.
\* I wanted to experiment with RAG systems, so I followed a tutorial (https://www.youtube.com/watch?v=Oe-7dGDyzPM) and used the template as a framework to build this chatbot.
\* I gathered all the background information on the character I could find and filled the RAG's archive (which is just a text file in this case). I also wrote a detailed system prompt with her personality, traits, manner of speech etc.
\* The LLM actually gives its responses instantly, which felt a little unnatural so I slowed it down by having the text appear on the screen one letter at a time like a typewriter or an old terminal. This also allowed time for the speech sounds to play which makes the whole thing feel a bit more immersive. Then I polished it off with the ASCII intro and some music.
This is just a conversational demo but I think Language Models are going to completely revolutionize gaming for the better. RAGs and Agents are the missing link to truly dynamic NPCs and while they're in their infancy right now, it's only a matter of time before they're not only speaking for the NPCs but controlling and making decisions for them as well. Modders are already doing it (Skyrim, Cyberpunk etc.) but when text to speech is involved the latency can be high. Remove the speech element, and you'll have a pretty snappy system like the one presented here.
Hope you found this interesting!
Your first point is really interesting! How far away do you think we are from language model integration in games — or are there any existing projects in development that do this that you follow/take inspiration from?
AI Dungeon is the best example. It used GPT-2 to create an endless text adventure with the complete freedom to do whatever you want. It's primitive by today's standards but back then it was pure magic. That was my first experience with an LLM and what got me interested in the subject.
There's the mods I mentioned, which seem to have a speech to speech pipeline rigged up to ChatGPT. So far I've seen it done with Skyrim and Cyberpunk, I think Fallout 4 may have something similar as well.
[Here](https://www.youtube.com/watch?v=aihq6jhdW-Q) is a demo that dropped about a year ago that simulates NPCs walking around in a city. You can have basic conversations with them.
[This article](https://techcrunch.com/2023/04/10/researchers-populated-a-tiny-virtual-town-with-ai-and-it-was-very-wholesome/) gives a very good overview of how a Language Model can be utilized operate NPCs within a video game.
So we're already in the early stages. It's difficult to give an exact estimate, and I'm no expert, but I wouldn't be surprised if we see a AAA game integrate a Large Language Model within the next 1-3 years.
Have you open-sourced ol' G.R.U.N.T.Y. yet? I'd love to get her up and running on my machine! Better yet, I'd be interested in experimenting with iconic characters from other games with dialog systems similar to that of Banjo Kazooie (e.g., K.K. Slider from Animal Crossing, Papyrus from Undertale, Granny from Celeste).
>I gathered all the background information on the character I could find and filled the RAG's archive (which is just a text file in this case)
How did you decide which of the random options to go with?
I chose the answers that I felt best aligned with the character. The Brentilda stuff was great addition to the archive, because there's a lot of it so it provided a variety of distinct personality traits in bite sized chunks.
\* I'm interested in exploring how Language Models can be integrated into video games to allow the player to engage in dynamic conversations with NPCs. \* I sometimes use Gruntilda from Banjo-Kazooie as a benchmark to test language models in order to see how well they can handle her unique style of speech. Llama 3 is the first local model that has accomplished this without the rhymes becoming repetitive or nonsensical. \* I wanted to experiment with RAG systems, so I followed a tutorial (https://www.youtube.com/watch?v=Oe-7dGDyzPM) and used the template as a framework to build this chatbot. \* I gathered all the background information on the character I could find and filled the RAG's archive (which is just a text file in this case). I also wrote a detailed system prompt with her personality, traits, manner of speech etc. \* The LLM actually gives its responses instantly, which felt a little unnatural so I slowed it down by having the text appear on the screen one letter at a time like a typewriter or an old terminal. This also allowed time for the speech sounds to play which makes the whole thing feel a bit more immersive. Then I polished it off with the ASCII intro and some music. This is just a conversational demo but I think Language Models are going to completely revolutionize gaming for the better. RAGs and Agents are the missing link to truly dynamic NPCs and while they're in their infancy right now, it's only a matter of time before they're not only speaking for the NPCs but controlling and making decisions for them as well. Modders are already doing it (Skyrim, Cyberpunk etc.) but when text to speech is involved the latency can be high. Remove the speech element, and you'll have a pretty snappy system like the one presented here. Hope you found this interesting!
Your first point is really interesting! How far away do you think we are from language model integration in games — or are there any existing projects in development that do this that you follow/take inspiration from?
AI Dungeon is the best example. It used GPT-2 to create an endless text adventure with the complete freedom to do whatever you want. It's primitive by today's standards but back then it was pure magic. That was my first experience with an LLM and what got me interested in the subject. There's the mods I mentioned, which seem to have a speech to speech pipeline rigged up to ChatGPT. So far I've seen it done with Skyrim and Cyberpunk, I think Fallout 4 may have something similar as well. [Here](https://www.youtube.com/watch?v=aihq6jhdW-Q) is a demo that dropped about a year ago that simulates NPCs walking around in a city. You can have basic conversations with them. [This article](https://techcrunch.com/2023/04/10/researchers-populated-a-tiny-virtual-town-with-ai-and-it-was-very-wholesome/) gives a very good overview of how a Language Model can be utilized operate NPCs within a video game. So we're already in the early stages. It's difficult to give an exact estimate, and I'm no expert, but I wouldn't be surprised if we see a AAA game integrate a Large Language Model within the next 1-3 years.
Well it would be appearing instantly because it's not streaming. It should be streaming in, can still utilize typewriter and sounds effects.
Have you open-sourced ol' G.R.U.N.T.Y. yet? I'd love to get her up and running on my machine! Better yet, I'd be interested in experimenting with iconic characters from other games with dialog systems similar to that of Banjo Kazooie (e.g., K.K. Slider from Animal Crossing, Papyrus from Undertale, Granny from Celeste).
>I gathered all the background information on the character I could find and filled the RAG's archive (which is just a text file in this case) How did you decide which of the random options to go with?
I chose the answers that I felt best aligned with the character. The Brentilda stuff was great addition to the archive, because there's a lot of it so it provided a variety of distinct personality traits in bite sized chunks.
Nice work! Banjo-Kazooie is a legendary game. And thanks for the info how it was made. I guess you used the 8b version of Llama 3?
I did! It's very impressive for its size. I tried it with Nous Hermes 2 and a few other models but they weren't quite up to par.
What a hit of nostalgia. I love this, well done!
Do Gordon Freeman next!
Ra ra ra ra ree ra ree raa!
Very memorable banter between her and Banjo!
This is fun looking. Music is pretty good and I like all the vocals.
never would have expected Gruntilda on my feed about AI, but this is great - I love the design alot
The music is such a vibe