History Trivia / AIDungeon2

Despite appearances [=AID2=] is not self-aware and does not learn from player inputs (any finetuning is done offline at the discretion of the developers). It may follow the "AI speaks directly to you" cliché, causing awe and confusion among players. You may notice the details given vary every time.

Its original version was powered by the XL-sized (1.5B internal parameters) '''General Pre-trained Transformer''' revision 2 (GPT-2), a neural network text predictor expensively ($ 150,000 USD) trained by [=OpenAI=] on a ~40GB corpus of uncensored Internet text linked with at least 3 karma on Reddit. Nick Walton (creator of [=AID2=] and a previous incarnation that used a less effective model) used transfer learning to finetune this model on the Choose Your Own Adventure format (with text scraped from the eponymous website). Several forks of the game exist, allowing local play (on CPU or beefy GPU -- with half precision floating point the XL model can fit onto 8GB VRAM).

It is powered by the XL-sized (1.5B internal parameters) '''General Pre-trained Transformer''' revision 2 (now with state caching!), a neural network text predictor expensively ($ 150,000 USD) trained by [=OpenAI=] on a ~40GB corpus of uncensored Internet text linked with at least 3 karma on Reddit. Nick Walton (creator of [=AID2=] and a previous incarnation that used a less effective model) used transfer learning to finetune this model on the Choose Your Own Adventure format (with text scraped from the eponymous website). Several forks of the game exist, allowing local play (on CPU or beefy GPU -- with half precision floating point the XL model can fit onto 8GB VRAM).

Despite appearances [=AID2=] is not self-aware and does not learn from player inputs (any finetuning is done offline at the discretion of the developers). It may follow the "AI speaks directly to you" cliché, causing awe and confusion among players. You may notice the details given vary every time.

Like most neural network models, GPT-2 is basically a black box. Machine Learning's primary focus is to build and train problem solvers rather than understand the solutions. Better ML architectures require relatively less training effort to perform well. The GPT-2 architecture itself is no longer state-of-the-art. Transformer-XL and [=XLNet=] are superior in design (T-XL allows for unlimited word tokens, while GPT-2 is hard-limited to 1024; [=XLNet=] also performs bidirectional prediction which is maybe overkill for [=AID2=] given the permutation overhead) but GPT-2 has been trained so much (three runs on a swarm of several hundred [=TPUs=]) it is still the best publicly available trained model.

In a nutshell, AI Dungeon 2 is a generative open-ended text adventure game with some templated scenarios and a limited textual memory to enforce some consistency.

It is powered by the XL-sized (1.5B internal parameters) General Pre-trained Transformer revision 2 (now with state caching!), a neural network text predictor expensively trained by [=OpenAI=] on a ~40GB corpus of uncensored Internet text linked with at least 3 karma on Reddit. Nick Walton (creator of [=AID2=]) used transfer learning to finetune this model on the Choose Your Own Adventure format (with text scraped from the eponymous website).
Several forks of the game exist, allowing local play (on CPU or beefy GPU -- with half precision the XL model can fit onto 8GB VRAM).

Despite appearances [=AID2=] is not self-aware and does not learn from player inputs (it may be trained offline using saved inputs at the discretion of the developers). It may follow the "AI speaks directly to you" cliché, causing awe and confusion among players. You may notice the details given vary every time.
The Transformer itself is an Attention-based architecture originally designed by Google Brain researchers (it is named Transformer because it converts word tokens to wavelet space to process them simultaneously, rather than sequentially word-for-word; Attention determines the relative importance of other tokens in the sequence in deciding which word should be generated next).

GPT-2 is basically a black box, Machine Learning's primary focus is to build and train problem solvers rather than understand the solutions. Better ML architectures require relatively less training effort to perform well.
The GPT-2 architecture itself is no longer state-of-the-art. Transformer-XL and [=XLNet=] are superior in design (T-XL allows for unlimited word tokens, while GPT-2 is hard-limited to 1024; [=XLNet=] also performs bidirectional prediction which is maybe overkill for [=AID2=] given the permutation overhead) but GPT-2 has been trained so much ($ 150,000 USD for three runs on several hundred [=TPUs=]) it is still the best publicly available trained model. And there you have it.

The Transformer itself is an Attention-based architecture originally designed by Google Brain researchers (it is named Transformer because it processes word tokens simultaneously after converting them to wavelet space rather than sequentially word-for-word).

The GPT-2 architecture itself is no longer state-of-the-art. Transformer-XL and [=XLNet=] are superior in design (T-XL allows for unlimited word tokens, while GPT-2 is limited to 1024; [=XLNet=] also performs bidirectional prediction which is maybe overkill for [=AID2=] given the permutation overhead) but GPT-2 has been trained so much ($ 150,000 USD for three runs on several hundred [=TPUs=]) it is still the best publicly available trained model. And there you have it.

It is powered by the XL-sized (1.5B internal parameters) General Pre-trained Transformer revision 2 (now with state caching!), a neural network text predictor expensively trained by [=OpenAI=] on a ~40GB corpus of uncensored Internet text linked with at least 3 karma on Reddit. Nick Walden (creator of [=AID2=]) used transfer learning to finetune this model on the Choose Your Own Adventure format (with text scraped from the eponymous website).