AI Referee

[GeneH]

I have been trying to get AI to ref for myself and a couple others and haven't quite gotten it to work right. wondering if anyone has had any luck.

basically, i am ridiculously slow at reading, not many folk near me to play Traveller with, and the only option i would have would be if I were to run it. i don't have the time to learn it well enough to do so. I started trying to get Gemini to do it, then Gemini-linked with Notebook LM, then Chatgpt, and all have the same problem, they dont consistently follow the instructions. i wanted to try Claude but the PC version doesn't have ai voice (though supposedly its rolling out). Anyone have an success with this? i could get into much more detail on what i have tried to get these to work but it would be a very long poste.

 

[rinku]

The chatbots run into many issues when trying to implement rule sets - one of those is that they don't usually have legal access to the rulebooks and are relying on secondary or tertiary sources, can get editions confused, and are basically trying to guess what you want to hear.

You may be better using them as a Narrator, than a Referee. That is, you describe to them the situation in defined, real world terms, and let them tell you the story. You keep control of the rule based stuff like dice rolls, and tell the bot what has happened, so that it can then progress the story.

 

[CyborgPrime]

I have been trying to get AI to ref for myself and a couple others and haven't quite gotten it to work right. wondering if anyone has had any luck.

basically, i am ridiculously slow at reading, not many folk near me to play Traveller with, and the only option i would have would be if I were to run it. i don't have the time to learn it well enough to do so. I started trying to get Gemini to do it, then Gemini-linked with Notebook LM, then Chatgpt, and all have the same problem, they dont consistently follow the instructions. i wanted to try Claude but the PC version doesn't have ai voice (though supposedly its rolling out). Anyone have an success with this? i could get into much more detail on what i have tried to get these to work but it would be a very long poste.

Hi Gene!

Check out my Virtual GM project here:

Virtual GM - AI Powered Game Master HOME

Craft solo RPG adventures with AI assistance, immerse in diverse worlds, and share your creativity in our imaginative community.

virtualgm.cyborgprime.com

It's currently a working prototype in Closed Alpha - and maintenance mode while I work out some technical difficulties.

But you can see more about my experiments from a couple of years ago - we've come very far and have work arounds for most issues.
Virtual Game Master Tests:
https://youtube.com/playlist?list=PL9zjFpmxR97ooy5wcB3KCq9gTWJTC903m&si=PX3E6VzOYGITGwei

Come to my Discord for multiplayer ChatGPT Virtual GM demos (link on my website https://www.cyborgprime.com)

Finally, if you are interested in learning how to leverage AI to help reduce GM prep time and cognitive load, play solo, create consistent portraits of characters and NPCs, AI Ethics, and more, join me at the AI Game Master Academy
(https://www.skool.com/ai-game-master-academy-1648/about).

Hope you will join us!

 

[GeneH]

The chatbots run into many issues when trying to implement rule sets - one of those is that they don't usually have legal access to the rulebooks and are relying on secondary or tertiary sources, can get editions confused, and are basically trying to guess what you want to hear.

You may be better using them as a Narrator, than a Referee. That is, you describe to them the situation in defined, real world terms, and let them tell you the story. You keep control of the rule based stuff like dice rolls, and tell the bot what has happened, so that it can then progress the story.

to be clear it's only supposed to be using uploaded documents and those would include the rule book and about $150 worth of other stuff that I've purchased since started to play this. PDF format some purchase here some on Drive through. Not using anything from the web except for some more stuff off the wiki

 

[rinku]

Yeah, but does the bot know that? You'll probably have to specifically exclude web based resources, at which point you may run into the opposite problem of the bot not having *enough* resources.

I'm not saying you can't train them to be book accurate Referees, but it would take careful compartmentalisation.

 

[To Tha Moon!]

ChatGPT worked great for me.

It's like it read the books.

 

[Talon Greyfeather]

You should be able to specify the resources the AI can use. I know copilot can do it as I have built AI Agents for our sales teams that scour the 3000+ contracts for the items our customers are looking for. I'm certain ChatGPT and the others can do it as well ... just ask them how to do it

 

[rinku]

I guess it can depend on your expectations.

At heart, all RPG games are conversations, and they are good at those.

 

[swordtart]

I guess it can depend on your expectations.

At heart, all RPG games are conversations, and they are good at those.

They are good at simulating those

AI's often better when the prompt includes clues as to how you want them to behave like you are talking to an actor.

You will probably do better if you ask it to describe the challenges in terms of a tv show and hand crank the rules interaction yourself (then you can guarantee they will be followed).

If you only want a rules lawyer, my experience is that AI can be hit and miss. Some can be kept on task, and they are quite good at trawling a lot of information to find you a citation, but you usually need to double check that it hasn't just made up a load of nonsense. In my job I don't get to do less reading, I just spend less time finding what it is I need to read.

In a game it perhaps doesn't matter as much as if you are looking for legal advice

Some games run perfectly well with referees who play fast and loose with the rules anyway. Traveller was always supposed to be "here is the situation - what do you do" and the rules were as light as possible. CT adventures often had "If the players try to bluff their way through, throw 6+ to succeed without reference to any skill or characteristic. It was just a way to steer the plot, not a test of whether the players happened to obtain the right skills during character generation.

 

[Galadrion]

I rather like the idea of an AI referee, but at present I see two main problems with the concept.

First, the legal climate around AIs and IP is such that the AI programmers are extremely leery of having AIs make direct use of copyrighted material... such as game books. This means that people trying to set up a chatbot to act as a referee have to go to greater efforts in order to get that chatbot to actually use those rules. This problem is surmountable, but it takes a knowledgeable, firm, and consistent hand in order to implement - it's not terribly convenient at this time.

And second, most chatbots are simply too accommodating to make good game referees. At the present time, they are programmed to be as helpful and cooperative with their users as possible... which is a problem when you consider that a significant part of the referee's role is to be the opposition for the players. I've seen this repeatedly; the chatbot throws a challenge in the direction of the player(s), the player(s) come up with a plan of action in order to overcome the challenge... and the chatbot simply rolls over and plays dead for them. In the long run, this does not make for a satisfying game.

Both of these problems are, I believe, solvable, although not particularly easily at the present time. The legal/social issues are, I think, easier at present, particularly if the operator runs the chatbot offline and uses a custom dataset (including access to the rulebooks). The issue of an overly agreeable chatbot, however, is a harder matter to resolve - it's going to take someone with a considerable amount of programming experience (and probably a significant amount of formal game theory, as well) in order to instill a proper OPFORS response set to override the built-in cooperativeness. And there is going to have to a careful balance struck on the second point - an opposing mindset is desirable, but going overboard into "kill 'em all!" territory is also a problem.

 

[Condottiere]

On the other hand, you could sweet talk the chatbot into resurrecting your character, when it dies during character generation.

 

[DeepSpaceAdventures]

If only the reading part is the problem, there are audio book versions of the rules and several adventures and background books available at:

AudioRPG

AudioRPG

audiorpg.net

And there is the superb video series by Seth where he explains the rules in detail:

Traveller Overview

A multi-part review and how-to series about the Mongoose Traveller 2e RPG.

www.youtube.com

 

[hopsnbaer]

I don't think they make a good full GM, but I think prepping or even assisting during a game they are fine. They can be really useful for reorganizing or pulling out relevant information from a badly written published adventure.

 

[Endie]

I’ve not even tried, but I’m absolutely confident that none of the LLM models I’ve used for work or personal use (currently Claude, chaptgpt, Gemini, Grok on pro subs, not counting coding or task--specific ones from suppliers like jetbrains or Atlassian) are ready to work as an AI referee beyond very short one-off encounters.

The biggest current problem is that they lack the context memory to reference beyond a core rule book and a sourcebook along with some limited adventure progress (even Gemini, which has a large CM. They will, therefore, start to lose context and you’ll get very frustrated with the hallucinations and resulting narrative and ludic inconsistencies.

This will change in the next year. But for now, as @hopsnbaer suggests, they’re powerful GM aids but not fitted to this use case.

 

[hopsnbaer]

I’ve not even tried, but I’m absolutely confident that none of the LLM models I’ve used for work or personal use (currently Claude, chaptgpt, Gemini, Grok on pro subs, not counting coding or task--specific ones from suppliers like jetbrains or Atlassian) are ready to work as an AI referee beyond very short one-off encounters.

The biggest current problem is that they lack the context memory to reference beyond a core rule book and a sourcebook along with some limited adventure progress (even Gemini, which has a large CM. They will, therefore, start to lose context and you’ll get very frustrated with the hallucinations and resulting narrative and ludic inconsistencies.

This will change in the next year. But for now, as @hopsnbaer suggests, they’re powerful GM aids but not fitted to this use case.

You can create a Canvas that can be edited and uploaded into each conversation with the current campaign data and update it as you go. It is still not perfect.

 

[hopsnbaer]

Here is an opening conversation I used AI to assist with for High and Dry -

Aran Dursan (glancing at the paperwork, then at the crew)
“So. You’re the ones claiming the Hollow Star.” (He exhales through his nose, shaking his head slightly.)

“You must have an interesting backer, because this ship’s been nothing but trouble for over two centuries. And now, it’s your trouble.” (He taps the papers.)
“The ship’s registry matches, the mortgage transfer checks out, and as far as McClellan Factors is concerned, it’s yours. No outstanding liens—at least, not on paper.”

(He places the Vilani Fuel Regulator on the desk in front of them, spinning it idly with his fingers.)

“Last crew ran this thing hard. They pulled some shady tricks to dodge their escrow account, and as far as we were concerned, they’d skipped out. When they finally showed their faces here on Marduk, they walked straight into a bounty hunter’s warrant. Got shipped back to the Imperium in chains. Left the ship dead in the water on a rock called Walston—one of the moons out in the gas giant’s orbit.”

(He leans back slightly.)
“Funny thing is, they only needed this little guy.” (gestures to the regulator) “Cr50. That’s all that was standing between them and a working jump drive. The ship’s been sitting there ever since.”

(He slides the component across the desk.)

“You’ll need this if you actually plan on flying that relic off the moon. Consider it a courtesy.” (His tone suggests it’s nothing of the sort.)

“Now, I don’t care what you do with the Hollow Star, but let me be clear—McClellan Factors is done with it. You need help getting it running? That’s your problem. You need a new mortgage? Not my department. But if there are any **outstanding debts or enemies attached to that ship, I suggest you settle them before you light the drives.””

(He steeples his fingers, watching their reactions.)

“Any questions?”

 

[actionman]

I have been trying to get AI to ref for myself and a couple others and haven't quite gotten it to work right. wondering if anyone has had any luck.

basically, i am ridiculously slow at reading, not many folk near me to play Traveller with, and the only option i would have would be if I were to run it. i don't have the time to learn it well enough to do so. I started trying to get Gemini to do it, then Gemini-linked with Notebook LM, then Chatgpt, and all have the same problem, they dont consistently follow the instructions. i wanted to try Claude but the PC version doesn't have ai voice (though supposedly its rolling out). Anyone have an success with this? i could get into much more detail on what i have tried to get these to work but it would be a very long poste.

I've tried this a few times. The shortcomings of the AI models most people have access to become apparent very quickly.

It's memory is short. The AI forgets context, rules, and instructions. It forgets which characters it's supposed to be playing and what they're like.
It is incapable of making decisions on its own, and it can't make even a tolerable attempt at character agency. It will always end up asking you what it should do. If you don't tell it what to do, it will just muddle around like a conversation with a boring person that goes nowhere.
It's just a very complex series of if-then statements, it has no understanding, so you'll get a lot of responses that don't make sense. It doesn't have a sense of cause and effect. Even using "ai rpg" interfaces, it soon went off the rails. It failed to maintain a consistent environment, like your character walks into a shop in a village, and then walks out of the shop and the village is completely different.

The best results I got were with novel writing ai interfaces, where I defined the story "this story is about such and such, Character 1 is like this, Character 2 is like that, the setting is like this, etc.", and even then I had to tell it how the story should progress in every scene.

I think the best it can do is provide an assist to solo play, and even then it will very probably be more trouble than its worth.