This disclosure describes a tabletop game assistant system configured to ingest and guide tangible games (such as board games, card games, etc.) using natural language interaction and image capture/visual display components. The system can include features enabling a game developer to “teach” the system the rules of a game using natural language, such as written instructions, to reduce or eliminate the need for writing dedicated code. The system may process images of a game board and/or tokens such as game pieces and/or cards to further generate game data in the form of a logical game model. The system can use the game data to guide human players of the game and, in some cases, participate as a player itself. The system may further be configured to observe a game and detect invalid actions, answer questions regarding the rules, and suggest moves. The system may provide additional utilities such as generating a random output (e.g., rolling virtual dice) and learning to recognize new game pieces.