Implement SQLite stats logging

[artemigkh]

Overview

This PR implements a more performant and structurally robust alternative to the existing CSV logging paradigm that drastically reduces the overhead of creating and managing hundreds of file handles, massively speeds up logging by reducing the raw amount of bytes needed to be written to disk, and structurally enforces a schema to avoid inconsistencies and escape character bugs in existing CSV logging.

The intended use-case is to speed up AI-only automation games as well as massively simplify downstream analysis of AI autoplay data by removing the need for parsing and aggregating hundreds of GB of logs into tabular data. Though of course the community may find other uses for this framework, as well as lower the barrier to entry for DLL modders to implement stats they may want tracked over autoplay runs.

Usage

Clear and straightforward instructions for getting started using this framework, as well as examples and other important information are thoroughly documented at the top of CvGameCoreDLL_Expansion2/SqliteLogger.h and this is the intended starting point for anyone wanting to understand or start using the logger.

Enabling

Logging is gated behind the SQLITE_LOGGING custom mod option and is off by default. Enable it by setting the option in the database:

UPDATE CustomModOptions SET Value = 1 WHERE Name = 'SQLITE_LOGGING';

As a quick simplified example of how the logger is invoked:

1. A stats table is registered:

TableDef kColumns;
kColumns.push_back(ColumnDef("Civ",        Database::COLTYPE_TEXT));
kColumns.push_back(ColumnDef("Technology", Database::COLTYPE_TEXT));
kColumns.push_back(ColumnDef("Action",     Database::COLTYPE_TEXT));
GET_SQLITE_LOGGER().RegisterTable("TechChoices", kColumns);

2. A row to that table is written

GET_SQLITE_LOGGER().BeginLogRow("TechChoices")
  .bind(strPlayerName.c_str())   // Civ
  .bind(szTechType)              // Technology
  .bind("CHOSEN")                // Action
  .execute();

Implementation Details

• A game UUID string has been added to the CvGame object in order to identify and persist game instances, potentially across game saves and loads
  • For write speed and storage performance reasons, the UUIDs are mapped to an integer lookup in the uuid_dictionary table which CvGame resolves and caches when MOD_SQLITE_LOGGING is enabled. Each game's UUID is stored there as a 32-char uppercase hex string (dashes stripped) in a TEXT column, keyed to the compact integer id that stat rows reference.
• Sqlite table registration and logging to stats.db is managed by the SqliteLogger singleton, which is accessed via GET_SQLITE_LOGGER() and is responsible for creating tables, storing/managing/verifying table schemas, and handling the validation and inserts for row writes
  • The stats.db is created in the cache directory alongside the other SQLite files
  • The SqliteLogger manager also caches prepared insert statements for each table and reuses them to speed up writes
  • Schemas are automatically migrated on a per-table basis. If a registered table's live schema differs from that in stats.db it is recreated with the new registered schema.
• Tables with a user-defined schema are registered to the singleton, and are automatically augmented with the game ID and current turn
• After a table is registered, a log row can be executed against it, which also automatically binds the game ID and current turn. This is done via a type-safe fluent builder API which validates each bound value against the registered column type and column counts.
  • Misuse such as type mismatch, incorrect number of values, unregistered table, etc. are safely caught, turned into no-ops, and trigger an assert on debug builds. For full details see CvGameCoreDLL_Expansion2/SqliteLogger.h
• All SqliteLogger operations are gated behind MOD_SQLITE_LOGGING and do not depend on GC.getLogging() being enabled. No cost when the flag is disabled as stats.db is lazily opened on first write
• The implementation is likely not multiplayer compatible for several reasons, which is probably unproblematic for its intended use-case (AI only autoplay games). But it may have to be looked at in the unlikely event that somebody wants to use it for MP debugging.

Currently Implemented Logging

Currently, the following game information is logged to stats.db:

• Technology research events: tracks technology start and completion times by turn, civ, and technology
• Religious belief choices and events: founding of a pantheon and religion, enhancement, reformation, conquering of a religion, and all belief choices associated with each event
• Policy choices: tracks branch and policy choices by turn and civ
• Game result: scores, victory civ, victory type at game completion
• Military summary: mirrors MilitarySummary.csv
• World state: mirrors WorldState_Log.csv

Future Work

Naturally this PR represents only a starting point, and much more work around this is planned in the future:

• Batch Insert / Transactions. The current volume of rows written is very low, so few performance gains would be realized from batch inserts. However, if logging of map state mirroring the current mapstate_turnx.json were to be implemented, which logs information about every plot every turn, this could significantly decrease write times
• More logging. The current tables represent some very low hanging fruit, but a great deal more can be harvested from the game state. Next candidates are city/civ economy logs, building/wonder completion records, and Instant Yield / Handicap Yield triggers.