[#120] Move the schema documentation to where it is useful (#136)

SQLite only keeps the text of the CREATE statement itself in sqlite_master, so
the 58 comment lines documenting the schema in Init.sql reached nobody: .schema
on a produced database showed bare column names.

Comments now go inside the CREATE statements, where SQLite preserves them, so a
produced .db is self-describing. Only facts needed to write a correct query and
not guessable from the column name are kept there; the prose moved into the
documentation, which ships in the release zip since #118.

The same fix applies to AssetBundle.sql, the ContentLayout*.sql files (which
already had good in-paren column notes plus a discarded prose block above each
statement) and ContentSummary.sql. AGENTS.md records the convention.

Documentation/analyzer.md was 81% schema reference, so the schema moved to a new
strictly formatted Documentation/analyzer-schema.md and analyzer.md became the
overview. contentlayout-database.md gained the same markdown-table format. The
schema version history moved out of the Init.sql comment into a table on the new
page, and serialized_files, types, property_names and property_types gained
documentation they never had.

No table, view or column changed, so PRAGMA user_version stays at 7.

Also fixed while here: the ResX file-reference encoding for Init.sql and
Finalize.sql (Windows-1252 since the initial commit, every other .sql is utf-8),
the duplicated DDL in the Commands/SerializedFile classes (AddPreloadDependency
declared a primary key the real DDL never had), and nine pre-existing broken
documentation links.

Author: SkowronskiAndrew

AGENTS.md

@@ -126,7 +126,7 @@ The feature libraries each build on UnityBinaryFormat and UnityFileSystem. See t
 
 **Handlers**: Type-specific handlers extract specialized properties for Unity object types and populate additional tables. For example Mesh, AnimationClip, Shader, BuildReport, MonoScript.
 
-**Views**: The database schema includes convenient views for seeing the data in useful ways, e.g. `object_view`. See `Documentation/analyzer.md` and `Documentation/addressables-build-reports.md` for schema details.
+**Views**: The database schema includes convenient views for seeing the data in useful ways, e.g. `object_view`. See `Documentation/analyzer-schema.md` for the core schema, and `Documentation/contentlayout-database.md`, `Documentation/buildreport.md` and `Documentation/addressables-build-reports.md` for the parts documented on their own pages.
 
 CLI entry point is `UnityDataTool/Program.cs` using System.CommandLine. Per-command documentation is in `Documentation/`.
 
@@ -135,9 +135,24 @@ CLI entry point is `UnityDataTool/Program.cs` using System.CommandLine. Per-comm
 ### Extending Analyze
 
 * New Unity types can be added by following the same pattern as the existing types, for example MonoScripts.
-* Any database schema change (new or changed tables, views, or columns) must bump `PRAGMA user_version` in `Analyzer/Resources/Init.sql` and extend the version-history comment above it.
+* Any database schema change (new or changed tables, views, or columns) must bump `PRAGMA user_version` in `Analyzer/Resources/Init.sql` and add a row to the version table in `Documentation/analyzer-schema.md`.
 * Analysis of additional file formats could be added, for example AssetBundle manifest files by following the pattern of Addressables build layout files are handled.
 
+#### Commenting the .sql resources
+
+SQLite only keeps the text of the `CREATE` statement itself in `sqlite_master`, so a comment placed
+*above* a statement is discarded and never reaches a produced database. Comments therefore go inside
+the statement:
+
+* One short note on the first line inside the `CREATE TABLE ( ... )` parentheses saying what a row
+  represents, and a trailing `--` note on a column only where the fact is needed to write a correct
+  query and is not guessable from the column name (a foreign key, a sentinel like `''`, an option
+  that leaves the column empty). Views get one purpose line as the first line of the body, after `AS`.
+* Full prose belongs in the documentation, not in the `.sql` file.
+* `CREATE INDEX` has no body, so the only comments left outside a statement are ones that explain the
+  code rather than the schema, such as why the ContentLayout indexes are created after population.
+* Keep `.sql` comments ASCII-only.
+
 ### Other Extensions
 
 The UnityFileSystem API and UnityBinaryFormat parsing can be useful for other analysis.  The "dump", "analyze" and "serialized-file" commands can be considered reference examples of how to use those lower level tools.

Analyzer/Properties/Resources.resx

@@ -131,10 +131,10 @@
     <value>..\Resources\BuildReport.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>
   </data>
   <data name="Finalize" type="System.Resources.ResXFileRef, System.Windows.Forms">
-    <value>..\Resources\Finalize.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;Windows-1252</value>
+    <value>..\Resources\Finalize.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>
   </data>
   <data name="Init" type="System.Resources.ResXFileRef, System.Windows.Forms">
-    <value>..\Resources\Init.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;Windows-1252</value>
+    <value>..\Resources\Init.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>
   </data>
   <data name="Mesh" type="System.Resources.ResXFileRef, System.Windows.Forms">
     <value>..\Resources\Mesh.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>

Analyzer/Resources/AssetBundle.sql

@@ -1,39 +1,28 @@
--- tables related to the AssetBundle and PreloadData objects
-
--- Do not confuse the AssetBundle Unity object (the source of much of this data)
--- with the archives table, which is general to any Unity Archive.
-
--- The "assets" that an AssetBundle explicitly exposes: each m_Container entry of the AssetBundle
--- object names an object (the addressable/asset name -> object it maps to). Populated only from
--- the AssetBundle object, so this table is empty for Player and ContentDirectory builds.
--- For scene bundles the entry names the scene and points at the synthetic Scene object (see
--- AssetBundleHandler / SerializedFileSQLiteWriter).
 CREATE TABLE IF NOT EXISTS assetbundle_assets(
-    object INTEGER,
-    name TEXT
+    -- The assets an AssetBundle explicitly exposes: one row per m_Container entry of the
+    -- AssetBundle object. Empty for Player and ContentDirectory builds, which have no such object.
+    object INTEGER,     -- objects.id; for a scene bundle this is the synthetic Scene object
+    name TEXT           -- the container path the asset is addressed by
 );
 
--- object depends on dependency. This table has three sources, only the first of which is truly
--- AssetBundle-specific:
---   * AssetBundleHandler: an asset's slice of the AssetBundle object's m_PreloadTable.
---   * SerializedFileSQLiteWriter: a scene object -> each object in the scene's SerializedFiles.
---   * PreloadDataHandler: the PreloadData object's m_Assets. PreloadData is a *separate* Unity
---     object (not part of the AssetBundle object) and also exists in Player builds (one per scene
---     in its sharedAssetsN.assets, plus one in globalgamemanagers.assets), so this table is NOT
---     empty there. Player builds have no scene object, so those rows hang off the PreloadData
---     object itself; scene bundles hang them off the synthetic Scene object.
 CREATE TABLE IF NOT EXISTS preload_dependencies(
-    object INTEGER,
-    dependency INTEGER
+    -- Objects that Unity preloads alongside another object. Populated for AssetBundle and Player
+    -- builds, but not ContentDirectory builds. See Documentation/analyzer-schema.md.
+    object INTEGER,     -- objects.id: an AssetBundle asset, a synthetic Scene, or a PreloadData object
+    dependency INTEGER  -- objects.id, or dangling_refs.id when the target was not analyzed
 );
 
 CREATE VIEW IF NOT EXISTS assetbundle_asset_view AS
+-- AssetBundle assets with their object columns resolved. Inner join, so an asset whose object was
+-- not analyzed is omitted here but still present in assetbundle_assets.
 SELECT
     a.name AS asset_name,
     o.*
 FROM assetbundle_assets a INNER JOIN object_view o ON o.id = a.object;
 
 CREATE VIEW IF NOT EXISTS preload_dependencies_view AS
+-- Preload dependencies of AssetBundle assets and scenes, with both sides resolved. Narrower than
+-- the table: Player-build rows and dangling dependencies drop out of the inner joins.
 SELECT a.id, a.asset_name, a.archive, a.type, od.id dep_id, od.archive dep_archive, od.name dep_name, od.type dep_type
 FROM assetbundle_asset_view a
 INNER JOIN preload_dependencies d ON a.id = d.object

Analyzer/Resources/ContentLayout.sql

@@ -1,8 +1,7 @@
--- Identity of the imported ContentLayout.json (see Documentation/contentlayout.md). The
--- content_layout* tables are only created when a ContentLayout.json is part of the analyzed
--- input. A single layout per database is supported.
 CREATE TABLE IF NOT EXISTS content_layout
 (
+    -- Identity of the imported ContentLayout.json. The content_layout* tables exist only when a
+    -- layout was part of the analyzed input. See Documentation/contentlayout-database.md.
     id INTEGER,                 -- always 0 (single layout per database)
     name TEXT,                  -- path of the imported ContentLayout.json
     version INTEGER,            -- schema version of the json file

Analyzer/Resources/ContentLayoutArtifactReferences.sql

@@ -1,9 +1,8 @@
--- Direct references between binary artifacts (ArtifactReferences in the json), e.g. a
--- serialized file referencing its .resS/.resource data files. References that go through a
--- loadable are not included, and the graph is never cyclical. References to other serialized
--- files are not recorded here either; those are in content_layout_serialized_file_dependencies.
 CREATE TABLE IF NOT EXISTS content_layout_artifact_references
 (
+    -- Direct references between binary artifacts, e.g. a content file to its .resS/.resource data
+    -- files. Never cyclical. Content-file-to-content-file edges live in
+    -- content_layout_serialized_file_dependencies instead.
     artifact_index INTEGER,           -- references content_layout_binary_artifacts.artifact_index
     referenced_artifact_index INTEGER,
     PRIMARY KEY (artifact_index, referenced_artifact_index)

Analyzer/Resources/ContentLayoutBinaryArtifacts.sql

@@ -1,11 +1,9 @@
--- The artifacts that make up the build output (BinaryArtifacts in the json): the serialized
--- files themselves plus the data files they use (.resS, .resource) and the manifest. This is the
--- standard place to find artifact sizes. When stored as a file the filename is the content hash
--- plus an extension derived from the category (content_layout_binary_artifacts_view adds it).
 CREATE TABLE IF NOT EXISTS content_layout_binary_artifacts
 (
+    -- Every artifact making up the build output: the content files plus the data files they use
+    -- (.resS, .resource) and the manifest. The standard place to find artifact sizes.
     artifact_index INTEGER,
-    content_hash TEXT,
+    content_hash TEXT,          -- the on-disk filename is content_hash plus an extension from category
     category TEXT,              -- 'texture' | 'mesh' | 'audio' | 'video' | 'contentfile' | 'manifest'
     size INTEGER,
     PRIMARY KEY (artifact_index)

Analyzer/Resources/ContentLayoutIndexes.sql

@@ -1,6 +1,4 @@
--- Created after the content_layout tables are populated, so inserts stay fast for very large
--- layouts. The content_hash and asset_path indexes carry the views; the rest serve reverse
--- lookups ("who depends on X", "which loadables live in file Y").
+-- Created after the content_layout tables are populated so inserts stay fast for very large layouts.
 CREATE INDEX content_layout_binary_artifacts_content_hash ON content_layout_binary_artifacts(content_hash);
 CREATE INDEX content_layout_source_assets_asset_path ON content_layout_source_assets(asset_path);
 CREATE INDEX content_layout_source_assets_file ON content_layout_source_assets(serialized_file_index);

Analyzer/Resources/ContentLayoutLoadableDependencies.sql

@@ -1,6 +1,6 @@
--- Loadable objects referenced from each serialized file (LoadableDependencies in the json).
 CREATE TABLE IF NOT EXISTS content_layout_loadable_dependencies
 (
+    -- The loadable objects each content file references.
     serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index
     object_id_hash TEXT              -- references content_layout_loadable_objects.object_id_hash
 );

Analyzer/Resources/ContentLayoutLoadableObjects.sql

@@ -1,17 +1,14 @@
--- The objects that can be loaded on demand (LoadableObjectIds in the json), identified
--- independently of the serialized file that contains them. Also records where each one came from
--- in the source project. The json's top-level RootAssets list is folded into the is_root_asset
--- flag. serialized_file_index is NULL when the object was dropped from the build (json value -1,
--- e.g. server build shader references).
 CREATE TABLE IF NOT EXISTS content_layout_loadable_objects
 (
+    -- The objects that can be loaded on demand, identified independently of the content file that
+    -- holds them, plus where each came from in the source project.
     object_id_hash TEXT,             -- hash of GUID, LFID and identifier_type
     guid TEXT,                       -- AssetDatabase GUID of the source asset
     asset_path TEXT,
     lfid INTEGER,                    -- local file id of the object in the source asset
     identifier_type INTEGER,
-    serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index, or NULL
-    output_lfid INTEGER,             -- local file id of the object in its output serialized file
+    serialized_file_index INTEGER,   -- content_layout_serialized_files.file_index; NULL if dropped from the build
+    output_lfid INTEGER,             -- local file id of the object in its output content file
     is_root_asset INTEGER,
     PRIMARY KEY (object_id_hash)
 );

Analyzer/Resources/ContentLayoutLoadableSceneDependencies.sql

@@ -1,6 +1,6 @@
--- Scenes referenced from each serialized file (LoadableSceneDependencies in the json).
 CREATE TABLE IF NOT EXISTS content_layout_loadable_scene_dependencies
 (
+    -- The scenes each content file references.
     serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index
     scene_path TEXT                  -- matches content_layout_loadable_scenes.path
 );