feat/labs migration by alanpeixinho · Pull Request #1968 · kernelci/dashboard

alanpeixinho · 2026-07-01T20:57:06Z

Summary

Migrates all read paths for lab information from JSONB misc fields (builds.misc->>'lab',
tests.misc->>'runtime') to the new labs table via lab_id foreign key, with JSONB fallback for rows not yet backfilled.
Every SQL query and Python helper that previously extracted lab names by parsing JSONB now uses COALESCE(labs.name, misc_fallback) through a LEFT JOIN on the labs table. This ensures:

New data (with lab_id populated by the ingester) reads from the structured FK
Historical data (without lab_id) still works via the JSONB fallback
After a full backfill, the JSONB fallbacks can be removed (all marked with TODO comments)

How to test

legacy database

Use a database where lab_id is NULL on existing rows
Verify all pages display lab names as before:
- Tree details page (boots/tests tabs) — lab column in test history
- Tree commits history page — build lab and test lab in filters and rows
- Hardware details page — lab in test/build summaries, history, and filters
- Build details page — lab column in test list
- Issue details page — lab column in test list
- Notifications/metrics endpoint — lab summary counts

Migrated database (test in a local database)

Run the migrations available
Run the ingester (which already fills lab_id)
- Verify all pages display lab names as before:
- Tree details page (boots/tests tabs) — lab column in test history
- Tree commits history page — build lab and test lab in filters and rows
- Hardware details page — lab in test/build summaries, history, and filters
- Build details page — lab column in test list
- Issue details page — lab column in test list
- Notifications/metrics endpoint — lab summary counts

Part of kernelci#1948 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

* For analysis queries we are going for lab column information first, and coalescing to json misc information. * All COALESCE expressions are marked with TODO comments for removal after the lab_id backfill is complete. Closes kernelci#1948 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

MarceloRobert · 2026-07-03T11:32:29Z



+def assign_lab_ids(builds_buf: list[Builds], tests_buf: list[Tests]) -> None:
+    """Resolve each instance's real lab name to a labs.id and set its lab_id FK.


If the lab name is unique in the database, why not use that as a primary key? In that case you'd make a first query to check which labs are not in the database yet, a second query to update the table if needed, but you wouldn't need the third query to get the id of the newly added labs

Mostly to avoid string keys, since this is an internal key, we might benefit more from integer indices.

Makes sense. I am worried that we are adding one query here, one query there and soon we will have a bloat of unnecessary queries in the ingester, but it's fine for now

It is a valid concern. If we plain to move some operations to ingestion time, we certainly are going to end up increasing "ingestion complexity". But I believe that, as long as we keep the ingester fast enough, simplifying the analysis step should be the goal.

MarceloRobert · 2026-07-03T11:40:51Z

                tests.number_value AS tests_number_value,
                tests.misc AS tests_misc,
                tests.environment_compatible AS tests_environment_compatible,
+                COALESCE(test_labs.name, tests.misc ->> 'runtime') AS tests_lab,


I think you forgot a TODO here

mentonin · 2026-07-03T01:37:53Z

    )
-    return list(result)
+    tests = []
+    for test in result:


Why not assign directly to lab like before?

Mostly because lab is now the column name, but thinking again. I might change the column name for lab_id, this way we keep the lab name free to use, and follow more closely the the foreign key nomenclature.

Changed mind, to really avoid rename hacking on the ORM, it was easier to just add a proper sql query instead.

mentonin · 2026-07-03T02:10:32Z

        result = []
        for row in rows:
            build_misc = row[11]
-            sanitized_build_misc = sanitize_dict(build_misc)


Why does this not use the fallback?

mentonin · 2026-07-03T02:12:12Z

    log_excerpt = models.CharField(max_length=16384, blank=True, null=True)
    misc = models.JSONField(blank=True, null=True)
+    lab = models.ForeignKey(
+        Labs, db_constraint=False, null=True, on_delete=models.DO_NOTHING


Shouldn't we use db_constraint=True?

ditto for Tests

I see all other tables are using db_constraint=False, so there must be a good reason for this, but I do not see it documented

For the kci keys, we unfortunately have some missing keys in a few rows. But since lab has only internal keys, I believe we can safely enforce constraints on them. Good point.

mentonin · 2026-07-03T11:41:38Z

+    build_lab_summary = builds_summary.labs.get(lab)
+    if not build_lab_summary:
+        build_lab_summary = StatusCount()
+        builds_summary.labs[lab] = build_lab_summary


Copy pattern from previous chunk for consistence and readability

The point is, this is the original implementation (pre-refactor)

misc = sanitize_dict(build.misc) or {} lab = misc.get("lab", UNKNOWN_STRING) if lab: build_lab_summary = builds_summary.labs.get(lab) if not build_lab_summary: build_lab_summary = StatusCount() builds_summary.labs[lab] = build_lab_summary setattr( builds_summary.labs[lab], status_key, getattr(builds_summary.labs[lab], status_key) + 1, )

The conditional for lab is never false, because we are assigning UNKNOWN_STRING for lab.
However, it seems that this function is no longer used, and might be a legacy function used before performance improvements on HardwareDetails endpoints.
I will confirm it, and if this is the case, remove it.
We might as well plan to perform some dead code elimination, and check for similar cases.

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Rename Builds/Tests FK field from lab to lab_id so lab is free for annotated lab names without conflicting with the ORM relation. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Keep lab as the standard Django FK field and resolve build test lab names via raw SQL, matching the hardware query pattern. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Enforce the lab FK db_constraint on builds and tests, drop the unused lab_id from the build tests response, and fix the tree query lab TODO. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

alanpeixinho added 2 commits June 29, 2026 17:31

feat: Add Lab dimension table

a4f30bf

Part of kernelci#1948 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

feat: ingester filling labs table and lab_id column

ec84431

Part of kernelci#1948 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

alanpeixinho marked this pull request as draft July 1, 2026 20:59

alanpeixinho changed the title ~~feat/labs migration~~ {feat/labs migration Jul 1, 2026

alanpeixinho changed the title ~~{feat/labs migration~~ [WIP] feat/labs migration Jul 1, 2026

alanpeixinho force-pushed the feat/labs-migration branch from 7ee4011 to 80d7ce6 Compare July 2, 2026 20:51

alanpeixinho marked this pull request as ready for review July 2, 2026 20:53

alanpeixinho force-pushed the feat/labs-migration branch from 80d7ce6 to 3404f84 Compare July 2, 2026 21:16

alanpeixinho changed the title ~~[WIP] feat/labs migration~~ feat/labs migration Jul 2, 2026

MarceloRobert reviewed Jul 3, 2026

View reviewed changes

mentonin suggested changes Jul 3, 2026

View reviewed changes

alanpeixinho added 7 commits July 7, 2026 10:49

fixup! feat: Change queries to use lab column information

c1bee99

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fixup! feat: Change queries to use lab column information

4d99007

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fix: Dead code removal

b2452b0

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fixup! feat: Change queries to use lab column information

779f8ec

Rename Builds/Tests FK field from lab to lab_id so lab is free for annotated lab names without conflicting with the ORM relation. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fixup! feat: Change queries to use lab column information

9600970

Keep lab as the standard Django FK field and resolve build test lab names via raw SQL, matching the hardware query pattern. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fixup! feat: Change queries to use lab column information

f2a593a

Enforce the lab FK db_constraint on builds and tests, drop the unused lab_id from the build tests response, and fix the tree query lab TODO. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fixup! feat: Change queries to use lab column information

c511998

alanpeixinho requested review from MarceloRobert and mentonin July 7, 2026 21:08



		def assign_lab_ids(builds_buf: list[Builds], tests_buf: list[Tests]) -> None:
		"""Resolve each instance's real lab name to a labs.id and set its lab_id FK.

Uh oh!

Conversation

alanpeixinho commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

legacy database

Migrated database (test in a local database)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alanpeixinho commented Jul 1, 2026 •

edited

Loading