Address issues related to the Web Connectivity 0.5 rollout

[hellais]

Web Connectivity 0.5 was rolled out on August 18th 2026 to all probes.

We started looking at the measurement data comparing the agreements of the fastpath analysis (i.e. the probe analysis) with that of the new pipeline (analysis_web_meausurement table), to identify differences.

To do this we divided measurements into 4 stratums:

• S00 -> both analysis agree with it not being blocked
• S01 -> fastpath says unblocked, analysis says blocked
• S11 -> both say blocked
• S10 -> fastpath says blocked, analysis says unblocked

As a baseline for comparison we drew a couple of windows of measurements from July, before web_connectivity was deployed.

{"start_time": "2026-07-03", "end_time": "2026-07-08"}

cnt	strata	perc
175553	S01	2.20559
451350	S11	5.67062
192595	S10	2.4197
7139948	S00	89.7041

{"start_time": "2026-06-03", "end_time": "2026-06-08"}

cnt	strata	perc
154211	S01	1.81242
470505	S11	5.52978
253414	S10	2.97834
7630436	S00	89.6795

As we can see the distributions are quite similar across the stratums.

We then examined for when web_connectivity was first deployed:
{"start_time": "2026-08-19", "end_time": "2026-08-20"}

cnt	strata	perc
476190	S01	29.4918
118238	S11	7.32282
28106	S10	1.74068
992118	S00	61.4447

As we can see the distribution is very skewed towards S01, which suggests the analysis_web_measurement believes measurements to be much more blocked than the probe.

[hellais]

We identified the source of the DNS related bugs to be that web_connectivity 0.5 includes in the DNS resolutions also the CNAME.
Since the data pipeline tries to determine consistency by either trying to find the address in control (which doesn't lookup CNAMES) or looking at TLS consistency results (which are keyed by IP), we get a mismatch.

This was fixed here: 716cb99

We then performed a backfill and recomputed the stratum, which now look like this:

cnt	strata	perc
301452	S01	18.6698
116582	S11	7.22025
29762	S10	1.84324
1166857	S00	72.2667

S01 has gone down, but not so much yet.

Therefore we breakdown the S01 stratum into the kind of blocking that it detected, finding the following:

|    cnt | strata   |   dns_state |   tcp_state |   tls_state | top_dns_rule_id              |         perc |
|-------:|:---------|------------:|------------:|------------:|:-----------------------------|-------------:|
| 260767 | S01      |           1 |           0 |           0 | failure_ctrl_ok              | 16.15        |
|  25172 | S01      |           1 |           0 |           0 | answer_unmatched             |  1.55897     |
|   5982 | S01      |           0 |           0 |           1 | tls_consistent_answer        |  0.370482    |
|   2991 | S01      |           1 |           0 |           0 | tls_inconsistent_not_in_ctrl |  0.185241    |
|   2296 | S01      |           1 |           0 |           1 | failure_ctrl_ok              |  0.142198    |
|   1644 | S01      |           1 |           0 |           0 | bogon_not_in_ctrl            |  0.101818    |
|    924 | S01      |           1 |           1 |           0 | failure_ctrl_ok              |  0.0572259   |
|    535 | S01      |           0 |           1 |           1 | tls_consistent_answer        |  0.0331341   |
|    418 | S01      |           1 |           0 |           1 | answer_unmatched             |  0.0258879   |
|    263 | S01      |           1 |           1 |           0 | answer_unmatched             |  0.0162883   |
|    188 | S01      |           1 |           1 |           1 | failure_ctrl_ok              |  0.0116434   |
|     81 | S01      |           1 |           1 |           0 | bogon_not_in_ctrl            |  0.00501656  |
|     61 | S01      |           1 |           1 |           1 | answer_unmatched             |  0.0037779   |
|     58 | S01      |           1 |           0 |           1 | bogon_not_in_ctrl            |  0.0035921   |
|     28 | S01      |           1 |           1 |           0 | tls_inconsistent_not_in_ctrl |  0.00173412  |
|     17 | S01      |           0 |           0 |           1 | answer_asn_matches_ctrl      |  0.00105286  |
|     11 | S01      |           0 |           0 |           1 | failure_ctrl_also_failing    |  0.000681261 |
|      6 | S01      |           0 |           0 |           1 | bogon_in_ctrl                |  0.000371597 |
|      3 | S01      |           0 |           1 |           1 | answer_asn_matches_ctrl      |  0.000185798 |
|      3 | S01      |           1 |           0 |           1 | tls_inconsistent_not_in_ctrl |  0.000185798 |
|      2 | S01      |           0 |           1 |           1 | failure_ctrl_also_failing    |  0.000123866 |
|      1 | S01      |           1 |           1 |           1 | bogon_not_in_ctrl            |  6.19328e-05 |
|      1 | S01      |           1 |           1 |           1 | tls_inconsistent_not_in_ctrl |  6.19328e-05 |

As we can see the most common source is that the DNS query fails on the probe, yet the control says it's ok. Sampling some measurement we can see what this behaviour looks like. For example in the following measurement: https://explorer.ooni.org/m/20260819122252.565771_RO_webconnectivity_011c98d67eae59a9

we can see how doh failed and the AAAA lookup produced a dns_no_answer error. These are both problematic from the perspective of analysis, as:

1. The doh failure indicates that DNS over HTTPS itself might be blocked, not that the target domain is filtered
2. The quad-A lookup produces no answers, likely because there is no quad-A record configured for this domain. This would not normally be considered a failure in the case of getaddrinfo since it combines both quad-A and A lookups in the same answer.

Because of these issues we are going for the time being to ignore every DNS answer that does not have dns_engine=getaddrinfo|system, see: #179

In the future we might change this if we come up with a better way of capturing these analysis results in here.

[hellais]

After having merged #179 the breakdowns look like the following:

cnt	strata	perc
74683	S01	4.625
108297	S11	6.70666
38068	S10	2.35749
1393720	S00	86.3109

It's still a bit far from the original breakdowns, but it's a significant improvement. There are probably more bugs lurking in the DNS sub-strata as seen here:

cnt	strata	dns_state	tcp_state	tls_state	top_dns_rule_id	perc
55128	S01	1	0	0	dns_system_answer_unmatched	3.41399
8012	S01	0	0	1	tls_consistent_answer	0.49617
4900	S01	1	0	0	dns_system_failure_ctrl_ok	0.303449
2991	S01	1	0	0	tls_inconsistent_not_in_ctrl	0.185228
1644	S01	1	0	0	bogon_not_in_ctrl	0.10181
700	S01	0	1	1	tls_consistent_answer	0.0433499
662	S01	1	0	1	dns_system_answer_unmatched	0.0409966
326	S01	1	1	0	dns_system_answer_unmatched	0.0201887
86	S01	1	1	1	dns_system_answer_unmatched	0.00532584
81	S01	1	1	0	bogon_not_in_ctrl	0.0050162
58	S01	1	0	1	bogon_not_in_ctrl	0.00359185
35	S01	0	0	1	answer_asn_matches_ctrl	0.00216749
28	S01	1	1	0	tls_inconsistent_not_in_ctrl	0.001734
11	S01	1	0	1	dns_system_failure_ctrl_ok	0.000681212
6	S01	0	0	1	bogon_in_ctrl	0.00037157
4	S01	0	0	1	dns_system_failure_ctrl_also_failing	0.000247714
3	S01	0	1	1	answer_asn_matches_ctrl	0.000185785
3	S01	1	0	1	tls_inconsistent_not_in_ctrl	0.000185785
3	S01	1	1	0	dns_system_failure_ctrl_ok	0.000185785
1	S01	1	1	1	bogon_not_in_ctrl	6.19284e-05
1	S01	1	1	1	tls_inconsistent_not_in_ctrl	6.19284e-05