Skip to content

Compress URL params with DEFLATE and a preset dictionary - #8

Merged
matthewnessworthy merged 1 commit into
mainfrom
compress-urls
Jun 10, 2026
Merged

matthewnessworthy merged 1 commit into
mainfrom
compress-urls

Conversation

@matthewnessworthy

Copy link
Copy Markdown
Member

Summary

json_encode + base64_encode + urlencode pushed template URLs past 1kb, causing SEO issues. URL params and template payloads are now raw-DEFLATE compressed and encoded as unpadded URL-safe base64 (RFC 4648 §5) — every output character is RFC 3986 unreserved, so urlencode() never expands a token.

New UrlCodec with a versioned token format:

prefix encoding
_1 DEFLATE with preset dictionary v1 — what compress() emits
_0 DEFLATE without dictionary — accepted
none legacy standard base64 — accepted, old URLs keep decoding
  • Dictionary v1 (3007 bytes, md5 b6b973310e98eb1286b280d1d65c45ea) was built from 23k real production template payloads: top payload shapes as contiguous skeletons, near-constant values (Now Hiring, de vânzare, SUPRAFAȚĂ, …), real bucket paths (horizon-files-prod/og-template/…, shared-folder-prod/…). It is a frozen byte-for-byte wire contract with the image service — never edit it; new vocabulary goes under prefix _2.
  • JSON is encoded with JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE before compression (kills \/ doubling on S3 paths).
  • ext-zlib / ext-json declared in composer.json.
  • Empty-params URLs (/-/ segment) and their signatures are byte-identical to before.

Results (real production corpus, n=2875)

legacy plain deflate dict v1
mean token 444 318 125
p95 token 562 405 193

The test-fixture template URL drops 822 → 251 chars; a real cde-prop payload tokenizes to 46 chars.

Tests

14 tests passing, including a pinned dictionary md5 (cross-language drift guard), roundtrips with unicode, legacy decode, and invalid-token rejection. Verified end-to-end against the lambda: PHP-signed URLs with _1 tokens pass signature validation and decode on both resize and template routes.

Deploy order

⚠️ RingierIMU/image-service-lambda#515 must deploy first — it decodes all three formats, so it ships safely ahead of this encoder.

🤖 Generated with Claude Code

base64+urlencoded JSON pushed template urls past 1kb. Url params and
template payloads are now raw DEFLATE compressed with a preset
dictionary and encoded as unpadded url-safe base64 (every character
RFC 3986 unreserved, so urlencode never expands the token):

  _1<base64url>  deflate with dictionary v1 (emitted)
  _0<base64url>  deflate without dictionary (accepted)
  <base64>       legacy standard base64 (accepted)

The dictionary was built from 23k production template payloads and is
a frozen byte-for-byte contract with the image service
(image-service-lambda lib/url-codec.js) — md5 pinned in tests on both
sides. Mean template token drops 444 -> 125 chars; the test-fixture
template url drops 822 -> 251 chars.

JSON is encoded with JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE
before compression. Requires ext-zlib (declared in composer.json).

The image service must deploy decode support
(RingierIMU/image-service-lambda#515) before this ships.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@matthewnessworthy
matthewnessworthy enabled auto-merge (squash) June 10, 2026 10:21
@matthewnessworthy
matthewnessworthy merged commit 9f6b832 into main Jun 10, 2026
2 checks passed
@matthewnessworthy
matthewnessworthy deleted the compress-urls branch June 10, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant