feat: obey robots.txt by default

[pasmud]

Fixes #3208 and #3156: Lightpanda ignored robots.txt unless --obey-robots was passed, letting crawls run against sites that explicitly disallow them (Gentoo, Haskell infra were hit hard).

The robots.txt gate was already fully implemented and correct; the only problem was the default. This change flips the option so robots.txt is respected out of the box in fetch, serve, mcp, and agent modes.

Changes

• src/Config.zig: obey_robots now defaults to true. The flag becomes --no-obey-robots to opt out; --obey-robots is still accepted as an explicit opt-in (backward compatible).
• src/help.zon: updated --no-obey-robots help text and default.
• README.md: examples no longer pass --obey-robots; documented the new default and the opt-out flag.
• src/cdp/domains/lp.zig: comment updated to reflect the new default.

Verification

• zig fmt --check passes on all changed files.
• A full zig build test could not run in this environment: the build fetches V8 and other dependencies from GitHub, which the sandbox network proxy rejects (HTTP fetch fails with HttpConnectionClosing/TemporaryNameServerFailure). The change is confined to the declarative CLI option table plus docs; the CLI framework's bool semantics (presence flips the field opposite its default; variants without a default flip to true) were verified against src/cli.zig.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[pasmud]

I have read the CLA Document and I hereby sign the CLA

[jappeace]

what a good idea!

[t4ccer]

Thanks for the fix!