[mathieuancelin]

Summary

The openapi document of the admin api (/apis/openapi.json) described the Scala classes of the entities, by reflection, instead of the json the api reads and writes: wrong field names (location instead of _loc, camelCase instead of snake_case), fields that do not exist, missing fields, wrong content types, undocumented operations and routes, few descriptions and no examples. This PR makes the document the reference of the admin api, and adds a suite that checks it against a running otoroshi.

The document

• The schemas of the core entities (and of the admin extensions shipped with otoroshi) are written by hand in otoroshi/conf/schemas/openapi/<group>.json, with a description on every schema and property and examples. They are read once by OpenApiSchemas, and each Resource of OtoroshiResources carries its own through ResourceVersion.schema (what GET .../_schema answers). The entities of third party extensions keep a schema reflected from their class.
• The operations of the generic api are built by OpenApiResourcePaths from what GenericApiController really does: json and yaml bodies, json patch / fields patch / form bodies for PATCH, ndjson for the bulk operations, the kind added to every answer, the list parameters (pagination, sorting, filters, projection), and the operations that were missing (POST {id}, DELETE of a collection, _schema).
• The other operations of the admin api (/api/..., and the /apis/... routes of specific controllers) moved from additionalPaths.json / additionalComponents.json to conf/schemas/openapi/paths/<family>.json and api-<family>.json, rewritten against the controllers, and the routes that were not documented are.
• Seven legacy /api routes that the generic api has no equivalent of are documented too, and the suite requires them: GET/DELETE /api/apikeys/{id}/quotas (read and reset the quotas of an api key), POST /api/certificates/{certId}/_renew, POST /api/certificates/_bundle (import a PEM bundle), POST /api/certificates/_dedupLetsEncrypt, DELETE /api/apps-sessions-module/{id} (log out every user of an auth module) and POST /api/new/resources (build complete entities from partial json or yaml definitions, nothing is stored).
• index.json lists every file, and the tags of the hand written operations with their description.
• tools.OpenApiSchemaDraftsSpec drafts the schema of a new entity from the json it writes, to complete by hand.

The suite

sbt "testOnly OpenApiConformanceTests" (otoroshi/test/openapi/, not wired in CI) starts an otoroshi (cluster leader, every shipped extension enabled), reads the document it serves, and checks:

• the document: a valid openapi 3.0.3 document, every $ref resolved, no unused schema, unique operation ids, declared tags, valid keywords, types, formats, enums, defaults and examples;
• every schema: a description on the schema, on every property, and on every value that can hold anything;
• every operation: routed by otoroshi (to the right controller), its parameters, body and responses described;
• every route of the admin api: documented;
• every entity, field by field: in process through its play json format (every documented field read and written back unchanged, every written field documented with its type, once per type of a polymorphic entity), then through the api, each documented operation being called the way the document describes it, in each documented content type;
• every other operation, called with values built from the document (the import gets the export of the instance).

It runs without sbt too, filtered by test name: org.scalatest.tools.Runner -s openapi.OpenApiConformanceSpec -z "<substring>", with otoroshi/conf first on the classpath.

Api behaviours it found

The document describes what the api should answer, it does not hide these: they are the 116 failures left of the suite (8377 tests, 8208 succeeded, 116 failed, 53 canceled). None of them is fixed here.

• The gateway answers a call without credentials (400) or with wrong credentials (401) without a Content-Type.
• A body that is not a json object ([]) is answered 500 on every entity.
• Every documentation endpoint (/apis/openapi.*, /apis/plugins.*, /apis/workflows/doc.*) answers 502 on the admin api host, the server the document declares (DocAction does not send back Otoroshi-State-Resp).
• GET /apis/proxy.otoroshi.io/v1/drafts/:kind (conf/routes) shadows the generic _count, _schema, _template and {id} of the drafts.
• 9 data exporter types have no template (console, s3, goreplayfile, goreplays3, tcp, udp, syslog, jms, user-analytics), nor have admins, admin sessions and auth module users.
• Any id reads the global configuration, and GET answers it without its id.
• PUT /api/cluster/quotas and POST /api/cluster/relay answer 500 on bodies or headers they do not expect.

Writing the schemas against the code found many more (500 on malformed inputs, settings lost on read, read defaults that differ from the defaults of a new instance, an ndjson export that cannot be imported back, docs that disagree with the formats...): they are listed for a review one by one, outside of this PR.

Choices to review

• The legacy entity crud of /api (apikeys, routes, groups...) is not required to be documented, /apis replaces it. Only the seven routes above, with no /apis equivalent, are.
• The deprecated entities (scripts, wasm plugins) are still served but not documented.
• The envelope query parameter of the generic api is not documented (it changes the shape of the answers).
• The operation id pathById is renamed patchById (typo), which changes the names generated by sdk generators.
• _schema answers the hand written schema of the entity, with the schemas it references under components.schemas.
• The example of the certificate holds a throwaway private key.
• The published openapi.json files are not regenerated here, the release does it.

Test plan

• sbt "testOnly OpenApiConformanceTests": 8208 succeeded, the 116 failures are the api behaviours above
• sbt "testOnly UnitTests": 7665 succeeded
• the seven legacy routes: 121 tests of the suite (coverage, schemas, live calls) succeeded
• functional.OpenApiSchemasSpec: 10 succeeded
• functional.ResponseCompressorSpec, functional.BrotliResponseCompressorSpec, functional.PluginsTestSpec (-l Browser): succeeded, but for the Swagger UI plugin test locally, that read a stale copy of the swagger ui bundle in target/web/public/test
• full CI