ITF-73: delimited text file steps (file.write, file.verify, file.delete)

CLAUDE.md

@@ -98,11 +98,12 @@ idm-test-framework/
 ├── pom.xml                  # Maven parent POM (multi-module)
 ├── idm-test-core/           # TestExecutor (waitUntil, executeWithRetry), OperationResult, ErrorType
 ├── idm-test-utils/          # Test data generators, assertions, EmbeddedLdapServer
-├── idm-test-engine/         # YAML scenario engine: loader, validator, engine, 14 step types (Spring-free)
+├── idm-test-engine/         # YAML scenario engine: loader, validator, engine, 17 step types (Spring-free)
 ├── idm-connector-ldap/      # eDirectory/LDAP (EdirectoryConnector)
 ├── idm-connector-database/  # Multi-DB raw SQL (MultiDatabaseConnector)
 ├── idm-connector-rest/      # IdmRestConnector, RestAuthenticator (basic/OAuth2), JDK HTTP client
 ├── idm-connector-soap/      # Role and Resource service connectors (JAX-WS)
+├── idm-connector-file/      # Delimited text files in named folders (DelimitedFileConnector, Commons CSV)
 ├── idm-connector-idm/       # Placeholder, no sources
 ├── idm-test-wiring/         # Spring wiring shared by runners: StepRegistry, EngineFactory, ScenarioNeeds,
 │                            # RealEnvironmentGuards, scenario sources, RunRecorder
@@ -130,6 +131,7 @@ Each connector does ONE thing:
 - **IdmRestConnector**: Thin wrapper around Spring's RestClient
 - **RestAuthenticator**: Authorization header from a REST target's settings (basic, or OAuth2 client credentials via Spring Security)
 - **RoleServiceConnector** / **ResourceServiceConnector**: SOAP clients for the NetIQ Role and Resource services
+- **DelimitedFileConnector**: delimited text files in named, access-controlled folders (`idm.test.files`)
 
 ### 3. Runner and Workbench (`idm-test-runner`)
 - **RunService** (`run/`): the one run path shared by `idmtest run` and the workbench (settings, secrets, guards, drivers, masking, `reports/<run-id>/`).

README.md

@@ -154,6 +154,9 @@ Actions change state; assertions check it and may retry.
 | `ldap.verify` | assertion | `dn`, and exactly one of `expect` (a value, a list as the exact set in any order, or `{equals, contains, excludes, count, matches, ignoreCase}`) or `exists` | `wait`, `directory` |
 | `db.execute` | action | `database`, `statement` | `params` |
 | `db.verify` | assertion | `database`, `query`, `expect` | `params`, `wait` |
+| `file.write` | action | `folder`, `file`, `rows` | `columns`, `delimiter`, `quote`, `encoding`, `lineEnding`, `header` |
+| `file.verify` | assertion | `folder`, `file` (an exact name or a glob), `expect` (`rowCount` and/or `row`) | `where`, `delimiter`, `quote`, `encoding`, `header`, `wait` |
+| `file.delete` | action | `folder`, `file` | — |
 | `roles.grant` | action | `user`, `roles` | `reason` |
 | `roles.revoke` | action | `assignments` | — |
 | `resources.grant` | action | `user`, `resource` | `reason` |
@@ -228,6 +231,7 @@ idm-test-framework/
 ├── idm-connector-database/  Multi-database JDBC connector (raw SQL)
 ├── idm-connector-rest/      REST connector, JDK HTTP client, target authentication (basic, OAuth2)
 ├── idm-connector-soap/      Role and Resource service connectors (NetIQ SOAP)
+├── idm-connector-file/      Delimited text files in named folders (Delimited Text driver)
 ├── idm-connector-idm/       Placeholder — no sources yet
 ├── idm-test-scenarios/      Java scenario tests, YAML scenarios, the JUnit adapter
 ├── docker-compose.yml       Optional local services; the test suite does not use them
@@ -298,7 +302,8 @@ Verified on 2026-10-20, after ITF-56 (the `idmtest ui` workbench), with `./mvnw
 | **Total** | **1407** |
 
 The scenarios module holds the Java workflow tests (provisioning, role assignment, modification,
-deprovisioning and multi-system flows), a corpus of 21 YAML scenarios converted from them, and the
+deprovisioning and multi-system flows), a corpus of 22 YAML scenarios (21 converted from them, and a
+delimited text file round trip), and the
 tests of the runners and their guards.
 
 **Worth being explicit about:** the default suite *simulates* IdM. The Java and YAML scenarios

docs/architecture/capabilities.md

@@ -14,12 +14,13 @@ Reviewed on **2026-09-25** against Git revision `fb35547c76bfecc0c4b4ca1c7fe71b8
 | YAML assertion `ldap.verify` | Implemented and registered | Assert directory attributes, or existence, with optional polling | [Verify step][verify] | Exactly one of `expect` or `exists`; presence checks do not constitute full nested schema validation. Multi-valued attributes compare as sets in any order, with `contains`/`excludes` and DN-aware values (ITF-38); `ldap.create`/`ldap.update` write lists as several values; count, matches and ignoreCase matchers, and add/remove of single values in ldap.update (ITF-39) |
 | Several LDAP directories | Implemented (ITF-34) | Check provisioning into more than one directory (eDirectory, Active Directory, …) | [LdapDirectories][ldap-directories], [LDAP steps][steps] | `idm.test.directories.<name>` and `directory:` on `ldap.*` steps, with a default directory; the older `idm.test.ldap` block is `default`. `ldap.create` takes `objectClass`. AD passwords, account state, binary attributes and paging are ITF-33 |
 | YAML LDAP, database and SOAP steps | Implemented and registered (ITF-8, ITF-9) | `ldap.update`, `ldap.delete`, `db.execute`, `db.verify`, `roles.grant`, `roles.revoke`, `resources.grant`, `resources.revoke` | [Step sources][steps], [adapter registry][adapter] | `db.verify` is retryable; the others are actions. Database steps run raw SQL against named databases. Set checks (has these, lacks those, exactly these) are written in the query; the authoring guide has recipes (ITF-41) |
+| YAML file steps | Implemented and registered (ITF-73) | Drop a CSV for the NetIQ Delimited Text driver and check the files it writes | [File steps][file-steps], [ITF-73 design][files-design] | `file.write`, `file.verify` (retryable), `file.delete` over named folders (`idm.test.files.<name>`) with declared access; never a path in a scenario. `file.verify` reads the newest matching file modified since the scenario started, selects rows with `where:` and asserts `rowCount`/`row` as `db.verify` does. Writes are temp-then-atomic-rename and never overwrite. Local folders and mounted shares only (no SFTP/SMB); one CSV section per file, no fixed-width files. The default suite proves the steps against a scratch folder, not against a real driver |
 | YAML REST steps | Implemented and registered (ITF-9) | `rest.request`, `rest.verify` against named targets | [Step sources][steps] | JSON bodies only; every HTTP status is data, and `rest.verify` asserts it. Redirects are not followed |
 | YAML `http.request` | Implemented and registered (ITF-22) | Raw string request and response, so REST and SOAP use one step | [HTTP step][http-step], [ITF-22 design][http-design] | Named target plus path only; never fails on HTTP status. `http.verify` checks a response with JavaScript conditions, retryable with `wait:` (ITF-24); checks read the parsed JSON (`response.json`) and XML or SOAP body (`response.xml`, ITF-25); checks can use containsAll, excludesAll and sameMembers, which say what was missing when false (ITF-41) |
 | YAML `set` | Implemented engine keyword | Bind derived variables | [Engine][engine], [validator][validator] | Bindings are processed in declaration order; duplicate/shadowing names are rejected by validation |
 | YAML `as` | Implemented engine parameter | Capture successful step output | [Engine][engine], [steps][create] | Create captures a DN string; `ldap.verify` captures every attribute except `userPassword` from the same read the check used, with a list for a multi-valued attribute. Requested null captures fail |
 | YAML step descriptions | Implemented (ITF-35) | Explain each step in results and failure messages | [Engine][engine] | Optional `description:` on every step and cleanup entry; text with `${…}` resolved, no `js:{}`; recorded per scenario, step and cleanup entry in the JSON results and quoted in failure summaries |
-| HTML report | Implemented (ITF-36) | Read and share a run's results | [HtmlReportWriter][report], [ScenarioSupport][support] | One self-contained file per run beside the JSON results, rewritten after each scenario; header with runner, profiles, filter and sanitised endpoints; failed scenarios open; secret-looking parameters masked. Per-step evidence for HTTP/REST, SQL and LDAP steps, masked and size-limited, in the report and the JSON (ITF-37), and for the SOAP role and resource steps (ITF-40) |
+| HTML report | Implemented (ITF-36) | Read and share a run's results | [HtmlReportWriter][report], [ScenarioSupport][support] | One self-contained file per run beside the JSON results, rewritten after each scenario; header with runner, profiles, filter and sanitised endpoints; failed scenarios open; secret-looking parameters masked. Per-step evidence for HTTP/REST, SQL and LDAP steps, masked and size-limited, in the report and the JSON (ITF-37), for the SOAP role and resource steps (ITF-40), and for the file steps, by folder name (ITF-73) |
 | Report language | Implemented (ITF-46) | Read the report and messages in English or Spanish | [Messages][messages], [ReportLocale][report-locale], [catalogs][catalogs] | `en` (default) or `es`, from `idm.test.report.locale`, then `IDM_TEST_LOCALE`, then `-Didm.test.locale`; any other value stops the run. Translates report labels, evidence, step, engine, validation and runner messages, and JUnit failure text; the JSON keeps the English `message` and adds `messageKey`/`messageArgs`. Author-written text, external-system errors, logs, step types and YAML keys stay as they are; a few framework texts are still English (see configuration.md) |
 | Fixtures | Implemented | Generate or supply scenario data | [Fixture factory][fixtures] | Only `random-user` is registered; `values` are literal overrides, not evaluated expressions |
 | `${path}` interpolation | Implemented | Resolve fixture/variable properties | [Interpolator][interpolator] | Traverses maps by dot-separated keys and substitutes strings; it is not JavaScript or typed object capture |
@@ -36,10 +37,11 @@ Reviewed on **2026-09-25** against Git revision `fb35547c76bfecc0c4b4ca1c7fe71b8
 | JDBC Java connector | Implemented | Named databases, raw SQL, verification | [MultiDatabaseConnector][db], [DB configuration][db-config] | Embedded H2 tests do not establish Oracle/PostgreSQL production compatibility |
 | REST Java connector | Implemented | Spring client with thin operations and IdM helpers | [IdmRestConnector][rest], [REST client configuration][rest-config] | JDK `HttpClient` pinned to HTTP/1.1 (ITF-22), shared by production and tests; default 10 s connect and 30 s overall read deadline; each target's `connect-timeout`/`read-timeout` apply (one client per distinct pair, ITF-43) |
 | REST authentication | Implemented (ITF-23) | Authenticate from target settings, never from scenario YAML | [RestAuthenticator][authenticator], [ITF-23 design][auth-design] | Basic, or OAuth2 client credentials through Spring Security's authorized-client manager (cached tokens), or none. A step's own `Authorization` header wins. A token server's 401 is reported as `401 Unauthorized` without its OAuth error code (Spring's default handling). Token-failure messages quote Spring's cause with the client secret and URL credentials masked and the length capped at 300 characters; a malformed token-uri is a configuration error (ITF-27) |
+| Delimited file connector | Implemented (ITF-73) | Named, access-controlled folders and delimited text | [DelimitedFileConnector][file-connector] | Apache Commons CSV; names stay inside their folder (separators, `..`, roots and symbolic links leading out are refused; reading never follows links); messages and evidence name the folder, never its path. The real-environment guards check folders without creating anything; the workbench's connection test writes and deletes a probe file in a write folder |
 | SOAP roles | Implemented | WSDL-based role assignment/removal | [Role connector][roles] | Uses generated JAX-WS stubs, not WebServiceTemplate. Sends HTTP basic authentication when `idm.test.soap.username`/`password` are set (ITF-23); `connection-timeout`/`read-timeout` applied (ITF-43) |
 | SOAP resources | Implemented Java connector | Resource grants/revokes | [Resource connector][resources], [resource tests][resource-tests] | Conditional on `idm.test.soap.resource-url`; used by `resources.grant`/`resources.revoke`; no default resource endpoint; basic authentication as for roles |
 | Dedicated `idm-connector-idm` module | Placeholder | Potential IdM-specific composition | [Module POM][idm-pom] | No main Java sources; existing IdmRestConnector belongs to the REST module |
-| Convert existing Java scenarios to YAML | Implemented (ITF-5) | Migrate suitable workflows while retaining Java tier | [Scenario corpus][corpus], [conversion findings][findings] | 21 YAML scenarios; the Java workflows remain alongside them |
+| Convert existing Java scenarios to YAML | Implemented (ITF-5) | Migrate suitable workflows while retaining Java tier | [Scenario corpus][corpus], [conversion findings][findings] | 21 YAML scenarios (22 with ITF-73's file round trip); the Java workflows remain alongside them |
 | Configuration profiles | Implemented templates | Environment-specific settings | [application.yml][config] | `dev`, `test`, and `prod` documents exist. They contain example endpoints, not verified deployed environments |
 | Embedded scenario environment | Implemented | Run without Docker or vendor systems | [Test base][base], [schema initializer][schema] | Test base forces `test` profile and dynamically overrides LDAP/role SOAP endpoints; schema initializer targets `test-db`. Spring Boot's own DataSource is excluded (ITF-32); TestSchemaInitializer alone creates the test-db schema |
 | Container execution | Implemented, optional (ITF-14) | Run the corpus against real protocol servers | [ContainerScenarios][containers] | OpenLDAP, PostgreSQL and WireMock via Testcontainers; needs Docker and is not part of the default suite. Starts no embedded LDAP or WireMock server (ITF-17) |
@@ -112,6 +114,9 @@ All five diagrams rendered from their Mermaid sources. The eight-slide PowerPoin
 [messages]: ../../idm-test-engine/src/main/java/com/idm/test/engine/i18n/Messages.java
 [report-locale]: ../../idm-test-engine/src/main/java/com/idm/test/engine/i18n/ReportLocale.java
 [catalogs]: ../../idm-test-engine/src/main/resources/com/idm/test/engine/i18n/
+[file-steps]: ../../idm-test-engine/src/main/java/com/idm/test/engine/steps/FileVerifyStep.java
+[files-design]: ../specs/2026-11-03-delimited-text-file-steps-design.md
+[file-connector]: ../../idm-connector-file/src/main/java/com/idm/test/connector/file/client/DelimitedFileConnector.java
 [ldap]: ../../idm-connector-ldap/src/main/java/com/idm/test/connector/ldap/client/EdirectoryConnector.java
 [ldap-directories]: ../../idm-connector-ldap/src/main/java/com/idm/test/connector/ldap/client/LdapDirectories.java
 [db]: ../../idm-connector-database/src/main/java/com/idm/test/connector/database/client/MultiDatabaseConnector.java

docs/guides/scenario-authoring.md

@@ -88,11 +88,17 @@ A guarded read such as `typeof created === 'undefined' ? null : created.dn` stil
 
 ## 4. Steps
 
-The framework has 14 step types across LDAP, databases, the SOAP role and resource services,
-REST and raw HTTP. Every step's parameters, what `as:` captures, how it fails and an example
+The framework has 17 step types across LDAP, databases, delimited text files, the SOAP role and
+resource services, REST and raw HTTP. Every step's parameters, what `as:` captures, how it fails and an example
 are in the [authoring reference](../reference/writing-scenarios.md#4-step-reference), which a
 test keeps in step with the code.
 
+For the NetIQ Identity Manager Driver for Delimited Text, the
+[file steps](../reference/writing-scenarios.md#files-delimited-text) write a CSV into a named folder
+(`file.write`), check the file the driver writes (`file.verify`, which waits for it) and remove
+written files in `cleanup:` (`file.delete`). Folders are declared in the settings, never as paths
+in the scenario; see [folders](../reference/configuration.md#delimited-text-folders).
+
 For calling a SOAP service or any API with a raw body, see
 [`http.request`](../reference/writing-scenarios.md#httprequest); for how REST targets
 authenticate, see [targets and configuration](../reference/writing-scenarios.md#7-targets-and-configuration).