Refactoring

Author: ilyario

README.md

@@ -1,43 +1,79 @@
 # DataFlow Operator
 
-Kubernetes operator for streaming data between different sources (Kafka, PostgreSQL, Trino) with support for message transformations.
+Kubernetes operator for streaming data between different sources and sinks with support for message transformations.
 
 **[Online documentation](https://dataflow-operator.github.io/docs/)**
 
-## Quick Start
+## Architecture
+
+The operator watches `DataFlow` custom resources and creates processor pods. Each processor reads from a source, applies optional transformations, and writes to a sink. Configuration uses the `type` + `config` format:
+
+```yaml
+apiVersion: dataflow.dataflow.io/v1
+kind: DataFlow
+metadata:
+  name: kafka-to-postgres
+spec:
+  source:
+    type: kafka
+    config:
+      brokers:
+        - localhost:9092
+      topic: input-topic
+      consumerGroup: dataflow-group
+  sink:
+    type: postgresql
+    config:
+      connectionString: "postgres://user:pass@localhost:5432/db?sslmode=disable"
+      table: output_table
+```
+
+### Supported connectors
+
+| Source | Sink |
+|--------|------|
+| Kafka | Kafka |
+| PostgreSQL | PostgreSQL |
+| — | ClickHouse |
+| — | Trino |
+| — | Nessie (Apache Iceberg) |
 
-### Prerequisites
+### Supported transformations
+
+Filter, Select, Remove, Mask, Flatten, Timestamp, SnakeCase, Router, Chain.
+
+## Prerequisites
 
 - Kubernetes 1.24+
 - Helm 3.0+
 - kubectl
-- Go 1.21+ (for local development)
+- Go 1.25+ (for local development)
 - Docker and docker-compose (for local development)
 
-#### Local Development
+## Quick Start
 
-For local development, you can run the operator locally:
-```bash
-task run
-```
+### Install via Helm
 
-Or use the script:
 ```bash
-./scripts/run-local.sh
+helm repo add dataflow-operator https://dataflow-operator.github.io/helm-charts
+helm repo update
+helm install dataflow-operator dataflow-operator/dataflow-operator
 ```
 
 ### Local Development Setup
 
-1. Start dependencies with UI interfaces:
+1. Start dependencies:
+
 ```bash
 docker-compose up -d
 ```
 
 Available UIs:
 - **Kafka UI**: http://localhost:8080
-- **pgAdmin**: http://localhost:5050 (admin@admin.com / admin)
+- **ClickHouse**: http://localhost:8123
 
 2. Run the operator:
+
 ```bash
 task run
 ```
@@ -46,27 +82,65 @@ task run
 
 ### Code Generation
 
-If you encounter issues with `task generate`, try:
+```bash
+task generate   # DeepCopy, CRD, RBAC manifests
+task manifests  # CRD and RBAC only
+```
+
+If you encounter issues with `task generate`:
 
 ```bash
-# Update controller-gen
 go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest
-
-# Then
 task generate
 ```
 
+### Building
+
+```bash
+task build  # builds bin/manager
+```
+
+Docker image (builds both operator and processor binaries):
+
+```bash
+docker build -t dataflow:local .
+```
+
 ### Testing
 
 ```bash
 # Unit tests
 task test
 
-# Integration tests (requires kind)
-./scripts/setup-kind.sh
+# Integration tests (requires Docker — uses testcontainers)
 task test-integration
 ```
 
+### Taskfile commands
+
+| Command | Description |
+|---------|-------------|
+| `task build` | Build `bin/manager` |
+| `task run` | Run operator locally |
+| `task generate` | Generate DeepCopy and manifests |
+| `task manifests` | Generate CRD and RBAC |
+| `task test` | Unit tests |
+| `task test-integration` | Integration tests (Docker required) |
+| `task install` | Install CRDs into cluster |
+| `task uninstall` | Remove CRDs from cluster |
+
+## Configuration samples
+
+Example manifests are in `config/samples/`:
+
+- `kafka-to-postgres.yaml` — Kafka to PostgreSQL
+- `kafka-to-clickhouse.yaml` — Kafka to ClickHouse
+- `kafka-to-trino.yaml` — Kafka to Trino
+- `postgres-to-kafka-router.yaml` — PostgreSQL to Kafka with Router
+- `clickhouse-to-clickhouse.yaml` — ClickHouse to ClickHouse
+
+See all samples: `ls config/samples/`
+
 ## License
 
 Apache License 2.0

Taskfile.yml

@@ -96,6 +96,50 @@ tasks:
     deps: [manifests, generate, fmt, vet]
     cmds:
       - go run ./main.go
+  sync-crd-to-helm:
+    desc: Sync generated CRD into Helm chart template (wraps CRD with Helm conditionals)
+    deps: [manifests]
+    vars:
+      CRD_SOURCE: config/crd/bases/dataflow.dataflow.io_dataflows.yaml
+      HELM_TARGET: ../helm-charts/charts/dataflow-operator/templates/crd.yaml
+    cmds:
+      - |
+        HEADER='{{- if .Values.crds.install }}'
+        FOOTER='{{- end }}'
+        LABELS_BLOCK='  labels:
+            {{- include "dataflow-operator.labels" . | nindent 4 }}'
+        KEEP_BLOCK='    {{- if .Values.crds.keep }}
+            "helm.sh/resource-policy": keep
+            {{- end }}'
+
+        {
+          echo "$HEADER"
+          awk '
+            /^---$/ { next }
+            /^metadata:$/ {
+              print
+              getline
+              # annotations block
+              if ($0 ~ /^  annotations:/) {
+                print
+                # print existing annotations
+                while (getline > 0 && $0 ~ /^    [^ ]/) {
+                  print
+                }
+                # insert keep annotation block
+                printf "%s\n", "'"$KEEP_BLOCK"'"
+                # insert labels block
+                printf "%s\n", "'"$LABELS_BLOCK"'"
+                print
+              }
+              next
+            }
+            { print }
+          ' "{{.CRD_SOURCE}}"
+          echo "$FOOTER"
+        } > "{{.HELM_TARGET}}"
+        echo "Synced CRD to {{.HELM_TARGET}}"
+
   # Deployment tasks
   install:
     desc: Install CRDs into the K8s cluster specified in ~/.kube/config
@@ -147,6 +191,11 @@ tasks:
     preconditions:
       - test -d {{.LOCALBIN}} || mkdir -p {{.LOCALBIN}}
 
+  test-helm-crd:
+    desc: Run Helm CRD template tests (requires helm CLI)
+    cmds:
+      - bash ../helm-charts/charts/dataflow-operator/tests/crd_test.sh
+
   tidy:
     desc: Run go mod tidy
     cmds: