Update readme and add crd ci

Author: ilyario

.github/workflows/crd.yml

@@ -0,0 +1,74 @@
+name: CRD
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'api/**'
+      - 'hack/**'
+  pull_request:
+    paths:
+      - 'api/**'
+      - 'hack/**'
+
+jobs:
+  generate-crd:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+
+      - name: Install controller-gen
+        run: go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0
+
+      - name: Generate CRD
+        run: |
+          controller-gen crd paths=./api/... output:crd:artifacts:config=config/crd/bases
+
+      - name: Verify CRD is up to date (PR)
+        if: github.event_name == 'pull_request'
+        run: |
+          git diff --exit-code config/crd || \
+            (echo "CRD is out of date. Run: task manifests" && exit 1)
+
+      - name: Commit and push CRD (push to main)
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        run: |
+          git config user.name "github-actions"
+          git config user.email "github-actions@github.com"
+          git add config/crd
+          git diff --staged --quiet || git commit -m "chore: regenerate CRD" && git push
+
+      - name: Create MR in helm-charts
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        env:
+          # Use HELM_CHARTS_TOKEN for cross-repo (PAT with repo scope). If unset, uses GITHUB_TOKEN (works when helm-charts is in same repo).
+          GH_TOKEN: ${{ secrets.HELM_CHARTS_TOKEN || secrets.GITHUB_TOKEN }}
+        run: |
+          set -e
+          HELM_REPO="dataflow-operator/helm-charts"
+          BRANCH="crd-sync-$(date +%Y%m%d-%H%M%S)"
+          mkdir -p helm-charts
+          git clone "https://x-access-token:${GH_TOKEN}@github.com/${HELM_REPO}.git" helm-charts
+          cd helm-charts
+          git config user.name "github-actions"
+          git config user.email "github-actions@github.com"
+          mkdir -p charts/dataflow-operator/crds
+          cp "$GITHUB_WORKSPACE/dataflow/config/crd/bases/dataflow.dataflow.io_dataflows.yaml" charts/dataflow-operator/crds/
+          git checkout -b "$BRANCH"
+          git add charts/dataflow-operator/crds
+          if git diff --staged --quiet; then
+            echo "CRD unchanged in helm-charts, skipping MR"
+            exit 0
+          fi
+          git commit -m "chore: sync CRD from dataflow-operator"
+          git push -u origin "$BRANCH"
+          gh pr create --repo "$HELM_REPO" \
+            --title "chore: sync CRD from dataflow-operator" \
+            --body "CRD regenerated from dataflow-operator after api/hack changes."

README.md

@@ -2,23 +2,7 @@
 
 Kubernetes operator for streaming data between different sources (Kafka, PostgreSQL, Trino) with support for message transformations.
 
-## Features
-
-- **Multiple Data Sources**: Kafka, PostgreSQL, Trino
-- **Message Transformations**:
-  - Timestamp - add timestamp to messages
-  - Flatten - expand arrays into separate messages
-  - Filter - filter by conditions
-  - Mask - mask sensitive data
-  - Router - route to different sinks
-  - Select - select specific fields
-  - Remove - remove fields
-  - SnakeCase - convert field names to snake_case
-  - CamelCase - convert field names to CamelCase
-- **Kubernetes Secrets Support**: Configure connectors using `SecretRef` for secure credential management
-- **Per-Resource Pod Deployment**: Each DataFlow resource creates a separate pod (Deployment) for processing
-- **Resource Management**: Configure CPU and memory resources for processor pods
-- **Pod Placement Control**: Configure nodeSelector, affinity, and tolerations for fine-grained pod placement
+**[Online documentation](https://dataflow-operator.github.io/docs/)**
 
 ## Quick Start
 
@@ -30,71 +14,11 @@ Kubernetes operator for streaming data between different sources (Kafka, Postgre
 - Go 1.21+ (for local development)
 - Docker and docker-compose (for local development)
 
-### Installation
-
-#### Installing CRD
-
-Before installing the operator, you need to install the Custom Resource Definition (CRD):
-
-```bash
-kubectl apply -f https://raw.githubusercontent.com/dataflow-operator/dataflow/refs/heads/main/config/crd/bases/dataflow.dataflow.io_dataflows.yaml
-```
-
-Or use a local file:
-
-```bash
-kubectl apply -f config/crd/bases/dataflow.dataflow.io_dataflows.yaml
-```
-
-#### Installation via Helm (Recommended)
-
-1. Install the operator from OCI registry:
-```bash
-helm install dataflow-operator oci://ghcr.io/dataflow-operator/helm-charts/dataflow-operator
-```
-
-2. For installation with custom settings:
-```bash
-helm install dataflow-operator oci://ghcr.io/dataflow-operator/helm-charts/dataflow-operator \
-  --set image.repository=your-registry/controller \
-  --set image.tag=v1.0.0 \
-  --set replicaCount=2
-```
-
-3. For installation in a specific namespace:
-```bash
-helm install dataflow-operator oci://ghcr.io/dataflow-operator/helm-charts/dataflow-operator \
-  --namespace dataflow-system \
-  --create-namespace
-```
-
-4. Check installation status:
-```bash
-kubectl get pods -l app.kubernetes.io/name=dataflow-operator
-```
-
-**Note**: For local development, you can also use the local chart:
-```bash
-helm install dataflow-operator ./helm-charts/dataflow-operator
-```
-
-#### Updating
-
-```bash
-helm upgrade dataflow-operator oci://ghcr.io/dataflow-operator/helm-charts/dataflow-operator
-```
-
-#### Uninstallation
-
-```bash
-helm uninstall dataflow-operator
-```
-
 #### Local Development
 
 For local development, you can run the operator locally:
 ```bash
-make run
+task run
 ```
 
 Or use the script:
@@ -115,93 +39,34 @@ Available UIs:
 
 2. Run the operator:
 ```bash
-make run
-```
-
-### Usage Examples
-
-See `config/samples/` for CRD manifest examples.
-
-#### Example with Kubernetes Secrets
-
-For secure credential storage, use Kubernetes Secrets:
-
-```bash
-kubectl apply -f config/samples/kafka-to-postgres-secrets.yaml
-```
-
-This example demonstrates using `SecretRef` for connector configuration. All connectors support configuration from Kubernetes Secrets.
-
-## Project Structure
-
-```
-dataflow/
-├── api/v1/              # CRD definitions
-├── internal/
-│   ├── connectors/      # Connectors for sources/sinks
-│   ├── transformers/    # Message transformations
-│   ├── processor/       # Message processor
-│   └── controller/      # Kubernetes controller
-├── config/samples/      # CRD examples
-├── docs/                # MkDocs documentation
-├── test/                # Tests and utilities
-└── scripts/             # Helper scripts
-```
-
-## Documentation
-
-Full documentation is available in `docs/`. To view:
-
-```bash
-mkdocs serve
+task run
 ```
 
-Or from the project root:
-
-```bash
-cd docs && mkdocs serve
-```
-
-Documentation is available in two languages:
-- **English**: Default language
-- **Russian**: Available in the navigation menu
-
 ## Development
 
 ### Code Generation
 
-If you encounter issues with `make generate`, try:
+If you encounter issues with `task generate`, try:
 
 ```bash
 # Update controller-gen
 go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest
 
 # Then
-make generate
+task generate
 ```
 
 ### Testing
 
 ```bash
 # Unit tests
-make test
+task test
 
 # Integration tests (requires kind)
 ./scripts/setup-kind.sh
-make test-integration
+task test-integration
 ```
 
-## Security
-
-DataFlow Operator supports configuring connectors from Kubernetes Secrets through `SecretRef` fields. This allows:
-
-- Secure storage of sensitive data (passwords, tokens, connection strings)
-- Centralized credential management
-- Secret rotation without changing DataFlow resources
-- Access control through Kubernetes RBAC
-
-See the [Connectors documentation](docs/en/connectors.md#using-kubernetes-secrets) for details.
-
 ## License
 
 Apache License 2.0

api/v1/dataflow_types.go

@@ -14,6 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
+
 package v1
 
 import (