Event-Driven Declarative Orchestrator

Get started with Kestra in 3 minutes.

Kestra Firecrawl Plugin

Why

• What user problem does this solve? Flows that feed RAG pipelines, monitor competitor pages or collect web data need clean, structured page content, and the Firecrawl API is asynchronous for crawls.
• Why would a team adopt this plugin in a workflow? It scrapes or crawls sites in one task, downloads the results to the internal storage before they expire and cancels the crawl when the task is killed so it stops consuming credits.
• What operational/business outcome does it enable? Web ingestion becomes a declarative, retryable step that plugs directly into AI, database and notification tasks.

What

• Provides plugin components under io.kestra.plugin.firecrawl.
• Includes classes such as Scrape and Crawl.

Running Kestra locally with this plugin

1. Build the shadow JAR: ./gradlew shadowJar. The output lands in build/libs/.
2. Run docker compose up. docker-compose.yml builds kestra/kestra:latest and mounts build/libs/ to /app/plugins/, so Kestra picks up the jar on startup.
3. Kestra UI is available at localhost:8080.

Plugins folder gotcha

Mounting a host folder onto /app/plugins/ replaces the container's plugins directory rather than adding to it. Core plugins (the ones logged as Registered N core plugins) are compiled into Kestra itself and aren't affected, but any additional plugin normally bundled in the base image under /app/plugins/ (e.g. the Python script plugin) gets hidden once the mount is in place. If a flow you're testing depends on another plugin, copy its jar into build/libs/ too before starting the container.

JFR startup error

On some hosts, command: server local fails with:

Unable to create JFR repository directory using base location (/tmp)

docker-compose.yml works around this by mounting /tmp as tmpfs. If you build your own compose file or run Kestra via docker run, add the same workaround, e.g. -v /tmp:/tmp or --tmpfs /tmp. Tracked upstream in kestra-io/kestra#17405.

Documentation

• Full documentation can be found under: kestra.io/docs
• Documentation for developing a plugin is included in the Plugin Developer Guide

License

Apache 2.0 © Kestra Technologies

Stay up to date

We release new versions every month. Give the main repository a star to stay up to date with the latest releases and get notified about future updates.