Data Export and Import Questions

[jerrac]

Background

I have a standalone Greptime 0.17.2 instance running in Docker via Docker Compose. There area few other services on that vm, and I have caused some of them to crash when hitting Greptime with too large of a query. (I'm still figuring out how to not do that...)

Due to that, I thought moving Greptime over to a vm on its own would help ensure I don't take other things down.

At the same time I thought I'd set up data storage on an S3 compatible target. I think if I ever need to cluster, having the data on S3 already would help.

My problem is that the data export is really really slow. As in my metrics table is something like 27GB, and my export has only created 19GB worth of data in over 20 hours. The docs mention watching system resources, but not really anything on how to adjust things based on what you see.

I tried exporting a second table at the same time, but that caused the docker container to run out of resources and get killed.

This is inside our data center, so networking should not be an issue. I think we're on 10G at the slowest.

My target storage is an nfs mount with lots of space. I initially started exporting the data the wrong way and it was getting sent to the local container storage, and that export was moving just as slow. So the speed issue should not be caused by NFS, I hope.

I do have the docker service limited with mem_limit: "8GB" and cpus: 4.0 settings. Which I'm guessing might be where the problem is? Specifically docker stats tells me that the container is using well over 200% cpu, and is using up almost all of it's available RAM.

Questions:

• Would increasing the memory and cpu limits help?
• Can the data exports be interrupted and then resumed, or do you have to start over from scratch? The current export was initially interrupted, but when I restarted it, I couldn't tell if it was overwriting the existing data, or if it had somehow known to resume from where it left off.
• Is there another way to get the data transferred? As in could I start using my new instance of Greptime, and at the same time get the old data pushed into it little by little?
• Any other tips? Places I probably misunderstood something?

Thanks!

[fengjiachun]

Thank you for the detailed context and questions. Here are a few suggestions that may help:

1. Prefer splitting the export by time ranges first
   For a 27 GB metrics table, doing one huge “all history” export is usually the slowest and most fragile option. The export tool supports time-range based export, so it’s better to break the data into several windows (for example: per month, or even per week/day if needed) and run multiple exports:

# Example: export a single database within a time range
greptime cli data export \ 
  --addr <your_greptime_addr> \ 
  --output-dir /path/to/backup/metrics-2024-01 \ 
  --database <your_database_name> \ 
  --start-time "2024-01-01 00:00:00" \ 
  --end-time "2024-01-31 23:59:59"

You can repeat this for different time windows until the full history is covered. This has two benefits:

Each export job is smaller and less likely to hit resource limits or fail halfway.

If one job fails, you only need to re-run that specific time range, not the whole 27 GB again.

2. About CPU / memory limits

Increasing the memory limit is the most effective way to avoid the container being OOM-killed during export.
The export process needs additional working memory for scanning data, buffering rows, encoding, and writing files. With only 8 GB available, a large table export (such as 27 GB of metrics) can easily exhaust the container’s memory and get terminated by Docker.

If possible, raising the memory limit (for example to 16–32 GB, depending on your VM capacity) will significantly reduce the risk of OOM and make the export more stable.
CPU limits can also be raised, but memory is the more critical factor for preventing crashes.

As in could I start using my new instance of Greptime, and at the same time get the old data pushed into it little by little?

Yes — as long as your new writes do not use timestamps that overlap with the historical data you are importing, you can absolutely start writing to the new Greptime instance while gradually importing the old data.

This approach is safe and commonly used when migrating from one instance to another without downtime. The only thing to avoid is writing new data that falls into the same timestamp ranges as the historical batches you are importing.

If your system always writes strictly increasing timestamps, then you can migrate history “little by little” without any interruption.

[jerrac]

Thanks! Doing small exports plus just pointing at the new instance for new data makes a lot of sense. I just wasn't sure I could do that exports/imports piecemeal. I'll get started on that next week, other stuff came up today.

[jerrac]

I just ran into the issue described here: #7442

And that got me wondering, for importing data, is the value of the --import-dir expected to be a directory local to the server you are importing data to?

[fengjiachun]

Quick update: the new Export/Import V2 work has now landed on main. The delivery was tracked in:

• tracking: split export/import v2 into 5 stacked PRs #7786

Compared with the older export/import flow discussed above, Export/Import V2 adds URI-based snapshots, resumable chunked export/import, snapshot verify/list/delete, progress reporting, parallelism controls, and local/MinIO object-store E2E coverage.

For a migration like the one described here, the recommended V2 flow is roughly:

1. Export in bounded time ranges with a chunk window, for example daily or weekly windows depending on data volume.
2. Verify the snapshot before importing.
3. Import into the new instance, using an explicit --state-path if you want a known resume-state location.
4. If the import is interrupted, rerun the same import command to resume unfinished data tasks.

Also, for the older --import-dir / local path question: yes, local filesystem paths need to be accessible from the GreptimeDB server doing the import, not only from the CLI environment. For containerized or remote deployments, object storage such as S3/MinIO is usually less error-prone because both the export and import sides can access the same snapshot URI.

We're also adding user-facing docs/examples for the V2 commands here:

• docs: add export import v2 guide docs#2585

[jerrac]

Nice. That will be helpful next time I have to migrate. I didn't end up getting everything from my old cluster moved, just didn't have time to figure it all out when other things were placed at a higher priority. I'll set your info as the answer since I think it really is at this point in time.

[silverfox23]

Hello! We have similar performance issues with data export/import. We’re seeing very slow GreptimeDB export/import performance when the dataset has a very high table count - even though the on-disk size is relatively small (e.g. ~102k tables / 12.3 GiB and ~144k tables / 8.1 GiB). Exports take at least 1-2 days in a 8 core and 32GiB memory Incus container. We've tried out both v1 and v2 exports with different parallelism options but the performance wouldn't improve in any meaningful way. The main bottleneck appears to be the sheer number of tables rather than total data volume.

We can't really reduce the amount of tables since we're using Netdata as our metrics exporter and it automatically created these tables. Is there a configuration or strategy to improve export/import performance specifically for high table cardinality?

[evenyag]

https://learn.netdata.cloud/docs/exporting-metrics

I guess the ingestion protocol is prometheus remote write, so they should be metrics tables.

[fengjiachun]

One thing we noticed is that export-v2 defaults to --parallelism 1, which means only one table is exported at a time.

If your previous test used the default value, could you try adding --parallelism 8 to the export command and let us know whether it improves the export time?

[rda0]

I am taking over for @silverfox23 who left our team in the mean time. Thank you for looking into this.

We tried to reconstruct some of the information you requested but are not 100% sure it is correct:

Could you provide the following information to help us identify where most of the time is spent?

* The complete commands used to test V1 and V2

This should be one of the commands used:

1735 2026-07-07T14:19:48+0000 time sudo -u greptimedb greptime cli data export-v2 create --addr 127.0.0.1:4000 --to file:///var/backups/greptime-v2/ --parallelism 100 --chunk-parallelism 12 --chunk-time-window 90d --start-time 2025-01-01T00:00:00Z --end-time 2026-04-30T23:59:59Z --timeout 72h --progress always --schemas opentsdb_autogen

The commands used to export other schemas probably used similar parameters.

* The storage backend used for the snapshot

This should be metric engine.

* The time spent before and after `Snapshot created` in the export logs
* The approximate time spent in the `Executing DDL (x/y)` stage during import

We could not find that in our logs.

Please also provide the table engine distribution:

MySQL root@localhost:opentsdb_10m> SELECT engine, COUNT(*)
                                -> FROM information_schema.tables
                                -> WHERE table_type = 'BASE TABLE'
                                -> GROUP BY engine;
+--------+----------+
| engine | count(*) |
+--------+----------+
| metric | 257109   |
| mito   | 958      |
+--------+----------+

2 rows in set
Time: 4.268s
MySQL root@localhost:opentsdb_10m> SELECT engine, COUNT(*)
                                -> FROM information_schema.tables
                                -> WHERE table_type = 'BASE TABLE' AND table_schema = 'opentsdb_10m'
                                -> GROUP BY engine;
+--------+----------+
| engine | count(*) |
+--------+----------+
| metric | 143813   |
+--------+----------+

1 row in set
Time: 3.879s

If most of the tables use the Metric Engine, please also provide the number of physical metric tables:

MySQL root@localhost:opentsdb_10m> SELECT COUNT(*)
                                -> FROM information_schema.tables
                                -> WHERE table_type = 'BASE TABLE'
                                ->   AND engine = 'metric'
                                ->   AND create_options LIKE '%physical_metric_table%';
+----------+
| count(*) |
+----------+
| 7        |
+----------+

1 row in set
Time: 4.071s
MySQL root@localhost:opentsdb_10m> SELECT COUNT(*)
                                -> FROM information_schema.tables
                                -> WHERE table_type = 'BASE TABLE' AND table_schema = 'opentsdb_10m'
                                ->   AND engine = 'metric'
                                ->   AND create_options LIKE '%physical_metric_table%';
+----------+
| count(*) |
+----------+
| 1        |
+----------+

1 row in set
Time: 3.920s

These export tests were done using imported data. The import was done using this script (via http://localhost:4000/v1/influxdb/write?db=${database}_${retention_policy}):

Import script

#!/usr/bin/env bash
database=opentsdb
export_directory=/var/backups/influxdb

# get measurements
echo "getting measurements in ${database}"
influx -database "${database}" -execute 'show measurements' -format 'column' | tail +4 > "${measurements_file}"

for retention_policy in autogen 1m 10m; do
    measurements_file="/root/influxdb_${retention_policy}_${database}_measurements.txt"

    # dropping database
    echo "dropping ${database}_${retention_policy} database"
    mysql -P 4002 -e "DROP DATABASE IF EXISTS ${database}_${retention_policy}"

    # create opentsdb database and metric table engines
    echo "creating ${database} database"
    mysql -P 4002 -e "CREATE DATABASE IF NOT EXISTS ${database}_${retention_policy}"
    mysql -P 4002 -e "CREATE TABLE IF NOT EXISTS \`${database}_${retention_policy}\`.\`greptime_physical_table\` ( \`_has_unstable_connection\` STRING NULL, \`_is_ephemeral\` STRING NULL, \`host\` STRING NULL, \`greptime_value\` DOUBLE NULL, \`greptime_timestamp\` TIMESTAMP(3) NOT NULL, TIME INDEX (\`greptime_timestamp\`), PRIMARY KEY (\`_has_unstable_connection\`, \`_is_ephemeral\`, \`host\`) ) ENGINE = metric WITH ( \`physical_metric_table\` = \`true\`, \`index.type\` = \`skipping\` )"

    while read -r measurement; do
            echo "creating ${measurement}"
            mysql -P 4002 -e "CREATE TABLE IF NOT EXISTS \`${database}_${retention_policy}\`.\`${measurement}\` ( \`_has_unstable_connection\` STRING NULL, \`_is_ephemeral\` STRING NULL, \`host\` STRING NULL, \`greptime_value\` DOUBLE NULL, \`greptime_timestamp\` TIMESTAMP(3) NOT NULL, TIME INDEX (\`greptime_timestamp\`), PRIMARY KEY (\`_has_unstable_connection\`, \`_is_ephemeral\`, \`host\`) ) ENGINE = metric WITH ( \`on_physical_table\` = \`greptime_physical_table\` )"
    done <"${measurements_file}"
done

for retention_policy in autogen 1m 10m; do
    lp_file="${export_directory}/${database}.${retention_policy}.lp"
    lp_directory="${export_directory}/${database}/${retention_policy}"

    echo "> cleaning up last export job"
    for file in $(ls ${lp_directory}); do rm "${lp_directory}/${file}"; done

    echo "> migrating ${database}_${retention_policy}"
    mkdir -p "${lp_directory}"

    # export lps for each month
    for year in 2025 2026; do
        for month in $(seq 1 12); do
            month_end_day=("31" "28" "31" "30" "31" "30" "31" "31" "30" "31" "30" "31")
            month_index=$((month-1))
            lp_month_file="${lp_file}.${year}.${month}"
            if [ ${#month} -eq 1 ]; then month="0${month}" ; fi
            month_start="${year}-${month}-01T00:00:00Z"
            month_end="${year}-${month}-${month_end_day[$month_index]}T23:59:59Z"
            echo "=== exporting ${month_start} to ${month_end} ==="

            # export InfluxDB data as line protocols
            echo "> get line protocol"
            if [ -f "${lp_month_file}.gz" ]; then
                echo "decompressing ${lp_month_file}.gz"
                gzip -d "${lp_month_file}.gz"
            else
                echo "exporting line protocol"
                influx_inspect export -datadir "/var/lib/influxdb/data/" -waldir "/var/lib/influxdb/wal/" -database "${database}" -retention "${retention_policy}" -out "${lp_month_file}" -lponly -start "${month_start}" -end "${month_end}"
            fi

            # change value= to greptime_value= to ensure consistency for prometheus write api
            echo "> changing value= to greptime_value= in ${lp_month_file}"
            sed -i 's/ value=/ greptime_value=/' "${lp_month_file}"


            # split line protocols into manageable chunks
            echo "> splitting line protocol up"
            split -l 100000 -d -a 10 "${lp_month_file}" "${lp_directory}/"

            # import line protocols
            echo "> importing line protocols"
            for file in $(ls ${lp_directory}); do echo "importing ${file}"; curl -i --retry 100 --retry-connrefused -X POST "http://localhost:4000/v1/influxdb/write?db=${database}_${retention_policy}"  --data-binary @"${lp_directory}/${file}"; sleep 1; done

            # make space
            echo "> deleting line protocols"
            for file in $(ls ${lp_directory}); do rm "${lp_directory}/${file}"; done

            echo "> compressing line protocol"
            gzip "${lp_month_file}"
            echo "======"
        done
    done
done

The data was created by the netdata exporter (via opentsdb:influxdb exporter) using the following export config:

[exporting:global]
    enabled = yes

[opentsdb:influxdb]
    enabled = yes
    destination = 127.0.0.1:4242
    update every = 10
    buffer on failures = 10
    timeout ms = 60000

which writes to the opentsdb.autogen retention policy (RP) in influxdb. The data in the two other RPs (opentsdb.1m, opentsdb.10m) were generated by automatic downsampling with 1 minute and 10 minute resolutions. The sql info above was taken from opentsdb_10m with ~144k tables:

MySQL root@localhost:(none)> show databases
+--------------------+
| Database           |
+--------------------+
| greptime_private   |
| information_schema |
| netdata            |
| opentsdb_10m       |
| opentsdb_1m        |
| opentsdb_autogen   |
| prometheus_10m     |
| prometheus_1m      |
| prometheus_autogen |
| public             |
+--------------------+

10 rows in set
Time: 0.020s
MySQL root@localhost:(none)> use opentsdb_10m
You are now connected to database "opentsdb_10m" as user "root"
Time: 0.003s
MySQL root@localhost:opentsdb_10m> show tables
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Tables_in_opentsdb_10m                                                                                                                                                                                     |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| greptime_physical_table                                                                                                                                                                                    |
| netdata.Infiniband.ib_cntbytes_ibp4s0_1.Received                                                                                                                                                           |
| netdata.Infiniband.ib_cntbytes_ibp4s0_1.Sent                                                                                                                                                               |
| netdata.Infiniband.ib_cntbytes_mlx4_0_1.Received                                                                                                                                                           |
| netdata.Infiniband.ib_cntbytes_mlx4_0_1.Sent                                                                                                                                                               |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Buffer_overrun                                                                                                                                                    |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Link_Downed                                                                                                                                                       |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Link_integrity_err                                                                                                                                                |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Link_minor_errors                                                                                                                                                 |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Link_recovered                                                                                                                                                    |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Pkts_malformated                                                                                                                                                  |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Pkts_missed_resource                                                                                                                                              |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Pkts_rcvd_discarded_                                                                                                                                              |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Pkts_rcvd_discarded_by_switch                                                                                                                                     |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Pkts_rcvd_with_EBP                                                                                                                                                |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Pkts_sent_discarded                                                                                                                                               |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Pkts_sent_discarded_by_switch                                                                                                                                     |
| netdata.Infiniband.ib_cnterrors_ibp4s0_1.Tick_Wait_to_send                                                                                                                                                 |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Buffer_overrun                                                                                                                                                    |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Link_Downed                                                                                                                                                       |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Link_integrity_err                                                                                                                                                |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Link_minor_errors                                                                                                                                                 |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Link_recovered                                                                                                                                                    |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Pkts_malformated                                                                                                                                                  |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Pkts_missed_resource                                                                                                                                              |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Pkts_rcvd_discarded_                                                                                                                                              |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Pkts_rcvd_discarded_by_switch                                                                                                                                     |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Pkts_rcvd_with_EBP                                                                                                                                                |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Pkts_sent_discarded                                                                                                                                               |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Pkts_sent_discarded_by_switch                                                                                                                                     |
| netdata.Infiniband.ib_cnterrors_mlx4_0_1.Tick_Wait_to_send                                                                                                                                                 |
| netdata.Infiniband.ib_cntpackets_ibp4s0_1.Mcast_rcvd                                                                                                                                                       |
| netdata.Infiniband.ib_cntpackets_ibp4s0_1.Mcast_sent                                                                                                                                                       |
| netdata.Infiniband.ib_cntpackets_ibp4s0_1.Received                                                                                                                                                         |
| netdata.Infiniband.ib_cntpackets_ibp4s0_1.Sent
...
143813 rows in set
Time: 20.730s

Some background information:

We started this project to evaluate GreptimeDB and other TSDBs as alternatives to InfluxDB. One of our use cases is storing historical monitoring data from our IT infrastructure.

During testing, I came to believe that importing all of our historical data from InfluxDB may not be the best approach. Most of it was generated by Netdata exporters over several years and across multiple Netdata versions. We also experienced occasional GreptimeDB crashes after importing the data.

[rda0]

For now, we have decided to continue with a fresh GreptimeDB instance and export new data using Prometheus Remote Write with the following Netdata configuration:

[exporting:global]
    enabled = yes

[opentsdb:influxdb]
    enabled = yes
    destination = 127.0.0.1:4242
    update every = 10
    buffer on failures = 10
    timeout ms = 60000

[prometheus_remote_write:greptimedb]
    enabled = yes
    destination = 127.0.0.1:4000
    remote write URL path = /v1/prometheus/write?db=netdata
    update every = 10
    buffer on failures = 2
    timeout ms = 5000

An export of the currently 37 days worth of data (40 GiB) took 70 minutes:

+phd-test-mon:~# mkdir /var/lib/greptime/export
+phd-test-mon:~# chown greptime:greptime /var/lib/greptime/export
+phd-test-mon:~# sudo -u greptime greptime cli data export-v2 create --addr 127.0.0.1:4000 --to file:///var/lib/greptime/export/ --parallelism 8 --schemas netdata --timeout 72h --progress always
2026-07-28T10:26:45.892953Z  INFO cli::database: Using system proxy(if any)
2026-07-28T10:26:45.893105Z  INFO cmd: Starting app: greptime-cli
2026-07-28T10:26:45.950792Z  INFO cli::data::export_v2::command: Exporting schemas: ["netdata"]
2026-07-28T10:26:45.959437Z  INFO cli::data::export_v2::command: Exported 1 schemas
2026-07-28T10:26:51.391891Z  INFO cli::data::export_v2::command: Exported DDL for schema netdata to schema/ddl/netdata.sql
2026-07-28T10:26:51.397093Z  INFO cli::data::export_v2::command: Snapshot created: bb227562-7058-46f0-9946-3a0389931ba9
Export data chunks [----------------------------------------] 0/1 (0%)                                                                                                                                                                                                                                                                                         2026-07-28T10:26:51.403544Z  INFO cli::data::export_v2::data: Executing sql: COPY DATABASE "greptime"."netdata" TO '/var/lib/greptime/export/data/netdata/1/' WITH (FORMAT='parquet', PARALLELISM=8);
2026-07-28T11:35:50.976595Z  INFO cli::data::export

This is on an 8 core VM with 64 GiB memory. Resource usage during the export:

Table details:

MySQL [(none)]> SELECT engine, COUNT(*)
    -> FROM information_schema.tables
    -> WHERE table_type = 'BASE TABLE' AND table_schema = 'netdata'
    -> GROUP BY engine;
+--------+----------+
| engine | count(*) |
+--------+----------+
| metric |      688 |
+--------+----------+
1 row in set (3.253 sec)

MySQL [(none)]> SELECT COUNT(*)
    -> FROM information_schema.tables
    -> WHERE table_type = 'BASE TABLE' AND table_schema = 'netdata'
    ->   AND engine = 'metric'
    ->   AND create_options LIKE '%physical_metric_table%';
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.089 sec)

I have yet to deploy GreptimeDB on our production monitoring server, which will produce more or other data. But this will take some time.

[fengjiachun]

@rda0 @silverfox23 An update on the high-table-count case. The Metric export/import optimization tracked in #9120 is now merged on main. Export now scans each physical Metric table once and writes the logical tables into a few packed objects instead of one file per table. Import recreates the logical tables with batched DDL.

We ran it on AWS with 100,000 Metric logical tables on 10 physical tables (32 rows each, 3.2 million rows). The machine was one 16 vCPU / 32 GiB EC2 instance, with the snapshot on S3 in the same region:

• Export: 58 s; the snapshot is about 507 MB in 5 objects.
• Import: 65 s, including creating all 100,000 logical tables through 790 batch DDL requests.
• After restore, the schema, data and DDL of every table matched the source.

This workload has very little data per table, so it measures the per-table overhead you ran into, not bulk data throughput. Your schemas also hold several GiB of data, which adds its own transfer time.

To try it before the next release, use a current greptime/greptimedb-nightly image on both the source and the target, and export with --pack-metric-data:

greptime cli data export-v2 create --addr <addr> --to s3://<bucket>/<path> --schemas opentsdb_10m --pack-metric-data --parallelism 4

Leave out --chunk-time-window, or keep the number of chunks small. import-v2 uses batch DDL automatically when the target supports it. Older importers reject packed snapshots, so the target must also run the new version.

The test conditions and full results are in #9573. The run itself is 38544464731. User documentation is in progress in GreptimeTeam/docs#2940.