Topo Imagery

Description

This is a collection of Python scripts used for processing topographic data in and for the cloud (AWS).

Two Docker containers are provided to run the Python scripts:

Container	Built from	Package	Contains
topo-imagery	Dockerfile.raster	geoprocessor-raster	GDAL
geoprocessor/pointcloud	Dockerfile.pointcloud	geoprocessor-pointcloud	PDAL

The containers are available in GitHub Packages.

Usage

The scripts have been implemented to be run inside the Docker container only. This is mainly because of the GDAL dependency.

Local

• Build the Docker image:

docker build --file Dockerfile.raster --tag=topo-imagery .

docker build --file Dockerfile.pointcloud --tag=geoprocessor/pointcloud .

• Example: running the standardise-validate command

This script standardises TIFF files to COGs with a creation of a STAC Item file per TIFF containing the metadata. The input TIFF file paths have to be passed through a json file in the following format:

[
  {
    "output": "tile_name",
    "input": ["./path/to/file.tiff"]
  }
]

where output is the desired output tile name and input is the path to one or several TIFFs. If more than one TIFF, the system will try to retile them into one single output file.

Some test data are available in /e2e/data/ along with the expected output.

Run docker run topo-imagery standardise-validate --help to get the list of the expected arguments.

• Example of local execution. This example uses the test data available on this repo and create the output will be created in a ~/tmp/ on the local machine (volume share with Docker):

docker run -v ${HOME}/tmp/:/tmp/:rw topo-imagery standardise-validate --preset webp --data-type uint8 --from-file ./tests/data/aerial.json --collection-id 123 --start-datetime 2023-01-01 --end-datetime 2023-01-01 --target /tmp/ --source-epsg 2193 --target-epsg 2193 --gsd 10 --create-footprints=true

To use an AWS test dataset (input located in an AWS S3 bucket), log into the AWS account and add the following arguments to the docker run command:

-v ${HOME}/.aws:/root/.aws/:ro -e AWS_PROFILE=your-profile

In the cloud

This package is designed to be run in a Kubernetes cluster using a workflow system. More information can be found in the linz/topo-workflows repository.

Versioning

GitHub Actions automatically handles publishing a container to the GitHub Package Registry (ghcr) and in a private AWS Elastic Container Registry (ECR).

A new container is published every time a change is merged to the master branch. This container will be tagged with the following:

• latest
• github version (example: v1.1.0-2-ga1154e8)

A new container is also published when a release is merged to master (see section bellow). This container will be tagged with the following:

• latest
• vX (example: v1)
• vX.Y (example: v1.2)
• vX.Y.Z (example: v1.2.4)

You can see the tags in the GitHub Packages page.

Releases

Managing

googleapis/release-please is used to support the release process. Based on what has been merged to master (fix, feat, feat!, fix! or refactor!), the library generates a changelog based on the commit messages and creates a Pull Request. This is triggered by this GitHub Action.

The repository is a uv workspace and each of its packages is versioned on its own:

Component	Version in	Git tag	Changelog
Repository (both containers)	pyproject.toml	vX.Y.Z	CHANGELOG.md
geoprocessor-<package>	packages/geoprocessor-<package>/pyproject.toml	geoprocessor-<package>-vX.Y.Z	packages/geoprocessor-<package>/CHANGELOG.md

The repository version takes every commit into account, so it bumps whenever any package does, and both containers keep sharing that single vX.Y.Z tag (see above). A package version only bumps when a user facing commit touches that package's directory, which lets the packages move at their own pace.

All of these bumps are batched into one release: X.Y.Z Pull Request, X.Y.Z being the new repository version. Merging it tags every component that was released.

Publishing

To publish a release, the Pull Request opened by release-please bot needs to be merged:

1. Open the PR and verify that the CHANGELOG contains what you expect in the release. If the latest change you expect is not there, double-check that a GitHub Actions is not currently running or failed.
2. Approve and merge the PR.
3. Once the Pull Request is merged to master a GitHub Action it creates the release and publish a new container tagged for this release.