DANDI S3 Log Extraction

Fast extraction of access summary data from DANDI S3 logs.

Developed for the DANDI Archive.

Installation

pip install dandi-s3-log-extraction

DANDI Usage

Usage on the DANDI archive logs requires a bit more customization than the generic package.

Begin by listing the IP addresses to exclude, such as all associated Drogon IPs (ask Yarik or Cody for the list), in ~/.s3-log-extraction/excluded_ips.txt. This file, read by the upstream s3-log-extraction package, is the only list of excluded addresses. Their requests are dropped during extraction and left out of every summary.

The file holds one IPv4 or IPv6 address per line. Anything after a # is a comment, and blank lines are ignored. Every entry must be a single exact address. Networks in CIDR notation, wildcards, and regular expressions are refused with an error.

Exclusion only works in one direction over time. Adding an address drops its requests from future extraction. Removing an address from the file does not bring its requests back, since they were never extracted. To include an address again, rerun the extraction from scratch with s3logextraction reset extraction followed by a full dandis3logextraction extract.

The S3_LOG_EXTRACTION_PASSWORD is no longer required by this package. It is only relevant if you opt in to the upstream s3-log-extraction encryption of the IP index and geolocation caches.

In fresh environments, the cache should be specified as:

dandis3logextraction config cache set /mnt/backup/dandi/s3-logs-extraction-cache

To run all the steps (such as for daily updates):

dandis3logextraction extract /mnt/backup/dandi/dandiarchive-logs
dandis3logextraction update summaries
s3logextraction update ip coordinates
dandis3logextraction update totals
dandis3logextraction update summaries --mode archive
dandis3logextraction update totals --mode archive

To check whether a remote extraction has anything to do, without running it, add --check to the same command:

dandis3logextraction extract s3://dandiarchive-logs --mode remote --inventory /path/to/inventory --check

It stops at the first log file that has not been extracted. It exits with status 3 when some are pending and 0 when none are, which keeps pending work apart from an error (status 1). A scheduled job can then start an extraction only when needed.

Requesters are geolocated while the summaries are generated, against the published IP ranges of known cloud services and VPNs and the local GeoLite2-City database of the upstream package. The database is downloaded on first use and refreshed once it is more than a week old, which needs the MAXMIND_ACCOUNT_ID and MAXMIND_LICENSE_KEY environment variables of a free MaxMind account; see the upstream README. The coordinates step comes from the upstream s3logextraction CLI and gives every region of the published summaries a coordinate for the maps.