Is your feature request related to a problem? Please describe.
Currently, when performing unbuffered, single-shot object uploads via the JSON (REST) API, the payload is streamed continuously in a single HTTP PUT request for maximum performance. However, because HTTP/1.1 requires all headers to be sent before the request body, we cannot append an on-the-fly computed CRC32C or MD5 checksum via the traditional x-goog-hash header. Achieving this currently requires passing over the payload twice (once to pre-compute the checksum and once to stream the data), creating a significant disk I/O performance penalty.
Describe the solution you'd like
To provide server-side validation against network data corruption without the double-read overhead, we should leverage the GCS JSON API's multipart trailing metadata feature.
Implementation Constraints
This trailing checksum strategy should be mutually exclusive with other kinds of checksum validation opt-in'ed by users.
b/521188176
Is your feature request related to a problem? Please describe.
Currently, when performing unbuffered, single-shot object uploads via the JSON (REST) API, the payload is streamed continuously in a single HTTP
PUTrequest for maximum performance. However, because HTTP/1.1 requires all headers to be sent before the request body, we cannot append an on-the-fly computed CRC32C or MD5 checksum via the traditionalx-goog-hashheader. Achieving this currently requires passing over the payload twice (once to pre-compute the checksum and once to stream the data), creating a significant disk I/O performance penalty.Describe the solution you'd like
To provide server-side validation against network data corruption without the double-read overhead, we should leverage the GCS JSON API's multipart trailing metadata feature.
Implementation Constraints
This trailing checksum strategy should be mutually exclusive with other kinds of checksum validation opt-in'ed by users.
b/521188176