Skip to content

aws login sometimes writes a login cache that can't be read back (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT) #10879

Open

Description

@kdrag0n

Describe the bug

About 1 in 256 aws login runs succeeds but leaves a cached token that every later command fails to load.

The login generates a P-256 DPoP key and stores it in the login cache as a SEC1 PEM. When the key's private scalar happens to start with a zero byte, awscrt on Linux exports it with a 31-byte privateKey instead of 32, and awscrt's own importer refuses a scalar that isn't exactly 32 bytes. The login itself works because it uses the key in memory, but everything after it reads the key from the cache and fails.

The root cause is in aws-c-cal, and I filed it there with a C repro and a patch: awslabs/aws-c-cal#262. Filing here as well because this is where it shows up for users, and because the CLI turns it into a traceback.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

A key written by aws login can be read back by the next command. If the cached key can't be loaded for any reason, I'd expect the "Unable to load private key from cached token" error instead of a traceback.

Current Behavior

After one login, every aws command using that profile failed like this until I ran aws login again:

File "awscli/botocore/credentials.py", line 2547, in refresh_credentials
File "awscli/botocore/credentials.py", line 2656, in _load_private_key
File "awscrt/crypto.py", line 279, in new_key_from_der_data
RuntimeError: 7178 (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT): Key format is not supported.

The dpopKey in the cache file decoded to this (values left out):

SEQUENCE len=118
  INTEGER len=1 value=1
  OCTET STRING len=31        <- privateKey, should be 32
  [0] OID prime256v1
  [1] BIT STRING len=66

Reproduction Steps

It takes a few hundred logins to hit by hand, so this does the same three calls as the CLI in a loop: generate the key (awscli/customizations/login/utils.py:70), export it as SEC1 (utils.py:202), and load it back (awscli/botocore/credentials.py:2656). Run it on Linux with the awscrt that the CLI pins (pip install awscrt==0.38.0 for 2.39.3):

from awscrt.crypto import EC, ECExportFormat, ECType

for i in range(1, 5001):
    der = EC.new_generate(ECType.P_256).export_key(ECExportFormat.SEC1)
    try:
        EC.new_key_from_der_data(der)
    except Exception as e:
        # der[6] is the length of the privateKey OCTET STRING
        print(f"key {i}: privateKey is {der[6]} bytes: {type(e).__name__}: {e}")
        break
else:
    print("no failure in 5000 keys")

Three runs:

key 99: privateKey is 31 bytes: RuntimeError: 7178 (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT): Key format is not supported.
key 142: privateKey is 31 bytes: RuntimeError: 7178 (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT): Key format is not supported.
key 737: privateKey is 31 bytes: RuntimeError: 7178 (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT): Key format is not supported.

On macOS the same loop doesn't fail (20,000 keys), because awscrt uses a different crypto backend there that always stores 32 bytes.

Possible Solution

The real fix is in aws-c-cal: pad the scalar on export, and accept a short one on import so caches already on disk become readable again. Details are in the issue linked above.

Two things I noticed on the CLI side (I haven't tried changing either):

  • _load_private_key catches ValueError, but awscrt raises RuntimeError for a key it can't parse, so the LoginTokenLoadError path is never taken:

    try:
    return EC.new_key_from_der_data(base64.b64decode(contents))
    except ValueError as e:
    raise LoginTokenLoadError(
    error_msg=f'Unable to load private key from cached token: {e}'
    )

  • The DPoP key is new on every login, so the login could load the exported key once before writing the cache, and generate another if that fails. That would also cover CLI versions that ship before a fixed awscrt.

Additional Information/Context

Workaround: run aws login again. The next key almost certainly has a non-zero first byte.

2.40.0 pins awscrt 0.38.1, which bundles the same aws-c-cal commit as 0.38.0, so I expect it to behave the same, but I haven't run 2.40.0.

CLI version used

aws-cli/2.39.3 (awscrt 0.38.0, aws-c-cal v1.0.0)

Environment details (OS name and version, etc.)

The failed login was on Linux. The repro above ran on Debian 13 (arm64), Python 3.13.5, awscrt 0.38.0 from PyPI.

Generated by AI tools, and reviewed by Danny Lin.

Activity

  1. added
    bugThis issue is a bug.
    needs-triageThis issue or PR still needs to be triaged.
    on Nov 11, 2026
  2. changed the title [-]`aws login` leaves an unusable credential cache about 1 time in 256 (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT on every later command)[/-] [+]`aws login` sometimes writes a login cache that no later command can read (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT)[/+] on Nov 11, 2026
  3. changed the title [-]`aws login` sometimes writes a login cache that no later command can read (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT)[/-] [+]`aws login` sometimes writes a login cache that can't be read back (AWS_ERROR_CAL_UNSUPPORTED_KEY_FORMAT)[/+] on Nov 11, 2026
  4. coughlmao commented on Nov 19, 2026

    @coughlmao

    Hi @kdrag0n, I'd like to work on this issue.

    I've reviewed the reproduction steps and the related aws-c-cal issue #262. I understand that the underlying serialization defect belongs to aws-c-cal, but there are opportunities to improve the AWS CLI's handling of malformed cached keys.

    I'd like to investigate a focused Python-side fix, particularly handling the RuntimeError raised by awscrt in _load_private_key, ensuring malformed cached keys produce the intended LoginTokenLoadError instead of an unhandled traceback, and adding regression tests.

    Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.needs-triageThis issue or PR still needs to be triaged.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions