Ambiguous logging of buckets with named profiles

[scaleoutsean]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

In logs, backend-link/head appears to drop namespace context:

• request intended for s3://#profile/bucket
• logs show head on s3://bucketshop
• credential chain falls back to IMDS unexpectedly

If one suspects it's the creds, they spend troubleshooting the profile for s3://bucketshop rather than for the actually used named profile.

Expected Behavior

E 05:47:32.518455 t[qKiWsHtn]: failed to HEAD remote bucket s3://#sg-bucketshop-bucketpipeline-storagegrid-sample/bucketshop...

Current Behavior

I'd expect to see namespace context, rather than s3://bucketshop.

```sh
$ ais ls
NAME				 PRESENT
ais://sg-bucketshop-bucketshop	 yes
Total: [AIS bucket: 1] ========

NAME									 PRESENT
s3://bucketshop								 yes
s3://#sg-bucketshop-bucketpipeline-storagegrid-sample/bucketshop	 yes
Total: [AWS buckets: 2] ========

Logs show remote bucket s3://bucketshop (technically correct, but it's not telling me that it's using the named profile, which may have different credentials):

$ kubectl logs -n ais ais-proxy-0 --since=3m
Defaulted container "aisnode" out of: aisnode, aisinit (init)
E 05:47:32.518455 t[qKiWsHtn]: failed to HEAD remote bucket s3://bucketshop, 
  err: aws-error[operation error S3: HeadBucket, 
  get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded]: HEAD /v1/buckets/bucketshop (called by p[snxwow3f4vd5e]) (p[snxwow3f4vd5e]: htrun.go:1541 <- prxbck.go:314 <- prxbck.go:291 <- proxy.go:2283 <- proxy.go:474])

Steps To Reproduce

Create 2 S3 profiles, one of them named, and do something wrong on the named one.

$ ais ls
NAME				 PRESENT
ais://sg-bucketshop-bucketshop	 yes
Total: [AIS bucket: 1] ========

NAME									 PRESENT
s3://bucketshop								 yes
s3://#sg-bucketshop-bucketpipeline-storagegrid-sample/bucketshop	 yes
Total: [AWS buckets: 2] ========

Now, when working on the bucket with named profile, logs refer to s3://bucketshop.

Possible Solution

No response

Additional Information/Context

No response

AIStore build/version

4.8.42242b334fd57e993919f04acf3c2011cf2db675

Environment details (OS name and version, etc.)

Ubuntu 24.04 LTS

[alex-aizman]

Now, when working on the bucket with named profile, logs refer to s3://bucketshop|

working - how? The entire AIS functionality is about working with buckets one way or another, that's why the question.

Specifically, which client have you used: s3 or native? what command did you run? have you configured "backend_bck" or just the namespace?

Secondly and separately, include ais show bucket output for all buckets involved.

And again, the real steps to reproduce.

[alex-aizman]

PS. the following snippet matches "failed to HEAD remote bucket" log:

807         // + cloud
808         bp := t.Backend(bck)
809         bpropsKV, code, err := bp.HeadBucket(ctx, bck)
810         if err != nil {
811                 if !inBMD {
812                         if code == http.StatusNotFound {
813                                 t.writeErr(w, r, err, code, Silent)
814                         } else {
815                                 err = cmn.NewErrFailedTo(t, "HEAD remote bucket", bck, err, code)
816                                 t._erris(w, r, err, code, cos.IsParseBool(apireq.query.Get(apc.QparamSilent)))
817                         }
818                         return
819                 }

and no, lines 815 and 816 do not strip #namespace if present. So, we are good here.

[scaleoutsean]

Summary:

$ ais ls
NAME				 PRESENT
ais://sg-bucketshop-bucketshop	 yes
Total: [AIS bucket: 1] ========

NAME									 PRESENT
s3://bucketshop								 yes
s3://#sg-bucketshop-bucketpipeline-storagegrid-sample/bucketshop	 yes
Total: [AWS buckets: 2] ========

The buckets:

$ ais show bucket ais://sg-bucketshop-bucketshop
PROPERTY				 VALUE
access					 GET,HEAD-OBJECT,PUT,APPEND,DELETE-OBJECT,MOVE-OBJECT,PROMOTE,UPDATE-OBJECT,HEAD-BUCKET,LIST-OBJECTS,PATCH,SET-BUCKET-ACL,LIST-BUCKETS,SHOW-CLUSTER,CREATE-BUCKET,DESTROY-BUCKET,MOVE-BUCKET,ADMIN
backend_bck.name			 bucketshop
backend_bck.provider			 aws
checksum.enable_read_range		 false
checksum.type				 xxhash2
checksum.validate_cold_get		 false
checksum.validate_obj_move		 false
checksum.validate_warm_get		 false
chunks.checkpoint_every			 0
chunks.chunk_size			 1GiB
chunks.flags				 0
chunks.max_monolithic_size		 1TiB
chunks.objsize_limit			 0B (auto-chunking disabled)
created					 2026-07-08T00:34:59+08:00
ec.bundle_multiplier			 2
ec.burst_buffer				 512
ec.compression				 never
ec.data_slices				 2
ec.disk_only				 false
ec.enabled				 false
ec.objsize_limit			 262144
ec.parity_slices			 2
features				 none
lru.batch_size				 32768
lru.capacity_upd_time			 10m
lru.dont_evict_time			 2h0m
lru.enabled				 false
mirror.burst_buffer			 512
mirror.copies				 2
mirror.enabled				 false
present					 yes
provider				 ais
rate_limit.backend.enabled		 false
rate_limit.backend.interval		 1m
rate_limit.backend.max_tokens		 1000
rate_limit.backend.num_retries		 3
rate_limit.backend.per_op_max_tokens	 
rate_limit.frontend.burst_size		 375
rate_limit.frontend.enabled		 false
rate_limit.frontend.interval		 1m
rate_limit.frontend.max_tokens		 1000
rate_limit.frontend.per_op_max_tokens	 
versioning.enabled			 false
versioning.synchronize			 false
versioning.validate_warm_get		 false
write_policy.data			 immediate
write_policy.md				 immediate


$ ais show bucket s3://#sg-bucketshop-bucketpipeline-storagegrid-sample/bucketshop
PROPERTY				 VALUE
access					 GET,HEAD-OBJECT,PUT,APPEND,DELETE-OBJECT,MOVE-OBJECT,PROMOTE,UPDATE-OBJECT,HEAD-BUCKET,LIST-OBJECTS,PATCH,SET-BUCKET-ACL,LIST-BUCKETS,SHOW-CLUSTER,CREATE-BUCKET,DESTROY-BUCKET,MOVE-BUCKET,ADMIN
backend_bck.name			 
backend_bck.provider			 
checksum.enable_read_range		 false
checksum.type				 xxhash2
checksum.validate_cold_get		 false
checksum.validate_obj_move		 false
checksum.validate_warm_get		 false
chunks.checkpoint_every			 0
chunks.chunk_size			 1GiB
chunks.flags				 0
chunks.max_monolithic_size		 1TiB
chunks.objsize_limit			 0B (auto-chunking disabled)
created					 2026-07-08T00:34:59+08:00
ec.bundle_multiplier			 2
ec.burst_buffer				 512
ec.compression				 never
ec.data_slices				 2
ec.disk_only				 false
ec.enabled				 false
ec.objsize_limit			 262144
ec.parity_slices			 2
extra.aws.cloud_region			 
extra.aws.endpoint			 https://192.168.1.211:10443
extra.aws.max_pagesize			 0
extra.aws.multipart_size		 0B
extra.aws.profile			 sg-bucketshop-bucketpipeline-storagegrid-sample
extra.custom				 
features				 none
lru.batch_size				 32768
lru.capacity_upd_time			 10m
lru.dont_evict_time			 2h0m
lru.enabled				 false
mirror.burst_buffer			 512
mirror.copies				 2
mirror.enabled				 false
present					 yes
provider				 aws
rate_limit.backend.enabled		 false
rate_limit.backend.interval		 1m
rate_limit.backend.max_tokens		 1000
rate_limit.backend.num_retries		 3
rate_limit.backend.per_op_max_tokens	 
rate_limit.frontend.burst_size		 375
rate_limit.frontend.enabled		 false
rate_limit.frontend.interval		 1m
rate_limit.frontend.max_tokens		 1000
rate_limit.frontend.per_op_max_tokens	 
versioning.enabled			 false
versioning.synchronize			 false
versioning.validate_warm_get		 false
write_policy.data			 immediate
write_policy.md				 immediate


$ ais show bucket s3://bucketshop
PROPERTY				 VALUE
access					 GET,HEAD-OBJECT,PUT,APPEND,DELETE-OBJECT,MOVE-OBJECT,PROMOTE,UPDATE-OBJECT,HEAD-BUCKET,LIST-OBJECTS,PATCH,SET-BUCKET-ACL,LIST-BUCKETS,SHOW-CLUSTER,CREATE-BUCKET,DESTROY-BUCKET,MOVE-BUCKET,ADMIN
backend_bck.name			 
backend_bck.provider			 
checksum.enable_read_range		 false
checksum.type				 xxhash2
checksum.validate_cold_get		 false
checksum.validate_obj_move		 false
checksum.validate_warm_get		 false
chunks.checkpoint_every			 0
chunks.chunk_size			 1GiB
chunks.flags				 0
chunks.max_monolithic_size		 1TiB
chunks.objsize_limit			 0B (auto-chunking disabled)
created					 2026-07-08T00:34:59+08:00
ec.bundle_multiplier			 2
ec.burst_buffer				 512
ec.compression				 never
ec.data_slices				 2
ec.disk_only				 false
ec.enabled				 false
ec.objsize_limit			 262144
ec.parity_slices			 2
extra.aws.cloud_region			 
extra.aws.endpoint			 https://192.168.1.211:10443
extra.aws.max_pagesize			 0
extra.aws.multipart_size		 0B
extra.aws.profile			 sg-bucketshop-bucketpipeline-storagegrid-sample
extra.custom				 
features				 none
lru.batch_size				 32768
lru.capacity_upd_time			 10m
lru.dont_evict_time			 2h0m
lru.enabled				 false
mirror.burst_buffer			 512
mirror.copies				 2
mirror.enabled				 false
present					 yes
provider				 aws
rate_limit.backend.enabled		 false
rate_limit.backend.interval		 1m
rate_limit.backend.max_tokens		 1000
rate_limit.backend.num_retries		 3
rate_limit.backend.per_op_max_tokens	 
rate_limit.frontend.burst_size		 375
rate_limit.frontend.enabled		 false
rate_limit.frontend.interval		 1m
rate_limit.frontend.max_tokens		 1000
rate_limit.frontend.per_op_max_tokens	 
versioning.enabled			 false
versioning.synchronize			 false
versioning.validate_warm_get		 false
write_policy.data			 immediate
write_policy.md				 immediate

By "working" I meant using native ais CLI commands such as those from #311, for example, I was trying to make the named S3 bucket the backend of AIS, using various ais bucket props commands.

ais bucket props set \
  ais://sg-bucketshop-bucketshop \
  backend_bck=s3://#sg-bucketshop-bucketpipeline-storagegrid-sample/bucketshop

Now (show bucket output above) they're configured and working, but earlier (because of that typo that I made by copy-pasting) I was seeing only s3://bucketshop-related errors in logs.

It's possible that something else may have triggered commands to s3://bucketshop, but I also don't remember seeing references to s3://#sg-bucketshop-bucketpipeline-storagegrid-sample/bucketshop in the log, so I wasn't just seeing the unexpected, but also not seeing the expected.

Since you checked and it looks like I may have gotten this one wrong, let's close this. I'll reopen if I can duplicate again and get full logs as per this guidance. I've re-installed AIS twice since the report (and it's working fine now) so now I can't really provide anything useful enough to justify leaving this issue open.

[alex-aizman]

let me digest your input and think of it later.. keeping it open for now