Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions skills/s3-access-denied-resolver/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Changelog

## 1.0.0

- Initial release. Read-only diagnosis of Amazon S3 403 AccessDenied. Walks the
ten layers that can deny a request (SCP/RCP, VPC endpoint policy, Block Public
Access, bucket policy, access point policy, identity policy, permissions
boundary, session policy, Object Ownership/ACLs, KMS key policy), names the
denying layer and statement, and proposes a policy diff for human review.
- Assigns each layer one verdict from a closed set (`DENIED_BY`,
`WOULD_ALSO_DENY`, `ALLOWED_BUT_UNVERIFIABLE`, `CANNOT_DETERMINE`,
`NOT_APPLICABLE`). There is no plain "allowed" verdict.
- Reads policy documents directly and uses IAM policy simulation only to
corroborate, because simulation does not evaluate VPC endpoint or session
policies.
- Separates same-account rules (an allow in the identity or the resource-based
policy is enough) from cross-account rules (both are required).
- Reads object metadata only and skips `HeadObject` unless it is already
permitted, because `s3:GetObject` also allows reading object contents.
- Checks causes outside the ten layers when every layer is clean: Requester
Pays, Object Lock, SSE-C blocking and access point network origin.
- Denial-message wording in `references/denial-message-hints.md` comes from the
Amazon S3 User Guide topic on 403 errors. KMS error wording is not documented
there and is not matched as fixed text.
- Functional scenarios plant one fault each:
- 01 bucket policy denies `s3:GetObject` to all principals except one named
role.
- 02 key policy denies `kms:Decrypt` to all principals except one named role,
while the reader's identity policy allows it.
- 03 permissions boundary allows only `s3:ListBucket`.
- 04 identity policy grants `s3:GetObject` on the bucket ARN instead of
`bucket/*`.
- 05 gateway endpoint policy allows only a different bucket, reached from a
private-subnet instance with SSM access.
- Not covered: an SCP/RCP scenario (needs an Organizations test account),
Block Public Access, access point, session policy and ACL scenarios.
- Falls back to AWS Config, the owning CloudFormation stack, CloudTrail and an
operator-supplied paste when the Agent Space refuses bucket-level S3 reads, and
names the source each verdict rests on.
128 changes: 128 additions & 0 deletions skills/s3-access-denied-resolver/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# S3 Access Denied Resolver

Read-only DevOps Agent skill that diagnoses an Amazon S3 403 AccessDenied. It
walks the ten layers that can deny an S3 request (SCP/RCP, VPC endpoint policy,
Block Public Access, bucket policy, access point policy, identity policy,
permissions boundary, session policy, Object Ownership/ACLs, KMS key policy),
names the layer and statement that denied the request, and proposes the
smallest fix as a diff for the operator to review.

Scope: S3 general purpose buckets. It is not for 503 SlowDown, CloudFront 403s,
Lake Formation access, or Bedrock/SageMaker role denials.

Taskei: V2468933301.

> **Non-production disclaimer.** Provided for demonstration and evaluation.
> Read-only: it never changes a bucket policy, key policy, role, endpoint or
> any other resource. Apply any proposed change through your own
> change-management and IaC process. Not an AWS service; no warranty.

## Prerequisites

- An AWS DevOps Agent Agent Space with the target account associated.
- The read-only IAM actions below. The Agent Space role can use only actions
that both its own policies and the DevOps Agent permission guardrail allow.

**Covered by `AIDevOpsAgentAccessPolicy`** (checked against policy version v16):

| Service | IAM actions |
|---|---|
| S3 | `s3:GetBucketPolicy`, `s3:GetBucketOwnershipControls`, `s3:GetBucketPublicAccessBlock`, `s3:GetEncryptionConfiguration`, `s3:GetBucketAcl`, `s3:GetAccessPointPolicy` |
| IAM | `iam:GetRole`, `iam:ListAttachedRolePolicies`, `iam:ListRolePolicies`, `iam:GetRolePolicy`, `iam:GetPolicyVersion` |
| KMS | `kms:GetKeyPolicy`, `kms:ListGrants` |
| EC2 | `ec2:DescribeVpcEndpoints` (through `ec2:Describe*`) |
| CloudTrail | `cloudtrail:LookupEvents` |
| Organizations (optional) | `organizations:ListPoliciesForTarget`, `organizations:DescribePolicy` (through `organizations:List*` and `Describe*`); only useful from the management or a delegated-administrator account |

**Not covered. Add as an inline policy on the Agent Space role.** The guardrail
allows all of them through `ReadOnlyAccess`:

| IAM action | Used for |
|---|---|
| `s3:GetAccountPublicAccessBlock` | Account-level Block Public Access (layer 3) |
| `s3:GetObjectAcl` | Object ACL and owner (layer 9) |
| `iam:SimulatePrincipalPolicy` | Corroborating the policy read; never the only evidence |
| `s3:GetObject` | `HeadObject` metadata for the object's encryption key. Optional, see the warning below |

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetAccountPublicAccessBlock",
"s3:GetObjectAcl",
"iam:SimulatePrincipalPolicy"
],
"Resource": "*"
}
]
}
```

Without these, the skill reports the affected layer as `CANNOT_DETERMINE` and
says which access would resolve it. It still diagnoses layers 1 to 8.

> **Agent Space tool limit.** In testing, the Agent Space refused direct bucket-level
> S3 reads (`GetBucketPolicy`, `GetBucketEncryption`, `GetBucketOwnershipControls`,
> `GetPublicAccessBlock`, `ListBuckets`) with "blocked by policy and was not
> executed", in many runs, including after adding the actions above. Added IAM permissions
> did not reliably lift it. The skill then falls back to AWS Config, the owning CloudFormation
> stack and CloudTrail, and otherwise reports `CANNOT_DETERMINE`. For those
> fallbacks, add the read actions below.

| IAM action | Used for |
|---|---|
| `config:GetResourceConfigHistory`, `config:BatchGetResourceConfig` | Bucket policy, Block Public Access, encryption and ACL from the last recorded configuration, where Config records S3 buckets |
| `cloudformation:GetTemplate`, `cloudformation:DescribeStackResources` | The policy a stack deployed, for stack-managed buckets |

> **`s3:GetObject` can read object contents.** `HeadObject` needs the
> `s3:GetObject` permission, which also allows reading the object body. The
> skill reads metadata only, but the permission does not enforce that. Do not
> grant it by default. If you grant it, scope `Resource` to the buckets under
> investigation. Without it, the skill uses the bucket's default encryption for
> layer 10 and says the object-level key was not checked.

IAM action names differ from API names in a few places:
`GetBucketEncryption` needs `s3:GetEncryptionConfiguration`,
`GetPublicAccessBlock` needs `s3:GetBucketPublicAccessBlock` (bucket) or
`s3:GetAccountPublicAccessBlock` (account, through S3 Control), and
`HeadObject` needs `s3:GetObject`.

## Setup

Add the `s3-access-denied-resolver` folder to your Agent Space as a skill. It
targets the `CHAT` and `INCIDENT_RCA` agent types.

## Sample prompts

- "GetObject on my-bucket/data.csv returns 403 AccessDenied from my Lambda. Which layer denies it?"
- "A role in another account cannot read objects from our bucket even though we added it to the bucket policy."
- "EC2 in a private subnet gets 403 on S3 through the gateway endpoint. Why?"

## Tested scenarios

The functional evals plant one fault each in a test account (a bucket, a reader
role and the object `test/hello.txt`) and ask the agent why the read fails:

| Planted fault | Expected verdict |
|---|---|
| Bucket policy denies `s3:GetObject` to all principals except one named role | Layer 4 `DENIED_BY` |
| Key policy denies `kms:Decrypt` to all principals except one named role; the reader's identity policy allows it | Layer 10 `DENIED_BY` |
| Permissions boundary allows only `s3:ListBucket` | Layer 7 `DENIED_BY` |
| Identity policy grants `s3:GetObject` on the bucket ARN, not `bucket/*` | Layer 6 `DENIED_BY` |
| Gateway endpoint policy allows only a different bucket, read from a private-subnet instance | Layer 2 `DENIED_BY` |

Not covered: an SCP/RCP scenario (needs an Organizations test account), and
Block Public Access, access point, session policy and ACL scenarios.

## Evaluation

`evals/` holds the `skill-eval` evals. The functional cases run one prompt per
scenario above, one prompt that asks the skill
to apply the fix itself, and four prompts it should not trigger on.

## Owner

saadnu-aws
89 changes: 89 additions & 0 deletions skills/s3-access-denied-resolver/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
---
name: s3-access-denied-resolver
description: >-
Diagnose an Amazon S3 403 AccessDenied. Use this skill whenever an S3 request
fails with AccessDenied or 403 Forbidden: GetObject, PutObject, ListBucket,
HeadObject or CopyObject calls, messages such as "is not authorized to
perform: s3:" or "explicit deny in a resource-based policy", a kms:Decrypt
denial on an SSE-KMS object, a cross-account bucket that cannot be read, or a
private-subnet instance that gets 403 through an S3 gateway endpoint. Walks
the ten layers that can deny an S3 request (SCP/RCP, VPC endpoint policy,
Block Public Access, bucket policy, access point policy, identity policy,
permissions boundary, session policy, Object Ownership/ACLs, KMS key policy),
names the denying layer and statement, and proposes the smallest fix for human
review. Not for 503 SlowDown, replication or lifecycle issues, CloudFront
403s, Lake Formation access, Bedrock or SageMaker role denials, or a
least-privilege review with no observed failure. Read-only.
metadata:
author: saadnu-aws
version: "1.0.0"
summary: "Read-only root-cause diagnosis for S3 403 AccessDenied: walks ten authorization layers, names the denying layer and statement, and proposes a fix for human review."
aws-devops-agent-skills.agent-types: "CHAT, INCIDENT_RCA"
aws-devops-agent-skills.aws-services: "Amazon S3, AWS IAM, Amazon VPC, AWS CloudTrail, AWS Config"
aws-devops-agent-skills.technical-domains: "Storage, Security"
---

# S3 Access Denied Resolver

Find out why an S3 request was denied. Walk the S3 authorization layers, name
the layer that denied the request, and propose a scoped fix for human review.
Read-only throughout. Never apply a policy change.

## Checklist

- [ ] **Step 1 - Classify:** confirm an observed S3 403/AccessDenied (not throttling, not CloudFront, not Lake Formation). Stop otherwise.
- [ ] **Step 2 - Pin the request:** principal ARN, action, bucket, key, request path (internet, gateway endpoint, interface endpoint, access point), and whether caller and bucket are in different accounts.
- [ ] **Step 3 - Read the denial:** if a CloudTrail event or error message exists, read the policy-type hint in it first (see [references/denial-message-hints.md](references/denial-message-hints.md); load it only when a message or CloudTrail `errorMessage` exists); it narrows the search.
- [ ] **Step 4 - Collect evidence, policy reads first:** read each layer's document directly. Use `SimulatePrincipalPolicy` only as corroboration. If a read is refused as "blocked by policy", do not retry it; load [references/layers.md](references/layers.md#when-a-direct-read-is-blocked) for the fallback sources.
- [ ] **Step 5 - Walk the layers:** evaluate all ten layers; do not stop at the first one that passes.
- [ ] **Step 6 - Assign verdicts** from the closed set below.
- [ ] **Step 7 - Propose a fix:** the smallest change at the denying layer, as a diff. Label any later blockers.
- [ ] **Step 8 - Validate before presenting:** confirm each `DENIED_BY` verdict rests on a policy statement you read, not on an assumption or a simulation result alone; confirm the proposed diff touches only the denying layer and does not widen access beyond the failing request; confirm every layer you could not read is `CANNOT_DETERMINE`.
- [ ] **Step 9 - Deliver the report** in the format below.

## The ten layers

S3 evaluates every layer that applies to the request. Any explicit deny wins, and the request also needs an allow. The layers, outermost first: 1 SCP/RCP, 2 VPC endpoint policy, 3 Block Public Access, 4 bucket policy, 5 access point policy, 6 identity policy, 7 permissions boundary, 8 session policy, 9 Object Ownership/ACLs, 10 KMS key policy and grants.

Load [references/layers.md](references/layers.md#the-ten-layers) at Step 4 for the API call that reads each layer and its typical denial. Load [references/layers.md](references/layers.md#account-rules) at Step 6, before assigning verdicts, for the same-account and cross-account allow rules. Load [references/layers.md](references/layers.md#other-causes-outside-the-ten-layers) only when every layer comes back clean.

## Verdicts (closed set)

- `DENIED_BY` - this layer denies the request. Name one root-cause layer: the one the denial message names, otherwise the first denying layer in table order.
- `WOULD_ALSO_DENY` - another layer that denies, or would still deny once the root cause is fixed.
- `ALLOWED_BUT_UNVERIFIABLE` - the document read allows the request, but conditions or context could not be fully checked.
- `CANNOT_DETERMINE` - the layer could not be read (missing permission, other account). State what is needed.
- `NOT_APPLICABLE` - the layer does not apply (for example, no endpoint in the path, SSE-S3 encryption).

There is no plain "allowed" verdict.

## Report format

1. **Root cause** - one sentence naming the layer and the statement (Sid or policy name).
2. **Layer table** - all ten layers with verdict and one-line evidence.
3. **Proposed fix** - JSON diff for the denying layer only, with a note that it is for human review.
4. **Subsequent blockers** - any `WOULD_ALSO_DENY` layers and their fixes.
5. **Could not check** - every `CANNOT_DETERMINE` layer and what access would resolve it.

## Gotchas

- A "blocked by policy" refusal on an S3 read is a platform limit, not a finding about the bucket. Use the fallback sources, state which one you used, and never infer a layer's content from the refusal.
- `SimulatePrincipalPolicy` ignores VPC endpoint policies and session policies and only partly evaluates SCPs and RCPs. A simulation result of "allowed" is never a verdict on its own.
- A cross-account caller outside the organization gets a bare "Access Denied" with no policy type, so the message cannot narrow the search. Walk every layer.
- A missing allow in the identity policy is not a finding when the bucket policy allows a same-account caller, and the reverse. Check both before flagging.
- A resource ARN of `arn:aws:s3:::bucket` covers bucket-level actions such as `ListBucket` only. Object actions need `arn:aws:s3:::bucket/*`.
- A KMS key policy that allows only the account root does not by itself give the role `kms:Decrypt`. The role's identity policy must also allow it.
- Fixing the root-cause layer often exposes the next denial (for example, the key policy after the bucket policy). Report those as `WOULD_ALSO_DENY` up front.

## Output discipline

Do not narrate API calls or paste raw policy documents. Speak only when starting,
asking a question, or delivering the report.

## Safety

Read-only. Never call `PutBucketPolicy`, `PutPublicAccessBlock`, `PutKeyPolicy`,
`PutRolePolicy`, or any other write API. Read object metadata only: never read
object contents, and skip `HeadObject` when it is not already permitted. Redact
account IDs and ARNs from example output when the user asks for a shareable
version.
Loading
Loading