Skip to content

storage: fix s3 batch delete with unicode filenames - #3487

Merged
syphar merged 1 commit into
rust-lang:mainfrom
syphar:delete-unicode
Aug 31, 2026
Merged

storage: fix s3 batch delete with unicode filenames#3487
syphar merged 1 commit into
rust-lang:mainfrom
syphar:delete-unicode

Conversation

@syphar

@syphar syphar commented Aug 31, 2026

Copy link
Copy Markdown
Member

The back cleanup of the s3 bucket finally showed the actual error we're getting: with certain unicode chars in filenames, the batch-delete fails. This fixes it.

The error is:

error running admin CLI: failed to delete files from s3: ["sources/luminance-gl/0.5.0/\u{1b}", "sources/luminance-gl/0.5.0/.cargo-ok", "sources/luminance-gl/0.5.0/.gitignore", "sources/luminance-gl/0.5.0/.travis.yml", "sources/luminance-gl/0.5.0/CHANGELOG.md", "sources/luminance-gl/0.5.0/Cargo.toml", "sources/luminance-gl/0.5.0/LICENSE", "sources/luminance-gl/0.5.0/README.md", "sources/luminance-gl/0.5.0/src/error.rs", "sources/luminance-gl/0.5.0/src/gl33/buffer.rs", "sources/luminance-gl/0.5.0/src/gl33/framebuffer.rs", "sources/luminance-gl/0.5.0/src/gl33/mod.rs", "sources/luminance-gl/0.5.0/src/gl33/pipeline.rs", "sources/luminance-gl/0.5.0/src/gl33/pixel.rs", "sources/luminance-gl/0.5.0/src/gl33/shader/mod.rs", "sources/luminance-gl/0.5.0/src/gl33/shader/program.rs", "sources/luminance-gl/0.5.0/src/gl33/shader/stage.rs", "sources/luminance-gl/0.5.0/src/gl33/shader/uniform.rs", "sources/luminance-gl/0.5.0/src/gl33/tessellation.rs", "sources/luminance-gl/0.5.0/src/gl33/texture.rs", "sources/luminance-gl/0.5.0/src/gl33/token.rs", "sources/luminance-gl/0.5.0/src/lib.rs"]

Caused by:
    0: service error
    1: unhandled error (MalformedXML)
    2: Error { code: "MalformedXML", message: "The XML you provided was not well-formed or did not validate against our published schema", s3_extended_request_id: "cgqB+l/AMGTBA621/hEC035ZfeX/PvdWyoQTrrst1L8+gZZvlU8IiyyZzl4ZdGAo0rFvyyd4M1A=", aws_request_id: "1GZ7TEFFEFPG6T34" }

Stack backtrace:
   0: <aws_smithy_runtime_api::client::result::SdkError<aws_sdk_s3::operation::delete_objects::DeleteObjectsError, aws_smithy_runtime_api::http::response::Response> as anyhow::context::ext::StdError>::
ext_context::<alloc::string::String>
   1: <futures_util::stream::try_stream::try_for_each::TryForEach<futures_util::stream::stream::map::Map<futures_util::stream::stream::chunks::Chunks<core::pin::Pin<alloc::boxed::Box<dyn futures_core::stream::Stream<Item = core::result::Result<alloc::string::String, anyhow::Error>> + core::marker::Send>>>, <docs_rs_storage::backends::s3::S3Backend as docs_rs_storage::backends::StorageBackendMethods>::delete_prefix::{closure#0}::{closure#0}>, <docs_rs_storage::backends::s3::S3Backend>::delete_batch_with_retry::{closure#0}, <docs_rs_storage::backends::s3::S3Backend as docs_rs_storage::backends::Sto
rageBackendMethods>::delete_prefix::{closure#0}::{closure#1}> as core::future::future::Future>::poll
   2: <docs_rs_storage::storage::non_blocking::AsyncStorage>::delete_prefix::{closure#0}::{closure#0}
   3: <tracing::instrument::Instrumented<<docs_rs_storage::storage::non_blocking::AsyncStorage>::delete_prefix::{closure#0}::{closure#0}> as core::future::future::Future>::poll
   4: docs_rs_admin::cleanup_s3::clean_prefix::<alloc::string::String>::{closure#0}
   5: <tracing::instrument::Instrumented<docs_rs_admin::cleanup_s3::cleanup_s3_bucket::{closure#0}::{closure#0}> as core::future::future::Future>::poll
   6: <docs_rs_admin::DatabaseSubcommand>::handle_args::{closure#0}
   7: <docs_rs_admin::CommandLine>::handle_args::{closure#0}
   8: <tokio::runtime::park::CachedParkThread>::block_on::<docs_rs_admin::main::{closure#0}>
   9: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::multi_thread::MultiThread>::block_on<docs_rs_admin::main::{closure#0}>::{closure#0}, core::result::Result<(), anyhow:
:Error>>
  10: <tokio::runtime::runtime::Runtime>::block_on::<docs_rs_admin::main::{closure#0}>
  11: docs_rs_admin::main
  12: std::sys::backtrace::__rust_begin_short_backtrace::<fn() -> core::result::Result<(), anyhow::Error>, core::result::Result<(), anyhow::Error>>
  13: std::rt::lang_start::<core::result::Result<(), anyhow::Error>>::{closure#0}
  14: std::rt::lang_start_internal
  15: main
  16: __libc_start_main
             at /build/glibc-Df2k3R/glibc-2.27/csu/../csu/libc-start.c:310:0
  17: _start

@syphar
syphar requested a review from a team as a code owner August 31, 2026 09:38
@syphar syphar self-assigned this Aug 31, 2026
@github-actions github-actions Bot added the S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed label Aug 31, 2026
@syphar syphar changed the title wip delete unicode storage: fix s3 batch delete with unicode filenames Aug 31, 2026
@syphar
syphar merged commit 16e9daf into rust-lang:main Aug 31, 2026
7 of 8 checks passed
@syphar
syphar deleted the delete-unicode branch August 31, 2026 10:35
@github-actions github-actions Bot added S-waiting-on-deploy This PR is ready to be merged, but is waiting for an admin to have time to deploy it and removed S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed labels Aug 31, 2026
@syphar syphar removed the S-waiting-on-deploy This PR is ready to be merged, but is waiting for an admin to have time to deploy it label Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants