Support Azure and shared obstore automation reads

CHANGELOG.md

@@ -11,15 +11,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Support Python 3.14 across all packages.
 - Test the lowest compatible direct dependencies on Python 3.11 in CI.
+- `tilebox-workflows`: Read Azure blobs in storage-event tasks. Use the account and container returned by the API.
+  Authenticate with Azure Identity, or use an account key or SAS token configured in the environment.
+- `tilebox-workflows`: Add `AzureBlobCache(account_name, container, prefix="jobs")` with the same authentication
+  as Azure automation reads. Cache groups share the store and credentials.
 
 ### Changed
 
+- `tilebox-workflows`: List storage locations through `StorageLocationService`. Read S3, GCS, and Azure objects
+  through obstore, with boto3, Google Auth, and Azure Identity handling credentials. Cloud reads now raise
+  obstore/Python exceptions instead of AWS/GCP SDK exceptions. Missing objects raise `FileNotFoundError`.
+- `tilebox-workflows`: Replace the Google storage SDK dependency with `google-auth[requests]` and add `azure-identity`.
+  Import cloud authentication libraries only when needed and reuse cloud storage clients between automation reads.
+- `tilebox-workflows`: Use obstore and Google Auth for `GoogleStorageCache`. Pass a bucket name instead of a
+  Google SDK bucket object. Cache groups share the same store and credentials.
+- `tilebox-workflows`: Use obstore for `AmazonS3Cache`, with the same boto3 session credentials as automation reads.
+  Cache groups share the store and credentials instead of creating new S3 clients.
 - Require Python 3.11 or newer across all packages, removing Python 3.10 compatibility code and typing backfills.
 - `tilebox-storage`: Replace legacy synchronous client patching with explicit wrappers using `asyncio.run()`.
   When called inside a running event loop (including notebooks), run the operation in a worker thread instead.
   Remove the internal `syncify` helper and the `nest-asyncio2` dependency from `tilebox-grpc`.
 - Raise dependency minimums to remove obsolete compatibility workarounds: boto3 1.40.2, OpenTelemetry 1.43.0
-  (logging instrumentation 0.64b0), grpcio 1.84.0, and pyqwest 0.7.0.
+  (logging instrumentation 0.64b0), and pyqwest 0.7.0.
 - `tilebox-datasets`: Require NumPy 1.25, pandas 2.2.2, xarray 2024.6, and Shapely 2.0.6 or newer.
 - Require protobuf 6.31.0 for repeated-field descriptors and protobuf-py 0.2.0 for integer-to-float serialization.
 - `tilebox-workflows`: Make interactive notebook progress an optional `notebook` extra; without it, jobs display
@@ -29,6 +42,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- `tilebox-workflows`: Use S3 bucket regions from API metadata or discover them with `HeadBucket`.
+  `AmazonS3Cache` also accepts `region=` to skip discovery. Let botocore refresh temporary credentials.
+- `tilebox-workflows`: Preserve cloud cache permission, network, and backend errors instead of reporting cache misses.
+- Require `grpcio>=1.84.0` for the memory-exhaustion fixes in
+  [GHSA-hf3w-6hpw-qp67](https://github.com/grpc/grpc/security/advisories/GHSA-hf3w-6hpw-qp67).
+  macOS CLI authentication may still emit native fork diagnostics during successful reads.
+- `tilebox-workflows`: Fix GCS automation reads when the API returns a bucket name without a project prefix.
 - Explicitly trust system certificates for Connect HTTP/1 transports with pyqwest 0.7 and newer.
 
 ## [0.62.0] - 2026-09-16

tilebox-grpc/_tilebox/grpc/replay.py

@@ -24,6 +24,7 @@
 from grpc.aio import (
     AioRpcError,
     ClientCallDetails,  # import from aio, since grpc.ClientCallDetails is an empty base class
+    Metadata,
 )
 
 RequestType = TypeVar("RequestType")
@@ -163,7 +164,7 @@ def unary_unary_call(
 
         if recorded_status != StatusCode.OK.value[0]:  # the recorded call was an error, so raise it again
             code = _STATUS_CODES[recorded_status]
-            error = AioRpcError(code, None, None, recorded_response.decode())
+            error = AioRpcError(code, Metadata(), Metadata(), recorded_response.decode())
             raise error
 
         return response_deserializer(base64.b64decode(recorded_response))

tilebox-grpc/pyproject.toml

@@ -21,7 +21,7 @@ classifiers = [
 ]
 requires-python = ">=3.11"
 dependencies = [
-    # Rate-limits the logging noise from https://github.com/grpc/grpc/issues/42293.
+    # Includes fixes for https://github.com/grpc/grpc/security/advisories/GHSA-hf3w-6hpw-qp67.
     "grpcio>=1.84.0",
     # connectrpc 0.11 switches from Google's protobuf runtime to protobuf-py
     "connectrpc>=0.10.1,<0.11.0",

tilebox-grpc/tests/test_fork_logging.py

@@ -0,0 +1,73 @@
+"""Check native gRPC diagnostics when a CLI starts during active RPCs.
+
+Run on macOS to check RPCs and report the known fork-log issue:
+
+uv run --isolated --no-project --with grpcio==1.84.0 --with pytest pytest -c /dev/null -p no:cacheprovider -rx tilebox-grpc/tests/test_fork_logging.py
+
+RPC failures fail the test. Known fork diagnostics produce XFAIL after the RPC checks pass.
+Linux may use vfork and not reproduce the diagnostics. This check uses no cloud credentials.
+"""
+
+import os
+import subprocess
+import sys
+from textwrap import dedent
+
+import pytest
+
+
+# RPCs must survive CLI subprocess launches; known fork-log noise is reported separately.
+@pytest.mark.skipif(sys.platform == "win32", reason="Windows does not use POSIX fork handlers")
+def test_cli_spawn_during_rpcs() -> None:
+    result = subprocess.run(  # noqa: S603
+        [
+            sys.executable,
+            "-c",
+            dedent("""
+                import subprocess
+                import sys
+                from concurrent.futures import ThreadPoolExecutor
+                from threading import Event
+
+                import grpc
+
+                with ThreadPoolExecutor(max_workers=4) as executor:
+                    server = grpc.server(executor)
+                    server.add_generic_rpc_handlers((grpc.method_handlers_generic_handler(
+                        "probe", {"echo": grpc.unary_unary_rpc_method_handler(lambda request, context: request)}
+                    ),))
+                    port = server.add_insecure_port("127.0.0.1:0")
+                    server.start()
+                    stop = Event()
+                    try:
+                        with grpc.insecure_channel(f"127.0.0.1:{port}") as channel:
+                            echo = channel.unary_unary("/probe/echo")
+                            assert echo(b"before", timeout=5) == b"before"
+
+                            def poll():
+                                while not stop.is_set():
+                                    assert echo(b"during", timeout=5) == b"during"
+
+                            poller = executor.submit(poll)
+                            try:
+                                for _ in range(25):
+                                    # Azure CLI authentication supplies cwd, which prevents posix_spawn.
+                                    subprocess.run([sys.executable, "-c", "pass"], cwd=".", check=True, timeout=5)
+                            finally:
+                                stop.set()
+                                poller.result(timeout=10)
+                            assert echo(b"after", timeout=5) == b"after"
+                    finally:
+                        stop.set()
+                        server.stop(0).wait(10)
+            """),
+        ],
+        capture_output=True,
+        text=True,
+        timeout=60,
+        env={**os.environ, "GRPC_VERBOSITY": "INFO"},
+        check=False,
+    )
+    assert result.returncode == 0, result.stderr
+    if "FD from fork parent still in poll list" in result.stderr:
+        pytest.xfail("Known gRPC fork diagnostics: https://github.com/grpc/grpc/issues/42293")

tilebox-workflows/README.md

@@ -90,6 +90,19 @@ unless the cache implementation explicitly provides that guarantee.
 Concurrency in one runtime avoids repeated process initialization and allows overlapping I/O or native code that
 releases Python's GIL. CPU-bound Python code still needs multiple runtime processes for parallel execution.
 
+## Reading automation objects
+
+Storage-event tasks can read objects from Amazon S3, Google Cloud Storage, Azure Blob Storage, or the local filesystem:
+
+```python
+content = self.trigger.storage.read(self.trigger.location)
+```
+
+The method returns bytes. Cloud reads use the credentials configured for the runner.
+
+On macOS, GCS or Azure CLI authentication can emit [gRPC fork diagnostics](https://github.com/grpc/grpc/issues/42293)
+even when a read succeeds. These messages alone do not indicate a failed read; do not downgrade gRPC to hide them.
+
 ## Documentation
 
 Check out the [Tilebox Workflows documentation](https://docs.tilebox.com/workflows/introduction) for more information.

tilebox-workflows/pyproject.toml

@@ -23,7 +23,10 @@ requires-python = ">=3.11"
 dependencies = [
     "tilebox-datasets>=0.56.0",
     "tilebox-grpc>=0.28.0",
-    "google-cloud-storage>=2.10",
+    # Require the security fixes even with an older tilebox-grpc release.
+    "grpcio>=1.84.0",
+    "google-auth[requests]>=2.29",
+    "azure-identity>=1.23",
     "opentelemetry-api>=1.43.0",
     "opentelemetry-exporter-otlp-proto-http>=1.43.0",
     "opentelemetry-sdk>=1.43.0",