Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 17 additions & 12 deletions Node/taskqueues-backup-images/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,32 +56,37 @@ $ firebase deploy

## IAM Policy

The sample declares required IAM roles declaratively in `functions/index.js` (`requiresRole("roles/cloudtasks.enqueuer")` and `requiresRole("roles/run.invoker")`). On deployment, Firebase CLI automatically provisions these IAM policy bindings.
The sample declares the IAM roles it needs in `functions/index.js`: `requiresRole("roles/cloudtasks.enqueuer")` and `requiresRole("roles/run.invoker")`. When you deploy, the Firebase CLI creates a service account for this codebase (its email starts with `firebase-fn-`), grants it those roles, and runs every function in the codebase as that account. The CLI prints the account's email when it creates it; you can also find it under **IAM & Admin > Service accounts** in the Google Cloud console. The commands below call it `${FUNCTIONS_SA}`.

If configuring manually or troubleshooting permission errors, ensure that your project has the following IAM bindings:
Cloud Tasks needs three bindings in total. `requiresRole` covers the first and third; you add the second by hand.

* Identity used to enqueue tasks to Cloud Tasks needs `cloudtasks.tasks.create` IAM permission (`roles/cloudtasks.enqueuer`):
* In our sample, this is the [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts).
* The identity that enqueues tasks needs `cloudtasks.tasks.create` (`roles/cloudtasks.enqueuer`). `requiresRole` grants it. To grant it yourself:

```
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--member=serviceAccount:${FUNCTIONS_SA} \
--role=roles/cloudtasks.enqueuer
```

* Identity used to enqueue tasks to Cloud Task needs permission to use the service account associated with a task in Cloud Tasks.
* In our sample, this is the [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts).
* The identity that enqueues tasks needs permission to act as the service account that Cloud Tasks uses to call the task queue function (`roles/iam.serviceAccountUser` on that service account). In this sample both are `${FUNCTIONS_SA}`, so the account needs the role on itself. `requiresRole` doesn't cover this one:

```
Please follow Google Cloud IAM documentation to add App Engine default service account as user of App Engine default service account.
gcloud iam service-accounts add-iam-policy-binding ${FUNCTIONS_SA} \
--member=serviceAccount:${FUNCTIONS_SA} \
--role=roles/iam.serviceAccountUser
```

* Identity used to trigger the Task Queue function needs `run.routes.invoke` permission.
* In our sample, this is the [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts).
* The identity that Cloud Tasks uses to call the task queue function needs `run.routes.invoke` (`roles/run.invoker`). `requiresRole` grants it. To grant it yourself:

```
gcloud functions add-iam-policy-binding backupapod \
--region=us-central1 \
--member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--member=serviceAccount:${FUNCTIONS_SA} \
--role=roles/run.invoker
```
```

If you remove the `requiresRole` calls, the functions run as the [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts) (`${PROJECT_NUMBER}-compute@developer.gserviceaccount.com`) instead. Use that email in the commands above and run all three.

If tasks fail with `401` or `403` (`PERMISSION_DENIED`) in the Cloud Tasks logs, check that the `roles/run.invoker` binding above exists. Cloud Tasks mints an OIDC token for the function's URL, and Cloud Run rejects the call if the token's service account can't invoke the function. If `enqueue` itself fails with `PERMISSION_DENIED`, the missing `roles/iam.serviceAccountUser` binding is the usual cause.

The sample deploys to `us-central1`, which is the region `getFunctions().taskQueue("backupapod")` assumes when you pass a bare function name. If you deploy the function to another region, pass the full resource name instead: `taskQueue("locations/europe-west1/functions/backupapod")`.
94 changes: 93 additions & 1 deletion Python/taskqueues-backup-images/README.md
Original file line number Diff line number Diff line change
@@ -1 +1,93 @@
# taskqueues-backup-images
# Back up images using a Task Queue Function (Python)

This sample shows how to set up a task queue function with the Firebase SDK for Cloud Functions in Python. It's the Python version of the [Node.js sample](../../Node/taskqueues-backup-images).

## Introduction

Task queue functions let you manage the execution, dispatch, and delivery of a large number of distributed tasks.

The sample uses them to back up every image from NASA's [Astronomy Picture of the Day](https://apod.nasa.gov/apod/astropix.html).

Task queue functions are powered by [Google Cloud Tasks](https://cloud.google.com/tasks). Learn more about [task queue functions](https://firebase.google.com/docs/functions/task-functions?gen=2nd).

## Functions

See [functions/main.py](functions/main.py) for the code. It defines two functions:

### 1. `backupapod`

A task queue function that backs up the Astronomy Picture of the Day ("apod") for a given date. Cloud Tasks calls this function once for every task enqueued on its queue.

Task queue functions let you control the rate limits and retry behavior of the queue. The sample uses these settings:

1. `RetryConfig(max_attempts=5)`: Each task is retried up to 5 times. This helps with transient errors such as network errors or a temporary outage of the NASA API.
2. `RetryConfig(min_backoff_seconds=60)`: Retries are spaced at least 60 seconds apart, so the 5 attempts aren't used up in a burst.
3. `RateLimits(max_concurrent_dispatches=10)`: At most 10 tasks run at a time. This keeps a steady stream of requests to the function and limits the number of active instances and cold starts.

You can configure this function with the following [parameter](https://firebase.google.com/docs/functions/config-env?gen=2nd#params):

* `BACKUP_BUCKET`: Name of the bucket that stores the "apod" images.

### 2. `enqueuebackuptasks`

An HTTP function that enqueues tasks on the queue. It uses the Firebase Admin SDK to create one task for each day to back up.

Because of the way Cloud Tasks authenticates requests to the task queue function, each task needs the function's Cloud Run URL. The `get_function_url` helper looks that URL up with the Cloud Functions API.

You can configure this function with the following parameters:

* `BACKUP_COUNT`: Number of days to back up, starting from 1995-06-17 (the first day of publication). Defaults to 100.
* `HOURLY_BATCH_SIZE`: Number of tasks to enqueue each hour. The NASA API allows 1000 requests per hour. Defaults to 600.

## Set up and deploy

### NASA Open API key

The sample fetches images with the [NASA Open APIs](https://api.nasa.gov/). Register for an API key, then provide it as the `NASA_API_KEY` parameter when the CLI prompts you during deployment, or add it to `functions/.env`:

```
NASA_API_KEY=your-api-key
```

### Deploy

```bash
firebase deploy
```

## IAM policy

The Node.js sample declares the roles it needs in code with `requiresRole`, and the Firebase CLI grants them at deploy time. The Python SDK has no equivalent, so you grant the roles yourself. The sample runs as the [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts), which needs these bindings:

* To enqueue tasks, `roles/cloudtasks.enqueuer` on the project:

```
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--role=roles/cloudtasks.enqueuer
```

* To act as the service account that Cloud Tasks uses to call the function, `roles/iam.serviceAccountUser` on that account. In this sample both accounts are the same, so the account needs the role on itself:

```
gcloud iam service-accounts add-iam-policy-binding ${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--role=roles/iam.serviceAccountUser
```

* To invoke the task queue function, `roles/run.invoker` on the function:

```
gcloud functions add-iam-policy-binding backupapod \
--region=us-central1 \
--member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--role=roles/run.invoker
Comment on lines +81 to +84

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For 2nd gen Cloud Functions, the underlying service is a Cloud Run service. The recommended and standard way to grant the invoker role is using gcloud run services add-iam-policy-binding instead of gcloud functions add-iam-policy-binding.

Suggested change
gcloud functions add-iam-policy-binding backupapod \
--region=us-east4 \
--member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--role=roles/run.invoker
gcloud run services add-iam-policy-binding backupapod \
--region=us-east4 \
--member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--role=roles/run.invoker

```

If tasks fail with `401` or `403` (`PERMISSION_DENIED`) in the Cloud Tasks logs, check that the `roles/run.invoker` binding exists and that `get_function_url` returns the function's Cloud Run URL for the region you deployed to. Cloud Tasks mints an OIDC token for that URL, and Cloud Run rejects the token if the URL doesn't match. Also check your SDK version: `firebase-functions` for Python older than 0.4.1 rejected Cloud Tasks tokens with an `aud` error ([firebase-functions-python#157](https://github.com/firebase/firebase-functions-python/issues/157)).

The sample deploys to `us-central1`, which is the region `functions.task_queue("backupapod")` assumes when you pass a bare function name. If you deploy the function to another region, pass the full resource name instead: `task_queue("locations/europe-west1/functions/backupapod")`, and pass the same region to `get_function_url`.

## Test

Open the URL of the `enqueuebackuptasks` function in a browser or with `curl`. The response reports how many tasks were enqueued. As Cloud Tasks dispatches them, images appear under `apod/` in your backup bucket and each run shows up in the `backupapod` function logs.
Loading