Repository navigation
docs(taskqueues): spell out the IAM bindings and add the Python README #1341
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
97021ab
docs(taskqueues): spell out the IAM bindings and add the Python README
jhuleatt 4ceaaf6
docs(taskqueues): use us-central1 in the Python README
jhuleatt e002520
docs(taskqueues): fix the troubleshooting notes
jhuleatt 031e901
docs(taskqueues): name the service account that requiresRole provisions
jhuleatt File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,93 @@ | ||
| # taskqueues-backup-images | ||
| # Back up images using a Task Queue Function (Python) | ||
|
|
||
| This sample shows how to set up a task queue function with the Firebase SDK for Cloud Functions in Python. It's the Python version of the [Node.js sample](../../Node/taskqueues-backup-images). | ||
|
|
||
| ## Introduction | ||
|
|
||
| Task queue functions let you manage the execution, dispatch, and delivery of a large number of distributed tasks. | ||
|
|
||
| The sample uses them to back up every image from NASA's [Astronomy Picture of the Day](https://apod.nasa.gov/apod/astropix.html). | ||
|
|
||
| Task queue functions are powered by [Google Cloud Tasks](https://cloud.google.com/tasks). Learn more about [task queue functions](https://firebase.google.com/docs/functions/task-functions?gen=2nd). | ||
|
|
||
| ## Functions | ||
|
|
||
| See [functions/main.py](functions/main.py) for the code. It defines two functions: | ||
|
|
||
| ### 1. `backupapod` | ||
|
|
||
| A task queue function that backs up the Astronomy Picture of the Day ("apod") for a given date. Cloud Tasks calls this function once for every task enqueued on its queue. | ||
|
|
||
| Task queue functions let you control the rate limits and retry behavior of the queue. The sample uses these settings: | ||
|
|
||
| 1. `RetryConfig(max_attempts=5)`: Each task is retried up to 5 times. This helps with transient errors such as network errors or a temporary outage of the NASA API. | ||
| 2. `RetryConfig(min_backoff_seconds=60)`: Retries are spaced at least 60 seconds apart, so the 5 attempts aren't used up in a burst. | ||
| 3. `RateLimits(max_concurrent_dispatches=10)`: At most 10 tasks run at a time. This keeps a steady stream of requests to the function and limits the number of active instances and cold starts. | ||
|
|
||
| You can configure this function with the following [parameter](https://firebase.google.com/docs/functions/config-env?gen=2nd#params): | ||
|
|
||
| * `BACKUP_BUCKET`: Name of the bucket that stores the "apod" images. | ||
|
|
||
| ### 2. `enqueuebackuptasks` | ||
|
|
||
| An HTTP function that enqueues tasks on the queue. It uses the Firebase Admin SDK to create one task for each day to back up. | ||
|
|
||
| Because of the way Cloud Tasks authenticates requests to the task queue function, each task needs the function's Cloud Run URL. The `get_function_url` helper looks that URL up with the Cloud Functions API. | ||
|
|
||
| You can configure this function with the following parameters: | ||
|
|
||
| * `BACKUP_COUNT`: Number of days to back up, starting from 1995-06-17 (the first day of publication). Defaults to 100. | ||
| * `HOURLY_BATCH_SIZE`: Number of tasks to enqueue each hour. The NASA API allows 1000 requests per hour. Defaults to 600. | ||
|
|
||
| ## Set up and deploy | ||
|
|
||
| ### NASA Open API key | ||
|
|
||
| The sample fetches images with the [NASA Open APIs](https://api.nasa.gov/). Register for an API key, then provide it as the `NASA_API_KEY` parameter when the CLI prompts you during deployment, or add it to `functions/.env`: | ||
|
|
||
| ``` | ||
| NASA_API_KEY=your-api-key | ||
| ``` | ||
|
|
||
| ### Deploy | ||
|
|
||
| ```bash | ||
| firebase deploy | ||
| ``` | ||
|
|
||
| ## IAM policy | ||
|
|
||
| The Node.js sample declares the roles it needs in code with `requiresRole`, and the Firebase CLI grants them at deploy time. The Python SDK has no equivalent, so you grant the roles yourself. The sample runs as the [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts), which needs these bindings: | ||
|
|
||
| * To enqueue tasks, `roles/cloudtasks.enqueuer` on the project: | ||
|
|
||
| ``` | ||
| gcloud projects add-iam-policy-binding $PROJECT_ID \ | ||
| --member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \ | ||
| --role=roles/cloudtasks.enqueuer | ||
| ``` | ||
|
|
||
| * To act as the service account that Cloud Tasks uses to call the function, `roles/iam.serviceAccountUser` on that account. In this sample both accounts are the same, so the account needs the role on itself: | ||
|
|
||
| ``` | ||
| gcloud iam service-accounts add-iam-policy-binding ${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \ | ||
| --member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \ | ||
| --role=roles/iam.serviceAccountUser | ||
| ``` | ||
|
|
||
| * To invoke the task queue function, `roles/run.invoker` on the function: | ||
|
|
||
| ``` | ||
| gcloud functions add-iam-policy-binding backupapod \ | ||
| --region=us-central1 \ | ||
| --member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \ | ||
| --role=roles/run.invoker | ||
| ``` | ||
|
|
||
| If tasks fail with `401` or `403` (`PERMISSION_DENIED`) in the Cloud Tasks logs, check that the `roles/run.invoker` binding exists and that `get_function_url` returns the function's Cloud Run URL for the region you deployed to. Cloud Tasks mints an OIDC token for that URL, and Cloud Run rejects the token if the URL doesn't match. Also check your SDK version: `firebase-functions` for Python older than 0.4.1 rejected Cloud Tasks tokens with an `aud` error ([firebase-functions-python#157](https://github.com/firebase/firebase-functions-python/issues/157)). | ||
|
|
||
| The sample deploys to `us-central1`, which is the region `functions.task_queue("backupapod")` assumes when you pass a bare function name. If you deploy the function to another region, pass the full resource name instead: `task_queue("locations/europe-west1/functions/backupapod")`, and pass the same region to `get_function_url`. | ||
|
|
||
| ## Test | ||
|
|
||
| Open the URL of the `enqueuebackuptasks` function in a browser or with `curl`. The response reports how many tasks were enqueued. As Cloud Tasks dispatches them, images appear under `apod/` in your backup bucket and each run shows up in the `backupapod` function logs. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For 2nd gen Cloud Functions, the underlying service is a Cloud Run service. The recommended and standard way to grant the invoker role is using
gcloud run services add-iam-policy-bindinginstead ofgcloud functions add-iam-policy-binding.