Skip to content
Prostochell-544Public

About

Turn a list of literature search queries into DOIs grouped by query, using Google Scholar APIs, Semantic Scholar and OpenAlex.

Topics

Resources

Stars

6 stars

Watchers

0 watching

Forks

Repository files navigation

Alcedo logo: a kingfisher in flight
Alcedo: open metadata extractor

Alcedo

Alcedo is the Latin name of the kingfisher, which dives and picks out exactly what it wants.

Alcedo turns a list of literature search queries into DOIs grouped by query. Put your boolean queries in a file and run one command. The results are a text file with one block per query. Then narrow them by year, language and journal quartile, and export them to Zotero, Mendeley, EndNote or ASReview.

Status: alpha. Options may change. See the roadmap and the changelog. Documentation in other languages: Russian, Ukrainian.

What it does

  • Takes up to 5 result pages (50 results) per query, or all of them if there are fewer.
  • Searches Google Scholar through a search API with your own key. Without a working API it uses Semantic Scholar and OpenAlex, plus direct Google Scholar search if you turned that on. See Which source to use.
  • A source that fails is left out, and the block says so.
  • Scholar gives links rather than DOIs, so Alcedo looks for the DOI in the URL, Europe PMC and publisher meta tags, then by title in Crossref and OpenAlex.
  • Saves after every query. --resume continues after an interruption and collects failed queries again.
  • Writes a second file listing each DOI once, with the queries that found it. Papers found by the most queries come first.
  • With --verify-dois, checks every DOI in the doi.org registry and marks the ones that do not exist.
  • alcedo filter narrows the results by year, language, record type, quartile and words, and explains every record it leaves out. It also removes repeats of the same work, links legal open copies, and writes a selection log with the numbers for a PRISMA 2020 flow diagram. See Filter the results.
  • alcedo export writes RIS and BibTeX for reference managers. See Export to a reference manager.

Install

Requires Python 3.10 or newer.

pip install alcedo-lit

For direct Google Scholar search, which most people will not need, install the extra as well: pip install "alcedo-lit[scholar]".

Then choose how to search. This explains the options and saves your key, which is not shown as you type:

alcedo setup

Quick start

The repository includes a sample: queries.txt has two queries about the kingfisher, and alcedo_results.txt and alcedo_results_unique.csv are their results. There is another set of queries in examples/queries.txt.

For your own search, create a queries.txt in any folder, with ; after each query:

biomimetic nacre ("brick-and-mortar" OR "layered structure") composite fracture toughness;
gecko-inspired dry adhesive (micropillar OR "hierarchical microstructure") adhesion;

From that folder, check the queries for common mistakes and then run them:

alcedo check
alcedo

The results go to alcedo_results.txt in the current folder:

=== Query: gecko-inspired dry adhesive (micropillar OR "hierarchical microstructure") adhesion ===
Source: OpenAlex
https://doi.org/10.1039/c3sm52921e
https://doi.org/10.1098/rsif.2018.0681

The Source: line appears when the results did not all come from a search API. Records with no DOI are listed as DOI not found: <title>, and a query that failed has a Query error line. If DOI lookups failed along the way, the final message says how many, because those records may still have a DOI.

Alcedo will not replace an existing results file: use --resume to continue it or --overwrite to start again.

alcedo_results_unique.csv lists each DOI once, with the number of queries that found it and the queries themselves. It is built from the results file and uses no extra searches. Cells are separated by ;, which Excel in European locales expects.

To check DOIs against the doi.org registry, add --verify-dois. This also works on results you already have: alcedo --resume --verify-dois. A DOI the registry does not know becomes DOI not registered: <doi>, and one that could not be checked stays in the results and is counted in the final message. The check uses no OpenAlex budget.

To check the wording of a query quickly, look at its first 10 results:

alcedo --pages 1 -i test.txt -o test_results.txt --overwrite

Read how to write queries before a large run. Some mistakes fail silently: a lowercase or, for example, is treated as AND. alcedo check finds the common ones.

Filter the results

alcedo filter narrows a results file by year, language, record type, quartile and words. It uses no search quota, so you can change the conditions as often as you like. It does look up each paper's data, in OpenAlex and then Crossref.

alcedo filter --year-from 2015 --language en --type article,review --quartile Q1,Q2 --sjr scimagojr.csv --exclude-word dental

--type takes the record types to keep: article, review, preprint, book-chapter, dissertation, conference-paper and others; alcedo filter --help lists them all. With a quartile condition, it also clears the unknown list of dissertations, preprints and books, which have no journal quartile.

It writes three lists, with the title, year, journal, quartile and authors of each record:

File What is in it
alcedo_filtered.csv Records that meet all the conditions
alcedo_excluded.csv Records that fail a condition, each with the reason
alcedo_unknown.csv Records whose data is missing, each with the reason. Check these by hand, because they may well fit
alcedo_duplicates.csv Repeats of a work that is already in the lists under another DOI
alcedo_no_access.csv Kept papers with no legal open copy, for interlibrary loan or to ask the authors
alcedo_selection_log.md Counts at every step, with the numbers for a PRISMA 2020 flow diagram

A repeat has exactly the same title as another record and either the same year or is a preprint, because a preprint and the article it became often differ by a year. The published version stays in the lists, with the queries that found both. Titles that are close but not the same, such as one starting with "Reprint of:", are only marked in the similar_to column, because they can be different works. Titles under four words, such as "Editorial", are never joined.

The open_access and open_access_url columns show whether a legal free copy exists and where, as OpenAlex reports it: gold, diamond, hybrid and bronze copies are at the publisher, green ones in a repository. A green copy is often the accepted manuscript, which can differ from the published paper. Records that only Crossref knows have no open access data. Alcedo links to these copies and does not download papers.

The selection log records the conditions, the date and the Alcedo version, and counts the records at every step: found by each source, duplicates removed, excluded with the reasons, left to check by hand, and kept. Its last section gives the numbers for the identification part of a PRISMA 2020 flow diagram, where Alcedo counts as an automation tool. Screening titles, abstracts and full texts is done by people, so record those numbers yourself.

Nothing is dropped silently. Records without a DOI always go to the unknown list, unless an excluded word appears in the title. Papers marked as retracted are excluded unless you add --keep-retracted, though only OpenAlex reports this. Language comes from OpenAlex, which detects it automatically and can be wrong. Excluded words are searched for in the title and abstract, and many records have no abstract.

The quartile comes from a file you download yourself: open the Scimago journal rank page and choose Download data. Alcedo does not include this file. A journal can have different quartiles in different fields, and Alcedo uses the best one, so check the field yourself if it matters.

Export to a reference manager

alcedo export writes the records as RIS and BibTeX, with authors, journal, year, pages and abstracts. Zotero, Mendeley and EndNote read both, and ASReview reads RIS for screening.

alcedo export

It takes alcedo_filtered.csv if you ran alcedo filter, and the whole results file otherwise, and writes alcedo_filtered.ris and alcedo_filtered.bib next to it. Use -i for another file and --format ris or --format bibtex for one format. Each record notes the queries that found it. Records without a DOI keep only their title, and about one in ten records from OpenAlex has no abstract.

Which source to use

With a working API key, a query uses only the API. Otherwise it uses the free sources together, in this order: Semantic Scholar, OpenAlex, and direct search if you turned it on. Each gives up to 50 results and repeats are removed, so a block can hold more than 50 records. A source that fails is left out, and the Source: line says so. If every source fails, the block has a Query error line and --resume collects it again.

The free options do not give the same results as an API. The figures below come from small tests with relevance judged by an AI assistant, not by a domain expert. Treat them as a rough guide. Results vary a lot by query.

Source What you need Cost at the time of writing How it did
Google Scholar via SearchApi Your key, no phone number 100 free requests to try it, then from $40 a month Best first results: 13 of the first 20 were exactly on topic. Same first results as a browser in our check
Google Scholar via Serper Your key, no card 2,500 free queries, which is about 500 queries of 50 results, then one-off packs from $50 for 50,000, valid for 6 months Same first 10 results as SearchApi on both test queries, and 13 of the first 20 exactly on topic. Later results differed
Google Scholar via SerpApi Your key and a phone number 250 free searches a month, paid plans from $25 a month Not measured; Alcedo has not yet been run against the live service
Semantic Scholar Nothing; a free key gives you your own limit Free 9 of the first 10 exactly on topic on one query, no results on a stricter one. Finds few but precise papers, most cited first. Not measured inside Alcedo
OpenAlex Nothing; a free key raises the daily budget Free 5 of the first 20 exactly on topic. Its share of on-topic results ranged from 34% against 73% for the API to about equal. It also found papers Scholar missed
Google Scholar directly pip install "alcedo-lit[scholar]" and your agreement Free Changes a lot between runs and networks, see below

A query with 50 results uses 3 requests on SearchApi and 5 on Serper. You create the key and accept the provider's terms, and the provider sees your queries. If you have neither a key nor a budget for one, the free catalogues still work, and direct search can add to them on a network Scholar accepts.

In a run where Scholar answered in full, the free sources together found 38 of the 45 exactly on-topic papers known from all our runs, against 29 for SearchApi. Their first results were weaker, with 9 of the first 20 exactly on topic against 13, and they took 15 minutes for 2 queries against 3.5, mostly on direct search and DOI lookups. A key gives you the best first results quickly, but going without one does not have to mean missing papers.

Publications that use results from Semantic Scholar should cite the Semantic Scholar Open Data Platform, as its API license asks.

Direct search

It is free, and Scholar may simply refuse it. Scholar often refuses VPNs and shared networks: in our test, one VPN provider was refused even in a browser and another was accepted. If it refuses you, try without the VPN, and check the same search in a browser.

An answer from Scholar does not mean the list is complete. In some of our tests, the script got 6 to 27 results for queries that give 50 through an API, mostly recent papers, with key works missing, and repeating one query gave different lists. In a later run it got all 50, nearly all of them also in the API's list. Parsing does not explain the difference, we do not know the cause, and you cannot tell in advance which you will get. So Alcedo puts direct results after the free catalogues, where they can only add to the list, and marks those blocks in the results file. Do not rely on direct search for a literature review. If you add a key later, alcedo --redo-free collects those blocks again.

Alcedo asks once before using direct search, and remembers your answer. alcedo setup changes it, and --direct-scholar or --no-direct-scholar overrides it for one run.

Keys and everyday commands

alcedo setup stores keys as plain text in your user settings (%APPDATA%\alcedo\config.json on Windows, ~/.config/alcedo/config.json elsewhere). Run it again and type - to remove a key. The SEARCHAPI_KEY, SERPER_API_KEY, SERPAPI_KEY, OPENALEX_API_KEY and SEMANTIC_SCHOLAR_API_KEY environment variables take priority over that file, and command line options take priority over both.

Alcedo remembers DOI lookups in a local cache, so a repeated run is faster. The cache holds the titles and links of records you searched for, and it stays on your computer: %LOCALAPPDATA%�lcedo on Windows, ~/.cache/alcedo elsewhere.

alcedo --resume          # continue after an interruption
alcedo --redo-free   # collect blocks from the free sources again, for example after adding a key
alcedo --source openalex # OpenAlex only
alcedo --no-semantic-scholar  # leave Semantic Scholar out of the free sources
alcedo --no-cache         # do not reuse or save DOI lookups from earlier runs
alcedo --help            # all options

Responsible use

  • Direct Google Scholar search is off until you agree to it. Scholar's robots.txt disallows automated search, and you are responsible for following Google's terms. Alcedo waits between queries and stops asking when Scholar refuses. The scholarly library it relies on sends a random browser User-Agent and may try to open a browser for a captcha, and it has no setting to turn that off.
  • Apart from that route, requests identify themselves as alcedo/<version>. With --mailto, your address goes to Crossref, OpenAlex, Semantic Scholar and Europe PMC, never to publisher sites.
  • Alcedo collects metadata. It does not download papers and will not integrate shadow libraries.

Development

pip install -e ".[dev]"
python -m pytest
ruff check .

To use the search from another program, listen to its events instead of reading the terminal:

from alcedo_lit import events
from alcedo_lit.cli import parse_args
from alcedo_lit.pipeline import run

with events.listening(print):  # each event has a kind, a text and data
    run(parse_args(["-i", "queries.txt"]))

The tests use no network. Git tracks the sample in the project root, so do not run your own queries there: queries.txt, alcedo_results.txt and alcedo_results_unique.csv would show up as changes. Use private/, which Git ignores, so your topic stays out of the repository.

License

The code is under the MIT license. The logo is not: see the third-party notices.

Special thanks

To kesha_xdd, who drew the Alcedo logo.

About

Turn a list of literature search queries into DOIs grouped by query, using Google Scholar APIs, Semantic Scholar and OpenAlex.

Topics

Resources

Stars

6 stars

Watchers

0 watching

Forks

Releases

Contributors

Languages