Skip to content

feat(arxiv): list new papers from Zenodo and journals too - #15

Merged
SproutSeeds merged 1 commit into
mainfrom
feat/recent-papers-more-sources
Sep 30, 2026
Merged

SproutSeeds merged 1 commit into
mainfrom
feat/recent-papers-more-sources

Conversation

@SproutSeeds

Copy link
Copy Markdown
Owner

Much new physics never reaches arXiv: many researchers post on Zenodo, including authors without arXiv access, and journal articles and other repositories reach OpenAlex. recent_papers now also searches both when given a query. A new sources argument selects arxiv, zenodo and openalex; a category-only listing stays on arXiv.

  • Zenodo: publications whose first version was uploaded inside the window. Every version is searched and only first versions are kept, so a new version of an older upload is not listed as new.
  • OpenAlex: works published inside the window that are not on arXiv (arXiv is searched directly) and, when Zenodo is searched, not on Zenodo. Datasets and software are left out.
  • Both services treat bare words as OR, so a plain query searches its phrase and then all words joined by AND.
  • Every paper passes the same window and topic checks as arXiv papers and carries its source. Sources take turns filling the list, so a busy source cannot crowd out the others. A title found in several sources is listed once, arXiv first.
  • A failing source becomes a note; the call fails only when every source fails. arXiv field syntax and the arxiv-only backend search arXiv only. Zenodo requests are paced under its guest limit of 30 a minute.

Checked live, 2026-09-30

Query Result
"reflection positivity", 30 days The arXiv paper listed alongside recent Zenodo uploads; Figshare copies of one work dropped as duplicates
"reflection positivity neural network", 60 days Nothing from any source
"neural network field theory", 45 days The three arXiv papers plus a Physical Review D article found through OpenAlex

Tests

New tests cover source merging and labels, first-version Zenodo queries, excluded arXiv and Zenodo works in OpenAlex, per-source failures, arXiv-only cases, turn-taking and duplicate titles. Full suite: 13048 passed, with the three known environment failures.

Limits

  • Zenodo uploads are not peer reviewed, and broad physics queries return many recent uploads; each is labeled.
  • OpenAlex can list a journal version of an older preprint as new when it has not linked the arXiv copy.
  • Sites without a public search API are not covered, for example ResearchGate and personal pages.

Much new physics never reaches arXiv: many researchers post on Zenodo,
including authors without arXiv access, and journal articles and other
repositories reach OpenAlex. recent_papers now also searches both when
given a query (a `sources` argument selects arxiv, zenodo and
openalex; a category listing stays on arXiv).

* Zenodo: publications whose first version was uploaded inside the
  window (every version is searched and only first versions are kept, so
  a new version of an older upload is not listed as new), newest first.
* OpenAlex: works published inside the window that are not on arXiv
  (arXiv is searched directly) and, when Zenodo is searched, not on
  Zenodo; datasets and software are left out.
* Both treat bare words as OR, so a plain query searches its phrase and
  then all words joined by AND.
* Every paper passes the same window and topic checks as arXiv papers
  and carries its source. Sources take turns filling the list, so a
  busy source cannot crowd out the others, and a title found in several
  sources is listed once, arXiv first. A failing source becomes a note;
  the call fails only when every source fails. arXiv field syntax and
  the arxiv-only backend search arXiv only. Zenodo requests are paced
  under its guest limit of 30 a minute.

Checked live on 2026-09-30: "reflection positivity" over 30 days lists
the arXiv paper alongside Zenodo uploads, with Figshare copies of the
same work dropped as duplicates; "reflection positivity neural network"
over 60 days returns nothing from any source; "neural network field
theory" adds a Physical Review D article. Full suite 13048 passed with
the three known environment failures.
@SproutSeeds
SproutSeeds merged commit b9c1eec into main Sep 30, 2026
24 checks passed
@SproutSeeds
SproutSeeds deleted the feat/recent-papers-more-sources branch September 30, 2026 04:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant