Skip to content

perf(updater): reuse the history chart for the dividend and benchmark metrics - #111

Merged
daggerok merged 1 commit into
mainfrom
perf/updater-speed
Nov 21, 2026
Merged

daggerok merged 1 commit into
mainfrom
perf/updater-speed

Conversation

@daggerok

Copy link
Copy Markdown
Owner

What

  • The dividends and benchmark blocks now read the full daily Yahoo chart that processFund already fetched (kept in memory per fund, taken once) instead of a second request per updated fund. With a HISTORY_RANGE window (not a full chart) the blocks still make their own request, as before
  • CONCURRENCY default 2 -> 4 (README table updated), so the r.jina.ai product page gate (3.2 s between starts, unchanged) stays busy while other workers wait on SEC and Yahoo
  • Tests: the Yahoo mock returns dividend events on every full daily chart (as Yahoo does); the request-count test now checks one chart per fund, two under HISTORY_RANGE=1y

Why

Measured full run before (fetch timing preload, VERBOSE, timestamps), 601 s for 96 funds:

  • worker pool 1 s -> 503 s: 110 r.jina.ai product page requests (direct wisdomtree.com gives HTML, not markdown, so the run latches onto the proxy after the first fund; avg 4.3 s, 14 retries after 551-byte error pages), 189 SEC requests (avg 1.4 s), 98 Yahoo history charts
  • after the pool, 98 s: 96 more Yahoo daily charts for the dividends block, identical to the history charts already fetched

Result

Run Time Yahoo requests Updated / unchanged / failed
before (main data) 10m01s 194 39 / 57 / 0
after (main data) 5m30s 98 38 / 58 / 0
after, rerun on the before output 5m11s 98 1 / 95 / 0, every index metric (dividend and benchmark keys included) identical

The one count difference is DGS, whose product page alternates between runs (it also flipped on the rerun); no 403 or 429 from SEC, Yahoo or the proxy, one SEC timeout that the retry recovered. Data was reset afterwards, nothing under api/ is committed.

Checks

  • bun test (66 pass)
  • bun build --target=bun scripts/update-data.ts --outfile=/dev/null
  • git diff --check
  • TC_MODE=esnext .claude/tools/etfs/tc/check.sh . (no output)
  • time ./scripts/update-data.ts full runs as above

… metrics

The dividends block made a second Yahoo daily chart request per updated fund
after the worker pool (about 100 s at REQUEST_SLEEP=2), although processFund
already fetched the same full daily chart (period1=0, events=div|split). That
answer is now kept in memory and handed to the dividends and benchmark blocks;
a HISTORY_RANGE window is not a full chart, so the blocks still ask for their
own then. CONCURRENCY default 2 -> 4 keeps the r.jina.ai product page gate
(3.2 s between starts, unchanged) busy instead of idle between SEC and Yahoo
requests.
@daggerok
daggerok merged commit 6fc7bb8 into main Nov 21, 2026
1 check passed
@daggerok
daggerok deleted the perf/updater-speed branch November 21, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant