Automated GitHub intelligence / Python + pandas + SQLite

Build lessnoise.See more signal.

GitHub ETL Engine turns newly created repositories into ranked, measurable signals, then pushes the same run into SQLite, CSV, Excel and daily email.

One pipeline / many outputs
GitHub ETL Engine / Portfolio Showcase30-day window · top 500 · daily run
01 / Why it exists

GitHub is full of signals.

Finding useful ones by hand means repeatedly searching, sorting, comparing and recording. This project turns that repeated work into one repeatable daily run.

01 / DISCOVER

Find fresh repos.

GitHub Search API pulls repositories created within the configured window, ordered by stars.

02 / UNDERSTAND

Measure momentum.

Raw fields become velocity, age, fork ratio and viral tiers so popularity has context.

03 / DELIVER

Make it readable.

One load becomes analysis tables, an Excel dashboard and a daily report people can consume.

02 / The run

One repo.
Five stops.

Follow one repository through the system. Each stage has one job. Output from one stage becomes input for the next.

01 / EXTRACT
↓

GitHub Search API

Pull up to 500 newly created repositories from the configured 30-day window.

02 / TRANSFORM
×

pandas

Clean descriptions, dedupe URLs, calculate age, velocity, fork ratio and tier.

03 / LOAD
▣

SQLite

Upsert current rows while retaining older rows for historical tracking.

04 / PUBLISH
↗

CSV + Excel

Generate analytical CSVs and patch workbook data without destroying dashboard design.

05 / REPORT
@

Email

Package daily KPIs, top repos, velocity and language mix into an HTML report.

475repos in snapshot
751Ktotal stars
79.41top stars / hour
24language groups
03 / Transformation

Popularity gets context.

Stars alone can reward older repositories. The engine adds age and velocity, letting fast-moving projects surface beside established ones.

Signal =
stars ÷ age

stars_per_hour = stars / age_hoursAge is calculated from repository creation time. Values are rounded before storage. Lower age + strong stars → stronger momentum signal.

Then it ranks the signal.

Every repository receives a viral_level based on star thresholds, from Normal through Trending, Rising Star, Viral, Super Viral, Iconic, Legendary and Hall of Fame.

description filterURL dedupeage_daysfork_ratioviral_levelrank
04 / What comes out

Raw API response becomes a decision surface.

The same run feeds storage, analysis tables, a presentation-ready dashboard and a report email. No manual copy-paste between stages.

GitHub Trending Intelligence
Latest load / 21 Nov 2026
475repos tracked
751Ktotal stars
79.41top velocity / hr
Repository momentumstars / hour
Language mix
0languages
Edit the DATA object in the script at the bottom to match your workbook.
Excel dashboard

Hand-designed workbook stays intact while data sheets and table ranges are refreshed.

Download workbook ↗
SQLite + CSV

SQLite keeps historical rows. CSVs expose repo, language, category, velocity, topic and activity datasets.

Open repo dataset ↗
05 / Daily automation

Run once.
Wake up
informed.

Automation is designed around one repeatable command. run_all.py executes extraction first, then Excel sync. A scheduler can trigger it daily; the report step packages the result for delivery.

01GitHub Search APIextract
02pipeline.pytransform + load
03update_excel.pypublish
04send_report.pyemail
06 / Under the hood

Small code.
Clear jobs.

Project stays modular: extraction + transformation + storage in pipeline.py; workbook sync in update_excel.py; orchestration in run_all.py; email reporting in send_report.py.

# pipeline.pyurl = "https://api.github.com/search/repositories"params = {    "q": f"created:>{since}",    "sort": "stars",    "order": "desc",} response = requests.get(url, params=params, ...) # transformdf["stars_per_hour"] = (    df["stars"] / age_h.clip(lower=1)).round(2) # loaddf.to_sql("repos_temp", conn, if_exists="replace", index=False)