Find fresh repos.
GitHub Search API pulls repositories created within the configured window, ordered by stars.
GitHub ETL Engine turns newly created repositories into ranked, measurable signals, then pushes the same run into SQLite, CSV, Excel and daily email.
Finding useful ones by hand means repeatedly searching, sorting, comparing and recording. This project turns that repeated work into one repeatable daily run.
GitHub Search API pulls repositories created within the configured window, ordered by stars.
Raw fields become velocity, age, fork ratio and viral tiers so popularity has context.
One load becomes analysis tables, an Excel dashboard and a daily report people can consume.
Follow one repository through the system. Each stage has one job. Output from one stage becomes input for the next.
Pull up to 500 newly created repositories from the configured 30-day window.
Clean descriptions, dedupe URLs, calculate age, velocity, fork ratio and tier.
Upsert current rows while retaining older rows for historical tracking.
Generate analytical CSVs and patch workbook data without destroying dashboard design.
Package daily KPIs, top repos, velocity and language mix into an HTML report.
Stars alone can reward older repositories. The engine adds age and velocity, letting fast-moving projects surface beside established ones.
Signal =
stars ÷ age
stars_per_hour = stars / age_hoursAge is calculated from repository creation time. Values are rounded before storage. Lower age + strong stars → stronger momentum signal.Every repository receives a viral_level based on star thresholds, from Normal through Trending, Rising Star, Viral, Super Viral, Iconic, Legendary and Hall of Fame.
The same run feeds storage, analysis tables, a presentation-ready dashboard and a report email. No manual copy-paste between stages.
Hand-designed workbook stays intact while data sheets and table ranges are refreshed.
Download workbook ↗SQLite keeps historical rows. CSVs expose repo, language, category, velocity, topic and activity datasets.
Open repo dataset ↗Automation is designed around one repeatable command. run_all.py executes extraction first, then Excel sync. A scheduler can trigger it daily; the report step packages the result for delivery.
Project stays modular: extraction + transformation + storage in pipeline.py; workbook sync in update_excel.py; orchestration in run_all.py; email reporting in send_report.py.
# pipeline.pyurl = "https://api.github.com/search/repositories"params = { "q": f"created:>{since}", "sort": "stars", "order": "desc",} response = requests.get(url, params=params, ...) # transformdf["stars_per_hour"] = ( df["stars"] / age_h.clip(lower=1)).round(2) # loaddf.to_sql("repos_temp", conn, if_exists="replace", index=False)