This was a fun one to build. So GitHub does not have an API endpoint for their trending page. So to work around this I used playwright to scrape the pages for various languages. Then I'd create a tab for a variety of language. What I ran into though was given the data likely doesn't change too often, going and scraping on every tab change was slow. To resolve this, I wrote a second script that runs on an hourly cron and then saves the data to a lowdb database with the preloaded db function. This now makes the main script run instantly as the entries are cached on my system.

Screen Shot 2021-05-02 at 10 25 49 AM

Scripts

Main Script:

// Menu: GitHub Trending
// Description: Show today's Trending GitHub Repos for various languages
const trendingDB = db("github-trending");
const langs = ["rust", "javascript", "typescript", "go", "python", "ruby"];
for (const lang of langs) {
onTab(lang, async () => {
const repo = await arg("Select a repo to open it", trendingDB.get(lang));
exec(`open ${repo}`);
});
}

Background puller script:

// Description: Pulls down trending repos from github and save to database
// Schedule: 0 * * * *
// Exclude: true
/** @type typeof import('playwright') */
const playwright = await npm("playwright");
let dbDefaults = {};
const langs = ["rust", "javascript", "typescript", "go", "python", "ruby"];
for (const lang of langs) {
dbDefaults[lang] = [];
}
const trendingDB = db("github-trending", dbDefaults);
const browser = await playwright.chromium.launch();
for (const lang of langs) {
const context = await browser.newContext();
const page = await context.newPage();
await page.goto(`https://github.com/trending/${lang}`);
const repos = await page.evaluate(() => {
const repos = document.querySelectorAll(".Box-row");
const results = [];
for (let repo of repos) {
const repoName = repo.querySelector("h1 a").getAttribute("href").slice(1);
let description = repo.querySelector("p")?.textContent.trim();
const starCount = repo
.querySelector("div span.d-inline-block.float-sm-right")
?.textContent.trim();
if (!description) {
description = starCount;
} else {
description = `${starCount} | ${description}`;
}
results.push({
name: repoName,
value: `https://github.com/${repoName}`,
description,
});
}
return results;
});
trendingDB.set(lang, repos).write();
}
await browser.close();