View GitHub Trending Repos

This was a fun one to build. So GitHub does not have an API endpoint for their trending page. So to work around this I used playwright to scrape the pages for various languages. Then I'd create a tab for a variety of language. What I ran into though was given the data likely doesn't change too often, going and scraping on every tab change was slow. To resolve this, I wrote a second script that runs on an hourly cron and then saves the data to a lowdb database with the preloaded db function. This now makes the main script run instantly as the entries are cached on my system.

Scripts

Main Script:

// Menu: GitHub Trending
// Description: Show today's Trending GitHub Repos for various languages

const trendingDB = db("github-trending");

const langs = ["rust", "javascript", "typescript", "go", "python", "ruby"];
for (const lang of langs) {
  onTab(lang, async () => {
    const repo = await arg("Select a repo to open it", trendingDB.get(lang));
    exec(`open ${repo}`);
  });
}

Background puller script:

// Description: Pulls down trending repos from github and save to database
// Schedule: 0 * * * *
// Exclude: true

/** @type typeof import('playwright') */
const playwright = await npm("playwright");

let dbDefaults = {};

const langs = ["rust", "javascript", "typescript", "go", "python", "ruby"];

for (const lang of langs) {
  dbDefaults[lang] = [];
}

const trendingDB = db("github-trending", dbDefaults);

const browser = await playwright.chromium.launch();

for (const lang of langs) {
  const context = await browser.newContext();

  const page = await context.newPage();

  await page.goto(`https://github.com/trending/${lang}`);

  const repos = await page.evaluate(() => {
    const repos = document.querySelectorAll(".Box-row");

    const results = [];

    for (let repo of repos) {
      const repoName = repo.querySelector("h1 a").getAttribute("href").slice(1);
      let description = repo.querySelector("p")?.textContent.trim();
      const starCount = repo
        .querySelector("div span.d-inline-block.float-sm-right")
        ?.textContent.trim();

      if (!description) {
        description = starCount;
      } else {
        description = `${starCount} | ${description}`;
      }

      results.push({
        name: repoName,
        value: `https://github.com/${repoName}`,
        description,
      });
    }

    return results;
  });

  trendingDB.set(lang, repos).write();
}
await browser.close();