Web Excursions 2021-09-06

Building Calliope: A Technical Journey Through MacStories’ Big Software Project

  • Calliope was built from the ground up to work in this paradigm, and

    • our full platform consists of eight separate (micro)services and three separate database deployments,

    • all managed seamlessly by Kubernetes.

  • To manage incoming traffic and load-balancing between services,

    • we run the incredible Ambassador Edge Stack.

    • This Kubernetes-native, open-source software project manages mapping URL paths to particular back-end services, and

    • load-balancing incoming traffic across multiple service instances (among other things).

  • Internally, we’ve built our platform on five services: an authentication service, a proxy service, an API service, a web server, and a Discord bot

  • We built our back-end services on Node.js, and our front-end website and web server on React via Next.js

  • Linode launched Linode Kubernetes Engine, or LKE, in mid 2020.

    • they don’t charge their customers for the server that runs the Kubernetes controllers.

    • You only pay for what you directly run your services on.

  • We use the open-source project Prometheus to collect metrics from throughout our system, and

    • the open-source project Grafana to graph these metrics in an understandable way

  • One of Calliope’s big features is the ability to create posts with a parent-child relationship.

    • For example, our weekly newsletter for Club MacStories members is made up of a variety of different sections.

    • Each of these sections are written independently, and generally by different authors

  • I’ve never liked the idea of “single-page web apps.”

    • Many React web apps these days are really just a single JavaScript-generated page

      • which fakes browser navigation and dynamically pulls content from the server after arriving on the client’s browser

  • I built an authentication proxy service which lives at the public URLs for each of our new websites

    • I deployed our Next.js instance onto a private network in our Kubernetes back end.

      • There’s no way to access the service from the public Internet, and

      • thus it can exist in a happy land of no authentication

  • Using some very simple syntax directly in their Markdown text, authors are able to target blocks of content at particular tiers of subscribers.

    • This means that within the same post, a Premier member might get some extra content that simply does not appear when viewed by a standard Club member.

  • Migrating from MailChimp

    • If you’ve ever had the misfortune of needing to inspect the HTML content of Mailchimp emails, you’ll know that it’s a terrifying mess.

      • Your text is buried under layers upon layers of <table><tbody><tr> and <td> elements.

      • Amongst these are clusters of ID-less and class-less <div>s and <span>s

    • For each issue of MacStories Weekly or our Monthly Log, we would forward the URL of the Mailchimp-generated web archive to my back-end parser service (this was all done using a custom front-end interface for parsing these issues).

      • The parser would pull the archive’s HTML contents and pass it to Cheerio,

      • which would transform it into a traversable virtual DOM tree.

  • Getting Wild with Faux Dynamism:

    • I wanted to have my cake and eat it too: develop in React but deploy as a static website.

    • Thankfully, this is exactly what the Next.js framework is designed to do.

    • Technically it still hydrates the page with JavaScript,

      • but with careful design you can cut down on the amount that you’re depending on JS.

  • As we continued building Calliope and found more and more needs for the project, it became clear that

    • we had use cases for automatically applying filters to our Markdown text as it went into the system, as well as

    • filters to our HTML text that was generated from the Markdown.

    • We even needed to apply filters to the HTML as it was coming out of the CMS,

      • in order to insert various dynamic elements based on the subscription tier of the requesting user