What’s tech stack/architecture/hosting situation look like at the moment? Has it hit any specific pain points in the last year? Is it likely to need change?
TL;DR; – For Funds/GWWC, the frontends are React (via NextJS) running on Vercel (previously a React SPA running on Netlify). The backend is a bunch of Node.js microservices running on Heroku, connected to a Postgres DB (running on RDS), and wired together with RabbitMQ. We’ve migrated most things to TypeScript, but a lot of the backend is still JS. A lot of business logic is written in SQL/plpgSQL.
---
EA Funds and GWWC have been on the same platform since 2017, and share the same backend.
The frontend was a pretty standard React SPA written in JavaScript, but we’re currently in the process of deprecating this for a NextJS app written in Typescript (which has been a huge win in terms of productivity). EA Funds has been ported (compare the new with the old) already, and the incoming GWWC developer will be working with me on porting the existing SPA functionality to the same NextJS-based system. The new sites are in a Git monorepo managed with Yarn Workspaces, meaning that eventually all EA Funds + CEA sites will be able to share components/login architecture/backend connections etc.
Down the React rabbithole a bit: We connect to the backend using Apollo to manage GraphQL queries, we use Immer for immutable state management as needed (though we don’t use Redux or any other global state management). UI components are provided with Material-UI.
I used to use Netlify to host the EA Funds/GWWC frontend, but we’ve moved to Vercel for their first-class NextJS support.
The backend is a collection of quasi-microservices running on Heroku, all written as Node.js apps:
An Express web server that handles our GraphQL endpoint, as well as webhooks/callbacks for various integrations (e.g. Stripe payments).
The GraphQL endpoint is provided by Postgraphile, which is essentially a way of generating a GraphQL schema by reflecting over our Postgres DB. This means that we get to leverage the data structure, foreign key relationships, and type safety of the existing database for free in GraphQL. This approach means that a lot of business logic (especially around user-facing CRUD and reporting) are written directly in SQL, implemented as views and functions.
The services are connected by a RabbitMQ message bus. Database events (e.g. “db.payment.updated”) and webhook calls (e.g. from payment processors) are pushed onto the RabbitMQ bus , so that other services can react to these events. So, if a user inserts a new payment, the Payments service can communicate with the appropriate payment processor, and when the payment status is updated as succeeded, the Emails service can send out a receipt.
The Postgres DB is hosted on Amazon RDS.
---
In addition to EA Funds/GWWC, I’ve also helped set up a bunch of other sites used by CEA (e.g. EffectiveAltruism.org, EAGlobal.org, CentreForEffectiveAltruism.org, the GivingWhatWeCan.org homepage). Most of these are currently using the Metalsmith static site generator, which generates static HTML files that are served via Netlify. This setup has been fantastic for performance and reliability, but eventually these sites will be ported over to the NextJS monorepo for better maintainability.
---
In terms of pain points, it’s generally been a pretty solid system. The biggest challenges have been maintenance. E.g. we’ve migrated to NextJS, partly for the improved performance and DX, but also because the previous SPA was running an outdated version of React, and because of the way the boilerplate I used was architected, upgrading to a more modern version (which many packages now require) was more trouble than it was worth. Similarly, all the static sites have historically been hosted in their own repositories, which has meant that they all have slightly different ways of doing things, and improvements made to one don’t propagate to the others. Hence, the move to a monorepo, where we can share components/logic between sites. Also, the more I use TypeScript, the more I hate using vanilla JS, so I guess that’s something of a pain point in the parts of the backend that haven’t been migrated yet!
The Forum is Mongo, Node, React app written in Typescript. We[1] started off on Meteor, which informs a lot of the design patterns, but have recently finished the process of removing it, which has greatly sped up build times. The project was initially build on top of the Vulcan framework, which was a relatively immature framework attempting to provide an even more full-service bootstrapping experience than Meteor. We got off of it several years ago, but it still informs our pattern which you might call UI-inclusive schemas.
Other technologies:
We use GraphQL as our API layer, which you can explore here.
Material UI provides the styling framework. (This is the framework, not the material design philosophy, which we differ quite a lot from.)
Our Editor is a customized CKEditor 5 editor
I’d highlight two pain points that I feel:
Our testing story is pretty poor. So much of what we do is hard to separate from the state of the app that it’s hard to make good, non-trivial unit tests. I think this is funged against by good type checking with TypeScript, but my inner Gary Bernhardt is disappointed.
Deploying to AWS takes way too long. I believe this will be fixed soon.
[1] “We” here being inclusive of the LW team, with whom we keep up to date.
What’s tech stack/architecture/hosting situation look like at the moment? Has it hit any specific pain points in the last year? Is it likely to need change?
TL;DR; – For Funds/GWWC, the frontends are React (via NextJS) running on Vercel (previously a React SPA running on Netlify). The backend is a bunch of Node.js microservices running on Heroku, connected to a Postgres DB (running on RDS), and wired together with RabbitMQ. We’ve migrated most things to TypeScript, but a lot of the backend is still JS. A lot of business logic is written in SQL/plpgSQL.
---
EA Funds and GWWC have been on the same platform since 2017, and share the same backend.
The frontend was a pretty standard React SPA written in JavaScript, but we’re currently in the process of deprecating this for a NextJS app written in Typescript (which has been a huge win in terms of productivity). EA Funds has been ported (compare the new with the old) already, and the incoming GWWC developer will be working with me on porting the existing SPA functionality to the same NextJS-based system. The new sites are in a Git monorepo managed with Yarn Workspaces, meaning that eventually all EA Funds + CEA sites will be able to share components/login architecture/backend connections etc.
Down the React rabbithole a bit: We connect to the backend using Apollo to manage GraphQL queries, we use Immer for immutable state management as needed (though we don’t use Redux or any other global state management). UI components are provided with Material-UI.
I used to use Netlify to host the EA Funds/GWWC frontend, but we’ve moved to Vercel for their first-class NextJS support.
The backend is a collection of quasi-microservices running on Heroku, all written as Node.js apps:
An Express web server that handles our GraphQL endpoint, as well as webhooks/callbacks for various integrations (e.g. Stripe payments).
The GraphQL endpoint is provided by Postgraphile, which is essentially a way of generating a GraphQL schema by reflecting over our Postgres DB. This means that we get to leverage the data structure, foreign key relationships, and type safety of the existing database for free in GraphQL. This approach means that a lot of business logic (especially around user-facing CRUD and reporting) are written directly in SQL, implemented as views and functions.
The services are connected by a RabbitMQ message bus. Database events (e.g. “db.payment.updated”) and webhook calls (e.g. from payment processors) are pushed onto the RabbitMQ bus , so that other services can react to these events. So, if a user inserts a new payment, the Payments service can communicate with the appropriate payment processor, and when the payment status is updated as succeeded, the Emails service can send out a receipt.
The Postgres DB is hosted on Amazon RDS.
---
In addition to EA Funds/GWWC, I’ve also helped set up a bunch of other sites used by CEA (e.g. EffectiveAltruism.org, EAGlobal.org, CentreForEffectiveAltruism.org, the GivingWhatWeCan.org homepage). Most of these are currently using the Metalsmith static site generator, which generates static HTML files that are served via Netlify. This setup has been fantastic for performance and reliability, but eventually these sites will be ported over to the NextJS monorepo for better maintainability.
---
In terms of pain points, it’s generally been a pretty solid system. The biggest challenges have been maintenance. E.g. we’ve migrated to NextJS, partly for the improved performance and DX, but also because the previous SPA was running an outdated version of React, and because of the way the boilerplate I used was architected, upgrading to a more modern version (which many packages now require) was more trouble than it was worth. Similarly, all the static sites have historically been hosted in their own repositories, which has meant that they all have slightly different ways of doing things, and improvements made to one don’t propagate to the others. Hence, the move to a monorepo, where we can share components/logic between sites. Also, the more I use TypeScript, the more I hate using vanilla JS, so I guess that’s something of a pain point in the parts of the backend that haven’t been migrated yet!
The Forum is Mongo, Node, React app written in Typescript. We[1] started off on Meteor, which informs a lot of the design patterns, but have recently finished the process of removing it, which has greatly sped up build times. The project was initially build on top of the Vulcan framework, which was a relatively immature framework attempting to provide an even more full-service bootstrapping experience than Meteor. We got off of it several years ago, but it still informs our pattern which you might call UI-inclusive schemas.
Other technologies:
We use GraphQL as our API layer, which you can explore here.
Apollo manages the client state
We’re hosted on AWS Elastic Beanstalk
Material UI provides the styling framework. (This is the framework, not the material design philosophy, which we differ quite a lot from.)
Our Editor is a customized CKEditor 5 editor
I’d highlight two pain points that I feel:
Our testing story is pretty poor. So much of what we do is hard to separate from the state of the app that it’s hard to make good, non-trivial unit tests. I think this is funged against by good type checking with TypeScript, but my inner Gary Bernhardt is disappointed.
Deploying to AWS takes way too long. I believe this will be fixed soon.
[1] “We” here being inclusive of the LW team, with whom we keep up to date.