The unfortunate naming of themes means that it probably bring some underlying expectations along with it. I think it is worth addressing this first and foremost. Traditionally a theme can be thought of as a single root level package that includes styling and templates and layouts for a website. You may be able to make child themes off of this and customize it, but that's about it. The flexibility in this system lies in the numerous amount of themes to choose from. One might search through many themes until they find one that does and looks like 90%+ of what is needed. The alternative here is a theme that literally has a built in website builder in it. None of this is required or should represent a Gatsby theme.
One might want to make a Gatsby theme when they can abstract out a part of a site. A Gatsby theme is just another plugin. The build script of Gatsby is effectively just an imperative list of things to do in order. Within these steps you might have multiple asynchronous processes that go on. In the end though it's just a big long list of things to do.
The bulk of the typical plugins that exist right now are generally more smaller primitives for each one of these life cycles. It may only run during one of the processes asynchronously. A theme is more likely to span multiple levels of these life cycles. There is no requirement to do that though. Rather than thinking of a theme as a root level with tweaks and adjustments off of it, I think it is better thought of as more of a horizontal abstraction.
You can pull pieces and parts out of a website and include that as a theme. For example, one of the themes that I made was gatsby-theme-recipes. It controls a recipes page and a page for each one of the recipes. It also sets up schema that enables dropping in data from any source into nodes that this theme can work with. The are also some sources that are built in and require less scaffolding to work.
Now with this theme, it is designed that we can put multiple sets of these type of themes side by side to build up one full website. You may have a theme for your articles, your recipes, your projects and what you're working on, the hardware and software that you use, a list of your experience similar to a resume or CV, a patreon stole donation page and anything else you may want to put on a personal website. Obviously this is not just a function that is available to personal sites and can be extrapolated to other websites as well.
A set of themes that a Nick DeJesus is working on integrates with the stripe API and lets you set up stripe payments much quicker. This could be added into a personal site as a very direct model similar to patreon where someone can support you. You can also add in a swag store and store if you are really going down the personal branding route. Richard Haines has themes that allows one to add a page with their bookmarks in Pocket or a page with their resume. These are all great examples of themes that have hit the right type of abstraction.
I have also created a set of themes. The current set includes a homepage, a set of pages for your articles (and other content such as notes of you choose), and a set of components for the header/footer.
These themes were written with the intent to allow a user to gradually settle into their Gatsby site. The current ecosystem seems to rely too heavily on shadowing and uses it as a crutch. Shadowing is a powerful and important feature, but the theme creator should provide a more gradual experience that eventually leads to shadowing if necessary.
These themes seek to accomplish our concept by using convention then configuration then shadowing if necessary. The convention is represented by the built-in data sources that it expects to consume, and the structure and location of the content. As the user wants to address more advanced configurations, they can use the Gatsby lifecycles to bring in data and put it into the existing data model. This would allow the user to make the content their own without the need to rely on any shadowing.
The other item that a user may want to modify is the styling and layout of the pages. A theme generally makes a basic assumption about the method of applying styles, be it css, Sass, css-in-js or what have you. For example, if they want to use Sass, but the library uses a css-in-js solution, the user will likely not pick this theme. Assuming that the user is okay with the method of styling that the theme uses, the real crux is changing the structure of the page. The only way they can modify the structures of pages is by shadowing components. This set of themes explores the middle ground.
These themes use theme-ui which has some built-in ability to customize colors, fonts and potentially some structural changes depending on how the theme creator builds the page. That can only get the user so far though. It also makes full use of the createPages lifecycles and template features. We can enable a gradual step into shadowing by providing multiple templates that can be used and composed based on convention, configuration, context and/or a combination of all three. From here we can move forward from a theme approach that is more than single-minded.
There has been no clear direction from Gatsby core on specific ways to and where to create these levels of abstraction. The open-ended nature of it allows for some creative use of themes, but it also means that every theme is built a little bit differently and inconsistently. The abstraction put forth in this article means that we do leave some of the plug-in configuration up to the user. The other themes and some of the official themes attempt to take care of more of that for the user, but it seems to mean that when you begin working outside of the box that they have created then you run into pain. I think the natural evolution of a website is to begin small and continue to build on top of it, and if we make that process of evolution difficult we are doing a disservice to the user.
Is there an alternative here? There has been some recent work on a new feature called Gatsby recipes, which allows someone to create a markdown file (mdx specifically) that serves both as documentation and a script that can be run and bootstrap Gatsby setup. I think this is a better workflow than starters as it will allow someone to settle into their setup, rather than giving them a full service kitchen and hoping they know how to cook in it.
With the combination of recipes and themes, I think we are entering the territory where someone can spin up a site very quickly and progressively get more comfortable modifying and extending their website. This may surface as adjusting configuration within a theme, or even removing themes and going completely custom on parts of their website. This is effectively progressive enhancement for website building.
Keep your eyes peeled on this space!
P.S. I would like to give a big shoutout to Chris for their work on creating themes and the help and pointers they provided in the more intricate parts of the code. Cheers!