However, in Vitessce, many other things complicate both the development and production setups.
Luckily, there are new ways to address these complications due to improvements in language standards and build tooling. Thanks to discussions with Trevor Manz, we have been able to greatly improve our build setup.
In the following sections I describe each requirement and how we approached it in the new setup, in the hopes that it may be useful to others.
Previous build setup
We previously used ejected create-react-app (CRA) scripts. The base CRA scripts work well for publishing web apps but are not meant for bundling a library. We had to eject the base scripts to add support for the library build. The resulting scripts consisted of two Webpack configurations which used many plugins that were difficult to maintain but supported many of our requirements.
While I previously made attempts to simplify our bundle scripts by using Rollup or Vite in place of CRA, these efforts were never successful. They usually required hours of debugging to find a perfect (ordered) set of plugins and parameter settings to support all of the requirements of Vitessce and its dependencies. It often felt like a game of whack-a-mole, fixing each issue just for another to emerge. After days of debugging, I achieved fully working setups on a few occasions. Unfortunately, even on these successful occasions (i.e., bundles, tests, and development server with no errors), I would still encounter issues preventing full adoption: the bundle size would be larger, or hot reloading in the development server would be unacceptably slow.
Incentives for this work
Further, it is difficult to estimate how long this work will take. It often requires major code refactoring, during which any unrelated code changes by other team members will need to be re-implemented in the refactored code base. For external consumers of the NPM package, bundle structure or format changes often constitute breaking changes. All of these factors mean that this work is often perfomed outside of normal work hours so that it does not prevent the development of new features or conflict with the work of other team members.
New build setup, by requirement
Bundle size and individual component bundles
One factor contributing to the overall bundle size is that we also included self-contained builds for individual React components. With such a large main bundle file, library consumers who only want to use individual components can achieve better development performance in their own environments by directly importing from the individual component bundles. However, this requirement adds redundant code to the bundle and contributes to the overall bundle size on NPM, as each individual component also requires a matrix of builds (minified/unminified, development/production, etc).
We addressed this by adopting a "monorepo" repository structure. This way, we can implement individual components in their own sub-packages which can be published to NPM as standalone bundles. Practically, this meant using PNPM for dependency management (as opposed to NPM), which implements many features for monorepo ("workspaces") management, including recursive installation and parallel script running.
Minimizing bundle steps in sub-packages
In the meantime, bundlers can be useful for generating the single-file, minified bundles used in the browser.
Historically, another role of a bundler has been to resolve non-JS and non-standard file imports.
For instance, it has become common to want to import CSS, SCSS, SVG, JSON, GLSL, and Web Worker files via JS-like syntax (e.g.,
import obj from './file.json').
A bundler provides the ability to transform non-standard source code into browser-compatible code.
On the other hand, in order to publish re-usable source code to NPM and avoid bundle steps, we need to avoid these non-standard imports.
We have achieved this by using only
tsc as a simple build step in our sub-packages, with only minor exceptions.
Where there are exceptions, the separation of concerns enabled by the monorepo structure allows more complex bundle steps to be scoped to the sub-repo code and its requirements, reducing complexity, improving performance, and allowing different bundler tools to be used in each context.
For example, we make exceptions in the following places:
- The main
vitesscepackage: For both backwards compatibility and to enable inclusion in HTML pages, we generate a matrix of single-file bundles using Vite and its library mode.
@vitessce/iconssub-package: To enable SVGs to be imported and transformed into React components automatically, we use Vite and vite-plugin-svgr to produce a plain JS bundle that can be consumed by other monorepo packages. (Note: This setup does not allow SVGs to be hot-reloaded during development, but we can probably assume they will not change frequently.)
@vitessce/workerssub-package: To enable Web Workers to be in-lined, we use Rollup + rollup-plugin-web-worker-loader. (This comes with the same hot reloading caveat as the
Website development with hot reloading
We have two website requirements in this project: a minimal development website (an
index.html page with a list of demos), and a documentation website.
These websites both need to consume the library code, ideally with hot reloading support during development.
In the same way that the monorepo's separation of concerns facilitates different bundling tools scoped to each sub-repo, it also enables different development servers in each "sub-website".
This allows us to use Vite development server mode and its React plugin for the minimal development website.
Meanwhile, we can continue to use Docusaurus and its internal Webpack configuration for the documentation website.
To enable hot reloading during development of both websites, rather than
tsc, we run
tsc --watch so that the
dist directory of each sub-package is updated on every source code change.
To further improve performance, we enable TypeScript's "incremental" mode with
To leverage the benefits of React 18, we use the TypeScript option
"jsx": "react-jsx", which also comes with the minor benefit that JSX code can be written without explicitly importing React at the top of each file.
To use React 18 in the context of Docusaurus and Material UI dependencies, which do not yet specify compatibility above React 17 in their
peerDependencies, we use PNPM's peerDependencyRules in the root of the repository to override them and ensure that we are only depending on React 18 (as two versions of React cannot be included in the same web page).
Because the Vitessce React component is intended to be consumed by a React application (or other React component library), we can assume the consumer is using their own copy of React.
In fact, including two copies of React in the same web application typically causes a website to crash.
To make React an external dependency in the main development and production bundles, we can use the
rollupOptions part of the Vite configuration:
external: ['react', 'react-dom'],
Then, we can use the React-less Vitessce bundle in different contexts: UMD, ESM with importmap, or d3.require.alias.
Managing and publishing sub-packages
To streamline the management of many sub-packages, we use PNPM and its meta-updater plugin.
This allows syncing changes to every sub-package's
package.json file, for instance to update the main
"version" or particular dependency version values.
GLSL shaders and glslify shader modules
To implement custom DeckGL shaders and extensions, we need to write GLSL shader code.
We use the glslify shader module system to leverage published GLSL functions such as colormaps.
Writing GLSL code in
.glsl files enables IDE support but introduces the need for a build step when they are imported into JS files.
The glslify module system also requires a build step to substitute strings like
#pragma glslify: plasma = require("glsl-colormap/plasma") with the actual colormap function.
As a workaround to prevent a build step, we can use the glslify CLI and simply include the shaders as strings in our JS files.
This is not ideal, so we will need to look into ways to address the IDE support and modularity that were lost.
To style our React components, we previously used a combination of SCSS and CSS-in-JS (via Material UI v4 + JSS). SCSS requires a build step to convert the source code to plain CSS code. Further, CSS imports in general also require a build step as they must be transformed from the JS-like import syntax to JS-compatible syntax. To address this, we have manually converted all SCSS to JSS using Material UI's makeStyles function and theming utilities. (Note: we have not upgraded to Material UI v5 which no longer uses JSS so this solution may not be advised in a new project.) Using JSS everywhere comes with the benefits of standardizing our styling code and facilitating direct style injection for consumers (i.e., no need for library users to include or think about a separate CSS file). While JSS auto-generates class names by default (which prevents naming conflicts), the jss-global-plugin enables global class names to be used to refer to elements defined by dependencies such as react-grid-layout.
To modernize our testing infrastructure, we have adopted Jest + Vitest + React Testing Library and upgraded to the latest version of Cypress. However, there were a few hiccups, namely using code that depends on Web Workers in unit tests and serving test fixtures as static files during unit tests. Jest + Vitest use jsdom rather than a real web browser during unit tests, for efficiency. However, jsdom lacks support for web workers, which can be addressed by including the jsdom-worker library in the test setup script. To serve static files during unit testing, we can hook into Vite's development server and add static file routes via serve-static.
It can be difficult to know what matters when it comes to the minified bundle size, and where to focus efforts when trying to reduce the size. Visualizing the bundle composition is a great way to understand which dependencies or source files are contributing. Luckily, rollup-plugin-visualizer is compatible with this build setup because Vite uses the rollup plugin API. From this visual analysis, I found that three different versions of the Zarr package were being included in the bundle, which I resolved using PNPM's overrides in the root of the monorepo.