bump version (#470)

But this time actually do that

.bumpversion.cfg

@@ -0,0 +1,15 @@
+[bumpversion]
+current_version = 0.2.2
+
+[bumpversion:file:setup.py]
+search = version="{current_version}"
+replace = version="{new_version}"
+
+[bumpversion:file:server/cli/cli.py]
+search = version="{current_version}"
+replace = version="{new_version}"
+
+[bumpversion:file:client/package.json]
+search = "version": "{current_version}"
+replace = "version": "{new_version}"
+

.gitignore

@@ -1,16 +1,37 @@
 # dependencies
-node_modules
+node_modules/
 
 # coverage
 coverage
 .nyc_output
 
+.pytest_cache/
+
 # production
-build
+build/
+dist/
+*.egg-info
+
+# Environments
+venv/
+cellxgene/
+
+# client build
+server/app/web/static/css/
+server/app/web/static/img/
+server/app/web/static/media/
+server/app/web/static/js/
+server/app/web/templates/index\.html
+
+# Jupyter Notebook
+.ipynb_checkpoints
+*.ipynb
 
 # misc
 .DS_Store
 npm-debug.log
 .vscode
-
+*.idea*
+__pycache__
+*.DS_Store*
 data

.travis.yml

@@ -0,0 +1,20 @@
+language: python
+python:
+  - "3.6"
+node_js:
+  - "8"
+cache:
+  pip: true
+install:
+  - set -eo pipefail
+  - pip install flake8
+  - ./bin/build-client
+  - pip install -e .
+  - pip install -r server/requirements-dev.txt
+script:
+  - set -eo pipefail
+  - flake8 server/app/
+  - flake8 server/cli/
+  - npm run --prefix client/ build
+  - npm run --prefix client/ test
+  - pytest -s server/test

MANIFEST.in

@@ -0,0 +1,4 @@
+recursive-include server/app/web/templates *
+recursive-include server/app/web/static *
+
+include server/requirements.txt

README.md

@@ -1,9 +1,228 @@
 # cellxgene
 
-A React + Redux web application for exploring large scale single cell RNA sequence data.
+> an interactive explorer for single-cell transcriptomics data
 
-##### Quickstart:
+`cellxgene` is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
 
-* `npm install`
+<img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
-* `npm start`
+
-* `localhost:3000`
+## getting started
+
+You'll need **python 3.6** and **Google Chrome**. The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).
+
+To install run
+
+```
+pip install cellxgene
+```
+
+To start exploring a dataset call
+
+```
+cellxgene launch dataset.h5ad --open
+```
+
+If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
+
+```
+cellxgene launch pbmc3k.h5ad --open
+```
+You should see your web browser open with the following
+
+<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+
+**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+
+There are several options available, such as:
+
+- `--layout` to specify the layout as  `tsne` or `umap`
+- `--title` to show a title on the explorer
+- `--open` to automatically open the web browser after launching (OS X only)
+
+To see all options call
+
+```
+cellxgene launch --help
+```
+
+There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+
+## data formatting
+
+### assumptions
+
+The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
+
+- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
+- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
+- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_tsne` for the `tsne` layout or `X_umap` for the `umap` layout)
+- any additional  `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments) 
+
+### prepare
+
+The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`. 
+
+To prepare from an existing `.h5ad` file use
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
+```
+
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. 
+
+Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
+```
+
+To see all options call
+
+```
+cellxgene prepare --help
+```
+
+**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
+
+```
+pip install cellxgene[louvain]
+```
+
+## conda and virtual environments
+
+If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.6
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+## FAQ
+
+> Someone sent me a directory of `10X-Genomics` data with a `mtx` file and I've never used `scanpy`, can I use `cellxgene`?
+
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+```
+
+Depending on the size of the dataset, this may take some time. Once it's done, call
+
+```
+cellxgene launch data-processed.h5ad --layout=umap --open
+```
+
+And your web browser should open with an interactive view of your data.
+
+> In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
+
+Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using 
+
+```
+pip install cellxgene[louvain]
+```
+
+> I ran `prepare` and I'm getting results that look unexpected
+
+You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
+```
+
+It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
+
+> I have extra metadata that I want to add to my dataset
+
+Currently this is not supported directly, but you should be able to do this manually using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
+
+> I tried to `pip install cellxgene` and got a weird error I don't understand
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.
+
+> How are you computing and sorting differential expression results?
+
+Currently we use a [Welch's *t*-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
+
+> I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
+
+This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
+
+## developer guide
+
+This project has made a few key design choices
+
+- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
+- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
+- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
+
+Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
+
+If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
+
+- python 3.6
+- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
+
+Then clone the project
+
+```
+git clone https://github.com/chanzuckerberg/cellxgene.git
+```
+
+Build the client web assets by calling this from inside the `cellxgene` folder
+
+```
+./bin/build-client
+```
+
+Install all requirements (we recommend doing this inside a virtual environment)
+
+```
+pip install -e .
+```
+
+You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
+
+If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://cziscience.slack.com/messages/CCTA8DF1T) and posting in the `#cellxgene-dev` channel.
+
+## development roadmap
+
+`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact) 
+
+- **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
+- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene ` to render the results of these analyses when they have been performed
+- **Deploy to web**  Many projects release public data browser websites  alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
+- **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data
+
+## contributing
+
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
+
+## inspiration and collaboration
+
+We've been heavily inspired by several other related single-cell visualization projects, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), and many others. We hope to explore collaborations where useful as this community works together on improving interactive visualization for single-cell data.
+
+We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.
+
+We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset. 
+
+We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
+
+## help and contact
+
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://cziscience.slack.com/messages/CCTA8DF1T) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+
+## reuse
+
+This project was started with the sole goal of empowering the scientific community to explore and understand their data. As such, we encourage other scientific tool builders in academia or industry to adopt the patterns, tools, and code from this project, and reach out to us with ideas or questions. All code is freely available for reuse under the [MIT license](https://opensource.org/licenses/MIT).

bin/build-client

@@ -0,0 +1,15 @@
+#!/bin/bash
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+CELLXGENE_DIR=$(dirname $DIR)
+
+cd $CELLXGENE_DIR
+
+npm install --prefix client/ client
+npm run  --prefix client build
+rm -rf server/app/web/static
+mkdir -p server/app/web/static/img
+cp client/build/index.html server/app/web/templates/
+cp -r client/build/static server/app/web/
+
+cp client/build/favicon.png server/app/web/static/img
+cp client/build/service-worker.js server/app/web/static/js/

bin/clean

@@ -0,0 +1,18 @@
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+CELLXGENE_DIR=$(dirname $DIR)
+
+echo "Uninstalling cellxgene"
+yes | pip uninstall cellxgene
+echo "removing node_modules"
+rm -rf $CELLXGENE_DIR/client/node_modules
+echo "removing client_build"
+rm -rf $CELLXGENE_DIR/client/build
+echo "removing dist"
+rm -rf $CELLXGENE_DIR/dist
+echo "removing egg-info"
+rm -rf $CELLXGENE_DIR/cellxgene.egg-info
+echo "removing static files"
+rm -f $CELLXGENE_DIR/server/app/web/templates/index.html
+rm -rf $CELLXGENE_DIR/server/app/web/static
+echo "cellxgene cleanup complete"
+

client/__tests__/util/actionHelpers.test.js

@@ -0,0 +1,60 @@
+import { rangeEncodeIndices } from "../../src/util/actionHelpers";
+
+describe("rangeEncodeIndices", () => {
+  test("small array edge cases", () => {
+    expect(rangeEncodeIndices([])).toMatchObject([]);
+    expect(rangeEncodeIndices([1])).toMatchObject([1]);
+    expect(rangeEncodeIndices([1, 99])).toMatchObject([1, 99]);
+    expect(rangeEncodeIndices([99, 1])).toMatchObject([1, 99]);
+    expect(rangeEncodeIndices([1, 100, 4])).toMatchObject([1, 4, 100]);
+  });
+
+  test("sorted flag", () => {
+    expect(rangeEncodeIndices([1, 9, 432], 10, true)).toMatchObject([
+      1,
+      9,
+      432
+    ]);
+    expect(rangeEncodeIndices([1, 9, 432], 10, false)).toMatchObject([
+      1,
+      9,
+      432
+    ]);
+    expect(rangeEncodeIndices([0, 1, 2, 3, 9, 10, 432], 2, true)).toMatchObject(
+      [[0, 3], [9, 10], 432]
+    );
+    expect(
+      rangeEncodeIndices([0, 1, 2, 3, 9, 10, 432], 2, false)
+    ).toMatchObject([[0, 3], [9, 10], 432]);
+  });
+
+  test("begin or end edge cases", () => {
+    expect(
+      rangeEncodeIndices([3, 4, 5, 6, 7, 10, 11, 12, 99], 3, false)
+    ).toMatchObject([[3, 7], [10, 12], 99]);
+    expect(
+      rangeEncodeIndices([3, 4, 5, 6, 7, 10, 11, 12], 3, false)
+    ).toMatchObject([[3, 7], [10, 12]]);
+    expect(
+      rangeEncodeIndices([0, 3, 4, 5, 6, 7, 10, 11, 12], 3, false)
+    ).toMatchObject([0, [3, 7], [10, 12]]);
+    expect(
+      rangeEncodeIndices([0, 3, 4, 5, 6, 7, 10, 11, 12, 99], 3, false)
+    ).toMatchObject([0, [3, 7], [10, 12], 99]);
+  });
+
+  test("minRangeLength", () => {
+    expect(
+      rangeEncodeIndices([3, 4, 5, 6, 7, 10, 11, 12, 99], 4, false)
+    ).toMatchObject([[3, 7], 10, 11, 12, 99]);
+    expect(
+      rangeEncodeIndices([3, 4, 5, 6, 7, 10, 11, 12], 4, false)
+    ).toMatchObject([[3, 7], 10, 11, 12]);
+    expect(
+      rangeEncodeIndices([0, 3, 4, 5, 6, 7, 10, 11, 12], 4, false)
+    ).toMatchObject([0, [3, 7], 10, 11, 12]);
+    expect(
+      rangeEncodeIndices([0, 3, 4, 5, 6, 7, 10, 11, 12, 99], 4, false)
+    ).toMatchObject([0, [3, 7], 10, 11, 12, 99]);
+  });
+});

client/__tests__/util/nameCreators.test.js

@@ -0,0 +1,68 @@
+/*
+Test cases for nameCreators.js.
+*/
+import {
+  layoutDimensionName,
+  obsAnnoDimensionName,
+  diffexpDimensionName,
+  userDefinedDimensionName,
+  makeContinuousDimensionName
+} from "../../src/util/nameCreators";
+
+describe("nameCreators", () => {
+  const nameCreators = [
+    layoutDimensionName,
+    obsAnnoDimensionName,
+    diffexpDimensionName,
+    userDefinedDimensionName
+  ];
+
+  test("check for namespace isolation", () => {
+    const foo = "foo";
+    nameCreators.forEach(fn => expect(fn(foo)).not.toBe(foo));
+
+    const bar = "bar";
+    nameCreators.forEach(fn => expect(fn(foo)).not.toBe(fn(bar)));
+
+    nameCreators.forEach(fn => {
+      const allOtherFn = nameCreators.filter(elmnt => elmnt !== fn);
+      allOtherFn.forEach(otherFn => expect(fn(foo)).not.toBe(otherFn(foo)));
+    });
+  });
+
+  test("check for legal keys", () => {
+    /* need namespace creators to return strings only */
+    nameCreators.forEach(fn => expect(fn("X")).toMatch(/X/));
+  });
+});
+
+describe("makeContinuousDimensionName", () => {
+  test("namespaces", () => {
+    expect(makeContinuousDimensionName({ isObs: true }, "OK")).toMatch(/OK/);
+    expect(makeContinuousDimensionName({ isDiffExp: true }, "OK")).toMatch(
+      /OK/
+    );
+    expect(makeContinuousDimensionName({ isUserDefined: true }, "OK")).toMatch(
+      /OK/
+    );
+  });
+
+  test("error handling", () => {
+    expect(() => makeContinuousDimensionName({}, "oops")).toThrow();
+    expect(() =>
+      makeContinuousDimensionName(
+        { isObs: false, isDiffExp: false, isUserDefined: false },
+        "oops"
+      )
+    ).toThrow();
+    expect(() =>
+      makeContinuousDimensionName({ isObs: true }, "OK")
+    ).not.toThrow();
+    expect(() =>
+      makeContinuousDimensionName({ isDiffExp: true }, "OK")
+    ).not.toThrow();
+    expect(() =>
+      makeContinuousDimensionName({ isUserDefined: true }, "OK")
+    ).not.toThrow();
+  });
+});

client/__tests__/util/stateManager/keyvalcache.test.js

@@ -0,0 +1,249 @@
+import _ from "lodash";
+import * as kvCache from "../../../src/util/stateManager/keyvalcache";
+
+/*
+This is PRIVATE to keyvalcache and must be kept in sync with
+any changs ot that module.  Need to Know - to enable error handling test
+*/
+const cachePrivateKey = "__kvcachekey__";
+
+/*
+helper function - promisify setTimeout()
+*/
+function timeout(ms) {
+  return new Promise(resolve => setTimeout(resolve, ms));
+}
+
+describe("kvcache API", () => {
+  /*
+  test the happy path create/set/get API
+  */
+
+  test("simple create", () => {
+    /* with defaults */
+    const kvc = kvCache.create();
+    expect(kvc).toBeDefined();
+    expect(kvc).toEqual(expect.objectContaining({}));
+    expect(kvCache.get(kvc, "test")).toBeUndefined();
+
+    /* with params */
+    const kvc1 = kvCache.create(/* lowWatermark */ 99, /* minTTL */ 0);
+    expect(kvc1).toBeDefined();
+    expect(kvc1).toEqual(expect.objectContaining({}));
+  });
+
+  test("set/get", () => {
+    /*
+    - check basic get/set functionality
+    - check set does not mutate source cache
+    */
+    const keyName = "foo";
+    const kvc1 = kvCache.create();
+    expect(kvc1).toBeDefined();
+    expect(kvCache.get(kvc1, keyName)).toBeUndefined();
+
+    const val2 = [2];
+    const kvc2 = kvCache.set(kvc1, keyName, val2);
+    expect(kvc2).toBeDefined();
+    expect(kvc2).not.toBe(kvc1);
+    expect(kvCache.get(kvc1, keyName)).toBeUndefined();
+    expect(kvCache.get(kvc2, keyName)).toBe(val2);
+
+    const val3 = [3];
+    const kvc3 = kvCache.set(kvc2, keyName, val3);
+    expect(kvc3).toBeDefined();
+    expect(kvc3).not.toBe(kvc1);
+    expect(kvc3).not.toBe(kvc2);
+    expect(kvCache.get(kvc1, keyName)).toBeUndefined();
+    expect(kvCache.get(kvc2, keyName)).toBe(val2);
+    expect(kvCache.get(kvc3, keyName)).toBe(val3);
+  });
+});
+
+describe("common error handling", () => {
+  /*
+  Test common error handlers
+  */
+
+  test("set() protection from namespace pollution", () => {
+    /*
+    Test that set() will not allow use of the private cache key
+    */
+    const kvc = kvCache.create();
+    expect(() => {
+      kvCache.set(kvc, cachePrivateKey, {});
+    }).toThrow();
+  });
+
+  test("create() does not accept bogus config", () => {
+    expect(() => {
+      kvCache.create([], {});
+    }).toThrow();
+    expect(() => {
+      kvCache.create(-99, 0);
+    }).toThrow();
+    expect(() => {
+      kvCache.create(100, -1);
+    }).toThrow();
+    expect(() => {
+      kvCache.create(1000, "foobar");
+    }).toThrow();
+    expect(() => {
+      kvCache.create(null, 8);
+    }).toThrow();
+  });
+});
+
+describe("map", () => {
+  /*
+  Test kvCache.map() - create new cache that is a transformation of an
+  existing cache
+  */
+  test("map of empty cache", () => {
+    const kvc = kvCache.create();
+    const callback = jest.fn();
+    const kvcMapped = kvCache.map(kvc, callback);
+    expect(callback).not.toHaveBeenCalled();
+    expect(kvcMapped).toBeDefined();
+    expect(kvcMapped).not.toBe(kvc); // immutable operation
+    expect(kvcMapped).toEqual(kvc);
+  });
+
+  test("map of non-empty cache", () => {
+    const key = "aKey";
+    const val = [0, 1, 2];
+    let kvc = kvCache.create();
+    kvc = kvCache.set(kvc, key, val);
+    const mockCB = jest.fn().mockImplementation(v => [...v]);
+    const kvcMapped = kvCache.map(kvc, mockCB);
+
+    expect(kvcMapped).toBeDefined();
+    expect(kvcMapped).not.toBe(kvc); // immutable operation
+    expect(_.isEqual(kvc, kvcMapped)).toBe(true);
+
+    expect(mockCB).toHaveBeenCalledTimes(1);
+    expect(mockCB).toHaveBeenLastCalledWith(val, key);
+  });
+});
+
+describe("flush", () => {
+  /*
+  test various cache flush behavior
+  */
+  test("flush - lowWatermark, disable minTTL", () => {
+    /*
+    verify lowWatermark functions correctly
+    */
+
+    // set lowWatermark to 2, set three times - only the final two
+    // should remain.
+    let kvc = kvCache.create(2, 0);
+    ["a", "b", "c"].forEach(k => {
+      kvc = kvCache.set(kvc, k, []);
+    });
+
+    expect(kvc).toEqual(
+      expect.objectContaining({
+        b: expect.arrayContaining([]),
+        c: expect.arrayContaining([])
+      })
+    );
+    expect(kvc).toEqual(
+      expect.not.objectContaining({
+        a: expect.arrayContaining([])
+      })
+    );
+  });
+
+  test("flush - minTTL, disable lowWatermark", async () => {
+    /*
+    verify minTTL functions correctly
+    */
+
+    // set minTTL to 1 ms
+    let kvc = kvCache.create(0, 10);
+    kvc = kvCache.set(kvc, "a", []);
+    await timeout(20);
+    ["b", "c"].forEach(k => {
+      kvc = kvCache.set(kvc, k, []);
+    });
+
+    expect(kvc).toEqual(
+      expect.objectContaining({
+        b: expect.arrayContaining([]),
+        c: expect.arrayContaining([])
+      })
+    );
+    expect(kvc).toEqual(
+      expect.not.objectContaining({
+        a: expect.arrayContaining([])
+      })
+    );
+  });
+
+  test("flush - minTTL and lowWatermark", async () => {
+    /*
+    verify minTTL functions correctly
+    */
+
+    // set lowwatermark to 3, minTTL to 1 ms
+    let kvc = kvCache.create(3, 10);
+    kvc = kvCache.set(kvc, "a", []);
+    // delay
+    await timeout(20);
+    ["b", "c"].forEach(k => {
+      kvc = kvCache.set(kvc, k, []);
+    });
+
+    expect(kvc).toEqual(
+      expect.objectContaining({
+        a: expect.arrayContaining([]),
+        b: expect.arrayContaining([]),
+        c: expect.arrayContaining([])
+      })
+    );
+
+    kvc = kvCache.set(kvc, "d", []);
+    expect(kvc).toEqual(
+      expect.objectContaining({
+        b: expect.arrayContaining([]),
+        c: expect.arrayContaining([]),
+        d: expect.arrayContaining([])
+      })
+    );
+    expect(kvc).toEqual(
+      expect.not.objectContaining({
+        a: expect.arrayContaining([])
+      })
+    );
+  });
+
+  test("manual flush", async () => {
+    let kvc = kvCache.create(1, 10);
+    ["a", "b", "c", "d"].forEach(k => {
+      kvc = kvCache.set(kvc, k, []);
+    });
+
+    // Before TTL has expired, should have all values in cache.
+    expect(kvc).toEqual(
+      expect.objectContaining({
+        a: expect.arrayContaining([]),
+        b: expect.arrayContaining([]),
+        c: expect.arrayContaining([])
+      })
+    );
+
+    // let TTL expire
+    await timeout(10);
+
+    // manually flush
+    const postFlushKvc = kvCache.flush(kvc);
+    expect(postFlushKvc).toBeDefined();
+    expect(postFlushKvc).not.toBe(kvc);
+    expect(postFlushKvc).toEqual(
+      expect.objectContaining({
+        d: expect.arrayContaining([])
+      })
+    );
+  });
+});

client/__tests__/util/stateManager/sampleResponses.js

@@ -0,0 +1,116 @@
+/* eslint no-bitwise: "off" */
+import _ from "lodash";
+
+/*
+test data mocking REST 0.2 API responses.  Used in several tests.
+*/
+const nObs = 10;
+const nVar = 32;
+const field4Categories = [83, true, "foo", 2.222222];
+const fieldDCategories = [99, false, "mumble", 3.1415];
+
+const aConfigResponse = {
+  config: {
+    features: [
+      { method: "POST", path: "/cluster/", available: false },
+      { method: "POST", path: "/layout/", available: false },
+      { method: "POST", path: "/diffexp/", available: false },
+      { method: "POST", path: "/saveLocal/", available: false }
+    ],
+    displayNames: {
+      engine: "the little engine that could",
+      dataset: "all your zeros are mine"
+    }
+  }
+};
+
+const aSchemaResponse = {
+  schema: {
+    dataframe: {
+      nObs,
+      nVar,
+      type: "float32"
+    },
+    annotations: {
+      obs: [
+        { name: "name", type: "string" },
+        { name: "field1", type: "int32" },
+        { name: "field2", type: "float32" },
+        { name: "field3", type: "boolean" },
+        {
+          name: "field4",
+          type: "categorical",
+          categories: field4Categories
+        }
+      ],
+      var: [
+        { name: "name", type: "string" },
+        { name: "fieldA", type: "int32" },
+        { name: "fieldB", type: "float32" },
+        { name: "fieldC", type: "boolean" },
+        {
+          name: "fieldD",
+          type: "categorical",
+          categories: fieldDCategories
+        }
+      ]
+    }
+  }
+};
+
+const anAnnotationsObsResponse = {
+  names: ["name", "field1", "field2", "field3", "field4"],
+  data: _()
+    .range(nObs)
+    .map(idx => [
+      idx,
+      `obs${idx}`,
+      2 * idx,
+      idx + 0.0133,
+      !!(idx & 1),
+      field4Categories[idx % field4Categories.length]
+    ])
+    .value()
+};
+
+const anAnnotationsVarResponse = {
+  names: ["fieldA", "fieldB", "fieldC", "fieldD", "name"],
+  data: _()
+    .range(nVar)
+    .map(idx => [
+      idx,
+      10 * idx,
+      idx + 2.90143,
+      !!(idx & 1),
+      fieldDCategories[idx % fieldDCategories.length],
+      `var${idx}`
+    ])
+    .value()
+};
+
+const aLayoutResponse = {
+  layout: {
+    ndims: 2,
+    coordinates: _()
+      .range(nObs)
+      .map(idx => [idx, Math.random(), Math.random()])
+      .value()
+  }
+};
+
+const aDataObsResponse = {
+  var: [2, 4, 29],
+  obs: _()
+    .range(nObs)
+    .map(idx => [idx, Math.random(), Math.random(), Math.random()])
+    .value()
+};
+
+export {
+  aLayoutResponse as layoutObs,
+  aDataObsResponse as dataObs,
+  anAnnotationsVarResponse as annotationsVar,
+  anAnnotationsObsResponse as annotationsObs,
+  aSchemaResponse as schema,
+  aConfigResponse as config
+};

client/__tests__/util/stateManager/universe.test.js

@@ -0,0 +1,121 @@
+import _ from "lodash";
+import * as Universe from "../../../src/util/stateManager/universe";
+import * as REST from "./sampleResponses";
+
+describe("createUniverseFromRestV02Response", () => {
+  /*
+  test createUniverseFromRestV02Response - this function converts
+  a set of REST 0.2 responses into a "new" Universe.
+
+  createUniverseFromRestV02Response(
+    configResponse,
+    schemaResponse,
+    annotationsObsResponse,
+    annotationsVarResponse,
+    layoutObsResponse
+  ) --> Universe
+
+  where:
+  configResponse: GET /.../config
+  schemaResponse: GET /.../schema
+  annotationsObsResponse: GET /.../annotations/obs
+  annotationsVarResponse: GET /.../annotations/var
+  layoutObsResponse: GET /.../layout/obs
+
+  See spec in docs/REST_API.md.
+  */
+
+  test("create from test data", () => {
+    /*
+    create a universe from sample data nad validate its shape & contents
+    */
+    const { nObs, nVar } = REST.schema.schema.dataframe;
+
+    const universe = Universe.createUniverseFromRestV02Response(
+      REST.config,
+      REST.schema,
+      REST.annotationsObs,
+      REST.annotationsVar,
+      REST.layoutObs
+    );
+
+    expect(universe).toBeDefined();
+    expect(universe).toMatchObject(
+      expect.objectContaining({
+        api: "0.2",
+        nObs,
+        nVar,
+        schema: REST.schema.schema,
+        obsAnnotations: expect.any(Array),
+        varAnnotations: expect.any(Array),
+        obsNameToIndexMap: expect.any(Object),
+        varNameToIndexMap: expect.any(Object),
+        obsLayout: expect.objectContaining({
+          X: expect.any(Float32Array),
+          Y: expect.any(Float32Array)
+        }),
+        varDataCache: expect.any(Object)
+      })
+    );
+
+    expect(universe.obsAnnotations).toHaveLength(nObs);
+    expect(_.keys(universe.obsNameToIndexMap)).toHaveLength(nObs);
+    expect(universe.obsLayout.X).toHaveLength(nObs);
+    expect(universe.obsLayout.Y).toHaveLength(nObs);
+    expect(universe.varAnnotations).toHaveLength(nVar);
+    expect(_.keys(universe.varNameToIndexMap)).toHaveLength(nVar);
+  });
+});
+
+describe("convertExpressionRESTv02ToObject", () => {
+  /*
+  test convertExpressionRESTv02ToObject
+
+  convertExpressionRESTv02ToObject(
+    universe,
+    response) --> { geneName: Float32Array, geneName: Float32Array, ... }
+
+  reponse is a /data/obs response:
+  {
+    var: [ varIndices fetched ],
+    obs: [
+      [ obsIndex, evalue, ... ],
+      ...
+    ]
+  }
+  */
+  test("create from response data", () => {
+    const universe = Universe.createUniverseFromRestV02Response(
+      REST.config,
+      REST.schema,
+      REST.annotationsObs,
+      REST.annotationsVar,
+      REST.layoutObs
+    );
+    const expression = Universe.convertExpressionRESTv02ToObject(
+      universe,
+      REST.dataObs
+    );
+
+    /* Check that the expected keys are present */
+    const expectedGeneNames = _.map(
+      REST.dataObs.var,
+      v => REST.annotationsVar.data[v][5]
+    );
+    expect(Object.keys(expression)).toEqual(
+      expect.arrayContaining(expectedGeneNames)
+    );
+
+    const expectedExpressionValues = _.map(
+      _.unzip(REST.dataObs.obs),
+      a => new Float32Array(a)
+    );
+
+    _.forEach(REST.dataObs.var, (varIdx, idx) => {
+      const varName = universe.varAnnotations[varIdx].name;
+      expect(varName).toBeDefined();
+      expect(varIdx).toBe(universe.varNameToIndexMap[varName]);
+      expect(expression[varName]).toEqual(expectedExpressionValues[idx + 1]);
+    });
+  });
+});

client/__tests__/util/stateManager/world.test.js

@@ -0,0 +1,255 @@
+import _ from "lodash";
+import * as Universe from "../../../src/util/stateManager/universe";
+import * as World from "../../../src/util/stateManager/world";
+import Crossfilter from "../../../src/util/typedCrossfilter";
+import * as REST from "./sampleResponses";
+import {
+  obsAnnoDimensionName,
+  layoutDimensionName
+} from "../../../src/util/nameCreators";
+import * as kvCache from "../../../src/util/stateManager/keyvalcache";
+
+/*
+Helper - creates universe, world, corssfilter and dimensionMap from
+the default REST test response.
+*/
+const defaultBigBang = () => {
+  /* create unverse, world, crossfilter and dimensionMap */
+  /* create universe */
+  const universe = Universe.createUniverseFromRestV02Response(
+    REST.config,
+    REST.schema,
+    REST.annotationsObs,
+    REST.annotationsVar,
+    REST.layoutObs
+  );
+  /* create world */
+  const world = World.createWorldFromEntireUniverse(universe);
+  /* create crossfilter */
+  const crossfilter = Crossfilter(world.obsAnnotations);
+  /* create dimension map */
+  const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+
+  return {
+    universe,
+    world,
+    crossfilter,
+    dimensionMap
+  };
+};
+
+describe("createWorldFromEntireUniverse", () => {
+  test("create from REST sample", () => {
+    const universe = Universe.createUniverseFromRestV02Response(
+      REST.config,
+      REST.schema,
+      REST.annotationsObs,
+      REST.annotationsVar,
+      REST.layoutObs
+    );
+    expect(universe).toBeDefined();
+
+    const world = World.createWorldFromEntireUniverse(universe);
+    expect(world).toBeDefined();
+
+    expect(world).toMatchObject(
+      expect.objectContaining({
+        api: "0.2",
+        nObs: universe.nObs,
+        nVar: universe.nVar,
+        schema: universe.schema,
+        obsAnnotations: universe.obsAnnotations,
+        varAnnotations: universe.varAnnotations,
+        obsLayout: universe.obsLayout,
+
+        summary: expect.objectContaining({
+          obs: _(REST.schema.schema.annotations.obs)
+            .filter(v => v.name !== "name")
+            .keyBy("name")
+            .mapValues(() => expect.any(Object))
+            .value(),
+          var: _(REST.schema.schema.annotations.var)
+            .filter(v => v.name !== "name")
+            .keyBy("name")
+            .mapValues(() => expect.any(Object))
+            .value()
+        }),
+
+        varDataCache: expect.any(Object),
+
+        obsIndex: null, // null indicating full universe
+        obsBackIndex: null
+      })
+    );
+  });
+});
+
+describe("createWorldFromCurrentSelection", () => {
+  test("create from REST sample", () => {
+    const {
+      universe,
+      world: originalWorld,
+      crossfilter,
+      dimensionMap
+    } = defaultBigBang();
+
+    /* mock a selection */
+    dimensionMap[obsAnnoDimensionName("field1")].filterRange([0, 5]);
+    dimensionMap[obsAnnoDimensionName("field3")].filterExact(false);
+
+    /* create the world from the selection */
+    const world = World.createWorldFromCurrentSelection(
+      universe,
+      originalWorld,
+      crossfilter
+    );
+    expect(world).toBeDefined();
+    expect(world.nObs).toEqual(crossfilter.countFiltered());
+
+    /*
+    calculate expected values and match against result
+    */
+
+    /* matchFilter must match the dimension filters above */
+    const matchFilter = val => val.field1 >= 0 && val.field1 < 5 && !val.field3;
+    const universeIndices = _()
+      .range(universe.nObs)
+      .filter(idx => matchFilter(universe.obsAnnotations[idx]))
+      .value();
+
+    const expected = {
+      nObs: universeIndices.length,
+      obsAnnotations: _.map(universeIndices, i => universe.obsAnnotations[i]),
+      obsLayout: {
+        X: new Float32Array(
+          _.map(universeIndices, i => universe.obsLayout.X[i])
+        ),
+        Y: new Float32Array(
+          _.map(universeIndices, i => universe.obsLayout.Y[i])
+        )
+      },
+      obsBackIndex: _.transform(
+        universeIndices,
+        (result, univIdx, worldIdx) => {
+          result[univIdx] = worldIdx;
+        },
+        new Uint32Array(universe.nObs).fill(-1)
+      ),
+      obsIndex: new Uint32Array(universeIndices)
+    };
+
+    expect(world).toMatchObject(
+      expect.objectContaining({
+        api: "0.2",
+        nObs: expected.nObs,
+        nVar: universe.nVar,
+        schema: universe.schema,
+        obsAnnotations: expected.obsAnnotations,
+        varAnnotations: universe.varAnnotations,
+        obsLayout: expected.obsLayout,
+        summary: {
+          obs: expect.any(Object) /* we could do better! */,
+          var: expect.any(Object) /* we could do better! */
+        },
+        varDataCache: expect.any(Object),
+        obsIndex: expected.obsIndex,
+        obsBackIndex: expected.obsBackIndex
+      })
+    );
+  });
+});
+
+describe("createObsDimensionMap", () => {
+  test("when universe eq world", () => {
+    /*
+    check for:
+    - creates a dimension for all obsAnnotations, PLUS X/Y layout
+    - check that dimension typing is sane
+    */
+
+    const { dimensionMap } = defaultBigBang();
+
+    const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
+    expect(dimensionMap).toBeDefined();
+    REST.annotationsObs.names.forEach(name => {
+      const dim = dimensionMap[obsAnnoDimensionName(name)];
+      if (name === "name") {
+        expect(dim).toBeUndefined();
+      } else {
+        const { type } = schemaByObsName[name];
+        if (type === "string" || type === "boolean" || type === "categorical") {
+          expect(dim).toBeInstanceOf(Crossfilter.EnumDimension);
+        } else {
+          expect(dim).toBeInstanceOf(Crossfilter.ScalarDimension);
+        }
+      }
+    });
+    expect(dimensionMap[layoutDimensionName("X")]).toBeInstanceOf(
+      Crossfilter.ScalarDimension
+    );
+    expect(dimensionMap[layoutDimensionName("Y")]).toBeInstanceOf(
+      Crossfilter.ScalarDimension
+    );
+  });
+});
+
+describe("subsetVarData", () => {
+  test("when world eq universe", () => {
+    const { universe, world } = defaultBigBang();
+    /* create a mock varData array for subsetting */
+    const sourceVarData = new Float32Array(universe.nObs);
+
+    /* expect literally the same object back */
+    const result = World.subsetVarData(world, universe, sourceVarData);
+    expect(result).toBe(sourceVarData);
+  });
+
+  test("when world neq universe", () => {
+    const { universe, world, crossfilter, dimensionMap } = defaultBigBang();
+    /* create a mock varData array for subsetting */
+    const sourceVarData = Float32Array.from(_.range(universe.nObs));
+
+    /* mock a selection */
+    dimensionMap[obsAnnoDimensionName("field1")].filterRange([0, 5]);
+    dimensionMap[obsAnnoDimensionName("field3")].filterExact(false);
+
+    /* create the world from the selection */
+    const newWorld = World.createWorldFromCurrentSelection(
+      universe,
+      world,
+      crossfilter
+    );
+    expect(newWorld.obsIndex).toMatchObject(new Uint32Array([0, 2]));
+
+    /* expect a subset */
+    const result = World.subsetVarData(newWorld, universe, sourceVarData);
+    expect(result).not.toBe(sourceVarData);
+    expect(result).toHaveLength(newWorld.nObs);
+    /* check that we have expected source var content */
+    expect(result).toMatchObject(new Float32Array([0, 2]));
+  });
+});
+
+describe("createVarDimension", () => {
+  /* create default universe */
+  const { world, crossfilter } = defaultBigBang();
+  /* create a mock var data cache */
+  const varDataCache = kvCache.set(
+    kvCache.create(),
+    "GENE",
+    Float32Array.from(_.range(world.nObs))
+  );
+  const result = World.createVarDimension(
+    world,
+    varDataCache,
+    crossfilter,
+    "GENE"
+  );
+  expect(result).toBeInstanceOf(Crossfilter.ScalarDimension);
+});
+
+describe("worldEqUniverse", () => {
+  const { universe, world } = defaultBigBang();
+  const result = World.worldEqUniverse(world, universe);
+  expect(result).toBe(true);
+});

client/__tests__/util/typedCrossfilter/bitArray.test.js

@@ -0,0 +1,185 @@
+// jshint esversion: 6
+
+import BitArray from "../../../src/util/typedCrossfilter/bitArray";
+
+const defaultTestLength = 8;
+
+describe("default select state", () => {
+  test("newly created Bitarray should be deselected", () => {
+    const ba = new BitArray(defaultTestLength);
+    expect(ba).toBeDefined();
+
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    const dim = ba.allocDimension();
+    expect(dim).toBeDefined();
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    ba.freeDimension(dim);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+  });
+});
+
+describe("select and deselect", () => {
+  test("selectAll and deselectAll", () => {
+    const ba = new BitArray(defaultTestLength);
+    expect(ba).toBeDefined();
+    const dim1 = ba.allocDimension();
+    expect(dim1).toBeDefined();
+    ba.selectAll(dim1);
+
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(true);
+    }
+
+    const dim2 = ba.allocDimension();
+    expect(dim2).toBeDefined();
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    ba.selectAll(dim2);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(true);
+    }
+
+    ba.deselectAll(dim1);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    ba.deselectAll(dim2);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    ba.selectAll(dim1);
+    ba.selectAll(dim2);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(true);
+    }
+
+    ba.freeDimension(dim1);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(true);
+    }
+
+    ba.freeDimension(dim2);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+  });
+
+  test("selectOne and deselectOne", () => {
+    const ba = new BitArray(defaultTestLength);
+    expect(ba).toBeDefined();
+    const dim = ba.allocDimension();
+    expect(dim).toBeDefined();
+
+    ba.selectOne(dim, 0);
+    expect(ba.isSelected(0)).toEqual(true);
+    for (let i = 1; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    ba.deselectOne(dim, 0);
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    ba.selectOne(dim, 1);
+    expect(ba.isSelected(1)).toEqual(true);
+    expect(ba.isSelected(0)).toEqual(false);
+    for (let i = 2; i < defaultTestLength; i += 1) {
+      expect(ba.isSelected(i)).toEqual(false);
+    }
+
+    ba.selectAll(dim);
+    ba.deselectOne(dim, defaultTestLength - 1);
+    expect(ba.isSelected(defaultTestLength - 1)).toEqual(false);
+    for (let i = 0; i < defaultTestLength - 1; i += 1) {
+      expect(ba.isSelected(i)).toEqual(true);
+    }
+  });
+});
+
+describe("selectionCount", () => {
+  test("simple", () => {
+    const ba = new BitArray(defaultTestLength);
+    expect(ba).toBeDefined();
+    const dim1 = ba.allocDimension();
+    expect(dim1).toBeDefined();
+    const dim2 = ba.allocDimension();
+    expect(dim2).toBeDefined();
+
+    expect(ba.selectionCount).toEqual(0);
+    ba.selectAll(dim1);
+    expect(ba.selectionCount).toEqual(0);
+    ba.selectAll(dim2);
+    expect(ba.selectionCount).toEqual(defaultTestLength);
+
+    for (let i = 0; i < defaultTestLength; i += 1) {
+      ba.deselectOne(dim1, i);
+      expect(ba.selectionCount).toEqual(defaultTestLength - i - 1);
+      expect(ba.selectionCount).toEqual(ba.countAllOnes());
+    }
+
+    ba.freeDimension(dim1);
+    ba.freeDimension(dim2);
+  });
+});
+
+describe("fillBySelection", () => {
+  test("sets values correctly", () => {
+    const ba = new BitArray(defaultTestLength);
+    expect(ba).toBeDefined();
+    const dim = ba.allocDimension();
+    expect(dim).toBeDefined();
+
+    const arr = new Int32Array(defaultTestLength);
+    arr.fill(0);
+    const truth = new Int32Array(defaultTestLength);
+    truth.fill(0);
+
+    // initial state should be deselected
+    ba.fillBySelection(arr, 1, 0);
+    expect(arr).toEqual(expect.not.arrayContaining([1]));
+
+    // selectAll
+    ba.selectAll(dim);
+    ba.fillBySelection(arr, 1, 0);
+    expect(arr).toEqual(expect.not.arrayContaining([0]));
+
+    // deselectOne
+    ba.deselectOne(dim, 3);
+    ba.fillBySelection(arr, 1, 0);
+    truth.fill(1);
+    truth[3] = 0;
+    expect(arr).toEqual(truth);
+
+    // deselectAll
+    ba.deselectAll(dim);
+    ba.fillBySelection(arr, 1, 0);
+    truth.fill(0);
+    expect(arr).toEqual(truth);
+
+    // selectOne
+    ba.selectOne(dim, 5);
+    ba.fillBySelection(arr, 6, 1);
+    truth.fill(1);
+    truth[5] = 6;
+    expect(arr).toEqual(truth);
+
+    // should be deselected after dimension disposal
+    ba.freeDimension(dim);
+    ba.fillBySelection(arr, 3, 9);
+    truth.fill(9);
+    expect(arr).toEqual(truth);
+  });
+});

client/__tests__/util/typedCrossfilter/positiveInterval.test.js

@@ -0,0 +1,141 @@
+// jshint esversion: 6
+
+// const PositiveIntervals = require("../../src/util/typedCrossfilter/positiveIntervals");
+import PositiveIntervals from "../../../src/util/typedCrossfilter/positiveIntervals";
+
+describe("canonicalize", () => {
+  test("empty", () => {
+    expect(PositiveIntervals.canonicalize([])).toEqual([]);
+  });
+
+  test("simple, already correct", () => {
+    expect(PositiveIntervals.canonicalize([[0, 1]])).toEqual([[0, 1]]);
+    expect(PositiveIntervals.canonicalize([[0, 1], [2, 3]])).toEqual([
+      [0, 1],
+      [2, 3]
+    ]);
+  });
+
+  test("non-canonical, need to be canonicalized", () => {
+    expect(PositiveIntervals.canonicalize([[0, 1], [1, 2]])).toEqual([[0, 2]]);
+    expect(PositiveIntervals.canonicalize([[1, 2], [2, 3]])).toEqual([[1, 3]]);
+  });
+});
+
+describe("union", () => {
+  test("empty range", () => {
+    expect(PositiveIntervals.union([], [])).toEqual([]);
+    expect(PositiveIntervals.union([], [[1, 2]])).toEqual([[1, 2]]);
+    expect(PositiveIntervals.union([], [[1, 2], [3, 4]])).toEqual([
+      [1, 2],
+      [3, 4]
+    ]);
+    expect(PositiveIntervals.union([[3, 4]], [])).toEqual([[3, 4]]);
+    expect(PositiveIntervals.union([[1, 2], [3, 4]], [])).toEqual([
+      [1, 2],
+      [3, 4]
+    ]);
+    expect(PositiveIntervals.union([[3, 3]], [])).toEqual([[3, 3]]);
+    expect(PositiveIntervals.union([], [[3, 3]])).toEqual([[3, 3]]);
+  });
+
+  test("simple ranges", () => {
+    expect(PositiveIntervals.union([[1, 2]], [[2, 3]])).toEqual([[1, 3]]);
+    expect(PositiveIntervals.union([[2, 3]], [[1, 2]])).toEqual([[1, 3]]);
+    expect(PositiveIntervals.union([[1, 2]], [[3, 4]])).toEqual([
+      [1, 2],
+      [3, 4]
+    ]);
+    expect(
+      PositiveIntervals.union([[1, 2], [3, 4]], [[6, 7], [19, 40]])
+    ).toEqual([[1, 2], [3, 4], [6, 7], [19, 40]]);
+    expect(PositiveIntervals.union([[1, 4]], [[1, 1], [3, 4]])).toEqual([
+      [1, 4]
+    ]);
+    expect(PositiveIntervals.union([[3, 3]], [[4, 4]])).toEqual([
+      [3, 3],
+      [4, 4]
+    ]);
+  });
+});
+
+describe("intersection", () => {
+  test("empty range", () => {
+    expect(PositiveIntervals.intersection([], [])).toEqual([]);
+    expect(PositiveIntervals.intersection([], [[1, 2]])).toEqual([]);
+    expect(PositiveIntervals.intersection([[1, 2]], [])).toEqual([]);
+  });
+
+  test("simple", () => {
+    expect(PositiveIntervals.intersection([[1, 2]], [[2, 3]])).toEqual([]);
+    expect(PositiveIntervals.intersection([[2, 3]], [[1, 2]])).toEqual([]);
+    expect(PositiveIntervals.intersection([[1, 10]], [[1, 10]])).toEqual([
+      [1, 10]
+    ]);
+    expect(PositiveIntervals.intersection([[1, 10]], [[2, 8]])).toEqual([
+      [2, 8]
+    ]);
+    expect(PositiveIntervals.intersection([[2, 8]], [[1, 10]])).toEqual([
+      [2, 8]
+    ]);
+    expect(PositiveIntervals.intersection([[1, 10]], [[2, 12]])).toEqual([
+      [2, 10]
+    ]);
+    expect(PositiveIntervals.intersection([[2, 12]], [[1, 10]])).toEqual([
+      [2, 10]
+    ]);
+    expect(PositiveIntervals.intersection([[1, 10]], [[1, 8]])).toEqual([
+      [1, 8]
+    ]);
+    expect(PositiveIntervals.intersection([[1, 8]], [[1, 10]])).toEqual([
+      [1, 8]
+    ]);
+    expect(PositiveIntervals.intersection([[1, 10]], [[1, 2], [6, 9]])).toEqual(
+      [[1, 2], [6, 9]]
+    );
+    expect(PositiveIntervals.intersection([[0, 2638]], [[1363, 2638]])).toEqual(
+      [[1363, 2638]]
+    );
+    expect(PositiveIntervals.intersection([[1, 2]], [[1, 2]])).toEqual([
+      [1, 2]
+    ]);
+  });
+});
+
+describe("difference", () => {
+  test("empty", () => {
+    expect(PositiveIntervals.difference([], [])).toEqual([]);
+    expect(PositiveIntervals.difference([], [[1, 10]])).toEqual([]);
+    expect(PositiveIntervals.difference([[1, 10]], [])).toEqual([[1, 10]]);
+  });
+
+  test("simple", () => {
+    expect(PositiveIntervals.difference([[1, 2], [3, 4]], [])).toEqual([
+      [1, 2],
+      [3, 4]
+    ]);
+    expect(PositiveIntervals.difference([[1, 2], [3, 10]], [[5, 10]])).toEqual([
+      [1, 2],
+      [3, 5]
+    ]);
+    expect(PositiveIntervals.difference([[1, 2], [3, 10]], [[0, 5]])).toEqual([
+      [5, 10]
+    ]);
+    expect(
+      PositiveIntervals.difference([[0, 2638]], [[0, 1363], [2055, 2638]])
+    ).toEqual([[1363, 2055]]);
+    expect(
+      PositiveIntervals.difference([[0, 1363], [2055, 2638]], [[0, 2638]])
+    ).toEqual([]);
+    expect(PositiveIntervals.difference([[0, 10]], [[0, 1]])).toEqual([
+      [1, 10]
+    ]);
+    expect(PositiveIntervals.difference([[0, 10]], [[1, 2]])).toEqual([
+      [0, 1],
+      [2, 10]
+    ]);
+    expect(PositiveIntervals.difference([[0, 10]], [[9, 10]])).toEqual([
+      [0, 9]
+    ]);
+  });
+});

client/__tests__/util/typedCrossfilter/sort.test.js

@@ -0,0 +1,56 @@
+import { sort, sortIndex } from "../../../src/util/typedCrossfilter/sort";
+
+function fillRange(arr, start = 0) {
+  const larr = arr;
+  for (let i = 0, len = larr.length; i < len; i += 1) {
+    larr[i] = i + start;
+  }
+  return larr;
+}
+
+function fillRand(arr) {
+  for (let i = 0, len = arr.length; i < len; i += 1) {
+    arr[i] = Math.random();
+  }
+  return arr;
+}
+
+describe("sort", () => {
+  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+    test(Type.name, () => {
+      expect(sort(Type.from([6, 5, 4, 3, 2, 1, 0]))).toMatchObject(
+        Type.from([0, 1, 2, 3, 4, 5, 6])
+      );
+      expect(sort(Type.from([6, 5, 4, 3, 2, 1]))).toMatchObject(
+        Type.from([1, 2, 3, 4, 5, 6])
+      );
+
+      const source = fillRand(new Type(1000));
+      expect(sort(Type.from(source))).toMatchObject(Type.from(source).sort());
+    })
+  );
+});
+
+describe("sortIndex", () => {
+  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+    test(Type.name, () => {
+      const source1 = Type.from([6, 5, 4, 3, 2, 1, 0]);
+      const index1 = fillRange(new Uint32Array(source1.length));
+      expect(sortIndex(index1, source1)).toMatchObject(
+        index1.sort((a, b) => source1[a] - source1[b])
+      );
+
+      const source2 = Type.from([6, 5, 4, 3, 2, 1]);
+      const index2 = fillRange(new Uint32Array(source2.length));
+      expect(sortIndex(index2, source2)).toMatchObject(
+        index2.sort((a, b) => source1[a] - source1[b])
+      );
+
+      const source3 = fillRand(new Type(1000));
+      const index3 = fillRange(new Uint32Array(source3.length));
+      expect(sortIndex(index3, source3)).toMatchObject(
+        index3.sort((a, b) => source1[a] - source1[b])
+      );
+    })
+  );
+});

client/__tests__/util/typedCrossfilter/typedCrossfilter.test.js

@@ -0,0 +1,462 @@
+// jshint esversion: 6
+import _ from "lodash";
+import crossfilter from "../../../src/util/typedCrossfilter";
+
+const someData = [
+  {
+    date: "2011-11-14T16:17:54Z",
+    quantity: 2,
+    total: 190,
+    tip: 100,
+    type: "tab",
+    productIDs: ["001"]
+  },
+  {
+    date: "2011-11-14T16:20:19Z",
+    quantity: 2,
+    total: 190,
+    tip: 100,
+    type: "tab",
+    productIDs: ["001", "005"]
+  },
+  {
+    date: "2011-11-14T16:28:54Z",
+    quantity: 1,
+    total: 300,
+    tip: 200,
+    type: "visa",
+    productIDs: ["004", "005"]
+  },
+  {
+    date: "2011-11-14T16:30:43Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001", "002"]
+  },
+  {
+    date: "2011-11-14T16:48:46Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["005"]
+  },
+  {
+    date: "2011-11-14T16:53:41Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001", "004", "005"]
+  },
+  {
+    date: "2011-11-14T16:54:06Z",
+    quantity: 1,
+    total: 100,
+    tip: 0,
+    type: "cash",
+    productIDs: ["001", "002", "003", "004", "005"]
+  },
+  {
+    date: "2011-11-14T16:58:03Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001"]
+  },
+  {
+    date: "2011-11-14T17:07:21Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["004", "005"]
+  },
+  {
+    date: "2011-11-14T17:22:59Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001", "002", "004", "005"]
+  },
+  {
+    date: "2011-11-14T17:25:45Z",
+    quantity: 2,
+    total: 200,
+    tip: 0,
+    type: "cash",
+    productIDs: ["002"]
+  },
+  {
+    date: "2011-11-14T17:29:52Z",
+    quantity: 1,
+    total: 200,
+    tip: 100,
+    type: "visa",
+    productIDs: ["004"]
+  }
+];
+
+function groupReduce(data, valueMap, valueReduce, valueInit) {
+  return _
+    .reduce(
+      data,
+      (acc, value) => {
+        const k = valueMap(value);
+        let r = _.find(acc, o => o.key === k);
+        if (!r) {
+          r = { key: k, value: valueInit() };
+          acc.push(r);
+        }
+        r.value = valueReduce(r.value, value);
+        return acc;
+      },
+      []
+    )
+    .sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
+}
+
+function groupCount(data, map) {
+  return groupReduce(data, map, (p, v) => p + 1, () => 0);
+}
+
+function groupSum(data, map) {
+  return groupReduce(data, map, (p, v) => (p += map(v)), () => 0);
+}
+
+var payments = null;
+beforeEach(() => {
+  payments = crossfilter(someData);
+});
+
+describe("typedCrossfilter", () => {
+  test("alloc and free", () => {
+    expect(payments).toBeDefined();
+    expect(payments.size()).toEqual(someData.length);
+    expect(payments.all()).toEqual(someData);
+
+    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    expect(quantity).toBeDefined();
+    expect(quantity.id()).toBeDefined();
+
+    quantity.dispose();
+    expect(payments.size()).toEqual(someData.length);
+    expect(payments.all()).toEqual(someData);
+  });
+
+  test("filterAll and filterNone", () => {
+    expect(payments).toBeDefined();
+    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const tip = payments.dimension(r => r.tip, Float32Array);
+    const total = payments.dimension(r => r.total, Float32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    expect(quantity).toBeDefined();
+    expect(tip).toBeDefined();
+    expect(total).toBeDefined();
+    expect(type).toBeDefined();
+
+    // initially, all should be filtered
+    expect(payments.allFiltered()).toHaveLength(payments.size());
+    expect(payments.allFiltered()).toEqual(payments.all());
+    expect(payments.countFiltered()).toEqual(someData.length);
+
+    // filterAll
+    tip.filterAll(); // should change nothing
+    expect(payments.allFiltered()).toEqual(payments.all());
+    expect(payments.countFiltered()).toEqual(someData.length);
+
+    // ditto
+    total.filterAll();
+    expect(payments.allFiltered()).toEqual(payments.all());
+    expect(payments.countFiltered()).toEqual(someData.length);
+
+    // filterNone
+    type.filterNone();
+    expect(payments.allFiltered()).toEqual([]);
+    expect(payments.countFiltered()).toEqual(0);
+
+    quantity.filterNone();
+    expect(payments.allFiltered()).toEqual([]);
+    expect(payments.countFiltered()).toEqual(0);
+
+    // invert the first none; should have no effect because type is
+    // still not filtered
+    quantity.filterAll();
+    expect(payments.allFiltered()).toEqual([]);
+    expect(payments.countFiltered()).toEqual(0);
+
+    // filter all of type; should select all
+    type.filterAll();
+    expect(payments.allFiltered()).toEqual(payments.all());
+    expect(payments.countFiltered()).toEqual(payments.size());
+  });
+
+  test("filterExact", () => {
+    expect(payments).toBeDefined();
+    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const tip = payments.dimension(r => r.tip, Float32Array);
+    const total = payments.dimension(r => r.total, Float32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    quantity.filterExact(1);
+    expect(payments.countFiltered()).toEqual(
+      _.countBy(someData, "quantity")[1]
+    );
+    expect(payments.allFiltered()).toEqual(_.filter(someData, { quantity: 1 }));
+
+    tip.filterExact(0);
+    expect(payments.allFiltered()).toEqual(
+      _.filter(someData, { tip: 0, quantity: 1 })
+    );
+
+    type.filterExact("cash");
+    expect(payments.allFiltered()).toEqual(
+      _.filter(someData, { tip: 0, quantity: 1, type: "cash" })
+    );
+  });
+
+  test("filterRange", () => {
+    expect(payments).toBeDefined();
+    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const tip = payments.dimension(r => r.tip, Float32Array);
+    const total = payments.dimension(r => r.total, Float32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    tip.filterRange([0, 91]);
+    expect(payments.allFiltered()).toEqual(
+      _(someData)
+        .filter(r => r.tip >= 0 && r.tip < 91)
+        .value()
+    );
+
+    tip.filterRange([0, 90]);
+    expect(payments.allFiltered()).toEqual(
+      _(someData)
+        .filter(r => r.tip >= 0 && r.tip < 90)
+        .value()
+    );
+
+    tip.filterRange([1, 90]);
+    expect(payments.allFiltered()).toEqual(
+      _(someData)
+        .filter(r => r.tip >= 1 && r.tip < 91)
+        .value()
+    );
+  });
+
+  test("filterEnum", () => {
+    expect(payments).toBeDefined();
+    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const tip = payments.dimension(r => r.tip, Float32Array);
+    const total = payments.dimension(r => r.total, Float32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    type.filterEnum(["tab", "cash"]);
+    expect(payments.allFiltered()).toEqual(
+      _(someData)
+        .filter(r => r.type === "cash" || r.type === "tab")
+        .value()
+    );
+
+    tip.filterEnum([0, 100]);
+    expect(payments.allFiltered()).toEqual(
+      _(someData)
+        .filter(r => r.type === "cash" || r.type === "tab")
+        .filter(r => r.tip === 0 || r.tip === 100)
+        .value()
+    );
+  });
+
+  test("more than 32 dimensions", () => {
+    expect(payments).toBeDefined();
+    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const tip = payments.dimension(r => r.tip, Float32Array);
+    const total = payments.dimension(r => r.total, Float32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    // Create a bunch of fake dimensions to ensure we can handle > 32
+    let dimMap = {};
+    for (let i = 0; i < 65; i++) {
+      dimMap[i] = payments.dimension(r => Math.random(), Float32Array);
+      expect(dimMap[i]).toBeDefined();
+      expect(dimMap[i].id()).toBeDefined();
+    }
+
+    // everything should start as selected/filtered
+    expect(payments.countFiltered()).toEqual(someData.length);
+
+    dimMap[0].filterAll();
+    dimMap[64].filterAll();
+    expect(payments.countFiltered()).toEqual(someData.length);
+
+    dimMap[33].filterNone();
+    expect(payments.allFiltered()).toEqual([]);
+
+    dimMap[33].filterAll();
+    expect(payments.allFiltered()).toEqual(someData);
+  });
+
+  test("group, default mapping, default reducer, no filter", () => {
+    expect(payments).toBeDefined();
+
+    var quantity = payments.dimension(r => r.quantity, Int32Array);
+    var tip = payments.dimension(r => r.tip, Int32Array);
+    var type = payments.dimension(r => r.type, "enum");
+    var total = payments.dimension(r => r.total, Int32Array);
+
+    _.each(
+      {
+        tip: tip.group(r => r),
+        type: type.group(),
+        total: total.group(),
+        quantity: quantity.group()
+      },
+      (grp, k) => {
+        const whatWeExpect = groupCount(someData, v => v[k]);
+        expect(grp.all()).toEqual(whatWeExpect);
+        expect(grp.size()).toEqual(whatWeExpect.length);
+        expect(grp.dispose()).toEqual(grp);
+      }
+    );
+  });
+
+  test("group, custom map, default reducer, no filters", () => {
+    expect(payments).toBeDefined();
+
+    // custom mapping in groups only works for scalar types.  Enums do not
+    // currently implement it.
+
+    const tip = payments.dimension(r => r.tip, Int32Array);
+    const totalX10 = payments.dimension(r => r.total * 10, Int32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    const paymentsByTip_A = tip.group();
+    const paymentsByTip_B = tip.group(r => 10 * r);
+    const paymentsByType = type.group(); // identity only
+    const paymentsByTotalX10_A = totalX10.group();
+    const paymentsByTotalX10_B = totalX10.group(r => r / 10);
+
+    expect(paymentsByTip_A.all()).toEqual(groupCount(someData, v => v.tip));
+    expect(paymentsByTip_B.all()).toEqual(
+      groupCount(someData, v => 10 * v.tip)
+    );
+    expect(paymentsByType.all()).toEqual(groupCount(someData, v => v.type));
+    expect(paymentsByTotalX10_A.all()).toEqual(
+      groupCount(someData, v => 10 * v.total)
+    );
+    expect(paymentsByTotalX10_B.all()).toEqual(
+      groupCount(someData, v => (10 * v.total) / 10)
+    );
+
+    for (let i of [
+      paymentsByTip_A,
+      paymentsByTip_B,
+      paymentsByType,
+      paymentsByTotalX10_A,
+      paymentsByTotalX10_B,
+      tip,
+      totalX10,
+      type
+    ]) {
+      expect(i.dispose()).toEqual(i);
+    }
+  });
+
+  test("group, default map, custom reducer, no filters", () => {
+    expect(payments).toBeDefined();
+
+    const total = payments.dimension(r => r.total, Float32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    const paymentsByTotal = total.group();
+    const paymentsByType = type.group();
+
+    // reduceCount
+    expect(paymentsByTotal.reduceCount()).toEqual(paymentsByTotal);
+    expect(paymentsByTotal.all()).toEqual(groupCount(someData, v => v.total));
+
+    // reduceSum
+    expect(paymentsByTotal.reduceSum(v => v.total)).toEqual(paymentsByTotal);
+    expect(paymentsByTotal.all()).toEqual(groupSum(someData, v => v.total));
+
+    // use custom reducers (my reducers) - count by three, init 1
+    expect(
+      paymentsByTotal.reduce((p, v) => (p += 3), (p, v) => (p -= 3), () => 1)
+    ).toEqual(paymentsByTotal);
+    expect(paymentsByTotal.all()).toEqual(
+      groupReduce(someData, v => v.total, (p, v) => p + 3, () => 1)
+    );
+
+    for (let i of [paymentsByTotal, paymentsByType, type]) {
+      expect(i.dispose()).toEqual(i);
+    }
+  });
+
+  test("group, default map, default reducer, filters", () => {
+    // From the docs:
+    // Note: a grouping intersects the crossfilter's current filters, except for the
+    // associated dimension's filter. Thus, group methods consider only records that
+    // satisfy every filter except this dimension's filter. So, if the crossfilter of
+    // payments is filtered by type and total, then group by total only observes the
+    // filter by type.
+
+    expect(payments).toBeDefined();
+
+    const tip = payments.dimension(r => r.tip, Int32Array);
+    const total = payments.dimension(r => r.total, Int32Array);
+    const type = payments.dimension(r => r.type, "enum");
+
+    const paymentsByTip = tip.group();
+    const paymentsByTotal = total.group();
+    const paymentsByType = type.group();
+
+    // 1. confirm that changing the filter on a dimension does NOT change that
+    // dimensions groups.
+    {
+      tip.filterAll(), total.filterAll(), type.filterAll();
+      let before = _.cloneDeep(paymentsByTip.all());
+      tip.filterExact(0);
+      expect(paymentsByTip.all()).toEqual(before);
+    }
+
+    // 2. confirm that changing a filter on a different dimension DOES change
+    // all other groups.
+    {
+      tip.filterAll(), total.filterAll(), type.filterAll();
+      const before = _.cloneDeep([paymentsByTotal.all(), paymentsByType.all()]);
+      tip.filterExact(0);
+      const after = [paymentsByTotal.all(), paymentsByType.all()];
+      expect(after).not.toEqual(before);
+      expect(after).toEqual([
+        groupReduce(
+          someData,
+          v => v.total,
+          (p, v) => (v.tip !== 0 ? p : p + 1),
+          () => 0
+        ),
+        groupReduce(
+          someData,
+          v => v.type,
+          (p, v) => (v.tip !== 0 ? p : p + 1),
+          () => 0
+        )
+      ]);
+    }
+
+    for (let i of [
+      paymentsByTip,
+      paymentsByTotal,
+      paymentsByType,
+      tip,
+      total,
+      type
+    ]) {
+      expect(i.dispose()).toEqual(i);
+    }
+  });
+});

client/__tests__/util/typedCrossfilter/util.test.js

@@ -0,0 +1,115 @@
+import {
+  fillRange,
+  sliceByIndex,
+  makeSortIndex,
+  lowerBound,
+  lowerBoundIndirect,
+  upperBoundIndirect
+} from "../../../src/util/typedCrossfilter/util";
+
+describe("fillRange", () => {
+  test("Array", () => {
+    expect(fillRange(new Array(6))).toMatchObject([0, 1, 2, 3, 4, 5]);
+    expect(fillRange(new Array(4), 1)).toMatchObject([1, 2, 3, 4]);
+    expect(fillRange([])).toMatchObject([]);
+  });
+
+  test("Uint32Array", () => {
+    expect(fillRange(new Uint32Array(6))).toMatchObject(
+      new Uint32Array([0, 1, 2, 3, 4, 5])
+    );
+    expect(fillRange(new Array(4), 1)).toMatchObject(
+      new Uint32Array([1, 2, 3, 4])
+    );
+    expect(fillRange([])).toMatchObject(new Uint32Array([]));
+  });
+});
+
+describe("sliceByIndex", () => {
+  test("Array", () => {
+    expect(sliceByIndex([0, 1, 2, 3, 4], [0, 1, 2])).toMatchObject([0, 1, 2]);
+    expect(sliceByIndex([0, 1, 2, 3, 4], [2, 1, 0])).toMatchObject([2, 1, 0]);
+    expect(sliceByIndex([0, 1, 2, 3, 4], [])).toMatchObject([]);
+    expect(sliceByIndex([], [])).toMatchObject([]);
+  });
+
+  test("Uint32Array", () => {
+    expect(
+      sliceByIndex([0, 1, 2, 3, 4], new Uint32Array([0, 1, 2]))
+    ).toMatchObject([0, 1, 2]);
+    expect(
+      sliceByIndex([0, 1, 2, 3, 4], new Uint32Array([2, 1, 0]))
+    ).toMatchObject([2, 1, 0]);
+    expect(sliceByIndex([0, 1, 2, 3, 4], new Uint32Array([]))).toMatchObject(
+      []
+    );
+    expect(sliceByIndex([], new Uint32Array([]))).toMatchObject([]);
+
+    expect(
+      sliceByIndex(new Uint32Array([0, 1, 2, 3, 4]), new Uint32Array([0, 1, 2]))
+    ).toMatchObject(new Uint32Array([0, 1, 2]));
+    expect(
+      sliceByIndex(new Uint32Array([0, 1, 2, 3, 4]), new Uint32Array([2, 1, 0]))
+    ).toMatchObject(new Uint32Array([2, 1, 0]));
+    expect(
+      sliceByIndex(
+        new Uint32Array([0, 1, 2, 3, 4]),
+        new Uint32Array(new Uint32Array([]))
+      )
+    ).toMatchObject(new Uint32Array([]));
+    expect(
+      sliceByIndex(new Uint32Array([]), new Uint32Array([]))
+    ).toMatchObject(new Uint32Array([]));
+  });
+
+  test("Float32Array", () => {
+    expect(
+      sliceByIndex(new Float32Array([0, 1, 2, 3, 4]), [0, 1, 2])
+    ).toMatchObject(new Float32Array([0, 1, 2]));
+    expect(
+      sliceByIndex(new Float32Array([0, 1, 2, 3, 4]), [2, 1, 0])
+    ).toMatchObject(new Float32Array([2, 1, 0]));
+    expect(sliceByIndex(new Float32Array([0, 1, 2, 3, 4]), [])).toMatchObject(
+      new Float32Array([])
+    );
+    expect(sliceByIndex(new Float32Array([]), [])).toMatchObject(
+      new Float32Array([])
+    );
+  });
+});
+
+describe("makeSortIndex", () => {
+  test("Array", () => {
+    expect(makeSortIndex([3, 2, 1, 0])).toMatchObject(
+      new Uint32Array([3, 2, 1, 0])
+    );
+    expect(makeSortIndex([3, 2, 1, 0, 4])).toMatchObject(
+      new Uint32Array([3, 2, 1, 0, 4])
+    );
+    expect(makeSortIndex([])).toMatchObject(new Uint32Array([]));
+  });
+
+  test("Float32Array", () => {
+    expect(makeSortIndex(new Float32Array([3, 2, 1, 0]))).toMatchObject(
+      new Uint32Array([3, 2, 1, 0])
+    );
+    expect(makeSortIndex(new Float32Array([3, 2, 1, 0, 4]))).toMatchObject(
+      new Uint32Array([3, 2, 1, 0, 4])
+    );
+    expect(makeSortIndex(new Float32Array([]))).toMatchObject(
+      new Uint32Array([])
+    );
+  });
+
+  test("Int32Array", () => {
+    expect(makeSortIndex(new Int32Array([3, 2, 1, 0]))).toMatchObject(
+      new Uint32Array([3, 2, 1, 0])
+    );
+    expect(makeSortIndex(new Int32Array([3, 2, 1, 0, 4]))).toMatchObject(
+      new Uint32Array([3, 2, 1, 0, 4])
+    );
+    expect(makeSortIndex(new Int32Array([]))).toMatchObject(
+      new Uint32Array([])
+    );
+  });
+});

client/configuration/babel/babel.dev.js

@@ -0,0 +1,14 @@
+module.exports = {
+  babelrc: false,
+  cacheDirectory: true,
+  presets: [
+    ["modern-browsers", { loose: true, modules: false }],
+    "@babel/preset-react"
+  ],
+  plugins: [
+    "@babel/plugin-proposal-function-bind",
+    "@babel/plugin-proposal-class-properties",
+    ["@babel/plugin-proposal-decorators", { legacy: true }],
+    "@babel/plugin-proposal-export-namespace-from"
+  ]
+};

client/configuration/babel/babel.prod.js

@@ -0,0 +1,15 @@
+module.exports = {
+  babelrc: false,
+  presets: [
+    ["modern-browsers", { loose: true, modules: false }],
+    "@babel/preset-react"
+  ],
+  plugins: [
+    "@babel/plugin-proposal-function-bind",
+    "@babel/plugin-proposal-class-properties",
+    ["@babel/plugin-proposal-decorators", { legacy: true }],
+    "@babel/plugin-proposal-export-namespace-from",
+    "@babel/plugin-transform-react-constant-elements",
+    "@babel/plugin-transform-runtime"
+  ]
+};

client/configuration/eslint/eslint.js

@@ -0,0 +1,58 @@
+module.exports = {
+  root: true,
+  parser: "babel-eslint",
+  extends: ["airbnb", "prettier", "prettier/react"],
+  env: { browser: true, commonjs: true, es6: true },
+  globals: { expect: true },
+  parserOptions: {
+    ecmaVersion: 2017,
+    sourceType: "module",
+    ecmaFeatures: {
+      jsx: true,
+      generators: true
+    }
+  },
+  rules: {
+    "no-magic-numbers": "off",
+    "no-nested-ternary": "off",
+    "func-style": "off",
+    "arrow-parens": "off",
+    "no-use-before-define": "off",
+    "react/jsx-filename-extension": "off",
+    "comma-dangle": "off",
+    "no-underscore-dangle": "off",
+    quotes: ["error", "double"],
+    "implicit-arrow-linebreak": "off",
+    "operator-linebreak": [
+      "error",
+      "after",
+      { overrides: { "?": "before", ":": "before" } }
+    ],
+    "no-console": "off",
+    "spaced-comment": ["error", "always", { exceptions: ["*"] }],
+    "no-param-reassign": "off",
+    "object-curly-newline": ["error", { consistent: true }],
+    "react/prop-types": [0],
+    "space-before-function-paren": "off",
+    "function-paren-newline": "off",
+    "prefer-destructuring": ["error", { object: true, array: false }]
+  },
+  overrides: [
+    {
+      files: ["**/*.test.js"],
+      env: {
+        jest: true // now **/*.test.js files' env has both es6 *and* jest
+      },
+      // Can't extend in overrides: https://github.com/eslint/eslint/issues/8813
+      // "extends": ["plugin:jest/recommended"]
+      plugins: ["jest"],
+      rules: {
+        "jest/no-disabled-tests": "warn",
+        "jest/no-focused-tests": "error",
+        "jest/no-identical-title": "error",
+        "jest/prefer-to-have-length": "warn",
+        "jest/valid-expect": "error"
+      }
+    }
+  ]
+};

client/configuration/webpack/webpack.config.dev.js

@@ -1,41 +1,61 @@
 // jshint esversion: 6
 const path = require("path");
-const autoprefixer = require("autoprefixer");
 const webpack = require("webpack");
 const HtmlWebpackPlugin = require("html-webpack-plugin");
 
 const src = path.resolve("src");
 const nodeModules = path.resolve("node_modules");
 
+const babelOptions = require("../babel/babel.dev");
+
 module.exports = {
+  mode: "development",
   devtool: "eval",
-  entry: [
+  entry: ["./src/index"],
-    "webpack-hot-middleware/client?quiet=true&noInfo=true",
-    require.resolve("react-hot-loader/patch"),
-    require.resolve("../polyfills/polyfills"),
-    path.join(src, "index")
-  ],
   output: {
     path: path.resolve("build"),
     pathinfo: true,
     filename: "static/js/bundle.js",
     publicPath: "/"
   },
-  resolve: { extensions: [".js", ".json"] },
   module: {
-    loaders: [
+    rules: [
       {
         test: /\.js$/,
         include: src,
         loader: "babel-loader",
-        query: require("../babel/babel.dev")
+        options: babelOptions
       },
       {
         test: /\.css$/,
-        include: [src, nodeModules],
+        include: src,
-        loader:
+        exclude: [path.resolve(src, "index.css")],
-          "style-loader!css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!postcss-loader"
+        loader: [
+          {
+            loader: "style-loader"
+          },
+          {
+            loader: "css-loader",
+            options: {
+              modules: true,
+              localIdentName: "[name]__[local]___[hash:base64:5]"
+            }
+          }
+        ]
       },
+      {
+        test: /index\.css$/,
+        include: [path.resolve(src, "index.css")],
+        loader: [
+          {
+            loader: "style-loader"
+          },
+          {
+            loader: "css-loader"
+          }
+        ]
+      },
+
       { test: /\.json$/, include: [src, nodeModules], loader: "json-loader" },
       {
         test: /\.(jpg|png|gif|eot|svg|ttf|woff|woff2)(\?.*)?$/,
@@ -57,20 +77,6 @@ module.exports = {
       template: path.resolve("index.html"),
       favicon: path.resolve("favicon.png")
     }),
-    new webpack.LoaderOptionsPlugin({
-      options: {
-        eslint: {
-          configFile: path.resolve("./configuration/eslint/eslint.js"),
-          useEslintrc: false
-        },
-        postcss() {
-          return [autoprefixer];
-        }
-      }
-    }),
-    new webpack.DefinePlugin({ "process.env.NODE_ENV": '"development"' }),
-    // Note: only CSS is currently hot reloaded
-    new webpack.HotModuleReplacementPlugin(),
     new webpack.NoEmitOnErrorsPlugin()
   ]
 };

client/configuration/webpack/webpack.config.prod.js

@@ -1,46 +1,64 @@
 // jshint esversion: 6
 const path = require("path");
-const autoprefixer = require("autoprefixer");
-const webpack = require("webpack");
 const HtmlWebpackPlugin = require("html-webpack-plugin");
-const ExtractTextPlugin = require("extract-text-webpack-plugin");
+const MiniCssExtractPlugin = require("mini-css-extract-plugin");
 const SWPrecacheWebpackPlugin = require("sw-precache-webpack-plugin");
-const CopyWebpackPlugin = require("copy-webpack-plugin");
 const HtmlWebpackInlineSourcePlugin = require("html-webpack-inline-source-plugin");
-const MinifyPlugin = require("babel-minify-webpack-plugin");
 
 const src = path.resolve("src");
 const nodeModules = path.resolve("node_modules");
 
+const babelOptions = require("../babel/babel.prod");
+
 const publicPath = "/";
 
 module.exports = {
+  mode: "production",
   bail: true,
+  cache: false,
   devtool: "cheap-source-map",
-  entry: [require.resolve("../polyfills/polyfills"), path.join(src, "index")],
+  entry: ["./src/index.js"],
   output: {
     path: path.resolve("build"),
     filename: "static/js/[name].[chunkhash:8].js",
-    chunkFilename: "static/js/[name].[chunkhash:8].chunk.js",
     publicPath
   },
-  resolve: { extensions: [".js", ".json"] },
   module: {
-    loaders: [
+    rules: [
       {
         test: /\.js$/,
         include: src,
         loader: "babel-loader",
-        query: require("../babel/babel.prod")
+        options: babelOptions
       },
       {
         test: /\.css$/,
-        include: [src, nodeModules],
+        include: src,
-        loader: ExtractTextPlugin.extract({
+        exclude: [path.resolve(src, "index.css")],
-          fallbackLoader: "style-loader",
+        use: [
-          loader:
+          MiniCssExtractPlugin.loader,
-            "css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]-autoprefixer!postcss-loader"
+          {
-        })
+            loader: "css-loader",
+            options: {
+              modules: true,
+              importLoaders: 1,
+              localIdentName: "[name]__[local]___[hash:base64:5]"
+            }
+          }
+        ]
+      },
+      {
+        test: /index\.css$/,
+        include: [path.resolve(src, "index.css")],
+        use: [
+          MiniCssExtractPlugin.loader,
+          {
+            loader: "css-loader",
+            options: {
+              importLoaders: 1
+            }
+          }
+        ]
       },
       {
         test: /\.json$/,
@@ -52,7 +70,10 @@ module.exports = {
         test: /\.(jpg|png|gif|eot|svg|ttf|woff|woff2)(\?.*)?$/,
         include: [src, nodeModules],
         loader: "file-loader",
-        query: { name: "static/media/[name].[hash:8].[ext]" }
+        options: {
+          name: "[name].[hash:8].[ext]",
+          outputPath: "static/media/"
+        }
       },
       {
         test: /\.(mp4|webm)(\?.*)?$/,
@@ -83,24 +104,16 @@ module.exports = {
       }
     }),
     new HtmlWebpackInlineSourcePlugin(),
-    new webpack.LoaderOptionsPlugin({
+    new MiniCssExtractPlugin({
-      options: {
+      filename: "static/css/[name].[contenthash:8].css"
-        eslint: {
-          configFile: path.resolve("./configuration/eslint/eslint.js"),
-          useEslintrc: false
-        },
-        postcss() {
-          return [autoprefixer];
-        }
-      }
     }),
-    new webpack.DefinePlugin({ "process.env.NODE_ENV": '"production"' }),
-    new webpack.optimize.OccurrenceOrderPlugin(),
-    new MinifyPlugin(),
-    new ExtractTextPlugin("static/css/[name].[contenthash:8].css"),
     new SWPrecacheWebpackPlugin({
       cacheId: "cellxgene",
       filename: "service-worker.js"
     })
-  ]
+  ],
+  performance: {
+    maxEntrypointSize: 2000000,
+    maxAssetSize: 2000000
+  }
 };

client/index.html

@@ -3,12 +3,12 @@
   <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>Cellx</title>
+    <title>cellxgene</title>
-    <link href="https://fonts.googleapis.com/css?family=Roboto:400,400i,700" rel="stylesheet">
+    <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
-      html, body, p, h1, h2, h3, h4, h5, h6, span, button, input {
+      html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
-        font-family: 'Roboto','Helvetica Neue','Helvetica','Arial',sans-serif;
+        font-family: 'Roboto Condensed','Helvetica Neue','Helvetica','Arial',sans-serif;
-        font-size: 16px;
+        font-size: 14px;
       }
       body {
         margin: 0;
@@ -24,7 +24,6 @@
     <noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>, please <strong>enable JS</strong> to make this app work.</noscript>
 
     <div id="root"></div>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/4.10.0/d3.js"></script>
+
-    <script src="//d3js.org/d3-scale-chromatic.v0.3.min.js"></script>
   </body>
 </html>

client/index_template.html

@@ -3,11 +3,20 @@
   <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>Cellx</title>
+    <title>cellxgene</title>
+    <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
-      html, body, p, h1, h2, h3, h4, h5, h6, span, button, input {
+      html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
-        font-family: Helvetica Neue,Helvetica,Arial,sans-serif;
+        font-family: 'Roboto Condensed','Helvetica Neue','Helvetica','Arial',sans-serif;
-        font-size: 16px;
+        font-size: 14px;
+      }
+      body {
+        margin: 0;
+        padding: 0;
+      }
+
+      * {
+        box-sizing: border-box;
       }
     </style>
   </head>
@@ -16,14 +25,11 @@
 window.CELLXGENE = {};
 window.CELLXGENE.API = {
   prefix: "{{ prefix | safe }}",
-  version: "v0.1/"
+  version: "v0.2/"
 };
-window.CELLXGENE.datasetTitle = "{{ datasetTitle }}";
 </script>
     <noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>, please <strong>enable JS</strong> to make this app work.</noscript>
 
     <div id="root"></div>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/4.10.0/d3.js"></script>
-    <script src="//d3js.org/d3-scale-chromatic.v0.3.min.js"></script>
   </body>
 </html>

client/package.json

@@ -0,0 +1,137 @@
+{
+  "name": "cellxgene",
+  "version": "0.2.2",
+  "license": "MIT",
+  "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
+  "repository": "https://github.com/chanzuckerberg/cellxgene",
+  "scripts": {
+    "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
+    "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
+    "clean": "rimraf build",
+    "start": "node server/development.js",
+    "lint": "eslint src",
+    "test": "jest"
+  },
+  "engineStrict": true,
+  "engines": {
+    "npm": ">=3.0.0"
+  },
+  "eslintConfig": {
+    "extends": "./configuration/eslint/eslint.js"
+  },
+  "nyc": {
+    "sourceMap": false,
+    "instrument": false
+  },
+  "resolutions": {
+    "eslint-scope": "3.7.1"
+  },
+  "dependencies": {
+    "@blueprintjs/core": "^3.8.0",
+    "@blueprintjs/icons": "^3.3.0",
+    "@blueprintjs/select": "^3.2.1",
+    "canvas-fit": "^1.5.0",
+    "d3": "^4.10.0",
+    "d3-scale-chromatic": "^1.3.0",
+    "font-color-contrast": "^1.0.3",
+    "gl-mat4": "^1.1.4",
+    "gl-matrix": "^2.7.1",
+    "key-pressed": "0.0.1",
+    "lodash": "^4.17.4",
+    "memoize-one": "^4.0.0",
+    "mouse-position": "^2.0.1",
+    "mouse-pressed": "^1.0.0",
+    "normalize.css": "^8.0.0",
+    "orbit-camera": "^1.0.0",
+    "query-string": "^6.1.0",
+    "react": "^16.6.0",
+    "react-autocomplete": "^1.7.2",
+    "react-dom": "^16.6.0",
+    "react-helmet": "^5.2.0",
+    "react-icons": "^3.2.2",
+    "react-redux": "^5.1.0",
+    "redux": "^4.0.1",
+    "redux-devtools-extension": "^2.13.5",
+    "redux-thunk": "^2.2.0",
+    "regl": "^1.3.9",
+    "scroll-speed": "^1.0.0",
+    "urijs": "^1.19.0"
+  },
+  "devDependencies": {
+    "@babel/core": "^7.1.5",
+    "@babel/plugin-proposal-class-properties": "^7.0.0",
+    "@babel/plugin-proposal-decorators": "^7.0.0",
+    "@babel/plugin-proposal-export-namespace-from": "^7.0.0",
+    "@babel/plugin-proposal-function-bind": "^7.0.0",
+    "@babel/plugin-transform-react-constant-elements": "^7.0.0",
+    "@babel/plugin-transform-runtime": "^7.1.0",
+    "@babel/preset-env": "^7.1.5",
+    "@babel/preset-react": "^7.0.0",
+    "@babel/register": "^7.0.0",
+    "@babel/runtime": "^7.1.5",
+    "babel-core": "^7.0.0-bridge.0",
+    "babel-eslint": "^10.0.1",
+    "babel-jest": "^23.6.0",
+    "babel-loader": "^8.0.0",
+    "babel-plugin-istanbul": "^5.1.0",
+    "babel-preset-modern-browsers": "^12.0.0",
+    "chalk": "^2.4.1",
+    "connect-history-api-fallback": "^1.3.0",
+    "copy-webpack-plugin": "^4.6.0",
+    "css-loader": "^1.0.1",
+    "eslint": "^5.8.0",
+    "eslint-config-airbnb": "^17.1.0",
+    "eslint-config-prettier": "^3.1.0",
+    "eslint-loader": "^2.1.1",
+    "eslint-plugin-filenames": "^1.3.2",
+    "eslint-plugin-import": "^2.14.0",
+    "eslint-plugin-jest": "^21.27.2",
+    "eslint-plugin-jsx-a11y": "^6.1.1",
+    "eslint-plugin-react": "^7.11.1",
+    "express": "^4.14.0",
+    "file-loader": "^2.0.0",
+    "html-webpack-inline-source-plugin": "0.0.10",
+    "html-webpack-plugin": "^3.2.0",
+    "jest": "^23.5.0",
+    "json-loader": "^0.5.4",
+    "mini-css-extract-plugin": "^0.4.1",
+    "nyc": "^13.0.1",
+    "rimraf": "^2.5.4",
+    "serve-favicon": "^2.3.0",
+    "style-loader": "^0.23.1",
+    "sw-precache-webpack-plugin": "^0.11.5",
+    "url-loader": "^1.1.0",
+    "webpack": "^4.25.1",
+    "webpack-cli": "^3.1.0",
+    "webpack-dev-middleware": "^3.1.3"
+  },
+  "jest": {
+    "testMatch": [
+      "**/__tests__/**/?(*.)(spec|test).js?(x)"
+    ],
+    "testURL": "http://localhost/"
+  },
+  "babel": {
+    "env": {
+      "test": {
+        "presets": [
+          "@babel/preset-env",
+          "@babel/preset-react"
+        ],
+        "plugins": [
+          "@babel/plugin-proposal-function-bind",
+          "@babel/plugin-proposal-class-properties",
+          [
+            "@babel/plugin-proposal-decorators",
+            {
+              "legacy": true
+            }
+          ],
+          "@babel/plugin-proposal-export-namespace-from",
+          "@babel/plugin-transform-react-constant-elements",
+          "@babel/plugin-transform-runtime"
+        ]
+      }
+    }
+  }
+}

client/server/development.js

@@ -32,13 +32,11 @@ app.use(historyApiFallback({ verbose: false }));
 
 app.use(
   require("webpack-dev-middleware")(compiler, {
-    noInfo: true,
+    logLevel: "warn",
     publicPath: config.output.publicPath
   })
 );
 
-app.use(require("webpack-hot-middleware")(compiler));
-
 app.use(favicon("./favicon.png"));
 
 app.get("*", (req, res) => {

client/src/actions/index.js

@@ -0,0 +1,314 @@
+// jshint esversion: 6
+import _ from "lodash";
+import * as globals from "../globals";
+import { Universe, kvCache } from "../util/stateManager";
+import {
+  catchErrorsWrap,
+  doJsonRequest,
+  rangeEncodeIndices,
+  dispatchNetworkErrorMessageToUser
+} from "../util/actionHelpers";
+
+/*
+Bootstrap application with the initial data loading.
+  * /config - application configuration
+  * /schema - schema of dataframe
+  * /annotations/obs - all metadata annotation
+*/
+const doInitialDataLoad = () =>
+  catchErrorsWrap(async dispatch => {
+    dispatch({ type: "initial data load start" });
+
+    try {
+      const requests = _([
+        "config",
+        "schema",
+        "annotations/obs",
+        "annotations/var",
+        "layout/obs"
+      ])
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doJsonRequest(url))
+        .value();
+      const results = await Promise.all(requests);
+
+      /* set config defaults */
+      const config = { ...globals.configDefaults, ...results[0].config };
+      const [, schema, obsAnno, varAnno, obsLayout] = [...results];
+      const universe = Universe.createUniverseFromRestV02Response(
+        config,
+        schema,
+        obsAnno,
+        varAnno,
+        obsLayout
+      );
+
+      dispatch({
+        type: "configuration load complete",
+        config
+      });
+      dispatch({
+        type: "initial data load complete (universe exists)",
+        universe
+      });
+    } catch (error) {
+      dispatch({ type: "initial data load error", error });
+    }
+  }, true);
+
+/*
+Set the view (world) to current selection.   Placeholder for an async action
+which also does re-layout.
+*/
+const regraph = () => (dispatch, getState) => {
+  const { universe, world, crossfilter } = getState().controls;
+  dispatch({
+    type: "set World to current selection",
+    universe,
+    world,
+    crossfilter
+  });
+};
+
+// Throws
+const dispatchExpressionErrors = (dispatch, res) => {
+  const msg = `Unexpected HTTP response while fetching expression data ${
+    res.status
+  }, ${res.statusText}`;
+  dispatchNetworkErrorMessageToUser(msg);
+  throw new Error(msg);
+};
+
+/*
+Fetch expression vectors for each gene in genes.   This is NOT an action
+function, but rather a helper to be called from an action helper that
+needs expression data.
+
+Transparently utilizes cached data if it is already present.
+*/
+async function _doRequestExpressionData(dispatch, getState, genes) {
+  const state = getState();
+  const { universe } = state.controls;
+  /* preload data already in cache */
+  let expressionData = _.transform(
+    genes,
+    (expData, g) => {
+      const data = kvCache.get(universe.varDataCache, g);
+      if (data) {
+        expData[g] = data;
+      }
+    },
+    {}
+  ); // --> { gene: data }
+  /* make a list of genes for which we do not have data */
+  const genesToFetch = _.filter(genes, g => expressionData[g] === undefined);
+
+  dispatch({ type: "expression load start" });
+
+  /* Fetch data for any genes not in cache */
+  if (genesToFetch.length) {
+    try {
+      // XXX: TODO - this could be using /data/var rather than /data/obs,
+      // as that would simplify the transformation in convertExpressionRESTv02ToObject
+      const res = await fetch(
+        `${globals.API.prefix}${globals.API.version}data/obs`,
+        {
+          method: "PUT",
+          body: JSON.stringify({
+            filter: {
+              var: {
+                annotation_value: [{ name: "name", values: genesToFetch }]
+              }
+            }
+          }),
+          headers: new Headers({
+            accept: "application/json",
+            "Content-Type": "application/json"
+          })
+        }
+      );
+
+      if (!res.ok || res.headers.get("Content-Type") !== "application/json") {
+        // WILL throw
+        return dispatchExpressionErrors(dispatch, res);
+      }
+
+      const data = await res.json();
+      expressionData = {
+        ...expressionData,
+        ...Universe.convertExpressionRESTv02ToObject(universe, data)
+      };
+    } catch (error) {
+      dispatch({ type: "expression load error", error });
+      throw error; // rethrow
+    }
+  }
+
+  dispatch({ type: "expression load success", expressionData });
+  return expressionData;
+}
+
+function requestSingleGeneExpressionCountsForColoringPOST(gene) {
+  return async (dispatch, getState) => {
+    dispatch({ type: "get single gene expression for coloring started" });
+    try {
+      await _doRequestExpressionData(dispatch, getState, [gene]);
+      const { world } = getState().controls;
+      dispatch({
+        type: "color by expression",
+        gene,
+        data: {
+          [gene]: kvCache.get(world.varDataCache, gene)
+        }
+      });
+    } catch (error) {
+      dispatch({
+        type: "get single gene expression for coloring error",
+        error
+      });
+    }
+  };
+}
+
+const requestUserDefinedGene = gene => async (dispatch, getState) => {
+  dispatch({ type: "request user defined gene started" });
+  try {
+    await await _doRequestExpressionData(dispatch, getState, [gene]);
+    const { world } = getState().controls;
+
+    /* then send the success case action through */
+    return dispatch({
+      type: "request user defined gene success",
+      data: {
+        genes: [gene],
+        expression: kvCache.get(world.varDataCache, gene)
+      }
+    });
+  } catch (error) {
+    return dispatch({
+      type: "request user defined gene error",
+      error
+    });
+  }
+};
+
+const dispatchDiffExpErrors = (dispatch, response) => {
+  switch (response.status) {
+    case 403:
+      dispatchNetworkErrorMessageToUser(
+        "Too many cells selected for differential experesion calculation - please make a smaller selection."
+      );
+      break;
+    case 501:
+      dispatchNetworkErrorMessageToUser(
+        "Differential expression is not implemented."
+      );
+      break;
+    default: {
+      const msg = `Unexpected differential expression HTTP response ${
+        response.status
+      }, ${response.statusText}`;
+      dispatchNetworkErrorMessageToUser(msg);
+      dispatch({
+        type: "request differential expression error",
+        error: new Error(msg)
+      });
+    }
+  }
+};
+
+const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
+  dispatch,
+  getState
+) => {
+  dispatch({ type: "request differential expression started" });
+  try {
+    /*
+    Steps:
+    1. get the most differentially expressed genes
+    2. get expression data for each
+    */
+    const state = getState();
+    const { universe } = state.controls;
+    const set1ByIndex = rangeEncodeIndices(
+      _.map(set1, s => universe.obsNameToIndexMap[s])
+    );
+    const set2ByIndex = rangeEncodeIndices(
+      _.map(set2, s => universe.obsNameToIndexMap[s])
+    );
+    const res = await fetch(
+      `${globals.API.prefix}${globals.API.version}diffexp/obs`,
+      {
+        method: "POST",
+        headers: new Headers({
+          Accept: "application/json",
+          "Content-Type": "application/json"
+        }),
+        body: JSON.stringify({
+          mode: "topN",
+          count: num_genes,
+          set1: { filter: { obs: { index: set1ByIndex } } },
+          set2: { filter: { obs: { index: set2ByIndex } } }
+        })
+      }
+    );
+
+    if (!res.ok || res.headers.get("Content-Type") !== "application/json") {
+      return dispatchDiffExpErrors(dispatch, res);
+    }
+
+    const data = await res.json();
+    // result is [ [varIdx, ...], ... ]
+    const topNGenes = _.map(data, r => universe.varAnnotations[r[0]].name);
+
+    /*
+    Kick off secondary action to fetch all of the expression data for the
+    topN expressed genes.
+    */
+    await _doRequestExpressionData(dispatch, getState, topNGenes);
+
+    /* then send the success case action through */
+    return dispatch({
+      type: "request differential expression success",
+      data
+    });
+  } catch (error) {
+    return dispatch({
+      type: "request differential expression error",
+      error
+    });
+  }
+};
+
+const resetInterface = () => (dispatch, getState) => {
+  const { universe } = getState().controls;
+
+  dispatch({
+    type: "clear all user defined genes"
+  });
+  dispatch({
+    type: "clear differential expression"
+  });
+  dispatch({
+    type: "reset colorscale"
+  });
+  dispatch({
+    type: "clear scatterplot"
+  });
+  dispatch({
+    type: "reset World to eq Universe",
+    universe
+  });
+  dispatch({
+    type: "increment graph render counter"
+  });
+};
+
+export default {
+  regraph,
+  resetInterface,
+  requestSingleGeneExpressionCountsForColoringPOST,
+  requestDifferentialExpression,
+  requestUserDefinedGene,
+  doInitialDataLoad
+};

client/src/components/app.js

@@ -1,47 +1,37 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import Helmet from "react-helmet";
-import Container from "./framework/container";
 import { connect } from "react-redux";
-import PulseLoader from "halogen/PulseLoader";
 
+import Container from "./framework/container";
 import LeftSideBar from "./leftsidebar";
-import Parallel from "./continuous/parallel";
+import Legend from "./continuousLegend";
-import Joy from "./joy/joy";
 import Graph from "./graph/graph";
-import * as globals from "../globals";
 import actions from "../actions";
 
-import SectionHeader from "./framework/sectionHeader";
+@connect(state => ({
-
+  loading: state.controls.loading,
-@connect(state => {
+  error: state.controls.error,
-  return {
+  graphRenderCounter: state.controls.graphRenderCounter
-    cells: state.cells
+}))
-  };
-})
 class App extends React.Component {
   constructor(props) {
     super(props);
     this.state = {};
   }
-  _onURLChanged() {
+
-    this.props.dispatch({ type: "url changed", url: document.location.href });
-  }
   componentDidMount() {
+    const { dispatch } = this.props;
+
     /* listen for url changes, fire one when we start the app up */
     window.addEventListener("popstate", this._onURLChanged);
     this._onURLChanged();
 
-    this.props.dispatch(actions.initialize());
+    dispatch(actions.doInitialDataLoad(window.location.search));
 
-    /*
-      first request includes query straight off the url bar for now
-    */
-    this.props.dispatch(actions.requestCells(window.location.search));
     /* listen for resize events */
     window.addEventListener("resize", () => {
-      this.props.dispatch({
+      dispatch({
         type: "window resize",
         data: {
           height: window.innerHeight,
@@ -49,7 +39,7 @@ class App extends React.Component {
         }
       });
     });
-    this.props.dispatch({
+    dispatch({
       type: "window resize",
       data: {
         height: window.innerHeight,
@@ -58,32 +48,31 @@ class App extends React.Component {
     });
   }
 
-  render() {
+  _onURLChanged() {
-    // console.log('app:', this.props, this.state)
+    const { dispatch } = this.props;
 
+    dispatch({ type: "url changed", url: document.location.href });
+  }
+
+  render() {
+    const { loading, error, graphRenderCounter } = this.props;
     return (
       <Container>
         <Helmet title="cellxgene" />
-        {this.props.cells.loading ? (
+        {loading ? (
           <div
-            style={{ position: "fixed", left: window.innerWidth / 2, top: 150 }}
+            style={{
+              position: "fixed",
+              fontWeight: 500,
+              top: window.innerHeight / 2,
+              left: window.innerWidth / 2 - 50
+            }}
           >
-            <PulseLoader color="rgb(0,0,0)" size="10px" margin="4px" />
+            loading cellxgene
-            <span
-              style={{ fontFamily: globals.accentFont, fontStyle: "italic" }}
-            >
-              loading cells
-            </span>
           </div>
         ) : null}
-        {this.props.cells.error ? "Error loading cells" : null}
-        {false ? (
-          <Joy data={this.state.expressions && this.state.expressions.data} />
-        ) : (
-          ""
-        )}
         <div>
-          <LeftSideBar />
+          {loading ? null : <LeftSideBar />}
           <div
             style={{
               padding: 15,
@@ -91,8 +80,8 @@ class App extends React.Component {
               marginLeft: 350 /* but responsive */
             }}
           >
-            <Graph />
+            {loading ? null : <Graph key={graphRenderCounter} />}
-            {/*<Parallel/>*/}
+            <Legend />
           </div>
         </div>
       </Container>

client/src/components/brushableHistogram/index.js

@@ -0,0 +1,359 @@
+/*
+https://bl.ocks.org/mbostock/4341954
+https://bl.ocks.org/mbostock/34f08d5e11952a80609169b7917d4172
+https://bl.ocks.org/SpaceActuary/2f004899ea1b2bd78d6f1dbb2febf771
+*/
+// jshint esversion: 6
+import React from "react";
+import _ from "lodash";
+import { Button, ButtonGroup, Tooltip } from "@blueprintjs/core";
+import { connect } from "react-redux";
+import * as d3 from "d3";
+import memoize from "memoize-one";
+import { kvCache } from "../../util/stateManager";
+import * as globals from "../../globals";
+import actions from "../../actions";
+
+@connect(state => ({
+  world: state.controls.world,
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+  crossfilter: state.controls.crossfilter,
+  differential: state.differential,
+  initializeRanges: _.get(state.controls.world, "summary.obs"),
+  colorAccessor: state.controls.colorAccessor,
+  colorScale: state.controls.colorScale,
+  obsAnnotations: _.get(state.controls.world, "obsAnnotations", null)
+}))
+class HistogramBrush extends React.Component {
+  calcHistogramCache = memoize((obsAnnotations, field, ranges) => {
+    const { world } = this.props;
+    const histogramCache = {};
+
+    histogramCache.y = d3
+      .scaleLinear()
+      .range([this.height - this.marginBottom, 0]);
+
+    if (obsAnnotations[0][field]) {
+      // recalculate expensive stuff
+      const allValuesForContinuousFieldAsArray = _.map(obsAnnotations, field);
+
+      histogramCache.x = d3
+        .scaleLinear()
+        .domain([ranges.min, ranges.max])
+        .range([0, this.width]);
+
+      histogramCache.bins = d3
+        .histogram()
+        .domain(histogramCache.x.domain())
+        .thresholds(40)(allValuesForContinuousFieldAsArray);
+
+      histogramCache.numValues = allValuesForContinuousFieldAsArray.length;
+    } else if (kvCache.get(world.varDataCache, field)) {
+      /* it's not in observations, so it's a gene, but let's check to make sure */
+      const varValues = kvCache.get(world.varDataCache, field);
+
+      histogramCache.x = d3
+        .scaleLinear()
+        .domain(
+          d3.extent(varValues)
+        ) /* replace this if we have ranges for genes back from server like we do for annotations on cells */
+        .range([0, this.width]);
+
+      histogramCache.bins = d3
+        .histogram()
+        .domain(histogramCache.x.domain())
+        .thresholds(40)(varValues);
+
+      histogramCache.numValues = varValues.length;
+    }
+
+    return histogramCache;
+  });
+
+  constructor(props) {
+    super(props);
+
+    this.width = 340;
+    this.height = 100;
+    this.marginBottom = 20;
+  }
+
+  componentDidMount() {
+    const { field } = this.props;
+    const { x, y, bins, numValues, svgRef } = this._histogram;
+
+    this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+  }
+
+  componentDidUpdate(prevProps) {
+    const { field, obsAnnotations } = this.props;
+    const { x, y, bins, numValues, svgRef } = this._histogram;
+
+    if (obsAnnotations !== prevProps.obsAnnotations) {
+      this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+    }
+  }
+
+  onBrush(selection, x) {
+    return () => {
+      const { dispatch, field, isObs, isUserDefined, isDiffExp } = this.props;
+
+      if (d3.event.selection) {
+        dispatch({
+          type: "continuous metadata histogram brush",
+          selection: field,
+          continuousNamespace: {
+            isObs,
+            isUserDefined,
+            isDiffExp
+          },
+          range: [x(d3.event.selection[0]), x(d3.event.selection[1])]
+        });
+      } else {
+        dispatch({
+          type: "continuous metadata histogram brush",
+          selection: field,
+          continuousNamespace: {
+            isObs,
+            isUserDefined,
+            isDiffExp
+          },
+          range: null
+        });
+      }
+    };
+  }
+
+  drawHistogram(svgRef) {
+    const { obsAnnotations, field, ranges } = this.props;
+    const histogramCache = this.calcHistogramCache(
+      obsAnnotations,
+      field,
+      ranges
+    );
+
+    const { x, y, bins, numValues } = histogramCache;
+
+    this._histogram = { x, y, bins, numValues, svgRef };
+  }
+
+  handleColorAction() {
+    const {
+      obsAnnotations,
+      dispatch,
+      field,
+      world,
+      initializeRanges
+    } = this.props;
+
+    if (obsAnnotations[0][field]) {
+      dispatch({
+        type: "color by continuous metadata",
+        colorAccessor: field,
+        rangeMaxForColorAccessor: initializeRanges[field].range.max
+      });
+    } else if (kvCache.get(world.varDataCache, field)) {
+      dispatch(actions.requestSingleGeneExpressionCountsForColoringPOST(field));
+    }
+  }
+
+  removeHistogram() {
+    const { dispatch, field, colorAccessor } = this.props;
+    dispatch({
+      type: "clear user defined gene",
+      data: field
+    });
+    if (field === colorAccessor) {
+      dispatch({
+        type: "reset colorscale"
+      });
+    }
+  }
+
+  handleSetGeneAsScatterplotX() {
+    return () => {
+      const { dispatch, field } = this.props;
+      dispatch({
+        type: "set scatterplot x",
+        data: field
+      });
+    };
+  }
+
+  handleSetGeneAsScatterplotY() {
+    return () => {
+      const { dispatch, field } = this.props;
+      dispatch({
+        type: "set scatterplot y",
+        data: field
+      });
+    };
+  }
+
+  renderAxesBrushBins(x, y, bins, numValues, svgRef, field) {
+    /* Remove everything */
+    d3.select(svgRef)
+      .selectAll("*")
+      .remove();
+
+    /* BINS */
+    d3.select(svgRef)
+      .insert("g", "*")
+      .attr("fill", "#bbb")
+      .selectAll("rect")
+      .data(bins)
+      .enter()
+      .append("rect")
+      .attr("class", "bar")
+      .attr("x", d => x(d.x0) + 1)
+      .attr("y", d => y(d.length / numValues))
+      .attr("width", d => Math.abs(x(d.x1) - x(d.x0) - 1))
+      .attr("height", d => y(0) - y(d.length / numValues));
+
+    /* BRUSH */
+    d3.select(svgRef)
+      .append("g")
+      .attr("class", "brush")
+      .call(
+        d3
+          .brushX()
+          .on("brush", this.onBrush(field, x.invert).bind(this))
+          .on("end", this.onBrush(field, x.invert).bind(this))
+      );
+
+    /* AXIS */
+    d3.select(svgRef)
+      .append("g")
+      .attr("class", "axis axis--x")
+      .attr("transform", `translate(0,${this.height - this.marginBottom})`)
+      .call(d3.axisBottom(x).ticks(5));
+
+    d3.select(svgRef)
+      .selectAll(".axis--x text")
+      .style("fill", "rgb(80,80,80)");
+
+    d3.select(svgRef)
+      .selectAll(".axis--x path")
+      .style("stroke", "rgb(230,230,230)");
+
+    d3.select(svgRef)
+      .selectAll(".axis--x line")
+      .style("stroke", "rgb(230,230,230)");
+  }
+
+  render() {
+    const {
+      field,
+      colorAccessor,
+      isUserDefined,
+      isDiffExp,
+      logFoldChange,
+      pval,
+      pvalAdj,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor,
+      zebra
+    } = this.props;
+
+    return (
+      <div
+        id={`histogram_${field}`}
+        style={{
+          padding: globals.leftSidebarSectionPadding,
+          backgroundColor: zebra ? globals.lightestGrey : "white"
+        }}
+      >
+        <div style={{ display: "flex", justifyContent: "flex-end" }}>
+          {isDiffExp ? (
+            <span>
+              <span
+                style={{ marginRight: 7 }}
+                className="bp3-icon-standard bp3-icon-scatter-plot"
+              />
+              <ButtonGroup style={{ marginRight: 7 }}>
+                <Button
+                  onClick={this.handleSetGeneAsScatterplotX(field).bind(this)}
+                  active={scatterplotXXaccessor === field}
+                  intent={scatterplotXXaccessor === field ? "primary" : "none"}
+                >
+                  plot x
+                </Button>
+                <Button
+                  onClick={this.handleSetGeneAsScatterplotY(field).bind(this)}
+                  active={scatterplotYYaccessor === field}
+                  intent={scatterplotYYaccessor === field ? "primary" : "none"}
+                >
+                  plot y
+                </Button>
+              </ButtonGroup>
+            </span>
+          ) : null}
+          {isUserDefined ? (
+            <Button
+              minimal
+              onClick={this.removeHistogram.bind(this)}
+              style={{
+                color: globals.blue,
+                cursor: "pointer",
+                marginLeft: 7
+              }}
+            >
+              remove
+            </Button>
+          ) : null}
+          <Tooltip content="Use as color scale" position="bottom">
+            <Button
+              onClick={this.handleColorAction.bind(this)}
+              active={colorAccessor === field}
+              intent={colorAccessor === field ? "primary" : "none"}
+              icon="tint"
+            />
+          </Tooltip>
+        </div>
+        <svg
+          width={this.width}
+          height={this.height}
+          id={`histogram_${field}_svg`}
+          ref={svgRef => {
+            this.drawHistogram(svgRef);
+          }}
+        />
+        <div
+          style={{
+            display: "flex",
+            justifyContent: "center"
+          }}
+        >
+          <span style={{ fontStyle: "italic" }}>{field}</span>
+        </div>
+
+        {isDiffExp ? (
+          <div
+            style={{
+              display: "flex",
+              justifyContent: "center",
+              alignItems: "baseline"
+            }}
+          >
+            <span>
+              <strong>log fold change:</strong>
+              {` ${logFoldChange.toPrecision(4)}`}
+            </span>
+            <span
+              style={{
+                marginLeft: 7,
+                padding: 2
+              }}
+            >
+              <strong>p-value (adj):</strong>
+              {pvalAdj < 0.0001 ? " < 0.0001" : ` ${pvalAdj.toFixed(4)}`}
+            </span>
+          </div>
+        ) : null}
+      </div>
+    );
+  }
+}
+
+export default HistogramBrush;

client/src/components/categorical/categorical.js

@@ -0,0 +1,37 @@
+// jshint esversion: 6
+import React from "react";
+import _ from "lodash";
+import { connect } from "react-redux";
+import * as globals from "../../globals";
+import Category from "./category";
+
+@connect(state => ({
+  categoricalSelectionState: state.controls.categoricalSelectionState
+}))
+class Categories extends React.Component {
+  render() {
+    const { categoricalSelectionState } = this.props;
+    if (!categoricalSelectionState) return null;
+
+    return (
+      <div
+        style={{
+          padding: globals.leftSidebarSectionPadding
+        }}
+      >
+        <p
+          style={Object.assign({}, globals.leftSidebarSectionHeading, {
+            marginTop: 4
+          })}
+        >
+          Categorical Metadata
+        </p>
+        {_.map(categoricalSelectionState, (catState, catName) => (
+          <Category key={catName} metadataField={catName} />
+        ))}
+      </div>
+    );
+  }
+}
+
+export default Categories;

client/src/components/categorical/category.js

@@ -0,0 +1,183 @@
+import React from "react";
+import _ from "lodash";
+import { connect } from "react-redux";
+import { FaChevronRight, FaChevronDown } from "react-icons/fa";
+import { Button, Tooltip } from "@blueprintjs/core";
+
+import * as globals from "../../globals";
+import Value from "./value";
+import alphabeticallySortedValues from "./util";
+
+@connect(state => ({
+  colorAccessor: state.controls.colorAccessor,
+  categoricalSelectionState: state.controls.categoricalSelectionState
+}))
+class Category extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      isChecked: true,
+      isExpanded: false
+    };
+  }
+
+  componentDidUpdate() {
+    const { categoricalSelectionState, metadataField } = this.props;
+    const cat = categoricalSelectionState[metadataField];
+    const categoryCount = {
+      // total number of options in this category
+      totalOptionCount: cat.numOptions,
+      // number of selected options in this category
+      selectedOptionCount: _.reduce(
+        cat.optionSelected,
+        (res, cond) => (cond ? res + 1 : res),
+        0
+      )
+    };
+    if (categoryCount.selectedOptionCount === categoryCount.totalOptionCount) {
+      /* everything is on, so not indeterminate */
+      this.checkbox.indeterminate = false;
+    } else if (categoryCount.selectedOptionCount === 0) {
+      /* nothing is on, so no */
+      this.checkbox.indeterminate = false;
+    } else if (
+      categoryCount.selectedOptionCount < categoryCount.totalOptionCount
+    ) {
+      /* to be explicit... */
+      this.checkbox.indeterminate = true;
+    }
+  }
+
+  handleColorChange() {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "color by categorical metadata",
+      colorAccessor: metadataField
+    });
+  }
+
+  toggleAll() {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "categorical metadata filter all of these",
+      metadataField
+    });
+    this.setState({ isChecked: true });
+  }
+
+  toggleNone() {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "categorical metadata filter none of these",
+      metadataField
+    });
+    this.setState({ isChecked: false });
+  }
+
+  handleToggleAllClick() {
+    const { isChecked } = this.state;
+    // || this.checkbox.indeterminate === false
+    if (isChecked) {
+      this.toggleNone();
+    } else if (!isChecked) {
+      this.toggleAll();
+    }
+  }
+
+  renderCategoryItems() {
+    const { categoricalSelectionState, metadataField } = this.props;
+
+    const cat = categoricalSelectionState[metadataField];
+    const optTuples = alphabeticallySortedValues([...cat.optionIndex]);
+    return _.map(optTuples, (tuple, i) => (
+      <Value
+        key={tuple[1]}
+        metadataField={metadataField}
+        optionIndex={tuple[1]}
+        i={i}
+      />
+    ));
+  }
+
+  render() {
+    const { isExpanded, isChecked } = this.state;
+    const {
+      metadataField,
+      colorAccessor,
+      categoricalSelectionState
+    } = this.props;
+    const { isTruncated } = categoricalSelectionState[metadataField];
+    return (
+      <div
+        style={{
+          maxWidth: globals.maxControlsWidth
+        }}
+      >
+        <div
+          style={{
+            display: "flex",
+            justifyContent: "space-between",
+            alignItems: "baseline"
+          }}
+        >
+          <div
+            style={{
+              display: "flex",
+              justifyContent: "flex-start",
+              alignItems: "baseline"
+            }}
+          >
+            <label className="bp3-control bp3-checkbox">
+              <input
+                onChange={this.handleToggleAllClick.bind(this)}
+                ref={el => {
+                  this.checkbox = el;
+                  return el;
+                }}
+                checked={isChecked}
+                type="checkbox"
+              />
+              <span className="bp3-control-indicator" />
+              {""}
+            </label>
+
+            <span
+              style={{
+                cursor: "pointer",
+                display: "inline-block"
+              }}
+              onClick={() => {
+                this.setState({ isExpanded: !isExpanded });
+              }}
+            >
+              {metadataField}
+              {isExpanded ? (
+                <FaChevronDown style={{ fontSize: 10, marginLeft: 5 }} />
+              ) : (
+                <FaChevronRight style={{ fontSize: 10, marginLeft: 5 }} />
+              )}
+            </span>
+          </div>
+          <Tooltip content="Use as color scale" position="bottom">
+            <Button
+              onClick={this.handleColorChange.bind(this)}
+              active={colorAccessor === metadataField}
+              intent={colorAccessor === metadataField ? "primary" : "none"}
+              icon={"tint"}
+            />
+          </Tooltip>
+        </div>
+        <div style={{ marginLeft: 26 }}>
+          {isExpanded ? this.renderCategoryItems() : null}
+        </div>
+        <div>
+          {isExpanded && isTruncated ? (
+            <p style={{ paddingLeft: 15 }}>... truncated list ...</p>
+          ) : null}
+        </div>
+      </div>
+    );
+  }
+}
+
+export default Category;

client/src/components/categorical/util.js

@@ -0,0 +1,11 @@
+// jshint esversion: 6
+
+// values is [ [optVal, optIdx], ...]
+// index is range array
+// return sorted index
+export default values =>
+  values.sort((a, b) => {
+    const textA = String(a[0]).toUpperCase();
+    const textB = String(b[0]).toUpperCase();
+    return textA < textB ? -1 : textA > textB ? 1 : 0;
+  });

client/src/components/categorical/value.js

@@ -0,0 +1,107 @@
+// jshint esversion: 6
+import { connect } from "react-redux";
+import React from "react";
+import _ from "lodash";
+
+@connect(state => ({
+  categoricalSelectionState: state.controls.categoricalSelectionState,
+  colorScale: state.controls.colorScale,
+  colorAccessor: state.controls.colorAccessor,
+  schema: _.get(state.controls.world, "schema", null)
+}))
+class CategoryValue extends React.Component {
+  toggleOff() {
+    const { dispatch, metadataField, optionIndex } = this.props;
+    dispatch({
+      type: "categorical metadata filter deselect",
+      metadataField,
+      optionIndex
+    });
+  }
+
+  toggleOn() {
+    const { dispatch, metadataField, optionIndex } = this.props;
+    dispatch({
+      type: "categorical metadata filter select",
+      metadataField,
+      optionIndex
+    });
+  }
+
+  render() {
+    const {
+      categoricalSelectionState,
+      metadataField,
+      optionIndex,
+      colorAccessor,
+      colorScale,
+      i,
+      schema
+    } = this.props;
+
+    if (!categoricalSelectionState) return null;
+
+    const category = categoricalSelectionState[metadataField];
+    const selected = category.optionSelected[optionIndex];
+    const count = category.optionCount[optionIndex];
+    const value = category.optionValue[optionIndex];
+    const displayString = String(category.optionValue[optionIndex]).valueOf();
+
+    /* this is the color scale, so add swatches below */
+    const c = metadataField === colorAccessor;
+    let categories = null;
+
+    if (c && schema) {
+      categories = _.filter(schema.annotations.obs, {
+        name: colorAccessor
+      })[0].categories;
+    }
+
+    return (
+      <div
+        key={i}
+        style={{
+          display: "flex",
+          alignItems: "baseline",
+          justifyContent: "space-between"
+        }}
+      >
+        <div
+          style={{
+            margin: 0,
+            padding: 0,
+            userSelect: "none"
+          }}
+        >
+          <label className="bp3-control bp3-checkbox">
+            <input
+              onChange={
+                selected ? this.toggleOff.bind(this) : this.toggleOn.bind(this)
+              }
+              checked={selected}
+              type="checkbox"
+            />
+            <span className="bp3-control-indicator" />
+            {displayString}
+          </label>
+        </div>
+        <span>
+          <span>{count}</span>
+          <svg
+            style={{
+              marginLeft: 5,
+              width: 11,
+              height: 11,
+              backgroundColor:
+                c && categories
+                  ? colorScale(categories.indexOf(value))
+                  : "inherit"
+            }}
+          />
+        </span>
+      </div>
+    );
+  }
+}
+
+export default CategoryValue;

client/src/components/continuous/continuous.js

@@ -0,0 +1,89 @@
+// jshint esversion: 6
+/* rc slider https://www.npmjs.com/package/rc-slider */
+
+import React from "react";
+import _ from "lodash";
+import { connect } from "react-redux";
+import * as globals from "../../globals";
+
+import HistogramBrush from "../brushableHistogram";
+
+@connect(state => ({
+  ranges: _.get(state.controls.world, "summary.obs", null),
+  metadata: _.get(state.controls.world, "obsAnnotations", null),
+  colorAccessor: state.controls.colorAccessor,
+  colorScale: state.controls.colorScale,
+  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null),
+  schema: _.get(state.controls.world, "schema", null)
+}))
+class Continuous extends React.Component {
+  constructor(props) {
+    super(props);
+    this.hasContinuous = false;
+    this.continuousChecked = false;
+
+    this.state = {};
+  }
+
+  componentDidUpdate() {}
+
+  handleColorAction(key) {
+    return () => {
+      const { dispatch, ranges } = this.props;
+      dispatch({
+        type: "color by continuous metadata",
+        colorAccessor: key,
+        rangeMaxForColorAccessor: ranges[key].range.max
+      });
+    };
+  }
+
+  render() {
+    const { ranges, obsAnnotations, schema } = this.props;
+    if (schema && !this.continuousChecked) {
+      this.hasContinuous = _.some(
+        schema.annotations.obs,
+        d => d.type === "int32" || d.type === "float32"
+      );
+      this.continuousChecked = true; /* only do this once */
+    }
+
+    /* initial value for iterator to simulate index, ranges is an object */
+    let zebra = -1;
+
+    return (
+      <div>
+        {this.hasContinuous ? (
+          <p
+            style={Object.assign({}, globals.leftSidebarSectionHeading, {
+              marginTop: 40,
+              paddingLeft: globals.leftSidebarSectionPadding
+            })}
+          >
+            Continuous metadata
+          </p>
+        ) : null}
+        {_.map(ranges, (value, key) => {
+          const isColorField = key.includes("color") || key.includes("Color");
+          zebra += 1;
+          if (value.range && key !== "name" && !isColorField) {
+            return (
+              <HistogramBrush
+                key={key}
+                field={key}
+                isObs
+                zebra={zebra % 2 === 0}
+                fieldValues={obsAnnotations}
+                ranges={value.range}
+                handleColorAction={this.handleColorAction(key).bind(this)}
+              />
+            );
+          }
+          return null;
+        })}
+      </div>
+    );
+  }
+}
+
+export default Continuous;

client/src/components/continuous/drawAxes.js

@@ -1,4 +1,5 @@
 // jshint esversion: 6
+import * as d3 from "d3";
 import styles from "./parallelCoordinates.css";
 import { yAxis, brushstart } from "./util";
 
@@ -19,13 +20,11 @@ const drawAxes = (
   ******************************************/
 
   function brush() {
-    var actives = [];
+    const actives = [];
     svg
       .selectAll(".parcoords_axis .parcoords_brush")
-      .filter(function(d) {
+      .filter(() => d3.brushSelection(this))
-        return d3.brushSelection(this);
+      .each(d => {
-      })
-      .each(function(d) {
         actives.push({
           dimension: d,
           extent: d3.brushSelection(this)
@@ -35,20 +34,18 @@ const drawAxes = (
     handleBrushAction(actives);
   }
 
-  var axes = svg
+  const axes = svg
     .selectAll(".parcoords_axis")
     .data(dimensions)
     .enter()
     .append("g")
     .attr("class", `${styles.axis} parcoords_axis`)
-    .attr("transform", (d, i) => {
+    .attr("transform", (d, i) => `translate(${xscale(i)})`);
-      return "translate(" + xscale(i) + ")";
-    });
 
   axes
     .append("g")
-    .each(function(d) {
+    .each(d => {
-      var renderAxis =
+      const renderAxis =
         "axis" in d
           ? d.axis.scale(d.scale) // custom axis
           : yAxis.scale(d.scale); // default axis
@@ -60,15 +57,13 @@ const drawAxes = (
     })
     .attr("class", styles.title)
     .attr("text-anchor", "start")
-    .text(function(d) {
+    .text(d => ("description" in d ? `${d.description}  🖌️` : `${d.key}  🖌️`));
-      return "description" in d ? d.description + "  🖌️" : d.key + "  🖌️";
-    });
 
   // Add and store a brush for each axis.
   axes
     .append("g")
     .attr("class", `${styles.brush} parcoords_brush`)
-    .each(function(d) {
+    .each(d => {
       d3.select(this).call(
         (d.brush = d3
           .brushY()

client/src/components/continuous/parallelCoordinates.css

@@ -7,24 +7,24 @@ body {
 }
 */
 
-.parcoords {
+:local(.parcoords) {
   display: block;
 }
 
-.parcoords svg,
+:local(.parcoords) svg,
-.parcoords canvas {
+:local(.parcoords) canvas {
   font: 10px sans-serif;
   position: absolute;
 }
 
-.parcoords canvas {
+:local(.parcoords) canvas {
   opacity: 0.9;
   pointer-events: none;
 }
 
-.axis .title {
+:local(.axis .title) {
   font-size: 10px;
-  transform: rotate(-21deg) translate(-5px,-6px);
+  transform: rotate(-21deg) translate(-5px, -6px);
   fill: #222;
   cursor: pointer;
 }
@@ -32,14 +32,14 @@ body {
 /* -webkit-filter: grayscale(100%);
  filter: grayscale(100%); */
 
-.axis line,
+:local(.axis) line,
-.axis path {
+:local(.axis) path {
   fill: none;
   stroke: #ccc;
   stroke-width: 1px;
 }
 
-.axis .tick text {
+:local(.axis .tick) text {
   fill: #222;
   pointer-events: none;
 }
@@ -52,39 +52,39 @@ old, from reference bl.ocks
 }
 */
 
-.axis:hover line,
+:local(.axis:hover) line,
-.axis:hover path,
+:local(.axis:hover) path,
-.axis.active line,
+:local(.axis.active) line,
-.axis.active path {
+:local(.axis.active) path {
   fill: none;
   stroke: #222;
   stroke-width: 1px;
 }
 
-.axis:hover .title {
+:local(.axis:hover .title) {
   font-weight: bold;
 }
 
-.axis:hover .tick text {
+:local(.axis:hover .tick) text {
   opacity: 1;
 }
 
-.axis.active .title {
+:local(.axis.active .title) {
   font-weight: bold;
 }
 
-.axis.active .tick text {
+:local(.axis.active .tick) text {
   opacity: 1;
   font-weight: bold;
 }
 
-.brush .extent {
+:local(.brush .extent) {
-  fill-opacity: .3;
+  fill-opacity: 0.3;
   stroke: #fff;
   stroke-width: 1px;
 }
 
-.pre {
+:local(.pre) {
   width: 100%;
   height: 300px;
   margin: 6px 12px;

client/src/components/continuous/setupParallelCoordinates.js

@@ -4,26 +4,28 @@
 ******************************************
 ******************************************/
 // jshint esversion: 6
-const setupParallelCoordinates = (width, height, margin) => {
+import * as d3 from "d3";
-  var container = d3.select("#parcoords");
 
-  var svg = container
+const setupParallelCoordinates = (width, height, margin) => {
+  const container = d3.select("#parcoords");
+
+  const svg = container
     .append("svg")
     .attr("width", width + margin.left + margin.right)
     .attr("height", height + margin.top + margin.bottom)
     .append("g")
-    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");
+    .attr("transform", `translate(${margin.left},${margin.top})`);
 
-  var canvas = container
+  const canvas = container
     .append("canvas")
     .attr("width", width * devicePixelRatio)
     .attr("height", height * devicePixelRatio)
-    .style("width", width + "px")
+    .style("width", `${width}px`)
-    .style("height", height + "px")
+    .style("height", `${height}px`)
-    .style("margin-top", margin.top + "px")
+    .style("margin-top", `${margin.top}px`)
-    .style("margin-left", margin.left + "px");
+    .style("margin-left", `${margin.left}px`);
 
-  var ctx = canvas.node().getContext("2d");
+  const ctx = canvas.node().getContext("2d");
   ctx.globalCompositeOperation = "darken";
   ctx.globalAlpha = 0.15;
   ctx.lineWidth = 1.5;

client/src/components/continuous/util.js

@@ -1,5 +1,6 @@
 // jshint esversion: 6
 import _ from "lodash";
+import * as d3 from "d3";
 
 const paddingRight = 120;
 const continuousChartWidth = 1200;
@@ -17,11 +18,10 @@ export const createDimensions = data => {
   _.each(data, (value, key) => {
     if (value.range) {
       newArr.push({
-        key: key /* room for confusion: lodash calls this key, it's also the name of the property parallel coords code is looking for */,
+        key /* room for confusion: lodash calls this key, it's also the name of the property parallel coords code is looking for */,
         type: {
-          within: (d, extent, dim) => {
+          within: (d, extent, dim) =>
-            return extent[0] <= dim.scale(d) && dim.scale(d) <= extent[1];
+            extent[0] <= dim.scale(d) && dim.scale(d) <= extent[1]
-          }
         },
         scale: d3
           .scaleSqrt()
@@ -39,19 +39,12 @@ export const brushstart = () => {
   d3.event.sourceEvent.stopPropagation();
 };
 
-export const d3_functor = v => {
+// Unused.
-  return typeof v === "function"
+// export const d3_functor = v => (typeof v === "function" ? v : () => v);
-    ? v
+export const project = (d, dimensions, xscale) =>
-    : () => {
+  dimensions.map((p, i) => {
-        return v;
-      };
-};
-
-export const project = (d, dimensions, xscale) => {
-  return dimensions.map((p, i) => {
     // check if data element has property and contains a value
     if (!(p.key in d) || d[p.key] === null) return null;
 
     return [xscale(i), p.scale(d[p.key])];
   });
-};

client/src/components/continuousLegend/index.js

@@ -0,0 +1,152 @@
+// jshint esversion: 6
+import React from "react";
+import { connect } from "react-redux";
+import * as d3 from "d3";
+import { interpolateViridis, interpolateCool } from "d3-scale-chromatic";
+
+// create continuous color legend
+// http://bl.ocks.org/syntagmatic/e8ccca52559796be775553b467593a9f
+const continuous = (selectorId, colorscale, colorAccessor) => {
+  const legendheight = 200;
+  const legendwidth = 80;
+  const margin = { top: 10, right: 60, bottom: 10, left: 2 };
+
+  const canvas = d3
+    .select(selectorId)
+    .style("height", `${legendheight}px`)
+    .style("width", `${legendwidth}px`)
+    .append("canvas")
+    .attr("height", legendheight - margin.top - margin.bottom)
+    .attr("width", 1)
+    .style("height", `${legendheight - margin.top - margin.bottom}px`)
+    .style("width", `${legendwidth - margin.left - margin.right}px`)
+    .style("position", "absolute")
+    .style("top", `${margin.top + 1}px`)
+    .style("left", `${margin.left + 1}px`)
+    .style(
+      "transform",
+      "scale(1,-1)"
+    ) /* flip it! dark is high value light is low.
+    we flip the color scale as well [1, 0] instead of [0, 1] */
+    .node();
+
+  const ctx = canvas.getContext("2d");
+
+  const legendscale = d3
+    .scaleLinear()
+    .range([1, legendheight - margin.top - margin.bottom])
+    .domain([
+      colorscale.domain()[1],
+      colorscale.domain()[0]
+    ]); /* we flip this to make viridis colors dark if high in the color scale */
+
+  // image data hackery based on http://bl.ocks.org/mbostock/048d21cf747371b11884f75ad896e5a5
+  const image = ctx.createImageData(1, legendheight);
+  d3.range(legendheight).forEach(i => {
+    const c = d3.rgb(colorscale(legendscale.invert(i)));
+    image.data[4 * i] = c.r;
+    image.data[4 * i + 1] = c.g;
+    image.data[4 * i + 2] = c.b;
+    image.data[4 * i + 3] = 255;
+  });
+  ctx.putImageData(image, 0, 0);
+
+  // A simpler way to do the above, but possibly slower. keep in mind the legend
+  // width is stretched because the width attr of the canvas is 1
+  // See http://stackoverflow.com/questions/4899799/whats-the-best-way-to-set-a-single-pixel-in-an-html5-canvas
+  /*
+  d3.range(legendheight).forEach(function(i) {
+    ctx.fillStyle = colorscale(legendscale.invert(i));
+    ctx.fillRect(0,i,1,1);
+  });
+  */
+
+  const legendaxis = d3
+    .axisRight()
+    .scale(legendscale)
+    .tickSize(6)
+    .ticks(8);
+
+  const svg = d3
+    .select(selectorId)
+    .append("svg")
+    .attr("height", `${legendheight}px`)
+    .attr("width", `${legendwidth}px`)
+    .style("position", "absolute")
+    .style("left", "0px")
+    .style("top", "0px");
+
+  svg
+    .append("g")
+    .attr("class", "axis")
+    .attr(
+      "transform",
+      `translate(${legendwidth - margin.left - margin.right + 3},${margin.top})`
+    )
+    .call(legendaxis);
+
+  // text label for the y axis
+  svg
+    .append("text")
+    .attr("transform", "rotate(-90)")
+    .attr("y", 2)
+    .attr("x", 0 - legendheight / 2)
+    .attr("dy", "1em")
+    .style("text-anchor", "middle")
+    .style("fill", "white")
+    .text(colorAccessor);
+};
+
+@connect(state => ({
+  colorAccessor: state.controls.colorAccessor,
+  colorScale: state.controls.colorScale,
+  responsive: state.responsive
+}))
+class ContinuousLegend extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {};
+  }
+
+  componentDidUpdate(prevProps) {
+    const { colorAccessor, responsive, colorScale } = this.props;
+    if (
+      prevProps.colorAccessor !== colorAccessor ||
+      prevProps.responsive.height !== responsive.height ||
+      prevProps.responsive.width !== responsive.width
+    ) {
+      /* always remove it, if it's not continuous we don't put it back. */
+      d3.select("#continuous_legend")
+        .selectAll("*")
+        .remove();
+    }
+
+    if (colorAccessor && colorScale && colorScale.range) {
+      /* fragile! continuous range is 0 to 1, not [#fa4b2c, ...], make this a flag? */
+      if (colorScale.range()[0][0] !== "#") {
+        continuous(
+          "#continuous_legend",
+          d3.scaleSequential(interpolateCool).domain(colorScale.domain()),
+          colorAccessor
+        );
+      }
+    }
+  }
+
+  render() {
+    const { colorAccessor, responsive } = this.props;
+    return (
+      <div
+        id="continuous_legend"
+        style={{
+          position: "fixed",
+          display: colorAccessor ? "inherit" : "none",
+          right: 0,
+          top: responsive.height / 2
+        }}
+      />
+    );
+  }
+}
+
+export default ContinuousLegend;

client/src/components/framework/buttons.css

@@ -1,4 +1,4 @@
-.btn {
+:local(.btn) {
   background: none;
   cursor: pointer;
   color: "#FFF";

client/src/components/framework/container.css

@@ -1,3 +1,3 @@
-.container {
+:local(.container) {
   min-height: 100%;
 }

client/src/components/framework/container.js

@@ -0,0 +1,11 @@
+// jshint esversion: 6
+import React from "react";
+
+import styles from "./container.css";
+
+const Container = props => {
+  const { children } = props;
+  return <div className={styles.container}>{children}</div>;
+};
+
+export default Container;

client/src/components/framework/toasters.js

@@ -0,0 +1,24 @@
+import { Position, Toaster, Intent } from "@blueprintjs/core";
+
+/** Singleton toaster instance. Create separate instances for different options. */
+
+const ErrorToastTopCenter = Toaster.create({
+  className: "recipe-toaster",
+  position: Position.TOP
+});
+
+/*
+A "user" error - eg, bad input
+*/
+export const postUserErrorToast = message =>
+  ErrorToastTopCenter.show({ message, intent: Intent.WARNING });
+
+/*
+a hard network error
+*/
+export const postNetworkErrorToast = message =>
+  ErrorToastTopCenter.show({
+    message,
+    timeout: 30000,
+    intent: Intent.DANGER
+  });

client/src/components/geneExpression/cellSetButtons.js

@@ -0,0 +1,52 @@
+// jshint esversion: 6
+import React from "react";
+import _ from "lodash";
+import { Button, Tooltip } from "@blueprintjs/core";
+import { connect } from "react-redux";
+
+@connect()
+class CellSetButton extends React.Component {
+  set() {
+    const {
+      differential,
+      crossfilter,
+      dispatch,
+      eitherCellSetOneOrTwo
+    } = this.props;
+
+    const set = _.map(crossfilter.allFiltered(), "name");
+
+    if (!differential.diffExp) {
+      /* diffexp needs to be cleared before we store a new set */
+      dispatch({
+        type: `store current cell selection as differential set ${eitherCellSetOneOrTwo}`,
+        data: set
+      });
+    }
+  }
+
+  render() {
+    const { differential, eitherCellSetOneOrTwo } = this.props;
+    const cellListName = `celllist${eitherCellSetOneOrTwo}`;
+    return (
+      <Tooltip
+        content="Save current selection for differential expression computation"
+        position="top"
+      >
+        <Button
+          style={{ marginRight: 10 }}
+          type="button"
+          onClick={this.set.bind(this)}
+        >
+          {eitherCellSetOneOrTwo}
+          {": "}
+          {differential[cellListName]
+            ? `${differential[cellListName].length} cells`
+            : "0 cells"}
+        </Button>
+      </Tooltip>
+    );
+  }
+}
+
+export default CellSetButton;

client/src/components/geneExpression/expressionButtons.js

@@ -0,0 +1,100 @@
+// jshint esversion: 6
+import React from "react";
+import _ from "lodash";
+import { Button, AnchorButton, Tooltip } from "@blueprintjs/core";
+import { connect } from "react-redux";
+import * as globals from "../../globals";
+import actions from "../../actions";
+import CellSetButton from "./cellSetButtons";
+
+@connect(state => ({
+  differential: state.differential,
+  world: state.controls.world,
+  crossfilter: state.controls.crossfilter,
+  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+}))
+class Expression extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {};
+  }
+
+  computeDiffExp() {
+    const { dispatch, differential } = this.props;
+    if (differential.celllist1 && differential.celllist2) {
+      dispatch(
+        actions.requestDifferentialExpression(
+          differential.celllist1,
+          differential.celllist2
+        )
+      );
+    }
+  }
+
+  clearDifferentialExpression() {
+    const { dispatch, differential } = this.props;
+    dispatch({
+      type: "clear differential expression",
+      diffExp: differential.diffExp
+    });
+    dispatch({
+      type: "clear scatterplot"
+    });
+  }
+
+  render() {
+    const { differential } = this.props;
+    if (!differential) {
+      return null;
+    }
+    const haveBothCellSets =
+      !!differential.celllist1 && !!differential.celllist2;
+    return (
+      <div
+        style={{
+          marginRight: 10,
+          marginBottom: 10,
+          paddingLeft: globals.leftSidebarSectionPadding
+        }}
+      >
+        <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
+        <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
+        {!differential.diffExp ? (
+          <Tooltip
+            content="Add two cells selections, see the top 15 differentially expressed genes between them"
+            position="bottom"
+          >
+            <AnchorButton
+              style={{ marginTop: 10 }}
+              disabled={!haveBothCellSets}
+              intent="primary"
+              fill
+              type="button"
+              onClick={this.computeDiffExp.bind(this)}
+            >
+              Compute Differential Expression
+            </AnchorButton>
+          </Tooltip>
+        ) : null}
+        {differential.diffExp ? (
+          <Tooltip
+            content="Remove differentially expressed gene list and clear cell selections"
+            position="bottom"
+          >
+            <Button
+              type="button"
+              fill
+              style={{ marginTop: 10 }}
+              intent="warning"
+              onClick={this.clearDifferentialExpression.bind(this)}
+            >
+              Clear Differential Expression
+            </Button>
+          </Tooltip>
+        ) : null}
+      </div>
+    );
+  }
+}
+
+export default Expression;

client/src/components/geneExpression/index.js

@@ -0,0 +1,168 @@
+// jshint esversion: 6
+/* rc slider https://www.npmjs.com/package/rc-slider */
+
+import React from "react";
+import _ from "lodash";
+import * as d3 from "d3";
+import { connect } from "react-redux";
+import { Button, Tooltip } from "@blueprintjs/core";
+import HistogramBrush from "../brushableHistogram";
+import * as globals from "../../globals";
+import actions from "../../actions";
+import { postUserErrorToast } from "../framework/toasters";
+import ExpressionButtons from "./expressionButtons";
+
+@connect(state => {
+  const metadata = _.get(state.controls.world, "obsAnnotations", null);
+  const ranges = _.get(state.controls.world, "summary.obs", null);
+  const initializeRanges = _.get(state.controls.world, "summary.obs");
+
+  return {
+    ranges,
+    metadata,
+    initializeRanges,
+    userDefinedGenes: state.controls.userDefinedGenes,
+    world: state.controls.world,
+    colorAccessor: state.controls.colorAccessor,
+    allGeneNames: state.controls.allGeneNames,
+    differential: state.differential
+  };
+})
+class GeneExpression extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      gene: ""
+    };
+  }
+
+  keyPress(e) {
+    if (e.keyCode === 13) {
+      this.handleClick();
+    }
+  }
+
+  handleClick() {
+    const { world, dispatch, userDefinedGenes } = this.props;
+    const { gene } = this.state;
+
+    if (userDefinedGenes.indexOf(gene) !== -1) {
+      postUserErrorToast("That gene already exists");
+    } else if (userDefinedGenes.length > 15) {
+      postUserErrorToast(
+        "That's too many genes, you can have at most 15 user defined genes"
+      );
+    } else if (!_.find(world.varAnnotations, { name: gene })) {
+      postUserErrorToast("That doesn't appear to be a valid gene name.");
+    } else {
+      dispatch(actions.requestUserDefinedGene(gene));
+      dispatch({
+        type: "user defined gene",
+        data: gene
+      });
+      this.setState({ gene: "" });
+    }
+  }
+
+  render() {
+    const { world, userDefinedGenes, differential } = this.props;
+    const { gene } = this.state;
+
+    return (
+      <div>
+        <div
+          style={{
+            marginTop: 30
+          }}
+        >
+          <p
+            style={Object.assign({}, globals.leftSidebarSectionHeading, {
+              paddingLeft: globals.leftSidebarSectionPadding,
+              margin: 0
+            })}
+          >
+            Selected Genes
+          </p>
+          <div
+            style={{ padding: globals.leftSidebarSectionPadding }}
+            className="bp3-control-group"
+          >
+            <div className="bp3-input-group bp3-fill">
+              <input
+                onKeyDown={this.keyPress.bind(this)}
+                onChange={e => {
+                  this.setState({ gene: e.target.value });
+                }}
+                value={gene}
+                type="text"
+                className="bp3-input"
+                placeholder="Enter a gene name"
+                style={{ paddingRight: 94 }}
+              />
+            </div>
+            <Tooltip
+              content="Add a gene to see its expression levels"
+              position="bottom"
+            >
+              <Button intent="primary" onClick={this.handleClick.bind(this)}>
+                Add
+              </Button>
+            </Tooltip>
+          </div>
+          {world && userDefinedGenes.length > 0
+            ? _.map(userDefinedGenes, (geneName, index) => {
+                const values = world.varDataCache[geneName];
+                if (!values) {
+                  return null;
+                }
+                return (
+                  <HistogramBrush
+                    key={geneName}
+                    field={geneName}
+                    zebra={index % 2 === 0}
+                    ranges={d3.extent(values)}
+                    isUserDefined
+                  />
+                );
+              })
+            : null}
+        </div>
+        <div>
+          <p
+            style={Object.assign({}, globals.leftSidebarSectionHeading, {
+              marginTop: 40,
+              paddingLeft: globals.leftSidebarSectionPadding
+            })}
+          >
+            Differentially Expressed Genes
+          </p>
+          <ExpressionButtons />
+          {differential.diffExp
+            ? _.map(differential.diffExp, (value, index) => {
+                const annotations = world.varAnnotations[value[0]];
+                const { name } = annotations;
+                const values = world.varDataCache[name];
+                if (!values) {
+                  return null;
+                }
+                return (
+                  <HistogramBrush
+                    key={name}
+                    field={name}
+                    zebra={index % 2 === 0}
+                    ranges={d3.extent(values)}
+                    isDiffExp
+                    logFoldChange={value[1]}
+                    pval={value[2]}
+                    pvalAdj={value[3]}
+                  />
+                );
+              })
+            : null}
+        </div>
+      </div>
+    );
+  }
+}
+
+export default GeneExpression;

client/src/components/graph/drawPointsRegl.js

@@ -38,15 +38,7 @@ export default function(regl) {
     uniforms: {
       distance: regl.prop("distance"),
       view: regl.prop("view"),
-      projection: (context, props) => {
+      projection: ({viewportWidth, viewportHeight}) => mat4.perspective([], Math.PI / 2, viewportWidth / viewportHeight, 0.01, 1000)
-        return mat4.perspective(
-          [],
-          Math.PI / 2,
-          context.viewportWidth * props.scale / context.viewportHeight,
-          0.01,
-          1000
-        );
-      }
     },
 
     count: regl.prop("count"),

client/src/components/graph/graph.css

@@ -1,4 +1,4 @@
-.graphCanvas path {
+:local(.graphCanvas) path {
   shape-rendering: crispEdges;
 }
 
@@ -16,8 +16,8 @@
   pointer-events: none;
 }
 
-.axis path,
+:local(.axis) path,
-.axis line {
+:local(.axis) line {
   fill: none;
   stroke: #000;
   stroke-width: 1px;
@@ -29,6 +29,6 @@ circle {
   fill: none;
 }
 
-.hidden {
+:local(.hidden) {
   display: none;
 }

client/src/components/graph/graph.js

@@ -0,0 +1,462 @@
+// jshint esversion: 6
+import React from "react";
+import _ from "lodash";
+import * as d3 from "d3";
+import { connect } from "react-redux";
+import mat4 from "gl-mat4";
+import _regl from "regl";
+import { Button, AnchorButton, Tooltip } from "@blueprintjs/core";
+import * as globals from "../../globals";
+import setupSVGandBrushElements from "./setupSVGandBrush";
+import actions from "../../actions";
+import _camera from "../../util/camera";
+import _drawPoints from "./drawPointsRegl";
+import scaleLinear from "../../util/scaleLinear";
+
+/* https://bl.ocks.org/mbostock/9078690 - quadtree for onClick / hover selections */
+
+@connect(state => ({
+  world: state.controls.world,
+  universe: state.controls.universe,
+  crossfilter: state.controls.crossfilter,
+  responsive: state.responsive,
+  colorRGB: _.get(state.controls, "colorRGB", null),
+  opacityForDeselectedCells: state.controls.opacityForDeselectedCells,
+  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+}))
+class Graph extends React.Component {
+  constructor(props) {
+    super(props);
+    this.count = 0;
+    this.inverse = mat4.identity([]);
+    this.graphPaddingTop = 0;
+    this.graphPaddingBottom = 45;
+    this.graphPaddingRight = globals.leftSidebarWidth;
+    this.renderCache = {
+      positions: null,
+      colors: null
+    };
+    this.state = {
+      svg: null,
+      brush: null,
+      mode: "brush"
+    };
+  }
+
+  componentDidMount() {
+    // setup canvas and camera
+    const camera = _camera(this.reglCanvas, { scale: true, rotate: false });
+    const regl = _regl(this.reglCanvas);
+
+    const drawPoints = _drawPoints(regl);
+
+    // preallocate buffers
+    const pointBuffer = regl.buffer();
+    const colorBuffer = regl.buffer();
+    const sizeBuffer = regl.buffer();
+
+    /* first time, but this duplicates above function, should be possile to avoid this */
+    const reglRender = regl.frame(() => {
+      this.reglDraw(
+        regl,
+        drawPoints,
+        sizeBuffer,
+        colorBuffer,
+        pointBuffer,
+        camera
+      );
+      camera.tick();
+    });
+
+    this.reglRenderState = "rendering";
+
+    this.setState({
+      regl,
+      drawPoints,
+      pointBuffer,
+      colorBuffer,
+      sizeBuffer,
+      camera,
+      reglRender
+    });
+  }
+
+  componentDidUpdate(prevProps) {
+    const {
+      world,
+      crossfilter,
+      selectionUpdate,
+      colorRGB,
+      responsive
+    } = this.props;
+    const {
+      reglRender,
+      mode,
+      regl,
+      drawPoints,
+      camera,
+      pointBuffer,
+      colorBuffer,
+      sizeBuffer,
+      svg
+    } = this.state;
+
+    if (reglRender && this.reglRenderState === "rendering" && mode !== "zoom") {
+      reglRender.cancel();
+      this.reglRenderState = "paused";
+    }
+
+    if (regl && world) {
+      /* update the regl state */
+      const { obsLayout } = world;
+      const cellCount = crossfilter.size();
+
+      // X/Y positions for each point - a cached value that only
+      // changes if we have loaded entirely new cell data
+      //
+      if (
+        !this.renderCache.positions ||
+        selectionUpdate !== prevProps.selectionUpdate
+      ) {
+        if (!this.renderCache.positions) {
+          this.renderCache.positions = new Float32Array(2 * cellCount);
+        }
+
+        const glScaleX = scaleLinear([0, 1], [-1, 1]);
+        const glScaleY = scaleLinear([0, 1], [1, -1]);
+
+        const offset = [d3.mean(obsLayout.X) - 0.5, d3.mean(obsLayout.Y) - 0.5];
+
+        for (
+          let i = 0, { positions } = this.renderCache;
+          i < cellCount;
+          i += 1
+        ) {
+          positions[2 * i] = glScaleX(obsLayout.X[i] - offset[0]);
+          positions[2 * i + 1] = glScaleY(obsLayout.Y[i] - offset[1]);
+        }
+        pointBuffer({
+          data: this.renderCache.positions,
+          dimension: 2
+        });
+
+        this.setState({
+          offset
+        });
+      }
+
+      // Colors for each point - a cached value that only changes when
+      // the cell metadata changes (done by updateCellColors middleware).
+      // NOTE: this is a slightly pessimistic assumption, as the metadata
+      // could have changed for some other reason, but for now color is
+      // the only metadata that changes client-side.  If this is problematic,
+      // we could add some sort of color-specific indicator to the app state.
+      if (!this.renderCache.colors || colorRGB !== prevProps.colorRGB) {
+        const rgb = colorRGB;
+        if (!this.renderCache.colors) {
+          this.renderCache.colors = new Float32Array(3 * rgb.length);
+        }
+        for (let i = 0, { colors } = this.renderCache; i < rgb.length; i += 1) {
+          colors.set(rgb[i], 3 * i);
+        }
+        colorBuffer({ data: this.renderCache.colors, dimension: 3 });
+      }
+
+      // Sizes for each point - this is presumed to change each time the
+      // component receives new props.   Almost always a true assumption, as
+      // most property upates are due to changes driving a crossfilter
+      // selection set change.
+      //
+      if (!this.renderCache.sizes) {
+        this.renderCache.sizes = new Float32Array(cellCount);
+      }
+
+      crossfilter.fillByIsFiltered(this.renderCache.sizes, 4, 0.2);
+      sizeBuffer({ data: this.renderCache.sizes, dimension: 1 });
+
+      this.count = cellCount;
+
+      regl._refresh();
+      this.reglDraw(
+        regl,
+        drawPoints,
+        sizeBuffer,
+        colorBuffer,
+        pointBuffer,
+        camera
+      );
+    }
+
+    if (
+      prevProps.responsive.height !== responsive.height ||
+      prevProps.responsive.width !== responsive.width ||
+      /* first time */
+      (responsive.height && responsive.width && !svg)
+    ) {
+      /* clear out whatever was on the div, even if nothing, but usually the brushes etc */
+      d3.select("#graphAttachPoint")
+        .selectAll("svg")
+        .remove();
+      const { svg: newSvg, brush } = setupSVGandBrushElements(
+        this.handleBrushSelectAction.bind(this),
+        this.handleBrushDeselectAction.bind(this),
+        responsive,
+        this.graphPaddingRight
+      );
+      this.setState({ svg: newSvg, brush });
+    }
+  }
+
+  reglDraw(regl, drawPoints, sizeBuffer, colorBuffer, pointBuffer, camera) {
+    regl.clear({
+      depth: 1,
+      color: [1, 1, 1, 1]
+    });
+    drawPoints({
+      size: sizeBuffer,
+      distance: camera.distance,
+      color: colorBuffer,
+      position: pointBuffer,
+      count: this.count,
+      view: camera.view()
+    });
+  }
+
+  restartReglLoop() {
+    const {
+      regl,
+      drawPoints,
+      sizeBuffer,
+      colorBuffer,
+      pointBuffer,
+      camera
+    } = this.state;
+    const reglRender = regl.frame(() => {
+      this.reglDraw(
+        regl,
+        drawPoints,
+        sizeBuffer,
+        colorBuffer,
+        pointBuffer,
+        camera
+      );
+      camera.tick();
+    });
+
+    this.reglRenderState = "rendering";
+
+    this.setState({
+      reglRender
+    });
+  }
+
+  handleBrushSelectAction() {
+    /*
+    This conditional handles procedural brush deselect. Brush emits
+    an event on procedural deselect because it is move: null
+    */
+
+    const { camera, offset } = this.state;
+    const { dispatch, responsive } = this.props;
+
+    if (d3.event.sourceEvent !== null) {
+      /*
+      No idea why d3 event scope works like this
+      but apparently
+      it does
+      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
+    */
+      const s = d3.event.selection;
+      const gl = this.state.regl._gl;
+      /*
+      event describing brush position:
+      @-------|
+      |       |
+      |       |
+      |-------@
+    */
+
+      // get aspect ratio
+      const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
+
+      // compute inverse view matrix
+      const inverse = mat4.invert([], camera.view());
+
+      // transform screen coordinates -> cell coordinates
+      const invert = pin => {
+        const x =
+          (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
+        const y =
+          2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
+        const pout = [
+          x * inverse[14] * aspect + inverse[12],
+          y * inverse[14] + inverse[13]
+        ];
+        return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
+      };
+
+      const brushCoords = {
+        northwest: invert([s[0][0], s[0][1]]),
+        southeast: invert([s[1][0], s[1][1]])
+      };
+
+      dispatch({
+        type: "graph brush selection change",
+        brushCoords
+      });
+    }
+  }
+
+  handleBrushDeselectAction() {
+    const { dispatch } = this.props;
+    const { svg, brush } = this.state;
+
+    if (d3.event && !d3.event.selection) {
+      dispatch({
+        type: "graph brush deselect"
+      });
+    }
+
+    if (!d3.event) {
+      /*
+      this line clears the brush procedurally, ie., zoom button clicked,
+      not a click away from brush on svg
+      */
+      svg.select(".graph_brush").call(brush.move, null);
+      dispatch({
+        type: "graph brush deselect"
+      });
+    }
+  }
+
+  handleOpacityRangeChange(e) {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "change opacity deselected cells in 2d graph background",
+      data: e.target.value
+    });
+  }
+
+  resetInterface() {
+    const { dispatch } = this.props;
+    dispatch(actions.resetInterface());
+  }
+
+  render() {
+    const { dispatch, responsive, crossfilter } = this.props;
+    const { mode } = this.state;
+    return (
+      <div id="graphWrapper">
+        <div
+          style={{
+            position: "fixed",
+            right: 0,
+            top: 0
+          }}
+        >
+          <div
+            style={{
+              padding: 10,
+              display: "flex",
+              justifyContent: "flex-end",
+              alignItems: "baseline"
+            }}
+          >
+            <Tooltip
+              content="Show only metadata and cells which are currently selected"
+              position="left"
+            >
+              <AnchorButton
+                type="button"
+                disabled={
+                  crossfilter &&
+                  (crossfilter.countFiltered() === 0 ||
+                    crossfilter.countFiltered() === crossfilter.size())
+                }
+                style={{ marginRight: 10 }}
+                onClick={() => {
+                  dispatch(actions.regraph());
+                  dispatch({ type: "increment graph render counter" });
+                }}
+              >
+                subset to current selection
+              </AnchorButton>
+            </Tooltip>
+            <Tooltip
+              content="Reset cellxgene, clearing all selections"
+              position="left"
+            >
+              <AnchorButton
+                disabled={
+                  false
+                  /* world && universe ? worldEqUniverse(world, universe) : false */
+                }
+                type="button"
+                intent="warning"
+                style={{ marginRight: 10 }}
+                onClick={this.resetInterface.bind(this)}
+              >
+                reset
+              </AnchorButton>
+            </Tooltip>
+            <div>
+              <div className="bp3-button-group">
+                <Tooltip content="Lasso cells" position="left">
+                  <Button
+                    className="bp3-button bp3-icon-select"
+                    type="button"
+                    active={mode === "brush"}
+                    onClick={() => {
+                      this.setState({ mode: "brush" });
+                    }}
+                  />
+                </Tooltip>
+                <Tooltip content="Pan and zoom" position="left">
+                  <Button
+                    type="button"
+                    className="bp3-button bp3-icon-zoom-in"
+                    active={mode === "zoom"}
+                    onClick={() => {
+                      this.handleBrushDeselectAction();
+                      this.restartReglLoop();
+                      this.setState({ mode: "zoom" });
+                    }}
+                    style={{
+                      cursor: "pointer"
+                    }}
+                  />
+                </Tooltip>
+              </div>
+            </div>
+          </div>
+        </div>
+        <div
+          style={{
+            zIndex: -9999,
+            position: "fixed",
+            top: 0,
+            right: 0
+          }}
+        >
+          <div
+            style={{
+              display: mode === "brush" ? "inherit" : "none"
+            }}
+            id="graphAttachPoint"
+          />
+          <div style={{ padding: 0, margin: 0 }}>
+            <canvas
+              width={responsive.width - this.graphPaddingRight}
+              height={responsive.height - this.graphPaddingTop}
+              ref={canvas => {
+                this.reglCanvas = canvas;
+              }}
+            />
+          </div>
+        </div>
+      </div>
+    );
+  }
+}
+
+export default Graph;

client/src/components/graph/setupSVGandBrush.js

@@ -0,0 +1,40 @@
+// jshint esversion: 6
+import * as d3 from "d3";
+import styles from "./graph.css";
+
+/******************************************
+*******************************************
+          put svg & brush in DOM
+*******************************************
+******************************************/
+
+export default (
+  handleBrushSelectAction,
+  handleBrushDeselectAction,
+  responsive,
+  graphPaddingRight
+) => {
+  const svg = d3
+    .select("#graphAttachPoint")
+    .append("svg")
+    .attr("width", responsive.width - graphPaddingRight)
+    .attr("height", responsive.height)
+    .attr("class", `${styles.graphSVG}`);
+
+  const brush = d3
+    .brush()
+    .extent([[0, 0], [responsive.width - graphPaddingRight, responsive.height]])
+    .on("brush", handleBrushSelectAction)
+    .on("end", handleBrushDeselectAction);
+
+  const brushContainer = svg
+    .append("g")
+    .attr("class", "graph_brush")
+    .call(brush);
+
+  return {
+    svg,
+    brushContainer,
+    brush
+  };
+};

client/src/components/leftsidebar.js

@@ -0,0 +1,76 @@
+// jshint esversion: 6
+import _ from "lodash";
+import React from "react";
+import { connect } from "react-redux";
+import Categorical from "./categorical/categorical";
+import Continuous from "./continuous/continuous";
+import GeneExpression from "./geneExpression";
+import * as globals from "../globals";
+import DynamicScatterplot from "./scatterplot/scatterplot";
+
+@connect(state => ({
+  responsive: state.responsive,
+  datasetTitle: _.get(state.config, "displayNames.dataset"),
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor
+}))
+class LeftSideBar extends React.Component {
+  render() {
+    const {
+      responsive,
+      datasetTitle,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor
+    } = this.props;
+
+    /*
+    this magic number should be made less fragile,
+    if cellxgene logo or tabs change, this must as well
+    */
+    const metadataSectionPadding = 0;
+    // scatterplotXXaccessor && scatterplotYYaccessor ? 450 : 0;
+
+    return (
+      <div
+        style={{
+          position: "fixed",
+          backgroundColor: "white",
+          /* x y blur spread color */
+          boxShadow: "1px 0px 6px 2px rgba(153,153,153,0.4)"
+        }}
+      >
+        <p
+          style={{
+            position: "fixed",
+            top: globals.cellxgeneTitleTopPadding,
+            left: globals.leftSidebarWidth + globals.cellxgeneTitleLeftPadding,
+            margin: 0,
+            fontSize: globals.largestFontSize,
+            color: globals.darkerGrey,
+            width: "100%"
+          }}
+        >
+          cellxgene: {datasetTitle}
+        </p>
+
+        <div
+          style={{
+            height: responsive.height - metadataSectionPadding,
+            width: globals.leftSidebarWidth,
+            overflowY: "auto",
+            overflowX: "hidden"
+          }}
+        >
+          <Categorical />
+          <GeneExpression />
+          <Continuous />
+        </div>
+        {scatterplotXXaccessor && scatterplotYYaccessor ? (
+          <DynamicScatterplot />
+        ) : null}
+      </div>
+    );
+  }
+}
+
+export default LeftSideBar;

client/src/components/scatterplot/drawPointsRegl.js

@@ -38,15 +38,7 @@ export default function(regl) {
     uniforms: {
       distance: regl.prop("distance"),
       view: regl.prop("view"),
-      projection: (context, props) => {
+      projection: () => mat4.perspective([], Math.PI / 2, 1, 0.01, 1000)
-        return mat4.perspective(
-          [],
-          Math.PI / 2,
-          context.viewportWidth * props.scale / context.viewportHeight,
-          0.01,
-          1000
-        );
-      }
     },
 
     count: regl.prop("count"),

client/src/components/scatterplot/scatterplot.css

@@ -1,15 +1,14 @@
-
+:local(.scatterplot) {
-.scatterplot {
   display: block;
 }
 
-.scatterplot svg,
+:local(.scatterplot) svg,
-.scatterplot canvas {
+:local(.scatterplot) canvas {
   font: 10px sans-serif;
   position: absolute;
 }
 
-.scatterplot canvas {
+:local(.scatterplot) canvas {
   opacity: 0.9;
   pointer-events: none;
 }

client/src/components/scatterplot/scatterplot.js

@@ -0,0 +1,377 @@
+// jshint esversion: 6
+// https://bl.ocks.org/Jverma/076377dd0125b1a508621441752735fc
+// https://peterbeshai.com/scatterplot-in-d3-with-voronoi-interaction.html
+
+import React from "react";
+import _ from "lodash";
+import { connect } from "react-redux";
+import { Button, ButtonGroup } from "@blueprintjs/core";
+import _regl from "regl";
+import * as d3 from "d3";
+import * as globals from "../../globals";
+
+import _camera from "../../util/camera";
+
+import setupScatterplot from "./setupScatterplot";
+import styles from "./scatterplot.css";
+
+import _drawPoints from "./drawPointsRegl";
+import scaleLinear from "../../util/scaleLinear";
+
+import { margin, width, height } from "./util";
+import { kvCache } from "../../util/stateManager";
+
+@connect(state => {
+  const {
+    world,
+    crossfilter,
+    scatterplotXXaccessor,
+    scatterplotYYaccessor
+  } = state.controls;
+  const expressionX =
+    world && scatterplotXXaccessor
+      ? kvCache.get(world.varDataCache, scatterplotXXaccessor)
+      : null;
+  const expressionY =
+    world && scatterplotYYaccessor
+      ? kvCache.get(world.varDataCache, scatterplotYYaccessor)
+      : null;
+
+  return {
+    world,
+
+    colorRGB: state.controls.colorRGB,
+    colorAccessor: state.controls.colorAccessor,
+    colorScale: state.controls.colorScale,
+
+    // Accessors are var/gene names (strings)
+    scatterplotXXaccessor,
+    scatterplotYYaccessor,
+    opacityForDeselectedCells: state.controls.opacityForDeselectedCells,
+
+    differential: state.differential,
+
+    expressionX,
+    expressionY,
+
+    crossfilter,
+    // updated whenever the crossfilter selection is updated
+    selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+  };
+})
+class Scatterplot extends React.Component {
+  constructor(props) {
+    super(props);
+    this.count = 0;
+    this.axes = false;
+    this.state = {
+      svg: null,
+      minimized: null,
+      xScale: null,
+      yScale: null
+    };
+  }
+
+  componentDidMount() {
+    const { svg } = setupScatterplot(width, height, margin);
+    let scales;
+    const { expressionX, expressionY } = this.props;
+
+    if (svg && expressionX && expressionY) {
+      scales = Scatterplot.setupScales(expressionX, expressionY);
+      this.drawAxesSVG(scales.xScale, scales.yScale, svg);
+    }
+
+    const camera = _camera(this.reglCanvas, { scale: true, rotate: false });
+    const regl = _regl(this.reglCanvas);
+
+    const drawPoints = _drawPoints(regl);
+
+    // preallocate buffers
+    const pointBuffer = regl.buffer();
+    const colorBuffer = regl.buffer();
+    const sizeBuffer = regl.buffer();
+
+    const reglRender = regl.frame(() => {
+      this.reglDraw(
+        regl,
+        drawPoints,
+        sizeBuffer,
+        colorBuffer,
+        pointBuffer,
+        camera
+      );
+      camera.tick();
+    });
+
+    this.reglRenderState = "rendering";
+
+    this.setState({
+      regl,
+      sizeBuffer,
+      pointBuffer,
+      colorBuffer,
+      svg,
+      xScale: scales ? scales.xScale : null,
+      yScale: scales ? scales.yScale : null,
+      reglRender,
+      camera,
+      drawPoints
+    });
+  }
+
+  componentDidUpdate(prevProps) {
+    const {
+      world,
+      crossfilter,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor,
+      expressionX,
+      expressionY,
+      colorRGB
+    } = this.props;
+    const {
+      reglRender,
+      xScale,
+      yScale,
+      regl,
+      pointBuffer,
+      colorBuffer,
+      sizeBuffer,
+      svg,
+      drawPoints,
+      camera
+    } = this.state;
+
+    if (
+      world &&
+      svg &&
+      xScale &&
+      yScale &&
+      scatterplotXXaccessor &&
+      scatterplotYYaccessor &&
+      (scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
+      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor || // was CLU now FTH1 etc
+        !this.axes) // clicked off the tab and back again, rerender
+    ) {
+      this.drawAxesSVG(xScale, yScale, svg);
+    }
+
+    if (reglRender && this.reglRenderState === "rendering") {
+      reglRender.cancel();
+      this.reglRenderState = "paused";
+    }
+
+    if (
+      world &&
+      regl &&
+      pointBuffer &&
+      colorBuffer &&
+      sizeBuffer &&
+      expressionX &&
+      expressionY &&
+      scatterplotXXaccessor &&
+      scatterplotYYaccessor &&
+      xScale &&
+      yScale
+    ) {
+      const cellCount = expressionX.length;
+      const positionsBuf = new Float32Array(2 * cellCount);
+      const colorsBuf = new Float32Array(3 * cellCount);
+      const sizesBuf = new Float32Array(cellCount);
+
+      const glScaleX = scaleLinear([0, width], [-0.95, 0.95]);
+      const glScaleY = scaleLinear([0, height], [-1, 1]);
+
+      /*
+        Construct Vectors
+      */
+      for (let i = 0; i < cellCount; i += 1) {
+        positionsBuf[2 * i] = glScaleX(xScale(expressionX[i]));
+        positionsBuf[2 * i + 1] = glScaleY(yScale(expressionY[i]));
+      }
+
+      for (let i = 0; i < cellCount; i += 1) {
+        colorsBuf.set(colorRGB[i], 3 * i);
+      }
+
+      crossfilter.fillByIsFiltered(sizesBuf, 4, 0.2);
+
+      pointBuffer({ data: positionsBuf, dimension: 2 });
+      colorBuffer({ data: colorsBuf, dimension: 3 });
+      sizeBuffer({ data: sizesBuf, dimension: 1 });
+      this.count = cellCount;
+
+      regl._refresh();
+      this.reglDraw(
+        regl,
+        drawPoints,
+        sizeBuffer,
+        colorBuffer,
+        pointBuffer,
+        camera
+      );
+    }
+
+    if (
+      expressionX &&
+      expressionY &&
+      (scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
+        scatterplotYYaccessor !== prevProps.scatterplotYYaccessor)
+    ) {
+      const scales = Scatterplot.setupScales(expressionX, expressionY);
+      this.setState(scales);
+    }
+  }
+
+  static setupScales(expressionX, expressionY) {
+    const xScale = d3
+      .scaleLinear()
+      .domain(d3.extent(expressionX))
+      .range([0, width]);
+    const yScale = d3
+      .scaleLinear()
+      .domain(d3.extent(expressionY))
+      .range([height, 0]);
+
+    return {
+      xScale,
+      yScale
+    };
+  }
+
+  reglDraw(regl, drawPoints, sizeBuffer, colorBuffer, pointBuffer, camera) {
+    regl.clear({
+      depth: 1,
+      color: [1, 1, 1, 1]
+    });
+
+    drawPoints({
+      size: sizeBuffer,
+      distance: camera.distance,
+      color: colorBuffer,
+      position: pointBuffer,
+      count: this.count,
+      view: camera.view()
+    });
+  }
+
+  drawAxesSVG(xScale, yScale, svg) {
+    const { scatterplotYYaccessor, scatterplotXXaccessor } = this.props;
+    svg.selectAll("*").remove();
+
+    // the axes are much cleaner and easier now. No need to rotate and orient
+    // the axis, just call axisBottom, axisLeft etc.
+    const xAxis = d3.axisBottom().scale(xScale);
+
+    const yAxis = d3.axisLeft().scale(yScale);
+
+    // adding axes is also simpler now, just translate x-axis to (0,height)
+    // and it's alread defined to be a bottom axis.
+    svg
+      .append("g")
+      .attr("transform", `translate(0,${height})`)
+      .attr("class", "x axis")
+      .call(xAxis);
+
+    // y-axis is translated to (0,0)
+    svg
+      .append("g")
+      .attr("transform", "translate(0,0)")
+      .attr("class", "y axis")
+      .call(yAxis);
+
+    // adding label. For x-axis, it's at (10, 10), and for y-axis at (width, height-10).
+    svg
+      .append("text")
+      .attr("x", 10)
+      .attr("y", 10)
+      .attr("class", "label")
+      .style("font-style", "italic")
+      .text(scatterplotYYaccessor);
+
+    svg
+      .append("text")
+      .attr("x", width)
+      .attr("y", height - 10)
+      .attr("text-anchor", "end")
+      .attr("class", "label")
+      .style("font-style", "italic")
+      .text(scatterplotXXaccessor);
+  }
+
+  render() {
+    const { dispatch } = this.props;
+    const { minimized } = this.state;
+
+    return (
+      <div
+        style={{
+          position: "fixed",
+          bottom: minimized ? -height + -margin.top : 0,
+          borderRadius: "3px 3px 0px 0px",
+          left: globals.leftSidebarWidth + globals.scatterplotMarginLeft,
+          padding: "0px 20px 20px 0px",
+          backgroundColor: "white",
+          /* x y blur spread color */
+          boxShadow: "0px 0px 6px 2px rgba(153,153,153,0.4)"
+        }}
+        id="scatterplot_wrapper"
+      >
+        <ButtonGroup
+          style={{
+            position: "absolute",
+            right: 5,
+            top: 5
+          }}
+        >
+          <Button
+            type="button"
+            minimal
+            onClick={() => {
+              this.setState({ minimized: !minimized });
+            }}
+          >
+            {minimized ? "show scatterplot" : "hide"}
+          </Button>
+          <Button
+            type="button"
+            minimal
+            onClick={() => {
+              dispatch({
+                type: "clear scatterplot"
+              });
+              dispatch({
+                type: "reset colorscale"
+              });
+            }}
+          >
+            remove
+          </Button>
+        </ButtonGroup>
+        <div
+          className={styles.scatterplot}
+          id="scatterplot"
+          style={{
+            width: `${width + margin.left + margin.right}px`,
+            height: `${height + margin.top + margin.bottom}px`
+          }}
+        >
+          <canvas
+            width={width}
+            height={height}
+            style={{
+              marginLeft: margin.left - 7,
+              marginTop: margin.top
+            }}
+            ref={canvas => {
+              this.reglCanvas = canvas;
+            }}
+          />
+        </div>
+      </div>
+    );
+  }
+}
+
+export default Scatterplot;

client/src/components/scatterplot/setupScatterplot.js

@@ -5,15 +5,17 @@
 ******************************************
 ******************************************/
 
-const setupScatterplot = (width, height, margin) => {
+import * as d3 from "d3";
-  var container = d3.select("#scatterplot");
 
-  var svg = container
+const setupScatterplot = (width, height, margin) => {
+  const container = d3.select("#scatterplot");
+
+  const svg = container
     .append("svg")
     .attr("width", width + margin.left + margin.right)
     .attr("height", height + margin.top + margin.bottom)
     .append("g")
-    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");
+    .attr("transform", `translate(${margin.left},${margin.top})`);
 
   return {
     svg

client/src/components/scatterplot/util.js

@@ -1,10 +1,5 @@
 // jshint esversion: 6
-import _ from "lodash";
+export const margin = { top: 40, right: 5, bottom: 20, left: 60 };
-
-const paddingRight = 120;
-const continuousChartWidth = 340;
-
-export const margin = { top: 66, right: 110, bottom: 20, left: 60 };
 export const width = 340;
 export const height = 340 - margin.top - margin.bottom;
 export const innerHeight = height - 2;

client/src/globals.js

@@ -28,9 +28,25 @@ export const continuous = [
   "Unmapped_short"
 ];
 
+/* if a categorical metadata field has more options than this, truncate */
+export const maxCategoricalOptionsToDisplay = 100;
+
+/*
+these are default values for configuration  the CLI may supply.
+See the REST API and CLI specs for more info.
+*/
+export const configDefaults = {
+  features: {},
+  displayNames: {},
+  parameters: {
+    "max-category-items": 1000
+  }
+};
+
 /* colors */
 export const blue = "#4a90e2";
 export const hcaBlue = "#1c7cc7";
+export const lightestGrey = "rgb(249,249,249)";
 export const lighterGrey = "rgb(245,245,245)";
 export const lightGrey = "rgb(211,211,211)";
 export const mediumGrey = "rgb(153,153,153)";
@@ -41,36 +57,48 @@ export const brightBlue = "#4a90e2";
 export const brightGreen = "#A2D729";
 export const darkGreen = "#448C4D";
 
+export const defaultCellColor = "rgb(0,0,0,1)";
+
+/* typography constants */
+
 export const tiniestFontSize = 12;
-
+export const largestFontSize = 24;
+export const uppercaseLetterSpacing = "0.04em";
 export const bolder = 700;
-
-export let API = {
-  // prefix: "http://api.clustering.czi.technology/api/",
-  //prefix: "http://tabulamuris.cxg.czi.technology/api/",
-   prefix: "http://pbmc3k.cxg.czi.technology/api/",
-  // prefix: "http://pbmc33k.cxg.czi.technology/api/",
-
-  // prefix: "http://api-staging.clustering.czi.technology/api/",
-  version: "v0.1/"
-};
-
-if (window.CELLXGENE && window.CELLXGENE.API) API = window.CELLXGENE.API;
-
-export let datasetTitle = "";
-
-if (window.CELLXGENE && window.CELLXGENE.datasetTitle)
-  datasetTitle = window.CELLXGENE.datasetTitle;
-
 export const accentFont = "Georgia,Times,Times New Roman,serif";
 export const maxParagraphWidth = 600;
+
+/* layout styles constants */
+
+export const cellxgeneTitleLeftPadding = 14;
+export const cellxgeneTitleTopPadding = 7;
+
 export const maxControlsWidth = 800;
 
 export const graphMargin = { top: 20, right: 10, bottom: 30, left: 40 };
-// export const graphWidth = 1440 /* window width */ - 410 /* sidebar */ - (15 + 15) /* left right padding */ /* but responsive */;
-// export const graphHeight = 500;
 export const graphWidth = 700;
 export const graphHeight = 700;
+export const scatterplotMarginLeft = 25;
+
+export const leftSidebarWidth = 365;
+export const leftSidebarSectionHeading = {
+  fontSize: 18,
+  textTransform: "uppercase",
+  fontWeight: 500,
+  letterSpacing: ".05em"
+};
+export const leftSidebarSectionPadding = 10;
+
+let _API = {
+  // prefix: "http://api.clustering.czi.technology/api/",
+  // prefix: "http://tabulamuris.cxg.czi.technology/api/",
+  // prefix: "http://api-staging.clustering.czi.technology/api/",
+  prefix: "http://localhost:5005/api/",
+  version: "v0.2/"
+};
+
+if (window.CELLXGENE && window.CELLXGENE.API) _API = window.CELLXGENE.API;
+export const API = _API;
 
 export const ordinalColors = [
   "#0ac115",

client/src/index.css

@@ -0,0 +1,9 @@
+/*
+>>>GLOBAL<<< CSS imports & definitions
+
+Do NOT put anything in this file that is not global.  We do not perform
+namespace mangling on these definitions.  See webpack config for specifics.
+*/
+@import "~@blueprintjs/core/lib/css/blueprint.css";
+@import "~@blueprintjs/icons/lib/css/blueprint-icons.css";
+@import "~@blueprintjs/select/lib/css/blueprint-select.css";

client/src/index.js

@@ -0,0 +1,21 @@
+// jshint esversion: 6
+/* eslint-disable no-console */
+import React from "react";
+import ReactDOM from "react-dom";
+import { Provider } from "react-redux";
+import { FocusStyleManager } from "@blueprintjs/core";
+
+import "./index.css";
+
+/* our code */
+import App from "./components/app";
+import store from "./reducers";
+
+FocusStyleManager.onlyShowFocusOnTabs();
+
+ReactDOM.render(
+  <Provider store={store}>
+    <App />
+  </Provider>,
+  document.getElementById("root")
+);

client/src/middleware/updateCellColors.js

@@ -0,0 +1,127 @@
+// jshint esversion: 6
+import _ from "lodash";
+import * as d3 from "d3";
+import {
+  interpolateViridis,
+  interpolateSpectral,
+  interpolateRainbow,
+  interpolateBlues,
+  interpolateCool
+} from "d3-scale-chromatic";
+import * as globals from "../globals";
+import parseRGB from "../util/parseRGB";
+
+/*
+  https://medium.com/@jacobp100/you-arent-using-redux-middleware-enough-94ffe991e6
+  storeInstance =>
+    functionToCallWithAnActionThatWillSendItToTheNextMiddleware =>
+    actionThatDispatchWasCalledWith =>
+    valueToUseAsTheReturnValueOfTheDispatchCall
+*/
+
+/*
+  What this file does:
+
+  1. fire a filter action anywhere in the app
+  2. ** this middleware checks to see the state of all the currently selected filters,
+     including the new one
+  3. ** create updated selection from a copy of all the cells presently on the client
+     (this may be a subset of 'all')
+  4. ** append that new selection to the action so that it magically appears in the reducer
+     just because the action was fired
+
+  This is nice because we keep a lot of filtering business logic centralized
+  (what it means in practice to be selected)
+*/
+
+const updateCellColorsMiddleware = store => next => action => {
+  const s = store.getState();
+
+  /*
+  this is a hardcoded map of the things we need to keep an eye on and update
+  global cell selection in response to
+  */
+  const filterJustChanged =
+    action.type === "color by expression" ||
+    action.type === "color by continuous metadata" ||
+    action.type === "color by categorical metadata";
+
+  if (!filterJustChanged || !s.controls.world.obsAnnotations) {
+    return next(
+      action
+    ); /* if the cells haven't loaded or the action wasn't a color change, bail */
+  }
+
+  const { obsAnnotations } = s.controls.world;
+  let colorScale;
+  const colorsByName = new Array(obsAnnotations.length);
+  const colorsByRGB = new Array(obsAnnotations.length);
+
+  /*
+  in plain language...
+   (a) once the cells have loaded.
+   (b) each time a user changes a color control we need to update cellsMetadata colors
+  This is available to all the draw functions as world.colorName[index] or world.colorRGB[index]
+  */
+
+  if (action.type === "color by categorical metadata") {
+    const categories = _.filter(s.controls.world.schema.annotations.obs, {
+      name: action.colorAccessor
+    })[0].categories;
+
+    colorScale = d3
+      .scaleSequential(interpolateRainbow)
+      .domain([0, categories.length]);
+
+    for (let i = 0; i < obsAnnotations.length; i += 1) {
+      const obs = obsAnnotations[i];
+      const c = colorScale(categories.indexOf(obs[action.colorAccessor]));
+      colorsByName[i] = c;
+      colorsByRGB[i] = parseRGB(c);
+    }
+  }
+
+  if (action.type === "color by continuous metadata") {
+    colorScale = d3
+      .scaleLinear()
+      .domain([0, action.rangeMaxForColorAccessor])
+      .range([1, 0]);
+
+    for (let i = 0; i < obsAnnotations.length; i += 1) {
+      const obs = obsAnnotations[i];
+      const c = interpolateCool(colorScale(obs[action.colorAccessor]));
+      colorsByName[i] = c;
+      colorsByRGB[i] = parseRGB(c);
+    }
+  }
+
+  if (action.type === "color by expression") {
+    const { gene, data } = action;
+    const expression = data[gene]; // Float32Array
+    colorScale = d3
+      .scaleLinear()
+      .domain([_.min(expression), _.max(expression)])
+      .range([
+        1,
+        0
+      ]); /* invert viridis... probably pass this scale through to others */
+
+    for (let i = 0, len = expression.length; i < len; i += 1) {
+      const c = interpolateCool(colorScale(expression[i]));
+      colorsByName[i] = c;
+      colorsByRGB[i] = parseRGB(c);
+    }
+  }
+
+  /*
+  append the result of all the filters to the action the user just triggered
+  */
+  const modifiedAction = Object.assign({}, action, {
+    colors: { name: colorsByName, rgb: colorsByRGB },
+    colorScale
+  });
+
+  return next(modifiedAction);
+};
+
+export default updateCellColorsMiddleware;

client/src/middleware/updateURLMiddleware.js

@@ -0,0 +1,87 @@
+// jshint esversion: 6
+// import uri from "urijs";
+
+/*
+NOTE: file currently not used, but retained as we expect to reinstate features in this
+area shortly.
+*/
+
+/*
+  https://medium.com/@jacobp100/you-arent-using-redux-middleware-enough-94ffe991e6
+  storeInstance
+    => functionToCallWithAnActionThatWillSendItToTheNextMiddleware
+    => actionThatDispatchWasCalledWith
+    => valueToUseAsTheReturnValueOfTheDispatchCall
+*/
+
+const updateURLMiddleware = (/* store */) => next => action => {
+  // const oldState = store.getState();
+  const nextAction = next(action);
+
+  if (action.type === "url changed") {
+    /* we don't handle pop state here - we handle it in the url reducer */
+    return nextAction;
+  }
+
+  // const state = store.getState();
+
+  /************************************************************************
+      *************************************************************************
+      1. Redux app state just changed. Clear URL, and then update it.
+      1a. We get the whole state tree to construct the url!
+      1b. But (see reducers/url.js) we try to centralize it because...
+      1c. ...the back button / initial load case ('url changed' return above)
+      means that we have to listen for 'url changed' and construct state
+      from the browser
+      *************************************************************************
+      ************************************************************************/
+
+  // const oldURI = URI(window.location.href)
+  // const newURI = URI(oldURI).setQuery({})
+
+  // if (window.location.search === "") {
+  //   newURL = uri.addQuery(category, value).toString(); /* #1 */
+  // } else if (uri.hasQuery(category, value) || uri.hasQuery(category, value, true)) { /* true param here means check arrays as well http://medialize.github.io/URI.js/docs.html#search-has */
+  //   newURL = uri.removeQuery(category, value).toString(); /* #4 */
+  // } else {
+  //   newURL = uri.addQuery(category, value).toString(); /* #2 & #3 are handled by URI */
+  // }
+  //
+  // window.history.pushState("", "", newURL)
+
+  //
+  // // Internal helper for working with URIs
+  // const oldURI = new URI(window.location.href);
+  // const newURI = new URI(oldURI).setQueryData({});
+  //
+  // newURI.setPath('/foo/bar');
+  //
+  // // Set the path based on state
+  // if (!state.isOnLandingPage && state.project.id) {
+  //   newURI.setPath(newURI.getPath() + state.project.id + '/');
+  //   newURI.addQueryData('baz', state.mode);
+  //   newURI.addQueryData('bat', state.selection.activePageID);
+  // } else {
+  //   newURI.setPath(newURI.getPath() + state.landingSection + '/');
+  // }
+  //
+  // // Avoid URL thrashing by replacing state while loading instead of pushing
+  // const newPath = newURI.toString();
+  // const oldPath = oldURI.toString();
+  // if (newPath !== oldPath) {
+  //   if (
+  //     (oldState.mode === 'asdf' &&
+  //       state.mode === 'asdf' &&
+  //       !oldState.isOnLandingPage) ||
+  //     oldState.isLoadingProject !== state.isLoadingProject
+  //   ) {
+  //     window.history.replaceState(null, null, newPath);
+  //   } else {
+  //     window.history.pushState(null, null, newPath);
+  //   }
+  // }
+
+  return nextAction;
+};
+
+export default updateURLMiddleware;

client/src/reducers/config.js

@@ -0,0 +1,34 @@
+// jshint esversion: 6
+const Config = (
+  state = {
+    displayNames: null,
+    features: null,
+    parameters: null
+  },
+  action
+) => {
+  switch (action.type) {
+    case "initial data load start":
+      return {
+        ...state,
+        loading: true,
+        error: null
+      };
+    case "configuration load complete":
+      return {
+        ...state,
+        loading: false,
+        error: null,
+        ...action.config
+      };
+    case "initial data load error":
+      return {
+        ...state,
+        error: action.error
+      };
+    default:
+      return state;
+  }
+};
+
+export default Config;

client/src/reducers/controls.js

@@ -0,0 +1,649 @@
+// jshint esversion: 6
+
+import _ from "lodash";
+import { World, kvCache } from "../util/stateManager";
+import parseRGB from "../util/parseRGB";
+import Crossfilter from "../util/typedCrossfilter";
+import * as globals from "../globals";
+import {
+  layoutDimensionName,
+  obsAnnoDimensionName,
+  userDefinedDimensionName,
+  diffexpDimensionName,
+  makeContinuousDimensionName
+} from "../util/nameCreators";
+import { fillRange } from "../util/typedCrossfilter/util";
+
+/*
+Selection state for categoricals are tracked in an Object that
+has two main components for each category:
+1. mapping of option value to an index
+2. array of bool selection state by index
+Remember that option values can be ANY js type, except undefined/null.
+
+  {
+    _category_name_1: {
+      // map of option value to index
+      optionIndex: Map([
+        optval1: index,
+        ...
+      ])
+
+      // index->selection true/false state
+      optionSelected: [ true/false, true/false, ... ]
+
+      // number of options
+      numOptions: number,
+
+      // isTruncated - true if the options for selection has
+      // been truncated (ie, was too large to implement)
+    }
+  }
+*/
+function topNoptions(summary) {
+  const counts = _.map(summary.categories, cat => summary.options[cat]);
+  const sortIndex = fillRange(new Array(summary.numOptions)).sort(
+    (a, b) => counts[b] - counts[a]
+  );
+  const sortedCategories = _.map(sortIndex, i => summary.categories[i]);
+  const sortedCounts = _.map(sortIndex, i => counts[i]);
+  const N = globals.maxCategoricalOptionsToDisplay;
+
+  if (sortedCategories.length < N) {
+    return [sortedCategories, sortedCounts];
+  }
+  return [sortedCategories.slice(0, N), sortedCounts.slice(0, N)];
+}
+
+function createCategoricalSelectionState(state, world) {
+  const res = {};
+  _.forEach(world.summary.obs, (value, key) => {
+    if (value.categories) {
+      const isColorField = key.includes("color") || key.includes("Color");
+      const isSelectableCategory =
+        !isColorField &&
+        key !== "name" &&
+        value.categories.length < state.maxCategoryItems;
+      if (isSelectableCategory) {
+        const [optionValue, optionCount] = topNoptions(value);
+        // const optionCount = Object.values(value.options);
+
+        const optionIndex = new Map(optionValue.map((v, i) => [v, i]));
+        const numOptions = optionIndex.size;
+        const optionSelected = new Array(numOptions).fill(true);
+        const isTruncated = optionValue.length < value.numOptions;
+        res[key] = {
+          optionValue, // array: of natively typed option values
+          optionIndex, // map: option value (native type) -> option index
+          optionSelected, // array: t/f selection state
+          numOptions, // number: of options
+          isTruncated, // bool: true if list was truncated
+          optionCount // array: cardinality of each option
+        };
+      }
+    }
+  });
+  return res;
+}
+
+/*
+given a categoricalSelectionState, return the list of all option values
+where selection state is true (ie, they are selected).
+*/
+function selectedValuesForCategory(categorySelectionState) {
+  const selectedValues = _([...categorySelectionState.optionIndex])
+    .filter(tuple => categorySelectionState.optionSelected[tuple[1]])
+    .map(tuple => tuple[0])
+    .value();
+  return selectedValues;
+}
+
+const Controls = (
+  state = {
+    // data loading flag
+    loading: false,
+    error: null,
+
+    // configuration
+    maxCategoryItems: globals.configDefaults.parameters["max-category-items"],
+
+    // the whole big bang
+    universe: null,
+
+    // all of the data + selection state
+    world: null,
+    colorName: null,
+    colorRGB: null,
+    categoricalSelectionState: null,
+    crossfilter: null,
+    dimensionMap: null,
+    userDefinedGenes: [],
+    diffexpGenes: [],
+
+    colorAccessor: null,
+    colorScale: null,
+
+    opacityForDeselectedCells: 0.2,
+    graphBrushSelection: null,
+    continuousSelection: null,
+    scatterplotXXaccessor: null, // just easier to read
+    scatterplotYYaccessor: null,
+    axesHaveBeenDrawn: false,
+    graphRenderCounter: 0 /* integer as <Component key={graphRenderCounter} - a change in key forces a remount */,
+    __storedStateForCelllist1__: null /* will need procedural control of brush ie., brush.extent https://bl.ocks.org/micahstubbs/3cda05ca68cba260cb81 */,
+    __storedStateForCelllist2__: null
+  },
+  action
+) => {
+  /*
+  For now, log anything looking like an error to the console.
+  */
+  if (action.error || /error/i.test(action.type)) {
+    console.error(action.error);
+  }
+
+  switch (action.type) {
+    /*****************************************************
+          Initialization, World/Universe management
+          and data loading.
+    ******************************************************/
+    case "configuration load complete": {
+      // there are a couple of configuration items we need to retain
+      return {
+        ...state,
+        maxCategoryItems: _.get(
+          state.config,
+          "parameters.max-category-items",
+          globals.configDefaults.parameters["max-category-items"]
+        )
+      };
+    }
+    case "initial data load start": {
+      return { ...state, loading: true };
+    }
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
+      const { userDefinedGenes, diffexpGenes } = state;
+      /* first light - create world & other data-driven defaults */
+      const { universe } = action;
+      const world = World.createWorldFromEntireUniverse(universe);
+      const colorName = new Array(universe.nObs).fill(globals.defaultCellColor);
+      const colorRGB = _.map(colorName, c => parseRGB(c));
+      const categoricalSelectionState = createCategoricalSelectionState(
+        state,
+        world
+      );
+      const crossfilter = Crossfilter(world.obsAnnotations);
+      const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+
+      const worldVarDataCache = world.varDataCache;
+
+      // dimensionMap = {
+      //     layout_X: dim-for-X,
+      //     obsAnno_name: dim for an annotation,
+      //     varData_userDefined_genename: dim for user defined expression,
+      //     varData_diffexp_genename: dim for diff-exp added gene expression
+      // }
+      /* var dimensions */
+      if (userDefinedGenes.length > 0) {
+        /*
+          verbose & slightly confusing that we also access this as an object
+          in controls rather than an array, should be abstracted into
+          util ie., createDimensionsFromBothListsOfGenes(userGenes, diffExp)
+        */
+        _.forEach(userDefinedGenes, gene => {
+          dimensionMap[
+            userDefinedDimensionName(gene)
+          ] = World.createVarDimension(
+            /* "__var__" + */
+            world,
+            worldVarDataCache,
+            crossfilter,
+            gene
+          );
+        });
+      }
+
+      if (diffexpGenes.length > 0) {
+        _.forEach(diffexpGenes, gene => {
+          dimensionMap[diffexpDimensionName(gene)] = World.createVarDimension(
+            /* "__var__" + */
+            world,
+            worldVarDataCache,
+            crossfilter,
+            gene
+          );
+        });
+      }
+
+      return {
+        ...state,
+        loading: false,
+        error: null,
+        universe,
+        world,
+        colorName,
+        colorRGB,
+        categoricalSelectionState,
+        crossfilter,
+        dimensionMap,
+        colorAccessor: null
+      };
+    }
+    case "set World to current selection": {
+      const { userDefinedGenes, diffexpGenes } = state;
+
+      /* Set viewable world to be the currently selected data */
+      const world = World.createWorldFromCurrentSelection(
+        action.universe,
+        action.world,
+        action.crossfilter
+      );
+      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
+      const colorRGB = _.map(colorName, c => parseRGB(c));
+      const categoricalSelectionState = createCategoricalSelectionState(
+        state,
+        world
+      );
+      const crossfilter = Crossfilter(world.obsAnnotations);
+      const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+
+      const worldVarDataCache = world.varDataCache;
+      /* var dimensions */
+
+      if (userDefinedGenes.length > 0) {
+        /*
+          verbose & slightly confusing that we also access this as an object
+          in controls rather than an array, should be abstracted into
+          util ie., createDimensionsFromBothListsOfGenes(userGenes, diffExp)
+        */
+        _.forEach(userDefinedGenes, gene => {
+          dimensionMap[
+            userDefinedDimensionName(gene)
+          ] = World.createVarDimension(
+            /* "__var__" + */
+            world,
+            worldVarDataCache,
+            crossfilter,
+            gene
+          );
+        });
+      }
+
+      if (diffexpGenes.length > 0) {
+        _.forEach(diffexpGenes, gene => {
+          dimensionMap[diffexpDimensionName(gene)] = World.createVarDimension(
+            /* "__var__" + */
+            world,
+            worldVarDataCache,
+            crossfilter,
+            gene
+          );
+        });
+      }
+
+      return {
+        ...state,
+        loading: false,
+        error: null,
+        world,
+        colorName,
+        colorRGB,
+        categoricalSelectionState,
+        crossfilter,
+        dimensionMap,
+        colorAccessor: null
+      };
+    }
+    case "expression load success": {
+      const { world, universe } = state;
+      let universeVarDataCache = universe.varDataCache;
+      let worldVarDataCache = world.varDataCache;
+      _.forEach(action.expressionData, (val, key) => {
+        universeVarDataCache = kvCache.set(universeVarDataCache, key, val);
+        if (kvCache.get(worldVarDataCache, key) === undefined) {
+          worldVarDataCache = kvCache.set(
+            worldVarDataCache,
+            key,
+            World.subsetVarData(world, universe, val)
+          );
+        }
+      });
+
+      return {
+        ...state,
+        universe: {
+          ...universe,
+          varDataCache: universeVarDataCache
+        },
+        world: {
+          ...world,
+          varDataCache: worldVarDataCache
+        }
+      };
+    }
+    case "request user defined gene success": {
+      const { world, crossfilter, dimensionMap, userDefinedGenes } = state;
+      const worldVarDataCache = world.varDataCache;
+      const _userDefinedGenes = userDefinedGenes.slice();
+      const gene = action.data.genes[0];
+
+      dimensionMap[userDefinedDimensionName(gene)] = World.createVarDimension(
+        /* "__var__" + */
+        world,
+        worldVarDataCache,
+        crossfilter,
+        gene
+      );
+
+      return {
+        ...state,
+        dimensionMap,
+        userDefinedGenes: _userDefinedGenes
+      };
+    }
+    case "request differential expression success": {
+      const { world, crossfilter, dimensionMap } = state;
+      const worldVarDataCache = world.varDataCache;
+      const _diffexpGenes = [];
+
+      action.data.forEach(d => {
+        _diffexpGenes.push(world.varAnnotations[d[0]].name);
+      });
+
+      _.forEach(_diffexpGenes, gene => {
+        dimensionMap[diffexpDimensionName(gene)] = World.createVarDimension(
+          /* "__var__" + */
+          world,
+          worldVarDataCache,
+          crossfilter,
+          gene
+        );
+      });
+
+      return {
+        ...state,
+        dimensionMap,
+        diffexpGenes: _diffexpGenes
+      };
+    }
+    case "clear differential expression": {
+      const { world, universe, dimensionMap } = state;
+      const _dimensionMap = dimensionMap;
+      const universeVarDataCache = universe.varDataCache;
+      const worldVarDataCache = world.varDataCache;
+
+      _.forEach(action.diffExp, values => {
+        const { name } = world.varAnnotations[values[0]];
+        // clean up crossfilter dimensions
+        const dimension = dimensionMap[diffexpDimensionName(name)];
+        dimension.dispose();
+        delete dimensionMap[diffexpDimensionName(name)];
+      });
+      return {
+        ...state,
+        dimensionMap: _dimensionMap,
+        diffexpGenes: [],
+        universe: {
+          ...universe,
+          varDataCache: universeVarDataCache
+        },
+        world: {
+          ...world,
+          varDataCache: worldVarDataCache
+        }
+      };
+    }
+    case "user defined gene": {
+      /*
+        this could also live in expression success with a conditional,
+        but that handles diffexp also
+      */
+      const newUserDefinedGenes = state.userDefinedGenes.slice();
+      newUserDefinedGenes.push(action.data);
+      return {
+        ...state,
+        userDefinedGenes: newUserDefinedGenes
+      };
+    }
+    case "clear user defined gene": {
+      const { userDefinedGenes, dimensionMap } = state;
+      const newUserDefinedGenes = _.filter(
+        userDefinedGenes,
+        d => d !== action.data
+      );
+
+      const dimension = dimensionMap[userDefinedDimensionName(action.data)];
+      dimension.dispose();
+      delete dimensionMap[userDefinedDimensionName(action.data)];
+
+      return {
+        ...state,
+        dimensionMap,
+        userDefinedGenes: newUserDefinedGenes
+      };
+    }
+    case "clear all user defined genes": {
+      const { userDefinedGenes, dimensionMap } = state;
+
+      _.forEach(userDefinedGenes, gene => {
+        const dimension = dimensionMap[userDefinedDimensionName(gene)];
+        dimension.dispose();
+        delete dimensionMap[userDefinedDimensionName(gene)];
+      });
+
+      return {
+        ...state,
+        dimensionMap,
+        userDefinedGenes: []
+      };
+    }
+    case "reset colorscale": {
+      const { world } = state;
+      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
+      const colorRGB = _.map(colorName, c => parseRGB(c));
+      return {
+        ...state,
+        colorName,
+        colorRGB,
+        colorAccessor: null
+      };
+    }
+    case "expression load error":
+    case "initial data load error": {
+      return {
+        ...state,
+        loading: false,
+        error: action.error
+      };
+    }
+
+    /*******************************
+             User Events
+     *******************************/
+    case "graph brush selection change": {
+      state.dimensionMap[layoutDimensionName("X")].filterRange([
+        action.brushCoords.northwest[0],
+        action.brushCoords.southeast[0]
+      ]);
+      state.dimensionMap[layoutDimensionName("Y")].filterRange([
+        action.brushCoords.southeast[1],
+        action.brushCoords.northwest[1]
+      ]);
+      return {
+        ...state,
+        graphBrushSelection: action.brushCoords
+      };
+    }
+    case "graph brush deselect": {
+      state.dimensionMap[layoutDimensionName("X")].filterAll();
+      state.dimensionMap[layoutDimensionName("Y")].filterAll();
+      return {
+        ...state,
+        graphBrushSelection: null
+      };
+    }
+    case "continuous metadata histogram brush": {
+      const name = makeContinuousDimensionName(
+        action.continuousNamespace,
+        action.selection
+      );
+
+      // action.selection: metadata name being selected
+      // action.range: filter range, or null if deselected
+      if (!action.range) {
+        state.dimensionMap[name].filterAll();
+      } else {
+        state.dimensionMap[name].filterRange(action.range);
+      }
+      return { ...state };
+    }
+    case "change opacity deselected cells in 2d graph background":
+      return {
+        ...state,
+        opacityForDeselectedCells: action.data
+      };
+    case "increment graph render counter": {
+      const c = state.graphRenderCounter + 1;
+      return {
+        ...state,
+        graphRenderCounter: c
+      };
+    }
+    /*******************************
+          Categorical metadata
+    *******************************/
+    case "categorical metadata filter select": {
+      const newOptionSelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].optionSelected
+      );
+      newOptionSelected[action.optionIndex] = true;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          optionSelected: newOptionSelected
+        }
+      };
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
+      state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
+        selectedValuesForCategory(cat)
+      );
+
+      return {
+        ...state,
+        categoricalSelectionState: newCategoricalSelectionState
+      };
+    }
+    case "categorical metadata filter deselect": {
+      const newOptionSelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].optionSelected
+      );
+      newOptionSelected[action.optionIndex] = false;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          optionSelected: newOptionSelected
+        }
+      };
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
+      state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
+        selectedValuesForCategory(cat)
+      );
+
+      return {
+        ...state,
+        categoricalSelectionState: newCategoricalSelectionState
+      };
+    }
+    case "categorical metadata filter none of these": {
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          optionSelected: Array.from(
+            state.categoricalSelectionState[action.metadataField].optionSelected
+          ).fill(false)
+        }
+      };
+      state.dimensionMap[
+        obsAnnoDimensionName(action.metadataField)
+      ].filterNone();
+      return {
+        ...state,
+        categoricalSelectionState: newCategoricalSelectionState
+      };
+    }
+    case "categorical metadata filter all of these": {
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          optionSelected: Array.from(
+            state.categoricalSelectionState[action.metadataField].optionSelected
+          ).fill(true)
+        }
+      };
+      state.dimensionMap[
+        obsAnnoDimensionName(action.metadataField)
+      ].filterAll();
+      return {
+        ...state,
+        categoricalSelectionState: newCategoricalSelectionState
+      };
+    }
+
+    /*******************************
+              Color Scale
+    *******************************/
+    case "color by categorical metadata":
+    case "color by continuous metadata": {
+      return {
+        ...state,
+        colorName: action.colors.name,
+        colorRGB: action.colors.rgb,
+        colorAccessor: action.colorAccessor,
+        colorScale: action.colorScale
+      };
+    }
+    case "color by expression": {
+      return {
+        ...state,
+        colorName: action.colors.name,
+        colorRGB: action.colors.rgb,
+        colorAccessor: action.gene,
+        colorScale: action.colorScale
+      };
+    }
+
+    /*******************************
+              Scatterplot
+    *******************************/
+    case "set scatterplot x":
+      return {
+        ...state,
+        scatterplotXXaccessor: action.data
+      };
+    case "set scatterplot y":
+      return {
+        ...state,
+        scatterplotYYaccessor: action.data
+      };
+    case "clear scatterplot":
+      return {
+        ...state,
+        scatterplotXXaccessor: null,
+        scatterplotYYaccessor: null
+      };
+
+    default:
+      return state;
+  }
+};
+
+export default Controls;

client/src/reducers/differential.js

@@ -1,4 +1,5 @@
 // jshint esversion: 6
+
 const Differential = (
   state = {
     diffExp: null,
@@ -11,29 +12,49 @@ const Differential = (
 ) => {
   switch (action.type) {
     case "request differential expression started":
-      return Object.assign({}, state, {
+      return {
+        ...state,
         loading: true,
         error: null
-      });
+      };
     case "request differential expression success":
-      return Object.assign({}, state, {
+      return {
+        ...state,
         error: null,
         loading: false,
         diffExp: action.data
-      });
+      };
     case "request differential expression error":
-      return Object.assign({}, state, {
+      return {
+        ...state,
         loading: false,
         error: action.data
-      });
+      };
     case "store current cell selection as differential set 1":
-      return Object.assign({}, state, {
+      return {
+        ...state,
         celllist1: action.data
-      });
+      };
     case "store current cell selection as differential set 2":
-      return Object.assign({}, state, {
+      return {
+        ...state,
         celllist2: action.data
-      });
+      };
+    case "clear differential expression":
+      return {
+        ...state,
+        diffExp: null,
+        celllist1: null,
+        celllist2: null
+      };
+    case "reset World to eq Universe":
+    case "set World to current selection":
+      return {
+        ...state,
+        diffExp: null,
+        celllist1: null,
+        celllist2: null
+      };
     default:
       return state;
   }

client/src/reducers/index.js

@@ -0,0 +1,27 @@
+// jshint esversion: 6
+import { combineReducers, createStore, applyMiddleware } from "redux";
+import thunk from "redux-thunk";
+import updateURLMiddleware from "../middleware/updateURLMiddleware";
+import updateCellColors from "../middleware/updateCellColors";
+import { composeWithDevTools } from "redux-devtools-extension";
+
+import config from "./config";
+import differential from "./differential";
+import responsive from "./responsive";
+import controls from "./controls";
+
+const Reducer = combineReducers({
+  config,
+  responsive,
+  controls,
+  differential
+});
+
+const store = createStore(
+  Reducer,
+  composeWithDevTools(
+    applyMiddleware(thunk, updateURLMiddleware, updateCellColors)
+  )
+);
+
+export default store;

client/src/reducers/responsive.js

@@ -8,10 +8,11 @@ const Responsive = (
 ) => {
   switch (action.type) {
     case "window resize":
-      return Object.assign({}, state, {
+      return {
+        ...state,
         width: action.data.width,
         height: action.data.height
-      });
+      };
     default:
       return state;
   }

client/src/util/actionHelpers.js

@@ -0,0 +1,98 @@
+import _ from "lodash";
+/* XXX: cough, cough, ... */
+import { postNetworkErrorToast } from "../components/framework/toasters";
+
+/*
+dispatch an action error to the user.   Currently we use
+async toasts.
+*/
+export const dispatchNetworkErrorMessageToUser = message =>
+  postNetworkErrorToast(message);
+
+/*
+Catch unexpected errors and make sure we don't lose them!
+*/
+export function catchErrorsWrap(fn, dispatchToUser = false) {
+  return (dispatch, getState) => {
+    fn(dispatch, getState).catch(error => {
+      console.error(error);
+      if (dispatchToUser) {
+        dispatchNetworkErrorMessageToUser(error.message);
+      }
+      dispatch({ type: "UNEXPECTED ERROR", error });
+    });
+  };
+}
+
+/*
+Wrapper to perform an async fetch and JSON decode response.
+*/
+export const doJsonRequest = async url => {
+  const res = await fetch(url, {
+    method: "get",
+    headers: new Headers({
+      "Content-Type": "application/json"
+    })
+  });
+  if (res.ok && res.headers.get("Content-Type") === "application/json") {
+    return res.json();
+  }
+  // else an error
+  const msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  dispatchNetworkErrorMessageToUser(msg);
+  throw new Error(msg);
+};
+
+/*
+This function "packs" filter index lists into the more efficient
+"range" form specified in the REST 0.2 spec.
+
+Specifically, it turns an array of indices [0, 1, 2, 10, 11, 14, ...]
+into a form that encodes runs of consecutive numbers as [min, max].
+Array may not be sorted, but will only contain uniq values.
+
+Parameters:
+   indices - input array of numbers (index)
+   minRangeLength - hint, min range length before it is encoded into range format.
+   sorted - boolean hint indicating array is presorted, ascending order
+
+So [1, 2, 3, 4, 10, 11, 14] -> [ [1, 4], [10, 11], 14]
+*/
+export const rangeEncodeIndices = (
+  indices,
+  minRangeLength = 3,
+  sorted = false
+) => {
+  if (indices.length === 0) {
+    return indices;
+  }
+
+  if (!sorted) {
+    indices = _.sortBy(indices);
+  }
+
+  const result = new Array(indices.length);
+  let resultTail = 0;
+
+  let i = 0;
+  while (i < indices.length) {
+    const begin = indices[i];
+    let current;
+    do {
+      current = indices[i];
+      i += 1;
+    } while (i < indices.length && indices[i] === current + 1);
+
+    if (current - begin + 1 >= minRangeLength) {
+      result[resultTail] = [begin, current];
+      resultTail += 1;
+    } else {
+      for (let j = begin; j <= current; j += 1, resultTail += 1) {
+        result[resultTail] = j;
+      }
+    }
+  }
+
+  result.length = resultTail;
+  return result;
+};

client/src/util/camera.js

@@ -1,9 +1,9 @@
 // jshint esversion: 6
-var createCamera = require("orbit-camera");
+const createCamera = require("orbit-camera");
-var createScroll = require("scroll-speed");
+const createScroll = require("scroll-speed");
-var mp = require("mouse-position");
+const mp = require("mouse-position");
-var mb = require("mouse-pressed");
+const mb = require("mouse-pressed");
-var key = require("key-pressed");
+const key = require("key-pressed");
 
 const panSpeed = 0.4;
 const scaleSpeed = 0.5;
@@ -17,20 +17,19 @@ function attachCamera(canvas, opts) {
   opts.scale = opts.scale !== false;
   opts.rotate = opts.rotate !== false;
 
-  var scroll = createScroll(canvas, opts.scale);
+  const scroll = createScroll(canvas, opts.scale);
-  var mbut = mb(canvas, opts.rotate);
+  const mbut = mb(canvas, opts.rotate);
-  var mpos = mp(canvas);
+  const mpos = mp(canvas);
-  var camera = createCamera([0, 0, 1], [0, 0, -1], [0, 1, 0]);
+  const camera = createCamera([0, 0, 1], [0, 0, -1], [0, 1, 0]);
 
   camera.tick = tick;
 
   return camera;
 
   function tick() {
-    var ctrl = key("<control>") || key("<alt>");
+    const ctrl = key("<control>") || key("<alt>");
-    var alt = key("<shift>");
+    const alt = key("<shift>");
-    var height = canvas.height;
+    const { height, width } = canvas;
-    var width = canvas.width;
 
     if (opts.rotate && mbut.left && ctrl && !alt) {
       camera.rotate(
@@ -41,23 +40,17 @@ function attachCamera(canvas, opts) {
 
     if ((opts.pan && mbut.right) || (mbut.left && !ctrl && !alt)) {
       camera.pan([
-        panSpeed *
+        ((panSpeed * (mpos[0] - mpos.prev[0])) / width) * camera.distance,
-          (mpos[0] - mpos.prev[0]) /
+        ((panSpeed * (mpos[1] - mpos.prev[1])) / height) * camera.distance
-          width *
-          Math.pow(camera.distance, 1),
-        panSpeed *
-          (mpos[1] - mpos.prev[1]) /
-          height *
-          Math.pow(camera.distance, 1)
       ]);
     }
 
     if (opts.scale && scroll[1]) {
-      camera.distance *= Math.exp(scroll[1] * scaleSpeed / height);
+      camera.distance *= Math.exp((scroll[1] * scaleSpeed) / height);
     }
 
     if (opts.scale && (mbut.middle || (mbut.left && !ctrl && alt))) {
-      var d = mpos.y - mpos.prevY;
+      const d = mpos.y - mpos.prevY;
       if (!d) return;
 
       camera.distance *= Math.exp(d / height);

client/src/util/nameCreators.js

@@ -0,0 +1,45 @@
+/*
+
+The original motivation for this file is name collision. It's possible
+to have two of the same gene, user defined and diffexpressed, on the client
+simultaneously, looking at the same data in the cache but with two
+dimensions and two crossfilters. It's also possible that someone could
+have a obsAnnotation named X, but we are using that for layout. So
+we namespace, and abstract to avoid proliferating strings throughout the
+codebase.
+
+*/
+
+const makeDimensionName = (namespace, key) => `${namespace}_${key}`;
+
+export const layoutDimensionName = key => makeDimensionName("layout", key);
+export const obsAnnoDimensionName = key => makeDimensionName("obsAnno", key);
+export const diffexpDimensionName = key =>
+  makeDimensionName("varData_diffexp", key);
+export const userDefinedDimensionName = key =>
+  makeDimensionName("varData_userDefined", key);
+
+/*
+    continuousNamespace = {
+    isObs: true,
+    isDiffExp: false,
+    isUserDefined: false
+  }
+
+  ie., makeContinuousDimensionName(continuousNamespace = {isObs: true}, "total_reads")
+  see: histogram brush, as it doesn't know what type of continuous it was with only field
+*/
+export const makeContinuousDimensionName = (continuousNamespace, key) => {
+  let name;
+  if (continuousNamespace.isObs) {
+    name = obsAnnoDimensionName(key);
+  } else if (continuousNamespace.isDiffExp) {
+    name = diffexpDimensionName(key);
+  } else if (continuousNamespace.isUserDefined) {
+    name = userDefinedDimensionName(key);
+  } else {
+    throw new Error("unknown continuous dimension");
+  }
+
+  return name;
+};

client/src/util/parseRGB.js

@@ -1,31 +1,30 @@
-// jshint esversion: 6
+// jshint esversion: 6
-import { scaleRGB } from "./scaleRGB";
+import scaleRGB from "./scaleRGB";
-
+
-// maintain a cache of already parsed RGB names, as it is reasonably expensive
+// maintain a cache of already parsed RGB names, as it is reasonably expensive
-// to do this operation.  This lets us have speed, but keep the pleasant ability
+// to do this operation.  This lets us have speed, but keep the pleasant ability
-// to talk about colors by their text description eg, 'rgb(0,0,1)'
+// to talk about colors by their text description eg, 'rgb(0,0,1)'
-//
+//
-const colorCache = new Object(null); // no prototype
+const colorCache = {};
-
+
-function parseColorName(c) {
+function parseColorName(c) {
-  if (c[0] !== "#") {
+  if (c[0] !== "#") {
-    const _c = c.replace(/[^\d,.]/g, "").split(",");
+    const _c = c.replace(/[^\d,.]/g, "").split(",");
-    return [scaleRGB(+_c[0]), scaleRGB(+_c[1]), scaleRGB(+_c[2])];
+    return [scaleRGB(+_c[0]), scaleRGB(+_c[1]), scaleRGB(+_c[2])];
-  } else {
+  }
-    var parsedHex = /^#?([a-f\d]{2})([a-f\d]{2})([a-f\d]{2})$/i.exec(c);
+  const parsedHex = /^#?([a-f\d]{2})([a-f\d]{2})([a-f\d]{2})$/i.exec(c);
-    return [
+  return [
-      scaleRGB(parseInt(parsedHex[1], 16)),
+    scaleRGB(parseInt(parsedHex[1], 16)),
-      scaleRGB(parseInt(parsedHex[2], 16)),
+    scaleRGB(parseInt(parsedHex[2], 16)),
-      scaleRGB(parseInt(parsedHex[3], 16))
+    scaleRGB(parseInt(parsedHex[3], 16))
-    ];
+  ];
-  }
+}
-}
+
-
+export default c => {
-export const parseRGB = c => {
+  let cv = colorCache[c];
-  var cv = colorCache[c];
+  if (!cv) {
-  if (!cv) {
+    cv = parseColorName(c);
-    cv = parseColorName(c);
+    colorCache[c] = cv;
-    colorCache[c] = cv;
+  }
-  }
+  return cv;
-  return cv;
+};
-};

client/src/util/scaleLinear.js

@@ -1,18 +1,16 @@
-// jshint esversion: 6
+// jshint esversion: 6
-
+
-// Substitute for a d3 linear scale - less flexible, more performant.
+// Substitute for a d3 linear scale - less flexible, more performant.
-// Returns a function which will scale a value.
+// Returns a function which will scale a value.
-//
+//
-// Example will scale [0,1] to [-1,1]
+// Example will scale [0,1] to [-1,1]
-//     var myScale = scaleLinear([0, 1], [-1, 1]);
+//     var myScale = scaleLinear([0, 1], [-1, 1]);
-//     myScale(0) === -1
+//     myScale(0) === -1
-// this is is equivalent to d3.scaleLinear().domain([0,1]).range([-1,1])
+// this is is equivalent to d3.scaleLinear().domain([0,1]).range([-1,1])
-
+
-export const scaleLinear = (domain, range) => {
+export default (domain, range) => {
-  const domainStart = domain[0];
+  const domainStart = domain[0];
-  const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+  const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
-  const rangeStart = range[0];
+  const rangeStart = range[0];
-  return function(value) {
+  return value => (value - domainStart) * scale + rangeStart;
-    return (value - domainStart) * scale + rangeStart;
+};
-  };
-};

client/src/util/scaleRGB.js

@@ -1,5 +1,5 @@
 // jshint esversion: 6
-export const scaleRGB = input => {
+export default input => {
   const outputMax = 1;
   const outputMin = 0;
 

client/src/util/stateManager/index.js

@@ -0,0 +1,19 @@
+// jshint esversion: 6
+
+/*
+Model manager providing an abstraction for the use of the reducer code.
+This module provides several buckets of functionality:
+  - schema and config driven tranformation of the dataframe wire protocol
+    into a format that is easy for the UI code to use.
+  - manage the universe/world abstraction:
+    + universe: all of the server-provided, read-only data
+    + world: subset of universe
+  - lazy access and caching of dataframe contents as needed
+
+This is all VERY tightly integrated with reducers and actions, and
+exists to support those concepts.
+*/
+
+export * as Universe from "./universe";
+export * as World from "./world";
+export * as kvCache from "./keyvalcache";

client/src/util/stateManager/keyvalcache.js

@@ -0,0 +1,122 @@
+// jshint esversion: 6
+import _ from "lodash";
+
+/*
+Very simple key/value cache for use by World & Universe.   Cache keys must
+be a string, and values are any JS non-primitive value.
+
+  * constructor(lowWatermark, minTTL):
+      - lowWatermark defines the number of cache elements below which
+        flushing will not occur.
+      - minTTL defines minimum time in milliseconds that cache entries will live.
+        A value of -1 disables automatic flushing (flush() can still
+        be called by external user).
+  * set() - add a key/val pair.
+  * get() - get a value or undefined if not present.
+  * flush(minAgeMs) - flush cache entries in excess of lowWatermark if those
+    entries are older than minAgeMs.
+
+*/
+
+const cachePrivateKey = "__kvcachekey__";
+const defaultLowWatermark = 32;
+const defaultMinTTL = 1000;
+
+function create(lowWatermark = defaultLowWatermark, minTTL = defaultMinTTL) {
+  if (typeof minTTL !== "number" || typeof lowWatermark !== "number") {
+    throw new TypeError(
+      "minTTL and lowWatermark parameters must be a primitive number"
+    );
+  }
+  if (lowWatermark < 0 || minTTL < 0) {
+    throw new RangeError(
+      "minTTL and lowWatermark parameters must be number greater than zero"
+    );
+  }
+
+  return {
+    [cachePrivateKey]: {
+      lowWatermark,
+      minTTL
+    }
+  };
+}
+
+function get(kvcache, key) {
+  if (key === cachePrivateKey) {
+    throw new RangeError(`key parameter may not have value ${cachePrivateKey}`);
+  }
+
+  const val = kvcache[key];
+  if (val) {
+    val[cachePrivateKey] = Date.now();
+  }
+  return val;
+}
+
+function set(kvcache, key, val) {
+  if (key === cachePrivateKey) {
+    throw new RangeError(`key parameter may not have value ${cachePrivateKey}`);
+  }
+
+  const newKvCache = { ...kvcache };
+  newKvCache[key] = val;
+  val[cachePrivateKey] = Date.now();
+  flushInPlace(newKvCache);
+  return newKvCache;
+}
+
+function flush(kvcache) {
+  const newKvCache = { ...kvcache };
+  flushInPlace(newKvCache);
+  return newKvCache;
+}
+
+/*
+Flush elements from cache IF cache size is greater than lowWatermark, and
+those elements are older than minAgeMS
+*/
+function flushInPlace(kvCache) {
+  const { lowWatermark, minTTL } = kvCache[cachePrivateKey];
+  const eol = Date.now() - minTTL;
+  const allKeys = _(kvCache)
+    .keys()
+    .filter(k => k !== cachePrivateKey)
+    .sortBy([k => kvCache[k][cachePrivateKey]])
+    .value();
+
+  if (allKeys.length > lowWatermark) {
+    const keysToDelete = _(allKeys)
+      .slice(0, allKeys.length - lowWatermark)
+      .filter(k => kvCache[k][cachePrivateKey] <= eol)
+      .value();
+    _.forEach(keysToDelete, k => delete kvCache[k]);
+  }
+
+  return kvCache;
+}
+
+/*
+use to create a cache that is a transformation of another cache.
+*/
+function map(srcKvCache, cb, createOptions) {
+  const keysInSrcKvCache = _(srcKvCache)
+    .keys()
+    .filter(k => k !== cachePrivateKey)
+    .value();
+  const lowWatermark = _.get(
+    createOptions,
+    "lowWatermark",
+    defaultLowWatermark
+  );
+  const minTTL = _.get(createOptions, "minTTL", defaultMinTTL);
+  const newKvCache = create(lowWatermark, minTTL);
+  _.forEach(keysInSrcKvCache, key => {
+    const val = cb(get(srcKvCache, key), key);
+    newKvCache[key] = val;
+    val[cachePrivateKey] = Date.now();
+  });
+  return newKvCache;
+}
+
+export { create, get, set, flush, map };

client/src/util/stateManager/summarizeAnnotations.js

@@ -0,0 +1,102 @@
+import _ from "lodash";
+
+/*
+Build and return obs/var summary using any annotation in the schema
+
+Summary information for each annotation, keyed by annotation name.
+Value will be an object, containing summary information.
+
+For continuous annotations (int, float, etc):
+  <annotation_name>: {
+    range {
+      min: <number>,
+      max: <number>
+    }
+  }
+
+For categorical annotations (boolean, string, category):
+  <annotatoin_name>: {
+    options: {
+      <option1>: <number>,
+      ...
+    },
+    numOptions: <number>
+  }
+
+Summarize will be returned for BOTH obs and var annotations.
+
+Example:
+  {
+    "Splice_sites_Annotated": {
+      "range": {
+        "min": 26,
+        "max": 1075869
+      }
+    },
+    "Selection": {
+      numOptions, 6,
+      "options": {
+        "Astrocytes(HEPACAM)": 714,
+        "Endothelial(BSC)": 123,
+        "Oligodendrocytes(GC)": 294,
+        "Neurons(Thy1)": 685,
+        "Microglia(CD45)": 1108,
+        "Unpanned": 665
+      }
+    }
+  }
+
+NOTE: will not summarize the required 'name' annotation, as that is
+specified as unique per element.
+
+TODO: XXX - this data structure coerces all metadata categories into a string
+(ie, stores values as an Object property in the `options` field).   This looses
+information (eg, type) for category types which are not strings.  Consider an
+alterative data structure that does not use the object property for non-string
+data types (and does not use _.countBy to summarize).
+*/
+function summarizeDimension(schema, annotations) {
+  return _(schema)
+    .filter(v => v.name !== "name")
+    .keyBy("name")
+    .mapValues(anno => {
+      const { name, type } = anno;
+      const continuous = type === "int32" || type === "float32";
+
+      if (!continuous) {
+        const categories = _.uniq(_.flatMap(annotations, name));
+        const options = _.countBy(annotations, name);
+        const numOptions = _.size(options);
+        return {
+          numOptions,
+          options,
+          categories
+        };
+      }
+
+      if (continuous) {
+        let min = Number.POSITIVE_INFINITY;
+        let max = Number.NEGATIVE_INFINITY;
+        _.forEach(annotations, obs => {
+          const val = Number(obs[name]);
+          min = val < min ? val : min;
+          max = val > max ? val : max;
+        });
+        return { range: { min, max } };
+      }
+
+      throw new Error("incomprehensible schema");
+    })
+    .value();
+}
+
+export default function summarizeAnnotations(
+  schema,
+  obsAnnotations,
+  varAnnotations
+) {
+  return {
+    obs: summarizeDimension(schema.annotations.obs, obsAnnotations),
+    var: summarizeDimension(schema.annotations.var, varAnnotations)
+  };
+}

client/src/util/stateManager/universe.js

@@ -0,0 +1,260 @@
+// jshint esversion: 6
+
+import _ from "lodash";
+import * as kvCache from "./keyvalcache";
+import summarizeAnnotations from "./summarizeAnnotations";
+
+/*
+Private helper function - create and return a template Universe
+*/
+function templateUniverse() {
+  /* default universe template */
+
+  /* varDataCache config - see kvCache for semantics */
+  const VarDataCacheLowWatermark = 32; // cache element count
+  const VarDataCacheTTLMs = 1000; // min cache time in MS
+
+  return {
+    api: null,
+    finalized: false, // XXX: may not be needed
+
+    nObs: 0,
+    nVar: 0,
+    schema: {},
+
+    /*
+    Annotations
+    */
+    obsAnnotations: [] /* all obs annotations, by obs index */,
+    varAnnotations: [] /* all var annotations, by var index */,
+    obsNameToIndexMap: {} /* reverse map 'name' to index */,
+    varNameToIndexMap: {} /* reverse map 'name' to index */,
+    summary: null /* derived data summaries XXX: consider exploding in place */,
+
+    obsLayout: { X: [], Y: [] } /* xy layout */,
+
+    /*
+    Cache of var data (expression), by var annotation name.   Data can be
+    accesses as a POJO, but if you want caching semantics, use the kvCache
+    API (eg., kvCache.get(), kvCache.set(), ...), which will maintain the
+    LRU semantics.
+    */
+    varDataCache: kvCache.create(VarDataCacheLowWatermark, VarDataCacheTTLMs)
+  };
+}
+
+/*
+This module implements functions that support storage of "Universe",
+aka all of the var/obs data and annotations.
+
+These functions are used exclusively by the actions and reducers to
+build an internal POJO for use by the rendering components.
+*/
+
+/*
+generate any client-side transformations or summarization that
+is independent of REST API response formats.
+*/
+function finalize(universe) {
+  /* A bit of sanity checking! */
+  const { nObs, nVar } = universe;
+  if (
+    nObs !== universe.obsAnnotations.length ||
+    nObs !== universe.obsLayout.X.length ||
+    nObs !== universe.obsLayout.Y.length ||
+    nVar !== universe.varAnnotations.length
+  ) {
+    throw new Error("Universe dimensionality mismatch - failed to load");
+  }
+  // TODO: add more sanity checks, such as:
+  //  - all annotations in the schema
+  //  - layout has supported number of dimensions
+  //  - ...
+
+  /*
+  Create all derived (convenience) data structures.
+  */
+  universe.obsNameToIndexMap = _.transform(
+    universe.obsAnnotations,
+    (acc, value, idx) => {
+      acc[value.name] = idx;
+    },
+    {}
+  );
+  universe.varNameToIndexMap = _.transform(
+    universe.varAnnotations,
+    (acc, value, idx) => {
+      acc[value.name] = idx;
+    },
+    {}
+  );
+  universe.finalized = true;
+  return universe;
+}
+
+function RESTv02AnnotationsResponseToInternal(response) {
+  /*
+  Source per the spec:
+  {
+    names: [
+      'tissue_type', 'sex', 'num_reads', 'clusters'
+    ],
+    data: [
+      [ 0, 'lung', 'F', 39844, 99 ],
+      [ 1, 'heart', 'M', 83, 1 ],
+      [ 49, 'spleen', null, 2, "unknown cluster" ],
+      // [ obsOrVarIndex, value, value, value, value ],
+      // ...
+    ]
+  }
+
+  Internal (target) format:
+  [
+    { __index__: 0, tissue_type: "lung", sex: "F", ... },
+    ...
+  ]
+  */
+  const { names, data } = response;
+  const keys = ["__index__", ...names];
+  return _(data)
+    .map(obs => _.zipObject(keys, obs))
+    .value();
+}
+
+function RESTv02LayoutResponseToInternal(response) {
+  /*
+  Source per the spec:
+  {
+    layout: {
+      ndims: 2,
+      coordinates: [
+        [ 0, 0.284483, 0.983744 ],
+        [ 1, 0.038844, 0.739444 ],
+        // [ obsOrVarIndex, X_coord, Y_coord ],
+        // ...
+      ]
+    }
+  }
+
+  Target (internal) format:
+  {
+    X: Float32Array(numObs),
+    Y: Float32Array(numObs)
+  }
+  In the same order as obsAnnotations
+  */
+  const { ndims, coordinates } = response.layout;
+  if (ndims !== 2) {
+    throw new Error("Unsupported layout dimensionality");
+  }
+
+  const layout = {
+    X: new Float32Array(coordinates.length),
+    Y: new Float32Array(coordinates.length)
+  };
+
+  for (let i = 0; i < coordinates.length; i += 1) {
+    const [idx, x, y] = coordinates[i];
+    layout.X[idx] = x;
+    layout.Y[idx] = y;
+  }
+  return layout;
+}
+
+function reconcileSchemaCategoriesWithSummary(universe) {
+  /*
+  where we treat types as (essentially) categorical metadata, update
+  the schema with data-derived categories (in addition to those in
+  the server declared schema).
+
+  For example, boolean defined fields in the schema do not contain
+  explicit declaration of categories (nor do string fields).  In these
+  cases, add a 'categories' field to the schema so it is accessible.
+  */
+
+  _.forEach(universe.schema.annotations.obs, s => {
+    if (
+      s.type === "string" ||
+      s.type === "boolean" ||
+      s.type === "categorical"
+    ) {
+      const categories = _.union(
+        _.get(s, "categories", []),
+        _.get(universe.summary.obs[s.name], "categories", [])
+      );
+      s.categories = categories;
+    }
+  });
+}
+
+export function createUniverseFromRestV02Response(
+  configResponse,
+  schemaResponse,
+  annotationsObsResponse,
+  annotationsVarResponse,
+  layoutObsResponse
+) {
+  /*
+  build & return universe from a REST 0.2 /config, /schema and /annotations/obs response
+  */
+  const { schema } = schemaResponse;
+  const universe = templateUniverse();
+
+  /* constants */
+  universe.api = "0.2";
+
+  /* schema related */
+  universe.schema = schema;
+  universe.nObs = schema.dataframe.nObs;
+  universe.nVar = schema.dataframe.nVar;
+
+  /* annotations */
+  universe.obsAnnotations = RESTv02AnnotationsResponseToInternal(
+    annotationsObsResponse
+  );
+  universe.varAnnotations = RESTv02AnnotationsResponseToInternal(
+    annotationsVarResponse
+  );
+
+  /* layout */
+  universe.obsLayout = RESTv02LayoutResponseToInternal(layoutObsResponse);
+
+  universe.summary = summarizeAnnotations(
+    universe.schema,
+    universe.obsAnnotations,
+    universe.varAnnotations
+  );
+
+  reconcileSchemaCategoriesWithSummary(universe);
+  return finalize(universe);
+}
+
+export function convertExpressionRESTv02ToObject(universe, response) {
+  /*
+  /data/obs response looks like:
+  {
+    var: [ varIndices fetched ],
+    obs: [
+      [ obsIndex, evalue, ... ],
+      ...
+    ]
+  }
+
+  convert expression toa simple Float32Array, and return
+  { geneName: array, geneName: array, ... }
+  NOTE: geneName, not varIndex
+  */
+  const vars = response.var;
+  const { obs } = response;
+  const result = {};
+  // XXX TODO: could this use _.unzip and have less code?
+  for (let varIdx = 0; varIdx < vars.length; varIdx += 1) {
+    const gene = universe.varAnnotations[vars[varIdx]].name;
+    const data = new Float32Array(universe.nObs);
+    for (let obsIdx = 0; obsIdx < obs.length; obsIdx += 1) {
+      data[obsIdx] = obs[obsIdx][varIdx + 1];
+    }
+    result[gene] = data;
+  }
+  return result;
+}

client/src/util/stateManager/world.js

@@ -0,0 +1,284 @@
+// jshint esversion: 6
+
+import _ from "lodash";
+import * as kvCache from "./keyvalcache";
+import summarizeAnnotations from "./summarizeAnnotations";
+import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
+import { sliceByIndex } from "../typedCrossfilter/util";
+
+/*
+World is a subset of universe.   Most code should use world, and should
+(generally) not use Universe.   World contains any per-obs or per-var data
+that must be consistent acorss the app when we view/manipulate subsets
+of Universe.
+
+Private API indicated by leading underscore in key name (eg, _foo).  Anything else
+is public.
+
+World contains several public keys, obsAnnotations, and obsLayout, which are
+arrays contianing information about an OBS in the same order/offset.  In
+other words, world.obsAnnotations[0] and world.obsLayout.X[0] refer to the same
+obs/cell.
+
+* obsAnnotations:
+
+  obsAnnotations will return an array of objects.  Each object contains all annotation
+  values for a given observation/cell, keyed by annotation name, PLUS a key
+  '__cellId__', containing a REST API ID for this obs/cell (referred to as the
+  obsIndex in the REST 0.2 spec or cellIndex in the 0.1 spec.
+
+  Example:  [ { __cellId__: 99, cluster: 'blue', numReads: 93933 } ]
+
+  NOTE: world.obsAnnotation should be identical to the old state.cells value,
+  EXCEPT that
+    * __cellIndex__ renamed to __index__
+    * __x__ and __y__ are now in world.obsLayout
+    * __color__ and __colorRBG__ should be moved to controls reducer
+
+* obsLayout:
+
+  obsLayout will return an object containing two arrays, containing X and Y
+  coordinates respectively.
+
+  Example: { X: [ 0.33, 0.23, ... ], Y: [ 0.8, 0.777, ... ]}
+
+* crossfilter - a crossfilter object across world.obsAnnotations
+
+* dimensionMap - an object mapping annotation names to dimensions on
+  the crossfilter
+
+*/
+
+/* varDataCache config - see kvCache for semantics */
+const VarDataCacheLowWatermark = 32; // cache element count
+const VarDataCacheTTLMs = 1000; // min cache time in MS
+
+function templateWorld() {
+  return {
+    // map from universe obsIndex to world offset.
+    // Undefined / null indicates identity mapping.
+    obsIndex: null,
+    obsBackIndex: null,
+
+    /* schema/version related */
+    api: null,
+    schema: null,
+    nObs: 0,
+    nVar: 0,
+
+    /* annotations */
+    obsAnnotations: null,
+    varAnnotations: null,
+
+    /* layout of graph */
+    obsLayout: null,
+
+    /* derived data summaries XXX: consider exploding in place */
+    summary: null,
+
+    varDataCache: kvCache.create(
+      VarDataCacheLowWatermark,
+      VarDataCacheTTLMs
+    ) /* cache of var data (expression) */
+  };
+}
+
+export function createWorldFromEntireUniverse(universe) {
+  if (!universe.finalized) {
+    throw new Error("World can't be created from an partial Universe");
+  }
+
+  const world = templateWorld();
+
+  // map from the universe obsIndex to our world offset.
+  // undefined/null indicates identity map.
+  // In other words   obsBackIndex[universeIdx] -> worldIdx
+  world.obsBackIndex = null;
+  // Map to the universe index for each element in world.
+  // Null indicates identity map (aka world === universe)
+  // In other wrods  obsIndex[worldIdx]  ->  universeIdx
+  world.obsIndex = null;
+
+  /*
+  public interface follows
+  */
+
+  /* Schema related */
+  world.api = universe.api;
+  world.schema = universe.schema;
+  world.nObs = universe.nObs;
+  world.nVar = universe.nVar;
+
+  /* annotations */
+  world.obsAnnotations = universe.obsAnnotations;
+  world.varAnnotations = universe.varAnnotations;
+
+  /* layout and display characteristics */
+  world.obsLayout = universe.obsLayout;
+
+  /* derived data & summaries */
+  world.summary = summarizeAnnotations(
+    world.schema,
+    world.obsAnnotations,
+    world.varAnnotations
+  );
+
+  /* build the varDataCache */
+  world.varDataCache = kvCache.map(
+    universe.varDataCache,
+    val => subsetVarData(world, universe, val),
+    { lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
+  );
+
+  return world;
+}
+
+export function createWorldFromCurrentSelection(universe, world, crossfilter) {
+  const newWorld = templateWorld();
+
+  /* these don't change as only OBS are selected in our current implementation */
+  newWorld.api = universe.api;
+  newWorld.nVar = universe.nVar;
+  newWorld.schema = universe.schema;
+  newWorld.varAnnotations = universe.varAnnotations;
+
+  /* build index maps and back maps based upon current selection state */
+  const obsBackIndex = new Uint32Array(universe.nObs);
+  obsBackIndex.fill(-1); // default - aka unused
+  const notSelected = obsBackIndex[0];
+  let nObs = 0;
+  for (let i = 0; i < universe.nObs; i += 1) {
+    if (crossfilter.isElementFiltered(i)) {
+      obsBackIndex[i] = nObs;
+      nObs += 1;
+    }
+  }
+  const obsIndex = new Uint32Array(nObs);
+  for (let i = 0; i < universe.nObs; i += 1) {
+    const worldIdx = obsBackIndex[i];
+    if (worldIdx !== notSelected) {
+      obsIndex[worldIdx] = i;
+    }
+  }
+
+  newWorld.nObs = nObs;
+  newWorld.obsIndex = obsIndex;
+  newWorld.obsBackIndex = obsBackIndex;
+
+  /* now slice */
+  newWorld.obsAnnotations = sliceByIndex(universe.obsAnnotations, obsIndex);
+  newWorld.obsLayout = {
+    X: sliceByIndex(universe.obsLayout.X, obsIndex),
+    Y: sliceByIndex(universe.obsLayout.Y, obsIndex)
+  };
+
+  /* derived data & summaries */
+  newWorld.summary = summarizeAnnotations(
+    newWorld.schema,
+    newWorld.obsAnnotations,
+    newWorld.varAnnotations
+  );
+
+  /* build the varDataCache */
+  newWorld.varDataCache = kvCache.map(
+    universe.varDataCache,
+    val => subsetVarData(newWorld, universe, val),
+    { lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
+  );
+  return newWorld;
+}
+
+/*
+ Deduce the correct crossfilter dimension type from a metadata
+ schema description.
+*/
+function deduceDimensionType(attributes, fieldName) {
+  let dimensionType;
+  const { type } = attributes;
+  if (type === "string" || type === "categorical" || type === "boolean") {
+    dimensionType = "enum";
+  } else if (type === "int32") {
+    dimensionType = Int32Array;
+  } else if (type === "float32") {
+    dimensionType = Float32Array;
+  } else {
+    /*
+    Currently not supporting boolean and categorical types.
+    */
+    console.error(
+      `Warning - REST API returned unknown metadata schema (${type}) for field ${fieldName}.`
+    );
+    // skip it - we don't know what to do with this type
+  }
+  return dimensionType;
+}
+
+/*
+  Return a crossfilter dimension for the specified world & named gene.
+
+  NOTE: this assumes that the expression data was already loaded,
+  by calling an appropriate action creator.
+
+  Caller needs to *save* this dimension somewhere for it to be later used.
+  Dimension must be destroyed by calling dimension.dispose()
+  when it is no longer needed
+  (it will not be garbage collected without this call)
+*/
+
+export function createVarDimension(
+  world,
+  _worldVarDataCache,
+  crossfilter,
+  geneName
+) {
+  return crossfilter.dimension(_worldVarDataCache[geneName], Float32Array);
+}
+
+export function createObsDimensionMap(crossfilter, world) {
+  /*
+  create and return a crossfilter dimension for every obs annotation
+  for which we have a supported type.
+  */
+  const { schema, obsLayout } = world;
+
+  // Create a crossfilter dimension for all obs annotations *except* 'name'
+  const dimensionMap = _(schema.annotations.obs)
+    .filter(anno => anno.name !== "name")
+    .transform((result, anno) => {
+      const dimType = deduceDimensionType(anno, anno.name);
+      // XXX if dimtype is a scalar, we may be able to do better?
+      if (dimType) {
+        result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
+          r => r[anno.name],
+          dimType
+        );
+      } // else ignore the annotation
+    }, {})
+    .value();
+
+  /*
+  Add crossfilter dimensions allowing filtering on layout
+  */
+  dimensionMap[layoutDimensionName("X")] = crossfilter.dimension(
+    obsLayout.X,
+    Float32Array
+  );
+  dimensionMap[layoutDimensionName("Y")] = crossfilter.dimension(
+    obsLayout.Y,
+    Float32Array
+  );
+
+  return dimensionMap;
+}
+
+export function worldEqUniverse(world, universe) {
+  return world.obsAnnotations === universe.obsAnnotations;
+}
+
+export function subsetVarData(world, universe, varData) {
+  // If world === universe, just return the entire varData array
+  if (worldEqUniverse(world, universe)) {
+    return varData;
+  }
+  return sliceByIndex(varData, world.obsIndex);
+}

client/src/util/typedCrossfilter/bitArray.js

@@ -0,0 +1,252 @@
+// jshint esversion: 6
+/* eslint no-bitwise: "off" */
+
+// BitArray is a 2D bitarray with size [length, nBitWidth].
+// Each bit is referred to as a `dimension`.  Dimensions may be
+// dynamically allocated and deallocated.  The overall length
+// of the BitArray is fixed at creation time (for simplicity).
+//
+// Organization of the bitarray is dimension-major.  As dimensions
+// are added, the underlying store is grown 32 bits at a time.
+// NOTE: currently does not deallocate / shrink.
+//
+// Primary operations on the BitArray are:
+//    - set & clear dimension
+//    - test dimension
+//    - various performance or convenience operations to optimize bulk ops
+//
+// The underlying data structure uses TypedArrays for performance.
+//
+class BitArray {
+  constructor(length) {
+    // Initially allocate a 32 bit wide array.  allocDimension() will expand
+    // as necessary.
+    //
+    // Int32Array is (counterintuitively) used to accomadate JS numeric casting
+    // (to/from primitive number type).
+    //
+
+    // Fixed for the life of this object.
+    this.length = length;
+
+    // Bitarray width.  width is always greater than dimensionCount/32.
+    this.width = 1; // underlying number of 32 bit arrays
+    this.dimensionCount = 0; // num allocated dimensions
+
+    this.bitmask = new Int32Array(this.width); // dimension allocation mask
+    this.bitarray = new Int32Array(this.width * this.length);
+  }
+
+  // Return the number of records that are selected, ie, have a one bit in
+  // all allocated dimensions.
+  //
+  get selectionCount() {
+    return this.countAllOnes();
+  }
+
+  // Count all records that have a 'one' bit in allocated dimensions.
+  //
+  countAllOnes() {
+    let count = 0;
+    const { bitarray, bitmask, length, width } = this;
+    for (let l = 0; l < length; l += 1) {
+      let dimensionsSet = 0;
+      for (let w = 0; w < width; w += 1) {
+        if (bitarray[w * length + l] === bitmask[w]) {
+          dimensionsSet += 1;
+        }
+      }
+      if (dimensionsSet === width) {
+        count += 1;
+      }
+    }
+    return count;
+  }
+
+  // count trailing zeros - hard to do fast in JS!
+  // https://en.wikipedia.org/wiki/Find_first_set#CTZ
+  static ctz(av) {
+    let c = 32;
+    let v = av;
+    v &= -v; // isolate lowest non-zero bit
+    if (v) c -= 1;
+    if (v & 0x0000ffff) c -= 16;
+    if (v & 0x00ff00ff) c -= 8;
+    if (v & 0x0f0f0f0f) c -= 4;
+    if (v & 0x33333333) c -= 2;
+    if (v & 0x55555555) c -= 1;
+    return c;
+  }
+
+  // find a free dimension.  Return undefined if none
+  _findFreeDimension() {
+    let dim;
+    for (let col = 0; col < this.width; col += 1) {
+      const lowestZeroBit = ~this.bitmask[col] & -~this.bitmask[col];
+      if (lowestZeroBit) {
+        this.bitmask[col] |= lowestZeroBit;
+        dim = 32 * col + BitArray.ctz(lowestZeroBit);
+      }
+    }
+    return dim;
+  }
+
+  // allocate and return the dimension ID (bit position)
+  //
+  allocDimension() {
+    let dim = this._findFreeDimension();
+
+    // if we did not find free dimension, expand the bitarray.
+    if (dim === undefined) {
+      this.width += 1;
+
+      const biggerBitArray = new Int32Array(this.width * this.length);
+      biggerBitArray.set(this.bitarray);
+      this.bitarray = biggerBitArray;
+
+      const biggerBitmask = new Int32Array(this.width);
+      biggerBitmask.set(this.bitmask);
+      this.bitmask = biggerBitmask;
+
+      dim = this._findFreeDimension();
+    }
+
+    this.dimensionCount += 1;
+    return dim;
+  }
+
+  // free a dimension for later use.  MUST deselect the dimension, as other
+  // code assume the column will be zero valued.
+  //
+  freeDimension(dim) {
+    // all selection tests assume unallocated dimensions are zero valued.
+    this.deselectAll(dim);
+    const col = dim >>> 5;
+    this.bitmask[col] &= ~(1 << dim % 32);
+    this.dimensionCount -= 1;
+  }
+
+  // return true if this index is selected in ALL dimensions.
+  //
+  isSelected(index) {
+    const { width, length, bitarray } = this;
+
+    for (let w = 0; w < width; w += 1) {
+      const bitmask = this.bitmask[w];
+      if (!bitmask || bitarray[w * length + index] !== bitmask) return false;
+    }
+    return true;
+  }
+
+  // return true if this index is selected in ALL dimensions IGNORING dim
+  //
+  isSelectedIgnoringDim(index, dim) {
+    const ignoreOffset = dim >>> 5;
+    const ignoreMask = ~(1 << dim % 32);
+
+    const { width, length, bitarray } = this;
+
+    for (let w = 0; w < width; w += 1) {
+      const bitmask = this.bitmask[w];
+      if (w === ignoreOffset) {
+        if (
+          bitmask &&
+          (bitarray[w * length + index] & ignoreMask) !== (bitmask & ignoreMask)
+        ) {
+          return false;
+        }
+      } else if (bitmask && bitarray[w * length + index] !== bitmask) {
+        return false;
+      }
+    }
+    return true;
+  }
+
+  // select index on dimension
+  //
+  selectOne(dim, index) {
+    const col = dim >>> 5;
+    const before = this.bitarray[col * this.length + index];
+    const after = before | (1 << dim % 32);
+    this.bitarray[col * this.length + index] = after;
+  }
+
+  // deselect index on dimension
+  //
+  deselectOne(dim, index) {
+    const col = dim >>> 5;
+    const before = this.bitarray[col * this.length + index];
+    const after = before & ~(1 << dim % 32);
+    this.bitarray[col * this.length + index] = after;
+  }
+
+  // select all indices on dimension.
+  //
+  selectAll(dim) {
+    const col = dim >> 5;
+    const one = 1 << dim % 32;
+    for (let i = col * this.length, len = i + this.length; i < len; i += 1) {
+      this.bitarray[i] |= one;
+    }
+  }
+
+  // deselect all indices on dimension
+  //
+  deselectAll(dim) {
+    const col = dim >> 5;
+    const zero = ~(1 << dim % 32);
+    for (let i = col * this.length, len = i + this.length; i < len; i += 1) {
+      this.bitarray[i] &= zero;
+    }
+  }
+
+  // select range of indices on a dimension, indirect through a sort map.
+  // Indirect functions are used to map between sort and natural order.
+  //
+  selectIndirectFromRange(dim, indirect, range) {
+    const col = dim >>> 5;
+    const first = range[0];
+    const last = range[1];
+    const one = 1 << dim % 32;
+    const offset = col * this.length;
+    for (let i = first; i < last; i += 1) {
+      this.bitarray[offset + indirect[i]] |= one;
+    }
+  }
+
+  // deselect range of indices on a dimension, indirect through a sort map.
+  //
+  deselectIndirectFromRange(dim, indirect, range) {
+    const col = dim >>> 5;
+    const first = range[0];
+    const last = range[1];
+    const zero = ~(1 << dim % 32);
+    const offset = col * this.length;
+    for (let i = first; i < last; i += 1) {
+      this.bitarray[offset + indirect[i]] &= zero;
+    }
+  }
+
+  // Fill the array with selected|deselected value based upon the
+  // current selection state.
+  //
+  fillBySelection(result, selectedValue, deselectedValue) {
+    // special case (width === 1) for performance
+    if (this.width === 1) {
+      const bitmask = this.bitmask[0];
+      for (let i = 0, len = this.length; i < len; i += 1) {
+        result[i] =
+          bitmask && this.bitarray[i] === bitmask
+            ? selectedValue
+            : deselectedValue;
+      }
+    } else {
+      for (let i = 0, len = this.length; i < len; i += 1) {
+        result[i] = this.isSelected(i) ? selectedValue : deselectedValue;
+      }
+    }
+    return result;
+  }
+}
+
+export default BitArray;

client/src/util/typedCrossfilter/index.js

@@ -0,0 +1,615 @@
+// jshint esversion: 6
+
+/*
+Typedarray Crossfilter - a re-implementation of a subset of crossfilter, with
+time/space optimizations predicated upon the following assumptions:
+  - dimensions are uniformly typed, and all values must be of that type
+  - dimension values must be a primitive type (int, float, string).  Arrays
+    or other complex types not supported.
+  - dimension creation requires call-provided type declaration
+  - no support for adding/removing data to an existing crossfilter.  If you
+    want to do that, you have to create the new crossfilter, using the new
+    data, from scratch.
+
+The actual backing store for a dimension is a TypedArray, enabling significant
+performance improvements over the original crossfilter.
+
+There are also a handful of new methods, primarily to take advantage of the
+performance (eg, crossfilter.fillBySelection)
+
+Helpful documents (this module tries to follow the original API as much
+as is feasable):
+    https://github.com/square/crossfilter/
+    http://square.github.io/crossfilter/
+
+There is also a newer, community supported fork of crossfilter, with a
+more complex API.  In a few cases, elements of that API were incorporated.
+    https://github.com/square/crossfilter/
+
+*/
+
+import PositiveIntervals from "./positiveIntervals";
+import BitArray from "./bitArray";
+import {
+  makeSortIndex,
+  lowerBound,
+  lowerBoundIndirect,
+  upperBoundIndirect
+} from "./util";
+
+class NotImplementedError extends Error {
+  constructor(...params) {
+    super(...params);
+
+    // Maintains proper stack trace for where our error was thrown (only available on V8)
+    if (Error.captureStackTrace) {
+      Error.captureStackTrace(this, NotImplementedError);
+    }
+  }
+}
+
+class TypedCrossfilter {
+  constructor(data) {
+    this.data = data;
+
+    // filters: array of { id, dimension }
+    this.filters = [];
+    this.selection = new BitArray(data.length);
+    this.updateTime = 0;
+  }
+
+  size() {
+    return this.data.length;
+  }
+
+  all() {
+    return this.data;
+  }
+
+  dimension(value, valueArrayType) {
+    const id = this.selection.allocDimension();
+    let dim;
+    if (valueArrayType === "enum") {
+      dim = new EnumDimension(value, this, id);
+    } else {
+      dim = new ScalarDimension(value, valueArrayType, this, id);
+    }
+    this.filters.push({ id, dim });
+    dim.filterAll();
+    return dim;
+  }
+
+  _freeDimension(id) {
+    this.selection.freeDimension(id);
+    this.filters = this.filters.filter(f => f._id !== id);
+  }
+
+  // return array of all records that are selected/filtered
+  // by all dimensions.
+  allFiltered() {
+    const { selection } = this;
+    const res = [];
+    for (let i = 0, len = this.data.length; i < len; i += 1) {
+      if (selection.isSelected(i)) {
+        res.push(this.data[i]);
+      }
+    }
+    return res;
+  }
+
+  countFiltered() {
+    return this.selection.selectionCount;
+  }
+
+  isElementFiltered(i) {
+    return this.selection.isSelected(i);
+  }
+
+  // fill array with one of two values, based upon selection state
+  fillByIsFiltered(array, selectedValue, deselectedValue) {
+    return this.selection.fillBySelection(
+      array,
+      selectedValue,
+      deselectedValue
+    );
+  }
+}
+
+// Base dimension type - value must be a scalar type (eg, int, float),
+// and value array must be a TypedArray.
+//
+class ScalarDimension {
+  constructor(value, ValueArrayType, xfltr, id) {
+    this.crossfilter = xfltr;
+    this._id = id;
+
+    // current selection filter, expressed as PostiveIntervals.
+    this.currentFilter = [];
+
+    // Two modes - caller can provide a pre-created value array,
+    // or a map function which will create it.
+    let array;
+    if (value instanceof ValueArrayType) {
+      if (value.length !== this.crossfilter.data.length) {
+        throw new RangeError(
+          "ScalarDimension values length must equal crossfilter data record count"
+        );
+      }
+      array = value;
+    } else if (value instanceof Function) {
+      // Create value array
+      array = this._createValueArray(
+        value,
+        new ValueArrayType(this.crossfilter.data.length)
+      );
+    } else {
+      throw new NotImplementedError(
+        "dimension value must be function or value array type"
+      );
+    }
+    this.value = array;
+
+    // create sort index
+    this.index = makeSortIndex(array);
+
+    // groups, if any
+    this.groups = [];
+  }
+
+  _createValueArray(value, array) {
+    // create dimension value array
+    const { data } = this.crossfilter;
+    const len = data.length;
+    const larray = array;
+    for (let i = 0; i < len; i += 1) {
+      larray[i] = value(data[i]);
+    }
+    return larray;
+  }
+
+  dispose() {
+    this.crossfilter._freeDimension(this._id);
+    return this;
+  }
+
+  id() {
+    return this._id;
+  }
+
+  // Argument is an array of intervals indicating records newly selected/filtered
+  //
+  _updateFilters(newFilter) {
+    const cNewFilter = PositiveIntervals.canonicalize(newFilter);
+
+    const adds = PositiveIntervals.difference(cNewFilter, this.currentFilter);
+    const dels = PositiveIntervals.difference(this.currentFilter, cNewFilter);
+
+    this.crossfilter.filters.forEach(f =>
+      f.dim.groups.forEach(grp => grp._updateReduceDel(this, dels))
+    );
+
+    dels.forEach(interval =>
+      this.crossfilter.selection.deselectIndirectFromRange(
+        this._id,
+        this.index,
+        interval
+      )
+    );
+
+    adds.forEach(interval =>
+      this.crossfilter.selection.selectIndirectFromRange(
+        this._id,
+        this.index,
+        interval
+      )
+    );
+
+    this.crossfilter.filters.forEach(f =>
+      f.dim.groups.forEach(grp => grp._updateReduceAdd(this, adds))
+    );
+
+    this.currentFilter = cNewFilter;
+    this.crossfilter.updateTime += 1;
+  }
+
+  // filter by value - exact match
+  filterExact(value) {
+    const newFilter = [
+      lowerBoundIndirect(this.value, this.index, value, 0, this.value.length),
+      upperBoundIndirect(this.value, this.index, value, 0, this.value.length)
+    ];
+    if (newFilter[0] <= newFilter[1]) {
+      this._updateFilters([newFilter]);
+    } else {
+      this._updateFilters([]);
+    }
+    return this;
+  }
+
+  // filter by a set of values, eg. enum.
+  filterEnum(values) {
+    const newFilter = [];
+    for (let v = 0, len = values.length; v < len; v += 1) {
+      const intv = [
+        lowerBoundIndirect(
+          this.value,
+          this.index,
+          values[v],
+          0,
+          this.value.length
+        ),
+        upperBoundIndirect(
+          this.value,
+          this.index,
+          values[v],
+          0,
+          this.value.length
+        )
+      ];
+      if (intv[0] <= intv[1]) newFilter.push(intv);
+    }
+    this._updateFilters(newFilter);
+    return this;
+  }
+
+  // filter by value range [lo, hi)
+  // lo: inclusive, hi: exclusive
+  filterRange(range) {
+    const newFilter = [];
+    const intv = [
+      lowerBoundIndirect(
+        this.value,
+        this.index,
+        range[0],
+        0,
+        this.value.length
+      ),
+      upperBoundIndirect(this.value, this.index, range[1], 0, this.value.length)
+    ];
+    if (intv[0] < intv[1]) newFilter.push(intv);
+    this._updateFilters(newFilter);
+    return this;
+  }
+
+  // select all - equivalent of selecting all in this dimension
+  filterAll() {
+    this._updateFilters([[0, this.value.length]]);
+    return this;
+  }
+
+  // select none
+  filterNone() {
+    this._updateFilters([]);
+  }
+
+  // return top k records, starting with offset, in descending order.
+  // Order is this dimension's sort order
+  top(k, offset = 0) {
+    const { data, selection } = this.crossfilter;
+    const { index } = this;
+    const len = index.length;
+    const ret = [];
+    let i = 0;
+    let skip = 0;
+    let found = 0;
+
+    // skip up to offset records
+    for (i = len - 1; i >= 0 && skip < offset; i -= 1) {
+      if (selection.isSelected(index[i])) {
+        skip += 1;
+      }
+    }
+
+    // grab up to k records
+    for (; i >= 0 && found < k; i -= 1) {
+      if (selection.isSelected(index[i])) {
+        ret.push(data[index[i]]);
+        found += 1;
+      }
+    }
+
+    return ret;
+  }
+
+  // return bottom k records, starting with offset, in ascending order.
+  // Order is this dimension's sort order
+  bottom(k, offset = 0) {
+    const { data, selection } = this.crossfilter;
+    const { index } = this;
+    const len = index.length;
+    const ret = [];
+    let skip = 0;
+    let found = 0;
+    let i = 0;
+
+    // skip up to offset records
+    for (i = 0; i < len && skip < offset; i += 1) {
+      if (selection.isSelected(index[i])) {
+        skip += 1;
+      }
+    }
+
+    // grab up to k records
+    for (; i < len && found < k; i += 1) {
+      if (selection.isSelected(index[i])) {
+        ret.push(data[index[i]]);
+        found += 1;
+      }
+    }
+
+    return ret;
+  }
+
+  group(groupValue) {
+    const grp = new ScalarGroup(groupValue, this.value.constructor, this);
+    this.groups.push(grp);
+    return grp;
+  }
+
+  _freeGroup(group) {
+    this.groups = this.groups.filter(e => e !== group);
+  }
+}
+
+// Ordered enumeration - supports any sortable enumerable type, eg,
+// strings, which can be mapped into an fixed numeric range [0..n).
+//
+class EnumDimension extends ScalarDimension {
+  constructor(value, xfltr, id) {
+    super(value, Uint32Array, xfltr, id);
+  }
+
+  _createValueArray(value, array) {
+    const { data } = this.crossfilter;
+    const len = data.length;
+    const larray = array;
+
+    // create enumeration table - mapping between the value
+    // and the enum.
+    const s = new Set();
+    for (let i = 0; i < len; i += 1) {
+      s.add(value(data[i]));
+    }
+    this.enumIndex = Array.from(s);
+    this.enumIndex.sort();
+
+    // create dimension value array
+    const enumLen = this.enumIndex.length;
+    for (let i = 0; i < len; i += 1) {
+      const v = value(data[i]);
+      const e = lowerBound(this.enumIndex, v, 0, enumLen);
+      larray[i] = e;
+    }
+    return larray;
+  }
+
+  filterExact(value) {
+    return super.filterExact(
+      lowerBound(this.enumIndex, value, 0, this.enumIndex.length)
+    );
+  }
+
+  filterEnum(values) {
+    return super.filterEnum(
+      values.map(v => lowerBound(this.enumIndex, v, 0, this.enumIndex.length))
+    );
+  }
+
+  filterRange(range) {
+    return super.filterEnum(
+      range.map(v => lowerBound(this.enumIndex, v, 0, this.enumIndex.length))
+    );
+  }
+
+  group(groupValue) {
+    const grp = new EnumGroup(groupValue, this.value.constructor, this);
+    this.groups.push(grp);
+    return grp;
+  }
+}
+
+// Groups!  Map/reduce
+//
+class ScalarGroup {
+  constructor(groupValue, groupValueType, dimension) {
+    // parent dimension
+    this.dimension = dimension;
+
+    // generate group names from dimension values
+    this.mapValue = this.constructor._map(
+      groupValue,
+      groupValueType,
+      dimension
+    );
+
+    // group index is mapping from data record index to group index
+    this.groupIndex = new Uint32Array(dimension.crossfilter.data.length);
+
+    // default to counting
+    this.reduceCount();
+
+    // Creates this.groups
+    this._reduce();
+  }
+
+  // internal support function - map all dimension values to group values.
+  //
+  static _map(groupValue, GroupValueType, dimension) {
+    // groupValue is optional.  Defaults to identity.  Used to perform
+    // initial map operation.
+    //
+    // identity: save some memory...
+    if (groupValue === undefined) return dimension.value;
+
+    const data = dimension.value;
+    const len = data.length;
+    const mapValue = new GroupValueType(dimension.value.length);
+    for (let i = 0; i < len; i += 1) {
+      mapValue[i] = groupValue(data[i]);
+    }
+    return mapValue;
+  }
+
+  // Update the group reduction incrementally.  Called when *any* dimension filter
+  // changes.  Guaranteed to be called AFTER the crossfilter is updated.
+  //
+  // Arguments:
+  //  * dim: the dimension that is changing
+  //  * intv: interval list of newly selected values on `dim` (adds)
+  //
+  _updateReduceAdd(dim, intv) {
+    // ignore updates to self, as we don't reduce inclusive of our filter
+    if (dim === this.dimension || intv.length === 0) return;
+
+    // Each item in the range was just added to `dim`.  It was NOT previously
+    // selected - reduceAdd if it is now selected.
+    const { data, selection } = this.dimension.crossfilter;
+    intv.forEach(rng => {
+      for (let r = rng[0]; r < rng[1]; r += 1) {
+        const i = dim.index[r];
+        if (selection.isSelectedIgnoringDim(i, this.dimension.id())) {
+          const group = this.groups[this.groupIndex[i]];
+          group.value = this.reduceAdd(group.value, data[i]);
+        }
+      }
+    });
+  }
+
+  // Update the group reduction incrementally.  Called when *any* dimension filter
+  // changes.   Guaranteed to be called BEFORE the crossfilter is updated.
+  //
+  // Arguments:
+  //  * dim: the dimension that is changing
+  //  * intv: interval list of previously selected values on `dim` (dels)
+  //
+  _updateReduceDel(dim, intv) {
+    // ignore updates to self, as we don't reduce inclusive of our filter
+    if (dim === this.dimension || intv.length === 0) return;
+
+    // Each item in the range will be remved from `dim`.  reduceRemove if it
+    // is currently selected.
+    const { data, selection } = this.dimension.crossfilter;
+    intv.forEach(rng => {
+      for (let r = rng[0]; r < rng[1]; r += 1) {
+        const i = dim.index[r];
+        if (selection.isSelectedIgnoringDim(i, this.dimension.id())) {
+          const group = this.groups[this.groupIndex[i]];
+          group.value = this.reduceRemove(group.value, data[i]);
+        }
+      }
+    });
+  }
+
+  // Reduce the entire data set, creating both the group index and the
+  // groups data.
+  //
+  _reduce() {
+    const { dimension } = this;
+    const { data } = dimension.crossfilter;
+
+    // Create groups
+    const groupNames = new Set(this.mapValue);
+    this.groups = [];
+    const groupIndexByName = {};
+    groupNames.forEach(name => {
+      this.groups.push({ key: name, value: this.reduceInitial() });
+      groupIndexByName[name] = this.groups.length - 1;
+    });
+
+    // Create groupIndex - index map between data record index and group index
+    for (let i = 0, len = this.mapValue.length; i < len; i += 1) {
+      this.groupIndex[i] = groupIndexByName[this.mapValue[i]];
+    }
+
+    // reduce all filtered records, IGNORING the current dimension's filter
+    const { selection } = dimension.crossfilter;
+    for (let i = 0, len = data.length; i < len; i += 1) {
+      if (selection.isSelectedIgnoringDim(i, dimension.id())) {
+        const group = this.groups[this.groupIndex[i]];
+        group.value = this.reduceAdd(group.value, data[i]);
+      }
+    }
+  }
+
+  dispose() {
+    this.dimension._freeGroup(this);
+    return this;
+  }
+
+  // return number of distinct values in the group, independent of any filters.
+  //
+  size() {
+    return this.groups.length;
+  }
+
+  // Set the reduce functions and return the grouping.
+  //
+  reduce(add, remove, initial) {
+    this.reduceAdd = add;
+    this.reduceRemove = remove;
+    this.reduceInitial = initial;
+    this._reduce();
+    return this;
+  }
+
+  // set the reduce functions to count records.
+  reduceCount() {
+    return this.reduce(p => p + 1, p => p - 1, () => 0);
+  }
+
+  // set the reduce functions to sum records using specified value accessor.
+  //
+  reduceSum(value) {
+    return this.reduce((p, v) => p + value(v), (p, v) => p - value(v), () => 0);
+  }
+
+  all() {
+    const res = [...this.groups];
+    res.sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
+    return res;
+  }
+}
+
+class EnumGroup extends ScalarGroup {
+  static _map(groupValue, groupValueType, dimension) {
+    // groupValue is optional.  Defaults to identity.  Used to perform
+    // initial map operation.
+    //
+    // identity: save some memory
+    if (groupValue === undefined) return dimension.value;
+
+    // non-identity mapping unsupported for EnumDimension/EnumGroup.
+    // XXX: this could be implemented, but would require another index
+    // array to map from the group names/keys back to the dimension values.
+    // With this, we just rely on the dimensions `enumIndex` to map from
+    // enumeration value to the record.
+    throw new NotImplementedError("enumerated group mapping not implemented");
+  }
+
+  all() {
+    const res = [];
+    this.groups.forEach(e =>
+      res.push({
+        // XXX: assumes identity group map - see comment in _map()
+        key: this.dimension.enumIndex[e.key],
+        value: e.value
+      })
+    );
+    res.sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
+    return res;
+  }
+}
+
+// Wrapper for backwards compat with crossfilter.
+//
+function crossfilter(data) {
+  return new TypedCrossfilter(data);
+}
+
+crossfilter.PositiveIntervals = PositiveIntervals;
+crossfilter.BitArray = BitArray;
+crossfilter.TypedCrossfilter = TypedCrossfilter;
+crossfilter.ScalarDimension = ScalarDimension;
+crossfilter.EnumDimension = EnumDimension;
+
+export default crossfilter;

client/src/util/typedCrossfilter/positiveIntervals.js

@@ -0,0 +1,127 @@
+// jshint esversion: 6
+
+// Interval operations - very simple version of interval set relationship
+// operators.   An interval is a multi-interval list of [min, max),
+// where min and max are mandatory.   Constraints:
+//    * min <= max, min >= 0
+//    * empty interval groups are OK, ie, []
+//    * Legal intervals:   [], [ [0, 1], ... ]
+//    * Not legal:   [ [] ]
+//
+// All intervals are represented by simple JS arrays/numbers.
+//
+// Code assumes intervals have a low cardinality; many operations are done
+// with a brute force scan.   Little attempt to reduce GC pressure.
+//
+class PositiveIntervals {
+  // Canonicalize - ensure that:
+  //  1. no overlapping intervals
+  //  2. sorted in order of interval min.
+  //
+  static canonicalize(A) {
+    if (A.length <= 1) return A;
+    const copy = A.slice();
+    copy.sort((a, b) => a[0] - b[0]);
+    const res = [];
+    res.push(copy[0]);
+    for (let i = 1, len = copy.length; i < len; i += 1) {
+      if (copy[i][0] > res[res.length - 1][1]) {
+        // non-overlapping, add to result
+        res.push(copy[i]);
+      } else if (copy[i][1] > res[res.length - 1][1]) {
+        // merge this into previous
+        res[res.length - 1][1] = copy[i][1]; // eslint-disable-line prefer-destructuring
+      }
+    }
+    return res;
+  }
+
+  // Return interval with values belonging to both A and B. Essentially
+  // a set union operation.
+  //
+  static union(A, B) {
+    return PositiveIntervals.canonicalize([...A, ...B]);
+  }
+
+  static _flatten(A, B) {
+    const points = []; /* point, A, start */
+    for (let a = 0; a < A.length; a += 1) {
+      points.push([A[a][0], true, true]);
+      points.push([A[a][1], true, false]);
+    }
+    for (let b = 0; b < B.length; b += 1) {
+      points.push([B[b][0], false, true]);
+      points.push([B[b][1], false, false]);
+    }
+    // Sort order: point, then start
+    points.sort((a, b) => (a[0] !== b[0] ? a[0] - b[0] : a[2] ? 1 : -1));
+    return points;
+  }
+
+  // A - B, ie, the interval with all values in A that are not in B.  Essentially
+  // a set difference operation.
+  //
+  static difference(A, B) {
+    // Corner cases
+    if (A.length === 0 || B.length === 0) {
+      return PositiveIntervals.canonicalize(A);
+    }
+
+    const cA = PositiveIntervals.canonicalize(A);
+    const cB = PositiveIntervals.canonicalize(B);
+
+    const points = PositiveIntervals._flatten(cA, cB);
+    const res = [];
+    let aDepth = 0;
+    let depth = 0;
+    let intervalStart;
+    for (let i = 0; i < points.length; i += 1) {
+      const p = points[i];
+      const delta = p[2] ? 1 : -1;
+      depth += delta;
+      if (p[1]) aDepth += delta;
+
+      if (i === points.length - 1 || p[0] !== points[i + 1][0]) {
+        if (aDepth === 1 && depth === 1) {
+          [intervalStart] = p;
+        } else if (intervalStart !== undefined) {
+          res.push([intervalStart, p[0]]);
+          intervalStart = undefined;
+        }
+      }
+    }
+    // guaranteed to be in canonical form
+    return res;
+  }
+
+  // Return interval with values belonging to A or B.  Essentially a set
+  // intersection.
+  //
+  static intersection(A, B) {
+    if (A.length === 0 || B.length === 0) {
+      return [];
+    }
+
+    const cA = PositiveIntervals.canonicalize(A);
+    const cB = PositiveIntervals.canonicalize(B);
+
+    const points = PositiveIntervals._flatten(cA, cB);
+    const res = [];
+    let depth = 0;
+    let intervalStart;
+    for (let i = 0; i < points.length; i += 1) {
+      const p = points[i];
+      depth += p[2] ? 1 : -1;
+      if (depth === 2) {
+        [intervalStart] = p;
+      } else if (intervalStart !== undefined) {
+        res.push([intervalStart, p[0]]);
+        intervalStart = undefined;
+      }
+    }
+    // guaranteed to be in canonical form
+    return res;
+  }
+}
+
+export default PositiveIntervals;

client/src/util/typedCrossfilter/sort.js

@@ -0,0 +1,100 @@
+const SmallArray = 32;
+
+function insertionsort(a, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    let j;
+    for (j = i; j > lo && a[j - 1] > x; j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
+function insertionsortIndirect(a, s, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    const t = s[x];
+    let j;
+    for (j = i; j > lo && s[a[j - 1]] > t; j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
+function quicksort(a, lo, hi) {
+  if (hi - lo < SmallArray) {
+    return insertionsort(a, lo, hi);
+  }
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (a[i] < p);
+      do {
+        j -= 1;
+      } while (a[j] > p);
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksort(a, lo, j);
+    quicksort(a, j + 1, hi);
+  }
+  return a;
+}
+
+function quicksortIndirect(a, s, lo, hi) {
+  if (hi - lo < SmallArray) {
+    return insertionsortIndirect(a, s, lo, hi);
+  }
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    const t = s[p];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (s[a[i]] < t);
+      do {
+        j -= 1;
+      } while (s[a[j]] > t);
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksortIndirect(a, s, lo, j);
+    quicksortIndirect(a, s, j + 1, hi);
+  }
+  return a;
+}
+
+// Convenience wrappers
+export function sort(arr, comparator = undefined) {
+  if (comparator !== undefined) {
+    // XXX for now
+    return arr.sort(arr, comparator);
+  }
+  return quicksort(arr, 0, arr.length - 1);
+}
+
+export function sortIndex(index, source) {
+  return quicksortIndirect(index, source, 0, index.length - 1);
+}

client/src/util/typedCrossfilter/util.js

@@ -0,0 +1,128 @@
+// jshint esversion: 6
+/* eslint no-bitwise: "off" */
+
+import { sortIndex } from "./sort";
+
+/*
+    Utility functions, private to this module.
+*/
+
+// fill an array or typedarray with a sequential range of numbers,
+// starting with `start`
+//
+export function fillRange(arr, start = 0) {
+  const larr = arr;
+  for (let i = 0, len = larr.length; i < len; i += 1) {
+    larr[i] = i + start;
+  }
+  return larr;
+}
+
+// slice out of one array into another, using an index array
+//
+export function sliceByIndex(src, index) {
+  if (index === undefined || index === null) {
+    return src;
+  }
+  const dst = new src.constructor(index.length);
+  for (let i = 0; i < index.length; i += 1) {
+    dst[i] = src[index[i]];
+  }
+  return dst;
+}
+
+export function makeSortIndex(src) {
+  const index = fillRange(new Uint32Array(src.length));
+  sortIndex(index, src);
+  return index;
+}
+
+// Search for `value` in the sorted array `arr`, in the range [first, last).
+// Return the first (left most) index where arr[index] >= value.
+//
+// In other words, return array index I where:
+//  arr[i] < value for all tarr[lo:I]
+//  arr[i] >= value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: lower_bound()
+//    Python: bisect.bisect_left()
+//
+// XXX: it is likely that there would be minimal performance hit from creating
+// a factory version of lowerBound that takes an accessor (rather than having
+// a special-cased version for lining the indirection).
+//
+export function lowerBound(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+// Inlined performance optimization - used to indirect through a sort map.
+//
+export function lowerBoundIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+// Search for `value in the sorted array `arr`, in the range [first, last).
+// Return the first value where arr[index] > value.
+//
+// In other words, return array index I, where:
+//  arr[i] <= value for all tarr[lo:I]
+//  arr[i] > value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: upper_bound()
+//    Python: bisect.bisect_right()
+//
+export function upperBound(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+// Inline performance optimization
+//
+export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}

configuration/babel/babel.dev.js

@@ -1,9 +0,0 @@
-module.exports = {
-  babelrc: false,
-  cacheDirectory: true,
-  presets: [["env", { loose: true, modules: false }], "stage-0", "react"],
-  plugins: [
-    "react-hot-loader/babel",
-    "babel-plugin-transform-decorators-legacy"
-  ]
-};

configuration/babel/babel.prod.js

@@ -1,10 +0,0 @@
-module.exports = {
-  babelrc: false,
-  presets: [["env", { loose: true, modules: false }], "stage-0", "react"],
-  plugins: [
-    "babel-plugin-transform-react-constant-elements",
-    "babel-plugin-transform-decorators-legacy"
-  ]
-    .map(require.resolve)
-    .concat([[require.resolve("babel-plugin-transform-runtime")]])
-};

configuration/babel/babel.test.js

@@ -1,9 +0,0 @@
-module.exports = {
-  babelrc: false,
-  presets: [
-    "babel-preset-es2015",
-    "babel-preset-stage-0",
-    "babel-preset-react"
-  ],
-  plugins: ["istanbul"]
-};

configuration/eslint/eslint.js

@@ -1,22 +0,0 @@
-module.exports = {
-  root: true,
-  parser: "babel-eslint",
-  extends: "airbnb",
-  env: { browser: true, commonjs: true, es6: true },
-  globals: { expect: true },
-  parserOptions: {
-    ecmaVersion: 6,
-    sourceType: "module",
-    ecmaFeatures: {
-      jsx: true,
-      generators: true
-    }
-  },
-  rules: {
-    "no-magic-numbers": "off",
-    "func-style": "off",
-    "arrow-parens": "off",
-    "no-use-before-define": "off",
-    "react/jsx-filename-extension": "off"
-  }
-};

configuration/polyfills/polyfills.js

@@ -1,6 +0,0 @@
-if (typeof Promise === "undefined") {
-  require("promise/lib/rejection-tracking").enable();
-  window.Promise = require("promise/lib/es6-extensions.js");
-}
-
-require("whatwg-fetch");

docs/REST_API.md

@@ -0,0 +1,725 @@
+# cellxgene REST API 0.2 specification
+
+Items marked as (_future_) are intended for future implementation, and are included in the design to round out the concept, and highlight what we would do when/if we needed more functionality. The (_future_) items are not currently used by the cellxgene web application, and may be omitted from any backend - see [Current Front-End Dependencies](#current-front-end-dependencies) for more details.
+
+_Caveat emptor, partial spec_: this is a sketch for a spec, not a full spec, and some shortcuts have been taken in the authorship. Best practices for a REST API are assumed but not documented here, such as API versioning, reasonable choices for HTTP response codes, etc. In addition, for clarity the JSON examples will not always have all required quoting (eg, on keys) - the actual implementation should use legal JSON/CSV.
+
+_Note to readers_: please submit bugs, errata, and requests for clarifications as github issues on this repo.
+
+## Terminology
+
+Terminology mapping between the proposed model and other others:
+
+- Data or dataframe - expression matrix
+- Observation (**obs**) - cell, a column in Seurat
+- Variable (**var**) - gene, a row in Seurat
+- Annotation - metadata, which may be attached to either observations or variables, and will have a name
+- Layout - N dimensional float coordinates (x,y,...) assigned to an obs or var
+- Cluster - integer id assigned to a obs or var
+
+Reverse map:
+
+- Cell - observation or obs
+- Gene - variable or var
+- Metadata - annotation
+- Expression - data
+- Expression matrix - dataframe
+
+## Underlying Data Model
+
+The underlying data model is typical dataframe plus observation/variable annotations (aka metadata), and is more or less the same as the ScanPy [AnnData](http://anndata.readthedocs.io/en/latest/index.html) or Bioconductor dataframe model:
+
+- Dataframe: expression matrix [nObs, nVar]
+  - Observations are cells
+  - Variables are genes (or equivalent variable, eg, isoforms)
+  - Observations and variables are identified by an matrix coordinate pair (index, base 0)
+  - Dataframe values must be defined (dense matrix)
+  - The underlying type of the dataframe is IEEE single precision float (IEEE 754 "decimal32" data type, float32 in many language environments)
+- Annotations (metadata):
+  - Observations/cells or variables/genes can be annotated
+  - Annotations are typed and described in a schema:
+    - Obs or var annotation
+    - Annotation name
+    - Annotation type (all encoding is JSON format):
+      - Float32 - scalar value, IEEE single precision float (decimal32), encoded as JSON number
+      - Int32 - scalar value, 2's complement signed Int 32 bit encoded as JSON number
+      - String - UTF-8 string, encoded as JSON string
+      - Boolean - boolean value, encoded as JSON true/false
+      - Categorical: value is an item from an _unordered_ enumeration. The enumeration schema is encoded as a JSON array. Enumeration categories may be a JSON number, string or boolean (or any combination thereof). Schema is encoded as a JSON array of enumeration category names.
+  - Annotation values will conform to their schema, _or_ be undefined.
+  - Annotation names are a string.
+  - There is one guaranteed annotation name on every dimension: "name". This should be used as the default display name, must be a string and must be unique for the dimension (ie, all observation names are unique and all variable names are unique).
+
+Missing from this model are ad hoc metadata/annotations not associated with a observation or variable - we can add that in the future if it is required (what ScanPy calls unstructured annotation).
+
+### Assumptions about the Server
+
+The following assumptions are made about server behavior:
+
+- The server will have a data schema for annotations
+- The server will have default algorithms for layout, clustering and differential expression (may add additional endpoints in the future)
+- The server will serve a default data set, including annotations, cluster assignment and 2D layout (put another way, the client can request this data without any specification about how it is generated).
+
+## Filters
+
+There are several common filters supported by various routes. Route descriptions supporting these filters will reference the filter types.
+
+### Annotation _Value_ Filters
+
+Filter by the value of specific observation or variable annotation values. For example, the `/data` routes support sub-selection by the value of the annotations, which might be used to filter by some specific metadata (eg, `tissue_type: lung`).
+
+Annotation value filters can be specified as a URL query parameter (using GET), or a JSON object in a request body (using PUT or POST)<sup>[1](#endnote-1)</sup>.
+
+For a GET URL query parameter:
+
+- Annotation name is encoded as `obs:name` or `var:name`<sup>[2](#endnote-2)</sup>.
+- Enumerated values (string, categorical, boolean) are encoded as option lists, ie, `var:tissue=lung, obs:tumor=true`
+- Scalar values (int32, float32) are encoded as ranges, ie, `obs:num_reads=1000,10000` where either min or max may be replaced with an asterisk to indicate a half-open range.
+- Index filters are not allowed within GET URL query parameter filters
+- Logically, filters are ANDed, except for repeated annotation names which are ORed. For example, `?X=A&X=B&Y=1` is evaluated as `((X==A or X==B) and Y==1)`
+
+Example selection for _lung_ and _heart_ tissue with more than 1000 reads:
+
+```
+    GET /data?obs:tissue=lung&obs:tissue=heart&obs:num_reads=1000,*
+```
+
+For a PUT / POST request body, the filter encoding is a simple JSON object:
+
+- Annotation names are embedded within either a obs or var object
+- Enumerated values (eg, string, boolean categorical) are encoded as an array of options
+- Scalar values are encoded as an object with a min and max field
+
+Example selection for _lung_ and _heart_ tissue with more than 1000 reads:
+
+```
+    PUT or POST
+    --
+    {
+      "filter": {
+        "obs": {
+          "annotation_value": [
+            { "name": "tissue", "values": ["lung", "heart"] },
+            { "name": "num_reads", "min": 1000 },
+            { "name": "tumor", "values": true }
+          ]
+        }
+      }
+    }
+```
+
+### Obs or Var _Index_ Filters
+
+Several PUT / POST routes allow filtering by observation or variable index, eg, access to a subset of the dataframe. Obs or var indexes may be specified individually or as a range.
+
+For example, the following filter requests observations 1, 99, 1000-2000 from the dataframe (all variables are implicitly requested because no var filter is provided):
+
+```
+PUT or POST
+--
+{
+  filter: {
+    obs: {
+      index: [
+        1, 99, [1000, 2000]
+      ]
+    }
+  }
+}
+```
+
+### Complex Filters
+
+Several routes support multiple filter types, which may be combined into a single PUT / POST body. Complex filters are additive.
+
+Example:
+
+```
+PUT or POST
+--
+{
+  "filter": {
+    "obs": {
+      "annotation_value": [
+        { "name": "sex", "values": [ "F" ] }
+      ],
+      "index": [
+        [0, 10000]
+      ]
+    },
+    "var": {
+      // ...
+    }
+  }
+}
+```
+
+## Routes
+
+Summary (details follow):
+
+- `/config` - return application configuration
+- `/schema` - data schema for the dataframe and dataframe annotations
+- `/annotations/obs`, `/annotations/var` - fetch observation or variable annotation values, with ability to filter by annotation name (ie, fetch a subset of annotation data)
+- `/data/obs`, `/data/var` - fetch dataframe values, with ability to subselect by annotation value and/or obs/var indices.
+- `/cluster/obs`, `/cluster/var` - fetch cluster assignments, with ability to request re-clustering of a data subset
+- `/layout/obs`, `/layout/var` - fetch layout information (N dimensional coordinates), with ability to request re-layout of a data subset
+- `/diffexp` - fetch differential expression stats for a subset of data.
+- `/saveLocal` - request that the server persist a copy of the caller-specified data to local storage, with caller-specified name.
+
+All routes will be implemented with a prefix: <code>/api/v<strong><em>major.minor</em></strong></code>, where <em>major.minor</em> is a semver version number. Currently <em>version </em> is <em>0.2</em>, so, for example, the config route will be <code>/api/v0.2/config</code>. Prefixes are omitted in this document for clarity and brevity (eg, <code>/config</code> should be read as <code>/api/v0.2/config</code>).
+
+The following routes are mandatory:
+
+- `/config` - GET
+- `/schema` - GET
+- `/annotations/obs`, `/annotations/var` - GET and PUT
+- `/data/obs`, `/data/var` - GET and PUT
+
+All other routes are optional. Routes that are optional will advertise their presence or non-presence in the `/config` response.
+
+### Common Error Codes
+
+Routes that request potentially long-running (non-interactive speed) computation should provide threshold hints in /config response, and must return a 403 error immediately when a request exceeds, in the server's view, a reasonable interactive compute time.
+
+Erroneously specified filters (eg, non-existent annotation name or other malformed filter) should return a 400 HTTP error.
+
+### GET /config
+
+Configuration information to assist in front-end adaptation to underlying engine, available functionality, interactive time limits, etc.
+
+Information will include:
+
+- API endpoint availability (eg, differential expression not available) and related limitations, including:
+  - Feature enabled/disabled:
+    - POST /cluster (re-clustering) supported: true, false
+    - PUT /layout (re-layout) supported: true, false
+    - POST /diffexp (differential expression calculation) available: true, false
+  - Interactive compute speed hints; _approximate_ limitation on request size before it becomes non-interactive:
+    - Re-clustering: if supported, this is an integer number of obs/vars at which the algorithm is non-interactive. Omitted if POST /cluster not supported
+    - Re-layout: integer number of obs/vars at which the re-layout is non-interactive. Omitted if PUT /layout not supported
+    - Diff Exp: integer number of obs/vars at which the differential expression compute is non-interactive. Omitted if POST /diffexp not supported
+- Human readable information:
+  - Data set name / title - for display purposes.
+  - Engine name and version - for display purposes.
+- System-wide configuration parameters, eg, configuration the end user has specified via a CLI or server configuration system.
+
+**Response body:**
+
+- `features` - encoded as array of features. All optional features must be included. Each feature will be identified by an HTTP method and path **prefix**, and will have availability status. If not specified as unavailable, the feature must be implemented.
+  - Limitations are optional, and specified as an additional parameter on a feature, encoded as a number value with key `interactiveLimit`. By convention, this is the maximum number of inputs at which point the request is likely to return an error indicating non-interactive compute request. For example, this might be the maximum number of observations specified in the request to `PUT /layout/obs`.
+  - mandatory routes do not need to be specified
+- `displayNames` - names will include `engine` and `dataset` display names, as a JSON string. These should be relatively short strings (eg, suitable for window titles or equivalent)
+- `parameters` - system configuration parameters, specified as a key/value pair in a simple object. See [Currently Defined Parameters](#currently-defined-parameters) for additional information.
+
+```
+GET /config
+--
+200
+{
+  "config": {
+    "features": [
+      // all /cluster/* paths not implemented
+      { "method": "POST", "path": "/cluster/", "available": false },
+      {
+        "method": "PUT",
+        "path": "/layout/obs",
+        "available": true,
+        "interactiveLimit": 10000
+      },
+      { "method": "PUT", "path": "/layout/var", "available": false }
+    ],
+    "displayNames": {
+      "engine": "ScanPy version 1.33",
+      "dataset": "/home/joe/mouse/blorth.csv"
+    },
+    "parameters": {
+      "max-category-items": 1000,
+      "verbose": false
+      // name: value
+    }
+  }
+}
+```
+
+### GET /schema
+
+Schema definition for dataframe and annotations.
+
+**URL Parameters:** none
+
+**Response body:** encoded as a JSON object:
+
+- Dataframe:
+  - Dimensionality and type of dataframe, including:
+    - nObs - number of observations
+    - nVar - number of variables
+    - type - dataframe element type (currently required to be float32)
+- Dataframe annotations, both obs & var
+  - name - encoded as a string
+  - type - one of `float32`, `string`, `int32`, `boolean` or `categorical`
+  - categories - meaningful only for `categorical` type. Specifies the complete list of category names.
+
+Example:
+
+```
+GET /schema
+--
+200
+{
+  "schema": {
+    "dataframe": {
+      "nObs": 383,
+      "nVar": 19944,
+      "type": "float32"
+    },
+    "annotations": {
+      "obs": [
+        { "name": "name", "type": "string" },
+        { "name": "tissue_type", "type": "string" },
+        { "name": "num_reads", "type": "int32" },
+        { "name": "sample_name", "type": "string" },
+        {
+          "name": "clusters",
+          "type": "categorical",
+          "categories"=[ 99, 1, "unknown cluster" ]
+        },
+        { "name": "QScore", "type": "float32" }
+      ],
+      "var": [
+        { "name": "name", "type": "string" },
+        { "name": "gene", "type": "string" }
+      ]
+    }
+  }
+}
+```
+
+### GET /annotations/obs, GET /annotations/var
+
+Fetch annotations (metadata) for all observations or all variables. A subset of annotations may be selected by providing a query parameter `annotation-name` with a URL query list of annotation names.
+
+Observations and variables are guaranteed to have a `name` annotation, which should contain a human-readable name for this element (eg. a gene or cell name). No other annotation names are required or may be assumed. The `name` annotation value must be unique for the dimension, ie, a given observation name is unique among all observation names and a given variable name is unique among all variables.
+
+**URL Query Parameters:**
+
+- annotations-name - URL query list of 1 or more annotation names. Only specified annotations are returned. If this query parameter is not specified, all annotations are returned.
+
+**Response code:**
+
+- 200 Success
+- 400 Bad Request - one or more of the names specified with `annotation-name` are not associated with an annotation.
+
+**Response body:** annotation description and values. Values conform to the schema returned by `/schema` and each record begins with the observation or variable index. Where the specific value is not defined, `null`will be returned. Values will be sorted by index.
+
+Example:
+
+```
+GET /annotations/obs?annotation-name=tissue_type&annotation-name=sex&annotation-name=num_reads&annotation-name=clusters
+--
+200
+{
+  "names": [
+    "tissue_type", "sex", "num_reads", "clusters"
+  ],
+  "data": [
+    [ 0, "lung", "F", 39844, 99 ],
+    [ 1, "heart", "M", 83, 1 ],
+    [ 49, "spleen", null, 2, "unknown cluster" ],
+    // [ obsOrVarIndex, value, value, value, value ],
+    // ...
+  ]
+}
+```
+
+### PUT /annotations/obs, (_future_) PUT /annotations/var
+
+Same as the `GET /annotations` routes, with additional ability to filter by observation or variable index and annotation value (complex filter)
+
+**Fragment:** same as `GET /annotations/{obs|var}`
+
+**URL Query Parameters:**
+
+- annotations-name - URL query list of one or more annotation names. Only specified annotations are returned. If this query parameter is not specified, all annotations are returned.
+
+**Request body:** Complex filters, including
+
+- Annotation value filter
+- Observation or (_future_) variable index filter
+
+**Response code:**
+
+- 200 Success
+- 400 Bad Request - malformed filter or one or more of the annotation-name identifiers were not associated with an annotation name.
+
+**Response body:**
+
+- Same as `GET /annotations/{obs|var}`
+
+### GET /data/obs, (future) GET /data/var
+
+Get data from the dataframe (ie expression values), optionally sub-selected by annotation value filter. Supports JSON (default) and CSV responses, with annotation value filtering to enable basic deep linking. Response data must be organized according to path-specified primary dimension:
+
+- observation-major (for `/data/obs`)
+- variable-major (for `/data/var`)
+
+**URL Query Parameters:**
+
+- Annotation value filter
+- Accept-type: specifies an acceptable media type for simple content type negotiation. Semantics are similar to HTTP Accept header, but included in query params to make it easy to cut & paste URL. If parameter is present, MUST take precedence over the value(s) specified in the Accept request header. If the service does not support the requested format, it must reply with a 406 - Not Acceptable error response. This parameter differs from an Accept header in that it only accepts one media type value and not a priority list (it is named accept-type and not accept because the behavior differs slightly). Acceptable values are the MIME types (same semantics as Accept-Type HTTP header):
+  - application/json
+  - text/csv
+
+**Response code:**
+
+- 200 - Success
+- 406 - Not Acceptable (accept-type unacceptable MIME type)
+- 400 - Bad Request - malformed filter
+
+**Response body:** contents of the filtered dataframe, in primary-axis-major order, encoded as a dense matrix (eg, `/data/obs` will return data by observation). Data is preceded by a list of secondary axis indices, using the same index encoding as Index Filters (ie, an array of indices and/or index ranges). In JSON, this will be encoded as an array of indices with key obs or var; in CSV, this will be the header of the CSV response body. In both encodings, data will be sorted by index (both primary and secondary axis).
+
+Example JSON response to a request which has filtered by annotation value:
+
+```
+GET /data/obs?accept-type=application/json&obs:tissue=lung
+--
+200
+{
+  "var": [ [ 0, 20000 ] ],
+  "obs": [
+    [ 1, 39483, 3902, 203, 0, 0, ..., 28 ],   // length == 20001
+    // [ obsIndex, value1, value2, ..., value20000 ]
+    // ...
+  ]
+}
+```
+
+### PUT /data/obs, (future) PUT /data/var
+
+Same as `GET /data/{obs|var}`, but also accepts sub-selection by obs/var index filter. Accept may be specified as an HTTP header, and will support JSON and CSV. JSON will be the default served if no Accept header is supplied.
+
+**URL Query Parameters:** none
+
+**Request body:** Complex filters, including
+
+- Annotation value filter
+- Observation or (_future_) variable index filter
+
+**Response code:**
+
+- 200 Success
+- 400 Bad Request - malformed filter
+- 406 - Not Acceptable (accept-type missing or unacceptable MIME type)
+
+**Response body:** same as `GET /data/{obs|var}`
+
+### (future) POST /cluster/obs, (future) POST /cluster/var
+
+Generate cluster assignments for the caller-specified subset of data, as indicated by the filter. This operation implicitly requests a re-clustering operation to be performed on the specified data.
+
+If re-clustering is not supported by the server, must return an HTTP 501 response. If, in the view of the server, the request will exceed a reasonable interactive time period, must immediately return HTTP 403 error (error return _before_ attempting computation).
+
+**URL Query Parameters:** none
+
+**Request body:** complex filter:
+
+- Annotation value filter
+- Obs/var index filter
+
+**Response code:**
+
+- 200 - Success
+- 400 Bad Request - malformed filter
+- 403 Forbidden - non-interactive request
+- 501 - Re-clustering is not implemented
+
+**Response body:**
+
+- For 200 Success, returns numeric cluster ID for each observation or variable, organized as an array of `[index, clusterId]`, an sorted by index:
+
+  ```
+  {
+    "cluster": [
+      [ 0, 24 ],
+      // [ obsOrVarIndex, clusterID ],
+      // ...
+    ]
+  }
+  ```
+
+* For 403 Forbidden, must return an explanation of refusal in the response body to allow differentiation from an authorization failure (see 403 response definition in [RFC 7231](https://tools.ietf.org/html/rfc7231#section-6.5.3))
+
+### GET /layout/obs, (_future_) GET /layout/var
+
+Get the _default_ layout for all observations or (_future_) all variables. Return dimensionality of layout and a coordinate list. The client must be prepared to accept dimensionality different than it natively displays, and should perform a best-effort attempt to display the higher dimensionality layout (eg, create a 2D view of a 3D layout).
+
+**Response body:** dimensionality and coordinate list for each observation or variable, where coordinates are encoded as floats [0,1], sorted by index. Example 2D response:
+
+```
+{
+  "layout": {
+    "ndims": 2,
+    "coordinates": [
+      [ 0, 0.284483, 0.983744 ],
+      [ 1, 0.038844, 0.739444 ],
+      // [ obsOrVarIndex, X_coord, Y_coord ],
+      // ...
+    ]
+  }
+}
+```
+
+**Response code:**
+
+- 200 - success
+- 500 Internal Server Error - unprepared data, layout has not been stored in the input data
+
+### (_future_) PUT /layout/obs, (_future_) PUT /layout/var
+
+Generate layout for the caller-specified subset of data, as indicated by the filter. This operation implicitly requests a re-layout operation to be performed on the specified data. This operation will _commonly_ return the same results for any given caller-specified filter, but this behavior is not guaranteed.
+
+If re-layout is not supported by the server, must return an HTTP 501 response. If, in the view of the server, the request will exceed a reasonable interactive time period, must immediately return HTTP 403 error (error return _before_ attempting computation).
+
+**URL Query Parameters:** none
+
+**Request Body:** complex filter
+
+- Annotation value filter
+- Observation or variable index filter
+
+**Response code:**
+
+- 200 - success
+- 400 Bad Request - malformed filter
+- 403 Forbidden - non-interactive request
+- 501 - re-layout is not implemented
+
+**Response body:**
+
+- For 200 Success, same as `GET /layout/{obs|var}`
+- For 403 Forbidden, must return an explanation of refusal in the response body to allow differentiation from an authorization failure (see 403 response definition in [RFC 7231](https://tools.ietf.org/html/rfc7231#section-6.5.3))
+
+### POST /diffexp/obs
+
+Generate differential expression (DE) statistics for two specified subsets of data, as indicated by the two provided observation complex filters (each filter provided either as annotation and/or obs/var index). **If only one subset of data is specified, the default assumption is that the second subset is all other observations (cells).** This operation implicitly requests a differential expression operation to be performed on the specified data.
+
+Two modes are provided:
+
+- `topN`: return top N differentially expressed variables (across all variables)
+- `varFilter`: return DE for caller-provided variable filter (_future_)
+
+Both modes perform calculations using a subset of observations, where each subset is defined by an observation filter (`set1` and `set2`). These filters must not include a variable filter.
+
+If differential expression is not supported by the server, must return an HTTP 501 response. If, in the view of the server, the request will exceed a reasonable interactive time period, must immediately return HTTP 403 error (error return _before_ attempting computation).
+
+**Request body:**
+
+- Mode: `topN` or `varFilter` (specified with complex variable filter). `varFilter` must not include an "obs" section for the filter. Can be specified as one of:
+
+  ```
+  { "mode": "topN", "count": N }
+  ```
+
+  or
+
+  ```
+      {
+        "mode": "varFilter",
+        "varFilter": {
+          "filter": {
+            "var": {
+              "annotation_value": [
+                { "name": "gene", "values": [ "FOXP2", "ACTN1" ] }
+              ]
+            }
+          }
+        }
+      }
+  ```
+
+- count: required for `topN` mode, and specifies the number of results to return.
+- varFilter: required for `varFilter` mode, and specifies a variable filter.
+- set1 - Complex filter - defines observation "set1".
+
+  ```
+  {
+     "set1" {
+         "filter": {
+             "obs": {
+                "annotation_value": [
+                     { "name": "tissue", "values": [lung, heart] },
+                     { "name": "num_reads", "min": 1000 },
+                     { "name": "tumor", "values": true }
+                ]
+             }
+         }
+     }
+  }
+  ```
+
+* set2 - Complex filter - defines observation "set2" (set 2 is optional, and defaults to all **other** observations, ie, those disjoint from set1)
+
+**Response code:**
+
+- 200 - success
+- 400 Bad Request - malformed filter
+- 403 Forbidden - non-interactive request
+- 501 - differential expression is not implemented
+
+**Response body:**
+
+- For 200 Success, differential expression statistics returned as array of arrays, where each contains the following values:
+
+  - **varIndex**: variable index for the computed results
+  - **logfoldchange**: log fold-change of the average expression between the two groups. Positive values indicate that the gene is more highly expressed in the first group,
+  - **pVal**: unadjusted p-value,
+  - **pValAdj**: adjusted p-value
+
+  Values ordered as:
+
+  _varIndex_, _logfoldchange_, _pVal_, _pValAdj_
+
+  For example:
+
+  ```
+  [
+    [ 1720, 2.4679039, 2.3124478092035228e-175, 4.250279073316075e-172 ]
+    // ...
+  ]
+  ```
+
+- For 403 Forbidden, must return an explanation of refusal in the response body to allow differentiation from an authorization failure (see 403 response definition in [RFC 7231](https://tools.ietf.org/html/rfc7231#section-6.5.3))
+
+Example:
+
+```
+POST /diffexp/obs
+{
+  "diffexp": {
+    "set1": {
+      "filter": {
+        "obs": { "index": [ [0, 10000] ] }
+      }
+    },
+    "set2": {
+      "filter": {
+        "obs": { "index": [28448, 4, [88888, 99999] ] }
+      }
+    },
+    "mode": "topN",
+    "count": 5
+  }
+}
+---
+200 - Success
+{
+  "diffexp": [
+    [ 328, -2.569489, 2.655706e-63, 3.642036e-57 ],
+    // [ varIdx, logfoldchange, pVal, pValAdj ],
+    // ...
+  ]
+}
+```
+
+### (future) POST /data/saveSelection
+
+Instruct the server to save or otherwise process the selected dataframe subset, as described the the filter. "Saved" data may optionally be given a name. If no name is provided, the server should pick a reasonable default. Action is synchronous - ie, a 200 return indicates completion of the request. Response will contain a human readable text message to inform the user the results of their action.
+
+The intent of this route is to provide the user with a means of "saving" or otherwise processing filtered/selected data, for other uses, in a server-local resource. The precise semantics and mechanism are server-dependent, and the client should make no assumptions about server-side UI, data formats or persistence mechanism. Additionally, the client should make no assumptions about the lifespan or long-term persistence of the "saved" data.
+
+For example, a server may provide:
+
+- Store data subset in a CSV for sharing with other toolchains
+- Persist a selection view within an HDF5 dataset region reference
+- Pipe the selected data to another toolchain for immediate processing (ie, Unix pipe-like behavior)
+
+(these are just examples - the server may implement any data export or integration UI, completely independent of client UI or client knowledge).
+
+**URL Query Parameters:** name parameter, which for portability should only contain characters legal in a Posix file name (`A–Z a–z 0–9 . _ -`).
+
+**Request body:** Complex filters, including
+
+- Annotation value filter
+- Observation or (_future_) variable index filter
+
+**Response code:**
+
+- 200 Success
+- 400 Bad Request - malformed filter
+- 500 - unable to perform requested action
+
+**Response body:**
+
+For 200 Success and 400 Bad Request:
+
+```
+{
+  message: "Your results have been saved to /tmp/foo.csv"
+}
+```
+
+Example:
+
+```
+POST /data/saveSelection?name=myFavCluster
+{
+  "filter": {
+    "obs": {
+      "annotation_value": [
+        { "name": "cluster", "values": [ "99" ] }
+      ],
+    }
+  }
+}
+--
+200 Success
+{
+  "message": "OK"
+}
+```
+
+## Current Front-End Dependencies
+
+The cellxgene web front-end currently uses the following routes & features.
+
+Routes:
+
+- `GET /config`
+- `GET /schema`
+- `GET /annotations/obs`
+- `GET /annotations/var`
+- `GET /layout/obs` - get the default layout
+- `PUT /data/obs` - request will contain a filter by var `name`
+- `POST /diffexp/obs` - mode `topN`, typically with a `count` of 10, and two sets defined by an obs index filter (`{ filter: { obs: { index: [...] } } }`)
+
+Requests include the following content negotiation headers:
+
+- `Accept: application/json` - CSV not currently used in any routes
+- `Accept-Encoding: gzip, deflate, br`
+
+Currently unused routes:
+
+- `/data/var`
+- `/cluster/*`
+- `/layout/var`
+- `/data/saveSelection`
+
+## Currently Defined Parameters
+
+The system currently defines the following parameters:
+
+- `max-category-items` - if any categorical annotation has a unique value count exceeding
+  this parameter, it will not be displayed to the user. Number.
+- `layout` - algorithm to use for graph layout. String.
+- `diffexp` - algorithm to used to calculate differential expression. String.
+- `title` - data set name. String.
+- `debug` - run in debug mode. Boolean.
+- `verbose` - more verbose logging. Boolean.
+
+<!-- Endnotes themselves at the bottom. -->
+
+## Notes
+
+<a name="endnote-1">[1]</a>: Filters can often be large and complex, and exceed the URL length limitations - the POST method provides a means for the client to accommodate these cases. The GET method is provided for those cases where this is not concern, and/or where the client requires deep linking (subject to this URL length limit).
+
+<a name="endnote-2">[2]</a>: Note to implementers -- this scheme requires careful use of URL encoding to ensure that both annotation names and annotation values are properly escaped. It requires that both normal URL encoding is applied, as well as ensuring that the annotation name has any embedded colon characters escaped.

docs/Release_Validation_Recipe.md

@@ -0,0 +1,75 @@
+# Pre-release Validation Plan
+_This document contains a general purpose test plan for validating cellxgene prior to release_.
+
+All steps are expected to pass with no errors or malfunctions.   Tester should check:
+* CLI for errors (eg, build error, engine error)
+* Browser console for errors
+* Expected CLI and/or browser UI function
+
+## [Backend Test](#backend-test)
+
+### Getting started build & install validation:
+Goal: validate build, install and example demo correctness, per Getting Started instructions.
+
+1. Clean clone of the cellxgene repo into a local directory
+2. Follow build & install instructions from the Getting Started guide
+    * Confirm no build or install errors
+    * Confirm `cellxgene --help` functions correctly
+3. Follow the example data set demo startup from Getting Started guide and confirm front-end data loads correctly
+4. Confirm all package (and other) version dependencies are correct and match 
+
+### Tabula Muris 
+Goal: basic functional validation of b/e functions using Tabula Muris data and the scanpy engine.
+
+1. Precondition: cellxgene built & installed.
+2. Load the Tabula Muris data set: `cellxgene --title 'T. Muris' scanpy directory-name/`
+3. Verify all metadata selectors display and have correct type/options:
+    * *TODO:* _need list of metadata and their type_
+    * ...
+4. Verify default graph display has expected layout.   _need screen shot of expected layout_
+5. Verify selection controls work as expected:
+    * continuous metadata field
+    * categorical metadata field
+    * graph/lasso select
+6. Verify color by metadata type
+7. Select two cell sets and confirm differential expression compute succeeds
+6. Verify expression scatter plot is correct 
+
+
+## [Front-end Compatibility Test](#frontend-compatability-test)
+Goal: verify front-end UI compatibility with a given browser variant/version/platform.
+
+1. Start back-end on PBM3K data set
+2. Load UI
+3. Verify all major UI modes/functions:
+    * Graph display
+    * Metadata selector display
+    * Title display
+    * Selection - single and multiple fields - correctly display in cluster graph
+    * Regraph & reset function correctly
+    * Differential expression calc & scatter plot display
+    * All graphs maintain consistent selection state
+    * All selection widgets (eg, continuous metadata selector) maintain correct status (consistent with graph displays)
+4. Verify overall performance is reasonable/interactive
+5. Verify no errors on CLI or browser console
+
+
+## [End-to-end Functional Test](#e2e-functional-test)
+Goal: confirm end-to-end functional behavior is as expected.
+
+### ScanPy engine
+
+1. Basic data load and display
+    * ...
+2. Select & multi-select of metadata and coordinates
+    * ...
+3. Color by metadata
+    * ...
+4. Regraph / reset
+    * ...
+5. Differential expression: scatterplot, top-N genes, etc.
+    * ...
+6. Color by expression
+    * ...
+7. Arbitrary gene expression
+    * ...