release 0.10.0 (#794 )

fix incompatibility of flask reload and port searching (#793 )
* WIP * add --developer; fix incompatibility of --port and --debug * put REST tests on separate ports * PR review
2026-09-26 21:38:11 +08:00 · 2019-05-29 16:55:43 -07:00 · 2019-05-29 16:38:57 -07:00 · 2019-05-29 12:23:30 -04:00 · 2019-05-24 21:01:11 -07:00 · 2019-05-24 21:00:54 -07:00
126 changed files with 13588 additions and 7237 deletions
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.7.0
+current_version = 0.10.0

 [bumpversion:file:setup.py]
 search = version="{current_version}"
@@ -27,6 +27,9 @@ server/app/web/templates/index\.html
 .ipynb_checkpoints
 *.ipynb

+# cefpython
+error.log
+
 # misc
 .DS_Store
 npm-debug.log
@@ -1,29 +1,31 @@
 language: python
-python:
-  - "3.6"
+dist: xenial
+sudo: required
 node_js:
-  - "8"
+  - 8
 cache:
-  pip: true
+  - pip
+  - npm
 install:
  - set -eo pipefail
  - pip install flake8
-  - make build
-  - make install
+  - make pydist
+  - make install-dist
  - pip install -r server/requirements-dev.txt
-  - docker build .

 jobs:
  include:
-  - name: "Branch Tests"
-    script:
-      - set -eo pipefail
-      - flake8 server
-      - black --check
-      - npm run --prefix client/ build
-      - npm run --prefix client/ unit-test
-      - pytest -s server/test
+  - name: "Branch Tests 3.7"
+    python: "3.7"
+    script: ./travis-build.sh
+  - name: "Branch Tests 3.6"
+    python: "3.6"
+    script: ./travis-build.sh
+  - name: "Docker Build"
+    install: skip
+    python: "3.6"
+    script: docker build .
  - name: "Smoke Tests"
-    if: branch = master AND type = cron
+    python: "3.6"
    script:
      - npm run --prefix client/ smoke-test
@@ -2,224 +2,59 @@

 > an interactive explorer for single-cell transcriptomics data

-`cellxgene` is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
+_cellxgene_ (pronounced "sell-by-jean") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.

 <img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">

-## getting started
+- Want to install and use cellxgene? Visit the [cellxgene docs](https://chanzuckerberg.github.io/cellxgene/).
+- Want to see where we are going? Check out [our roadmap](ROADMAP.md).
+- Want to contribute? See our [contributors guide](#Contributing)

-You'll need **python 3.6** and **Google Chrome**. (_Warning_: Python 3.7 is **not** supported at this time)
-The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).
+## quick start

-To install run
+To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](https://chanzuckerberg.github.io/cellxgene/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)

-```
+Install the package.
+``` bash
 pip install cellxgene
 ```

-To start exploring a dataset call
+Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file

-```
-cellxgene launch dataset.h5ad --open
+``` bash
+curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
 ```

-If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
-
-```
+Launch _cellxgene_
+``` bash
 cellxgene launch pbmc3k.h5ad --open
 ```

-You should see your web browser open with the following
+To learn more about what you can do with _cellxgene_, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-stared/) guide.

-<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+## get in touch

-**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!

-There are several options available, such as:
+## where we are going

- `--layout` to specify the layout as `tsne` or `umap`
- `--title` to show a title on the explorer
- `--open` to automatically open the web browser after launching (OS X only)
+Our goal is to enable teams of computational and experimental
+biologists to collaboratively gain insight into their single-cell RNA-seq data.

-To see all options call
+There are 4 key features we plan to implement in the near term.

-```
-cellxgene launch --help
-```
+- Click install and launch
+- Manual annotation workflows
+- Toggle embeddings
+- Gene information

-There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+For more detail on these features and where we are going, see [our roadmap](ROADMAP.md).

-## data formatting
+## contributing

-### assumptions
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!

-The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
-
- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_tsne` for the `tsne` layout or `X_umap` for the `umap` layout)
- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
-
-### prepare
-
-The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
-
-To prepare from an existing `.h5ad` file use
-
-```
-cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
-```
-
-This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
-
-Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
-
-```
-cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
-```
-
-To see all options call
-
-```
-cellxgene prepare --help
-```
-
-**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
-
-```
-pip install cellxgene[louvain]
-```
-
-If the aforementioned optional package installation fails, you can also install these packages directly:
-
-```
-pip install python-igraph louvain>=0.6
-```
-
-## conda and virtual environments
-
-If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
-
-```
-conda create --yes -n cellxgene python=3.6
-conda activate cellxgene
-pip install cellxgene
-```
-
-Or you can create a virtual environment by using
-
-```
-ENV_NAME=cellxgene
-python3.6 -m venv ${ENV_NAME}
-source ${ENV_NAME}/bin/activate
-pip install cellxgene
-```
-
-## docker
-
-We have included a dockerfile to conveniently run cellxgene from docker.
-
-1. Build the image `docker build . -t cellxgene`
-2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
-   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
-
-## FAQ
-
-<details>
-
-<summary> questions about data formatting </summary>
-  
-<hr>
-  
-> Someone sent me a directory of `10X-Genomics` data with a `mtx` file and I've never used `scanpy`, can I use `cellxgene`?
-
-Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
-
-```
-cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
-```
-
-Depending on the size of the dataset, this may take some time. Once it's done, call
-
-```
-cellxgene launch data-processed.h5ad --layout=umap --open
-```
-
-And your web browser should open with an interactive view of your data.
-
-<hr>
-
-> In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
-
-Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using
-
-```
-pip install cellxgene[louvain]
-```
-
-<hr>
-
-> I ran `prepare` and I'm getting results that look unexpected
-
-You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
-
-```
-cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
-```
-
-It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
-
-<hr>
-
-> I have extra metadata that I want to add to my dataset
-
-Currently this is not supported directly, but you should be able to do this manually using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
-
-<hr>
-
-> What part of the anndata objects does cellxgene pull in for visualization?
-
- `.obs` and `.var` annotations are use to extract metadata for filtering
- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
- `.obsm` is used for layout
-
-</details>
-
-<details>
-  
-<summary> questions about installing and building </summary>
-
-<hr>
-
-> I tried to `pip install cellxgene` and got a weird error about missing paths to an HDF5 library?  
-
-You probably just need to install HDF5 first. If you're on a mac, you can simply `brew install hdf5` and then try `pip install cellxgene` again.  
-
-> I tried to `pip install cellxgene` and got another weird error I don't understand  
-
-This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.  
-
-<hr>
-
-> I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
-
-This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
-
-</details>
-
-<details>
-
-<summary> questions about algorithms </summary>
-
-<hr>
-
-> How are you computing and sorting differential expression results?
-
-Currently we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
-
-</details>
-
-## developer guide
+### developer guide

 This project has made a few key design choices

@@ -231,7 +66,7 @@ Depending on your background and interests, you might want to contribute to the

 If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine

- python 3.6
+- python 3.6+
 - node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)

 Then clone the project
@@ -240,10 +75,10 @@ Then clone the project
 git clone https://github.com/chanzuckerberg/cellxgene.git
 ```

-Build the client web assets by calling this from inside the `cellxgene` folder
+Build the client web assets by calling `make` from inside the `cellxgene` folder

 ```
-./bin/build-client
+make
 ```

 Install all requirements (we recommend doing this inside a virtual environment)
@@ -256,20 +91,7 @@ You can start the app while developing either by calling `cellxgene` or by calli

 If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.

-## development roadmap
-
-`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact)
-
- **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene` to render the results of these analyses when they have been performed
- **Deploy to web** Many projects release public data browser websites alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
- **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data
-
-## contributing
-
-We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
-
-## inspiration and collaboration
+## inspiration

 We've been heavily inspired by several other related single-cell visualization projects, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), and many others. We hope to explore collaborations where useful as this community works together on improving interactive visualization for single-cell data.

@@ -279,9 +101,13 @@ We have been working closely with the [`scanpy`](https://github.com/theislab/sca

 We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)

-## help and contact
+## core team

-Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+- Colin Megill, frontend & product design
+- Charlotte Weaver, software engineer
+- Bruce Martin, software engineer
+- Sidney Bell, computational biologist
+- Justin Kiggins, product manager

 ## reuse

@@ -0,0 +1,54 @@
+# cellxgene roadmap
+
+We are very exited for _cellxgene_ to become a valuable tool in collaborations
+between computational biologists and experimental biologists working on
+single-cell transcriptomics data. _cellxgene_ is in active development, and we
+would love to include the community as we plan new features to work on. If you
+have questions of feedback about this roadmap, please submit an issue on
+GitHub.
+
+Please note: this roadmap is subject to change.
+
+*Last updated: April 11, 2019*
+
+## what we are building now
+
+In the near term, our goal is to enable teams of computational and experimental
+biologists to collaboratively explore and annotate their single-cell RNA-seq data.
+
+There are 4 key features we plan to implement in the near term.
+
+- Click install and launch
+- Manual annotation workflows
+- Toggle embeddings
+- Gene information
+
+### simple install and launch
+
+The command line interface for installing and launching cellxgene is a barrier
+for users who are not used to Python or using the command line. We plan to
+support installation and launch of cellxgene on Mac and Windows. See
+[Issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for more details.
+
+### manual annotation workflows
+
+The exploratory visualization that cellxgene offers is critical for manual
+annotation workflows, especially in collaborative environments. We plan to
+support manually annotate cells with labels (i.e., cell type or QC flags) for
+downstream analysis. See [Issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524)
+for more details.
+
+### toggle embeddings
+
+While a single dataset may have multiple embeddings calculated (tSNE, umap, in
+situ coordinates, trajectories, etc), cellxgene currently requires the user to select the
+embedding to use in the main layout at launch. We plan to support letting users
+toggle between any embedding present in a file from the cellxgene interface.
+See [Issue #594](https://github.com/chanzuckerberg/cellxgene/issues/594) for details.
+
+### gene information
+
+Differential expression returns only the names of genes, but no additional information
+about gene metadata, function, or known associations. We plan to help users learn
+more about genes they discover by exposing additional gene metadata. See
+[Issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for details.
@@ -0,0 +1,6 @@
+export const jest_env = process.env.JEST_ENV || "dev";
+export const appPort = process.env.JEST_CXG_PORT || 3000;
+export const appUrlBase = `http://localhost:${appPort}`;
+export const DEV = jest_env === "dev";
+export const DEBUG = jest_env === "debug";
+export const DATASET = "pbmc3k";
@@ -0,0 +1,122 @@
+export const datasets = {
+  pbmc3k: {
+    title: "pbmc3k",
+    dataframe: {
+      nObs: "2638",
+      nVar: "1838",
+      type: "float32"
+    },
+    categorical: {
+      louvain: {
+        "B cells": "342",
+        "CD14+ Monocytes": "480",
+        "CD4 T cells": "1144",
+        "CD8 T cells": "316",
+        "Dendritic cells": "37",
+        "FCGR3A+ Monocytes": "150",
+        Megakaryocytes: "15",
+        "NK cells": "154"
+      }
+    },
+    continuous: {
+      n_genes: "int32",
+      percent_mito: "float32",
+      n_counts: "float32"
+    },
+    cellsets: {
+      lasso: [
+        {
+          "coordinates-as-percent": { x1: 0.05, y1: 0.25, x2: 0.15, y2: 0.35 },
+          count: "104"
+        }
+      ],
+      categorical: [
+        {
+          metadata: "louvain",
+          values: ["B cells", "Megakaryocytes"],
+          count: "357"
+        }
+      ],
+      continuous: [
+        {
+          metadata: "n_genes",
+          "coordinates-as-percent": { x1: 0.25, y1: 0.5, x2: 0.55, y2: 0.5 },
+          count: "1537"
+        }
+      ]
+    },
+
+    diffexp: {
+      cellset1: [
+        { kind: "categorical", metadata: "louvain", values: ["B cells"] }
+      ],
+      cellset2: [
+        {
+          kind: "categorical",
+          metadata: "louvain",
+          values: ["CD4 T cells", "NK cells"]
+        }
+      ],
+      "gene-results": [
+        "HLA-DRB1",
+        "HLA-DPB1",
+        "CD79A",
+        "HLA-DPA1",
+        "HLA-DQA1",
+        "CD79B",
+        "HLA-DQB1",
+        "MS4A1",
+        "IL32",
+        "CD37"
+      ]
+    },
+
+    genes: {
+      bulkadd: ["S100A8", "FCGR3A", "LGALS2", "GSTP1"],
+      search: "ACD"
+    },
+    subset: {
+      cellset1: [
+        {
+          kind: "categorical",
+          metadata: "louvain",
+          values: ["B cells", "Megakaryocytes"]
+        }
+      ],
+      count: "357",
+      categorical: {
+        louvain: {
+          "B cells": "342",
+          Megakaryocytes: "15"
+        }
+      },
+      lasso: {
+        "coordinates-as-percent": { x1: 0.45, y1: 0.05, x2: 0.5, y2: 0.1 },
+        count: "76"
+      }
+    },
+    scatter: {
+      genes: { x: "S100A8", y: "FCGR3A" }
+    },
+    pan: {
+      "coordinates-as-percent": { x1: 0.75, y1: 0.75, x2: 0.35, y2: 0.35 }
+    },
+    features: {
+      panzoom: {
+        lasso: {
+          "coordinates-as-percent": { x1: 0.3, y1: 0.3, x2: 0.5, y2: 0.5 },
+          count: "24"
+        }
+      }
+    },
+    clip: {
+      min: "30",
+      max: "70",
+      metadata: "n_genes",
+      gene: "S100A8",
+      "coordinates-as-percent": { x1: 0.25, y1: 0.5, x2: 0.55, y2: 0.5 },
+      count: "392",
+      "gene-cell-count": "421"
+    }
+  }
+};
@@ -1,185 +1,336 @@
+/*
+Smoke test suite that will be run in Travis CI
+
+Tests included in this file are expected to be relatively stable and test core features
+ */
 import puppeteer from "puppeteer";
+import { appUrlBase, DEBUG, DEV, DATASET } from "./config";
+import { puppeteerUtils, cellxgeneActions } from "./puppeteerUtils";
+import { datasets } from "./data";

-const jest_env = process.env.JEST_ENV || "dev";
-const appPort = process.env.JEST_CXG_PORT || 3000;
-const appUrlBase = `http://localhost:${appPort}`;
-const DEV = jest_env === "dev";
-
-let browser;
-let page;
+let browser, page, utils, cxgActions, spy;
 const browserViewport = { width: 1280, height: 960 };
+let data = datasets[DATASET];
+
+if (DEBUG) jest.setTimeout(100000);
+if (DEV) jest.setTimeout(10000);

 beforeAll(async () => {
-  const browser_params = DEV
+  const browserParams = DEV
+    ? { headless: false, slowMo: 5 }
+    : DEBUG
    ? { headless: false, slowMo: 100, devtools: true }
    : {};
-  browser = await puppeteer.launch(browser_params);
+  browser = await puppeteer.launch(browserParams);
  page = await browser.newPage();
-  page.setViewport(browserViewport);
-  if (DEV) page.on("console", msg => console.log("PAGE LOG:", msg.text()));
+  await page.setViewport(browserViewport);
+  if (DEV || DEBUG) {
+    page.on("console", async msg => {
+      // If there is a console.error but an error is not thrown, this will ensure the test fails
+      if (msg.type() === "error") {
+        const errorMsgText = await Promise.all(
+          // TODO can we do this without internal properties?
+          msg.args().map(arg => arg._remoteObject.description)
+        );
+        throw new Error(`Console error: ${errorMsgText}`);
+      }
+      console.log(`PAGE LOG: ${msg.text()}`);
+    });
+  }
+  page.on("pageerror", err => {
+    throw new Error(`Console error: ${err}`);
+  });
+  utils = puppeteerUtils(page);
+  cxgActions = cellxgeneActions(page);
+});
+
+beforeEach(async () => {
+  await page.goto(appUrlBase);
 });

 afterAll(() => {
-  if (!DEV) {
+  if (!DEBUG) {
    browser.close();
  }
 });

-const getOneElementInnerHTML = async function(selector) {
-  let text = await page.$eval(selector, el => el.innerHTML);
-  return text;
-};
-
-const drag = async function(el_box, start, end, lasso = false) {
-  const x1 = el_box.content[0].x + start.x;
-  const x2 = el_box.content[0].x + end.x;
-  const y1 = el_box.content[0].y + start.y;
-  const y2 = el_box.content[0].y + end.y;
-  await page.mouse.move(x1, y1);
-  await page.mouse.down();
-  if (lasso) {
-    await page.mouse.move(x2, y1);
-    await page.mouse.move(x2, y2);
-    await page.mouse.move(x1, y2);
-    await page.mouse.move(x1, y1);
-  } else {
-    await page.mouse.move(x2, y2);
-  }
-  await page.mouse.up();
-};
-
-describe("did launch", () => {
+describe("did launch", async () => {
  test("page launched", async () => {
-    await page.goto(appUrlBase);
-    let el = await getOneElementInnerHTML("[data-testid='header']");
-    expect(el).toBe("cellxgene: pbmc3k");
+    let el = await utils.getOneElementInnerHTML("[data-testid='header']");
+    expect(el).toBe(data.title);
  });
 });

-describe("search for genes", () => {
-  test("search for known gene and add to metadata", async () => {
-    await page.goto(appUrlBase);
-    await page.waitForSelector("[ data-testid='gene-search']");
-    // blueprint's  typeahead is treating typing weird, clicking & waiting first solves this
-    await page.click("[data-testid='gene-search']");
-    await page.waitFor(200);
-    await page.type("[data-testid='gene-search']", "ACD");
-    await page.keyboard.press("Enter");
-    await page.waitForSelector("[data-testid='histogram-ACD']");
+describe("metadata loads", async () => {
+  test("categories and values from dataset appear", async () => {
+    for (const label in data.categorical) {
+      await utils.waitByID(`category-${label}`);
+      const categoryName = await utils.getOneElementInnerText(
+        `[data-testid="category-${label}"]`
+      );
+      expect(categoryName).toMatch(label);
+      await utils.clickOn(`category-expand-${label}`);
+      const categories = await cxgActions.getAllCategoriesAndCounts(label);
+      expect(Object.keys(categories)).toMatchObject(
+        Object.keys(data.categorical[label])
+      );
+      expect(Object.values(categories)).toMatchObject(
+        Object.values(data.categorical[label])
+      );
+    }
+  });
+
+  test("continuous data appears", async () => {
+    for (const label in data.continuous) {
+      await utils.waitByID(`histogram-${label}`);
+    }
  });
 });

-describe("select cells and diffexp", () => {
-  test("selects cells from layout and adds to cell set 1", async () => {
-    await page.goto(appUrlBase);
-    const layout = await page.waitForSelector("[data-testid='layout']");
-    const size = await layout.boxModel();
-    const cellset1 = {
-      start: {
-        x: Math.floor(size.width * 0.25),
-        y: Math.floor(size.height * 0.25)
-      },
-      end: {
-        x: Math.floor(size.width * 0.35),
-        y: Math.floor(size.height * 0.35)
-      }
-    };
-    await drag(size, cellset1.start, cellset1.end, true);
-    await page.click("[data-testid='cellset-button-1");
-    let button = await getOneElementInnerHTML("[data-testid='cellset-button-1");
-    expect(button).toMatch(/26 cells/);
+describe("cell selection", async () => {
+  test("selects all cells cellset 1", async () => {
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.dataframe.nObs);
  });

-  test("selects cells from layout and adds to cell set 2", async () => {
-    await page.goto(appUrlBase);
-    const layout = await page.waitForSelector("[data-testid='layout']");
-    const size = await layout.boxModel();
-    const cellset2 = {
-      start: {
-        x: Math.floor(size.width * 0.45),
-        y: Math.floor(size.height * 0.45)
-      },
-      end: {
-        x: Math.floor(size.width * 0.55),
-        y: Math.floor(size.height * 0.55)
-      }
-    };
-    await drag(size, cellset2.start, cellset2.end, true);
-    await page.click("[data-testid='cellset-button-2");
-    let button = await getOneElementInnerHTML("[data-testid='cellset-button-2");
-    expect(button).toMatch(/49 cells/);
+  test("selects all cells cellset 2", async () => {
+    const cellCount = await cxgActions.cellSet(2);
+    expect(cellCount).toBe(data.dataframe.nObs);
  });

-  test("selects cells, saves them and performs diffexp", async () => {
-    await page.goto(appUrlBase);
-    const layout = await page.waitForSelector("[data-testid='layout']");
-    const size = await layout.boxModel();
-    const cellset1 = {
-      start: {
-        x: Math.floor(size.width * 0.25),
-        y: Math.floor(size.height * 0.25)
-      },
-      end: {
-        x: Math.floor(size.width * 0.35),
-        y: Math.floor(size.height * 0.35)
-      }
-    };
-    await drag(size, cellset1.start, cellset1.end, true);
-    await page.click("[data-testid='cellset-button-1");
-    const cellset2 = {
-      start: {
-        x: Math.floor(size.width * 0.45),
-        y: Math.floor(size.height * 0.45)
-      },
-      end: {
-        x: Math.floor(size.width * 0.55),
-        y: Math.floor(size.height * 0.55)
-      }
-    };
-    await drag(size, cellset2.start, cellset2.end, true);
-    await page.click("[data-testid='cellset-button-2");
-    await page.click("[data-testid='diffexp-button");
-    await page.waitForSelector("[data-testclass='histogram-diffexp']");
-    const diffexps = await page.$$eval(
-      "[data-testclass='histogram-diffexp']",
-      divs => {
-        return divs.map(div =>
-          div.id.substring("histogram-".length, div.id.length)
+  test("selects cells via lasso", async () => {
+    for (const cellset of data.cellsets.lasso) {
+      const cellset1 = await cxgActions.calcDragCoordinates(
+        "layout-graph",
+        cellset["coordinates-as-percent"]
+      );
+      await cxgActions.drag("layout-graph", cellset1.start, cellset1.end, true);
+      const cellCount = await cxgActions.cellSet(1);
+      expect(cellCount).toBe(cellset.count);
+    }
+  });
+
+  test("selects cells via categorical", async () => {
+    for (const cellset of data.cellsets.categorical) {
+      await utils.clickOn(`category-expand-${cellset.metadata}`);
+      await utils.clickOn(`category-select-${cellset.metadata}`);
+      for (const val of cellset.values) {
+        await utils.clickOn(
+          `categorical-value-select-${cellset.metadata}-${val}`
        );
      }
-    );
-    expect(diffexps).toMatchObject([
-      "HLA-DPA1",
-      "HLA-DQA1",
-      "HLA-DRB1",
-      "HLA-DMA",
-      "CST3",
-      "HLA-DPB1",
-      "HLA-DQB1",
-      "LGALS2",
-      "FCER1A",
-      "LTB"
-    ]);
+      const cellCount = await cxgActions.cellSet(1);
+      expect(cellCount).toBe(cellset.count);
+    }
+  });
+
+  test("selects cells via continuous", async () => {
+    for (const cellset of data.cellsets.continuous) {
+      const histId = `histogram-${cellset.metadata}-plot-brush`;
+      const coords = await cxgActions.calcDragCoordinates(
+        histId,
+        cellset["coordinates-as-percent"]
+      );
+      await cxgActions.drag(histId, coords.start, coords.end);
+      const cellCount = await cxgActions.cellSet(1);
+      expect(cellCount).toBe(cellset.count);
+    }
  });
 });

-describe("brushable histogram", () => {
-  test("can brush historgram", async () => {
-    await page.goto(appUrlBase);
-    const hist = await page.waitForSelector(
-      "[data-testid='histogram_n_genes_svg-brush'] > .overlay"
+describe("gene entry", async () => {
+  test("search for single gene", async () => {
+    // blueprint's  typeahead is treating typing weird, clicking & waiting first solves this
+    await utils.typeInto("gene-search", data.genes.search);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(
+      `[data-testid='histogram-${data.genes.search}']`
    );
-    const hist_size = await hist.boxModel();
-    const draghist = {
-      start: {
-        x: Math.floor(hist_size.width * 0.25),
-        y: Math.floor(hist_size.height * 0.5)
-      },
-      end: {
-        x: Math.floor(hist_size.width * 0.55),
-        y: Math.floor(hist_size.height * 0.5)
-      }
-    };
-    await drag(hist_size, draghist.start, draghist.end);
+  });
+
+  test("bulk add genes", async () => {
+    await cxgActions.reset();
+    const testGenes = data.genes.bulkadd;
+    await utils.clickOn("section-bulk-add");
+    await utils.typeInto("input-bulk-add", testGenes.join(","));
+    await page.keyboard.press("Enter");
+    const userGeneHist = await cxgActions.getAllHistograms(
+      "histogram-user-gene"
+    );
+    expect(userGeneHist).toEqual(expect.arrayContaining(testGenes));
+  });
+});
+
+describe("diffexp", async () => {
+  test("selects cells, saves them and performs diffexp", async () => {
+    for (const select of data.diffexp.cellset1) {
+      if (select.kind === "categorical") {
+        await cxgActions.selectCategory(select.metadata, select.values, true);
+      }
+    }
+    await cxgActions.cellSet(1);
+    for (const select of data.diffexp.cellset2) {
+      if (select.kind === "categorical") {
+        await cxgActions.selectCategory(select.metadata, select.values, true);
+      }
+    }
+    await cxgActions.cellSet(2);
+    await utils.clickOn("diffexp-button");
+    const diffExpHists = await cxgActions.getAllHistograms("histogram-diffexp");
+    expect(diffExpHists).toEqual(
+      expect.arrayContaining(data.diffexp["gene-results"])
+    );
+  });
+});
+
+describe("subset/reset", async () => {
+  test("subset - cell count matches", async () => {
+    for (const select of data.subset.cellset1) {
+      if (select.kind === "categorical") {
+        await cxgActions.selectCategory(select.metadata, select.values, true);
+      }
+    }
+    await utils.clickOn("subset-button");
+    for (const label in data.subset.categorical) {
+      const categories = await cxgActions.getAllCategoriesAndCounts(label);
+      expect(Object.keys(categories)).toMatchObject(
+        Object.keys(data.subset.categorical[label])
+      );
+      expect(Object.values(categories)).toMatchObject(
+        Object.values(data.subset.categorical[label])
+      );
+    }
+  });
+
+  test("reset after subset", async () => {
+    for (const select of data.subset.cellset1) {
+      if (select.kind === "categorical") {
+        await cxgActions.selectCategory(select.metadata, select.values, true);
+      }
+    }
+    await utils.clickOn("subset-button");
+    for (const label in data.subset.categorical) {
+      const categories = await cxgActions.getAllCategoriesAndCounts(label);
+      expect(Object.keys(categories)).toMatchObject(
+        Object.keys(data.subset.categorical[label])
+      );
+      expect(Object.values(categories)).toMatchObject(
+        Object.values(data.subset.categorical[label])
+      );
+    }
+    await cxgActions.reset();
+    for (const label in data.categorical) {
+      await utils.waitByID(`category-${label}`);
+      const categoryName = await utils.getOneElementInnerText(
+        `[data-testid="category-${label}"]`
+      );
+      expect(categoryName).toMatch(label);
+      const categories = await cxgActions.getAllCategoriesAndCounts(label);
+      expect(Object.keys(categories)).toMatchObject(
+        Object.keys(data.categorical[label])
+      );
+      expect(Object.values(categories)).toMatchObject(
+        Object.values(data.categorical[label])
+      );
+    }
+  });
+
+  test("lasso after subset", async () => {
+    for (const select of data.subset.cellset1) {
+      if (select.kind === "categorical") {
+        await cxgActions.selectCategory(select.metadata, select.values, true);
+      }
+    }
+    await utils.clickOn("subset-button");
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.subset.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.subset.lasso.count);
+  });
+});
+
+describe("scatter plot", async () => {
+  test("scatter plot appears", async () => {
+    await cxgActions.reset();
+    const testGenes = data.scatter.genes;
+    await utils.clickOn("section-bulk-add");
+    await utils.typeInto("input-bulk-add", Object.values(testGenes).join(","));
+    await page.keyboard.press("Enter");
+    await utils.clickOn(`plot-x-${data.scatter.genes.x}`);
+    await utils.clickOn(`plot-y-${data.scatter.genes.y}`);
+    await utils.waitByID("scatterplot");
+  });
+});
+
+describe("clipping", async () => {
+  test("clip continuous", async () => {
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.metadata}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip.count);
+  });
+
+  test("clip gene", async () => {
+    await utils.typeInto("gene-search", data.clip.gene);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(`[data-testid='histogram-${data.clip.gene}']`);
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.gene}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip["gene-cell-count"]);
+  });
+});
+
+// interact with UI elements just that they do not break
+describe("ui elements don't error", async () => {
+  test("color by", async () => {
+    for (const label in data.categorical) {
+      await utils.clickOn(`colorby-${label}`);
+    }
+    for (const label in data.continuous) {
+      await utils.clickOn(`colorby-${label}`);
+    }
+  });
+
+  test("color by for gene", async () => {
+    await utils.typeInto("gene-search", data.genes.search);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(
+      `[data-testid='histogram-${data.genes.search}']`
+    );
+    await utils.clickOn(`colorby-${data.genes.search}`);
+  });
+
+  test("pan and zoom", async () => {
+    await utils.clickOn("mode-pan-zoom");
+    const panCoords = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.pan["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      panCoords.start,
+      panCoords.end,
+      false
+    );
+    await page.evaluate(`window.scrollBy(0, 1000);`);
  });
 });
@@ -0,0 +1,10 @@
+{
+  "preset": "jest-puppeteer",
+  "testMatch": [
+    "**/__tests__/**/?(*.)(spec|test).js?(x)"
+  ],
+  "testURL": "http://localhost/",
+  "setupFiles": [
+    "../setupMissingGlobals.js"
+  ]
+}
@@ -0,0 +1,118 @@
+/*
+NOT run in Travis CI
+
+UX tests using puppeteer to be run locally.
+
+To run locally, ensure you are running the client is running on port 3000.
+Then run jest --verbose false --config __tests__/e2e/e2eJestConfig.json feature.
+ */
+
+import puppeteer from "puppeteer";
+import { appUrlBase, DEBUG, DEV, DATASET } from "./config";
+import { puppeteerUtils, cellxgeneActions } from "./puppeteerUtils";
+import { datasets } from "./data";
+
+let browser, page, utils, cxgActions, spy;
+const browserViewport = { width: 1280, height: 960 };
+let data = datasets[DATASET].features;
+
+if (DEBUG) jest.setTimeout(100000);
+if (DEV) jest.setTimeout(10000);
+
+beforeAll(async () => {
+  const browserParams = DEV
+    ? { headless: false, slowMo: 5 }
+    : DEBUG
+    ? { headless: false, slowMo: 100, devtools: true }
+    : {};
+  browser = await puppeteer.launch(browserParams);
+  page = await browser.newPage();
+  await page.setViewport(browserViewport);
+  if (DEV || DEBUG) {
+    page.on("console", msg => console.log(`PAGE LOG: ${msg.text()}`));
+  }
+  page.on("pageerror", err => {
+    throw new Error(`Console error: ${err}`);
+  });
+  utils = puppeteerUtils(page);
+  cxgActions = cellxgeneActions(page);
+});
+
+beforeEach(async () => {
+  await page.goto(appUrlBase);
+});
+
+afterAll(() => {
+  if (!DEBUG) {
+    browser.close();
+  }
+});
+
+describe("zoom interaction", async () => {
+  // Skip this test since UI is to hide lasso path when switching modes
+  test.skip("lasso visible after switching modes to pan/zoom", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    await utils.clickOn("mode-pan-zoom");
+    await utils.waitByID("lasso-element", { visible: true });
+  });
+
+  test("pan zoom mode resets lasso selection", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    const initialCount = await cxgActions.cellSet(1);
+    expect(initialCount).toBe(data.panzoom.lasso.count);
+    await utils.clickOn("mode-pan-zoom");
+    await utils.clickOn("mode-lasso");
+    const modeSwitchCount = await cxgActions.cellSet(1);
+    expect(modeSwitchCount).toBe(initialCount);
+  });
+
+  test("lasso moves after pan", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    const initialCount = await cxgActions.cellSet(1);
+    expect(initialCount).toBe(data.panzoom.lasso.count);
+    await utils.clickOn("mode-pan-zoom");
+    const panCoords = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      panCoords.start,
+      panCoords.end,
+      false
+    );
+    await utils.clickOn("mode-lasso");
+    const panCount = await cxgActions.cellSet(2);
+    expect(panCount).toBe(initialCount);
+  });
+});
@@ -0,0 +1,190 @@
+export const puppeteerUtils = puppeteerPage => ({
+  async waitByID(testid, props = {}) {
+    return await puppeteerPage.waitForSelector(
+      `[data-testid='${testid}']`,
+      props
+    );
+  },
+
+  async waitByClass(testclass, props = {}) {
+    return await puppeteerPage.waitForSelector(
+      `[data-testclass='${testclass}']`,
+      props
+    );
+  },
+
+  async typeInto(testid, text) {
+    // only works for text without special characters
+    await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
+    // type ahead can be annoying if you don't pause before you type
+    await puppeteerPage.click(selector);
+    await puppeteerPage.waitFor(200);
+    await puppeteerPage.type(selector, text);
+  },
+
+  async clearInputAndTypeInto(testid, text) {
+    await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
+    // only works for text without special characters
+    // type ahead can be annoying if you don't pause before you type
+    await puppeteerPage.click(selector);
+    await puppeteerPage.waitFor(200);
+    // select all
+
+    await puppeteerPage.click(selector, {clickCount: 3})
+    await puppeteerPage.keyboard.type("Backspace")
+    await puppeteerPage.type(selector, text);
+  },
+
+  async clickOn(testid) {
+    await this.waitByID(testid);
+    await puppeteerPage.click(`[data-testid='${testid}']`);
+    await puppeteerPage.waitFor(50);
+  },
+
+  async getOneElementInnerHTML(selector) {
+    await puppeteerPage.waitForSelector(selector);
+    let text = await puppeteerPage.$eval(selector, el => el.innerHTML);
+    return text;
+  },
+
+  async getOneElementInnerText(selector) {
+    await puppeteerPage.waitForSelector(selector);
+    let text = await puppeteerPage.$eval(selector, el => el.innerText);
+    return text;
+  }
+});
+
+export const cellxgeneActions = puppeteerPage => ({
+  async drag(testid, start, end, lasso = false) {
+    const layout = await puppeteerUtils(puppeteerPage).waitByID(testid);
+    const elBox = await layout.boxModel();
+    const x1 = elBox.content[0].x + start.x;
+    const x2 = elBox.content[0].x + end.x;
+    const y1 = elBox.content[0].y + start.y;
+    const y2 = elBox.content[0].y + end.y;
+    await puppeteerPage.mouse.move(x1, y1);
+    await puppeteerPage.mouse.down();
+    if (lasso) {
+      await puppeteerPage.mouse.move(x2, y1);
+      await puppeteerPage.mouse.move(x2, y2);
+      await puppeteerPage.mouse.move(x1, y2);
+      await puppeteerPage.mouse.move(x1, y1);
+    } else {
+      await puppeteerPage.mouse.move(x2, y2);
+    }
+    await puppeteerPage.mouse.up();
+  },
+
+  async getAllHistograms(testclass) {
+    await puppeteerUtils(puppeteerPage).waitByClass(testclass);
+    const histograms = await puppeteerPage.$$eval(
+      `[data-testclass=${testclass}]`,
+      els => {
+        return els.map(el => {
+          return el.dataset.testid.substring(
+            "histogram_".length,
+            el.dataset.testid.length
+          );
+        });
+      }
+    );
+    return histograms;
+  },
+
+  async getAllCategoriesAndCounts(category) {
+    await puppeteerUtils(puppeteerPage).waitByClass("categorical-row");
+    const categories = await puppeteerPage.$$eval(
+      `[data-testid="category-${category}"] [data-testclass='categorical-row']`,
+      els => {
+        let result = {};
+        els.forEach(el => {
+          const cat = el.querySelector("[data-testclass='categorical-value']")
+            .innerText;
+          const count = el.querySelector(
+            "[data-testclass='categorical-value-count']"
+          ).innerText;
+          result[cat] = count;
+        });
+        return result;
+      }
+    );
+    return categories;
+  },
+
+  async cellSet(num) {
+    await puppeteerUtils(puppeteerPage).clickOn(`cellset-button-${num}`);
+    return await puppeteerUtils(puppeteerPage).getOneElementInnerText(
+      `[data-testid='cellset-count-${num}']`
+    );
+  },
+
+  async resetCategory(category) {
+    const checkboxId = `category-select-${category}`;
+    await puppeteerUtils(puppeteerPage).waitByID(checkboxId);
+    const checkedPseudoclass = await puppeteerPage.$eval(
+      `[data-testid='${checkboxId}']`,
+      el => {
+        return el.matches(":checked");
+      }
+    );
+    if (!checkedPseudoclass) {
+      await puppeteerUtils(puppeteerPage).clickOn(checkboxId);
+    }
+    try {
+      const categoryRow = await puppeteerUtils(puppeteerPage).waitByID(
+        `category-expand-${category}`
+      );
+      const isExpanded = await categoryRow.$(
+        "[data-testclass='category-expand-is-expanded']"
+      );
+      if (isExpanded) {
+        await puppeteerUtils(puppeteerPage).clickOn(
+          `category-expand-${category}`
+        );
+      }
+    } catch {}
+  },
+
+  async calcDragCoordinates(testid, coordinateAsPercent) {
+    const el = await puppeteerUtils(puppeteerPage).waitByID(testid);
+    const size = await el.boxModel();
+    const coords = {
+      start: {
+        x: Math.floor(size.width * coordinateAsPercent.x1),
+        y: Math.floor(size.height * coordinateAsPercent.y1)
+      },
+      end: {
+        x: Math.floor(size.width * coordinateAsPercent.x2),
+        y: Math.floor(size.height * coordinateAsPercent.y2)
+      }
+    };
+    return coords;
+  },
+
+  async selectCategory(category, values, reset = true) {
+    if (reset) await this.resetCategory(category);
+    await puppeteerUtils(puppeteerPage).clickOn(`category-expand-${category}`);
+    await puppeteerUtils(puppeteerPage).clickOn(`category-select-${category}`);
+    for (const val of values) {
+      await puppeteerUtils(puppeteerPage).clickOn(
+        `categorical-value-select-${category}-${val}`
+      );
+    }
+  },
+
+  async reset() {
+    await puppeteerUtils(puppeteerPage).clickOn("reset");
+    // loading state never actually happens, reset is too fast
+    await page.waitFor(200);
+  },
+
+  async clip(min = 0, max = 100) { 
+    await puppeteerUtils(puppeteerPage).clickOn("visualization-settings");
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto("clip-min-input", min);
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto("clip-max-input", max);
+    await puppeteerUtils(puppeteerPage).clickOn("clip-commit");
+  }
+
+});
@@ -0,0 +1,48 @@
+import cascadeReducers from "../../src/reducers/cascade";
+
+describe("create", () => {
+  test("from Array", () => {
+    expect(cascadeReducers([["foo", () => 0]])).toBeInstanceOf(Function);
+  });
+
+  test("from Map", () => {
+    expect(cascadeReducers(new Map([["foo", () => 0]]))).toBeInstanceOf(
+      Function
+    );
+  });
+});
+
+describe("cascade", () => {
+  test("expected arguments provided & cascade ordering", () => {
+    const topLevelState = {};
+    const topLevelAction = { type: "test" };
+
+    const reducer = cascadeReducers([
+      [
+        "foo",
+        (currentState, action, nextSharedState, prevSharedState) => {
+          expect(currentState).toBeUndefined();
+          expect(action).toEqual(topLevelAction);
+          expect(nextSharedState).toStrictEqual({});
+          expect(prevSharedState).toBe(topLevelState);
+          return 0;
+        }
+      ],
+      [
+        "bar",
+        (currentState, action, nextSharedState, prevSharedState) => {
+          expect(currentState).toBeUndefined();
+          expect(action).toEqual(topLevelAction);
+          expect(nextSharedState).toStrictEqual({ foo: 0 });
+          expect(prevSharedState).toBe(topLevelState);
+          return 99;
+        }
+      ]
+    ]);
+
+    const nextState = reducer(topLevelState, topLevelAction);
+    expect(nextState).toStrictEqual({ foo: 0, bar: 99 });
+    expect(topLevelState).toStrictEqual({});
+    expect(topLevelAction).toStrictEqual({ type: "test" });
+  });
+});
@@ -0,0 +1,87 @@
+import undoable from "../../src/reducers/undoable";
+
+describe("create", () => {
+  test("no keys", () => {
+    expect(() => undoable(() => {})).toThrow();
+    expect(() => undoable(() => {}, null)).toThrow();
+    expect(() => undoable(() => {}, [])).toThrow();
+    expect(() => undoable(() => {}, [], {})).toThrow();
+  });
+
+  test("simple", () => {
+    expect(undoable(() => {}, ["foo"])).toBeInstanceOf(Function);
+    expect(undoable(() => {}, ["foo"], {})).toBeInstanceOf(Function);
+  });
+
+  test("handles undefined initial state", () => {
+    expect(
+      undoable(() => {}, ["a"])(undefined, { type: "test" })
+    ).toMatchObject({});
+  });
+});
+
+describe("undo", () => {
+  test("expected state modifications", () => {
+    const initialState = { a: 0, b: 1000 };
+    const reducer = state => {
+      return { a: state.a + 1, b: state.b + 1 };
+    };
+    const undoableReducer = undoable(reducer, ["a"]);
+
+    const s1 = undoableReducer(initialState, { type: "test" });
+    expect(s1).toMatchObject({ a: 1, b: 1001 });
+
+    // test that only specified keys are undone
+    const s2 = undoableReducer(s1, { type: "@@undoable/undo" });
+    expect(s2).toMatchObject({ a: 0, b: 1001 });
+
+    // test backstop when no more history
+    const s3 = undoableReducer(s2, { type: "@@undoable/undo" });
+    expect(s3).toMatchObject({ a: 0, b: 1001 });
+  });
+});
+
+describe("redo", () => {
+  const initialState = { a: 0, b: 1000 };
+  const reducer = state => {
+    return { a: state.a + 1, b: state.b + 1 };
+  };
+  let UR;
+
+  beforeEach(() => {
+    UR = undoable(reducer, ["a"]);
+  });
+
+  test("expected state modifications", () => {
+    const s1 = UR(initialState, { type: "test" });
+    expect(s1).toMatchObject({ a: 1, b: 1001 });
+
+    // verify undo->redo reverts state.
+    const s2 = UR(UR(s1, { type: "@@undoable/undo" }), {
+      type: "@@undoable/redo"
+    });
+    expect(s2).toMatchObject({ a: 1, b: 1001 });
+
+    // verify backstop when no redo future
+    const s3 = UR(s2, { type: "@@undoable/redo" });
+    expect(s3).toMatchObject({ a: 1, b: 1001 });
+  });
+
+  test("history cleared", () => {
+    // verify future cleared upon a normal state transition
+    const s1 = UR(initialState, { type: "test" });
+    expect(s1).toMatchObject({ a: 1, b: 1001 });
+    const s2 = UR(s1, { type: "@@undoable/undo" });
+    expect(s2).toMatchObject({ a: 0, b: 1001 });
+    const s3 = UR(s2, { type: "test" });
+    expect(s3).toMatchObject({ a: 1, b: 1002 });
+    const s4 = UR(s3, { type: "@@undoable/redo" });
+    expect(s4).toMatchObject({ a: 1, b: 1002 });
+  });
+});
+
+/*
+TODO:
+- historyLimit is enforced
+- action filters
+*/
@@ -532,6 +532,42 @@ describe("dataframe factories", () => {
      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
    });
  });
+
+  describe("mapColumns", () => {
+    test("identity", () => {
+      const dfA = Dataframe.Dataframe.create(
+        [3, 3],
+        [
+          new Array(3).fill(0),
+          new Int16Array(3).fill(99),
+          new Float64Array(3).fill(1.1)
+        ]
+      );
+      const dfB = dfA.mapColumns((col, idx) => {
+        expect(dfA.icol(idx).asArray()).toBe(col);
+        return col;
+      });
+      expect(dfA).not.toBe(dfB);
+      expect(dfA.dims).toEqual(dfB.dims);
+      for (let c = 0; c < dfA.dims[1]; c += 1) {
+        expect(dfA.icol(c).asArray()).toBe(dfB.icol(c).asArray());
+      }
+    });
+
+    test("transform", () => {
+      const dfA = Dataframe.Dataframe.create(
+        [3, 3],
+        [new Array(3).fill(0), new Array(3).fill(0), new Array(3).fill(0)]
+      );
+      const dfB = dfA.mapColumns(() => {
+        return new Array(3).fill(1);
+      });
+      expect(dfA).not.toBe(dfB);
+      expect(dfB.iat(0, 0)).toEqual(1);
+      expect(dfB.iat(0, 1)).toEqual(1);
+      expect(dfB.iat(0, 2)).toEqual(1);
+    });
+  });
 });

 describe("dataframe col", () => {
@@ -0,0 +1,29 @@
+import quantile from "../../src/util/quantile";
+
+describe("quantile", () => {
+	test("single q", () => {
+		const arr = new Float32Array([9, 3, 5, 6, 0]);
+		expect(quantile([1.0], arr)).toMatchObject([9]);
+		expect(quantile([0.9], arr)).toMatchObject([9]);
+		expect(quantile([0.8], arr)).toMatchObject([9]);
+		expect(quantile([0.7], arr)).toMatchObject([6]);
+		expect(quantile([0.6], arr)).toMatchObject([6]);
+		expect(quantile([0.5], arr)).toMatchObject([5]);
+		expect(quantile([0.4], arr)).toMatchObject([5]);
+		expect(quantile([0.3], arr)).toMatchObject([3]);
+		expect(quantile([0.2], arr)).toMatchObject([3]);
+		expect(quantile([0.1], arr)).toMatchObject([0]);
+		expect(quantile([0], arr)).toMatchObject([0]);
+	});
+
+	test("multi q", () => {
+		const arr = new Float32Array([9, 3, 5, 6, 0]);
+		expect(quantile([0, 0.25, 0.5, 0.75, 1.0], arr)).toMatchObject([
+			0,
+			3,
+			5,
+			6,
+			9
+		]);
+	});
+});
@@ -0,0 +1,42 @@
+import { range, rangeFill } from "../../src/util/range";
+
+describe("range", () => {
+	test("no defaults", () => {
+		expect(range(0, 3, 1)).toMatchObject([0, 1, 2]);
+	});
+
+	test("range(stop)", () => {
+		expect(range(3)).toMatchObject([0, 1, 2]);
+		expect(range(0)).toMatchObject([]);
+		expect(range(1)).toMatchObject([0]);
+	});
+
+	test("range(start,stop)", () => {
+		expect(range(0, 0)).toMatchObject([]);
+		expect(range(0, 2)).toMatchObject([0, 1]);
+		expect(range(4, 8)).toMatchObject([4, 5, 6, 7]);
+	});
+
+	test("range(start, stop, step", () => {
+		expect(range(4, 0, -1)).toMatchObject([4, 3, 2, 1]);
+		expect(range(0, 4, 2)).toMatchObject([0, 2]);
+	});
+});
+
+describe("rangefill", () => {
+	test("rangeFill(arr)", () => {
+		expect(rangeFill(new Int32Array(3))).toMatchObject(
+			new Int32Array([0, 1, 2])
+		);
+	});
+	test("rangeFill(arr, start)", () => {
+		expect(rangeFill(new Int32Array(2), 1)).toMatchObject(
+			new Int32Array([1, 2])
+		);
+	});
+	test("rangeFill(arr, start, step)", () => {
+		expect(rangeFill(new Int32Array(3), 2, -1)).toMatchObject(
+			new Int32Array([2, 1, 0])
+		);
+	});
+});
@@ -34,28 +34,38 @@ const aSchemaResponse = {
      type: "float32"
    },
    annotations: {
-      obs: [
-        { name: "name", type: "string" },
-        { name: "field1", type: "int32" },
-        { name: "field2", type: "float32" },
-        { name: "field3", type: "boolean" },
-        {
-          name: "field4",
-          type: "categorical",
-          categories: field4Categories
-        }
-      ],
-      var: [
-        { name: "name", type: "string" },
-        { name: "fieldA", type: "int32" },
-        { name: "fieldB", type: "float32" },
-        { name: "fieldC", type: "boolean" },
-        {
-          name: "fieldD",
-          type: "categorical",
-          categories: fieldDCategories
-        }
-      ]
+      obs: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "field1", type: "int32" },
+          { name: "field2", type: "float32" },
+          { name: "field3", type: "boolean" },
+          {
+            name: "field4",
+            type: "categorical",
+            categories: field4Categories
+          }
+        ]
+      },
+      var: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "fieldA", type: "int32" },
+          { name: "fieldB", type: "float32" },
+          { name: "fieldC", type: "boolean" },
+          {
+            name: "fieldD",
+            type: "categorical",
+            categories: fieldDCategories
+          }
+        ]
+      }
+    },
+    layout: {
+      obs: [{ name: "umap", type: "float32", dims: ["umap_0", "umap_1"] }],
+      var: []
    }
  }
 };
@@ -162,29 +172,7 @@ const aLayoutFBSResponse = (() => {
    new Float32Array(nObs).fill(Math.random()),
    new Float32Array(nObs).fill(Math.random())
  ];
-  const builder = new flatbuffers.Builder(1024);
-
-  const cols = _.map(coords, carr => {
-    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
-    NetEncoding.Float32Array.startFloat32Array(builder);
-    NetEncoding.Float32Array.addData(builder, cdv);
-    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
-
-    NetEncoding.Column.startColumn(builder);
-    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
-    NetEncoding.Column.addU(builder, floatArr);
-    return NetEncoding.Column.endColumn(builder);
-  });
-
-  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
-
-  NetEncoding.Matrix.startMatrix(builder);
-  NetEncoding.Matrix.addNRows(builder, nObs);
-  NetEncoding.Matrix.addNCols(builder, coords.length);
-  NetEncoding.Matrix.addColumns(builder, columns);
-  const matrix = NetEncoding.Matrix.endMatrix(builder);
-  builder.finish(matrix);
-  return builder.asUint8Array();
+  return encodeMatrix(coords, ["umap_0", "umap_1"]);
 })();

 const aDataObsResponse = {
@@ -53,13 +53,15 @@ describe("createUniverseFromResponse", () => {

    expect(universe.obsAnnotations.dims).toEqual([
      nObs,
-      REST.schema.schema.annotations.obs.length
+      REST.schema.schema.annotations.obs.columns.length
    ]);
    expect(universe.obsLayout.dims).toEqual([nObs, 2]);
-    expect(universe.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(universe.obsLayout.colIndex.keys()).toEqual(
+      universe.schema.layout.obs[0].dims
+    );
    expect(universe.varAnnotations.dims).toEqual([
      nVar,
-      REST.schema.schema.annotations.var.length
+      REST.schema.schema.annotations.var.columns.length
    ]);
    expect(universe.varData.isEmpty()).toBeTruthy();
  });
@@ -3,6 +3,7 @@ import * as Universe from "../../../src/util/stateManager/universe";
 import * as World from "../../../src/util/stateManager/world";
 import * as Dataframe from "../../../src/util/dataframe";
 import Crossfilter from "../../../src/util/typedCrossfilter";
+import { DimTypes } from "../../../src/util/typedCrossfilter/crossfilter";
 import * as REST from "./sampleResponses";
 import {
  obsAnnoDimensionName,
@@ -26,15 +27,16 @@ const defaultBigBang = () => {
  /* create world */
  const world = World.createWorldFromEntireUniverse(universe);
  /* create crossfilter */
-  const crossfilter = Crossfilter(world.obsAnnotations);
-  /* create dimension map */
-  const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+  const crossfilter = World.createObsDimensions(
+    new Crossfilter(world.obsAnnotations),
+    world,
+    REST.schema.schema.layout.obs[0].dims
+  );

  return {
    universe,
    world,
-    crossfilter,
-    dimensionMap
+    crossfilter
  };
 };

@@ -57,10 +59,15 @@ describe("createWorldFromEntireUniverse", () => {
        nObs: universe.nObs,
        nVar: universe.nVar,
        schema: universe.schema,
-        obsAnnotations: universe.obsAnnotations,
-        varAnnotations: universe.varAnnotations,
-        obsLayout: universe.obsLayout,
-        varData: expect.any(Dataframe.Dataframe)
+        obsAnnotations: expect.any(Dataframe.Dataframe),
+        varAnnotations: expect.any(Dataframe.Dataframe),
+        obsLayout: expect.any(Dataframe.Dataframe),
+        varData: expect.any(Dataframe.Dataframe),
+        clipQuantiles: { min: 0, max: 1 },
+        unclipped: {
+          obsAnnotations: expect.any(Dataframe.Dataframe),
+          varData: expect.any(Dataframe.Dataframe)
+        }
      })
    );
  });
@@ -71,22 +78,25 @@ describe("createWorldFromCurrentSelection", () => {
    const {
      universe,
      world: originalWorld,
-      crossfilter,
-      dimensionMap
+      crossfilter: originalCrossfilter
    } = defaultBigBang();

    /* mock a selection */
-    dimensionMap[obsAnnoDimensionName("field1")].filterRange([0, 5]);
-    dimensionMap[obsAnnoDimensionName("field3")].filterExact(false);
+    const crossfilter = originalCrossfilter
+      .select(obsAnnoDimensionName("field1"), { mode: "range", lo: 0, hi: 5 })
+      .select(obsAnnoDimensionName("field3"), {
+        mode: "exact",
+        values: [false]
+      });

    /* create the world from the selection */
-    const world = World.createWorldFromCurrentSelection(
+    const world = World.createWorldBySelection(
      universe,
      originalWorld,
      crossfilter
    );
    expect(world).toBeDefined();
-    expect(world.nObs).toEqual(crossfilter.countFiltered());
+    expect(world.nObs).toEqual(crossfilter.countSelected());

    /*
    calculate expected values and match against result
@@ -108,10 +118,15 @@ describe("createWorldFromCurrentSelection", () => {
        nObs: matchingIndices.length,
        nVar: universe.nVar,
        schema: universe.schema,
+        clipQuantiles: { min: 0, max: 1 },
        obsAnnotations: expect.any(Dataframe.Dataframe),
-        varAnnotations: universe.varAnnotations,
+        varAnnotations: expect.any(Dataframe.Dataframe),
        obsLayout: expect.any(Dataframe.Dataframe),
-        varData: expect.any(Dataframe.Dataframe)
+        varData: expect.any(Dataframe.Dataframe),
+        unclipped: {
+          obsAnnotations: expect.any(Dataframe.Dataframe),
+          varData: expect.any(Dataframe.Dataframe)
+        }
      })
    );

@@ -124,7 +139,9 @@ describe("createWorldFromCurrentSelection", () => {
    expect(world.obsLayout.rowIndex.keys()).toEqual(
      new Int32Array(matchingIndices)
    );
-    expect(world.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(world.obsLayout.colIndex.keys()).toEqual(
+      world.schema.layout.obs[0].dims
+    );
  });
 });

@@ -136,43 +153,36 @@ describe("createObsDimensionMap", () => {
    - check that dimension typing is sane
    */

-    const { dimensionMap } = defaultBigBang();
+    const { crossfilter } = defaultBigBang();
    const annotationNames = _.map(
-      REST.schema.schema.annotations.obs,
+      REST.schema.schema.annotations.obs.columns,
      c => c.name
    );
-    const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
-    expect(dimensionMap).toBeDefined();
+    const obsIndexColName = REST.schema.schema.annotations.obs.index;
+    const schemaByObsName = _.keyBy(
+      REST.schema.schema.annotations.obs.columns,
+      "name"
+    );
+    expect(crossfilter).toBeDefined();
    annotationNames.forEach(name => {
-      const dim = dimensionMap[obsAnnoDimensionName(name)];
-      if (name === "name") {
+      const dim = crossfilter.dimensions[obsAnnoDimensionName(name)];
+      if (name === obsIndexColName) {
        expect(dim).toBeUndefined();
      } else {
        const { type } = schemaByObsName[name];
        if (type === "string" || type === "boolean" || type === "categorical") {
-          expect(dim).toBeInstanceOf(Crossfilter.EnumDimension);
+          expect(dim.dim).toBeInstanceOf(DimTypes.enum);
        } else {
-          expect(dim).toBeInstanceOf(Crossfilter.ScalarDimension);
+          expect(dim.dim).toBeInstanceOf(DimTypes.scalar);
        }
      }
    });
-    expect(dimensionMap[layoutDimensionName("XY")]).toBeInstanceOf(
-      Crossfilter.SpatialDimension
-    );
+    expect(
+      crossfilter.dimensions[layoutDimensionName("XY")].dim
+    ).toBeInstanceOf(DimTypes.spatial);
  });
 });

-describe("createVarDataDimension", () => {
-  /* create default universe */
-  const { world, crossfilter } = defaultBigBang();
-  world.varData = world.varData.withCol(
-    "GENE",
-    Float32Array.from(_.range(world.nObs))
-  );
-  const result = World.createVarDataDimension(world, crossfilter, "GENE");
-  expect(result).toBeInstanceOf(Crossfilter.ScalarDimension);
-});
-
 describe("worldEqUniverse", () => {
  const { universe, world } = defaultBigBang();
  const result = World.worldEqUniverse(world, universe);
@@ -0,0 +1,434 @@
+import _ from "lodash";
+import { polygonContains } from "d3";
+
+import Crossfilter from "../../../src/util/typedCrossfilter";
+
+const someData = [
+  {
+    date: "2011-11-14T16:17:54Z",
+    quantity: 2,
+    total: 190,
+    tip: 100,
+    type: "tab",
+    productIDs: ["001"],
+    coords: [0, 0],
+    nonFinite: 0.0
+  },
+  {
+    date: "2011-11-14T16:20:19Z",
+    quantity: 2,
+    total: 190,
+    tip: 100,
+    type: "tab",
+    productIDs: ["001", "005"],
+    coords: [0.4, 0.4],
+    nonFinite: Number.NaN
+  },
+  {
+    date: "2011-11-14T16:28:54Z",
+    quantity: 1,
+    total: 300,
+    tip: 200,
+    type: "visa",
+    productIDs: ["004", "005"],
+    coords: [0.3, 0.1],
+    nonFinite: Number.POSITIVE_INFINITY
+  },
+  {
+    date: "2011-11-14T16:30:43Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001", "002"],
+    coords: [0.392, 0.1],
+    nonFinite: Number.NEGATIVE_INFINITY
+  },
+  {
+    date: "2011-11-14T16:48:46Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["005"],
+    coords: [0.7, 0.0482],
+    nonFinite: 1.0
+  },
+  {
+    date: "2011-11-14T16:53:41Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001", "004", "005"],
+    coords: [0.9999, 1.0],
+    nonFinite: Number.NaN
+  },
+  {
+    date: "2011-11-14T16:54:06Z",
+    quantity: 1,
+    total: 100,
+    tip: 0,
+    type: "cash",
+    productIDs: ["001", "002", "003", "004", "005"],
+    coords: [0.384, 0.6938],
+    nonFinite: 99.0
+  },
+  {
+    date: "2011-11-14T16:58:03Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001"],
+    coords: [0.4822, 0.482],
+    nonFinite: Number.NaN
+  },
+  {
+    date: "2011-11-14T17:07:21Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["004", "005"],
+    coords: [0.2234, 0],
+    nonFinite: Number.NaN
+  },
+  {
+    date: "2011-11-14T17:22:59Z",
+    quantity: 2,
+    total: 90,
+    tip: 0,
+    type: "tab",
+    productIDs: ["001", "002", "004", "005"],
+    coords: [0.382, 0.38485],
+    nonFinite: -1
+  },
+  {
+    date: "2011-11-14T17:25:45Z",
+    quantity: 2,
+    total: 200,
+    tip: 0,
+    type: "cash",
+    productIDs: ["002"],
+    coords: [0.998, 0.8472],
+    nonFinite: 0.0
+  },
+  {
+    date: "2011-11-14T17:29:52Z",
+    quantity: 1,
+    total: 200,
+    tip: 100,
+    type: "visa",
+    productIDs: ["004"],
+    coords: [0.8273, 0.3384],
+    nonFinite: 0.0
+  }
+];
+
+let payments = null;
+beforeEach(() => {
+  payments = new Crossfilter(someData);
+});
+
+describe("ImmutableTypedCrossfilter", () => {
+  test("create crossfilter", () => {
+    expect(payments).toBeDefined();
+    expect(payments.size()).toEqual(someData.length);
+    expect(payments.all()).toEqual(someData);
+
+    const p = payments
+      .addDimension("quantity", "scalar", (i, d) => d[i].quantity, Int32Array)
+      .select("quantity", { mode: "all" });
+    expect(p).toBeDefined();
+    expect(p.all()).toEqual(someData);
+    expect(p.size()).toEqual(someData.length);
+    expect(p.isElementSelected(0)).toBeTruthy();
+    expect(p.countSelected()).toEqual(someData.length);
+    expect(p.allSelected()).toEqual(someData);
+  });
+
+  test("immutability", () => {
+    /*
+    the following should return a new crossfilter:
+    - addDimension()
+    - delDimension()
+    - select
+    */
+    const p2 = payments.addDimension(
+      "quantity",
+      "scalar",
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
+
+    expect(payments).not.toBe(p2);
+    const p3 = p2.select("quantity", { mode: "all" });
+    expect(p3).not.toBe(p2);
+
+    const p4 = p3.delDimension("quantity");
+    expect(p4).not.toBe(p3);
+  });
+
+  test("select all and none", () => {
+    let p = payments
+      .addDimension("quantity", "scalar", (i, d) => d[i].quantity, Int32Array)
+      .addDimension("tip", "scalar", (i, d) => d[i].tip, Float32Array)
+      .addDimension("total", "scalar", (i, d) => d[i].total, Float32Array)
+      .addDimension("type", "enum", (i, d) => d[i].type);
+    expect(p).toBeDefined();
+
+    /* expect all records to be selected - default init state */
+    expect(p.allSelected()).toEqual(someData);
+    expect(p.countSelected()).toEqual(someData.length);
+    expect(p.allSelectedMask()).toEqual(
+      new Uint8Array(someData.length).fill(1)
+    );
+    expect(p.fillByIsSelected(new Uint8Array(someData.length), 99, 0)).toEqual(
+      new Uint8Array(someData.length).fill(99)
+    );
+    for (let i = 0; i < someData.length; i += 1) {
+      expect(p.isElementSelected(i)).toBeTruthy();
+    }
+
+    /* expect a selectAll on one dimension to change nothing */
+    p = p.select("tip", { mode: "all" });
+    expect(p.allSelected()).toEqual(someData);
+
+    /* ditto */
+    p = p.select("quantity", { mode: "all" });
+    expect(p.allSelected()).toEqual(someData);
+
+    /* select none on one dimension */
+    p = p.select("type", { mode: "none" });
+    expect(p.allSelected()).toEqual([]);
+    expect(p.countSelected()).toEqual(0);
+    expect(p.allSelectedMask()).toEqual(
+      new Uint8Array(someData.length).fill(0)
+    );
+    expect(p.fillByIsSelected(new Uint8Array(someData.length), 99, 0)).toEqual(
+      new Uint8Array(someData.length).fill(0)
+    );
+    for (let i = 0; i < someData.length; i += 1) {
+      expect(p.isElementSelected(i)).toBeFalsy();
+    }
+
+    p = p.select("quantity", { mode: "none" });
+    expect(p.allSelected()).toEqual([]);
+
+    // invert the first none; should have no effect because type is
+    // still not filtered.
+    p = p.select("quantity", { mode: "all" });
+    expect(p.allSelected()).toEqual([]);
+
+    /* select all of type; should select all records */
+    p = p.select("type", { mode: "all" });
+    expect(p.allSelected()).toEqual(someData);
+  });
+
+  describe("scalar dimension", () => {
+    let p;
+    beforeEach(() => {
+      p = payments
+        .addDimension("quantity", "scalar", (i, d) => d[i].quantity, Int32Array)
+        .addDimension("tip", "scalar", (i, d) => d[i].tip, Float32Array)
+        .select("tip", { mode: "all" });
+    });
+
+    /*
+    select modes:  all, none, exact, range
+    */
+    test("all", () => {
+      expect(p.select("quantity", { mode: "all" }).countSelected()).toEqual(
+        someData.length
+      );
+    });
+    test("none", () => {
+      expect(p.select("quantity", { mode: "none" }).countSelected()).toEqual(0);
+    });
+    test.each([[[]], [[2]], [[2, 1]], [[9, 82]], [[0, 1]]])("exact: %p", v =>
+      expect(
+        p.select("quantity", { mode: "exact", values: v }).countSelected()
+      ).toEqual(_.filter(someData, d => v.includes(d.quantity)).length)
+    );
+    test.each([[0, 1], [1, 2], [0, 99], [99, 100000]])("range %p", (lo, hi) =>
+      expect(
+        p.select("quantity", { mode: "range", lo, hi }).countSelected()
+      ).toEqual(
+        _.filter(someData, d => d.quantity >= lo && d.quantity < hi).length
+      )
+    );
+    test("bad mode", () => {
+      expect(() => p.select("type", { mode: "bad mode" })).toThrow(Error);
+    });
+  });
+
+  describe("enum dimension", () => {
+    let p;
+    beforeEach(() => {
+      p = payments.addDimension("type", "enum", (i, d) => d[i].type);
+    });
+
+    test("all", () => {
+      expect(p.select("type", { mode: "all" }).countSelected()).toEqual(
+        someData.length
+      );
+    });
+    test("none", () => {
+      expect(p.select("type", { mode: "none" }).countSelected()).toEqual(0);
+    });
+    test.each([
+      [[]],
+      [["tab"]],
+      [["visa"]],
+      [["visa", "tab"]],
+      [["cash", "tab", "visa"]]
+    ])("exact: %p", v =>
+      expect(
+        p.select("type", { mode: "exact", values: v }).countSelected()
+      ).toEqual(_.filter(someData, d => v.includes(d.type)).length)
+    );
+    test("range", () => {
+      expect(() => p.select("type", { mode: "range", lo: 0, hi: 9 })).toThrow(
+        Error
+      );
+    });
+    test("bad mode", () => {
+      expect(() => p.select("type", { mode: "bad mode" })).toThrow(Error);
+    });
+  });
+
+  describe("spatial dimension", () => {
+    let p;
+    beforeEach(() => {
+      const X = someData.map(r => r.coords[0]);
+      const Y = someData.map(r => r.coords[1]);
+      p = payments.addDimension("coords", "spatial", X, Y);
+    });
+
+    test("all", () => {
+      expect(p.select("coords", { mode: "all" }).countSelected()).toEqual(
+        someData.length
+      );
+    });
+    test("none", () => {
+      expect(p.select("coords", { mode: "none" }).countSelected()).toEqual(0);
+    });
+    test.each([[0, 0, 1, 1], [0, 0, 0.5, 0.5], [0.5, 0.5, 1, 1]])(
+      "within-rect %d %d %d %d",
+      (minX, minY, maxX, maxY) => {
+        expect(
+          p
+            .select("coords", { mode: "within-rect", minX, minY, maxX, maxY })
+            .allSelected()
+        ).toEqual(
+          _.filter(someData, d => {
+            const [x, y] = d.coords;
+            return minX <= x && x < maxX && minY <= y && y < maxY;
+          })
+        );
+      }
+    );
+
+    test.each([
+      [[[0, 0], [0, 1], [1, 1], [1, 0]]],
+      [[[0, 0], [0, 0.5], [0.5, 0.5], [0.5, 0]]]
+    ])("within-polygon %p", polygon => {
+      expect(
+        p.select("coords", { mode: "within-polygon", polygon }).allSelected()
+      ).toEqual(_.filter(someData, d => polygonContains(polygon, d.coords)));
+    });
+  });
+
+  describe("non-finite scalars", () => {
+    let p;
+    beforeEach(() => {
+      p = payments
+        .addDimension("quantity", "scalar", (i, d) => d[i].quantity, Int32Array)
+        .addDimension(
+          "nonFinite",
+          "scalar",
+          (i, d) => d[i].nonFinite,
+          Float32Array
+        )
+        .select("quantity", { mode: "all" });
+    });
+
+    test("all or none", () => {
+      expect(p.select("nonFinite", { mode: "all" }).countSelected()).toEqual(
+        someData.length
+      );
+      expect(p.select("nonFinite", { mode: "none" }).countSelected()).toEqual(
+        0
+      );
+    });
+
+    test("exact", () => {
+      expect(
+        p.select("nonFinite", { mode: "exact", values: [0] }).countSelected()
+      ).toEqual(3);
+      expect(
+        p.select("nonFinite", { mode: "exact", values: [1] }).countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.POSITIVE_INFINITY]
+          })
+          .countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.NEGATIVE_INFINITY]
+          })
+          .countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", { mode: "exact", values: [Number.NaN] })
+          .countSelected()
+      ).toEqual(4);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.POSITIVE_INFINITY, 0, 1, 99]
+          })
+          .countSelected()
+      ).toEqual(6);
+    });
+
+    test("range", () => {
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: 0,
+            hi: Number.POSITIVE_INFINITY
+          })
+          .countSelected()
+      ).toEqual(5);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: 0,
+            hi: Number.NaN
+          })
+          .countSelected()
+      ).toEqual(6);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: Number.NEGATIVE_INFINITY,
+            hi: Number.POSITIVE_INFINITY
+          })
+          .countSelected()
+      ).toEqual(7);
+    });
+  });
+});
@@ -1,4 +1,22 @@
-import { sort, sortIndex } from "../../../src/util/typedCrossfilter/sort";
+import {
+  sortArray,
+  sortIndex,
+  lowerBound,
+  upperBound,
+  lowerBoundIndirect,
+  upperBoundIndirect
+} from "../../../src/util/typedCrossfilter/sort";
+
+/*
+Sort tests should keep in mind that there are separate code
+paths for:
+  - small vs. large arrays (insertionsort only)
+  - float-only typed arrays vs. other array types (non-finite handling)
+  - indexed vs. direct sort
+*/
+
+const pInf = Number.POSITIVE_INFINITY;
+const nInf = Number.NEGATIVE_INFINITY;

 function fillRange(arr, start = 0) {
  const larr = arr;
@@ -15,42 +33,223 @@ function fillRand(arr) {
  return arr;
 }

-describe("sort", () => {
-  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
-    test(Type.name, () => {
-      expect(sort(Type.from([6, 5, 4, 3, 2, 1, 0]))).toMatchObject(
-        Type.from([0, 1, 2, 3, 4, 5, 6])
-      );
-      expect(sort(Type.from([6, 5, 4, 3, 2, 1]))).toMatchObject(
-        Type.from([1, 2, 3, 4, 5, 6])
-      );
+describe("sortArray", () => {
+  describe("JS vals", () => {
+    [
+      [true, false],
+      ["a", "b", "0", "1"],
+      [0, "a", true, null, undefined, 3.1415],
+      fillRand(new Array(1000)),
+      ["a", NaN, null, pInf]
+    ].map((val, idx) =>
+      test(`JS vals ${idx}`, () => {
+        expect(sortArray(val)).toMatchObject(val.sort());
+      })
+    );
+  });

-      const source = fillRand(new Type(1000));
-      expect(sort(Type.from(source))).toMatchObject(Type.from(source).sort());
-    })
-  );
+  describe("finite numbers", () => {
+    [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+      test(Type.name, () => {
+        expect(sortArray(Type.from([6, 5, 4, 3, 2, 1, 0]))).toMatchObject(
+          Type.from([0, 1, 2, 3, 4, 5, 6])
+        );
+
+        expect(sortArray(Type.from([6, 5, 4, 3, 2, 1]))).toMatchObject(
+          Type.from([1, 2, 3, 4, 5, 6])
+        );
+
+        const source = fillRand(new Type(1000));
+        expect(sortArray(Type.from(source))).toMatchObject(
+          Type.from(source).sort()
+        );
+      })
+    );
+  });
+
+  describe("non-finite numbers", () => {
+    test("inifinity", () => {
+      expect(sortArray(new Float32Array([pInf, nInf, 0, 1, 2]))).toMatchObject(
+        new Float32Array([nInf, 0, 1, 2, pInf])
+      );
+      expect(
+        sortArray(new Float32Array([pInf, nInf, pInf, nInf]))
+      ).toMatchObject(new Float32Array([nInf, nInf, pInf, pInf]));
+      expect(
+        sortArray(new Float32Array([pInf, nInf, pInf, nInf, pInf]))
+      ).toMatchObject(new Float32Array([nInf, nInf, pInf, pInf, pInf]));
+      expect(
+        sortArray(
+          new Float32Array(100).fill(Infinity, 0, 50).fill(-Infinity, 50, 100)
+        )
+      ).toMatchObject(
+        new Float32Array(100).fill(-Infinity, 0, 50).fill(Infinity, 50, 100)
+      );
+    });
+    test("NaN", () => {
+      expect(sortArray(new Float64Array([NaN, 2, 1, 0]))).toMatchObject(
+        new Float64Array([0, 1, 2, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, 1, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, NaN, 1, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, 1, NaN, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN, NaN])
+      );
+      expect(
+        sortArray(fillRange(new Float32Array(100)).fill(NaN, 0, 10))
+      ).toMatchObject(fillRange(new Float32Array(100), 10).fill(NaN, 90, 100));
+    });
+
+    test("mixed numbers", () => {
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, NaN, NaN]))
+      ).toMatchObject(new Float32Array([nInf, pInf, NaN, NaN, NaN]));
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, NaN, 1, NaN, 2]))
+      ).toMatchObject(new Float32Array([nInf, 1, 2, pInf, NaN, NaN, NaN]));
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, 0, 1, NaN, 2]))
+      ).toMatchObject(new Float32Array([nInf, 0, 1, 2, pInf, NaN, NaN]));
+      expect(
+        sortArray(
+          fillRange(new Float32Array(100))
+            .fill(NaN, 0, 10)
+            .fill(Infinity, 10, 20)
+        )
+      ).toMatchObject(
+        fillRange(new Float32Array(100), 20)
+          .fill(Infinity, 80, 90)
+          .fill(NaN, 90, 100)
+      );
+    });
+  });
 });

 describe("sortIndex", () => {
-  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
-    test(Type.name, () => {
-      const source1 = Type.from([6, 5, 4, 3, 2, 1, 0]);
+  describe("finite numbers", () => {
+    [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+      test(Type.name, () => {
+        const source1 = Type.from([6, 5, 4, 3, 2, 1, 0]);
+        const index1 = fillRange(new Uint32Array(source1.length));
+        expect(sortIndex(index1, source1)).toMatchObject(
+          index1.sort((a, b) => source1[a] - source1[b])
+        );
+
+        const source2 = Type.from([6, 5, 4, 3, 2, 1]);
+        const index2 = fillRange(new Uint32Array(source2.length));
+        expect(sortIndex(index2, source2)).toMatchObject(
+          index2.sort((a, b) => source1[a] - source1[b])
+        );
+
+        const source3 = fillRand(new Type(1000));
+        const index3 = fillRange(new Uint32Array(source3.length));
+        expect(sortIndex(index3, source3)).toMatchObject(
+          index3.sort((a, b) => source1[a] - source1[b])
+        );
+      })
+    );
+  });
+
+  describe("non-finite numbers", () => {
+    test("mixed numbers", () => {
+      const source1 = new Float32Array([NaN, pInf, nInf, NaN, 1, NaN, 2]);
      const index1 = fillRange(new Uint32Array(source1.length));
      expect(sortIndex(index1, source1)).toMatchObject(
-        index1.sort((a, b) => source1[a] - source1[b])
+        new Uint32Array([2, 4, 6, 1, 0, 3, 5])
      );

-      const source2 = Type.from([6, 5, 4, 3, 2, 1]);
+      const source2 = new Float32Array([NaN, pInf, nInf, 0, 1, NaN, 2]);
      const index2 = fillRange(new Uint32Array(source2.length));
      expect(sortIndex(index2, source2)).toMatchObject(
-        index2.sort((a, b) => source1[a] - source1[b])
+        new Uint32Array([2, 3, 4, 6, 1, 0, 5])
      );
+    });
+  });
+});

-      const source3 = fillRand(new Type(1000));
-      const index3 = fillRange(new Uint32Array(source3.length));
-      expect(sortIndex(index3, source3)).toMatchObject(
-        index3.sort((a, b) => source1[a] - source1[b])
-      );
-    })
-  );
+describe("lowerBound", () => {
+  test("non-float path", () => {
+    expect(lowerBound([], 0, 0, 0)).toEqual(0);
+
+    expect(lowerBound([0, 1, 2, 3], -1, 0, 4)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3], 0, 0, 4)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3], 1, 0, 4)).toEqual(1);
+    expect(lowerBound([0, 1, 2, 3], 3, 0, 4)).toEqual(3);
+    expect(lowerBound([0, 1, 2, 3], 4, 0, 4)).toEqual(4);
+
+    expect(lowerBound([0, 1, 2, 3, 4], -1, 0, 5)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3, 4], 0, 0, 5)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3, 4], 2, 0, 5)).toEqual(2);
+    expect(lowerBound([0, 1, 2, 3, 4], 4, 0, 5)).toEqual(4);
+    expect(lowerBound([0, 1, 2, 3, 4], 5, 0, 5)).toEqual(5);
+
+    expect(lowerBound([0, 2, 4, 6, 8], 5, 0, 5)).toEqual(3);
+    expect(lowerBound([0, 2, 2, 2, 8], 5, 0, 5)).toEqual(4);
+
+    expect(lowerBound([0, 1, 2, 3, 4, 5, 6, 7, 8], 3, 2, 4)).toEqual(3);
+    expect(lowerBound([0, 1, 2, 3, 4, 5, 6, 7, 8], 99, 2, 4)).toEqual(4);
+  });
+
+  test("float path, finites", () => {
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
+
+    expect(lowerBound(new Float32Array([]), 0, 0, 0)).toEqual(0);
+
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), -1, 0, 4)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 0, 0, 4)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 3, 0, 4)).toEqual(3);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 4, 0, 4)).toEqual(4);
+
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), -1, 0, 5)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 0, 0, 5)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 2, 0, 5)).toEqual(2);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 4, 0, 5)).toEqual(4);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 5, 0, 5)).toEqual(5);
+
+    expect(lowerBound(new Float32Array([0, 2, 4, 6, 8]), 5, 0, 5)).toEqual(3);
+    expect(lowerBound(new Float32Array([0, 2, 2, 2, 8]), 5, 0, 5)).toEqual(4);
+
+    expect(
+      lowerBound(new Float32Array([0, 1, 2, 3, 4, 5, 6, 7, 8]), 3, 2, 4)
+    ).toEqual(3);
+    expect(
+      lowerBound(new Float32Array([0, 1, 2, 3, 4, 5, 6, 7, 8]), 99, 2, 4)
+    ).toEqual(4);
+  });
+
+  test("float path, non-finite", () => {
+    expect(
+      lowerBound(
+        new Float32Array([-Infinity, 0, 1, Infinity, NaN]),
+        -Infinity,
+        0,
+        5
+      )
+    ).toEqual(0);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 0, 0, 5)
+    ).toEqual(1);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 1, 0, 5)
+    ).toEqual(2);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 2, 0, 5)
+    ).toEqual(3);
+    expect(
+      lowerBound(
+        new Float32Array([-Infinity, 0, 1, Infinity, NaN]),
+        Infinity,
+        0,
+        5
+      )
+    ).toEqual(3);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), NaN, 0, 5)
+    ).toEqual(4);
+  });
 });
@@ -1,590 +0,0 @@
-// jshint esversion: 6
-import _ from "lodash";
-import crossfilter from "../../../src/util/typedCrossfilter";
-
-const someData = [
-  {
-    date: "2011-11-14T16:17:54Z",
-    quantity: 2,
-    total: 190,
-    tip: 100,
-    type: "tab",
-    productIDs: ["001"]
-  },
-  {
-    date: "2011-11-14T16:20:19Z",
-    quantity: 2,
-    total: 190,
-    tip: 100,
-    type: "tab",
-    productIDs: ["001", "005"]
-  },
-  {
-    date: "2011-11-14T16:28:54Z",
-    quantity: 1,
-    total: 300,
-    tip: 200,
-    type: "visa",
-    productIDs: ["004", "005"]
-  },
-  {
-    date: "2011-11-14T16:30:43Z",
-    quantity: 2,
-    total: 90,
-    tip: 0,
-    type: "tab",
-    productIDs: ["001", "002"]
-  },
-  {
-    date: "2011-11-14T16:48:46Z",
-    quantity: 2,
-    total: 90,
-    tip: 0,
-    type: "tab",
-    productIDs: ["005"]
-  },
-  {
-    date: "2011-11-14T16:53:41Z",
-    quantity: 2,
-    total: 90,
-    tip: 0,
-    type: "tab",
-    productIDs: ["001", "004", "005"]
-  },
-  {
-    date: "2011-11-14T16:54:06Z",
-    quantity: 1,
-    total: 100,
-    tip: 0,
-    type: "cash",
-    productIDs: ["001", "002", "003", "004", "005"]
-  },
-  {
-    date: "2011-11-14T16:58:03Z",
-    quantity: 2,
-    total: 90,
-    tip: 0,
-    type: "tab",
-    productIDs: ["001"]
-  },
-  {
-    date: "2011-11-14T17:07:21Z",
-    quantity: 2,
-    total: 90,
-    tip: 0,
-    type: "tab",
-    productIDs: ["004", "005"]
-  },
-  {
-    date: "2011-11-14T17:22:59Z",
-    quantity: 2,
-    total: 90,
-    tip: 0,
-    type: "tab",
-    productIDs: ["001", "002", "004", "005"]
-  },
-  {
-    date: "2011-11-14T17:25:45Z",
-    quantity: 2,
-    total: 200,
-    tip: 0,
-    type: "cash",
-    productIDs: ["002"]
-  },
-  {
-    date: "2011-11-14T17:29:52Z",
-    quantity: 1,
-    total: 200,
-    tip: 100,
-    type: "visa",
-    productIDs: ["004"]
-  }
-];
-
-function groupReduce(data, valueMap, valueReduce, valueInit) {
-  return _.reduce(
-    data,
-    (acc, value) => {
-      const k = valueMap(value);
-      let r = _.find(acc, o => o.key === k);
-      if (!r) {
-        r = { key: k, value: valueInit() };
-        acc.push(r);
-      }
-      r.value = valueReduce(r.value, value);
-      return acc;
-    },
-    []
-  ).sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
-}
-
-function groupCount(data, map) {
-  return groupReduce(data, map, p => p + 1, () => 0);
-}
-
-function groupSum(data, map) {
-  return groupReduce(
-    data,
-    map,
-    (p, v) => {
-      p += map(v);
-      return p;
-    },
-    () => 0
-  );
-}
-
-let payments = null;
-beforeEach(() => {
-  payments = crossfilter(someData);
-});
-
-describe("typedCrossfilter", () => {
-  test("alloc and free", () => {
-    expect(payments).toBeDefined();
-    expect(payments.size()).toEqual(someData.length);
-    expect(payments.all()).toEqual(someData);
-
-    const quantity = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].quantity,
-      Int32Array
-    );
-    expect(quantity).toBeDefined();
-    expect(quantity.id()).toBeDefined();
-
-    quantity.dispose();
-    expect(payments.size()).toEqual(someData.length);
-    expect(payments.all()).toEqual(someData);
-  });
-
-  test("filterAll and filterNone", () => {
-    expect(payments).toBeDefined();
-    const quantity = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].quantity,
-      Int32Array
-    );
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Float32Array
-    );
-    const total = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total,
-      Float32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    expect(quantity).toBeDefined();
-    expect(tip).toBeDefined();
-    expect(total).toBeDefined();
-    expect(type).toBeDefined();
-
-    // initially, all should be filtered
-    expect(payments.allFiltered()).toHaveLength(payments.size());
-    expect(payments.allFiltered()).toEqual(payments.all());
-    expect(payments.countFiltered()).toEqual(someData.length);
-
-    // filterAll
-    tip.filterAll(); // should change nothing
-    expect(payments.allFiltered()).toEqual(payments.all());
-    expect(payments.countFiltered()).toEqual(someData.length);
-
-    // ditto
-    total.filterAll();
-    expect(payments.allFiltered()).toEqual(payments.all());
-    expect(payments.countFiltered()).toEqual(someData.length);
-
-    // filterNone
-    type.filterNone();
-    expect(payments.allFiltered()).toEqual([]);
-    expect(payments.countFiltered()).toEqual(0);
-
-    quantity.filterNone();
-    expect(payments.allFiltered()).toEqual([]);
-    expect(payments.countFiltered()).toEqual(0);
-
-    // invert the first none; should have no effect because type is
-    // still not filtered
-    quantity.filterAll();
-    expect(payments.allFiltered()).toEqual([]);
-    expect(payments.countFiltered()).toEqual(0);
-
-    // filter all of type; should select all
-    type.filterAll();
-    expect(payments.allFiltered()).toEqual(payments.all());
-    expect(payments.countFiltered()).toEqual(payments.size());
-  });
-
-  test("filterExact", () => {
-    expect(payments).toBeDefined();
-    const quantity = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].quantity,
-      Int32Array
-    );
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Float32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    quantity.filterExact(1);
-    expect(payments.countFiltered()).toEqual(
-      _.countBy(someData, "quantity")[1]
-    );
-    expect(payments.allFiltered()).toEqual(_.filter(someData, { quantity: 1 }));
-
-    tip.filterExact(0);
-    expect(payments.allFiltered()).toEqual(
-      _.filter(someData, { tip: 0, quantity: 1 })
-    );
-
-    type.filterExact("cash");
-    expect(payments.allFiltered()).toEqual(
-      _.filter(someData, { tip: 0, quantity: 1, type: "cash" })
-    );
-  });
-
-  test("filterRange", () => {
-    expect(payments).toBeDefined();
-    const quantity = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].quantity,
-      Int32Array
-    );
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Float32Array
-    );
-    const total = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total,
-      Float32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    tip.filterRange([0, 91]);
-    expect(payments.allFiltered()).toEqual(
-      _(someData)
-        .filter(r => r.tip >= 0 && r.tip < 91)
-        .value()
-    );
-
-    tip.filterRange([0, 90]);
-    expect(payments.allFiltered()).toEqual(
-      _(someData)
-        .filter(r => r.tip >= 0 && r.tip < 90)
-        .value()
-    );
-
-    tip.filterRange([1, 90]);
-    expect(payments.allFiltered()).toEqual(
-      _(someData)
-        .filter(r => r.tip >= 1 && r.tip < 91)
-        .value()
-    );
-  });
-
-  test("filterEnum", () => {
-    expect(payments).toBeDefined();
-    const quantity = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].quantity,
-      Int32Array
-    );
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Float32Array
-    );
-    const total = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total,
-      Float32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    type.filterEnum(["tab", "cash"]);
-    expect(payments.allFiltered()).toEqual(
-      _(someData)
-        .filter(r => r.type === "cash" || r.type === "tab")
-        .value()
-    );
-
-    tip.filterEnum([0, 100]);
-    expect(payments.allFiltered()).toEqual(
-      _(someData)
-        .filter(r => r.type === "cash" || r.type === "tab")
-        .filter(r => r.tip === 0 || r.tip === 100)
-        .value()
-    );
-  });
-
-  test("more than 32 dimensions", () => {
-    expect(payments).toBeDefined();
-    const quantity = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].quantity,
-      Int32Array
-    );
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Float32Array
-    );
-    const total = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total,
-      Float32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    // Create a bunch of fake dimensions to ensure we can handle > 32
-    let dimMap = {};
-    for (let i = 0; i < 65; i++) {
-      dimMap[i] = payments.dimension(
-        crossfilter.ScalarDimension,
-        () => Math.random(),
-        Float32Array
-      );
-      expect(dimMap[i]).toBeDefined();
-      expect(dimMap[i].id()).toBeDefined();
-    }
-
-    // everything should start as selected/filtered
-    expect(payments.countFiltered()).toEqual(someData.length);
-
-    dimMap[0].filterAll();
-    dimMap[64].filterAll();
-    expect(payments.countFiltered()).toEqual(someData.length);
-
-    dimMap[33].filterNone();
-    expect(payments.allFiltered()).toEqual([]);
-
-    dimMap[33].filterAll();
-    expect(payments.allFiltered()).toEqual(someData);
-  });
-
-  test("group, default mapping, default reducer, no filter", () => {
-    expect(payments).toBeDefined();
-
-    const quantity = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].quantity,
-      Int32Array
-    );
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Int32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-    const total = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total,
-      Int32Array
-    );
-
-    _.each(
-      {
-        tip: tip.group(r => r),
-        type: type.group(),
-        total: total.group(),
-        quantity: quantity.group()
-      },
-      (grp, k) => {
-        const whatWeExpect = groupCount(someData, v => v[k]);
-        expect(grp.all()).toEqual(whatWeExpect);
-        expect(grp.size()).toEqual(whatWeExpect.length);
-        expect(grp.dispose()).toEqual(grp);
-      }
-    );
-  });
-
-  test("group, custom map, default reducer, no filters", () => {
-    expect(payments).toBeDefined();
-
-    // custom mapping in groups only works for scalar types.  Enums do not
-    // currently implement it.
-
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Int32Array
-    );
-    const totalX10 = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total * 10,
-      Int32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    const paymentsByTip_A = tip.group();
-    const paymentsByTip_B = tip.group(r => 10 * r);
-    const paymentsByType = type.group(); // identity only
-    const paymentsByTotalX10_A = totalX10.group();
-    const paymentsByTotalX10_B = totalX10.group(r => r / 10);
-
-    expect(paymentsByTip_A.all()).toEqual(groupCount(someData, v => v.tip));
-    expect(paymentsByTip_B.all()).toEqual(
-      groupCount(someData, v => 10 * v.tip)
-    );
-    expect(paymentsByType.all()).toEqual(groupCount(someData, v => v.type));
-    expect(paymentsByTotalX10_A.all()).toEqual(
-      groupCount(someData, v => 10 * v.total)
-    );
-    expect(paymentsByTotalX10_B.all()).toEqual(
-      groupCount(someData, v => (10 * v.total) / 10)
-    );
-
-    for (let i of [
-      paymentsByTip_A,
-      paymentsByTip_B,
-      paymentsByType,
-      paymentsByTotalX10_A,
-      paymentsByTotalX10_B,
-      tip,
-      totalX10,
-      type
-    ]) {
-      expect(i.dispose()).toEqual(i);
-    }
-  });
-
-  test("group, default map, custom reducer, no filters", () => {
-    expect(payments).toBeDefined();
-
-    const total = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total,
-      Float32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    const paymentsByTotal = total.group();
-    const paymentsByType = type.group();
-
-    // reduceCount
-    expect(paymentsByTotal.reduceCount()).toEqual(paymentsByTotal);
-    expect(paymentsByTotal.all()).toEqual(groupCount(someData, v => v.total));
-
-    // reduceSum
-    expect(paymentsByTotal.reduceSum(v => v.total)).toEqual(paymentsByTotal);
-    expect(paymentsByTotal.all()).toEqual(groupSum(someData, v => v.total));
-
-    // use custom reducers (my reducers) - count by three, init 1
-    expect(
-      paymentsByTotal.reduce((p, v) => (p += 3), (p, v) => (p -= 3), () => 1)
-    ).toEqual(paymentsByTotal);
-    expect(paymentsByTotal.all()).toEqual(
-      groupReduce(someData, v => v.total, (p, v) => p + 3, () => 1)
-    );
-
-    for (let i of [paymentsByTotal, paymentsByType, type]) {
-      expect(i.dispose()).toEqual(i);
-    }
-  });
-
-  test("group, default map, default reducer, filters", () => {
-    // From the docs:
-    // Note: a grouping intersects the crossfilter's current filters, except for the
-    // associated dimension's filter. Thus, group methods consider only records that
-    // satisfy every filter except this dimension's filter. So, if the crossfilter of
-    // payments is filtered by type and total, then group by total only observes the
-    // filter by type.
-
-    expect(payments).toBeDefined();
-
-    const tip = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].tip,
-      Int32Array
-    );
-    const total = payments.dimension(
-      crossfilter.ScalarDimension,
-      (i, data) => data[i].total,
-      Int32Array
-    );
-    const type = payments.dimension(
-      crossfilter.EnumDimension,
-      (i, data) => data[i].type
-    );
-
-    const paymentsByTip = tip.group();
-    const paymentsByTotal = total.group();
-    const paymentsByType = type.group();
-
-    // 1. confirm that changing the filter on a dimension does NOT change that
-    // dimensions groups.
-    {
-      tip.filterAll(), total.filterAll(), type.filterAll();
-      let before = _.cloneDeep(paymentsByTip.all());
-      tip.filterExact(0);
-      expect(paymentsByTip.all()).toEqual(before);
-    }
-
-    // 2. confirm that changing a filter on a different dimension DOES change
-    // all other groups.
-    {
-      tip.filterAll(), total.filterAll(), type.filterAll();
-      const before = _.cloneDeep([paymentsByTotal.all(), paymentsByType.all()]);
-      tip.filterExact(0);
-      const after = [paymentsByTotal.all(), paymentsByType.all()];
-      expect(after).not.toEqual(before);
-      expect(after).toEqual([
-        groupReduce(
-          someData,
-          v => v.total,
-          (p, v) => (v.tip !== 0 ? p : p + 1),
-          () => 0
-        ),
-        groupReduce(
-          someData,
-          v => v.type,
-          (p, v) => (v.tip !== 0 ? p : p + 1),
-          () => 0
-        )
-      ]);
-    }
-
-    for (let i of [
-      paymentsByTip,
-      paymentsByTotal,
-      paymentsByType,
-      tip,
-      total,
-      type
-    ]) {
-      expect(i.dispose()).toEqual(i);
-    }
-  });
-});
@@ -1,11 +1,8 @@
 import {
-  fillRange,
  sliceByIndex,
-  makeSortIndex,
-  lowerBound,
-  lowerBoundIndirect,
-  upperBoundIndirect
+  makeSortIndex
 } from "../../../src/util/typedCrossfilter/util";
+import { rangeFill as fillRange } from "../../../src/util/range";

 describe("fillRange", () => {
  test("Array", () => {
@@ -7,8 +7,10 @@ module.exports = {
  ],
  plugins: [
    "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
    ["@babel/plugin-proposal-decorators", { legacy: true }],
-    "@babel/plugin-proposal-export-namespace-from"
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
+    "@babel/plugin-proposal-export-namespace-from",
+    "@babel/plugin-proposal-optional-chaining",
+    "@babel/plugin-proposal-nullish-coalescing-operator"
  ]
 };
@@ -6,10 +6,12 @@ module.exports = {
  ],
  plugins: [
    "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
    ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
    "@babel/plugin-proposal-export-namespace-from",
    "@babel/plugin-transform-react-constant-elements",
-    "@babel/plugin-transform-runtime"
+    "@babel/plugin-transform-runtime",
+    "@babel/plugin-proposal-optional-chaining",
+    "@babel/plugin-proposal-nullish-coalescing-operator"
  ]
 };
@@ -77,6 +77,9 @@ module.exports = {
      template: path.resolve("index.html"),
      favicon: path.resolve("favicon.png")
    }),
-    new webpack.NoEmitOnErrorsPlugin()
+    new webpack.NoEmitOnErrorsPlugin(),
+    new webpack.DefinePlugin({
+      __REACT_DEVTOOLS_GLOBAL_HOOK__: "({ isDisabled: true })"
+    })
  ]
 };
@@ -1,35 +1,42 @@
 <!DOCTYPE html>
 <html lang="en">
-  <head>
+<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>cellxgene</title>
    <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
    <style>
-      html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
-        font-family: 'Roboto Condensed','Helvetica Neue','Helvetica','Arial',sans-serif;
-        font-size: 14px;
-      }
-      body {
-        margin: 0;
-        padding: 0;
-      }
+        html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
+            font-family: 'Roboto Condensed', 'Helvetica Neue', 'Helvetica', 'Arial', sans-serif;
+            font-size: 14px;
+        }

-      * {
-        box-sizing: border-box;
-      }
+        body {
+            margin: 0;
+            padding: 0;
+        }
+
+        * {
+            box-sizing: border-box;
+        }
    </style>
-  </head>
-  <body>
-<script type="text/javascript">
-window.CELLXGENE = {};
-window.CELLXGENE.API = {
-  prefix: window.location.href + "api/",
-  version: "v0.2/"
-};
-</script>
-    <noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>, please <strong>enable JS</strong> to make this app work.</noscript>

-    <div id="root"></div>
-  </body>
+</head>
+<body>
+<script type="text/javascript">
+  window.CELLXGENE = {};
+  window.CELLXGENE.API = {
+    prefix: window.location.href + "api/",
+    version: "v0.2/"
+  };
+</script>
+<noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>,
+    please <strong>enable JS</strong> to make this app work.
+</noscript>
+
+<div id="root"></div>
+{% for script in SCRIPTS %}
+<script type="text/javascript" src="{{script | safe}}"></script>
+{% endfor %}
+</body>
 </html>
@@ -1,6 +1,6 @@
 {
  "name": "cellxgene",
-  "version": "0.7.0",
+  "version": "0.10.0",
  "license": "MIT",
  "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
  "repository": "https://github.com/chanzuckerberg/cellxgene",
@@ -9,12 +9,13 @@
    "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
    "clean": "rimraf build",
    "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
-    "e2e": "jest e2e",
+    "e2e": "node node_modules/jest/bin/jest.js --verbose false --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
    "lint": "eslint src",
    "smoke-test": "start-server-and-test start-server-for-test :5000 e2e",
    "start": "node server/development.js",
    "start-server-for-test": "cellxgene launch -p 5000 ../example-dataset/pbmc3k.h5ad",
-    "unit-test": "jest --testPathIgnorePatterns e2e"
+    "test": "node node_modules/jest/bin/jest.js",
+    "unit-test": "node node_modules/jest/bin/jest.js --testPathIgnorePatterns e2e"
  },
  "engineStrict": true,
  "engines": {
@@ -30,9 +31,9 @@
    "eslint-scope": "3.7.1"
  },
  "dependencies": {
-    "@blueprintjs/core": "^3.8.0",
-    "@blueprintjs/icons": "^3.3.0",
-    "@blueprintjs/select": "^3.2.1",
+    "@blueprintjs/core": "^3.15.1",
+    "@blueprintjs/icons": "^3.8.0",
+    "@blueprintjs/select": "^3.8.0",
    "canvas-fit": "^1.5.0",
    "d3": "^4.10.0",
    "d3-scale-chromatic": "^1.3.0",
@@ -40,76 +41,77 @@
    "font-color-contrast": "^1.0.3",
    "fuzzysort": "^1.1.4",
    "gl-mat4": "^1.1.4",
-    "gl-matrix": "^2.7.1",
+    "gl-vec3": "^1.1.3",
+    "gl-matrix": "^3.0.0",
    "is-number": "^7.0.0",
    "key-pressed": "0.0.1",
    "lodash": "^4.17.4",
-    "memoize-one": "^4.0.0",
+    "memoize-one": "^5.0.4",
    "mouse-position": "^2.0.1",
    "mouse-pressed": "^1.0.0",
-    "normalize.css": "^8.0.0",
    "orbit-camera": "^1.0.0",
-    "query-string": "^6.1.0",
-    "react": "^16.6.0",
+    "query-string": "^6.5.0",
+    "react": "^16.8.6",
    "react-autocomplete": "^1.7.2",
-    "react-dom": "^16.6.0",
-    "react-helmet": "^5.2.0",
-    "react-icons": "^3.2.2",
-    "react-redux": "^5.1.0",
+    "react-dom": "^16.8.6",
+    "react-helmet": "^5.2.1",
+    "react-icons": "^3.7.0",
+    "react-redux": "^7.0.3",
    "redux": "^4.0.1",
-    "redux-devtools-extension": "^2.13.5",
    "redux-thunk": "^2.2.0",
-    "regl": "^1.3.9",
+    "regl": "^1.3.11",
    "scroll-speed": "^1.0.0",
    "urijs": "^1.19.0"
  },
  "devDependencies": {
-    "@babel/core": "^7.1.5",
-    "@babel/plugin-proposal-class-properties": "^7.0.0",
-    "@babel/plugin-proposal-decorators": "^7.0.0",
-    "@babel/plugin-proposal-export-namespace-from": "^7.0.0",
-    "@babel/plugin-proposal-function-bind": "^7.0.0",
-    "@babel/plugin-transform-react-constant-elements": "^7.0.0",
-    "@babel/plugin-transform-runtime": "^7.1.0",
-    "@babel/preset-env": "^7.1.5",
+    "@babel/core": "^7.4.4",
+    "@babel/plugin-proposal-class-properties": "^7.4.4",
+    "@babel/plugin-proposal-decorators": "^7.4.4",
+    "@babel/plugin-proposal-export-namespace-from": "^7.2.0",
+    "@babel/plugin-proposal-function-bind": "^7.2.0",
+    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.4.4",
+    "@babel/plugin-proposal-optional-chaining": "^7.2.0",
+    "@babel/plugin-transform-react-constant-elements": "^7.2.0",
+    "@babel/plugin-transform-runtime": "^7.4.4",
+    "@babel/preset-env": "^7.4.4",
    "@babel/preset-react": "^7.0.0",
-    "@babel/register": "^7.0.0",
-    "@babel/runtime": "^7.1.5",
-    "babel-core": "^7.0.0-bridge.0",
+    "@babel/register": "^7.4.4",
+    "@babel/runtime": "^7.4.4",
    "babel-eslint": "^10.0.1",
-    "babel-jest": "^23.6.0",
-    "babel-loader": "^8.0.0",
-    "babel-preset-modern-browsers": "^12.0.0",
+    "babel-jest": "^24.8.0",
+    "babel-loader": "^8.0.6",
+    "babel-preset-modern-browsers": "^14.0.0",
    "chalk": "^2.4.2",
    "connect-history-api-fallback": "^1.6.0",
-    "copy-webpack-plugin": "^4.6.0",
-    "css-loader": "^1.0.1",
-    "eslint": "^5.13.0",
+    "copy-webpack-plugin": "^5.0.3",
+    "css-loader": "^2.1.1",
+    "eslint": "^5.16.0",
    "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^4.0.0",
+    "eslint-config-prettier": "^4.2.0",
    "eslint-loader": "^2.1.2",
    "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.16.0",
-    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-import": "^2.17.2",
+    "eslint-plugin-jest": "^22.5.1",
    "eslint-plugin-jsx-a11y": "^6.2.1",
-    "eslint-plugin-react": "^7.12.4",
+    "eslint-plugin-react": "^7.13.0",
    "express": "^4.14.0",
-    "file-loader": "^2.0.0",
+    "file-loader": "^3.0.1",
    "html-webpack-inline-source-plugin": "0.0.10",
    "html-webpack-plugin": "^3.2.0",
-    "jest": "^24.1.0",
+    "jest": "^24.8.0",
+    "jest-puppeteer": "^4.1.1",
    "json-loader": "^0.5.4",
-    "mini-css-extract-plugin": "^0.4.1",
-    "puppeteer": "^1.12.1",
+    "mini-css-extract-plugin": "^0.6.0",
+    "puppeteer": "^1.16.0",
    "rimraf": "^2.6.3",
    "serve-favicon": "^2.3.0",
-    "start-server-and-test": "^1.7.11",
+    "start-server-and-test": "^1.9.0",
    "style-loader": "^0.23.1",
    "sw-precache-webpack-plugin": "^0.11.5",
    "url-loader": "^1.1.0",
-    "webpack": "^4.25.1",
-    "webpack-cli": "^3.1.0",
-    "webpack-dev-middleware": "^3.1.3"
+    "webpack": "^4.31.0",
+    "webpack-cli": "^3.3.2",
+    "webpack-dev-middleware": "^3.6.2"
  },
  "jest": {
    "testMatch": [
@@ -129,16 +131,23 @@
        ],
        "plugins": [
          "@babel/plugin-proposal-function-bind",
-          "@babel/plugin-proposal-class-properties",
          [
            "@babel/plugin-proposal-decorators",
            {
              "legacy": true
            }
          ],
+          [
+            "@babel/plugin-proposal-class-properties",
+            {
+              "loose": true
+            }
+          ],
          "@babel/plugin-proposal-export-namespace-from",
          "@babel/plugin-transform-react-constant-elements",
-          "@babel/plugin-transform-runtime"
+          "@babel/plugin-transform-runtime",
+          "@babel/plugin-proposal-optional-chaining",
+          "@babel/plugin-proposal-nullish-coalescing-operator"
        ]
      }
    }
@@ -21,24 +21,32 @@ const doInitialDataLoad = () =>
    dispatch({ type: "initial data load start" });

    try {
-      const requestJson = _(["config", "schema"])
+      /*
+      Step 1 - config & schema, all JSON
+      */
+      const requestJson = ["config", "schema"]
        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doJsonRequest(url))
-        .value();
-      const requestBinary = _([
-        "annotations/obs",
-        "annotations/var?annotation-name=name",
-        "layout/obs"
-      ])
-        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doBinaryRequest(url))
-        .value();
-
-      const results = await Promise.all(_.concat(requestJson, requestBinary));
-
+        .map(url => doJsonRequest(url));
+      const stepOneResults = await Promise.all(requestJson);
      /* set config defaults */
-      const config = { ...globals.configDefaults, ...results[0].config };
-      const [, schema, obsAnno, varAnno, obsLayout] = [...results];
+      const config = { ...globals.configDefaults, ...stepOneResults[0].config };
+      const schema = stepOneResults[1];
+
+      /*
+      Step 2 - dataframes, all binary.  NOTE: uses results of step 1.
+      */
+      /* only load names for var annotations, if possible*/
+      const varIndexName = schema?.schema?.annotations?.var?.index;
+      const varAnnotationsQuery = varIndexName
+        ? `?annotation-name=${varIndexName}`
+        : "";
+      const varAnnotationsURL = `annotations/var${varAnnotationsQuery}`;
+      const requestBinary = ["annotations/obs", varAnnotationsURL, "layout/obs"]
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doBinaryRequest(url));
+      const stepTwoResults = await Promise.all(requestBinary);
+      const [obsAnno, varAnno, obsLayout] = [...stepTwoResults];
+
      const universe = Universe.createUniverseFromResponse(
        config,
        schema,
@@ -65,7 +73,7 @@ Set the view (world) to current selection.   Placeholder for an async action
 which also does re-layout.
 */
 const regraph = () => (dispatch, getState) => {
-  const { universe, world, crossfilter } = getState().controls;
+  const { universe, world, crossfilter } = getState();
  dispatch({
    type: "set World to current selection",
    universe,
@@ -91,6 +99,10 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  const state = getState();
+  const { universe } = state;
+  const varIndexName = universe.schema.annotations.var.index;
+
  /* helper for this function only */
  const fetchData = async geneNames => {
    const res = await fetch(
@@ -100,7 +112,7 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
        body: JSON.stringify({
          filter: {
            var: {
-              annotation_value: [{ name: "name", values: geneNames }]
+              annotation_value: [{ name: varIndexName, values: geneNames }]
            }
          }
        }),
@@ -123,8 +135,6 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
    return Universe.convertDataFBStoObject(universe, data);
  };

-  const state = getState();
-  const { universe } = state.controls;
  /* preload data already in cache */
  let expressionData = _.transform(
    genes,
@@ -164,7 +174,7 @@ function requestSingleGeneExpressionCountsForColoringPOST(gene) {
    dispatch({ type: "get single gene expression for coloring started" });
    try {
      await _doRequestExpressionData(dispatch, getState, [gene]);
-      const { world } = getState().controls;
+      const { world } = getState();
      dispatch({
        type: "color by expression",
        gene,
@@ -185,7 +195,7 @@ const requestUserDefinedGene = gene => async (dispatch, getState) => {
  dispatch({ type: "request user defined gene started" });
  try {
    await await _doRequestExpressionData(dispatch, getState, [gene]);
-    const { world } = getState().controls;
+    const { world } = getState();

    /* then send the success case action through */
    return dispatch({
@@ -240,7 +250,8 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
    2. get expression data for each
    */
    const state = getState();
-    const { universe } = state.controls;
+    const { universe } = state;
+    const varIndexName = universe.schema.annotations.var.index;

    // Legal values are null, Array or TypedArray.  Null is initial state.
    if (!set1) set1 = [];
@@ -277,7 +288,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
    const data = await res.json();
    // result is [ [varIdx, ...], ... ]
    const topNGenes = _.map(data, r =>
-      universe.varAnnotations.at(r[0], "name")
+      universe.varAnnotations.at(r[0], varIndexName)
    );

    /*
@@ -300,8 +311,11 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
 };

 const resetInterface = () => (dispatch, getState) => {
-  const { universe } = getState().controls;
+  const { universe } = getState();

+  dispatch({
+    type: "user reset start"
+  });
  dispatch({
    type: "clear all user defined genes"
  });
@@ -321,6 +335,9 @@ const resetInterface = () => (dispatch, getState) => {
  dispatch({
    type: "increment graph render counter"
  });
+  dispatch({
+    type: "user reset end"
+  });
 };

 export default {
@@ -5,69 +5,65 @@ https://bl.ocks.org/SpaceActuary/2f004899ea1b2bd78d6f1dbb2febf771
 */
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { Button, ButtonGroup, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
 import * as d3 from "d3";
 import memoize from "memoize-one";
 import * as globals from "../../globals";
 import actions from "../../actions";
-import finiteExtent from "../../util/finiteExtent";
+import { makeContinuousDimensionName } from "../../util/nameCreators";

@connect(state => ({
-  world: state.controls.world,
+  world: state.world,
  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
-  crossfilter: state.controls.crossfilter,
+  continuousSelection: state.continuousSelection,
  differential: state.differential,
-  colorAccessor: state.controls.colorAccessor,
-  colorScale: state.controls.colorScale,
-  obsAnnotations: _.get(state.controls.world, "obsAnnotations", null)
+  colorAccessor: state.colors.colorAccessor
 }))
 class HistogramBrush extends React.Component {
-  calcHistogramCache = memoize((obsAnnotations, field, rangeMin, rangeMax) => {
-    const { world } = this.props;
-    const histogramCache = {};
+  static getColumn(world, field, clipped = true) {
+    /*
+    Return the underlying Dataframe column for our field.   By default, 
+    returns the clipped column.   If clipped===false, will return the 
+    unclipped column.
+    */
+    const obsAnnotations = clipped
+      ? world.obsAnnotations
+      : world.unclipped.obsAnnotations;
+    const varData = clipped ? world.varData : world.unclipped.varData;
+    if (obsAnnotations.hasCol(field)) {
+      return obsAnnotations.col(field);
+    }
+    return varData.col(field);
+  }

+  calcHistogramCache = memoize((col, field) => {
+    /*
+     recalculate expensive stuff, notably bins, summaries, etc.
+    */
+    const histogramCache = {};
+    const values = col.asArray();
+    const summary = col.summarize();
+    const { min: domainMin, max: domainMax } = summary;
+    histogramCache.x = d3
+      .scaleLinear()
+      .domain([domainMin, domainMax])
+      .range([0, this.width - this.marginRight]);
+
+    histogramCache.bins = d3
+      .histogram()
+      .domain(histogramCache.x.domain())
+      .thresholds(40)(values);
+
+    const yMax = histogramCache.bins
+      .map(b => b.length)
+      .reduce((a, b) => Math.max(a, b));
    histogramCache.y = d3
      .scaleLinear()
+      .domain([0, yMax])
      .range([this.height - this.marginBottom, 0]);

-    if (obsAnnotations.hasCol(field)) {
-      // recalculate expensive stuff
-      const allValuesForContinuousFieldAsArray = obsAnnotations
-        .col(field)
-        .asArray();
-
-      histogramCache.x = d3
-        .scaleLinear()
-        .domain([rangeMin, rangeMax])
-        .range([0, this.width]);
-
-      histogramCache.bins = d3
-        .histogram()
-        .domain(histogramCache.x.domain())
-        .thresholds(40)(allValuesForContinuousFieldAsArray);
-
-      histogramCache.numValues = allValuesForContinuousFieldAsArray.length;
-    } else if (world.varData.hasCol(field)) {
-      const varValues = world.varData.col(field).asArray();
-
-      histogramCache.x = d3
-        .scaleLinear()
-        .domain(
-          finiteExtent(varValues)
-        ) /* replace this if we have ranges for genes back from server like we do for annotations on cells */
-        .range([0, this.width]);
-
-      histogramCache.bins = d3
-        .histogram()
-        .domain(histogramCache.x.domain())
-        .thresholds(40)(varValues);
-
-      histogramCache.numValues = varValues.length;
-    }
-
    return histogramCache;
  });

@@ -76,32 +72,98 @@ class HistogramBrush extends React.Component {

    this.width = 340;
    this.height = 100;
-    this.marginBottom = 20;
+    this.marginBottom = 20; // space for X axis & labels
+    this.marginRight = 40; // space for Y axis & labels
  }

  componentDidMount() {
    const { field } = this.props;
-    const { x, y, bins, numValues, svgRef } = this._histogram;
+    const { x, y, bins, svgRef } = this._histogram;

-    this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+    this.renderAxesBrushBins(x, y, bins, svgRef, field);
  }

  componentDidUpdate(prevProps) {
-    const { field, obsAnnotations } = this.props;
-    const { x, y, bins, numValues, svgRef } = this._histogram;
+    const { field, world, continuousSelection } = this.props;
+    const { x, y, bins, svgRef } = this._histogram;
+    let { brushXselection, brushX } = this.state;
+    let forceBrushUpdate = false;

-    if (obsAnnotations !== prevProps.obsAnnotations) {
-      this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+    /*
+    Update our axis if the underlying dataframe column has changed
+    */
+    const dfColumn = HistogramBrush.getColumn(world, field);
+    const oldDfColumn = HistogramBrush.getColumn(
+      prevProps.world,
+      prevProps.field
+    );
+    if (dfColumn !== oldDfColumn) {
+      ({ brushXselection, brushX } = this.renderAxesBrushBins(
+        x,
+        y,
+        bins,
+        svgRef,
+        field
+      ));
+      forceBrushUpdate = true;
+    }
+
+    /*
+    if the selection has changed, ensure that the brush correctly reflects
+    the underlying selection.
+    */
+    if (
+      forceBrushUpdate ||
+      continuousSelection !== prevProps.continuousSelection
+    ) {
+      const { isObs, isUserDefined, isDiffExp } = this.props;
+      const myName = makeContinuousDimensionName(
+        { isObs, isUserDefined, isDiffExp },
+        field
+      );
+      const range = continuousSelection[myName];
+      if (brushXselection) {
+        const selection = d3.brushSelection(brushXselection.node());
+        if (!range && selection) {
+          /* no active selection - clear brush */
+          brushXselection.call(brushX.move, null);
+        } else if (range && !selection) {
+          /* there is an active selection, but no brush - set the brush */
+          const x0 = x(range[0]);
+          const x1 = x(range[1]);
+          brushXselection.call(brushX.move, [x0, x1]);
+        } else if (range && selection) {
+          /* there is an active selection and a brush - make sure they match */
+          const moveDeltaThreshold = 1;
+          const x0 = x(range[0]);
+          const x1 = x(range[1]);
+          const dX0 = Math.abs(x0 - selection[0]);
+          const dX1 = Math.abs(x1 - selection[1]);
+          /*
+          only update the brush if it is grossly incorrect,
+          as defined by the moveDeltaThreshold
+          */
+          if (dX0 > moveDeltaThreshold || dX1 > moveDeltaThreshold) {
+            brushXselection.call(brushX.move, [x0, x1]);
+          }
+        }
+      }
    }
  }

-  onBrush(selection, x) {
+  onBrush(selection, x, eventType) {
+    const type = `continuous metadata histogram ${eventType}`;
    return () => {
      const { dispatch, field, isObs, isUserDefined, isDiffExp } = this.props;

+      // ignore programmatically generated events
+      if (!d3.event.sourceEvent) return;
+      // ignore cascading events, which are programmatically generated
+      if (d3.event.sourceEvent.sourceEvent) return;
+
      if (d3.event.selection) {
        dispatch({
-          type: "continuous metadata histogram brush",
+          type,
          selection: field,
          continuousNamespace: {
            isObs,
@@ -112,7 +174,7 @@ class HistogramBrush extends React.Component {
        });
      } else {
        dispatch({
-          type: "continuous metadata histogram brush",
+          type,
          selection: field,
          continuousNamespace: {
            isObs,
@@ -125,24 +187,73 @@ class HistogramBrush extends React.Component {
    };
  }

+  onBrushEnd(selection, x) {
+    return () => {
+      const { dispatch, field, isObs, isUserDefined, isDiffExp } = this.props;
+      const minAllowedBrushSize = 10;
+      const smallAmountToAvoidInfiniteLoop = 0.1;
+
+      // ignore programmatically generated events
+      if (!d3.event.sourceEvent) return;
+      // ignore cascading events, which are programmatically generated
+      if (d3.event.sourceEvent.sourceEvent) return;
+
+      if (d3.event.selection) {
+        let _range;
+
+        if (
+          d3.event.selection[1] - d3.event.selection[0] >
+          minAllowedBrushSize
+        ) {
+          _range = [x(d3.event.selection[0]), x(d3.event.selection[1])];
+        } else {
+          /* the user selected range is too small and will be hidden #587, so take control of it procedurally */
+          /* https://stackoverflow.com/questions/12354729/d3-js-limit-size-of-brush */
+
+          const procedurallyResizedBrushWidth =
+            d3.event.selection[0] +
+            minAllowedBrushSize +
+            smallAmountToAvoidInfiniteLoop; //
+
+          _range = [x(d3.event.selection[0]), x(procedurallyResizedBrushWidth)];
+        }
+
+        dispatch({
+          type: "continuous metadata histogram end",
+          selection: field,
+          continuousNamespace: {
+            isObs,
+            isUserDefined,
+            isDiffExp
+          },
+          range: _range
+        });
+      } else {
+        dispatch({
+          type: "continuous metadata histogram cancel",
+          selection: field,
+          continuousNamespace: {
+            isObs,
+            isUserDefined,
+            isDiffExp
+          }
+        });
+      }
+    };
+  }
+
  drawHistogram(svgRef) {
-    const { obsAnnotations, field, ranges } = this.props;
-    const histogramCache = this.calcHistogramCache(
-      obsAnnotations,
-      field,
-      ranges.min,
-      ranges.max
-    );
-
-    const { x, y, bins, numValues } = histogramCache;
-
-    this._histogram = { x, y, bins, numValues, svgRef };
+    const { field, world } = this.props;
+    const col = HistogramBrush.getColumn(world, field);
+    const histogramCache = this.calcHistogramCache(col, field);
+    const { x, y, bins } = histogramCache;
+    this._histogram = { x, y, bins, svgRef };
  }

  handleColorAction() {
-    const { obsAnnotations, dispatch, field, world, ranges } = this.props;
+    const { dispatch, field, world, ranges } = this.props;

-    if (obsAnnotations.hasCol(field)) {
+    if (world.obsAnnotations.hasCol(field)) {
      dispatch({
        type: "color by continuous metadata",
        colorAccessor: field,
@@ -204,83 +315,113 @@ class HistogramBrush extends React.Component {
    };
  }

-  renderAxesBrushBins(x, y, bins, numValues, svgRef, field) {
+  renderAxesBrushBins(x, y, bins, svgRef, field) {
+    const svg = d3.select(svgRef);
+
    /* Remove everything */
-    d3.select(svgRef)
-      .selectAll("*")
-      .remove();
+    svg.selectAll("*").remove();

    /* BINS */
-    d3.select(svgRef)
+    svg
      .insert("g", "*")
      .attr("fill", "#bbb")
      .selectAll("rect")
      .data(bins)
      .enter()
      .append("rect")
-      .attr("class", "bar")
      .attr("x", d => x(d.x0) + 1)
-      .attr("y", d => y(d.length / numValues))
+      .attr("y", d => y(d.length))
      .attr("width", d => Math.abs(x(d.x1) - x(d.x0) - 1))
-      .attr("height", d => y(0) - y(d.length / numValues));
+      .attr("height", d => y(0) - y(d.length));

    /* BRUSH */
-    d3.select(svgRef)
+    const brushX = d3
+      .brushX()
+      .extent([[0, 0], [this.width - this.marginRight, this.height]])
+      /*
+      emit start so that the Undoable history can save an undo point
+      upon drag start, and ignore the subsequent intermediate drag events.
+      */
+      .on("start", this.onBrush(field, x.invert, "start").bind(this))
+      .on("brush", this.onBrush(field, x.invert, "brush").bind(this))
+      .on("end", this.onBrushEnd(field, x.invert).bind(this));
+    const brushXselection = d3
+      .select(svgRef)
      .append("g")
      .attr("class", "brush")
-      .attr("data-testid", `${svgRef.id}-brush`)
-      .call(
-        d3
-          .brushX()
-          .on("brush", this.onBrush(field, x.invert).bind(this))
-          .on("end", this.onBrush(field, x.invert).bind(this))
-      );
+      .attr("data-testid", `${svgRef.dataset.testid}-brush`)
+      .call(brushX);

-    /* AXIS */
-    d3.select(svgRef)
+    /* X AXIS */
+    svg
      .append("g")
      .attr("class", "axis axis--x")
      .attr("transform", `translate(0,${this.height - this.marginBottom})`)
      .call(d3.axisBottom(x).ticks(5));

-    d3.select(svgRef)
-      .selectAll(".axis--x text")
-      .style("fill", "rgb(80,80,80)");
+    /* Y AXIS */
+    svg
+      .append("g")
+      .attr("class", "axis axis--y")
+      .attr("transform", `translate(${this.width - this.marginRight},0)`)
+      .call(d3.axisRight(y).ticks(3));

-    d3.select(svgRef)
-      .selectAll(".axis--x path")
-      .style("stroke", "rgb(230,230,230)");
+    /* axis style */
+    svg.selectAll(".axis text").style("fill", "rgb(80,80,80)");
+    svg.selectAll(".axis path").style("stroke", "rgb(230,230,230)");
+    svg.selectAll(".axis line").style("stroke", "rgb(230,230,230)");

-    d3.select(svgRef)
-      .selectAll(".axis--x line")
-      .style("stroke", "rgb(230,230,230)");
+    const newState = { brushX, brushXselection };
+    this.setState(newState);
+    return newState;
  }

  render() {
    const {
      field,
+      world,
      colorAccessor,
      isUserDefined,
      isDiffExp,
      logFoldChange,
-      pval,
      pvalAdj,
      scatterplotXXaccessor,
      scatterplotYYaccessor,
      zebra
    } = this.props;
+    const fieldForId = field.replace(/\s/g, "_");
+    const {
+      min: unclippedRangeMin,
+      max: unclippedRangeMax
+    } = HistogramBrush.getColumn(world, field, false).summarize();
+    const unclippedRangeMinColor =
+      world.clipQuantiles.min === 0 ? "#bbb" : globals.blue;
+    const unclippedRangeMaxColor =
+      world.clipQuantiles.max === 1 ? "#bbb" : globals.blue;

    return (
      <div
-        id={`histogram_${field}`}
+        id={`histogram_${fieldForId}`}
        data-testid={`histogram-${field}`}
-        data-testclass={isDiffExp ? `histogram-diffexp` : ""}
+        data-testclass={
+          isDiffExp
+            ? "histogram-diffexp"
+            : isUserDefined
+            ? "histogram-user-gene"
+            : "histogram-continuous-metadata"
+        }
        style={{
          padding: globals.leftSidebarSectionPadding,
          backgroundColor: zebra ? globals.lightestGrey : "white"
        }}
      >
-        <div style={{ display: "flex", justifyContent: "flex-end" }}>
+        <div
+          style={{
+            display: "flex",
+            justifyContent: "flex-end",
+            paddingBottom: "8px"
+          }}
+        >
          {isDiffExp || isUserDefined ? (
            <span>
              <span
@@ -289,6 +430,7 @@ class HistogramBrush extends React.Component {
              />
              <ButtonGroup style={{ marginRight: 7 }}>
                <Button
+                  data-testid={`plot-x-${field}`}
                  onClick={this.handleSetGeneAsScatterplotX(field).bind(this)}
                  active={scatterplotXXaccessor === field}
                  intent={scatterplotXXaccessor === field ? "primary" : "none"}
@@ -296,6 +438,7 @@ class HistogramBrush extends React.Component {
                  plot x
                </Button>
                <Button
+                  data-testid={`plot-y-${field}`}
                  onClick={this.handleSetGeneAsScatterplotY(field).bind(this)}
                  active={scatterplotYYaccessor === field}
                  intent={scatterplotYYaccessor === field ? "primary" : "none"}
@@ -323,6 +466,8 @@ class HistogramBrush extends React.Component {
              onClick={this.handleColorAction.bind(this)}
              active={colorAccessor === field}
              intent={colorAccessor === field ? "primary" : "none"}
+              data-testclass="colorby"
+              data-testid={`colorby-${field}`}
              icon="tint"
            />
          </Tooltip>
@@ -330,7 +475,9 @@ class HistogramBrush extends React.Component {
        <svg
          width={this.width}
          height={this.height}
-          id={`histogram_${field}_svg`}
+          id={`histogram_${fieldForId}_svg`}
+          data-testclass="histogram-plot"
+          data-testid={`histogram-${field}-plot`}
          ref={svgRef => {
            this.drawHistogram(svgRef);
          }}
@@ -338,10 +485,21 @@ class HistogramBrush extends React.Component {
        <div
          style={{
            display: "flex",
-            justifyContent: "center"
+            justifyContent: "space-between"
          }}
        >
-          <span style={{ fontStyle: "italic" }}>{field}</span>
+          <span style={{ color: unclippedRangeMinColor }}>
+            min {unclippedRangeMin.toPrecision(4)}
+          </span>
+          <span
+            data-testclass="brushable-histogram-field-name"
+            style={{ fontStyle: "italic" }}
+          >
+            {field}
+          </span>
+          <span style={{ color: unclippedRangeMaxColor }}>
+            max {unclippedRangeMax.toPrecision(4)}
+          </span>
        </div>

        {isDiffExp ? (
@@ -6,12 +6,12 @@ import * as globals from "../../globals";
 import Category from "./category";

@connect(state => ({
-  categoricalSelectionState: state.controls.categoricalSelectionState
+  categoricalSelection: state.categoricalSelection
 }))
 class Categories extends React.Component {
  render() {
-    const { categoricalSelectionState } = this.props;
-    if (!categoricalSelectionState) return null;
+    const { categoricalSelection } = this.props;
+    if (!categoricalSelection) return null;

    return (
      <div
@@ -26,7 +26,7 @@ class Categories extends React.Component {
        >
          Categorical Metadata
        </p>
-        {_.map(categoricalSelectionState, (catState, catName) => (
+        {_.map(categoricalSelection, (catState, catName) => (
          <Category key={catName} metadataField={catName} />
        ))}
      </div>
@@ -9,8 +9,8 @@ import Value from "./value";
 import sortedCategoryValues from "./util";

@connect(state => ({
-  colorAccessor: state.controls.colorAccessor,
-  categoricalSelectionState: state.controls.categoricalSelectionState
+  colorAccessor: state.colors.colorAccessor,
+  categoricalSelection: state.categoricalSelection
 }))
 class Category extends React.Component {
  constructor(props) {
@@ -21,38 +21,43 @@ class Category extends React.Component {
    };
  }

-  componentDidUpdate() {
-    const { categoricalSelectionState, metadataField } = this.props;
-    const cat = categoricalSelectionState[metadataField];
-    const categoryCount = {
-      // total number of categories in this dimension
-      totalCatCount: cat.numCategories,
-      // number of selected options in this category
-      selectedCatCount: _.reduce(
-        cat.categorySelected,
-        (res, cond) => (cond ? res + 1 : res),
-        0
-      )
-    };
-    if (categoryCount.selectedCatCount === categoryCount.totalCatCount) {
-      /* everything is on, so not indeterminate */
-      this.checkbox.indeterminate = false;
-    } else if (categoryCount.selectedCatCount === 0) {
-      /* nothing is on, so no */
-      this.checkbox.indeterminate = false;
-    } else if (categoryCount.selectedCatCount < categoryCount.totalCatCount) {
-      /* to be explicit... */
-      this.checkbox.indeterminate = true;
+  componentDidUpdate(prevProps) {
+    const { categoricalSelection, metadataField } = this.props;
+    if (categoricalSelection !== prevProps.categoricalSelection) {
+      const cat = categoricalSelection[metadataField];
+      const categoryCount = {
+        // total number of categories in this dimension
+        totalCatCount: cat.numCategoryValues,
+        // number of selected options in this category
+        selectedCatCount: _.reduce(
+          cat.categoryValueSelected,
+          (res, cond) => (cond ? res + 1 : res),
+          0
+        )
+      };
+      if (categoryCount.selectedCatCount === categoryCount.totalCatCount) {
+        /* everything is on, so not indeterminate */
+        this.checkbox.indeterminate = false;
+        this.setState({ isChecked: true }); // eslint-disable-line react/no-did-update-set-state
+      } else if (categoryCount.selectedCatCount === 0) {
+        /* nothing is on, so no */
+        this.checkbox.indeterminate = false;
+        this.setState({ isChecked: false }); // eslint-disable-line react/no-did-update-set-state
+      } else if (categoryCount.selectedCatCount < categoryCount.totalCatCount) {
+        /* to be explicit... */
+        this.checkbox.indeterminate = true;
+        this.setState({ isChecked: false });
+      }
    }
  }

-  handleColorChange() {
+  handleColorChange = () => {
    const { dispatch, metadataField } = this.props;
    dispatch({
      type: "color by categorical metadata",
      colorAccessor: metadataField
    });
-  }
+  };

  toggleAll() {
    const { dispatch, metadataField } = this.props;
@@ -83,10 +88,10 @@ class Category extends React.Component {
  }

  renderCategoryItems() {
-    const { categoricalSelectionState, metadataField } = this.props;
+    const { categoricalSelection, metadataField } = this.props;

-    const cat = categoricalSelectionState[metadataField];
-    const optTuples = sortedCategoryValues([...cat.categoryIndices]);
+    const cat = categoricalSelection[metadataField];
+    const optTuples = sortedCategoryValues([...cat.categoryValueIndices]);
    return _.map(optTuples, (tuple, i) => (
      <Value
        optTuples={optTuples}
@@ -100,17 +105,15 @@ class Category extends React.Component {

  render() {
    const { isExpanded, isChecked } = this.state;
-    const {
-      metadataField,
-      colorAccessor,
-      categoricalSelectionState
-    } = this.props;
-    const { isTruncated } = categoricalSelectionState[metadataField];
+    const { metadataField, colorAccessor, categoricalSelection } = this.props;
+    const { isTruncated } = categoricalSelection[metadataField];
    return (
      <div
        style={{
          maxWidth: globals.maxControlsWidth
        }}
+        data-testclass="category"
+        data-testid={`category-${metadataField}`}
      >
        <div
          style={{
@@ -128,6 +131,8 @@ class Category extends React.Component {
          >
            <label className="bp3-control bp3-checkbox">
              <input
+                data-testclass="category-select"
+                data-testid={`category-select-${metadataField}`}
                onChange={this.handleToggleAllClick.bind(this)}
                ref={el => {
                  this.checkbox = el;
@@ -141,6 +146,7 @@ class Category extends React.Component {
            </label>

            <span
+              data-testid={`category-expand-${metadataField}`}
              style={{
                cursor: "pointer",
                display: "inline-block"
@@ -151,18 +157,26 @@ class Category extends React.Component {
            >
              {metadataField}
              {isExpanded ? (
-                <FaChevronDown style={{ fontSize: 10, marginLeft: 5 }} />
+                <FaChevronDown
+                  data-testclass="category-expand-is-expanded"
+                  style={{ fontSize: 10, marginLeft: 5 }}
+                />
              ) : (
-                <FaChevronRight style={{ fontSize: 10, marginLeft: 5 }} />
+                <FaChevronRight
+                  data-testclass="category-expand-is-not-expanded"
+                  style={{ fontSize: 10, marginLeft: 5 }}
+                />
              )}
            </span>
          </div>
          <Tooltip content="Use as color scale" position="bottom">
            <Button
-              onClick={this.handleColorChange.bind(this)}
+              data-testclass="colorby"
+              data-testid={`colorby-${metadataField}`}
+              onClick={this.handleColorChange}
              active={colorAccessor === metadataField}
              intent={colorAccessor === metadataField ? "primary" : "none"}
-              icon={"tint"}
+              icon="tint"
            />
          </Tooltip>
        </div>
@@ -1,25 +1,16 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import * as d3 from "d3";

@connect()
 class Occupancy extends React.Component {
  render() {
-    const {
-      occupancy,
-      colorScale,
-      categoricalSelectionState,
-      colorAccessor,
-      schema
-    } = this.props;
+    const { occupancy, colorScale, colorAccessor, schema, world } = this.props;
    const width = 100;
    const height = 11;

-    const categories = _.filter(schema.annotations.obs, {
-      name: colorAccessor
-    })[0].categories;
+    const categories = schema.annotations.obsByName[colorAccessor]?.categories;

    const x = d3
      .scaleLinear()
@@ -28,24 +19,23 @@ class Occupancy extends React.Component {
      .range([0, width]);

    let currentOffset = 0;
+    const dfColumn = world.obsAnnotations.col(colorAccessor);
+    const categoryValues = dfColumn.summarize().categories;
+    const stacks = categoryValues.map(d => {
+      const o = occupancy.get(d);

-    const stacks = categoricalSelectionState[colorAccessor].categoryValues.map(
-      d => {
-        const o = occupancy.get(d);
+      const scaledValue = x(o);

-        const scaledValue = x(o);
-
-        const stackItem = {
-          key: d,
-          value: o || 0,
-          rectWidth: o ? scaledValue : 0,
-          offset: currentOffset,
-          fill: o ? colorScale(categories.indexOf(d)) : "rgb(255,255,255)"
-        };
-        currentOffset += o ? scaledValue : 0;
-        return stackItem;
-      }
-    );
+      const stackItem = {
+        key: d,
+        value: o || 0,
+        rectWidth: o ? scaledValue : 0,
+        offset: currentOffset,
+        fill: o ? colorScale(categories.indexOf(d)) : "rgb(255,255,255)"
+      };
+      currentOffset += o ? scaledValue : 0;
+      return stackItem;
+    });

    return (
      <svg
@@ -5,7 +5,6 @@
 // return sorted index

 import isNumber from "is-number";
-import _ from "lodash";

 const sortedCategoryValues = values => {
  /* this sort could be memoized for perf */
@@ -13,7 +12,7 @@ const sortedCategoryValues = values => {
  const strings = [];
  const ints = [];

-  _.forEach(values, v => {
+  values.forEach(v => {
    if (isNumber(v[0])) {
      ints.push(v);
    } else {
@@ -1,17 +1,16 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
-import _ from "lodash";
 import Occupancy from "./occupancy";
 import { countCategoryValues2D } from "../../util/stateManager/worldUtil";
 import * as globals from "../../globals";

@connect(state => ({
-  categoricalSelectionState: state.controls.categoricalSelectionState,
-  colorScale: state.controls.colorScale,
-  colorAccessor: state.controls.colorAccessor,
-  schema: _.get(state.controls.world, "schema", null),
-  world: state.controls.world
+  categoricalSelection: state.categoricalSelection,
+  colorScale: state.colors.scale,
+  colorAccessor: state.colors.colorAccessor,
+  schema: state.world?.schema,
+  world: state.world
 }))
 class CategoryValue extends React.Component {
  toggleOff() {
@@ -34,7 +33,7 @@ class CategoryValue extends React.Component {

  render() {
    const {
-      categoricalSelectionState,
+      categoricalSelection,
      metadataField,
      categoryIndex,
      colorAccessor,
@@ -44,11 +43,11 @@ class CategoryValue extends React.Component {
      world
    } = this.props;

-    if (!categoricalSelectionState) return null;
+    if (!categoricalSelection) return null;

-    const category = categoricalSelectionState[metadataField];
-    const selected = category.categorySelected[categoryIndex];
-    const count = category.categoryCounts[categoryIndex];
+    const category = categoricalSelection[metadataField];
+    const selected = category.categoryValueSelected[categoryIndex];
+    const count = category.categoryValueCounts[categoryIndex];
    const value = category.categoryValues[categoryIndex];
    const displayString = String(
      category.categoryValues[categoryIndex]
@@ -60,16 +59,10 @@ class CategoryValue extends React.Component {
    let occupancy = null;

    if (isColorBy && schema) {
-      categories = _.filter(schema.annotations.obs, {
-        name: colorAccessor
-      })[0].categories;
+      categories = schema.annotations.obsByName[colorAccessor]?.categories;
    }

-    if (
-      colorAccessor &&
-      !isColorBy &&
-      categoricalSelectionState[colorAccessor]
-    ) {
+    if (colorAccessor && !isColorBy && categoricalSelection[colorAccessor]) {
      occupancy = countCategoryValues2D(
        metadataField,
        colorAccessor,
@@ -85,6 +78,7 @@ class CategoryValue extends React.Component {
          alignItems: "baseline",
          justifyContent: "space-between"
        }}
+        data-testclass="categorical-row"
      >
        <div
          style={{
@@ -101,16 +95,23 @@ class CategoryValue extends React.Component {
              onChange={
                selected ? this.toggleOff.bind(this) : this.toggleOn.bind(this)
              }
+              data-testclass="categorical-value-select"
+              data-testid={`categorical-value-select-${metadataField}-${displayString}`}
              checked={selected}
              type="checkbox"
            />
            <span className="bp3-control-indicator" />
-            {displayString}
+            <span
+              data-testid={`categorical-value-${metadataField}-${displayString}`}
+              data-testclass="categorical-value"
+            >
+              {displayString}
+            </span>
          </label>
          <span style={{ flexShrink: 0 }}>
            {colorAccessor &&
            !isColorBy &&
-            categoricalSelectionState[colorAccessor] ? (
+            categoricalSelection[colorAccessor] ? (
              <Occupancy
                occupancy={occupancy.get(
                  category.categoryValues[categoryIndex]
@@ -121,7 +122,12 @@ class CategoryValue extends React.Component {
          </span>
        </div>
        <span>
-          <span>{count}</span>
+          <span
+            data-testclass="categorical-value-count"
+            data-testid={`categorical-value-count-${metadataField}-${displayString}`}
+          >
+            {count}
+          </span>
          <svg
            style={{
              marginLeft: 5,
@@ -9,11 +9,10 @@ import * as globals from "../../globals";
 import HistogramBrush from "../brushableHistogram";

@connect(state => ({
-  obsAnnotations: _.get(state.controls.world, "obsAnnotations", null),
-  colorAccessor: state.controls.colorAccessor,
-  colorScale: state.controls.colorScale,
-  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null),
-  schema: _.get(state.controls.world, "schema", null)
+  obsAnnotations: state.world?.obsAnnotations,
+  colorAccessor: state.colors.colorAccessor,
+  colorScale: state.colors.scale,
+  schema: state.world?.schema
 }))
 class Continuous extends React.Component {
  constructor(props) {
@@ -65,18 +64,16 @@ class Continuous extends React.Component {
        ) : null}
        {obsAnnotations
          ? _.map(obsAnnotations.colIndex.keys(), key => {
-              const summary = obsAnnotations.col(key).summarize();
              const isColorField =
                key.includes("color") || key.includes("Color");
+              if (key === schema.annotations.obs.index || isColorField)
+                return null;
+
+              const summary = obsAnnotations.col(key).summarize();
              const nonFiniteExtent =
                summary.min === undefined || summary.max === undefined;
-              zebra += 1;
-              if (
-                !summary.categorical &&
-                key !== "name" &&
-                !isColorField &&
-                !nonFiniteExtent
-              ) {
+              if (!summary.categorical && !nonFiniteExtent) {
+                zebra += 1;
                return (
                  <HistogramBrush
                    key={key}
@@ -98,8 +98,8 @@ const continuous = (selectorId, colorscale, colorAccessor) => {
 };

@connect(state => ({
-  colorAccessor: state.controls.colorAccessor,
-  colorScale: state.controls.colorScale,
+  colorAccessor: state.colors.colorAccessor,
+  colorScale: state.colors.scale,
  responsive: state.responsive
 }))
 class ContinuousLegend extends React.Component {
@@ -112,6 +112,7 @@ class ContinuousLegend extends React.Component {
    const { colorAccessor, responsive, colorScale } = this.props;
    if (
      prevProps.colorAccessor !== colorAccessor ||
+      prevProps.colorScale !== colorScale ||
      prevProps.responsive.height !== responsive.height ||
      prevProps.responsive.width !== responsive.width
    ) {
@@ -0,0 +1,18 @@
+import React from "react";
+import * as globals from "../../globals";
+
+const Logo = props => {
+  const { size } = props;
+  return (
+    <svg width={size} height={size} viewBox={`0 0 48 48`} fill="none">
+      <rect width="48" height="48" fill="white" />
+      <rect width="48" height="48" fill={globals.logoColor} />
+      <rect x="19" y="19" width="22" height="22" fill="white" />
+      <rect x="24" y="24" width="12" height="12" fill={globals.logoColor} />
+      <rect x="7" y="19" width="7" height="22" fill="white" />
+      <rect x="19" y="7" width="22" height="7" fill="white" />
+    </svg>
+  );
+};
+
+export default Logo;
@@ -32,6 +32,9 @@ class CellSetButton extends React.Component {
  render() {
    const { differential, eitherCellSetOneOrTwo } = this.props;
    const cellListName = `celllist${eitherCellSetOneOrTwo}`;
+    let cells_selected = differential[cellListName]
+      ? differential[cellListName].length
+      : 0;
    return (
      <Tooltip
        content="Save current selection for differential expression computation"
@@ -46,9 +49,10 @@ class CellSetButton extends React.Component {
        >
          {eitherCellSetOneOrTwo}
          {": "}
-          {differential[cellListName]
-            ? `${differential[cellListName].length} cells`
-            : "0 cells"}
+          <span data-testid={`cellset-count-${eitherCellSetOneOrTwo}`}>
+            {cells_selected}
+          </span>
+          {" cells"}
        </AnchorButton>
      </Tooltip>
    );
@@ -9,9 +9,8 @@ import CellSetButton from "./cellSetButtons";

@connect(state => ({
  differential: state.differential,
-  world: state.controls.world,
-  crossfilter: state.controls.crossfilter,
-  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+  world: state.world,
+  crossfilter: state.crossfilter
 }))
 class Expression extends React.Component {
  constructor(props) {
@@ -22,7 +22,6 @@ import {
  keepAroundErrorToast
 } from "../framework/toasters";
 import ExpressionButtons from "./expressionButtons";
-import finiteExtent from "../../util/finiteExtent";

 const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
  if (!modifiers.matchesPredicate) {
@@ -35,6 +34,7 @@ const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
    <MenuItem
      active={modifiers.active}
      disabled={modifiers.disabled}
+      data-testid={`suggest-menu-item-${geneName}`}
      // Use of annotations in this way is incorrect and dataset specific.
      // See https://github.com/chanzuckerberg/cellxgene/issues/483
      // label={gene.n_counts}
@@ -57,11 +57,11 @@ const filterGenes = (query, genes) =>

@connect(state => {
  return {
-    obsAnnotations: _.get(state.controls.world, "obsAnnotations", null),
+    obsAnnotations: state.world?.obsAnnotations,
    userDefinedGenes: state.controls.userDefinedGenes,
    userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
-    world: state.controls.world,
-    colorAccessor: state.controls.colorAccessor,
+    world: state.world,
+    colorAccessor: state.colors.colorAccessor,
    differential: state.differential
  };
 })
@@ -85,7 +85,8 @@ class GeneExpression extends React.Component {
    */
    const { world } = this.props;
    const { varAnnotations } = world;
-    const geneNames = varAnnotations.col("name").asArray();
+    const varIndexName = world.schema.annotations.var.index;
+    const geneNames = varAnnotations.col(varIndexName).asArray();
    if (geneNames.length > 0) {
      const placeholder = [];
      let len = geneNames.length;
@@ -107,6 +108,7 @@ class GeneExpression extends React.Component {

  handleClick(g) {
    const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
    const gene = g.target;
    if (userDefinedGenes.indexOf(gene) !== -1) {
      postUserErrorToast("That gene already exists");
@@ -114,19 +116,22 @@ class GeneExpression extends React.Component {
      postUserErrorToast(
        "That's too many genes, you can have at most 15 user defined genes"
      );
-    } else if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
+    } else if (
+      world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+    ) {
      postUserErrorToast("That doesn't appear to be a valid gene name.");
    } else {
-      dispatch(actions.requestUserDefinedGene(gene));
-      dispatch({
-        type: "user defined gene",
-        data: gene
-      });
+      dispatch({ type: "single user defined gene start" });
+      dispatch(actions.requestUserDefinedGene(gene)).then(
+        () => dispatch({ type: "single user defined gene complete" }),
+        () => dispatch({ type: "single user defined gene error" })
+      );
    }
  }

  handleBulkAddClick() {
    const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
    const { bulkAdd } = this.state;

    /*
@@ -136,25 +141,28 @@ class GeneExpression extends React.Component {
    if (bulkAdd !== "") {
      const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");

-      genes.forEach(gene => {
-        if (gene.length === 0) {
-          keepAroundErrorToast("Must enter a gene name.");
-        } else if (userDefinedGenes.indexOf(gene) !== -1) {
-          keepAroundErrorToast("That gene already exists");
-        } else if (
-          world.varAnnotations.col("name").indexOf(gene) === undefined
-        ) {
-          keepAroundErrorToast(
-            `${gene} doesn't appear to be a valid gene name.`
-          );
-        } else {
-          dispatch(actions.requestUserDefinedGene(gene));
-          dispatch({
-            type: "user defined gene",
-            data: gene
-          });
-        }
-      });
+      dispatch({ type: "bulk user defined gene start" });
+      Promise.all(
+        genes.map(gene => {
+          if (gene.length === 0) {
+            return keepAroundErrorToast("Must enter a gene name.");
+          }
+          if (userDefinedGenes.indexOf(gene) !== -1) {
+            return keepAroundErrorToast("That gene already exists");
+          }
+          if (
+            world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+          ) {
+            return keepAroundErrorToast(
+              `${gene} doesn't appear to be a valid gene name.`
+            );
+          }
+          return dispatch(actions.requestUserDefinedGene(gene));
+        })
+      ).then(
+        () => dispatch({ type: "bulk user defined gene complete" }),
+        () => dispatch({ type: "bulk user defined gene error" })
+      );
    }

    this.setState({ bulkAdd: "" });
@@ -167,7 +175,7 @@ class GeneExpression extends React.Component {
      userDefinedGenesLoading,
      differential
    } = this.props;
-
+    const varIndexName = world?.schema?.annotations?.var?.index;
    const { tab, bulkAdd } = this.state;

    return (
@@ -195,6 +203,7 @@ class GeneExpression extends React.Component {
              style={{ marginRight: 5 }}
              minimal
              small
+              data-testid="tab-autosuggest"
              onClick={() => {
                this.setState({ tab: "autosuggest" });
              }}
@@ -205,6 +214,7 @@ class GeneExpression extends React.Component {
              active={tab === "bulkadd"}
              minimal
              small
+              data-testid="section-bulk-add"
              onClick={() => {
                this.setState({ tab: "bulkadd" });
              }}
@@ -221,9 +231,9 @@ class GeneExpression extends React.Component {
              }}
            >
              <Suggest
-                closeOnSelect
-                openOnKeyDown
                resetOnSelect
+                closeOnSelect
+                resetOnClose
                itemDisabled={
                  userDefinedGenesLoading ? () => true : () => false
                }
@@ -240,7 +250,7 @@ class GeneExpression extends React.Component {
                itemRenderer={renderGene.bind(this)}
                items={
                  world && world.varAnnotations
-                    ? world.varAnnotations.col("name").asArray()
+                    ? world.varAnnotations.col(varIndexName).asArray()
                    : ["No genes"]
                }
                popoverProps={{ minimal: true }}
@@ -272,6 +282,7 @@ class GeneExpression extends React.Component {
                        this.setState({ bulkAdd: e.target.value });
                      }}
                      id="text-input-bulk-add"
+                      data-testid="input-bulk-add"
                      placeholder={this.placeholderGeneNames()}
                      value={bulkAdd}
                    />
@@ -318,7 +329,7 @@ class GeneExpression extends React.Component {
          <ExpressionButtons />
          {differential.diffExp
            ? _.map(differential.diffExp, (value, index) => {
-                const name = world.varAnnotations.at(value[0], "name");
+                const name = world.varAnnotations.at(value[0], varIndexName);
                const values = world.varData.col(name);
                if (!values) {
                  return null;
@@ -1,5 +1,6 @@
 // jshint esversion: 6
 const mat4 = require("gl-mat4");
+const vec3 = require("gl-vec3");

 // opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js

@@ -38,7 +39,20 @@ export default function(regl) {
    uniforms: {
      distance: regl.prop("distance"),
      view: regl.prop("view"),
-      projection: ({viewportWidth, viewportHeight}) => mat4.perspective([], Math.PI / 2, viewportWidth / viewportHeight, 0.01, 1000)
+      projection: ({ viewportWidth, viewportHeight }) => {
+        const aspectRatio = viewportWidth / viewportHeight;
+        let m = mat4.perspective(
+          [],
+          Math.PI / 2,
+          viewportWidth / viewportHeight,
+          0.01,
+          1000
+        );
+        if (aspectRatio < 1) {
+          m = mat4.scale(m, m, vec3.fromValues(1, 1, 1 / aspectRatio));
+        }
+        return m;
+      }
    },

    count: regl.prop("count"),
@@ -3,7 +3,7 @@
 import * as d3 from "d3";

 const Lasso = () => {
-  const dispatch = d3.dispatch("start", "end");
+  const dispatch = d3.dispatch("start", "end", "cancel");

  const polygonToPath = polygon =>
    `M${polygon.map(d => d.join(",")).join("L")}`;
@@ -28,6 +28,7 @@ const Lasso = () => {

      lassoPath = g
        .append("path")
+        .attr("data-testid", "lasso-element")
        .attr("fill", "#0bb")
        .attr("fill-opacity", 0.1)
        .attr("stroke", "#0bb")
@@ -81,6 +82,7 @@ const Lasso = () => {
        lassoPath.remove();
        lassoPath = null;
        lassoPolygon = null;
+        dispatch.call("cancel");
      }
    };

@@ -114,6 +116,23 @@ const Lasso = () => {
        closePath = null;
      }
    };
+
+    lasso.move = polygon => {
+      if (polygon !== lassoPolygon || polygon.length !== lassoPolygon.length) {
+        lasso.reset();
+
+        lassoPolygon = polygon;
+        lassoPath = g
+          .append("path")
+          .attr("data-testid", "lasso-element")
+          .attr("fill", "#0bb")
+          .attr("fill-opacity", 0.1)
+          .attr("stroke", "#0bb")
+          .attr("stroke-dasharray", "3, 3");
+
+        lassoPath.attr("d", `${polygonToPath(lassoPolygon)}Z`);
+      }
+    };
  };

  lasso.on = (type, callback) => {
@@ -10,12 +10,13 @@ import Lasso from "./setupLasso";
 ******************************************/

 export default (
-  handleBrushSelectAction,
-  handleBrushDeselectAction,
+  selectionToolType,
+  handleStartAction,
+  handleDragAction,
+  handleEndAction,
+  handleCancelAction,
  responsive,
-  graphPaddingRight,
-  handleLassoStart,
-  handleLassoEnd
+  graphPaddingRight
 ) => {
  const svg = d3
    .select("#graphAttachPoint")
@@ -25,27 +26,37 @@ export default (
    .attr("height", responsive.height)
    .attr("class", `${styles.graphSVG}`);

-  const brush = d3
-    .brush()
-    .extent([[0, 0], [responsive.width - graphPaddingRight, responsive.height]])
-    .on("brush", handleBrushSelectAction)
-    .on("end", handleBrushDeselectAction);
+  if (selectionToolType === "brush") {
+    const brush = d3
+      .brush()
+      .extent([
+        [0, 0],
+        [responsive.width - graphPaddingRight, responsive.height]
+      ])
+      .on("start", handleStartAction)
+      .on("brush", handleDragAction)
+      // FYI, brush doesn't generate cancel
+      .on("end", handleEndAction);

-  const brushContainer = svg
-    .append("g")
-    .attr("class", "graph_brush")
-    .call(brush);
+    const brushContainer = svg
+      .append("g")
+      .attr("class", "graph_brush")
+      .call(brush);

-  const lassoInstance = Lasso()
-    .on("end", handleLassoEnd)
-    .on("start", handleLassoStart);
+    return { svg, container: brushContainer, tool: brush };
+  }

-  const lasso = svg.call(lassoInstance);
+  if (selectionToolType === "lasso") {
+    const lasso = Lasso()
+      .on("end", handleEndAction)
+      // FYI, Lasso doesn't generate drag
+      .on("start", handleStartAction)
+      .on("cancel", handleCancelAction);

-  return {
-    svg,
-    brushContainer,
-    brush,
-    lasso
-  };
+    const lassoContainer = svg.call(lasso);
+
+    return { svg, container: lassoContainer, tool: lasso };
+  }
+
+  throw new Error("unknown graph selection tool");
 };
@@ -1,26 +0,0 @@
-// jshint esversion: 6
-// createExpressionsCountsMap () {
-//
-//   const CHANGE_ME_MAGIC_GENE_INDEX = 5;
-//
-//   const expressionsCountsMap = {};
-//
-//   /* currently selected gene */
-//   expressionsCountsMap.geneName = this.state.expressions.data.genes[3];
-//
-//   let maxExpressionValue = 0;
-//
-//   /* create map of expressions for every cell */
-//   this.state.expressions.data.cells.map((c) => {
-//     /* cellname = 234 */
-//     expressionsCountsMap[c.cellname] = c["e"][CHANGE_ME_MAGIC_GENE_INDEX];
-//     /* collect the maximum value as we iterate */
-//     if (c["e"][CHANGE_ME_MAGIC_GENE_INDEX] > maxExpressionValue) {
-//       maxExpressionValue = c["e"][CHANGE_ME_MAGIC_GENE_INDEX]
-//     }
-//   })
-//
-//   expressionsCountsMap.maxValue = maxExpressionValue;
-//
-//   return expressionsCountsMap;
-// }
@@ -1,5 +1,4 @@
 // jshint esversion: 6
-import _ from "lodash";
 import React from "react";
 import { connect } from "react-redux";
 import Categorical from "./categorical/categorical";
@@ -7,10 +6,11 @@ import Continuous from "./continuous/continuous";
 import GeneExpression from "./geneExpression";
 import * as globals from "../globals";
 import DynamicScatterplot from "./scatterplot/scatterplot";
+import Logo from "./framework/logo.js";

@connect(state => ({
  responsive: state.responsive,
-  datasetTitle: _.get(state.config, "displayNames.dataset"),
+  datasetTitle: state.config?.displayNames?.dataset,
  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
  scatterplotYYaccessor: state.controls.scatterplotYYaccessor
 }))
@@ -28,7 +28,6 @@ class LeftSideBar extends React.Component {
    if cellxgene logo or tabs change, this must as well
    */
    const metadataSectionPadding = 0;
-    // scatterplotXXaccessor && scatterplotYYaccessor ? 450 : 0;

    return (
      <div
@@ -40,20 +39,49 @@ class LeftSideBar extends React.Component {
        }}
      >
        <p
-          data-testid="header"
          style={{
            position: "fixed",
            top: globals.cellxgeneTitleTopPadding,
            left: globals.leftSidebarWidth + globals.cellxgeneTitleLeftPadding,
-            margin: 0,
-            fontSize: globals.largestFontSize,
-            color: globals.darkerGrey,
-            width: "100%"
+            margin: 0
          }}
        >
-          cellxgene: {datasetTitle}
+          <Logo size={32} />
+          <span
+            style={{
+              fontSize: 28,
+              position: "relative",
+              top: -4,
+              fontWeight: "bold",
+              marginLeft: 5,
+              color: globals.logoColor,
+              userSelect: "none"
+            }}
+          >
+            cell<span
+              style={{
+                position: "relative",
+                top: 1,
+                fontWeight: 300,
+                fontSize: 24
+              }}
+            >
+              ×
+            </span>gene
+          </span>
+          <span
+            data-testid="header"
+            style={{
+              fontSize: 16,
+              display: "block",
+              position: "relative",
+              marginTop: 10,
+              top: -4
+            }}
+          >
+            {datasetTitle}
+          </span>
        </p>
-
        <div
          style={{
            height: responsive.height - metadataSectionPadding,
@@ -22,12 +22,8 @@ import { margin, width, height } from "./util";
 import finiteExtent from "../../util/finiteExtent";

@connect(state => {
-  const {
-    world,
-    crossfilter,
-    scatterplotXXaccessor,
-    scatterplotYYaccessor
-  } = state.controls;
+  const { world, crossfilter } = state;
+  const { scatterplotXXaccessor, scatterplotYYaccessor } = state.controls;
  const expressionX =
    world &&
    scatterplotXXaccessor &&
@@ -44,9 +40,9 @@ import finiteExtent from "../../util/finiteExtent";
  return {
    world,

-    colorRGB: state.controls.colorRGB,
-    colorAccessor: state.controls.colorAccessor,
-    colorScale: state.controls.colorScale,
+    colorRGB: state.colors.rgb,
+    colorScale: state.colors.scale,
+    colorAccessor: state.colors.colorAccessor,

    // Accessors are var/gene names (strings)
    scatterplotXXaccessor,
@@ -58,9 +54,7 @@ import finiteExtent from "../../util/finiteExtent";
    expressionX,
    expressionY,

-    crossfilter,
-    // updated whenever the crossfilter selection is updated
-    selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+    crossfilter
  };
 })
 class Scatterplot extends React.Component {
@@ -136,8 +130,7 @@ class Scatterplot extends React.Component {
      scatterplotYYaccessor,
      expressionX,
      expressionY,
-      colorRGB,
-      selectionUpdate
+      colorRGB
    } = this.props;
    const {
      reglRender,
@@ -152,7 +145,8 @@ class Scatterplot extends React.Component {

    if (
      scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
-      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor // was CLU now FTH1 etc
+      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor || // was CLU now FTH1 etc
+      world !== prevProps.world // shape or clip of world changed
    ) {
      const scales = Scatterplot.setupScales(expressionX, expressionY);
      this.drawAxesSVG(scales.xScale, scales.yScale, svg);
@@ -210,11 +204,11 @@ class Scatterplot extends React.Component {

      // Sizes for each point - updates are triggered only when selected
      // obs change
-      if (!renderCache.sizes || selectionUpdate !== prevProps.selctionUpdate) {
+      if (!renderCache.sizes || crossfilter !== prevProps.crossfilter) {
        if (!renderCache.sizes) {
          renderCache.sizes = new Float32Array(cellCount);
        }
-        crossfilter.fillByIsFiltered(renderCache.sizes, 4, 0.2);
+        crossfilter.fillByIsSelected(renderCache.sizes, 4, 0.2);
        sizeBuffer({ data: renderCache.sizes, dimension: 1 });
      }

@@ -270,9 +264,15 @@ class Scatterplot extends React.Component {

    // the axes are much cleaner and easier now. No need to rotate and orient
    // the axis, just call axisBottom, axisLeft etc.
-    const xAxis = d3.axisBottom().scale(xScale);
+    const xAxis = d3
+      .axisBottom()
+      .ticks(7)
+      .scale(xScale);

-    const yAxis = d3.axisLeft().scale(yScale);
+    const yAxis = d3
+      .axisLeft()
+      .ticks(7)
+      .scale(yScale);

    // adding axes is also simpler now, just translate x-axis to (0,height)
    // and it's alread defined to be a bottom axis.
@@ -345,14 +345,12 @@ class Scatterplot extends React.Component {
          <Button
            type="button"
            minimal
-            onClick={() => {
+            data-testid="clear-scatterplot"
+            onClick={() =>
              dispatch({
                type: "clear scatterplot"
-              });
-              dispatch({
-                type: "reset colorscale"
-              });
-            }}
+              })
+            }
          >
            remove
          </Button>
@@ -368,6 +366,7 @@ class Scatterplot extends React.Component {
          <canvas
            width={width}
            height={height}
+            data-testid="scatterplot"
            style={{
              marginLeft: margin.left - 7,
              marginTop: margin.top
@@ -14,6 +14,7 @@ const setupScatterplot = (width, height, margin) => {
    .append("svg")
    .attr("width", width + margin.left + margin.right)
    .attr("height", height + margin.top + margin.bottom)
+    .attr("data-testid", "scatterplot-svg")
    .append("g")
    .attr("transform", `translate(${margin.left},${margin.top})`);

@@ -1,35 +1,5 @@
 import { Colors } from "@blueprintjs/core";

-// jshint esversion: 6
-/* these will be either (preferably) specified or inferred */
-export const categories = [
-  "Sample.type",
-  "Selection",
-  "Location",
-  "Sample.name",
-  "Class",
-  "Neoplastic"
-];
-export const continuous = [
-  "Total_reads",
-  "Unique_reads",
-  "Unique_reads_percent",
-  "ERCC_reads",
-  "Non_ERCC_reads",
-  "ERCC_to_non_ERCC",
-  "Genes_detected",
-  "Multimapping_reads_percent",
-  "Splice_sites_AT.AC",
-  "Splice_sites_Annotated",
-  "Splice_sites_GC.AG",
-  "Splice_sites_GT.AG",
-  "Splice_sites_non_canonical",
-  "Splice_sites_total",
-  "Unmapped_mismatch",
-  "Unmapped_other",
-  "Unmapped_short"
-];
-
 /* if a categorical metadata field has more options than this, truncate */
 export const maxCategoricalOptionsToDisplay = 100;

@@ -61,6 +31,7 @@ export const darkGreen = "#448C4D";

 export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";
+export const logoColor = "black"; /* logo pink: "#E9429A" */

 /* typography constants */

@@ -1,150 +0,0 @@
-// jshint esversion: 6
-import _ from "lodash";
-import * as d3 from "d3";
-import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
-import * as globals from "../globals";
-import parseRGB from "../util/parseRGB";
-import finiteExtent from "../util/finiteExtent";
-
-/*
-  https://medium.com/@jacobp100/you-arent-using-redux-middleware-enough-94ffe991e6
-  storeInstance =>
-    functionToCallWithAnActionThatWillSendItToTheNextMiddleware =>
-    actionThatDispatchWasCalledWith =>
-    valueToUseAsTheReturnValueOfTheDispatchCall
-*/
-
-/*
-  What this file does:
-
-  1. fire a filter action anywhere in the app
-  2. ** this middleware checks to see the state of all the currently selected filters,
-     including the new one
-  3. ** create updated selection from a copy of all the cells presently on the client
-     (this may be a subset of 'all')
-  4. ** append that new selection to the action so that it magically appears in the reducer
-     just because the action was fired
-
-  This is nice because we keep a lot of filtering business logic centralized
-  (what it means in practice to be selected)
-*/
-
-const updateCellColorsMiddleware = store => next => action => {
-  const s = store.getState();
-
-  /*
-  this is a hardcoded map of the things we need to keep an eye on and update
-  global cell selection in response to
-  */
-  const filterJustChanged =
-    action.type === "color by expression" ||
-    action.type === "color by continuous metadata" ||
-    action.type === "color by categorical metadata";
-
-  const obsAnnotations = _.get(s.controls, "world.obsAnnotations", null);
-  if (!filterJustChanged || !obsAnnotations) {
-    return next(
-      action
-    ); /* if the cells haven't loaded or the action wasn't a color change, bail */
-  }
-
-  let colorScale;
-  const colorsByRGB = new Array(obsAnnotations.length);
-
-  /*
-  in plain language...
-   (a) once the cells have loaded.
-   (b) each time a user changes a color control we need to update cellsMetadata colors
-  This is available to all the draw functions as controls.colorRGB[index]
-  */
-
-  if (action.type === "color by categorical metadata") {
-    const { categories } = _.filter(s.controls.world.schema.annotations.obs, {
-      name: action.colorAccessor
-    })[0];
-
-    colorScale = d3
-      .scaleSequential(interpolateRainbow)
-      .domain([0, categories.length]);
-
-    /* pre-create colors - much faster than doing it for each obs */
-    const colors = _.transform(categories, (acc, cat, idx) => {
-      acc[cat] = parseRGB(colorScale(idx));
-    });
-
-    const key = action.colorAccessor;
-    const col = obsAnnotations.col(key).asArray();
-    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
-      const cat = col[i];
-      colorsByRGB[i] = colors[cat];
-    }
-  }
-
-  if (action.type === "color by continuous metadata") {
-    const colorBins = 100;
-    const { min, max } = action.rangeForColorAccessor;
-    colorScale = d3
-      .scaleQuantile()
-      .domain([min, max])
-      .range(_.range(colorBins - 1, -1, -1));
-
-    /* pre-create colors - much faster than doing it for each obs */
-    const colors = new Array(colorBins);
-    for (let i = 0; i < colorBins; i += 1) {
-      colors[i] = parseRGB(interpolateCool(i / colorBins));
-    }
-
-    const key = action.colorAccessor;
-    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
-    const col = obsAnnotations.col(key).asArray();
-    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
-      const val = col[i];
-      if (Number.isFinite(val)) {
-        const c = colorScale(val);
-        colorsByRGB[i] = colors[c];
-      } else {
-        colorsByRGB[i] = nonFiniteColor;
-      }
-    }
-  }
-
-  if (action.type === "color by expression") {
-    const { gene, data } = action;
-    const expression = data[gene]; // Float32Array
-    const colorBins = 100;
-    const [min, max] = finiteExtent(expression);
-    colorScale = d3
-      .scaleQuantile()
-      .domain([min, max])
-      .range(_.range(colorBins - 1, -1, -1));
-
-    /* pre-create colors - much faster than doing it for each obs */
-    const colors = new Array(colorBins);
-    for (let i = 0; i < colorBins; i += 1) {
-      colors[i] = parseRGB(interpolateCool(i / colorBins));
-    }
-    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
-
-    for (let i = 0, len = expression.length; i < len; i += 1) {
-      const e = expression[i];
-      if (Number.isFinite(e)) {
-        const c = colorScale(e);
-        colorsByRGB[i] = colors[c];
-      } else {
-        colorsByRGB[i] = nonFiniteColor;
-      }
-    }
-  }
-
-  /*
-  append the result of all the filters to the action the user just triggered
-  */
-  const modifiedAction = Object.assign({}, action, {
-    colors: { rgb: colorsByRGB },
-    colorScale
-  });
-
-  return next(modifiedAction);
-};
-
-export default updateCellColorsMiddleware;
@@ -1,87 +0,0 @@
-// jshint esversion: 6
-// import uri from "urijs";
-
-/*
-NOTE: file currently not used, but retained as we expect to reinstate features in this
-area shortly.
-*/
-
-/*
-  https://medium.com/@jacobp100/you-arent-using-redux-middleware-enough-94ffe991e6
-  storeInstance
-    => functionToCallWithAnActionThatWillSendItToTheNextMiddleware
-    => actionThatDispatchWasCalledWith
-    => valueToUseAsTheReturnValueOfTheDispatchCall
-*/
-
-const updateURLMiddleware = (/* store */) => next => action => {
-  // const oldState = store.getState();
-  const nextAction = next(action);
-
-  if (action.type === "url changed") {
-    /* we don't handle pop state here - we handle it in the url reducer */
-    return nextAction;
-  }
-
-  // const state = store.getState();
-
-  /************************************************************************
-      *************************************************************************
-      1. Redux app state just changed. Clear URL, and then update it.
-      1a. We get the whole state tree to construct the url!
-      1b. But (see reducers/url.js) we try to centralize it because...
-      1c. ...the back button / initial load case ('url changed' return above)
-      means that we have to listen for 'url changed' and construct state
-      from the browser
-      *************************************************************************
-      ************************************************************************/
-
-  // const oldURI = URI(window.location.href)
-  // const newURI = URI(oldURI).setQuery({})
-
-  // if (window.location.search === "") {
-  //   newURL = uri.addQuery(category, value).toString(); /* #1 */
-  // } else if (uri.hasQuery(category, value) || uri.hasQuery(category, value, true)) { /* true param here means check arrays as well http://medialize.github.io/URI.js/docs.html#search-has */
-  //   newURL = uri.removeQuery(category, value).toString(); /* #4 */
-  // } else {
-  //   newURL = uri.addQuery(category, value).toString(); /* #2 & #3 are handled by URI */
-  // }
-  //
-  // window.history.pushState("", "", newURL)
-
-  //
-  // // Internal helper for working with URIs
-  // const oldURI = new URI(window.location.href);
-  // const newURI = new URI(oldURI).setQueryData({});
-  //
-  // newURI.setPath('/foo/bar');
-  //
-  // // Set the path based on state
-  // if (!state.isOnLandingPage && state.project.id) {
-  //   newURI.setPath(newURI.getPath() + state.project.id + '/');
-  //   newURI.addQueryData('baz', state.mode);
-  //   newURI.addQueryData('bat', state.selection.activePageID);
-  // } else {
-  //   newURI.setPath(newURI.getPath() + state.landingSection + '/');
-  // }
-  //
-  // // Avoid URL thrashing by replacing state while loading instead of pushing
-  // const newPath = newURI.toString();
-  // const oldPath = oldURI.toString();
-  // if (newPath !== oldPath) {
-  //   if (
-  //     (oldState.mode === 'asdf' &&
-  //       state.mode === 'asdf' &&
-  //       !oldState.isOnLandingPage) ||
-  //     oldState.isLoadingProject !== state.isLoadingProject
-  //   ) {
-  //     window.history.replaceState(null, null, newPath);
-  //   } else {
-  //     window.history.pushState(null, null, newPath);
-  //   }
-  // }
-
-  return nextAction;
-};
-
-export default updateURLMiddleware;
@@ -0,0 +1,44 @@
+export default function cascadeReducers(arg) {
+  /*
+  Combined a set of cascading reducers into a single reducer.  Cascading
+  reducers are reducers which may rely on state computed by another reducer.
+  Therefore, they:
+  - must be composed in a particular order (currently, this is a simple
+    linear list of reducers, run in list order)
+  - must have access to partially updated "next state" so they can further
+    derive state.
+
+  Parameter is one of:
+  - a Map object
+  - an array of tuples, [ [key1, reducer1], [key2, reducer2], ... ]
+  Ie,  cascadeReducers([ ["a", reduceA], ["b", reduceB] ])
+
+  Each reducer will be called with the sigature:
+      (prevState, action, sharedNextState, sharedPrevState) => newState
+
+  cascadeReducers will build a composite newState object, much
+  like combinedReducers.  Additional semantics:
+  - reducers guaranteed to be called in order
+  - each reducer will receive shared objects
+  */
+  const reducers = arg instanceof Map ? arg : new Map(arg);
+  const reducerKeys = [...reducers.keys()];
+  return (prevState, action) => {
+    const nextState = {};
+    let stateChange = false;
+    for (let i = 0, l = reducerKeys.length; i < l; i += 1) {
+      const key = reducerKeys[i];
+      const reducer = reducers.get(key);
+      const prevStateForKey = prevState ? prevState[key] : undefined;
+      const nextStateForKey = reducer(
+        prevStateForKey,
+        action,
+        nextState,
+        prevState
+      );
+      nextState[key] = nextStateForKey;
+      stateChange = stateChange || nextStateForKey !== prevStateForKey;
+    }
+    return stateChange ? nextState : prevState;
+  };
+}
@@ -0,0 +1,105 @@
+import { ControlsHelpers } from "../util/stateManager";
+import * as globals from "../globals";
+
+function maxCategoryItems(state) {
+  return (
+    state.config.parameters?.["max-category-items"] ??
+    globals.configDefaults.parameters["max-category-items"]
+  );
+}
+
+const CategoricalSelection = (
+  state,
+  action,
+  nextSharedState,
+  prevSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)":
+    case "set World to current selection":
+    case "reset World to eq Universe":
+    case "set clip quantiles": {
+      const { world } = nextSharedState;
+      return ControlsHelpers.createCategoricalSelection(
+        maxCategoryItems(prevSharedState),
+        world
+      );
+    }
+
+    case "categorical metadata filter select": {
+      /*
+      Set the specific category in this field to false
+      */
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
+      );
+      newCategoryValueSelected[action.categoryIndex] = true;
+      const newCategoricalSelection = {
+        ...state,
+        [action.metadataField]: {
+          ...state[action.metadataField],
+          categoryValueSelected: newCategoryValueSelected
+        }
+      };
+      return newCategoricalSelection;
+    }
+
+    case "categorical metadata filter deselect": {
+      /*
+      Set the specific category in this field to false
+      */
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
+      );
+      newCategoryValueSelected[action.categoryIndex] = false;
+      const newCategoricalSelection = {
+        ...state,
+        [action.metadataField]: {
+          ...state[action.metadataField],
+          categoryValueSelected: newCategoryValueSelected
+        }
+      };
+      return newCategoricalSelection;
+    }
+
+    case "categorical metadata filter none of these": {
+      /*
+      set all categories in this field to false.
+      */
+      const newCategoricalSelection = {
+        ...state,
+        [action.metadataField]: {
+          ...state[action.metadataField],
+          categorySelected: false,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
+          ).fill(false)
+        }
+      };
+      return newCategoricalSelection;
+    }
+
+    case "categorical metadata filter all of these": {
+      /*
+      set all categories in this field to true.
+      */
+      const newCategoricalSelection = {
+        ...state,
+        [action.metadataField]: {
+          ...state[action.metadataField],
+          categorySelected: true,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
+          ).fill(true)
+        }
+      };
+      return newCategoricalSelection;
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default CategoricalSelection;
@@ -0,0 +1,100 @@
+import { createColors } from "../util/stateManager";
+
+const ColorsReducer = (
+  state = {
+    colorMode: null,
+    colorAccessor: null,
+    rgb: null,
+    scale: null
+  },
+  action,
+  nextSharedState,
+  prevSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
+      const { world } = nextSharedState;
+      const colorMode = null;
+      const colorAccessor = null;
+      const { rgb, scale } = createColors(world, colorMode);
+      return {
+        ...state,
+        colorAccessor,
+        colorMode,
+        rgb,
+        scale
+      };
+    }
+
+    case "set clip quantiles":
+    case "set World to current selection": {
+      const { colorMode, colorAccessor } = state;
+      const { world } = nextSharedState;
+      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      return {
+        ...state,
+        rgb,
+        scale
+      };
+    }
+
+    case "reset colorscale": {
+      const { world } = prevSharedState;
+      const { rgb, scale } = createColors(world);
+      return {
+        ...state,
+        colorMode: null,
+        colorAccessor: null,
+        rgb,
+        scale
+      };
+    }
+
+    case "color by categorical metadata":
+    case "color by continuous metadata": {
+      const { world } = prevSharedState;
+
+      /* toggle between this mode and reset */
+      const resetCurrent =
+        action.type === state.colorMode &&
+        action.colorAccessor === state.colorAccessor;
+      const colorMode = !resetCurrent ? action.type : null;
+      const colorAccessor = !resetCurrent ? action.colorAccessor : null;
+
+      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      return {
+        ...state,
+        colorMode,
+        colorAccessor,
+        rgb,
+        scale
+      };
+    }
+
+    case "color by expression": {
+      const { world } = prevSharedState;
+
+      /* toggle between this mode and reset */
+      const resetCurrent =
+        action.type === state.colorMode && action.gene === state.colorAccessor;
+      const colorMode = !resetCurrent ? action.type : null;
+      const colorAccessor = !resetCurrent ? action.gene : null;
+
+      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      return {
+        ...state,
+        colorMode,
+        colorAccessor,
+        rgb,
+        scale
+      };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default ColorsReducer;
@@ -0,0 +1,35 @@
+import { makeContinuousDimensionName } from "../util/nameCreators";
+
+const ContinuousSelection = (state = {}, action) => {
+  switch (action.type) {
+    case "reset World to eq Universe":
+    case "set clip quantiles": {
+      return {};
+    }
+    case "continuous metadata histogram start":
+    case "continuous metadata histogram brush":
+    case "continuous metadata histogram end": {
+      const name = makeContinuousDimensionName(
+        action.continuousNamespace,
+        action.selection
+      );
+      return {
+        ...state,
+        [name]: action.range
+      };
+    }
+    case "continuous metadata histogram cancel": {
+      const name = makeContinuousDimensionName(
+        action.continuousNamespace,
+        action.selection
+      );
+      const { [name]: deletedField, ...newState } = state;
+      return newState;
+    }
+    default: {
+      return state;
+    }
+  }
+};
+
+export default ContinuousSelection;
@@ -2,17 +2,7 @@

 import _ from "lodash";

-import { World, WorldUtil, ControlsHelper } from "../util/stateManager";
-import parseRGB from "../util/parseRGB";
-import Crossfilter from "../util/typedCrossfilter";
-import * as globals from "../globals";
-import {
-  layoutDimensionName,
-  obsAnnoDimensionName,
-  userDefinedDimensionName,
-  diffexpDimensionName,
-  makeContinuousDimensionName
-} from "../util/nameCreators";
+import { WorldUtil } from "../util/stateManager";

 const Controls = (
  state = {
@@ -20,38 +10,23 @@ const Controls = (
    loading: false,
    error: null,

-    // configuration
-    maxCategoryItems: globals.configDefaults.parameters["max-category-items"],
-
-    // the whole big bang
-    universe: null,
-    fullUniverseCache: null,
-
    // all of the data + selection state
-    world: null,
-    colorRGB: null,
-    categoricalSelectionState: null,
-    crossfilter: null,
-    dimensionMap: null,
    userDefinedGenes: [],
    userDefinedGenesLoading: false,
    diffexpGenes: [],

-    colorAccessor: null,
-    colorScale: null,
    resettingInterface: false,

    opacityForDeselectedCells: 0.2,
-    graphBrushSelection: null,
-    continuousSelection: null,
    scatterplotXXaccessor: null, // just easier to read
    scatterplotYYaccessor: null,
-    axesHaveBeenDrawn: false,
    graphRenderCounter: 0 /* integer as <Component key={graphRenderCounter} - a change in key forces a remount */,
    __storedStateForCelllist1__: null /* will need procedural control of brush ie., brush.extent https://bl.ocks.org/micahstubbs/3cda05ca68cba260cb81 */,
    __storedStateForCelllist2__: null
  },
-  action
+  action,
+  nextSharedState,
+  prevSharedState
 ) => {
  /*
  For now, log anything looking like an error to the console.
@@ -65,199 +40,32 @@ const Controls = (
          Initialization, World/Universe management
          and data loading.
    ******************************************************/
-    case "configuration load complete": {
-      // there are a couple of configuration items we need to retain
-      return {
-        ...state,
-        maxCategoryItems: _.get(
-          state.config,
-          "parameters.max-category-items",
-          globals.configDefaults.parameters["max-category-items"]
-        )
-      };
-    }
    case "initial data load start": {
      return { ...state, loading: true };
    }
    case "initial data load complete (universe exists)": {
      /* first light - create world & other data-driven defaults */
-      const { universe } = action;
-      const world = World.createWorldFromEntireUniverse(universe);
-      const colorRGB = new Array(universe.nObs).fill(
-        parseRGB(globals.defaultCellColor)
-      );
-      const categoricalSelectionState = ControlsHelper.createCategoricalSelectionState(
-        state,
-        world
-      );
-      const crossfilter = Crossfilter(world.obsAnnotations);
-      const dimensionMap = World.createObsDimensionMap(crossfilter, world);
      WorldUtil.clearCaches();
-
      return {
        ...state,
        loading: false,
        error: null,
-        universe,
-        fullUniverseCache: { world, crossfilter, dimensionMap },
-        world,
-        colorRGB,
-        categoricalSelectionState,
-        crossfilter,
-        dimensionMap,
-        colorAccessor: null,
        resettingInterface: false
      };
    }
    case "reset World to eq Universe": {
-      const {
-        userDefinedGenes,
-        diffexpGenes,
-        universe,
-        fullUniverseCache
-      } = state;
-      const { world, crossfilter } = fullUniverseCache;
-      // reset all crossfilter dimensions
-      _.forEach(fullUniverseCache.dimensionMap, dim => dim.filterAll());
-      const colorRGB = new Array(universe.nObs).fill(
-        parseRGB(globals.defaultCellColor)
-      );
-      const categoricalSelectionState = ControlsHelper.createCategoricalSelectionState(
-        state,
-        world
-      );
-
-      /* free dimensions not in cache (otherwise they leak) */
-      _.forEach(state.dimensionMap, (dim, dimName) => {
-        if (!fullUniverseCache.dimensionMap[dimName]) {
-          dim.dispose();
-        }
-      });
-      const dimensionMap = {
-        ...fullUniverseCache.dimensionMap,
-        ...ControlsHelper.createGenesDimMap(
-          userDefinedGenes,
-          diffexpGenes,
-          world,
-          crossfilter
-        )
-      };
      WorldUtil.clearCaches();
-
      return {
        ...state,
-        world,
-        colorRGB,
-        categoricalSelectionState,
-        crossfilter,
-        dimensionMap,
-        colorAccessor: null,
        resettingInterface: false
      };
    }
    case "set World to current selection": {
-      const { userDefinedGenes, diffexpGenes } = state;
-
-      /* Set viewable world to be the currently selected data */
-      const world = World.createWorldFromCurrentSelection(
-        action.universe,
-        action.world,
-        action.crossfilter
-      );
-      const colorRGB = new Array(world.nObs).fill(
-        parseRGB(globals.defaultCellColor)
-      );
-      const categoricalSelectionState = ControlsHelper.createCategoricalSelectionState(
-        state,
-        world
-      );
-      const crossfilter = Crossfilter(world.obsAnnotations);
-      const dimensionMap = {
-        ...World.createObsDimensionMap(crossfilter, world),
-        ...ControlsHelper.createGenesDimMap(
-          userDefinedGenes,
-          diffexpGenes,
-          world,
-          crossfilter
-        )
-      };
      WorldUtil.clearCaches();
-
      return {
        ...state,
        loading: false,
-        error: null,
-        world,
-        colorRGB,
-        categoricalSelectionState,
-        crossfilter,
-        dimensionMap,
-        colorAccessor: null
-      };
-    }
-    case "expression load success": {
-      const { world, universe } = state;
-      let universeVarData = universe.varData;
-      let worldVarData = world.varData;
-
-      // Load new expression data into the varData dataframes, if
-      // not already present.
-      _.forEach(action.expressionData, (val, key) => {
-        // If not already in universe.varData, save entire expression column
-        if (!universeVarData.hasCol(key)) {
-          universeVarData = universeVarData.withCol(key, val);
-        }
-
-        // If not already in world.varData, save sliced expression column
-        if (!worldVarData.hasCol(key)) {
-          // Slice if world !== universe, else just use whole column.
-          // Use the obsAnnotation index as the cut key, as we keep
-          // all world dataframes in sync.
-          let worldValSlice = val;
-          if (!World.worldEqUniverse(world, universe)) {
-            worldValSlice = universeVarData
-              .subset(world.obsAnnotations.rowIndex.keys(), [key], null)
-              .icol(0)
-              .asArray();
-          }
-
-          // Now build world's varData dataframe
-          worldVarData = worldVarData.withCol(
-            key,
-            worldValSlice,
-            world.obsAnnotations.rowIndex
-          );
-        }
-      });
-
-      // Prune size of varData "cache" if getting out of hand....
-      const { userDefinedGenes, diffexpGenes } = state;
-      const allTheGenesWeNeed = _.uniq(
-        [].concat(
-          userDefinedGenes,
-          diffexpGenes,
-          Object.keys(action.expressionData)
-        )
-      );
-      universeVarData = ControlsHelper.pruneVarDataCache(
-        universeVarData,
-        allTheGenesWeNeed
-      );
-      worldVarData = ControlsHelper.pruneVarDataCache(
-        worldVarData,
-        allTheGenesWeNeed
-      );
-
-      return {
-        ...state,
-        universe: {
-          ...universe,
-          varData: universeVarData
-        },
-        world: {
-          ...world,
-          varData: worldVarData
-        }
+        error: null
      };
    }
    case "request user defined gene started": {
@@ -273,114 +81,51 @@ const Controls = (
      };
    }
    case "request user defined gene success": {
-      const { world, crossfilter, dimensionMap, userDefinedGenes } = state;
-      const _userDefinedGenes = userDefinedGenes.slice();
-      const gene = action.data.genes[0];
-
-      dimensionMap[
-        userDefinedDimensionName(gene)
-      ] = World.createVarDataDimension(world, crossfilter, gene);
-
+      const { userDefinedGenes } = state;
+      const _userDefinedGenes = _.uniq(
+        userDefinedGenes.concat(action.data.genes)
+      );
      return {
        ...state,
-        dimensionMap,
        userDefinedGenes: _userDefinedGenes,
        userDefinedGenesLoading: false
      };
    }
    case "request differential expression success": {
-      const { world, crossfilter, dimensionMap } = state;
+      const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
      const _diffexpGenes = [];
-
      action.data.forEach(d => {
-        _diffexpGenes.push(world.varAnnotations.at(d[0], "name"));
+        _diffexpGenes.push(world.varAnnotations.at(d[0], varIndexName));
      });
-
-      _.forEach(_diffexpGenes, gene => {
-        dimensionMap[diffexpDimensionName(gene)] = World.createVarDataDimension(
-          world,
-          crossfilter,
-          gene
-        );
-      });
-
      return {
        ...state,
-        dimensionMap,
        diffexpGenes: _diffexpGenes
      };
    }
    case "clear differential expression": {
-      const { world, dimensionMap } = state;
-      const _dimensionMap = dimensionMap;
-      _.forEach(action.diffExp, values => {
-        const name = world.varAnnotations.at(values[0], "name");
-        // clean up crossfilter dimensions
-        const dimension = dimensionMap[diffexpDimensionName(name)];
-        dimension.dispose();
-        delete dimensionMap[diffexpDimensionName(name)];
-      });
      return {
        ...state,
-        dimensionMap: _dimensionMap,
        diffexpGenes: []
      };
    }
-    case "user defined gene": {
-      /*
-        this could also live in expression success with a conditional,
-        but that handles diffexp also
-      */
-      const newUserDefinedGenes = state.userDefinedGenes.slice();
-      newUserDefinedGenes.push(action.data);
-      return {
-        ...state,
-        userDefinedGenes: newUserDefinedGenes
-      };
-    }
    case "clear user defined gene": {
-      const { userDefinedGenes, dimensionMap } = state;
+      const { userDefinedGenes } = state;
      const newUserDefinedGenes = _.filter(
        userDefinedGenes,
        d => d !== action.data
      );
-
-      const dimension = dimensionMap[userDefinedDimensionName(action.data)];
-      dimension.dispose();
-      delete dimensionMap[userDefinedDimensionName(action.data)];
-
      return {
        ...state,
-        dimensionMap,
        userDefinedGenes: newUserDefinedGenes
      };
    }
    case "clear all user defined genes": {
-      const { userDefinedGenes, dimensionMap } = state;
-
-      _.forEach(userDefinedGenes, gene => {
-        const dimension = dimensionMap[userDefinedDimensionName(gene)];
-        dimension.dispose();
-        delete dimensionMap[userDefinedDimensionName(gene)];
-      });
-
      return {
        ...state,
-        dimensionMap,
        userDefinedGenes: []
      };
    }
-    case "reset colorscale": {
-      const { world } = state;
-      const colorRGB = new Array(world.nObs).fill(
-        parseRGB(globals.defaultCellColor)
-      );
-      return {
-        ...state,
-        colorRGB,
-        colorAccessor: null
-      };
-    }
    case "expression load error":
    case "initial data load error": {
      return {
@@ -393,52 +138,6 @@ const Controls = (
    /*******************************
             User Events
     *******************************/
-    case "graph brush selection change": {
-      state.dimensionMap[layoutDimensionName("XY")].filterWithinRect(
-        action.brushCoords.northwest,
-        action.brushCoords.southeast
-      );
-      return {
-        ...state,
-        graphBrushSelection: action.brushCoords
-      };
-    }
-    case "lasso deselect":
-    case "graph brush deselect": {
-      state.dimensionMap[layoutDimensionName("XY")].filterAll();
-      return {
-        ...state,
-        graphBrushSelection: null
-      };
-    }
-    case "lasso selection": {
-      const { polygon } = action;
-      const dXY = state.dimensionMap[layoutDimensionName("XY")];
-      if (polygon.length < 3) {
-        // single point or a line is not a polygon, and is therefore a deselect
-        dXY.filterAll();
-      } else {
-        dXY.filterWithinPolygon(polygon);
-      }
-      return {
-        ...state
-      };
-    }
-    case "continuous metadata histogram brush": {
-      const name = makeContinuousDimensionName(
-        action.continuousNamespace,
-        action.selection
-      );
-
-      // action.selection: metadata name being selected
-      // action.range: filter range, or null if deselected
-      if (!action.range) {
-        state.dimensionMap[name].filterAll();
-      } else {
-        state.dimensionMap[name].filterRange(action.range);
-      }
-      return { ...state };
-    }
    case "change opacity deselected cells in 2d graph background":
      return {
        ...state,
@@ -457,116 +156,6 @@ const Controls = (
        resettingInterface: true
      };
    }
-    /*******************************
-          Categorical metadata
-    *******************************/
-    case "categorical metadata filter select": {
-      const newCategorySelected = Array.from(
-        state.categoricalSelectionState[action.metadataField].categorySelected
-      );
-      newCategorySelected[action.categoryIndex] = true;
-      const newCategoricalSelectionState = {
-        ...state.categoricalSelectionState,
-        [action.metadataField]: {
-          ...state.categoricalSelectionState[action.metadataField],
-          categorySelected: newCategorySelected
-        }
-      };
-
-      // update the filter to match all selected options
-      const cat = newCategoricalSelectionState[action.metadataField];
-      state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        ControlsHelper.selectedValuesForCategory(cat)
-      );
-
-      return {
-        ...state,
-        categoricalSelectionState: newCategoricalSelectionState
-      };
-    }
-    case "categorical metadata filter deselect": {
-      const newCategorySelected = Array.from(
-        state.categoricalSelectionState[action.metadataField].categorySelected
-      );
-      newCategorySelected[action.categoryIndex] = false;
-      const newCategoricalSelectionState = {
-        ...state.categoricalSelectionState,
-        [action.metadataField]: {
-          ...state.categoricalSelectionState[action.metadataField],
-          categorySelected: newCategorySelected
-        }
-      };
-
-      // update the filter to match all selected options
-      const cat = newCategoricalSelectionState[action.metadataField];
-      state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        ControlsHelper.selectedValuesForCategory(cat)
-      );
-
-      return {
-        ...state,
-        categoricalSelectionState: newCategoricalSelectionState
-      };
-    }
-    case "categorical metadata filter none of these": {
-      const newCategoricalSelectionState = {
-        ...state.categoricalSelectionState,
-        [action.metadataField]: {
-          ...state.categoricalSelectionState[action.metadataField],
-          categorySelected: Array.from(
-            state.categoricalSelectionState[action.metadataField]
-              .categorySelected
-          ).fill(false)
-        }
-      };
-      state.dimensionMap[
-        obsAnnoDimensionName(action.metadataField)
-      ].filterNone();
-      return {
-        ...state,
-        categoricalSelectionState: newCategoricalSelectionState
-      };
-    }
-    case "categorical metadata filter all of these": {
-      const newCategoricalSelectionState = {
-        ...state.categoricalSelectionState,
-        [action.metadataField]: {
-          ...state.categoricalSelectionState[action.metadataField],
-          categorySelected: Array.from(
-            state.categoricalSelectionState[action.metadataField]
-              .categorySelected
-          ).fill(true)
-        }
-      };
-      state.dimensionMap[
-        obsAnnoDimensionName(action.metadataField)
-      ].filterAll();
-      return {
-        ...state,
-        categoricalSelectionState: newCategoricalSelectionState
-      };
-    }
-
-    /*******************************
-              Color Scale
-    *******************************/
-    case "color by categorical metadata":
-    case "color by continuous metadata": {
-      return {
-        ...state,
-        colorRGB: action.colors.rgb,
-        colorAccessor: action.colorAccessor,
-        colorScale: action.colorScale
-      };
-    }
-    case "color by expression": {
-      return {
-        ...state,
-        colorRGB: action.colors.rgb,
-        colorAccessor: action.gene,
-        colorScale: action.colorScale
-      };
-    }

    /*******************************
              Scatterplot
@@ -0,0 +1,215 @@
+import _ from "lodash";
+
+import Crossfilter from "../util/typedCrossfilter";
+import { World, ControlsHelpers } from "../util/stateManager";
+import {
+  layoutDimensionName,
+  obsAnnoDimensionName,
+  userDefinedDimensionName,
+  diffexpDimensionName,
+  makeContinuousDimensionName
+} from "../util/nameCreators";
+
+const XYDimName = layoutDimensionName("XY");
+
+const CrossfilterReducer = (
+  state = null,
+  action,
+  nextSharedState,
+  prevSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)": {
+      const { world, layoutChoice } = nextSharedState;
+      const crossfilter = World.createObsDimensions(
+        new Crossfilter(world.obsAnnotations),
+        world,
+        layoutChoice.currentDimNames
+      );
+      return crossfilter;
+    }
+
+    case "reset World to eq Universe": {
+      const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
+      const { world } = nextSharedState;
+      const crossfilter = ControlsHelpers.createGeneDimensions(
+        userDefinedGenes,
+        diffexpGenes,
+        world,
+        prevSharedState.resetCache.crossfilter
+      );
+      return crossfilter;
+    }
+
+    case "set clip quantiles":
+    case "set World to current selection": {
+      const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
+      const { world, layoutChoice } = nextSharedState;
+      let crossfilter = new Crossfilter(world.obsAnnotations);
+      crossfilter = World.createObsDimensions(
+        crossfilter,
+        world,
+        layoutChoice.currentDimNames
+      );
+      crossfilter = ControlsHelpers.createGeneDimensions(
+        userDefinedGenes,
+        diffexpGenes,
+        world,
+        crossfilter
+      );
+      return crossfilter;
+    }
+
+    case "set layout choice": {
+      /*
+      when switching layouts:
+      - delete the existing XY index
+      - add the new XY index (which implicitly selects all on it)
+      */
+      const { world, layoutChoice } = nextSharedState;
+      return state
+        .delDimension(layoutDimensionName("XY"))
+        .addDimension(
+          layoutDimensionName("XY"),
+          "spatial",
+          world.obsLayout.col(layoutChoice.currentDimNames[0]).asArray(),
+          world.obsLayout.col(layoutChoice.currentDimNames[1]).asArray()
+        );
+    }
+
+    case "request user defined gene success": {
+      const { world } = prevSharedState;
+      const gene = action.data.genes[0];
+      return state.addDimension(
+        userDefinedDimensionName(gene),
+        "scalar",
+        world.varData.col(gene).asArray(),
+        Float32Array
+      );
+    }
+
+    case "request differential expression success": {
+      const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
+      const genes = _.map(action.data, d =>
+        world.varAnnotations.at(d[0], varIndexName)
+      );
+      const crossfilter = _.reduce(
+        genes,
+        (xfltr, gene) =>
+          xfltr.addDimension(
+            diffexpDimensionName(gene),
+            "scalar",
+            world.varData.col(gene).asArray(),
+            Float32Array
+          ),
+        state
+      );
+      return crossfilter;
+    }
+
+    case "clear differential expression": {
+      const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
+      const crossfilter = _.reduce(
+        action.diffExp,
+        (xfltr, values) => {
+          const name = world.varAnnotations.at(values[0], varIndexName);
+          return xfltr.delDimension(diffexpDimensionName(name));
+        },
+        state
+      );
+      return crossfilter;
+    }
+
+    case "clear user defined gene": {
+      return state.delDimension(userDefinedDimensionName(action.data));
+    }
+
+    case "clear all user defined genes": {
+      const { userDefinedGenes } = prevSharedState.controls;
+      const crossfilter = _.reduce(
+        userDefinedGenes,
+        (xfltr, gene) => xfltr.delDimension(userDefinedDimensionName(gene)),
+        state
+      );
+      return crossfilter;
+    }
+
+    case "graph brush end":
+    case "graph brush change": {
+      const [minX, maxY] = action.brushCoords.northwest;
+      const [maxX, minY] = action.brushCoords.southeast;
+      return state.select(XYDimName, {
+        mode: "within-rect",
+        minX,
+        minY,
+        maxX,
+        maxY
+      });
+    }
+
+    case "graph lasso end": {
+      const { polygon } = action;
+      return state.select(XYDimName, {
+        mode: "within-polygon",
+        polygon
+      });
+    }
+
+    case "graph lasso cancel":
+    case "graph brush cancel":
+    case "graph lasso deselect":
+    case "graph brush deselect": {
+      return state.select(XYDimName, { mode: "all" });
+    }
+
+    case "continuous metadata histogram start":
+    case "continuous metadata histogram brush":
+    case "continuous metadata histogram cancel":
+    case "continuous metadata histogram end": {
+      const name = makeContinuousDimensionName(
+        action.continuousNamespace,
+        action.selection
+      );
+      // action.selection: metadata name being selected
+      // action.range: filter range, or null if deselected
+      if (!action.range) {
+        return state.select(name, { mode: "all" });
+      }
+      const [lo, hi] = action.range;
+      const newState = state.select(name, { mode: "range", lo, hi });
+      return newState;
+    }
+
+    case "categorical metadata filter select":
+    case "categorical metadata filter deselect": {
+      const { categoricalSelection } = nextSharedState;
+      const { world } = prevSharedState;
+      const cat = categoricalSelection[action.metadataField];
+      const col = world.obsAnnotations.col(action.metadataField);
+      return state.select(obsAnnoDimensionName(action.metadataField), {
+        mode: "exact",
+        values: ControlsHelpers.selectedValuesForCategory(cat, col)
+      });
+    }
+
+    case "categorical metadata filter none of these": {
+      return state.select(obsAnnoDimensionName(action.metadataField), {
+        mode: "none"
+      });
+    }
+
+    case "categorical metadata filter all of these": {
+      return state.select(obsAnnoDimensionName(action.metadataField), {
+        mode: "all"
+      });
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default CrossfilterReducer;
@@ -0,0 +1,61 @@
+const GraphSelection = (
+  state = {
+    tool: "lasso", // what selection tool mode (lasso, brush, ...)
+    selection: { mode: "all" } // current selection, which is tool specific
+  },
+  action
+) => {
+  switch (action.type) {
+    case "set clip quantiles":
+    case "reset World to eq Universe":
+    case "set layout choice": {
+      return {
+        ...state,
+        selection: {
+          mode: "all"
+        }
+      };
+    }
+
+    case "graph brush end":
+    case "graph brush change": {
+      const { brushCoords } = action;
+      return {
+        ...state,
+        selection: {
+          mode: "within-rect",
+          brushCoords
+        }
+      };
+    }
+
+    case "graph lasso end": {
+      const { polygon } = action;
+      return {
+        ...state,
+        selection: {
+          mode: "within-polygon",
+          polygon
+        }
+      };
+    }
+
+    case "graph lasso cancel":
+    case "graph brush cancel":
+    case "graph lasso deselect":
+    case "graph brush deselect": {
+      return {
+        ...state,
+        selection: {
+          mode: "all"
+        }
+      };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default GraphSelection;
@@ -1,27 +1,54 @@
-// jshint esversion: 6
-import { combineReducers, createStore, applyMiddleware } from "redux";
+import { createStore, applyMiddleware } from "redux";
 import thunk from "redux-thunk";
-import { composeWithDevTools } from "redux-devtools-extension";
-import updateURLMiddleware from "../middleware/updateURLMiddleware";
-import updateCellColors from "../middleware/updateCellColors";

+import cascadeReducers from "./cascade";
+import undoable from "./undoable";
 import config from "./config";
+import universe from "./universe";
+import world from "./world";
+import categoricalSelection from "./categoricalSelection";
+import continuousSelection from "./continuousSelection";
+import graphSelection from "./graphSelection";
+import crossfilter from "./crossfilter";
+import colors from "./colors";
 import differential from "./differential";
+import layoutChoice from "./layoutChoice";
 import responsive from "./responsive";
 import controls from "./controls";
+import resetCache from "./resetCache";

-const Reducer = combineReducers({
-  config,
-  responsive,
-  controls,
-  differential
-});
+import undoableConfig from "./undoableConfig";

-const store = createStore(
-  Reducer,
-  composeWithDevTools(
-    applyMiddleware(thunk, updateURLMiddleware, updateCellColors)
-  )
+const Reducer = undoable(
+    cascadeReducers([
+        ["config", config],
+        ["universe", universe],
+        ["world", world],
+        ["layoutChoice", layoutChoice],
+        ["categoricalSelection", categoricalSelection],
+        ["continuousSelection", continuousSelection],
+        ["graphSelection", graphSelection],
+        ["crossfilter", crossfilter],
+        ["colors", colors],
+        ["controls", controls],
+        ["differential", differential],
+        ["responsive", responsive],
+        ["resetCache", resetCache]
+    ]),
+    [
+        "world",
+        "categoricalSelection",
+        "continuousSelection",
+        "graphSelection",
+        "crossfilter",
+        "colors",
+        "controls",
+        "differential",
+        "layoutChoice"
+    ],
+    undoableConfig
 );

+const store = createStore(Reducer, applyMiddleware(thunk));
+
 export default store;
@@ -0,0 +1,50 @@
+/*
+we have a UI heuristic to pick the default layout, based on assumptions
+about commonly used names.  Preferentially, pick in the following order:
+
+  1. "umap"
+  2. "tsne"
+  3. "pca"
+  4. give up, use the first available
+*/
+function bestDefaultLayout(layouts) {
+  const preferredNames = ["umap", "tsne", "pca"];
+  const idx = preferredNames.findIndex(name => layouts.indexOf(name) !== -1);
+  if (idx !== -1) return preferredNames[idx];
+  return layouts[0];
+}
+
+const LayoutChoice = (
+  state = {
+    available: [], // all available choices
+    current: undefined, // name of the current layout, eg, 'umap'
+    currentDimNames: [] // dimension name
+  },
+  action,
+  nextSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
+      // set default to default
+      const { schema } = nextSharedState.world;
+      const available = schema.layout.obs.map(v => v.name);
+      const current = bestDefaultLayout(available);
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { available, current, currentDimNames };
+    }
+
+    case "set layout choice": {
+      const { schema } = nextSharedState.world;
+      const current = action.layoutChoice;
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { ...state, current, currentDimNames };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default LayoutChoice;
@@ -0,0 +1,31 @@
+/*
+Reducer which caches derived state to be used in a reset or other 
+recomputation.
+
+Currently this only caches the baseline (full universe) world & crossfilter, 
+for use in a Reset.
+*/
+const ResetCacheReducer = (
+  state = {
+    world: null,
+    crossfilter: null
+  },
+  action,
+  nextSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)": {
+      const { world, crossfilter } = nextSharedState;
+      return {
+        ...state,
+        world,
+        crossfilter
+      };
+    }
+    default: {
+      return state;
+    }
+  }
+};
+
+export default ResetCacheReducer;
@@ -0,0 +1,287 @@
+/*
+A redo/undo meta reducer for Redux.   Designed to work well with the cascadeReducer().
+
+Requires three parameters:
+* reducer - a reducer, which MUST return an object as state.
+* undoableKeys - an array of object keys (strings).   If any of these keys
+  are in the object/state returned by the reducer, they will be treated as
+  state to be made "undoable".
+* options - an optional object, which may contain the following parameters:
+    * historyLimit: max number of historical states to remember (aka max undo depth)
+    * actionFilter: filter function, (state, action, filterState) => value.
+      See below for details.
+    * debug: if truish, will print helpful log messages about history manipulation
+
+This meta reducer accepts three actions types:
+* @@undoable/undo - move back in history
+* @@undoable/redo - move forward in history
+* @@undoable/clear - clear history
+
+---
+
+Action filter - controls the undoable reducer side-effects.  If not
+specified, the action filter defaults to "save", ie, pushes a redo
+point upon each action.
+
+The action filter callback has access to the current action, the entire
+undoable reducer state, and any state it wants to manage ("filterState").
+This filter state will be passed to each action filter call, and any
+value returned (via @@undoable/filterState field described below) will be
+MERGED into the current filter state.
+
+An object must be returned (the "undoable action"), indicating desired
+history state processing.  The undoable action object contents, by key:
+
+  @@undoable/filterAction: required.   Can be one of:
+      "skip" - reduce the current action, but no other side effects.
+        Same as returning false.
+      "clear" - reduce the current action, and clear history state.
+      "save" - push the previous state onto the history stack (ie,
+        before reducing the action)
+      "stashPending" - reduce action, save state as pending.  Does not
+        not commit it to history.  Along with cancelPending and applyPending,
+        can be used to delay commit of history (eg, for multi-action
+        groupings, asynch operations, etc).
+      "cancelPending" - reduce action, cancel any pending state save.
+      "applyPending" - commit any pending state to the history stack,
+        then reduce action.
+
+  @@undoable/filterState: optional.  If this value is set, it will be
+    MERGED into the current filter state. The value and semantics of any
+    filter state are entirely at the discretion of the action filter.
+
+*/
+import fromEntries from "../util/fromEntries";
+
+const historyKeyPrefix = "@@undoable/";
+const pastKey = `${historyKeyPrefix}past`;
+const futureKey = `${historyKeyPrefix}future`;
+const filterStateKey = `${historyKeyPrefix}filterState`;
+const filterActionKey = `${historyKeyPrefix}filterAction`;
+const pendingKey = `${historyKeyPrefix}pending`;
+const defaultHistoryLimit = -100;
+
+const Undoable = (reducer, undoableKeys, options = {}) => {
+  const { debug } = options;
+  let { historyLimit } = options;
+  if (!historyLimit) historyLimit = defaultHistoryLimit;
+  if (historyLimit > 0) historyLimit = -historyLimit;
+  const actionFilter =
+    options.actionFilter || (() => ({ [filterActionKey]: "save" }));
+
+  if (!Array.isArray(undoableKeys) || undoableKeys.length === 0)
+    throw new Error("undoable keys array must be specified");
+  const undoableKeysSet = new Set(undoableKeys);
+
+  /*
+  Undo the current to previous history
+  */
+  function undo(currentState) {
+    const past = currentState[pastKey];
+    const future = currentState[futureKey];
+    if (past.length === 0) return currentState;
+    const currentUndoableState = Object.entries(currentState).filter(kv =>
+      undoableKeysSet.has(kv[0])
+    );
+    const newPast = [...past];
+    const newState = newPast.pop();
+    const newFuture = push(future, currentUndoableState);
+    const nextState = {
+      ...currentState,
+      ...fromEntries(newState),
+      [pastKey]: newPast,
+      [futureKey]: newFuture,
+      [pendingKey]: null
+    };
+    return nextState;
+  }
+
+  /*
+  Replay future, previously undone.
+  */
+  function redo(currentState) {
+    const past = currentState[pastKey] || [];
+    const future = currentState[futureKey] || [];
+    if (future.length === 0) return currentState;
+    const currentUndoableState = Object.entries(currentState).filter(kv =>
+      undoableKeysSet.has(kv[0])
+    );
+    const newFuture = [...future];
+    const newState = newFuture.pop();
+    const newPast = push(past, currentUndoableState);
+    const nextState = {
+      ...currentState,
+      ...fromEntries(newState),
+      [pastKey]: newPast,
+      [futureKey]: newFuture,
+      [pendingKey]: null
+    };
+    return nextState;
+  }
+
+  /*
+  Clear the history state.  No side-effects on current state.
+  */
+  function clear(currentState) {
+    return {
+      ...currentState,
+      [pastKey]: [],
+      [futureKey]: [],
+      [filterStateKey]: {},
+      [pendingKey]: null
+    };
+  }
+
+  /*
+  Reduce current action, with no history side-effects
+  */
+  function skip(currentState, action, filterState) {
+    const past = currentState[pastKey] || [];
+    const pending = currentState[pendingKey];
+    const res = reducer(currentState, action);
+    return {
+      ...res,
+      [pastKey]: past,
+      [futureKey]: [],
+      [filterStateKey]: filterState,
+      [pendingKey]: pending
+    };
+  }
+
+  /*
+  Save current state in the history, then reduce action.
+  */
+  function save(currentState, action, filterState) {
+    const past = currentState[pastKey] || [];
+    const currentUndoableState = Object.entries(currentState).filter(kv =>
+      undoableKeysSet.has(kv[0])
+    );
+    const res = reducer(currentState, action);
+    const newPast = push(past, currentUndoableState, historyLimit);
+    const nextState = {
+      ...res,
+      [pastKey]: newPast,
+      [futureKey]: [],
+      [filterStateKey]: filterState,
+      [pendingKey]: null
+    };
+    return nextState;
+  }
+
+  /*
+  Save current state as pending history change.  No other side effects.
+  */
+  function stashPending(currentState) {
+    const currentUndoableState = Object.entries(currentState).filter(kv =>
+      undoableKeysSet.has(kv[0])
+    );
+    return {
+      ...currentState,
+      [pendingKey]: currentUndoableState
+    };
+  }
+
+  /*
+  Cancel pending history state change.  No other side effects.
+  */
+  function cancelPending(currentState) {
+    return {
+      ...currentState,
+      [pendingKey]: null
+    };
+  }
+
+  /*
+  Push pending state onto the history stack
+  */
+  function applyPending(currentState) {
+    const past = currentState[pastKey] || [];
+    const pendingState = currentState[pendingKey];
+    const newPast = push(past, pendingState, historyLimit);
+    const nextState = {
+      ...currentState,
+      [pastKey]: newPast,
+      [futureKey]: [],
+      [pendingKey]: null
+    };
+    return nextState;
+  }
+
+  return (
+    currentState = {
+      [pastKey]: [],
+      [futureKey]: [],
+      [filterStateKey]: {},
+      [pendingKey]: null
+    },
+    action
+  ) => {
+    const aType = action.type;
+    switch (aType) {
+      case "@@undoable/undo": {
+        return undo(currentState, action);
+      }
+
+      case "@@undoable/redo": {
+        return redo(currentState, action);
+      }
+
+      case "@@undoable/clear": {
+        return clear(currentState, action);
+      }
+
+      default: {
+        const currentFilterState = currentState[filterStateKey];
+        const actionFilterResp = actionFilter(
+          currentState,
+          action,
+          currentFilterState
+        );
+        const {
+          [filterActionKey]: filterAction,
+          [filterStateKey]: filterStateUpdate
+        } = actionFilterResp;
+        const nextFilterState = { ...currentFilterState, ...filterStateUpdate };
+
+        switch (filterAction) {
+          case "clear":
+            if (debug) console.log("---- CLEAR HISTO", action.type);
+            return clear(skip(currentState, action, nextFilterState));
+
+          case "save":
+            if (debug) console.log("---- SAVE HISTO", action.type);
+            return save(currentState, action, nextFilterState);
+
+          case "stashPending":
+            if (debug) console.log("---- STASH PENDING", action.type);
+            return skip(stashPending(currentState), action, nextFilterState);
+
+          case "cancelPending":
+            if (debug) console.log("---- CANCEL PENDING", action.type);
+            return skip(cancelPending(currentState), action, nextFilterState);
+
+          case "applyPending":
+            if (debug) console.log("---- APPLY PENDING", action.type);
+            return skip(applyPending(currentState), action, nextFilterState);
+
+          case "skip":
+          default:
+            return skip(currentState, action, nextFilterState);
+        }
+      }
+    }
+  };
+};
+
+function push(arr, val, limit = undefined) {
+  /*
+  functional array push, with a max length limit to the new array.
+  Like Array.push, except it returns new array and discards as needed
+  to enforce the length limit.
+  */
+  const narr = arr.slice(limit);
+  narr.push(val);
+  return narr;
+}
+
+export default Undoable;
@@ -0,0 +1,259 @@
+import StateMachine from "../util/statemachine";
+import createFsmTransitions from "./undoableFsm";
+
+const actionKey = "@@undoable/filterAction";
+const stateKey = "@@undoable/filterState";
+
+/*
+these actions will not affect history
+*/
+const skipOnActions = new Set([
+  "url changed",
+  "interface reset started",
+  "initial data load start",
+  "configuration load complete",
+  "increment graph render counter",
+  "window resize",
+  "user reset start",
+  "reset colorscale",
+
+  "graph brush change",
+  "continuous metadata histogram brush",
+
+  "expression load start",
+  "expression load success",
+  "expression load error",
+
+  "request user defined gene started",
+  "request user defined gene success",
+  "clear all user defined genes",
+
+  "get single gene expression for coloring started",
+  "get single gene expression for coloring error"
+]);
+
+/*
+identical, repeated occurances of these action types will be debounced.
+Entire action must be identical (all keys).
+*/
+const debounceOnActions = new Set([
+  "color by categorical metadata",
+  "color by continuous metadata",
+  "color by expression"
+]);
+
+/*
+history will be cleared when these actions occur
+*/
+const clearOnActions = new Set([
+  "initial data load complete (universe exists)",
+  "reset World to eq Universe",
+  "initial data load error",
+  "user reset end"
+]);
+
+/*
+An immediate history save will be done for these
+*/
+const saveOnActions = new Set([
+  "categorical metadata filter select",
+  "categorical metadata filter deselect",
+  "categorical metadata filter all of these",
+  "categorical metadata filter none of these",
+
+  "color by categorical metadata",
+  "color by continuous metadata",
+  "color by expression",
+
+  "set scatterplot x",
+  "set scatterplot y",
+
+  "store current cell selection as differential set 1",
+  "store current cell selection as differential set 2",
+
+  "set World to current selection",
+  "set clip quantiles",
+
+  "set layout choice"
+]);
+
+/**
+StateMachine - processing complex action handling - see FSM graph for
+actual structure, in undoableFsm.js
+**/
+
+/*
+Default FSM actions.  Used to side-effect transitions in the graph.
+See graph definition for the transitions that use each.
+
+Signature:  (fsm, transition, reducerState, reducerAction) => undoableAction
+*/
+const stashPending = fsm => ({
+  [actionKey]: "stashPending",
+  [stateKey]: { fsm }
+});
+const cancelPending = () => ({
+  [actionKey]: "cancelPending",
+  [stateKey]: { fsm: null }
+});
+const applyPending = () => ({
+  [actionKey]: "applyPending",
+  [stateKey]: { fsm: null }
+});
+const skip = (fsm, transition) => ({
+  [actionKey]: "skip",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
+const clear = () => ({ [actionKey]: "clear", [stateKey]: { fsm: null } });
+const save = (fsm, transition) => ({
+  [actionKey]: "save",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
+
+/*
+Error handler for state transitions that are unexpected.  Called by
+StateMachine when it doesn't know what to do.
+
+Signature:  (fsm, event, from) => undoableAction
+*/
+const onFsmError = (fsm, event, from) => {
+  console.error(`FSM error [event: "${event}", state: "${from}"]`, fsm);
+  // In production, try to recover gracefully if we have unexpected state
+  return clear(fsm);
+};
+
+/*
+Definition of the transition graph mapping action types to history side effects.
+*/
+const fsmTransitions = createFsmTransitions(
+  stashPending,
+  cancelPending,
+  applyPending,
+  skip,
+  clear,
+  save
+);
+/* State machine we clone whenever we need to run it */
+const seedFsm = new StateMachine("init", fsmTransitions, onFsmError);
+
+/*
+See undoable.js for description action filter interface description.
+
+Basic approach:
+  * trivial handlers for skip, clear & save cases to keep config simple.
+  * only implement complex state machines where absolutely required (eg,
+    multi-event seleciton and the like)
+*/
+const actionFilter = debug => (state, action, prevFilterState) => {
+  const actionType = action.type;
+  const filterState = {
+    ...prevFilterState,
+    prevAction: action
+  };
+  if (skipOnActions.has(actionType)) {
+    return { [actionKey]: "skip", [stateKey]: filterState };
+  }
+  if (
+    debounceOnActions.has(actionType) &&
+    shallowObjectEq(action, prevFilterState.prevAction)
+  ) {
+    return { [actionKey]: "skip", [stateKey]: filterState };
+  }
+  if (clearOnActions.has(actionType)) {
+    return { [actionKey]: "clear", [stateKey]: filterState };
+  }
+  if (saveOnActions.has(actionType)) {
+    return { [actionKey]: "save", [stateKey]: filterState };
+  }
+
+  /*
+    Else, something more complex OR unknown to us....
+    */
+  if (seedFsm.events.has(actionType)) {
+    let { fsm } = filterState;
+    if (!fsm) {
+      /* no active FSM, so create one in init state */
+      fsm = seedFsm.clone("init");
+    }
+    return fsm.next(action.type, { state, action });
+  }
+
+  /* else, we have no idea what this is - skip it */
+  if (debug) console.log("**** ACTION FILTER EVENT HANDLER MISS", actionType);
+  return { [actionKey]: "skip", [stateKey]: filterState };
+};
+
+/*
+return true if objA and objB are ===, OR if:
+  - are both objects and not null
+  - have same own properties
+  - all values are strict equal (===)
+*/
+function shallowObjectEq(objA, objB) {
+  if (objA === objB) return true;
+  if (!objA || !objB) return false;
+  if (!shallowArrayEq(Object.keys(objA), Object.keys(objB))) return false;
+  if (!shallowArrayEq(Object.values(objA), Object.values(objB))) return false;
+  return true;
+}
+
+/*
+return true if arrA and arrB contain the same strict-equal values,
+in the same order.
+*/
+function shallowArrayEq(arrA, arrB) {
+  if (arrA.length !== arrB.length) return false;
+  for (let i = 0, l = arrA.length; i < l; i += 1) {
+    if (arrA[i] !== arrB[i]) return false;
+  }
+  return true;
+}
+
+/* configuration for the undoable meta reducer */
+const debug = false; // set truish for undoble debugging
+const undoableConfig = {
+  debug,
+  historyLimit: 50, // maximum history size
+  actionFilter: actionFilter(debug)
+};
+
+/*
+this code is strictly for sanity checking configuration, and is only
+enabled when we are debugging the undoable configuration (ie, debug === true).
+*/
+if (debug) {
+  /*
+  Confirm no intersection between the various trivial rejection action filters
+  */
+  if (
+    new Set([...skipOnActions].filter(x => clearOnActions.has(x))).size > 0 ||
+    new Set([...skipOnActions].filter(x => saveOnActions.has(x))).size > 0 ||
+    new Set([...clearOnActions].filter(x => saveOnActions.has(x))).size > 0
+  ) {
+    console.error(
+      "Undoable misconfiguration - action filters have redundant events"
+    );
+  }
+
+  /*
+  Confirm that no FSM events are blocked by a trivial rejection filter.
+  If this occurs, the FSM can't ever see the events needed to process
+  state transitions.
+  */
+  const trivialFilters = new Set([
+    ...skipOnActions,
+    ...clearOnActions,
+    ...saveOnActions
+  ]);
+  const trivialOverlapWithFsm = new Set(
+    [...trivialFilters].filter(x => seedFsm.events.has(x))
+  );
+  if (trivialOverlapWithFsm.size > 0) {
+    console.error(
+      "Undoable misconfiguration - trivival action filter blocking FSM filter",
+      [...trivialOverlapWithFsm]
+    );
+  }
+}
+
+export default undoableConfig;
@@ -0,0 +1,206 @@
+/*
+State transition graph for complex action/history interactions.
+
+Assumed configuration from undoableConfig:
+    * By convention, "init" is used as the start state for all, and "done"
+      as the final state.
+    * Unexpected states will result in an error, plus a clear and cancelPending
+      side-effect.
+
+TODO: is is possible there is a more concise format for this, as it is
+a fairly repetitive pattern.
+
+These events are largely one of two types:
+a) async operations or multi-event options that should only be committed
+upon some success criteria, otherwise cancelled.
+
+b) compound actions that should be collapsed into a single history change.
+
+*/
+
+const createFsmTransitions = (
+  stashPending,
+  cancelPending,
+  applyPending,
+  skip,
+  clear,
+  save
+) => {
+  return [
+    /* graph selection brushing */
+    {
+      event: "graph brush start",
+      from: "init",
+      to: "graph brush in progress",
+      action: stashPending
+    },
+    {
+      event: "graph brush cancel",
+      from: "graph brush in progress",
+      to: "done",
+      action: applyPending
+    },
+    {
+      event: "graph brush deselect",
+      from: "graph brush in progress",
+      to: "done",
+      /* if current selection is all, cancelPending.  Else, applyPending */
+      action: (fsm, transition, data) =>
+        data.state.graphSelection.selection.mode === "all"
+          ? cancelPending()
+          : applyPending()
+    },
+    {
+      event: "graph brush end",
+      from: "graph brush in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* graph selection lasso */
+    {
+      event: "graph lasso start",
+      from: "init",
+      to: "graph lasso in progress",
+      action: stashPending
+    },
+    {
+      event: "graph lasso cancel",
+      from: "graph lasso in progress",
+      to: "done",
+      action: applyPending
+    },
+    {
+      event: "graph lasso deselect",
+      from: "graph lasso in progress",
+      to: "done",
+      /* if current selection is all, cancelPending.  Else, applyPending */
+      action: (fsm, transition, data) =>
+        data.state.graphSelection.selection.mode === "all"
+          ? cancelPending()
+          : applyPending()
+    },
+    {
+      event: "graph lasso end",
+      from: "graph lasso in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Continuous metadata histogram brush selection */
+    {
+      event: "continuous metadata histogram start",
+      from: "init",
+      to: "continuous histo select in progress",
+      action: stashPending
+    },
+    {
+      event: "continuous metadata histogram cancel",
+      from: "continuous histo select in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "continuous metadata histogram end",
+      from: "continuous histo select in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Single gene request by user */
+    {
+      event: "single user defined gene start",
+      from: "init",
+      to: "single user gene request in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "single user gene request in progress",
+      to: "single user gene error in progress",
+      action: skip
+    },
+    {
+      event: "single user defined gene error",
+      from: "single user gene error in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "single user defined gene complete",
+      from: "single user gene request in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Bulk gene request by user */
+    {
+      event: "bulk user defined gene start",
+      from: "init",
+      to: "bulk user gene request in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "bulk user gene request in progress",
+      to: "bulk user gene request error in progress",
+      action: skip
+    },
+    {
+      event: "bulk user defined gene error",
+      from: "bulk user gene request error in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "bulk user defined gene complete",
+      from: "bulk user gene request in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Compute Differential Expression button user action */
+    {
+      event: "request differential expression started",
+      from: "init",
+      to: "diffexp in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "diffexp in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "request differential expression success",
+      from: "diffexp in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Clear Differential Expression button user action */
+    {
+      event: "clear differential expression",
+      from: "init",
+      to: "CDE Button in progress",
+      action: stashPending
+    },
+    {
+      event: "clear scatterplot",
+      from: "CDE Button in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* clear scatter plot button (eg, on scatterplot view) */
+    {
+      event: "clear scatterplot",
+      from: "init",
+      to: "done",
+      action: save
+    }
+  ];
+};
+
+export default createFsmTransitions;
@@ -0,0 +1,47 @@
+import { ControlsHelpers } from "../util/stateManager";
+
+const Universe = (state = null, action, nextSharedState, prevSharedState) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)": {
+      const { universe } = action;
+      return universe;
+    }
+
+    case "expression load success": {
+      let { varData } = state;
+
+      // Lazy load new expression data into the varData dataframe, if
+      // not already present.
+      //
+      Object.entries(action.expressionData).forEach(([key, val]) => {
+        // If not already in universe.varData, save entire expression column
+        if (!varData.hasCol(key)) {
+          varData = varData.withCol(key, val);
+        }
+      });
+
+      // Prune size of varData "cache" if getting out of hand....
+      //
+      const { userDefinedGenes, diffexpGenes } = prevSharedState;
+      const allTheGenesWeNeed = [
+        ...new Set(
+          userDefinedGenes,
+          diffexpGenes,
+          Object.keys(action.expressionData)
+        )
+      ];
+      varData = ControlsHelpers.pruneVarDataCache(varData, allTheGenesWeNeed);
+
+      return {
+        ...state,
+        varData
+      };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default Universe;
@@ -0,0 +1,131 @@
+import { World, ControlsHelpers } from "../util/stateManager";
+import clip from "../util/clip";
+import quantile from "../util/quantile";
+
+const WorldReducer = (
+  state = null,
+  action,
+  nextSharedState,
+  prevSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)": {
+      const { universe } = nextSharedState;
+      const world = World.createWorldFromEntireUniverse(universe);
+      return world;
+    }
+
+    case "reset World to eq Universe": {
+      return prevSharedState.resetCache.world;
+    }
+
+    case "set World to current selection": {
+      /* Set viewable world to be the currently selected data */
+      const world = World.createWorldBySelection(
+        action.universe,
+        action.world,
+        action.crossfilter
+      );
+      return world;
+    }
+
+    case "set clip quantiles": {
+      const world = World.createWorldWithNewClip(
+        prevSharedState.universe,
+        state,
+        prevSharedState.crossfilter,
+        action.clipQuantiles
+      );
+      return world;
+    }
+
+    case "expression load success": {
+      const { universe } = nextSharedState;
+      const universeVarData = universe.varData;
+      let unclippedVarData = state.unclipped.varData;
+
+      // Lazy load new expression data into the unclipped varData dataframe, if
+      // not already present.
+      //
+      Object.entries(action.expressionData).forEach(([key, val]) => {
+        // If not already in world.varData, save sliced expression column
+        if (!unclippedVarData.hasCol(key)) {
+          // Slice if world !== universe, else just use whole column.
+          // Use the obsAnnotation index as the cut key, as we keep
+          // all world dataframes in sync.
+          let worldValSlice = val;
+          if (!World.worldEqUniverse(state, universe)) {
+            worldValSlice = universeVarData
+              .subset(state.obsAnnotations.rowIndex.keys(), [key], null)
+              .icol(0)
+              .asArray();
+          }
+
+          // Now build world's varData dataframe
+          unclippedVarData = unclippedVarData.withCol(
+            key,
+            worldValSlice,
+            state.obsAnnotations.rowIndex
+          );
+        }
+      });
+
+      // Prune size of varData unclipped dataframe if getting out of hand....
+      //
+      const { userDefinedGenes, diffexpGenes } = prevSharedState;
+      const allTheGenesWeNeed = [
+        ...new Set(
+          userDefinedGenes,
+          diffexpGenes,
+          Object.keys(action.expressionData)
+        )
+      ];
+      unclippedVarData = ControlsHelpers.pruneVarDataCache(
+        unclippedVarData,
+        allTheGenesWeNeed
+      );
+
+      // at this point, we have the unclipped data in unclippedVarData.
+      // Now create clipped.
+      //   - Drop columns no longer needed
+      //   - Add new columns
+      //
+      let clippedVarData = state.varData;
+      const keysToDrop = clippedVarData.colIndex
+        .keys()
+        .filter(k => !unclippedVarData.hasCol(k));
+      const keysToAdd = unclippedVarData.colIndex
+        .keys()
+        .filter(k => !clippedVarData.hasCol(k));
+      keysToDrop.forEach(k => {
+        clippedVarData = clippedVarData.dropCol(k);
+      });
+      keysToAdd.forEach(k => {
+        const data = unclippedVarData.col(k).asArray();
+        const q = [state.clipQuantiles.min, state.clipQuantiles.max];
+        const [qMinVal, qMaxVal] = quantile(q, data);
+        const clippedData = clip(data, qMinVal, qMaxVal, Number.NaN);
+        clippedVarData = clippedVarData.withCol(
+          k,
+          clippedData,
+          state.obsAnnotations.rowIndex
+        );
+      });
+
+      return {
+        ...state,
+        varData: clippedVarData,
+        unclipped: {
+          ...state.unclipped,
+          varData: unclippedVarData
+        }
+      };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default WorldReducer;
@@ -5,10 +5,9 @@ const mp = require("mouse-position");
 const mb = require("mouse-pressed");
 const key = require("key-pressed");

-const panSpeed = 0.4;
+const panSpeed = 1.0; // changed from 0.4 to 1.0 per issue #722
 const scaleSpeed = 0.5;
 const scaleMax = 3;
-// const scaleMin = 1.15
 const scaleMin = 1.03;

 function attachCamera(canvas, opts) {
@@ -0,0 +1,25 @@
+/*
+clip - clip all values in a Array or TypedArray, IN PLACE.
+
+Values in array are clipped if less than `lower` or greater than `upper`.
+
+If `setTo` is undefined, values less than `lower` will be set to `lower`,
+and values greater than `upper` will be set to `upper`.  
+
+If `setTo` is not undefined, values outside the [lower, upper] range will be set to 
+`setTo`.
+
+*/
+export default function clip(arr, lower, upper, setTo) {
+	const lowerSet = setTo === undefined ? lower : setTo;
+	const upperSet = setTo === undefined ? upper : setTo;
+	for (let i = 0, l = arr.length; i < l; i += 1) {
+		const v = arr[i];
+		if (v < lower) {
+			arr[i] = lowerSet;
+		} else if (v > upper) {
+			arr[i] = upperSet;
+		}
+	}
+	return arr;
+}
@@ -1,6 +1,6 @@
 import { IdentityInt32Index, isLabelIndex } from "./labelIndex";
 // weird cross-dependency that we should clean up someday...
-import { sort } from "../typedCrossfilter/sort";
+import { sortArray } from "../typedCrossfilter/sort";
 import { isTypedArray, isArrayOrTypedArray, callOnceLazy } from "./util";
 import { summarizeContinuous, summarizeCategorical } from "./summarize";

@@ -63,7 +63,13 @@ class Dataframe {
  Constructors & factories
  **/

-  constructor(dims, columnarData, rowIndex = null, colIndex = null) {
+  constructor(
+    dims,
+    columnarData,
+    rowIndex = null,
+    colIndex = null,
+    __columnsAccessor = [] // private interface
+  ) {
    /*
    The base constructor is relatively hard to use - as an alternative,
    see factory methods and clone/slice, below.
@@ -74,6 +80,9 @@ class Dataframe {
        or TypedArray of length nRows.
      * rowIndex/colIndex - null (create default index using offsets as key),
        or a caller-provided index.
+      * __columnsAccessor - private interface, do not specify.  Used internally
+        to improve caching of column accessors when possible (eg, clone(), 
+        dropCol(), withCol()).
    All columns and indices must have appropriate dimensionality.
    */
    const [nRows, nCols] = dims;
@@ -94,7 +103,7 @@ class Dataframe {
    this.rowIndex = rowIndex;
    this.colIndex = colIndex;

-    this.__compile();
+    this.__compile(__columnsAccessor);
  }

  static __errorChecks(dims, columnarData, rowIndex, colIndex) {
@@ -135,97 +144,107 @@ class Dataframe {
    }
  }

-  __compile() {
+  static __compileColumn(column, getOffset, getLabel) {
+    /*
+      Each column accessor is a function which will lookup data by
+      index (ie, is equivalent to dataframe.get(row, col), where 'col'
+      is fixed.
+
+      In addition, each column accessor has several functions:
+
+      asArray() -- return the entire column as a native Array or TypedArray.
+        Crucially, this native array only supports label indexing.
+        Example:
+          const arr = df.col('a').asArray();
+
+      has(rlabel) -- return boolean indicating of the row label
+        is contained within the column.  Example:
+          const isInColumn = df.col('a').includes(99)
+        For the default offset indexing, this is identical to:
+          const isInColumn = (99 > 0) && (99 < df.nRows);
+
+      ihas(roffset) -- same as has(), but accepts a row offset
+        instead of a row label.
+
+      indexOf(value) -- return the label (not offset) of the first instance of
+        'value' in the column.  If you want the offset, just use the builtin JS
+        indexOf() function, available on both Array and TypedArray.
+
+      iget(offset) -- return the value at 'offset'
+
+    */
+    const { length } = column;
+
+    /* get value by row label */
+    const get = function get(rlabel) {
+      return column[getOffset(rlabel)];
+    };
+
+    /* get value by row offset */
+    const iget = function iget(roffset) {
+      return column[roffset];
+    };
+
+    /* full column array access */
+    const asArray = function asArray() {
+      return column;
+    };
+
+    /* test for row label inclusion in column */
+    const has = function has(rlabel) {
+      const offset = getOffset(rlabel);
+      return offset >= 0 && offset < length;
+    };
+
+    const ihas = function ihas(offset) {
+      return offset >= 0 && offset < length;
+    };
+
+    /*
+    return first label (index) at which the value is found in this column,
+    or undefined if not found.
+
+    NOTE: not found return is DIFFERENT than the default Array.indexOf as
+    -1 is a plausible Dataframe row/col label.
+    */
+    const indexOf = function indexOf(value) {
+      const offset = column.indexOf(value);
+      if (offset === -1) {
+        return undefined;
+      }
+      return getLabel(offset);
+    };
+
+    /*
+    Summarize the column data. Lazy eval;
+    */
+    const summarize = callOnceLazy(() =>
+      isTypedArray(column)
+        ? summarizeContinuous(column)
+        : summarizeCategorical(column)
+    );
+
+    get.summarize = summarize;
+    get.asArray = asArray;
+    get.has = has;
+    get.ihas = ihas;
+    get.indexOf = indexOf;
+    get.iget = iget;
+    return get;
+  }
+
+  __compile(accessors) {
    /*
    Compile data accessors for each column.

-    Each column accessor is a function which will lookup data by
-    index (ie, is equivalent to dataframe.get(row, col), where 'col'
-    is fixed.
-
-    In addition, each column accessor has several functions:
-
-    asArray() -- return the entire column as a native Array or TypedArray.
-      Crucially, this native array only supports label indexing.
-      Example:
-        const arr = df.col('a').asArray();
-
-    has(rlabel) -- return boolean indicating of the row label
-      is contained within the column.  Example:
-        const isInColumn = df.col('a').includes(99)
-      For the default offset indexing, this is identical to:
-        const isInColumn = (99 > 0) && (99 < df.nRows);
-
-    ihas(roffset) -- same as has(), but accepts a row offset
-      instead of a row label.
-
-    indexOf(value) -- return the label (not offset) of the first instance of
-      'value' in the column.  If you want the offset, just use the builtin JS
-      indexOf() function, available on both Array and TypedArray.
-
-    iget(offset) -- return the value at 'offset'
-
+    Use an existing accessor if provided, else compile a new one.
    */
    const { getOffset, getLabel } = this.rowIndex;
-    this.__columnsAccessor = this.__columns.map(column => {
-      const { length } = column;
-
-      /* get value by row label */
-      const get = function get(rlabel) {
-        return column[getOffset(rlabel)];
-      };
-
-      /* get value by row offset */
-      const iget = function iget(roffset) {
-        return column[roffset];
-      };
-
-      /* full column array access */
-      const asArray = function asArray() {
-        return column;
-      };
-
-      /* test for row label inclusion in column */
-      const has = function has(rlabel) {
-        const offset = getOffset(rlabel);
-        return offset >= 0 && offset < length;
-      };
-
-      const ihas = function ihas(offset) {
-        return offset >= 0 && offset < length;
-      };
-
-      /*
-      return first label (index) at which the value is found in this column,
-      or undefined if not found.
-
-      NOTE: not found return is DIFFERENT than the default Array.indexOf as
-      -1 is a plausible Dataframe row/col label.
-      */
-      const indexOf = function indexOf(value) {
-        const offset = column.indexOf(value);
-        if (offset === -1) {
-          return undefined;
-        }
-        return getLabel(offset);
-      };
-
-      /*
-      Summarize the column data. Lazy eval;
-      */
-      const summarize = callOnceLazy(() =>
-        isTypedArray(column)
-          ? summarizeContinuous(column)
-          : summarizeCategorical(column)
-      );
-
-      get.summarize = summarize;
-      get.asArray = asArray;
-      get.has = has;
-      get.ihas = ihas;
-      get.indexOf = indexOf;
-      get.iget = iget;
-      return get;
+    this.__columnsAccessor = this.__columns.map((column, idx) => {
+      if (accessors[idx]) {
+        return accessors[idx];
+      }
+      return Dataframe.__compileColumn(column, getOffset, getLabel);
    });
  }

@@ -237,7 +256,8 @@ class Dataframe {
      this.dims,
      [...this.__columns],
      this.rowIndex,
-      this.colIndex
+      this.colIndex,
+      [...this.__columnsAccessor]
    );
  }

@@ -273,7 +293,14 @@ class Dataframe {
    const columns = [...this.__columns];
    columns.push(colData);
    const colIndex = this.colIndex.withLabel(label);
-    return new this.constructor(dims, columns, rowIndex, colIndex);
+    const columnsAccessor = [...this.__columnsAccessor];
+    return new this.constructor(
+      dims,
+      columns,
+      rowIndex,
+      colIndex,
+      columnsAccessor
+    );
  }

  dropCol(label) {
@@ -287,7 +314,15 @@ class Dataframe {
    const columns = [...this.__columns];
    columns.splice(coffset, 1);
    const colIndex = this.colIndex.dropLabel(label);
-    return new this.constructor(dims, columns, this.rowIndex, colIndex);
+    const columnsAccessor = [...this.__columnsAccessor];
+    columnsAccessor.splice(coffset, 1);
+    return new this.constructor(
+      dims,
+      columns,
+      this.rowIndex,
+      colIndex,
+      columnsAccessor
+    );
  }

  static empty(rowIndex = null, colIndex = null) {
@@ -316,7 +351,7 @@ class Dataframe {
      if (!offsets) {
        return [null, null];
      }
-      const sortedOffsets = sort(offsets);
+      const sortedOffsets = sortArray(offsets);
      const sortedLabels = new Array(sortedOffsets.length);
      for (let i = 0, l = sortedOffsets.length; i < l; i += 1) {
        sortedLabels[i] = index.getLabel(sortedOffsets[i]);
@@ -543,14 +578,34 @@ class Dataframe {
  /****
  Functional (map/reduce/etc) data access

-  XXX: not yet implemented, as there is no clear use case.   Can easily
+  TODO: most are not yet implemented, as there is no clear use case.   Can easily
  add these as useful.
  ****/

+  mapColumns(callback) {
+    /*
+    map all columns in the dataframe, returning a new dataframe comprised of the
+    return values, with the same index as the original dataframe.
+
+    callback MUST not modify the column, but instead return a mutated copy.
+    */
+    const columns = this.__columns.map(callback);
+    const columnsAccessor = columns.map((c, idx) =>
+      this.__columns[idx] === c ? this.__columnsAccessor[idx] : undefined
+    );
+    return new this.constructor(
+      this.dims,
+      columns,
+      this.rowIndex,
+      this.colIndex,
+      columnsAccessor
+    );
+  }
+
  /*
  Map & reduce of column or row

-  XXX TODO remainder of map/reduce functions:  mapCol, mapRow, reduceRow, ...
+  TODO remainder of map/reduce functions:  mapCol, mapRow, reduceRow, ...
  */
  /* comment out until we have a use for this

@@ -3,6 +3,8 @@ Label indexing - map a label to & from an integer offset.  See Dataframe
 for how this is used.
 **/

+import { rangeFill as fillRange } from "../range";
+
 /*
 Private utility functions
 */
@@ -21,14 +23,6 @@ function extent(tarr) {
  return [min, max];
 }

-function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, l = larr.length; i < l; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 /* eslint-disable class-methods-use-this */
 class IdentityInt32Index {
  /*
@@ -1,32 +1,56 @@
 /*
 Private dataframe support functions
+
+TODO / XXX: for scalar/continuous data, this uses a naive method
+of computing quantiles.  Would be good to switch from sort to 
+partition at some point.
 */

+import quantile from "../quantile";
+import { sortArray } from "../typedCrossfilter/sort";
+
+// [ 0, 0.01, 0.02, ..., 1.0]
+const centileNames = new Array(101).fill(0).map((v, idx) => idx / 100);
+
 export function summarizeContinuous(col) {
  let min;
  let max;
  let nan = 0;
  let pinf = 0;
  let ninf = 0;
+  let percentiles;
  if (col) {
-    for (let r = 0, l = col.length; r < l; r += 1) {
-      const val = Number(col[r]);
-      if (Number.isFinite(val)) {
-        if (min === undefined) {
-          min = val;
-          max = val;
-        } else {
-          min = val < min ? val : min;
-          max = val > max ? val : max;
-        }
-      } else if (Number.isNaN(val)) {
-        nan += 1;
-      } else if (val > 0) {
-        pinf += 1;
-      } else {
-        ninf += 1;
+    // -Inf < finite < Inf < NaN
+    const sortedCol = sortArray(new col.constructor(col));
+
+    // count non-finites, which are at each end of sorted data
+    for (let i = sortedCol.length - 1; i >= 0; i -= 1) {
+      if (!Number.isNaN(sortedCol[i])) {
+        nan = sortedCol.length - i - 1;
+        break;
      }
    }
+    for (let i = 0, l = sortedCol.length; i < l; i += 1) {
+      if (sortedCol[i] !== Number.NEGATIVE_INFINITY) {
+        ninf = i;
+        break;
+      }
+    }
+    for (let i = sortedCol.length - nan - 1; i >= 0; i -= 1) {
+      if (sortedCol[i] !== Number.POSITIVE_INFINITY) {
+        pinf = sortedCol.length - i - nan - 1;
+        break;
+      }
+    }
+
+    // compute percentiles on finite data ONLY
+    const sortedColFiniteOnly = sortedCol.slice(
+      ninf,
+      sortedCol.length - nan - pinf
+    );
+    percentiles = quantile(centileNames, sortedColFiniteOnly, true);
+    min = percentiles[0];
+    max = percentiles[100];
  }
  return {
    categorical: false,
@@ -34,7 +58,8 @@ export function summarizeContinuous(col) {
    max,
    nan,
    pinf,
-    ninf
+    ninf,
+    percentiles
  };
 }

@@ -2,16 +2,7 @@
 Private utility code for dataframe
 */

-export function isTypedArray(x) {
-  return (
-    ArrayBuffer.isView(x) &&
-    Object.prototype.toString.call(x) !== "[object DataView]"
-  );
-}
-
-export function isArrayOrTypedArray(x) {
-  return Array.isArray(x) || isTypedArray(x);
-}
+export { isTypedArray, isArrayOrTypedArray } from "../typeHelpers";

 export function callOnceLazy(f) {
  let value;
@@ -0,0 +1,13 @@
+export default function fromEntries(arr) {
+	/*
+	Similar to Object.fromEntries, but only handles array.
+	This could be replaced with the standard fucnction once it
+	is widely available.   As of 3/20/2019, it has not yet
+	been released in the Chrome stable channel.
+	*/
+	const obj = {};
+	for (let i = 0, l = arr.length; i < l; i += 1) {
+		obj[arr[i][0]] = arr[i][1];
+	}
+	return obj;
+}
@@ -0,0 +1,29 @@
+/*
+quantiles - calculate quantiles for the typed array.
+
+Currently interpolates to 'lower' value.
+
+Arguments:
+
+	* quantArr - array of quantiles to compute, where values:  0 <= value <= 1.0
+	* tarr - a typed array
+	* sorted - option bool.  If false (default), will assume array is not sorted.
+	 	If true, will assume it is sorted.
+
+*/
+
+import { sortArray } from "./typedCrossfilter/sort";
+
+export default function quantile(quantArr, tarr, sorted = false) {
+	/*
+	start with the naive (sort) implementation.  Later, use a faster partition
+	*/
+	const arr = sorted ? tarr : sortArray(new tarr.constructor(tarr)); // copy
+	const len = arr.length;
+	return quantArr.map(q => {
+		if (q === 1) {
+			return arr[len - 1];
+		}
+		return arr[Math.floor(q * len)];
+	});
+}
@@ -0,0 +1,45 @@
+/*
+Array range creation
+
+range(start, stop, step) -> Array
+	This is identical to https://docs.python.org/3/library/functions.html#func-range
+	Returns new array filled with a range of numbers.
+
+	Usage:
+
+		range(stop) - start defaults to zero, step defaults to 1
+		range(start, stop, [step]) - step defaults to 1
+
+	Examples:
+		range(3) -> [0, 1, 2]
+		range(1, 3) -> [1, 2]
+		range(1, 5, 2) -> [1, 3]
+
+
+rangeFill(array, start, step) -> array
+	Fill entire array with values, from start, by step.  Returns first array.
+	start defaults to zero, step defaults to one.
+
+*/
+
+function _doFill(arr, start, step, count) {
+	for (let idx = 0, val = start; idx < count; idx += 1, val += step) {
+		arr[idx] = val;
+	}
+	return arr;
+}
+
+export function rangeFill(arr, start = 0, step = 1) {
+	return _doFill(arr, start, step, arr.length);
+}
+
+export function range(start, stop, step) {
+	if (start === undefined) return [];
+	if (stop === undefined) {
+		stop = start;
+		start = 0;
+	}
+	step = step || 1; // catch undefind and zero
+	const len = Math.max(Math.ceil((stop - start) / step), 0);
+	return _doFill(new Array(len), start, step, len);
+}
@@ -9,8 +9,14 @@
 // this is is equivalent to d3.scaleLinear().domain([0,1]).range([-1,1])

 export default (domain, range) => {
-  const domainStart = domain[0];
-  const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
-  const rangeStart = range[0];
-  return value => (value - domainStart) * scale + rangeStart;
+	const domainStart = domain[0];
+	const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+	const invScale = 1 / scale;
+	const rangeStart = range[0];
+	const f = value => (value - domainStart) * scale + rangeStart;
+
+	// inverter
+	f.invert = value => (value - rangeStart) * invScale + domainStart;
+
+	return f;
 };
@@ -0,0 +1,120 @@
+/*
+Helper functions for the embedded graph colors
+*/
+import * as d3 from "d3";
+import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
+import * as globals from "../../globals";
+import parseRGB from "../parseRGB";
+import finiteExtent from "../finiteExtent";
+import { range } from "../range";
+
+/*
+create new colors state object.   Paramters:
+  - world - current world object
+  - mode - color-by mode.  One of: null, "color by expression",
+    "color by continuous metadata", "color by categorical metadata"
+  -
+*/
+function createColors(world, colorMode = null, colorAccessor = null) {
+  switch (colorMode) {
+    case "color by categorical metadata": {
+      return createColorsByCategoricalMetadata(world, colorAccessor);
+    }
+    case "color by continuous metadata": {
+      return createColorsByContinuousMetadata(world, colorAccessor);
+    }
+    case "color by expression": {
+      return createColorsByExpression(world, colorAccessor);
+    }
+    default: {
+      const defaultCellColor = parseRGB(globals.defaultCellColor);
+      return {
+        rgb: new Array(world.nObs).fill(defaultCellColor),
+        scale: undefined
+      };
+    }
+  }
+}
+
+function createColorsByCategoricalMetadata(world, accessor) {
+  const { categories } = world.schema.annotations.obsByName[accessor];
+
+  const scale = d3
+    .scaleSequential(interpolateRainbow)
+    .domain([0, categories.length]);
+
+  /* pre-create colors - much faster than doing it for each obs */
+  const colors = categories.reduce((acc, cat, idx) => {
+    acc[cat] = parseRGB(scale(idx));
+    return acc;
+  }, {});
+
+  const rgb = new Array(world.nObs);
+  const data = world.obsAnnotations.col(accessor).asArray();
+  for (let i = 0, len = world.obsAnnotations.length; i < len; i += 1) {
+    const cat = data[i];
+    rgb[i] = colors[cat];
+  }
+  return { rgb, scale };
+}
+
+function createColorsByContinuousMetadata(world, accessor) {
+  const colorBins = 100;
+  const col = world.obsAnnotations.col(accessor);
+  const { min, max } = col.summarize();
+  const scale = d3
+    .scaleQuantile()
+    .domain([min, max])
+    .range(range(colorBins - 1, -1, -1));
+
+  /* pre-create colors - much faster than doing it for each obs */
+  const colors = new Array(colorBins);
+  for (let i = 0; i < colorBins; i += 1) {
+    colors[i] = parseRGB(interpolateCool(i / colorBins));
+  }
+
+  const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
+  const rgb = new Array(world.nObs);
+  const data = col.asArray();
+  for (let i = 0, len = world.obsAnnotations.length; i < len; i += 1) {
+    const val = data[i];
+    if (Number.isFinite(val)) {
+      const c = scale(val);
+      rgb[i] = colors[c];
+    } else {
+      rgb[i] = nonFiniteColor;
+    }
+  }
+  return { rgb, scale };
+}
+
+function createColorsByExpression(world, accessor) {
+  const expression = world.varData.col(accessor).asArray();
+  const colorBins = 100;
+  const [min, max] = finiteExtent(expression);
+  const scale = d3
+    .scaleQuantile()
+    .domain([min, max])
+    .range(range(colorBins - 1, -1, -1));
+
+  /* pre-create colors - much faster than doing it for each obs */
+  const colors = new Array(colorBins);
+  for (let i = 0; i < colorBins; i += 1) {
+    colors[i] = parseRGB(interpolateCool(i / colorBins));
+  }
+  const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
+
+  const rgb = new Array(world.nObs);
+  for (let i = 0, len = expression.length; i < len; i += 1) {
+    const e = expression[i];
+    if (Number.isFinite(e)) {
+      const c = scale(e);
+      rgb[i] = colors[c];
+    } else {
+      rgb[i] = nonFiniteColor;
+    }
+  }
+  return { rgb, scale };
+}
+
+export default createColors;
@@ -5,12 +5,11 @@ Helper functions for the controls reducer
 import _ from "lodash";

 import * as globals from "../../globals";
-import { fillRange } from "../typedCrossfilter/util";
+import { rangeFill as fillRange } from "../range";
 import {
  userDefinedDimensionName,
  diffexpDimensionName
 } from "../nameCreators";
-import * as World from "./world";

 /*
 Selection state for categoricals are tracked in an Object that
@@ -22,16 +21,16 @@ Remember that option values can be ANY js type, except undefined/null.
  {
    _category_name_1: {
      // map of option value to index
-      categoryIndices: Map([
+      categoryValueIndices: Map([
        catval1: index,
        ...
      ])

      // index->selection true/false state
-      categorySelected: [ true/false, true/false, ... ]
+      categoryValueSelected: [ true/false, true/false, ... ]

      // number of options
-      numCategories: number,
+      numCategoryValues: number,

      // isTruncated - true if the options for selection has
      // been truncated (ie, was too large to implement)
@@ -55,29 +54,33 @@ function topNCategories(summary) {
  return [sortedCategories.slice(0, N), sortedCounts.slice(0, N)];
 }

-export function createCategoricalSelectionState(state, world) {
+export function createCategoricalSelection(maxCategoryItems, world) {
  const res = {};
+  const obsIndexName = world.schema.annotations.obs.index;
  _.forEach(world.obsAnnotations.colIndex.keys(), key => {
    const summary = world.obsAnnotations.col(key).summarize();
    if (summary.categories) {
      const isColorField = key.includes("color") || key.includes("Color");
      const isSelectableCategory =
        !isColorField &&
-        key !== "name" &&
-        summary.categories.length < state.maxCategoryItems;
+        key !== obsIndexName &&
+        summary.categories.length < maxCategoryItems;
      if (isSelectableCategory) {
-        const [categoryValues, categoryCounts] = topNCategories(summary);
-        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
-        const numCategories = categoryIndices.size;
-        const categorySelected = new Array(numCategories).fill(true);
+        const [categoryValues, categoryValueCounts] = topNCategories(summary);
+        const categoryValueIndices = new Map(
+          categoryValues.map((v, i) => [v, i])
+        );
+        const numCategoryValues = categoryValueIndices.size;
+        const categoryValueSelected = new Array(numCategoryValues).fill(true);
        const isTruncated = categoryValues.length < summary.numCategories;
        res[key] = {
          categoryValues, // array: of natively typed category values
-          categoryIndices, // map: category value (native type) -> category index
-          categorySelected, // array: t/f selection state
-          numCategories, // number: of categories
+          categoryValueIndices, // map: category value (native type) -> category index
+          categoryValueSelected, // array: t/f selection state
+          numCategoryValues, // number: of values in the category
          isTruncated, // bool: true if list was truncated
-          categoryCounts // array: cardinality of each category
+          categoryValueCounts, // array: cardinality of each category,
+          categorySelected: true // bool - default state for entire category
        };
      }
    }
@@ -86,41 +89,61 @@ export function createCategoricalSelectionState(state, world) {
 }

 /*
-given a categoricalSelectionState, return the list of all category values
+given a categoricalSelection, return the list of all category values
 where selection state is true (ie, they are selected).
 */
-export function selectedValuesForCategory(categorySelectionState) {
-  const selectedValues = _([...categorySelectionState.categoryIndices])
-    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
-    .map(tuple => tuple[0])
-    .value();
-  return selectedValues;
+export function selectedValuesForCategory(categorySelectionState, dfColumn) {
+  const {
+    categorySelected,
+    categoryValueSelected,
+    categoryValueIndices
+  } = categorySelectionState;
+  let selectedValues;
+  if (categorySelected) {
+    selectedValues = new Set(dfColumn.summarize().categories);
+  } else {
+    selectedValues = new Set();
+  }
+  categoryValueIndices.forEach((catIndex, catValue) => {
+    if (!categoryValueSelected[catIndex]) {
+      selectedValues.delete(catValue);
+    } else {
+      selectedValues.add(catValue);
+    }
+  });
+  return [...selectedValues.values()];
 }

 /*
-build a crossfilter dimension map for all gene expression related dimensions.
+build a crossfilter dimensions for all gene expression related dimensions.
 */
-export function createGenesDimMap(
+export function createGeneDimensions(
  userDefinedGenes,
  diffexpGenes,
  world,
  crossfilter
 ) {
-  function _createGenesDimMap(genes, nameCreator) {
-    return genes.reduce((acc, gene) => {
-      acc[nameCreator(gene)] = World.createVarDataDimension(
-        world,
-        crossfilter,
-        gene
-      );
-      return acc;
-    }, {});
-  }
-
-  return {
-    ..._createGenesDimMap(userDefinedGenes, userDefinedDimensionName),
-    ..._createGenesDimMap(diffexpGenes, diffexpDimensionName)
-  };
+  crossfilter = userDefinedGenes.reduce(
+    (xflt, gene) =>
+      xflt.addDimension(
+        userDefinedDimensionName(gene),
+        "scalar",
+        world.varData.col(gene).asArray(),
+        Float32Array
+      ),
+    crossfilter
+  );
+  crossfilter = diffexpGenes.reduce(
+    (xflt, gene) =>
+      xflt.addDimension(
+        diffexpDimensionName(gene),
+        "scalar",
+        world.varData.col(gene).asArray(),
+        Float32Array
+      ),
+    crossfilter
+  );
+  return crossfilter;
 }

 export function pruneVarDataCache(varData, needed) {
@@ -14,7 +14,8 @@ This is all VERY tightly integrated with reducers and actions, and
 exists to support those concepts.
 */

+export { default as createColors } from "./colorHelpers";
 export * as Universe from "./universe";
 export * as World from "./world";
 export * as WorldUtil from "./worldUtil";
-export * as ControlsHelper from "./controlsHelpers";
+export * as ControlsHelpers from "./controlsHelpers";
@@ -4,6 +4,8 @@ import _ from "lodash";

 import decodeMatrixFBS from "./matrix";
 import * as Dataframe from "../dataframe";
+import fromEntries from "../fromEntries";
+import { isFpTypedArray } from "../typeHelpers";

 /*
 Private helper function - create and return a template Universe
@@ -37,14 +39,58 @@ These functions are used exclusively by the actions and reducers to
 build an internal POJO for use by the rendering components.
 */

+function promoteTypedArray(o) {
+  /*
+  Decide what internal data type to use for the data returned from 
+  the server.
+
+  TODO - future optimization: not all int32/uint32 data series require
+  promotion to float64.  We COULD simply look at the data to decide. 
+  */
+  if (isFpTypedArray(o) || Array.isArray(o)) return o;
+
+  let TyepdArrayCtor;
+  switch (o.constructor) {
+    case Int8Array:
+    case Uint8Array:
+    case Uint8ClampedArray:
+    case Int16Array:
+    case Uint16Array:
+      TyepdArrayCtor = Float32Array;
+      break;
+
+    case Int32Array:
+    case Uint32Array:
+      TyepdArrayCtor = Float64Array;
+      break;
+
+    default:
+      throw new Error("Unexpected data type returned from server.");
+  }
+  if (o.constructor === TyepdArrayCtor) return o;
+  return new TyepdArrayCtor(o);
+}
+
 function AnnotationsFBSToDataframe(arrayBuffer) {
  /*
  Convert a Matrix FBS to a Dataframe.
+
+  The application has strong assumptions that all scalar data will be
+  stored as a float32 or float64 (regardless of underlying data types).
+  For example, clipping of value ranges (eg, user-selected percentiles)
+  depends on the ability to use NaN in any numeric type.
+
+  All float data from the server is left as is.  All non-float is promoted
+  to an appropriate float.
  */
-  const fbs = decodeMatrixFBS(arrayBuffer);
+  const fbs = decodeMatrixFBS(arrayBuffer, true); // leave in place
+  const columns = fbs.columns.map(c => {
+    if (isFpTypedArray(c) || Array.isArray(c)) return c;
+    return promoteTypedArray(c);
+  });
  const df = new Dataframe.Dataframe(
    [fbs.nRows, fbs.nCols],
-    fbs.columns,
+    columns,
    null,
    new Dataframe.KeyIndex(fbs.colIdx)
  );
@@ -53,11 +99,16 @@ function AnnotationsFBSToDataframe(arrayBuffer) {

 function LayoutFBSToDataframe(arrayBuffer) {
  const fbs = decodeMatrixFBS(arrayBuffer, true);
+  if (fbs.columns.length < 2 || !fbs.columns.every(isFpTypedArray)) {
+    // We have strong assumptions about the shape & type of layout data.
+    throw new Error("Unexpected layout data type returned from server");
+  }
+
  const df = new Dataframe.Dataframe(
    [fbs.nRows, fbs.nCols],
    fbs.columns,
    null,
-    new Dataframe.KeyIndex(["X", "Y"])
+    new Dataframe.KeyIndex(fbs.colIdx)
  );
  return df;
 }
@@ -73,15 +124,15 @@ function reconcileSchemaCategoriesWithSummary(universe) {
  cases, add a 'categories' field to the schema so it is accessible.
  */

-  _.forEach(universe.schema.annotations.obs, s => {
+  universe.schema.annotations.obs.columns.forEach(s => {
    if (
      s.type === "string" ||
      s.type === "boolean" ||
      s.type === "categorical"
    ) {
      const categories = _.union(
-        _.get(s, "categories", []),
-        _.get(universe.obsAnnotations.col(s.name).summarize(), "categories", [])
+        s.categories ?? [],
+        universe.obsAnnotations.col(s.name).summarize().categories ?? []
      );
      s.categories = categories;
    }
@@ -105,6 +156,9 @@ export function createUniverseFromResponse(
  universe.schema = schema;
  universe.nObs = schema.dataframe.nObs;
  universe.nVar = schema.dataframe.nVar;
+  /* add defaults, as we can't assume back-end will fully populate schema */
+  if (!schema.layout.var) schema.layout.var = [];
+  if (!schema.layout.obs) schema.layout.obs = [];

  /* annotations */
  universe.obsAnnotations = AnnotationsFBSToDataframe(annotationsObsResponse);
@@ -122,6 +176,20 @@ export function createUniverseFromResponse(
  }

  reconcileSchemaCategoriesWithSummary(universe);
+
+  /* Index schema for ease of use */
+  universe.schema.annotations.obsByName = fromEntries(
+    universe.schema.annotations.obs.columns.map(v => [v.name, v])
+  );
+  universe.schema.annotations.varByName = fromEntries(
+    universe.schema.annotations.var.columns.map(v => [v.name, v])
+  );
+  universe.schema.layout.obsByName = fromEntries(
+    universe.schema.layout.obs.map(v => [v.name, v])
+  );
+  universe.schema.layout.varByName = fromEntries(
+    universe.schema.layout.var.map(v => [v.name, v])
+  );
  return universe;
 }

@@ -140,8 +208,14 @@ export function convertDataFBStoObject(universe, arrayBuffer) {
  const { colIdx, columns } = fbs;
  const result = {};

+  if (!columns.every(isFpTypedArray)) {
+    // We have strong assumptions that all var data is float
+    throw new Error("Unexpected non-floating point response from server.");
+  }
+
+  const varIndexName = universe.schema.annotations.var.index;
  for (let c = 0; c < colIdx.length; c += 1) {
-    const varName = universe.varAnnotations.at(colIdx[c], "name");
+    const varName = universe.varAnnotations.at(colIdx[c], varIndexName);
    result[varName] = columns[c];
  }
  return result;
@@ -1,9 +1,14 @@
 // jshint esversion: 6

-import _ from "lodash";
-import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
-import Crossfilter from "../typedCrossfilter";
+import clip from "../clip";
+import {
+  layoutDimensionName,
+  obsAnnoDimensionName,
+  diffexpDimensionName,
+  userDefinedDimensionName
+} from "../nameCreators";
 import * as Dataframe from "../dataframe";
+import ImmutableTypedCrossfilter from "../typedCrossfilter/crossfilter";

 /*

@@ -21,6 +26,8 @@ Notable keys in the world object:

 * schema: data schema from the server

+* clipQuantiles: the quantiles used to clip all data in world.
+
 * obsAnnotations:

  Dataframe containing obs annotations.  Columns are indexed by annotation
@@ -39,78 +46,192 @@ Notable keys in the world object:
 * varData: a cache of expression columns, stored in a Dataframe.  Cache
  managed by controls reducer.

+* unclipped: will contain unclipped variants of all potentiall clipped
+  dataframes (obsAnnotations, varData).
+
 */

 function templateWorld() {
+  const obsAnnotations = Dataframe.Dataframe.empty();
+  const varAnnotations = Dataframe.Dataframe.empty();
+  const obsLayout = Dataframe.Dataframe.empty();
+  const varData = Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex());
  return {
    /* schema/version related */
    schema: null,
    nObs: 0,
    nVar: 0,
+    clipQuantiles: { min: 0, max: 1 },

    /* annotations */
-    obsAnnotations: Dataframe.Dataframe.empty(),
-    varAnnotations: Dataframe.Dataframe.empty(),
+    obsAnnotations,
+    varAnnotations,

    /* layout of graph. Dataframe. */
-    obsLayout: Dataframe.Dataframe.empty(),
+    obsLayout,

-    /*
-    Var data columns - subset of all data (may be empty)
-    */
-    varData: Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex())
+    /* Var data columns - subset of all data (may be empty) */
+    varData,
+
+    /* unclipped dataframes - subset, but not value clipped */
+    unclipped: {
+      obsAnnotations,
+      varData
+    }
  };
 }

+function clipDataframe(
+  df,
+  lowerQuantile,
+  upperQuantile,
+  quantileF,
+  clipPredicate = () => true,
+  value = Number.NaN
+) {
+  /*
+  For all columns in the dataframe, clip all values above or below specified 
+  quantiles to `value` if clipPredicate returns True for that column (if it
+  returns false, skip the column entirely).
+
+  Returns a clipped copy - does not mutate original.
+
+  clipPredicate must have signature: (dataframe, colIndex, colLabel) => boolean
+  True signifies that the column should be clipped; false indicates that the
+  column should be left intact/unchanged.
+
+  quantileF must have signature:  (label, qval) => number
+  */
+  if (lowerQuantile < 0) lowerQuantile = 0;
+  if (upperQuantile > 1) upperQuantile = 1;
+  if (lowerQuantile === 0 && upperQuantile === 1) return df;
+
+  const keys = df.colIndex.keys();
+  return df.mapColumns((col, colIdx) => {
+    const colLabel = keys[colIdx];
+    if (!clipPredicate(df, colIdx, colLabel)) return col;
+
+    const colMin = quantileF(colLabel, lowerQuantile);
+    const colMax = quantileF(colLabel, upperQuantile);
+    const newCol = clip(col.slice(), colMin, colMax, value);
+    return newCol;
+  });
+}
+
+/*
+Create World with contents eq entire universe.   Commonly used to initialize World.
+If clipQuantiles
+*/
 export function createWorldFromEntireUniverse(universe) {
  const world = templateWorld();

-  /*
-  public interface follows
-  */
-
  /* Schema related */
  world.schema = universe.schema;
  world.nObs = universe.nObs;
  world.nVar = universe.nVar;
+  world.clipQuantiles = { min: 0, max: 1 };

-  /* annotation dataframes */
-  world.obsAnnotations = universe.obsAnnotations;
-  world.varAnnotations = universe.varAnnotations;
+  /* dataframes: annotations and layout */
+  world.obsAnnotations = universe.obsAnnotations.clone();
+  world.varAnnotations = universe.varAnnotations.clone();
+  world.obsLayout = universe.obsLayout.clone();

-  /* layout and display characteristics dataframe */
-  world.obsLayout = universe.obsLayout;
-
-  /*
-  Var data columns - subset of all
-  */
+  /* Var dataframe - contains a subset of all var columns */
  world.varData = universe.varData.clone();

+  /* save unclipped copies of potentially clipped dataframes */
+  world.unclipped = {
+    obsAnnotations: world.obsAnnotations.clone(),
+    varData: world.varData.clone()
+  };
+
  return world;
 }

-export function createWorldFromCurrentSelection(universe, world, crossfilter) {
-  const newWorld = templateWorld();
+/*
+clip dataframes based on quantiles.

-  /* these don't change as only OBS are selected in our current implementation */
-  newWorld.nVar = universe.nVar;
-  newWorld.schema = universe.schema;
-  newWorld.varAnnotations = universe.varAnnotations;
+This is an in-place operation on the world object provided as an argument.
+The values in world.unclipped are clipped and assigned to world.obsAnnotations
+and world.varData.
+*/
+function setClippedDataframes(world) {
+  const { schema } = world;
+  const isContinuousObsAnnotation = (df, idx, label) =>
+    deduceDimensionType(schema.annotations.obsByName[label], label) !== "enum";
+  const obsQuantile = (label, q) =>
+    world.unclipped.obsAnnotations.col(label).summarize().percentiles[100 * q];
+  world.obsAnnotations = clipDataframe(
+    world.unclipped.obsAnnotations,
+    world.clipQuantiles.min,
+    world.clipQuantiles.max,
+    obsQuantile,
+    isContinuousObsAnnotation
+  );

-  /* now subset/cut obs */
-  const mask = crossfilter.allFilteredMask();
-  newWorld.obsAnnotations = world.obsAnnotations.isubsetMask(mask);
+  const varDataQuantile = (label, q) =>
+    world.unclipped.varData.col(label).summarize().percentiles[100 * q];
+  world.varData = clipDataframe(
+    world.unclipped.varData,
+    world.clipQuantiles.min,
+    world.clipQuantiles.max,
+    varDataQuantile,
+    () => true
+  );
+}
+
+/*
+Subset the current world based upon the current selection, maintaining any existing
+clip.  Returns new world.  Parameters:
+  * unvierse
+  * world - the current world
+  * crossfilter - the selection state
+*/
+export function createWorldBySelection(universe, world, crossfilter) {
+  const newWorld = { ...world, obsLayout: null, unclipped: {}, varData: null };
+
+  /* subset unclipped dataframes based upon current selection */
+  const mask = crossfilter.allSelectedMask();
  newWorld.obsLayout = world.obsLayout.isubsetMask(mask);
-  newWorld.nObs = newWorld.obsAnnotations.dims[0];
-
-  /*
-  Var data columns - subset of all
-  */
-  if (world.varData.isEmpty()) {
-    newWorld.varData = world.varData.clone();
+  newWorld.unclipped.obsAnnotations = world.unclipped.obsAnnotations.isubsetMask(
+    mask
+  );
+  if (world.unclipped.varData.isEmpty()) {
+    newWorld.unclipped.varData = world.unclipped.varData.clone();
  } else {
-    newWorld.varData = world.varData.isubsetMask(mask);
+    newWorld.unclipped.varData = world.unclipped.varData.isubsetMask(mask);
  }
+  /* subsetting changings dimension size */
+  newWorld.nObs = newWorld.unclipped.obsAnnotations.dims[0];
+
+  /* and now clip */
+  setClippedDataframes(newWorld);
+  return newWorld;
+}
+
+/*
+Change clip quantiles on the current world, returning a new world.
+Parameters:
+  * universe
+  * world - current world
+  * clipQuantiles - new clip
+*/
+export function createWorldWithNewClip(
+  universe,
+  world,
+  crossfilter,
+  clipQuantiles
+) {
+  const newWorld = { ...world, obsAnnotation: null, varData: null };
+  newWorld.clipQuantiles = clipQuantiles;
+  newWorld.obsLayout = world.obsLayout.clone();
+  newWorld.unclipped = {
+    obsAnnotations: world.unclipped.obsAnnotations.clone(),
+    varData: world.unclipped.varData.clone()
+  };
+
+  /* and now clip */
+  setClippedDataframes(newWorld);
  return newWorld;
 }

@@ -139,74 +260,50 @@ function deduceDimensionType(attributes, fieldName) {
  return dimensionType;
 }

-/*
-  Return a crossfilter dimension for the specified world & named gene.
-
-  NOTE: this assumes that the expression data was already loaded,
-  by calling an appropriate action creator.
-
-  Caller needs to *save* this dimension somewhere for it to be later used.
-  Dimension must be destroyed by calling dimension.dispose()
-  when it is no longer needed
-  (it will not be garbage collected without this call)
-*/
-export function createVarDataDimension(world, crossfilter, name) {
-  return crossfilter.dimension(
-    Crossfilter.ScalarDimension,
-    world.varData.col(name).asArray(),
-    Float32Array
-  );
-}
-
-export function createObsDimensionMap(crossfilter, world) {
+export function createObsDimensions(crossfilter, world, XYdimNames) {
  /*
-  create and return a crossfilter dimension for every obs annotation
-  for which we have a supported type.
+  create and return a crossfilter with a dimension for every obs annotation
+  for which we have a supported type, *except* for the index column, indicated
+  by schema.annotations.obs.index.
  */
  const { schema, obsLayout, obsAnnotations } = world;
-
-  // Create a crossfilter dimension for all obs annotations *except* 'name'
-  const dimensionMap = _(schema.annotations.obs)
-    .filter(anno => anno.name !== "name")
-    .transform((result, anno) => {
-      const dimType = deduceDimensionType(anno, anno.name);
-      const colData = obsAnnotations.col(anno.name).asArray();
-      if (dimType === "enum") {
-        result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
-          Crossfilter.EnumDimension,
-          colData
-        );
-      } else if (dimType) {
-        result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
-          Crossfilter.ScalarDimension,
-          colData,
-          dimType
-        );
-      } // else ignore the annotation
-    }, {})
-    .value();
-
-  /*
-  Add crossfilter dimensions allowing filtering on layout
-  */
-  dimensionMap[layoutDimensionName("XY")] = crossfilter.dimension(
-    Crossfilter.SpatialDimension,
-    obsLayout.col("X").asArray(),
-    obsLayout.col("Y").asArray()
+  const indexName = schema.annotations.obs.index;
+  const annoList = schema.annotations.obs.columns.filter(
+    anno => anno.name !== indexName
  );
+  crossfilter = annoList.reduce((xfltr, anno) => {
+    const dimType = deduceDimensionType(anno, anno.name);
+    const colData = obsAnnotations.col(anno.name).asArray();
+    const name = obsAnnoDimensionName(anno.name);
+    if (dimType === "enum") {
+      return xfltr.addDimension(name, "enum", colData);
+    }
+    if (dimType) {
+      return xfltr.addDimension(name, "scalar", colData, dimType);
+    }
+    return xfltr;
+  }, crossfilter);

-  return dimensionMap;
+  return crossfilter.addDimension(
+    layoutDimensionName("XY"),
+    "spatial",
+    obsLayout.col(XYdimNames[0]).asArray(),
+    obsLayout.col(XYdimNames[1]).asArray()
+  );
 }

 export function worldEqUniverse(world, universe) {
-  return world.obsAnnotations === universe.obsAnnotations;
+  return (
+    world.obsAnnotations === universe.obsAnnotations ||
+    world.obsAnnotations.rowIndex === universe.obsAnnotations.rowIndex
+  );
 }

 export function getSelectedByIndex(crossfilter) {
  /*
  return array of obsIndex, containing all selected obs/cells.
  */
-  const selected = crossfilter.allFilteredMask(); // array of bool-ish
+  const selected = crossfilter.allSelectedMask(); // array of bool-ish
  const keys = crossfilter.data.rowIndex.keys(); // row keys, aka universe rowIndex

  const set = new Int32Array(selected.length);
@@ -0,0 +1,95 @@
+/*
+Very simple FSM for use in reducer, etc.
+
+To create a state machine:
+  new StateMachine(initialState, transitions, onErrorCallback) -> statemachine
+
+Where:
+  * initialState - a caller-specified value that represents the initial state of
+    the FSM.
+  * transitions - an array of objects, representing FSM transitions (graph edges),
+    having the form:
+      {
+        to: state_name_transitioning_to,
+        from: state_name_transitioning_from,
+        event: value_that_will_cause_transition,
+        action: optional_callback_upon_transition
+      }
+    The transition will be provided to the action callback, so other data
+    may be stored in the transition object for use by the action callback.
+  * onErrorCallback - a callback function called if the FSM receives an event
+    for which it has no defined transition.
+
+
+Interface:
+  * states - property containing the state names. A Set(), contianing the
+    union of to: and from: values.
+  * events - property containing all of the accepted event values.  Set().
+  * graph - a Map of Maps, organized as  graph[eventValue][fromStateValue]
+  * clone() - clone the entire statemachine.
+  * next(eventValue) - drive the FSM to the next state.   If the event
+    matches a transition with a defined action, the action callback is
+    called, and the action return value is returned by next().  If no
+    transition is defined, onErrorCallback is called.
+
+Example:
+
+  const transitions = [
+    { from: "A", to: "B", event: "yo", action: () => 42 }
+  ];
+  const fsm = new StateMachine("A", transitions, () => { throw new Error("oops") });
+  fsm.next("yo");   // returns 42
+
+
+*/
+export default class StateMachine {
+  constructor(initState, transitions, onError) {
+    this.onError = onError || (() => undefined);
+    this.state = initState;
+
+    // all states
+    this.states = new Set(
+      transitions.reduce((names, tsn) => {
+        names.push(tsn.from);
+        names.push(tsn.to);
+        return names;
+      }, [])
+    );
+
+    // all transition names (aka events)
+    this.events = new Set(transitions.map(tsn => tsn.event));
+
+    // the transition graph.
+    // graph[event][from] -> transition
+    this.graph = transitions.reduce((graph, tsn) => {
+      const { event, from } = tsn;
+      if (!graph.has(event)) graph.set(event, new Map());
+      const tsnMap = graph.get(event);
+      tsnMap.set(from, tsn);
+      return graph;
+    }, new Map());
+  }
+
+  clone(initState) {
+    const fsm = new StateMachine(initState, []);
+    fsm.onError = this.onError;
+    fsm.states = this.states;
+    fsm.events = this.events;
+    fsm.graph = this.graph;
+    return fsm;
+  }
+
+  next(event, data) {
+    const { graph, state } = this;
+    const tsnMap = graph.get(event);
+    if (!tsnMap) return this.onError(this, event, state, undefined);
+
+    const transition = tsnMap.get(state);
+    if (!transition) return this.onError(this, event, state, undefined);
+
+    this.state = transition.to;
+    return transition.action
+      ? transition.action(this, transition, data)
+      : undefined;
+  }
+}
@@ -0,0 +1,29 @@
+/*
+Various type and schema related helper functions.
+*/
+
+/*
+Utility function to test for a typed array
+*/
+export function isTypedArray(x) {
+	return (
+		ArrayBuffer.isView(x) &&
+		Object.prototype.toString.call(x) !== "[object DataView]"
+	);
+}
+
+/*
+Test for float typed array, ie, Float32TypedArray or Float64TypedArray
+*/
+export function isFpTypedArray(x) {
+	let constructor;
+	const isFloatArray =
+		x &&
+		({ constructor } = x) &&
+		(constructor === Float32Array || constructor === Float64Array);
+	return isFloatArray;
+}
+
+export function isArrayOrTypedArray(x) {
+	return Array.isArray(x) || isTypedArray(x);
+}
@@ -211,6 +211,19 @@ class BitArray {
    }
  }

+  // select range of indices on a dimension
+  //
+  selectFromRange(dim, range) {
+    const col = dim >>> 5;
+    const first = range[0];
+    const last = range[1];
+    const one = 1 << dim % 32;
+    const offset = col * this.length;
+    for (let i = first; i < last; i += 1) {
+      this.bitarray[offset + i] |= one;
+    }
+  }
+
  // select range of indices on a dimension, indirect through a sort map.
  // Indirect functions are used to map between sort and natural order.
  //
@@ -225,6 +238,19 @@ class BitArray {
    }
  }

+  // deselect range of indices on a dimension
+  //
+  deselectFromRange(dim, range) {
+    const col = dim >>> 5;
+    const first = range[0];
+    const last = range[1];
+    const zero = ~(1 << dim % 32);
+    const offset = col * this.length;
+    for (let i = first; i < last; i += 1) {
+      this.bitarray[offset + i] &= zero;
+    }
+  }
+
  // deselect range of indices on a dimension, indirect through a sort map.
  //
  deselectIndirectFromRange(dim, indirect, range) {
@@ -0,0 +1,598 @@
+import { polygonContains } from "d3";
+
+import PositiveIntervals from "./positiveIntervals";
+import BitArray from "./bitArray";
+import {
+  sortArray,
+  lowerBound,
+  lowerBoundIndirect,
+  upperBoundIndirect
+} from "./sort";
+import { makeSortIndex } from "./util";
+
+class NotImplementedError extends Error {
+  constructor(...params) {
+    super(...params);
+
+    // Maintains proper stack trace for where our error was thrown (only available on V8)
+    if (Error.captureStackTrace) {
+      Error.captureStackTrace(this, NotImplementedError);
+    }
+  }
+}
+
+export default class ImmutableTypedCrossfilter {
+  constructor(data, dimensions = {}, selectionCache = null) {
+    /*
+    Typically, parameter 'data' is one of:
+      - Array of objects/records
+      - Dataframe (util/dataframe)
+    Other parameters are only used internally.
+
+    Object field description:
+    - data: reference to the array of records in the crossfilter
+    - selectionBitArray: bit array containing the flatted selection state
+      of all dimensions.  This is lazily created and is effectively
+      a perfomance cache.  Methods which return a new crossfilter,
+      such as select(), addDimention() and delDimension(), will pass
+      the cache forward to the new object, as the typical "immutable API"
+      usage pattern is to retain the new crossfilter and discard the old.
+    - dimensions: contains each dimension and its current state:
+        - id: bit offset in the cached bit array
+        - dim: the dimension object
+        - name: the dimension name
+        - selection: the dimension's current selection
+    */
+    this.data = data;
+    this.selectionCache = selectionCache; /* BitArray */
+    this.dimensions = dimensions; /* name: { id, dim, name, selection } */
+  }
+
+  size() {
+    return this.data.length;
+  }
+
+  all() {
+    return this.data;
+  }
+
+  dimensionNames() {
+    /* return array of all dimensions (by name) */
+    return Object.keys(this.dimensions);
+  }
+
+  hasDimension(name) {
+    return !!this.dimensions[name];
+  }
+
+  addDimension(name, type, ...rest) {
+    /*
+    Add a new dimension to this crossfilter, of type DimensionType.
+    Remainder of parameters are dimension-type-specific.
+    */
+    const { data, selectionCache } = this;
+
+    if (this.dimensions[name] !== undefined) {
+      throw new Error(`Adding duplicate dimension name ${name}`);
+    }
+
+    this.selectionCache = null; // pass ownership to new crossfilter
+
+    let id;
+    if (selectionCache) {
+      id = selectionCache.allocDimension();
+      selectionCache.selectAll(id);
+    }
+    const DimensionType = DimTypes[type];
+    const dim = new DimensionType(name, data, ...rest);
+    const dimensions = {
+      ...this.dimensions,
+      [name]: {
+        id,
+        dim,
+        name,
+        selection: dim.select({ mode: "all" })
+      }
+    };
+    return new ImmutableTypedCrossfilter(data, dimensions, selectionCache);
+  }
+
+  delDimension(name) {
+    const { data, selectionCache } = this;
+    const dimensions = { ...this.dimensions };
+    if (dimensions[name] === undefined) {
+      throw new ReferenceError(`Unable to delete unknown dimension ${name}`);
+    }
+
+    const { id } = dimensions[name];
+    delete dimensions[name];
+    this.selectionCache = null; // pass ownership to new crossfilter
+    if (selectionCache) {
+      selectionCache.freeDimension(id);
+    }
+    return new ImmutableTypedCrossfilter(data, dimensions, selectionCache);
+  }
+
+  select(name, spec) {
+    /*
+    select on named dimension, as indicated by `spec`.   Spec is an object
+    specifying the selection, and must contain at least a `mode` field.
+
+    Examples:
+      select("foo", {mode: "all"});
+      select("bar", {mode: "none"});
+      select("mumble", {mode: "exact", values: "blue"});
+      select("mumble", {mode: "exact", values: ["red", "green", "blue"]});
+      select("blort", {mode: "range", lo: 0, hi: 999.99});
+    */
+    const { data, selectionCache } = this;
+    this.selectionCache = null;
+    const dimensions = { ...this.dimensions };
+    const { dim, id, selection: oldSelection } = dimensions[name];
+    const newSelection = dim.select(spec);
+    newSelection.ranges = PositiveIntervals.canonicalize(newSelection.ranges);
+    dimensions[name] = { id, dim, name, selection: newSelection };
+    ImmutableTypedCrossfilter._dimSelnHasUpdated(
+      selectionCache,
+      id,
+      newSelection,
+      oldSelection
+    );
+    return new ImmutableTypedCrossfilter(data, dimensions, selectionCache);
+  }
+
+  static _dimSelnHasUpdated(selectionCache, id, newSeln, oldSeln) {
+    /*
+    Selection has updated from oldSeln to newSeln.   Update the
+    bit array if it exists.  If not, we will lazy create it when
+    needed.
+    */
+    if (selectionCache) {
+      /*
+      if both new and old selection use the same index, we can
+      perform an incremental update. If the index changed, we have
+      to do a suboptimal full deselect/select.
+      */
+      let adds;
+      let dels;
+      if (newSeln.index === oldSeln.index) {
+        adds = PositiveIntervals.difference(newSeln.ranges, oldSeln.ranges);
+        dels = PositiveIntervals.difference(oldSeln.ranges, newSeln.ranges);
+      } else {
+        // console.log("suboptimal selection update - index changed");
+        adds = newSeln.ranges;
+        dels = oldSeln.ranges;
+      }
+
+      /*
+      allow dimensions to return selected ranges in either dimension sort
+      order (indirect via index), or in original record order.
+
+      If sort index exists in the dimension, assume sort ordered ranges.
+      */
+      if (oldSeln.index) {
+        dels.forEach(interval =>
+          selectionCache.deselectIndirectFromRange(id, oldSeln.index, interval)
+        );
+      } else {
+        dels.forEach(interval =>
+          selectionCache.deselectFromRange(id, interval)
+        );
+      }
+
+      if (newSeln.index) {
+        adds.forEach(interval =>
+          selectionCache.selectIndirectFromRange(id, newSeln.index, interval)
+        );
+      } else {
+        adds.forEach(interval => selectionCache.selectFromRange(id, interval));
+      }
+    }
+  }
+
+  _getSelectionCache() {
+    if (!this.selectionCache) {
+      // console.log("...rebuilding crossfilter cache...");
+      const selectionCache = new BitArray(this.data.length);
+      Object.keys(this.dimensions).forEach(name => {
+        const { selection } = this.dimensions[name];
+        const id = selectionCache.allocDimension();
+        this.dimensions[name].id = id;
+        const { ranges, index } = selection;
+        ranges.forEach(range => {
+          if (index) {
+            selectionCache.selectIndirectFromRange(id, index, range);
+          } else {
+            selectionCache.selectFromRange(id, range);
+          }
+        });
+      });
+      this.selectionCache = selectionCache;
+    }
+    return this.selectionCache;
+  }
+
+  allSelected() {
+    /*
+    return array of all records currently selected by all dimensions
+    */
+    const selectionCache = this._getSelectionCache();
+    const { data } = this;
+    if (Array.isArray(data)) {
+      const res = [];
+      for (let i = 0, len = data.length; i < len; i += 1) {
+        if (selectionCache.isSelected(i)) {
+          res.push(data[i]);
+        }
+      }
+      return res;
+    }
+    /* else, Dataframe-like */
+    return data.isubsetMask(this.allSelectedMask());
+  }
+
+  allSelectedMask() {
+    /*
+    return Uint8Array containing selection state (truthy/falsey) for each record.
+    */
+    const selectionCache = this._getSelectionCache();
+    return selectionCache.fillBySelection(
+      new Uint8Array(this.data.length),
+      1,
+      0
+    );
+  }
+
+  countSelected() {
+    /*
+    return number of records selected on all dimensions
+    */
+    const selectionCache = this._getSelectionCache();
+    return selectionCache.selectionCount();
+  }
+
+  isElementSelected(i) {
+    /*
+    return truthy/falsey if this record is selected on all dimensions
+    */
+    const selectionCache = this._getSelectionCache();
+    return selectionCache.isSelected(i);
+  }
+
+  fillByIsSelected(array, selectedValue, deselectedValue) {
+    /*
+    fill array with one of two values, based upon selection state.
+    */
+    const selectionCache = this._getSelectionCache();
+    return selectionCache.fillBySelection(
+      array,
+      selectedValue,
+      deselectedValue
+    );
+  }
+}
+
+/*
+Base dimension object.
+
+A Dimension is an index, accessed via a select() method.  The protocol
+for a dimension:
+  - constructor - first param is name, remainder is whatever params are
+    required to initialize the dimension.
+  - select - one and only param is the selection specifier.  Returns an
+    array of record IDs.
+  - name - the dimension name/label.
+*/
+class _ImmutableBaseDimension {
+  constructor(name) {
+    this.name = name;
+  }
+
+  select(spec) {
+    const { mode } = spec;
+    if (mode === undefined) {
+      throw new Error("select spec does not contain 'mode'");
+    }
+    throw new Error(
+      `select mode ${mode} not implemented by dimension ${this.name}`
+    );
+  }
+}
+
+class ImmutableScalarDimension extends _ImmutableBaseDimension {
+  constructor(name, data, value, ValueArrayType) {
+    super(name);
+
+    // Three modes - caller can provide a pre-created value array,
+    // a map function which will create it, or another array which
+    // will used with an identity map function.
+    let array;
+    if (value instanceof ValueArrayType) {
+      // user has provided the final typed array - just use it
+      if (value.length !== data.length) {
+        throw new RangeError(
+          "ScalarDimension values length must equal crossfilter data record count"
+        );
+      }
+      array = value;
+    } else if (value instanceof Function) {
+      // Create value array from user-provided map function.
+      array = this._createValueArray(
+        data,
+        value,
+        new ValueArrayType(data.length)
+      );
+    } else if (isArrayOrTypedArray(value)) {
+      // Create value array from user-provided array.  Typically used
+      // only by enumerated dimensions
+      array = this._createValueArray(
+        data,
+        i => value[i],
+        new ValueArrayType(data.length)
+      );
+    } else {
+      throw new NotImplementedError(
+        "dimension value must be function or value array type"
+      );
+    }
+    this.value = array;
+
+    // create sort index
+    this.index = makeSortIndex(array);
+  }
+
+  /* eslint-disable class-methods-use-this */
+  _createValueArray(data, mapf, array) {
+    // create dimension value array
+    const len = data.length;
+    const larray = array;
+    for (let i = 0; i < len; i += 1) {
+      larray[i] = mapf(i, data);
+    }
+    return larray;
+  }
+  /* eslint-enable class-methods-use-this */
+
+  select(spec) {
+    const { mode } = spec;
+    const { index } = this;
+    switch (mode) {
+      case "all":
+        return { ranges: [[0, this.value.length]], index };
+      case "none":
+        return { ranges: [], index };
+      case "exact":
+        return this.selectExact(spec);
+      case "range":
+        return this.selectRange(spec);
+      default:
+        return super.select(spec);
+    }
+  }
+
+  selectExact(spec) {
+    const { value, index } = this;
+    let { values } = spec;
+    if (!Array.isArray(values)) {
+      values = [values];
+    }
+    const ranges = [];
+    for (let v = 0, len = values.length; v < len; v += 1) {
+      const r = [
+        lowerBoundIndirect(value, index, values[v], 0, value.length),
+        upperBoundIndirect(value, index, values[v], 0, value.length)
+      ];
+      if (r[0] <= r[1]) {
+        ranges.push(r);
+      }
+    }
+    return { ranges, index };
+  }
+
+  selectRange(spec) {
+    const { value, index } = this;
+    /* [lo, hi) */
+    const { lo, hi } = spec;
+    const ranges = [];
+    const r = [
+      lowerBoundIndirect(value, index, lo, 0, value.length),
+      lowerBoundIndirect(value, index, hi, 0, value.length)
+    ];
+    if (r[0] < r[1]) ranges.push(r);
+    return { ranges, index };
+  }
+}
+
+class ImmutableEnumDimension extends ImmutableScalarDimension {
+  constructor(name, data, value) {
+    super(name, data, value, Uint32Array);
+  }
+
+  _createValueArray(data, mapf, array) {
+    const len = data.length;
+    const larray = array;
+
+    // create enumeration table - mapping between the value
+    // and the enum.
+    const s = new Set();
+    for (let i = 0; i < len; i += 1) {
+      s.add(mapf(i, data));
+    }
+    const enumIndex = sortArray(Array.from(s));
+    this.enumIndex = enumIndex;
+
+    // create dimension value array
+    const enumLen = enumIndex.length;
+    for (let i = 0; i < len; i += 1) {
+      const v = mapf(i, data);
+      const e = lowerBound(enumIndex, v, 0, enumLen);
+      larray[i] = e;
+    }
+    return larray;
+  }
+
+  selectExact(spec) {
+    const { enumIndex } = this;
+    const { values } = spec;
+    return super.selectExact({
+      mode: spec.mode,
+      values: values.map(v => lowerBound(enumIndex, v, 0, enumIndex.length))
+    });
+  }
+
+  /* eslint-disable class-methods-use-this */
+  selectRange() {
+    throw new Error("range selection unsupported on Enumerated dimension");
+  }
+  /* eslint-enable class-methods-use-this */
+}
+
+class ImmutableSpatialDimension extends _ImmutableBaseDimension {
+  constructor(name, data, X, Y) {
+    super(name);
+
+    if (X.length !== Y.length && X.length !== data.length) {
+      throw new RangeError(
+        "SpatialDimension values must have same dimensionality as crossfilter"
+      );
+    }
+    this.X = X;
+    this.Y = Y;
+
+    this.Xindex = makeSortIndex(X);
+    this.Yindex = makeSortIndex(Y);
+  }
+
+  select(spec) {
+    const { mode } = spec;
+    switch (mode) {
+      case "all":
+        return { ranges: [[0, this.X.length]], index: null };
+      case "none":
+        return { ranges: [], index: null };
+      case "within-rect":
+        return this.selectWithinRect(spec);
+      case "within-polygon":
+        return this.selectWithinPolygon(spec);
+      default:
+        return super.select(spec);
+    }
+  }
+
+  selectWithinRect(spec) {
+    /*
+      { mode: "within-rect", minX: 1, minY: 0, maxX: 3, maxY: 9 }
+    */
+    const { minX, minY, maxX, maxY } = spec;
+    const { X, Y } = this;
+    const ranges = [];
+    let start = -1;
+    for (let i = 0, l = X.length; i < l; i += 1) {
+      const x = X[i];
+      const y = Y[i];
+      const inside = minX <= x && x < maxX && minY <= y && y < maxY;
+      if (inside && start === -1) start = i;
+      if (!inside && start !== -1) {
+        ranges.push([start, i]);
+        start = -1;
+      }
+    }
+    if (start !== -1) ranges.push([start, X.length]);
+    return { ranges, index: null };
+  }
+
+  /*
+  Relatively brute force filter by polygon.
+
+  Currently uses d3.polygonContains() to test for polygon inclusion, which itself
+  uses a ray casting (crossing number) algorithm.  There are a series of optimizations
+  to make this faster:
+    * first sliced by X or Y, using an index on the axis
+    * then the polygon bounding box is used for trivial rejection
+    * then the polygon test is applied
+  */
+
+  selectWithinPolygon(spec) {
+    /*
+      { mode: "within-polygon", polygon: [ [x0, y0], ... ] }
+    */
+    const { polygon } = spec;
+    const [minX, minY, maxX, maxY] = polygonBoundingBox(polygon);
+    const { X, Y, Xindex, Yindex } = this;
+    const { length } = X;
+    let slice;
+    let index;
+    if (maxY - minY > maxX - minX) {
+      slice = [
+        lowerBoundIndirect(X, Xindex, minX, 0, length),
+        lowerBoundIndirect(X, Xindex, maxX, 0, length)
+      ];
+      index = Xindex;
+    } else {
+      slice = [
+        lowerBoundIndirect(Y, Yindex, minY, 0, length),
+        lowerBoundIndirect(Y, Yindex, maxY, 0, length)
+      ];
+      index = Yindex;
+    }
+
+    const ranges = [];
+    let start = -1;
+    for (let i = slice[0], e = slice[1]; i < e; i += 1) {
+      const rid = index[i];
+      const x = X[rid];
+      const y = Y[rid];
+      const inside =
+        minX <= x &&
+        x < maxX &&
+        minY <= y &&
+        y < maxY &&
+        withinPolygon(polygon, x, y);
+
+      if (inside && start === -1) start = i;
+      if (!inside && start !== -1) {
+        ranges.push([start, i]);
+        start = -1;
+      }
+    }
+    if (start !== -1) ranges.push([start, slice[1]]);
+    return { ranges, index };
+  }
+}
+
+/* Helpers */
+export const DimTypes = {
+  scalar: ImmutableScalarDimension,
+  enum: ImmutableEnumDimension,
+  spatial: ImmutableSpatialDimension
+};
+
+function isArrayOrTypedArray(x) {
+  return (
+    Array.isArray(x) ||
+    (ArrayBuffer.isView(x) &&
+      Object.prototype.toString.call(x) !== "[object DataView]")
+  );
+}
+
+/* return bounding box of the polygon */
+function polygonBoundingBox(polygon) {
+  let minX = Number.MAX_VALUE;
+  let minY = Number.MAX_VALUE;
+  let maxX = Number.MIN_VALUE;
+  let maxY = Number.MIN_VALUE;
+  for (let i = 0, l = polygon.length; i < l; i += 1) {
+    const point = polygon[i];
+    const [x, y] = point;
+    if (x < minX) minX = x;
+    if (y < minY) minY = y;
+    if (x > maxX) maxX = x;
+    if (y > maxY) maxY = y;
+  }
+  return [minX, minY, maxX, maxY];
+}
+
+function withinPolygon(polygon, x, y) {
+  // TODO XXX replace
+  return polygonContains(polygon, [x, y]);
+}
@@ -1,7 +1,5 @@
-// jshint esversion: 6
-
 /*
-Typedarray Crossfilter - a re-implementation of a subset of crossfilter, with
+Crossfilter - a re-implementation of a subset of crossfilter, with
 time/space optimizations predicated upon the following assumptions:
  - dimensions are uniformly typed, and all values must be of that type
  - dimension values must be a primitive type (int, float, string).  Arrays
@@ -11,14 +9,18 @@ time/space optimizations predicated upon the following assumptions:
    want to do that, you have to create the new crossfilter, using the new
    data, from scratch.

-The actual backing store for a dimension is a TypedArray, enabling significant
+In addition, this implementation is easier to use with a "redux" style
+app, as all operations on the crossfilter are immutable (ie, return a
+new crossfilter).
+
+The actual backing store for a dimension is a TypedArray, enabling
 performance improvements over the original crossfilter.

 There are also a handful of new methods, primarily to take advantage of the
 performance (eg, crossfilter.fillBySelection)

-Helpful documents (this module tries to follow the original API as much
-as is feasable):
+Helpful documents (this module follows similar concepts as the original,
+but deviates from the API):
    https://github.com/square/crossfilter/
    http://square.github.io/crossfilter/

@@ -26,766 +28,7 @@ There is also a newer, community supported fork of crossfilter, with a
 more complex API.  In a few cases, elements of that API were incorporated.
    https://github.com/square/crossfilter/

+See test cases for some concrete examples.
 */
-// XXX replace
-import { polygonContains } from "d3";

-import PositiveIntervals from "./positiveIntervals";
-import BitArray from "./bitArray";
-import {
-  makeSortIndex,
-  lowerBound,
-  lowerBoundIndirect,
-  upperBoundIndirect
-} from "./util";
-
-function isArrayOrTypedArray(x) {
-  return (
-    Array.isArray(x) ||
-    (ArrayBuffer.isView(x) &&
-      Object.prototype.toString.call(x) !== "[object DataView]")
-  );
-}
-
-class NotImplementedError extends Error {
-  constructor(...params) {
-    super(...params);
-
-    // Maintains proper stack trace for where our error was thrown (only available on V8)
-    if (Error.captureStackTrace) {
-      Error.captureStackTrace(this, NotImplementedError);
-    }
-  }
-}
-
-class TypedCrossfilter {
-  constructor(data) {
-    /*
-    Typically, data is one of:
-      - Array of objects/records
-      - Dataframe (util/dataframe)
-    */
-    this.data = data;
-
-    // filters: array of { id, dimension }
-    this.filters = [];
-    this.selection = new BitArray(data.length);
-    this.updateTime = 0;
-  }
-
-  size() {
-    return this.data.length;
-  }
-
-  all() {
-    return this.data;
-  }
-
-  /*
-  Create a crossfilter dimension, upon which filtering (subselection) can
-  be done.  Each dimension is typed, and has a particular set of filtering
-  semantics.
-    * ScalarDimension - backed by TypedArray values, supporting filtering
-      by value (within a value range, or one or more exact values)
-    * EnumDimension - backed by an enumeration (eg, strings, bools), filtering
-      by one or more enum categories.
-    * SpatialDimension - backed by 2D points, filter by containment within
-      various shapes (currently supports within Rectangle and within Polygon).
-  Call this method to create a dimension, passing arguments appropriate for
-  the dimension constructor.
-  */
-  dimension(DimensionType, ...rest) {
-    const id = this.selection.allocDimension();
-    const dim = new DimensionType(this, id, ...rest);
-    this.filters.push({ id, dim });
-    dim.filterAll();
-    return dim;
-  }
-
-  _freeDimension(id) {
-    this.selection.freeDimension(id);
-    this.filters = this.filters.filter(f => f._id !== id);
-  }
-
-  // return array of all records that are selected/filtered
-  // by all dimensions.
-  allFiltered() {
-    const { data, selection } = this;
-    if (Array.isArray(data)) {
-      const res = [];
-      for (let i = 0, len = data.length; i < len; i += 1) {
-        if (selection.isSelected(i)) {
-          res.push(data[i]);
-        }
-      }
-      return res;
-    }
-    /* else, Dataframe-like */
-    return data.isubsetMask(this.allFilteredMask());
-  }
-
-  // return Uint8array containing selection state (truthy/falsey) for each record.
-  //
-  allFilteredMask() {
-    return this.selection.fillBySelection(
-      new Uint8Array(this.data.length),
-      1,
-      0
-    );
-  }
-
-  countFiltered() {
-    return this.selection.selectionCount();
-  }
-
-  isElementFiltered(i) {
-    return this.selection.isSelected(i);
-  }
-
-  // fill array with one of two values, based upon selection state
-  fillByIsFiltered(array, selectedValue, deselectedValue) {
-    return this.selection.fillBySelection(
-      array,
-      selectedValue,
-      deselectedValue
-    );
-  }
-}
-
-// Base dimension type - not exported.
-class _Dimension {
-  constructor(xfltr, id) {
-    this.crossfilter = xfltr;
-    this._id = id;
-    this.groups = [];
-  }
-
-  dispose() {
-    this.crossfilter._freeDimension(this._id);
-    return this;
-  }
-
-  id() {
-    return this._id;
-  }
-
-  _filterUpdate() {
-    this.crossfilter.updateTime += 1;
-  }
-}
-
-// Scalar dimension type - value must be a scalar type (eg, int, float),
-// and value array must be a TypedArray.
-//
-class ScalarDimension extends _Dimension {
-  constructor(xfltr, id, value, ValueArrayType) {
-    super(xfltr, id);
-
-    // current selection filter, expressed as PostiveIntervals.
-    this.currentFilter = [];
-
-    // Two modes - caller can provide a pre-created value array,
-    // or a map function which will create it.
-    let array;
-    if (value instanceof ValueArrayType) {
-      // user has provided the final typed array - just use it
-      if (value.length !== this.crossfilter.data.length) {
-        throw new RangeError(
-          "ScalarDimension values length must equal crossfilter data record count"
-        );
-      }
-      array = value;
-    } else if (value instanceof Function) {
-      // Create value array from user-provided map function.
-      array = this._createValueArray(
-        value,
-        new ValueArrayType(this.crossfilter.data.length)
-      );
-    } else if (isArrayOrTypedArray(value)) {
-      // Create value array from user-provided array.  Typically used
-      // only by enumerated dimensions
-      array = this._createValueArray(
-        i => value[i],
-        new ValueArrayType(this.crossfilter.data.length)
-      );
-    } else {
-      throw new NotImplementedError(
-        "dimension value must be function or value array type"
-      );
-    }
-    this.value = array;
-
-    // create sort index
-    this.index = makeSortIndex(array);
-  }
-
-  _createValueArray(value, array) {
-    // create dimension value array
-    const { data } = this.crossfilter;
-    const len = data.length;
-    const larray = array;
-    for (let i = 0; i < len; i += 1) {
-      larray[i] = value(i, data);
-    }
-    return larray;
-  }
-
-  // Argument is an array of intervals indicating records newly selected/filtered
-  //
-  _updateFilters(newFilter) {
-    const cNewFilter = PositiveIntervals.canonicalize(newFilter);
-
-    const adds = PositiveIntervals.difference(cNewFilter, this.currentFilter);
-    const dels = PositiveIntervals.difference(this.currentFilter, cNewFilter);
-
-    this.crossfilter.filters.forEach(f =>
-      f.dim.groups.forEach(grp => grp._updateReduceDel(this, dels))
-    );
-
-    dels.forEach(interval =>
-      this.crossfilter.selection.deselectIndirectFromRange(
-        this._id,
-        this.index,
-        interval
-      )
-    );
-
-    adds.forEach(interval =>
-      this.crossfilter.selection.selectIndirectFromRange(
-        this._id,
-        this.index,
-        interval
-      )
-    );
-
-    this.crossfilter.filters.forEach(f =>
-      f.dim.groups.forEach(grp => grp._updateReduceAdd(this, adds))
-    );
-
-    this.currentFilter = cNewFilter;
-    this._filterUpdate();
-  }
-
-  // filter by value - exact match
-  filterExact(value) {
-    const newFilter = [
-      lowerBoundIndirect(this.value, this.index, value, 0, this.value.length),
-      upperBoundIndirect(this.value, this.index, value, 0, this.value.length)
-    ];
-    if (newFilter[0] <= newFilter[1]) {
-      this._updateFilters([newFilter]);
-    } else {
-      this._updateFilters([]);
-    }
-    return this;
-  }
-
-  // filter by a set of values, eg. enum.
-  filterEnum(values) {
-    const newFilter = [];
-    for (let v = 0, len = values.length; v < len; v += 1) {
-      const intv = [
-        lowerBoundIndirect(
-          this.value,
-          this.index,
-          values[v],
-          0,
-          this.value.length
-        ),
-        upperBoundIndirect(
-          this.value,
-          this.index,
-          values[v],
-          0,
-          this.value.length
-        )
-      ];
-      if (intv[0] <= intv[1]) newFilter.push(intv);
-    }
-    this._updateFilters(newFilter);
-    return this;
-  }
-
-  // filter by value range [lo, hi)
-  // lo: inclusive, hi: exclusive
-  filterRange(range) {
-    const newFilter = [];
-    const intv = [
-      lowerBoundIndirect(
-        this.value,
-        this.index,
-        range[0],
-        0,
-        this.value.length
-      ),
-      upperBoundIndirect(this.value, this.index, range[1], 0, this.value.length)
-    ];
-    if (intv[0] < intv[1]) newFilter.push(intv);
-    this._updateFilters(newFilter);
-    return this;
-  }
-
-  // select all - equivalent of selecting all in this dimension
-  filterAll() {
-    this._updateFilters([[0, this.value.length]]);
-    return this;
-  }
-
-  // select none
-  filterNone() {
-    this._updateFilters([]);
-  }
-
-  // return top k records, starting with offset, in descending order.
-  // Order is this dimension's sort order
-  top(k, offset = 0) {
-    const { data, selection } = this.crossfilter;
-    const { index } = this;
-    const len = index.length;
-    const ret = [];
-    let i = 0;
-    let skip = 0;
-    let found = 0;
-
-    // skip up to offset records
-    for (i = len - 1; i >= 0 && skip < offset; i -= 1) {
-      if (selection.isSelected(index[i])) {
-        skip += 1;
-      }
-    }
-
-    // grab up to k records
-    for (; i >= 0 && found < k; i -= 1) {
-      if (selection.isSelected(index[i])) {
-        ret.push(data[index[i]]);
-        found += 1;
-      }
-    }
-
-    return ret;
-  }
-
-  // return bottom k records, starting with offset, in ascending order.
-  // Order is this dimension's sort order
-  bottom(k, offset = 0) {
-    const { data, selection } = this.crossfilter;
-    const { index } = this;
-    const len = index.length;
-    const ret = [];
-    let skip = 0;
-    let found = 0;
-    let i = 0;
-
-    // skip up to offset records
-    for (i = 0; i < len && skip < offset; i += 1) {
-      if (selection.isSelected(index[i])) {
-        skip += 1;
-      }
-    }
-
-    // grab up to k records
-    for (; i < len && found < k; i += 1) {
-      if (selection.isSelected(index[i])) {
-        ret.push(data[index[i]]);
-        found += 1;
-      }
-    }
-
-    return ret;
-  }
-
-  group(groupValue) {
-    const grp = new ScalarGroup(groupValue, this.value.constructor, this);
-    this.groups.push(grp);
-    return grp;
-  }
-
-  _freeGroup(group) {
-    this.groups = this.groups.filter(e => e !== group);
-  }
-}
-
-// Ordered enumeration - supports any sortable enumerable type, eg,
-// strings, which can be mapped into an fixed numeric range [0..n).
-//
-class EnumDimension extends ScalarDimension {
-  constructor(xfltr, id, value) {
-    super(xfltr, id, value, Uint32Array);
-  }
-
-  _createValueArray(value, array) {
-    const { data } = this.crossfilter;
-    const len = data.length;
-    const larray = array;
-
-    // create enumeration table - mapping between the value
-    // and the enum.
-    const s = new Set();
-    for (let i = 0; i < len; i += 1) {
-      s.add(value(i, data));
-    }
-    this.enumIndex = Array.from(s);
-    this.enumIndex.sort();
-
-    // create dimension value array
-    const enumLen = this.enumIndex.length;
-    for (let i = 0; i < len; i += 1) {
-      const v = value(i, data);
-      const e = lowerBound(this.enumIndex, v, 0, enumLen);
-      larray[i] = e;
-    }
-    return larray;
-  }
-
-  filterExact(value) {
-    return super.filterExact(
-      lowerBound(this.enumIndex, value, 0, this.enumIndex.length)
-    );
-  }
-
-  filterEnum(values) {
-    return super.filterEnum(
-      values.map(v => lowerBound(this.enumIndex, v, 0, this.enumIndex.length))
-    );
-  }
-
-  filterRange(range) {
-    return super.filterEnum(
-      range.map(v => lowerBound(this.enumIndex, v, 0, this.enumIndex.length))
-    );
-  }
-
-  group(groupValue) {
-    const grp = new EnumGroup(groupValue, this.value.constructor, this);
-    this.groups.push(grp);
-    return grp;
-  }
-}
-
-/*
-Super simple 2D spatial dimension, supporting basic "filter within"
-operations.
-*/
-class SpatialDimension extends _Dimension {
-  constructor(xfltr, id, X, Y) {
-    super(xfltr, id);
-
-    if (X.length !== Y.length && X.length !== this.crossfilter.data.length) {
-      throw new RangeError(
-        "SpatialDimension values must have same dimensionality as crossfilter"
-      );
-    }
-    this.X = X;
-    this.Y = Y;
-
-    this.Xindex = makeSortIndex(X);
-    this.Yindex = makeSortIndex(Y);
-  }
-
-  filterAll() {
-    this.crossfilter.selection.selectAll(this._id);
-    this._filterUpdate();
-  }
-
-  filterNone() {
-    this.crossfilter.selection.deselectAll(this._id);
-    this._filterUpdate();
-  }
-
-  /*
-  this could be smarter, but we don't currently use it...
-  */
-  filterWithinRect(northwest, southeast) {
-    const [x0, y0] = northwest;
-    const [x1, y1] = southeast;
-    const { X, Y } = this;
-    const seln = this.crossfilter.selection;
-    const { _id } = this;
-    seln.deselectAll(_id);
-    for (let i = 0, l = this.X.length; i < l; i += 1) {
-      const x = X[i];
-      const y = Y[i];
-      if (x0 <= x && x < x1 && y0 <= y && y < y1) {
-        seln.selectOne(_id, i);
-      }
-    }
-    this._filterUpdate();
-  }
-
-  /*
-  Relatively brute force filter by polygon.   Polygon is array of points, where
-  each point is [x,y].   Eg, [[x0,y0], [x1,y1], ...].
-
-  Currently uses d3.polygonContains() to test for polygon inclusion, which itself
-  uses a ray casting (crossing number) algorithm.  There are a series of optimizations
-  to make this faster:
-    * first sliced by X or Y, using an index on the axis
-    * then the polygon bounding box is used for trivial rejection
-    * then the polygon test is applied
-  */
-  filterWithinPolygon(polygon) {
-    /* return bounding box of the polygon */
-    function polygonBoundingBox(pg) {
-      let minX = Number.MAX_VALUE;
-      let minY = Number.MAX_VALUE;
-      let maxX = Number.MIN_VALUE;
-      let maxY = Number.MIN_VALUE;
-      for (let i = 0, l = pg.length; i < l; i += 1) {
-        const p = pg[i];
-        const x = p[0];
-        const y = p[1];
-        if (x < minX) minX = x;
-        if (y < minY) minY = y;
-        if (x > maxX) maxX = x;
-        if (y > maxY) maxY = y;
-      }
-      return [minX, minY, maxX, maxY];
-    }
-
-    const [minX, minY, maxX, maxY] = polygonBoundingBox(polygon);
-    const { X, Y } = this;
-    let slice;
-    let index;
-    if (maxY - minY > maxX - minX) {
-      slice = [
-        lowerBoundIndirect(X, this.Xindex, minX, 0, X.length),
-        upperBoundIndirect(X, this.Xindex, maxX, 0, X.length)
-      ];
-      index = this.Xindex;
-    } else {
-      slice = [
-        lowerBoundIndirect(Y, this.Yindex, minY, 0, Y.length),
-        upperBoundIndirect(Y, this.Yindex, maxY, 0, Y.length)
-      ];
-      index = this.Yindex;
-    }
-
-    const seln = this.crossfilter.selection;
-    const { _id } = this;
-    const testWithin = polygonContains; // d3.polygonContains()
-    seln.deselectAll(_id);
-
-    for (let i = slice[0], e = slice[1]; i < e; i += 1) {
-      const rid = index[i];
-      const x = X[rid];
-      const y = Y[rid];
-      if (
-        minX <= x &&
-        x < maxX &&
-        minY <= y &&
-        y < maxY &&
-        testWithin(polygon, [x, y])
-      ) {
-        seln.selectOne(_id, rid);
-      }
-    }
-    this._filterUpdate();
-  }
-}
-
-// Groups!  Map/reduce
-//
-class ScalarGroup {
-  constructor(groupValue, groupValueType, dimension) {
-    // parent dimension
-    this.dimension = dimension;
-
-    // generate group names from dimension values
-    this.mapValue = this.constructor._map(
-      groupValue,
-      groupValueType,
-      dimension
-    );
-
-    // group index is mapping from data record index to group index
-    this.groupIndex = new Uint32Array(dimension.crossfilter.data.length);
-
-    // default to counting
-    this.reduceCount();
-
-    // Creates this.groups
-    this._reduce();
-  }
-
-  // internal support function - map all dimension values to group values.
-  //
-  static _map(groupValue, GroupValueType, dimension) {
-    // groupValue is optional.  Defaults to identity.  Used to perform
-    // initial map operation.
-    //
-    // identity: save some memory...
-    if (groupValue === undefined) return dimension.value;
-
-    const data = dimension.value;
-    const len = data.length;
-    const mapValue = new GroupValueType(dimension.value.length);
-    for (let i = 0; i < len; i += 1) {
-      mapValue[i] = groupValue(data[i]);
-    }
-    return mapValue;
-  }
-
-  // Update the group reduction incrementally.  Called when *any* dimension filter
-  // changes.  Guaranteed to be called AFTER the crossfilter is updated.
-  //
-  // Arguments:
-  //  * dim: the dimension that is changing
-  //  * intv: interval list of newly selected values on `dim` (adds)
-  //
-  _updateReduceAdd(dim, intv) {
-    // ignore updates to self, as we don't reduce inclusive of our filter
-    if (dim === this.dimension || intv.length === 0) return;
-
-    // Each item in the range was just added to `dim`.  It was NOT previously
-    // selected - reduceAdd if it is now selected.
-    const { data, selection } = this.dimension.crossfilter;
-    intv.forEach(rng => {
-      for (let r = rng[0]; r < rng[1]; r += 1) {
-        const i = dim.index[r];
-        if (selection.isSelectedIgnoringDim(i, this.dimension.id())) {
-          const group = this.groups[this.groupIndex[i]];
-          group.value = this.reduceAdd(group.value, data[i]);
-        }
-      }
-    });
-  }
-
-  // Update the group reduction incrementally.  Called when *any* dimension filter
-  // changes.   Guaranteed to be called BEFORE the crossfilter is updated.
-  //
-  // Arguments:
-  //  * dim: the dimension that is changing
-  //  * intv: interval list of previously selected values on `dim` (dels)
-  //
-  _updateReduceDel(dim, intv) {
-    // ignore updates to self, as we don't reduce inclusive of our filter
-    if (dim === this.dimension || intv.length === 0) return;
-
-    // Each item in the range will be remved from `dim`.  reduceRemove if it
-    // is currently selected.
-    const { data, selection } = this.dimension.crossfilter;
-    intv.forEach(rng => {
-      for (let r = rng[0]; r < rng[1]; r += 1) {
-        const i = dim.index[r];
-        if (selection.isSelectedIgnoringDim(i, this.dimension.id())) {
-          const group = this.groups[this.groupIndex[i]];
-          group.value = this.reduceRemove(group.value, data[i]);
-        }
-      }
-    });
-  }
-
-  // Reduce the entire data set, creating both the group index and the
-  // groups data.
-  //
-  _reduce() {
-    const { dimension } = this;
-    const { data } = dimension.crossfilter;
-
-    // Create groups
-    const groupNames = new Set(this.mapValue);
-    this.groups = [];
-    const groupIndexByName = {};
-    groupNames.forEach(name => {
-      this.groups.push({ key: name, value: this.reduceInitial() });
-      groupIndexByName[name] = this.groups.length - 1;
-    });
-
-    // Create groupIndex - index map between data record index and group index
-    for (let i = 0, len = this.mapValue.length; i < len; i += 1) {
-      this.groupIndex[i] = groupIndexByName[this.mapValue[i]];
-    }
-
-    // reduce all filtered records, IGNORING the current dimension's filter
-    const { selection } = dimension.crossfilter;
-    for (let i = 0, len = data.length; i < len; i += 1) {
-      if (selection.isSelectedIgnoringDim(i, dimension.id())) {
-        const group = this.groups[this.groupIndex[i]];
-        group.value = this.reduceAdd(group.value, data[i]);
-      }
-    }
-  }
-
-  dispose() {
-    this.dimension._freeGroup(this);
-    return this;
-  }
-
-  // return number of distinct values in the group, independent of any filters.
-  //
-  size() {
-    return this.groups.length;
-  }
-
-  // Set the reduce functions and return the grouping.
-  //
-  reduce(add, remove, initial) {
-    this.reduceAdd = add;
-    this.reduceRemove = remove;
-    this.reduceInitial = initial;
-    this._reduce();
-    return this;
-  }
-
-  // set the reduce functions to count records.
-  reduceCount() {
-    return this.reduce(p => p + 1, p => p - 1, () => 0);
-  }
-
-  // set the reduce functions to sum records using specified value accessor.
-  //
-  reduceSum(value) {
-    return this.reduce((p, v) => p + value(v), (p, v) => p - value(v), () => 0);
-  }
-
-  all() {
-    const res = [...this.groups];
-    res.sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
-    return res;
-  }
-}
-
-class EnumGroup extends ScalarGroup {
-  static _map(groupValue, groupValueType, dimension) {
-    // groupValue is optional.  Defaults to identity.  Used to perform
-    // initial map operation.
-    //
-    // identity: save some memory
-    if (groupValue === undefined) return dimension.value;
-
-    // non-identity mapping unsupported for EnumDimension/EnumGroup.
-    // XXX: this could be implemented, but would require another index
-    // array to map from the group names/keys back to the dimension values.
-    // With this, we just rely on the dimensions `enumIndex` to map from
-    // enumeration value to the record.
-    throw new NotImplementedError("enumerated group mapping not implemented");
-  }
-
-  all() {
-    const res = [];
-    this.groups.forEach(e =>
-      res.push({
-        // XXX: assumes identity group map - see comment in _map()
-        key: this.dimension.enumIndex[e.key],
-        value: e.value
-      })
-    );
-    res.sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
-    return res;
-  }
-}
-
-// Wrapper for backwards compat with crossfilter.
-//
-function crossfilter(data) {
-  return new TypedCrossfilter(data);
-}
-
-crossfilter.PositiveIntervals = PositiveIntervals;
-crossfilter.BitArray = BitArray;
-crossfilter.TypedCrossfilter = TypedCrossfilter;
-crossfilter.ScalarDimension = ScalarDimension;
-crossfilter.EnumDimension = EnumDimension;
-crossfilter.SpatialDimension = SpatialDimension;
-
-export default crossfilter;
+export { default } from "./crossfilter";
@@ -1,5 +1,29 @@
-const SmallArray = 32;
+import { isTypedArray, isFpTypedArray } from "../typeHelpers";

+/* eslint no-bitwise: "off" */
+
+/*
+ ** fast sort and search, with separate code paths for floats (NaN ordering),
+ ** indirect and direct search/sort.
+ */
+
+/*
+Comparators for float sort.   -Infinity < finite < Infinity < NaN
+*/
+function lt(a, b) {
+  if (Number.isNaN(b)) return !Number.isNaN(a);
+  return a < b;
+}
+
+function gt(a, b) {
+  if (Number.isNaN(a)) return !Number.isNaN(b);
+  return a > b;
+}
+
+/*
+insertion sort, used for small arrays (controlled by SMALL_ARRAY constant)
+*/
+const SMALL_ARRAY = 32;
 function insertionsort(a, lo, hi) {
  for (let i = lo + 1; i < hi + 1; i += 1) {
    const x = a[i];
@@ -12,6 +36,18 @@ function insertionsort(a, lo, hi) {
  return a;
 }

+function insertionsortFloats(a, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    let j;
+    for (j = i; j > lo && gt(a[j - 1], x); j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
 function insertionsortIndirect(a, s, lo, hi) {
  for (let i = lo + 1; i < hi + 1; i += 1) {
    const x = a[i];
@@ -25,8 +61,24 @@ function insertionsortIndirect(a, s, lo, hi) {
  return a;
 }

+function insertionsortFloatsIndirect(a, s, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    const t = s[x];
+    let j;
+    for (j = i; j > lo && gt(s[a[j - 1]], t); j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
+/*
+Quicksort - used for larger arrays
+*/
 function quicksort(a, lo, hi) {
-  if (hi - lo < SmallArray) {
+  if (hi - lo < SMALL_ARRAY) {
    return insertionsort(a, lo, hi);
  }
  if (lo < hi) {
@@ -55,8 +107,38 @@ function quicksort(a, lo, hi) {
  return a;
 }

+function quicksortFloats(a, lo, hi) {
+  if (hi - lo < SMALL_ARRAY) {
+    return insertionsortFloats(a, lo, hi);
+  }
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (lt(a[i], p));
+      do {
+        j -= 1;
+      } while (gt(a[j], p));
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksortFloats(a, lo, j);
+    quicksortFloats(a, j + 1, hi);
+  }
+  return a;
+}
+
 function quicksortIndirect(a, s, lo, hi) {
-  if (hi - lo < SmallArray) {
+  if (hi - lo < SMALL_ARRAY) {
    return insertionsortIndirect(a, s, lo, hi);
  }
  if (lo < hi) {
@@ -86,15 +168,248 @@ function quicksortIndirect(a, s, lo, hi) {
  return a;
 }

-// Convenience wrappers
-export function sort(arr, comparator = undefined) {
-  if (comparator !== undefined) {
-    // XXX for now
-    return arr.sort(arr, comparator);
+function quicksortFloatsIndirect(a, s, lo, hi) {
+  if (hi - lo < SMALL_ARRAY) {
+    return insertionsortFloatsIndirect(a, s, lo, hi);
  }
-  return quicksort(arr, 0, arr.length - 1);
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    const t = s[p];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (lt(s[a[i]], t));
+      do {
+        j -= 1;
+      } while (gt(s[a[j]], t));
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksortFloatsIndirect(a, s, lo, j);
+    quicksortFloatsIndirect(a, s, j + 1, hi);
+  }
+  return a;
+}
+
+/*
+Convenience wrappers, handling optimization paths and default
+handlers for NaN comparisons.  Sorts in place.
+*/
+export function sortArray(arr) {
+  if (Array.isArray(arr)) {
+    return quicksort(arr, 0, arr.length - 1);
+  }
+  if (isTypedArray(arr)) {
+    if (isFpTypedArray(arr)) {
+      return quicksortFloats(arr, 0, arr.length - 1);
+    }
+    return quicksort(arr, 0, arr.length - 1);
+  }
+  /* else unsupported */
+  throw new Error("sortArray received unsupported object type");
 }

 export function sortIndex(index, source) {
+  if (isFpTypedArray(source))
+    return quicksortFloatsIndirect(index, source, 0, index.length - 1);
  return quicksortIndirect(index, source, 0, index.length - 1);
 }
+
+// Search for `value` in the sorted array `arr`, in the range [first, last).
+// Return the first (left most) index where arr[index] >= value.
+//
+// In other words, return array index I where:
+//  arr[i] < value for all tarr[lo:I]
+//  arr[i] >= value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: lower_bound()
+//    Python: bisect.bisect_left()
+//
+function lowerBoundNonFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+// lowerBound, but with NaN handling
+//
+// If the underlying array is a Float32Array or Float64Array, will enforce
+// the ordering -Infinity < finite < Infinity < NaN.
+//
+function lowerBoundFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (lt(valueArray[middle], value)) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+export function lowerBound(valueArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return lowerBoundFloat(valueArray, value, first, last);
+  }
+  return lowerBoundNonFloat(valueArray, value, first, last);
+}
+
+// Inlined performance optimization - used to indirect through a sort map.
+//
+function lowerBoundNonFloatIndirect(
+  valueArray,
+  indexArray,
+  value,
+  first,
+  last
+) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+function lowerBoundFloatIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (lt(valueArray[indexArray[middle]], value)) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+export function lowerBoundIndirect(valueArray, indexArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return lowerBoundFloatIndirect(valueArray, indexArray, value, first, last);
+  }
+  return lowerBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
+}
+
+// Search for `value in the sorted array `arr`, in the range [first, last).
+// Return the first value where arr[index] > value.
+//
+// In other words, return array index I, where:
+//  arr[i] <= value for all tarr[lo:I]
+//  arr[i] > value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: upper_bound()
+//    Python: bisect.bisect_right()
+//
+function upperBoundNonFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+function upperBoundFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (gt(valueArray[middle], value)) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+export function upperBound(valueArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return upperBoundFloat(valueArray, value, first, last);
+  }
+  return upperBoundNonFloat(valueArray, value, first, last);
+}
+
+// Inline performance optimization
+//
+function upperBoundNonFloatIndirect(
+  valueArray,
+  indexArray,
+  value,
+  first,
+  last
+) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+function upperBoundFloatIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (gt(valueArray[indexArray[middle]], value)) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return upperBoundFloatIndirect(valueArray, indexArray, value, first, last);
+  }
+  return upperBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
+}
@@ -1,23 +1,12 @@
 // jshint esversion: 6
-/* eslint no-bitwise: "off" */

 import { sortIndex } from "./sort";
+import { rangeFill as fillRange } from "../range";

 /*
    Utility functions, private to this module.
 */

-// fill an array or typedarray with a sequential range of numbers,
-// starting with `start`
-//
-export function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, len = larr.length; i < len; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 // slice out of one array into another, using an index array
 //
 export function sliceByIndex(src, index) {
@@ -36,93 +25,3 @@ export function makeSortIndex(src) {
  sortIndex(index, src);
  return index;
 }
-
-// Search for `value` in the sorted array `arr`, in the range [first, last).
-// Return the first (left most) index where arr[index] >= value.
-//
-// In other words, return array index I where:
-//  arr[i] < value for all tarr[lo:I]
-//  arr[i] >= value for all tarr[I:last]
-//
-// The same semantics/behavior as:
-//    C++: lower_bound()
-//    Python: bisect.bisect_left()
-//
-// XXX: it is likely that there would be minimal performance hit from creating
-// a factory version of lowerBound that takes an accessor (rather than having
-// a special-cased version for lining the indirection).
-//
-export function lowerBound(valueArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[middle] < value) {
-      lfirst = middle + 1;
-    } else {
-      llast = middle;
-    }
-  }
-  return lfirst;
-}
-
-// Inlined performance optimization - used to indirect through a sort map.
-//
-export function lowerBoundIndirect(valueArray, indexArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[indexArray[middle]] < value) {
-      lfirst = middle + 1;
-    } else {
-      llast = middle;
-    }
-  }
-  return lfirst;
-}
-
-// Search for `value in the sorted array `arr`, in the range [first, last).
-// Return the first value where arr[index] > value.
-//
-// In other words, return array index I, where:
-//  arr[i] <= value for all tarr[lo:I]
-//  arr[i] > value for all tarr[I:last]
-//
-// The same semantics/behavior as:
-//    C++: upper_bound()
-//    Python: bisect.bisect_right()
-//
-export function upperBound(valueArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[middle] > value) {
-      llast = middle;
-    } else {
-      lfirst = middle + 1;
-    }
-  }
-  return lfirst;
-}
-
-// Inline performance optimization
-//
-export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[indexArray[middle]] > value) {
-      llast = middle;
-    } else {
-      lfirst = middle + 1;
-    }
-  }
-  return lfirst;
-}
@@ -0,0 +1,91 @@
+# Developer guidelines
+
+### Requirements
+- npm
+- Python 3.6+
+- Chrome
+
+[See dev section of README](../README.md)
+
+**All instructions are expected to be run from the top level cellxgene directory unless otherwise specified.**
+
+## Server dev
+### Install
+* Build the client and put static files in place: `make build-for-server-dev`
+* Install from local files: `make install-dev`
+
+### Launch
+* `cellxgene launch [options] <datafile>`
+
+### Reloading
+If you install cellxgene using `make install-dev` the server will be restarted every time you make changes on the server code. If changes affects the client, the browser must be reloaded.
+
+### Linter
+We use `flake8` to lint code. Travis CI runs `flake8 server`.
+
+### Test
+1. Install development requirements `pip install -r server/requirements-dev.txt`
+2. Run tests `pytest server/test`
+
+### Tips
+* Install in a virtualenv
+* May need to rebuild/reinstall when you make client changes
+
+## Client dev
+### Install
+1. Install prereqs for client: `npm install --prefix client/ client`
+2. Install cellxgene server: `pip install -e .` Caveat: this will not build the production client package - you must use the [server install](#install) instructions above to serve web assets.
+
+### Launch
+To launch with hot reloading you need to launch the server and the client separately. Node's hot reloading starts the client on its own node server and auto-refreshes when changes are made.
+1. Launch server (the client relies on the REST API being available): `cellxgene launch [options] <datafile>`
+2. Launch client: in `client/` directory run `npm run start`
+3. Client will be served on localhost:3000
+
+### Build
+To build only the client: `make build-client`
+
+### Linter
+We use `eslint` to lint the code and `prettier` as our code formatter.
+
+### Test
+In `client/` directory run `npm run unit-test`
+
+### Tips
+* You can also install/launch the server side code from npm scrips (requires python3.6 with virtualenv) in `client/` directory run `npm run backend-dev`
+
+## Running tests
+Client and server tests run on Travis CI for every push, PR, and commit to master on github. End to end tests run nightly on master only. 
+
+### Server unit tests
+Install development requirements `pip install -r server/requirements-dev.txt`
+Run tests `pytest server/test`
+
+### Client unit tests
+In `client/` directory run `npm run unit-test`
+
+### End to end tests
+
+End to end tests use two env variables:
+* `JEST_ENV` - environment to run end to end tests. Default `dev`
+    * `prod` - run headless with no slowdown, chromium will not open.
+    * `dev` - opens chromimum, runs tests with minimal slowdown, close on exit.
+    * `debug` - opens chromium, runs tests with 100ms slowdown, dev tools open, chrome stays open on exit.
+* `JEST_CXG_PORT` - port that end to end tests are being run on. Default `3000` (client hosted port).
+
+On CI the end to end tests are run with `JEST_ENV` set to `prod` using the `smoke-test` npm script
+
+To run end to end tests as they will be run on CI
+1. cellxgene should be built and installed as [specified in server dev](#install)
+2. `export JEST_ENV='prod'`
+3. `export JEST_CXG_PORT='5000'`
+4. Run `npm run --prefix client/ smoke-test`
+
+Run end to end tests interactively during development
+1. cellxgene should be installed as [specified in client dev](#install-1)
+2. Follow [launch](#launch-1) instructions for client dev with dataset `example-dataset/pbmc3k`
+3. Run `npm run --prefix client/ e2e`
+4. To debug a failing test `export JEST_ENV='debug'` and re-run.
+
+
+
@@ -43,24 +43,53 @@ Follow these steps to create a release.
 8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
 and that you have write access to the cellxgene pypi package):
    - Build the distribution and upload to test pypi `make release-stage-2`
-    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Test the test installation in a fresh virtual environment using `make install-release-test`
    - Upload the package to real pypi using `make release-stage-final`
-    - [optional] Test the installation in a fresh virtual environment using
+    - Test the installation in a fresh virtual environment using
      `pip install cellxgene`
-    - **Troubleshooting**:
-        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
-        if you accidentally burned a release number you want to use on prod, you have a couple options. 
-            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
-             then upload to prod `make release-stage-final`. 
-            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
-            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
-            PART=patch until you get to an unburned version.
-        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
-        go to the cellxgene package, click manage, then in the options drop down, click delete and 
-        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
-        and not a major regression, you can just release a patch. 
-    

 The optional steps are for testing purposes, and are recommended
 for publishing any major releases, and any releases that significantly
 change the packaging (e.g. new bundled files, new dependencies, etc.)
+
+## Troubleshooting
+### Fails to upload to test.pypi
+ 
+ _PyPi doesn't allow you to reupload a release with the same version number_ 
+If you accidentally burned a release number you want to use on prod, you have a few options:
+1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+ then upload to prod `make release-stage-final`. 
+2) OPTION 2: (DANGER) release directly to prod: `make release-directly-to-prod`.  
+3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+PART=patch until you get to an unburned version.
+    
+### The release doesn't install or fails your tests when you install it
+ 
+Delete it from pypi - Go to pypi.org -> sign in -> go to the cellxgene package -> click manage -> then in the options drop down click delete -> follow the instructions. You will not be able to use that release number again. If it is a minor bug and not a major regression, you can just release a patch. 
+### If you need to run stage final on a different computer than stage 2
+ If you run stage final without running stage 2 first, the dist will not have been build on the computer running stage final. The solution is to run `make release-directly-to-prod`. This both builds the distribution files and then releases directly to prod pypi.org. 
+
+## Stage Details
+### Stage 1 - `make release-stage-1`
+    1. Pip installs requirements-dev
+    2. Bumps version by [PART]
+    3. Deletes build directory, client/build, dist and cellxgene.egg-info
+    4. Creates the package-lock.json
+
+### Stage 2 - `make release-stage-2`
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to test.pypi.org
+
+### Stage final - `make release-stage-final`
+    ** Does not build distribution **
+    1. Uploads to pypi.org
+    
+### (DANGER) Release directly to prod `make release-directly-to-prod`
+    ** builds distribution and uploads directly to prod **
+    Only use this if you are directed to by the troubleshooting guide
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to pypi.org
@@ -3,6 +3,8 @@ show_downloads: false
 baseurl: /cellxgene

 nav:
+  - title: Getting Started
+    url: getting-started.html
  - title: Data
    url: data.html
  - title: FAQ
@@ -38,7 +38,15 @@ Currently this is not supported directly, but you should be able to do this your

 - `.obs` and `.var` annotations are use to extract metadata for filtering
 - `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
- `.obsm` is used for layout
+- `.obsm` is used for layout. If an embedding has more than two components, the first two will be used for visualization.
+
+#### I have a BIG dataset - how can I make cellxgene run as fast as possible?
+
+If your dataset requires gigabytes of disk space, you may need to select an appropriate storage format in order to effectively utilize `cellxgene`. Tips and tricks:
+
+- `cellxgene` is optimized for columnar data access. For large datasets, format the expression matrix (`.X`) as either a [SciPy CSC sparse matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) or a dense Numpy array (whichever creates a smaller `h5ad` file). If you are using `cellxgene prepare`, include the `--sparse` flag to ensure `.X` is formatted as a CSC sparse matrix (by default, `.X` will be a dense matrix).
+- `cellxgene` start time is directly proportional to `h5ad` file size and the speed of your file system. Expect that large (eg, million cell) datasets will take minutes to load, even on relatively fast computers with a high performance local hard drive. Once loaded, exploring metadata should still be quick.
+- If your dataset size exceeds the size of memory (RAM) on the host computer, differential expression calculations will be extremely slow (or fail, if you run out of virtual memory).

 # Algorithms

@@ -48,12 +56,12 @@ We use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) impl

 # Problems, errors, & bugs

-#### How do I create a Python 3.6 environment for _cellxgene_?
+#### How do I create a Python environment for _cellxgene_?

 If you use conda and want to create a [conda environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for _cellxgene_ you can use the following commands

 ```
-conda create --yes -n cellxgene python=3.6
+conda create --yes -n cellxgene python=3.7
 conda activate cellxgene
 pip install cellxgene
 ```
@@ -62,7 +70,7 @@ Or you can create a virtual environment by using

 ```
 ENV_NAME=cellxgene
-python3.6 -m venv ${ENV_NAME}
+python3.7 -m venv ${ENV_NAME}
 source ${ENV_NAME}/bin/activate
 pip install cellxgene
 ```
@@ -0,0 +1,116 @@
+## getting started
+
+You'll need **python 3.6+** and **Google Chrome**.
+The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know.
+
+To install run
+
+```
+pip install cellxgene
+```
+
+To start exploring a dataset call
+
+```
+cellxgene launch dataset.h5ad --open
+```
+
+If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
+
+```
+cellxgene launch pbmc3k.h5ad --open
+```
+
+On Mac OS and Ubuntu, you should see your web browser open with the following
+
+<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+
+**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+
+There are several options available, such as:
+
+- `--layout` to specify the layout as `tsne`, `umap`, `diffmap`, `phate`, `draw_graph_fa`, or `draw_graph_fr`
+- `--title` to show a title on the explorer
+- `--open` to automatically open the web browser after launching (OS X only)
+
+To see all options call
+
+```
+cellxgene launch --help
+```
+
+There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+
+## data formatting
+
+### requirements
+
+The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
+
+- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
+- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
+- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_umap` for the `umap` layout)
+- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
+
+### prepare
+
+The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
+
+To prepare from an existing `.h5ad` file use
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
+```
+
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
+
+Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
+```
+
+To see all options call
+
+```
+cellxgene prepare --help
+```
+
+**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
+
+```
+pip install cellxgene[louvain]
+```
+
+If the aforementioned optional package installation fails, you can also install these packages directly:
+
+```
+pip install python-igraph louvain>=0.6
+```
+
+## conda and virtual environments
+
+If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.7
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3.7 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+## docker
+
+We have included a dockerfile to conveniently run cellxgene from docker.
+
+1. Build the image `docker build . -t cellxgene`
+2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
+   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
@@ -2,15 +2,15 @@ _cellxgene_ is an interactive data explorer for single-cell transcriptomics data

 ## features

-#### Flexible selections, coloring, and differential expression of your selected sets of cells
+#### flexible selections, coloring, and differential expression of your selected sets of cells
 <img src="diffexp.gif" width="600"/>

-#### Single-gene analyses (e.g. expression analysis)
+#### single-gene analyses (e.g. expression analysis)
 <img src="customGene.gif" width="600" />

-## getting started
+## quick start

-_cellxgene_ **only** supports Python 3.6. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
+To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-environment-for-cellxgene)

 Install the package.
 ``` bash
@@ -25,13 +25,17 @@ curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/m

 Launch _cellxgene_
 ``` bash
-cellxgene launch pbmc3k.h5ad
+cellxgene launch pbmc3k.h5ad --open
 ```

+To explore more datasets already formatted for _cellxgene_, see [Data](data) or
+visit [Getting Started](getting-started) to learn more about formatting your own
+data for _cellxgene_.
+
 ## getting help

 We'd love to hear from you!

-For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cziscience-slack.herokuapp.com/) and say "hi!".
+For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and say "hi!".

 For any errors, [report bugs on Github](https://github.com/chanzuckerberg/cellxgene/issues).
@@ -66,17 +66,20 @@ release-stage-1 : dev-env bump clean-lite gen-package-lock
 # build dist and release to dev pypi
 release-stage-2 : dev-env pydist twine
 	@echo "Dist built and uploaded to test.pypi.org"
-	@echo "Test the install `make install-release-test` and then upload to Pypi prod"
-	@echo "`make twine-prod`"
+	@echo "Test the install:"
+	@echo "    make install-release-test"
+	@echo "Then upload to Pypi prod:"
+	@echo "    make twine-prod"

 release-stage-final: twine-prod
 	@echo "Release uploaded to pypi.org"

 # DANGER: releases directly to prod
 # use this if you accidently burned a test release version number,
-release-burned : dev-env pydist twine-prod
+release-directly-to-prod : dev-env pydist twine-prod
 	@echo "Dist built and uploaded to pypi.org"
-	@echo "Test the install `make install-release`"
+	@echo "Test the install:"
+	@echo "    make install-release"

 dev-env :
 	pip install -r server/requirements-dev.txt
@@ -111,15 +114,19 @@ install-dev : uninstall

 # install from test.pypi to test your release
 install-release-test : uninstall
-	pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
+	pip install --no-cache-dir --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
 	@echo "Installed cellxgene from test.pypi.org, now run and smoke test"

 # install from pypi to test your release
 install-release : uninstall
-	pip install cellxgene
+	pip install --no-cache-dir cellxgene
 	@echo "Installed cellxgene from pypi.org"

+# install from dist
+install-dist : uninstall
+	pip install dist/cellxgene*.tar.gz
+
 uninstall :
-	yes | pip uninstall cellxgene || true
+	pip uninstall -y cellxgene || :

 .PHONY : install install-dev install-release-test install-release uninstall
@@ -5,27 +5,34 @@ from flask_caching import Cache
 from flask_compress import Compress
 from flask_cors import CORS

-from .rest_api.rest import get_api_resources
-from .util.utils import Float32JSONEncoder
-from .web import webapp
+from server.app.rest_api.rest import get_api_resources
+from server.app.util.utils import Float32JSONEncoder
+from server.app.web import webapp

-REACTIVE_LIMIT = 1_000_000

-app = Flask(__name__, static_folder="web/static")
-app.json_encoder = Float32JSONEncoder
-cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860_000})
-Compress(app)
-CORS(app)
+class Server:
+    def __init__(self):
+        self.data = None
+        self.cache = Cache(config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860_000})
+        self.app = None

-# Config
-SECRET_KEY = os.environ.get("CXG_SECRET_KEY", default="SparkleAndShine")
+    def create_app(self):
+        self.app = Flask(__name__, static_folder="web/static")
+        self.app.json_encoder = Float32JSONEncoder
+        self.cache.init_app(self.app)
+        Compress(self.app)
+        CORS(self.app)

-app.config.update(SECRET_KEY=SECRET_KEY)
+        # Config
+        SECRET_KEY = os.environ.get("CXG_SECRET_KEY", default="SparkleAndShine")
+        self.app.config.update(SECRET_KEY=SECRET_KEY)
+        self.app.config.update(SCRIPTS=[])

-# Application Data
-data = None
+        resources = get_api_resources()
+        self.app.register_blueprint(webapp.bp)
+        self.app.register_blueprint(resources.blueprint)
+        self.app.add_url_rule("/", endpoint="index")

-resources = get_api_resources()
-app.register_blueprint(webapp.bp)
-app.register_blueprint(resources.blueprint)
-app.add_url_rule("/", endpoint="index")
+    def attach_data(self, data, title="Demo"):
+        self.app.config.update(DATASET_TITLE=title)
+        self.app.data = data
@@ -11,13 +11,27 @@ Sort order for methods


 class CXGDriver(metaclass=ABCMeta):
-    def __init__(self, data, args):
-        self.data = self._load_data(data)
-        self.layout_method = args["layout"]
-        self.diffexp_method = args["diffexp"]
-        self.max_category_items = args["max_category_items"]
-        self.diffexp_lfc_cutoff = args["diffexp_lfc_cutoff"]
-        self.cluster = None
+    def __init__(self, data=None, args={}):
+        self.config = self._get_default_config()
+        self.config.update(args)
+        if data:
+            self._load_data(data)
+        else:
+            self.data = None
+
+    def update(self, data=None, args={}):
+        self.config.update(args)
+        if data:
+            self._load_data(data)
+
+    @staticmethod
+    def _get_default_config():
+        return {
+            "layout": None,
+            "diffexp": None,
+            "max_category_items": None,
+            "diffexp_lfc_cutoff": None
+        }

    @property
    def features(self):
@@ -27,18 +41,15 @@ class CXGDriver(metaclass=ABCMeta):
            "diffexp": {"available": False},
        }
        # TODO - Interactive limit should be generated from the actual available methods see GH issue #94
-        if self.layout_method:
+        if self.config["layout"]:
            # TODO handle "var" when gene layout becomes available
            features["layout"]["obs"] = {"available": True, "interactiveLimit": 50000}
-        if self.diffexp_method:
+        if self.config["diffexp"]:
            features["diffexp"] = {"available": True, "interactiveLimit": 50000}
-        if self.cluster:
-            features["cluster"] = {"available": True, "interactiveLimit": 50000}
        return features

-    @staticmethod
    @abstractmethod
-    def _load_data(data):
+    def _load_data(self, data):
        pass

    @abstractmethod
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Bruce Martin	c6252825f3	release 0.10.0 (#794 )	2019-05-29 16:55:43 -07:00
Bruce Martin	4b96b3a635	fix incompatibility of flask reload and port searching (#793 ) * WIP * add --developer; fix incompatibility of --port and --debug * put REST tests on separate ports * PR review	2019-05-29 16:38:57 -07:00
Colin Megill	862d8feb5e	x (#792 )	2019-05-29 12:23:30 -04:00
Bruce Martin	1ef77d1596	fix misconfiguration for history management (#787 )	2019-05-24 21:01:11 -07:00
Bruce Martin	3dc45d6330	do not hard-wire column names in annotations (#785 ) * enforce column name uniqueness for obs and var * parameterize the column name containing obs and var user-readable names * use the new annotation index value from schema * update f/e unit tests * PR review suggestions * lint	2019-05-24 21:00:54 -07:00
Bruce Martin	a8c2e408d1	update to latest anndata and remove restriction on scipy (#790 )	2019-05-24 11:23:35 -07:00
Bruce Martin	e941c1a496	scaling omitted from event handlers (#789 ) * scaling omitted from event handlers * fix smoke tests	2019-05-24 07:00:04 -07:00
Colin Megill	a657eb3152	Logo (#782 ) * logo, black * fixes * remove template, move header	2019-05-23 11:47:24 -04:00
Bruce Martin	ef7c26e799	correctly handle selection of trunctated categories (#781 )	2019-05-23 08:46:22 -07:00
Bruce Martin	49af278de7	cleanup memoiziation in graph component (#783 )	2019-05-22 17:37:18 -07:00
Bruce Martin	2357d0c1b8	layout change UI (#776 ) * add layout to schema * add layout choice action and reducer * multi layout UI * update unit tests * add missing file * update test schema * fix duplicate test id * fix tabs * PR lint * fix pytest	2019-05-22 13:21:33 -07:00
Charlotte Weaver	63af79d3f8	Add developer guidelines (#769 ) * Add developer guidelines * minor formatting * PR clarifications/lint * more pr fixes * link fix * below->above * pr suggestions	2019-05-21 13:57:34 -07:00
Bruce Martin	fcc05f6a00	coordinate system fixes for embedded graph (#768 ) * change pan speed to 1 per issue #722 * correct handle scaling of graph when aspect ratio less than one * add package lock * add invert to our scale functions * correctly transform to/from gl coordinates * remove unused import * fix naming of import * update smoke tests	2019-05-20 14:22:41 -07:00
Bruce Martin	de3407d875	change scripts to support windows (#775 )	2019-05-20 11:42:57 -07:00
Charlotte Weaver	2d4e827bea	wait for element before getting text/html (#777 )	2019-05-20 11:35:53 -07:00
Bruce Martin	1fa4838863	npm (js) package dependency updates (#765 ) * JS package dependency updates * additional package updates * more package version updates * more js package updates * more JS dependency updates	2019-05-20 10:11:49 -07:00
Charlotte Weaver	ab4c74a321	remove psutil (#773 )	2019-05-18 10:53:45 -07:00
Charlotte Weaver	82d65addec	always run smoke tests (#772 )	2019-05-18 10:50:17 -07:00
Charlotte Weaver	e2ad28a510	exclude recent scipy versions (#770 )	2019-05-17 15:13:22 -07:00
Bruce Martin	efa1709158	add multi-layout support to back-end (#766 ) * add multi-layout support to back-end * remove obsolete code * temporary code to apply heuristic choice of default layout * fix tests * update python tests * more py lint * PR review changes * more PR lint * PR lint	2019-05-16 14:49:22 -07:00
Charlotte WeaverandTony Tung	d6040f687a	port retry (#761 ) * WIP * import find_available_port method * move method to utils so I can add to eventually add to gui * add fixed-port flag to tests * Update server/utils/utils.py Co-Authored-By: Tony Tung <tonytung@merly.org> * pr review suggestions * pr review suggestions * fix outdated package.json * update error message * simplify find_available_port function * Auto scan for ports unless port is specified. * fix tests * fix comment for find_available_port * lint error * differentiate port error from generic os error * add errno to OSerror * pr review fixes * raise e -> raise * oserror -> socket error	2019-05-14 14:04:13 -07:00
Bruce Martin	b9a1e30652	large file size guardrails (#763 ) * large file guardrails * fix lint * PR review * remove unused import * use standard slice for CSR * revert change	2019-05-13 18:13:16 -07:00
Bruce Martin	7adac5d004	create occupancy stacks for all category values, not just top N values (#764 )	2019-05-13 11:22:46 -07:00
Charlotte Weaver	2354731083	install from dist instead of build on travis (#760 )	2019-05-09 15:32:06 -07:00
Charlotte Weaver	d522cc8f91	Add clipping test to smoke tests (#757 ) * Add clipping test to smoke tests * devtools on in debug	2019-05-09 15:31:55 -07:00
Charlotte Weaver	86eb01eb2c	improve release process (#752 ) * Add --no-cache-dir to make release-install target Prevents installing from cache so you get the freshest release * Testing releases is not optional * Updated release documentation	2019-05-08 09:36:00 -07:00
Charlotte Weaver	8a94b1e086	fix #754 (#755 )	2019-05-07 12:43:20 -07:00
Bruce Martin	846b8d15bd	lodash cleanup (#747 ) * add own range() function * lodash cleanup * remove redundant fill range implementations * remove use of _.get * sync test babel config with build * update tests to match new range implementation	2019-05-06 20:28:32 -04:00
Charlotte Weaver	c12cb2424a	release bugfix (#749 ) * add __init__.py * bump version	2019-05-06 10:16:36 -07:00
Bruce Martin	1471d6b214	release 0.9.0 (#746 )	2019-05-04 08:41:56 -07:00
Charlotte Weaver	98b63fa9ea	Moved to python threads (#745 ) So we could use daemon threads	2019-05-03 14:48:48 -07:00
Bruce Martin	0aa0f641ab	improve selection interaction with clip changes (#744 ) * brush interactions with underlying dataframe updates improved * improve comment * reset selection state upon clip	2019-05-03 11:14:30 -07:00
Charlotte Weaver	77a4495b28	Update README.md (#743 ) Fixes #702	2019-05-02 16:54:39 -07:00
Bruce Martin	b521a17ffd	improve column access speed for sparse matrices (#742 ) * improve column access speed for sparse matrices * add FAQ entry about data format performance * add note about using --sparse flag for prepare command * clean up for PR review * Update docs/faq.md Co-Authored-By: bkmartinjr <bruce@chanzuckerberg.com> * improvements to big data faq	2019-05-01 15:44:01 -07:00
Colin Megill	86bf64e793	avoid overflow on clip dialogue (#740 )	2019-05-01 15:41:54 -04:00
Charlotte Weaver	8a72010768	PyQt5 -> PySide2 (#738 )	2019-05-01 09:03:16 -07:00
Sidney Bell	a08e19bbd0	Clip continuous values based on percentile cutoffs (#672 ) * Add numeric inputs for percentiles * Define initial values for percentile cutoffs in world reducer * add percentil to crossfilter dimensions * worldEqUniverse now handles cloned worlds * add Dataframe.mapColumns * Wire up handlers for percentile inputs * World reducer and stateManager know about continuousPercentileMin/Max * Create world as universe clone (not pointer) to avoid clobbering vals * Define basic actions for setting continuousPercentileMin/Max * Under the hood, deal with percentiles between 0 and 1 * Move percentile inputs to visualization settings menu * Fix padding for undo/redo buttons * Trigger world rebuild from percentile actions * BROKEN - pseudocode for clamping dataframe by percentiles upon world rebuild * fix error handling on clip quantiles; start world clipping implementation * more unclipped reorg * rename crossfilter.percentile to quantile * simplify schema access * update continuous legend when scale changes * update color cache when clip changes * clip obs annotations and var data when clip quantile changes * use own fromEntries * fix tests * stable non-finite float sort/search * clarify comments * fix syntax typo * use new stand-alone clip * clip expresssion data * add select tests for non-finite scalars * basic styles * clip UI now requires explicit commit * reset enable/disable accounts for clip percentiles * better error messages * fix bug in undo interaction with programatic min brush selection * small refactoring * support clipping of int data * do not perform unnecessary summarizations * improve caching of dataframe compiled columns * add percentile precompute to Dataframe.summarize * use Dataframe.summarize for clip percentiles * remove obsolete quantile code from corssfilter * histogram scale and label Y axis, add unclipped X range labels * layout tweaks * scatterplot now updates when clip changes * improve comments * remove debugging comment * rework clip number entry validation for usability * ui tweaks to histogram colors and layout * enable undo/redo for clip user action * refine UI on clip value entry * api cleanup * update confusing comment * clarify purpose of isValidDigitKeyEvent * fix misleading comment * apply appropriate button-group classes; do not mix span and div * variable name and comment changes suggested in PR review * rename sort to sortArray; remove unused and dead code path * naming changes suggested in PR review * code review improvements for clarity * more small changes from PR review * lint fixes for PR review * fix spelling error * clarify that function performs in-place modification of world * add comment to clarify intent of range operation * fix bad indents in comments * clean up __columnsAccessor comments and code * improve comments around clipPredicate * field name consistency * improve comment on quantiles params	2019-04-30 16:20:10 -07:00
Charlotte Weaver	9f10d8095a	GUI app (experimental) (#730 ) * add default config * first pass * flake8 * cleanup * first pass at using qthreads * cleanup * WIP * better error handling * improved UI * bugfix * fix merge bugs * import order * cleanup * make gui requirements optional * pr review requested changes * Update server/gui/main.py Co-Authored-By: csweaver <charlottesweaver@gmail.com> * pr review request * qt child class methods -> camelCase * whitespace	2019-04-30 12:52:15 -07:00
Charlotte Weaver	ea187f48e0	add scripts from cli (#680 ) * add scripts from cli * add warning when including scripts * flake8 fixes * confirm scripts injection	2019-04-22 12:30:57 -07:00
Charlotte Weaver	9c6273eb94	core library (#711 ) * move app creation to function * create engine without load * flake 8 fixes * cleanup original scanpy test * add default config * handle missing data * test data changes * unify update * load data isn't static anymore * make app a class	2019-04-22 12:24:07 -07:00
Justin Kiggins	1f735abe2b	updates the roadmap & reorganizes the README (#712 )	2019-04-18 13:10:03 -07:00
Colin Megill	b878b0f93c	Gene typeahead stale state (#714 ) * reimplementing suggest * resolve stale state	2019-04-17 14:03:10 -04:00
Charlotte Weaver	ad0a3c939c	faster ci (#713 ) * parallelize docker build * cache npm too * testing skip install	2019-04-15 16:34:40 -07:00
Charlotte Weaver	d581a0d460	color by gene smoketest (#707 ) * add test for color by gene expression gene expression and metadata color by are handled differently * error on console.error not just on thrown errors	2019-04-11 17:06:36 -07:00
Bruce Martin	9044b8d85d	fix color-by regression in toggle (#705 ) * fix color-by regression in toggle * fix incorrect field reference	2019-04-11 12:47:38 -07:00
Charlotte Weaver	73852dd6f4	use flask's json (#703 )	2019-04-09 12:59:09 -07:00
Bruce Martin	5e02408732	latent bug in color toggle (#701 )	2019-04-09 10:50:35 -07:00
Sidney Bell	34e5a91dc6	Add `calculate_qc_metrics` to `prepare (#697 ) * Calculate QC metrics * Add QC metrics to prepare section of readme * Add pointer to scanpy qc metrics function * Don't explicitly pass qc flag as arg * Add explicit toggle for run-qc/skip-qc * Move qc metrics calculation to separate step/function	2019-04-09 08:56:07 -07:00
Bruce Martin	7275d9d4dc	Graph selection state management and history bug fixes (#679 ) * save graph selection in redux state * fix old graph brush select regressions * refactor graph brush selection to work with undo/redo * update tests to match new crossfilter spatial select API * graph selection state now in redux * remove dead code * sync graph selection with redux state; improvements to undoable machinery * fix regression in undoable * differentiate graph selection cancel from deselect action * simplify calculation * remove debugging code * fix responsive repaint bug in graph selection tool * undoable debugging and code cleanliness * undoable action filter state now merges, rather than replaces * improve comments * add debounce to undoable action filter; improve comments and debug sanity check code * comments * fix undoable bug with clear scatterplot actions * disable undoable debug flag * cleanup API and comments around statemachine * add test id attribute to lasso * add better error handling for gene fetch requests	2019-04-08 15:53:12 -07:00
Charlotte Weaver	c9a8e3ea42	Add manual UX tests (#699 )	2019-04-08 13:23:17 -07:00
Bruce Martin	71505abe4c	toggle color-by when repeatedly picked by the user (#696 ) * toggle color-by when repeatedly picked * remove debugging code	2019-04-08 10:57:47 -07:00
Sidney Bell	c9a56fa73a	Update scanpy version (#688 ) Update to version 1.3.7	2019-04-04 12:16:30 -07:00
Charlotte Weaver	3167e38993	update docs to reflect python 3.7 support (#685 )	2019-04-02 09:24:50 -07:00
Charlotte Weaver	ce8326ddea	bump version (#684 )	2019-04-01 14:42:22 -07:00
Charlotte Weaver	5d4b11d287	Better smoke tests (#669 )	2019-03-28 16:47:42 -07:00
Charlotte Weaver	096d477893	use Math.abs when calculating lasso area (#673 )	2019-03-27 21:02:31 -07:00
Bruce Martin	153f240c43	brushable histogram brush selection responds to state updates (#670 ) * add continuousSelection reducer and make histo respond to state changes * correctly handle uninitialized state for brush move * PR review tweaks	2019-03-25 15:02:39 -07:00
Gökçen Eraslan	df46db965a	bin/build-client.sh does not exist. (#671 )	2019-03-25 10:26:27 -07:00
Bruce Martin	7ef5203564	Undo/redo (#659 ) * immutable crossfilter * first cut at reducer refactor with cascade model * add initial redo/undo implementation * small optimization * integrate expression with history * add tests for new reducers and fix a couple of small initialization bugs * treat tiny lasso selections as a clear * better function name for clarity * fix undo for differential expression * remove logging * fix regression due to bad merge * cleanup and comments for clarity * improve undoable configuration for flexibility * fix stale comments * remove debugging code from production build * rename categoricalSelectionState * rename file * improve comments	2019-03-22 14:53:48 -07:00
Bruce Martin	9420abfacc	add Babel support for JS optioning chaining and nullish coallescing operators (#664 )	2019-03-22 14:51:51 -07:00
Sidney Bell	16f93397ae	Support `diffmap` and `phate` layouts. Explicitly handle embeddings with >2 components. (#662 ) * Add diffmap and phate to supported embeddings * Explicitly pull the first two components of any given layout	2019-03-21 19:06:05 -07:00
Colin Megill	016a4a422a	Procedurally resize brush (#650 ) * add on brush end event * brush snap move * resize brush	2019-03-20 14:46:20 -07:00
Bruce Martin	996b06cecc	refactoring - immutable crossfilter (#647 ) * immutable crossfilter * PR review changes	2019-03-20 14:32:35 -07:00
Bruce Martin	571b7387e7	remove dead code from globals (#651 )	2019-03-19 06:22:39 -07:00
Charlotte Weaver	b6d468376a	py37 fixes (#646 ) * add python version for nightly build * update numpy version required older versions interact poorly on anaconda + python 3.7	2019-03-14 14:51:55 -07:00
Charlotte Weaver	d8fc7e40a1	Support python3.7 (#645 ) * Support python3.7 * add 3.7 env to travis	2019-03-14 12:08:46 -07:00
Charlotte Weaver	e875ed739b	More informative out-of-memory error (#644 ) * more informative error message: memory error * flake fix	2019-03-12 14:19:00 -07:00
Bruce Martin	aa5ce4a2f1	correctly toggle group selection in categorical metadata (#640 )	2019-03-12 10:49:29 -07:00
Bruce Martin	80969012c9	do not reset color scale when dismissing scatterplot (#637 )	2019-03-12 10:49:01 -07:00
Bruce Martin	caaee7e9bf	do not reset color-by when subsetting world (#636 ) * do not reset colors when subsetting to world * revert diffexp state change	2019-03-12 10:48:35 -07:00
Charlotte Weaver	f96fd36ecb	Update test ui names (#638 ) * Fixed changed testid * Added debug mode for running tests	2019-03-12 10:17:09 -07:00
Bruce Martin	495dc55144	remove obsolete URL middleware (#639 )	2019-03-11 16:03:33 -07:00
Colin Megill	8795f0f32c	Add test ids and classes (#633 ) * data test ids and classes * suggest	2019-03-07 13:01:41 -08:00
Colin Megill	a0f54b4871	Add menu (#631 ) * add menu * Added library versions to config and tests * add template version number	2019-03-06 11:56:46 -05:00
Charlotte Weaver	68dfbcc2eb	Add force graph to docs and example dataset (#630 )	2019-03-05 20:59:54 -08:00
Alex Wolf	82493d1019	Add forced directed graph drawing to allowed layout options (#626 ) * added forced directed graph drawing layout options * added line breaks for 120 character limit	2019-03-05 13:10:07 -08:00
Bruce Martin	3d6df0c044	implement improved disable/enable of Reset UI (#628 )	2019-03-05 08:38:55 -08:00
Charlotte Weaver	2583016693	Fix bug where prod would automatically run after release stage 2 (#625 ) * Fix bug where prod would automatically run after release stage 2 * fixed make release-burned	2019-03-01 20:20:16 -08:00