release 0.10.1 (#797)

* add gutter to embedding canvas

* improve layout scale and translate

* fix lint

* pin tables to version 3.5.1

* fix lasso coordinate smoke tests

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.8.0
+current_version = 0.10.1
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"

.gitignore

@@ -27,6 +27,9 @@ server/app/web/templates/index\.html
 .ipynb_checkpoints
 *.ipynb
 
+# cefpython
+error.log
+
 # misc
 .DS_Store
 npm-debug.log

.travis.yml

@@ -4,14 +4,14 @@ sudo: required
 node_js:
   - 8
 cache:
-  pip: true
+  - pip
+  - npm
 install:
   - set -eo pipefail
   - pip install flake8
-  - make build
-  - make install
+  - make pydist
+  - make install-dist
   - pip install -r server/requirements-dev.txt
-  - docker build .
 
 jobs:
   include:
@@ -21,8 +21,11 @@ jobs:
   - name: "Branch Tests 3.6"
     python: "3.6"
     script: ./travis-build.sh
+  - name: "Docker Build"
+    install: skip
+    python: "3.6"
+    script: docker build .
   - name: "Smoke Tests"
     python: "3.6"
-    if: branch = master AND type = cron
     script:
       - npm run --prefix client/ smoke-test

README.md

@@ -2,224 +2,61 @@
 
 > an interactive explorer for single-cell transcriptomics data
 
-`cellxgene` is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
+[![DOI](https://zenodo.org/badge/105615409.svg)](https://zenodo.org/badge/latestdoi/105615409)
+
+_cellxgene_ (pronounced "sell-by-jean") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
 
 <img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
 
-## getting started
+- Want to install and use cellxgene? Visit the [cellxgene docs](https://chanzuckerberg.github.io/cellxgene/).
+- Want to see where we are going? Check out [our roadmap](ROADMAP.md).
+- Want to contribute? See our [contributors guide](#Contributing)
 
-You'll need **python 3.6** and **Google Chrome**. (_Warning_: Python 3.7 is **not** supported at this time)
-The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).
+## quick start
 
-To install run
+To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](https://chanzuckerberg.github.io/cellxgene/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
 
-```
+Install the package.
+``` bash
 pip install cellxgene
 ```
 
-To start exploring a dataset call
+Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
 
-```
-cellxgene launch dataset.h5ad --open
+``` bash
+curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
 ```
 
-If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
-
-```
+Launch _cellxgene_
+``` bash
 cellxgene launch pbmc3k.h5ad --open
 ```
 
-You should see your web browser open with the following
+To learn more about what you can do with _cellxgene_, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-stared/) guide.
 
-<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+## get in touch
 
-**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
 
-There are several options available, such as:
+## where we are going
 
-- `--layout` to specify the layout as `tsne`, `umap`, `diffmap`, `phate`, `draw_graph_fa`, or `draw_graph_fr`
-- `--title` to show a title on the explorer
-- `--open` to automatically open the web browser after launching (OS X only)
+Our goal is to enable teams of computational and experimental
+biologists to collaboratively gain insight into their single-cell RNA-seq data.
 
-To see all options call
+There are 4 key features we plan to implement in the near term.
 
-```
-cellxgene launch --help
-```
+- Click install and launch
+- Manual annotation workflows
+- Toggle embeddings
+- Gene information
 
-There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+For more detail on these features and where we are going, see [our roadmap](ROADMAP.md).
 
-## data formatting
+## contributing
 
-### assumptions
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
 
-The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
-
-- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
-- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
-- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_umap` for the `umap` layout)
-- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
-
-### prepare
-
-The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
-
-To prepare from an existing `.h5ad` file use
-
-```
-cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
-```
-
-This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
-
-Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
-
-```
-cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
-```
-
-To see all options call
-
-```
-cellxgene prepare --help
-```
-
-**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
-
-```
-pip install cellxgene[louvain]
-```
-
-If the aforementioned optional package installation fails, you can also install these packages directly:
-
-```
-pip install python-igraph louvain>=0.6
-```
-
-## conda and virtual environments
-
-If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
-
-```
-conda create --yes -n cellxgene python=3.6
-conda activate cellxgene
-pip install cellxgene
-```
-
-Or you can create a virtual environment by using
-
-```
-ENV_NAME=cellxgene
-python3.6 -m venv ${ENV_NAME}
-source ${ENV_NAME}/bin/activate
-pip install cellxgene
-```
-
-## docker
-
-We have included a dockerfile to conveniently run cellxgene from docker.
-
-1. Build the image `docker build . -t cellxgene`
-2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
-   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
-
-## FAQ
-
-<details>
-
-<summary> questions about data formatting </summary>
-  
-<hr>
-  
-> Someone sent me a directory of `10X-Genomics` data with a `mtx` file and I've never used `scanpy`, can I use `cellxgene`?
-
-Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
-
-```
-cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
-```
-
-Depending on the size of the dataset, this may take some time. Once it's done, call
-
-```
-cellxgene launch data-processed.h5ad --layout=umap --open
-```
-
-And your web browser should open with an interactive view of your data.
-
-<hr>
-
-> In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
-
-Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using
-
-```
-pip install cellxgene[louvain]
-```
-
-<hr>
-
-> I ran `prepare` and I'm getting results that look unexpected
-
-You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
-
-```
-cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
-```
-
-It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
-
-<hr>
-
-> I have extra metadata that I want to add to my dataset
-
-Currently this is not supported directly, but you should be able to do this manually using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
-
-<hr>
-
-> What part of the anndata objects does cellxgene pull in for visualization?
-
-- `.obs` and `.var` annotations are use to extract metadata for filtering
-- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
-- `.obsm` is used for layout
-
-</details>
-
-<details>
-  
-<summary> questions about installing and building </summary>
-
-<hr>
-
-> I tried to `pip install cellxgene` and got a weird error about missing paths to an HDF5 library?  
-
-You probably just need to install HDF5 first. If you're on a mac, you can simply `brew install hdf5` and then try `pip install cellxgene` again.  
-
-> I tried to `pip install cellxgene` and got another weird error I don't understand  
-
-This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.  
-
-<hr>
-
-> I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
-
-This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
-
-</details>
-
-<details>
-
-<summary> questions about algorithms </summary>
-
-<hr>
-
-> How are you computing and sorting differential expression results?
-
-Currently we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
-
-</details>
-
-## developer guide
+### developer guide
 
 This project has made a few key design choices
 
@@ -231,7 +68,7 @@ Depending on your background and interests, you might want to contribute to the
 
 If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
 
-- python 3.6
+- python 3.6+
 - node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
 
 Then clone the project
@@ -256,20 +93,7 @@ You can start the app while developing either by calling `cellxgene` or by calli
 
 If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.
 
-## development roadmap
-
-`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact)
-
-- **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
-- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene` to render the results of these analyses when they have been performed
-- **Deploy to web** Many projects release public data browser websites alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
-- **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data
-
-## contributing
-
-We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
-
-## inspiration and collaboration
+## inspiration
 
 We've been heavily inspired by several other related single-cell visualization projects, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), and many others. We hope to explore collaborations where useful as this community works together on improving interactive visualization for single-cell data.
 
@@ -279,9 +103,13 @@ We have been working closely with the [`scanpy`](https://github.com/theislab/sca
 
 We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
 
-## help and contact
+## core team
 
-Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+- Colin Megill, frontend & product design
+- Charlotte Weaver, software engineer
+- Bruce Martin, software engineer
+- Sidney Bell, computational biologist
+- Justin Kiggins, product manager
 
 ## reuse
 

ROADMAP.md

@@ -0,0 +1,54 @@
+# cellxgene roadmap
+
+We are very exited for _cellxgene_ to become a valuable tool in collaborations
+between computational biologists and experimental biologists working on
+single-cell transcriptomics data. _cellxgene_ is in active development, and we
+would love to include the community as we plan new features to work on. If you
+have questions of feedback about this roadmap, please submit an issue on
+GitHub.
+
+Please note: this roadmap is subject to change.
+
+*Last updated: April 11, 2019*
+
+## what we are building now
+
+In the near term, our goal is to enable teams of computational and experimental
+biologists to collaboratively explore and annotate their single-cell RNA-seq data.
+
+There are 4 key features we plan to implement in the near term.
+
+- Click install and launch
+- Manual annotation workflows
+- Toggle embeddings
+- Gene information
+
+### simple install and launch
+
+The command line interface for installing and launching cellxgene is a barrier
+for users who are not used to Python or using the command line. We plan to
+support installation and launch of cellxgene on Mac and Windows. See
+[Issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for more details.
+
+### manual annotation workflows
+
+The exploratory visualization that cellxgene offers is critical for manual
+annotation workflows, especially in collaborative environments. We plan to
+support manually annotate cells with labels (i.e., cell type or QC flags) for
+downstream analysis. See [Issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524)
+for more details.
+
+### toggle embeddings
+
+While a single dataset may have multiple embeddings calculated (tSNE, umap, in
+situ coordinates, trajectories, etc), cellxgene currently requires the user to select the
+embedding to use in the main layout at launch. We plan to support letting users
+toggle between any embedding present in a file from the cellxgene interface.
+See [Issue #594](https://github.com/chanzuckerberg/cellxgene/issues/594) for details.
+
+### gene information
+
+Differential expression returns only the names of genes, but no additional information
+about gene metadata, function, or known associations. We plan to help users learn
+more about genes they discover by exposing additional gene metadata. See
+[Issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for details.

client/__tests__/e2e/data.js

@@ -1,6 +1,6 @@
 export const datasets = {
   pbmc3k: {
-    title: "cellxgene: pbmc3k",
+    title: "pbmc3k",
     dataframe: {
       nObs: "2638",
       nVar: "1838",
@@ -26,8 +26,8 @@ export const datasets = {
     cellsets: {
       lasso: [
         {
-          "coordinates-as-percent": { x1: 0.25, y1: 0.25, x2: 0.35, y2: 0.35 },
-          count: "26"
+          "coordinates-as-percent": { x1: 0.05, y1: 0.25, x2: 0.15, y2: 0.35 },
+          count: "101"
         }
       ],
       categorical: [
@@ -91,8 +91,8 @@ export const datasets = {
         }
       },
       lasso: {
-        "coordinates-as-percent": { x1: 0.45, y1: 0.45, x2: 0.5, y2: 0.5 },
-        count: "67"
+        "coordinates-as-percent": { x1: 0.45, y1: 0.05, x2: 0.65, y2: 0.15 },
+        count: "46"
       }
     },
     scatter: {
@@ -100,6 +100,23 @@ export const datasets = {
     },
     pan: {
       "coordinates-as-percent": { x1: 0.75, y1: 0.75, x2: 0.35, y2: 0.35 }
+    },
+    features: {
+      panzoom: {
+        lasso: {
+          "coordinates-as-percent": { x1: 0.3, y1: 0.3, x2: 0.5, y2: 0.5 },
+          count: "24"
+        }
+      }
+    },
+    clip: {
+      min: "30",
+      max: "70",
+      metadata: "n_genes",
+      gene: "S100A8",
+      "coordinates-as-percent": { x1: 0.25, y1: 0.5, x2: 0.55, y2: 0.5 },
+      count: "392",
+      "gene-cell-count": "421"
     }
   }
 };

client/__tests__/e2e/e2e.test.js

@@ -1,3 +1,8 @@
+/*
+Smoke test suite that will be run in Travis CI
+
+Tests included in this file are expected to be relatively stable and test core features
+ */
 import puppeteer from "puppeteer";
 import { appUrlBase, DEBUG, DEV, DATASET } from "./config";
 import { puppeteerUtils, cellxgeneActions } from "./puppeteerUtils";
@@ -20,7 +25,17 @@ beforeAll(async () => {
   page = await browser.newPage();
   await page.setViewport(browserViewport);
   if (DEV || DEBUG) {
-    page.on("console", msg => console.log(`PAGE LOG: ${msg.text()}`));
+    page.on("console", async msg => {
+      // If there is a console.error but an error is not thrown, this will ensure the test fails
+      if (msg.type() === "error") {
+        const errorMsgText = await Promise.all(
+          // TODO can we do this without internal properties?
+          msg.args().map(arg => arg._remoteObject.description)
+        );
+        throw new Error(`Console error: ${errorMsgText}`);
+      }
+      console.log(`PAGE LOG: ${msg.text()}`);
+    });
   }
   page.on("pageerror", err => {
     throw new Error(`Console error: ${err}`);
@@ -167,7 +182,6 @@ describe("diffexp", async () => {
     );
   });
 });
-//
 
 describe("subset/reset", async () => {
   test("subset - cell count matches", async () => {
@@ -256,6 +270,35 @@ describe("scatter plot", async () => {
   });
 });
 
+describe("clipping", async () => {
+  test("clip continuous", async () => {
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.metadata}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip.count);
+  });
+
+  test("clip gene", async () => {
+    await utils.typeInto("gene-search", data.clip.gene);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(`[data-testid='histogram-${data.clip.gene}']`);
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.gene}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip["gene-cell-count"]);
+  });
+});
+
 // interact with UI elements just that they do not break
 describe("ui elements don't error", async () => {
   test("color by", async () => {
@@ -267,6 +310,15 @@ describe("ui elements don't error", async () => {
     }
   });
 
+  test("color by for gene", async () => {
+    await utils.typeInto("gene-search", data.genes.search);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(
+      `[data-testid='histogram-${data.genes.search}']`
+    );
+    await utils.clickOn(`colorby-${data.genes.search}`);
+  });
+
   test("pan and zoom", async () => {
     await utils.clickOn("mode-pan-zoom");
     const panCoords = await cxgActions.calcDragCoordinates(

client/__tests__/e2e/feature.test.js

@@ -0,0 +1,118 @@
+/*
+NOT run in Travis CI
+
+UX tests using puppeteer to be run locally.
+
+To run locally, ensure you are running the client is running on port 3000.
+Then run jest --verbose false --config __tests__/e2e/e2eJestConfig.json feature.
+ */
+
+import puppeteer from "puppeteer";
+import { appUrlBase, DEBUG, DEV, DATASET } from "./config";
+import { puppeteerUtils, cellxgeneActions } from "./puppeteerUtils";
+import { datasets } from "./data";
+
+let browser, page, utils, cxgActions, spy;
+const browserViewport = { width: 1280, height: 960 };
+let data = datasets[DATASET].features;
+
+if (DEBUG) jest.setTimeout(100000);
+if (DEV) jest.setTimeout(10000);
+
+beforeAll(async () => {
+  const browserParams = DEV
+    ? { headless: false, slowMo: 5 }
+    : DEBUG
+    ? { headless: false, slowMo: 100, devtools: true }
+    : {};
+  browser = await puppeteer.launch(browserParams);
+  page = await browser.newPage();
+  await page.setViewport(browserViewport);
+  if (DEV || DEBUG) {
+    page.on("console", msg => console.log(`PAGE LOG: ${msg.text()}`));
+  }
+  page.on("pageerror", err => {
+    throw new Error(`Console error: ${err}`);
+  });
+  utils = puppeteerUtils(page);
+  cxgActions = cellxgeneActions(page);
+});
+
+beforeEach(async () => {
+  await page.goto(appUrlBase);
+});
+
+afterAll(() => {
+  if (!DEBUG) {
+    browser.close();
+  }
+});
+
+describe("zoom interaction", async () => {
+  // Skip this test since UI is to hide lasso path when switching modes
+  test.skip("lasso visible after switching modes to pan/zoom", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    await utils.clickOn("mode-pan-zoom");
+    await utils.waitByID("lasso-element", { visible: true });
+  });
+
+  test("pan zoom mode resets lasso selection", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    const initialCount = await cxgActions.cellSet(1);
+    expect(initialCount).toBe(data.panzoom.lasso.count);
+    await utils.clickOn("mode-pan-zoom");
+    await utils.clickOn("mode-lasso");
+    const modeSwitchCount = await cxgActions.cellSet(1);
+    expect(modeSwitchCount).toBe(initialCount);
+  });
+
+  test("lasso moves after pan", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    const initialCount = await cxgActions.cellSet(1);
+    expect(initialCount).toBe(data.panzoom.lasso.count);
+    await utils.clickOn("mode-pan-zoom");
+    const panCoords = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      panCoords.start,
+      panCoords.end,
+      false
+    );
+    await utils.clickOn("mode-lasso");
+    const panCount = await cxgActions.cellSet(2);
+    expect(panCount).toBe(initialCount);
+  });
+});

client/__tests__/e2e/puppeteerUtils.js

@@ -1,21 +1,40 @@
 export const puppeteerUtils = puppeteerPage => ({
-  async waitByID(testid) {
-    return await puppeteerPage.waitForSelector(`[data-testid='${testid}']`);
+  async waitByID(testid, props = {}) {
+    return await puppeteerPage.waitForSelector(
+      `[data-testid='${testid}']`,
+      props
+    );
   },
 
-  async waitByClass(testclass) {
+  async waitByClass(testclass, props = {}) {
     return await puppeteerPage.waitForSelector(
-      `[data-testclass='${testclass}']`
+      `[data-testclass='${testclass}']`,
+      props
     );
   },
 
   async typeInto(testid, text) {
     // only works for text without special characters
     await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
     // type ahead can be annoying if you don't pause before you type
-    await puppeteerPage.click(`[data-testid='${testid}']`);
+    await puppeteerPage.click(selector);
     await puppeteerPage.waitFor(200);
-    await puppeteerPage.type(`[data-testid='${testid}']`, text);
+    await puppeteerPage.type(selector, text);
+  },
+
+  async clearInputAndTypeInto(testid, text) {
+    await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
+    // only works for text without special characters
+    // type ahead can be annoying if you don't pause before you type
+    await puppeteerPage.click(selector);
+    await puppeteerPage.waitFor(200);
+    // select all
+
+    await puppeteerPage.click(selector, {clickCount: 3})
+    await puppeteerPage.keyboard.type("Backspace")
+    await puppeteerPage.type(selector, text);
   },
 
   async clickOn(testid) {
@@ -25,11 +44,13 @@ export const puppeteerUtils = puppeteerPage => ({
   },
 
   async getOneElementInnerHTML(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerHTML);
     return text;
   },
 
   async getOneElementInnerText(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerText);
     return text;
   }
@@ -157,5 +178,13 @@ export const cellxgeneActions = puppeteerPage => ({
     await puppeteerUtils(puppeteerPage).clickOn("reset");
     // loading state never actually happens, reset is too fast
     await page.waitFor(200);
+  },
+
+  async clip(min = 0, max = 100) { 
+    await puppeteerUtils(puppeteerPage).clickOn("visualization-settings");
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto("clip-min-input", min);
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto("clip-max-input", max);
+    await puppeteerUtils(puppeteerPage).clickOn("clip-commit");
   }
+
 });

client/__tests__/util/dataframe/dataframe.test.js

@@ -532,6 +532,42 @@ describe("dataframe factories", () => {
       expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
     });
   });
+
+  describe("mapColumns", () => {
+    test("identity", () => {
+      const dfA = Dataframe.Dataframe.create(
+        [3, 3],
+        [
+          new Array(3).fill(0),
+          new Int16Array(3).fill(99),
+          new Float64Array(3).fill(1.1)
+        ]
+      );
+      const dfB = dfA.mapColumns((col, idx) => {
+        expect(dfA.icol(idx).asArray()).toBe(col);
+        return col;
+      });
+      expect(dfA).not.toBe(dfB);
+      expect(dfA.dims).toEqual(dfB.dims);
+      for (let c = 0; c < dfA.dims[1]; c += 1) {
+        expect(dfA.icol(c).asArray()).toBe(dfB.icol(c).asArray());
+      }
+    });
+
+    test("transform", () => {
+      const dfA = Dataframe.Dataframe.create(
+        [3, 3],
+        [new Array(3).fill(0), new Array(3).fill(0), new Array(3).fill(0)]
+      );
+      const dfB = dfA.mapColumns(() => {
+        return new Array(3).fill(1);
+      });
+      expect(dfA).not.toBe(dfB);
+      expect(dfB.iat(0, 0)).toEqual(1);
+      expect(dfB.iat(0, 1)).toEqual(1);
+      expect(dfB.iat(0, 2)).toEqual(1);
+    });
+  });
 });
 
 describe("dataframe col", () => {

client/__tests__/util/quantile.test.js

@@ -0,0 +1,29 @@
+import quantile from "../../src/util/quantile";
+
+describe("quantile", () => {
+	test("single q", () => {
+		const arr = new Float32Array([9, 3, 5, 6, 0]);
+		expect(quantile([1.0], arr)).toMatchObject([9]);
+		expect(quantile([0.9], arr)).toMatchObject([9]);
+		expect(quantile([0.8], arr)).toMatchObject([9]);
+		expect(quantile([0.7], arr)).toMatchObject([6]);
+		expect(quantile([0.6], arr)).toMatchObject([6]);
+		expect(quantile([0.5], arr)).toMatchObject([5]);
+		expect(quantile([0.4], arr)).toMatchObject([5]);
+		expect(quantile([0.3], arr)).toMatchObject([3]);
+		expect(quantile([0.2], arr)).toMatchObject([3]);
+		expect(quantile([0.1], arr)).toMatchObject([0]);
+		expect(quantile([0], arr)).toMatchObject([0]);
+	});
+
+	test("multi q", () => {
+		const arr = new Float32Array([9, 3, 5, 6, 0]);
+		expect(quantile([0, 0.25, 0.5, 0.75, 1.0], arr)).toMatchObject([
+			0,
+			3,
+			5,
+			6,
+			9
+		]);
+	});
+});

client/__tests__/util/range.test.js

@@ -0,0 +1,42 @@
+import { range, rangeFill } from "../../src/util/range";
+
+describe("range", () => {
+	test("no defaults", () => {
+		expect(range(0, 3, 1)).toMatchObject([0, 1, 2]);
+	});
+
+	test("range(stop)", () => {
+		expect(range(3)).toMatchObject([0, 1, 2]);
+		expect(range(0)).toMatchObject([]);
+		expect(range(1)).toMatchObject([0]);
+	});
+
+	test("range(start,stop)", () => {
+		expect(range(0, 0)).toMatchObject([]);
+		expect(range(0, 2)).toMatchObject([0, 1]);
+		expect(range(4, 8)).toMatchObject([4, 5, 6, 7]);
+	});
+
+	test("range(start, stop, step", () => {
+		expect(range(4, 0, -1)).toMatchObject([4, 3, 2, 1]);
+		expect(range(0, 4, 2)).toMatchObject([0, 2]);
+	});
+});
+
+describe("rangefill", () => {
+	test("rangeFill(arr)", () => {
+		expect(rangeFill(new Int32Array(3))).toMatchObject(
+			new Int32Array([0, 1, 2])
+		);
+	});
+	test("rangeFill(arr, start)", () => {
+		expect(rangeFill(new Int32Array(2), 1)).toMatchObject(
+			new Int32Array([1, 2])
+		);
+	});
+	test("rangeFill(arr, start, step)", () => {
+		expect(rangeFill(new Int32Array(3), 2, -1)).toMatchObject(
+			new Int32Array([2, 1, 0])
+		);
+	});
+});

client/__tests__/util/stateManager/sampleResponses.js

@@ -34,28 +34,38 @@ const aSchemaResponse = {
       type: "float32"
     },
     annotations: {
-      obs: [
-        { name: "name", type: "string" },
-        { name: "field1", type: "int32" },
-        { name: "field2", type: "float32" },
-        { name: "field3", type: "boolean" },
-        {
-          name: "field4",
-          type: "categorical",
-          categories: field4Categories
-        }
-      ],
-      var: [
-        { name: "name", type: "string" },
-        { name: "fieldA", type: "int32" },
-        { name: "fieldB", type: "float32" },
-        { name: "fieldC", type: "boolean" },
-        {
-          name: "fieldD",
-          type: "categorical",
-          categories: fieldDCategories
-        }
-      ]
+      obs: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "field1", type: "int32" },
+          { name: "field2", type: "float32" },
+          { name: "field3", type: "boolean" },
+          {
+            name: "field4",
+            type: "categorical",
+            categories: field4Categories
+          }
+        ]
+      },
+      var: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "fieldA", type: "int32" },
+          { name: "fieldB", type: "float32" },
+          { name: "fieldC", type: "boolean" },
+          {
+            name: "fieldD",
+            type: "categorical",
+            categories: fieldDCategories
+          }
+        ]
+      }
+    },
+    layout: {
+      obs: [{ name: "umap", type: "float32", dims: ["umap_0", "umap_1"] }],
+      var: []
     }
   }
 };
@@ -162,29 +172,7 @@ const aLayoutFBSResponse = (() => {
     new Float32Array(nObs).fill(Math.random()),
     new Float32Array(nObs).fill(Math.random())
   ];
-  const builder = new flatbuffers.Builder(1024);
-
-  const cols = _.map(coords, carr => {
-    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
-    NetEncoding.Float32Array.startFloat32Array(builder);
-    NetEncoding.Float32Array.addData(builder, cdv);
-    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
-
-    NetEncoding.Column.startColumn(builder);
-    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
-    NetEncoding.Column.addU(builder, floatArr);
-    return NetEncoding.Column.endColumn(builder);
-  });
-
-  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
-
-  NetEncoding.Matrix.startMatrix(builder);
-  NetEncoding.Matrix.addNRows(builder, nObs);
-  NetEncoding.Matrix.addNCols(builder, coords.length);
-  NetEncoding.Matrix.addColumns(builder, columns);
-  const matrix = NetEncoding.Matrix.endMatrix(builder);
-  builder.finish(matrix);
-  return builder.asUint8Array();
+  return encodeMatrix(coords, ["umap_0", "umap_1"]);
 })();
 
 const aDataObsResponse = {

client/__tests__/util/stateManager/universe.test.js

@@ -53,13 +53,15 @@ describe("createUniverseFromResponse", () => {
 
     expect(universe.obsAnnotations.dims).toEqual([
       nObs,
-      REST.schema.schema.annotations.obs.length
+      REST.schema.schema.annotations.obs.columns.length
     ]);
     expect(universe.obsLayout.dims).toEqual([nObs, 2]);
-    expect(universe.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(universe.obsLayout.colIndex.keys()).toEqual(
+      universe.schema.layout.obs[0].dims
+    );
     expect(universe.varAnnotations.dims).toEqual([
       nVar,
-      REST.schema.schema.annotations.var.length
+      REST.schema.schema.annotations.var.columns.length
     ]);
     expect(universe.varData.isEmpty()).toBeTruthy();
   });

client/__tests__/util/stateManager/world.test.js

@@ -29,7 +29,8 @@ const defaultBigBang = () => {
   /* create crossfilter */
   const crossfilter = World.createObsDimensions(
     new Crossfilter(world.obsAnnotations),
-    world
+    world,
+    REST.schema.schema.layout.obs[0].dims
   );
 
   return {
@@ -58,10 +59,15 @@ describe("createWorldFromEntireUniverse", () => {
         nObs: universe.nObs,
         nVar: universe.nVar,
         schema: universe.schema,
-        obsAnnotations: universe.obsAnnotations,
-        varAnnotations: universe.varAnnotations,
-        obsLayout: universe.obsLayout,
-        varData: expect.any(Dataframe.Dataframe)
+        obsAnnotations: expect.any(Dataframe.Dataframe),
+        varAnnotations: expect.any(Dataframe.Dataframe),
+        obsLayout: expect.any(Dataframe.Dataframe),
+        varData: expect.any(Dataframe.Dataframe),
+        clipQuantiles: { min: 0, max: 1 },
+        unclipped: {
+          obsAnnotations: expect.any(Dataframe.Dataframe),
+          varData: expect.any(Dataframe.Dataframe)
+        }
       })
     );
   });
@@ -76,7 +82,7 @@ describe("createWorldFromCurrentSelection", () => {
     } = defaultBigBang();
 
     /* mock a selection */
-    let crossfilter = originalCrossfilter
+    const crossfilter = originalCrossfilter
       .select(obsAnnoDimensionName("field1"), { mode: "range", lo: 0, hi: 5 })
       .select(obsAnnoDimensionName("field3"), {
         mode: "exact",
@@ -84,7 +90,7 @@ describe("createWorldFromCurrentSelection", () => {
       });
 
     /* create the world from the selection */
-    const world = World.createWorldFromCurrentSelection(
+    const world = World.createWorldBySelection(
       universe,
       originalWorld,
       crossfilter
@@ -112,10 +118,15 @@ describe("createWorldFromCurrentSelection", () => {
         nObs: matchingIndices.length,
         nVar: universe.nVar,
         schema: universe.schema,
+        clipQuantiles: { min: 0, max: 1 },
         obsAnnotations: expect.any(Dataframe.Dataframe),
-        varAnnotations: universe.varAnnotations,
+        varAnnotations: expect.any(Dataframe.Dataframe),
         obsLayout: expect.any(Dataframe.Dataframe),
-        varData: expect.any(Dataframe.Dataframe)
+        varData: expect.any(Dataframe.Dataframe),
+        unclipped: {
+          obsAnnotations: expect.any(Dataframe.Dataframe),
+          varData: expect.any(Dataframe.Dataframe)
+        }
       })
     );
 
@@ -128,7 +139,9 @@ describe("createWorldFromCurrentSelection", () => {
     expect(world.obsLayout.rowIndex.keys()).toEqual(
       new Int32Array(matchingIndices)
     );
-    expect(world.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(world.obsLayout.colIndex.keys()).toEqual(
+      world.schema.layout.obs[0].dims
+    );
   });
 });
 
@@ -142,14 +155,18 @@ describe("createObsDimensionMap", () => {
 
     const { crossfilter } = defaultBigBang();
     const annotationNames = _.map(
-      REST.schema.schema.annotations.obs,
+      REST.schema.schema.annotations.obs.columns,
       c => c.name
     );
-    const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
+    const obsIndexColName = REST.schema.schema.annotations.obs.index;
+    const schemaByObsName = _.keyBy(
+      REST.schema.schema.annotations.obs.columns,
+      "name"
+    );
     expect(crossfilter).toBeDefined();
     annotationNames.forEach(name => {
       const dim = crossfilter.dimensions[obsAnnoDimensionName(name)];
-      if (name === "name") {
+      if (name === obsIndexColName) {
         expect(dim).toBeUndefined();
       } else {
         const { type } = schemaByObsName[name];

client/__tests__/util/typedCrossfilter/crossfilter.test.js

@@ -11,7 +11,8 @@ const someData = [
     tip: 100,
     type: "tab",
     productIDs: ["001"],
-    coords: [0, 0]
+    coords: [0, 0],
+    nonFinite: 0.0
   },
   {
     date: "2011-11-14T16:20:19Z",
@@ -20,7 +21,8 @@ const someData = [
     tip: 100,
     type: "tab",
     productIDs: ["001", "005"],
-    coords: [0.4, 0.4]
+    coords: [0.4, 0.4],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T16:28:54Z",
@@ -29,7 +31,8 @@ const someData = [
     tip: 200,
     type: "visa",
     productIDs: ["004", "005"],
-    coords: [0.3, 0.1]
+    coords: [0.3, 0.1],
+    nonFinite: Number.POSITIVE_INFINITY
   },
   {
     date: "2011-11-14T16:30:43Z",
@@ -38,7 +41,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001", "002"],
-    coords: [0.392, 0.1]
+    coords: [0.392, 0.1],
+    nonFinite: Number.NEGATIVE_INFINITY
   },
   {
     date: "2011-11-14T16:48:46Z",
@@ -47,7 +51,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["005"],
-    coords: [0.7, 0.0482]
+    coords: [0.7, 0.0482],
+    nonFinite: 1.0
   },
   {
     date: "2011-11-14T16:53:41Z",
@@ -56,7 +61,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001", "004", "005"],
-    coords: [0.9999, 1.0]
+    coords: [0.9999, 1.0],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T16:54:06Z",
@@ -65,7 +71,8 @@ const someData = [
     tip: 0,
     type: "cash",
     productIDs: ["001", "002", "003", "004", "005"],
-    coords: [0.384, 0.6938]
+    coords: [0.384, 0.6938],
+    nonFinite: 99.0
   },
   {
     date: "2011-11-14T16:58:03Z",
@@ -74,7 +81,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001"],
-    coords: [0.4822, 0.482]
+    coords: [0.4822, 0.482],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T17:07:21Z",
@@ -83,7 +91,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["004", "005"],
-    coords: [0.2234, 0]
+    coords: [0.2234, 0],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T17:22:59Z",
@@ -92,7 +101,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001", "002", "004", "005"],
-    coords: [0.382, 0.38485]
+    coords: [0.382, 0.38485],
+    nonFinite: -1
   },
   {
     date: "2011-11-14T17:25:45Z",
@@ -101,7 +111,8 @@ const someData = [
     tip: 0,
     type: "cash",
     productIDs: ["002"],
-    coords: [0.998, 0.8472]
+    coords: [0.998, 0.8472],
+    nonFinite: 0.0
   },
   {
     date: "2011-11-14T17:29:52Z",
@@ -110,7 +121,8 @@ const someData = [
     tip: 100,
     type: "visa",
     productIDs: ["004"],
-    coords: [0.8273, 0.3384]
+    coords: [0.8273, 0.3384],
+    nonFinite: 0.0
   }
 ];
 
@@ -304,15 +316,15 @@ describe("ImmutableTypedCrossfilter", () => {
     });
     test.each([[0, 0, 1, 1], [0, 0, 0.5, 0.5], [0.5, 0.5, 1, 1]])(
       "within-rect %d %d %d %d",
-      (x0, y0, x1, y1) => {
+      (minX, minY, maxX, maxY) => {
         expect(
           p
-            .select("coords", { mode: "within-rect", x0, y0, x1, y1 })
+            .select("coords", { mode: "within-rect", minX, minY, maxX, maxY })
             .allSelected()
         ).toEqual(
           _.filter(someData, d => {
             const [x, y] = d.coords;
-            return x0 <= x && x < x1 && y0 <= y && y < y1;
+            return minX <= x && x < maxX && minY <= y && y < maxY;
           })
         );
       }
@@ -327,4 +339,96 @@ describe("ImmutableTypedCrossfilter", () => {
       ).toEqual(_.filter(someData, d => polygonContains(polygon, d.coords)));
     });
   });
+
+  describe("non-finite scalars", () => {
+    let p;
+    beforeEach(() => {
+      p = payments
+        .addDimension("quantity", "scalar", (i, d) => d[i].quantity, Int32Array)
+        .addDimension(
+          "nonFinite",
+          "scalar",
+          (i, d) => d[i].nonFinite,
+          Float32Array
+        )
+        .select("quantity", { mode: "all" });
+    });
+
+    test("all or none", () => {
+      expect(p.select("nonFinite", { mode: "all" }).countSelected()).toEqual(
+        someData.length
+      );
+      expect(p.select("nonFinite", { mode: "none" }).countSelected()).toEqual(
+        0
+      );
+    });
+
+    test("exact", () => {
+      expect(
+        p.select("nonFinite", { mode: "exact", values: [0] }).countSelected()
+      ).toEqual(3);
+      expect(
+        p.select("nonFinite", { mode: "exact", values: [1] }).countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.POSITIVE_INFINITY]
+          })
+          .countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.NEGATIVE_INFINITY]
+          })
+          .countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", { mode: "exact", values: [Number.NaN] })
+          .countSelected()
+      ).toEqual(4);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.POSITIVE_INFINITY, 0, 1, 99]
+          })
+          .countSelected()
+      ).toEqual(6);
+    });
+
+    test("range", () => {
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: 0,
+            hi: Number.POSITIVE_INFINITY
+          })
+          .countSelected()
+      ).toEqual(5);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: 0,
+            hi: Number.NaN
+          })
+          .countSelected()
+      ).toEqual(6);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: Number.NEGATIVE_INFINITY,
+            hi: Number.POSITIVE_INFINITY
+          })
+          .countSelected()
+      ).toEqual(7);
+    });
+  });
 });

client/__tests__/util/typedCrossfilter/sort.test.js

@@ -1,4 +1,22 @@
-import { sort, sortIndex } from "../../../src/util/typedCrossfilter/sort";
+import {
+  sortArray,
+  sortIndex,
+  lowerBound,
+  upperBound,
+  lowerBoundIndirect,
+  upperBoundIndirect
+} from "../../../src/util/typedCrossfilter/sort";
+
+/*
+Sort tests should keep in mind that there are separate code
+paths for:
+  - small vs. large arrays (insertionsort only)
+  - float-only typed arrays vs. other array types (non-finite handling)
+  - indexed vs. direct sort
+*/
+
+const pInf = Number.POSITIVE_INFINITY;
+const nInf = Number.NEGATIVE_INFINITY;
 
 function fillRange(arr, start = 0) {
   const larr = arr;
@@ -15,42 +33,223 @@ function fillRand(arr) {
   return arr;
 }
 
-describe("sort", () => {
-  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
-    test(Type.name, () => {
-      expect(sort(Type.from([6, 5, 4, 3, 2, 1, 0]))).toMatchObject(
-        Type.from([0, 1, 2, 3, 4, 5, 6])
-      );
-      expect(sort(Type.from([6, 5, 4, 3, 2, 1]))).toMatchObject(
-        Type.from([1, 2, 3, 4, 5, 6])
-      );
+describe("sortArray", () => {
+  describe("JS vals", () => {
+    [
+      [true, false],
+      ["a", "b", "0", "1"],
+      [0, "a", true, null, undefined, 3.1415],
+      fillRand(new Array(1000)),
+      ["a", NaN, null, pInf]
+    ].map((val, idx) =>
+      test(`JS vals ${idx}`, () => {
+        expect(sortArray(val)).toMatchObject(val.sort());
+      })
+    );
+  });
 
-      const source = fillRand(new Type(1000));
-      expect(sort(Type.from(source))).toMatchObject(Type.from(source).sort());
-    })
-  );
+  describe("finite numbers", () => {
+    [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+      test(Type.name, () => {
+        expect(sortArray(Type.from([6, 5, 4, 3, 2, 1, 0]))).toMatchObject(
+          Type.from([0, 1, 2, 3, 4, 5, 6])
+        );
+
+        expect(sortArray(Type.from([6, 5, 4, 3, 2, 1]))).toMatchObject(
+          Type.from([1, 2, 3, 4, 5, 6])
+        );
+
+        const source = fillRand(new Type(1000));
+        expect(sortArray(Type.from(source))).toMatchObject(
+          Type.from(source).sort()
+        );
+      })
+    );
+  });
+
+  describe("non-finite numbers", () => {
+    test("inifinity", () => {
+      expect(sortArray(new Float32Array([pInf, nInf, 0, 1, 2]))).toMatchObject(
+        new Float32Array([nInf, 0, 1, 2, pInf])
+      );
+      expect(
+        sortArray(new Float32Array([pInf, nInf, pInf, nInf]))
+      ).toMatchObject(new Float32Array([nInf, nInf, pInf, pInf]));
+      expect(
+        sortArray(new Float32Array([pInf, nInf, pInf, nInf, pInf]))
+      ).toMatchObject(new Float32Array([nInf, nInf, pInf, pInf, pInf]));
+      expect(
+        sortArray(
+          new Float32Array(100).fill(Infinity, 0, 50).fill(-Infinity, 50, 100)
+        )
+      ).toMatchObject(
+        new Float32Array(100).fill(-Infinity, 0, 50).fill(Infinity, 50, 100)
+      );
+    });
+    test("NaN", () => {
+      expect(sortArray(new Float64Array([NaN, 2, 1, 0]))).toMatchObject(
+        new Float64Array([0, 1, 2, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, 1, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, NaN, 1, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, 1, NaN, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN, NaN])
+      );
+      expect(
+        sortArray(fillRange(new Float32Array(100)).fill(NaN, 0, 10))
+      ).toMatchObject(fillRange(new Float32Array(100), 10).fill(NaN, 90, 100));
+    });
+
+    test("mixed numbers", () => {
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, NaN, NaN]))
+      ).toMatchObject(new Float32Array([nInf, pInf, NaN, NaN, NaN]));
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, NaN, 1, NaN, 2]))
+      ).toMatchObject(new Float32Array([nInf, 1, 2, pInf, NaN, NaN, NaN]));
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, 0, 1, NaN, 2]))
+      ).toMatchObject(new Float32Array([nInf, 0, 1, 2, pInf, NaN, NaN]));
+      expect(
+        sortArray(
+          fillRange(new Float32Array(100))
+            .fill(NaN, 0, 10)
+            .fill(Infinity, 10, 20)
+        )
+      ).toMatchObject(
+        fillRange(new Float32Array(100), 20)
+          .fill(Infinity, 80, 90)
+          .fill(NaN, 90, 100)
+      );
+    });
+  });
 });
 
 describe("sortIndex", () => {
-  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
-    test(Type.name, () => {
-      const source1 = Type.from([6, 5, 4, 3, 2, 1, 0]);
+  describe("finite numbers", () => {
+    [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+      test(Type.name, () => {
+        const source1 = Type.from([6, 5, 4, 3, 2, 1, 0]);
+        const index1 = fillRange(new Uint32Array(source1.length));
+        expect(sortIndex(index1, source1)).toMatchObject(
+          index1.sort((a, b) => source1[a] - source1[b])
+        );
+
+        const source2 = Type.from([6, 5, 4, 3, 2, 1]);
+        const index2 = fillRange(new Uint32Array(source2.length));
+        expect(sortIndex(index2, source2)).toMatchObject(
+          index2.sort((a, b) => source1[a] - source1[b])
+        );
+
+        const source3 = fillRand(new Type(1000));
+        const index3 = fillRange(new Uint32Array(source3.length));
+        expect(sortIndex(index3, source3)).toMatchObject(
+          index3.sort((a, b) => source1[a] - source1[b])
+        );
+      })
+    );
+  });
+
+  describe("non-finite numbers", () => {
+    test("mixed numbers", () => {
+      const source1 = new Float32Array([NaN, pInf, nInf, NaN, 1, NaN, 2]);
       const index1 = fillRange(new Uint32Array(source1.length));
       expect(sortIndex(index1, source1)).toMatchObject(
-        index1.sort((a, b) => source1[a] - source1[b])
+        new Uint32Array([2, 4, 6, 1, 0, 3, 5])
       );
 
-      const source2 = Type.from([6, 5, 4, 3, 2, 1]);
+      const source2 = new Float32Array([NaN, pInf, nInf, 0, 1, NaN, 2]);
       const index2 = fillRange(new Uint32Array(source2.length));
       expect(sortIndex(index2, source2)).toMatchObject(
-        index2.sort((a, b) => source1[a] - source1[b])
+        new Uint32Array([2, 3, 4, 6, 1, 0, 5])
       );
+    });
+  });
+});
 
-      const source3 = fillRand(new Type(1000));
-      const index3 = fillRange(new Uint32Array(source3.length));
-      expect(sortIndex(index3, source3)).toMatchObject(
-        index3.sort((a, b) => source1[a] - source1[b])
-      );
-    })
-  );
+describe("lowerBound", () => {
+  test("non-float path", () => {
+    expect(lowerBound([], 0, 0, 0)).toEqual(0);
+
+    expect(lowerBound([0, 1, 2, 3], -1, 0, 4)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3], 0, 0, 4)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3], 1, 0, 4)).toEqual(1);
+    expect(lowerBound([0, 1, 2, 3], 3, 0, 4)).toEqual(3);
+    expect(lowerBound([0, 1, 2, 3], 4, 0, 4)).toEqual(4);
+
+    expect(lowerBound([0, 1, 2, 3, 4], -1, 0, 5)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3, 4], 0, 0, 5)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3, 4], 2, 0, 5)).toEqual(2);
+    expect(lowerBound([0, 1, 2, 3, 4], 4, 0, 5)).toEqual(4);
+    expect(lowerBound([0, 1, 2, 3, 4], 5, 0, 5)).toEqual(5);
+
+    expect(lowerBound([0, 2, 4, 6, 8], 5, 0, 5)).toEqual(3);
+    expect(lowerBound([0, 2, 2, 2, 8], 5, 0, 5)).toEqual(4);
+
+    expect(lowerBound([0, 1, 2, 3, 4, 5, 6, 7, 8], 3, 2, 4)).toEqual(3);
+    expect(lowerBound([0, 1, 2, 3, 4, 5, 6, 7, 8], 99, 2, 4)).toEqual(4);
+  });
+
+  test("float path, finites", () => {
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
+
+    expect(lowerBound(new Float32Array([]), 0, 0, 0)).toEqual(0);
+
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), -1, 0, 4)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 0, 0, 4)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 3, 0, 4)).toEqual(3);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 4, 0, 4)).toEqual(4);
+
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), -1, 0, 5)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 0, 0, 5)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 2, 0, 5)).toEqual(2);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 4, 0, 5)).toEqual(4);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 5, 0, 5)).toEqual(5);
+
+    expect(lowerBound(new Float32Array([0, 2, 4, 6, 8]), 5, 0, 5)).toEqual(3);
+    expect(lowerBound(new Float32Array([0, 2, 2, 2, 8]), 5, 0, 5)).toEqual(4);
+
+    expect(
+      lowerBound(new Float32Array([0, 1, 2, 3, 4, 5, 6, 7, 8]), 3, 2, 4)
+    ).toEqual(3);
+    expect(
+      lowerBound(new Float32Array([0, 1, 2, 3, 4, 5, 6, 7, 8]), 99, 2, 4)
+    ).toEqual(4);
+  });
+
+  test("float path, non-finite", () => {
+    expect(
+      lowerBound(
+        new Float32Array([-Infinity, 0, 1, Infinity, NaN]),
+        -Infinity,
+        0,
+        5
+      )
+    ).toEqual(0);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 0, 0, 5)
+    ).toEqual(1);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 1, 0, 5)
+    ).toEqual(2);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 2, 0, 5)
+    ).toEqual(3);
+    expect(
+      lowerBound(
+        new Float32Array([-Infinity, 0, 1, Infinity, NaN]),
+        Infinity,
+        0,
+        5
+      )
+    ).toEqual(3);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), NaN, 0, 5)
+    ).toEqual(4);
+  });
 });

client/__tests__/util/typedCrossfilter/util.test.js

@@ -1,11 +1,8 @@
 import {
-  fillRange,
   sliceByIndex,
-  makeSortIndex,
-  lowerBound,
-  lowerBoundIndirect,
-  upperBoundIndirect
+  makeSortIndex
 } from "../../../src/util/typedCrossfilter/util";
+import { rangeFill as fillRange } from "../../../src/util/range";
 
 describe("fillRange", () => {
   test("Array", () => {

client/configuration/babel/babel.dev.js

@@ -7,8 +7,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-proposal-optional-chaining",
     "@babel/plugin-proposal-nullish-coalescing-operator"

client/configuration/babel/babel.prod.js

@@ -6,8 +6,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-transform-react-constant-elements",
     "@babel/plugin-transform-runtime",

client/index_template.html

@@ -1,35 +1,42 @@
 <!DOCTYPE html>
 <html lang="en">
-  <head>
+<head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <title>cellxgene</title>
     <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
-      html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
-        font-family: 'Roboto Condensed','Helvetica Neue','Helvetica','Arial',sans-serif;
-        font-size: 14px;
-      }
-      body {
-        margin: 0;
-        padding: 0;
-      }
+        html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
+            font-family: 'Roboto Condensed', 'Helvetica Neue', 'Helvetica', 'Arial', sans-serif;
+            font-size: 14px;
+        }
 
-      * {
-        box-sizing: border-box;
-      }
+        body {
+            margin: 0;
+            padding: 0;
+        }
+
+        * {
+            box-sizing: border-box;
+        }
     </style>
-  </head>
-  <body>
-<script type="text/javascript">
-window.CELLXGENE = {};
-window.CELLXGENE.API = {
-  prefix: window.location.href + "api/",
-  version: "v0.2/"
-};
-</script>
-    <noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>, please <strong>enable JS</strong> to make this app work.</noscript>
 
-    <div id="root"></div>
-  </body>
+</head>
+<body>
+<script type="text/javascript">
+  window.CELLXGENE = {};
+  window.CELLXGENE.API = {
+    prefix: window.location.href + "api/",
+    version: "v0.2/"
+  };
+</script>
+<noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>,
+    please <strong>enable JS</strong> to make this app work.
+</noscript>
+
+<div id="root"></div>
+{% for script in SCRIPTS %}
+<script type="text/javascript" src="{{script | safe}}"></script>
+{% endfor %}
+</body>
 </html>

client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cellxgene",
-  "version": "0.8.0",
+  "version": "0.10.1",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
@@ -9,13 +9,13 @@
     "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
     "clean": "rimraf build",
     "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
-    "e2e": "jest --verbose false --config __tests__/e2e/e2eJestConfig.json e2e",
+    "e2e": "node node_modules/jest/bin/jest.js --verbose false --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
     "lint": "eslint src",
     "smoke-test": "start-server-and-test start-server-for-test :5000 e2e",
     "start": "node server/development.js",
     "start-server-for-test": "cellxgene launch -p 5000 ../example-dataset/pbmc3k.h5ad",
-    "test": "jest",
-    "unit-test": "jest --testPathIgnorePatterns e2e"
+    "test": "node node_modules/jest/bin/jest.js",
+    "unit-test": "node node_modules/jest/bin/jest.js --testPathIgnorePatterns e2e"
   },
   "engineStrict": true,
   "engines": {
@@ -31,9 +31,9 @@
     "eslint-scope": "3.7.1"
   },
   "dependencies": {
-    "@blueprintjs/core": "^3.8.0",
-    "@blueprintjs/icons": "^3.3.0",
-    "@blueprintjs/select": "^3.2.1",
+    "@blueprintjs/core": "^3.15.1",
+    "@blueprintjs/icons": "^3.8.0",
+    "@blueprintjs/select": "^3.8.0",
     "canvas-fit": "^1.5.0",
     "d3": "^4.10.0",
     "d3-scale-chromatic": "^1.3.0",
@@ -41,79 +41,77 @@
     "font-color-contrast": "^1.0.3",
     "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
-    "gl-matrix": "^2.7.1",
+    "gl-vec3": "^1.1.3",
+    "gl-matrix": "^3.0.0",
     "is-number": "^7.0.0",
     "key-pressed": "0.0.1",
     "lodash": "^4.17.4",
-    "memoize-one": "^4.0.0",
+    "memoize-one": "^5.0.4",
     "mouse-position": "^2.0.1",
     "mouse-pressed": "^1.0.0",
-    "normalize.css": "^8.0.0",
     "orbit-camera": "^1.0.0",
-    "query-string": "^6.1.0",
-    "react": "^16.6.0",
+    "query-string": "^6.5.0",
+    "react": "^16.8.6",
     "react-autocomplete": "^1.7.2",
-    "react-dom": "^16.6.0",
-    "react-helmet": "^5.2.0",
-    "react-icons": "^3.2.2",
-    "react-redux": "^5.1.0",
+    "react-dom": "^16.8.6",
+    "react-helmet": "^5.2.1",
+    "react-icons": "^3.7.0",
+    "react-redux": "^7.0.3",
     "redux": "^4.0.1",
-    "redux-devtools-extension": "^2.13.5",
     "redux-thunk": "^2.2.0",
-    "regl": "^1.3.9",
+    "regl": "^1.3.11",
     "scroll-speed": "^1.0.0",
     "urijs": "^1.19.0"
   },
   "devDependencies": {
-    "@babel/core": "^7.1.5",
-    "@babel/plugin-proposal-class-properties": "^7.0.0",
-    "@babel/plugin-proposal-decorators": "^7.0.0",
-    "@babel/plugin-proposal-export-namespace-from": "^7.0.0",
-    "@babel/plugin-proposal-function-bind": "^7.0.0",
-    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.2.0",
+    "@babel/core": "^7.4.4",
+    "@babel/plugin-proposal-class-properties": "^7.4.4",
+    "@babel/plugin-proposal-decorators": "^7.4.4",
+    "@babel/plugin-proposal-export-namespace-from": "^7.2.0",
+    "@babel/plugin-proposal-function-bind": "^7.2.0",
+    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.4.4",
     "@babel/plugin-proposal-optional-chaining": "^7.2.0",
-    "@babel/plugin-transform-react-constant-elements": "^7.0.0",
-    "@babel/plugin-transform-runtime": "^7.1.0",
-    "@babel/preset-env": "^7.1.5",
+    "@babel/plugin-transform-react-constant-elements": "^7.2.0",
+    "@babel/plugin-transform-runtime": "^7.4.4",
+    "@babel/preset-env": "^7.4.4",
     "@babel/preset-react": "^7.0.0",
-    "@babel/register": "^7.0.0",
-    "@babel/runtime": "^7.1.5",
-    "babel-core": "^7.0.0-bridge.0",
+    "@babel/register": "^7.4.4",
+    "@babel/runtime": "^7.4.4",
     "babel-eslint": "^10.0.1",
-    "babel-jest": "^23.6.0",
-    "babel-loader": "^8.0.0",
-    "babel-preset-modern-browsers": "^12.0.0",
+    "babel-jest": "^24.8.0",
+    "babel-loader": "^8.0.6",
+    "babel-preset-modern-browsers": "^14.0.0",
     "chalk": "^2.4.2",
     "connect-history-api-fallback": "^1.6.0",
-    "copy-webpack-plugin": "^4.6.0",
-    "css-loader": "^1.0.1",
-    "eslint": "^5.13.0",
+    "copy-webpack-plugin": "^5.0.3",
+    "css-loader": "^2.1.1",
+    "eslint": "^5.16.0",
     "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^4.0.0",
+    "eslint-config-prettier": "^4.2.0",
     "eslint-loader": "^2.1.2",
     "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.16.0",
-    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-import": "^2.17.2",
+    "eslint-plugin-jest": "^22.5.1",
     "eslint-plugin-jsx-a11y": "^6.2.1",
-    "eslint-plugin-react": "^7.12.4",
+    "eslint-plugin-react": "^7.13.0",
     "express": "^4.14.0",
-    "file-loader": "^2.0.0",
+    "file-loader": "^3.0.1",
     "html-webpack-inline-source-plugin": "0.0.10",
     "html-webpack-plugin": "^3.2.0",
-    "jest": "^24.1.0",
-    "jest-puppeteer": "^4.1.0",
+    "jest": "^24.8.0",
+    "jest-puppeteer": "^4.1.1",
     "json-loader": "^0.5.4",
-    "mini-css-extract-plugin": "^0.4.1",
-    "puppeteer": "^1.12.1",
+    "mini-css-extract-plugin": "^0.6.0",
+    "puppeteer": "^1.16.0",
     "rimraf": "^2.6.3",
     "serve-favicon": "^2.3.0",
-    "start-server-and-test": "^1.7.11",
+    "start-server-and-test": "^1.9.0",
     "style-loader": "^0.23.1",
     "sw-precache-webpack-plugin": "^0.11.5",
     "url-loader": "^1.1.0",
-    "webpack": "^4.25.1",
-    "webpack-cli": "^3.1.0",
-    "webpack-dev-middleware": "^3.1.3"
+    "webpack": "^4.31.0",
+    "webpack-cli": "^3.3.2",
+    "webpack-dev-middleware": "^3.6.2"
   },
   "jest": {
     "testMatch": [
@@ -133,16 +131,23 @@
         ],
         "plugins": [
           "@babel/plugin-proposal-function-bind",
-          "@babel/plugin-proposal-class-properties",
           [
             "@babel/plugin-proposal-decorators",
             {
               "legacy": true
             }
           ],
+          [
+            "@babel/plugin-proposal-class-properties",
+            {
+              "loose": true
+            }
+          ],
           "@babel/plugin-proposal-export-namespace-from",
           "@babel/plugin-transform-react-constant-elements",
-          "@babel/plugin-transform-runtime"
+          "@babel/plugin-transform-runtime",
+          "@babel/plugin-proposal-optional-chaining",
+          "@babel/plugin-proposal-nullish-coalescing-operator"
         ]
       }
     }

client/src/actions/index.js

@@ -21,24 +21,32 @@ const doInitialDataLoad = () =>
     dispatch({ type: "initial data load start" });
 
     try {
-      const requestJson = _(["config", "schema"])
+      /*
+      Step 1 - config & schema, all JSON
+      */
+      const requestJson = ["config", "schema"]
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doJsonRequest(url))
-        .value();
-      const requestBinary = _([
-        "annotations/obs",
-        "annotations/var?annotation-name=name",
-        "layout/obs"
-      ])
-        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doBinaryRequest(url))
-        .value();
-
-      const results = await Promise.all(_.concat(requestJson, requestBinary));
-
+        .map(url => doJsonRequest(url));
+      const stepOneResults = await Promise.all(requestJson);
       /* set config defaults */
-      const config = { ...globals.configDefaults, ...results[0].config };
-      const [, schema, obsAnno, varAnno, obsLayout] = [...results];
+      const config = { ...globals.configDefaults, ...stepOneResults[0].config };
+      const schema = stepOneResults[1];
+
+      /*
+      Step 2 - dataframes, all binary.  NOTE: uses results of step 1.
+      */
+      /* only load names for var annotations, if possible*/
+      const varIndexName = schema?.schema?.annotations?.var?.index;
+      const varAnnotationsQuery = varIndexName
+        ? `?annotation-name=${varIndexName}`
+        : "";
+      const varAnnotationsURL = `annotations/var${varAnnotationsQuery}`;
+      const requestBinary = ["annotations/obs", varAnnotationsURL, "layout/obs"]
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doBinaryRequest(url));
+      const stepTwoResults = await Promise.all(requestBinary);
+      const [obsAnno, varAnno, obsLayout] = [...stepTwoResults];
+
       const universe = Universe.createUniverseFromResponse(
         config,
         schema,
@@ -91,6 +99,10 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  const state = getState();
+  const { universe } = state;
+  const varIndexName = universe.schema.annotations.var.index;
+
   /* helper for this function only */
   const fetchData = async geneNames => {
     const res = await fetch(
@@ -100,7 +112,7 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
         body: JSON.stringify({
           filter: {
             var: {
-              annotation_value: [{ name: "name", values: geneNames }]
+              annotation_value: [{ name: varIndexName, values: geneNames }]
             }
           }
         }),
@@ -123,8 +135,6 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
     return Universe.convertDataFBStoObject(universe, data);
   };
 
-  const state = getState();
-  const { universe } = state;
   /* preload data already in cache */
   let expressionData = _.transform(
     genes,
@@ -241,6 +251,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     */
     const state = getState();
     const { universe } = state;
+    const varIndexName = universe.schema.annotations.var.index;
 
     // Legal values are null, Array or TypedArray.  Null is initial state.
     if (!set1) set1 = [];
@@ -277,7 +288,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     const data = await res.json();
     // result is [ [varIdx, ...], ... ]
     const topNGenes = _.map(data, r =>
-      universe.varAnnotations.at(r[0], "name")
+      universe.varAnnotations.at(r[0], varIndexName)
     );
 
     /*
@@ -302,6 +313,9 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
 const resetInterface = () => (dispatch, getState) => {
   const { universe } = getState();
 
+  dispatch({
+    type: "user reset start"
+  });
   dispatch({
     type: "clear all user defined genes"
   });
@@ -321,6 +335,9 @@ const resetInterface = () => (dispatch, getState) => {
   dispatch({
     type: "increment graph render counter"
   });
+  dispatch({
+    type: "user reset end"
+  });
 };
 
 export default {

client/src/components/brushableHistogram/index.js

@@ -5,14 +5,12 @@ https://bl.ocks.org/SpaceActuary/2f004899ea1b2bd78d6f1dbb2febf771
 */
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { Button, ButtonGroup, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
 import * as d3 from "d3";
 import memoize from "memoize-one";
 import * as globals from "../../globals";
 import actions from "../../actions";
-import finiteExtent from "../../util/finiteExtent";
 import { makeContinuousDimensionName } from "../../util/nameCreators";
 
 @connect(state => ({
@@ -21,53 +19,51 @@ import { makeContinuousDimensionName } from "../../util/nameCreators";
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
   continuousSelection: state.continuousSelection,
   differential: state.differential,
-  colorAccessor: state.colors.colorAccessor,
-  obsAnnotations: _.get(state.world, "obsAnnotations", null)
+  colorAccessor: state.colors.colorAccessor
 }))
 class HistogramBrush extends React.Component {
-  calcHistogramCache = memoize((obsAnnotations, field, rangeMin, rangeMax) => {
-    const { world } = this.props;
-    const histogramCache = {};
+  static getColumn(world, field, clipped = true) {
+    /*
+    Return the underlying Dataframe column for our field.   By default, 
+    returns the clipped column.   If clipped===false, will return the 
+    unclipped column.
+    */
+    const obsAnnotations = clipped
+      ? world.obsAnnotations
+      : world.unclipped.obsAnnotations;
+    const varData = clipped ? world.varData : world.unclipped.varData;
+    if (obsAnnotations.hasCol(field)) {
+      return obsAnnotations.col(field);
+    }
+    return varData.col(field);
+  }
 
+  calcHistogramCache = memoize((col, field) => {
+    /*
+     recalculate expensive stuff, notably bins, summaries, etc.
+    */
+    const histogramCache = {};
+    const values = col.asArray();
+    const summary = col.summarize();
+    const { min: domainMin, max: domainMax } = summary;
+    histogramCache.x = d3
+      .scaleLinear()
+      .domain([domainMin, domainMax])
+      .range([0, this.width - this.marginRight]);
+
+    histogramCache.bins = d3
+      .histogram()
+      .domain(histogramCache.x.domain())
+      .thresholds(40)(values);
+
+    const yMax = histogramCache.bins
+      .map(b => b.length)
+      .reduce((a, b) => Math.max(a, b));
     histogramCache.y = d3
       .scaleLinear()
+      .domain([0, yMax])
       .range([this.height - this.marginBottom, 0]);
 
-    if (obsAnnotations.hasCol(field)) {
-      // recalculate expensive stuff
-      const allValuesForContinuousFieldAsArray = obsAnnotations
-        .col(field)
-        .asArray();
-
-      histogramCache.x = d3
-        .scaleLinear()
-        .domain([rangeMin, rangeMax])
-        .range([0, this.width]);
-
-      histogramCache.bins = d3
-        .histogram()
-        .domain(histogramCache.x.domain())
-        .thresholds(40)(allValuesForContinuousFieldAsArray);
-
-      histogramCache.numValues = allValuesForContinuousFieldAsArray.length;
-    } else if (world.varData.hasCol(field)) {
-      const varValues = world.varData.col(field).asArray();
-
-      histogramCache.x = d3
-        .scaleLinear()
-        .domain(
-          finiteExtent(varValues)
-        ) /* replace this if we have ranges for genes back from server like we do for annotations on cells */
-        .range([0, this.width]);
-
-      histogramCache.bins = d3
-        .histogram()
-        .domain(histogramCache.x.domain())
-        .thresholds(40)(varValues);
-
-      histogramCache.numValues = varValues.length;
-    }
-
     return histogramCache;
   });
 
@@ -76,36 +72,56 @@ class HistogramBrush extends React.Component {
 
     this.width = 340;
     this.height = 100;
-    this.marginBottom = 20;
+    this.marginBottom = 20; // space for X axis & labels
+    this.marginRight = 40; // space for Y axis & labels
   }
 
   componentDidMount() {
     const { field } = this.props;
-    const { x, y, bins, numValues, svgRef } = this._histogram;
+    const { x, y, bins, svgRef } = this._histogram;
 
-    this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+    this.renderAxesBrushBins(x, y, bins, svgRef, field);
   }
 
   componentDidUpdate(prevProps) {
-    const { field, obsAnnotations, continuousSelection } = this.props;
-    const { x, y, bins, numValues, svgRef } = this._histogram;
+    const { field, world, continuousSelection } = this.props;
+    const { x, y, bins, svgRef } = this._histogram;
+    let { brushXselection, brushX } = this.state;
+    let forceBrushUpdate = false;
 
-    if (obsAnnotations !== prevProps.obsAnnotations) {
-      this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+    /*
+    Update our axis if the underlying dataframe column has changed
+    */
+    const dfColumn = HistogramBrush.getColumn(world, field);
+    const oldDfColumn = HistogramBrush.getColumn(
+      prevProps.world,
+      prevProps.field
+    );
+    if (dfColumn !== oldDfColumn) {
+      ({ brushXselection, brushX } = this.renderAxesBrushBins(
+        x,
+        y,
+        bins,
+        svgRef,
+        field
+      ));
+      forceBrushUpdate = true;
     }
 
     /*
     if the selection has changed, ensure that the brush correctly reflects
     the underlying selection.
     */
-    if (continuousSelection !== prevProps.continuousSelection) {
+    if (
+      forceBrushUpdate ||
+      continuousSelection !== prevProps.continuousSelection
+    ) {
       const { isObs, isUserDefined, isDiffExp } = this.props;
       const myName = makeContinuousDimensionName(
         { isObs, isUserDefined, isDiffExp },
         field
       );
       const range = continuousSelection[myName];
-      const { brushXselection, brushX } = this.state;
       if (brushXselection) {
         const selection = d3.brushSelection(brushXselection.node());
         if (!range && selection) {
@@ -124,7 +140,7 @@ class HistogramBrush extends React.Component {
           const dX0 = Math.abs(x0 - selection[0]);
           const dX1 = Math.abs(x1 - selection[1]);
           /*
-          only update the brush if it is grossly incorrect, 
+          only update the brush if it is grossly incorrect,
           as defined by the moveDeltaThreshold
           */
           if (dX0 > moveDeltaThreshold || dX1 > moveDeltaThreshold) {
@@ -142,6 +158,8 @@ class HistogramBrush extends React.Component {
 
       // ignore programmatically generated events
       if (!d3.event.sourceEvent) return;
+      // ignore cascading events, which are programmatically generated
+      if (d3.event.sourceEvent.sourceEvent) return;
 
       if (d3.event.selection) {
         dispatch({
@@ -172,12 +190,13 @@ class HistogramBrush extends React.Component {
   onBrushEnd(selection, x) {
     return () => {
       const { dispatch, field, isObs, isUserDefined, isDiffExp } = this.props;
-      const { brushXselection } = this.state;
       const minAllowedBrushSize = 10;
       const smallAmountToAvoidInfiniteLoop = 0.1;
 
       // ignore programmatically generated events
       if (!d3.event.sourceEvent) return;
+      // ignore cascading events, which are programmatically generated
+      if (d3.event.sourceEvent.sourceEvent) return;
 
       if (d3.event.selection) {
         let _range;
@@ -197,11 +216,6 @@ class HistogramBrush extends React.Component {
             smallAmountToAvoidInfiniteLoop; //
 
           _range = [x(d3.event.selection[0]), x(procedurallyResizedBrushWidth)];
-
-          d3.event.target.move(brushXselection, [
-            d3.event.selection[0],
-            procedurallyResizedBrushWidth
-          ]);
         }
 
         dispatch({
@@ -216,37 +230,30 @@ class HistogramBrush extends React.Component {
         });
       } else {
         dispatch({
-          type: "continuous metadata histogram end",
+          type: "continuous metadata histogram cancel",
           selection: field,
           continuousNamespace: {
             isObs,
             isUserDefined,
             isDiffExp
-          },
-          range: null
+          }
         });
       }
     };
   }
 
   drawHistogram(svgRef) {
-    const { obsAnnotations, field, ranges } = this.props;
-    const histogramCache = this.calcHistogramCache(
-      obsAnnotations,
-      field,
-      ranges.min,
-      ranges.max
-    );
-
-    const { x, y, bins, numValues } = histogramCache;
-
-    this._histogram = { x, y, bins, numValues, svgRef };
+    const { field, world } = this.props;
+    const col = HistogramBrush.getColumn(world, field);
+    const histogramCache = this.calcHistogramCache(col, field);
+    const { x, y, bins } = histogramCache;
+    this._histogram = { x, y, bins, svgRef };
   }
 
   handleColorAction() {
-    const { obsAnnotations, dispatch, field, world, ranges } = this.props;
+    const { dispatch, field, world, ranges } = this.props;
 
-    if (obsAnnotations.hasCol(field)) {
+    if (world.obsAnnotations.hasCol(field)) {
       dispatch({
         type: "color by continuous metadata",
         colorAccessor: field,
@@ -308,29 +315,29 @@ class HistogramBrush extends React.Component {
     };
   }
 
-  renderAxesBrushBins(x, y, bins, numValues, svgRef, field) {
+  renderAxesBrushBins(x, y, bins, svgRef, field) {
+    const svg = d3.select(svgRef);
+
     /* Remove everything */
-    d3.select(svgRef)
-      .selectAll("*")
-      .remove();
+    svg.selectAll("*").remove();
 
     /* BINS */
-    d3.select(svgRef)
+    svg
       .insert("g", "*")
       .attr("fill", "#bbb")
       .selectAll("rect")
       .data(bins)
       .enter()
       .append("rect")
-      .attr("class", "bar")
       .attr("x", d => x(d.x0) + 1)
-      .attr("y", d => y(d.length / numValues))
+      .attr("y", d => y(d.length))
       .attr("width", d => Math.abs(x(d.x1) - x(d.x0) - 1))
-      .attr("height", d => y(0) - y(d.length / numValues));
+      .attr("height", d => y(0) - y(d.length));
 
     /* BRUSH */
     const brushX = d3
       .brushX()
+      .extent([[0, 0], [this.width - this.marginRight, this.height]])
       /*
       emit start so that the Undoable history can save an undo point
       upon drag start, and ignore the subsequent intermediate drag events.
@@ -345,45 +352,56 @@ class HistogramBrush extends React.Component {
       .attr("data-testid", `${svgRef.dataset.testid}-brush`)
       .call(brushX);
 
-    /* AXIS */
-    d3.select(svgRef)
+    /* X AXIS */
+    svg
       .append("g")
       .attr("class", "axis axis--x")
       .attr("transform", `translate(0,${this.height - this.marginBottom})`)
       .call(d3.axisBottom(x).ticks(5));
 
-    d3.select(svgRef)
-      .selectAll(".axis--x text")
-      .style("fill", "rgb(80,80,80)");
+    /* Y AXIS */
+    svg
+      .append("g")
+      .attr("class", "axis axis--y")
+      .attr("transform", `translate(${this.width - this.marginRight},0)`)
+      .call(d3.axisRight(y).ticks(3));
 
-    d3.select(svgRef)
-      .selectAll(".axis--x path")
-      .style("stroke", "rgb(230,230,230)");
+    /* axis style */
+    svg.selectAll(".axis text").style("fill", "rgb(80,80,80)");
+    svg.selectAll(".axis path").style("stroke", "rgb(230,230,230)");
+    svg.selectAll(".axis line").style("stroke", "rgb(230,230,230)");
 
-    d3.select(svgRef)
-      .selectAll(".axis--x line")
-      .style("stroke", "rgb(230,230,230)");
-
-    this.setState({ brushX, brushXselection });
+    const newState = { brushX, brushXselection };
+    this.setState(newState);
+    return newState;
   }
 
   render() {
     const {
       field,
+      world,
       colorAccessor,
       isUserDefined,
       isDiffExp,
       logFoldChange,
-      pval,
       pvalAdj,
       scatterplotXXaccessor,
       scatterplotYYaccessor,
       zebra
     } = this.props;
-    const field_for_id = field.replace(/\s/g, "_");
+    const fieldForId = field.replace(/\s/g, "_");
+    const {
+      min: unclippedRangeMin,
+      max: unclippedRangeMax
+    } = HistogramBrush.getColumn(world, field, false).summarize();
+    const unclippedRangeMinColor =
+      world.clipQuantiles.min === 0 ? "#bbb" : globals.blue;
+    const unclippedRangeMaxColor =
+      world.clipQuantiles.max === 1 ? "#bbb" : globals.blue;
+
     return (
       <div
-        id={`histogram_${field_for_id}`}
+        id={`histogram_${fieldForId}`}
         data-testid={`histogram-${field}`}
         data-testclass={
           isDiffExp
@@ -397,7 +415,13 @@ class HistogramBrush extends React.Component {
           backgroundColor: zebra ? globals.lightestGrey : "white"
         }}
       >
-        <div style={{ display: "flex", justifyContent: "flex-end" }}>
+        <div
+          style={{
+            display: "flex",
+            justifyContent: "flex-end",
+            paddingBottom: "8px"
+          }}
+        >
           {isDiffExp || isUserDefined ? (
             <span>
               <span
@@ -451,7 +475,7 @@ class HistogramBrush extends React.Component {
         <svg
           width={this.width}
           height={this.height}
-          id={`histogram_${field_for_id}_svg`}
+          id={`histogram_${fieldForId}_svg`}
           data-testclass="histogram-plot"
           data-testid={`histogram-${field}-plot`}
           ref={svgRef => {
@@ -461,15 +485,21 @@ class HistogramBrush extends React.Component {
         <div
           style={{
             display: "flex",
-            justifyContent: "center"
+            justifyContent: "space-between"
           }}
         >
+          <span style={{ color: unclippedRangeMinColor }}>
+            min {unclippedRangeMin.toPrecision(4)}
+          </span>
           <span
             data-testclass="brushable-histogram-field-name"
             style={{ fontStyle: "italic" }}
           >
             {field}
           </span>
+          <span style={{ color: unclippedRangeMaxColor }}>
+            max {unclippedRangeMax.toPrecision(4)}
+          </span>
         </div>
 
         {isDiffExp ? (

client/src/components/categorical/category.js

@@ -27,10 +27,10 @@ class Category extends React.Component {
       const cat = categoricalSelection[metadataField];
       const categoryCount = {
         // total number of categories in this dimension
-        totalCatCount: cat.numCategories,
+        totalCatCount: cat.numCategoryValues,
         // number of selected options in this category
         selectedCatCount: _.reduce(
-          cat.categorySelected,
+          cat.categoryValueSelected,
           (res, cond) => (cond ? res + 1 : res),
           0
         )
@@ -91,7 +91,7 @@ class Category extends React.Component {
     const { categoricalSelection, metadataField } = this.props;
 
     const cat = categoricalSelection[metadataField];
-    const optTuples = sortedCategoryValues([...cat.categoryIndices]);
+    const optTuples = sortedCategoryValues([...cat.categoryValueIndices]);
     return _.map(optTuples, (tuple, i) => (
       <Value
         optTuples={optTuples}

client/src/components/categorical/occupancy.js

@@ -1,25 +1,16 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import * as d3 from "d3";
 
 @connect()
 class Occupancy extends React.Component {
   render() {
-    const {
-      occupancy,
-      colorScale,
-      categoricalSelection,
-      colorAccessor,
-      schema
-    } = this.props;
+    const { occupancy, colorScale, colorAccessor, schema, world } = this.props;
     const width = 100;
     const height = 11;
 
-    const categories = _.filter(schema.annotations.obs, {
-      name: colorAccessor
-    })[0].categories;
+    const categories = schema.annotations.obsByName[colorAccessor]?.categories;
 
     const x = d3
       .scaleLinear()
@@ -28,8 +19,9 @@ class Occupancy extends React.Component {
       .range([0, width]);
 
     let currentOffset = 0;
-
-    const stacks = categoricalSelection[colorAccessor].categoryValues.map(d => {
+    const dfColumn = world.obsAnnotations.col(colorAccessor);
+    const categoryValues = dfColumn.summarize().categories;
+    const stacks = categoryValues.map(d => {
       const o = occupancy.get(d);
 
       const scaledValue = x(o);

client/src/components/categorical/util.js

@@ -5,7 +5,6 @@
 // return sorted index
 
 import isNumber from "is-number";
-import _ from "lodash";
 
 const sortedCategoryValues = values => {
   /* this sort could be memoized for perf */
@@ -13,7 +12,7 @@ const sortedCategoryValues = values => {
   const strings = [];
   const ints = [];
 
-  _.forEach(values, v => {
+  values.forEach(v => {
     if (isNumber(v[0])) {
       ints.push(v);
     } else {

client/src/components/categorical/value.js

@@ -1,7 +1,6 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
-import _ from "lodash";
 import Occupancy from "./occupancy";
 import { countCategoryValues2D } from "../../util/stateManager/worldUtil";
 import * as globals from "../../globals";
@@ -10,7 +9,7 @@ import * as globals from "../../globals";
   categoricalSelection: state.categoricalSelection,
   colorScale: state.colors.scale,
   colorAccessor: state.colors.colorAccessor,
-  schema: _.get(state.world, "schema", null),
+  schema: state.world?.schema,
   world: state.world
 }))
 class CategoryValue extends React.Component {
@@ -47,8 +46,8 @@ class CategoryValue extends React.Component {
     if (!categoricalSelection) return null;
 
     const category = categoricalSelection[metadataField];
-    const selected = category.categorySelected[categoryIndex];
-    const count = category.categoryCounts[categoryIndex];
+    const selected = category.categoryValueSelected[categoryIndex];
+    const count = category.categoryValueCounts[categoryIndex];
     const value = category.categoryValues[categoryIndex];
     const displayString = String(
       category.categoryValues[categoryIndex]
@@ -60,9 +59,7 @@ class CategoryValue extends React.Component {
     let occupancy = null;
 
     if (isColorBy && schema) {
-      categories = _.filter(schema.annotations.obs, {
-        name: colorAccessor
-      })[0].categories;
+      categories = schema.annotations.obsByName[colorAccessor]?.categories;
     }
 
     if (colorAccessor && !isColorBy && categoricalSelection[colorAccessor]) {

client/src/components/continuous/continuous.js

@@ -9,10 +9,10 @@ import * as globals from "../../globals";
 import HistogramBrush from "../brushableHistogram";
 
 @connect(state => ({
-  obsAnnotations: _.get(state.world, "obsAnnotations", null),
+  obsAnnotations: state.world?.obsAnnotations,
   colorAccessor: state.colors.colorAccessor,
   colorScale: state.colors.scale,
-  schema: _.get(state.world, "schema", null)
+  schema: state.world?.schema
 }))
 class Continuous extends React.Component {
   constructor(props) {
@@ -64,18 +64,16 @@ class Continuous extends React.Component {
         ) : null}
         {obsAnnotations
           ? _.map(obsAnnotations.colIndex.keys(), key => {
-              const summary = obsAnnotations.col(key).summarize();
               const isColorField =
                 key.includes("color") || key.includes("Color");
+              if (key === schema.annotations.obs.index || isColorField)
+                return null;
+
+              const summary = obsAnnotations.col(key).summarize();
               const nonFiniteExtent =
                 summary.min === undefined || summary.max === undefined;
-              zebra += 1;
-              if (
-                !summary.categorical &&
-                key !== "name" &&
-                !isColorField &&
-                !nonFiniteExtent
-              ) {
+              if (!summary.categorical && !nonFiniteExtent) {
+                zebra += 1;
                 return (
                   <HistogramBrush
                     key={key}

client/src/components/continuousLegend/index.js

@@ -112,6 +112,7 @@ class ContinuousLegend extends React.Component {
     const { colorAccessor, responsive, colorScale } = this.props;
     if (
       prevProps.colorAccessor !== colorAccessor ||
+      prevProps.colorScale !== colorScale ||
       prevProps.responsive.height !== responsive.height ||
       prevProps.responsive.width !== responsive.width
     ) {

client/src/components/framework/logo.js

@@ -0,0 +1,18 @@
+import React from "react";
+import * as globals from "../../globals";
+
+const Logo = props => {
+  const { size } = props;
+  return (
+    <svg width={size} height={size} viewBox={`0 0 48 48`} fill="none">
+      <rect width="48" height="48" fill="white" />
+      <rect width="48" height="48" fill={globals.logoColor} />
+      <rect x="19" y="19" width="22" height="22" fill="white" />
+      <rect x="24" y="24" width="12" height="12" fill={globals.logoColor} />
+      <rect x="7" y="19" width="7" height="22" fill="white" />
+      <rect x="19" y="7" width="22" height="7" fill="white" />
+    </svg>
+  );
+};
+
+export default Logo;

client/src/components/geneExpression/index.js

@@ -22,7 +22,6 @@ import {
   keepAroundErrorToast
 } from "../framework/toasters";
 import ExpressionButtons from "./expressionButtons";
-import finiteExtent from "../../util/finiteExtent";
 
 const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
   if (!modifiers.matchesPredicate) {
@@ -58,7 +57,7 @@ const filterGenes = (query, genes) =>
 
 @connect(state => {
   return {
-    obsAnnotations: _.get(state.world, "obsAnnotations", null),
+    obsAnnotations: state.world?.obsAnnotations,
     userDefinedGenes: state.controls.userDefinedGenes,
     userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
     world: state.world,
@@ -86,7 +85,8 @@ class GeneExpression extends React.Component {
     */
     const { world } = this.props;
     const { varAnnotations } = world;
-    const geneNames = varAnnotations.col("name").asArray();
+    const varIndexName = world.schema.annotations.var.index;
+    const geneNames = varAnnotations.col(varIndexName).asArray();
     if (geneNames.length > 0) {
       const placeholder = [];
       let len = geneNames.length;
@@ -108,6 +108,7 @@ class GeneExpression extends React.Component {
 
   handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
     const gene = g.target;
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
@@ -115,17 +116,22 @@ class GeneExpression extends React.Component {
       postUserErrorToast(
         "That's too many genes, you can have at most 15 user defined genes"
       );
-    } else if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
+    } else if (
+      world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+    ) {
       postUserErrorToast("That doesn't appear to be a valid gene name.");
     } else {
       dispatch({ type: "single user defined gene start" });
-      dispatch(actions.requestUserDefinedGene(gene));
-      dispatch({ type: "single user defined gene complete" });
+      dispatch(actions.requestUserDefinedGene(gene)).then(
+        () => dispatch({ type: "single user defined gene complete" }),
+        () => dispatch({ type: "single user defined gene error" })
+      );
     }
   }
 
   handleBulkAddClick() {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
     const { bulkAdd } = this.state;
 
     /*
@@ -136,22 +142,27 @@ class GeneExpression extends React.Component {
       const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");
 
       dispatch({ type: "bulk user defined gene start" });
-      genes.forEach(gene => {
-        if (gene.length === 0) {
-          keepAroundErrorToast("Must enter a gene name.");
-        } else if (userDefinedGenes.indexOf(gene) !== -1) {
-          keepAroundErrorToast("That gene already exists");
-        } else if (
-          world.varAnnotations.col("name").indexOf(gene) === undefined
-        ) {
-          keepAroundErrorToast(
-            `${gene} doesn't appear to be a valid gene name.`
-          );
-        } else {
-          dispatch(actions.requestUserDefinedGene(gene));
-        }
-      });
-      dispatch({ type: "bulk user defined gene complete" });
+      Promise.all(
+        genes.map(gene => {
+          if (gene.length === 0) {
+            return keepAroundErrorToast("Must enter a gene name.");
+          }
+          if (userDefinedGenes.indexOf(gene) !== -1) {
+            return keepAroundErrorToast("That gene already exists");
+          }
+          if (
+            world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+          ) {
+            return keepAroundErrorToast(
+              `${gene} doesn't appear to be a valid gene name.`
+            );
+          }
+          return dispatch(actions.requestUserDefinedGene(gene));
+        })
+      ).then(
+        () => dispatch({ type: "bulk user defined gene complete" }),
+        () => dispatch({ type: "bulk user defined gene error" })
+      );
     }
 
     this.setState({ bulkAdd: "" });
@@ -164,7 +175,7 @@ class GeneExpression extends React.Component {
       userDefinedGenesLoading,
       differential
     } = this.props;
-
+    const varIndexName = world?.schema?.annotations?.var?.index;
     const { tab, bulkAdd } = this.state;
 
     return (
@@ -220,9 +231,9 @@ class GeneExpression extends React.Component {
               }}
             >
               <Suggest
-                closeOnSelect
-                openOnKeyDown
                 resetOnSelect
+                closeOnSelect
+                resetOnClose
                 itemDisabled={
                   userDefinedGenesLoading ? () => true : () => false
                 }
@@ -239,7 +250,7 @@ class GeneExpression extends React.Component {
                 itemRenderer={renderGene.bind(this)}
                 items={
                   world && world.varAnnotations
-                    ? world.varAnnotations.col("name").asArray()
+                    ? world.varAnnotations.col(varIndexName).asArray()
                     : ["No genes"]
                 }
                 popoverProps={{ minimal: true }}
@@ -318,7 +329,7 @@ class GeneExpression extends React.Component {
           <ExpressionButtons />
           {differential.diffExp
             ? _.map(differential.diffExp, (value, index) => {
-                const name = world.varAnnotations.at(value[0], "name");
+                const name = world.varAnnotations.at(value[0], varIndexName);
                 const values = world.varData.col(name);
                 if (!values) {
                   return null;

client/src/components/graph/drawPointsRegl.js

@@ -1,5 +1,6 @@
 // jshint esversion: 6
 const mat4 = require("gl-mat4");
+const vec3 = require("gl-vec3");
 
 // opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js
 
@@ -38,7 +39,20 @@ export default function(regl) {
     uniforms: {
       distance: regl.prop("distance"),
       view: regl.prop("view"),
-      projection: ({viewportWidth, viewportHeight}) => mat4.perspective([], Math.PI / 2, viewportWidth / viewportHeight, 0.01, 1000)
+      projection: ({ viewportWidth, viewportHeight }) => {
+        const aspectRatio = viewportWidth / viewportHeight;
+        let m = mat4.perspective(
+          [],
+          Math.PI / 2,
+          viewportWidth / viewportHeight,
+          0.01,
+          1000
+        );
+        if (aspectRatio < 1) {
+          m = mat4.scale(m, m, vec3.fromValues(1, 1, 1 / aspectRatio));
+        }
+        return m;
+      }
     },
 
     count: regl.prop("count"),

client/src/components/graph/setupLasso.js

@@ -3,7 +3,7 @@
 import * as d3 from "d3";
 
 const Lasso = () => {
-  const dispatch = d3.dispatch("start", "end");
+  const dispatch = d3.dispatch("start", "end", "cancel");
 
   const polygonToPath = polygon =>
     `M${polygon.map(d => d.join(",")).join("L")}`;
@@ -28,6 +28,7 @@ const Lasso = () => {
 
       lassoPath = g
         .append("path")
+        .attr("data-testid", "lasso-element")
         .attr("fill", "#0bb")
         .attr("fill-opacity", 0.1)
         .attr("stroke", "#0bb")
@@ -81,6 +82,7 @@ const Lasso = () => {
         lassoPath.remove();
         lassoPath = null;
         lassoPolygon = null;
+        dispatch.call("cancel");
       }
     };
 
@@ -114,6 +116,23 @@ const Lasso = () => {
         closePath = null;
       }
     };
+
+    lasso.move = polygon => {
+      if (polygon !== lassoPolygon || polygon.length !== lassoPolygon.length) {
+        lasso.reset();
+
+        lassoPolygon = polygon;
+        lassoPath = g
+          .append("path")
+          .attr("data-testid", "lasso-element")
+          .attr("fill", "#0bb")
+          .attr("fill-opacity", 0.1)
+          .attr("stroke", "#0bb")
+          .attr("stroke-dasharray", "3, 3");
+
+        lassoPath.attr("d", `${polygonToPath(lassoPolygon)}Z`);
+      }
+    };
   };
 
   lasso.on = (type, callback) => {

client/src/components/graph/setupSVGandBrush.js

@@ -10,12 +10,13 @@ import Lasso from "./setupLasso";
 ******************************************/
 
 export default (
-  handleBrushSelectAction,
-  handleBrushDeselectAction,
+  selectionToolType,
+  handleStartAction,
+  handleDragAction,
+  handleEndAction,
+  handleCancelAction,
   responsive,
-  graphPaddingRight,
-  handleLassoStart,
-  handleLassoEnd
+  graphPaddingRight
 ) => {
   const svg = d3
     .select("#graphAttachPoint")
@@ -25,27 +26,37 @@ export default (
     .attr("height", responsive.height)
     .attr("class", `${styles.graphSVG}`);
 
-  const brush = d3
-    .brush()
-    .extent([[0, 0], [responsive.width - graphPaddingRight, responsive.height]])
-    .on("brush", handleBrushSelectAction)
-    .on("end", handleBrushDeselectAction);
+  if (selectionToolType === "brush") {
+    const brush = d3
+      .brush()
+      .extent([
+        [0, 0],
+        [responsive.width - graphPaddingRight, responsive.height]
+      ])
+      .on("start", handleStartAction)
+      .on("brush", handleDragAction)
+      // FYI, brush doesn't generate cancel
+      .on("end", handleEndAction);
 
-  const brushContainer = svg
-    .append("g")
-    .attr("class", "graph_brush")
-    .call(brush);
+    const brushContainer = svg
+      .append("g")
+      .attr("class", "graph_brush")
+      .call(brush);
 
-  const lassoInstance = Lasso()
-    .on("end", handleLassoEnd)
-    .on("start", handleLassoStart);
+    return { svg, container: brushContainer, tool: brush };
+  }
 
-  const lasso = svg.call(lassoInstance);
+  if (selectionToolType === "lasso") {
+    const lasso = Lasso()
+      .on("end", handleEndAction)
+      // FYI, Lasso doesn't generate drag
+      .on("start", handleStartAction)
+      .on("cancel", handleCancelAction);
 
-  return {
-    svg,
-    brushContainer,
-    brush,
-    lasso
-  };
+    const lassoContainer = svg.call(lasso);
+
+    return { svg, container: lassoContainer, tool: lasso };
+  }
+
+  throw new Error("unknown graph selection tool");
 };

client/src/components/graph/util.js

@@ -1,26 +0,0 @@
-// jshint esversion: 6
-// createExpressionsCountsMap () {
-//
-//   const CHANGE_ME_MAGIC_GENE_INDEX = 5;
-//
-//   const expressionsCountsMap = {};
-//
-//   /* currently selected gene */
-//   expressionsCountsMap.geneName = this.state.expressions.data.genes[3];
-//
-//   let maxExpressionValue = 0;
-//
-//   /* create map of expressions for every cell */
-//   this.state.expressions.data.cells.map((c) => {
-//     /* cellname = 234 */
-//     expressionsCountsMap[c.cellname] = c["e"][CHANGE_ME_MAGIC_GENE_INDEX];
-//     /* collect the maximum value as we iterate */
-//     if (c["e"][CHANGE_ME_MAGIC_GENE_INDEX] > maxExpressionValue) {
-//       maxExpressionValue = c["e"][CHANGE_ME_MAGIC_GENE_INDEX]
-//     }
-//   })
-//
-//   expressionsCountsMap.maxValue = maxExpressionValue;
-//
-//   return expressionsCountsMap;
-// }

client/src/components/leftsidebar.js

@@ -1,5 +1,4 @@
 // jshint esversion: 6
-import _ from "lodash";
 import React from "react";
 import { connect } from "react-redux";
 import Categorical from "./categorical/categorical";
@@ -7,10 +6,11 @@ import Continuous from "./continuous/continuous";
 import GeneExpression from "./geneExpression";
 import * as globals from "../globals";
 import DynamicScatterplot from "./scatterplot/scatterplot";
+import Logo from "./framework/logo.js";
 
 @connect(state => ({
   responsive: state.responsive,
-  datasetTitle: _.get(state.config, "displayNames.dataset"),
+  datasetTitle: state.config?.displayNames?.dataset,
   scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor
 }))
@@ -28,7 +28,6 @@ class LeftSideBar extends React.Component {
     if cellxgene logo or tabs change, this must as well
     */
     const metadataSectionPadding = 0;
-    // scatterplotXXaccessor && scatterplotYYaccessor ? 450 : 0;
 
     return (
       <div
@@ -40,20 +39,49 @@ class LeftSideBar extends React.Component {
         }}
       >
         <p
-          data-testid="header"
           style={{
             position: "fixed",
             top: globals.cellxgeneTitleTopPadding,
             left: globals.leftSidebarWidth + globals.cellxgeneTitleLeftPadding,
-            margin: 0,
-            fontSize: globals.largestFontSize,
-            color: globals.darkerGrey,
-            width: "100%"
+            margin: 0
           }}
         >
-          cellxgene: {datasetTitle}
+          <Logo size={32} />
+          <span
+            style={{
+              fontSize: 28,
+              position: "relative",
+              top: -4,
+              fontWeight: "bold",
+              marginLeft: 5,
+              color: globals.logoColor,
+              userSelect: "none"
+            }}
+          >
+            cell<span
+              style={{
+                position: "relative",
+                top: 1,
+                fontWeight: 300,
+                fontSize: 24
+              }}
+            >
+              ×
+            </span>gene
+          </span>
+          <span
+            data-testid="header"
+            style={{
+              fontSize: 16,
+              display: "block",
+              position: "relative",
+              marginTop: 10,
+              top: -4
+            }}
+          >
+            {datasetTitle}
+          </span>
         </p>
-
         <div
           style={{
             height: responsive.height - metadataSectionPadding,

client/src/components/scatterplot/scatterplot.js

@@ -145,7 +145,8 @@ class Scatterplot extends React.Component {
 
     if (
       scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
-      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor // was CLU now FTH1 etc
+      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor || // was CLU now FTH1 etc
+      world !== prevProps.world // shape or clip of world changed
     ) {
       const scales = Scatterplot.setupScales(expressionX, expressionY);
       this.drawAxesSVG(scales.xScale, scales.yScale, svg);
@@ -263,9 +264,15 @@ class Scatterplot extends React.Component {
 
     // the axes are much cleaner and easier now. No need to rotate and orient
     // the axis, just call axisBottom, axisLeft etc.
-    const xAxis = d3.axisBottom().scale(xScale);
+    const xAxis = d3
+      .axisBottom()
+      .ticks(7)
+      .scale(xScale);
 
-    const yAxis = d3.axisLeft().scale(yScale);
+    const yAxis = d3
+      .axisLeft()
+      .ticks(7)
+      .scale(yScale);
 
     // adding axes is also simpler now, just translate x-axis to (0,height)
     // and it's alread defined to be a bottom axis.

client/src/globals.js

@@ -31,6 +31,7 @@ export const darkGreen = "#448C4D";
 
 export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";
+export const logoColor = "black"; /* logo pink: "#E9429A" */
 
 /* typography constants */
 

client/src/reducers/categoricalSelection.js

@@ -1,12 +1,9 @@
-import _ from "lodash";
-
 import { ControlsHelpers } from "../util/stateManager";
 import * as globals from "../globals";
 
 function maxCategoryItems(state) {
-  return _.get(
-    state.config,
-    "parameters.max-category-items",
+  return (
+    state.config.parameters?.["max-category-items"] ??
     globals.configDefaults.parameters["max-category-items"]
   );
 }
@@ -20,7 +17,8 @@ const CategoricalSelection = (
   switch (action.type) {
     case "initial data load complete (universe exists)":
     case "set World to current selection":
-    case "reset World to eq Universe": {
+    case "reset World to eq Universe":
+    case "set clip quantiles": {
       const { world } = nextSharedState;
       return ControlsHelpers.createCategoricalSelection(
         maxCategoryItems(prevSharedState),
@@ -32,15 +30,15 @@ const CategoricalSelection = (
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
-        state[action.metadataField].categorySelected
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = true;
+      newCategoryValueSelected[action.categoryIndex] = true;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -50,15 +48,15 @@ const CategoricalSelection = (
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
-        state[action.metadataField].categorySelected
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = false;
+      newCategoryValueSelected[action.categoryIndex] = false;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -72,8 +70,9 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
-            state[action.metadataField].categorySelected
+          categorySelected: false,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(false)
         }
       };
@@ -88,8 +87,9 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
-            state[action.metadataField].categorySelected
+          categorySelected: true,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(true)
         }
       };

client/src/reducers/colors.js

@@ -27,6 +27,7 @@ const ColorsReducer = (
       };
     }
 
+    case "set clip quantiles":
     case "set World to current selection": {
       const { colorMode, colorAccessor } = state;
       const { world } = nextSharedState;
@@ -53,15 +54,19 @@ const ColorsReducer = (
     case "color by categorical metadata":
     case "color by continuous metadata": {
       const { world } = prevSharedState;
-      const { rgb, scale } = createColors(
-        world,
-        action.type,
-        action.colorAccessor
-      );
+
+      /* toggle between this mode and reset */
+      const resetCurrent =
+        action.type === state.colorMode &&
+        action.colorAccessor === state.colorAccessor;
+      const colorMode = !resetCurrent ? action.type : null;
+      const colorAccessor = !resetCurrent ? action.colorAccessor : null;
+
+      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
       return {
         ...state,
-        colorMode: action.type,
-        colorAccessor: action.colorAccessor,
+        colorMode,
+        colorAccessor,
         rgb,
         scale
       };
@@ -69,11 +74,18 @@ const ColorsReducer = (
 
     case "color by expression": {
       const { world } = prevSharedState;
-      const { rgb, scale } = createColors(world, action.type, action.gene);
+
+      /* toggle between this mode and reset */
+      const resetCurrent =
+        action.type === state.colorMode && action.gene === state.colorAccessor;
+      const colorMode = !resetCurrent ? action.type : null;
+      const colorAccessor = !resetCurrent ? action.gene : null;
+
+      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
       return {
         ...state,
-        colorMode: action.type,
-        colorAccessor: action.gene,
+        colorMode,
+        colorAccessor,
         rgb,
         scale
       };

client/src/reducers/continuousSelection.js

@@ -2,7 +2,8 @@ import { makeContinuousDimensionName } from "../util/nameCreators";
 
 const ContinuousSelection = (state = {}, action) => {
   switch (action.type) {
-    case "reset World to eq Universe": {
+    case "reset World to eq Universe":
+    case "set clip quantiles": {
       return {};
     }
     case "continuous metadata histogram start":
@@ -17,6 +18,14 @@ const ContinuousSelection = (state = {}, action) => {
         [name]: action.range
       };
     }
+    case "continuous metadata histogram cancel": {
+      const name = makeContinuousDimensionName(
+        action.continuousNamespace,
+        action.selection
+      );
+      const { [name]: deletedField, ...newState } = state;
+      return newState;
+    }
     default: {
       return state;
     }

client/src/reducers/controls.js

@@ -93,9 +93,10 @@ const Controls = (
     }
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const _diffexpGenes = [];
       action.data.forEach(d => {
-        _diffexpGenes.push(world.varAnnotations.at(d[0], "name"));
+        _diffexpGenes.push(world.varAnnotations.at(d[0], varIndexName));
       });
       return {
         ...state,

client/src/reducers/crossfilter.js

@@ -10,6 +10,8 @@ import {
   makeContinuousDimensionName
 } from "../util/nameCreators";
 
+const XYDimName = layoutDimensionName("XY");
+
 const CrossfilterReducer = (
   state = null,
   action,
@@ -18,10 +20,11 @@ const CrossfilterReducer = (
 ) => {
   switch (action.type) {
     case "initial data load complete (universe exists)": {
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       const crossfilter = World.createObsDimensions(
         new Crossfilter(world.obsAnnotations),
-        world
+        world,
+        layoutChoice.currentDimNames
       );
       return crossfilter;
     }
@@ -38,11 +41,16 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "set clip quantiles":
     case "set World to current selection": {
       const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       let crossfilter = new Crossfilter(world.obsAnnotations);
-      crossfilter = World.createObsDimensions(crossfilter, world);
+      crossfilter = World.createObsDimensions(
+        crossfilter,
+        world,
+        layoutChoice.currentDimNames
+      );
       crossfilter = ControlsHelpers.createGeneDimensions(
         userDefinedGenes,
         diffexpGenes,
@@ -52,6 +60,23 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "set layout choice": {
+      /*
+      when switching layouts:
+      - delete the existing XY index
+      - add the new XY index (which implicitly selects all on it)
+      */
+      const { world, layoutChoice } = nextSharedState;
+      return state
+        .delDimension(layoutDimensionName("XY"))
+        .addDimension(
+          layoutDimensionName("XY"),
+          "spatial",
+          world.obsLayout.col(layoutChoice.currentDimNames[0]).asArray(),
+          world.obsLayout.col(layoutChoice.currentDimNames[1]).asArray()
+        );
+    }
+
     case "request user defined gene success": {
       const { world } = prevSharedState;
       const gene = action.data.genes[0];
@@ -65,8 +90,9 @@ const CrossfilterReducer = (
 
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const genes = _.map(action.data, d =>
-        world.varAnnotations.at(d[0], "name")
+        world.varAnnotations.at(d[0], varIndexName)
       );
       const crossfilter = _.reduce(
         genes,
@@ -84,10 +110,11 @@ const CrossfilterReducer = (
 
     case "clear differential expression": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const crossfilter = _.reduce(
         action.diffExp,
         (xfltr, values) => {
-          const name = world.varAnnotations.at(values[0], "name");
+          const name = world.varAnnotations.at(values[0], varIndexName);
           return xfltr.delDimension(diffexpDimensionName(name));
         },
         state
@@ -109,40 +136,37 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
-    case "graph brush selection change": {
-      const name = layoutDimensionName("XY");
-      const [x0, y0] = action.brushCoords.northwest;
-      const [x1, y1] = action.brushCoords.southeast;
-      return state.select(name, {
+    case "graph brush end":
+    case "graph brush change": {
+      const [minX, maxY] = action.brushCoords.northwest;
+      const [maxX, minY] = action.brushCoords.southeast;
+      return state.select(XYDimName, {
         mode: "within-rect",
-        x0,
-        y0,
-        x1,
-        y1
+        minX,
+        minY,
+        maxX,
+        maxY
       });
     }
 
-    case "lasso deselect":
-    case "graph brush deselect": {
-      const name = layoutDimensionName("XY");
-      return state.select(name, { mode: "all" });
-    }
-
-    case "lasso selection": {
+    case "graph lasso end": {
       const { polygon } = action;
-      const name = layoutDimensionName("XY");
-      if (polygon.length < 3) {
-        // single point or a line is not a polygon, and is therefore a deselect
-        return state.select(name, { mode: "all" });
-      }
-      return state.select(name, {
+      return state.select(XYDimName, {
         mode: "within-polygon",
         polygon
       });
     }
 
+    case "graph lasso cancel":
+    case "graph brush cancel":
+    case "graph lasso deselect":
+    case "graph brush deselect": {
+      return state.select(XYDimName, { mode: "all" });
+    }
+
     case "continuous metadata histogram start":
     case "continuous metadata histogram brush":
+    case "continuous metadata histogram cancel":
     case "continuous metadata histogram end": {
       const name = makeContinuousDimensionName(
         action.continuousNamespace,
@@ -161,10 +185,12 @@ const CrossfilterReducer = (
     case "categorical metadata filter select":
     case "categorical metadata filter deselect": {
       const { categoricalSelection } = nextSharedState;
+      const { world } = prevSharedState;
       const cat = categoricalSelection[action.metadataField];
+      const col = world.obsAnnotations.col(action.metadataField);
       return state.select(obsAnnoDimensionName(action.metadataField), {
         mode: "exact",
-        values: ControlsHelpers.selectedValuesForCategory(cat)
+        values: ControlsHelpers.selectedValuesForCategory(cat, col)
       });
     }
 

client/src/reducers/graphSelection.js

@@ -0,0 +1,61 @@
+const GraphSelection = (
+  state = {
+    tool: "lasso", // what selection tool mode (lasso, brush, ...)
+    selection: { mode: "all" } // current selection, which is tool specific
+  },
+  action
+) => {
+  switch (action.type) {
+    case "set clip quantiles":
+    case "reset World to eq Universe":
+    case "set layout choice": {
+      return {
+        ...state,
+        selection: {
+          mode: "all"
+        }
+      };
+    }
+
+    case "graph brush end":
+    case "graph brush change": {
+      const { brushCoords } = action;
+      return {
+        ...state,
+        selection: {
+          mode: "within-rect",
+          brushCoords
+        }
+      };
+    }
+
+    case "graph lasso end": {
+      const { polygon } = action;
+      return {
+        ...state,
+        selection: {
+          mode: "within-polygon",
+          polygon
+        }
+      };
+    }
+
+    case "graph lasso cancel":
+    case "graph brush cancel":
+    case "graph lasso deselect":
+    case "graph brush deselect": {
+      return {
+        ...state,
+        selection: {
+          mode: "all"
+        }
+      };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default GraphSelection;

client/src/reducers/index.js

@@ -8,80 +8,45 @@ import universe from "./universe";
 import world from "./world";
 import categoricalSelection from "./categoricalSelection";
 import continuousSelection from "./continuousSelection";
+import graphSelection from "./graphSelection";
 import crossfilter from "./crossfilter";
 import colors from "./colors";
 import differential from "./differential";
+import layoutChoice from "./layoutChoice";
 import responsive from "./responsive";
 import controls from "./controls";
 import resetCache from "./resetCache";
 
-const ignoredActions = new Set([
-  // these actions will not affect history, ie, we will
-  // not snapshot history upon these actions.  These take
-  // precedent over `clearHistoryUponActions`
-  "url changed",
-  "interface reset started",
-  "initial data load start",
-  "configuration load complete",
-  "increment graph render counter",
-  "window resize",
-
-  "lasso started",
-
-  "request differential expression success",
-
-  "expression load start",
-  "expression load success",
-  "expression load error",
-
-  "continuous metadata histogram brush",
-  "continuous metadata histogram end",
-
-  "request user defined gene started",
-  "request user defined gene success",
-  "request user defined gene error",
-  "bulk user defined gene complete",
-  "single user defined gene complete"
-]);
-
-const clearOnActions = new Set([
-  // history will be cleared when these actions occur
-  "initial data load complete (universe exists)",
-  "reset World to eq Universe",
-  "initial data load error"
-]);
-
-/* configuration for the undoable meta reducer */
-const undoableConfig = {
-  historyLimit: 50, // maximum history size
-  skipActionFilter: (state, action) => ignoredActions.has(action.type),
-  clearOnActionFilter: (state, action) => clearOnActions.has(action.type)
-};
+import undoableConfig from "./undoableConfig";
 
 const Reducer = undoable(
-  cascadeReducers([
-    ["config", config],
-    ["universe", universe],
-    ["world", world],
-    ["categoricalSelection", categoricalSelection],
-    ["continuousSelection", continuousSelection],
-    ["crossfilter", crossfilter],
-    ["colors", colors],
-    ["controls", controls],
-    ["differential", differential],
-    ["responsive", responsive],
-    ["resetCache", resetCache]
-  ]),
-  [
-    "world",
-    "categoricalSelection",
-    "continuousSelection",
-    "crossfilter",
-    "colors",
-    "controls",
-    "differential"
-  ],
-  undoableConfig
+    cascadeReducers([
+        ["config", config],
+        ["universe", universe],
+        ["world", world],
+        ["layoutChoice", layoutChoice],
+        ["categoricalSelection", categoricalSelection],
+        ["continuousSelection", continuousSelection],
+        ["graphSelection", graphSelection],
+        ["crossfilter", crossfilter],
+        ["colors", colors],
+        ["controls", controls],
+        ["differential", differential],
+        ["responsive", responsive],
+        ["resetCache", resetCache]
+    ]),
+    [
+        "world",
+        "categoricalSelection",
+        "continuousSelection",
+        "graphSelection",
+        "crossfilter",
+        "colors",
+        "controls",
+        "differential",
+        "layoutChoice"
+    ],
+    undoableConfig
 );
 
 const store = createStore(Reducer, applyMiddleware(thunk));

client/src/reducers/layoutChoice.js

@@ -0,0 +1,50 @@
+/*
+we have a UI heuristic to pick the default layout, based on assumptions
+about commonly used names.  Preferentially, pick in the following order:
+
+  1. "umap"
+  2. "tsne"
+  3. "pca"
+  4. give up, use the first available
+*/
+function bestDefaultLayout(layouts) {
+  const preferredNames = ["umap", "tsne", "pca"];
+  const idx = preferredNames.findIndex(name => layouts.indexOf(name) !== -1);
+  if (idx !== -1) return preferredNames[idx];
+  return layouts[0];
+}
+
+const LayoutChoice = (
+  state = {
+    available: [], // all available choices
+    current: undefined, // name of the current layout, eg, 'umap'
+    currentDimNames: [] // dimension name
+  },
+  action,
+  nextSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
+      // set default to default
+      const { schema } = nextSharedState.world;
+      const available = schema.layout.obs.map(v => v.name);
+      const current = bestDefaultLayout(available);
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { available, current, currentDimNames };
+    }
+
+    case "set layout choice": {
+      const { schema } = nextSharedState.world;
+      const current = action.layoutChoice;
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { ...state, current, currentDimNames };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default LayoutChoice;

client/src/reducers/resetCache.js

@@ -1,7 +1,10 @@
 /*
-Reducer which caches derived state to be used in a reset
-*/
+Reducer which caches derived state to be used in a reset or other 
+recomputation.
 
+Currently this only caches the baseline (full universe) world & crossfilter, 
+for use in a Reset.
+*/
 const ResetCacheReducer = (
   state = {
     world: null,

client/src/reducers/undoable.js

@@ -8,35 +8,74 @@ Requires three parameters:
   state to be made "undoable".
 * options - an optional object, which may contain the following parameters:
     * historyLimit: max number of historical states to remember (aka max undo depth)
-    * skipActionFilter: filter function, (state, action) => bool.  If it returns
-      truthy, the current state will not be pushed onto the history stack.
-    * clearOnActionFilter: filter function, (state, action) => bool.  If it returns
-      truthy, the history state will be cleared as part of handling this action.
-
-skipActionFilter has precedence over clearOnActionFilter.
+    * actionFilter: filter function, (state, action, filterState) => value.
+      See below for details.
+    * debug: if truish, will print helpful log messages about history manipulation
 
 This meta reducer accepts three actions types:
 * @@undoable/undo - move back in history
 * @@undoable/redo - move forward in history
 * @@undoable/clear - clear history
+
+---
+
+Action filter - controls the undoable reducer side-effects.  If not
+specified, the action filter defaults to "save", ie, pushes a redo
+point upon each action.
+
+The action filter callback has access to the current action, the entire
+undoable reducer state, and any state it wants to manage ("filterState").
+This filter state will be passed to each action filter call, and any
+value returned (via @@undoable/filterState field described below) will be
+MERGED into the current filter state.
+
+An object must be returned (the "undoable action"), indicating desired
+history state processing.  The undoable action object contents, by key:
+
+  @@undoable/filterAction: required.   Can be one of:
+      "skip" - reduce the current action, but no other side effects.
+        Same as returning false.
+      "clear" - reduce the current action, and clear history state.
+      "save" - push the previous state onto the history stack (ie,
+        before reducing the action)
+      "stashPending" - reduce action, save state as pending.  Does not
+        not commit it to history.  Along with cancelPending and applyPending,
+        can be used to delay commit of history (eg, for multi-action
+        groupings, asynch operations, etc).
+      "cancelPending" - reduce action, cancel any pending state save.
+      "applyPending" - commit any pending state to the history stack,
+        then reduce action.
+
+  @@undoable/filterState: optional.  If this value is set, it will be
+    MERGED into the current filter state. The value and semantics of any
+    filter state are entirely at the discretion of the action filter.
+
 */
+import fromEntries from "../util/fromEntries";
 
 const historyKeyPrefix = "@@undoable/";
 const pastKey = `${historyKeyPrefix}past`;
 const futureKey = `${historyKeyPrefix}future`;
+const filterStateKey = `${historyKeyPrefix}filterState`;
+const filterActionKey = `${historyKeyPrefix}filterAction`;
+const pendingKey = `${historyKeyPrefix}pending`;
 const defaultHistoryLimit = -100;
 
 const Undoable = (reducer, undoableKeys, options = {}) => {
+  const { debug } = options;
   let { historyLimit } = options;
   if (!historyLimit) historyLimit = defaultHistoryLimit;
   if (historyLimit > 0) historyLimit = -historyLimit;
-  const skipActionFilter = options.skipActionFilter || (() => false);
-  const clearOnActionFilter = options.clearOnActionFilter || (() => false);
+  const actionFilter =
+    options.actionFilter || (() => ({ [filterActionKey]: "save" }));
 
   if (!Array.isArray(undoableKeys) || undoableKeys.length === 0)
     throw new Error("undoable keys array must be specified");
   const undoableKeysSet = new Set(undoableKeys);
 
+  /*
+  Undo the current to previous history
+  */
   function undo(currentState) {
     const past = currentState[pastKey];
     const future = currentState[futureKey];
@@ -51,11 +90,15 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
       ...currentState,
       ...fromEntries(newState),
       [pastKey]: newPast,
-      [futureKey]: newFuture
+      [futureKey]: newFuture,
+      [pendingKey]: null
     };
     return nextState;
   }
 
+  /*
+  Replay future, previously undone.
+  */
   function redo(currentState) {
     const past = currentState[pastKey] || [];
     const future = currentState[futureKey] || [];
@@ -70,30 +113,45 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
       ...currentState,
       ...fromEntries(newState),
       [pastKey]: newPast,
-      [futureKey]: newFuture
+      [futureKey]: newFuture,
+      [pendingKey]: null
     };
     return nextState;
   }
 
+  /*
+  Clear the history state.  No side-effects on current state.
+  */
   function clear(currentState) {
     return {
       ...currentState,
       [pastKey]: [],
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: {},
+      [pendingKey]: null
     };
   }
 
-  function skip(currentState, action) {
+  /*
+  Reduce current action, with no history side-effects
+  */
+  function skip(currentState, action, filterState) {
     const past = currentState[pastKey] || [];
+    const pending = currentState[pendingKey];
     const res = reducer(currentState, action);
     return {
       ...res,
       [pastKey]: past,
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: filterState,
+      [pendingKey]: pending
     };
   }
 
-  function save(currentState, action) {
+  /*
+  Save current state in the history, then reduce action.
+  */
+  function save(currentState, action, filterState) {
     const past = currentState[pastKey] || [];
     const currentUndoableState = Object.entries(currentState).filter(kv =>
       undoableKeysSet.has(kv[0])
@@ -103,7 +161,48 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
     const nextState = {
       ...res,
       [pastKey]: newPast,
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: filterState,
+      [pendingKey]: null
+    };
+    return nextState;
+  }
+
+  /*
+  Save current state as pending history change.  No other side effects.
+  */
+  function stashPending(currentState) {
+    const currentUndoableState = Object.entries(currentState).filter(kv =>
+      undoableKeysSet.has(kv[0])
+    );
+    return {
+      ...currentState,
+      [pendingKey]: currentUndoableState
+    };
+  }
+
+  /*
+  Cancel pending history state change.  No other side effects.
+  */
+  function cancelPending(currentState) {
+    return {
+      ...currentState,
+      [pendingKey]: null
+    };
+  }
+
+  /*
+  Push pending state onto the history stack
+  */
+  function applyPending(currentState) {
+    const past = currentState[pastKey] || [];
+    const pendingState = currentState[pendingKey];
+    const newPast = push(past, pendingState, historyLimit);
+    const nextState = {
+      ...currentState,
+      [pastKey]: newPast,
+      [futureKey]: [],
+      [pendingKey]: null
     };
     return nextState;
   }
@@ -111,7 +210,9 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
   return (
     currentState = {
       [pastKey]: [],
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: {},
+      [pendingKey]: null
     },
     action
   ) => {
@@ -120,20 +221,53 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
       case "@@undoable/undo": {
         return undo(currentState, action);
       }
+
       case "@@undoable/redo": {
         return redo(currentState, action);
       }
+
       case "@@undoable/clear": {
         return clear(currentState, action);
       }
+
       default: {
-        if (skipActionFilter(currentState, action)) {
-          return skip(currentState, action);
+        const currentFilterState = currentState[filterStateKey];
+        const actionFilterResp = actionFilter(
+          currentState,
+          action,
+          currentFilterState
+        );
+        const {
+          [filterActionKey]: filterAction,
+          [filterStateKey]: filterStateUpdate
+        } = actionFilterResp;
+        const nextFilterState = { ...currentFilterState, ...filterStateUpdate };
+
+        switch (filterAction) {
+          case "clear":
+            if (debug) console.log("---- CLEAR HISTO", action.type);
+            return clear(skip(currentState, action, nextFilterState));
+
+          case "save":
+            if (debug) console.log("---- SAVE HISTO", action.type);
+            return save(currentState, action, nextFilterState);
+
+          case "stashPending":
+            if (debug) console.log("---- STASH PENDING", action.type);
+            return skip(stashPending(currentState), action, nextFilterState);
+
+          case "cancelPending":
+            if (debug) console.log("---- CANCEL PENDING", action.type);
+            return skip(cancelPending(currentState), action, nextFilterState);
+
+          case "applyPending":
+            if (debug) console.log("---- APPLY PENDING", action.type);
+            return skip(applyPending(currentState), action, nextFilterState);
+
+          case "skip":
+          default:
+            return skip(currentState, action, nextFilterState);
         }
-        if (clearOnActionFilter(currentState, action)) {
-          return clear(skip(currentState, action));
-        }
-        return save(currentState, action);
       }
     }
   };
@@ -150,18 +284,4 @@ function push(arr, val, limit = undefined) {
   return narr;
 }
 
-function fromEntries(arr) {
-  /*
-  Similar to Object.fromEntries, but only handles array.
-  This could be replaced with the standard fucnction once it
-  is widely available.   As of 3/20/2019, it has not yet
-  been released in the Chrome stable channel.
-  */
-  const obj = {};
-  for (let i = 0, l = arr.length; i < l; i += 1) {
-    obj[arr[i][0]] = arr[i][1];
-  }
-  return obj;
-}
-
 export default Undoable;

client/src/reducers/undoableConfig.js

@@ -0,0 +1,259 @@
+import StateMachine from "../util/statemachine";
+import createFsmTransitions from "./undoableFsm";
+
+const actionKey = "@@undoable/filterAction";
+const stateKey = "@@undoable/filterState";
+
+/*
+these actions will not affect history
+*/
+const skipOnActions = new Set([
+  "url changed",
+  "interface reset started",
+  "initial data load start",
+  "configuration load complete",
+  "increment graph render counter",
+  "window resize",
+  "user reset start",
+  "reset colorscale",
+
+  "graph brush change",
+  "continuous metadata histogram brush",
+
+  "expression load start",
+  "expression load success",
+  "expression load error",
+
+  "request user defined gene started",
+  "request user defined gene success",
+  "clear all user defined genes",
+
+  "get single gene expression for coloring started",
+  "get single gene expression for coloring error"
+]);
+
+/*
+identical, repeated occurances of these action types will be debounced.
+Entire action must be identical (all keys).
+*/
+const debounceOnActions = new Set([
+  "color by categorical metadata",
+  "color by continuous metadata",
+  "color by expression"
+]);
+
+/*
+history will be cleared when these actions occur
+*/
+const clearOnActions = new Set([
+  "initial data load complete (universe exists)",
+  "reset World to eq Universe",
+  "initial data load error",
+  "user reset end"
+]);
+
+/*
+An immediate history save will be done for these
+*/
+const saveOnActions = new Set([
+  "categorical metadata filter select",
+  "categorical metadata filter deselect",
+  "categorical metadata filter all of these",
+  "categorical metadata filter none of these",
+
+  "color by categorical metadata",
+  "color by continuous metadata",
+  "color by expression",
+
+  "set scatterplot x",
+  "set scatterplot y",
+
+  "store current cell selection as differential set 1",
+  "store current cell selection as differential set 2",
+
+  "set World to current selection",
+  "set clip quantiles",
+
+  "set layout choice"
+]);
+
+/**
+StateMachine - processing complex action handling - see FSM graph for
+actual structure, in undoableFsm.js
+**/
+
+/*
+Default FSM actions.  Used to side-effect transitions in the graph.
+See graph definition for the transitions that use each.
+
+Signature:  (fsm, transition, reducerState, reducerAction) => undoableAction
+*/
+const stashPending = fsm => ({
+  [actionKey]: "stashPending",
+  [stateKey]: { fsm }
+});
+const cancelPending = () => ({
+  [actionKey]: "cancelPending",
+  [stateKey]: { fsm: null }
+});
+const applyPending = () => ({
+  [actionKey]: "applyPending",
+  [stateKey]: { fsm: null }
+});
+const skip = (fsm, transition) => ({
+  [actionKey]: "skip",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
+const clear = () => ({ [actionKey]: "clear", [stateKey]: { fsm: null } });
+const save = (fsm, transition) => ({
+  [actionKey]: "save",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
+
+/*
+Error handler for state transitions that are unexpected.  Called by
+StateMachine when it doesn't know what to do.
+
+Signature:  (fsm, event, from) => undoableAction
+*/
+const onFsmError = (fsm, event, from) => {
+  console.error(`FSM error [event: "${event}", state: "${from}"]`, fsm);
+  // In production, try to recover gracefully if we have unexpected state
+  return clear(fsm);
+};
+
+/*
+Definition of the transition graph mapping action types to history side effects.
+*/
+const fsmTransitions = createFsmTransitions(
+  stashPending,
+  cancelPending,
+  applyPending,
+  skip,
+  clear,
+  save
+);
+/* State machine we clone whenever we need to run it */
+const seedFsm = new StateMachine("init", fsmTransitions, onFsmError);
+
+/*
+See undoable.js for description action filter interface description.
+
+Basic approach:
+  * trivial handlers for skip, clear & save cases to keep config simple.
+  * only implement complex state machines where absolutely required (eg,
+    multi-event seleciton and the like)
+*/
+const actionFilter = debug => (state, action, prevFilterState) => {
+  const actionType = action.type;
+  const filterState = {
+    ...prevFilterState,
+    prevAction: action
+  };
+  if (skipOnActions.has(actionType)) {
+    return { [actionKey]: "skip", [stateKey]: filterState };
+  }
+  if (
+    debounceOnActions.has(actionType) &&
+    shallowObjectEq(action, prevFilterState.prevAction)
+  ) {
+    return { [actionKey]: "skip", [stateKey]: filterState };
+  }
+  if (clearOnActions.has(actionType)) {
+    return { [actionKey]: "clear", [stateKey]: filterState };
+  }
+  if (saveOnActions.has(actionType)) {
+    return { [actionKey]: "save", [stateKey]: filterState };
+  }
+
+  /*
+    Else, something more complex OR unknown to us....
+    */
+  if (seedFsm.events.has(actionType)) {
+    let { fsm } = filterState;
+    if (!fsm) {
+      /* no active FSM, so create one in init state */
+      fsm = seedFsm.clone("init");
+    }
+    return fsm.next(action.type, { state, action });
+  }
+
+  /* else, we have no idea what this is - skip it */
+  if (debug) console.log("**** ACTION FILTER EVENT HANDLER MISS", actionType);
+  return { [actionKey]: "skip", [stateKey]: filterState };
+};
+
+/*
+return true if objA and objB are ===, OR if:
+  - are both objects and not null
+  - have same own properties
+  - all values are strict equal (===)
+*/
+function shallowObjectEq(objA, objB) {
+  if (objA === objB) return true;
+  if (!objA || !objB) return false;
+  if (!shallowArrayEq(Object.keys(objA), Object.keys(objB))) return false;
+  if (!shallowArrayEq(Object.values(objA), Object.values(objB))) return false;
+  return true;
+}
+
+/*
+return true if arrA and arrB contain the same strict-equal values,
+in the same order.
+*/
+function shallowArrayEq(arrA, arrB) {
+  if (arrA.length !== arrB.length) return false;
+  for (let i = 0, l = arrA.length; i < l; i += 1) {
+    if (arrA[i] !== arrB[i]) return false;
+  }
+  return true;
+}
+
+/* configuration for the undoable meta reducer */
+const debug = false; // set truish for undoble debugging
+const undoableConfig = {
+  debug,
+  historyLimit: 50, // maximum history size
+  actionFilter: actionFilter(debug)
+};
+
+/*
+this code is strictly for sanity checking configuration, and is only
+enabled when we are debugging the undoable configuration (ie, debug === true).
+*/
+if (debug) {
+  /*
+  Confirm no intersection between the various trivial rejection action filters
+  */
+  if (
+    new Set([...skipOnActions].filter(x => clearOnActions.has(x))).size > 0 ||
+    new Set([...skipOnActions].filter(x => saveOnActions.has(x))).size > 0 ||
+    new Set([...clearOnActions].filter(x => saveOnActions.has(x))).size > 0
+  ) {
+    console.error(
+      "Undoable misconfiguration - action filters have redundant events"
+    );
+  }
+
+  /*
+  Confirm that no FSM events are blocked by a trivial rejection filter.
+  If this occurs, the FSM can't ever see the events needed to process
+  state transitions.
+  */
+  const trivialFilters = new Set([
+    ...skipOnActions,
+    ...clearOnActions,
+    ...saveOnActions
+  ]);
+  const trivialOverlapWithFsm = new Set(
+    [...trivialFilters].filter(x => seedFsm.events.has(x))
+  );
+  if (trivialOverlapWithFsm.size > 0) {
+    console.error(
+      "Undoable misconfiguration - trivival action filter blocking FSM filter",
+      [...trivialOverlapWithFsm]
+    );
+  }
+}
+
+export default undoableConfig;

client/src/reducers/undoableFsm.js

@@ -0,0 +1,206 @@
+/*
+State transition graph for complex action/history interactions.
+
+Assumed configuration from undoableConfig:
+    * By convention, "init" is used as the start state for all, and "done"
+      as the final state.
+    * Unexpected states will result in an error, plus a clear and cancelPending
+      side-effect.
+
+TODO: is is possible there is a more concise format for this, as it is
+a fairly repetitive pattern.
+
+These events are largely one of two types:
+a) async operations or multi-event options that should only be committed
+upon some success criteria, otherwise cancelled.
+
+b) compound actions that should be collapsed into a single history change.
+
+*/
+
+const createFsmTransitions = (
+  stashPending,
+  cancelPending,
+  applyPending,
+  skip,
+  clear,
+  save
+) => {
+  return [
+    /* graph selection brushing */
+    {
+      event: "graph brush start",
+      from: "init",
+      to: "graph brush in progress",
+      action: stashPending
+    },
+    {
+      event: "graph brush cancel",
+      from: "graph brush in progress",
+      to: "done",
+      action: applyPending
+    },
+    {
+      event: "graph brush deselect",
+      from: "graph brush in progress",
+      to: "done",
+      /* if current selection is all, cancelPending.  Else, applyPending */
+      action: (fsm, transition, data) =>
+        data.state.graphSelection.selection.mode === "all"
+          ? cancelPending()
+          : applyPending()
+    },
+    {
+      event: "graph brush end",
+      from: "graph brush in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* graph selection lasso */
+    {
+      event: "graph lasso start",
+      from: "init",
+      to: "graph lasso in progress",
+      action: stashPending
+    },
+    {
+      event: "graph lasso cancel",
+      from: "graph lasso in progress",
+      to: "done",
+      action: applyPending
+    },
+    {
+      event: "graph lasso deselect",
+      from: "graph lasso in progress",
+      to: "done",
+      /* if current selection is all, cancelPending.  Else, applyPending */
+      action: (fsm, transition, data) =>
+        data.state.graphSelection.selection.mode === "all"
+          ? cancelPending()
+          : applyPending()
+    },
+    {
+      event: "graph lasso end",
+      from: "graph lasso in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Continuous metadata histogram brush selection */
+    {
+      event: "continuous metadata histogram start",
+      from: "init",
+      to: "continuous histo select in progress",
+      action: stashPending
+    },
+    {
+      event: "continuous metadata histogram cancel",
+      from: "continuous histo select in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "continuous metadata histogram end",
+      from: "continuous histo select in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Single gene request by user */
+    {
+      event: "single user defined gene start",
+      from: "init",
+      to: "single user gene request in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "single user gene request in progress",
+      to: "single user gene error in progress",
+      action: skip
+    },
+    {
+      event: "single user defined gene error",
+      from: "single user gene error in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "single user defined gene complete",
+      from: "single user gene request in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Bulk gene request by user */
+    {
+      event: "bulk user defined gene start",
+      from: "init",
+      to: "bulk user gene request in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "bulk user gene request in progress",
+      to: "bulk user gene request error in progress",
+      action: skip
+    },
+    {
+      event: "bulk user defined gene error",
+      from: "bulk user gene request error in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "bulk user defined gene complete",
+      from: "bulk user gene request in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Compute Differential Expression button user action */
+    {
+      event: "request differential expression started",
+      from: "init",
+      to: "diffexp in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "diffexp in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "request differential expression success",
+      from: "diffexp in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Clear Differential Expression button user action */
+    {
+      event: "clear differential expression",
+      from: "init",
+      to: "CDE Button in progress",
+      action: stashPending
+    },
+    {
+      event: "clear scatterplot",
+      from: "CDE Button in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* clear scatter plot button (eg, on scatterplot view) */
+    {
+      event: "clear scatterplot",
+      from: "init",
+      to: "done",
+      action: save
+    }
+  ];
+};
+
+export default createFsmTransitions;

client/src/reducers/universe.js

@@ -1,5 +1,3 @@
-import _ from "lodash";
-
 import { ControlsHelpers } from "../util/stateManager";
 
 const Universe = (state = null, action, nextSharedState, prevSharedState) => {
@@ -12,9 +10,10 @@ const Universe = (state = null, action, nextSharedState, prevSharedState) => {
     case "expression load success": {
       let { varData } = state;
 
-      // Load new expression data into the varData dataframes, if
+      // Lazy load new expression data into the varData dataframe, if
       // not already present.
-      _.forEach(action.expressionData, (val, key) => {
+      //
+      Object.entries(action.expressionData).forEach(([key, val]) => {
         // If not already in universe.varData, save entire expression column
         if (!varData.hasCol(key)) {
           varData = varData.withCol(key, val);
@@ -22,14 +21,15 @@ const Universe = (state = null, action, nextSharedState, prevSharedState) => {
       });
 
       // Prune size of varData "cache" if getting out of hand....
+      //
       const { userDefinedGenes, diffexpGenes } = prevSharedState;
-      const allTheGenesWeNeed = _.uniq(
-        [].concat(
+      const allTheGenesWeNeed = [
+        ...new Set(
           userDefinedGenes,
           diffexpGenes,
           Object.keys(action.expressionData)
         )
-      );
+      ];
       varData = ControlsHelpers.pruneVarDataCache(varData, allTheGenesWeNeed);
 
       return {

client/src/reducers/world.js

@@ -1,6 +1,6 @@
-import _ from "lodash";
-
 import { World, ControlsHelpers } from "../util/stateManager";
+import clip from "../util/clip";
+import quantile from "../util/quantile";
 
 const WorldReducer = (
   state = null,
@@ -21,7 +21,7 @@ const WorldReducer = (
 
     case "set World to current selection": {
       /* Set viewable world to be the currently selected data */
-      const world = World.createWorldFromCurrentSelection(
+      const world = World.createWorldBySelection(
         action.universe,
         action.world,
         action.crossfilter
@@ -29,16 +29,27 @@ const WorldReducer = (
       return world;
     }
 
+    case "set clip quantiles": {
+      const world = World.createWorldWithNewClip(
+        prevSharedState.universe,
+        state,
+        prevSharedState.crossfilter,
+        action.clipQuantiles
+      );
+      return world;
+    }
+
     case "expression load success": {
       const { universe } = nextSharedState;
       const universeVarData = universe.varData;
-      let worldVarData = state.varData;
+      let unclippedVarData = state.unclipped.varData;
 
-      // Load new expression data into the varData dataframes, if
+      // Lazy load new expression data into the unclipped varData dataframe, if
       // not already present.
-      _.forEach(action.expressionData, (val, key) => {
+      //
+      Object.entries(action.expressionData).forEach(([key, val]) => {
         // If not already in world.varData, save sliced expression column
-        if (!worldVarData.hasCol(key)) {
+        if (!unclippedVarData.hasCol(key)) {
           // Slice if world !== universe, else just use whole column.
           // Use the obsAnnotation index as the cut key, as we keep
           // all world dataframes in sync.
@@ -51,7 +62,7 @@ const WorldReducer = (
           }
 
           // Now build world's varData dataframe
-          worldVarData = worldVarData.withCol(
+          unclippedVarData = unclippedVarData.withCol(
             key,
             worldValSlice,
             state.obsAnnotations.rowIndex
@@ -59,23 +70,55 @@ const WorldReducer = (
         }
       });
 
-      // Prune size of varData "cache" if getting out of hand....
+      // Prune size of varData unclipped dataframe if getting out of hand....
+      //
       const { userDefinedGenes, diffexpGenes } = prevSharedState;
-      const allTheGenesWeNeed = _.uniq(
-        [].concat(
+      const allTheGenesWeNeed = [
+        ...new Set(
           userDefinedGenes,
           diffexpGenes,
           Object.keys(action.expressionData)
         )
-      );
-      worldVarData = ControlsHelpers.pruneVarDataCache(
-        worldVarData,
+      ];
+      unclippedVarData = ControlsHelpers.pruneVarDataCache(
+        unclippedVarData,
         allTheGenesWeNeed
       );
 
+      // at this point, we have the unclipped data in unclippedVarData.
+      // Now create clipped.
+      //   - Drop columns no longer needed
+      //   - Add new columns
+      //
+      let clippedVarData = state.varData;
+      const keysToDrop = clippedVarData.colIndex
+        .keys()
+        .filter(k => !unclippedVarData.hasCol(k));
+      const keysToAdd = unclippedVarData.colIndex
+        .keys()
+        .filter(k => !clippedVarData.hasCol(k));
+      keysToDrop.forEach(k => {
+        clippedVarData = clippedVarData.dropCol(k);
+      });
+      keysToAdd.forEach(k => {
+        const data = unclippedVarData.col(k).asArray();
+        const q = [state.clipQuantiles.min, state.clipQuantiles.max];
+        const [qMinVal, qMaxVal] = quantile(q, data);
+        const clippedData = clip(data, qMinVal, qMaxVal, Number.NaN);
+        clippedVarData = clippedVarData.withCol(
+          k,
+          clippedData,
+          state.obsAnnotations.rowIndex
+        );
+      });
+
       return {
         ...state,
-        varData: worldVarData
+        varData: clippedVarData,
+        unclipped: {
+          ...state.unclipped,
+          varData: unclippedVarData
+        }
       };
     }
 

client/src/util/camera.js

@@ -5,10 +5,9 @@ const mp = require("mouse-position");
 const mb = require("mouse-pressed");
 const key = require("key-pressed");
 
-const panSpeed = 0.4;
+const panSpeed = 1.0; // changed from 0.4 to 1.0 per issue #722
 const scaleSpeed = 0.5;
 const scaleMax = 3;
-// const scaleMin = 1.15
 const scaleMin = 1.03;
 
 function attachCamera(canvas, opts) {

client/src/util/clip.js

@@ -0,0 +1,25 @@
+/*
+clip - clip all values in a Array or TypedArray, IN PLACE.
+
+Values in array are clipped if less than `lower` or greater than `upper`.
+
+If `setTo` is undefined, values less than `lower` will be set to `lower`,
+and values greater than `upper` will be set to `upper`.  
+
+If `setTo` is not undefined, values outside the [lower, upper] range will be set to 
+`setTo`.
+
+*/
+export default function clip(arr, lower, upper, setTo) {
+	const lowerSet = setTo === undefined ? lower : setTo;
+	const upperSet = setTo === undefined ? upper : setTo;
+	for (let i = 0, l = arr.length; i < l; i += 1) {
+		const v = arr[i];
+		if (v < lower) {
+			arr[i] = lowerSet;
+		} else if (v > upper) {
+			arr[i] = upperSet;
+		}
+	}
+	return arr;
+}

client/src/util/dataframe/dataframe.js

@@ -1,6 +1,6 @@
 import { IdentityInt32Index, isLabelIndex } from "./labelIndex";
 // weird cross-dependency that we should clean up someday...
-import { sort } from "../typedCrossfilter/sort";
+import { sortArray } from "../typedCrossfilter/sort";
 import { isTypedArray, isArrayOrTypedArray, callOnceLazy } from "./util";
 import { summarizeContinuous, summarizeCategorical } from "./summarize";
 
@@ -63,7 +63,13 @@ class Dataframe {
   Constructors & factories
   **/
 
-  constructor(dims, columnarData, rowIndex = null, colIndex = null) {
+  constructor(
+    dims,
+    columnarData,
+    rowIndex = null,
+    colIndex = null,
+    __columnsAccessor = [] // private interface
+  ) {
     /*
     The base constructor is relatively hard to use - as an alternative,
     see factory methods and clone/slice, below.
@@ -74,6 +80,9 @@ class Dataframe {
         or TypedArray of length nRows.
       * rowIndex/colIndex - null (create default index using offsets as key),
         or a caller-provided index.
+      * __columnsAccessor - private interface, do not specify.  Used internally
+        to improve caching of column accessors when possible (eg, clone(), 
+        dropCol(), withCol()).
     All columns and indices must have appropriate dimensionality.
     */
     const [nRows, nCols] = dims;
@@ -94,7 +103,7 @@ class Dataframe {
     this.rowIndex = rowIndex;
     this.colIndex = colIndex;
 
-    this.__compile();
+    this.__compile(__columnsAccessor);
   }
 
   static __errorChecks(dims, columnarData, rowIndex, colIndex) {
@@ -135,97 +144,107 @@ class Dataframe {
     }
   }
 
-  __compile() {
+  static __compileColumn(column, getOffset, getLabel) {
+    /*
+      Each column accessor is a function which will lookup data by
+      index (ie, is equivalent to dataframe.get(row, col), where 'col'
+      is fixed.
+
+      In addition, each column accessor has several functions:
+
+      asArray() -- return the entire column as a native Array or TypedArray.
+        Crucially, this native array only supports label indexing.
+        Example:
+          const arr = df.col('a').asArray();
+
+      has(rlabel) -- return boolean indicating of the row label
+        is contained within the column.  Example:
+          const isInColumn = df.col('a').includes(99)
+        For the default offset indexing, this is identical to:
+          const isInColumn = (99 > 0) && (99 < df.nRows);
+
+      ihas(roffset) -- same as has(), but accepts a row offset
+        instead of a row label.
+
+      indexOf(value) -- return the label (not offset) of the first instance of
+        'value' in the column.  If you want the offset, just use the builtin JS
+        indexOf() function, available on both Array and TypedArray.
+
+      iget(offset) -- return the value at 'offset'
+
+    */
+    const { length } = column;
+
+    /* get value by row label */
+    const get = function get(rlabel) {
+      return column[getOffset(rlabel)];
+    };
+
+    /* get value by row offset */
+    const iget = function iget(roffset) {
+      return column[roffset];
+    };
+
+    /* full column array access */
+    const asArray = function asArray() {
+      return column;
+    };
+
+    /* test for row label inclusion in column */
+    const has = function has(rlabel) {
+      const offset = getOffset(rlabel);
+      return offset >= 0 && offset < length;
+    };
+
+    const ihas = function ihas(offset) {
+      return offset >= 0 && offset < length;
+    };
+
+    /*
+    return first label (index) at which the value is found in this column,
+    or undefined if not found.
+
+    NOTE: not found return is DIFFERENT than the default Array.indexOf as
+    -1 is a plausible Dataframe row/col label.
+    */
+    const indexOf = function indexOf(value) {
+      const offset = column.indexOf(value);
+      if (offset === -1) {
+        return undefined;
+      }
+      return getLabel(offset);
+    };
+
+    /*
+    Summarize the column data. Lazy eval;
+    */
+    const summarize = callOnceLazy(() =>
+      isTypedArray(column)
+        ? summarizeContinuous(column)
+        : summarizeCategorical(column)
+    );
+
+    get.summarize = summarize;
+    get.asArray = asArray;
+    get.has = has;
+    get.ihas = ihas;
+    get.indexOf = indexOf;
+    get.iget = iget;
+    return get;
+  }
+
+  __compile(accessors) {
     /*
     Compile data accessors for each column.
 
-    Each column accessor is a function which will lookup data by
-    index (ie, is equivalent to dataframe.get(row, col), where 'col'
-    is fixed.
-
-    In addition, each column accessor has several functions:
-
-    asArray() -- return the entire column as a native Array or TypedArray.
-      Crucially, this native array only supports label indexing.
-      Example:
-        const arr = df.col('a').asArray();
-
-    has(rlabel) -- return boolean indicating of the row label
-      is contained within the column.  Example:
-        const isInColumn = df.col('a').includes(99)
-      For the default offset indexing, this is identical to:
-        const isInColumn = (99 > 0) && (99 < df.nRows);
-
-    ihas(roffset) -- same as has(), but accepts a row offset
-      instead of a row label.
-
-    indexOf(value) -- return the label (not offset) of the first instance of
-      'value' in the column.  If you want the offset, just use the builtin JS
-      indexOf() function, available on both Array and TypedArray.
-
-    iget(offset) -- return the value at 'offset'
-
+    Use an existing accessor if provided, else compile a new one.
     */
     const { getOffset, getLabel } = this.rowIndex;
-    this.__columnsAccessor = this.__columns.map(column => {
-      const { length } = column;
-
-      /* get value by row label */
-      const get = function get(rlabel) {
-        return column[getOffset(rlabel)];
-      };
-
-      /* get value by row offset */
-      const iget = function iget(roffset) {
-        return column[roffset];
-      };
-
-      /* full column array access */
-      const asArray = function asArray() {
-        return column;
-      };
-
-      /* test for row label inclusion in column */
-      const has = function has(rlabel) {
-        const offset = getOffset(rlabel);
-        return offset >= 0 && offset < length;
-      };
-
-      const ihas = function ihas(offset) {
-        return offset >= 0 && offset < length;
-      };
-
-      /*
-      return first label (index) at which the value is found in this column,
-      or undefined if not found.
-
-      NOTE: not found return is DIFFERENT than the default Array.indexOf as
-      -1 is a plausible Dataframe row/col label.
-      */
-      const indexOf = function indexOf(value) {
-        const offset = column.indexOf(value);
-        if (offset === -1) {
-          return undefined;
-        }
-        return getLabel(offset);
-      };
-
-      /*
-      Summarize the column data. Lazy eval;
-      */
-      const summarize = callOnceLazy(() =>
-        isTypedArray(column)
-          ? summarizeContinuous(column)
-          : summarizeCategorical(column)
-      );
-
-      get.summarize = summarize;
-      get.asArray = asArray;
-      get.has = has;
-      get.ihas = ihas;
-      get.indexOf = indexOf;
-      get.iget = iget;
-      return get;
+    this.__columnsAccessor = this.__columns.map((column, idx) => {
+      if (accessors[idx]) {
+        return accessors[idx];
+      }
+      return Dataframe.__compileColumn(column, getOffset, getLabel);
     });
   }
 
@@ -237,7 +256,8 @@ class Dataframe {
       this.dims,
       [...this.__columns],
       this.rowIndex,
-      this.colIndex
+      this.colIndex,
+      [...this.__columnsAccessor]
     );
   }
 
@@ -273,7 +293,14 @@ class Dataframe {
     const columns = [...this.__columns];
     columns.push(colData);
     const colIndex = this.colIndex.withLabel(label);
-    return new this.constructor(dims, columns, rowIndex, colIndex);
+    const columnsAccessor = [...this.__columnsAccessor];
+    return new this.constructor(
+      dims,
+      columns,
+      rowIndex,
+      colIndex,
+      columnsAccessor
+    );
   }
 
   dropCol(label) {
@@ -287,7 +314,15 @@ class Dataframe {
     const columns = [...this.__columns];
     columns.splice(coffset, 1);
     const colIndex = this.colIndex.dropLabel(label);
-    return new this.constructor(dims, columns, this.rowIndex, colIndex);
+    const columnsAccessor = [...this.__columnsAccessor];
+    columnsAccessor.splice(coffset, 1);
+    return new this.constructor(
+      dims,
+      columns,
+      this.rowIndex,
+      colIndex,
+      columnsAccessor
+    );
   }
 
   static empty(rowIndex = null, colIndex = null) {
@@ -316,7 +351,7 @@ class Dataframe {
       if (!offsets) {
         return [null, null];
       }
-      const sortedOffsets = sort(offsets);
+      const sortedOffsets = sortArray(offsets);
       const sortedLabels = new Array(sortedOffsets.length);
       for (let i = 0, l = sortedOffsets.length; i < l; i += 1) {
         sortedLabels[i] = index.getLabel(sortedOffsets[i]);
@@ -543,14 +578,34 @@ class Dataframe {
   /****
   Functional (map/reduce/etc) data access
 
-  XXX: not yet implemented, as there is no clear use case.   Can easily
+  TODO: most are not yet implemented, as there is no clear use case.   Can easily
   add these as useful.
   ****/
 
+  mapColumns(callback) {
+    /*
+    map all columns in the dataframe, returning a new dataframe comprised of the
+    return values, with the same index as the original dataframe.
+
+    callback MUST not modify the column, but instead return a mutated copy.
+    */
+    const columns = this.__columns.map(callback);
+    const columnsAccessor = columns.map((c, idx) =>
+      this.__columns[idx] === c ? this.__columnsAccessor[idx] : undefined
+    );
+    return new this.constructor(
+      this.dims,
+      columns,
+      this.rowIndex,
+      this.colIndex,
+      columnsAccessor
+    );
+  }
+
   /*
   Map & reduce of column or row
 
-  XXX TODO remainder of map/reduce functions:  mapCol, mapRow, reduceRow, ...
+  TODO remainder of map/reduce functions:  mapCol, mapRow, reduceRow, ...
   */
   /* comment out until we have a use for this
 

client/src/util/dataframe/labelIndex.js

@@ -3,6 +3,8 @@ Label indexing - map a label to & from an integer offset.  See Dataframe
 for how this is used.
 **/
 
+import { rangeFill as fillRange } from "../range";
+
 /*
 Private utility functions
 */
@@ -21,14 +23,6 @@ function extent(tarr) {
   return [min, max];
 }
 
-function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, l = larr.length; i < l; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 /* eslint-disable class-methods-use-this */
 class IdentityInt32Index {
   /*

client/src/util/dataframe/summarize.js

@@ -1,32 +1,56 @@
 /*
 Private dataframe support functions
+
+TODO / XXX: for scalar/continuous data, this uses a naive method
+of computing quantiles.  Would be good to switch from sort to 
+partition at some point.
 */
 
+import quantile from "../quantile";
+import { sortArray } from "../typedCrossfilter/sort";
+
+// [ 0, 0.01, 0.02, ..., 1.0]
+const centileNames = new Array(101).fill(0).map((v, idx) => idx / 100);
+
 export function summarizeContinuous(col) {
   let min;
   let max;
   let nan = 0;
   let pinf = 0;
   let ninf = 0;
+  let percentiles;
   if (col) {
-    for (let r = 0, l = col.length; r < l; r += 1) {
-      const val = Number(col[r]);
-      if (Number.isFinite(val)) {
-        if (min === undefined) {
-          min = val;
-          max = val;
-        } else {
-          min = val < min ? val : min;
-          max = val > max ? val : max;
-        }
-      } else if (Number.isNaN(val)) {
-        nan += 1;
-      } else if (val > 0) {
-        pinf += 1;
-      } else {
-        ninf += 1;
+    // -Inf < finite < Inf < NaN
+    const sortedCol = sortArray(new col.constructor(col));
+
+    // count non-finites, which are at each end of sorted data
+    for (let i = sortedCol.length - 1; i >= 0; i -= 1) {
+      if (!Number.isNaN(sortedCol[i])) {
+        nan = sortedCol.length - i - 1;
+        break;
       }
     }
+    for (let i = 0, l = sortedCol.length; i < l; i += 1) {
+      if (sortedCol[i] !== Number.NEGATIVE_INFINITY) {
+        ninf = i;
+        break;
+      }
+    }
+    for (let i = sortedCol.length - nan - 1; i >= 0; i -= 1) {
+      if (sortedCol[i] !== Number.POSITIVE_INFINITY) {
+        pinf = sortedCol.length - i - nan - 1;
+        break;
+      }
+    }
+
+    // compute percentiles on finite data ONLY
+    const sortedColFiniteOnly = sortedCol.slice(
+      ninf,
+      sortedCol.length - nan - pinf
+    );
+    percentiles = quantile(centileNames, sortedColFiniteOnly, true);
+    min = percentiles[0];
+    max = percentiles[100];
   }
   return {
     categorical: false,
@@ -34,7 +58,8 @@ export function summarizeContinuous(col) {
     max,
     nan,
     pinf,
-    ninf
+    ninf,
+    percentiles
   };
 }
 

client/src/util/dataframe/util.js

@@ -2,16 +2,7 @@
 Private utility code for dataframe
 */
 
-export function isTypedArray(x) {
-  return (
-    ArrayBuffer.isView(x) &&
-    Object.prototype.toString.call(x) !== "[object DataView]"
-  );
-}
-
-export function isArrayOrTypedArray(x) {
-  return Array.isArray(x) || isTypedArray(x);
-}
+export { isTypedArray, isArrayOrTypedArray } from "../typeHelpers";
 
 export function callOnceLazy(f) {
   let value;

client/src/util/fromEntries.js

@@ -0,0 +1,13 @@
+export default function fromEntries(arr) {
+	/*
+	Similar to Object.fromEntries, but only handles array.
+	This could be replaced with the standard fucnction once it
+	is widely available.   As of 3/20/2019, it has not yet
+	been released in the Chrome stable channel.
+	*/
+	const obj = {};
+	for (let i = 0, l = arr.length; i < l; i += 1) {
+		obj[arr[i][0]] = arr[i][1];
+	}
+	return obj;
+}

client/src/util/quantile.js

@@ -0,0 +1,29 @@
+/*
+quantiles - calculate quantiles for the typed array.
+
+Currently interpolates to 'lower' value.
+
+Arguments:
+
+	* quantArr - array of quantiles to compute, where values:  0 <= value <= 1.0
+	* tarr - a typed array
+	* sorted - option bool.  If false (default), will assume array is not sorted.
+	 	If true, will assume it is sorted.
+
+*/
+
+import { sortArray } from "./typedCrossfilter/sort";
+
+export default function quantile(quantArr, tarr, sorted = false) {
+	/*
+	start with the naive (sort) implementation.  Later, use a faster partition
+	*/
+	const arr = sorted ? tarr : sortArray(new tarr.constructor(tarr)); // copy
+	const len = arr.length;
+	return quantArr.map(q => {
+		if (q === 1) {
+			return arr[len - 1];
+		}
+		return arr[Math.floor(q * len)];
+	});
+}

client/src/util/range.js

@@ -0,0 +1,45 @@
+/*
+Array range creation
+
+range(start, stop, step) -> Array
+	This is identical to https://docs.python.org/3/library/functions.html#func-range
+	Returns new array filled with a range of numbers.
+
+	Usage:
+
+		range(stop) - start defaults to zero, step defaults to 1
+		range(start, stop, [step]) - step defaults to 1
+
+	Examples:
+		range(3) -> [0, 1, 2]
+		range(1, 3) -> [1, 2]
+		range(1, 5, 2) -> [1, 3]
+
+
+rangeFill(array, start, step) -> array
+	Fill entire array with values, from start, by step.  Returns first array.
+	start defaults to zero, step defaults to one.
+
+*/
+
+function _doFill(arr, start, step, count) {
+	for (let idx = 0, val = start; idx < count; idx += 1, val += step) {
+		arr[idx] = val;
+	}
+	return arr;
+}
+
+export function rangeFill(arr, start = 0, step = 1) {
+	return _doFill(arr, start, step, arr.length);
+}
+
+export function range(start, stop, step) {
+	if (start === undefined) return [];
+	if (stop === undefined) {
+		stop = start;
+		start = 0;
+	}
+	step = step || 1; // catch undefind and zero
+	const len = Math.max(Math.ceil((stop - start) / step), 0);
+	return _doFill(new Array(len), start, step, len);
+}

client/src/util/scaleLinear.js

@@ -9,8 +9,14 @@
 // this is is equivalent to d3.scaleLinear().domain([0,1]).range([-1,1])
 
 export default (domain, range) => {
-  const domainStart = domain[0];
-  const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
-  const rangeStart = range[0];
-  return value => (value - domainStart) * scale + rangeStart;
+	const domainStart = domain[0];
+	const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+	const invScale = 1 / scale;
+	const rangeStart = range[0];
+	const f = value => (value - domainStart) * scale + rangeStart;
+
+	// inverter
+	f.invert = value => (value - rangeStart) * invScale + domainStart;
+
+	return f;
 };

client/src/util/stateManager/colorHelpers.js

@@ -1,12 +1,12 @@
 /*
 Helper functions for the embedded graph colors
 */
-import _ from "lodash";
 import * as d3 from "d3";
 import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
 import * as globals from "../../globals";
 import parseRGB from "../parseRGB";
 import finiteExtent from "../finiteExtent";
+import { range } from "../range";
 
 /*
 create new colors state object.   Paramters:
@@ -37,9 +37,7 @@ function createColors(world, colorMode = null, colorAccessor = null) {
 }
 
 function createColorsByCategoricalMetadata(world, accessor) {
-  const { categories } = _.filter(world.schema.annotations.obs, {
-    name: accessor
-  })[0];
+  const { categories } = world.schema.annotations.obsByName[accessor];
 
   const scale = d3
     .scaleSequential(interpolateRainbow)
@@ -67,7 +65,7 @@ function createColorsByContinuousMetadata(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);
@@ -97,7 +95,7 @@ function createColorsByExpression(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);

client/src/util/stateManager/controlsHelpers.js

@@ -5,7 +5,7 @@ Helper functions for the controls reducer
 import _ from "lodash";
 
 import * as globals from "../../globals";
-import { fillRange } from "../typedCrossfilter/util";
+import { rangeFill as fillRange } from "../range";
 import {
   userDefinedDimensionName,
   diffexpDimensionName
@@ -21,16 +21,16 @@ Remember that option values can be ANY js type, except undefined/null.
   {
     _category_name_1: {
       // map of option value to index
-      categoryIndices: Map([
+      categoryValueIndices: Map([
         catval1: index,
         ...
       ])
 
       // index->selection true/false state
-      categorySelected: [ true/false, true/false, ... ]
+      categoryValueSelected: [ true/false, true/false, ... ]
 
       // number of options
-      numCategories: number,
+      numCategoryValues: number,
 
       // isTruncated - true if the options for selection has
       // been truncated (ie, was too large to implement)
@@ -56,27 +56,31 @@ function topNCategories(summary) {
 
 export function createCategoricalSelection(maxCategoryItems, world) {
   const res = {};
+  const obsIndexName = world.schema.annotations.obs.index;
   _.forEach(world.obsAnnotations.colIndex.keys(), key => {
     const summary = world.obsAnnotations.col(key).summarize();
     if (summary.categories) {
       const isColorField = key.includes("color") || key.includes("Color");
       const isSelectableCategory =
         !isColorField &&
-        key !== "name" &&
+        key !== obsIndexName &&
         summary.categories.length < maxCategoryItems;
       if (isSelectableCategory) {
-        const [categoryValues, categoryCounts] = topNCategories(summary);
-        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
-        const numCategories = categoryIndices.size;
-        const categorySelected = new Array(numCategories).fill(true);
+        const [categoryValues, categoryValueCounts] = topNCategories(summary);
+        const categoryValueIndices = new Map(
+          categoryValues.map((v, i) => [v, i])
+        );
+        const numCategoryValues = categoryValueIndices.size;
+        const categoryValueSelected = new Array(numCategoryValues).fill(true);
         const isTruncated = categoryValues.length < summary.numCategories;
         res[key] = {
           categoryValues, // array: of natively typed category values
-          categoryIndices, // map: category value (native type) -> category index
-          categorySelected, // array: t/f selection state
-          numCategories, // number: of categories
+          categoryValueIndices, // map: category value (native type) -> category index
+          categoryValueSelected, // array: t/f selection state
+          numCategoryValues, // number: of values in the category
           isTruncated, // bool: true if list was truncated
-          categoryCounts // array: cardinality of each category
+          categoryValueCounts, // array: cardinality of each category,
+          categorySelected: true // bool - default state for entire category
         };
       }
     }
@@ -88,12 +92,26 @@ export function createCategoricalSelection(maxCategoryItems, world) {
 given a categoricalSelection, return the list of all category values
 where selection state is true (ie, they are selected).
 */
-export function selectedValuesForCategory(categorySelectionState) {
-  const selectedValues = _([...categorySelectionState.categoryIndices])
-    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
-    .map(tuple => tuple[0])
-    .value();
-  return selectedValues;
+export function selectedValuesForCategory(categorySelectionState, dfColumn) {
+  const {
+    categorySelected,
+    categoryValueSelected,
+    categoryValueIndices
+  } = categorySelectionState;
+  let selectedValues;
+  if (categorySelected) {
+    selectedValues = new Set(dfColumn.summarize().categories);
+  } else {
+    selectedValues = new Set();
+  }
+  categoryValueIndices.forEach((catIndex, catValue) => {
+    if (!categoryValueSelected[catIndex]) {
+      selectedValues.delete(catValue);
+    } else {
+      selectedValues.add(catValue);
+    }
+  });
+  return [...selectedValues.values()];
 }
 
 /*

client/src/util/stateManager/universe.js

@@ -4,6 +4,8 @@ import _ from "lodash";
 
 import decodeMatrixFBS from "./matrix";
 import * as Dataframe from "../dataframe";
+import fromEntries from "../fromEntries";
+import { isFpTypedArray } from "../typeHelpers";
 
 /*
 Private helper function - create and return a template Universe
@@ -37,14 +39,58 @@ These functions are used exclusively by the actions and reducers to
 build an internal POJO for use by the rendering components.
 */
 
+function promoteTypedArray(o) {
+  /*
+  Decide what internal data type to use for the data returned from 
+  the server.
+
+  TODO - future optimization: not all int32/uint32 data series require
+  promotion to float64.  We COULD simply look at the data to decide. 
+  */
+  if (isFpTypedArray(o) || Array.isArray(o)) return o;
+
+  let TyepdArrayCtor;
+  switch (o.constructor) {
+    case Int8Array:
+    case Uint8Array:
+    case Uint8ClampedArray:
+    case Int16Array:
+    case Uint16Array:
+      TyepdArrayCtor = Float32Array;
+      break;
+
+    case Int32Array:
+    case Uint32Array:
+      TyepdArrayCtor = Float64Array;
+      break;
+
+    default:
+      throw new Error("Unexpected data type returned from server.");
+  }
+  if (o.constructor === TyepdArrayCtor) return o;
+  return new TyepdArrayCtor(o);
+}
+
 function AnnotationsFBSToDataframe(arrayBuffer) {
   /*
   Convert a Matrix FBS to a Dataframe.
+
+  The application has strong assumptions that all scalar data will be
+  stored as a float32 or float64 (regardless of underlying data types).
+  For example, clipping of value ranges (eg, user-selected percentiles)
+  depends on the ability to use NaN in any numeric type.
+
+  All float data from the server is left as is.  All non-float is promoted
+  to an appropriate float.
   */
-  const fbs = decodeMatrixFBS(arrayBuffer);
+  const fbs = decodeMatrixFBS(arrayBuffer, true); // leave in place
+  const columns = fbs.columns.map(c => {
+    if (isFpTypedArray(c) || Array.isArray(c)) return c;
+    return promoteTypedArray(c);
+  });
   const df = new Dataframe.Dataframe(
     [fbs.nRows, fbs.nCols],
-    fbs.columns,
+    columns,
     null,
     new Dataframe.KeyIndex(fbs.colIdx)
   );
@@ -53,11 +99,16 @@ function AnnotationsFBSToDataframe(arrayBuffer) {
 
 function LayoutFBSToDataframe(arrayBuffer) {
   const fbs = decodeMatrixFBS(arrayBuffer, true);
+  if (fbs.columns.length < 2 || !fbs.columns.every(isFpTypedArray)) {
+    // We have strong assumptions about the shape & type of layout data.
+    throw new Error("Unexpected layout data type returned from server");
+  }
+
   const df = new Dataframe.Dataframe(
     [fbs.nRows, fbs.nCols],
     fbs.columns,
     null,
-    new Dataframe.KeyIndex(["X", "Y"])
+    new Dataframe.KeyIndex(fbs.colIdx)
   );
   return df;
 }
@@ -73,15 +124,15 @@ function reconcileSchemaCategoriesWithSummary(universe) {
   cases, add a 'categories' field to the schema so it is accessible.
   */
 
-  _.forEach(universe.schema.annotations.obs, s => {
+  universe.schema.annotations.obs.columns.forEach(s => {
     if (
       s.type === "string" ||
       s.type === "boolean" ||
       s.type === "categorical"
     ) {
       const categories = _.union(
-        _.get(s, "categories", []),
-        _.get(universe.obsAnnotations.col(s.name).summarize(), "categories", [])
+        s.categories ?? [],
+        universe.obsAnnotations.col(s.name).summarize().categories ?? []
       );
       s.categories = categories;
     }
@@ -105,6 +156,9 @@ export function createUniverseFromResponse(
   universe.schema = schema;
   universe.nObs = schema.dataframe.nObs;
   universe.nVar = schema.dataframe.nVar;
+  /* add defaults, as we can't assume back-end will fully populate schema */
+  if (!schema.layout.var) schema.layout.var = [];
+  if (!schema.layout.obs) schema.layout.obs = [];
 
   /* annotations */
   universe.obsAnnotations = AnnotationsFBSToDataframe(annotationsObsResponse);
@@ -122,6 +176,20 @@ export function createUniverseFromResponse(
   }
 
   reconcileSchemaCategoriesWithSummary(universe);
+
+  /* Index schema for ease of use */
+  universe.schema.annotations.obsByName = fromEntries(
+    universe.schema.annotations.obs.columns.map(v => [v.name, v])
+  );
+  universe.schema.annotations.varByName = fromEntries(
+    universe.schema.annotations.var.columns.map(v => [v.name, v])
+  );
+  universe.schema.layout.obsByName = fromEntries(
+    universe.schema.layout.obs.map(v => [v.name, v])
+  );
+  universe.schema.layout.varByName = fromEntries(
+    universe.schema.layout.var.map(v => [v.name, v])
+  );
   return universe;
 }
 
@@ -140,8 +208,14 @@ export function convertDataFBStoObject(universe, arrayBuffer) {
   const { colIdx, columns } = fbs;
   const result = {};
 
+  if (!columns.every(isFpTypedArray)) {
+    // We have strong assumptions that all var data is float
+    throw new Error("Unexpected non-floating point response from server.");
+  }
+
+  const varIndexName = universe.schema.annotations.var.index;
   for (let c = 0; c < colIdx.length; c += 1) {
-    const varName = universe.varAnnotations.at(colIdx[c], "name");
+    const varName = universe.varAnnotations.at(colIdx[c], varIndexName);
     result[varName] = columns[c];
   }
   return result;

client/src/util/stateManager/world.js

@@ -1,7 +1,14 @@
 // jshint esversion: 6
 
-import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
+import clip from "../clip";
+import {
+  layoutDimensionName,
+  obsAnnoDimensionName,
+  diffexpDimensionName,
+  userDefinedDimensionName
+} from "../nameCreators";
 import * as Dataframe from "../dataframe";
+import ImmutableTypedCrossfilter from "../typedCrossfilter/crossfilter";
 
 /*
 
@@ -19,6 +26,8 @@ Notable keys in the world object:
 
 * schema: data schema from the server
 
+* clipQuantiles: the quantiles used to clip all data in world.
+
 * obsAnnotations:
 
   Dataframe containing obs annotations.  Columns are indexed by annotation
@@ -37,78 +46,192 @@ Notable keys in the world object:
 * varData: a cache of expression columns, stored in a Dataframe.  Cache
   managed by controls reducer.
 
+* unclipped: will contain unclipped variants of all potentiall clipped
+  dataframes (obsAnnotations, varData).
+
 */
 
 function templateWorld() {
+  const obsAnnotations = Dataframe.Dataframe.empty();
+  const varAnnotations = Dataframe.Dataframe.empty();
+  const obsLayout = Dataframe.Dataframe.empty();
+  const varData = Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex());
   return {
     /* schema/version related */
     schema: null,
     nObs: 0,
     nVar: 0,
+    clipQuantiles: { min: 0, max: 1 },
 
     /* annotations */
-    obsAnnotations: Dataframe.Dataframe.empty(),
-    varAnnotations: Dataframe.Dataframe.empty(),
+    obsAnnotations,
+    varAnnotations,
 
     /* layout of graph. Dataframe. */
-    obsLayout: Dataframe.Dataframe.empty(),
+    obsLayout,
 
-    /*
-    Var data columns - subset of all data (may be empty)
-    */
-    varData: Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex())
+    /* Var data columns - subset of all data (may be empty) */
+    varData,
+
+    /* unclipped dataframes - subset, but not value clipped */
+    unclipped: {
+      obsAnnotations,
+      varData
+    }
   };
 }
 
+function clipDataframe(
+  df,
+  lowerQuantile,
+  upperQuantile,
+  quantileF,
+  clipPredicate = () => true,
+  value = Number.NaN
+) {
+  /*
+  For all columns in the dataframe, clip all values above or below specified 
+  quantiles to `value` if clipPredicate returns True for that column (if it
+  returns false, skip the column entirely).
+
+  Returns a clipped copy - does not mutate original.
+
+  clipPredicate must have signature: (dataframe, colIndex, colLabel) => boolean
+  True signifies that the column should be clipped; false indicates that the
+  column should be left intact/unchanged.
+
+  quantileF must have signature:  (label, qval) => number
+  */
+  if (lowerQuantile < 0) lowerQuantile = 0;
+  if (upperQuantile > 1) upperQuantile = 1;
+  if (lowerQuantile === 0 && upperQuantile === 1) return df;
+
+  const keys = df.colIndex.keys();
+  return df.mapColumns((col, colIdx) => {
+    const colLabel = keys[colIdx];
+    if (!clipPredicate(df, colIdx, colLabel)) return col;
+
+    const colMin = quantileF(colLabel, lowerQuantile);
+    const colMax = quantileF(colLabel, upperQuantile);
+    const newCol = clip(col.slice(), colMin, colMax, value);
+    return newCol;
+  });
+}
+
+/*
+Create World with contents eq entire universe.   Commonly used to initialize World.
+If clipQuantiles
+*/
 export function createWorldFromEntireUniverse(universe) {
   const world = templateWorld();
 
-  /*
-  public interface follows
-  */
-
   /* Schema related */
   world.schema = universe.schema;
   world.nObs = universe.nObs;
   world.nVar = universe.nVar;
+  world.clipQuantiles = { min: 0, max: 1 };
 
-  /* annotation dataframes */
-  world.obsAnnotations = universe.obsAnnotations;
-  world.varAnnotations = universe.varAnnotations;
+  /* dataframes: annotations and layout */
+  world.obsAnnotations = universe.obsAnnotations.clone();
+  world.varAnnotations = universe.varAnnotations.clone();
+  world.obsLayout = universe.obsLayout.clone();
 
-  /* layout and display characteristics dataframe */
-  world.obsLayout = universe.obsLayout;
-
-  /*
-  Var data columns - subset of all
-  */
+  /* Var dataframe - contains a subset of all var columns */
   world.varData = universe.varData.clone();
 
+  /* save unclipped copies of potentially clipped dataframes */
+  world.unclipped = {
+    obsAnnotations: world.obsAnnotations.clone(),
+    varData: world.varData.clone()
+  };
+
   return world;
 }
 
-export function createWorldFromCurrentSelection(universe, world, crossfilter) {
-  const newWorld = templateWorld();
+/*
+clip dataframes based on quantiles.
 
-  /* these don't change as only OBS are selected in our current implementation */
-  newWorld.nVar = universe.nVar;
-  newWorld.schema = universe.schema;
-  newWorld.varAnnotations = universe.varAnnotations;
+This is an in-place operation on the world object provided as an argument.
+The values in world.unclipped are clipped and assigned to world.obsAnnotations
+and world.varData.
+*/
+function setClippedDataframes(world) {
+  const { schema } = world;
+  const isContinuousObsAnnotation = (df, idx, label) =>
+    deduceDimensionType(schema.annotations.obsByName[label], label) !== "enum";
+  const obsQuantile = (label, q) =>
+    world.unclipped.obsAnnotations.col(label).summarize().percentiles[100 * q];
+  world.obsAnnotations = clipDataframe(
+    world.unclipped.obsAnnotations,
+    world.clipQuantiles.min,
+    world.clipQuantiles.max,
+    obsQuantile,
+    isContinuousObsAnnotation
+  );
 
-  /* now subset/cut obs */
+  const varDataQuantile = (label, q) =>
+    world.unclipped.varData.col(label).summarize().percentiles[100 * q];
+  world.varData = clipDataframe(
+    world.unclipped.varData,
+    world.clipQuantiles.min,
+    world.clipQuantiles.max,
+    varDataQuantile,
+    () => true
+  );
+}
+
+/*
+Subset the current world based upon the current selection, maintaining any existing
+clip.  Returns new world.  Parameters:
+  * unvierse
+  * world - the current world
+  * crossfilter - the selection state
+*/
+export function createWorldBySelection(universe, world, crossfilter) {
+  const newWorld = { ...world, obsLayout: null, unclipped: {}, varData: null };
+
+  /* subset unclipped dataframes based upon current selection */
   const mask = crossfilter.allSelectedMask();
-  newWorld.obsAnnotations = world.obsAnnotations.isubsetMask(mask);
   newWorld.obsLayout = world.obsLayout.isubsetMask(mask);
-  newWorld.nObs = newWorld.obsAnnotations.dims[0];
-
-  /*
-  Var data columns - subset of all
-  */
-  if (world.varData.isEmpty()) {
-    newWorld.varData = world.varData.clone();
+  newWorld.unclipped.obsAnnotations = world.unclipped.obsAnnotations.isubsetMask(
+    mask
+  );
+  if (world.unclipped.varData.isEmpty()) {
+    newWorld.unclipped.varData = world.unclipped.varData.clone();
   } else {
-    newWorld.varData = world.varData.isubsetMask(mask);
+    newWorld.unclipped.varData = world.unclipped.varData.isubsetMask(mask);
   }
+  /* subsetting changings dimension size */
+  newWorld.nObs = newWorld.unclipped.obsAnnotations.dims[0];
+
+  /* and now clip */
+  setClippedDataframes(newWorld);
+  return newWorld;
+}
+
+/*
+Change clip quantiles on the current world, returning a new world.
+Parameters:
+  * universe
+  * world - current world
+  * clipQuantiles - new clip
+*/
+export function createWorldWithNewClip(
+  universe,
+  world,
+  crossfilter,
+  clipQuantiles
+) {
+  const newWorld = { ...world, obsAnnotation: null, varData: null };
+  newWorld.clipQuantiles = clipQuantiles;
+  newWorld.obsLayout = world.obsLayout.clone();
+  newWorld.unclipped = {
+    obsAnnotations: world.unclipped.obsAnnotations.clone(),
+    varData: world.unclipped.varData.clone()
+  };
+
+  /* and now clip */
+  setClippedDataframes(newWorld);
   return newWorld;
 }
 
@@ -137,13 +260,17 @@ function deduceDimensionType(attributes, fieldName) {
   return dimensionType;
 }
 
-export function createObsDimensions(crossfilter, world) {
+export function createObsDimensions(crossfilter, world, XYdimNames) {
   /*
   create and return a crossfilter with a dimension for every obs annotation
-  for which we have a supported type, *except* 'name'
+  for which we have a supported type, *except* for the index column, indicated
+  by schema.annotations.obs.index.
   */
   const { schema, obsLayout, obsAnnotations } = world;
-  const annoList = schema.annotations.obs.filter(anno => anno.name !== "name");
+  const indexName = schema.annotations.obs.index;
+  const annoList = schema.annotations.obs.columns.filter(
+    anno => anno.name !== indexName
+  );
   crossfilter = annoList.reduce((xfltr, anno) => {
     const dimType = deduceDimensionType(anno, anno.name);
     const colData = obsAnnotations.col(anno.name).asArray();
@@ -160,13 +287,16 @@ export function createObsDimensions(crossfilter, world) {
   return crossfilter.addDimension(
     layoutDimensionName("XY"),
     "spatial",
-    obsLayout.col("X").asArray(),
-    obsLayout.col("Y").asArray()
+    obsLayout.col(XYdimNames[0]).asArray(),
+    obsLayout.col(XYdimNames[1]).asArray()
   );
 }
 
 export function worldEqUniverse(world, universe) {
-  return world.obsAnnotations === universe.obsAnnotations;
+  return (
+    world.obsAnnotations === universe.obsAnnotations ||
+    world.obsAnnotations.rowIndex === universe.obsAnnotations.rowIndex
+  );
 }
 
 export function getSelectedByIndex(crossfilter) {

client/src/util/statemachine/index.js

@@ -0,0 +1,95 @@
+/*
+Very simple FSM for use in reducer, etc.
+
+To create a state machine:
+  new StateMachine(initialState, transitions, onErrorCallback) -> statemachine
+
+Where:
+  * initialState - a caller-specified value that represents the initial state of
+    the FSM.
+  * transitions - an array of objects, representing FSM transitions (graph edges),
+    having the form:
+      {
+        to: state_name_transitioning_to,
+        from: state_name_transitioning_from,
+        event: value_that_will_cause_transition,
+        action: optional_callback_upon_transition
+      }
+    The transition will be provided to the action callback, so other data
+    may be stored in the transition object for use by the action callback.
+  * onErrorCallback - a callback function called if the FSM receives an event
+    for which it has no defined transition.
+
+
+Interface:
+  * states - property containing the state names. A Set(), contianing the
+    union of to: and from: values.
+  * events - property containing all of the accepted event values.  Set().
+  * graph - a Map of Maps, organized as  graph[eventValue][fromStateValue]
+  * clone() - clone the entire statemachine.
+  * next(eventValue) - drive the FSM to the next state.   If the event
+    matches a transition with a defined action, the action callback is
+    called, and the action return value is returned by next().  If no
+    transition is defined, onErrorCallback is called.
+
+Example:
+
+  const transitions = [
+    { from: "A", to: "B", event: "yo", action: () => 42 }
+  ];
+  const fsm = new StateMachine("A", transitions, () => { throw new Error("oops") });
+  fsm.next("yo");   // returns 42
+
+
+*/
+export default class StateMachine {
+  constructor(initState, transitions, onError) {
+    this.onError = onError || (() => undefined);
+    this.state = initState;
+
+    // all states
+    this.states = new Set(
+      transitions.reduce((names, tsn) => {
+        names.push(tsn.from);
+        names.push(tsn.to);
+        return names;
+      }, [])
+    );
+
+    // all transition names (aka events)
+    this.events = new Set(transitions.map(tsn => tsn.event));
+
+    // the transition graph.
+    // graph[event][from] -> transition
+    this.graph = transitions.reduce((graph, tsn) => {
+      const { event, from } = tsn;
+      if (!graph.has(event)) graph.set(event, new Map());
+      const tsnMap = graph.get(event);
+      tsnMap.set(from, tsn);
+      return graph;
+    }, new Map());
+  }
+
+  clone(initState) {
+    const fsm = new StateMachine(initState, []);
+    fsm.onError = this.onError;
+    fsm.states = this.states;
+    fsm.events = this.events;
+    fsm.graph = this.graph;
+    return fsm;
+  }
+
+  next(event, data) {
+    const { graph, state } = this;
+    const tsnMap = graph.get(event);
+    if (!tsnMap) return this.onError(this, event, state, undefined);
+
+    const transition = tsnMap.get(state);
+    if (!transition) return this.onError(this, event, state, undefined);
+
+    this.state = transition.to;
+    return transition.action
+      ? transition.action(this, transition, data)
+      : undefined;
+  }
+}

client/src/util/typeHelpers.js

@@ -0,0 +1,29 @@
+/*
+Various type and schema related helper functions.
+*/
+
+/*
+Utility function to test for a typed array
+*/
+export function isTypedArray(x) {
+	return (
+		ArrayBuffer.isView(x) &&
+		Object.prototype.toString.call(x) !== "[object DataView]"
+	);
+}
+
+/*
+Test for float typed array, ie, Float32TypedArray or Float64TypedArray
+*/
+export function isFpTypedArray(x) {
+	let constructor;
+	const isFloatArray =
+		x &&
+		({ constructor } = x) &&
+		(constructor === Float32Array || constructor === Float64Array);
+	return isFloatArray;
+}
+
+export function isArrayOrTypedArray(x) {
+	return Array.isArray(x) || isTypedArray(x);
+}

client/src/util/typedCrossfilter/crossfilter.js

@@ -2,13 +2,13 @@ import { polygonContains } from "d3";
 
 import PositiveIntervals from "./positiveIntervals";
 import BitArray from "./bitArray";
-import { sort } from "./sort";
 import {
-  makeSortIndex,
+  sortArray,
   lowerBound,
   lowerBoundIndirect,
   upperBoundIndirect
-} from "./util";
+} from "./sort";
+import { makeSortIndex } from "./util";
 
 class NotImplementedError extends Error {
   constructor(...params) {
@@ -61,6 +61,10 @@ export default class ImmutableTypedCrossfilter {
     return Object.keys(this.dimensions);
   }
 
+  hasDimension(name) {
+    return !!this.dimensions[name];
+  }
+
   addDimension(name, type, ...rest) {
     /*
     Add a new dimension to this crossfilter, of type DimensionType.
@@ -284,15 +288,15 @@ class _ImmutableBaseDimension {
     this.name = name;
   }
 
-  /* eslint-disable class-methods-use-this */
   select(spec) {
     const { mode } = spec;
     if (mode === undefined) {
       throw new Error("select spec does not contain 'mode'");
     }
-    throw new Error(`select mode ${mode} not implemented`);
+    throw new Error(
+      `select mode ${mode} not implemented by dimension ${this.name}`
+    );
   }
-  /* eslint-enable class-methods-use-this */
 }
 
 class ImmutableScalarDimension extends _ImmutableBaseDimension {
@@ -414,7 +418,7 @@ class ImmutableEnumDimension extends ImmutableScalarDimension {
     for (let i = 0; i < len; i += 1) {
       s.add(mapf(i, data));
     }
-    const enumIndex = sort(Array.from(s));
+    const enumIndex = sortArray(Array.from(s));
     this.enumIndex = enumIndex;
 
     // create dimension value array
@@ -477,16 +481,16 @@ class ImmutableSpatialDimension extends _ImmutableBaseDimension {
 
   selectWithinRect(spec) {
     /*
-      { mode: "within-rect", x0: 1, y0: 0, x1: 3, y1: 9 }
+      { mode: "within-rect", minX: 1, minY: 0, maxX: 3, maxY: 9 }
     */
-    const { x0, y0, x1, y1 } = spec;
+    const { minX, minY, maxX, maxY } = spec;
     const { X, Y } = this;
     const ranges = [];
     let start = -1;
     for (let i = 0, l = X.length; i < l; i += 1) {
       const x = X[i];
       const y = Y[i];
-      const inside = x0 <= x && x < x1 && y0 <= y && y < y1;
+      const inside = minX <= x && x < maxX && minY <= y && y < maxY;
       if (inside && start === -1) start = i;
       if (!inside && start !== -1) {
         ranges.push([start, i]);

client/src/util/typedCrossfilter/sort.js

@@ -1,5 +1,29 @@
-const SmallArray = 32;
+import { isTypedArray, isFpTypedArray } from "../typeHelpers";
 
+/* eslint no-bitwise: "off" */
+
+/*
+ ** fast sort and search, with separate code paths for floats (NaN ordering),
+ ** indirect and direct search/sort.
+ */
+
+/*
+Comparators for float sort.   -Infinity < finite < Infinity < NaN
+*/
+function lt(a, b) {
+  if (Number.isNaN(b)) return !Number.isNaN(a);
+  return a < b;
+}
+
+function gt(a, b) {
+  if (Number.isNaN(a)) return !Number.isNaN(b);
+  return a > b;
+}
+
+/*
+insertion sort, used for small arrays (controlled by SMALL_ARRAY constant)
+*/
+const SMALL_ARRAY = 32;
 function insertionsort(a, lo, hi) {
   for (let i = lo + 1; i < hi + 1; i += 1) {
     const x = a[i];
@@ -12,6 +36,18 @@ function insertionsort(a, lo, hi) {
   return a;
 }
 
+function insertionsortFloats(a, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    let j;
+    for (j = i; j > lo && gt(a[j - 1], x); j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
 function insertionsortIndirect(a, s, lo, hi) {
   for (let i = lo + 1; i < hi + 1; i += 1) {
     const x = a[i];
@@ -25,8 +61,24 @@ function insertionsortIndirect(a, s, lo, hi) {
   return a;
 }
 
+function insertionsortFloatsIndirect(a, s, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    const t = s[x];
+    let j;
+    for (j = i; j > lo && gt(s[a[j - 1]], t); j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
+/*
+Quicksort - used for larger arrays
+*/
 function quicksort(a, lo, hi) {
-  if (hi - lo < SmallArray) {
+  if (hi - lo < SMALL_ARRAY) {
     return insertionsort(a, lo, hi);
   }
   if (lo < hi) {
@@ -55,8 +107,38 @@ function quicksort(a, lo, hi) {
   return a;
 }
 
+function quicksortFloats(a, lo, hi) {
+  if (hi - lo < SMALL_ARRAY) {
+    return insertionsortFloats(a, lo, hi);
+  }
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (lt(a[i], p));
+      do {
+        j -= 1;
+      } while (gt(a[j], p));
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksortFloats(a, lo, j);
+    quicksortFloats(a, j + 1, hi);
+  }
+  return a;
+}
+
 function quicksortIndirect(a, s, lo, hi) {
-  if (hi - lo < SmallArray) {
+  if (hi - lo < SMALL_ARRAY) {
     return insertionsortIndirect(a, s, lo, hi);
   }
   if (lo < hi) {
@@ -86,15 +168,248 @@ function quicksortIndirect(a, s, lo, hi) {
   return a;
 }
 
-// Convenience wrappers
-export function sort(arr, comparator = undefined) {
-  if (comparator !== undefined) {
-    // XXX for now
-    return arr.sort(arr, comparator);
+function quicksortFloatsIndirect(a, s, lo, hi) {
+  if (hi - lo < SMALL_ARRAY) {
+    return insertionsortFloatsIndirect(a, s, lo, hi);
   }
-  return quicksort(arr, 0, arr.length - 1);
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    const t = s[p];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (lt(s[a[i]], t));
+      do {
+        j -= 1;
+      } while (gt(s[a[j]], t));
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksortFloatsIndirect(a, s, lo, j);
+    quicksortFloatsIndirect(a, s, j + 1, hi);
+  }
+  return a;
+}
+
+/*
+Convenience wrappers, handling optimization paths and default
+handlers for NaN comparisons.  Sorts in place.
+*/
+export function sortArray(arr) {
+  if (Array.isArray(arr)) {
+    return quicksort(arr, 0, arr.length - 1);
+  }
+  if (isTypedArray(arr)) {
+    if (isFpTypedArray(arr)) {
+      return quicksortFloats(arr, 0, arr.length - 1);
+    }
+    return quicksort(arr, 0, arr.length - 1);
+  }
+  /* else unsupported */
+  throw new Error("sortArray received unsupported object type");
 }
 
 export function sortIndex(index, source) {
+  if (isFpTypedArray(source))
+    return quicksortFloatsIndirect(index, source, 0, index.length - 1);
   return quicksortIndirect(index, source, 0, index.length - 1);
 }
+
+// Search for `value` in the sorted array `arr`, in the range [first, last).
+// Return the first (left most) index where arr[index] >= value.
+//
+// In other words, return array index I where:
+//  arr[i] < value for all tarr[lo:I]
+//  arr[i] >= value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: lower_bound()
+//    Python: bisect.bisect_left()
+//
+function lowerBoundNonFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+// lowerBound, but with NaN handling
+//
+// If the underlying array is a Float32Array or Float64Array, will enforce
+// the ordering -Infinity < finite < Infinity < NaN.
+//
+function lowerBoundFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (lt(valueArray[middle], value)) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+export function lowerBound(valueArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return lowerBoundFloat(valueArray, value, first, last);
+  }
+  return lowerBoundNonFloat(valueArray, value, first, last);
+}
+
+// Inlined performance optimization - used to indirect through a sort map.
+//
+function lowerBoundNonFloatIndirect(
+  valueArray,
+  indexArray,
+  value,
+  first,
+  last
+) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+function lowerBoundFloatIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (lt(valueArray[indexArray[middle]], value)) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+export function lowerBoundIndirect(valueArray, indexArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return lowerBoundFloatIndirect(valueArray, indexArray, value, first, last);
+  }
+  return lowerBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
+}
+
+// Search for `value in the sorted array `arr`, in the range [first, last).
+// Return the first value where arr[index] > value.
+//
+// In other words, return array index I, where:
+//  arr[i] <= value for all tarr[lo:I]
+//  arr[i] > value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: upper_bound()
+//    Python: bisect.bisect_right()
+//
+function upperBoundNonFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+function upperBoundFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (gt(valueArray[middle], value)) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+export function upperBound(valueArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return upperBoundFloat(valueArray, value, first, last);
+  }
+  return upperBoundNonFloat(valueArray, value, first, last);
+}
+
+// Inline performance optimization
+//
+function upperBoundNonFloatIndirect(
+  valueArray,
+  indexArray,
+  value,
+  first,
+  last
+) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+function upperBoundFloatIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (gt(valueArray[indexArray[middle]], value)) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return upperBoundFloatIndirect(valueArray, indexArray, value, first, last);
+  }
+  return upperBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
+}

client/src/util/typedCrossfilter/util.js

@@ -1,23 +1,12 @@
 // jshint esversion: 6
-/* eslint no-bitwise: "off" */
 
 import { sortIndex } from "./sort";
+import { rangeFill as fillRange } from "../range";
 
 /*
     Utility functions, private to this module.
 */
 
-// fill an array or typedarray with a sequential range of numbers,
-// starting with `start`
-//
-export function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, len = larr.length; i < len; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 // slice out of one array into another, using an index array
 //
 export function sliceByIndex(src, index) {
@@ -36,93 +25,3 @@ export function makeSortIndex(src) {
   sortIndex(index, src);
   return index;
 }
-
-// Search for `value` in the sorted array `arr`, in the range [first, last).
-// Return the first (left most) index where arr[index] >= value.
-//
-// In other words, return array index I where:
-//  arr[i] < value for all tarr[lo:I]
-//  arr[i] >= value for all tarr[I:last]
-//
-// The same semantics/behavior as:
-//    C++: lower_bound()
-//    Python: bisect.bisect_left()
-//
-// XXX: it is likely that there would be minimal performance hit from creating
-// a factory version of lowerBound that takes an accessor (rather than having
-// a special-cased version for lining the indirection).
-//
-export function lowerBound(valueArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[middle] < value) {
-      lfirst = middle + 1;
-    } else {
-      llast = middle;
-    }
-  }
-  return lfirst;
-}
-
-// Inlined performance optimization - used to indirect through a sort map.
-//
-export function lowerBoundIndirect(valueArray, indexArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[indexArray[middle]] < value) {
-      lfirst = middle + 1;
-    } else {
-      llast = middle;
-    }
-  }
-  return lfirst;
-}
-
-// Search for `value in the sorted array `arr`, in the range [first, last).
-// Return the first value where arr[index] > value.
-//
-// In other words, return array index I, where:
-//  arr[i] <= value for all tarr[lo:I]
-//  arr[i] > value for all tarr[I:last]
-//
-// The same semantics/behavior as:
-//    C++: upper_bound()
-//    Python: bisect.bisect_right()
-//
-export function upperBound(valueArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[middle] > value) {
-      llast = middle;
-    } else {
-      lfirst = middle + 1;
-    }
-  }
-  return lfirst;
-}
-
-// Inline performance optimization
-//
-export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[indexArray[middle]] > value) {
-      llast = middle;
-    } else {
-      lfirst = middle + 1;
-    }
-  }
-  return lfirst;
-}

dev_docs/developer_guidelines.md

@@ -0,0 +1,91 @@
+# Developer guidelines
+
+### Requirements
+- npm
+- Python 3.6+
+- Chrome
+
+[See dev section of README](../README.md)
+
+**All instructions are expected to be run from the top level cellxgene directory unless otherwise specified.**
+
+## Server dev
+### Install
+* Build the client and put static files in place: `make build-for-server-dev`
+* Install from local files: `make install-dev`
+
+### Launch
+* `cellxgene launch [options] <datafile>`
+
+### Reloading
+If you install cellxgene using `make install-dev` the server will be restarted every time you make changes on the server code. If changes affects the client, the browser must be reloaded.
+
+### Linter
+We use `flake8` to lint code. Travis CI runs `flake8 server`.
+
+### Test
+1. Install development requirements `pip install -r server/requirements-dev.txt`
+2. Run tests `pytest server/test`
+
+### Tips
+* Install in a virtualenv
+* May need to rebuild/reinstall when you make client changes
+
+## Client dev
+### Install
+1. Install prereqs for client: `npm install --prefix client/ client`
+2. Install cellxgene server: `pip install -e .` Caveat: this will not build the production client package - you must use the [server install](#install) instructions above to serve web assets.
+
+### Launch
+To launch with hot reloading you need to launch the server and the client separately. Node's hot reloading starts the client on its own node server and auto-refreshes when changes are made.
+1. Launch server (the client relies on the REST API being available): `cellxgene launch [options] <datafile>`
+2. Launch client: in `client/` directory run `npm run start`
+3. Client will be served on localhost:3000
+
+### Build
+To build only the client: `make build-client`
+
+### Linter
+We use `eslint` to lint the code and `prettier` as our code formatter.
+
+### Test
+In `client/` directory run `npm run unit-test`
+
+### Tips
+* You can also install/launch the server side code from npm scrips (requires python3.6 with virtualenv) in `client/` directory run `npm run backend-dev`
+
+## Running tests
+Client and server tests run on Travis CI for every push, PR, and commit to master on github. End to end tests run nightly on master only. 
+
+### Server unit tests
+Install development requirements `pip install -r server/requirements-dev.txt`
+Run tests `pytest server/test`
+
+### Client unit tests
+In `client/` directory run `npm run unit-test`
+
+### End to end tests
+
+End to end tests use two env variables:
+* `JEST_ENV` - environment to run end to end tests. Default `dev`
+    * `prod` - run headless with no slowdown, chromium will not open.
+    * `dev` - opens chromimum, runs tests with minimal slowdown, close on exit.
+    * `debug` - opens chromium, runs tests with 100ms slowdown, dev tools open, chrome stays open on exit.
+* `JEST_CXG_PORT` - port that end to end tests are being run on. Default `3000` (client hosted port).
+
+On CI the end to end tests are run with `JEST_ENV` set to `prod` using the `smoke-test` npm script
+
+To run end to end tests as they will be run on CI
+1. cellxgene should be built and installed as [specified in server dev](#install)
+2. `export JEST_ENV='prod'`
+3. `export JEST_CXG_PORT='5000'`
+4. Run `npm run --prefix client/ smoke-test`
+
+Run end to end tests interactively during development
+1. cellxgene should be installed as [specified in client dev](#install-1)
+2. Follow [launch](#launch-1) instructions for client dev with dataset `example-dataset/pbmc3k`
+3. Run `npm run --prefix client/ e2e`
+4. To debug a failing test `export JEST_ENV='debug'` and re-run.
+
+
+

dev_docs/release_process.md

@@ -43,24 +43,53 @@ Follow these steps to create a release.
 8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
 and that you have write access to the cellxgene pypi package):
     - Build the distribution and upload to test pypi `make release-stage-2`
-    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Test the test installation in a fresh virtual environment using `make install-release-test`
     - Upload the package to real pypi using `make release-stage-final`
-    - [optional] Test the installation in a fresh virtual environment using
+    - Test the installation in a fresh virtual environment using
       `pip install cellxgene`
-    - **Troubleshooting**:
-        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
-        if you accidentally burned a release number you want to use on prod, you have a couple options. 
-            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
-             then upload to prod `make release-stage-final`. 
-            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
-            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
-            PART=patch until you get to an unburned version.
-        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
-        go to the cellxgene package, click manage, then in the options drop down, click delete and 
-        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
-        and not a major regression, you can just release a patch. 
-    
 
 The optional steps are for testing purposes, and are recommended
 for publishing any major releases, and any releases that significantly
 change the packaging (e.g. new bundled files, new dependencies, etc.)
+
+## Troubleshooting
+### Fails to upload to test.pypi
+ 
+ _PyPi doesn't allow you to reupload a release with the same version number_ 
+If you accidentally burned a release number you want to use on prod, you have a few options:
+1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+ then upload to prod `make release-stage-final`. 
+2) OPTION 2: (DANGER) release directly to prod: `make release-directly-to-prod`.  
+3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+PART=patch until you get to an unburned version.
+    
+### The release doesn't install or fails your tests when you install it
+ 
+Delete it from pypi - Go to pypi.org -> sign in -> go to the cellxgene package -> click manage -> then in the options drop down click delete -> follow the instructions. You will not be able to use that release number again. If it is a minor bug and not a major regression, you can just release a patch. 
+### If you need to run stage final on a different computer than stage 2
+ If you run stage final without running stage 2 first, the dist will not have been build on the computer running stage final. The solution is to run `make release-directly-to-prod`. This both builds the distribution files and then releases directly to prod pypi.org. 
+
+## Stage Details
+### Stage 1 - `make release-stage-1`
+    1. Pip installs requirements-dev
+    2. Bumps version by [PART]
+    3. Deletes build directory, client/build, dist and cellxgene.egg-info
+    4. Creates the package-lock.json
+
+### Stage 2 - `make release-stage-2`
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to test.pypi.org
+
+### Stage final - `make release-stage-final`
+    ** Does not build distribution **
+    1. Uploads to pypi.org
+    
+### (DANGER) Release directly to prod `make release-directly-to-prod`
+    ** builds distribution and uploads directly to prod **
+    Only use this if you are directed to by the troubleshooting guide
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to pypi.org

docs/_config.yml

@@ -3,6 +3,8 @@ show_downloads: false
 baseurl: /cellxgene
 
 nav:
+  - title: Getting Started
+    url: getting-started.html
   - title: Data
     url: data.html
   - title: FAQ

docs/faq.md

@@ -38,7 +38,15 @@ Currently this is not supported directly, but you should be able to do this your
 
 - `.obs` and `.var` annotations are use to extract metadata for filtering
 - `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
-- `.obsm` is used for layout
+- `.obsm` is used for layout. If an embedding has more than two components, the first two will be used for visualization.
+
+#### I have a BIG dataset - how can I make cellxgene run as fast as possible?
+
+If your dataset requires gigabytes of disk space, you may need to select an appropriate storage format in order to effectively utilize `cellxgene`. Tips and tricks:
+
+- `cellxgene` is optimized for columnar data access. For large datasets, format the expression matrix (`.X`) as either a [SciPy CSC sparse matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) or a dense Numpy array (whichever creates a smaller `h5ad` file). If you are using `cellxgene prepare`, include the `--sparse` flag to ensure `.X` is formatted as a CSC sparse matrix (by default, `.X` will be a dense matrix).
+- `cellxgene` start time is directly proportional to `h5ad` file size and the speed of your file system. Expect that large (eg, million cell) datasets will take minutes to load, even on relatively fast computers with a high performance local hard drive. Once loaded, exploring metadata should still be quick.
+- If your dataset size exceeds the size of memory (RAM) on the host computer, differential expression calculations will be extremely slow (or fail, if you run out of virtual memory).
 
 # Algorithms
 
@@ -48,12 +56,12 @@ We use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) impl
 
 # Problems, errors, & bugs
 
-#### How do I create a Python 3.6 environment for _cellxgene_?
+#### How do I create a Python environment for _cellxgene_?
 
 If you use conda and want to create a [conda environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for _cellxgene_ you can use the following commands
 
 ```
-conda create --yes -n cellxgene python=3.6
+conda create --yes -n cellxgene python=3.7
 conda activate cellxgene
 pip install cellxgene
 ```
@@ -62,7 +70,7 @@ Or you can create a virtual environment by using
 
 ```
 ENV_NAME=cellxgene
-python3.6 -m venv ${ENV_NAME}
+python3.7 -m venv ${ENV_NAME}
 source ${ENV_NAME}/bin/activate
 pip install cellxgene
 ```

docs/getting-started.md

@@ -0,0 +1,116 @@
+## getting started
+
+You'll need **python 3.6+** and **Google Chrome**.
+The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know.
+
+To install run
+
+```
+pip install cellxgene
+```
+
+To start exploring a dataset call
+
+```
+cellxgene launch dataset.h5ad --open
+```
+
+If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
+
+```
+cellxgene launch pbmc3k.h5ad --open
+```
+
+On Mac OS and Ubuntu, you should see your web browser open with the following
+
+<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+
+**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+
+There are several options available, such as:
+
+- `--layout` to specify the layout as `tsne`, `umap`, `diffmap`, `phate`, `draw_graph_fa`, or `draw_graph_fr`
+- `--title` to show a title on the explorer
+- `--open` to automatically open the web browser after launching (OS X only)
+
+To see all options call
+
+```
+cellxgene launch --help
+```
+
+There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+
+## data formatting
+
+### requirements
+
+The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
+
+- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
+- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
+- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_umap` for the `umap` layout)
+- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
+
+### prepare
+
+The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
+
+To prepare from an existing `.h5ad` file use
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
+```
+
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
+
+Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
+```
+
+To see all options call
+
+```
+cellxgene prepare --help
+```
+
+**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
+
+```
+pip install cellxgene[louvain]
+```
+
+If the aforementioned optional package installation fails, you can also install these packages directly:
+
+```
+pip install python-igraph louvain>=0.6
+```
+
+## conda and virtual environments
+
+If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.7
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3.7 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+## docker
+
+We have included a dockerfile to conveniently run cellxgene from docker.
+
+1. Build the image `docker build . -t cellxgene`
+2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
+   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser

docs/index.md

@@ -2,15 +2,15 @@ _cellxgene_ is an interactive data explorer for single-cell transcriptomics data
 
 ## features
 
-#### Flexible selections, coloring, and differential expression of your selected sets of cells
+#### flexible selections, coloring, and differential expression of your selected sets of cells
 <img src="diffexp.gif" width="600"/>
 
-#### Single-gene analyses (e.g. expression analysis)
+#### single-gene analyses (e.g. expression analysis)
 <img src="customGene.gif" width="600" />
 
-## getting started
+## quick start
 
-_cellxgene_ **only** supports Python 3.6. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
+To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-environment-for-cellxgene)
 
 Install the package.
 ``` bash
@@ -25,13 +25,17 @@ curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/m
 
 Launch _cellxgene_
 ``` bash
-cellxgene launch pbmc3k.h5ad
+cellxgene launch pbmc3k.h5ad --open
 ```
 
+To explore more datasets already formatted for _cellxgene_, see [Data](data) or
+visit [Getting Started](getting-started) to learn more about formatting your own
+data for _cellxgene_.
+
 ## getting help
 
 We'd love to hear from you!
 
-For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cziscience-slack.herokuapp.com/) and say "hi!".
+For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and say "hi!".
 
 For any errors, [report bugs on Github](https://github.com/chanzuckerberg/cellxgene/issues).

makefile

@@ -76,7 +76,7 @@ release-stage-final: twine-prod
 
 # DANGER: releases directly to prod
 # use this if you accidently burned a test release version number,
-release-burned : dev-env pydist twine-prod
+release-directly-to-prod : dev-env pydist twine-prod
 	@echo "Dist built and uploaded to pypi.org"
 	@echo "Test the install:"
 	@echo "    make install-release"
@@ -114,14 +114,18 @@ install-dev : uninstall
 
 # install from test.pypi to test your release
 install-release-test : uninstall
-	pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
+	pip install --no-cache-dir --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
 	@echo "Installed cellxgene from test.pypi.org, now run and smoke test"
 
 # install from pypi to test your release
 install-release : uninstall
-	pip install cellxgene
+	pip install --no-cache-dir cellxgene
 	@echo "Installed cellxgene from pypi.org"
 
+# install from dist
+install-dist : uninstall
+	pip install dist/cellxgene*.tar.gz
+
 uninstall :
 	pip uninstall -y cellxgene || :
 

server/app/app.py

@@ -5,27 +5,34 @@ from flask_caching import Cache
 from flask_compress import Compress
 from flask_cors import CORS
 
-from .rest_api.rest import get_api_resources
-from .util.utils import Float32JSONEncoder
-from .web import webapp
+from server.app.rest_api.rest import get_api_resources
+from server.app.util.utils import Float32JSONEncoder
+from server.app.web import webapp
 
-REACTIVE_LIMIT = 1_000_000
 
-app = Flask(__name__, static_folder="web/static")
-app.json_encoder = Float32JSONEncoder
-cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860_000})
-Compress(app)
-CORS(app)
+class Server:
+    def __init__(self):
+        self.data = None
+        self.cache = Cache(config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860_000})
+        self.app = None
 
-# Config
-SECRET_KEY = os.environ.get("CXG_SECRET_KEY", default="SparkleAndShine")
+    def create_app(self):
+        self.app = Flask(__name__, static_folder="web/static")
+        self.app.json_encoder = Float32JSONEncoder
+        self.cache.init_app(self.app)
+        Compress(self.app)
+        CORS(self.app)
 
-app.config.update(SECRET_KEY=SECRET_KEY)
+        # Config
+        SECRET_KEY = os.environ.get("CXG_SECRET_KEY", default="SparkleAndShine")
+        self.app.config.update(SECRET_KEY=SECRET_KEY)
+        self.app.config.update(SCRIPTS=[])
 
-# Application Data
-data = None
+        resources = get_api_resources()
+        self.app.register_blueprint(webapp.bp)
+        self.app.register_blueprint(resources.blueprint)
+        self.app.add_url_rule("/", endpoint="index")
 
-resources = get_api_resources()
-app.register_blueprint(webapp.bp)
-app.register_blueprint(resources.blueprint)
-app.add_url_rule("/", endpoint="index")
+    def attach_data(self, data, title="Demo"):
+        self.app.config.update(DATASET_TITLE=title)
+        self.app.data = data

server/app/driver/driver.py

@@ -11,13 +11,27 @@ Sort order for methods
 
 
 class CXGDriver(metaclass=ABCMeta):
-    def __init__(self, data, args):
-        self.data = self._load_data(data)
-        self.layout_method = args["layout"]
-        self.diffexp_method = args["diffexp"]
-        self.max_category_items = args["max_category_items"]
-        self.diffexp_lfc_cutoff = args["diffexp_lfc_cutoff"]
-        self.cluster = None
+    def __init__(self, data=None, args={}):
+        self.config = self._get_default_config()
+        self.config.update(args)
+        if data:
+            self._load_data(data)
+        else:
+            self.data = None
+
+    def update(self, data=None, args={}):
+        self.config.update(args)
+        if data:
+            self._load_data(data)
+
+    @staticmethod
+    def _get_default_config():
+        return {
+            "layout": None,
+            "diffexp": None,
+            "max_category_items": None,
+            "diffexp_lfc_cutoff": None
+        }
 
     @property
     def features(self):
@@ -27,18 +41,15 @@ class CXGDriver(metaclass=ABCMeta):
             "diffexp": {"available": False},
         }
         # TODO - Interactive limit should be generated from the actual available methods see GH issue #94
-        if self.layout_method:
+        if self.config["layout"]:
             # TODO handle "var" when gene layout becomes available
             features["layout"]["obs"] = {"available": True, "interactiveLimit": 50000}
-        if self.diffexp_method:
+        if self.config["diffexp"]:
             features["diffexp"] = {"available": True, "interactiveLimit": 50000}
-        if self.cluster:
-            features["cluster"] = {"available": True, "interactiveLimit": 50000}
         return features
 
-    @staticmethod
     @abstractmethod
-    def _load_data(data):
+    def _load_data(self, data):
         pass
 
     @abstractmethod

server/app/rest_api/rest.py

@@ -63,7 +63,7 @@ class ConfigAPI(Resource):
                     "dataset": current_app.config["DATASET_TITLE"],
                 },
                 "parameters": {
-                    "max_category_items": current_app.data.max_category_items
+                    "max_category_items": current_app.data.config["max_category_items"]
                 },
                 "library_versions": {
                     "scanpy": pkg_resources.get_distribution("scanpy").version,

server/app/scanpy_engine/scanpy_engine.py

@@ -1,18 +1,20 @@
 import warnings
 
 import numpy as np
+import pandas
 from pandas.core.dtypes.dtypes import CategoricalDtype
-import scanpy.api as sc
+import anndata
+from scipy import sparse
 
 from server.app.driver.driver import CXGDriver
-from server.app.util.constants import Axis, DEFAULT_TOP_N
+from server.app.util.constants import Axis, DEFAULT_TOP_N, MAX_LAYOUTS
 from server.app.util.errors import (
     FilterError,
     JSONEncodingValueError,
     PrepareError,
     ScanpyFileError,
 )
-from server.app.util.utils import jsonify_scanpy
+from server.app.util.utils import jsonify_scanpy, requires_data
 from server.app.scanpy_engine.diffexp import diffexp_ttest
 from server.app.util.fbs.matrix import encode_matrix_fbs
 
@@ -27,53 +29,82 @@ Sort order for methods
 
 
 class ScanpyEngine(CXGDriver):
-    def __init__(self, data, args):
+    def __init__(self, data=None, args={}):
         super().__init__(data, args)
-        self._alias_annotation_names(Axis.OBS, args["obs_names"])
-        self._alias_annotation_names(Axis.VAR, args["var_names"])
-        self._validate_data_types()
-        self._validate_data_calculations()
-        self.cell_count = self.data.shape[0]
-        self.gene_count = self.data.shape[1]
-        self.layout_options = ["umap", "tsne"]
-        self.diffexp_options = ["ttest"]
-        self._create_schema()
+        if self.data:
+            self._validate_and_initialize()
 
-    def _alias_annotation_names(self, axis, name):
+    def update(self, data=None, args={}):
+        super().__init__(data, args)
+        if self.data:
+            self._validate_and_initialize()
+
+    @staticmethod
+    def _get_default_config():
+        return {
+            "layout": [],
+            "diffexp": "ttest",
+            "max_category_items": 100,
+            "obs_names": None,
+            "var_names": None,
+            "diffexp_lfc_cutoff": 0.01,
+        }
+
+    @staticmethod
+    def _create_unique_column_name(df, col_name_prefix):
+        """ given the columns of a dataframe, and a name prefix, return a column name which
+            does not exist in the dataframe, AND which is prefixed by `prefix`
+
+            The approach is to append a numeric suffix, starting at zero and increasing by
+            one, until an unused name is found (eg, prefix_0, prefix_1, ...).
         """
-        Do all user-specified annotation aliasing.
+        suffix = 0
+        while f"{col_name_prefix}{suffix}" in df:
+            suffix += 1
+        return f"{col_name_prefix}{suffix}"
 
-        As a *critical* side-effect, ensure the indices are simple number ranges
-        (accomplished by calling pandas.DataFrame.reset_index())
+    def _alias_annotation_names(self):
         """
-        if name == "name":
-            # a noop, so skip it
-            return
+        The front-end relies on the existance of a unique, human-readable
+        index for obs & var (eg, var is typically gene name, obs the cell name).
+        The user can specify these via the --obs-names and --var-names config.
+        If they are not specified, use the existing index to create them, giving
+        the resulting column a unique name (eg, "name").
 
-        ax_name = str(axis)
-        df_axis = getattr(self.data, ax_name)
-        if name is None:
-            # reset index to simple range; alias "name" to point at the
-            # previously specified index.
-            df_axis.reset_index(inplace=True)
-            df_axis.rename(inplace=True, columns={"index": "name"})
-        elif name in df_axis.columns:
-            if name not in df_axis.columns:
+        In both cases, enforce that the result is unique, and communicate the
+        index column name to the front-end via the obs_names and var_names config
+        (which is incorporated into the schema).
+        """
+        for (ax_name, config_name) in ((Axis.OBS, "obs_names"), (Axis.VAR, "var_names")):
+            name = self.config[config_name]
+            df_axis = getattr(self.data, str(ax_name))
+            if name is None:
+                # Default: create unique names from index
+                if not df_axis.index.is_unique:
+                    raise KeyError(
+                        f"Values in {ax_name}.index must be unique. "
+                        "Please prepare data to contain unique index values, or specify an "
+                        "alternative with --{ax_name}-name."
+                    )
+                name = self._create_unique_column_name(df_axis.columns, "name_")
+                self.config[config_name] = name
+                # reset index to simple range; alias name to point at the
+                # previously specified index.
+                df_axis.rename_axis(name, inplace=True)
+                df_axis.reset_index(inplace=True)
+            elif name in df_axis.columns:
+                # User has specified alternative column for unique names, and it exists
+                if not df_axis[name].is_unique:
+                    raise KeyError(
+                        f"Values in {ax_name}.{name} must be unique. "
+                        "Please prepare data to contain unique values."
+                    )
+                df_axis.reset_index(drop=True, inplace=True)
+            else:
+                # user specified a non-existent column name
                 raise KeyError(
                     f"Annotation name {name}, specified in --{ax_name}-name does not exist."
                 )
-            if not df_axis[name].is_unique:
-                raise KeyError(
-                    f"Values in -{ax_name}-name must be unique. "
-                    "Please prepare data to contain unique values."
-                )
-            # reset index to simple range; alias user-specified annotation to "name"
-            df_axis.reset_index(drop=True, inplace=True)
-            df_axis.rename(inplace=True, columns={name: "name"})
-        else:
-            raise KeyError(
-                f"Annotation name {name}, specified in --{ax_name}_name does not exist."
-            )
 
     @staticmethod
     def _can_cast_to_float32(ann):
@@ -95,6 +126,7 @@ class ScanpyEngine(CXGDriver):
                 return True
         return False
 
+    @requires_data
     def _create_schema(self):
         self.schema = {
             "dataframe": {
@@ -102,7 +134,17 @@ class ScanpyEngine(CXGDriver):
                 "nVar": self.gene_count,
                 "type": str(self.data.X.dtype),
             },
-            "annotations": {"obs": [], "var": []},
+            "annotations": {
+                "obs": {
+                    "index": self.config["obs_names"],
+                    "columns": []
+                },
+                "var": {
+                    "index": self.config["var_names"],
+                    "columns": []
+                }
+            },
+            "layout": {"obs": []}
         }
         for ax in Axis:
             curr_axis = getattr(self.data, str(ax))
@@ -126,15 +168,21 @@ class ScanpyEngine(CXGDriver):
                     raise TypeError(
                         f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene."
                     )
-                self.schema["annotations"][ax].append(ann_schema)
+                self.schema["annotations"][ax]["columns"].append(ann_schema)
 
-    @staticmethod
-    def _load_data(data):
-        # Based on benchmarking, cache=True has no impact on perf.
-        # Note: as of current scanpy/anndata release, setting backed='r' will
-        # result in an error.  https://github.com/theislab/anndata/issues/79
+        for layout in self.config['layout']:
+            layout_schema = {
+                "name": layout,
+                "type": "float32",
+                "dims": [f"{layout}_0", f"{layout}_1"]
+            }
+            self.schema["layout"]["obs"].append(layout_schema)
+
+    def _load_data(self, data):
+        # as of AnnData 0.6.19, backed mode performs initial load fast, but at the
+        # cost of significantly slower access to X data.
         try:
-            result = sc.read(data, cache=True)
+            self.data = anndata.read_h5ad(data)
         except ValueError:
             raise ScanpyFileError(
                 "File must be in the .h5ad format. Please read "
@@ -151,9 +199,71 @@ class ScanpyEngine(CXGDriver):
                 f"Error while loading file: {e}, File must be in the .h5ad format, please check "
                 f"that your input and try again."
             )
-        return result
 
+    @requires_data
+    def _validate_and_initialize(self):
+        # var and obs column names must be unique
+        if not self.data.obs.columns.is_unique or not self.data.var.columns.is_unique:
+            raise KeyError(f"All annotation column names must be unique.")
+
+        self._alias_annotation_names()
+        self._validate_data_types()
+        self.cell_count = self.data.shape[0]
+        self.gene_count = self.data.shape[1]
+        self._default_and_validate_layouts()
+        self._create_schema()
+
+    @requires_data
+    def _default_and_validate_layouts(self):
+        """ function:
+            a) generate list of default layouts, if not already user specified
+            b) validate layouts are legal.  remove/warn on any that are not
+            c) cap total list of layouts at global const MAX_LAYOUTS
+        """
+        layouts = self.config['layout']
+        # handle default
+        if layouts is None or len(layouts) == 0:
+            # load default layouts from the data.
+            layouts = [key[2:] for key in self.data.obsm_keys() if type(key) == str and key.startswith("X_")]
+            if len(layouts) == 0:
+                raise PrepareError(f"Unable to find any precomputed layouts within the dataset.")
+
+        # remove invalid layouts
+        valid_layouts = []
+        obsm_keys = self.data.obsm_keys()
+        for layout in layouts:
+            layout_name = f"X_{layout}"
+            if layout_name not in obsm_keys:
+                warnings.warn(f"Ignoring unknown layout name: {layout}.")
+            elif not self._is_valid_layout(self.data.obsm[layout_name]):
+                warnings.warn(f"Ignoring layout due to malformed shape or data type: {layout}")
+            else:
+                valid_layouts.append(layout)
+
+        if len(valid_layouts) == 0:
+            raise PrepareError(f"No valid layout data.")
+
+        # cap layouts to MAX_LAYOUTS
+        self.config['layout'] = valid_layouts[0:MAX_LAYOUTS]
+
+    @requires_data
+    def _is_valid_layout(self, arr):
+        """ return True if this layout data is a valid array for front-end presentation:
+            * ndarray, with shape (n_obs, >= 2), dtype float/int/uint
+            * contains only finite values
+        """
+        is_valid = type(arr) == np.ndarray and arr.dtype.kind in "fiu"
+        is_valid = is_valid and arr.shape[0] == self.data.n_obs and arr.shape[1] >= 2
+        is_valid = is_valid and np.all(np.isfinite(arr))
+        return is_valid
+
+    @requires_data
     def _validate_data_types(self):
+        if sparse.isspmatrix(self.data.X) and not sparse.isspmatrix_csc(self.data.X):
+            warnings.warn(
+                f"Scanpy data matrix is sparse, but not a CSC (columnar) matrix.  "
+                f"Performance may be improved by using CSC."
+            )
         if self.data.X.dtype != "float32":
             warnings.warn(
                 f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
@@ -176,7 +286,7 @@ class ScanpyEngine(CXGDriver):
                     )
                 if isinstance(datatype, CategoricalDtype):
                     category_num = len(curr_axis[ann].dtype.categories)
-                    if category_num > 500 and category_num > self.max_category_items:
+                    if category_num > 500 and category_num > self.config['max_category_items']:
                         warnings.warn(
                             f"{str(ax).title()} annotation '{ann}' has {category_num} categories, this may be "
                             f"cumbersome or slow to display. We recommend setting the "
@@ -184,19 +294,6 @@ class ScanpyEngine(CXGDriver):
                             f"annotations with more than 500 categories in the UI"
                         )
 
-    def _validate_data_calculations(self):
-        layout_key = f"X_{self.layout_method}"
-        try:
-            assert layout_key in self.data.obsm_keys()
-        except AssertionError:
-            raise PrepareError(
-                f"Cannot find a field with coordinates for the {self.layout_method} layout requested. A different"
-                f" layout may have been computed. The requested layout must be pre-calculated and saved "
-                f"back in the h5ad file. You can run "
-                f"`cellxgene prepare --layout {self.layout_method} <datafile>` "
-                f"to solve this problem. "
-            )
-
     @staticmethod
     def _annotation_filter_to_mask(filter, d_axis, count):
         mask = np.ones((count,), dtype=bool)
@@ -220,7 +317,7 @@ class ScanpyEngine(CXGDriver):
         mask = np.zeros((count,), dtype=bool)
         for i in filter:
             if type(i) == list:
-                mask[i[0] : i[1]] = True
+                mask[i[0]: i[1]] = True
             else:
                 mask[i] = True
         return mask
@@ -241,6 +338,7 @@ class ScanpyEngine(CXGDriver):
             )
         return mask
 
+    @requires_data
     def _filter_to_mask(self, filter, use_slices=True):
         if use_slices:
             obs_selector = slice(0, self.data.n_obs)
@@ -260,6 +358,7 @@ class ScanpyEngine(CXGDriver):
                 )
         return obs_selector, var_selector
 
+    @requires_data
     def annotation_to_fbs_matrix(self, axis, fields=None):
         if axis == Axis.OBS:
             df = self.data.obs
@@ -269,6 +368,23 @@ class ScanpyEngine(CXGDriver):
             df = df[fields]
         return encode_matrix_fbs(df, col_idx=df.columns)
 
+    @staticmethod
+    def slice_columns(X, var_mask):
+        """
+        Slice columns from the matrix X, as specified by the mask
+        Semantically equivalent to X[:, var_mask], but handles sparse
+        matrices in a more performant manner.
+        """
+        if var_mask is None:    # noop
+            return X
+        if sparse.issparse(X):  # use tuned getcol/hstack for performance
+            indices = np.nonzero(var_mask)[0]
+            cols = [X.getcol(i) for i in indices]
+            return sparse.hstack(cols, format="csc")
+        else:   # else, just use standard slicing, which is fine for dense arrays
+            return X[:, var_mask]
+
+    @requires_data
     def data_frame_to_fbs_matrix(self, filter, axis):
         """
         Retrieves data 'X' and returns in a flatbuffer Matrix.
@@ -290,11 +406,10 @@ class ScanpyEngine(CXGDriver):
             raise FilterError("filtering on obs unsupported")
 
         # Currently only handles VAR dimension
-        X = self.data._X
-        if var_selector is not None:
-            X = X[:, var_selector]
+        X = self.slice_columns(self.data._X, var_selector)
         return encode_matrix_fbs(X, col_idx=np.nonzero(var_selector)[0], row_idx=None)
 
+    @requires_data
     def diffexp_topN(self, obsFilterA, obsFilterB, top_n=None, interactive_limit=None):
         if Axis.VAR in obsFilterA or Axis.VAR in obsFilterB:
             raise FilterError("Observation filters may not contain vaiable conditions")
@@ -310,7 +425,7 @@ class ScanpyEngine(CXGDriver):
         if top_n is None:
             top_n = DEFAULT_TOP_N
         result = diffexp_ttest(
-            self.data, obs_mask_A, obs_mask_B, top_n, self.diffexp_lfc_cutoff
+            self.data, obs_mask_A, obs_mask_B, top_n, self.config['diffexp_lfc_cutoff']
         )
         try:
             return jsonify_scanpy(result)
@@ -319,6 +434,7 @@ class ScanpyEngine(CXGDriver):
                 "Error encoding differential expression to JSON"
             )
 
+    @requires_data
     def layout_to_fbs_matrix(self):
         """
         Return the default 2-D layout for cells as a FBS Matrix.
@@ -326,17 +442,33 @@ class ScanpyEngine(CXGDriver):
         Caveats:
         * does not support filtering
         * only returns Matrix in columnar layout
+
+        All embeddings must be individually centered & scaled (isotropically)
+        to a [0, 1] range.
         """
         try:
-            full_embedding = self.data.obsm[f"X_{self.layout_method}"]
-            if full_embedding.shape[1] > 2:
-                warnings.warn(f"Warning: found {full_embedding.shape[1]} \
-                components of embedding. Using the first two for layout display.")
-            df_layout = full_embedding[:, :2]
+            layout_data = []
+            for layout in self.config["layout"]:
+                full_embedding = self.data.obsm[f"X_{layout}"]
+                embedding = full_embedding[:, :2]
+
+                # scale isotropically
+                min = embedding.min(axis=0)
+                max = embedding.max(axis=0)
+                scale = np.amax(max - min)
+                normalized_layout = (embedding - min) / scale
+
+                # translate to center on both axis
+                translate = 0.5 - ((max - min) / scale / 2)
+                normalized_layout = normalized_layout + translate
+
+                normalized_layout = normalized_layout.astype(dtype=np.float32)
+                layout_data.append(pandas.DataFrame(normalized_layout, columns=[f"{layout}_0", f"{layout}_1"]))
+
         except ValueError as e:
             raise PrepareError(
-                f"Layout has not been calculated using {self.layout_method}, "
+                f"Layout has not been calculated using {self.config['layout']}, "
                 f"please prepare your datafile and relaunch cellxgene") from e
 
-        normalized_layout = (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min())
-        return encode_matrix_fbs(normalized_layout.astype(dtype=np.float32), col_idx=None, row_idx=None)
+        df = pandas.concat(layout_data, axis=1, copy=False)
+        return encode_matrix_fbs(df, col_idx=df.columns, row_idx=None)

server/app/util/constants.py

@@ -30,3 +30,6 @@ class DiffExpMode(AugmentedEnum):
 JSON_NaN_to_num_warning_msg = (
     "JSON encoding failure - please verify all data are finite values (no NaN or Infinities)"
 )
+REACTIVE_LIMIT = 1_000_000
+
+MAX_LAYOUTS = 30

server/app/util/errors.py

@@ -50,3 +50,12 @@ class ScanpyFileError(Exception):
 
     def __init__(self, message):
         self.message = message
+
+
+class DriverError(Exception):
+    """
+    Raised when file loaded into scanpy is misformatted
+    """
+
+    def __init__(self, message):
+        self.message = message

server/app/util/utils.py

@@ -1,6 +1,10 @@
-import json
+from functools import wraps
+
+from flask import json
 from numpy import float32, integer
 
+from server.app.util.errors import DriverError
+
 
 class Float32JSONEncoder(json.JSONEncoder):
     def __init__(self, *args, **kwargs):
@@ -28,3 +32,12 @@ def custom_format_warning(msg, *args, **kwargs):
 
 def jsonify_scanpy(data):
     return json.dumps(data, cls=Float32JSONEncoder, allow_nan=False)
+
+
+def requires_data(func):
+    @wraps(func)
+    def wrapped_function(self, *args, **kwargs):
+        if self.data is None:
+            raise DriverError(f"error data must be loaded before you call {func.__name__}")
+        return func(self, *args, **kwargs)
+    return wrapped_function

server/app/web/webapp.py

@@ -8,7 +8,8 @@ bp = Blueprint("webapp", __name__, template_folder="templates")
 @bp.route("/")
 def index():
     dataset_title = current_app.config["DATASET_TITLE"]
-    return render_template("index.html", datasetTitle=dataset_title)
+    scripts = current_app.config["SCRIPTS"]
+    return render_template("index.html", datasetTitle=dataset_title, SCRIPTS=scripts)
 
 
 @bp.route("/favicon.png")

server/cli/cli.py

@@ -5,7 +5,7 @@ from .prepare import prepare
 
 
 @click.group(name="cellxgene", context_settings=dict(max_content_width=85))
-@click.version_option(version="0.8.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
+@click.version_option(version="0.10.1", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
 def cli():
     pass
 

server/cli/launch.py

@@ -1,14 +1,21 @@
+import errno
 import logging
 from os import devnull
-from os.path import splitext, basename
+from os.path import splitext, basename, getsize
 import sys
 import warnings
 import webbrowser
 
 import click
 
+from server.app.app import Server
 from server.app.util.errors import ScanpyFileError
 from server.app.util.utils import custom_format_warning
+from server.utils.utils import find_available_port, is_port_available
+
+
+# anything bigger than this will generate a special message
+BIG_FILE_SIZE_THRESHOLD = 100 * 2**20  # 100MB
 
 
 @click.command()
@@ -16,10 +23,10 @@ from server.app.util.utils import custom_format_warning
 @click.option(
     "--layout",
     "-l",
-    type=click.Choice(["umap", "tsne", "draw_graph_fa", "draw_graph_fr", "diffmap", "phate"]),
-    default="umap",
+    default=[],
+    multiple=True,
     show_default=True,
-    help="Method for layout."
+    help="Layout name, eg, 'umap'."
 )
 @click.option(
     "--diffexp",
@@ -48,7 +55,8 @@ from server.app.util.utils import custom_format_warning
     show_default=True,
     help="Open the web browser after launch.",
 )
-@click.option("--port", "-p", help="Port to run server on.", metavar="", default=5005, show_default=True)
+@click.option("--port", "-p", help="Port to run server on, if not specified cellxgene will find an available port.",
+              metavar="", show_default=True)
 @click.option("--obs-names", default=None, metavar="", help="Name of annotation field to use for observations.")
 @click.option("--var-names", default=None, metavar="", help="Name of annotation to use for variables.")
 @click.option("--host", default="127.0.0.1", help="Host IP address")
@@ -65,20 +73,28 @@ from server.app.util.utils import custom_format_warning
     show_default=True,
     help="Relative expression cutoff used when selecting top N differentially expressed genes",
 )
+@click.option(
+    "--scripts",
+    default=[],
+    multiple=True,
+    help="Additional script files to include in html page",
+    show_default=True,
+)
 def launch(
-    data,
-    layout,
-    diffexp,
-    title,
-    verbose,
-    debug,
-    obs_names,
-    var_names,
-    open_browser,
-    port,
-    host,
-    max_category_items,
-    diffexp_lfc_cutoff,
+        data,
+        layout,
+        diffexp,
+        title,
+        verbose,
+        debug,
+        obs_names,
+        var_names,
+        open_browser,
+        port,
+        host,
+        max_category_items,
+        diffexp_lfc_cutoff,
+        scripts,
 ):
     """Launch the cellxgene data viewer.
     This web app lets you explore single-cell expression data.
@@ -105,6 +121,19 @@ def launch(
     else:
         warnings.formatwarning = custom_format_warning
 
+    if scripts:
+        click.echo(r"""
+/ / /\ \ \__ _ _ __ _ __ (_)_ __   __ _
+\ \/  \/ / _` | '__| '_ \| | '_ \ / _` |
+ \  /\  / (_| | |  | | | | | | | | (_| |
+  \/  \/ \__,_|_|  |_| |_|_|_| |_|\__, |
+                                  |___/
+The --scripts flag is intended for developers to include google analytics etc. You could be opening yourself to a
+security risk by including the --scripts flag. Make sure you trust the scripts that you are including.
+        """)
+        scripts_pretty = ", ".join(scripts)
+        click.confirm(f"Are you sure you want to inject these scripts: {scripts_pretty}?", abort=True)
+
     if not verbose:
         sys.tracebacklimit = 0
 
@@ -112,19 +141,36 @@ def launch(
         file_parts = splitext(basename(data))
         title = file_parts[0]
 
+    if port:
+        if debug:
+            raise click.ClickException("--port and --debug may not be used together (try --verbose for error logging).")
+        if not is_port_available(host, int(port)):
+            raise click.ClickException(
+                f"The port selected {port} is in use, please specify an open port using the --port flag."
+            )
+    else:
+        port = find_available_port(host)
+
     # Setup app
     cellxgene_url = f"http://{host}:{port}"
 
     # Import Flask app
-    from server.app.app import app
+    server = Server()
 
-    app.config.update(DATASET_TITLE=title)
+    server.create_app()
+    server.app.config.update(SCRIPTS=scripts)
 
     if not verbose:
         log = logging.getLogger("werkzeug")
         log.setLevel(logging.ERROR)
 
-    click.echo(f"[cellxgene] Loading data from {basename(data)}, this may take awhile...")
+    file_size = getsize(data)
+
+    # if a big file, let the user know it may take a while to load.
+    if file_size > BIG_FILE_SIZE_THRESHOLD:
+        click.echo(f"[cellxgene] Loading data from {basename(data)}, this may take awhile...")
+    else:
+        click.echo(f"[cellxgene] Loading data from {basename(data)}.")
 
     # Fix for anaconda python. matplotlib typically expects python to be installed as a framework TKAgg is usually
     # available and fixes this issue. See https://matplotlib.org/faq/virtualenv_faq.html
@@ -143,7 +189,7 @@ def launch(
     }
 
     try:
-        app.data = ScanpyEngine(data, args)
+        server.attach_data(ScanpyEngine(data, args), title=title)
     except ScanpyFileError as e:
         raise click.ClickException(f"{e}")
 
@@ -159,4 +205,9 @@ def launch(
         f = open(devnull, "w")
         sys.stdout = f
 
-    app.run(host=host, debug=debug, port=port, threaded=True)
+    try:
+        server.app.run(host=host, debug=debug, port=port, threaded=True, use_debugger=False)
+    except OSError as e:
+        if e.errno == errno.EADDRINUSE:
+            raise click.ClickException("Port is in use, please specify an open port using the --port flag.") from e
+        raise

server/cli/prepare.py

@@ -30,6 +30,10 @@ from scipy.sparse.csc import csc_matrix
 @click.option("--overwrite", default=False, is_flag=True, help="Allow file overwriting.", show_default=True)
 @click.option("--set-obs-names", default="", help="Named field to set as index for obs.", metavar="<name>")
 @click.option("--set-var-names", default="", help="Named field to set as index for var.", metavar="<name>")
+@click.option(
+    "--run-qc/--skip-qc", default=True, is_flag=True,
+    help="Whether to calculate QC metrics (saved to adata.obs and adata.var). \
+    See scanpy.pp.calculate_qc_metrics for details.", show_default=True)
 @click.option(
     "--make-obs-names-unique", default=True, is_flag=True, help="Ensure obs index is unique.", show_default=True
 )
@@ -46,6 +50,7 @@ def prepare(
     overwrite,
     set_obs_names,
     set_var_names,
+    run_qc,
     make_obs_names_unique,
     make_var_names_unique,
 ):
@@ -63,7 +68,7 @@ def prepare(
     import matplotlib
 
     matplotlib.use("Agg")
-    import scanpy.api as sc
+    import scanpy as sc
 
     # scanpy settings
     sc.settings.verbosity = 0
@@ -115,7 +120,11 @@ def prepare(
             click.echo("Warning: obs index is not unique")
         if not adata._var.index.is_unique:
             click.echo("Warning: var index is not unique")
+        return adata
 
+    def calculate_qc_metrics(adata):
+        if run_qc:
+            sc.pp.calculate_qc_metrics(adata, inplace=True)
         return adata
 
     def make_sparse(adata):
@@ -171,7 +180,12 @@ def prepare(
                 sc.pl.tsne(adata, color="louvain", palette=palette, save="_louvain")
 
     def show_step(item):
+        if run_qc:
+            qc_name = "Calculating QC metrics"
+        else:
+            qc_name = "Skipping QC"
         names = {
+            "calculate_qc_metrics": qc_name,
             "make_sparse": "Ensuring sparsity",
             "run_recipe": f'Running preprocessing recipe "{recipe}"',
             "run_pca": "Running PCA",
@@ -182,7 +196,7 @@ def prepare(
         if item is not None:
             return names[item.__name__]
 
-    steps = [make_sparse, run_recipe, run_pca, run_neighbors, run_louvain, run_layout]
+    steps = [calculate_qc_metrics, make_sparse, run_recipe, run_pca, run_neighbors, run_louvain, run_layout]
 
     click.echo(f"[cellxgene] Loading data from {data}, please wait...")
     adata = load_data(data)

server/gui/browser.py

@@ -0,0 +1,97 @@
+# flake8: noqa F403, F405
+from cefpython3 import cefpython as cef
+from PySide2.QtCore import *
+from PySide2.QtGui import *
+from PySide2.QtWidgets import *
+
+from server.gui.utils import WINDOWS, LINUX
+
+WindowUtils = cef.WindowUtils()
+
+# OS differences
+# noinspection PyUnresolvedReferences
+CefWidgetParent = QWidget
+if LINUX:
+    # noinspection PyUnresolvedReferences
+    CefWidgetParent = QX11EmbedContainer
+
+
+class CefWidget(CefWidgetParent):
+    def __init__(self, parent=None):
+        super(CefWidget, self).__init__(parent)
+        self.parent = parent
+        self.browser = None
+        # TODO test without this on linux
+        self.hidden_window = None  # Required for PyQt5 on Linux
+        self.show()
+
+    def focusInEvent(self, event):
+        # This event seems to never get called on Linux, as CEF is
+        # stealing all focus due to Issue #284.
+        if self.browser:
+            if WINDOWS:
+                WindowUtils.OnSetFocus(self.getHandle(), 0, 0, 0)
+            self.browser.SetFocus(True)
+
+    def focusOutEvent(self, event):
+        # This event seems to never get called on Linux, as CEF is
+        # stealing all focus due to Issue #284.
+        if self.browser:
+            self.browser.SetFocus(False)
+
+    def embedBrowser(self):
+        if LINUX:
+            self.hidden_window = QWindow()
+        window_info = cef.WindowInfo()
+        rect = [0, 0, self.width(), self.height()]
+        window_info.SetAsChild(self.getHandle(), rect)
+        # TODO better splash
+        self.browser = cef.CreateBrowserSync(window_info)
+
+    def getHandle(self):
+        if self.hidden_window:
+            # PyQt5 on Linux
+            return int(self.hidden_window.winId())
+        else:
+            return int(self.winId())
+
+    def moveEvent(self, _):
+        self.x = 0
+        self.y = 0
+        if self.browser:
+            if WINDOWS:
+                WindowUtils.OnSize(self.getHandle(), 0, 0, 0)
+            elif LINUX:
+                self.browser.SetBounds(self.x, self.y,
+                                       self.width(), self.height())
+            self.browser.NotifyMoveOrResizeStarted()
+
+    def resizeEvent(self, event):
+        size = event.size()
+        if self.browser:
+            if WINDOWS:
+                WindowUtils.OnSize(self.getHandle(), 0, 0, 0)
+            elif LINUX:
+                self.browser.SetBounds(self.x, self.y,
+                                       size.width(), size.height())
+            self.browser.NotifyMoveOrResizeStarted()
+
+
+class CefApplication(QApplication):
+    def __init__(self, args):
+        super(CefApplication, self).__init__(args)
+        if not cef.GetAppSetting("external_message_pump"):
+            self.timer = self.createTimer()
+
+    def createTimer(self):
+        timer = QTimer()
+        timer.timeout.connect(self.onTimer)
+        timer.start(10)
+        return timer
+
+    def onTimer(self):
+        cef.MessageLoopWork()
+
+    def stopTimer(self):
+        # Stop the timer after Qt's message loop has ended
+        self.timer.stop()

server/gui/main.py

@@ -0,0 +1,219 @@
+# flake8: noqa F403, F405
+from os.path import splitext, basename
+import sys
+import threading
+
+from cefpython3 import cefpython as cef
+from PySide2.QtCore import *
+from PySide2.QtWidgets import *
+
+from server.app.app import Server
+from server.gui.browser import CefWidget, CefApplication
+from server.gui.workers import DataLoadWorker, ServerRunWorker
+from server.gui.utils import WINDOWS, LINUX, MAC, FileLoadSignals
+from server.utils.constants import MODES
+
+
+# Configuration
+# TODO remember this or calculate it?
+WIDTH = 1024
+HEIGHT = 768
+
+# noinspection PyUnresolvedReferences
+class MainWindow(QMainWindow):
+    def __init__(self):
+        super(MainWindow, self).__init__(None)
+        self.cef_widget = None
+        self.data_widget = None
+        self.server = Server()
+        self.server.create_app()
+        self.runServer()
+        self.setWindowTitle("cellxgene")
+
+        # Strong focus - accepts focus by tab & click
+        self.setFocusPolicy(Qt.StrongFocus)
+        self.setupLayout()
+        self.setupMenu()
+
+    def setupLayout(self):
+        self.resize(WIDTH, HEIGHT)
+        self.cef_widget = CefWidget(self)
+        self.data_widget = LoadWidget(self)
+        self.stacked_layout = QStackedLayout()
+        self.stacked_layout.addWidget(self.data_widget)
+        self.stacked_layout.addWidget(self.cef_widget)
+        main_layout = QVBoxLayout()
+        main_layout.setContentsMargins(0, 0, 0, 0)
+        main_layout.setSpacing(0)
+        main_layout.addLayout(self.stacked_layout)
+        frame = QFrame()
+        frame.setLayout(main_layout)
+        self.setCentralWidget(frame)
+
+        if WINDOWS:
+            # On Windows with PyQt5 main window must be shown first
+            # before CEF browser is embedded, otherwise window is
+            # not resized and application hangs during resize.
+            self.show()
+
+        # Browser can be embedded only after layout was set up
+        self.cef_widget.embedBrowser()
+
+        if LINUX:
+            # On Linux with PyQt5 the QX11EmbedContainer widget is
+            # no longer available. An equivalent in Qt5 is to create
+            # a hidden window, embed CEF browser in it and then
+            # create a container for that hidden window and replace
+            # cef widget in the layout with the container.
+            self.container = QWidget.createWindowContainer(
+                self.cef_widget.hidden_window, parent=self)
+            stacked_layout.addWidget(self.container, 1, 0)
+
+    def setupMenu(self):
+        main_menu = self.menuBar()
+        file_menu = main_menu.addMenu('File')
+        load_action = QAction("Load file...", self)
+        load_action.setStatusTip("Load file")
+        load_action.setShortcut("Ctrl+O")
+        load_action.triggered.connect(self.showLoad)
+        file_menu.addAction(load_action)
+
+    def showLoad(self):
+        self.stacked_layout.setCurrentIndex(0)
+
+    def closeEvent(self, event):
+        # Close browser (force=True) and free CEF reference
+        if self.cef_widget.browser:
+            self.cef_widget.browser.CloseBrowser(True)
+            self.clearBrowserReferences()
+
+    def runServer(self):
+        worker = ServerRunWorker(self.server.app, host="127.0.0.1", port=8000)
+        self.httpd = threading.Thread(target=worker.run, daemon=True)
+        self.httpd.start()
+
+    def clearBrowserReferences(self):
+        # Clear browser references that you keep anywhere in your
+        # code. All references must be cleared for CEF to shutdown cleanly.
+        self.cef_widget.browser = None
+
+
+class LoadWidget(QFrame):
+    def __init__(self, parent):
+        super(LoadWidget, self).__init__(parent=parent)
+        # Init layout
+        self.MAX_CONTENT_WIDTH = 500
+        load_ui_layout = QVBoxLayout()
+        h_margin = (WIDTH - self.MAX_CONTENT_WIDTH) // 2
+        if h_margin < 10:
+            h_margin = 10
+        load_ui_layout.setContentsMargins(h_margin, 20, h_margin, 20)
+        logo_layout = QHBoxLayout()
+        logo_layout.setContentsMargins(0, 0, 0, 20)
+
+        load_layout = QGridLayout()
+        load_layout.setContentsMargins(0, 0, 0, 0)
+        load_layout.setSpacing(0)
+        message_layout = QHBoxLayout()
+        message_layout.setContentsMargins(0, 0, 0, 0)
+
+        self.title = ""
+        self.label = QLabel("cellxgene")
+        logo_layout.addWidget(self.label)
+
+        # UI section
+        # TODO add load spinner
+        # TODO add cancel button to send back to browser (if available)
+        self.embedding_label = QLabel("embedding: ")
+        load_layout.addWidget(self.embedding_label, 0, 0)
+        self.file_label = QLabel("file: ")
+        load_layout.addWidget(self.file_label, 0, 1)
+        self.embeddings = QComboBox(self)
+        self.embeddings.currentIndexChanged.connect(self.updateEmbedding)
+        self.embeddings.addItems(MODES)
+        self.embedding_selection = MODES[0]
+        load_layout.addWidget(self.embeddings, 1, 0)
+
+        self.load = QPushButton("Open...")
+        self.load.clicked.connect(self.onLoad)
+        load_layout.addWidget(self.load, 1, 1)
+
+        # Error section
+        self.error_label = QLabel("")
+        self.error_label.setWordWrap(True)
+        self.error_label.setFixedWidth(self.MAX_CONTENT_WIDTH)
+        message_layout.addWidget(self.error_label, alignment=Qt.AlignTop)
+
+        # Layout
+        for l in [logo_layout, load_layout, message_layout ]:
+            load_ui_layout.addLayout(l)
+
+        load_ui_layout.setStretch(2, 10)
+        self.setLayout(load_ui_layout)
+
+        self.signals = FileLoadSignals()
+        self.signals.selectedFile.connect(self.createScanpyEngine)
+
+    def updateEmbedding(self, idx):
+        self.embedding_selection = MODES[idx]
+
+    def createScanpyEngine(self, file_name):
+        worker = DataLoadWorker(file_name, self.embedding_selection)
+        worker.signals.result.connect(self.onDataSuccess)
+        worker.signals.error.connect(self.onDataError)
+        self.load_worker = threading.Thread(target=worker.run, daemon=True)
+        self.load_worker.start()
+
+    def onLoad(self):
+        options = QFileDialog.Options()
+        # options |= QFileDialog.DontUseNativeDialog
+        file_name, _ = QFileDialog.getOpenFileName(self,
+                                                  "Open H5AD File", "", "H5AD Files (*.h5ad)", options=options)
+        self.title = splitext(basename(file_name))[0]
+        if file_name:
+            self.signals.selectedFile.emit(file_name)
+
+
+    def onDataSuccess(self, data):
+        self.window().server.attach_data(data, self.title)
+        self.navigateToLocation()
+        # Reveal browser
+        self.window().stacked_layout.setCurrentIndex(1)
+
+    def onDataError(self, err):
+        self.error_label.setText(f"Error: {err}")
+        self.error_label.resize(self.MAX_CONTENT_WIDTH, self.error_label.height())
+
+    def navigateToLocation(self, location="http://localhost:8000/"):
+        self.window().cef_widget.browser.Navigate(location)
+
+
+def main():
+    # This generates an error.log file on error
+    sys.excepthook = cef.ExceptHook  # To shutdown all CEF processes on error
+    settings = {}
+    # Instead of timer loop
+    if MAC:
+        settings["external_message_pump"] = True
+
+    # Create and launch cef browser and qt window
+    cef.Initialize(settings)
+    app = CefApplication(sys.argv)
+    main_window = MainWindow()
+    main_window.show()
+    main_window.activateWindow()
+    main_window.raise_()
+    app.exec_()
+
+    # Clean up on close
+    if not cef.GetAppSetting("external_message_pump"):
+        app.stopTimer()
+    # TODO clean up threads when we switch threading model
+    del main_window  # Just to be safe, similarly to "del app"
+    del app  # Must destroy app object before calling Shutdown
+    cef.Shutdown()
+    sys.exit(0)
+
+
+if __name__ == '__main__':
+    main()

server/gui/utils.py

@@ -0,0 +1,25 @@
+import platform
+
+from PySide2.QtCore import QObject, Signal
+
+# Detect OS
+WINDOWS = (platform.system() == "Windows")
+LINUX = (platform.system() == "Linux")
+MAC = (platform.system() == "Darwin")
+
+
+class WorkerSignals(QObject):
+    """
+    Defines the signals available from a running worker thread.
+    Supported signals are:
+        finished
+        error - `str` error message
+        result - `object` data returned from processing, anything
+    """
+    finished = Signal()
+    error = Signal(str)
+    result = Signal(object)
+
+
+class FileLoadSignals(QObject):
+    selectedFile = Signal(str)

server/gui/workers.py

@@ -0,0 +1,47 @@
+import traceback
+
+from server.gui.utils import WorkerSignals
+
+
+class DataLoadWorker():
+    def __init__(self, data_file, layout="umap", *args, **kwargs):
+        super(DataLoadWorker, self).__init__()
+        self.data_file = data_file
+        self.layout = layout
+        self.signals = WorkerSignals()
+
+    def run(self):
+        if not self.data_file:
+            self.signals.finished.emit()
+            return
+
+        # delayed import to speed load
+        from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
+        args = {
+            "layout": self.layout,
+            "diffexp": "ttest",
+            "max_category_items": 100,
+            "diffexp_lfc_cutoff": 0.01,
+            "obs_names": None,
+            "var_names": None,
+        }
+        try:
+            data_results = ScanpyEngine(self.data_file, args)
+        except Exception as e:
+            traceback.print_exc()
+            self.signals.error.emit(str(e))
+        else:
+            self.signals.result.emit(data_results)
+        finally:
+            self.signals.finished.emit()
+
+
+class ServerRunWorker():
+    def __init__(self, app, host, port, *args, **kwargs):
+        super(ServerRunWorker, self).__init__()
+        self.app = app
+        self.host = host
+        self.port = port
+
+    def run(self):
+        self.app.run(host=self.host, debug=False, port=self.port, threaded=True)

server/requirements.txt

@@ -1,4 +1,4 @@
-anndata>=0.6.13
+anndata>=0.6.20
 click>=6.7
 Flask>=1.0.2
 Flask-Caching>=1.4.0
@@ -9,7 +9,7 @@ flatbuffers>=1.10.0
 matplotlib>=2.2
 numpy>=1.15.2
 pandas>=0.23.1
-scanpy>=1.3.2
+scanpy>=1.3.7
 scipy>=1.1.0
 scikit-learn>=0.19.1,!=0.20.0
-tables>=3.5.1
+tables==3.5.1

server/test/schema.json

@@ -5,46 +5,61 @@
     "type": "float32"
   },
   "annotations": {
+    "obs": {
+      "index": "name_0",
+      "columns": [
+        {
+          "name": "name_0",
+          "type": "string"
+        },
+        {
+          "name": "n_genes",
+          "type": "int32"
+        },
+        {
+          "name": "percent_mito",
+          "type": "float32"
+        },
+        {
+          "name": "n_counts",
+          "type": "float32"
+        },
+        {
+          "name": "louvain",
+          "type": "categorical",
+          "categories": [
+            "CD4 T cells",
+            "CD14+ Monocytes",
+            "B cells",
+            "CD8 T cells",
+            "NK cells",
+            "FCGR3A+ Monocytes",
+            "Dendritic cells",
+            "Megakaryocytes"
+          ]
+        }
+      ]
+    },
+    "var": {
+      "index": "name_0",
+      "columns": [
+        {
+          "name": "name_0",
+          "type": "string"
+        },
+        {
+          "name": "n_cells",
+          "type": "int32"
+        }
+      ]
+    }
+  },
+  "layout": {
     "obs": [
       {
-        "name": "name",
-        "type": "string"
-      },
-      {
-        "name": "n_genes",
-        "type": "int32"
-      },
-      {
-        "name": "percent_mito",
-        "type": "float32"
-      },
-      {
-        "name": "n_counts",
-        "type": "float32"
-      },
-      {
-        "name": "louvain",
-        "type": "categorical",
-        "categories": [
-          "CD4 T cells",
-          "CD14+ Monocytes",
-          "B cells",
-          "CD8 T cells",
-          "NK cells",
-          "FCGR3A+ Monocytes",
-          "Dendritic cells",
-          "Megakaryocytes"
-        ]
-      }
-    ],
-    "var": [
-      {
-        "name": "name",
-        "type": "string"
-      },
-      {
-        "name": "n_cells",
-        "type": "int32"
+        "name": "umap",
+        "type": "float32",
+        "dims": ["umap_0", "umap_1"]
       }
     ]
   }

server/test/test_api.py

@@ -19,11 +19,12 @@ class EndPoints(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
-        cls.ps = Popen(["cellxgene", "launch", "example-dataset/pbmc3k.h5ad", "--debug"])
+        cls.ps = Popen(["cellxgene", "launch", "example-dataset/pbmc3k.h5ad", "--verbose", "--port", "5005"])
         session = requests.Session()
         for i in range(90):
             try:
-                session.get(f"{URL_BASE}schema")
+                result = session.get(f"{URL_BASE}schema")
+                cls.schema = result.json()
             except requests.exceptions.ConnectionError:
                 time.sleep(1)
 
@@ -45,7 +46,8 @@ class EndPoints(unittest.TestCase):
         self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["schema"]["dataframe"]["nObs"], 2638)
-        self.assertEqual(len(result_data["schema"]["annotations"]["obs"]), 5)
+        self.assertEqual(len(result_data["schema"]["annotations"]["obs"]), 2)
+        self.assertEqual(len(result_data["schema"]["annotations"]["obs"]["columns"]), 5)
 
     def test_config(self):
         endpoint = "config"
@@ -67,9 +69,11 @@ class EndPoints(unittest.TestCase):
         self.assertEqual(result.headers["Content-Type"], "application/octet-stream")
         df = decode_fbs.decode_matrix_FBS(result.content)
         self.assertEqual(df['n_rows'], 2638)
-        self.assertEqual(df['n_cols'], 2)
+        self.assertEqual(df['n_cols'], 8)
         self.assertIsNotNone(df['columns'])
-        self.assertIsNone(df['col_idx'])
+        self.assertListEqual(df['col_idx'], [
+            'pca_0', 'pca_1', 'tsne_0', 'tsne_1', 'umap_0', 'umap_1', 'draw_graph_fr_0', 'draw_graph_fr_1'
+        ])
         self.assertIsNone(df['row_idx'])
         self.assertEqual(len(df['columns']), df['n_cols'])
 
@@ -93,7 +97,8 @@ class EndPoints(unittest.TestCase):
         self.assertIsNotNone(df['col_idx'])
         self.assertIsNone(df['row_idx'])
         self.assertEqual(len(df['columns']), df['n_cols'])
-        self.assertListEqual(df['col_idx'], ['name', 'n_genes', 'percent_mito', 'n_counts', 'louvain'])
+        obs_index_col_name = self.schema["schema"]["annotations"]["obs"]["index"]
+        self.assertListEqual(df['col_idx'], [obs_index_col_name, 'n_genes', 'percent_mito', 'n_counts', 'louvain'])
 
     def test_get_annotations_obs_keys_fbs(self):
         endpoint = "annotations/obs"
@@ -163,7 +168,8 @@ class EndPoints(unittest.TestCase):
         self.assertIsNotNone(df['col_idx'])
         self.assertIsNone(df['row_idx'])
         self.assertEqual(len(df['columns']), df['n_cols'])
-        self.assertListEqual(df['col_idx'], ['name', 'n_cells'])
+        var_index_col_name = self.schema["schema"]["annotations"]["var"]["index"]
+        self.assertListEqual(df['col_idx'], [var_index_col_name, 'n_cells'])
 
     def test_get_annotations_var_keys_fbs(self):
         endpoint = "annotations/var"
@@ -245,7 +251,8 @@ class EndPoints(unittest.TestCase):
         endpoint = f"data/var"
         url = f"{URL_BASE}{endpoint}"
         header = {"Accept": "application/octet-stream"}
-        var_filter = {"filter": {"var": {"annotation_value": [{"name": "name", "values": ["RER1"]}]}}}
+        index_col_name = self.schema["schema"]["annotations"]["var"]["index"]
+        var_filter = {"filter": {"var": {"annotation_value": [{"name": index_col_name, "values": ["RER1"]}]}}}
         result = self.session.put(url, headers=header, json=var_filter)
         self.assertEqual(result.status_code, HTTPStatus.OK)
         self.assertEqual(result.headers["Content-Type"], "application/octet-stream")