bump version to 0.11.0 (#850)

* bump version to 0.11.0

* manually fix version number

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.8.0
+current_version = 0.11.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"
@@ -13,3 +13,7 @@ replace = version="{new_version}"
 search = "version": "{current_version}"
 replace = "version": "{new_version}"
 
+[bumpversion:file:server/__init__.py]
+search = "__version__ = "{current_version}"
+replace = "__version__ = "{new_version}"
+

.gitignore

@@ -27,6 +27,9 @@ server/app/web/templates/index\.html
 .ipynb_checkpoints
 *.ipynb
 
+# cefpython
+error.log
+
 # misc
 .DS_Store
 npm-debug.log

.travis.yml

@@ -4,14 +4,14 @@ sudo: required
 node_js:
   - 8
 cache:
-  pip: true
+  - pip
+  - npm
 install:
   - set -eo pipefail
   - pip install flake8
-  - make build
-  - make install
+  - make pydist
+  - make install-dist
   - pip install -r server/requirements-dev.txt
-  - docker build .
 
 jobs:
   include:
@@ -21,8 +21,11 @@ jobs:
   - name: "Branch Tests 3.6"
     python: "3.6"
     script: ./travis-build.sh
+  - name: "Docker Build"
+    install: skip
+    python: "3.6"
+    script: docker build .
   - name: "Smoke Tests"
     python: "3.6"
-    if: branch = master AND type = cron
     script:
       - npm run --prefix client/ smoke-test

CONTRIBUTING.md

@@ -0,0 +1,49 @@
+# contributing to cellxgene
+
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress.
+
+All code contributions and dependencies must be compatible with the project's [open-source license (MIT)](LICENSE.txt).
+
+This project adheres to the Contributor Covenant
+[code of conduct](https://github.com/chanzuckerberg/.github/tree/master/CODE_OF_CONDUCT.md).
+By participating, you are expected to uphold this code. Please report unacceptable behavior
+to opensource@chanzuckerberg.com.
+
+And finally, if you have any questions about any of this stuff, just ask! :)
+
+## developer guide
+
+This project has made a few key design choices
+
+- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
+- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
+- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
+
+Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
+
+If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
+
+- python 3.6+
+- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
+
+Then clone the project
+
+```
+git clone https://github.com/chanzuckerberg/cellxgene.git
+```
+
+Build the client web assets by calling `make` from inside the `cellxgene` folder
+
+```
+make
+```
+
+Install all requirements (we recommend doing this inside a virtual environment)
+
+```
+pip install -e .
+```
+
+You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
+
+If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.

README.md

@@ -2,274 +2,77 @@
 
 > an interactive explorer for single-cell transcriptomics data
 
-`cellxgene` is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
+[![DOI](https://zenodo.org/badge/105615409.svg)](https://zenodo.org/badge/latestdoi/105615409)
+
+_cellxgene_ (pronounced "sell-by-jean") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
 
 <img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
 
-## getting started
+- Want to install and use cellxgene? Visit the [cellxgene docs](https://chanzuckerberg.github.io/cellxgene/).
+- Want to see where we are going? Check out [our roadmap](ROADMAP.md).
+- Want to contribute? See our [contributors guide](CONTRIBUTING.md).
 
-You'll need **python 3.6** and **Google Chrome**. (_Warning_: Python 3.7 is **not** supported at this time)
-The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).
+## quick start
 
-To install run
+To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](https://chanzuckerberg.github.io/cellxgene/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
 
-```
+Install the package.
+``` bash
 pip install cellxgene
 ```
 
-To start exploring a dataset call
+Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
 
-```
-cellxgene launch dataset.h5ad --open
+``` bash
+curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
 ```
 
-If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
-
-```
+Launch _cellxgene_
+``` bash
 cellxgene launch pbmc3k.h5ad --open
 ```
 
-You should see your web browser open with the following
+To learn more about what you can do with _cellxgene_, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-started.html) guide.
 
-<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+## get in touch
 
-**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
-
-There are several options available, such as:
-
-- `--layout` to specify the layout as `tsne`, `umap`, `diffmap`, `phate`, `draw_graph_fa`, or `draw_graph_fr`
-- `--title` to show a title on the explorer
-- `--open` to automatically open the web browser after launching (OS X only)
-
-To see all options call
-
-```
-cellxgene launch --help
-```
-
-There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
-
-## data formatting
-
-### assumptions
-
-The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
-
-- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
-- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
-- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_umap` for the `umap` layout)
-- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
-
-### prepare
-
-The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
-
-To prepare from an existing `.h5ad` file use
-
-```
-cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
-```
-
-This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
-
-Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
-
-```
-cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
-```
-
-To see all options call
-
-```
-cellxgene prepare --help
-```
-
-**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
-
-```
-pip install cellxgene[louvain]
-```
-
-If the aforementioned optional package installation fails, you can also install these packages directly:
-
-```
-pip install python-igraph louvain>=0.6
-```
-
-## conda and virtual environments
-
-If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
-
-```
-conda create --yes -n cellxgene python=3.6
-conda activate cellxgene
-pip install cellxgene
-```
-
-Or you can create a virtual environment by using
-
-```
-ENV_NAME=cellxgene
-python3.6 -m venv ${ENV_NAME}
-source ${ENV_NAME}/bin/activate
-pip install cellxgene
-```
-
-## docker
-
-We have included a dockerfile to conveniently run cellxgene from docker.
-
-1. Build the image `docker build . -t cellxgene`
-2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
-   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
-
-## FAQ
-
-<details>
-
-<summary> questions about data formatting </summary>
-  
-<hr>
-  
-> Someone sent me a directory of `10X-Genomics` data with a `mtx` file and I've never used `scanpy`, can I use `cellxgene`?
-
-Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
-
-```
-cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
-```
-
-Depending on the size of the dataset, this may take some time. Once it's done, call
-
-```
-cellxgene launch data-processed.h5ad --layout=umap --open
-```
-
-And your web browser should open with an interactive view of your data.
-
-<hr>
-
-> In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
-
-Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using
-
-```
-pip install cellxgene[louvain]
-```
-
-<hr>
-
-> I ran `prepare` and I'm getting results that look unexpected
-
-You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
-
-```
-cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
-```
-
-It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
-
-<hr>
-
-> I have extra metadata that I want to add to my dataset
-
-Currently this is not supported directly, but you should be able to do this manually using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
-
-<hr>
-
-> What part of the anndata objects does cellxgene pull in for visualization?
-
-- `.obs` and `.var` annotations are use to extract metadata for filtering
-- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
-- `.obsm` is used for layout
-
-</details>
-
-<details>
-  
-<summary> questions about installing and building </summary>
-
-<hr>
-
-> I tried to `pip install cellxgene` and got a weird error about missing paths to an HDF5 library?  
-
-You probably just need to install HDF5 first. If you're on a mac, you can simply `brew install hdf5` and then try `pip install cellxgene` again.  
-
-> I tried to `pip install cellxgene` and got another weird error I don't understand  
-
-This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.  
-
-<hr>
-
-> I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
-
-This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
-
-</details>
-
-<details>
-
-<summary> questions about algorithms </summary>
-
-<hr>
-
-> How are you computing and sorting differential expression results?
-
-Currently we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
-
-</details>
-
-## developer guide
-
-This project has made a few key design choices
-
-- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
-- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
-- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
-
-Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
-
-If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
-
-- python 3.6
-- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
-
-Then clone the project
-
-```
-git clone https://github.com/chanzuckerberg/cellxgene.git
-```
-
-Build the client web assets by calling `make` from inside the `cellxgene` folder
-
-```
-make
-```
-
-Install all requirements (we recommend doing this inside a virtual environment)
-
-```
-pip install -e .
-```
-
-You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
-
-If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.
-
-## development roadmap
-
-`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact)
-
-- **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
-- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene` to render the results of these analyses when they have been performed
-- **Deploy to web** Many projects release public data browser websites alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
-- **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. Have feature requests or bugs? Please submit these as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
 
 ## contributing
 
-We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
+We warmly welcome contributions from the community! Please see our [contributing guide](CONTRIBUTING.md) and don't hesitate to open an issue or send a pull request to improve cellxgene.
 
-## inspiration and collaboration
+This project adheres to the Contributor Covenant [code of conduct](https://github.com/chanzuckerberg/.github/blob/master/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.
+
+## where we are going
+
+Our goal is to enable teams of computational and experimental
+biologists to collaboratively gain insight into their single-cell RNA-seq data.
+
+There are 4 key features we plan to implement in the near term.
+
+- Click install and launch
+- Manual annotation workflows
+- Toggle embeddings
+- Gene information
+
+For more detail on these features and where we are going, see [our roadmap](ROADMAP.md).
+
+## risks of hosting cellxgene
+
+_cellxgene_ is built on standard web technologies, but is currently designed as a single-user desktop application.
+We've done this so we can prioritize the features on [our roadmap](ROADMAP.md).
+
+Some of our users have experimented with hosting _cellxgene_ either for their lab or for public use, but please note that the _cellxgene_ team does not officially support, troubleshoot, or maintain any web deployments at this time.
+
+If do you choose setup _cellxgene_ as a hosted service, you should be aware of the following risks:
+
+- `$ cellxgene launch` uses Flask's development server, which is not recommended for hosted deployment (see the [Flask documentation](http://flask.pocoo.org/docs/1.0/tutorial/deploy/#run-with-a-production-server))
+- We have no testing or official support for deployments where multiple users are accessing the same _cellxgene_ instance.
+- Your _cellxgene_ instance is likely to hang or crash if too many people access it at the same time, especially if they using functions that call the Python backend (such as differential expression, updating the layout, or coloring by gene).
+- _cellxgene_ only supports one instance per dataset
+
+## inspiration
 
 We've been heavily inspired by several other related single-cell visualization projects, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), and many others. We hope to explore collaborations where useful as this community works together on improving interactive visualization for single-cell data.
 
@@ -279,9 +82,13 @@ We have been working closely with the [`scanpy`](https://github.com/theislab/sca
 
 We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
 
-## help and contact
+## core team
 
-Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+- Colin Megill, frontend & product design
+- Charlotte Weaver, software engineer
+- Bruce Martin, software engineer
+- Sidney Bell, computational biologist
+- Justin Kiggins, product manager
 
 ## reuse
 

ROADMAP.md

@@ -0,0 +1,46 @@
+# cellxgene roadmap
+
+cellxgene makes it easier for biologists to collaboratively explore and understand their single-cell RNA-seq data. 
+In the near term, we are focused on continuing to enable fast, interactive exploration of single-cell data, supporting collaborative workflows in single-cell analysis, and improving user support. 
+If you have questions or feedback about this roadmap, please submit an issue on GitHub.
+Please note: this roadmap is subject to change.
+
+Last updated: June 25, 2019
+
+## Fast, interactive exploration of single-cell data
+
+### Exposing Relationships Between Metadata and Data
+Biologists need to understand how variables (stored in metadata) are associated with one another and how they relate to changes in gene expression. 
+Building upon visualization features that reveal categorical metadata relationships (cluster occupancy) and gene expression relationships (scatterplot), we plan to add exploratory visualization components that enable investigation of relationships between metadata and gene expression. 
+See [issue #616](https://github.com/chanzuckerberg/cellxgene/issues/616) for more details.
+
+### Contextualizing Genes
+While exploring a transcriptomics dataset, scientists need to understand the biological context of genes.
+This context may be provided by user-defined gene metadata or publicly available gene databases. 
+We plan to support augmenting gene names with additional information that is useful to biologists. 
+See [issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for more detail.
+
+## Support collaborative workflows in single-cell analysis
+
+### Manual Annotations
+cellxgene offers exploratory visualizations that are critical for manual annotation workflows, especially in collaborative environments. 
+We plan to support manually annotating cells with labels (i.e., cell type or QC flags), and their easy export for downstream analysis. 
+See [issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524) for more details.
+
+### Simple Click to Launch
+
+Many biologists prefer not to interact with the command line and need an OS-native experience when using cellxgene. 
+We plan to implement a point-and-click installation and launch experience so that users can easily load data into cellxgene. 
+See [issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for details.
+
+### Python API
+For computational biologists, saving h5ad files then loading them into cellxgene is a point of friction.  
+We plan to support importing cellxgene as a Python package so that users can launch cellxgene directly from an interactive environment (such as Jupyter, IPython, or Spyder), and pass data to and from the cellxgene UI. 
+
+## Improving user support
+
+### Improved documentation
+cellxgene has some specific expectations about how data is stored. 
+We want to ensure that new users can get started easily and learn how to use cellxgene with their own data. 
+We plan to improve documentation on getting started, installation, data, and contributing. 
+See [issue #533](https://github.com/chanzuckerberg/cellxgene/issues/533) for more details.

cellxgene-osx.spec

@@ -0,0 +1,41 @@
+# -*- mode: python ; coding: utf-8 -*-
+
+block_cipher = None
+
+
+a = Analysis(['server/gui/main.py'],
+             pathex=['/Users/charlotteweaver/Documents/Git/cellxgene'],
+             binaries=[('/System/Library/Frameworks/Tk.framework/Tk', 'tk'), ('/System/Library/Frameworks/Tcl.framework/Tcl', 'tcl')],
+             datas=[('server/app/web/templates/', 'server/app/web/templates/'), ('server/app/web/static/', 'server/app/web/static/')],
+             hiddenimports=['sklearn', 'sklearn.utils._cython_blas', 'sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree', 'sklearn.tree._utils'],
+             hookspath=['server/gui/'],
+             runtime_hooks=[],
+             excludes=[],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+exe = EXE(pyz,
+          a.scripts,
+          [],
+          exclude_binaries=True,
+          name='cellxgene',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          console=False , icon='server/gui/images/cxg_icons.icns')
+coll = COLLECT(exe,
+               a.binaries,
+               a.zipfiles,
+               a.datas,
+               strip=False,
+               upx=True,
+               upx_exclude=[],
+               name='cellxgene')
+app = BUNDLE(coll,
+             name='cellxgene.app',
+             icon='server/gui/images/cxg_icons.icns',
+             bundle_identifier=None)

cellxgene-windows.spec

@@ -0,0 +1,36 @@
+# -*- mode: python -*-
+
+block_cipher = None
+
+
+a = Analysis(['server\\gui\\main.py'],
+             pathex=['C:\\Users\\Charlotte\\Documents\\git\\cellxgene'],
+             binaries=[],
+             datas=[('server/app/web/templates/', 'server/app/web/templates'), ('server/app/web/static/', 'server/app/web/static')],
+             hiddenimports=[],
+             hookspath=['server/gui/'],
+             runtime_hooks=[],
+             excludes=[],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+exe = EXE(pyz,
+          a.scripts,
+          [],
+          exclude_binaries=True,
+          name='cellxgene',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          console=False , icon='server\\gui\\images\\icon.ico')
+coll = COLLECT(exe,
+               a.binaries,
+               a.zipfiles,
+               a.datas,
+               strip=False,
+               upx=True,
+               name='cellxgene')

client/__tests__/e2e/data.js

@@ -1,6 +1,6 @@
 export const datasets = {
   pbmc3k: {
-    title: "cellxgene: pbmc3k",
+    title: "pbmc3k",
     dataframe: {
       nObs: "2638",
       nVar: "1838",
@@ -26,8 +26,8 @@ export const datasets = {
     cellsets: {
       lasso: [
         {
-          "coordinates-as-percent": { x1: 0.25, y1: 0.25, x2: 0.35, y2: 0.35 },
-          count: "26"
+          "coordinates-as-percent": { x1: 0.05, y1: 0.25, x2: 0.15, y2: 0.35 },
+          count: "71"
         }
       ],
       categorical: [
@@ -91,8 +91,8 @@ export const datasets = {
         }
       },
       lasso: {
-        "coordinates-as-percent": { x1: 0.45, y1: 0.45, x2: 0.5, y2: 0.5 },
-        count: "67"
+        "coordinates-as-percent": { x1: 0.45, y1: 0.05, x2: 0.65, y2: 0.15 },
+        count: "36"
       }
     },
     scatter: {
@@ -100,6 +100,23 @@ export const datasets = {
     },
     pan: {
       "coordinates-as-percent": { x1: 0.75, y1: 0.75, x2: 0.35, y2: 0.35 }
+    },
+    features: {
+      panzoom: {
+        lasso: {
+          "coordinates-as-percent": { x1: 0.3, y1: 0.3, x2: 0.5, y2: 0.5 },
+          count: "24"
+        }
+      }
+    },
+    clip: {
+      min: "30",
+      max: "70",
+      metadata: "n_genes",
+      gene: "S100A8",
+      "coordinates-as-percent": { x1: 0.25, y1: 0.5, x2: 0.55, y2: 0.5 },
+      count: "392",
+      "gene-cell-count": "421"
     }
   }
 };

client/__tests__/e2e/e2e.test.js

@@ -1,3 +1,8 @@
+/*
+Smoke test suite that will be run in Travis CI
+
+Tests included in this file are expected to be relatively stable and test core features
+ */
 import puppeteer from "puppeteer";
 import { appUrlBase, DEBUG, DEV, DATASET } from "./config";
 import { puppeteerUtils, cellxgeneActions } from "./puppeteerUtils";
@@ -20,7 +25,17 @@ beforeAll(async () => {
   page = await browser.newPage();
   await page.setViewport(browserViewport);
   if (DEV || DEBUG) {
-    page.on("console", msg => console.log(`PAGE LOG: ${msg.text()}`));
+    page.on("console", async msg => {
+      // If there is a console.error but an error is not thrown, this will ensure the test fails
+      if (msg.type() === "error") {
+        const errorMsgText = await Promise.all(
+          // TODO can we do this without internal properties?
+          msg.args().map(arg => arg._remoteObject.description)
+        );
+        throw new Error(`Console error: ${errorMsgText}`);
+      }
+      console.log(`PAGE LOG: ${msg.text()}`);
+    });
   }
   page.on("pageerror", err => {
     throw new Error(`Console error: ${err}`);
@@ -39,14 +54,14 @@ afterAll(() => {
   }
 });
 
-describe("did launch", async () => {
+describe("did launch", () => {
   test("page launched", async () => {
     let el = await utils.getOneElementInnerHTML("[data-testid='header']");
     expect(el).toBe(data.title);
   });
 });
 
-describe("metadata loads", async () => {
+describe("metadata loads", () => {
   test("categories and values from dataset appear", async () => {
     for (const label in data.categorical) {
       await utils.waitByID(`category-${label}`);
@@ -72,7 +87,7 @@ describe("metadata loads", async () => {
   });
 });
 
-describe("cell selection", async () => {
+describe("cell selection", () => {
   test("selects all cells cellset 1", async () => {
     const cellCount = await cxgActions.cellSet(1);
     expect(cellCount).toBe(data.dataframe.nObs);
@@ -123,7 +138,7 @@ describe("cell selection", async () => {
   });
 });
 
-describe("gene entry", async () => {
+describe("gene entry", () => {
   test("search for single gene", async () => {
     // blueprint's  typeahead is treating typing weird, clicking & waiting first solves this
     await utils.typeInto("gene-search", data.genes.search);
@@ -139,14 +154,17 @@ describe("gene entry", async () => {
     await utils.clickOn("section-bulk-add");
     await utils.typeInto("input-bulk-add", testGenes.join(","));
     await page.keyboard.press("Enter");
-    const userGeneHist = await cxgActions.getAllHistograms(
-      "histogram-user-gene"
+
+    const allHistograms = await cxgActions.getAllHistograms(
+      "histogram-user-gene",
+      testGenes
     );
-    expect(userGeneHist).toEqual(expect.arrayContaining(testGenes));
+    expect(allHistograms).toEqual(expect.arrayContaining(testGenes));
+    expect(allHistograms.length).toEqual(testGenes.length);
   });
 });
 
-describe("diffexp", async () => {
+describe("diffexp", () => {
   test("selects cells, saves them and performs diffexp", async () => {
     for (const select of data.diffexp.cellset1) {
       if (select.kind === "categorical") {
@@ -161,15 +179,18 @@ describe("diffexp", async () => {
     }
     await cxgActions.cellSet(2);
     await utils.clickOn("diffexp-button");
-    const diffExpHists = await cxgActions.getAllHistograms("histogram-diffexp");
-    expect(diffExpHists).toEqual(
+    const allHistograms = await cxgActions.getAllHistograms(
+      "histogram-diffexp",
+      data.diffexp["gene-results"]
+    );
+    expect(allHistograms).toEqual(
       expect.arrayContaining(data.diffexp["gene-results"])
     );
+    expect(allHistograms.length).toEqual(data.diffexp["gene-results"].length);
   });
 });
-//
 
-describe("subset/reset", async () => {
+describe("subset/reset", () => {
   test("subset - cell count matches", async () => {
     for (const select of data.subset.cellset1) {
       if (select.kind === "categorical") {
@@ -243,7 +264,7 @@ describe("subset/reset", async () => {
   });
 });
 
-describe("scatter plot", async () => {
+describe("scatter plot", () => {
   test("scatter plot appears", async () => {
     await cxgActions.reset();
     const testGenes = data.scatter.genes;
@@ -256,8 +277,37 @@ describe("scatter plot", async () => {
   });
 });
 
+describe("clipping", () => {
+  test("clip continuous", async () => {
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.metadata}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip.count);
+  });
+
+  test("clip gene", async () => {
+    await utils.typeInto("gene-search", data.clip.gene);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(`[data-testid='histogram-${data.clip.gene}']`);
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.gene}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip["gene-cell-count"]);
+  });
+});
+
 // interact with UI elements just that they do not break
-describe("ui elements don't error", async () => {
+describe("ui elements don't error", () => {
   test("color by", async () => {
     for (const label in data.categorical) {
       await utils.clickOn(`colorby-${label}`);
@@ -267,6 +317,15 @@ describe("ui elements don't error", async () => {
     }
   });
 
+  test("color by for gene", async () => {
+    await utils.typeInto("gene-search", data.genes.search);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(
+      `[data-testid='histogram-${data.genes.search}']`
+    );
+    await utils.clickOn(`colorby-${data.genes.search}`);
+  });
+
   test("pan and zoom", async () => {
     await utils.clickOn("mode-pan-zoom");
     const panCoords = await cxgActions.calcDragCoordinates(

client/__tests__/e2e/feature.test.js

@@ -0,0 +1,118 @@
+/*
+NOT run in Travis CI
+
+UX tests using puppeteer to be run locally.
+
+To run locally, ensure you are running the client is running on port 3000.
+Then run jest --verbose false --config __tests__/e2e/e2eJestConfig.json feature.
+ */
+
+import puppeteer from "puppeteer";
+import { appUrlBase, DEBUG, DEV, DATASET } from "./config";
+import { puppeteerUtils, cellxgeneActions } from "./puppeteerUtils";
+import { datasets } from "./data";
+
+let browser, page, utils, cxgActions, spy;
+const browserViewport = { width: 1280, height: 960 };
+let data = datasets[DATASET].features;
+
+if (DEBUG) jest.setTimeout(100000);
+if (DEV) jest.setTimeout(10000);
+
+beforeAll(async () => {
+  const browserParams = DEV
+    ? { headless: false, slowMo: 5 }
+    : DEBUG
+    ? { headless: false, slowMo: 100, devtools: true }
+    : {};
+  browser = await puppeteer.launch(browserParams);
+  page = await browser.newPage();
+  await page.setViewport(browserViewport);
+  if (DEV || DEBUG) {
+    page.on("console", msg => console.log(`PAGE LOG: ${msg.text()}`));
+  }
+  page.on("pageerror", err => {
+    throw new Error(`Console error: ${err}`);
+  });
+  utils = puppeteerUtils(page);
+  cxgActions = cellxgeneActions(page);
+});
+
+beforeEach(async () => {
+  await page.goto(appUrlBase);
+});
+
+afterAll(() => {
+  if (!DEBUG) {
+    browser.close();
+  }
+});
+
+describe("zoom interaction", async () => {
+  // Skip this test since UI is to hide lasso path when switching modes
+  test.skip("lasso visible after switching modes to pan/zoom", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    await utils.clickOn("mode-pan-zoom");
+    await utils.waitByID("lasso-element", { visible: true });
+  });
+
+  test("pan zoom mode resets lasso selection", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    const initialCount = await cxgActions.cellSet(1);
+    expect(initialCount).toBe(data.panzoom.lasso.count);
+    await utils.clickOn("mode-pan-zoom");
+    await utils.clickOn("mode-lasso");
+    const modeSwitchCount = await cxgActions.cellSet(1);
+    expect(modeSwitchCount).toBe(initialCount);
+  });
+
+  test("lasso moves after pan", async () => {
+    const lassoSelection = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      lassoSelection.start,
+      lassoSelection.end,
+      true
+    );
+    await utils.waitByID("lasso-element", { visible: true });
+    const initialCount = await cxgActions.cellSet(1);
+    expect(initialCount).toBe(data.panzoom.lasso.count);
+    await utils.clickOn("mode-pan-zoom");
+    const panCoords = await cxgActions.calcDragCoordinates(
+      "layout-graph",
+      data.panzoom.lasso["coordinates-as-percent"]
+    );
+    await cxgActions.drag(
+      "layout-graph",
+      panCoords.start,
+      panCoords.end,
+      false
+    );
+    await utils.clickOn("mode-lasso");
+    const panCount = await cxgActions.cellSet(2);
+    expect(panCount).toBe(initialCount);
+  });
+});

client/__tests__/e2e/puppeteerUtils.js

@@ -1,21 +1,60 @@
 export const puppeteerUtils = puppeteerPage => ({
-  async waitByID(testid) {
-    return await puppeteerPage.waitForSelector(`[data-testid='${testid}']`);
+  async waitByID(testid, props = {}) {
+    return await puppeteerPage.waitForSelector(
+      `[data-testid='${testid}']`,
+      props
+    );
   },
 
-  async waitByClass(testclass) {
+  async waitByClass(testclass, props = {}) {
     return await puppeteerPage.waitForSelector(
-      `[data-testclass='${testclass}']`
+      `[data-testclass='${testclass}']`,
+      props
     );
   },
 
+  async waitForAllByIds(testids, props = {}) {
+    await Promise.all(
+      testids.map(testid =>
+        puppeteerPage.waitForSelector(`[data-testid='${testid}']`)
+      )
+    );
+  },
+
+  async getAllByClass(testclass, props = {}) {
+    const elements = await puppeteerPage.$$eval(
+      `[data-testclass=${testclass}]`,
+      els => {
+        return els.map(el => {
+          return el.dataset.testid;
+        });
+      }
+    );
+    return elements;
+  },
+
   async typeInto(testid, text) {
     // only works for text without special characters
     await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
     // type ahead can be annoying if you don't pause before you type
-    await puppeteerPage.click(`[data-testid='${testid}']`);
+    await puppeteerPage.click(selector);
     await puppeteerPage.waitFor(200);
-    await puppeteerPage.type(`[data-testid='${testid}']`, text);
+    await puppeteerPage.type(selector, text);
+  },
+
+  async clearInputAndTypeInto(testid, text) {
+    await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
+    // only works for text without special characters
+    // type ahead can be annoying if you don't pause before you type
+    await puppeteerPage.click(selector);
+    await puppeteerPage.waitFor(200);
+    // select all
+
+    await puppeteerPage.click(selector, { clickCount: 3 });
+    await puppeteerPage.keyboard.type("Backspace");
+    await puppeteerPage.type(selector, text);
   },
 
   async clickOn(testid) {
@@ -25,11 +64,13 @@ export const puppeteerUtils = puppeteerPage => ({
   },
 
   async getOneElementInnerHTML(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerHTML);
     return text;
   },
 
   async getOneElementInnerText(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerText);
     return text;
   }
@@ -56,20 +97,16 @@ export const cellxgeneActions = puppeteerPage => ({
     await puppeteerPage.mouse.up();
   },
 
-  async getAllHistograms(testclass) {
-    await puppeteerUtils(puppeteerPage).waitByClass(testclass);
-    const histograms = await puppeteerPage.$$eval(
-      `[data-testclass=${testclass}]`,
-      els => {
-        return els.map(el => {
-          return el.dataset.testid.substring(
-            "histogram_".length,
-            el.dataset.testid.length
-          );
-        });
-      }
+  async getAllHistograms(testclass, testids) {
+    const histTestIds = testids.map(tid => `histogram-${tid}`);
+    // these load asynchronously, so we need to wait for each histogram individually
+    await puppeteerUtils(puppeteerPage).waitForAllByIds(histTestIds);
+    const allHistograms = await puppeteerUtils(puppeteerPage).getAllByClass(
+      testclass
+    );
+    return allHistograms.map(hist =>
+      hist.substr("histogram_".length, hist.length)
     );
-    return histograms;
   },
 
   async getAllCategoriesAndCounts(category) {
@@ -157,5 +194,18 @@ export const cellxgeneActions = puppeteerPage => ({
     await puppeteerUtils(puppeteerPage).clickOn("reset");
     // loading state never actually happens, reset is too fast
     await page.waitFor(200);
+  },
+
+  async clip(min = 0, max = 100) {
+    await puppeteerUtils(puppeteerPage).clickOn("visualization-settings");
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto(
+      "clip-min-input",
+      min
+    );
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto(
+      "clip-max-input",
+      max
+    );
+    await puppeteerUtils(puppeteerPage).clickOn("clip-commit");
   }
 });

client/__tests__/util/dataframe/dataframe.test.js

@@ -532,6 +532,42 @@ describe("dataframe factories", () => {
       expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
     });
   });
+
+  describe("mapColumns", () => {
+    test("identity", () => {
+      const dfA = Dataframe.Dataframe.create(
+        [3, 3],
+        [
+          new Array(3).fill(0),
+          new Int16Array(3).fill(99),
+          new Float64Array(3).fill(1.1)
+        ]
+      );
+      const dfB = dfA.mapColumns((col, idx) => {
+        expect(dfA.icol(idx).asArray()).toBe(col);
+        return col;
+      });
+      expect(dfA).not.toBe(dfB);
+      expect(dfA.dims).toEqual(dfB.dims);
+      for (let c = 0; c < dfA.dims[1]; c += 1) {
+        expect(dfA.icol(c).asArray()).toBe(dfB.icol(c).asArray());
+      }
+    });
+
+    test("transform", () => {
+      const dfA = Dataframe.Dataframe.create(
+        [3, 3],
+        [new Array(3).fill(0), new Array(3).fill(0), new Array(3).fill(0)]
+      );
+      const dfB = dfA.mapColumns(() => {
+        return new Array(3).fill(1);
+      });
+      expect(dfA).not.toBe(dfB);
+      expect(dfB.iat(0, 0)).toEqual(1);
+      expect(dfB.iat(0, 1)).toEqual(1);
+      expect(dfB.iat(0, 2)).toEqual(1);
+    });
+  });
 });
 
 describe("dataframe col", () => {

client/__tests__/util/dataframe/histogram.test.js

@@ -0,0 +1,69 @@
+import * as Dataframe from "../../../src/util/dataframe";
+
+describe("Dataframe column histogram", () => {
+  test("categorical by categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("cat").histogram(df.col("name"));
+    expect(h1).toMatchObject(
+      new Map([
+        ["n1", new Map([["c1", 1]])],
+        ["n2", new Map([["c2", 1]])],
+        ["n3", new Map([["c3", 1]])]
+      ])
+    );
+    // memoized?
+    expect(df.col("cat").histogram(df.col("name"))).toMatchObject(h1);
+  });
+
+  test("continuous by categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("value").histogram(3, [0, 2], df.col("name"));
+    expect(h1).toMatchObject(
+      new Map([["n1", [1, 0, 0]], ["n2", [0, 1, 0]], ["n3", [0, 0, 1]]])
+    );
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2], df.col("name"))).toMatchObject(
+      h1
+    );
+  });
+
+  test("categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("cat").histogram();
+    expect(h1).toMatchObject(new Map([["c1", 1], ["c2", 1], ["c3", 1]]));
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2])).toMatchObject(h1);
+  });
+
+  test("continuous", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("value").histogram(3, [0, 2]);
+    expect(h1).toMatchObject([1, 1, 1]);
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2])).toMatchObject(h1);
+  });
+});

client/__tests__/util/quantile.test.js

@@ -0,0 +1,29 @@
+import quantile from "../../src/util/quantile";
+
+describe("quantile", () => {
+	test("single q", () => {
+		const arr = new Float32Array([9, 3, 5, 6, 0]);
+		expect(quantile([1.0], arr)).toMatchObject([9]);
+		expect(quantile([0.9], arr)).toMatchObject([9]);
+		expect(quantile([0.8], arr)).toMatchObject([9]);
+		expect(quantile([0.7], arr)).toMatchObject([6]);
+		expect(quantile([0.6], arr)).toMatchObject([6]);
+		expect(quantile([0.5], arr)).toMatchObject([5]);
+		expect(quantile([0.4], arr)).toMatchObject([5]);
+		expect(quantile([0.3], arr)).toMatchObject([3]);
+		expect(quantile([0.2], arr)).toMatchObject([3]);
+		expect(quantile([0.1], arr)).toMatchObject([0]);
+		expect(quantile([0], arr)).toMatchObject([0]);
+	});
+
+	test("multi q", () => {
+		const arr = new Float32Array([9, 3, 5, 6, 0]);
+		expect(quantile([0, 0.25, 0.5, 0.75, 1.0], arr)).toMatchObject([
+			0,
+			3,
+			5,
+			6,
+			9
+		]);
+	});
+});

client/__tests__/util/range.test.js

@@ -0,0 +1,42 @@
+import { range, rangeFill } from "../../src/util/range";
+
+describe("range", () => {
+	test("no defaults", () => {
+		expect(range(0, 3, 1)).toMatchObject([0, 1, 2]);
+	});
+
+	test("range(stop)", () => {
+		expect(range(3)).toMatchObject([0, 1, 2]);
+		expect(range(0)).toMatchObject([]);
+		expect(range(1)).toMatchObject([0]);
+	});
+
+	test("range(start,stop)", () => {
+		expect(range(0, 0)).toMatchObject([]);
+		expect(range(0, 2)).toMatchObject([0, 1]);
+		expect(range(4, 8)).toMatchObject([4, 5, 6, 7]);
+	});
+
+	test("range(start, stop, step", () => {
+		expect(range(4, 0, -1)).toMatchObject([4, 3, 2, 1]);
+		expect(range(0, 4, 2)).toMatchObject([0, 2]);
+	});
+});
+
+describe("rangefill", () => {
+	test("rangeFill(arr)", () => {
+		expect(rangeFill(new Int32Array(3))).toMatchObject(
+			new Int32Array([0, 1, 2])
+		);
+	});
+	test("rangeFill(arr, start)", () => {
+		expect(rangeFill(new Int32Array(2), 1)).toMatchObject(
+			new Int32Array([1, 2])
+		);
+	});
+	test("rangeFill(arr, start, step)", () => {
+		expect(rangeFill(new Int32Array(3), 2, -1)).toMatchObject(
+			new Int32Array([2, 1, 0])
+		);
+	});
+});

client/__tests__/util/stateManager/sampleResponses.js

@@ -34,28 +34,38 @@ const aSchemaResponse = {
       type: "float32"
     },
     annotations: {
-      obs: [
-        { name: "name", type: "string" },
-        { name: "field1", type: "int32" },
-        { name: "field2", type: "float32" },
-        { name: "field3", type: "boolean" },
-        {
-          name: "field4",
-          type: "categorical",
-          categories: field4Categories
-        }
-      ],
-      var: [
-        { name: "name", type: "string" },
-        { name: "fieldA", type: "int32" },
-        { name: "fieldB", type: "float32" },
-        { name: "fieldC", type: "boolean" },
-        {
-          name: "fieldD",
-          type: "categorical",
-          categories: fieldDCategories
-        }
-      ]
+      obs: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "field1", type: "int32" },
+          { name: "field2", type: "float32" },
+          { name: "field3", type: "boolean" },
+          {
+            name: "field4",
+            type: "categorical",
+            categories: field4Categories
+          }
+        ]
+      },
+      var: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "fieldA", type: "int32" },
+          { name: "fieldB", type: "float32" },
+          { name: "fieldC", type: "boolean" },
+          {
+            name: "fieldD",
+            type: "categorical",
+            categories: fieldDCategories
+          }
+        ]
+      }
+    },
+    layout: {
+      obs: [{ name: "umap", type: "float32", dims: ["umap_0", "umap_1"] }],
+      var: []
     }
   }
 };
@@ -162,29 +172,7 @@ const aLayoutFBSResponse = (() => {
     new Float32Array(nObs).fill(Math.random()),
     new Float32Array(nObs).fill(Math.random())
   ];
-  const builder = new flatbuffers.Builder(1024);
-
-  const cols = _.map(coords, carr => {
-    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
-    NetEncoding.Float32Array.startFloat32Array(builder);
-    NetEncoding.Float32Array.addData(builder, cdv);
-    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
-
-    NetEncoding.Column.startColumn(builder);
-    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
-    NetEncoding.Column.addU(builder, floatArr);
-    return NetEncoding.Column.endColumn(builder);
-  });
-
-  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
-
-  NetEncoding.Matrix.startMatrix(builder);
-  NetEncoding.Matrix.addNRows(builder, nObs);
-  NetEncoding.Matrix.addNCols(builder, coords.length);
-  NetEncoding.Matrix.addColumns(builder, columns);
-  const matrix = NetEncoding.Matrix.endMatrix(builder);
-  builder.finish(matrix);
-  return builder.asUint8Array();
+  return encodeMatrix(coords, ["umap_0", "umap_1"]);
 })();
 
 const aDataObsResponse = {

client/__tests__/util/stateManager/universe.test.js

@@ -53,13 +53,15 @@ describe("createUniverseFromResponse", () => {
 
     expect(universe.obsAnnotations.dims).toEqual([
       nObs,
-      REST.schema.schema.annotations.obs.length
+      REST.schema.schema.annotations.obs.columns.length
     ]);
     expect(universe.obsLayout.dims).toEqual([nObs, 2]);
-    expect(universe.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(universe.obsLayout.colIndex.keys()).toEqual(
+      universe.schema.layout.obs[0].dims
+    );
     expect(universe.varAnnotations.dims).toEqual([
       nVar,
-      REST.schema.schema.annotations.var.length
+      REST.schema.schema.annotations.var.columns.length
     ]);
     expect(universe.varData.isEmpty()).toBeTruthy();
   });

client/__tests__/util/stateManager/world.test.js

@@ -29,7 +29,8 @@ const defaultBigBang = () => {
   /* create crossfilter */
   const crossfilter = World.createObsDimensions(
     new Crossfilter(world.obsAnnotations),
-    world
+    world,
+    REST.schema.schema.layout.obs[0].dims
   );
 
   return {
@@ -58,10 +59,15 @@ describe("createWorldFromEntireUniverse", () => {
         nObs: universe.nObs,
         nVar: universe.nVar,
         schema: universe.schema,
-        obsAnnotations: universe.obsAnnotations,
-        varAnnotations: universe.varAnnotations,
-        obsLayout: universe.obsLayout,
-        varData: expect.any(Dataframe.Dataframe)
+        obsAnnotations: expect.any(Dataframe.Dataframe),
+        varAnnotations: expect.any(Dataframe.Dataframe),
+        obsLayout: expect.any(Dataframe.Dataframe),
+        varData: expect.any(Dataframe.Dataframe),
+        clipQuantiles: { min: 0, max: 1 },
+        unclipped: {
+          obsAnnotations: expect.any(Dataframe.Dataframe),
+          varData: expect.any(Dataframe.Dataframe)
+        }
       })
     );
   });
@@ -76,7 +82,7 @@ describe("createWorldFromCurrentSelection", () => {
     } = defaultBigBang();
 
     /* mock a selection */
-    let crossfilter = originalCrossfilter
+    const crossfilter = originalCrossfilter
       .select(obsAnnoDimensionName("field1"), { mode: "range", lo: 0, hi: 5 })
       .select(obsAnnoDimensionName("field3"), {
         mode: "exact",
@@ -84,7 +90,7 @@ describe("createWorldFromCurrentSelection", () => {
       });
 
     /* create the world from the selection */
-    const world = World.createWorldFromCurrentSelection(
+    const world = World.createWorldBySelection(
       universe,
       originalWorld,
       crossfilter
@@ -112,10 +118,15 @@ describe("createWorldFromCurrentSelection", () => {
         nObs: matchingIndices.length,
         nVar: universe.nVar,
         schema: universe.schema,
+        clipQuantiles: { min: 0, max: 1 },
         obsAnnotations: expect.any(Dataframe.Dataframe),
-        varAnnotations: universe.varAnnotations,
+        varAnnotations: expect.any(Dataframe.Dataframe),
         obsLayout: expect.any(Dataframe.Dataframe),
-        varData: expect.any(Dataframe.Dataframe)
+        varData: expect.any(Dataframe.Dataframe),
+        unclipped: {
+          obsAnnotations: expect.any(Dataframe.Dataframe),
+          varData: expect.any(Dataframe.Dataframe)
+        }
       })
     );
 
@@ -128,7 +139,9 @@ describe("createWorldFromCurrentSelection", () => {
     expect(world.obsLayout.rowIndex.keys()).toEqual(
       new Int32Array(matchingIndices)
     );
-    expect(world.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(world.obsLayout.colIndex.keys()).toEqual(
+      world.schema.layout.obs[0].dims
+    );
   });
 });
 
@@ -142,14 +155,18 @@ describe("createObsDimensionMap", () => {
 
     const { crossfilter } = defaultBigBang();
     const annotationNames = _.map(
-      REST.schema.schema.annotations.obs,
+      REST.schema.schema.annotations.obs.columns,
       c => c.name
     );
-    const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
+    const obsIndexColName = REST.schema.schema.annotations.obs.index;
+    const schemaByObsName = _.keyBy(
+      REST.schema.schema.annotations.obs.columns,
+      "name"
+    );
     expect(crossfilter).toBeDefined();
     annotationNames.forEach(name => {
       const dim = crossfilter.dimensions[obsAnnoDimensionName(name)];
-      if (name === "name") {
+      if (name === obsIndexColName) {
         expect(dim).toBeUndefined();
       } else {
         const { type } = schemaByObsName[name];

client/__tests__/util/typedCrossfilter/crossfilter.test.js

@@ -11,7 +11,8 @@ const someData = [
     tip: 100,
     type: "tab",
     productIDs: ["001"],
-    coords: [0, 0]
+    coords: [0, 0],
+    nonFinite: 0.0
   },
   {
     date: "2011-11-14T16:20:19Z",
@@ -20,7 +21,8 @@ const someData = [
     tip: 100,
     type: "tab",
     productIDs: ["001", "005"],
-    coords: [0.4, 0.4]
+    coords: [0.4, 0.4],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T16:28:54Z",
@@ -29,7 +31,8 @@ const someData = [
     tip: 200,
     type: "visa",
     productIDs: ["004", "005"],
-    coords: [0.3, 0.1]
+    coords: [0.3, 0.1],
+    nonFinite: Number.POSITIVE_INFINITY
   },
   {
     date: "2011-11-14T16:30:43Z",
@@ -38,7 +41,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001", "002"],
-    coords: [0.392, 0.1]
+    coords: [0.392, 0.1],
+    nonFinite: Number.NEGATIVE_INFINITY
   },
   {
     date: "2011-11-14T16:48:46Z",
@@ -47,7 +51,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["005"],
-    coords: [0.7, 0.0482]
+    coords: [0.7, 0.0482],
+    nonFinite: 1.0
   },
   {
     date: "2011-11-14T16:53:41Z",
@@ -56,7 +61,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001", "004", "005"],
-    coords: [0.9999, 1.0]
+    coords: [0.9999, 1.0],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T16:54:06Z",
@@ -65,7 +71,8 @@ const someData = [
     tip: 0,
     type: "cash",
     productIDs: ["001", "002", "003", "004", "005"],
-    coords: [0.384, 0.6938]
+    coords: [0.384, 0.6938],
+    nonFinite: 99.0
   },
   {
     date: "2011-11-14T16:58:03Z",
@@ -74,7 +81,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001"],
-    coords: [0.4822, 0.482]
+    coords: [0.4822, 0.482],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T17:07:21Z",
@@ -83,7 +91,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["004", "005"],
-    coords: [0.2234, 0]
+    coords: [0.2234, 0],
+    nonFinite: Number.NaN
   },
   {
     date: "2011-11-14T17:22:59Z",
@@ -92,7 +101,8 @@ const someData = [
     tip: 0,
     type: "tab",
     productIDs: ["001", "002", "004", "005"],
-    coords: [0.382, 0.38485]
+    coords: [0.382, 0.38485],
+    nonFinite: -1
   },
   {
     date: "2011-11-14T17:25:45Z",
@@ -101,7 +111,8 @@ const someData = [
     tip: 0,
     type: "cash",
     productIDs: ["002"],
-    coords: [0.998, 0.8472]
+    coords: [0.998, 0.8472],
+    nonFinite: 0.0
   },
   {
     date: "2011-11-14T17:29:52Z",
@@ -110,7 +121,8 @@ const someData = [
     tip: 100,
     type: "visa",
     productIDs: ["004"],
-    coords: [0.8273, 0.3384]
+    coords: [0.8273, 0.3384],
+    nonFinite: 0.0
   }
 ];
 
@@ -304,15 +316,15 @@ describe("ImmutableTypedCrossfilter", () => {
     });
     test.each([[0, 0, 1, 1], [0, 0, 0.5, 0.5], [0.5, 0.5, 1, 1]])(
       "within-rect %d %d %d %d",
-      (x0, y0, x1, y1) => {
+      (minX, minY, maxX, maxY) => {
         expect(
           p
-            .select("coords", { mode: "within-rect", x0, y0, x1, y1 })
+            .select("coords", { mode: "within-rect", minX, minY, maxX, maxY })
             .allSelected()
         ).toEqual(
           _.filter(someData, d => {
             const [x, y] = d.coords;
-            return x0 <= x && x < x1 && y0 <= y && y < y1;
+            return minX <= x && x < maxX && minY <= y && y < maxY;
           })
         );
       }
@@ -327,4 +339,96 @@ describe("ImmutableTypedCrossfilter", () => {
       ).toEqual(_.filter(someData, d => polygonContains(polygon, d.coords)));
     });
   });
+
+  describe("non-finite scalars", () => {
+    let p;
+    beforeEach(() => {
+      p = payments
+        .addDimension("quantity", "scalar", (i, d) => d[i].quantity, Int32Array)
+        .addDimension(
+          "nonFinite",
+          "scalar",
+          (i, d) => d[i].nonFinite,
+          Float32Array
+        )
+        .select("quantity", { mode: "all" });
+    });
+
+    test("all or none", () => {
+      expect(p.select("nonFinite", { mode: "all" }).countSelected()).toEqual(
+        someData.length
+      );
+      expect(p.select("nonFinite", { mode: "none" }).countSelected()).toEqual(
+        0
+      );
+    });
+
+    test("exact", () => {
+      expect(
+        p.select("nonFinite", { mode: "exact", values: [0] }).countSelected()
+      ).toEqual(3);
+      expect(
+        p.select("nonFinite", { mode: "exact", values: [1] }).countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.POSITIVE_INFINITY]
+          })
+          .countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.NEGATIVE_INFINITY]
+          })
+          .countSelected()
+      ).toEqual(1);
+      expect(
+        p
+          .select("nonFinite", { mode: "exact", values: [Number.NaN] })
+          .countSelected()
+      ).toEqual(4);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "exact",
+            values: [Number.POSITIVE_INFINITY, 0, 1, 99]
+          })
+          .countSelected()
+      ).toEqual(6);
+    });
+
+    test("range", () => {
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: 0,
+            hi: Number.POSITIVE_INFINITY
+          })
+          .countSelected()
+      ).toEqual(5);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: 0,
+            hi: Number.NaN
+          })
+          .countSelected()
+      ).toEqual(6);
+      expect(
+        p
+          .select("nonFinite", {
+            mode: "range",
+            lo: Number.NEGATIVE_INFINITY,
+            hi: Number.POSITIVE_INFINITY
+          })
+          .countSelected()
+      ).toEqual(7);
+    });
+  });
 });

client/__tests__/util/typedCrossfilter/sort.test.js

@@ -1,4 +1,22 @@
-import { sort, sortIndex } from "../../../src/util/typedCrossfilter/sort";
+import {
+  sortArray,
+  sortIndex,
+  lowerBound,
+  upperBound,
+  lowerBoundIndirect,
+  upperBoundIndirect
+} from "../../../src/util/typedCrossfilter/sort";
+
+/*
+Sort tests should keep in mind that there are separate code
+paths for:
+  - small vs. large arrays (insertionsort only)
+  - float-only typed arrays vs. other array types (non-finite handling)
+  - indexed vs. direct sort
+*/
+
+const pInf = Number.POSITIVE_INFINITY;
+const nInf = Number.NEGATIVE_INFINITY;
 
 function fillRange(arr, start = 0) {
   const larr = arr;
@@ -15,42 +33,223 @@ function fillRand(arr) {
   return arr;
 }
 
-describe("sort", () => {
-  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
-    test(Type.name, () => {
-      expect(sort(Type.from([6, 5, 4, 3, 2, 1, 0]))).toMatchObject(
-        Type.from([0, 1, 2, 3, 4, 5, 6])
-      );
-      expect(sort(Type.from([6, 5, 4, 3, 2, 1]))).toMatchObject(
-        Type.from([1, 2, 3, 4, 5, 6])
-      );
+describe("sortArray", () => {
+  describe("JS vals", () => {
+    [
+      [true, false],
+      ["a", "b", "0", "1"],
+      [0, "a", true, null, undefined, 3.1415],
+      fillRand(new Array(1000)),
+      ["a", NaN, null, pInf]
+    ].map((val, idx) =>
+      test(`JS vals ${idx}`, () => {
+        expect(sortArray(val)).toMatchObject(val.sort());
+      })
+    );
+  });
 
-      const source = fillRand(new Type(1000));
-      expect(sort(Type.from(source))).toMatchObject(Type.from(source).sort());
-    })
-  );
+  describe("finite numbers", () => {
+    [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+      test(Type.name, () => {
+        expect(sortArray(Type.from([6, 5, 4, 3, 2, 1, 0]))).toMatchObject(
+          Type.from([0, 1, 2, 3, 4, 5, 6])
+        );
+
+        expect(sortArray(Type.from([6, 5, 4, 3, 2, 1]))).toMatchObject(
+          Type.from([1, 2, 3, 4, 5, 6])
+        );
+
+        const source = fillRand(new Type(1000));
+        expect(sortArray(Type.from(source))).toMatchObject(
+          Type.from(source).sort()
+        );
+      })
+    );
+  });
+
+  describe("non-finite numbers", () => {
+    test("inifinity", () => {
+      expect(sortArray(new Float32Array([pInf, nInf, 0, 1, 2]))).toMatchObject(
+        new Float32Array([nInf, 0, 1, 2, pInf])
+      );
+      expect(
+        sortArray(new Float32Array([pInf, nInf, pInf, nInf]))
+      ).toMatchObject(new Float32Array([nInf, nInf, pInf, pInf]));
+      expect(
+        sortArray(new Float32Array([pInf, nInf, pInf, nInf, pInf]))
+      ).toMatchObject(new Float32Array([nInf, nInf, pInf, pInf, pInf]));
+      expect(
+        sortArray(
+          new Float32Array(100).fill(Infinity, 0, 50).fill(-Infinity, 50, 100)
+        )
+      ).toMatchObject(
+        new Float32Array(100).fill(-Infinity, 0, 50).fill(Infinity, 50, 100)
+      );
+    });
+    test("NaN", () => {
+      expect(sortArray(new Float64Array([NaN, 2, 1, 0]))).toMatchObject(
+        new Float64Array([0, 1, 2, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, 1, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, NaN, 1, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN, NaN])
+      );
+      expect(sortArray(new Float32Array([NaN, 2, 1, NaN, 0]))).toMatchObject(
+        new Float32Array([0, 1, 2, NaN, NaN])
+      );
+      expect(
+        sortArray(fillRange(new Float32Array(100)).fill(NaN, 0, 10))
+      ).toMatchObject(fillRange(new Float32Array(100), 10).fill(NaN, 90, 100));
+    });
+
+    test("mixed numbers", () => {
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, NaN, NaN]))
+      ).toMatchObject(new Float32Array([nInf, pInf, NaN, NaN, NaN]));
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, NaN, 1, NaN, 2]))
+      ).toMatchObject(new Float32Array([nInf, 1, 2, pInf, NaN, NaN, NaN]));
+      expect(
+        sortArray(new Float32Array([NaN, pInf, nInf, 0, 1, NaN, 2]))
+      ).toMatchObject(new Float32Array([nInf, 0, 1, 2, pInf, NaN, NaN]));
+      expect(
+        sortArray(
+          fillRange(new Float32Array(100))
+            .fill(NaN, 0, 10)
+            .fill(Infinity, 10, 20)
+        )
+      ).toMatchObject(
+        fillRange(new Float32Array(100), 20)
+          .fill(Infinity, 80, 90)
+          .fill(NaN, 90, 100)
+      );
+    });
+  });
 });
 
 describe("sortIndex", () => {
-  [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
-    test(Type.name, () => {
-      const source1 = Type.from([6, 5, 4, 3, 2, 1, 0]);
+  describe("finite numbers", () => {
+    [Array, Float32Array, Uint32Array, Int32Array, Float64Array].map(Type =>
+      test(Type.name, () => {
+        const source1 = Type.from([6, 5, 4, 3, 2, 1, 0]);
+        const index1 = fillRange(new Uint32Array(source1.length));
+        expect(sortIndex(index1, source1)).toMatchObject(
+          index1.sort((a, b) => source1[a] - source1[b])
+        );
+
+        const source2 = Type.from([6, 5, 4, 3, 2, 1]);
+        const index2 = fillRange(new Uint32Array(source2.length));
+        expect(sortIndex(index2, source2)).toMatchObject(
+          index2.sort((a, b) => source1[a] - source1[b])
+        );
+
+        const source3 = fillRand(new Type(1000));
+        const index3 = fillRange(new Uint32Array(source3.length));
+        expect(sortIndex(index3, source3)).toMatchObject(
+          index3.sort((a, b) => source1[a] - source1[b])
+        );
+      })
+    );
+  });
+
+  describe("non-finite numbers", () => {
+    test("mixed numbers", () => {
+      const source1 = new Float32Array([NaN, pInf, nInf, NaN, 1, NaN, 2]);
       const index1 = fillRange(new Uint32Array(source1.length));
       expect(sortIndex(index1, source1)).toMatchObject(
-        index1.sort((a, b) => source1[a] - source1[b])
+        new Uint32Array([2, 4, 6, 1, 0, 3, 5])
       );
 
-      const source2 = Type.from([6, 5, 4, 3, 2, 1]);
+      const source2 = new Float32Array([NaN, pInf, nInf, 0, 1, NaN, 2]);
       const index2 = fillRange(new Uint32Array(source2.length));
       expect(sortIndex(index2, source2)).toMatchObject(
-        index2.sort((a, b) => source1[a] - source1[b])
+        new Uint32Array([2, 3, 4, 6, 1, 0, 5])
       );
+    });
+  });
+});
 
-      const source3 = fillRand(new Type(1000));
-      const index3 = fillRange(new Uint32Array(source3.length));
-      expect(sortIndex(index3, source3)).toMatchObject(
-        index3.sort((a, b) => source1[a] - source1[b])
-      );
-    })
-  );
+describe("lowerBound", () => {
+  test("non-float path", () => {
+    expect(lowerBound([], 0, 0, 0)).toEqual(0);
+
+    expect(lowerBound([0, 1, 2, 3], -1, 0, 4)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3], 0, 0, 4)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3], 1, 0, 4)).toEqual(1);
+    expect(lowerBound([0, 1, 2, 3], 3, 0, 4)).toEqual(3);
+    expect(lowerBound([0, 1, 2, 3], 4, 0, 4)).toEqual(4);
+
+    expect(lowerBound([0, 1, 2, 3, 4], -1, 0, 5)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3, 4], 0, 0, 5)).toEqual(0);
+    expect(lowerBound([0, 1, 2, 3, 4], 2, 0, 5)).toEqual(2);
+    expect(lowerBound([0, 1, 2, 3, 4], 4, 0, 5)).toEqual(4);
+    expect(lowerBound([0, 1, 2, 3, 4], 5, 0, 5)).toEqual(5);
+
+    expect(lowerBound([0, 2, 4, 6, 8], 5, 0, 5)).toEqual(3);
+    expect(lowerBound([0, 2, 2, 2, 8], 5, 0, 5)).toEqual(4);
+
+    expect(lowerBound([0, 1, 2, 3, 4, 5, 6, 7, 8], 3, 2, 4)).toEqual(3);
+    expect(lowerBound([0, 1, 2, 3, 4, 5, 6, 7, 8], 99, 2, 4)).toEqual(4);
+  });
+
+  test("float path, finites", () => {
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
+
+    expect(lowerBound(new Float32Array([]), 0, 0, 0)).toEqual(0);
+
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), -1, 0, 4)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 0, 0, 4)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 3, 0, 4)).toEqual(3);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 4, 0, 4)).toEqual(4);
+
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), -1, 0, 5)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 0, 0, 5)).toEqual(0);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 2, 0, 5)).toEqual(2);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 4, 0, 5)).toEqual(4);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 5, 0, 5)).toEqual(5);
+
+    expect(lowerBound(new Float32Array([0, 2, 4, 6, 8]), 5, 0, 5)).toEqual(3);
+    expect(lowerBound(new Float32Array([0, 2, 2, 2, 8]), 5, 0, 5)).toEqual(4);
+
+    expect(
+      lowerBound(new Float32Array([0, 1, 2, 3, 4, 5, 6, 7, 8]), 3, 2, 4)
+    ).toEqual(3);
+    expect(
+      lowerBound(new Float32Array([0, 1, 2, 3, 4, 5, 6, 7, 8]), 99, 2, 4)
+    ).toEqual(4);
+  });
+
+  test("float path, non-finite", () => {
+    expect(
+      lowerBound(
+        new Float32Array([-Infinity, 0, 1, Infinity, NaN]),
+        -Infinity,
+        0,
+        5
+      )
+    ).toEqual(0);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 0, 0, 5)
+    ).toEqual(1);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 1, 0, 5)
+    ).toEqual(2);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), 2, 0, 5)
+    ).toEqual(3);
+    expect(
+      lowerBound(
+        new Float32Array([-Infinity, 0, 1, Infinity, NaN]),
+        Infinity,
+        0,
+        5
+      )
+    ).toEqual(3);
+    expect(
+      lowerBound(new Float32Array([-Infinity, 0, 1, Infinity, NaN]), NaN, 0, 5)
+    ).toEqual(4);
+  });
 });

client/__tests__/util/typedCrossfilter/util.test.js

@@ -1,11 +1,8 @@
 import {
-  fillRange,
   sliceByIndex,
-  makeSortIndex,
-  lowerBound,
-  lowerBoundIndirect,
-  upperBoundIndirect
+  makeSortIndex
 } from "../../../src/util/typedCrossfilter/util";
+import { rangeFill as fillRange } from "../../../src/util/range";
 
 describe("fillRange", () => {
   test("Array", () => {

client/configuration/babel/babel.dev.js

@@ -7,8 +7,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-proposal-optional-chaining",
     "@babel/plugin-proposal-nullish-coalescing-operator"

client/configuration/babel/babel.prod.js

@@ -6,8 +6,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-transform-react-constant-elements",
     "@babel/plugin-transform-runtime",

client/index.html

@@ -3,7 +3,7 @@
   <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>cellxgene</title>
+    <title>cell&times;gene</title>
     <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
       html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {

client/index_template.html

@@ -1,35 +1,42 @@
 <!DOCTYPE html>
 <html lang="en">
-  <head>
+<head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>cellxgene</title>
+    <title>cell&times;gene</title>
     <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
-      html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
-        font-family: 'Roboto Condensed','Helvetica Neue','Helvetica','Arial',sans-serif;
-        font-size: 14px;
-      }
-      body {
-        margin: 0;
-        padding: 0;
-      }
+        html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {
+            font-family: 'Roboto Condensed', 'Helvetica Neue', 'Helvetica', 'Arial', sans-serif;
+            font-size: 14px;
+        }
 
-      * {
-        box-sizing: border-box;
-      }
+        body {
+            margin: 0;
+            padding: 0;
+        }
+
+        * {
+            box-sizing: border-box;
+        }
     </style>
-  </head>
-  <body>
-<script type="text/javascript">
-window.CELLXGENE = {};
-window.CELLXGENE.API = {
-  prefix: window.location.href + "api/",
-  version: "v0.2/"
-};
-</script>
-    <noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>, please <strong>enable JS</strong> to make this app work.</noscript>
 
-    <div id="root"></div>
-  </body>
+</head>
+<body>
+<script type="text/javascript">
+  window.CELLXGENE = {};
+  window.CELLXGENE.API = {
+    prefix: window.location.href + "api/",
+    version: "v0.2/"
+  };
+</script>
+<noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>,
+    please <strong>enable JS</strong> to make this app work.
+</noscript>
+
+<div id="root"></div>
+{% for script in SCRIPTS %}
+<script type="text/javascript" src="{{script | safe}}"></script>
+{% endfor %}
+</body>
 </html>

client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cellxgene",
-  "version": "0.8.0",
+  "version": "0.11.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
@@ -9,13 +9,13 @@
     "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
     "clean": "rimraf build",
     "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
-    "e2e": "jest --verbose false --config __tests__/e2e/e2eJestConfig.json e2e",
+    "e2e": "node node_modules/jest/bin/jest.js --verbose false --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
     "lint": "eslint src",
     "smoke-test": "start-server-and-test start-server-for-test :5000 e2e",
     "start": "node server/development.js",
     "start-server-for-test": "cellxgene launch -p 5000 ../example-dataset/pbmc3k.h5ad",
-    "test": "jest",
-    "unit-test": "jest --testPathIgnorePatterns e2e"
+    "test": "node node_modules/jest/bin/jest.js",
+    "unit-test": "node node_modules/jest/bin/jest.js --testPathIgnorePatterns e2e"
   },
   "engineStrict": true,
   "engines": {
@@ -31,9 +31,9 @@
     "eslint-scope": "3.7.1"
   },
   "dependencies": {
-    "@blueprintjs/core": "^3.8.0",
-    "@blueprintjs/icons": "^3.3.0",
-    "@blueprintjs/select": "^3.2.1",
+    "@blueprintjs/core": "^3.15.1",
+    "@blueprintjs/icons": "^3.8.0",
+    "@blueprintjs/select": "^3.8.0",
     "canvas-fit": "^1.5.0",
     "d3": "^4.10.0",
     "d3-scale-chromatic": "^1.3.0",
@@ -41,79 +41,77 @@
     "font-color-contrast": "^1.0.3",
     "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
-    "gl-matrix": "^2.7.1",
+    "gl-matrix": "^3.0.0",
+    "gl-vec3": "^1.1.3",
     "is-number": "^7.0.0",
     "key-pressed": "0.0.1",
-    "lodash": "^4.17.4",
-    "memoize-one": "^4.0.0",
+    "lodash": "^4.17.14",
+    "memoize-one": "^5.0.4",
     "mouse-position": "^2.0.1",
     "mouse-pressed": "^1.0.0",
-    "normalize.css": "^8.0.0",
     "orbit-camera": "^1.0.0",
-    "query-string": "^6.1.0",
-    "react": "^16.6.0",
+    "query-string": "^6.5.0",
+    "react": "^16.8.6",
     "react-autocomplete": "^1.7.2",
-    "react-dom": "^16.6.0",
-    "react-helmet": "^5.2.0",
-    "react-icons": "^3.2.2",
-    "react-redux": "^5.1.0",
+    "react-dom": "^16.8.6",
+    "react-helmet": "^5.2.1",
+    "react-icons": "^3.7.0",
+    "react-redux": "^7.0.3",
     "redux": "^4.0.1",
-    "redux-devtools-extension": "^2.13.5",
     "redux-thunk": "^2.2.0",
-    "regl": "^1.3.9",
+    "regl": "^1.3.11",
     "scroll-speed": "^1.0.0",
     "urijs": "^1.19.0"
   },
   "devDependencies": {
-    "@babel/core": "^7.1.5",
-    "@babel/plugin-proposal-class-properties": "^7.0.0",
-    "@babel/plugin-proposal-decorators": "^7.0.0",
-    "@babel/plugin-proposal-export-namespace-from": "^7.0.0",
-    "@babel/plugin-proposal-function-bind": "^7.0.0",
-    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.2.0",
+    "@babel/core": "^7.4.4",
+    "@babel/plugin-proposal-class-properties": "^7.4.4",
+    "@babel/plugin-proposal-decorators": "^7.4.4",
+    "@babel/plugin-proposal-export-namespace-from": "^7.2.0",
+    "@babel/plugin-proposal-function-bind": "^7.2.0",
+    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.4.4",
     "@babel/plugin-proposal-optional-chaining": "^7.2.0",
-    "@babel/plugin-transform-react-constant-elements": "^7.0.0",
-    "@babel/plugin-transform-runtime": "^7.1.0",
-    "@babel/preset-env": "^7.1.5",
+    "@babel/plugin-transform-react-constant-elements": "^7.2.0",
+    "@babel/plugin-transform-runtime": "^7.4.4",
+    "@babel/preset-env": "^7.4.4",
     "@babel/preset-react": "^7.0.0",
-    "@babel/register": "^7.0.0",
-    "@babel/runtime": "^7.1.5",
-    "babel-core": "^7.0.0-bridge.0",
+    "@babel/register": "^7.4.4",
+    "@babel/runtime": "^7.4.4",
     "babel-eslint": "^10.0.1",
-    "babel-jest": "^23.6.0",
-    "babel-loader": "^8.0.0",
-    "babel-preset-modern-browsers": "^12.0.0",
+    "babel-jest": "^24.8.0",
+    "babel-loader": "^8.0.6",
+    "babel-preset-modern-browsers": "^14.0.0",
     "chalk": "^2.4.2",
     "connect-history-api-fallback": "^1.6.0",
-    "copy-webpack-plugin": "^4.6.0",
-    "css-loader": "^1.0.1",
-    "eslint": "^5.13.0",
+    "copy-webpack-plugin": "^5.0.3",
+    "css-loader": "^2.1.1",
+    "eslint": "^5.16.0",
     "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^4.0.0",
+    "eslint-config-prettier": "^4.2.0",
     "eslint-loader": "^2.1.2",
     "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.16.0",
-    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-import": "^2.17.2",
+    "eslint-plugin-jest": "^22.5.1",
     "eslint-plugin-jsx-a11y": "^6.2.1",
-    "eslint-plugin-react": "^7.12.4",
+    "eslint-plugin-react": "^7.13.0",
     "express": "^4.14.0",
-    "file-loader": "^2.0.0",
+    "file-loader": "^3.0.1",
     "html-webpack-inline-source-plugin": "0.0.10",
     "html-webpack-plugin": "^3.2.0",
-    "jest": "^24.1.0",
-    "jest-puppeteer": "^4.1.0",
+    "jest": "^24.8.0",
+    "jest-puppeteer": "^4.1.1",
     "json-loader": "^0.5.4",
-    "mini-css-extract-plugin": "^0.4.1",
-    "puppeteer": "^1.12.1",
+    "mini-css-extract-plugin": "^0.6.0",
+    "puppeteer": "^1.16.0",
     "rimraf": "^2.6.3",
     "serve-favicon": "^2.3.0",
-    "start-server-and-test": "^1.7.11",
+    "start-server-and-test": "^1.9.0",
     "style-loader": "^0.23.1",
     "sw-precache-webpack-plugin": "^0.11.5",
     "url-loader": "^1.1.0",
-    "webpack": "^4.25.1",
-    "webpack-cli": "^3.1.0",
-    "webpack-dev-middleware": "^3.1.3"
+    "webpack": "^4.31.0",
+    "webpack-cli": "^3.3.2",
+    "webpack-dev-middleware": "^3.6.2"
   },
   "jest": {
     "testMatch": [
@@ -133,16 +131,23 @@
         ],
         "plugins": [
           "@babel/plugin-proposal-function-bind",
-          "@babel/plugin-proposal-class-properties",
           [
             "@babel/plugin-proposal-decorators",
             {
               "legacy": true
             }
           ],
+          [
+            "@babel/plugin-proposal-class-properties",
+            {
+              "loose": true
+            }
+          ],
           "@babel/plugin-proposal-export-namespace-from",
           "@babel/plugin-transform-react-constant-elements",
-          "@babel/plugin-transform-runtime"
+          "@babel/plugin-transform-runtime",
+          "@babel/plugin-proposal-optional-chaining",
+          "@babel/plugin-proposal-nullish-coalescing-operator"
         ]
       }
     }

client/src/actions/index.js

@@ -21,24 +21,32 @@ const doInitialDataLoad = () =>
     dispatch({ type: "initial data load start" });
 
     try {
-      const requestJson = _(["config", "schema"])
+      /*
+      Step 1 - config & schema, all JSON
+      */
+      const requestJson = ["config", "schema"]
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doJsonRequest(url))
-        .value();
-      const requestBinary = _([
-        "annotations/obs",
-        "annotations/var?annotation-name=name",
-        "layout/obs"
-      ])
-        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doBinaryRequest(url))
-        .value();
-
-      const results = await Promise.all(_.concat(requestJson, requestBinary));
-
+        .map(url => doJsonRequest(url));
+      const stepOneResults = await Promise.all(requestJson);
       /* set config defaults */
-      const config = { ...globals.configDefaults, ...results[0].config };
-      const [, schema, obsAnno, varAnno, obsLayout] = [...results];
+      const config = { ...globals.configDefaults, ...stepOneResults[0].config };
+      const schema = stepOneResults[1];
+
+      /*
+      Step 2 - dataframes, all binary.  NOTE: uses results of step 1.
+      */
+      /* only load names for var annotations, if possible*/
+      const varIndexName = schema?.schema?.annotations?.var?.index;
+      const varAnnotationsQuery = varIndexName
+        ? `?annotation-name=${varIndexName}`
+        : "";
+      const varAnnotationsURL = `annotations/var${varAnnotationsQuery}`;
+      const requestBinary = ["annotations/obs", varAnnotationsURL, "layout/obs"]
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doBinaryRequest(url));
+      const stepTwoResults = await Promise.all(requestBinary);
+      const [obsAnno, varAnno, obsLayout] = [...stepTwoResults];
+
       const universe = Universe.createUniverseFromResponse(
         config,
         schema,
@@ -91,6 +99,10 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  const state = getState();
+  const { universe } = state;
+  const varIndexName = universe.schema.annotations.var.index;
+
   /* helper for this function only */
   const fetchData = async geneNames => {
     const res = await fetch(
@@ -100,7 +112,7 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
         body: JSON.stringify({
           filter: {
             var: {
-              annotation_value: [{ name: "name", values: geneNames }]
+              annotation_value: [{ name: varIndexName, values: geneNames }]
             }
           }
         }),
@@ -123,8 +135,6 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
     return Universe.convertDataFBStoObject(universe, data);
   };
 
-  const state = getState();
-  const { universe } = state;
   /* preload data already in cache */
   let expressionData = _.transform(
     genes,
@@ -241,6 +251,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     */
     const state = getState();
     const { universe } = state;
+    const varIndexName = universe.schema.annotations.var.index;
 
     // Legal values are null, Array or TypedArray.  Null is initial state.
     if (!set1) set1 = [];
@@ -277,7 +288,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     const data = await res.json();
     // result is [ [varIdx, ...], ... ]
     const topNGenes = _.map(data, r =>
-      universe.varAnnotations.at(r[0], "name")
+      universe.varAnnotations.at(r[0], varIndexName)
     );
 
     /*
@@ -302,6 +313,9 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
 const resetInterface = () => (dispatch, getState) => {
   const { universe } = getState();
 
+  dispatch({
+    type: "user reset start"
+  });
   dispatch({
     type: "clear all user defined genes"
   });
@@ -321,6 +335,9 @@ const resetInterface = () => (dispatch, getState) => {
   dispatch({
     type: "increment graph render counter"
   });
+  dispatch({
+    type: "user reset end"
+  });
 };
 
 export default {

client/src/components/app.js

@@ -4,9 +4,11 @@ import Helmet from "react-helmet";
 import { connect } from "react-redux";
 
 import Container from "./framework/container";
-import LeftSideBar from "./leftsidebar";
+import LeftSideBar from "./leftSidebar";
 import Legend from "./continuousLegend";
 import Graph from "./graph/graph";
+import MenuBar from "./menubar";
+
 import actions from "../actions";
 
 @connect(state => ({
@@ -71,18 +73,23 @@ class App extends React.Component {
             loading cellxgene
           </div>
         ) : null}
-        <div>
-          {loading ? null : <LeftSideBar />}
+        {error ? (
           <div
             style={{
-              padding: 15,
-              width: 1440 - 410 /* but responsive */,
-              marginLeft: 350 /* but responsive */
+              position: "fixed",
+              fontWeight: 500,
+              top: window.innerHeight / 2,
+              left: window.innerWidth / 2 - 50
             }}
           >
-            {loading ? null : <Graph key={graphRenderCounter} />}
-            <Legend />
+            error loading
           </div>
+        ) : null}
+        <div>
+          {loading ? null : <LeftSideBar />}
+          {loading ? null : <MenuBar />}
+          {loading ? null : <Graph key={graphRenderCounter} />}
+          <Legend />
         </div>
       </Container>
     );

client/src/components/brushableHistogram/index.js

@@ -5,14 +5,12 @@ https://bl.ocks.org/SpaceActuary/2f004899ea1b2bd78d6f1dbb2febf771
 */
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { Button, ButtonGroup, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
 import * as d3 from "d3";
 import memoize from "memoize-one";
 import * as globals from "../../globals";
 import actions from "../../actions";
-import finiteExtent from "../../util/finiteExtent";
 import { makeContinuousDimensionName } from "../../util/nameCreators";
 
 @connect(state => ({
@@ -21,53 +19,51 @@ import { makeContinuousDimensionName } from "../../util/nameCreators";
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
   continuousSelection: state.continuousSelection,
   differential: state.differential,
-  colorAccessor: state.colors.colorAccessor,
-  obsAnnotations: _.get(state.world, "obsAnnotations", null)
+  colorAccessor: state.colors.colorAccessor
 }))
 class HistogramBrush extends React.Component {
-  calcHistogramCache = memoize((obsAnnotations, field, rangeMin, rangeMax) => {
-    const { world } = this.props;
-    const histogramCache = {};
+  static getColumn(world, field, clipped = true) {
+    /*
+    Return the underlying Dataframe column for our field.   By default, 
+    returns the clipped column.   If clipped===false, will return the 
+    unclipped column.
+    */
+    const obsAnnotations = clipped
+      ? world.obsAnnotations
+      : world.unclipped.obsAnnotations;
+    const varData = clipped ? world.varData : world.unclipped.varData;
+    if (obsAnnotations.hasCol(field)) {
+      return obsAnnotations.col(field);
+    }
+    return varData.col(field);
+  }
 
+  calcHistogramCache = memoize((col, field) => {
+    /*
+     recalculate expensive stuff, notably bins, summaries, etc.
+    */
+    const histogramCache = {};
+    const values = col.asArray();
+    const summary = col.summarize();
+    const { min: domainMin, max: domainMax } = summary;
+    histogramCache.x = d3
+      .scaleLinear()
+      .domain([domainMin, domainMax])
+      .range([0, this.width - this.marginRight]);
+
+    histogramCache.bins = d3
+      .histogram()
+      .domain(histogramCache.x.domain())
+      .thresholds(40)(values);
+
+    const yMax = histogramCache.bins
+      .map(b => b.length)
+      .reduce((a, b) => Math.max(a, b));
     histogramCache.y = d3
       .scaleLinear()
+      .domain([0, yMax])
       .range([this.height - this.marginBottom, 0]);
 
-    if (obsAnnotations.hasCol(field)) {
-      // recalculate expensive stuff
-      const allValuesForContinuousFieldAsArray = obsAnnotations
-        .col(field)
-        .asArray();
-
-      histogramCache.x = d3
-        .scaleLinear()
-        .domain([rangeMin, rangeMax])
-        .range([0, this.width]);
-
-      histogramCache.bins = d3
-        .histogram()
-        .domain(histogramCache.x.domain())
-        .thresholds(40)(allValuesForContinuousFieldAsArray);
-
-      histogramCache.numValues = allValuesForContinuousFieldAsArray.length;
-    } else if (world.varData.hasCol(field)) {
-      const varValues = world.varData.col(field).asArray();
-
-      histogramCache.x = d3
-        .scaleLinear()
-        .domain(
-          finiteExtent(varValues)
-        ) /* replace this if we have ranges for genes back from server like we do for annotations on cells */
-        .range([0, this.width]);
-
-      histogramCache.bins = d3
-        .histogram()
-        .domain(histogramCache.x.domain())
-        .thresholds(40)(varValues);
-
-      histogramCache.numValues = varValues.length;
-    }
-
     return histogramCache;
   });
 
@@ -76,36 +72,56 @@ class HistogramBrush extends React.Component {
 
     this.width = 340;
     this.height = 100;
-    this.marginBottom = 20;
+    this.marginBottom = 20; // space for X axis & labels
+    this.marginRight = 40; // space for Y axis & labels
   }
 
   componentDidMount() {
     const { field } = this.props;
-    const { x, y, bins, numValues, svgRef } = this._histogram;
+    const { x, y, bins, svgRef } = this._histogram;
 
-    this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+    this.renderAxesBrushBins(x, y, bins, svgRef, field);
   }
 
   componentDidUpdate(prevProps) {
-    const { field, obsAnnotations, continuousSelection } = this.props;
-    const { x, y, bins, numValues, svgRef } = this._histogram;
+    const { field, world, continuousSelection } = this.props;
+    const { x, y, bins, svgRef } = this._histogram;
+    let { brushXselection, brushX } = this.state;
+    let forceBrushUpdate = false;
 
-    if (obsAnnotations !== prevProps.obsAnnotations) {
-      this.renderAxesBrushBins(x, y, bins, numValues, svgRef, field);
+    /*
+    Update our axis if the underlying dataframe column has changed
+    */
+    const dfColumn = HistogramBrush.getColumn(world, field);
+    const oldDfColumn = HistogramBrush.getColumn(
+      prevProps.world,
+      prevProps.field
+    );
+    if (dfColumn !== oldDfColumn) {
+      ({ brushXselection, brushX } = this.renderAxesBrushBins(
+        x,
+        y,
+        bins,
+        svgRef,
+        field
+      ));
+      forceBrushUpdate = true;
     }
 
     /*
     if the selection has changed, ensure that the brush correctly reflects
     the underlying selection.
     */
-    if (continuousSelection !== prevProps.continuousSelection) {
+    if (
+      forceBrushUpdate ||
+      continuousSelection !== prevProps.continuousSelection
+    ) {
       const { isObs, isUserDefined, isDiffExp } = this.props;
       const myName = makeContinuousDimensionName(
         { isObs, isUserDefined, isDiffExp },
         field
       );
       const range = continuousSelection[myName];
-      const { brushXselection, brushX } = this.state;
       if (brushXselection) {
         const selection = d3.brushSelection(brushXselection.node());
         if (!range && selection) {
@@ -124,7 +140,7 @@ class HistogramBrush extends React.Component {
           const dX0 = Math.abs(x0 - selection[0]);
           const dX1 = Math.abs(x1 - selection[1]);
           /*
-          only update the brush if it is grossly incorrect, 
+          only update the brush if it is grossly incorrect,
           as defined by the moveDeltaThreshold
           */
           if (dX0 > moveDeltaThreshold || dX1 > moveDeltaThreshold) {
@@ -142,6 +158,8 @@ class HistogramBrush extends React.Component {
 
       // ignore programmatically generated events
       if (!d3.event.sourceEvent) return;
+      // ignore cascading events, which are programmatically generated
+      if (d3.event.sourceEvent.sourceEvent) return;
 
       if (d3.event.selection) {
         dispatch({
@@ -172,12 +190,13 @@ class HistogramBrush extends React.Component {
   onBrushEnd(selection, x) {
     return () => {
       const { dispatch, field, isObs, isUserDefined, isDiffExp } = this.props;
-      const { brushXselection } = this.state;
       const minAllowedBrushSize = 10;
       const smallAmountToAvoidInfiniteLoop = 0.1;
 
       // ignore programmatically generated events
       if (!d3.event.sourceEvent) return;
+      // ignore cascading events, which are programmatically generated
+      if (d3.event.sourceEvent.sourceEvent) return;
 
       if (d3.event.selection) {
         let _range;
@@ -197,11 +216,6 @@ class HistogramBrush extends React.Component {
             smallAmountToAvoidInfiniteLoop; //
 
           _range = [x(d3.event.selection[0]), x(procedurallyResizedBrushWidth)];
-
-          d3.event.target.move(brushXselection, [
-            d3.event.selection[0],
-            procedurallyResizedBrushWidth
-          ]);
         }
 
         dispatch({
@@ -216,37 +230,30 @@ class HistogramBrush extends React.Component {
         });
       } else {
         dispatch({
-          type: "continuous metadata histogram end",
+          type: "continuous metadata histogram cancel",
           selection: field,
           continuousNamespace: {
             isObs,
             isUserDefined,
             isDiffExp
-          },
-          range: null
+          }
         });
       }
     };
   }
 
   drawHistogram(svgRef) {
-    const { obsAnnotations, field, ranges } = this.props;
-    const histogramCache = this.calcHistogramCache(
-      obsAnnotations,
-      field,
-      ranges.min,
-      ranges.max
-    );
-
-    const { x, y, bins, numValues } = histogramCache;
-
-    this._histogram = { x, y, bins, numValues, svgRef };
+    const { field, world } = this.props;
+    const col = HistogramBrush.getColumn(world, field);
+    const histogramCache = this.calcHistogramCache(col, field);
+    const { x, y, bins } = histogramCache;
+    this._histogram = { x, y, bins, svgRef };
   }
 
   handleColorAction() {
-    const { obsAnnotations, dispatch, field, world, ranges } = this.props;
+    const { dispatch, field, world, ranges } = this.props;
 
-    if (obsAnnotations.hasCol(field)) {
+    if (world.obsAnnotations.hasCol(field)) {
       dispatch({
         type: "color by continuous metadata",
         colorAccessor: field,
@@ -308,29 +315,29 @@ class HistogramBrush extends React.Component {
     };
   }
 
-  renderAxesBrushBins(x, y, bins, numValues, svgRef, field) {
+  renderAxesBrushBins(x, y, bins, svgRef, field) {
+    const svg = d3.select(svgRef);
+
     /* Remove everything */
-    d3.select(svgRef)
-      .selectAll("*")
-      .remove();
+    svg.selectAll("*").remove();
 
     /* BINS */
-    d3.select(svgRef)
+    svg
       .insert("g", "*")
       .attr("fill", "#bbb")
       .selectAll("rect")
       .data(bins)
       .enter()
       .append("rect")
-      .attr("class", "bar")
       .attr("x", d => x(d.x0) + 1)
-      .attr("y", d => y(d.length / numValues))
+      .attr("y", d => y(d.length))
       .attr("width", d => Math.abs(x(d.x1) - x(d.x0) - 1))
-      .attr("height", d => y(0) - y(d.length / numValues));
+      .attr("height", d => y(0) - y(d.length));
 
     /* BRUSH */
     const brushX = d3
       .brushX()
+      .extent([[0, 0], [this.width - this.marginRight, this.height]])
       /*
       emit start so that the Undoable history can save an undo point
       upon drag start, and ignore the subsequent intermediate drag events.
@@ -345,45 +352,56 @@ class HistogramBrush extends React.Component {
       .attr("data-testid", `${svgRef.dataset.testid}-brush`)
       .call(brushX);
 
-    /* AXIS */
-    d3.select(svgRef)
+    /* X AXIS */
+    svg
       .append("g")
       .attr("class", "axis axis--x")
       .attr("transform", `translate(0,${this.height - this.marginBottom})`)
       .call(d3.axisBottom(x).ticks(5));
 
-    d3.select(svgRef)
-      .selectAll(".axis--x text")
-      .style("fill", "rgb(80,80,80)");
+    /* Y AXIS */
+    svg
+      .append("g")
+      .attr("class", "axis axis--y")
+      .attr("transform", `translate(${this.width - this.marginRight},0)`)
+      .call(d3.axisRight(y).ticks(3));
 
-    d3.select(svgRef)
-      .selectAll(".axis--x path")
-      .style("stroke", "rgb(230,230,230)");
+    /* axis style */
+    svg.selectAll(".axis text").style("fill", "rgb(80,80,80)");
+    svg.selectAll(".axis path").style("stroke", "rgb(230,230,230)");
+    svg.selectAll(".axis line").style("stroke", "rgb(230,230,230)");
 
-    d3.select(svgRef)
-      .selectAll(".axis--x line")
-      .style("stroke", "rgb(230,230,230)");
-
-    this.setState({ brushX, brushXselection });
+    const newState = { brushX, brushXselection };
+    this.setState(newState);
+    return newState;
   }
 
   render() {
     const {
       field,
+      world,
       colorAccessor,
       isUserDefined,
       isDiffExp,
       logFoldChange,
-      pval,
       pvalAdj,
       scatterplotXXaccessor,
       scatterplotYYaccessor,
       zebra
     } = this.props;
-    const field_for_id = field.replace(/\s/g, "_");
+    const fieldForId = field.replace(/\s/g, "_");
+    const {
+      min: unclippedRangeMin,
+      max: unclippedRangeMax
+    } = HistogramBrush.getColumn(world, field, false).summarize();
+    const unclippedRangeMinColor =
+      world.clipQuantiles.min === 0 ? "#bbb" : globals.blue;
+    const unclippedRangeMaxColor =
+      world.clipQuantiles.max === 1 ? "#bbb" : globals.blue;
+
     return (
       <div
-        id={`histogram_${field_for_id}`}
+        id={`histogram_${fieldForId}`}
         data-testid={`histogram-${field}`}
         data-testclass={
           isDiffExp
@@ -397,7 +415,13 @@ class HistogramBrush extends React.Component {
           backgroundColor: zebra ? globals.lightestGrey : "white"
         }}
       >
-        <div style={{ display: "flex", justifyContent: "flex-end" }}>
+        <div
+          style={{
+            display: "flex",
+            justifyContent: "flex-end",
+            paddingBottom: "8px"
+          }}
+        >
           {isDiffExp || isUserDefined ? (
             <span>
               <span
@@ -451,7 +475,7 @@ class HistogramBrush extends React.Component {
         <svg
           width={this.width}
           height={this.height}
-          id={`histogram_${field_for_id}_svg`}
+          id={`histogram_${fieldForId}_svg`}
           data-testclass="histogram-plot"
           data-testid={`histogram-${field}-plot`}
           ref={svgRef => {
@@ -461,15 +485,21 @@ class HistogramBrush extends React.Component {
         <div
           style={{
             display: "flex",
-            justifyContent: "center"
+            justifyContent: "space-between"
           }}
         >
+          <span style={{ color: unclippedRangeMinColor }}>
+            min {unclippedRangeMin.toPrecision(4)}
+          </span>
           <span
             data-testclass="brushable-histogram-field-name"
             style={{ fontStyle: "italic" }}
           >
             {field}
           </span>
+          <span style={{ color: unclippedRangeMaxColor }}>
+            max {unclippedRangeMax.toPrecision(4)}
+          </span>
         </div>
 
         {isDiffExp ? (

client/src/components/categorical/categorical.css

@@ -0,0 +1,3 @@
+:local(.value):hover {
+  background: rgba(167, 182, 194, 0.3);
+}

client/src/components/categorical/categorical.js

@@ -19,13 +19,6 @@ class Categories extends React.Component {
           padding: globals.leftSidebarSectionPadding
         }}
       >
-        <p
-          style={Object.assign({}, globals.leftSidebarSectionHeading, {
-            marginTop: 4
-          })}
-        >
-          Categorical Metadata
-        </p>
         {_.map(categoricalSelection, (catState, catName) => (
           <Category key={catName} metadataField={catName} />
         ))}

client/src/components/categorical/category.js

@@ -27,10 +27,10 @@ class Category extends React.Component {
       const cat = categoricalSelection[metadataField];
       const categoryCount = {
         // total number of categories in this dimension
-        totalCatCount: cat.numCategories,
+        totalCatCount: cat.numCategoryValues,
         // number of selected options in this category
         selectedCatCount: _.reduce(
-          cat.categorySelected,
+          cat.categoryValueSelected,
           (res, cond) => (cond ? res + 1 : res),
           0
         )
@@ -91,7 +91,7 @@ class Category extends React.Component {
     const { categoricalSelection, metadataField } = this.props;
 
     const cat = categoricalSelection[metadataField];
-    const optTuples = sortedCategoryValues([...cat.categoryIndices]);
+    const optTuples = sortedCategoryValues([...cat.categoryValueIndices]);
     return _.map(optTuples, (tuple, i) => (
       <Value
         optTuples={optTuples}

client/src/components/categorical/occupancy.js

@@ -1,69 +1,192 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import * as d3 from "d3";
+import {
+  Popover,
+  PopoverInteractionKind,
+  Position,
+  Classes
+} from "@blueprintjs/core";
 
 @connect()
 class Occupancy extends React.Component {
-  render() {
-    const {
-      occupancy,
-      colorScale,
-      categoricalSelection,
-      colorAccessor,
-      schema
-    } = this.props;
-    const width = 100;
-    const height = 11;
+  _WIDTH = 100;
 
-    const categories = _.filter(schema.annotations.obs, {
-      name: colorAccessor
-    })[0].categories;
+  _HEIGHT = 11;
+
+  createHistogram = () => {
+    /* 
+      Knowing that colorScale is based off continous data, 
+      createHistogram fetches the continous data in relation to the cells releveant to the catagory value.
+      It then seperates that data into 50 bins for drawing the mini-histogram
+    */
+    const {
+      world,
+      metadataField,
+      colorAccessor,
+      category,
+      categoryIndex
+    } = this.props;
+
+    if (!this.canvas) return;
+
+    const groupBy = world.obsAnnotations.col(metadataField);
+
+    const col =
+      world.obsAnnotations.col(colorAccessor) ||
+      world.varData.col(colorAccessor);
+
+    const range = col.summarize();
+
+    const histogramMap = col.histogram(
+      50,
+      [range.min, range.max],
+      groupBy
+    ); /* Because the signature changes we really need different names for histogram to differentiate signatures  */
+
+    const bins = histogramMap.get(category.categoryValues[categoryIndex]);
+
+    const xScale = d3
+      .scaleLinear()
+      .domain([0, bins.length])
+      .range([0, this._WIDTH]);
+
+    const largestBin = Math.max(...bins);
+
+    const yScale = d3
+      .scaleLinear()
+      .domain([0, largestBin])
+      .range([0, this._HEIGHT]);
+
+    const ctx = this.canvas.getContext("2d");
+
+    ctx.fillStyle = "#000";
+
+    let x;
+    let y;
+
+    const rectWidth = this._WIDTH / bins.length;
+
+    for (let i = 0, { length } = bins; i < length; i += 1) {
+      x = xScale(i);
+      y = yScale(bins[i]);
+      ctx.fillRect(x, this._HEIGHT - y, rectWidth, y);
+    }
+  };
+
+  createOccupancyStack = () => {
+    /* 
+      Knowing that the color scale is based off of catagorical data, 
+      createOccupancyStack obtains a map showing the number if cells per colored value
+      Using the colorScale a stack of colored bars is drawn representing the map
+     */
+    const {
+      world,
+      metadataField,
+      colorAccessor,
+      category,
+      categoryIndex,
+      schema,
+      colorScale
+    } = this.props;
+
+    const ctx = this.canvas?.getContext("2d");
+
+    if (!ctx) return;
+
+    const groupBy = world.obsAnnotations.col(metadataField);
+    const occupancyMap = world.obsAnnotations
+      .col(colorAccessor)
+      .histogram(groupBy);
+
+    const occupancy = occupancyMap.get(category.categoryValues[categoryIndex]);
 
     const x = d3
       .scaleLinear()
       /* get all the keys d[1] as an array, then find the sum */
-      .domain([0, d3.sum(Array.from(occupancy, d => d[1]))])
-      .range([0, width]);
+      .domain([0, d3.sum(Array.from(occupancy.values()))])
+      .range([0, this._WIDTH]);
+    const categories = schema.annotations.obsByName[colorAccessor]?.categories;
 
     let currentOffset = 0;
+    const dfColumn = world.obsAnnotations.col(colorAccessor);
+    const categoryValues = dfColumn.summarize().categories;
 
-    const stacks = categoricalSelection[colorAccessor].categoryValues.map(d => {
-      const o = occupancy.get(d);
+    let o;
+    let scaledValue;
+    let value;
 
-      const scaledValue = x(o);
-
-      const stackItem = {
-        key: d,
-        value: o || 0,
-        rectWidth: o ? scaledValue : 0,
-        offset: currentOffset,
-        fill: o ? colorScale(categories.indexOf(d)) : "rgb(255,255,255)"
-      };
+    for (let i = 0, { length } = categoryValues; i < length; i += 1) {
+      value = categoryValues[i];
+      o = occupancy.get(value);
+      scaledValue = x(o);
+      ctx.fillStyle = o
+        ? colorScale(categories.indexOf(value))
+        : "rgb(255,255,255)";
+      ctx.fillRect(currentOffset, 0, o ? scaledValue : 0, this._HEIGHT);
       currentOffset += o ? scaledValue : 0;
-      return stackItem;
-    });
+    }
+  };
+
+  render() {
+    const {
+      colorAccessor,
+      categoricalSelection,
+      category,
+      categoryIndex
+    } = this.props;
+
+    this.canvas?.getContext("2d").clearRect(0, 0, this._WIDTH, this._HEIGHT);
+
+    const colorByIsCatagoricalData = !!categoricalSelection[colorAccessor];
 
     return (
-      <svg
-        style={{
-          marginRight: 5,
-          width,
-          height
+      <Popover
+        interactionKind={PopoverInteractionKind.HOVER_TARGET_ONLY}
+        hoverOpenDelay={1500}
+        hoverCloseDelay={200}
+        position={Position.LEFT}
+        modifiers={{
+          preventOverflow: { enabled: false },
+          hide: { enabled: false }
         }}
+        lazy
+        usePortal
+        disabled={colorByIsCatagoricalData}
+        popoverClassName={Classes.POPOVER_CONTENT_SIZING}
       >
-        {stacks.map(d => (
-          <rect
-            key={d.key}
-            width={d.rectWidth}
-            height={height}
-            x={d.offset}
-            title={d.metadataField}
-            fill={d.fill}
-          />
-        ))}
-      </svg>
+        <canvas
+          className="bp3-popover-targer"
+          style={{
+            marginRight: 5,
+            width: this._WIDTH,
+            height: this._HEIGHT,
+            borderBottom: colorByIsCatagoricalData
+              ? ""
+              : "solid rgb(230, 230, 230) 0.25px"
+          }}
+          width={this._WIDTH}
+          height={this._HEIGHT}
+          ref={ref => {
+            this.canvas = ref;
+            if (colorByIsCatagoricalData) this.createOccupancyStack();
+            else this.createHistogram();
+          }}
+        />
+        <div key="text" style={{ fontFamily: "Roboto", fontSize: "14px" }}>
+          <p style={{ margin: "0" }}>
+            This histograms shows the distribution of{" "}
+            <strong>{colorAccessor}</strong> within{" "}
+            <strong>{category.categoryValues[categoryIndex]}</strong>.
+            <br />
+            <br />
+            The x axis is the same for each histogram, while the y axis is
+            scaled to the largest bin within this histogram instead of the
+            largest bin within the whole category.
+          </p>
+        </div>
+      </Popover>
     );
   }
 }

client/src/components/categorical/util.js

@@ -5,7 +5,6 @@
 // return sorted index
 
 import isNumber from "is-number";
-import _ from "lodash";
 
 const sortedCategoryValues = values => {
   /* this sort could be memoized for perf */
@@ -13,7 +12,7 @@ const sortedCategoryValues = values => {
   const strings = [];
   const ints = [];
 
-  _.forEach(values, v => {
+  values.forEach(v => {
     if (isNumber(v[0])) {
       ints.push(v);
     } else {

client/src/components/categorical/value.js

@@ -1,36 +1,80 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
-import _ from "lodash";
 import Occupancy from "./occupancy";
-import { countCategoryValues2D } from "../../util/stateManager/worldUtil";
 import * as globals from "../../globals";
+import styles from "./categorical.css";
 
 @connect(state => ({
   categoricalSelection: state.categoricalSelection,
   colorScale: state.colors.scale,
   colorAccessor: state.colors.colorAccessor,
-  schema: _.get(state.world, "schema", null),
+  schema: state.world?.schema,
   world: state.world
 }))
 class CategoryValue extends React.Component {
-  toggleOff() {
+  toggleOff = () => {
     const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter deselect",
       metadataField,
       categoryIndex
     });
-  }
+  };
 
-  toggleOn() {
+  shouldComponentUpdate = nextProps => {
+    /* 
+    Checks to see if at least one of the following changed:
+    * world state
+    * the color accessor (what is currently being colored by)
+    * if this catagorical value's selection status has changed
+    
+    If and only if true, update the component
+    */
+    const { props } = this;
+    const { metadataField, categoryIndex, categoricalSelection } = props;
+    const { categoricalSelection: newCategoricalSelection } = nextProps;
+
+    const valueSelectionChange =
+      categoricalSelection[metadataField].categoryValueSelected[
+        categoryIndex
+      ] !==
+      newCategoricalSelection[metadataField].categoryValueSelected[
+        categoryIndex
+      ];
+
+    const worldChange = props.world !== nextProps.world;
+    const colorAccessorChange = props.colorAccessor !== nextProps.colorAccessor;
+
+    return valueSelectionChange || worldChange || colorAccessorChange;
+  };
+
+  toggleOn = () => {
     const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter select",
       metadataField,
       categoryIndex
     });
-  }
+  };
+
+  handleMouseEnter = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "category value mouse hover start",
+      metadataField,
+      categoryIndex
+    });
+  };
+
+  handleMouseExit = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "category value mouse hover end",
+      metadataField,
+      categoryIndex
+    });
+  };
 
   render() {
     const {
@@ -40,15 +84,14 @@ class CategoryValue extends React.Component {
       colorAccessor,
       colorScale,
       i,
-      schema,
-      world
+      schema
     } = this.props;
 
     if (!categoricalSelection) return null;
 
     const category = categoricalSelection[metadataField];
-    const selected = category.categorySelected[categoryIndex];
-    const count = category.categoryCounts[categoryIndex];
+    const selected = category.categoryValueSelected[categoryIndex];
+    const count = category.categoryValueCounts[categoryIndex];
     const value = category.categoryValues[categoryIndex];
     const displayString = String(
       category.categoryValues[categoryIndex]
@@ -57,31 +100,26 @@ class CategoryValue extends React.Component {
     /* this is the color scale, so add swatches below */
     const isColorBy = metadataField === colorAccessor;
     let categories = null;
-    let occupancy = null;
 
     if (isColorBy && schema) {
-      categories = _.filter(schema.annotations.obs, {
-        name: colorAccessor
-      })[0].categories;
-    }
-
-    if (colorAccessor && !isColorBy && categoricalSelection[colorAccessor]) {
-      occupancy = countCategoryValues2D(
-        metadataField,
-        colorAccessor,
-        world.obsAnnotations
-      );
+      categories = schema.annotations.obsByName[colorAccessor]?.categories;
     }
 
     return (
       <div
         key={i}
+        className={styles.value}
+        data-testclass="categorical-row"
         style={{
+          padding: "4px 7px",
           display: "flex",
           alignItems: "baseline",
-          justifyContent: "space-between"
+          justifyContent: "space-between",
+          marginBottom: "2px",
+          borderRadius: "2px"
         }}
-        data-testclass="categorical-row"
+        onMouseEnter={this.handleMouseEnter}
+        onMouseLeave={this.handleMouseExit}
       >
         <div
           style={{
@@ -93,34 +131,32 @@ class CategoryValue extends React.Component {
             justifyContent: "space-between"
           }}
         >
-          <label className="bp3-control bp3-checkbox">
-            <input
-              onChange={
-                selected ? this.toggleOff.bind(this) : this.toggleOn.bind(this)
-              }
-              data-testclass="categorical-value-select"
-              data-testid={`categorical-value-select-${metadataField}-${displayString}`}
-              checked={selected}
-              type="checkbox"
-            />
-            <span className="bp3-control-indicator" />
+          <div style={{ display: "flex" }}>
+            <label className="bp3-control bp3-checkbox" style={{ margin: 0 }}>
+              <input
+                onChange={selected ? this.toggleOff : this.toggleOn}
+                data-testclass="categorical-value-select"
+                data-testid={`categorical-value-select-${metadataField}-${displayString}`}
+                checked={selected}
+                type="checkbox"
+              />
+              <span
+                className="bp3-control-indicator"
+                onMouseEnter={this.handleMouseExit}
+                onMouseLeave={this.handleMouseEnter}
+              />
+            </label>
             <span
               data-testid={`categorical-value-${metadataField}-${displayString}`}
               data-testclass="categorical-value"
+              style={{ wordBreak: "break-all" }}
             >
               {displayString}
             </span>
-          </label>
+          </div>
           <span style={{ flexShrink: 0 }}>
-            {colorAccessor &&
-            !isColorBy &&
-            categoricalSelection[colorAccessor] ? (
-              <Occupancy
-                occupancy={occupancy.get(
-                  category.categoryValues[categoryIndex]
-                )}
-                {...this.props}
-              />
+            {colorAccessor && !isColorBy ? (
+              <Occupancy category={category} {...this.props} />
             ) : null}
           </span>
         </div>

client/src/components/continuous/continuous.js

@@ -9,10 +9,10 @@ import * as globals from "../../globals";
 import HistogramBrush from "../brushableHistogram";
 
 @connect(state => ({
-  obsAnnotations: _.get(state.world, "obsAnnotations", null),
+  obsAnnotations: state.world?.obsAnnotations,
   colorAccessor: state.colors.colorAccessor,
   colorScale: state.colors.scale,
-  schema: _.get(state.world, "schema", null)
+  schema: state.world?.schema
 }))
 class Continuous extends React.Component {
   constructor(props) {
@@ -48,7 +48,7 @@ class Continuous extends React.Component {
     }
 
     /* initial value for iterator to simulate index, ranges is an object */
-    let zebra = -1;
+    let zebra = 0;
 
     return (
       <div>
@@ -64,18 +64,19 @@ class Continuous extends React.Component {
         ) : null}
         {obsAnnotations
           ? _.map(obsAnnotations.colIndex.keys(), key => {
-              const summary = obsAnnotations.col(key).summarize();
               const isColorField =
                 key.includes("color") || key.includes("Color");
+              if (key === schema.annotations.obs.index || isColorField)
+                return null;
+
+              const summary = obsAnnotations.col(key).summarize();
               const nonFiniteExtent =
-                summary.min === undefined || summary.max === undefined;
-              zebra += 1;
-              if (
-                !summary.categorical &&
-                key !== "name" &&
-                !isColorField &&
-                !nonFiniteExtent
-              ) {
+                summary.min === undefined ||
+                summary.max === undefined ||
+                Number.isNaN(summary.min) ||
+                Number.isNaN(summary.max);
+              if (!summary.categorical && !nonFiniteExtent) {
+                zebra += 1;
                 return (
                   <HistogramBrush
                     key={key}

client/src/components/continuousLegend/index.js

@@ -2,7 +2,7 @@
 import React from "react";
 import { connect } from "react-redux";
 import * as d3 from "d3";
-import { interpolateViridis, interpolateCool } from "d3-scale-chromatic";
+import { interpolateCool } from "d3-scale-chromatic";
 
 // create continuous color legend
 // http://bl.ocks.org/syntagmatic/e8ccca52559796be775553b467593a9f
@@ -112,6 +112,7 @@ class ContinuousLegend extends React.Component {
     const { colorAccessor, responsive, colorScale } = this.props;
     if (
       prevProps.colorAccessor !== colorAccessor ||
+      prevProps.colorScale !== colorScale ||
       prevProps.responsive.height !== responsive.height ||
       prevProps.responsive.width !== responsive.width
     ) {

client/src/components/framework/logo.js

@@ -0,0 +1,18 @@
+import React from "react";
+import * as globals from "../../globals";
+
+const Logo = props => {
+  const { size } = props;
+  return (
+    <svg width={size} height={size} viewBox="0 0 48 48" fill="none">
+      <rect width="48" height="48" fill="white" />
+      <rect width="48" height="48" fill={globals.logoColor} />
+      <rect x="19" y="19" width="22" height="22" fill="white" />
+      <rect x="24" y="24" width="12" height="12" fill={globals.logoColor} />
+      <rect x="7" y="19" width="7" height="22" fill="white" />
+      <rect x="19" y="7" width="22" height="7" fill="white" />
+    </svg>
+  );
+};
+
+export default Logo;

client/src/components/geneExpression/expressionButtons.js

@@ -1,101 +0,0 @@
-// jshint esversion: 6
-import React from "react";
-import _ from "lodash";
-import { Button, AnchorButton, Tooltip } from "@blueprintjs/core";
-import { connect } from "react-redux";
-import * as globals from "../../globals";
-import actions from "../../actions";
-import CellSetButton from "./cellSetButtons";
-
-@connect(state => ({
-  differential: state.differential,
-  world: state.world,
-  crossfilter: state.crossfilter
-}))
-class Expression extends React.Component {
-  constructor(props) {
-    super(props);
-    this.state = {};
-  }
-
-  computeDiffExp() {
-    const { dispatch, differential } = this.props;
-    if (differential.celllist1 && differential.celllist2) {
-      dispatch(
-        actions.requestDifferentialExpression(
-          differential.celllist1,
-          differential.celllist2
-        )
-      );
-    }
-  }
-
-  clearDifferentialExpression() {
-    const { dispatch, differential } = this.props;
-    dispatch({
-      type: "clear differential expression",
-      diffExp: differential.diffExp
-    });
-    dispatch({
-      type: "clear scatterplot"
-    });
-  }
-
-  render() {
-    const { differential } = this.props;
-    if (!differential) {
-      return null;
-    }
-    const haveBothCellSets =
-      !!differential.celllist1 && !!differential.celllist2;
-    return (
-      <div
-        style={{
-          marginRight: 10,
-          marginBottom: 10,
-          paddingLeft: globals.leftSidebarSectionPadding
-        }}
-      >
-        <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
-        <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
-        {!differential.diffExp ? (
-          <Tooltip
-            content="Add two cells selections, see the top 15 differentially expressed genes between them"
-            position="bottom"
-          >
-            <AnchorButton
-              style={{ marginTop: 10 }}
-              disabled={!haveBothCellSets}
-              intent="primary"
-              data-testid="diffexp-button"
-              loading={differential.loading}
-              fill
-              type="button"
-              onClick={this.computeDiffExp.bind(this)}
-            >
-              Compute Differential Expression
-            </AnchorButton>
-          </Tooltip>
-        ) : null}
-        {differential.diffExp ? (
-          <Tooltip
-            content="Remove differentially expressed gene list and clear cell selections"
-            position="bottom"
-          >
-            <Button
-              type="button"
-              fill
-              style={{ marginTop: 10 }}
-              intent="warning"
-              onClick={this.clearDifferentialExpression.bind(this)}
-            >
-              Clear Differential Expression
-            </Button>
-          </Tooltip>
-        ) : null}
-      </div>
-    );
-  }
-}
-
-export default Expression;

client/src/components/geneExpression/index.js

@@ -21,8 +21,6 @@ import {
   postUserErrorToast,
   keepAroundErrorToast
 } from "../framework/toasters";
-import ExpressionButtons from "./expressionButtons";
-import finiteExtent from "../../util/finiteExtent";
 
 const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
   if (!modifiers.matchesPredicate) {
@@ -58,7 +56,7 @@ const filterGenes = (query, genes) =>
 
 @connect(state => {
   return {
-    obsAnnotations: _.get(state.world, "obsAnnotations", null),
+    obsAnnotations: state.world?.obsAnnotations,
     userDefinedGenes: state.controls.userDefinedGenes,
     userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
     world: state.world,
@@ -86,7 +84,8 @@ class GeneExpression extends React.Component {
     */
     const { world } = this.props;
     const { varAnnotations } = world;
-    const geneNames = varAnnotations.col("name").asArray();
+    const varIndexName = world.schema.annotations.var.index;
+    const geneNames = varAnnotations.col(varIndexName).asArray();
     if (geneNames.length > 0) {
       const placeholder = [];
       let len = geneNames.length;
@@ -108,6 +107,7 @@ class GeneExpression extends React.Component {
 
   handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
     const gene = g.target;
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
@@ -115,17 +115,22 @@ class GeneExpression extends React.Component {
       postUserErrorToast(
         "That's too many genes, you can have at most 15 user defined genes"
       );
-    } else if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
+    } else if (
+      world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+    ) {
       postUserErrorToast("That doesn't appear to be a valid gene name.");
     } else {
       dispatch({ type: "single user defined gene start" });
-      dispatch(actions.requestUserDefinedGene(gene));
-      dispatch({ type: "single user defined gene complete" });
+      dispatch(actions.requestUserDefinedGene(gene)).then(
+        () => dispatch({ type: "single user defined gene complete" }),
+        () => dispatch({ type: "single user defined gene error" })
+      );
     }
   }
 
   handleBulkAddClick() {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
     const { bulkAdd } = this.state;
 
     /*
@@ -136,22 +141,27 @@ class GeneExpression extends React.Component {
       const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");
 
       dispatch({ type: "bulk user defined gene start" });
-      genes.forEach(gene => {
-        if (gene.length === 0) {
-          keepAroundErrorToast("Must enter a gene name.");
-        } else if (userDefinedGenes.indexOf(gene) !== -1) {
-          keepAroundErrorToast("That gene already exists");
-        } else if (
-          world.varAnnotations.col("name").indexOf(gene) === undefined
-        ) {
-          keepAroundErrorToast(
-            `${gene} doesn't appear to be a valid gene name.`
-          );
-        } else {
-          dispatch(actions.requestUserDefinedGene(gene));
-        }
-      });
-      dispatch({ type: "bulk user defined gene complete" });
+      Promise.all(
+        genes.map(gene => {
+          if (gene.length === 0) {
+            return keepAroundErrorToast("Must enter a gene name.");
+          }
+          if (userDefinedGenes.indexOf(gene) !== -1) {
+            return keepAroundErrorToast("That gene already exists");
+          }
+          if (
+            world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+          ) {
+            return keepAroundErrorToast(
+              `${gene} doesn't appear to be a valid gene name.`
+            );
+          }
+          return dispatch(actions.requestUserDefinedGene(gene));
+        })
+      ).then(
+        () => dispatch({ type: "bulk user defined gene complete" }),
+        () => dispatch({ type: "bulk user defined gene error" })
+      );
     }
 
     this.setState({ bulkAdd: "" });
@@ -164,24 +174,12 @@ class GeneExpression extends React.Component {
       userDefinedGenesLoading,
       differential
     } = this.props;
-
+    const varIndexName = world?.schema?.annotations?.var?.index;
     const { tab, bulkAdd } = this.state;
 
     return (
       <div>
-        <div
-          style={{
-            marginTop: 30
-          }}
-        >
-          <p
-            style={Object.assign({}, globals.leftSidebarSectionHeading, {
-              paddingLeft: globals.leftSidebarSectionPadding,
-              margin: 0
-            })}
-          >
-            Selected Genes
-          </p>
+        <div>
           <div
             style={{
               padding: globals.leftSidebarSectionPadding
@@ -197,7 +195,7 @@ class GeneExpression extends React.Component {
                 this.setState({ tab: "autosuggest" });
               }}
             >
-              Autosuggest
+              Autosuggest genes
             </Button>
             <Button
               active={tab === "bulkadd"}
@@ -220,9 +218,9 @@ class GeneExpression extends React.Component {
               }}
             >
               <Suggest
-                closeOnSelect
-                openOnKeyDown
                 resetOnSelect
+                closeOnSelect
+                resetOnClose
                 itemDisabled={
                   userDefinedGenesLoading ? () => true : () => false
                 }
@@ -239,7 +237,7 @@ class GeneExpression extends React.Component {
                 itemRenderer={renderGene.bind(this)}
                 items={
                   world && world.varAnnotations
-                    ? world.varAnnotations.col("name").asArray()
+                    ? world.varAnnotations.col(varIndexName).asArray()
                     : ["No genes"]
                 }
                 popoverProps={{ minimal: true }}
@@ -249,7 +247,7 @@ class GeneExpression extends React.Component {
                 data-testid={"add-gene"}
                 loading={userDefinedGenesLoading}
               >
-                Add
+                Add gene
               </Button>
             </ControlGroup>
           ) : null}
@@ -280,7 +278,7 @@ class GeneExpression extends React.Component {
                       onClick={this.handleBulkAddClick.bind(this)}
                       loading={userDefinedGenesLoading}
                     >
-                      Add
+                      Add genes
                     </Button>
                   </ControlGroup>
                 </FormGroup>
@@ -307,18 +305,9 @@ class GeneExpression extends React.Component {
             : null}
         </div>
         <div>
-          <p
-            style={Object.assign({}, globals.leftSidebarSectionHeading, {
-              marginTop: 40,
-              paddingLeft: globals.leftSidebarSectionPadding
-            })}
-          >
-            Differentially Expressed Genes
-          </p>
-          <ExpressionButtons />
           {differential.diffExp
             ? _.map(differential.diffExp, (value, index) => {
-                const name = world.varAnnotations.at(value[0], "name");
+                const name = world.varAnnotations.at(value[0], varIndexName);
                 const values = world.varData.col(name);
                 if (!values) {
                   return null;

client/src/components/graph/drawPointsRegl.js

@@ -1,5 +1,6 @@
 // jshint esversion: 6
 const mat4 = require("gl-mat4");
+const vec3 = require("gl-vec3");
 
 // opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js
 
@@ -38,7 +39,20 @@ export default function(regl) {
     uniforms: {
       distance: regl.prop("distance"),
       view: regl.prop("view"),
-      projection: ({viewportWidth, viewportHeight}) => mat4.perspective([], Math.PI / 2, viewportWidth / viewportHeight, 0.01, 1000)
+      projection: ({ viewportWidth, viewportHeight }) => {
+        const aspectRatio = viewportWidth / viewportHeight;
+        let m = mat4.perspective(
+          [],
+          Math.PI / 2,
+          viewportWidth / viewportHeight,
+          0.01,
+          1000
+        );
+        if (aspectRatio < 1) {
+          m = mat4.scale(m, m, vec3.fromValues(1, 1, 1 / aspectRatio));
+        }
+        return m;
+      }
     },
 
     count: regl.prop("count"),

client/src/components/graph/graph.js

@@ -1,66 +1,109 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import * as d3 from "d3";
 import { connect } from "react-redux";
 import mat4 from "gl-mat4";
 import _regl from "regl";
-import {
-  Button,
-  AnchorButton,
-  Tooltip,
-  Popover,
-  Menu,
-  MenuItem,
-  Position
-} from "@blueprintjs/core";
+import memoize from "memoize-one";
 
 import * as globals from "../../globals";
 import setupSVGandBrushElements from "./setupSVGandBrush";
+import setupCentroidSVG from "./setupCentroidSVG";
 import actions from "../../actions";
 import _camera from "../../util/camera";
 import _drawPoints from "./drawPointsRegl";
 import scaleLinear from "../../util/scaleLinear";
-import { World } from "../../util/stateManager";
 
 /* https://bl.ocks.org/mbostock/9078690 - quadtree for onClick / hover selections */
 
 @connect(state => ({
   world: state.world,
-  universe: state.universe,
   crossfilter: state.crossfilter,
   responsive: state.responsive,
   colorRGB: state.colors.rgb,
-  opacityForDeselectedCells: state.controls.opacityForDeselectedCells,
-  resettingInterface: state.controls.resettingInterface,
-  userDefinedGenes: state.controls.userDefinedGenes,
-  diffexpGenes: state.controls.diffexpGenes,
-  colorAccessor: state.colors.colorAccessor,
-  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
-  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
-  celllist1: state.differential.celllist1,
-  celllist2: state.differential.celllist2,
-  library_versions: _.get(state.config, "library_versions", null),
-  undoDisabled: state["@@undoable/past"].length === 0,
-  redoDisabled: state["@@undoable/future"].length === 0
+  selectionTool: state.graphSelection.tool,
+  currentSelection: state.graphSelection.selection,
+  layoutChoice: state.layoutChoice,
+  centroidLabel: state.centroidLabel,
+  graphInteractionMode: state.controls.graphInteractionMode,
+  colorAccessor: state.colors.colorAccessor
 }))
 class Graph extends React.Component {
+  computePointPositions = memoize((X, Y, scaleX, scaleY) => {
+    /*
+    compute webgl coordinate buffer for each point
+    */
+    const positions = new Float32Array(2 * X.length);
+    for (let i = 0, len = X.length; i < len; i += 1) {
+      positions[2 * i] = scaleX(X[i]);
+      positions[2 * i + 1] = scaleY(Y[i]);
+    }
+    return positions;
+  });
+
+  computePointColors = memoize(rgb => {
+    /*
+    compute webgl colors for each point
+    */
+    const colors = new Float32Array(3 * rgb.length);
+    for (let i = 0, len = rgb.length; i < len; i += 1) {
+      colors.set(rgb[i], 3 * i);
+    }
+    return colors;
+  });
+
+  computePointSizesFromCrossfilter = memoize((len, crossfilter) => {
+    const sizes = new Float32Array(len);
+    crossfilter.fillByIsSelected(sizes, 4, 0.2);
+
+    return sizes;
+  });
+
+  computePointSizes = memoize(
+    (len, crossfilter, metadataField, categoryField) => {
+      /*
+    compute webgl dot size for each point
+    */
+
+      const selectionSizes = this.computePointSizesFromCrossfilter(
+        len,
+        crossfilter
+      );
+      let sizes;
+
+      if (metadataField && categoryField) {
+        sizes = selectionSizes.slice();
+        const valuesArr = crossfilter.data.col(metadataField).asArray();
+
+        for (let i = 0; i < len; i += 1) {
+          if (valuesArr[i] === categoryField) {
+            sizes[i] = 10;
+          }
+        }
+      } else {
+        sizes = selectionSizes;
+      }
+      return sizes;
+    }
+  );
+
   constructor(props) {
     super(props);
     this.count = 0;
-    this.inverse = mat4.identity([]);
     this.graphPaddingTop = 0;
-    this.graphPaddingBottom = 45;
     this.graphPaddingRight = globals.leftSidebarWidth;
     this.renderCache = {
+      X: null,
+      Y: null,
       positions: null,
       colors: null,
       sizes: null
     };
     this.state = {
-      svg: null,
-      brush: null,
-      mode: "lasso"
+      toolSVG: null,
+      centroidSVG: null,
+      tool: null,
+      container: null
     };
   }
 
@@ -76,6 +119,20 @@ class Graph extends React.Component {
     const colorBuffer = regl.buffer();
     const sizeBuffer = regl.buffer();
 
+    // preallocate coordinate system transformation between data and gl
+    const fractionToUse = 0.93; // fraction of dimension to use
+    const shiftForMenuBar = 0.05;
+    const transform = {
+      glScaleX: scaleLinear([0, 1], [-1 * fractionToUse, 1 * fractionToUse]),
+      glScaleY: scaleLinear(
+        [0, 1],
+        [
+          (1 + shiftForMenuBar) * fractionToUse,
+          (-1 + shiftForMenuBar) * fractionToUse
+        ]
+      )
+    };
+
     /* first time, but this duplicates above function, should be possile to avoid this */
     const reglRender = regl.frame(() => {
       this.reglDraw(
@@ -98,82 +155,91 @@ class Graph extends React.Component {
       colorBuffer,
       sizeBuffer,
       camera,
-      reglRender
+      reglRender,
+      transform
     });
   }
 
   componentDidUpdate(prevProps) {
     const { renderCache } = this;
-    const { world, crossfilter, colorRGB, responsive } = this.props;
     const {
-      reglRender,
-      mode,
-      regl,
-      drawPoints,
-      camera,
-      pointBuffer,
-      colorBuffer,
-      sizeBuffer,
-      svg
-    } = this.state;
+      world,
+      crossfilter,
+      colorRGB,
+      responsive,
+      selectionTool,
+      currentSelection,
+      layoutChoice,
+      graphInteractionMode,
+      colorAccessor,
+      centroidLabel
+    } = this.props;
+    const { reglRender, mode, regl, toolSVG, centroidSVG } = this.state;
+    let stateChanges = {};
 
-    if (reglRender && this.reglRenderState === "rendering" && mode !== "zoom") {
-      reglRender.cancel();
-      this.reglRenderState = "paused";
+    if (reglRender) {
+      if (
+        // If it IS RENDERING and it is NOT IN ZOOM mode, stop rendering.
+        this.reglRenderState === "rendering" &&
+        graphInteractionMode !== "zoom"
+      ) {
+        reglRender.cancel();
+        this.reglRenderState = "paused";
+      }
+
+      if (
+        // If it is NOT RENDERING and it IS IN ZOOM mode, start rendering
+        this.reglRenderState !== "rendering" &&
+        graphInteractionMode === "zoom"
+      ) {
+        this.restartReglLoop();
+        this.reglRenderState = "rendering";
+      }
     }
 
     if (regl && world) {
       /* update the regl state */
       const { obsLayout, nObs } = world;
-      const X = obsLayout.col("X").asArray();
-      const Y = obsLayout.col("Y").asArray();
+      const {
+        drawPoints,
+        transform,
+        camera,
+        pointBuffer,
+        colorBuffer,
+        sizeBuffer
+      } = this.state;
 
-      // X/Y positions for each point - a cached value that only
-      // changes if we have loaded entirely new cell data
-      //
-      if (!renderCache.positions || world !== prevProps.world) {
-        renderCache.positions = new Float32Array(2 * nObs);
-
-        const glScaleX = scaleLinear([0, 1], [-1, 1]);
-        const glScaleY = scaleLinear([0, 1], [1, -1]);
-
-        const offset = [d3.mean(X) - 0.5, d3.mean(Y) - 0.5];
-
-        for (let i = 0, { positions } = renderCache; i < nObs; i += 1) {
-          positions[2 * i] = glScaleX(X[i] - offset[0]);
-          positions[2 * i + 1] = glScaleY(Y[i] - offset[1]);
-        }
-        pointBuffer({
-          data: renderCache.positions,
-          dimension: 2
-        });
-
-        this.setState({
-          offset
-        });
+      /* coordinates for each point */
+      const { glScaleX, glScaleY } = transform;
+      const X = obsLayout.col(layoutChoice.currentDimNames[0]).asArray();
+      const Y = obsLayout.col(layoutChoice.currentDimNames[1]).asArray();
+      const newPositions = this.computePointPositions(X, Y, glScaleX, glScaleY);
+      if (renderCache.positions !== newPositions) {
+        /* update our cache & GL if the buffer changes */
+        renderCache.positions = newPositions;
+        pointBuffer({ data: newPositions, dimension: 2 });
       }
 
-      // Colors for each point - a cached value that only changes when
-      // the cell metadata changes.
-      if (!renderCache.colors || colorRGB !== prevProps.colorRGB) {
-        const rgb = colorRGB;
-        if (!renderCache.colors) {
-          renderCache.colors = new Float32Array(3 * rgb.length);
-        }
-        for (let i = 0, { colors } = renderCache; i < rgb.length; i += 1) {
-          colors.set(rgb[i], 3 * i);
-        }
-        colorBuffer({ data: renderCache.colors, dimension: 3 });
+      /* colors for each point */
+      const newColors = this.computePointColors(colorRGB);
+      if (renderCache.colors !== newColors) {
+        /* update our cache & GL if the buffer changes */
+        renderCache.colors = newColors;
+        colorBuffer({ data: newColors, dimension: 3 });
       }
 
-      // Sizes for each point - updates are triggered only when selected
-      // obs change
-      if (!renderCache.sizes || crossfilter !== prevProps.crossfilter) {
-        if (!renderCache.sizes) {
-          renderCache.sizes = new Float32Array(nObs);
-        }
-        crossfilter.fillByIsSelected(renderCache.sizes, 4, 0.2);
-        sizeBuffer({ data: renderCache.sizes, dimension: 1 });
+      /* sizes for each point */
+      const { metadataField, categoryField } = centroidLabel;
+      const newSizes = this.computePointSizes(
+        nObs,
+        crossfilter,
+        metadataField,
+        categoryField
+      );
+      if (renderCache.sizes !== newSizes) {
+        /* update our cache & GL if the buffer changes */
+        renderCache.size = newSizes;
+        sizeBuffer({ data: newSizes, dimension: 1 });
       }
 
       this.count = nObs;
@@ -189,77 +255,183 @@ class Graph extends React.Component {
       );
     }
 
-    if (
-      prevProps.responsive.height !== responsive.height ||
-      prevProps.responsive.width !== responsive.width ||
-      /* first time */
-      (responsive.height && responsive.width && !svg)
-    ) {
+    const createToolSVG = () => {
       /* clear out whatever was on the div, even if nothing, but usually the brushes etc */
       d3.select("#graphAttachPoint")
-        .selectAll("svg")
+        .select("#tool")
         .remove();
-      const { svg: newSvg, brush } = setupSVGandBrushElements(
-        this.handleBrushSelectAction.bind(this),
-        this.handleBrushDeselectAction.bind(this),
+
+      let handleStart;
+      let handleDrag;
+      let handleEnd;
+      let handleCancel;
+      if (selectionTool === "brush") {
+        handleStart = this.handleBrushStartAction.bind(this);
+        handleDrag = this.handleBrushDragAction.bind(this);
+        handleEnd = this.handleBrushEndAction.bind(this);
+      } else {
+        handleStart = this.handleLassoStart.bind(this);
+        handleEnd = this.handleLassoEnd.bind(this);
+        handleCancel = this.handleLassoCancel.bind(this);
+      }
+
+      const { svg: newToolSVG, tool, container } = setupSVGandBrushElements(
+        selectionTool,
+        handleStart,
+        handleDrag,
+        handleEnd,
+        handleCancel,
         responsive,
         this.graphPaddingRight,
-        this.handleLassoStart.bind(this),
-        this.handleLassoEnd.bind(this)
+        graphInteractionMode
       );
-      this.setState({ svg: newSvg, brush });
+
+      stateChanges = { ...stateChanges, toolSVG: newToolSVG, tool, container };
+    };
+
+    const createCentroidSVG = () => {
+      d3.select("#graphAttachPoint")
+        .select("#centroid-container")
+        .remove();
+
+      if (centroidLabel.metadataField === "" || !centroidLabel.centroidXY) {
+        return;
+      }
+
+      const centroidScreen = this.mapPointToScreen(centroidLabel.centroidXY);
+
+      const newCentroidSVG = setupCentroidSVG(
+        responsive,
+        this.graphPaddingRight,
+        centroidScreen,
+        centroidLabel.categoryField,
+        colorAccessor
+      );
+
+      stateChanges = { ...stateChanges, centroidSVG: newCentroidSVG };
+    };
+
+    // Centroid SVG creation is disabled for now but should go into the first and third cases if enabled
+    if (
+      prevProps.responsive.height !== responsive.height ||
+      prevProps.responsive.width !== responsive.width
+    ) {
+      // If the window size has changed we want to recreate all SVGs
+      createToolSVG();
+    } else if (
+      (responsive.height && responsive.width && !toolSVG) ||
+      selectionTool !== prevProps.selectionTool ||
+      prevProps.graphInteractionMode !== graphInteractionMode
+    ) {
+      // first time or change of selection tool6
+      createToolSVG();
+    } else if (
+      centroidLabel !== prevProps.centroidLabel ||
+      (responsive.height && responsive.width && !centroidSVG)
+    ) {
+      // First time for centroid or label change
+    }
+
+    /*
+    if the selection tool or state has changed, ensure that the selection
+    tool correctly reflects the underlying selection.
+    */
+    if (
+      currentSelection !== prevProps.currentSelection ||
+      graphInteractionMode !== prevProps.graphInteractionMode ||
+      stateChanges.toolSVG
+    ) {
+      const { tool, container } = this.state;
+      this.selectionToolUpdate(
+        stateChanges.tool ? stateChanges.tool : tool,
+        stateChanges.container ? stateChanges.container : container
+      );
+    }
+
+    if (Object.keys(stateChanges).length > 0) {
+      this.setState(stateChanges);
     }
   }
 
-  isResetDisabled = () => {
+  brushToolUpdate(tool, container) {
     /*
-    Reset should be disabled when all of the following are true:
-      * nothing is selected in the crossfilter
-      * world EQ universe
-      * nothing is colored by
-      * there are no userDefinedGenes or diffexpGenes displayed
-      * scatterplot is not displayed
-      * nothing in cellset1 or cellset2
+    this is called from componentDidUpdate(), so be very careful using
+    anything from this.state, which may be updated asynchronously.
     */
-    const {
-      crossfilter,
-      world,
-      universe,
-      userDefinedGenes,
-      diffexpGenes,
-      colorAccessor,
-      scatterplotXXaccessor,
-      scatterplotYYaccessor,
-      celllist1,
-      celllist2
-    } = this.props;
+    const { currentSelection } = this.props;
+    if (container) {
+      const toolCurrentSelection = d3.brushSelection(container.node());
 
-    if (!crossfilter || !world || !universe) {
-      return false;
+      if (currentSelection.mode === "within-rect") {
+        /*
+        if there is a selection, make sure the brush tool matches
+        */
+        const screenCoords = [
+          this.mapPointToScreen(currentSelection.brushCoords.northwest),
+          this.mapPointToScreen(currentSelection.brushCoords.southeast)
+        ];
+        if (!toolCurrentSelection) {
+          /* tool is not selected, so just move the brush */
+          container.call(tool.move, screenCoords);
+        } else {
+          /* there is an active selection and a brush - make sure they match */
+          /* this just sums the difference of each dimension, of each point */
+          let delta = 0;
+          for (let x = 0; x < 2; x += 1) {
+            for (let y = 0; y < 2; y += 1) {
+              delta += Math.abs(
+                screenCoords[x][y] - toolCurrentSelection[x][y]
+              );
+            }
+          }
+          if (delta > 0) {
+            container.call(tool.move, screenCoords);
+          }
+        }
+      } else if (toolCurrentSelection) {
+        /* no selection, so clear the brush tool if it is set */
+        container.call(tool.move, null);
+      }
     }
-    const nothingSelected = crossfilter.countSelected() === crossfilter.size();
-    const nothingColoredBy = !colorAccessor;
-    const noGenes = userDefinedGenes.length === 0 && diffexpGenes.length === 0;
-    const scatterNotDpl = !scatterplotXXaccessor || !scatterplotYYaccessor;
-    const nothingInCellsets = !celllist1 && !celllist2;
+  }
 
-    return (
-      nothingSelected &&
-      World.worldEqUniverse(world, universe) &&
-      nothingColoredBy &&
-      noGenes &&
-      scatterNotDpl &&
-      nothingInCellsets
-    );
-  };
+  lassoToolUpdate(tool) {
+    /*
+    this is called from componentDidUpdate(), so be very careful using
+    anything from this.state, which may be updated asynchronously.
+    */
+    const { currentSelection } = this.props;
+    if (currentSelection.mode === "within-polygon") {
+      /*
+      if there is a current selection, make sure the lasso tool matches
+      */
+      const polygon = currentSelection.polygon.map(p =>
+        this.mapPointToScreen(p)
+      );
+      tool.move(polygon);
+    } else {
+      tool.reset();
+    }
+  }
 
-  resetInterface = () => {
-    const { dispatch } = this.props;
-    dispatch({
-      type: "interface reset started"
-    });
-    dispatch(actions.resetInterface());
-  };
+  selectionToolUpdate(tool, container) {
+    /*
+    this is called from componentDidUpdate(), so be very careful using
+    anything from this.state, which may be updated asynchronously.
+    */
+    const { selectionTool } = this.props;
+    switch (selectionTool) {
+      case "brush":
+        this.brushToolUpdate(tool, container);
+        break;
+      case "lasso":
+        this.lassoToolUpdate(tool, container);
+        break;
+      default:
+        /* punt? */
+        break;
+    }
+  }
 
   reglDraw(regl, drawPoints, sizeBuffer, colorBuffer, pointBuffer, camera) {
     regl.clear({
@@ -304,14 +476,20 @@ class Graph extends React.Component {
     });
   }
 
-  invertPoint(pin) {
+  mapScreenToPoint(pin) {
+    /*
+    Map an XY coordinates from screen domain to cell/point range,
+    accounting for current pan/zoom camera.
+    */
     const { responsive } = this.props;
-    const { regl, camera, offset } = this.state;
+    const { regl, camera, transform } = this.state;
+    const { glScaleX, glScaleY } = transform;
 
     const gl = regl._gl;
 
     // get aspect ratio
     const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
+    const scale = aspect < 1 ? 1 / aspect : 1;
 
     // compute inverse view matrix
     const inverse = mat4.invert([], camera.view());
@@ -320,17 +498,48 @@ class Graph extends React.Component {
     const x = (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
     const y = 2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
     const pout = [
-      x * inverse[14] * aspect + inverse[12],
-      y * inverse[14] + inverse[13]
+      x * inverse[14] * aspect * scale + inverse[12],
+      -(y * inverse[14] * scale + inverse[13])
     ];
-    return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
+
+    const xy = [glScaleX.invert(pout[0]), glScaleY.invert(pout[1])];
+    return xy;
   }
 
-  handleBrushSelectAction() {
+  mapPointToScreen(xyCell) {
     /*
-      This conditional handles procedural brush deselect. Brush emits
-      an event on procedural deselect because it is move: null
+    Map an XY coordinate from cell/point domain to screen range.  Inverse
+    of mapScreenToPoint()
     */
+    const { responsive } = this.props;
+    const { regl, camera, transform } = this.state;
+    const { glScaleX, glScaleY } = transform;
+
+    const gl = regl._gl;
+
+    // get aspect ratio
+    const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
+    const scale = aspect < 1 ? 1 / aspect : 1;
+
+    // compute inverse view matrix
+    const inverse = mat4.invert([], camera.view());
+
+    // variable names are choosen to reflect inverse of those used
+    // in mapScreenToPoint().
+    const pout = [glScaleX(xyCell[0]), glScaleY(xyCell[1])];
+    const x = (pout[0] - inverse[12]) / aspect / scale / inverse[14];
+    const y = (-pout[1] - inverse[13]) / scale / inverse[14];
+
+    const pin = [
+      Math.round(((x + 1) * (responsive.width - this.graphPaddingRight)) / 2),
+      Math.round(
+        -((y + 1) / 2 - 1) * (responsive.height - this.graphPaddingTop)
+      )
+    ];
+    return pin;
+  }
+
+  handleBrushDragAction() {
     /*
       event describing brush position:
       @-------|
@@ -338,79 +547,105 @@ class Graph extends React.Component {
       |       |
       |-------@
     */
+    // ignore programatically generated events
+    if (d3.event.sourceEvent === null || !d3.event.selection) return;
+
+    const { dispatch } = this.props;
+    const s = d3.event.selection;
+    const brushCoords = {
+      northwest: this.mapScreenToPoint([s[0][0], s[0][1]]),
+      southeast: this.mapScreenToPoint([s[1][0], s[1][1]])
+    };
+
+    dispatch({
+      type: "graph brush change",
+      brushCoords
+    });
+  }
+
+  handleBrushStartAction() {
+    // Ignore programatically generated events.
+    if (!d3.event.sourceEvent) return;
+
+    const { dispatch } = this.props;
+    dispatch({ type: "graph brush start" });
+  }
+
+  handleBrushEndAction() {
+    // Ignore programatically generated events.
+    if (!d3.event.sourceEvent) return;
+
     /*
-      No idea why d3 event scope works like this
-      but apparently
-      it does
-      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
+    coordinates will be included if selection made, null
+    if selection cleared.
     */
     const { dispatch } = this.props;
-
-    if (d3.event.sourceEvent !== null) {
-      const s = d3.event.selection;
-
+    const s = d3.event.selection;
+    if (s) {
       const brushCoords = {
-        northwest: this.invertPoint([s[0][0], s[0][1]]),
-        southeast: this.invertPoint([s[1][0], s[1][1]])
+        northwest: this.mapScreenToPoint(s[0]),
+        southeast: this.mapScreenToPoint(s[1])
       };
-
       dispatch({
-        type: "graph brush selection change",
+        type: "graph brush end",
         brushCoords
       });
+    } else {
+      dispatch({
+        type: "graph brush deselect"
+      });
     }
   }
 
   handleBrushDeselectAction() {
     const { dispatch } = this.props;
-    const { svg, brush } = this.state;
-
-    if (d3.event && !d3.event.selection) {
-      dispatch({
-        type: "graph brush deselect"
-      });
-    }
-
-    if (!d3.event) {
-      /*
-      this line clears the brush procedurally, ie., zoom button clicked,
-      not a click away from brush on svg
-      */
-      svg.select(".graph_brush").call(brush.move, null);
-      dispatch({
-        type: "graph brush deselect"
-      });
-    }
+    dispatch({
+      type: "graph brush deselect"
+    });
   }
 
   handleLassoStart() {
     const { dispatch } = this.props;
-    // reset selected points when starting a new polygon
-    // making it easier for the user to make the next selection
     dispatch({
-      type: "lasso started"
+      type: "graph lasso start"
     });
   }
 
   // when a lasso is completed, filter to the points within the lasso polygon
   handleLassoEnd(polygon) {
-    const minimumPolygoneArea = 10;
+    const minimumPolygonArea = 10;
     const { dispatch } = this.props;
 
     if (
       polygon.length < 3 ||
-      Math.abs(d3.polygonArea(polygon)) < minimumPolygoneArea
+      Math.abs(d3.polygonArea(polygon)) < minimumPolygonArea
     ) {
       // if less than three points, or super small area, treat as a clear selection.
-      dispatch({ type: "lasso deselect" });
+      dispatch({ type: "graph lasso deselect" });
     } else {
       dispatch({
-        type: "lasso selection",
-        polygon: polygon.map(xy => this.invertPoint(xy)) // transform the polygon
+        type: "graph lasso end",
+        polygon: polygon.map(xy => this.mapScreenToPoint(xy)) // transform the polygon
       });
     }
   }
 
+  handleLassoCancel() {
+    const { dispatch } = this.props;
+    dispatch({ type: "graph lasso cancel" });
+  }
+
+  handleLassoDeselectAction() {
+    const { dispatch } = this.props;
+    dispatch({ type: "graph lasso deselect" });
+  }
+
+  handleDeselectAction() {
+    const { selectionTool } = this.props;
+    if (selectionTool === "brush") this.handleBrushDeselectAction();
+    if (selectionTool === "lasso") this.handleLassoDeselectAction();
+  }
+
   handleOpacityRangeChange(e) {
     const { dispatch } = this.props;
     dispatch({
@@ -420,199 +655,19 @@ class Graph extends React.Component {
   }
 
   render() {
-    const {
-      dispatch,
-      responsive,
-      crossfilter,
-      resettingInterface,
-      library_versions,
-      undoDisabled,
-      redoDisabled
-    } = this.props;
-    const { mode } = this.state;
+    const { responsive, graphInteractionMode } = this.props;
+
     return (
       <div id="graphWrapper">
-        <div
-          style={{
-            position: "fixed",
-            right: 0,
-            top: 0
-          }}
-        >
-          <div
-            style={{
-              padding: 10,
-              display: "flex",
-              justifyContent: "flex-end",
-              alignItems: "baseline"
-            }}
-          >
-            <Tooltip
-              content="Show only metadata and cells which are currently selected"
-              position="left"
-            >
-              <AnchorButton
-                type="button"
-                data-testid="subset-button"
-                disabled={
-                  crossfilter &&
-                  (crossfilter.countSelected() === 0 ||
-                    crossfilter.countSelected() === crossfilter.size())
-                }
-                style={{ marginRight: 10 }}
-                onClick={() => {
-                  dispatch(actions.regraph());
-                  dispatch({ type: "increment graph render counter" });
-                }}
-              >
-                subset to current selection
-              </AnchorButton>
-            </Tooltip>
-            <Tooltip
-              content="Reset cellxgene, clearing all selections"
-              position="left"
-            >
-              <AnchorButton
-                disabled={this.isResetDisabled()}
-                type="button"
-                loading={resettingInterface}
-                intent="warning"
-                style={{ marginRight: 10 }}
-                onClick={this.resetInterface}
-                data-testid="reset"
-                data-testclass={`resetting-${resettingInterface}`}
-              >
-                reset
-              </AnchorButton>
-            </Tooltip>
-            <div>
-              <div className="bp3-button-group">
-                <Tooltip content="Lasso selection" position="left">
-                  <Button
-                    type="button"
-                    data-testid="mode-lasso"
-                    className="bp3-button bp3-icon-polygon-filter"
-                    active={mode === "lasso"}
-                    onClick={() => {
-                      this.handleBrushDeselectAction();
-                      // this.restartReglLoop();
-                      this.setState({ mode: "lasso" });
-                    }}
-                    style={{
-                      cursor: "pointer"
-                    }}
-                  />
-                </Tooltip>
-                <Tooltip content="Pan and zoom" position="left">
-                  <Button
-                    type="button"
-                    data-testid="mode-pan-zoom"
-                    className="bp3-button bp3-icon-zoom-in"
-                    active={mode === "zoom"}
-                    onClick={() => {
-                      this.handleBrushDeselectAction();
-                      this.restartReglLoop();
-                      this.setState({ mode: "zoom" });
-                    }}
-                    style={{
-                      cursor: "pointer"
-                    }}
-                  />
-                </Tooltip>
-                <Tooltip content="Undo" position="left">
-                  <AnchorButton
-                    type="button"
-                    className="bp3-button bp3-icon-undo"
-                    disabled={undoDisabled}
-                    onClick={() => {
-                      dispatch({ type: "@@undoable/undo" });
-                    }}
-                    style={{
-                      cursor: "pointer"
-                    }}
-                  />
-                </Tooltip>
-                <Tooltip content="Redo" position="left">
-                  <AnchorButton
-                    type="button"
-                    className="bp3-button bp3-icon-redo"
-                    disabled={redoDisabled}
-                    onClick={() => {
-                      dispatch({ type: "@@undoable/redo" });
-                    }}
-                    style={{
-                      cursor: "pointer"
-                    }}
-                  />
-                </Tooltip>
-              </div>
-            </div>
-            <div style={{ marginLeft: 10 }}>
-              <Popover
-                content={
-                  <Menu>
-                    <MenuItem
-                      href="https://chanzuckerberg.github.io/cellxgene/faq.html"
-                      target="_blank"
-                      icon="help"
-                      text="FAQ"
-                    />
-                    <MenuItem
-                      href="https://join-cziscience-slack.herokuapp.com/"
-                      target="_blank"
-                      icon="chat"
-                      text="Chat"
-                    />
-                    <MenuItem
-                      href="https://chanzuckerberg.github.io/cellxgene/"
-                      target="_blank"
-                      icon="book"
-                      text="Docs"
-                    />
-                    <MenuItem
-                      href="https://github.com/chanzuckerberg/cellxgene"
-                      target="_blank"
-                      icon="git-branch"
-                      text="Github"
-                    />
-                    <MenuItem
-                      target="_blank"
-                      text={`cellxgene v${
-                        library_versions && library_versions.cellxgene
-                          ? library_versions.cellxgene
-                          : null
-                      }`}
-                    />
-                    <MenuItem text="MIT License" />
-                  </Menu>
-                }
-                position={Position.BOTTOM_RIGHT}
-              >
-                <Button
-                  type="button"
-                  className="bp3-button bp3-icon-cog"
-                  style={{
-                    cursor: "pointer"
-                  }}
-                />
-              </Popover>
-            </div>
-          </div>
-        </div>
         <div
           style={{
             zIndex: -9999,
             position: "fixed",
-            top: 0,
+            top: this.graphPaddingTop,
             right: 0
           }}
         >
-          <div
-            style={{
-              display: mode === "lasso" ? "inherit" : "none"
-            }}
-            id="graphAttachPoint"
-          />
+          <div id="graphAttachPoint" />
           <div style={{ padding: 0, margin: 0 }}>
             <canvas
               width={responsive.width - this.graphPaddingRight}

client/src/components/graph/setupCentroidSVG.js

@@ -0,0 +1,34 @@
+import * as d3 from "d3";
+import styles from "./graph.css";
+
+export default (responsive, graphPaddingRight, xy, text, colorBy) => {
+  const containerWidth = responsive.width - graphPaddingRight;
+
+  const svg = d3
+    .select("#graphAttachPoint")
+    .append("svg")
+    .attr("id", "centroid-container")
+    .attr("data-testid", "centroid-overlay")
+    .attr("width", containerWidth)
+    .attr("height", responsive.height)
+    .attr("class", `${styles.graphSVG}`)
+    .style("z-index", 998)
+    .style("pointer-events", "none");
+  //  TODO: Create own styles, ask Colin for an explanation on the css
+  // For now I'm going to put centroid z-index at 998 and lasso on 999
+
+  const label = svg
+    .append("g")
+    .attr("transform", `translate(${xy[0]}, ${xy[1]})`);
+
+  label
+    .append("text")
+    .attr("text-anchor", "middle")
+    .text(text)
+    .style("font-family", "Roboto Condensed")
+    .style("font-size", "18px")
+    .style("font-weight", "700")
+    .style("fill", colorBy ? "black" : "rgb(32, 178, 212)");
+
+  return svg;
+};

client/src/components/graph/setupLasso.js

@@ -3,7 +3,7 @@
 import * as d3 from "d3";
 
 const Lasso = () => {
-  const dispatch = d3.dispatch("start", "end");
+  const dispatch = d3.dispatch("start", "end", "cancel");
 
   const polygonToPath = polygon =>
     `M${polygon.map(d => d.join(",")).join("L")}`;
@@ -28,6 +28,7 @@ const Lasso = () => {
 
       lassoPath = g
         .append("path")
+        .attr("data-testid", "lasso-element")
         .attr("fill", "#0bb")
         .attr("fill-opacity", 0.1)
         .attr("stroke", "#0bb")
@@ -81,6 +82,7 @@ const Lasso = () => {
         lassoPath.remove();
         lassoPath = null;
         lassoPolygon = null;
+        dispatch.call("cancel");
       }
     };
 
@@ -114,6 +116,23 @@ const Lasso = () => {
         closePath = null;
       }
     };
+
+    lasso.move = polygon => {
+      if (polygon !== lassoPolygon || polygon.length !== lassoPolygon.length) {
+        lasso.reset();
+
+        lassoPolygon = polygon;
+        lassoPath = g
+          .append("path")
+          .attr("data-testid", "lasso-element")
+          .attr("fill", "#0bb")
+          .attr("fill-opacity", 0.1)
+          .attr("stroke", "#0bb")
+          .attr("stroke-dasharray", "3, 3");
+
+        lassoPath.attr("d", `${polygonToPath(lassoPolygon)}Z`);
+      }
+    };
   };
 
   lasso.on = (type, callback) => {

client/src/components/graph/setupSVGandBrush.js

@@ -10,42 +10,57 @@ import Lasso from "./setupLasso";
 ******************************************/
 
 export default (
-  handleBrushSelectAction,
-  handleBrushDeselectAction,
+  selectionToolType,
+  handleStartAction,
+  handleDragAction,
+  handleEndAction,
+  handleCancelAction,
   responsive,
   graphPaddingRight,
-  handleLassoStart,
-  handleLassoEnd
+  graphInteractionMode
 ) => {
   const svg = d3
     .select("#graphAttachPoint")
     .append("svg")
+    .attr("id", "tool")
     .attr("data-testid", "layout-overlay")
     .attr("width", responsive.width - graphPaddingRight)
     .attr("height", responsive.height)
-    .attr("class", `${styles.graphSVG}`);
+    .attr("class", `${styles.graphSVG}`)
+    .style("z-index", 999)
+    .style("display", graphInteractionMode === "select" ? "inherit" : "none");
 
-  const brush = d3
-    .brush()
-    .extent([[0, 0], [responsive.width - graphPaddingRight, responsive.height]])
-    .on("brush", handleBrushSelectAction)
-    .on("end", handleBrushDeselectAction);
+  if (selectionToolType === "brush") {
+    const brush = d3
+      .brush()
+      .extent([
+        [0, 0],
+        [responsive.width - graphPaddingRight, responsive.height]
+      ])
+      .on("start", handleStartAction)
+      .on("brush", handleDragAction)
+      // FYI, brush doesn't generate cancel
+      .on("end", handleEndAction);
 
-  const brushContainer = svg
-    .append("g")
-    .attr("class", "graph_brush")
-    .call(brush);
+    const brushContainer = svg
+      .append("g")
+      .attr("class", "graph_brush")
+      .call(brush);
 
-  const lassoInstance = Lasso()
-    .on("end", handleLassoEnd)
-    .on("start", handleLassoStart);
+    return { svg, container: brushContainer, tool: brush };
+  }
 
-  const lasso = svg.call(lassoInstance);
+  if (selectionToolType === "lasso") {
+    const lasso = Lasso()
+      .on("end", handleEndAction)
+      // FYI, Lasso doesn't generate drag
+      .on("start", handleStartAction)
+      .on("cancel", handleCancelAction);
 
-  return {
-    svg,
-    brushContainer,
-    brush,
-    lasso
-  };
+    const lassoContainer = svg.call(lasso);
+
+    return { svg, container: lassoContainer, tool: lasso };
+  }
+
+  throw new Error("unknown graph selection tool");
 };

client/src/components/graph/util.js

@@ -1,26 +0,0 @@
-// jshint esversion: 6
-// createExpressionsCountsMap () {
-//
-//   const CHANGE_ME_MAGIC_GENE_INDEX = 5;
-//
-//   const expressionsCountsMap = {};
-//
-//   /* currently selected gene */
-//   expressionsCountsMap.geneName = this.state.expressions.data.genes[3];
-//
-//   let maxExpressionValue = 0;
-//
-//   /* create map of expressions for every cell */
-//   this.state.expressions.data.cells.map((c) => {
-//     /* cellname = 234 */
-//     expressionsCountsMap[c.cellname] = c["e"][CHANGE_ME_MAGIC_GENE_INDEX];
-//     /* collect the maximum value as we iterate */
-//     if (c["e"][CHANGE_ME_MAGIC_GENE_INDEX] > maxExpressionValue) {
-//       maxExpressionValue = c["e"][CHANGE_ME_MAGIC_GENE_INDEX]
-//     }
-//   })
-//
-//   expressionsCountsMap.maxValue = maxExpressionValue;
-//
-//   return expressionsCountsMap;
-// }

client/src/components/leftSidebar/index.js

@@ -1,16 +1,15 @@
 // jshint esversion: 6
-import _ from "lodash";
 import React from "react";
 import { connect } from "react-redux";
-import Categorical from "./categorical/categorical";
-import Continuous from "./continuous/continuous";
-import GeneExpression from "./geneExpression";
-import * as globals from "../globals";
-import DynamicScatterplot from "./scatterplot/scatterplot";
+import Categorical from "../categorical/categorical";
+import Continuous from "../continuous/continuous";
+import GeneExpression from "../geneExpression";
+import * as globals from "../../globals";
+import DynamicScatterplot from "../scatterplot/scatterplot";
+import TopLeftLogoAndTitle from "./topLeftLogoAndTitle";
 
 @connect(state => ({
   responsive: state.responsive,
-  datasetTitle: _.get(state.config, "displayNames.dataset"),
   scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor
 }))
@@ -18,7 +17,6 @@ class LeftSideBar extends React.Component {
   render() {
     const {
       responsive,
-      datasetTitle,
       scatterplotXXaccessor,
       scatterplotYYaccessor
     } = this.props;
@@ -27,8 +25,7 @@ class LeftSideBar extends React.Component {
     this magic number should be made less fragile,
     if cellxgene logo or tabs change, this must as well
     */
-    const metadataSectionPadding = 0;
-    // scatterplotXXaccessor && scatterplotYYaccessor ? 450 : 0;
+    const logoRelatedPadding = 50;
 
     return (
       <div
@@ -36,27 +33,14 @@ class LeftSideBar extends React.Component {
           position: "fixed",
           backgroundColor: "white",
           /* x y blur spread color */
-          boxShadow: "1px 0px 6px 2px rgba(153,153,153,0.4)"
+          boxShadow: "-3px 0px 6px 2px rgba(153,153,153,0.4)"
         }}
       >
-        <p
-          data-testid="header"
-          style={{
-            position: "fixed",
-            top: globals.cellxgeneTitleTopPadding,
-            left: globals.leftSidebarWidth + globals.cellxgeneTitleLeftPadding,
-            margin: 0,
-            fontSize: globals.largestFontSize,
-            color: globals.darkerGrey,
-            width: "100%"
-          }}
-        >
-          cellxgene: {datasetTitle}
-        </p>
-
+        <TopLeftLogoAndTitle />
         <div
           style={{
-            height: responsive.height - metadataSectionPadding,
+            height: responsive.height - logoRelatedPadding,
+            marginTop: logoRelatedPadding,
             width: globals.leftSidebarWidth,
             overflowY: "auto",
             overflowX: "hidden"

client/src/components/leftSidebar/topLeftLogoAndTitle.js

@@ -0,0 +1,71 @@
+// jshint esversion: 6
+import React from "react";
+import { connect } from "react-redux";
+import * as globals from "../../globals";
+import Logo from "../framework/logo";
+
+@connect(state => ({
+  responsive: state.responsive,
+  datasetTitle: state.config?.displayNames?.dataset ?? "",
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor
+}))
+class LeftSideBar extends React.Component {
+  render() {
+    const { datasetTitle } = this.props;
+
+    const paddingToAvoidScrollBar = 15;
+
+    return (
+      <div
+        style={{
+          paddingLeft: 8,
+          paddingTop: 8,
+          width: globals.leftSidebarWidth - paddingToAvoidScrollBar,
+          position: "absolute",
+          backgroundColor: "white",
+          zIndex: 8888
+          /* x y blur spread color */
+          // boxShadow: "-5px -1px 4px 2px rgba(225,225,225,0.4)"
+        }}
+      >
+        <Logo size={30} />
+        <span
+          style={{
+            fontSize: 28,
+            position: "relative",
+            top: -6,
+            fontWeight: "bold",
+            marginLeft: 5,
+            color: globals.logoColor,
+            userSelect: "none"
+          }}
+        >
+          cell<span
+            style={{
+              position: "relative",
+              top: 1,
+              fontWeight: 300,
+              fontSize: 24
+            }}
+          >
+            ×
+          </span>gene
+        </span>
+        <span
+          data-testid="header"
+          style={{
+            fontSize: 14,
+            position: "relative",
+            marginLeft: 7,
+            top: -8
+          }}
+        >
+          {datasetTitle}
+        </span>
+      </div>
+    );
+  }
+}
+
+export default LeftSideBar;

client/src/components/menubar/cellSetButtons.js

@@ -32,7 +32,7 @@ class CellSetButton extends React.Component {
   render() {
     const { differential, eitherCellSetOneOrTwo } = this.props;
     const cellListName = `celllist${eitherCellSetOneOrTwo}`;
-    let cells_selected = differential[cellListName]
+    const cellsSelected = differential[cellListName]
       ? differential[cellListName].length
       : 0;
     return (
@@ -41,7 +41,6 @@ class CellSetButton extends React.Component {
         position="top"
       >
         <AnchorButton
-          style={{ marginRight: 10 }}
           type="button"
           disabled={differential.diffExp}
           onClick={this.set.bind(this)}
@@ -50,7 +49,7 @@ class CellSetButton extends React.Component {
           {eitherCellSetOneOrTwo}
           {": "}
           <span data-testid={`cellset-count-${eitherCellSetOneOrTwo}`}>
-            {cells_selected}
+            {cellsSelected}
           </span>
           {" cells"}
         </AnchorButton>

client/src/components/menubar/clip.js

@@ -0,0 +1,130 @@
+// jshint esversion: 6
+import React from "react";
+import {
+  Position,
+  Button,
+  Popover,
+  NumericInput,
+  Icon
+} from "@blueprintjs/core";
+
+function Clip(props) {
+  const {
+    pendingClipPercentiles,
+    clipPercentileMin,
+    clipPercentileMax,
+    handleClipOpening,
+    handleClipClosing,
+    handleClipCommit,
+    isClipDisabled,
+    handleClipOnKeyPress,
+    handleClipPercentileMaxValueChange,
+    handleClipPercentileMinValueChange
+  } = props;
+
+  const clipMin =
+    pendingClipPercentiles?.clipPercentileMin ?? clipPercentileMin;
+  const clipMax =
+    pendingClipPercentiles?.clipPercentileMax ?? clipPercentileMax;
+  const activeClipClass =
+    clipPercentileMin > 0 || clipPercentileMax < 100
+      ? " bp3-intent-warning"
+      : "";
+
+  return (
+    <div
+      className="bp3-button-group"
+      style={{
+        marginLeft: 10
+      }}
+    >
+      <Popover
+        target={
+          <Button
+            type="button"
+            data-testid="visualization-settings"
+            className={`bp3-button bp3-icon-timeline-bar-chart ${activeClipClass}`}
+            style={{
+              cursor: "pointer"
+            }}
+          />
+        }
+        position={Position.BOTTOM_RIGHT}
+        onOpening={handleClipOpening}
+        onClosing={handleClipClosing}
+        content={
+          <div
+            style={{
+              display: "flex",
+              justifyContent: "flex-start",
+              alignItems: "flex-start",
+              flexDirection: "column",
+              padding: 10
+            }}
+          >
+            <div>Clip all continuous values to percentile range</div>
+            <div
+              style={{
+                display: "flex",
+                justifyContent: "space-between",
+                alignItems: "center",
+                paddingTop: 5,
+                paddingBottom: 5
+              }}
+            >
+              <NumericInput
+                style={{ width: 50 }}
+                data-testid="clip-min-input"
+                onValueChange={handleClipPercentileMinValueChange}
+                onKeyPress={handleClipOnKeyPress}
+                value={clipMin}
+                min={0}
+                max={100}
+                fill={false}
+                minorStepSize={null}
+                rightElement={
+                  <div style={{ padding: "4px 2px" }}>
+                    <Icon icon="percentage" intent="primary" iconSize={14} />
+                  </div>
+                }
+              />
+              <span style={{ marginRight: 5, marginLeft: 5 }}> - </span>
+              <NumericInput
+                style={{ width: 50 }}
+                data-testid="clip-max-input"
+                onValueChange={handleClipPercentileMaxValueChange}
+                onKeyPress={handleClipOnKeyPress}
+                value={clipMax}
+                min={0}
+                max={100}
+                fill={false}
+                minorStepSize={null}
+                rightElement={
+                  <div style={{ padding: "4px 2px" }}>
+                    <Icon icon="percentage" intent="primary" iconSize={14} />
+                  </div>
+                }
+              />
+              <Button
+                type="button"
+                data-testid="clip-commit"
+                className="bp3-button"
+                disabled={isClipDisabled()}
+                style={{
+                  cursor: "pointer",
+                  marginRight: 5,
+                  marginLeft: 5
+                }}
+                onClick={handleClipCommit}
+              >
+                Clip
+              </Button>
+            </div>
+          </div>
+        }
+      />
+    </div>
+  );
+}
+
+export default Clip;

client/src/components/menubar/index.js

@@ -0,0 +1,464 @@
+// jshint esversion: 6
+import React from "react";
+import { connect } from "react-redux";
+import {
+  Button,
+  AnchorButton,
+  Tooltip,
+  Popover,
+  Position,
+  RadioGroup,
+  Radio
+} from "@blueprintjs/core";
+import { World } from "../../util/stateManager";
+import actions from "../../actions";
+import CellSetButton from "./cellSetButtons";
+import InformationMenu from "./infoMenu";
+import UndoRedoReset from "./undoRedoReset";
+import Clip from "./clip";
+
+@connect(state => ({
+  universe: state.universe,
+  world: state.world,
+  loading: state.controls.loading,
+  crossfilter: state.crossfilter,
+  differential: state.differential,
+  resettingInterface: state.controls.resettingInterface,
+  layoutChoice: state.layoutChoice,
+  graphInteractionMode: state.controls.graphInteractionMode,
+  clipPercentileMin: Math.round(100 * (state.world?.clipQuantiles?.min ?? 0)),
+  clipPercentileMax: Math.round(100 * (state.world?.clipQuantiles?.max ?? 1)),
+  userDefinedGenes: state.controls.userDefinedGenes,
+  diffexpGenes: state.controls.diffexpGenes,
+  colorAccessor: state.colors.colorAccessor,
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+  celllist1: state.differential.celllist1,
+  celllist2: state.differential.celllist2,
+  libraryVersions: state.config?.library_versions, // eslint-disable-line camelcase
+  undoDisabled: state["@@undoable/past"].length === 0,
+  redoDisabled: state["@@undoable/future"].length === 0
+}))
+class MenuBar extends React.Component {
+  static isValidDigitKeyEvent(e) {
+    /*
+    Return true if this event is necessary to enter a percent number input.
+    Return false if not.
+
+    Returns true for events with keys: backspace, control, alt, meta, [0-9],
+    or events that don't have a key.
+    */
+    if (e.key === null) return true;
+    if (e.ctrlKey || e.altKey || e.metaKey) return true;
+
+    // concept borrowed from blueprint's numericInputUtils:
+    // keys that print a single character when pressed have a `key` name of
+    // length 1. every other key has a longer `key` name (e.g. "Backspace",
+    // "ArrowUp", "Shift"). since none of those keys can print a character
+    // to the field--and since they may have important native behaviors
+    // beyond printing a character--we don't want to disable their effects.
+    const isSingleCharKey = e.key.length === 1;
+    if (!isSingleCharKey) return true;
+
+    const key = e.key.charCodeAt(0) - 48; /* "0" */
+    return key >= 0 && key <= 9;
+  }
+
+  constructor(props) {
+    super(props);
+    this.state = {
+      pendingClipPercentiles: null
+    };
+  }
+
+  isClipDisabled = () => {
+    /*
+    return true if clip button should be disabled.
+    */
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMin = pendingClipPercentiles?.clipPercentileMin;
+    const clipPercentileMax = pendingClipPercentiles?.clipPercentileMax;
+
+    const { world } = this.props;
+    const currentClipMin = 100 * world?.clipQuantiles?.min;
+    const currentClipMax = 100 * world?.clipQuantiles?.max;
+
+    // if you change this test, be careful with logic around
+    // comparisons between undefined / NaN handling.
+    const isDisabled =
+      !(clipPercentileMin < clipPercentileMax) ||
+      (clipPercentileMin === currentClipMin &&
+        clipPercentileMax === currentClipMax);
+
+    return isDisabled;
+  };
+
+  isResetDisabled = () => {
+    /*
+    Reset should be disabled when all of the following are true:
+      * nothing is selected in the crossfilter
+      * world EQ universe
+      * nothing is colored by
+      * there are no userDefinedGenes or diffexpGenes displayed
+      * scatterplot is not displayed
+      * nothing in cellset1 or cellset2
+      * clip percentiles are [0,100]
+    */
+    const {
+      crossfilter,
+      world,
+      universe,
+      userDefinedGenes,
+      diffexpGenes,
+      colorAccessor,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor,
+      celllist1,
+      celllist2,
+      clipPercentileMin,
+      clipPercentileMax
+    } = this.props;
+
+    if (!crossfilter || !world || !universe) {
+      return false;
+    }
+    const nothingSelected = crossfilter.countSelected() === crossfilter.size();
+    const nothingColoredBy = !colorAccessor;
+    const noGenes = userDefinedGenes.length === 0 && diffexpGenes.length === 0;
+    const scatterNotDpl = !scatterplotXXaccessor || !scatterplotYYaccessor;
+    const nothingInCellsets = !celllist1 && !celllist2;
+
+    return (
+      nothingSelected &&
+      World.worldEqUniverse(world, universe) &&
+      nothingColoredBy &&
+      noGenes &&
+      scatterNotDpl &&
+      nothingInCellsets &&
+      clipPercentileMax === 100 &&
+      clipPercentileMin === 0
+    );
+  };
+
+  resetInterface = () => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "interface reset started"
+    });
+    dispatch(actions.resetInterface());
+  };
+
+  handleClipOnKeyPress = e => {
+    /*
+    allow only numbers, plus other critical keys which
+    may be required to make a number
+    */
+    if (!MenuBar.isValidDigitKeyEvent(e)) {
+      e.preventDefault();
+    }
+  };
+
+  handleClipPercentileMinValueChange = v => {
+    /*
+    Ignore anything that isn't a legit number
+    */
+    if (!Number.isFinite(v)) return;
+
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMax = pendingClipPercentiles?.clipPercentileMax;
+
+    /*
+    clamp to [0, currentClipPercentileMax]
+    */
+    if (v <= 0) v = 0;
+    if (v > 100) v = 100;
+    const clipPercentileMin = Math.round(v); // paranoia
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipPercentileMaxValueChange = v => {
+    /*
+    Ignore anything that isn't a legit number
+    */
+    if (!Number.isFinite(v)) return;
+
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMin = pendingClipPercentiles?.clipPercentileMin;
+
+    /*
+    clamp to [0, 100]
+    */
+    if (v < 0) v = 0;
+    if (v > 100) v = 100;
+    const clipPercentileMax = Math.round(v); // paranoia
+
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipCommit = () => {
+    const { dispatch } = this.props;
+    const { pendingClipPercentiles } = this.state;
+    const { clipPercentileMin, clipPercentileMax } = pendingClipPercentiles;
+    const min = clipPercentileMin / 100;
+    const max = clipPercentileMax / 100;
+    dispatch({
+      type: "set clip quantiles",
+      clipQuantiles: { min, max }
+    });
+  };
+
+  handleClipOpening = () => {
+    const { clipPercentileMin, clipPercentileMax } = this.props;
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipClosing = () => {
+    this.setState({ pendingClipPercentiles: null });
+  };
+
+  handleLayoutChoiceChange = e => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "set layout choice",
+      layoutChoice: e.currentTarget.value
+    });
+  };
+
+  computeDiffExp = () => {
+    const { dispatch, differential } = this.props;
+    if (differential.celllist1 && differential.celllist2) {
+      dispatch(
+        actions.requestDifferentialExpression(
+          differential.celllist1,
+          differential.celllist2
+        )
+      );
+    }
+  };
+
+  clearDifferentialExpression = () => {
+    const { dispatch, differential } = this.props;
+    dispatch({
+      type: "clear differential expression",
+      diffExp: differential.diffExp
+    });
+    dispatch({
+      type: "clear scatterplot"
+    });
+  };
+
+  render() {
+    const {
+      dispatch,
+      differential,
+      crossfilter,
+      resettingInterface,
+      libraryVersions,
+      undoDisabled,
+      redoDisabled,
+      selectionTool,
+      clipPercentileMin,
+      clipPercentileMax,
+      layoutChoice,
+      graphInteractionMode
+    } = this.props;
+    const { pendingClipPercentiles } = this.state;
+
+    const haveBothCellSets =
+      !!differential.celllist1 && !!differential.celllist2;
+
+    // constants used to create selection tool button
+    let selectionTooltip;
+    let selectionButtonClass;
+    if (selectionTool === "brush") {
+      selectionTooltip = "Brush selection";
+      selectionButtonClass = "bp3-icon-select";
+    } else {
+      selectionTooltip = "Lasso selection";
+      selectionButtonClass = "bp3-icon-polygon-filter";
+    }
+
+    return (
+      <div
+        style={{
+          position: "fixed",
+          right: 8,
+          top: 8
+        }}
+      >
+        <div className="bp3-button-group" style={{ marginRight: 10 }}>
+          <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
+          <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
+          {!differential.diffExp ? (
+            <Tooltip
+              content="Add two cells selections, see the top 15 differentially expressed genes between them"
+              position="bottom"
+            >
+              <AnchorButton
+                disabled={!haveBothCellSets}
+                intent="primary"
+                data-testid="diffexp-button"
+                loading={differential.loading}
+                icon="left-join"
+                fill
+                type="button"
+                onClick={this.computeDiffExp}
+              >
+                Compute Differential Expression
+              </AnchorButton>
+            </Tooltip>
+          ) : null}
+
+          {differential.diffExp ? (
+            <Tooltip
+              content="Remove differentially expressed gene list and clear cell selections"
+              position="bottom"
+            >
+              <Button
+                type="button"
+                fill
+                intent="warning"
+                onClick={this.clearDifferentialExpression}
+              >
+                Clear Differential Expression
+              </Button>
+            </Tooltip>
+          ) : null}
+        </div>
+        <Tooltip
+          content="Show only metadata and cells which are currently selected"
+          position="left"
+        >
+          <AnchorButton
+            type="button"
+            data-testid="subset-button"
+            disabled={
+              crossfilter &&
+              (crossfilter.countSelected() === 0 ||
+                crossfilter.countSelected() === crossfilter.size())
+            }
+            style={{
+              marginRight: 10
+            }}
+            onClick={() => {
+              dispatch(actions.regraph());
+              dispatch({ type: "increment graph render counter" });
+            }}
+          >
+            subset to current selection
+          </AnchorButton>
+        </Tooltip>
+        <div className="bp3-button-group">
+          <Tooltip content={selectionTooltip} position="left">
+            <Button
+              type="button"
+              data-testid="mode-lasso"
+              className={`bp3-button ${selectionButtonClass}`}
+              active={graphInteractionMode === "select"}
+              onClick={() => {
+                dispatch({
+                  type: "change graph interaction mode",
+                  data: "select"
+                });
+              }}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
+          <Tooltip content="Pan and zoom" position="left">
+            <Button
+              type="button"
+              data-testid="mode-pan-zoom"
+              className="bp3-button bp3-icon-zoom-in"
+              active={graphInteractionMode === "zoom"}
+              onClick={() => {
+                dispatch({
+                  type: "change graph interaction mode",
+                  data: "zoom"
+                });
+              }}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
+        </div>
+        <div
+          className="bp3-button-group"
+          style={{
+            marginLeft: 10
+          }}
+        >
+          <Popover
+            target={
+              <Button
+                type="button"
+                data-testid="layout-choice"
+                className="bp3-button bp3-icon-heatmap"
+                style={{
+                  cursor: "pointer"
+                }}
+              />
+            }
+            position={Position.BOTTOM_RIGHT}
+            content={
+              <div
+                style={{
+                  display: "flex",
+                  justifyContent: "flex-start",
+                  alignItems: "flex-start",
+                  flexDirection: "column",
+                  padding: 10
+                }}
+              >
+                <RadioGroup
+                  label="Layout Choice"
+                  onChange={this.handleLayoutChoiceChange}
+                  selectedValue={layoutChoice.current}
+                >
+                  {layoutChoice.available.map(name => (
+                    <Radio label={name} value={name} key={name} />
+                  ))}
+                </RadioGroup>
+              </div>
+            }
+          />
+        </div>
+        <Clip
+          pendingClipPercentiles={pendingClipPercentiles}
+          clipPercentileMin={clipPercentileMin}
+          clipPercentileMax={clipPercentileMax}
+          handleClipOpening={this.handleClipOpening}
+          handleClipClosing={this.handleClipClosing}
+          handleClipCommit={this.handleClipCommit}
+          isClipDisabled={this.isClipDisabled}
+          handleClipOnKeyPress={this.handleClipOnKeyPress}
+          handleClipPercentileMaxValueChange={
+            this.handleClipPercentileMaxValueChange
+          }
+          handleClipPercentileMinValueChange={
+            this.handleClipPercentileMinValueChange
+          }
+        />
+        <UndoRedoReset
+          dispatch={dispatch}
+          isResetDisabled={this.isResetDisabled}
+          resetInterface={this.resetInterface}
+          resettingInterface={resettingInterface}
+          undoDisabled={undoDisabled}
+          redoDisabled={redoDisabled}
+        />
+        <InformationMenu libraryVersions={libraryVersions} />
+      </div>
+    );
+  }
+}
+
+export default MenuBar;

client/src/components/menubar/infoMenu.js

@@ -0,0 +1,61 @@
+// jshint esversion: 6
+import React from "react";
+import { Button, Popover, Menu, MenuItem, Position } from "@blueprintjs/core";
+
+function InformationMenu(props) {
+  const { libraryVersions } = props;
+  return (
+    <div style={{ marginLeft: 10 }} className="bp3-button-group">
+      <Popover
+        content={
+          <Menu>
+            <MenuItem
+              href="https://chanzuckerberg.github.io/cellxgene/faq.html"
+              target="_blank"
+              icon="help"
+              text="FAQ"
+            />
+            <MenuItem
+              href="https://join-cellxgene-users.herokuapp.com/"
+              target="_blank"
+              icon="chat"
+              text="Chat"
+            />
+            <MenuItem
+              href="https://chanzuckerberg.github.io/cellxgene/"
+              target="_blank"
+              icon="book"
+              text="Docs"
+            />
+            <MenuItem
+              href="https://github.com/chanzuckerberg/cellxgene"
+              target="_blank"
+              icon="git-branch"
+              text="Github"
+            />
+            <MenuItem
+              target="_blank"
+              text={`cellxgene v${
+                libraryVersions && libraryVersions.cellxgene
+                  ? libraryVersions.cellxgene
+                  : null
+              }`}
+            />
+            <MenuItem text="MIT License" />
+          </Menu>
+        }
+        position={Position.BOTTOM_RIGHT}
+      >
+        <Button
+          type="button"
+          className="bp3-button bp3-icon-info-sign"
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Popover>
+    </div>
+  );
+}
+
+export default InformationMenu;

client/src/components/menubar/undoRedoReset.js

@@ -0,0 +1,64 @@
+// jshint esversion: 6
+import React from "react";
+import { AnchorButton, Tooltip } from "@blueprintjs/core";
+
+function InformationMenu(props) {
+  const {
+    resettingInterface,
+    undoDisabled,
+    redoDisabled,
+    resetInterface,
+    isResetDisabled,
+    dispatch
+  } = props;
+  return (
+    <div style={{ marginLeft: 10 }} className="bp3-button-group">
+      <Tooltip content="Undo" position="left">
+        <AnchorButton
+          type="button"
+          className="bp3-button bp3-icon-undo"
+          disabled={undoDisabled}
+          onClick={() => {
+            dispatch({ type: "@@undoable/undo" });
+          }}
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Tooltip>
+      <Tooltip content="Redo" position="left">
+        <AnchorButton
+          type="button"
+          className="bp3-button bp3-icon-redo"
+          disabled={redoDisabled}
+          onClick={() => {
+            dispatch({ type: "@@undoable/redo" });
+          }}
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Tooltip>
+      <Tooltip
+        content="Reset cellxgene, clearing all selections"
+        position="left"
+      >
+        <AnchorButton
+          disabled={isResetDisabled()}
+          style={{ marginLeft: 10 }}
+          type="button"
+          loading={resettingInterface}
+          intent="none"
+          icon="refresh"
+          onClick={resetInterface}
+          data-testid="reset"
+          data-testclass={`resetting-${resettingInterface}`}
+        >
+          reset
+        </AnchorButton>
+      </Tooltip>
+    </div>
+  );
+}
+
+export default InformationMenu;

client/src/components/scatterplot/scatterplot.js

@@ -3,7 +3,6 @@
 // https://peterbeshai.com/scatterplot-in-d3-with-voronoi-interaction.html
 
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import { Button, ButtonGroup } from "@blueprintjs/core";
 import _regl from "regl";
@@ -145,7 +144,8 @@ class Scatterplot extends React.Component {
 
     if (
       scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
-      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor // was CLU now FTH1 etc
+      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor || // was CLU now FTH1 etc
+      world !== prevProps.world // shape or clip of world changed
     ) {
       const scales = Scatterplot.setupScales(expressionX, expressionY);
       this.drawAxesSVG(scales.xScale, scales.yScale, svg);
@@ -263,9 +263,15 @@ class Scatterplot extends React.Component {
 
     // the axes are much cleaner and easier now. No need to rotate and orient
     // the axis, just call axisBottom, axisLeft etc.
-    const xAxis = d3.axisBottom().scale(xScale);
+    const xAxis = d3
+      .axisBottom()
+      .ticks(7)
+      .scale(xScale);
 
-    const yAxis = d3.axisLeft().scale(yScale);
+    const yAxis = d3
+      .axisLeft()
+      .ticks(7)
+      .scale(yScale);
 
     // adding axes is also simpler now, just translate x-axis to (0,height)
     // and it's alread defined to be a bottom axis.

client/src/globals.js

@@ -31,6 +31,7 @@ export const darkGreen = "#448C4D";
 
 export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";
+export const logoColor = "black"; /* logo pink: "#E9429A" */
 
 /* typography constants */
 
@@ -72,114 +73,3 @@ let _API = {
 
 if (window.CELLXGENE && window.CELLXGENE.API) _API = window.CELLXGENE.API;
 export const API = _API;
-
-export const ordinalColors = [
-  "#0ac115",
-  "#c10ab6",
-  "#c1710a",
-  "#0a5ac1",
-  "#c1150a",
-  "#0ab6c1",
-  "#5ac10a",
-  "#710ac1",
-  "#0ac171",
-  "#c10a5a",
-  "#b6c10a",
-  "#150ac1",
-  "#b2ffb7",
-  "#ffb2fa",
-  "#ffddb2",
-  "#b2d4ff",
-  "#ffb7b2",
-  "#b2faff",
-  "#d4ffb2",
-  "#ddb2ff",
-  "#b2ffdd",
-  "#ffb2d4",
-  "#faffb2",
-  "#b7b2ff",
-  "#27a908",
-  "#8b08a9",
-  "#a93a08",
-  "#0877a9",
-  "#a90827",
-  "#08a98b",
-  "#77a908",
-  "#3a08a9",
-  "#08a93a",
-  "#a90877",
-  "#a98b08",
-  "#0827a9",
-  "#00ff0f",
-  "#ff00ef",
-  "#ff8e00",
-  "#0070ff",
-  "#ff0f00",
-  "#00efff",
-  "#70ff00",
-  "#8e00ff",
-  "#00ff8e",
-  "#ff0070",
-  "#efff00",
-  "#0f00ff",
-  "#006606",
-  "#66005f",
-  "#663900",
-  "#002c66",
-  "#660600",
-  "#005f66",
-  "#2c6600",
-  "#390066",
-  "#006639",
-  "#66002c",
-  "#5f6600",
-  "#060066",
-  "#83ff65",
-  "#e165ff",
-  "#ff9565",
-  "#65cfff",
-  "#ff6583",
-  "#65ffe1",
-  "#cfff65",
-  "#9565ff",
-  "#65ff95",
-  "#ff65cf",
-  "#ffe165",
-  "#6583ff",
-  "#009909",
-  "#99008f",
-  "#995500",
-  "#004399",
-  "#990900",
-  "#008f99",
-  "#439900",
-  "#550099",
-  "#009955",
-  "#990043",
-  "#8f9900",
-  "#090099",
-  "#d9fecc",
-  "#f1ccfe",
-  "#fed7cc",
-  "#ccf3fe",
-  "#feccd9",
-  "#ccfef1",
-  "#f3fecc",
-  "#d7ccfe",
-  "#ccfed7",
-  "#feccf3",
-  "#fef1cc",
-  "#ccd9fe",
-  "#47ea51",
-  "#ea47e0",
-  "#eaa247",
-  "#478fea",
-  "#ea5147",
-  "#47e0ea",
-  "#8fea47",
-  "#a247ea",
-  "#47eaa2",
-  "#ea478f",
-  "#e0ea47",
-  "#5147ea"
-];

client/src/reducers/categoricalSelection.js

@@ -1,12 +1,9 @@
-import _ from "lodash";
-
 import { ControlsHelpers } from "../util/stateManager";
 import * as globals from "../globals";
 
 function maxCategoryItems(state) {
-  return _.get(
-    state.config,
-    "parameters.max-category-items",
+  return (
+    state.config.parameters?.["max-category-items"] ??
     globals.configDefaults.parameters["max-category-items"]
   );
 }
@@ -20,7 +17,8 @@ const CategoricalSelection = (
   switch (action.type) {
     case "initial data load complete (universe exists)":
     case "set World to current selection":
-    case "reset World to eq Universe": {
+    case "reset World to eq Universe":
+    case "set clip quantiles": {
       const { world } = nextSharedState;
       return ControlsHelpers.createCategoricalSelection(
         maxCategoryItems(prevSharedState),
@@ -32,15 +30,15 @@ const CategoricalSelection = (
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
-        state[action.metadataField].categorySelected
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = true;
+      newCategoryValueSelected[action.categoryIndex] = true;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -50,15 +48,15 @@ const CategoricalSelection = (
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
-        state[action.metadataField].categorySelected
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = false;
+      newCategoryValueSelected[action.categoryIndex] = false;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -72,8 +70,9 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
-            state[action.metadataField].categorySelected
+          categorySelected: false,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(false)
         }
       };
@@ -88,8 +87,9 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
-            state[action.metadataField].categorySelected
+          categorySelected: true,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(true)
         }
       };

client/src/reducers/centroidLabel.js

@@ -0,0 +1,44 @@
+import calcCentroid from "../util/centroid";
+
+const initialState = {
+  metadataField: "",
+  categoryIndex: -1,
+  categoryField: "",
+  centroidXY: [-1, -1]
+};
+
+const CentroidLabel = (state = initialState, action, sharedNextState) => {
+  const { categoricalSelection, world, layoutChoice } = sharedNextState;
+  const { metadataField, categoryIndex } = action;
+  const categoryField =
+    categoricalSelection?.[metadataField]?.categoryValues[categoryIndex];
+  switch (action.type) {
+    case "category value mouse hover start":
+      return {
+        ...state,
+        metadataField,
+        categoryIndex,
+        categoryField,
+        centroidXY: null /* calcCentroid(  This function call is computationally heavy and also leading to large GC. Before reimplementation, look into optimization and memoization
+          world,
+          metadataField,
+          categoryField,
+          layoutChoice.currentDimNames
+        ) */
+      };
+
+    case "category value mouse hover end":
+      if (
+        metadataField === state.metadataField &&
+        categoryIndex === state.categoryIndex
+      ) {
+        return initialState;
+      }
+      return state;
+
+    default:
+      return state;
+  }
+};
+
+export default CentroidLabel;

client/src/reducers/colors.js

@@ -1,4 +1,4 @@
-import { createColors } from "../util/stateManager";
+import { ColorHelpers } from "../util/stateManager";
 
 const ColorsReducer = (
   state = {
@@ -17,7 +17,7 @@ const ColorsReducer = (
       const { world } = nextSharedState;
       const colorMode = null;
       const colorAccessor = null;
-      const { rgb, scale } = createColors(world, colorMode);
+      const { rgb, scale } = ColorHelpers.createColors(world, colorMode);
       return {
         ...state,
         colorAccessor,
@@ -27,10 +27,25 @@ const ColorsReducer = (
       };
     }
 
+    case "set clip quantiles":
     case "set World to current selection": {
+      const { world: prevWorld, controls: prevControls } = prevSharedState;
+      const resetColorState = ColorHelpers.checkIfColorByDiffexpAndResetColors(
+        prevControls,
+        state,
+        prevWorld
+      );
+      if (resetColorState) {
+        return resetColorState;
+      }
+
       const { colorMode, colorAccessor } = state;
       const { world } = nextSharedState;
-      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
         rgb,
@@ -39,29 +54,32 @@ const ColorsReducer = (
     }
 
     case "reset colorscale": {
-      const { world } = prevSharedState;
-      const { rgb, scale } = createColors(world);
       return {
         ...state,
-        colorMode: null,
-        colorAccessor: null,
-        rgb,
-        scale
+        ...ColorHelpers.resetColors(prevSharedState.world)
       };
     }
 
     case "color by categorical metadata":
     case "color by continuous metadata": {
       const { world } = prevSharedState;
-      const { rgb, scale } = createColors(
+
+      /* toggle between this mode and reset */
+      const resetCurrent =
+        action.type === state.colorMode &&
+        action.colorAccessor === state.colorAccessor;
+      const colorMode = !resetCurrent ? action.type : null;
+      const colorAccessor = !resetCurrent ? action.colorAccessor : null;
+
+      const { rgb, scale } = ColorHelpers.createColors(
         world,
-        action.type,
-        action.colorAccessor
+        colorMode,
+        colorAccessor
       );
       return {
         ...state,
-        colorMode: action.type,
-        colorAccessor: action.colorAccessor,
+        colorMode,
+        colorAccessor,
         rgb,
         scale
       };
@@ -69,16 +87,40 @@ const ColorsReducer = (
 
     case "color by expression": {
       const { world } = prevSharedState;
-      const { rgb, scale } = createColors(world, action.type, action.gene);
+
+      /* toggle between this mode and reset */
+      const resetCurrent =
+        action.type === state.colorMode && action.gene === state.colorAccessor;
+      const colorMode = !resetCurrent ? action.type : null;
+      const colorAccessor = !resetCurrent ? action.gene : null;
+
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
-        colorMode: action.type,
-        colorAccessor: action.gene,
+        colorMode,
+        colorAccessor,
         rgb,
         scale
       };
     }
 
+    case "clear differential expression": {
+      const { world: prevWorld, controls: prevControls } = prevSharedState;
+      const resetColorState = ColorHelpers.checkIfColorByDiffexpAndResetColors(
+        prevControls,
+        state,
+        prevWorld
+      );
+      if (resetColorState) {
+        return resetColorState;
+      }
+      return state;
+    }
+
     default: {
       return state;
     }

client/src/reducers/continuousSelection.js

@@ -2,7 +2,8 @@ import { makeContinuousDimensionName } from "../util/nameCreators";
 
 const ContinuousSelection = (state = {}, action) => {
   switch (action.type) {
-    case "reset World to eq Universe": {
+    case "reset World to eq Universe":
+    case "set clip quantiles": {
       return {};
     }
     case "continuous metadata histogram start":
@@ -17,6 +18,14 @@ const ContinuousSelection = (state = {}, action) => {
         [name]: action.range
       };
     }
+    case "continuous metadata histogram cancel": {
+      const name = makeContinuousDimensionName(
+        action.continuousNamespace,
+        action.selection
+      );
+      const { [name]: deletedField, ...newState } = state;
+      return newState;
+    }
     default: {
       return state;
     }

client/src/reducers/controls.js

@@ -16,7 +16,7 @@ const Controls = (
     diffexpGenes: [],
 
     resettingInterface: false,
-
+    graphInteractionMode: "select",
     opacityForDeselectedCells: 0.2,
     scatterplotXXaccessor: null, // just easier to read
     scatterplotYYaccessor: null,
@@ -93,9 +93,10 @@ const Controls = (
     }
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const _diffexpGenes = [];
       action.data.forEach(d => {
-        _diffexpGenes.push(world.varAnnotations.at(d[0], "name"));
+        _diffexpGenes.push(world.varAnnotations.at(d[0], varIndexName));
       });
       return {
         ...state,
@@ -137,6 +138,11 @@ const Controls = (
     /*******************************
              User Events
      *******************************/
+    case "change graph interaction mode":
+      return {
+        ...state,
+        graphInteractionMode: action.data
+      };
     case "change opacity deselected cells in 2d graph background":
       return {
         ...state,

client/src/reducers/crossfilter.js

@@ -10,6 +10,8 @@ import {
   makeContinuousDimensionName
 } from "../util/nameCreators";
 
+const XYDimName = layoutDimensionName("XY");
+
 const CrossfilterReducer = (
   state = null,
   action,
@@ -18,10 +20,11 @@ const CrossfilterReducer = (
 ) => {
   switch (action.type) {
     case "initial data load complete (universe exists)": {
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       const crossfilter = World.createObsDimensions(
         new Crossfilter(world.obsAnnotations),
-        world
+        world,
+        layoutChoice.currentDimNames
       );
       return crossfilter;
     }
@@ -38,11 +41,16 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "set clip quantiles":
     case "set World to current selection": {
       const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       let crossfilter = new Crossfilter(world.obsAnnotations);
-      crossfilter = World.createObsDimensions(crossfilter, world);
+      crossfilter = World.createObsDimensions(
+        crossfilter,
+        world,
+        layoutChoice.currentDimNames
+      );
       crossfilter = ControlsHelpers.createGeneDimensions(
         userDefinedGenes,
         diffexpGenes,
@@ -52,6 +60,23 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "set layout choice": {
+      /*
+      when switching layouts:
+      - delete the existing XY index
+      - add the new XY index (which implicitly selects all on it)
+      */
+      const { world, layoutChoice } = nextSharedState;
+      return state
+        .delDimension(layoutDimensionName("XY"))
+        .addDimension(
+          layoutDimensionName("XY"),
+          "spatial",
+          world.obsLayout.col(layoutChoice.currentDimNames[0]).asArray(),
+          world.obsLayout.col(layoutChoice.currentDimNames[1]).asArray()
+        );
+    }
+
     case "request user defined gene success": {
       const { world } = prevSharedState;
       const gene = action.data.genes[0];
@@ -65,8 +90,9 @@ const CrossfilterReducer = (
 
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const genes = _.map(action.data, d =>
-        world.varAnnotations.at(d[0], "name")
+        world.varAnnotations.at(d[0], varIndexName)
       );
       const crossfilter = _.reduce(
         genes,
@@ -84,10 +110,11 @@ const CrossfilterReducer = (
 
     case "clear differential expression": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const crossfilter = _.reduce(
         action.diffExp,
         (xfltr, values) => {
-          const name = world.varAnnotations.at(values[0], "name");
+          const name = world.varAnnotations.at(values[0], varIndexName);
           return xfltr.delDimension(diffexpDimensionName(name));
         },
         state
@@ -109,40 +136,37 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
-    case "graph brush selection change": {
-      const name = layoutDimensionName("XY");
-      const [x0, y0] = action.brushCoords.northwest;
-      const [x1, y1] = action.brushCoords.southeast;
-      return state.select(name, {
+    case "graph brush end":
+    case "graph brush change": {
+      const [minX, maxY] = action.brushCoords.northwest;
+      const [maxX, minY] = action.brushCoords.southeast;
+      return state.select(XYDimName, {
         mode: "within-rect",
-        x0,
-        y0,
-        x1,
-        y1
+        minX,
+        minY,
+        maxX,
+        maxY
       });
     }
 
-    case "lasso deselect":
-    case "graph brush deselect": {
-      const name = layoutDimensionName("XY");
-      return state.select(name, { mode: "all" });
-    }
-
-    case "lasso selection": {
+    case "graph lasso end": {
       const { polygon } = action;
-      const name = layoutDimensionName("XY");
-      if (polygon.length < 3) {
-        // single point or a line is not a polygon, and is therefore a deselect
-        return state.select(name, { mode: "all" });
-      }
-      return state.select(name, {
+      return state.select(XYDimName, {
         mode: "within-polygon",
         polygon
       });
     }
 
+    case "graph lasso cancel":
+    case "graph brush cancel":
+    case "graph lasso deselect":
+    case "graph brush deselect": {
+      return state.select(XYDimName, { mode: "all" });
+    }
+
     case "continuous metadata histogram start":
     case "continuous metadata histogram brush":
+    case "continuous metadata histogram cancel":
     case "continuous metadata histogram end": {
       const name = makeContinuousDimensionName(
         action.continuousNamespace,
@@ -161,10 +185,12 @@ const CrossfilterReducer = (
     case "categorical metadata filter select":
     case "categorical metadata filter deselect": {
       const { categoricalSelection } = nextSharedState;
+      const { world } = prevSharedState;
       const cat = categoricalSelection[action.metadataField];
+      const col = world.obsAnnotations.col(action.metadataField);
       return state.select(obsAnnoDimensionName(action.metadataField), {
         mode: "exact",
-        values: ControlsHelpers.selectedValuesForCategory(cat)
+        values: ControlsHelpers.selectedValuesForCategory(cat, col)
       });
     }
 

client/src/reducers/graphSelection.js

@@ -0,0 +1,61 @@
+const GraphSelection = (
+  state = {
+    tool: "lasso", // what selection tool mode (lasso, brush, ...)
+    selection: { mode: "all" } // current selection, which is tool specific
+  },
+  action
+) => {
+  switch (action.type) {
+    case "set clip quantiles":
+    case "reset World to eq Universe":
+    case "set layout choice": {
+      return {
+        ...state,
+        selection: {
+          mode: "all"
+        }
+      };
+    }
+
+    case "graph brush end":
+    case "graph brush change": {
+      const { brushCoords } = action;
+      return {
+        ...state,
+        selection: {
+          mode: "within-rect",
+          brushCoords
+        }
+      };
+    }
+
+    case "graph lasso end": {
+      const { polygon } = action;
+      return {
+        ...state,
+        selection: {
+          mode: "within-polygon",
+          polygon
+        }
+      };
+    }
+
+    case "graph lasso cancel":
+    case "graph brush cancel":
+    case "graph lasso deselect":
+    case "graph brush deselect": {
+      return {
+        ...state,
+        selection: {
+          mode: "all"
+        }
+      };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default GraphSelection;

client/src/reducers/index.js

@@ -8,78 +8,45 @@ import universe from "./universe";
 import world from "./world";
 import categoricalSelection from "./categoricalSelection";
 import continuousSelection from "./continuousSelection";
+import graphSelection from "./graphSelection";
 import crossfilter from "./crossfilter";
 import colors from "./colors";
 import differential from "./differential";
+import layoutChoice from "./layoutChoice";
 import responsive from "./responsive";
 import controls from "./controls";
 import resetCache from "./resetCache";
+import centroidLabel from "./centroidLabel";
 
-const ignoredActions = new Set([
-  // these actions will not affect history, ie, we will
-  // not snapshot history upon these actions.  These take
-  // precedent over `clearHistoryUponActions`
-  "url changed",
-  "interface reset started",
-  "initial data load start",
-  "configuration load complete",
-  "increment graph render counter",
-  "window resize",
-
-  "lasso started",
-
-  "request differential expression success",
-
-  "expression load start",
-  "expression load success",
-  "expression load error",
-
-  "continuous metadata histogram brush",
-  "continuous metadata histogram end",
-
-  "request user defined gene started",
-  "request user defined gene success",
-  "request user defined gene error",
-  "bulk user defined gene complete",
-  "single user defined gene complete"
-]);
-
-const clearOnActions = new Set([
-  // history will be cleared when these actions occur
-  "initial data load complete (universe exists)",
-  "reset World to eq Universe",
-  "initial data load error"
-]);
-
-/* configuration for the undoable meta reducer */
-const undoableConfig = {
-  historyLimit: 50, // maximum history size
-  skipActionFilter: (state, action) => ignoredActions.has(action.type),
-  clearOnActionFilter: (state, action) => clearOnActions.has(action.type)
-};
+import undoableConfig from "./undoableConfig";
 
 const Reducer = undoable(
   cascadeReducers([
     ["config", config],
     ["universe", universe],
     ["world", world],
+    ["layoutChoice", layoutChoice],
     ["categoricalSelection", categoricalSelection],
     ["continuousSelection", continuousSelection],
+    ["graphSelection", graphSelection],
     ["crossfilter", crossfilter],
     ["colors", colors],
     ["controls", controls],
     ["differential", differential],
     ["responsive", responsive],
+    ["centroidLabel", centroidLabel],
     ["resetCache", resetCache]
   ]),
   [
     "world",
     "categoricalSelection",
     "continuousSelection",
+    "graphSelection",
     "crossfilter",
     "colors",
     "controls",
-    "differential"
+    "differential",
+    "layoutChoice"
   ],
   undoableConfig
 );

client/src/reducers/layoutChoice.js

@@ -0,0 +1,50 @@
+/*
+we have a UI heuristic to pick the default layout, based on assumptions
+about commonly used names.  Preferentially, pick in the following order:
+
+  1. "umap"
+  2. "tsne"
+  3. "pca"
+  4. give up, use the first available
+*/
+function bestDefaultLayout(layouts) {
+  const preferredNames = ["umap", "tsne", "pca"];
+  const idx = preferredNames.findIndex(name => layouts.indexOf(name) !== -1);
+  if (idx !== -1) return preferredNames[idx];
+  return layouts[0];
+}
+
+const LayoutChoice = (
+  state = {
+    available: [], // all available choices
+    current: undefined, // name of the current layout, eg, 'umap'
+    currentDimNames: [] // dimension name
+  },
+  action,
+  nextSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
+      // set default to default
+      const { schema } = nextSharedState.world;
+      const available = schema.layout.obs.map(v => v.name);
+      const current = bestDefaultLayout(available);
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { available, current, currentDimNames };
+    }
+
+    case "set layout choice": {
+      const { schema } = nextSharedState.world;
+      const current = action.layoutChoice;
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { ...state, current, currentDimNames };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default LayoutChoice;

client/src/reducers/resetCache.js

@@ -1,7 +1,10 @@
 /*
-Reducer which caches derived state to be used in a reset
-*/
+Reducer which caches derived state to be used in a reset or other 
+recomputation.
 
+Currently this only caches the baseline (full universe) world & crossfilter, 
+for use in a Reset.
+*/
 const ResetCacheReducer = (
   state = {
     world: null,

client/src/reducers/undoable.js

@@ -8,35 +8,74 @@ Requires three parameters:
   state to be made "undoable".
 * options - an optional object, which may contain the following parameters:
     * historyLimit: max number of historical states to remember (aka max undo depth)
-    * skipActionFilter: filter function, (state, action) => bool.  If it returns
-      truthy, the current state will not be pushed onto the history stack.
-    * clearOnActionFilter: filter function, (state, action) => bool.  If it returns
-      truthy, the history state will be cleared as part of handling this action.
-
-skipActionFilter has precedence over clearOnActionFilter.
+    * actionFilter: filter function, (state, action, filterState) => value.
+      See below for details.
+    * debug: if truish, will print helpful log messages about history manipulation
 
 This meta reducer accepts three actions types:
 * @@undoable/undo - move back in history
 * @@undoable/redo - move forward in history
 * @@undoable/clear - clear history
+
+---
+
+Action filter - controls the undoable reducer side-effects.  If not
+specified, the action filter defaults to "save", ie, pushes a redo
+point upon each action.
+
+The action filter callback has access to the current action, the entire
+undoable reducer state, and any state it wants to manage ("filterState").
+This filter state will be passed to each action filter call, and any
+value returned (via @@undoable/filterState field described below) will be
+MERGED into the current filter state.
+
+An object must be returned (the "undoable action"), indicating desired
+history state processing.  The undoable action object contents, by key:
+
+  @@undoable/filterAction: required.   Can be one of:
+      "skip" - reduce the current action, but no other side effects.
+        Same as returning false.
+      "clear" - reduce the current action, and clear history state.
+      "save" - push the previous state onto the history stack (ie,
+        before reducing the action)
+      "stashPending" - reduce action, save state as pending.  Does not
+        not commit it to history.  Along with cancelPending and applyPending,
+        can be used to delay commit of history (eg, for multi-action
+        groupings, asynch operations, etc).
+      "cancelPending" - reduce action, cancel any pending state save.
+      "applyPending" - commit any pending state to the history stack,
+        then reduce action.
+
+  @@undoable/filterState: optional.  If this value is set, it will be
+    MERGED into the current filter state. The value and semantics of any
+    filter state are entirely at the discretion of the action filter.
+
 */
+import fromEntries from "../util/fromEntries";
 
 const historyKeyPrefix = "@@undoable/";
 const pastKey = `${historyKeyPrefix}past`;
 const futureKey = `${historyKeyPrefix}future`;
+const filterStateKey = `${historyKeyPrefix}filterState`;
+const filterActionKey = `${historyKeyPrefix}filterAction`;
+const pendingKey = `${historyKeyPrefix}pending`;
 const defaultHistoryLimit = -100;
 
 const Undoable = (reducer, undoableKeys, options = {}) => {
+  const { debug } = options;
   let { historyLimit } = options;
   if (!historyLimit) historyLimit = defaultHistoryLimit;
   if (historyLimit > 0) historyLimit = -historyLimit;
-  const skipActionFilter = options.skipActionFilter || (() => false);
-  const clearOnActionFilter = options.clearOnActionFilter || (() => false);
+  const actionFilter =
+    options.actionFilter || (() => ({ [filterActionKey]: "save" }));
 
   if (!Array.isArray(undoableKeys) || undoableKeys.length === 0)
     throw new Error("undoable keys array must be specified");
   const undoableKeysSet = new Set(undoableKeys);
 
+  /*
+  Undo the current to previous history
+  */
   function undo(currentState) {
     const past = currentState[pastKey];
     const future = currentState[futureKey];
@@ -51,11 +90,15 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
       ...currentState,
       ...fromEntries(newState),
       [pastKey]: newPast,
-      [futureKey]: newFuture
+      [futureKey]: newFuture,
+      [pendingKey]: null
     };
     return nextState;
   }
 
+  /*
+  Replay future, previously undone.
+  */
   function redo(currentState) {
     const past = currentState[pastKey] || [];
     const future = currentState[futureKey] || [];
@@ -70,30 +113,45 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
       ...currentState,
       ...fromEntries(newState),
       [pastKey]: newPast,
-      [futureKey]: newFuture
+      [futureKey]: newFuture,
+      [pendingKey]: null
     };
     return nextState;
   }
 
+  /*
+  Clear the history state.  No side-effects on current state.
+  */
   function clear(currentState) {
     return {
       ...currentState,
       [pastKey]: [],
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: {},
+      [pendingKey]: null
     };
   }
 
-  function skip(currentState, action) {
+  /*
+  Reduce current action, with no history side-effects
+  */
+  function skip(currentState, action, filterState) {
     const past = currentState[pastKey] || [];
+    const pending = currentState[pendingKey];
     const res = reducer(currentState, action);
     return {
       ...res,
       [pastKey]: past,
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: filterState,
+      [pendingKey]: pending
     };
   }
 
-  function save(currentState, action) {
+  /*
+  Save current state in the history, then reduce action.
+  */
+  function save(currentState, action, filterState) {
     const past = currentState[pastKey] || [];
     const currentUndoableState = Object.entries(currentState).filter(kv =>
       undoableKeysSet.has(kv[0])
@@ -103,7 +161,48 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
     const nextState = {
       ...res,
       [pastKey]: newPast,
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: filterState,
+      [pendingKey]: null
+    };
+    return nextState;
+  }
+
+  /*
+  Save current state as pending history change.  No other side effects.
+  */
+  function stashPending(currentState) {
+    const currentUndoableState = Object.entries(currentState).filter(kv =>
+      undoableKeysSet.has(kv[0])
+    );
+    return {
+      ...currentState,
+      [pendingKey]: currentUndoableState
+    };
+  }
+
+  /*
+  Cancel pending history state change.  No other side effects.
+  */
+  function cancelPending(currentState) {
+    return {
+      ...currentState,
+      [pendingKey]: null
+    };
+  }
+
+  /*
+  Push pending state onto the history stack
+  */
+  function applyPending(currentState) {
+    const past = currentState[pastKey] || [];
+    const pendingState = currentState[pendingKey];
+    const newPast = push(past, pendingState, historyLimit);
+    const nextState = {
+      ...currentState,
+      [pastKey]: newPast,
+      [futureKey]: [],
+      [pendingKey]: null
     };
     return nextState;
   }
@@ -111,7 +210,9 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
   return (
     currentState = {
       [pastKey]: [],
-      [futureKey]: []
+      [futureKey]: [],
+      [filterStateKey]: {},
+      [pendingKey]: null
     },
     action
   ) => {
@@ -120,20 +221,53 @@ const Undoable = (reducer, undoableKeys, options = {}) => {
       case "@@undoable/undo": {
         return undo(currentState, action);
       }
+
       case "@@undoable/redo": {
         return redo(currentState, action);
       }
+
       case "@@undoable/clear": {
         return clear(currentState, action);
       }
+
       default: {
-        if (skipActionFilter(currentState, action)) {
-          return skip(currentState, action);
+        const currentFilterState = currentState[filterStateKey];
+        const actionFilterResp = actionFilter(
+          currentState,
+          action,
+          currentFilterState
+        );
+        const {
+          [filterActionKey]: filterAction,
+          [filterStateKey]: filterStateUpdate
+        } = actionFilterResp;
+        const nextFilterState = { ...currentFilterState, ...filterStateUpdate };
+
+        switch (filterAction) {
+          case "clear":
+            if (debug) console.log("---- CLEAR HISTO", action.type);
+            return clear(skip(currentState, action, nextFilterState));
+
+          case "save":
+            if (debug) console.log("---- SAVE HISTO", action.type);
+            return save(currentState, action, nextFilterState);
+
+          case "stashPending":
+            if (debug) console.log("---- STASH PENDING", action.type);
+            return skip(stashPending(currentState), action, nextFilterState);
+
+          case "cancelPending":
+            if (debug) console.log("---- CANCEL PENDING", action.type);
+            return skip(cancelPending(currentState), action, nextFilterState);
+
+          case "applyPending":
+            if (debug) console.log("---- APPLY PENDING", action.type);
+            return skip(applyPending(currentState), action, nextFilterState);
+
+          case "skip":
+          default:
+            return skip(currentState, action, nextFilterState);
         }
-        if (clearOnActionFilter(currentState, action)) {
-          return clear(skip(currentState, action));
-        }
-        return save(currentState, action);
       }
     }
   };
@@ -150,18 +284,4 @@ function push(arr, val, limit = undefined) {
   return narr;
 }
 
-function fromEntries(arr) {
-  /*
-  Similar to Object.fromEntries, but only handles array.
-  This could be replaced with the standard fucnction once it
-  is widely available.   As of 3/20/2019, it has not yet
-  been released in the Chrome stable channel.
-  */
-  const obj = {};
-  for (let i = 0, l = arr.length; i < l; i += 1) {
-    obj[arr[i][0]] = arr[i][1];
-  }
-  return obj;
-}
-
 export default Undoable;

client/src/reducers/undoableConfig.js

@@ -0,0 +1,263 @@
+import StateMachine from "../util/statemachine";
+import createFsmTransitions from "./undoableFsm";
+
+const actionKey = "@@undoable/filterAction";
+const stateKey = "@@undoable/filterState";
+
+/*
+these actions will not affect history
+*/
+const skipOnActions = new Set([
+  "url changed",
+  "interface reset started",
+  "initial data load start",
+  "configuration load complete",
+  "increment graph render counter",
+  "window resize",
+  "user reset start",
+  "reset colorscale",
+
+  "graph brush change",
+  "continuous metadata histogram brush",
+
+  "expression load start",
+  "expression load success",
+  "expression load error",
+
+  "request user defined gene started",
+  "request user defined gene success",
+  "clear all user defined genes",
+
+  "get single gene expression for coloring started",
+  "get single gene expression for coloring error",
+
+  "category value mouse hover start",
+  "category value mouse hover end"
+]);
+
+/*
+identical, repeated occurances of these action types will be debounced.
+Entire action must be identical (all keys).
+*/
+const debounceOnActions = new Set([
+  "color by categorical metadata",
+  "color by continuous metadata",
+  "color by expression"
+]);
+
+/*
+history will be cleared when these actions occur
+*/
+const clearOnActions = new Set([
+  "initial data load complete (universe exists)",
+  "reset World to eq Universe",
+  "initial data load error",
+  "user reset end"
+]);
+
+/*
+An immediate history save will be done for these
+*/
+const saveOnActions = new Set([
+  "categorical metadata filter select",
+  "categorical metadata filter deselect",
+  "categorical metadata filter all of these",
+  "categorical metadata filter none of these",
+
+  "color by categorical metadata",
+  "color by continuous metadata",
+  "color by expression",
+
+  "set scatterplot x",
+  "set scatterplot y",
+
+  "store current cell selection as differential set 1",
+  "store current cell selection as differential set 2",
+
+  "set World to current selection",
+  "set clip quantiles",
+
+  "set layout choice",
+  "change graph interaction mode"
+]);
+
+/**
+StateMachine - processing complex action handling - see FSM graph for
+actual structure, in undoableFsm.js
+**/
+
+/*
+Default FSM actions.  Used to side-effect transitions in the graph.
+See graph definition for the transitions that use each.
+
+Signature:  (fsm, transition, reducerState, reducerAction) => undoableAction
+*/
+const stashPending = fsm => ({
+  [actionKey]: "stashPending",
+  [stateKey]: { fsm }
+});
+const cancelPending = () => ({
+  [actionKey]: "cancelPending",
+  [stateKey]: { fsm: null }
+});
+const applyPending = () => ({
+  [actionKey]: "applyPending",
+  [stateKey]: { fsm: null }
+});
+const skip = (fsm, transition) => ({
+  [actionKey]: "skip",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
+const clear = () => ({ [actionKey]: "clear", [stateKey]: { fsm: null } });
+const save = (fsm, transition) => ({
+  [actionKey]: "save",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
+
+/*
+Error handler for state transitions that are unexpected.  Called by
+StateMachine when it doesn't know what to do.
+
+Signature:  (fsm, event, from) => undoableAction
+*/
+const onFsmError = (fsm, event, from) => {
+  console.error(`FSM error [event: "${event}", state: "${from}"]`, fsm);
+  // In production, try to recover gracefully if we have unexpected state
+  return clear(fsm);
+};
+
+/*
+Definition of the transition graph mapping action types to history side effects.
+*/
+const fsmTransitions = createFsmTransitions(
+  stashPending,
+  cancelPending,
+  applyPending,
+  skip,
+  clear,
+  save
+);
+/* State machine we clone whenever we need to run it */
+const seedFsm = new StateMachine("init", fsmTransitions, onFsmError);
+
+/*
+See undoable.js for description action filter interface description.
+
+Basic approach:
+  * trivial handlers for skip, clear & save cases to keep config simple.
+  * only implement complex state machines where absolutely required (eg,
+    multi-event seleciton and the like)
+*/
+const actionFilter = debug => (state, action, prevFilterState) => {
+  const actionType = action.type;
+  const filterState = {
+    ...prevFilterState,
+    prevAction: action
+  };
+  if (skipOnActions.has(actionType)) {
+    return { [actionKey]: "skip", [stateKey]: filterState };
+  }
+  if (
+    debounceOnActions.has(actionType) &&
+    shallowObjectEq(action, prevFilterState.prevAction)
+  ) {
+    return { [actionKey]: "skip", [stateKey]: filterState };
+  }
+  if (clearOnActions.has(actionType)) {
+    return { [actionKey]: "clear", [stateKey]: filterState };
+  }
+  if (saveOnActions.has(actionType)) {
+    return { [actionKey]: "save", [stateKey]: filterState };
+  }
+
+  /*
+    Else, something more complex OR unknown to us....
+    */
+  if (seedFsm.events.has(actionType)) {
+    let { fsm } = filterState;
+    if (!fsm) {
+      /* no active FSM, so create one in init state */
+      fsm = seedFsm.clone("init");
+    }
+    return fsm.next(action.type, { state, action });
+  }
+
+  /* else, we have no idea what this is - skip it */
+  if (debug) console.log("**** ACTION FILTER EVENT HANDLER MISS", actionType);
+  return { [actionKey]: "skip", [stateKey]: filterState };
+};
+
+/*
+return true if objA and objB are ===, OR if:
+  - are both objects and not null
+  - have same own properties
+  - all values are strict equal (===)
+*/
+function shallowObjectEq(objA, objB) {
+  if (objA === objB) return true;
+  if (!objA || !objB) return false;
+  if (!shallowArrayEq(Object.keys(objA), Object.keys(objB))) return false;
+  if (!shallowArrayEq(Object.values(objA), Object.values(objB))) return false;
+  return true;
+}
+
+/*
+return true if arrA and arrB contain the same strict-equal values,
+in the same order.
+*/
+function shallowArrayEq(arrA, arrB) {
+  if (arrA.length !== arrB.length) return false;
+  for (let i = 0, l = arrA.length; i < l; i += 1) {
+    if (arrA[i] !== arrB[i]) return false;
+  }
+  return true;
+}
+
+/* configuration for the undoable meta reducer */
+const debug = false; // set truish for undoble debugging
+const undoableConfig = {
+  debug,
+  historyLimit: 50, // maximum history size
+  actionFilter: actionFilter(debug)
+};
+
+/*
+this code is strictly for sanity checking configuration, and is only
+enabled when we are debugging the undoable configuration (ie, debug === true).
+*/
+if (debug) {
+  /*
+  Confirm no intersection between the various trivial rejection action filters
+  */
+  if (
+    new Set([...skipOnActions].filter(x => clearOnActions.has(x))).size > 0 ||
+    new Set([...skipOnActions].filter(x => saveOnActions.has(x))).size > 0 ||
+    new Set([...clearOnActions].filter(x => saveOnActions.has(x))).size > 0
+  ) {
+    console.error(
+      "Undoable misconfiguration - action filters have redundant events"
+    );
+  }
+
+  /*
+  Confirm that no FSM events are blocked by a trivial rejection filter.
+  If this occurs, the FSM can't ever see the events needed to process
+  state transitions.
+  */
+  const trivialFilters = new Set([
+    ...skipOnActions,
+    ...clearOnActions,
+    ...saveOnActions
+  ]);
+  const trivialOverlapWithFsm = new Set(
+    [...trivialFilters].filter(x => seedFsm.events.has(x))
+  );
+  if (trivialOverlapWithFsm.size > 0) {
+    console.error(
+      "Undoable misconfiguration - trivival action filter blocking FSM filter",
+      [...trivialOverlapWithFsm]
+    );
+  }
+}
+
+export default undoableConfig;

client/src/reducers/undoableFsm.js

@@ -0,0 +1,206 @@
+/*
+State transition graph for complex action/history interactions.
+
+Assumed configuration from undoableConfig:
+    * By convention, "init" is used as the start state for all, and "done"
+      as the final state.
+    * Unexpected states will result in an error, plus a clear and cancelPending
+      side-effect.
+
+TODO: is is possible there is a more concise format for this, as it is
+a fairly repetitive pattern.
+
+These events are largely one of two types:
+a) async operations or multi-event options that should only be committed
+upon some success criteria, otherwise cancelled.
+
+b) compound actions that should be collapsed into a single history change.
+
+*/
+
+const createFsmTransitions = (
+  stashPending,
+  cancelPending,
+  applyPending,
+  skip,
+  clear,
+  save
+) => {
+  return [
+    /* graph selection brushing */
+    {
+      event: "graph brush start",
+      from: "init",
+      to: "graph brush in progress",
+      action: stashPending
+    },
+    {
+      event: "graph brush cancel",
+      from: "graph brush in progress",
+      to: "done",
+      action: applyPending
+    },
+    {
+      event: "graph brush deselect",
+      from: "graph brush in progress",
+      to: "done",
+      /* if current selection is all, cancelPending.  Else, applyPending */
+      action: (fsm, transition, data) =>
+        data.state.graphSelection.selection.mode === "all"
+          ? cancelPending()
+          : applyPending()
+    },
+    {
+      event: "graph brush end",
+      from: "graph brush in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* graph selection lasso */
+    {
+      event: "graph lasso start",
+      from: "init",
+      to: "graph lasso in progress",
+      action: stashPending
+    },
+    {
+      event: "graph lasso cancel",
+      from: "graph lasso in progress",
+      to: "done",
+      action: applyPending
+    },
+    {
+      event: "graph lasso deselect",
+      from: "graph lasso in progress",
+      to: "done",
+      /* if current selection is all, cancelPending.  Else, applyPending */
+      action: (fsm, transition, data) =>
+        data.state.graphSelection.selection.mode === "all"
+          ? cancelPending()
+          : applyPending()
+    },
+    {
+      event: "graph lasso end",
+      from: "graph lasso in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Continuous metadata histogram brush selection */
+    {
+      event: "continuous metadata histogram start",
+      from: "init",
+      to: "continuous histo select in progress",
+      action: stashPending
+    },
+    {
+      event: "continuous metadata histogram cancel",
+      from: "continuous histo select in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "continuous metadata histogram end",
+      from: "continuous histo select in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Single gene request by user */
+    {
+      event: "single user defined gene start",
+      from: "init",
+      to: "single user gene request in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "single user gene request in progress",
+      to: "single user gene error in progress",
+      action: skip
+    },
+    {
+      event: "single user defined gene error",
+      from: "single user gene error in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "single user defined gene complete",
+      from: "single user gene request in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Bulk gene request by user */
+    {
+      event: "bulk user defined gene start",
+      from: "init",
+      to: "bulk user gene request in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "bulk user gene request in progress",
+      to: "bulk user gene request error in progress",
+      action: skip
+    },
+    {
+      event: "bulk user defined gene error",
+      from: "bulk user gene request error in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "bulk user defined gene complete",
+      from: "bulk user gene request in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Compute Differential Expression button user action */
+    {
+      event: "request differential expression started",
+      from: "init",
+      to: "diffexp in progress",
+      action: stashPending
+    },
+    {
+      event: "request user defined gene error",
+      from: "diffexp in progress",
+      to: "done",
+      action: cancelPending
+    },
+    {
+      event: "request differential expression success",
+      from: "diffexp in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* Clear Differential Expression button user action */
+    {
+      event: "clear differential expression",
+      from: "init",
+      to: "CDE Button in progress",
+      action: stashPending
+    },
+    {
+      event: "clear scatterplot",
+      from: "CDE Button in progress",
+      to: "done",
+      action: applyPending
+    },
+
+    /* clear scatter plot button (eg, on scatterplot view) */
+    {
+      event: "clear scatterplot",
+      from: "init",
+      to: "done",
+      action: save
+    }
+  ];
+};
+
+export default createFsmTransitions;

client/src/reducers/universe.js

@@ -1,5 +1,3 @@
-import _ from "lodash";
-
 import { ControlsHelpers } from "../util/stateManager";
 
 const Universe = (state = null, action, nextSharedState, prevSharedState) => {
@@ -12,9 +10,10 @@ const Universe = (state = null, action, nextSharedState, prevSharedState) => {
     case "expression load success": {
       let { varData } = state;
 
-      // Load new expression data into the varData dataframes, if
+      // Lazy load new expression data into the varData dataframe, if
       // not already present.
-      _.forEach(action.expressionData, (val, key) => {
+      //
+      Object.entries(action.expressionData).forEach(([key, val]) => {
         // If not already in universe.varData, save entire expression column
         if (!varData.hasCol(key)) {
           varData = varData.withCol(key, val);
@@ -22,14 +21,15 @@ const Universe = (state = null, action, nextSharedState, prevSharedState) => {
       });
 
       // Prune size of varData "cache" if getting out of hand....
+      //
       const { userDefinedGenes, diffexpGenes } = prevSharedState;
-      const allTheGenesWeNeed = _.uniq(
-        [].concat(
+      const allTheGenesWeNeed = [
+        ...new Set(
           userDefinedGenes,
           diffexpGenes,
           Object.keys(action.expressionData)
         )
-      );
+      ];
       varData = ControlsHelpers.pruneVarDataCache(varData, allTheGenesWeNeed);
 
       return {

client/src/reducers/world.js

@@ -1,6 +1,6 @@
-import _ from "lodash";
-
 import { World, ControlsHelpers } from "../util/stateManager";
+import clip from "../util/clip";
+import quantile from "../util/quantile";
 
 const WorldReducer = (
   state = null,
@@ -21,7 +21,7 @@ const WorldReducer = (
 
     case "set World to current selection": {
       /* Set viewable world to be the currently selected data */
-      const world = World.createWorldFromCurrentSelection(
+      const world = World.createWorldBySelection(
         action.universe,
         action.world,
         action.crossfilter
@@ -29,16 +29,27 @@ const WorldReducer = (
       return world;
     }
 
+    case "set clip quantiles": {
+      const world = World.createWorldWithNewClip(
+        prevSharedState.universe,
+        state,
+        prevSharedState.crossfilter,
+        action.clipQuantiles
+      );
+      return world;
+    }
+
     case "expression load success": {
       const { universe } = nextSharedState;
       const universeVarData = universe.varData;
-      let worldVarData = state.varData;
+      let unclippedVarData = state.unclipped.varData;
 
-      // Load new expression data into the varData dataframes, if
+      // Lazy load new expression data into the unclipped varData dataframe, if
       // not already present.
-      _.forEach(action.expressionData, (val, key) => {
+      //
+      Object.entries(action.expressionData).forEach(([key, val]) => {
         // If not already in world.varData, save sliced expression column
-        if (!worldVarData.hasCol(key)) {
+        if (!unclippedVarData.hasCol(key)) {
           // Slice if world !== universe, else just use whole column.
           // Use the obsAnnotation index as the cut key, as we keep
           // all world dataframes in sync.
@@ -51,7 +62,7 @@ const WorldReducer = (
           }
 
           // Now build world's varData dataframe
-          worldVarData = worldVarData.withCol(
+          unclippedVarData = unclippedVarData.withCol(
             key,
             worldValSlice,
             state.obsAnnotations.rowIndex
@@ -59,23 +70,55 @@ const WorldReducer = (
         }
       });
 
-      // Prune size of varData "cache" if getting out of hand....
+      // Prune size of varData unclipped dataframe if getting out of hand....
+      //
       const { userDefinedGenes, diffexpGenes } = prevSharedState;
-      const allTheGenesWeNeed = _.uniq(
-        [].concat(
+      const allTheGenesWeNeed = [
+        ...new Set(
           userDefinedGenes,
           diffexpGenes,
           Object.keys(action.expressionData)
         )
-      );
-      worldVarData = ControlsHelpers.pruneVarDataCache(
-        worldVarData,
+      ];
+      unclippedVarData = ControlsHelpers.pruneVarDataCache(
+        unclippedVarData,
         allTheGenesWeNeed
       );
 
+      // at this point, we have the unclipped data in unclippedVarData.
+      // Now create clipped.
+      //   - Drop columns no longer needed
+      //   - Add new columns
+      //
+      let clippedVarData = state.varData;
+      const keysToDrop = clippedVarData.colIndex
+        .keys()
+        .filter(k => !unclippedVarData.hasCol(k));
+      const keysToAdd = unclippedVarData.colIndex
+        .keys()
+        .filter(k => !clippedVarData.hasCol(k));
+      keysToDrop.forEach(k => {
+        clippedVarData = clippedVarData.dropCol(k);
+      });
+      keysToAdd.forEach(k => {
+        const data = unclippedVarData.col(k).asArray();
+        const q = [state.clipQuantiles.min, state.clipQuantiles.max];
+        const [qMinVal, qMaxVal] = quantile(q, data);
+        const clippedData = clip(data, qMinVal, qMaxVal, Number.NaN);
+        clippedVarData = clippedVarData.withCol(
+          k,
+          clippedData,
+          state.obsAnnotations.rowIndex
+        );
+      });
+
       return {
         ...state,
-        varData: worldVarData
+        varData: clippedVarData,
+        unclipped: {
+          ...state.unclipped,
+          varData: unclippedVarData
+        }
       };
     }
 

client/src/util/camera.js

@@ -5,10 +5,9 @@ const mp = require("mouse-position");
 const mb = require("mouse-pressed");
 const key = require("key-pressed");
 
-const panSpeed = 0.4;
+const panSpeed = 1.0; // changed from 0.4 to 1.0 per issue #722
 const scaleSpeed = 0.5;
 const scaleMax = 3;
-// const scaleMin = 1.15
 const scaleMin = 1.03;
 
 function attachCamera(canvas, opts) {

client/src/util/centroid.js

@@ -0,0 +1,58 @@
+import quantile from "./quantile";
+
+/*
+  Centroid coordinate calculation
+*/
+const calcMeanCentroid = (world, annoName, annoValue, layoutDimNames) => {
+  const centroid = { x: 0, y: 0, size: 0 };
+  const annoArray = world.obsAnnotations.col(annoName).asArray();
+  const layoutXArray = world.obsLayout.col(layoutDimNames[0]).asArray();
+  const layoutYArray = world.obsLayout.col(layoutDimNames[1]).asArray();
+
+  for (let i = 0, len = annoArray.length; i < len; i += 1) {
+    if (annoArray[i] === annoValue) {
+      centroid.x += layoutXArray[i];
+      centroid.y += layoutYArray[i];
+      centroid.size += 1;
+    }
+  }
+
+  if (centroid[2] !== 0) {
+    centroid.x /= centroid.size;
+    centroid.y /= centroid.size;
+  }
+
+  return [centroid.x, centroid.y];
+};
+
+const calcMedianCentroid = (world, annoName, annoValue, layoutDimNames) => {
+  const centroidX = [];
+  const centroidY = [];
+  let hasFinite = false;
+
+  const annoArray = world.obsAnnotations.col(annoName).asArray();
+  const layoutXArray = world.obsLayout.col(layoutDimNames[0]).asArray();
+  const layoutYArray = world.obsLayout.col(layoutDimNames[1]).asArray();
+
+  for (let i = 0, len = annoArray.length; i < len; i += 1) {
+    if (annoArray[i] === annoValue) {
+      hasFinite =
+        Number.isFinite(layoutXArray[i]) || Number.isFinite(layoutYArray[i])
+          ? true
+          : hasFinite;
+      centroidX.push(layoutXArray[i]);
+      centroidY.push(layoutYArray[i]);
+    }
+  }
+
+  if (hasFinite) {
+    const medianX = quantile([0.5], Float64Array.from(centroidX));
+    const medianY = quantile([0.5], Float64Array.from(centroidY));
+
+    return [medianX, medianY];
+  }
+
+  return null;
+};
+
+export default calcMedianCentroid;

client/src/util/clip.js

@@ -0,0 +1,25 @@
+/*
+clip - clip all values in a Array or TypedArray, IN PLACE.
+
+Values in array are clipped if less than `lower` or greater than `upper`.
+
+If `setTo` is undefined, values less than `lower` will be set to `lower`,
+and values greater than `upper` will be set to `upper`.  
+
+If `setTo` is not undefined, values outside the [lower, upper] range will be set to 
+`setTo`.
+
+*/
+export default function clip(arr, lower, upper, setTo) {
+	const lowerSet = setTo === undefined ? lower : setTo;
+	const upperSet = setTo === undefined ? upper : setTo;
+	for (let i = 0, l = arr.length; i < l; i += 1) {
+		const v = arr[i];
+		if (v < lower) {
+			arr[i] = lowerSet;
+		} else if (v > upper) {
+			arr[i] = upperSet;
+		}
+	}
+	return arr;
+}

client/src/util/dataframe/dataframe.js

@@ -1,8 +1,19 @@
 import { IdentityInt32Index, isLabelIndex } from "./labelIndex";
 // weird cross-dependency that we should clean up someday...
-import { sort } from "../typedCrossfilter/sort";
-import { isTypedArray, isArrayOrTypedArray, callOnceLazy } from "./util";
+import { sortArray } from "../typedCrossfilter/sort";
+import {
+  isTypedArray,
+  isArrayOrTypedArray,
+  callOnceLazy,
+  memoize
+} from "./util";
 import { summarizeContinuous, summarizeCategorical } from "./summarize";
+import {
+  histogramCategorical,
+  hashCategorical,
+  histogramContinuous,
+  hashContinuous
+} from "./histogram";
 
 /*
 Dataframe is an immutable 2D matrix similiar to Python Pandas Dataframe,
@@ -59,11 +70,28 @@ Dataframe
 **/
 
 class Dataframe {
+  /**
+  memoization helpers.
+  **/
+  static __DataframeId__ = 0;
+
+  static __getId() {
+    const id = Dataframe.__DataframeId__;
+    Dataframe.__DataframeId__ += 1;
+    return id;
+  }
+
   /**
   Constructors & factories
   **/
 
-  constructor(dims, columnarData, rowIndex = null, colIndex = null) {
+  constructor(
+    dims,
+    columnarData,
+    rowIndex = null,
+    colIndex = null,
+    __columnsAccessor = [] // private interface
+  ) {
     /*
     The base constructor is relatively hard to use - as an alternative,
     see factory methods and clone/slice, below.
@@ -74,6 +102,9 @@ class Dataframe {
         or TypedArray of length nRows.
       * rowIndex/colIndex - null (create default index using offsets as key),
         or a caller-provided index.
+      * __columnsAccessor - private interface, do not specify.  Used internally
+        to improve caching of column accessors when possible (eg, clone(), 
+        dropCol(), withCol()).
     All columns and indices must have appropriate dimensionality.
     */
     const [nRows, nCols] = dims;
@@ -93,8 +124,9 @@ class Dataframe {
     this.length = nRows; // convenience accessor for row dimension
     this.rowIndex = rowIndex;
     this.colIndex = colIndex;
+    this.__id = Dataframe.__getId();
 
-    this.__compile();
+    this.__compile(__columnsAccessor);
   }
 
   static __errorChecks(dims, columnarData, rowIndex, colIndex) {
@@ -135,97 +167,126 @@ class Dataframe {
     }
   }
 
-  __compile() {
+  static __compileColumn(column, getRowByOffset, getRowByLabel) {
+    /*
+      Each column accessor is a function which will lookup data by
+      index (ie, is equivalent to dataframe.get(row, col), where 'col'
+      is fixed.
+
+      In addition, each column accessor has several functions:
+
+      asArray() -- return the entire column as a native Array or TypedArray.
+        Crucially, this native array only supports label indexing.
+        Example:
+          const arr = df.col('a').asArray();
+
+      has(rlabel) -- return boolean indicating of the row label
+        is contained within the column.  Example:
+          const isInColumn = df.col('a').includes(99)
+        For the default offset indexing, this is identical to:
+          const isInColumn = (99 > 0) && (99 < df.nRows);
+
+      ihas(roffset) -- same as has(), but accepts a row offset
+        instead of a row label.
+
+      indexOf(value) -- return the label (not offset) of the first instance of
+        'value' in the column.  If you want the offset, just use the builtin JS
+        indexOf() function, available on both Array and TypedArray.
+
+      iget(offset) -- return the value at 'offset'
+
+      ... and more ...
+
+    */
+    const { length } = column;
+    const __id = Dataframe.__getId();
+
+    /* get value by row label */
+    const get = function get(rlabel) {
+      return column[getRowByOffset(rlabel)];
+    };
+
+    /* get value by row offset */
+    const iget = function iget(roffset) {
+      return column[roffset];
+    };
+
+    /* full column array access */
+    const asArray = function asArray() {
+      return column;
+    };
+
+    /* test for row label inclusion in column */
+    const has = function has(rlabel) {
+      const offset = getRowByOffset(rlabel);
+      return offset >= 0 && offset < length;
+    };
+
+    const ihas = function ihas(offset) {
+      return offset >= 0 && offset < length;
+    };
+
+    /*
+    return first label (index) at which the value is found in this column,
+    or undefined if not found.
+
+    NOTE: not found return is DIFFERENT than the default Array.indexOf as
+    -1 is a plausible Dataframe row/col label.
+    */
+    const indexOf = function indexOf(value) {
+      const offset = column.indexOf(value);
+      if (offset === -1) {
+        return undefined;
+      }
+      return getRowByLabel(offset);
+    };
+
+    /*
+    Summarize the column data. Lazy eval;
+    */
+    const summarize = callOnceLazy(() =>
+      isTypedArray(column)
+        ? summarizeContinuous(column)
+        : summarizeCategorical(column)
+    );
+
+    /*
+    Create histogram bins for this column.  Memoized.
+    */
+    if (isTypedArray(column)) {
+      const mFn = memoize(histogramContinuous, hashContinuous);
+      get.histogram = (bins, domain, by) => mFn(get, bins, domain, by);
+    } else {
+      const mFn = memoize(histogramCategorical, hashCategorical);
+      get.histogram = by => mFn(get, by);
+    }
+
+    get.summarize = summarize;
+    get.asArray = asArray;
+    get.has = has;
+    get.ihas = ihas;
+    get.indexOf = indexOf;
+    get.iget = iget;
+    get.__id = __id;
+
+    return get;
+  }
+
+  __compile(accessors) {
     /*
     Compile data accessors for each column.
 
-    Each column accessor is a function which will lookup data by
-    index (ie, is equivalent to dataframe.get(row, col), where 'col'
-    is fixed.
-
-    In addition, each column accessor has several functions:
-
-    asArray() -- return the entire column as a native Array or TypedArray.
-      Crucially, this native array only supports label indexing.
-      Example:
-        const arr = df.col('a').asArray();
-
-    has(rlabel) -- return boolean indicating of the row label
-      is contained within the column.  Example:
-        const isInColumn = df.col('a').includes(99)
-      For the default offset indexing, this is identical to:
-        const isInColumn = (99 > 0) && (99 < df.nRows);
-
-    ihas(roffset) -- same as has(), but accepts a row offset
-      instead of a row label.
-
-    indexOf(value) -- return the label (not offset) of the first instance of
-      'value' in the column.  If you want the offset, just use the builtin JS
-      indexOf() function, available on both Array and TypedArray.
-
-    iget(offset) -- return the value at 'offset'
-
+    Use an existing accessor if provided, else compile a new one.
     */
-    const { getOffset, getLabel } = this.rowIndex;
-    this.__columnsAccessor = this.__columns.map(column => {
-      const { length } = column;
-
-      /* get value by row label */
-      const get = function get(rlabel) {
-        return column[getOffset(rlabel)];
-      };
-
-      /* get value by row offset */
-      const iget = function iget(roffset) {
-        return column[roffset];
-      };
-
-      /* full column array access */
-      const asArray = function asArray() {
-        return column;
-      };
-
-      /* test for row label inclusion in column */
-      const has = function has(rlabel) {
-        const offset = getOffset(rlabel);
-        return offset >= 0 && offset < length;
-      };
-
-      const ihas = function ihas(offset) {
-        return offset >= 0 && offset < length;
-      };
-
-      /*
-      return first label (index) at which the value is found in this column,
-      or undefined if not found.
-
-      NOTE: not found return is DIFFERENT than the default Array.indexOf as
-      -1 is a plausible Dataframe row/col label.
-      */
-      const indexOf = function indexOf(value) {
-        const offset = column.indexOf(value);
-        if (offset === -1) {
-          return undefined;
-        }
-        return getLabel(offset);
-      };
-
-      /*
-      Summarize the column data. Lazy eval;
-      */
-      const summarize = callOnceLazy(() =>
-        isTypedArray(column)
-          ? summarizeContinuous(column)
-          : summarizeCategorical(column)
-      );
-
-      get.summarize = summarize;
-      get.asArray = asArray;
-      get.has = has;
-      get.ihas = ihas;
-      get.indexOf = indexOf;
-      get.iget = iget;
-      return get;
+    const {
+      getOffset: getRowByOffset,
+      getLabel: getRowByLabel
+    } = this.rowIndex;
+    this.__columnsAccessor = this.__columns.map((column, idx) => {
+      if (accessors[idx]) {
+        return accessors[idx];
+      }
+      return Dataframe.__compileColumn(column, getRowByOffset, getRowByLabel);
     });
   }
 
@@ -237,7 +298,8 @@ class Dataframe {
       this.dims,
       [...this.__columns],
       this.rowIndex,
-      this.colIndex
+      this.colIndex,
+      [...this.__columnsAccessor]
     );
   }
 
@@ -273,7 +335,14 @@ class Dataframe {
     const columns = [...this.__columns];
     columns.push(colData);
     const colIndex = this.colIndex.withLabel(label);
-    return new this.constructor(dims, columns, rowIndex, colIndex);
+    const columnsAccessor = [...this.__columnsAccessor];
+    return new this.constructor(
+      dims,
+      columns,
+      rowIndex,
+      colIndex,
+      columnsAccessor
+    );
   }
 
   dropCol(label) {
@@ -287,7 +356,15 @@ class Dataframe {
     const columns = [...this.__columns];
     columns.splice(coffset, 1);
     const colIndex = this.colIndex.dropLabel(label);
-    return new this.constructor(dims, columns, this.rowIndex, colIndex);
+    const columnsAccessor = [...this.__columnsAccessor];
+    columnsAccessor.splice(coffset, 1);
+    return new this.constructor(
+      dims,
+      columns,
+      this.rowIndex,
+      colIndex,
+      columnsAccessor
+    );
   }
 
   static empty(rowIndex = null, colIndex = null) {
@@ -316,7 +393,7 @@ class Dataframe {
       if (!offsets) {
         return [null, null];
       }
-      const sortedOffsets = sort(offsets);
+      const sortedOffsets = sortArray(offsets);
       const sortedLabels = new Array(sortedOffsets.length);
       for (let i = 0, l = sortedOffsets.length; i < l; i += 1) {
         sortedLabels[i] = index.getLabel(sortedOffsets[i]);
@@ -543,14 +620,34 @@ class Dataframe {
   /****
   Functional (map/reduce/etc) data access
 
-  XXX: not yet implemented, as there is no clear use case.   Can easily
+  TODO: most are not yet implemented, as there is no clear use case.   Can easily
   add these as useful.
   ****/
 
+  mapColumns(callback) {
+    /*
+    map all columns in the dataframe, returning a new dataframe comprised of the
+    return values, with the same index as the original dataframe.
+
+    callback MUST not modify the column, but instead return a mutated copy.
+    */
+    const columns = this.__columns.map(callback);
+    const columnsAccessor = columns.map((c, idx) =>
+      this.__columns[idx] === c ? this.__columnsAccessor[idx] : undefined
+    );
+    return new this.constructor(
+      this.dims,
+      columns,
+      this.rowIndex,
+      this.colIndex,
+      columnsAccessor
+    );
+  }
+
   /*
   Map & reduce of column or row
 
-  XXX TODO remainder of map/reduce functions:  mapCol, mapRow, reduceRow, ...
+  TODO remainder of map/reduce functions:  mapCol, mapRow, reduceRow, ...
   */
   /* comment out until we have a use for this
 

client/src/util/dataframe/histogram.js

@@ -0,0 +1,135 @@
+/*
+Dataframe histogram
+*/
+import { isTypedArray } from "./util";
+
+function _histogramContinuous(column, bins, min, max) {
+  const valBins = new Array(bins).fill(0);
+  if (!column) {
+    return valBins;
+  }
+  const binWidth = (max - min) / (bins - 1);
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const val = colArray[r];
+    if (val <= max && val >= min) {
+      // ensure test excludes NaN values
+      const valBin = (val - min) / binWidth;
+      valBins[valBin] += 1;
+    }
+  }
+  return valBins;
+}
+
+function _histogramContinuousBy(column, bins, min, max, by) {
+  const byMap = new Map();
+  if (!column || !by) {
+    return byMap;
+  }
+  const binWidth = (max - min) / (bins - 1);
+  const byArray = by.asArray();
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const byBin = byArray[r];
+    let valBins = byMap.get(byBin);
+    if (valBins === undefined) {
+      valBins = new Array(bins).fill(0);
+      byMap.set(byBin, valBins);
+    }
+    const val = colArray[r];
+    if (val <= max && val >= min) {
+      // ensure test excludes NaN values
+      const valBin = (val - min) / binWidth;
+      valBins[Math.floor(valBin)] += 1;
+    }
+  }
+  return byMap;
+}
+
+function _histogramCategorical(column) {
+  const valMap = new Map();
+  if (!column) {
+    return valMap;
+  }
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const valBin = colArray[r];
+    let curCount = valMap.get(valBin);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    valMap.set(valBin, curCount + 1);
+  }
+  return valMap;
+}
+
+function _histogramCategoricalBy(column, by) {
+  const byMap = new Map();
+  if (!column || !by) {
+    return byMap;
+  }
+  const byArray = by.asArray();
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const byBin = byArray[r];
+    let valMap = byMap.get(byBin);
+    if (valMap === undefined) {
+      valMap = new Map();
+      byMap.set(byBin, valMap);
+    }
+    const valBin = colArray[r];
+    let curCount = valMap.get(valBin);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    valMap.set(valBin, curCount + 1);
+  }
+  return byMap;
+}
+
+/*
+Count category occupancy.  Optional group-by category.
+*/
+export function histogramCategorical(column, by) {
+  if (by && isTypedArray(by)) {
+    throw new Error("Group by column must be categorical");
+  }
+  return by
+    ? _histogramCategoricalBy(column, by)
+    : _histogramCategorical(column);
+}
+
+/*
+Memoization hash for histogramCategorical()
+*/
+export function hashCategorical(column, by) {
+  if (by) {
+    return `${column.__id}:${by.__id}`;
+  }
+  return `${column.__id}:`;
+}
+
+/*
+Bin counts for continuous/scalar values, with optional group-by category.
+Values outside domain are ignored.
+*/
+export function histogramContinuous(column, bins = 40, domain = [0, 1], by) {
+  if (by && isTypedArray(by)) {
+    throw new Error("Group by column must be categorical");
+  }
+  const [min, max] = domain;
+  return by
+    ? _histogramContinuousBy(column, bins, min, max, by)
+    : _histogramContinuous(column, bins, min, max);
+}
+
+/*
+Memoization hash for histogramContinuous
+*/
+export function hashContinuous(column, bins = "", domain = [0, 0], by) {
+  const [min, max] = domain;
+  if (by) {
+    return `${column.__id}:${bins}:${min}:${max}:${by.__id}`;
+  }
+  return `${column.__id}::${bins}:${min}:${max}`;
+}

client/src/util/dataframe/labelIndex.js

@@ -3,6 +3,8 @@ Label indexing - map a label to & from an integer offset.  See Dataframe
 for how this is used.
 **/
 
+import { rangeFill as fillRange } from "../range";
+
 /*
 Private utility functions
 */
@@ -21,14 +23,6 @@ function extent(tarr) {
   return [min, max];
 }
 
-function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, l = larr.length; i < l; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 /* eslint-disable class-methods-use-this */
 class IdentityInt32Index {
   /*

client/src/util/dataframe/summarize.js

@@ -1,32 +1,56 @@
 /*
 Private dataframe support functions
+
+TODO / XXX: for scalar/continuous data, this uses a naive method
+of computing quantiles.  Would be good to switch from sort to 
+partition at some point.
 */
 
+import quantile from "../quantile";
+import { sortArray } from "../typedCrossfilter/sort";
+
+// [ 0, 0.01, 0.02, ..., 1.0]
+const centileNames = new Array(101).fill(0).map((v, idx) => idx / 100);
+
 export function summarizeContinuous(col) {
   let min;
   let max;
   let nan = 0;
   let pinf = 0;
   let ninf = 0;
+  let percentiles;
   if (col) {
-    for (let r = 0, l = col.length; r < l; r += 1) {
-      const val = Number(col[r]);
-      if (Number.isFinite(val)) {
-        if (min === undefined) {
-          min = val;
-          max = val;
-        } else {
-          min = val < min ? val : min;
-          max = val > max ? val : max;
-        }
-      } else if (Number.isNaN(val)) {
-        nan += 1;
-      } else if (val > 0) {
-        pinf += 1;
-      } else {
-        ninf += 1;
+    // -Inf < finite < Inf < NaN
+    const sortedCol = sortArray(new col.constructor(col));
+
+    // count non-finites, which are at each end of sorted data
+    for (let i = sortedCol.length - 1; i >= 0; i -= 1) {
+      if (!Number.isNaN(sortedCol[i])) {
+        nan = sortedCol.length - i - 1;
+        break;
       }
     }
+    for (let i = 0, l = sortedCol.length; i < l; i += 1) {
+      if (sortedCol[i] !== Number.NEGATIVE_INFINITY) {
+        ninf = i;
+        break;
+      }
+    }
+    for (let i = sortedCol.length - nan - 1; i >= 0; i -= 1) {
+      if (sortedCol[i] !== Number.POSITIVE_INFINITY) {
+        pinf = sortedCol.length - i - nan - 1;
+        break;
+      }
+    }
+
+    // compute percentiles on finite data ONLY
+    const sortedColFiniteOnly = sortedCol.slice(
+      ninf,
+      sortedCol.length - nan - pinf
+    );
+    percentiles = quantile(centileNames, sortedColFiniteOnly, true);
+    min = percentiles[0];
+    max = percentiles[100];
   }
   return {
     categorical: false,
@@ -34,7 +58,8 @@ export function summarizeContinuous(col) {
     max,
     nan,
     pinf,
-    ninf
+    ninf,
+    percentiles
   };
 }
 

client/src/util/dataframe/util.js

@@ -2,18 +2,13 @@
 Private utility code for dataframe
 */
 
-export function isTypedArray(x) {
-  return (
-    ArrayBuffer.isView(x) &&
-    Object.prototype.toString.call(x) !== "[object DataView]"
-  );
-}
-
-export function isArrayOrTypedArray(x) {
-  return Array.isArray(x) || isTypedArray(x);
-}
+export { isTypedArray, isArrayOrTypedArray } from "../typeHelpers";
 
 export function callOnceLazy(f) {
+  /*
+  call function once, and save the result, regardless of arguments (this is not
+  the same as typical memoization).
+  */
   let value;
   let calledOnce = false;
   const result = function result(...args) {
@@ -23,6 +18,25 @@ export function callOnceLazy(f) {
     }
     return value;
   };
-
   return result;
 }
+
+export function memoize(fn, hashFn) {
+  /* 
+  function memoization, with user-provided hash.  hashFn must return a
+  key which will be unique as a Map key (ie, obeys "sameValueZero" algorithm
+  as defined in the JS spec).  For more info on hash key, see:
+  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map#Key_equality
+  */
+  const cache = new Map();
+  const wrap = function wrap(...args) {
+    const key = hashFn(...args);
+    if (cache.has(key)) {
+      return cache.get(key);
+    }
+    const result = fn(...args);
+    cache.set(key, result);
+    return result;
+  };
+  return wrap;
+}

client/src/util/fromEntries.js

@@ -0,0 +1,13 @@
+export default function fromEntries(arr) {
+	/*
+	Similar to Object.fromEntries, but only handles array.
+	This could be replaced with the standard fucnction once it
+	is widely available.   As of 3/20/2019, it has not yet
+	been released in the Chrome stable channel.
+	*/
+	const obj = {};
+	for (let i = 0, l = arr.length; i < l; i += 1) {
+		obj[arr[i][0]] = arr[i][1];
+	}
+	return obj;
+}

client/src/util/quantile.js

@@ -0,0 +1,29 @@
+/*
+quantiles - calculate quantiles for the typed array.
+
+Currently interpolates to 'lower' value.
+
+Arguments:
+
+	* quantArr - array of quantiles to compute, where values:  0 <= value <= 1.0
+	* tarr - a typed array
+	* sorted - option bool.  If false (default), will assume array is not sorted.
+	 	If true, will assume it is sorted.
+
+*/
+
+import { sortArray } from "./typedCrossfilter/sort";
+
+export default function quantile(quantArr, tarr, sorted = false) {
+	/*
+	start with the naive (sort) implementation.  Later, use a faster partition
+	*/
+	const arr = sorted ? tarr : sortArray(new tarr.constructor(tarr)); // copy
+	const len = arr.length;
+	return quantArr.map(q => {
+		if (q === 1) {
+			return arr[len - 1];
+		}
+		return arr[Math.floor(q * len)];
+	});
+}

client/src/util/range.js

@@ -0,0 +1,45 @@
+/*
+Array range creation
+
+range(start, stop, step) -> Array
+	This is identical to https://docs.python.org/3/library/functions.html#func-range
+	Returns new array filled with a range of numbers.
+
+	Usage:
+
+		range(stop) - start defaults to zero, step defaults to 1
+		range(start, stop, [step]) - step defaults to 1
+
+	Examples:
+		range(3) -> [0, 1, 2]
+		range(1, 3) -> [1, 2]
+		range(1, 5, 2) -> [1, 3]
+
+
+rangeFill(array, start, step) -> array
+	Fill entire array with values, from start, by step.  Returns first array.
+	start defaults to zero, step defaults to one.
+
+*/
+
+function _doFill(arr, start, step, count) {
+	for (let idx = 0, val = start; idx < count; idx += 1, val += step) {
+		arr[idx] = val;
+	}
+	return arr;
+}
+
+export function rangeFill(arr, start = 0, step = 1) {
+	return _doFill(arr, start, step, arr.length);
+}
+
+export function range(start, stop, step) {
+	if (start === undefined) return [];
+	if (stop === undefined) {
+		stop = start;
+		start = 0;
+	}
+	step = step || 1; // catch undefind and zero
+	const len = Math.max(Math.ceil((stop - start) / step), 0);
+	return _doFill(new Array(len), start, step, len);
+}

client/src/util/scaleLinear.js

@@ -9,8 +9,14 @@
 // this is is equivalent to d3.scaleLinear().domain([0,1]).range([-1,1])
 
 export default (domain, range) => {
-  const domainStart = domain[0];
-  const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
-  const rangeStart = range[0];
-  return value => (value - domainStart) * scale + rangeStart;
+	const domainStart = domain[0];
+	const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+	const invScale = 1 / scale;
+	const rangeStart = range[0];
+	const f = value => (value - domainStart) * scale + rangeStart;
+
+	// inverter
+	f.invert = value => (value - rangeStart) * invScale + domainStart;
+
+	return f;
 };

client/src/util/stateManager/colorHelpers.js

@@ -1,12 +1,12 @@
 /*
 Helper functions for the embedded graph colors
 */
-import _ from "lodash";
 import * as d3 from "d3";
 import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
 import * as globals from "../../globals";
 import parseRGB from "../parseRGB";
 import finiteExtent from "../finiteExtent";
+import { range } from "../range";
 
 /*
 create new colors state object.   Paramters:
@@ -15,7 +15,7 @@ create new colors state object.   Paramters:
     "color by continuous metadata", "color by categorical metadata"
   -
 */
-function createColors(world, colorMode = null, colorAccessor = null) {
+export function createColors(world, colorMode = null, colorAccessor = null) {
   switch (colorMode) {
     case "color by categorical metadata": {
       return createColorsByCategoricalMetadata(world, colorAccessor);
@@ -37,9 +37,7 @@ function createColors(world, colorMode = null, colorAccessor = null) {
 }
 
 function createColorsByCategoricalMetadata(world, accessor) {
-  const { categories } = _.filter(world.schema.annotations.obs, {
-    name: accessor
-  })[0];
+  const { categories } = world.schema.annotations.obsByName[accessor];
 
   const scale = d3
     .scaleSequential(interpolateRainbow)
@@ -67,7 +65,7 @@ function createColorsByContinuousMetadata(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);
@@ -97,7 +95,7 @@ function createColorsByExpression(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);
@@ -119,4 +117,26 @@ function createColorsByExpression(world, accessor) {
   return { rgb, scale };
 }
 
-export default createColors;
+export const resetColors = world => {
+  const { rgb, scale } = createColors(world);
+  return {
+    colorMode: null,
+    colorAccessor: null,
+    rgb,
+    scale
+  };
+};
+
+export const checkIfColorByDiffexpAndResetColors = (
+  prevControls,
+  state,
+  prevWorld
+) => {
+  if (prevControls.diffexpGenes.includes(state.colorAccessor)) {
+    return {
+      ...state,
+      ...resetColors(prevWorld)
+    };
+  }
+  return null;
+};

client/src/util/stateManager/controlsHelpers.js

@@ -5,7 +5,7 @@ Helper functions for the controls reducer
 import _ from "lodash";
 
 import * as globals from "../../globals";
-import { fillRange } from "../typedCrossfilter/util";
+import { rangeFill as fillRange } from "../range";
 import {
   userDefinedDimensionName,
   diffexpDimensionName
@@ -21,16 +21,16 @@ Remember that option values can be ANY js type, except undefined/null.
   {
     _category_name_1: {
       // map of option value to index
-      categoryIndices: Map([
+      categoryValueIndices: Map([
         catval1: index,
         ...
       ])
 
       // index->selection true/false state
-      categorySelected: [ true/false, true/false, ... ]
+      categoryValueSelected: [ true/false, true/false, ... ]
 
       // number of options
-      numCategories: number,
+      numCategoryValues: number,
 
       // isTruncated - true if the options for selection has
       // been truncated (ie, was too large to implement)
@@ -56,27 +56,31 @@ function topNCategories(summary) {
 
 export function createCategoricalSelection(maxCategoryItems, world) {
   const res = {};
+  const obsIndexName = world.schema.annotations.obs.index;
   _.forEach(world.obsAnnotations.colIndex.keys(), key => {
     const summary = world.obsAnnotations.col(key).summarize();
     if (summary.categories) {
       const isColorField = key.includes("color") || key.includes("Color");
       const isSelectableCategory =
         !isColorField &&
-        key !== "name" &&
+        key !== obsIndexName &&
         summary.categories.length < maxCategoryItems;
       if (isSelectableCategory) {
-        const [categoryValues, categoryCounts] = topNCategories(summary);
-        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
-        const numCategories = categoryIndices.size;
-        const categorySelected = new Array(numCategories).fill(true);
+        const [categoryValues, categoryValueCounts] = topNCategories(summary);
+        const categoryValueIndices = new Map(
+          categoryValues.map((v, i) => [v, i])
+        );
+        const numCategoryValues = categoryValueIndices.size;
+        const categoryValueSelected = new Array(numCategoryValues).fill(true);
         const isTruncated = categoryValues.length < summary.numCategories;
         res[key] = {
           categoryValues, // array: of natively typed category values
-          categoryIndices, // map: category value (native type) -> category index
-          categorySelected, // array: t/f selection state
-          numCategories, // number: of categories
+          categoryValueIndices, // map: category value (native type) -> category index
+          categoryValueSelected, // array: t/f selection state
+          numCategoryValues, // number: of values in the category
           isTruncated, // bool: true if list was truncated
-          categoryCounts // array: cardinality of each category
+          categoryValueCounts, // array: cardinality of each category,
+          categorySelected: true // bool - default state for entire category
         };
       }
     }
@@ -88,12 +92,26 @@ export function createCategoricalSelection(maxCategoryItems, world) {
 given a categoricalSelection, return the list of all category values
 where selection state is true (ie, they are selected).
 */
-export function selectedValuesForCategory(categorySelectionState) {
-  const selectedValues = _([...categorySelectionState.categoryIndices])
-    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
-    .map(tuple => tuple[0])
-    .value();
-  return selectedValues;
+export function selectedValuesForCategory(categorySelectionState, dfColumn) {
+  const {
+    categorySelected,
+    categoryValueSelected,
+    categoryValueIndices
+  } = categorySelectionState;
+  let selectedValues;
+  if (categorySelected) {
+    selectedValues = new Set(dfColumn.summarize().categories);
+  } else {
+    selectedValues = new Set();
+  }
+  categoryValueIndices.forEach((catIndex, catValue) => {
+    if (!categoryValueSelected[catIndex]) {
+      selectedValues.delete(catValue);
+    } else {
+      selectedValues.add(catValue);
+    }
+  });
+  return [...selectedValues.values()];
 }
 
 /*

client/src/util/stateManager/index.js

@@ -14,7 +14,7 @@ This is all VERY tightly integrated with reducers and actions, and
 exists to support those concepts.
 */
 
-export { default as createColors } from "./colorHelpers";
+export * as ColorHelpers from "./colorHelpers";
 export * as Universe from "./universe";
 export * as World from "./world";
 export * as WorldUtil from "./worldUtil";

client/src/util/stateManager/universe.js

@@ -4,6 +4,8 @@ import _ from "lodash";
 
 import decodeMatrixFBS from "./matrix";
 import * as Dataframe from "../dataframe";
+import fromEntries from "../fromEntries";
+import { isFpTypedArray } from "../typeHelpers";
 
 /*
 Private helper function - create and return a template Universe
@@ -37,14 +39,58 @@ These functions are used exclusively by the actions and reducers to
 build an internal POJO for use by the rendering components.
 */
 
+function promoteTypedArray(o) {
+  /*
+  Decide what internal data type to use for the data returned from 
+  the server.
+
+  TODO - future optimization: not all int32/uint32 data series require
+  promotion to float64.  We COULD simply look at the data to decide. 
+  */
+  if (isFpTypedArray(o) || Array.isArray(o)) return o;
+
+  let TyepdArrayCtor;
+  switch (o.constructor) {
+    case Int8Array:
+    case Uint8Array:
+    case Uint8ClampedArray:
+    case Int16Array:
+    case Uint16Array:
+      TyepdArrayCtor = Float32Array;
+      break;
+
+    case Int32Array:
+    case Uint32Array:
+      TyepdArrayCtor = Float64Array;
+      break;
+
+    default:
+      throw new Error("Unexpected data type returned from server.");
+  }
+  if (o.constructor === TyepdArrayCtor) return o;
+  return new TyepdArrayCtor(o);
+}
+
 function AnnotationsFBSToDataframe(arrayBuffer) {
   /*
   Convert a Matrix FBS to a Dataframe.
+
+  The application has strong assumptions that all scalar data will be
+  stored as a float32 or float64 (regardless of underlying data types).
+  For example, clipping of value ranges (eg, user-selected percentiles)
+  depends on the ability to use NaN in any numeric type.
+
+  All float data from the server is left as is.  All non-float is promoted
+  to an appropriate float.
   */
-  const fbs = decodeMatrixFBS(arrayBuffer);
+  const fbs = decodeMatrixFBS(arrayBuffer, true); // leave in place
+  const columns = fbs.columns.map(c => {
+    if (isFpTypedArray(c) || Array.isArray(c)) return c;
+    return promoteTypedArray(c);
+  });
   const df = new Dataframe.Dataframe(
     [fbs.nRows, fbs.nCols],
-    fbs.columns,
+    columns,
     null,
     new Dataframe.KeyIndex(fbs.colIdx)
   );
@@ -53,11 +99,16 @@ function AnnotationsFBSToDataframe(arrayBuffer) {
 
 function LayoutFBSToDataframe(arrayBuffer) {
   const fbs = decodeMatrixFBS(arrayBuffer, true);
+  if (fbs.columns.length < 2 || !fbs.columns.every(isFpTypedArray)) {
+    // We have strong assumptions about the shape & type of layout data.
+    throw new Error("Unexpected layout data type returned from server");
+  }
+
   const df = new Dataframe.Dataframe(
     [fbs.nRows, fbs.nCols],
     fbs.columns,
     null,
-    new Dataframe.KeyIndex(["X", "Y"])
+    new Dataframe.KeyIndex(fbs.colIdx)
   );
   return df;
 }
@@ -73,15 +124,15 @@ function reconcileSchemaCategoriesWithSummary(universe) {
   cases, add a 'categories' field to the schema so it is accessible.
   */
 
-  _.forEach(universe.schema.annotations.obs, s => {
+  universe.schema.annotations.obs.columns.forEach(s => {
     if (
       s.type === "string" ||
       s.type === "boolean" ||
       s.type === "categorical"
     ) {
       const categories = _.union(
-        _.get(s, "categories", []),
-        _.get(universe.obsAnnotations.col(s.name).summarize(), "categories", [])
+        s.categories ?? [],
+        universe.obsAnnotations.col(s.name).summarize().categories ?? []
       );
       s.categories = categories;
     }
@@ -105,6 +156,9 @@ export function createUniverseFromResponse(
   universe.schema = schema;
   universe.nObs = schema.dataframe.nObs;
   universe.nVar = schema.dataframe.nVar;
+  /* add defaults, as we can't assume back-end will fully populate schema */
+  if (!schema.layout.var) schema.layout.var = [];
+  if (!schema.layout.obs) schema.layout.obs = [];
 
   /* annotations */
   universe.obsAnnotations = AnnotationsFBSToDataframe(annotationsObsResponse);
@@ -122,6 +176,20 @@ export function createUniverseFromResponse(
   }
 
   reconcileSchemaCategoriesWithSummary(universe);
+
+  /* Index schema for ease of use */
+  universe.schema.annotations.obsByName = fromEntries(
+    universe.schema.annotations.obs.columns.map(v => [v.name, v])
+  );
+  universe.schema.annotations.varByName = fromEntries(
+    universe.schema.annotations.var.columns.map(v => [v.name, v])
+  );
+  universe.schema.layout.obsByName = fromEntries(
+    universe.schema.layout.obs.map(v => [v.name, v])
+  );
+  universe.schema.layout.varByName = fromEntries(
+    universe.schema.layout.var.map(v => [v.name, v])
+  );
   return universe;
 }
 
@@ -140,8 +208,14 @@ export function convertDataFBStoObject(universe, arrayBuffer) {
   const { colIdx, columns } = fbs;
   const result = {};
 
+  if (!columns.every(isFpTypedArray)) {
+    // We have strong assumptions that all var data is float
+    throw new Error("Unexpected non-floating point response from server.");
+  }
+
+  const varIndexName = universe.schema.annotations.var.index;
   for (let c = 0; c < colIdx.length; c += 1) {
-    const varName = universe.varAnnotations.at(colIdx[c], "name");
+    const varName = universe.varAnnotations.at(colIdx[c], varIndexName);
     result[varName] = columns[c];
   }
   return result;

client/src/util/stateManager/world.js

@@ -1,7 +1,14 @@
 // jshint esversion: 6
 
-import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
+import clip from "../clip";
+import {
+  layoutDimensionName,
+  obsAnnoDimensionName,
+  diffexpDimensionName,
+  userDefinedDimensionName
+} from "../nameCreators";
 import * as Dataframe from "../dataframe";
+import ImmutableTypedCrossfilter from "../typedCrossfilter/crossfilter";
 
 /*
 
@@ -19,6 +26,8 @@ Notable keys in the world object:
 
 * schema: data schema from the server
 
+* clipQuantiles: the quantiles used to clip all data in world.
+
 * obsAnnotations:
 
   Dataframe containing obs annotations.  Columns are indexed by annotation
@@ -37,78 +46,192 @@ Notable keys in the world object:
 * varData: a cache of expression columns, stored in a Dataframe.  Cache
   managed by controls reducer.
 
+* unclipped: will contain unclipped variants of all potentiall clipped
+  dataframes (obsAnnotations, varData).
+
 */
 
 function templateWorld() {
+  const obsAnnotations = Dataframe.Dataframe.empty();
+  const varAnnotations = Dataframe.Dataframe.empty();
+  const obsLayout = Dataframe.Dataframe.empty();
+  const varData = Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex());
   return {
     /* schema/version related */
     schema: null,
     nObs: 0,
     nVar: 0,
+    clipQuantiles: { min: 0, max: 1 },
 
     /* annotations */
-    obsAnnotations: Dataframe.Dataframe.empty(),
-    varAnnotations: Dataframe.Dataframe.empty(),
+    obsAnnotations,
+    varAnnotations,
 
     /* layout of graph. Dataframe. */
-    obsLayout: Dataframe.Dataframe.empty(),
+    obsLayout,
 
-    /*
-    Var data columns - subset of all data (may be empty)
-    */
-    varData: Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex())
+    /* Var data columns - subset of all data (may be empty) */
+    varData,
+
+    /* unclipped dataframes - subset, but not value clipped */
+    unclipped: {
+      obsAnnotations,
+      varData
+    }
   };
 }
 
+function clipDataframe(
+  df,
+  lowerQuantile,
+  upperQuantile,
+  quantileF,
+  clipPredicate = () => true,
+  value = Number.NaN
+) {
+  /*
+  For all columns in the dataframe, clip all values above or below specified 
+  quantiles to `value` if clipPredicate returns True for that column (if it
+  returns false, skip the column entirely).
+
+  Returns a clipped copy - does not mutate original.
+
+  clipPredicate must have signature: (dataframe, colIndex, colLabel) => boolean
+  True signifies that the column should be clipped; false indicates that the
+  column should be left intact/unchanged.
+
+  quantileF must have signature:  (label, qval) => number
+  */
+  if (lowerQuantile < 0) lowerQuantile = 0;
+  if (upperQuantile > 1) upperQuantile = 1;
+  if (lowerQuantile === 0 && upperQuantile === 1) return df;
+
+  const keys = df.colIndex.keys();
+  return df.mapColumns((col, colIdx) => {
+    const colLabel = keys[colIdx];
+    if (!clipPredicate(df, colIdx, colLabel)) return col;
+
+    const colMin = quantileF(colLabel, lowerQuantile);
+    const colMax = quantileF(colLabel, upperQuantile);
+    const newCol = clip(col.slice(), colMin, colMax, value);
+    return newCol;
+  });
+}
+
+/*
+Create World with contents eq entire universe.   Commonly used to initialize World.
+If clipQuantiles
+*/
 export function createWorldFromEntireUniverse(universe) {
   const world = templateWorld();
 
-  /*
-  public interface follows
-  */
-
   /* Schema related */
   world.schema = universe.schema;
   world.nObs = universe.nObs;
   world.nVar = universe.nVar;
+  world.clipQuantiles = { min: 0, max: 1 };
 
-  /* annotation dataframes */
-  world.obsAnnotations = universe.obsAnnotations;
-  world.varAnnotations = universe.varAnnotations;
+  /* dataframes: annotations and layout */
+  world.obsAnnotations = universe.obsAnnotations.clone();
+  world.varAnnotations = universe.varAnnotations.clone();
+  world.obsLayout = universe.obsLayout.clone();
 
-  /* layout and display characteristics dataframe */
-  world.obsLayout = universe.obsLayout;
-
-  /*
-  Var data columns - subset of all
-  */
+  /* Var dataframe - contains a subset of all var columns */
   world.varData = universe.varData.clone();
 
+  /* save unclipped copies of potentially clipped dataframes */
+  world.unclipped = {
+    obsAnnotations: world.obsAnnotations.clone(),
+    varData: world.varData.clone()
+  };
+
   return world;
 }
 
-export function createWorldFromCurrentSelection(universe, world, crossfilter) {
-  const newWorld = templateWorld();
+/*
+clip dataframes based on quantiles.
 
-  /* these don't change as only OBS are selected in our current implementation */
-  newWorld.nVar = universe.nVar;
-  newWorld.schema = universe.schema;
-  newWorld.varAnnotations = universe.varAnnotations;
+This is an in-place operation on the world object provided as an argument.
+The values in world.unclipped are clipped and assigned to world.obsAnnotations
+and world.varData.
+*/
+function setClippedDataframes(world) {
+  const { schema } = world;
+  const isContinuousObsAnnotation = (df, idx, label) =>
+    deduceDimensionType(schema.annotations.obsByName[label], label) !== "enum";
+  const obsQuantile = (label, q) =>
+    world.unclipped.obsAnnotations.col(label).summarize().percentiles[100 * q];
+  world.obsAnnotations = clipDataframe(
+    world.unclipped.obsAnnotations,
+    world.clipQuantiles.min,
+    world.clipQuantiles.max,
+    obsQuantile,
+    isContinuousObsAnnotation
+  );
 
-  /* now subset/cut obs */
+  const varDataQuantile = (label, q) =>
+    world.unclipped.varData.col(label).summarize().percentiles[100 * q];
+  world.varData = clipDataframe(
+    world.unclipped.varData,
+    world.clipQuantiles.min,
+    world.clipQuantiles.max,
+    varDataQuantile,
+    () => true
+  );
+}
+
+/*
+Subset the current world based upon the current selection, maintaining any existing
+clip.  Returns new world.  Parameters:
+  * unvierse
+  * world - the current world
+  * crossfilter - the selection state
+*/
+export function createWorldBySelection(universe, world, crossfilter) {
+  const newWorld = { ...world, obsLayout: null, unclipped: {}, varData: null };
+
+  /* subset unclipped dataframes based upon current selection */
   const mask = crossfilter.allSelectedMask();
-  newWorld.obsAnnotations = world.obsAnnotations.isubsetMask(mask);
   newWorld.obsLayout = world.obsLayout.isubsetMask(mask);
-  newWorld.nObs = newWorld.obsAnnotations.dims[0];
-
-  /*
-  Var data columns - subset of all
-  */
-  if (world.varData.isEmpty()) {
-    newWorld.varData = world.varData.clone();
+  newWorld.unclipped.obsAnnotations = world.unclipped.obsAnnotations.isubsetMask(
+    mask
+  );
+  if (world.unclipped.varData.isEmpty()) {
+    newWorld.unclipped.varData = world.unclipped.varData.clone();
   } else {
-    newWorld.varData = world.varData.isubsetMask(mask);
+    newWorld.unclipped.varData = world.unclipped.varData.isubsetMask(mask);
   }
+  /* subsetting changings dimension size */
+  newWorld.nObs = newWorld.unclipped.obsAnnotations.dims[0];
+
+  /* and now clip */
+  setClippedDataframes(newWorld);
+  return newWorld;
+}
+
+/*
+Change clip quantiles on the current world, returning a new world.
+Parameters:
+  * universe
+  * world - current world
+  * clipQuantiles - new clip
+*/
+export function createWorldWithNewClip(
+  universe,
+  world,
+  crossfilter,
+  clipQuantiles
+) {
+  const newWorld = { ...world, obsAnnotation: null, varData: null };
+  newWorld.clipQuantiles = clipQuantiles;
+  newWorld.obsLayout = world.obsLayout.clone();
+  newWorld.unclipped = {
+    obsAnnotations: world.unclipped.obsAnnotations.clone(),
+    varData: world.unclipped.varData.clone()
+  };
+
+  /* and now clip */
+  setClippedDataframes(newWorld);
   return newWorld;
 }
 
@@ -137,13 +260,17 @@ function deduceDimensionType(attributes, fieldName) {
   return dimensionType;
 }
 
-export function createObsDimensions(crossfilter, world) {
+export function createObsDimensions(crossfilter, world, XYdimNames) {
   /*
   create and return a crossfilter with a dimension for every obs annotation
-  for which we have a supported type, *except* 'name'
+  for which we have a supported type, *except* for the index column, indicated
+  by schema.annotations.obs.index.
   */
   const { schema, obsLayout, obsAnnotations } = world;
-  const annoList = schema.annotations.obs.filter(anno => anno.name !== "name");
+  const indexName = schema.annotations.obs.index;
+  const annoList = schema.annotations.obs.columns.filter(
+    anno => anno.name !== indexName
+  );
   crossfilter = annoList.reduce((xfltr, anno) => {
     const dimType = deduceDimensionType(anno, anno.name);
     const colData = obsAnnotations.col(anno.name).asArray();
@@ -160,13 +287,16 @@ export function createObsDimensions(crossfilter, world) {
   return crossfilter.addDimension(
     layoutDimensionName("XY"),
     "spatial",
-    obsLayout.col("X").asArray(),
-    obsLayout.col("Y").asArray()
+    obsLayout.col(XYdimNames[0]).asArray(),
+    obsLayout.col(XYdimNames[1]).asArray()
   );
 }
 
 export function worldEqUniverse(world, universe) {
-  return world.obsAnnotations === universe.obsAnnotations;
+  return (
+    world.obsAnnotations === universe.obsAnnotations ||
+    world.obsAnnotations.rowIndex === universe.obsAnnotations.rowIndex
+  );
 }
 
 export function getSelectedByIndex(crossfilter) {

client/src/util/statemachine/index.js

@@ -0,0 +1,95 @@
+/*
+Very simple FSM for use in reducer, etc.
+
+To create a state machine:
+  new StateMachine(initialState, transitions, onErrorCallback) -> statemachine
+
+Where:
+  * initialState - a caller-specified value that represents the initial state of
+    the FSM.
+  * transitions - an array of objects, representing FSM transitions (graph edges),
+    having the form:
+      {
+        to: state_name_transitioning_to,
+        from: state_name_transitioning_from,
+        event: value_that_will_cause_transition,
+        action: optional_callback_upon_transition
+      }
+    The transition will be provided to the action callback, so other data
+    may be stored in the transition object for use by the action callback.
+  * onErrorCallback - a callback function called if the FSM receives an event
+    for which it has no defined transition.
+
+
+Interface:
+  * states - property containing the state names. A Set(), contianing the
+    union of to: and from: values.
+  * events - property containing all of the accepted event values.  Set().
+  * graph - a Map of Maps, organized as  graph[eventValue][fromStateValue]
+  * clone() - clone the entire statemachine.
+  * next(eventValue) - drive the FSM to the next state.   If the event
+    matches a transition with a defined action, the action callback is
+    called, and the action return value is returned by next().  If no
+    transition is defined, onErrorCallback is called.
+
+Example:
+
+  const transitions = [
+    { from: "A", to: "B", event: "yo", action: () => 42 }
+  ];
+  const fsm = new StateMachine("A", transitions, () => { throw new Error("oops") });
+  fsm.next("yo");   // returns 42
+
+
+*/
+export default class StateMachine {
+  constructor(initState, transitions, onError) {
+    this.onError = onError || (() => undefined);
+    this.state = initState;
+
+    // all states
+    this.states = new Set(
+      transitions.reduce((names, tsn) => {
+        names.push(tsn.from);
+        names.push(tsn.to);
+        return names;
+      }, [])
+    );
+
+    // all transition names (aka events)
+    this.events = new Set(transitions.map(tsn => tsn.event));
+
+    // the transition graph.
+    // graph[event][from] -> transition
+    this.graph = transitions.reduce((graph, tsn) => {
+      const { event, from } = tsn;
+      if (!graph.has(event)) graph.set(event, new Map());
+      const tsnMap = graph.get(event);
+      tsnMap.set(from, tsn);
+      return graph;
+    }, new Map());
+  }
+
+  clone(initState) {
+    const fsm = new StateMachine(initState, []);
+    fsm.onError = this.onError;
+    fsm.states = this.states;
+    fsm.events = this.events;
+    fsm.graph = this.graph;
+    return fsm;
+  }
+
+  next(event, data) {
+    const { graph, state } = this;
+    const tsnMap = graph.get(event);
+    if (!tsnMap) return this.onError(this, event, state, undefined);
+
+    const transition = tsnMap.get(state);
+    if (!transition) return this.onError(this, event, state, undefined);
+
+    this.state = transition.to;
+    return transition.action
+      ? transition.action(this, transition, data)
+      : undefined;
+  }
+}

client/src/util/typeHelpers.js

@@ -0,0 +1,29 @@
+/*
+Various type and schema related helper functions.
+*/
+
+/*
+Utility function to test for a typed array
+*/
+export function isTypedArray(x) {
+	return (
+		ArrayBuffer.isView(x) &&
+		Object.prototype.toString.call(x) !== "[object DataView]"
+	);
+}
+
+/*
+Test for float typed array, ie, Float32TypedArray or Float64TypedArray
+*/
+export function isFpTypedArray(x) {
+	let constructor;
+	const isFloatArray =
+		x &&
+		({ constructor } = x) &&
+		(constructor === Float32Array || constructor === Float64Array);
+	return isFloatArray;
+}
+
+export function isArrayOrTypedArray(x) {
+	return Array.isArray(x) || isTypedArray(x);
+}

client/src/util/typedCrossfilter/crossfilter.js

@@ -2,13 +2,13 @@ import { polygonContains } from "d3";
 
 import PositiveIntervals from "./positiveIntervals";
 import BitArray from "./bitArray";
-import { sort } from "./sort";
 import {
-  makeSortIndex,
+  sortArray,
   lowerBound,
   lowerBoundIndirect,
   upperBoundIndirect
-} from "./util";
+} from "./sort";
+import { makeSortIndex } from "./util";
 
 class NotImplementedError extends Error {
   constructor(...params) {
@@ -61,6 +61,10 @@ export default class ImmutableTypedCrossfilter {
     return Object.keys(this.dimensions);
   }
 
+  hasDimension(name) {
+    return !!this.dimensions[name];
+  }
+
   addDimension(name, type, ...rest) {
     /*
     Add a new dimension to this crossfilter, of type DimensionType.
@@ -284,15 +288,15 @@ class _ImmutableBaseDimension {
     this.name = name;
   }
 
-  /* eslint-disable class-methods-use-this */
   select(spec) {
     const { mode } = spec;
     if (mode === undefined) {
       throw new Error("select spec does not contain 'mode'");
     }
-    throw new Error(`select mode ${mode} not implemented`);
+    throw new Error(
+      `select mode ${mode} not implemented by dimension ${this.name}`
+    );
   }
-  /* eslint-enable class-methods-use-this */
 }
 
 class ImmutableScalarDimension extends _ImmutableBaseDimension {
@@ -414,7 +418,7 @@ class ImmutableEnumDimension extends ImmutableScalarDimension {
     for (let i = 0; i < len; i += 1) {
       s.add(mapf(i, data));
     }
-    const enumIndex = sort(Array.from(s));
+    const enumIndex = sortArray(Array.from(s));
     this.enumIndex = enumIndex;
 
     // create dimension value array
@@ -477,16 +481,16 @@ class ImmutableSpatialDimension extends _ImmutableBaseDimension {
 
   selectWithinRect(spec) {
     /*
-      { mode: "within-rect", x0: 1, y0: 0, x1: 3, y1: 9 }
+      { mode: "within-rect", minX: 1, minY: 0, maxX: 3, maxY: 9 }
     */
-    const { x0, y0, x1, y1 } = spec;
+    const { minX, minY, maxX, maxY } = spec;
     const { X, Y } = this;
     const ranges = [];
     let start = -1;
     for (let i = 0, l = X.length; i < l; i += 1) {
       const x = X[i];
       const y = Y[i];
-      const inside = x0 <= x && x < x1 && y0 <= y && y < y1;
+      const inside = minX <= x && x < maxX && minY <= y && y < maxY;
       if (inside && start === -1) start = i;
       if (!inside && start !== -1) {
         ranges.push([start, i]);

client/src/util/typedCrossfilter/sort.js

@@ -1,5 +1,29 @@
-const SmallArray = 32;
+import { isTypedArray, isFpTypedArray } from "../typeHelpers";
 
+/* eslint no-bitwise: "off" */
+
+/*
+ ** fast sort and search, with separate code paths for floats (NaN ordering),
+ ** indirect and direct search/sort.
+ */
+
+/*
+Comparators for float sort.   -Infinity < finite < Infinity < NaN
+*/
+function lt(a, b) {
+  if (Number.isNaN(b)) return !Number.isNaN(a);
+  return a < b;
+}
+
+function gt(a, b) {
+  if (Number.isNaN(a)) return !Number.isNaN(b);
+  return a > b;
+}
+
+/*
+insertion sort, used for small arrays (controlled by SMALL_ARRAY constant)
+*/
+const SMALL_ARRAY = 32;
 function insertionsort(a, lo, hi) {
   for (let i = lo + 1; i < hi + 1; i += 1) {
     const x = a[i];
@@ -12,6 +36,18 @@ function insertionsort(a, lo, hi) {
   return a;
 }
 
+function insertionsortFloats(a, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    let j;
+    for (j = i; j > lo && gt(a[j - 1], x); j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
 function insertionsortIndirect(a, s, lo, hi) {
   for (let i = lo + 1; i < hi + 1; i += 1) {
     const x = a[i];
@@ -25,8 +61,24 @@ function insertionsortIndirect(a, s, lo, hi) {
   return a;
 }
 
+function insertionsortFloatsIndirect(a, s, lo, hi) {
+  for (let i = lo + 1; i < hi + 1; i += 1) {
+    const x = a[i];
+    const t = s[x];
+    let j;
+    for (j = i; j > lo && gt(s[a[j - 1]], t); j -= 1) {
+      a[j] = a[j - 1];
+    }
+    a[j] = x;
+  }
+  return a;
+}
+
+/*
+Quicksort - used for larger arrays
+*/
 function quicksort(a, lo, hi) {
-  if (hi - lo < SmallArray) {
+  if (hi - lo < SMALL_ARRAY) {
     return insertionsort(a, lo, hi);
   }
   if (lo < hi) {
@@ -55,8 +107,38 @@ function quicksort(a, lo, hi) {
   return a;
 }
 
+function quicksortFloats(a, lo, hi) {
+  if (hi - lo < SMALL_ARRAY) {
+    return insertionsortFloats(a, lo, hi);
+  }
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (lt(a[i], p));
+      do {
+        j -= 1;
+      } while (gt(a[j], p));
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksortFloats(a, lo, j);
+    quicksortFloats(a, j + 1, hi);
+  }
+  return a;
+}
+
 function quicksortIndirect(a, s, lo, hi) {
-  if (hi - lo < SmallArray) {
+  if (hi - lo < SMALL_ARRAY) {
     return insertionsortIndirect(a, s, lo, hi);
   }
   if (lo < hi) {
@@ -86,15 +168,248 @@ function quicksortIndirect(a, s, lo, hi) {
   return a;
 }
 
-// Convenience wrappers
-export function sort(arr, comparator = undefined) {
-  if (comparator !== undefined) {
-    // XXX for now
-    return arr.sort(arr, comparator);
+function quicksortFloatsIndirect(a, s, lo, hi) {
+  if (hi - lo < SMALL_ARRAY) {
+    return insertionsortFloatsIndirect(a, s, lo, hi);
   }
-  return quicksort(arr, 0, arr.length - 1);
+  if (lo < hi) {
+    // partition
+    const mid = Math.floor((lo + hi) / 2);
+    const p = a[mid];
+    const t = s[p];
+    let i = lo - 1;
+    let j = hi + 1;
+    while (i < j) {
+      do {
+        i += 1;
+      } while (lt(s[a[i]], t));
+      do {
+        j -= 1;
+      } while (gt(s[a[j]], t));
+      if (i < j) {
+        const tmp = a[i];
+        a[i] = a[j];
+        a[j] = tmp;
+      }
+    }
+    // sort
+    quicksortFloatsIndirect(a, s, lo, j);
+    quicksortFloatsIndirect(a, s, j + 1, hi);
+  }
+  return a;
+}
+
+/*
+Convenience wrappers, handling optimization paths and default
+handlers for NaN comparisons.  Sorts in place.
+*/
+export function sortArray(arr) {
+  if (Array.isArray(arr)) {
+    return quicksort(arr, 0, arr.length - 1);
+  }
+  if (isTypedArray(arr)) {
+    if (isFpTypedArray(arr)) {
+      return quicksortFloats(arr, 0, arr.length - 1);
+    }
+    return quicksort(arr, 0, arr.length - 1);
+  }
+  /* else unsupported */
+  throw new Error("sortArray received unsupported object type");
 }
 
 export function sortIndex(index, source) {
+  if (isFpTypedArray(source))
+    return quicksortFloatsIndirect(index, source, 0, index.length - 1);
   return quicksortIndirect(index, source, 0, index.length - 1);
 }
+
+// Search for `value` in the sorted array `arr`, in the range [first, last).
+// Return the first (left most) index where arr[index] >= value.
+//
+// In other words, return array index I where:
+//  arr[i] < value for all tarr[lo:I]
+//  arr[i] >= value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: lower_bound()
+//    Python: bisect.bisect_left()
+//
+function lowerBoundNonFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+// lowerBound, but with NaN handling
+//
+// If the underlying array is a Float32Array or Float64Array, will enforce
+// the ordering -Infinity < finite < Infinity < NaN.
+//
+function lowerBoundFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (lt(valueArray[middle], value)) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+export function lowerBound(valueArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return lowerBoundFloat(valueArray, value, first, last);
+  }
+  return lowerBoundNonFloat(valueArray, value, first, last);
+}
+
+// Inlined performance optimization - used to indirect through a sort map.
+//
+function lowerBoundNonFloatIndirect(
+  valueArray,
+  indexArray,
+  value,
+  first,
+  last
+) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] < value) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+function lowerBoundFloatIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (lt(valueArray[indexArray[middle]], value)) {
+      lfirst = middle + 1;
+    } else {
+      llast = middle;
+    }
+  }
+  return lfirst;
+}
+
+export function lowerBoundIndirect(valueArray, indexArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return lowerBoundFloatIndirect(valueArray, indexArray, value, first, last);
+  }
+  return lowerBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
+}
+
+// Search for `value in the sorted array `arr`, in the range [first, last).
+// Return the first value where arr[index] > value.
+//
+// In other words, return array index I, where:
+//  arr[i] <= value for all tarr[lo:I]
+//  arr[i] > value for all tarr[I:last]
+//
+// The same semantics/behavior as:
+//    C++: upper_bound()
+//    Python: bisect.bisect_right()
+//
+function upperBoundNonFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[middle] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+function upperBoundFloat(valueArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (gt(valueArray[middle], value)) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+export function upperBound(valueArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return upperBoundFloat(valueArray, value, first, last);
+  }
+  return upperBoundNonFloat(valueArray, value, first, last);
+}
+
+// Inline performance optimization
+//
+function upperBoundNonFloatIndirect(
+  valueArray,
+  indexArray,
+  value,
+  first,
+  last
+) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (valueArray[indexArray[middle]] > value) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+function upperBoundFloatIndirect(valueArray, indexArray, value, first, last) {
+  let lfirst = first;
+  let llast = last;
+  // this is just a binary search
+  while (lfirst < llast) {
+    const middle = (lfirst + llast) >>> 1;
+    if (gt(valueArray[indexArray[middle]], value)) {
+      llast = middle;
+    } else {
+      lfirst = middle + 1;
+    }
+  }
+  return lfirst;
+}
+
+export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
+  if (isFpTypedArray(valueArray)) {
+    return upperBoundFloatIndirect(valueArray, indexArray, value, first, last);
+  }
+  return upperBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
+}

client/src/util/typedCrossfilter/util.js

@@ -1,23 +1,12 @@
 // jshint esversion: 6
-/* eslint no-bitwise: "off" */
 
 import { sortIndex } from "./sort";
+import { rangeFill as fillRange } from "../range";
 
 /*
     Utility functions, private to this module.
 */
 
-// fill an array or typedarray with a sequential range of numbers,
-// starting with `start`
-//
-export function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, len = larr.length; i < len; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 // slice out of one array into another, using an index array
 //
 export function sliceByIndex(src, index) {
@@ -36,93 +25,3 @@ export function makeSortIndex(src) {
   sortIndex(index, src);
   return index;
 }
-
-// Search for `value` in the sorted array `arr`, in the range [first, last).
-// Return the first (left most) index where arr[index] >= value.
-//
-// In other words, return array index I where:
-//  arr[i] < value for all tarr[lo:I]
-//  arr[i] >= value for all tarr[I:last]
-//
-// The same semantics/behavior as:
-//    C++: lower_bound()
-//    Python: bisect.bisect_left()
-//
-// XXX: it is likely that there would be minimal performance hit from creating
-// a factory version of lowerBound that takes an accessor (rather than having
-// a special-cased version for lining the indirection).
-//
-export function lowerBound(valueArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[middle] < value) {
-      lfirst = middle + 1;
-    } else {
-      llast = middle;
-    }
-  }
-  return lfirst;
-}
-
-// Inlined performance optimization - used to indirect through a sort map.
-//
-export function lowerBoundIndirect(valueArray, indexArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[indexArray[middle]] < value) {
-      lfirst = middle + 1;
-    } else {
-      llast = middle;
-    }
-  }
-  return lfirst;
-}
-
-// Search for `value in the sorted array `arr`, in the range [first, last).
-// Return the first value where arr[index] > value.
-//
-// In other words, return array index I, where:
-//  arr[i] <= value for all tarr[lo:I]
-//  arr[i] > value for all tarr[I:last]
-//
-// The same semantics/behavior as:
-//    C++: upper_bound()
-//    Python: bisect.bisect_right()
-//
-export function upperBound(valueArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[middle] > value) {
-      llast = middle;
-    } else {
-      lfirst = middle + 1;
-    }
-  }
-  return lfirst;
-}
-
-// Inline performance optimization
-//
-export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
-  let lfirst = first;
-  let llast = last;
-  // this is just a binary search
-  while (lfirst < llast) {
-    const middle = (lfirst + llast) >>> 1;
-    if (valueArray[indexArray[middle]] > value) {
-      llast = middle;
-    } else {
-      lfirst = middle + 1;
-    }
-  }
-  return lfirst;
-}

dev_docs/developer_guidelines.md

@@ -0,0 +1,91 @@
+# Developer guidelines
+
+### Requirements
+- npm
+- Python 3.6+
+- Chrome
+
+[See dev section of README](../README.md)
+
+**All instructions are expected to be run from the top level cellxgene directory unless otherwise specified.**
+
+## Server dev
+### Install
+* Build the client and put static files in place: `make build-for-server-dev`
+* Install from local files: `make install-dev`
+
+### Launch
+* `cellxgene launch [options] <datafile>`
+
+### Reloading
+If you install cellxgene using `make install-dev` the server will be restarted every time you make changes on the server code. If changes affects the client, the browser must be reloaded.
+
+### Linter
+We use `flake8` to lint code. Travis CI runs `flake8 server`.
+
+### Test
+1. Install development requirements `pip install -r server/requirements-dev.txt`
+2. Run tests `pytest server/test`
+
+### Tips
+* Install in a virtualenv
+* May need to rebuild/reinstall when you make client changes
+
+## Client dev
+### Install
+1. Install prereqs for client: `npm install --prefix client/ client`
+2. Install cellxgene server: `pip install -e .` Caveat: this will not build the production client package - you must use the [server install](#install) instructions above to serve web assets.
+
+### Launch
+To launch with hot reloading you need to launch the server and the client separately. Node's hot reloading starts the client on its own node server and auto-refreshes when changes are made.
+1. Launch server (the client relies on the REST API being available): `cellxgene launch [options] <datafile>`
+2. Launch client: in `client/` directory run `npm run start`
+3. Client will be served on localhost:3000
+
+### Build
+To build only the client: `make build-client`
+
+### Linter
+We use `eslint` to lint the code and `prettier` as our code formatter.
+
+### Test
+In `client/` directory run `npm run unit-test`
+
+### Tips
+* You can also install/launch the server side code from npm scrips (requires python3.6 with virtualenv) in `client/` directory run `npm run backend-dev`
+
+## Running tests
+Client and server tests run on Travis CI for every push, PR, and commit to master on github. End to end tests run nightly on master only. 
+
+### Server unit tests
+Install development requirements `pip install -r server/requirements-dev.txt`
+Run tests `pytest server/test`
+
+### Client unit tests
+In `client/` directory run `npm run unit-test`
+
+### End to end tests
+
+End to end tests use two env variables:
+* `JEST_ENV` - environment to run end to end tests. Default `dev`
+    * `prod` - run headless with no slowdown, chromium will not open.
+    * `dev` - opens chromimum, runs tests with minimal slowdown, close on exit.
+    * `debug` - opens chromium, runs tests with 100ms slowdown, dev tools open, chrome stays open on exit.
+* `JEST_CXG_PORT` - port that end to end tests are being run on. Default `3000` (client hosted port).
+
+On CI the end to end tests are run with `JEST_ENV` set to `prod` using the `smoke-test` npm script
+
+To run end to end tests as they will be run on CI
+1. cellxgene should be built and installed as [specified in server dev](#install)
+2. `export JEST_ENV='prod'`
+3. `export JEST_CXG_PORT='5000'`
+4. Run `npm run --prefix client/ smoke-test`
+
+Run end to end tests interactively during development
+1. cellxgene should be installed as [specified in client dev](#install-1)
+2. Follow [launch](#launch-1) instructions for client dev with dataset `example-dataset/pbmc3k`
+3. Run `npm run --prefix client/ e2e`
+4. To debug a failing test `export JEST_ENV='debug'` and re-run.
+
+
+

dev_docs/release_process.md

@@ -43,24 +43,53 @@ Follow these steps to create a release.
 8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
 and that you have write access to the cellxgene pypi package):
     - Build the distribution and upload to test pypi `make release-stage-2`
-    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Test the test installation in a fresh virtual environment using `make install-release-test`
     - Upload the package to real pypi using `make release-stage-final`
-    - [optional] Test the installation in a fresh virtual environment using
+    - Test the installation in a fresh virtual environment using
       `pip install cellxgene`
-    - **Troubleshooting**:
-        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
-        if you accidentally burned a release number you want to use on prod, you have a couple options. 
-            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
-             then upload to prod `make release-stage-final`. 
-            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
-            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
-            PART=patch until you get to an unburned version.
-        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
-        go to the cellxgene package, click manage, then in the options drop down, click delete and 
-        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
-        and not a major regression, you can just release a patch. 
-    
 
 The optional steps are for testing purposes, and are recommended
 for publishing any major releases, and any releases that significantly
 change the packaging (e.g. new bundled files, new dependencies, etc.)
+
+## Troubleshooting
+### Fails to upload to test.pypi
+ 
+ _PyPi doesn't allow you to reupload a release with the same version number_ 
+If you accidentally burned a release number you want to use on prod, you have a few options:
+1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+ then upload to prod `make release-stage-final`. 
+2) OPTION 2: (DANGER) release directly to prod: `make release-directly-to-prod`.  
+3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+PART=patch until you get to an unburned version.
+    
+### The release doesn't install or fails your tests when you install it
+ 
+Delete it from pypi - Go to pypi.org -> sign in -> go to the cellxgene package -> click manage -> then in the options drop down click delete -> follow the instructions. You will not be able to use that release number again. If it is a minor bug and not a major regression, you can just release a patch. 
+### If you need to run stage final on a different computer than stage 2
+ If you run stage final without running stage 2 first, the dist will not have been build on the computer running stage final. The solution is to run `make release-directly-to-prod`. This both builds the distribution files and then releases directly to prod pypi.org. 
+
+## Stage Details
+### Stage 1 - `make release-stage-1`
+    1. Pip installs requirements-dev
+    2. Bumps version by [PART]
+    3. Deletes build directory, client/build, dist and cellxgene.egg-info
+    4. Creates the package-lock.json
+
+### Stage 2 - `make release-stage-2`
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to test.pypi.org
+
+### Stage final - `make release-stage-final`
+    ** Does not build distribution **
+    1. Uploads to pypi.org
+    
+### (DANGER) Release directly to prod `make release-directly-to-prod`
+    ** builds distribution and uploads directly to prod **
+    Only use this if you are directed to by the troubleshooting guide
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to pypi.org

docs/_config.yml

@@ -3,6 +3,8 @@ show_downloads: false
 baseurl: /cellxgene
 
 nav:
+  - title: Getting Started
+    url: getting-started.html
   - title: Data
     url: data.html
   - title: FAQ

docs/data.md

@@ -1,29 +1,94 @@
----
-layout: default
-title: data
-description: Data
----
+---
+layout: default
+title: data
+description: Data
+---
+
+# Using `cellxgene prepare`
+
+#### What is `cellxgene prepare`?
+
+`prepare` offers an easy command line interface (CLI) to preliminarily wrangle your data into the required format for previewing it with `cellxgene`.
+
+#### What is `cellxgene prepare` _not_?
+
+`cellxgene prepare` is not meant as a way to formally process or analyze your data. It's simply a utility for quickly wrangling your data into cellxgene-compatible format and computing a "vanilla" embedding so you can try out `cellxgene` and get a general sense of a dataset.
+
+#### What input formats does it accept?
+
+Currently, we accept `h5ad` and `loom` files, as well as `10x` directories, and are hoping to accept more formats in the future.
+
+While we'd like to support quick conversion from seurat and bioconductor, these packages don't currently output a python-parseable intermediate file type. In the meantime, you might check out the [converters](https://satijalab.org/seurat/v3.0/conversion_vignette.html) that are under early development.
+
+#### What can `cellxgene prepare` do?
+
+`prepare` uses scanpy to:
+
+- Handle simple data normalization (from a [recipe](https://www.pydoc.io/pypi/scanpy-0.2.3/autoapi/preprocessing/recipes/index.html))
+- Do basic preprocessing to run PCA and compute the neighbor graph
+- Infer clusters
+- Reduce dimensionality to generate embeddings.  
+  You can control which steps to run and their methods (when applicable), via the CLI. The CLI also includes options for computing QC metrics, enforcing matrix sparcity, specifying index names, and plotting output.
+
+**To see a full list of available arguments and options, run `cellxgene prepare --help`.**
+
+#### How do I use `cellxgene prepare`?
+
+As a quick example, let's construct a command to use `prepare` to take a raw expression matrix and generate a processed `h5ad` ready to visualize with cellxgene.
+
+We'll start off using the raw data from the pbmc3k dataset. This dataset is described [here](https://icb-scanpy.readthedocs-hosted.com/en/stable/api/scanpy.datasets.pbmc3k.html), and is available as part of the scanpy API. For this example, we'll assume this raw data is stored in a file called `pbmc3k-raw.h5ad`.
+
+Our `prepare` compose our command looks like this:  
+<img src="prepare-cmd-example.jpg" width="700" />
+
+Let's look at what `prepare` is doing to our data, and how each step relates to the command above. You can see a walkthrough of what's going on under the hood for this example in [this notebook](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-prepare-example.ipynb).
+
+**1 - Compute quality control metrics and store this in our `AnnData` object for later inspection (A)**  
+**2 - Normalize the expression matrix using a basic preprocessing recipe (B)**  
+**3 - Do some preprocessing to run PCA and compute the neighbor graph (auto)**  
+**4 - Infer clusters with the Louvain algorithm and store these labels to visualize later (auto)**  
+**5 - Compute and store umap and tsne embeddings (C)**  
+**6 - Write results to file (D)**
+
+# Example datasets to use with cellxgene
+
 
-
-# data vignette: how to use cellxgene prepare
-
-#### coming soon!
-
-# example datasets to use with cellxgene
-
-### Examination of single cells from primary human pancreas tissue
-cells: 2,544  
-tissue(s): pancreas  
-data: [Human Cell Atlas Data Portal](https://prod.data.humancellatlas.org/explore/projects?filter=%5B%7B%22facetName%22%3A%22organ%22%2C%22terms%22%3A%5B%22pancreas%22%5D%7D%2C%7B%22facetName%22%3A%22project%22%2C%22terms%22%3A%5B%22Single+cell+transcriptome+analysis+of+human+pancreas%22%5D%7D%5D)  
-paper: [Enge, Martin, et al.](https://www.cell.com/cell/fulltext/S0092-8674(17)31053-X?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS009286741731053X%3Fshowall%3Dtrue)
-
-### Tabula Muris
-cells: 53,800  
-tissue(s): muscle, pancreas, bone, large intestine, heart, brain, fat, mammary gland, tongue , diaphragm, bladder, spleen, thymus, lung , skin, liver, trachea, kidney  
-data: [Tabula Muris Data](https://github.com/czbiohub/tabula-muris-vignettes/tree/master/data)  
-paper: [Tabula Muris Consortium](https://www.nature.com/articles/s41586-018-0590-4)  
-
-### Transcriptional profiling of 1.3 million brain cells
-cells: 1,330,000  
-tissue(s): brain  
-data: [10x Genomics](https://community.10xgenomics.com/t5/10x-Blog/Our-1-3-million-single-cell-dataset-is-ready-to-download/ba-p/276)  
+**To download and use these datasets, run:**  
+`curl -O [URL]`  
+`unzip [filename.zip]`  
+`cellxgene launch [filename.h5ad] --open`
+
+### Peripheral blood mononuclear cells
+
+Healthy human PBMCs (10X).
+
+- Source: [10X genomics](https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k)
+- Cells: 2,638
+- File size: 19MB
+- [Raw data](http://cf.10xgenomics.com/samples/cell-exp/1.1.0/pbmc3k/pbmc3k_filtered_gene_bc_matrices.tar.gz)
+- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-processing.ipynb)
+- Download: `curl -O https://cellxgene-example-data.czi.technology/pbmc3k.h5ad.zip`
+
+### Tabula muris
+
+20 organs and tissues from healthy mice (Smart-Seq2).  
+Rich metadata and annotations.
+
+- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/237446v2)
+- Cells: 45,423
+- File size: 174MB
+- [Raw data](https://figshare.com/projects/Tabula_Muris_Transcriptomic_characterization_of_20_organs_and_tissues_from_Mus_musculus_at_single_cell_resolution/27733)
+- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/tabula-muris-processing.ipynb)
+- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris.h5ad.zip`
+
+### Tabula muris senis
+
+22 organs and tissues from healthy mice at ages 3mo, 18mo, 21mo, and 24mo (Smart-Seq2).  
+Rich metadata and annotations.
+
+- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/661728v1)
+- Cells: 81,478
+- File size: 3.9GB
+- Raw data [geo link coming soon!]
+- [Processing](https://www.biorxiv.org/content/10.1101/661728v1)
+- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris-senis.h5ad.zip`

docs/faq.md

@@ -38,7 +38,15 @@ Currently this is not supported directly, but you should be able to do this your
 
 - `.obs` and `.var` annotations are use to extract metadata for filtering
 - `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
-- `.obsm` is used for layout
+- `.obsm` is used for layout. If an embedding has more than two components, the first two will be used for visualization.
+
+#### I have a BIG dataset - how can I make cellxgene run as fast as possible?
+
+If your dataset requires gigabytes of disk space, you may need to select an appropriate storage format in order to effectively utilize `cellxgene`. Tips and tricks:
+
+- `cellxgene` is optimized for columnar data access. For large datasets, format the expression matrix (`.X`) as either a [SciPy CSC sparse matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) or a dense Numpy array (whichever creates a smaller `h5ad` file). If you are using `cellxgene prepare`, include the `--sparse` flag to ensure `.X` is formatted as a CSC sparse matrix (by default, `.X` will be a dense matrix).
+- `cellxgene` start time is directly proportional to `h5ad` file size and the speed of your file system. Expect that large (eg, million cell) datasets will take minutes to load, even on relatively fast computers with a high performance local hard drive. Once loaded, exploring metadata should still be quick.
+- If your dataset size exceeds the size of memory (RAM) on the host computer, differential expression calculations will be extremely slow (or fail, if you run out of virtual memory).
 
 # Algorithms
 
@@ -48,12 +56,12 @@ We use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) impl
 
 # Problems, errors, & bugs
 
-#### How do I create a Python 3.6 environment for _cellxgene_?
+#### How do I create a Python environment for _cellxgene_?
 
 If you use conda and want to create a [conda environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for _cellxgene_ you can use the following commands
 
 ```
-conda create --yes -n cellxgene python=3.6
+conda create --yes -n cellxgene python=3.7
 conda activate cellxgene
 pip install cellxgene
 ```
@@ -62,7 +70,7 @@ Or you can create a virtual environment by using
 
 ```
 ENV_NAME=cellxgene
-python3.6 -m venv ${ENV_NAME}
+python3.7 -m venv ${ENV_NAME}
 source ${ENV_NAME}/bin/activate
 pip install cellxgene
 ```