bumpversion to 0.7.0 (#623)

* initial dataframe commit

* initial dataframe port of core app

* rename variables for clarity

* remove unused import

* comment out unused code

* fix array handling bug in crossfilter dimension creation

* allow creation of empty dataframes

* handle non-existent columns

* handle non-existent columns

* revise tests for new dataframe

* comments for clarity

* comments for clarity

* generate bulk add placeholder with real gene names

* fix bug in gene name adding

* more dataframe unit tests

* fix bug - subset from current world, not universe

* put cut and pasted code into a single function

* improve caching of crossfilter

* remove cascading update bug from graph

* more performance work

* improve state handling for scatterplot

* performance optimization of critical path

* add column summarization

* dataframe utils

* add callOnceLazy

* fix tests

* minor updates found during review

* fix misspelling

* remove RESTv02 from function names

* comment cleanup

* cut/icut col parameter defaults to null

* break up large test

* improve tests and comments on dataframe at/has functions

* add Dataframe withCol/dropCol

* expression varData now stored in a dataframe

* dead code cleanup

* use dataframe.summarize()

* test cases for Dataframe.col.summarize

* update test cases for new dataframe summarize

* improve naming

* use new hasCol API

* add comments

* add more Dataframe.withCol tests

* add ability to specify row index in cut operation

* retire subsetVarData function

* correctly handle expression subsetting

* lint and improve comments

* rename cut to subset

* create helper file for controls reducer

.bumpversion.cfg

@@ -1,6 +1,5 @@
 [bumpversion]
-current_version = 0.0.2
-
+current_version = 0.7.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"

.dockerignore

@@ -0,0 +1,5 @@
+bin
+client
+dist
+docs
+server

.gitignore

@@ -35,3 +35,7 @@ npm-debug.log
 __pycache__
 *.DS_Store*
 data
+
+# Jekyll
+docs/_site/
+docs/Gemfile.lock

.travis.yml

@@ -8,13 +8,22 @@ cache:
 install:
   - set -eo pipefail
   - pip install flake8
-  - ./bin/build-client
-  - pip install -e .
+  - make build
+  - make install
   - pip install -r server/requirements-dev.txt
-script:
-  - set -eo pipefail
-  - flake8 server/app/
-  - flake8 server/cli/
-  - npm run --prefix client/ build
-  - npm run --prefix client/ test
-  - pytest -s server/test
+  - docker build .
+
+jobs:
+  include:
+  - name: "Branch Tests"
+    script:
+      - set -eo pipefail
+      - flake8 server
+      - black --check
+      - npm run --prefix client/ build
+      - npm run --prefix client/ unit-test
+      - pytest -s server/test
+  - name: "Smoke Tests"
+    if: branch = master AND type = cron
+    script:
+      - npm run --prefix client/ smoke-test

Dockerfile

@@ -0,0 +1,10 @@
+FROM ubuntu:bionic
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+RUN apt-get update && \
+    apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev && \
+    pip3 install cellxgene
+
+ENTRYPOINT ["cellxgene"]

README.md

@@ -1,111 +1,288 @@
 # cellxgene
 
-### An interactive, performant explorer for single cell transcriptomics data.
+> an interactive explorer for single-cell transcriptomics data
 
-<img align="right" width="350" height="218" src="./example-dataset/cellxgene-demo.gif" pad="50px">
-cellxgene is an open-source experiment in how to bring powerful tools from modern web development to visualize and explore large single-cell transcriptomics datasets.
-Started in the context of the Human Cell Atlas Consortium, cellxgene hopes to both enable scientists to explore their data and to equip developers with scalable, reusable patterns and frameworks for visualizing large scientific datasets.
+`cellxgene` is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
 
-## Features
+<img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
 
-- **Visualization at scale:** built with [WebGL](https://www.khronos.org/webgl/), [React](https://reactjs.org/) & [Redux](https://redux.js.org/) to handle visualization of at least 1 million cells.
+## getting started
 
-- **Interactive exploration:** select, cross-filter, and compare subsets of your data with performant indexing and data handling.
+You'll need **python 3.6** and **Google Chrome**. (_Warning_: Python 3.7 is **not** supported at this time)
+The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).
 
-- **Flexible API:** the cellxgene client-server model is designed to support a range of existing analysis packages for backend computational tasks (eg scanpy), integrated with client-side visualization via a [REST API](https://restfulapi.net/).
+To install run
 
-## Getting Started
+```
+pip install cellxgene
+```
 
-**Requirements**
+To start exploring a dataset call
+
+```
+cellxgene launch dataset.h5ad --open
+```
+
+If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
+
+```
+cellxgene launch pbmc3k.h5ad --open
+```
+
+You should see your web browser open with the following
+
+<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+
+**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+
+There are several options available, such as:
+
+- `--layout` to specify the layout as `tsne` or `umap`
+- `--title` to show a title on the explorer
+- `--open` to automatically open the web browser after launching (OS X only)
+
+To see all options call
+
+```
+cellxgene launch --help
+```
+
+There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+
+## data formatting
+
+### assumptions
+
+The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
+
+- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
+- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
+- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_tsne` for the `tsne` layout or `X_umap` for the `umap` layout)
+- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
+
+### prepare
+
+The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
+
+To prepare from an existing `.h5ad` file use
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
+```
+
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
+
+Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
+```
+
+To see all options call
+
+```
+cellxgene prepare --help
+```
+
+**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
+
+```
+pip install cellxgene[louvain]
+```
+
+If the aforementioned optional package installation fails, you can also install these packages directly:
+
+```
+pip install python-igraph louvain>=0.6
+```
+
+## conda and virtual environments
+
+If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.6
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3.6 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+## docker
+
+We have included a dockerfile to conveniently run cellxgene from docker.
+
+1. Build the image `docker build . -t cellxgene`
+2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
+   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
+
+## FAQ
+
+<details>
+
+<summary> questions about data formatting </summary>
+  
+<hr>
+  
+> Someone sent me a directory of `10X-Genomics` data with a `mtx` file and I've never used `scanpy`, can I use `cellxgene`?
+
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+```
+
+Depending on the size of the dataset, this may take some time. Once it's done, call
+
+```
+cellxgene launch data-processed.h5ad --layout=umap --open
+```
+
+And your web browser should open with an interactive view of your data.
+
+<hr>
+
+> In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
+
+Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using
+
+```
+pip install cellxgene[louvain]
+```
+
+<hr>
+
+> I ran `prepare` and I'm getting results that look unexpected
+
+You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
+```
+
+It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
+
+<hr>
+
+> I have extra metadata that I want to add to my dataset
+
+Currently this is not supported directly, but you should be able to do this manually using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
+
+<hr>
+
+> What part of the anndata objects does cellxgene pull in for visualization?
+
+- `.obs` and `.var` annotations are use to extract metadata for filtering
+- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
+- `.obsm` is used for layout
+
+</details>
+
+<details>
+  
+<summary> questions about installing and building </summary>
+
+<hr>
+
+> I tried to `pip install cellxgene` and got a weird error about missing paths to an HDF5 library?  
+
+You probably just need to install HDF5 first. If you're on a mac, you can simply `brew install hdf5` and then try `pip install cellxgene` again.  
+
+> I tried to `pip install cellxgene` and got another weird error I don't understand  
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.  
+
+<hr>
+
+> I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
+
+This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
+
+</details>
+
+<details>
+
+<summary> questions about algorithms </summary>
+
+<hr>
+
+> How are you computing and sorting differential expression results?
+
+Currently we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
+
+</details>
+
+## developer guide
+
+This project has made a few key design choices
+
+- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
+- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
+- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
+
+Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
+
+If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
 
-- OS: OSX, Windows, Linux -- the developers are currently testing on OSX and Windows (via WSL using Ubuntu). It should work on other platforms but if you are using something different and need help, please let us know.
 - python 3.6
-- python3 tkinter
-- npm
-- Google Chrome
+- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
 
-**Clone project**
+Then clone the project
 
-    git clone https://github.com/chanzuckerberg/cellxgene.git
+```
+git clone https://github.com/chanzuckerberg/cellxgene.git
+```
 
-**Install client**
+Build the client web assets by calling this from inside the `cellxgene` folder
 
-    cd cellxgene
-    ./bin/build-client
+```
+./bin/build-client
+```
 
-**To use with virtual env for python**
-(optional, but recommended)
+Install all requirements (we recommend doing this inside a virtual environment)
 
-    ENV_NAME=cellxgene
-    python3 -m venv ${ENV_NAME}
-    source ${ENV_NAME}/bin/activate
+```
+pip install -e .
+```
 
-**Install server**
+You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
 
-     pip install -e .
+If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.
 
-**Run (with demo data)**
+## development roadmap
 
-    cellxgene launch --title PBMC3K example-dataset/pbmc3k.h5ad
+`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact)
 
-**Help**
+- **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
+- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene` to render the results of these analyses when they have been performed
+- **Deploy to web** Many projects release public data browser websites alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
+- **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data
 
-    cellxgene --help
+## contributing
 
-_For help with the scanpy engine_
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
 
-    cellxgene scanpy --help
+## inspiration and collaboration
 
-## Using your own data
+We've been heavily inspired by several other related single-cell visualization projects, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), and many others. We hope to explore collaborations where useful as this community works together on improving interactive visualization for single-cell data.
 
-### Scanpy
+We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.
 
-To prepare your data you will need to format your data into AnnData format using scanpy and calculate PCA and nearest neighbors and save in h5ad format.
+We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
 
-1.  [Load data into scanpy](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)
+We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
 
-    - Ensure that `obs`'s index is the cell names: `print(data.obs_names)` should show your cell indices. If it shows gene names, you may need to just call `data.transpose()`.
+## help and contact
 
-2.  Calculate PCA
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
 
-    sc.pp.pca(data) ## sc is scanpy.api
+## reuse
 
-3.  Calculate nearest neighbors (depending on layout algorithm)
-
-    ```
-    # For umap layout algorithm, you need to use the "umap" method for neighbors
-    sc.pp.neighbors(data, method="umap", metric="euclidean", use_rep="X_pca")
-
-    # For tsne layout algorithm, you can use either "umap" or "gauss"; we recommend "gauss"
-    sc.pp.neighbors(data, method="gauss", metric="euclidean", use_rep="X_pca")
-    ```
-
-4.  Save file
-
-    ```
-    # cellxgene requires file to be named data.h5ad
-    data.write("data.h5ad")
-    ```
-
-## Contributing
-
-We warmly welcome contributions from the community. Please submit any bug reports and feature requests through github issues. Please submit any direct contributions via a branch + pull request.
-
-## Inspiration and collaboration
-
-We’ve been inspired by several other related efforts in this space, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), & many others; we hope to explore collaborations where useful.
-
-## Help/Contact
-
-Have questions, suggestions, or comments? You can contact us by joining [CZI Science Slack](https://cziscience.slack.com/messages/CCTA8DF1T) and posting in the #cellxgene channel. Please submit any feature requests or bugs as an issue in github. We'd love to hear from you!
-
-## Reuse
-
-This project was started with the sole goal of empowering the scientific community to explore and understand their data. As such, we whole-heartedly encourage other scientific tool builders to adopt the patterns, tools, and code from this project, and reach out to us with ideas or questions using Github Issues or Pull Requests. All code is freely available for reuse under the [MIT license](https://opensource.org/licenses/MIT).
-
-## Acknowledgements
-
-cellxgene is inspired by many innovative projects. We would like to specifically thank:
-
-- Alex Wolf for the demo dataset.
-- Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for API inspiration.
+This project was started with the sole goal of empowering the scientific community to explore and understand their data. As such, we encourage other scientific tool builders in academia or industry to adopt the patterns, tools, and code from this project, and reach out to us with ideas or questions. All code is freely available for reuse under the [MIT license](https://opensource.org/licenses/MIT).

bin/build-client

@@ -1,15 +0,0 @@
-#!/bin/bash
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-CELLXGENE_DIR=$(dirname $DIR)
-
-cd $CELLXGENE_DIR
-
-npm install --prefix client/ client
-npm run  --prefix client build
-rm -rf server/app/web/static
-mkdir -p server/app/web/static/img
-cp client/build/index.html server/app/web/templates/
-cp -r client/build/static server/app/web/
-
-cp client/build/favicon.png server/app/web/static/img
-cp client/build/service-worker.js server/app/web/static/js/

bin/clean

@@ -1,16 +0,0 @@
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-CELLXGENE_DIR=$(dirname $DIR)
-
-echo "Uninstalling cellxgene"
-yes | pip uninstall cellxgene
-echo "removing node_modules"
-rm -rf $CELLXGENE_DIR/client/node_modules
-echo "removing client_build"
-rm -rf $CELLXGENE_DIR/client/build
-echo "removing egg-info"
-rm -rf $CELLXGENE_DIR/cellxgene.egg-info
-echo "removing static files"
-rm -f $CELLXGENE_DIR/server/app/web/templates/index.html
-rm -rf $CELLXGENE_DIR/server/app/web/static
-echo "cellxgene cleanup complete"
-

client/__tests__/e2e/e2e.test.js

@@ -0,0 +1,185 @@
+import puppeteer from "puppeteer";
+
+const jest_env = process.env.JEST_ENV || "dev";
+const appPort = process.env.JEST_CXG_PORT || 3000;
+const appUrlBase = `http://localhost:${appPort}`;
+const DEV = jest_env === "dev";
+
+let browser;
+let page;
+const browserViewport = { width: 1280, height: 960 };
+
+beforeAll(async () => {
+  const browser_params = DEV
+    ? { headless: false, slowMo: 100, devtools: true }
+    : {};
+  browser = await puppeteer.launch(browser_params);
+  page = await browser.newPage();
+  page.setViewport(browserViewport);
+  if (DEV) page.on("console", msg => console.log("PAGE LOG:", msg.text()));
+});
+
+afterAll(() => {
+  if (!DEV) {
+    browser.close();
+  }
+});
+
+const getOneElementInnerHTML = async function(selector) {
+  let text = await page.$eval(selector, el => el.innerHTML);
+  return text;
+};
+
+const drag = async function(el_box, start, end, lasso = false) {
+  const x1 = el_box.content[0].x + start.x;
+  const x2 = el_box.content[0].x + end.x;
+  const y1 = el_box.content[0].y + start.y;
+  const y2 = el_box.content[0].y + end.y;
+  await page.mouse.move(x1, y1);
+  await page.mouse.down();
+  if (lasso) {
+    await page.mouse.move(x2, y1);
+    await page.mouse.move(x2, y2);
+    await page.mouse.move(x1, y2);
+    await page.mouse.move(x1, y1);
+  } else {
+    await page.mouse.move(x2, y2);
+  }
+  await page.mouse.up();
+};
+
+describe("did launch", () => {
+  test("page launched", async () => {
+    await page.goto(appUrlBase);
+    let el = await getOneElementInnerHTML("[data-testid='header']");
+    expect(el).toBe("cellxgene: pbmc3k");
+  });
+});
+
+describe("search for genes", () => {
+  test("search for known gene and add to metadata", async () => {
+    await page.goto(appUrlBase);
+    await page.waitForSelector("[ data-testid='gene-search']");
+    // blueprint's  typeahead is treating typing weird, clicking & waiting first solves this
+    await page.click("[data-testid='gene-search']");
+    await page.waitFor(200);
+    await page.type("[data-testid='gene-search']", "ACD");
+    await page.keyboard.press("Enter");
+    await page.waitForSelector("[data-testid='histogram-ACD']");
+  });
+});
+
+describe("select cells and diffexp", () => {
+  test("selects cells from layout and adds to cell set 1", async () => {
+    await page.goto(appUrlBase);
+    const layout = await page.waitForSelector("[data-testid='layout']");
+    const size = await layout.boxModel();
+    const cellset1 = {
+      start: {
+        x: Math.floor(size.width * 0.25),
+        y: Math.floor(size.height * 0.25)
+      },
+      end: {
+        x: Math.floor(size.width * 0.35),
+        y: Math.floor(size.height * 0.35)
+      }
+    };
+    await drag(size, cellset1.start, cellset1.end, true);
+    await page.click("[data-testid='cellset-button-1");
+    let button = await getOneElementInnerHTML("[data-testid='cellset-button-1");
+    expect(button).toMatch(/26 cells/);
+  });
+
+  test("selects cells from layout and adds to cell set 2", async () => {
+    await page.goto(appUrlBase);
+    const layout = await page.waitForSelector("[data-testid='layout']");
+    const size = await layout.boxModel();
+    const cellset2 = {
+      start: {
+        x: Math.floor(size.width * 0.45),
+        y: Math.floor(size.height * 0.45)
+      },
+      end: {
+        x: Math.floor(size.width * 0.55),
+        y: Math.floor(size.height * 0.55)
+      }
+    };
+    await drag(size, cellset2.start, cellset2.end, true);
+    await page.click("[data-testid='cellset-button-2");
+    let button = await getOneElementInnerHTML("[data-testid='cellset-button-2");
+    expect(button).toMatch(/49 cells/);
+  });
+
+  test("selects cells, saves them and performs diffexp", async () => {
+    await page.goto(appUrlBase);
+    const layout = await page.waitForSelector("[data-testid='layout']");
+    const size = await layout.boxModel();
+    const cellset1 = {
+      start: {
+        x: Math.floor(size.width * 0.25),
+        y: Math.floor(size.height * 0.25)
+      },
+      end: {
+        x: Math.floor(size.width * 0.35),
+        y: Math.floor(size.height * 0.35)
+      }
+    };
+    await drag(size, cellset1.start, cellset1.end, true);
+    await page.click("[data-testid='cellset-button-1");
+    const cellset2 = {
+      start: {
+        x: Math.floor(size.width * 0.45),
+        y: Math.floor(size.height * 0.45)
+      },
+      end: {
+        x: Math.floor(size.width * 0.55),
+        y: Math.floor(size.height * 0.55)
+      }
+    };
+    await drag(size, cellset2.start, cellset2.end, true);
+    await page.click("[data-testid='cellset-button-2");
+    await page.click("[data-testid='diffexp-button");
+    await page.waitForSelector("[data-testclass='histogram-diffexp']");
+    const diffexps = await page.$$eval(
+      "[data-testclass='histogram-diffexp']",
+      divs => {
+        return divs.map(div =>
+          div.id.substring("histogram-".length, div.id.length)
+        );
+      }
+    );
+    expect(diffexps).toMatchObject([
+      "HLA-DPA1",
+      "HLA-DQA1",
+      "HLA-DRB1",
+      "HLA-DMA",
+      "CST3",
+      "HLA-DPB1",
+      "HLA-DQB1",
+      "LGALS2",
+      "FCER1A",
+      "LTB"
+    ]);
+  });
+});
+
+describe("brushable histogram", () => {
+  test("can brush historgram", async () => {
+    await page.goto(appUrlBase);
+    const hist = await page.waitForSelector(
+      "[data-testid='histogram_n_genes_svg-brush'] > .overlay"
+    );
+    const hist_size = await hist.boxModel();
+    const draghist = {
+      start: {
+        x: Math.floor(hist_size.width * 0.25),
+        y: Math.floor(hist_size.height * 0.5)
+      },
+      end: {
+        x: Math.floor(hist_size.width * 0.55),
+        y: Math.floor(hist_size.height * 0.5)
+      }
+    };
+    await drag(hist_size, draghist.start, draghist.end);
+  });
+});

client/__tests__/setupMissingGlobals.js

@@ -0,0 +1,9 @@
+/*
+Define globals which are present in the client, but not in node (and therefore not in
+the jest test environment).
+*/
+
+import { TextDecoder, TextEncoder } from "util";
+
+global.TextDecoder = TextDecoder;
+global.TextEncoder = TextEncoder;

client/__tests__/util/dataframe/dataframe.test.js

@@ -0,0 +1,616 @@
+import * as Dataframe from "../../../src/util/dataframe";
+
+describe("dataframe constructor", () => {
+  test("empty dataframe", () => {
+    const df = new Dataframe.Dataframe([0, 0], []);
+    expect(df).toBeDefined();
+    expect(df.dims).toEqual([0, 0]);
+    expect(df).toHaveLength(0);
+    expect(df.icol(0)).not.toBeDefined();
+  });
+
+  test("create with default indices", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 2],
+      [new Int32Array(3).fill(0), new Int32Array(3).fill(1)]
+    );
+
+    expect(df).toBeDefined();
+    expect(df.dims).toEqual([3, 2]);
+    expect(df.rowIndex).toBeInstanceOf(Dataframe.IdentityInt32Index);
+    expect(df.colIndex).toBeInstanceOf(Dataframe.IdentityInt32Index);
+    expect(df.at(0, 0)).toEqual(0);
+    expect(df.at(2, 1)).toEqual(1);
+    expect(df.iat(0, 0)).toEqual(0);
+    expect(df.iat(2, 1)).toEqual(1);
+  });
+
+  test("create with labelled indices", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 2],
+      [new Int32Array([0, 1, 2]), new Int32Array([3, 4, 5])],
+      new Dataframe.DenseInt32Index([2, 1, 0]),
+      new Dataframe.KeyIndex(["A", "B"])
+    );
+
+    expect(df).toBeDefined();
+    expect(df.dims).toEqual([3, 2]);
+
+    expect(df.rowIndex).toBeInstanceOf(Dataframe.DenseInt32Index);
+    expect(df.colIndex).toBeInstanceOf(Dataframe.KeyIndex);
+    expect(df.rowIndex.keys()).toEqual(new Int32Array([2, 1, 0]));
+    expect(df.colIndex.keys()).toEqual(["A", "B"]);
+
+    expect(df.at(0, "A")).toEqual(2);
+    expect(df.at(2, "B")).toEqual(3);
+    expect(df.iat(0, 0)).toEqual(0);
+    expect(df.iat(2, 1)).toEqual(5);
+  });
+});
+
+describe("simple data access", () => {
+  const df = new Dataframe.Dataframe(
+    [4, 2],
+    [
+      new Float64Array([0.0, Number.NaN, Number.POSITIVE_INFINITY, 3.14159]),
+      ["red", "blue", "green", "nan"]
+    ],
+    new Dataframe.DenseInt32Index([3, 2, 1, 0]),
+    new Dataframe.KeyIndex(["numbers", "colors"])
+  );
+
+  test("iat", () => {
+    expect(df).toBeDefined();
+
+    // present
+    expect(df.iat(0, 0)).toEqual(0.0);
+    expect(df.iat(0, 1)).toEqual("red");
+    expect(df.iat(1, 0)).toEqual(Number.NaN);
+    expect(df.iat(1, 1)).toEqual("blue");
+    expect(df.iat(2, 0)).toEqual(Number.POSITIVE_INFINITY);
+    expect(df.iat(2, 1)).toEqual("green");
+    expect(df.iat(3, 0)).toEqual(3.14159);
+    expect(df.iat(3, 1)).toEqual("nan");
+
+    // labels out of range have no defined behavior
+  });
+
+  test("at", () => {
+    expect(df).toBeDefined();
+
+    // present
+    expect(df.at(3, "numbers")).toEqual(0.0);
+    expect(df.at(3, "colors")).toEqual("red");
+    expect(df.at(2, "numbers")).toEqual(Number.NaN);
+    expect(df.at(2, "colors")).toEqual("blue");
+    expect(df.at(1, "numbers")).toEqual(Number.POSITIVE_INFINITY);
+    expect(df.at(1, "colors")).toEqual("green");
+    expect(df.at(0, "numbers")).toEqual(3.14159);
+    expect(df.at(0, "colors")).toEqual("nan");
+
+    // labels out of range have no defined behavior
+  });
+
+  test("ihas", () => {
+    expect(df).toBeDefined();
+
+    // present
+    expect(df.ihas(0, 0)).toBeTruthy();
+    expect(df.ihas(1, 1)).toBeTruthy();
+    expect(df.ihas(3, 1)).toBeTruthy();
+
+    // not present
+    expect(df.ihas(-1, -1)).toBeFalsy();
+    expect(df.ihas(0, 99)).toBeFalsy();
+    expect(df.ihas(99, 0)).toBeFalsy();
+    expect(df.ihas(99, 99)).toBeFalsy();
+    expect(df.ihas(-1, 0)).toBeFalsy();
+    expect(df.ihas(0, -1)).toBeFalsy();
+  });
+
+  test("has", () => {
+    expect(df).toBeDefined();
+
+    // present
+    expect(df.has(3, "numbers")).toBeTruthy();
+    expect(df.has(0, "numbers")).toBeTruthy();
+    expect(df.has(3, "colors")).toBeTruthy();
+    expect(df.has(0, "colors")).toBeTruthy();
+
+    // not present
+    expect(df.has(3, "foo")).toBeFalsy();
+    expect(df.has(-1, "numbers")).toBeFalsy();
+    expect(df.has(-1, -1)).toBeFalsy();
+    expect(df.has(null, null)).toBeFalsy();
+    expect(df.has(0, "foo")).toBeFalsy();
+    expect(df.has(99, "numbers")).toBeFalsy();
+    expect(df.has(99, "foo")).toBeFalsy();
+  });
+});
+
+describe("dataframe subsetting", () => {
+  describe("subset", () => {
+    const sourceDf = new Dataframe.Dataframe(
+      [3, 4],
+      [
+        new Int32Array([0, 1, 2]),
+        ["A", "B", "C"],
+        new Float32Array([4.4, 5.5, 6.6]),
+        ["red", "green", "blue"]
+      ],
+      null,
+      new Dataframe.KeyIndex(["int32", "string", "float32", "colors"])
+    );
+
+    test("all rows, one column", () => {
+      const dfA = sourceDf.subset(null, ["colors"]);
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([3, 1]);
+      expect(dfA.iat(0, 0)).toEqual("red");
+      expect(dfA.at(2, "colors")).toEqual("blue");
+      expect(dfA.col("colors").asArray()).toEqual(["red", "green", "blue"]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "green", "blue"]);
+      expect(dfA.col("colors").asArray()).toEqual(
+        sourceDf.col("colors").asArray()
+      );
+      expect(dfA.rowIndex.keys()).toEqual(sourceDf.rowIndex.keys());
+      expect(dfA.colIndex.keys()).toEqual(["colors"]);
+    });
+
+    test("all rows, two columns", () => {
+      const dfB = sourceDf.subset(null, ["colors", "float32"]);
+      expect(dfB).toBeDefined();
+      expect(dfB.dims).toEqual([3, 2]);
+      expect(dfB.iat(0, 0)).toBeCloseTo(4.4);
+      expect(dfB.iat(0, 1)).toEqual("red");
+      expect(dfB.at(2, "colors")).toEqual("blue");
+      expect(dfB.at(2, "float32")).toBeCloseTo(6.6);
+      expect(dfB.col("colors").asArray()).toEqual(["red", "green", "blue"]);
+      expect(dfB.col("float32").asArray()).toEqual(
+        new Float32Array([4.4, 5.5, 6.6])
+      );
+      expect(dfB.icol(0).asArray()).toEqual(dfB.col("float32").asArray());
+      expect(dfB.icol(1).asArray()).toEqual(dfB.col("colors").asArray());
+      expect(dfB.col("colors").asArray()).toEqual(
+        sourceDf.col("colors").asArray()
+      );
+      expect(dfB.col("float32").asArray()).toEqual(
+        sourceDf.col("float32").asArray()
+      );
+      expect(dfB.rowIndex.keys()).toEqual(sourceDf.rowIndex.keys());
+      expect(dfB.colIndex.keys()).toEqual(["float32", "colors"]);
+    });
+
+    test("one row, all columns", () => {
+      const dfC = sourceDf.subset([1], null);
+      expect(dfC).toBeDefined();
+      expect(dfC.dims).toEqual([1, 4]);
+      expect(dfC.iat(0, 0)).toEqual(1);
+      expect(dfC.iat(0, 1)).toEqual("B");
+      expect(dfC.iat(0, 2)).toBeCloseTo(5.5);
+      expect(dfC.iat(0, 3)).toEqual("green");
+      expect(dfC.rowIndex.keys()).toEqual(new Int32Array([1]));
+      expect(dfC.colIndex.keys()).toEqual(sourceDf.colIndex.keys());
+    });
+
+    test("two rows, all columns", () => {
+      const dfD = sourceDf.subset([0, 2], null);
+      expect(dfD).toBeDefined();
+      expect(dfD.dims).toEqual([2, 4]);
+      expect(dfD.icol(0).asArray()).toEqual(new Int32Array([0, 2]));
+      expect(dfD.icol(1).asArray()).toEqual(["A", "C"]);
+      expect(dfD.icol(2).asArray()).toEqual(new Float32Array([4.4, 6.6]));
+      expect(dfD.icol(3).asArray()).toEqual(["red", "blue"]);
+      expect(dfD.rowIndex.keys()).toEqual(new Int32Array([0, 2]));
+      expect(dfD.colIndex.keys()).toEqual(sourceDf.colIndex.keys());
+    });
+
+    test("all rows, all columns", () => {
+      const dfE = sourceDf.subset(null, null);
+      expect(dfE).toBeDefined();
+      expect(dfE.dims).toEqual([3, 4]);
+      expect(dfE.icol(0).asArray()).toEqual(sourceDf.icol(0).asArray());
+      expect(dfE.icol(1).asArray()).toEqual(sourceDf.icol(1).asArray());
+      expect(dfE.icol(2).asArray()).toEqual(sourceDf.icol(2).asArray());
+      expect(dfE.icol(3).asArray()).toEqual(sourceDf.icol(3).asArray());
+      expect(dfE.rowIndex.keys()).toEqual(sourceDf.rowIndex.keys());
+      expect(dfE.colIndex.keys()).toEqual(sourceDf.colIndex.keys());
+    });
+
+    test("two rows, two colums", () => {
+      const dfF = sourceDf.subset([0, 2], ["int32", "float32"]);
+      expect(dfF).toBeDefined();
+      expect(dfF.dims).toEqual([2, 2]);
+      expect(dfF.icol(0).asArray()).toEqual(new Int32Array([0, 2]));
+      expect(dfF.icol(1).asArray()).toEqual(new Float32Array([4.4, 6.6]));
+      expect(dfF.rowIndex.keys()).toEqual(new Int32Array([0, 2]));
+      expect(dfF.colIndex.keys()).toEqual(["int32", "float32"]);
+    });
+
+    test("withRowIndex", () => {
+      const df = sourceDf.subset(
+        null,
+        ["int32", "float32"],
+        new Dataframe.DenseInt32Index([3, 2, 1])
+      );
+      expect(df.colIndex).toBeInstanceOf(Dataframe.KeyIndex);
+      expect(df.rowIndex).toBeInstanceOf(Dataframe.DenseInt32Index);
+      expect(df.at(3, "int32")).toEqual(df.iat(0, 0));
+    });
+
+    test("withRowIndex error checks", () => {
+      expect(() =>
+        sourceDf.subset(null, ["red"], new Dataframe.IdentityInt32Index(1))
+      ).toThrow(RangeError);
+      expect(() =>
+        sourceDf.subset(null, ["red"], new Dataframe.DenseInt32Index([0, 1]))
+      ).toThrow(RangeError);
+      expect(() =>
+        sourceDf.subset(null, ["red"], new Dataframe.KeyIndex([0, 1, 2, 3]))
+      ).toThrow(RangeError);
+    });
+  });
+
+  test("isubsetMask", () => {
+    const sourceDf = new Dataframe.Dataframe(
+      [3, 4],
+      [
+        new Int32Array([0, 1, 2]),
+        ["A", "B", "C"],
+        new Float32Array([4.4, 5.5, 6.6]),
+        ["red", "green", "blue"]
+      ],
+      new Dataframe.DenseInt32Index([2, 4, 6]),
+      new Dataframe.KeyIndex(["int32", "string", "float32", "colors"])
+    );
+
+    const dfA = sourceDf.isubsetMask(
+      new Uint8Array([0, 1, 1]),
+      new Uint8Array([1, 0, 0, 1])
+    );
+    expect(dfA.dims).toEqual([2, 2]);
+    expect(dfA.icol(0).asArray()).toEqual(new Int32Array([1, 2]));
+    expect(dfA.icol(1).asArray()).toEqual(["green", "blue"]);
+    expect(dfA.rowIndex.keys()).toEqual(new Int32Array([4, 6]));
+    expect(dfA.colIndex.keys()).toEqual(["int32", "colors"]);
+  });
+});
+
+describe("dataframe factories", () => {
+  test("create", () => {
+    const df = Dataframe.Dataframe.create(
+      [3, 3],
+      [
+        new Array(3).fill(0),
+        new Int16Array(3).fill(99),
+        new Float64Array(3).fill(1.1)
+      ]
+    );
+
+    expect(df).toBeDefined();
+    expect(df.dims).toEqual([3, 3]);
+    expect(df).toHaveLength(3);
+    expect(df.iat(0, 0)).toEqual(0);
+    expect(df.iat(1, 1)).toEqual(99);
+    expect(df.iat(2, 2)).toBeCloseTo(1.1);
+    expect(df.iat(0, 0)).toEqual(df.at(0, 0));
+    expect(df.iat(1, 1)).toEqual(df.at(1, 1));
+    expect(df.iat(2, 2)).toEqual(df.at(2, 2));
+  });
+
+  test("clone", () => {
+    const dfA = new Dataframe.Dataframe(
+      [3, 2],
+      [new Int32Array([0, 1, 2]), new Int32Array([3, 4, 5])],
+      new Dataframe.DenseInt32Index([2, 1, 0]),
+      new Dataframe.KeyIndex(["A", "B"])
+    );
+
+    const dfB = dfA.clone();
+    expect(dfB).not.toBe(dfA);
+    expect(dfB.dims).toEqual(dfA.dims);
+    expect(dfB).toHaveLength(dfA.length);
+    expect(dfB.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    expect(dfB.colIndex.keys()).toEqual(dfA.colIndex.keys());
+    for (let i = 0, l = dfB.dims[1]; i < l; i += 1) {
+      expect(dfB.icol(i).asArray()).toEqual(dfA.icol(i).asArray());
+    }
+  });
+
+  describe("withCol", () => {
+    test("KeyIndex", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 2],
+        [["red", "blue"], [true, false]],
+        null,
+        new Dataframe.KeyIndex(["colors", "bools"])
+      );
+      const dfA = df.withCol("numbers", [1, 0]);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 3]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.icol(1).asArray()).toEqual([true, false]);
+      expect(dfA.icol(2).asArray()).toEqual([1, 0]);
+      expect(dfA.col("numbers").asArray()).toEqual([1, 0]);
+      expect(dfA.colIndex.keys()).toEqual(["colors", "bools", "numbers"]);
+      expect(df.colIndex.keys()).toEqual(["colors", "bools"]);
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    test("DenseInt32Index", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 2],
+        [["red", "blue"], [true, false]],
+        null,
+        new Dataframe.DenseInt32Index([74, 75])
+      );
+      const dfA = df.withCol(72, [1, 0]);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 3]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.icol(1).asArray()).toEqual([true, false]);
+      expect(dfA.icol(2).asArray()).toEqual([1, 0]);
+      expect(dfA.col(74).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.col(75).asArray()).toEqual([true, false]);
+      expect(dfA.col(72).asArray()).toEqual([1, 0]);
+      expect(dfA.colIndex.keys()).toEqual(new Int32Array([74, 75, 72]));
+      expect(df.colIndex.keys()).toEqual(new Int32Array([74, 75]));
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    test("DenseInt32Index promote", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 2],
+        [["red", "blue"], [true, false]],
+        null,
+        new Dataframe.DenseInt32Index([74, 75])
+      );
+      const dfA = df.withCol(999, [1, 0]);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 3]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.icol(1).asArray()).toEqual([true, false]);
+      expect(dfA.icol(2).asArray()).toEqual([1, 0]);
+      expect(dfA.col(74).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.col(75).asArray()).toEqual([true, false]);
+      expect(dfA.col(999).asArray()).toEqual([1, 0]);
+      expect(dfA.colIndex.keys()).toEqual(new Int32Array([74, 75, 999]));
+      expect(df.colIndex.keys()).toEqual(new Int32Array([74, 75]));
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    test("IdentityInt32Index with last", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 2],
+        [["red", "blue"], [true, false]],
+        null,
+        null
+      );
+      const dfA = df.withCol(2, [1, 0]);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 3]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.icol(1).asArray()).toEqual([true, false]);
+      expect(dfA.icol(2).asArray()).toEqual([1, 0]);
+      expect(dfA.col(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.col(1).asArray()).toEqual([true, false]);
+      expect(dfA.col(2).asArray()).toEqual([1, 0]);
+      expect(dfA.colIndex.keys()).toEqual(new Int32Array([0, 1, 2]));
+      expect(df.colIndex.keys()).toEqual(new Int32Array([0, 1]));
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    test("IdentityInt32Index promote", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 2],
+        [["red", "blue"], [true, false]],
+        null,
+        null
+      );
+      const dfA = df.withCol(99, [1, 0]);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 3]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.icol(1).asArray()).toEqual([true, false]);
+      expect(dfA.icol(2).asArray()).toEqual([1, 0]);
+      expect(dfA.col(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.col(1).asArray()).toEqual([true, false]);
+      expect(dfA.col(99).asArray()).toEqual([1, 0]);
+      expect(dfA.colIndex.keys()).toEqual(new Int32Array([0, 1, 99]));
+      expect(df.colIndex.keys()).toEqual(new Int32Array([0, 1]));
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    describe("handle column dimensions correctly", () => {
+      /*
+      there are two conditions:
+      - empty dataframe - will accept an add of any dimensionality
+      - non-empty dataframe - added column must match row-count dimension
+      */
+      test("empty.withCol", () => {
+        const edf = Dataframe.Dataframe.empty();
+        const df = edf.withCol("foo", [1, 2, 3]);
+
+        expect(edf).toBeDefined();
+        expect(df).toBeDefined();
+        expect(edf).not.toEqual(df);
+        expect(df.dims).toEqual([3, 1]);
+        expect(df.icol(0).asArray()).toEqual([1, 2, 3]);
+      });
+
+      test("withCol dimension check", () => {
+        const dfA = new Dataframe.Dataframe([1, 1], [["a"]]);
+        expect(() => {
+          dfA.withCol(1, []);
+        }).toThrow(RangeError);
+      });
+    });
+  });
+
+  describe("dropCol", () => {
+    test("KeyIndex", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 3],
+        [["red", "blue"], [true, false], [1, 0]],
+        null,
+        new Dataframe.KeyIndex(["colors", "bools", "numbers"])
+      );
+      const dfA = df.dropCol("colors");
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 2]);
+      expect(dfA.icol(0).asArray()).toEqual([true, false]);
+      expect(dfA.icol(1).asArray()).toEqual([1, 0]);
+      expect(dfA.col("numbers").asArray()).toEqual([1, 0]);
+      expect(dfA.colIndex.keys()).toEqual(["bools", "numbers"]);
+      expect(df.colIndex.keys()).toEqual(["colors", "bools", "numbers"]);
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    test("IdentityInt32Index drop first", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 3],
+        [["red", "blue"], [true, false], [1, 0]],
+        null,
+        null
+      );
+      const dfA = df.dropCol(0);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 2]);
+      expect(dfA.icol(0).asArray()).toEqual([true, false]);
+      expect(dfA.icol(1).asArray()).toEqual([1, 0]);
+      expect(df.col(1).asArray()).toEqual(dfA.col(1).asArray());
+      expect(df.col(2).asArray()).toEqual(dfA.col(2).asArray());
+      expect(dfA.colIndex.keys()).toEqual(new Int32Array([1, 2]));
+      expect(df.colIndex.keys()).toEqual(new Int32Array([0, 1, 2]));
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    test("IdentityInt32Index drop last", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 3],
+        [["red", "blue"], [true, false], [1, 0]],
+        null,
+        null
+      );
+      const dfA = df.dropCol(2);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 2]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.icol(1).asArray()).toEqual([true, false]);
+      expect(df.col(0).asArray()).toEqual(dfA.col(0).asArray());
+      expect(df.col(1).asArray()).toEqual(dfA.col(1).asArray());
+      expect(dfA.colIndex.keys()).toEqual(new Int32Array([0, 1]));
+      expect(df.colIndex.keys()).toEqual(new Int32Array([0, 1, 2]));
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+
+    test("DenseInt32Index", () => {
+      const df = new Dataframe.Dataframe(
+        [2, 3],
+        [["red", "blue"], [true, false], [1, 0]],
+        null,
+        new Dataframe.DenseInt32Index([102, 101, 100])
+      );
+      const dfA = df.dropCol(101);
+
+      expect(dfA).toBeDefined();
+      expect(dfA.dims).toEqual([2, 2]);
+      expect(dfA.icol(0).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.icol(1).asArray()).toEqual([1, 0]);
+      expect(dfA.col(100).asArray()).toEqual([1, 0]);
+      expect(dfA.col(102).asArray()).toEqual(["red", "blue"]);
+      expect(dfA.colIndex.keys()).toEqual(new Int32Array([102, 100]));
+      expect(df.colIndex.keys()).toEqual(new Int32Array([102, 101, 100]));
+      expect(df.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+    });
+  });
+});
+
+describe("dataframe col", () => {
+  let df = null;
+  beforeEach(() => {
+    df = new Dataframe.Dataframe(
+      [2, 2],
+      [[true, false], [1, 0]],
+      null,
+      new Dataframe.KeyIndex(["A", "B"])
+    );
+  });
+
+  test("col", () => {
+    expect(df).toBeDefined();
+    expect(df.col("A")).toBe(df.icol(0));
+    expect(df.col("B")).toBe(df.icol(1));
+    expect(df.col("undefined")).toBeUndefined();
+    expect(df.icol("undefined")).toBeUndefined();
+
+    const colA = df.col("A");
+    expect(colA).toBeInstanceOf(Function);
+    expect(colA.asArray).toBeInstanceOf(Function);
+    expect(colA.has).toBeInstanceOf(Function);
+    expect(colA.ihas).toBeInstanceOf(Function);
+    expect(colA.indexOf).toBeInstanceOf(Function);
+    expect(colA.iget).toBeInstanceOf(Function);
+  });
+
+  test("col.asArray", () => {
+    expect(df).toBeDefined();
+    expect(df.col("A").asArray()).toEqual([true, false]);
+    expect(df.icol(0).asArray()).toEqual([true, false]);
+    expect(df.col("B").asArray()).toEqual([1, 0]);
+    expect(df.icol(1).asArray()).toEqual([1, 0]);
+  });
+
+  test("col.has", () => {
+    expect(df).toBeDefined();
+    expect(df.col("A").has(-1)).toBe(false);
+    expect(df.col("A").has(0)).toBe(true);
+    expect(df.col("A").has(1)).toBe(true);
+    expect(df.col("A").has(2)).toBe(false);
+    expect(df.col("B").has(-1)).toBe(false);
+    expect(df.col("B").has(0)).toBe(true);
+    expect(df.col("B").has(1)).toBe(true);
+    expect(df.col("B").has(2)).toBe(false);
+  });
+
+  test("col.ihas", () => {
+    expect(df).toBeDefined();
+    expect(df.col("A").ihas(-1)).toBe(false);
+    expect(df.col("A").ihas(0)).toBe(true);
+    expect(df.col("A").ihas(1)).toBe(true);
+    expect(df.col("A").ihas(2)).toBe(false);
+    expect(df.col("B").ihas(-1)).toBe(false);
+    expect(df.col("B").ihas(0)).toBe(true);
+    expect(df.col("B").ihas(1)).toBe(true);
+    expect(df.col("B").ihas(2)).toBe(false);
+  });
+
+  test("col.iget", () => {
+    expect(df).toBeDefined();
+    expect(df.col("A").iget(0)).toEqual(df.iat(0, 0));
+    expect(df.col("B").iget(1)).toEqual(df.iat(1, 1));
+  });
+
+  test("col.indexOf", () => {
+    expect(df).toBeDefined();
+    expect(df.col("A").indexOf(true)).toEqual(0);
+    expect(df.col("A").indexOf(false)).toEqual(1);
+    expect(df.col("A").indexOf(99)).toBeUndefined();
+    expect(df.col("A").indexOf(undefined)).toBeUndefined();
+    expect(df.col("A").indexOf(1)).toBeUndefined();
+
+    expect(df.col("B").indexOf(1)).toEqual(0);
+    expect(df.col("B").indexOf(0)).toEqual(1);
+    expect(df.col("B").indexOf(99)).toBeUndefined();
+    expect(df.col("B").indexOf(undefined)).toBeUndefined();
+    expect(df.col("B").indexOf(true)).toBeUndefined();
+  });
+});

client/__tests__/util/dataframe/summarize.test.js

@@ -0,0 +1,253 @@
+import * as Dataframe from "../../../src/util/dataframe";
+
+function float32Conversion(f) {
+  return new Float32Array([f])[0];
+}
+
+describe("Dataframe column summary", () => {
+  test("empty column test", () => {
+    const df = Dataframe.Dataframe.create([0, 1], [[]]);
+    const summary = df.icol(0).summarize();
+    expect(summary).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: [],
+        categoryCounts: new Map(),
+        numCategories: 0
+      })
+    );
+  });
+
+  test("simple test", () => {
+    const df = new Dataframe.Dataframe(
+      [1, 6],
+      [
+        ["n1"],
+        ["hi"],
+        [true],
+        new Float32Array([39.3]),
+        new Int32Array([99]),
+        [1]
+      ],
+      null,
+      new Dataframe.KeyIndex([
+        "name",
+        "nameString",
+        "nameBoolean",
+        "nameFloat32",
+        "nameInt32",
+        "nameCategorical"
+      ])
+    );
+
+    expect(df.icol(0).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: ["n1"],
+        categoryCounts: new Map([["n1", 1]]),
+        numCategories: 1
+      })
+    );
+    expect(df.icol(1).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: ["hi"],
+        categoryCounts: new Map([["hi", 1]]),
+        numCategories: 1
+      })
+    );
+    expect(df.icol(2).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: [true],
+        categoryCounts: new Map([[true, 1]]),
+        numCategories: 1
+      })
+    );
+    expect(df.icol(3).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: false,
+        min: float32Conversion(39.3),
+        max: float32Conversion(39.3),
+        nan: 0,
+        ninf: 0,
+        pinf: 0
+      })
+    );
+    expect(df.icol(4).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: false,
+        min: 99,
+        max: 99,
+        nan: 0,
+        ninf: 0,
+        pinf: 0
+      })
+    );
+    expect(df.icol(5).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: [1],
+        categoryCounts: new Map([[1, 1]]),
+        numCategories: 1
+      })
+    );
+  });
+
+  test("multi test", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 6],
+      [
+        ["n0", "n1", "n2"],
+        ["hi", "hi", "bye"],
+        [false, true, true],
+        new Float32Array([39.3, 39.3, 0]),
+        new Int32Array([99, 99, 99]),
+        [1, false, "0"]
+      ],
+      null,
+      new Dataframe.KeyIndex([
+        "name",
+        "nameString",
+        "nameBoolean",
+        "nameFloat32",
+        "nameInt32",
+        "nameCategorical"
+      ])
+    );
+
+    expect(df.icol(0).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining(["n0", "n1", "n2"]),
+        categoryCounts: new Map([["n0", 1], ["n1", 1], ["n2", 1]]),
+        numCategories: 3
+      })
+    );
+    expect(df.icol(1).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining(["hi", "bye"]),
+        categoryCounts: new Map([["hi", 2], ["bye", 1]]),
+        numCategories: 2
+      })
+    );
+    expect(df.icol(2).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining([true, false]),
+        categoryCounts: new Map([[true, 2], [false, 1]]),
+        numCategories: 2
+      })
+    );
+    expect(df.icol(3).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: false,
+        min: 0,
+        max: float32Conversion(39.3),
+        nan: 0,
+        ninf: 0,
+        pinf: 0
+      })
+    );
+    expect(df.icol(4).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: false,
+        min: 99,
+        max: 99,
+        nan: 0,
+        ninf: 0,
+        pinf: 0
+      })
+    );
+    expect(df.icol(5).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining([1, false, "0"]),
+        categoryCounts: new Map([[1, 1], [false, 1], ["0", 1]]),
+        numCategories: 3
+      })
+    );
+  });
+
+  test("non-finite numbers", () => {
+    const df = new Dataframe.Dataframe(
+      [4, 6],
+      [
+        ["n0", "n1", "n2", "n2"],
+        ["hi", "hi", "bye", "bye"],
+        [false, true, true, true],
+        new Float32Array([
+          39.3,
+          Number.NEGATIVE_INFINITY,
+          Number.NaN,
+          Number.POSITIVE_INFINITY
+        ]),
+        new Int32Array([99, 99, 99, 99]),
+        [1, false, "0", "0"]
+      ],
+      null,
+      new Dataframe.KeyIndex([
+        "name",
+        "nameString",
+        "nameBoolean",
+        "nameFloat32",
+        "nameInt32",
+        "nameCategorical"
+      ])
+    );
+
+    expect(df.icol(0).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining(["n0", "n1", "n2"]),
+        categoryCounts: new Map([["n0", 1], ["n1", 1], ["n2", 2]]),
+        numCategories: 3
+      })
+    );
+    expect(df.icol(1).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining(["hi", "bye"]),
+        categoryCounts: new Map([["hi", 2], ["bye", 1]]),
+        numCategories: 2
+      })
+    );
+    expect(df.icol(2).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining([true, false]),
+        categoryCounts: new Map([[true, 2], [false, 1]]),
+        numCategories: 2
+      })
+    );
+    expect(df.icol(3).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: false,
+        min: float32Conversion(39.3),
+        max: float32Conversion(39.3),
+        nan: 1,
+        ninf: 1,
+        pinf: 1
+      })
+    );
+    expect(df.icol(4).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: false,
+        min: 99,
+        max: 99,
+        nan: 0,
+        ninf: 0,
+        pinf: 0
+      })
+    );
+    expect(df.icol(5).summarize()).toEqual(
+      expect.objectContaining({
+        categorical: true,
+        categories: expect.arrayContaining([1, false, "0"]),
+        categoryCounts: new Map([[1, 1], [false, 1], ["0", 1]]),
+        numCategories: 3
+      })
+    );
+  });
+});

client/__tests__/util/stateManager/keyvalcache.test.js

@@ -1,249 +0,0 @@
-import _ from "lodash";
-import * as kvCache from "../../../src/util/stateManager/keyvalcache";
-
-/*
-This is PRIVATE to keyvalcache and must be kept in sync with
-any changs ot that module.  Need to Know - to enable error handling test
-*/
-const cachePrivateKey = "__kvcachekey__";
-
-/*
-helper function - promisify setTimeout()
-*/
-function timeout(ms) {
-  return new Promise(resolve => setTimeout(resolve, ms));
-}
-
-describe("kvcache API", () => {
-  /*
-  test the happy path create/set/get API
-  */
-
-  test("simple create", () => {
-    /* with defaults */
-    const kvc = kvCache.create();
-    expect(kvc).toBeDefined();
-    expect(kvc).toEqual(expect.objectContaining({}));
-    expect(kvCache.get(kvc, "test")).toBeUndefined();
-
-    /* with params */
-    const kvc1 = kvCache.create(/* lowWatermark */ 99, /* minTTL */ 0);
-    expect(kvc1).toBeDefined();
-    expect(kvc1).toEqual(expect.objectContaining({}));
-  });
-
-  test("set/get", () => {
-    /*
-    - check basic get/set functionality
-    - check set does not mutate source cache
-    */
-    const keyName = "foo";
-    const kvc1 = kvCache.create();
-    expect(kvc1).toBeDefined();
-    expect(kvCache.get(kvc1, keyName)).toBeUndefined();
-
-    const val2 = [2];
-    const kvc2 = kvCache.set(kvc1, keyName, val2);
-    expect(kvc2).toBeDefined();
-    expect(kvc2).not.toBe(kvc1);
-    expect(kvCache.get(kvc1, keyName)).toBeUndefined();
-    expect(kvCache.get(kvc2, keyName)).toBe(val2);
-
-    const val3 = [3];
-    const kvc3 = kvCache.set(kvc2, keyName, val3);
-    expect(kvc3).toBeDefined();
-    expect(kvc3).not.toBe(kvc1);
-    expect(kvc3).not.toBe(kvc2);
-    expect(kvCache.get(kvc1, keyName)).toBeUndefined();
-    expect(kvCache.get(kvc2, keyName)).toBe(val2);
-    expect(kvCache.get(kvc3, keyName)).toBe(val3);
-  });
-});
-
-describe("common error handling", () => {
-  /*
-  Test common error handlers
-  */
-
-  test("set() protection from namespace pollution", () => {
-    /*
-    Test that set() will not allow use of the private cache key
-    */
-    const kvc = kvCache.create();
-    expect(() => {
-      kvCache.set(kvc, cachePrivateKey, {});
-    }).toThrow();
-  });
-
-  test("create() does not accept bogus config", () => {
-    expect(() => {
-      kvCache.create([], {});
-    }).toThrow();
-    expect(() => {
-      kvCache.create(-99, 0);
-    }).toThrow();
-    expect(() => {
-      kvCache.create(100, -1);
-    }).toThrow();
-    expect(() => {
-      kvCache.create(1000, "foobar");
-    }).toThrow();
-    expect(() => {
-      kvCache.create(null, 8);
-    }).toThrow();
-  });
-});
-
-describe("map", () => {
-  /*
-  Test kvCache.map() - create new cache that is a transformation of an
-  existing cache
-  */
-  test("map of empty cache", () => {
-    const kvc = kvCache.create();
-    const callback = jest.fn();
-    const kvcMapped = kvCache.map(kvc, callback);
-    expect(callback).not.toHaveBeenCalled();
-    expect(kvcMapped).toBeDefined();
-    expect(kvcMapped).not.toBe(kvc); // immutable operation
-    expect(kvcMapped).toEqual(kvc);
-  });
-
-  test("map of non-empty cache", () => {
-    const key = "aKey";
-    const val = [0, 1, 2];
-    let kvc = kvCache.create();
-    kvc = kvCache.set(kvc, key, val);
-    const mockCB = jest.fn().mockImplementation(v => [...v]);
-    const kvcMapped = kvCache.map(kvc, mockCB);
-
-    expect(kvcMapped).toBeDefined();
-    expect(kvcMapped).not.toBe(kvc); // immutable operation
-    expect(_.isEqual(kvc, kvcMapped)).toBe(true);
-
-    expect(mockCB).toHaveBeenCalledTimes(1);
-    expect(mockCB).toHaveBeenLastCalledWith(val, key);
-  });
-});
-
-describe("flush", () => {
-  /*
-  test various cache flush behavior
-  */
-  test("flush - lowWatermark, disable minTTL", () => {
-    /*
-    verify lowWatermark functions correctly
-    */
-
-    // set lowWatermark to 2, set three times - only the final two
-    // should remain.
-    let kvc = kvCache.create(2, 0);
-    ["a", "b", "c"].forEach(k => {
-      kvc = kvCache.set(kvc, k, []);
-    });
-
-    expect(kvc).toEqual(
-      expect.objectContaining({
-        b: expect.arrayContaining([]),
-        c: expect.arrayContaining([])
-      })
-    );
-    expect(kvc).toEqual(
-      expect.not.objectContaining({
-        a: expect.arrayContaining([])
-      })
-    );
-  });
-
-  test("flush - minTTL, disable lowWatermark", async () => {
-    /*
-    verify minTTL functions correctly
-    */
-
-    // set minTTL to 1 ms
-    let kvc = kvCache.create(0, 10);
-    kvc = kvCache.set(kvc, "a", []);
-    await timeout(20);
-    ["b", "c"].forEach(k => {
-      kvc = kvCache.set(kvc, k, []);
-    });
-
-    expect(kvc).toEqual(
-      expect.objectContaining({
-        b: expect.arrayContaining([]),
-        c: expect.arrayContaining([])
-      })
-    );
-    expect(kvc).toEqual(
-      expect.not.objectContaining({
-        a: expect.arrayContaining([])
-      })
-    );
-  });
-
-  test("flush - minTTL and lowWatermark", async () => {
-    /*
-    verify minTTL functions correctly
-    */
-
-    // set lowwatermark to 3, minTTL to 1 ms
-    let kvc = kvCache.create(3, 10);
-    kvc = kvCache.set(kvc, "a", []);
-    // delay
-    await timeout(20);
-    ["b", "c"].forEach(k => {
-      kvc = kvCache.set(kvc, k, []);
-    });
-
-    expect(kvc).toEqual(
-      expect.objectContaining({
-        a: expect.arrayContaining([]),
-        b: expect.arrayContaining([]),
-        c: expect.arrayContaining([])
-      })
-    );
-
-    kvc = kvCache.set(kvc, "d", []);
-    expect(kvc).toEqual(
-      expect.objectContaining({
-        b: expect.arrayContaining([]),
-        c: expect.arrayContaining([]),
-        d: expect.arrayContaining([])
-      })
-    );
-    expect(kvc).toEqual(
-      expect.not.objectContaining({
-        a: expect.arrayContaining([])
-      })
-    );
-  });
-
-  test("manual flush", async () => {
-    let kvc = kvCache.create(1, 10);
-    ["a", "b", "c", "d"].forEach(k => {
-      kvc = kvCache.set(kvc, k, []);
-    });
-
-    // Before TTL has expired, should have all values in cache.
-    expect(kvc).toEqual(
-      expect.objectContaining({
-        a: expect.arrayContaining([]),
-        b: expect.arrayContaining([]),
-        c: expect.arrayContaining([])
-      })
-    );
-
-    // let TTL expire
-    await timeout(10);
-
-    // manually flush
-    const postFlushKvc = kvCache.flush(kvc);
-    expect(postFlushKvc).toBeDefined();
-    expect(postFlushKvc).not.toBe(kvc);
-    expect(postFlushKvc).toEqual(
-      expect.objectContaining({
-        d: expect.arrayContaining([])
-      })
-    );
-  });
-});

client/__tests__/util/stateManager/sampleResponses.js

@@ -1,5 +1,7 @@
 /* eslint no-bitwise: "off" */
 import _ from "lodash";
+import { flatbuffers } from "flatbuffers";
+import { NetEncoding } from "../../../src/util/stateManager/matrix_generated";
 
 /*
 test data mocking REST 0.2 API responses.  Used in several tests.
@@ -58,7 +60,7 @@ const aSchemaResponse = {
   }
 };
 
-const anAnnotationsObsResponse = {
+const anAnnotationsObsJSONResponse = {
   names: ["name", "field1", "field2", "field3", "field4"],
   data: _()
     .range(nObs)
@@ -73,7 +75,7 @@ const anAnnotationsObsResponse = {
     .value()
 };
 
-const anAnnotationsVarResponse = {
+const anAnnotationsVarJSONResponse = {
   names: ["fieldA", "fieldB", "fieldC", "fieldD", "name"],
   data: _()
     .range(nVar)
@@ -88,15 +90,102 @@ const anAnnotationsVarResponse = {
     .value()
 };
 
-const aLayoutResponse = {
-  layout: {
-    ndims: 2,
-    coordinates: _()
-      .range(nObs)
-      .map(idx => [idx, Math.random(), Math.random()])
-      .value()
+function encodeTypedArray(builder, uType, uData) {
+  const uTypeName = NetEncoding.TypedArray[uType];
+  const ArrayType = NetEncoding[uTypeName];
+  const dv = ArrayType.createDataVector(builder, uData);
+  builder.startObject(1);
+  builder.addFieldOffset(0, dv, 0);
+  return builder.endObject();
+}
+
+function encodeMatrix(columns, colIndex = undefined) {
+  const utf8Encoder = new TextEncoder("utf-8");
+  const builder = new flatbuffers.Builder(1024);
+  const cols = _.map(columns, carr => {
+    let uType;
+    let tarr;
+    if (_.every(carr, _.isNumber)) {
+      uType = NetEncoding.TypedArray.Float32Array;
+      tarr = encodeTypedArray(builder, uType, new Float32Array(carr));
+    } else {
+      uType = NetEncoding.TypedArray.JSONEncodedArray;
+      const json = JSON.stringify(carr);
+      const jsonUTF8 = utf8Encoder.encode(json);
+      tarr = encodeTypedArray(builder, uType, jsonUTF8);
+    }
+    NetEncoding.Column.startColumn(builder);
+    NetEncoding.Column.addUType(builder, uType);
+    NetEncoding.Column.addU(builder, tarr);
+    return NetEncoding.Column.endColumn(builder);
+  });
+
+  const encColumns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+  let encColIndex;
+  if (colIndex) {
+    encColIndex = encodeTypedArray(
+      builder,
+      NetEncoding.TypedArray.JSONEncodedArray,
+      utf8Encoder.encode(JSON.stringify(colIndex))
+    );
   }
-};
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, columns[0].length);
+  NetEncoding.Matrix.addNCols(builder, columns.length);
+  NetEncoding.Matrix.addColumns(builder, encColumns);
+  if (colIndex) {
+    NetEncoding.Matrix.addColIndexType(
+      builder,
+      NetEncoding.TypedArray.JSONEncodedArray
+    );
+    NetEncoding.Matrix.addColIndex(builder, encColIndex);
+  }
+  const root = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(root);
+  return builder.asUint8Array();
+}
+
+const anAnnotationsObsFBSResponse = (() => {
+  const columns = _.zip(...anAnnotationsObsJSONResponse.data).slice(1);
+  return encodeMatrix(columns, anAnnotationsObsJSONResponse.names);
+})();
+
+const anAnnotationsVarFBSResponse = (() => {
+  const columns = _.zip(...anAnnotationsVarJSONResponse.data).slice(1);
+  return encodeMatrix(columns, anAnnotationsVarJSONResponse.names);
+})();
+
+const aLayoutFBSResponse = (() => {
+  const coords = [
+    new Float32Array(nObs).fill(Math.random()),
+    new Float32Array(nObs).fill(Math.random())
+  ];
+  const builder = new flatbuffers.Builder(1024);
+
+  const cols = _.map(coords, carr => {
+    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
+    NetEncoding.Float32Array.startFloat32Array(builder);
+    NetEncoding.Float32Array.addData(builder, cdv);
+    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
+
+    NetEncoding.Column.startColumn(builder);
+    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
+    NetEncoding.Column.addU(builder, floatArr);
+    return NetEncoding.Column.endColumn(builder);
+  });
+
+  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, nObs);
+  NetEncoding.Matrix.addNCols(builder, coords.length);
+  NetEncoding.Matrix.addColumns(builder, columns);
+  const matrix = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(matrix);
+  return builder.asUint8Array();
+})();
 
 const aDataObsResponse = {
   var: [2, 4, 29],
@@ -107,10 +196,10 @@ const aDataObsResponse = {
 };
 
 export {
-  aLayoutResponse as layoutObs,
+  aLayoutFBSResponse as layoutObs,
   aDataObsResponse as dataObs,
-  anAnnotationsVarResponse as annotationsVar,
-  anAnnotationsObsResponse as annotationsObs,
+  anAnnotationsVarFBSResponse as annotationsVar,
+  anAnnotationsObsFBSResponse as annotationsObs,
   aSchemaResponse as schema,
   aConfigResponse as config
 };

client/__tests__/util/stateManager/universe.test.js

@@ -1,13 +1,13 @@
-import _ from "lodash";
 import * as Universe from "../../../src/util/stateManager/universe";
+import * as Dataframe from "../../../src/util/dataframe";
 import * as REST from "./sampleResponses";
 
-describe("createUniverseFromRestV02Response", () => {
+describe("createUniverseFromResponse", () => {
   /*
-  test createUniverseFromRestV02Response - this function converts
+  test createUniverseFromResponse - this function converts
   a set of REST 0.2 responses into a "new" Universe.
 
-  createUniverseFromRestV02Response(
+  createUniverseFromResponse(
     configResponse,
     schemaResponse,
     annotationsObsResponse,
@@ -30,8 +30,7 @@ describe("createUniverseFromRestV02Response", () => {
     create a universe from sample data nad validate its shape & contents
     */
     const { nObs, nVar } = REST.schema.schema.dataframe;
-
-    const universe = Universe.createUniverseFromRestV02Response(
+    const universe = Universe.createUniverseFromResponse(
       REST.config,
       REST.schema,
       REST.annotationsObs,
@@ -42,80 +41,26 @@ describe("createUniverseFromRestV02Response", () => {
     expect(universe).toBeDefined();
     expect(universe).toMatchObject(
       expect.objectContaining({
-        api: "0.2",
         nObs,
         nVar,
         schema: REST.schema.schema,
-        obsAnnotations: expect.any(Array),
-        varAnnotations: expect.any(Array),
-        obsNameToIndexMap: expect.any(Object),
-        varNameToIndexMap: expect.any(Object),
-        obsLayout: expect.objectContaining({
-          X: expect.any(Float32Array),
-          Y: expect.any(Float32Array)
-        }),
-        varDataCache: expect.any(Object)
+        obsAnnotations: expect.any(Dataframe.Dataframe),
+        varAnnotations: expect.any(Dataframe.Dataframe),
+        obsLayout: expect.any(Dataframe.Dataframe),
+        varData: expect.any(Dataframe.Dataframe)
       })
     );
 
-    expect(universe.obsAnnotations).toHaveLength(nObs);
-    expect(_.keys(universe.obsNameToIndexMap)).toHaveLength(nObs);
-    expect(universe.obsLayout.X).toHaveLength(nObs);
-    expect(universe.obsLayout.Y).toHaveLength(nObs);
-    expect(universe.varAnnotations).toHaveLength(nVar);
-    expect(_.keys(universe.varNameToIndexMap)).toHaveLength(nVar);
-  });
-});
-
-describe("convertExpressionRESTv02ToObject", () => {
-  /*
-  test convertExpressionRESTv02ToObject
-
-  convertExpressionRESTv02ToObject(
-    universe,
-    response) --> { geneName: Float32Array, geneName: Float32Array, ... }
-
-  reponse is a /data/obs response:
-  {
-    var: [ varIndices fetched ],
-    obs: [
-      [ obsIndex, evalue, ... ],
-      ...
-    ]
-  }
-  */
-  test("create from response data", () => {
-    const universe = Universe.createUniverseFromRestV02Response(
-      REST.config,
-      REST.schema,
-      REST.annotationsObs,
-      REST.annotationsVar,
-      REST.layoutObs
-    );
-    const expression = Universe.convertExpressionRESTv02ToObject(
-      universe,
-      REST.dataObs
-    );
-
-    /* Check that the expected keys are present */
-    const expectedGeneNames = _.map(
-      REST.dataObs.var,
-      v => REST.annotationsVar.data[v][5]
-    );
-    expect(Object.keys(expression)).toEqual(
-      expect.arrayContaining(expectedGeneNames)
-    );
-
-    const expectedExpressionValues = _.map(
-      _.unzip(REST.dataObs.obs),
-      a => new Float32Array(a)
-    );
-
-    _.forEach(REST.dataObs.var, (varIdx, idx) => {
-      const varName = universe.varAnnotations[varIdx].name;
-      expect(varName).toBeDefined();
-      expect(varIdx).toBe(universe.varNameToIndexMap[varName]);
-      expect(expression[varName]).toEqual(expectedExpressionValues[idx + 1]);
-    });
+    expect(universe.obsAnnotations.dims).toEqual([
+      nObs,
+      REST.schema.schema.annotations.obs.length
+    ]);
+    expect(universe.obsLayout.dims).toEqual([nObs, 2]);
+    expect(universe.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(universe.varAnnotations.dims).toEqual([
+      nVar,
+      REST.schema.schema.annotations.var.length
+    ]);
+    expect(universe.varData.isEmpty()).toBeTruthy();
   });
 });

client/__tests__/util/stateManager/world.test.js

@@ -1,13 +1,13 @@
 import _ from "lodash";
 import * as Universe from "../../../src/util/stateManager/universe";
 import * as World from "../../../src/util/stateManager/world";
+import * as Dataframe from "../../../src/util/dataframe";
 import Crossfilter from "../../../src/util/typedCrossfilter";
 import * as REST from "./sampleResponses";
 import {
   obsAnnoDimensionName,
   layoutDimensionName
 } from "../../../src/util/nameCreators";
-import * as kvCache from "../../../src/util/stateManager/keyvalcache";
 
 /*
 Helper - creates universe, world, corssfilter and dimensionMap from
@@ -16,7 +16,7 @@ the default REST test response.
 const defaultBigBang = () => {
   /* create unverse, world, crossfilter and dimensionMap */
   /* create universe */
-  const universe = Universe.createUniverseFromRestV02Response(
+  const universe = Universe.createUniverseFromResponse(
     REST.config,
     REST.schema,
     REST.annotationsObs,
@@ -40,7 +40,7 @@ const defaultBigBang = () => {
 
 describe("createWorldFromEntireUniverse", () => {
   test("create from REST sample", () => {
-    const universe = Universe.createUniverseFromRestV02Response(
+    const universe = Universe.createUniverseFromResponse(
       REST.config,
       REST.schema,
       REST.annotationsObs,
@@ -54,31 +54,13 @@ describe("createWorldFromEntireUniverse", () => {
 
     expect(world).toMatchObject(
       expect.objectContaining({
-        api: "0.2",
         nObs: universe.nObs,
         nVar: universe.nVar,
         schema: universe.schema,
         obsAnnotations: universe.obsAnnotations,
         varAnnotations: universe.varAnnotations,
         obsLayout: universe.obsLayout,
-
-        summary: expect.objectContaining({
-          obs: _(REST.schema.schema.annotations.obs)
-            .filter(v => v.name !== "name")
-            .keyBy("name")
-            .mapValues(() => expect.any(Object))
-            .value(),
-          var: _(REST.schema.schema.annotations.var)
-            .filter(v => v.name !== "name")
-            .keyBy("name")
-            .mapValues(() => expect.any(Object))
-            .value()
-        }),
-
-        varDataCache: expect.any(Object),
-
-        obsIndex: null, // null indicating full universe
-        obsBackIndex: null
+        varData: expect.any(Dataframe.Dataframe)
       })
     );
   });
@@ -111,51 +93,38 @@ describe("createWorldFromCurrentSelection", () => {
     */
 
     /* matchFilter must match the dimension filters above */
-    const matchFilter = val => val.field1 >= 0 && val.field1 < 5 && !val.field3;
-    const universeIndices = _()
-      .range(universe.nObs)
-      .filter(idx => matchFilter(universe.obsAnnotations[idx]))
-      .value();
-
-    const expected = {
-      nObs: universeIndices.length,
-      obsAnnotations: _.map(universeIndices, i => universe.obsAnnotations[i]),
-      obsLayout: {
-        X: new Float32Array(
-          _.map(universeIndices, i => universe.obsLayout.X[i])
-        ),
-        Y: new Float32Array(
-          _.map(universeIndices, i => universe.obsLayout.Y[i])
-        )
-      },
-      obsBackIndex: _.transform(
-        universeIndices,
-        (result, univIdx, worldIdx) => {
-          result[univIdx] = worldIdx;
-        },
-        new Uint32Array(universe.nObs).fill(-1)
-      ),
-      obsIndex: new Uint32Array(universeIndices)
+    const matchFilter = (df, row) => {
+      const field1 = df.at(row, "field1");
+      const field3 = df.at(row, "field3");
+      return field1 >= 0 && field1 < 5 && !field3;
     };
+    const matchingIndices = _()
+      .range(universe.nObs)
+      .filter(idx => matchFilter(universe.obsAnnotations, idx))
+      .value();
 
     expect(world).toMatchObject(
       expect.objectContaining({
-        api: "0.2",
-        nObs: expected.nObs,
+        nObs: matchingIndices.length,
         nVar: universe.nVar,
         schema: universe.schema,
-        obsAnnotations: expected.obsAnnotations,
+        obsAnnotations: expect.any(Dataframe.Dataframe),
         varAnnotations: universe.varAnnotations,
-        obsLayout: expected.obsLayout,
-        summary: {
-          obs: expect.any(Object) /* we could do better! */,
-          var: expect.any(Object) /* we could do better! */
-        },
-        varDataCache: expect.any(Object),
-        obsIndex: expected.obsIndex,
-        obsBackIndex: expected.obsBackIndex
+        obsLayout: expect.any(Dataframe.Dataframe),
+        varData: expect.any(Dataframe.Dataframe)
       })
     );
+
+    expect(world.obsAnnotations.rowIndex.keys()).toEqual(
+      new Int32Array(matchingIndices)
+    );
+    expect(world.obsAnnotations.colIndex.keys()).toEqual(
+      universe.obsAnnotations.colIndex.keys()
+    );
+    expect(world.obsLayout.rowIndex.keys()).toEqual(
+      new Int32Array(matchingIndices)
+    );
+    expect(world.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
   });
 });
 
@@ -168,10 +137,13 @@ describe("createObsDimensionMap", () => {
     */
 
     const { dimensionMap } = defaultBigBang();
-
+    const annotationNames = _.map(
+      REST.schema.schema.annotations.obs,
+      c => c.name
+    );
     const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
     expect(dimensionMap).toBeDefined();
-    REST.annotationsObs.names.forEach(name => {
+    annotationNames.forEach(name => {
       const dim = dimensionMap[obsAnnoDimensionName(name)];
       if (name === "name") {
         expect(dim).toBeUndefined();
@@ -184,67 +156,20 @@ describe("createObsDimensionMap", () => {
         }
       }
     });
-    expect(dimensionMap[layoutDimensionName("X")]).toBeInstanceOf(
-      Crossfilter.ScalarDimension
-    );
-    expect(dimensionMap[layoutDimensionName("Y")]).toBeInstanceOf(
-      Crossfilter.ScalarDimension
+    expect(dimensionMap[layoutDimensionName("XY")]).toBeInstanceOf(
+      Crossfilter.SpatialDimension
     );
   });
 });
 
-describe("subsetVarData", () => {
-  test("when world eq universe", () => {
-    const { universe, world } = defaultBigBang();
-    /* create a mock varData array for subsetting */
-    const sourceVarData = new Float32Array(universe.nObs);
-
-    /* expect literally the same object back */
-    const result = World.subsetVarData(world, universe, sourceVarData);
-    expect(result).toBe(sourceVarData);
-  });
-
-  test("when world neq universe", () => {
-    const { universe, world, crossfilter, dimensionMap } = defaultBigBang();
-    /* create a mock varData array for subsetting */
-    const sourceVarData = Float32Array.from(_.range(universe.nObs));
-
-    /* mock a selection */
-    dimensionMap[obsAnnoDimensionName("field1")].filterRange([0, 5]);
-    dimensionMap[obsAnnoDimensionName("field3")].filterExact(false);
-
-    /* create the world from the selection */
-    const newWorld = World.createWorldFromCurrentSelection(
-      universe,
-      world,
-      crossfilter
-    );
-    expect(newWorld.obsIndex).toMatchObject(new Uint32Array([0, 2]));
-
-    /* expect a subset */
-    const result = World.subsetVarData(newWorld, universe, sourceVarData);
-    expect(result).not.toBe(sourceVarData);
-    expect(result).toHaveLength(newWorld.nObs);
-    /* check that we have expected source var content */
-    expect(result).toMatchObject(new Float32Array([0, 2]));
-  });
-});
-
-describe("createVarDimension", () => {
+describe("createVarDataDimension", () => {
   /* create default universe */
   const { world, crossfilter } = defaultBigBang();
-  /* create a mock var data cache */
-  const varDataCache = kvCache.set(
-    kvCache.create(),
+  world.varData = world.varData.withCol(
     "GENE",
     Float32Array.from(_.range(world.nObs))
   );
-  const result = World.createVarDimension(
-    world,
-    varDataCache,
-    crossfilter,
-    "GENE"
-  );
+  const result = World.createVarDataDimension(world, crossfilter, "GENE");
   expect(result).toBeInstanceOf(Crossfilter.ScalarDimension);
 });
 

client/__tests__/util/stateManager/worldUtil.test.js

@@ -0,0 +1,62 @@
+import {
+  countCategoryValues2D,
+  clearCaches
+} from "../../../src/util/stateManager/worldUtil";
+import * as Dataframe from "../../../src/util/dataframe";
+
+describe("WorldUtil cache management", () => {
+  test("empty", () => {
+    const count = countCategoryValues2D(
+      "a",
+      "b",
+      new Dataframe.Dataframe([0, 0], [])
+    );
+    expect(count).toMatchObject(new Map());
+    expect(count.size).toBe(0);
+  });
+
+  test("simple couts", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 2],
+      [[0, 0, 1], [false, true, false]],
+      null,
+      new Dataframe.KeyIndex(["a", "b"])
+    );
+    const count = countCategoryValues2D("a", "b", df);
+    expect(count).toMatchObject(
+      new Map([
+        [0, new Map([[true, 1], [false, 1]])],
+        [1, new Map([[false, 1]])]
+      ])
+    );
+  });
+
+  test("memo cache clear", () => {
+    clearCaches();
+    const df1 = new Dataframe.Dataframe([0, 0], []);
+    const df2 = new Dataframe.Dataframe(
+      [3, 2],
+      [[0, 0, 1], [false, true, false]],
+      null,
+      new Dataframe.KeyIndex(["a", "b"])
+    );
+
+    const count1 = countCategoryValues2D("a", "b", df1);
+    const count2 = countCategoryValues2D("a", "b", df1);
+    const count3 = countCategoryValues2D("a", "b", df1.clone());
+    const count4 = countCategoryValues2D("a", "b", df2);
+
+    clearCaches();
+    const count10 = countCategoryValues2D("a", "b", df1);
+    const count11 = countCategoryValues2D("a", "b", df2);
+
+    expect(count1).toEqual(count2);
+    expect(count1).toEqual(count3);
+    expect(count1).toEqual(count10);
+    expect(count1).not.toBe(count3);
+    expect(count1).not.toBe(count10);
+
+    expect(count4).toEqual(count11);
+    expect(count4).not.toBe(count11);
+  });
+});

client/__tests__/util/typedCrossfilter/bitArray.test.js

@@ -118,16 +118,16 @@ describe("selectionCount", () => {
     const dim2 = ba.allocDimension();
     expect(dim2).toBeDefined();
 
-    expect(ba.selectionCount).toEqual(0);
+    expect(ba.selectionCount()).toEqual(0);
     ba.selectAll(dim1);
-    expect(ba.selectionCount).toEqual(0);
+    expect(ba.selectionCount()).toEqual(0);
     ba.selectAll(dim2);
-    expect(ba.selectionCount).toEqual(defaultTestLength);
+    expect(ba.selectionCount()).toEqual(defaultTestLength);
 
     for (let i = 0; i < defaultTestLength; i += 1) {
       ba.deselectOne(dim1, i);
-      expect(ba.selectionCount).toEqual(defaultTestLength - i - 1);
-      expect(ba.selectionCount).toEqual(ba.countAllOnes());
+      expect(ba.selectionCount()).toEqual(defaultTestLength - i - 1);
+      expect(ba.selectionCount()).toEqual(ba.countAllOnes());
     }
 
     ba.freeDimension(dim1);

client/__tests__/util/typedCrossfilter/typedCrossfilter.test.js

@@ -102,33 +102,39 @@ const someData = [
 ];
 
 function groupReduce(data, valueMap, valueReduce, valueInit) {
-  return _
-    .reduce(
-      data,
-      (acc, value) => {
-        const k = valueMap(value);
-        let r = _.find(acc, o => o.key === k);
-        if (!r) {
-          r = { key: k, value: valueInit() };
-          acc.push(r);
-        }
-        r.value = valueReduce(r.value, value);
-        return acc;
-      },
-      []
-    )
-    .sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
+  return _.reduce(
+    data,
+    (acc, value) => {
+      const k = valueMap(value);
+      let r = _.find(acc, o => o.key === k);
+      if (!r) {
+        r = { key: k, value: valueInit() };
+        acc.push(r);
+      }
+      r.value = valueReduce(r.value, value);
+      return acc;
+    },
+    []
+  ).sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
 }
 
 function groupCount(data, map) {
-  return groupReduce(data, map, (p, v) => p + 1, () => 0);
+  return groupReduce(data, map, p => p + 1, () => 0);
 }
 
 function groupSum(data, map) {
-  return groupReduce(data, map, (p, v) => (p += map(v)), () => 0);
+  return groupReduce(
+    data,
+    map,
+    (p, v) => {
+      p += map(v);
+      return p;
+    },
+    () => 0
+  );
 }
 
-var payments = null;
+let payments = null;
 beforeEach(() => {
   payments = crossfilter(someData);
 });
@@ -139,7 +145,11 @@ describe("typedCrossfilter", () => {
     expect(payments.size()).toEqual(someData.length);
     expect(payments.all()).toEqual(someData);
 
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
     expect(quantity).toBeDefined();
     expect(quantity.id()).toBeDefined();
 
@@ -150,10 +160,25 @@ describe("typedCrossfilter", () => {
 
   test("filterAll and filterNone", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total,
+      Float32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     expect(quantity).toBeDefined();
     expect(tip).toBeDefined();
@@ -198,10 +223,20 @@ describe("typedCrossfilter", () => {
 
   test("filterExact", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Float32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     quantity.filterExact(1);
     expect(payments.countFiltered()).toEqual(
@@ -222,10 +257,25 @@ describe("typedCrossfilter", () => {
 
   test("filterRange", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total,
+      Float32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     tip.filterRange([0, 91]);
     expect(payments.allFiltered()).toEqual(
@@ -251,10 +301,25 @@ describe("typedCrossfilter", () => {
 
   test("filterEnum", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total,
+      Float32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     type.filterEnum(["tab", "cash"]);
     expect(payments.allFiltered()).toEqual(
@@ -274,15 +339,34 @@ describe("typedCrossfilter", () => {
 
   test("more than 32 dimensions", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total,
+      Float32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     // Create a bunch of fake dimensions to ensure we can handle > 32
     let dimMap = {};
     for (let i = 0; i < 65; i++) {
-      dimMap[i] = payments.dimension(r => Math.random(), Float32Array);
+      dimMap[i] = payments.dimension(
+        crossfilter.ScalarDimension,
+        () => Math.random(),
+        Float32Array
+      );
       expect(dimMap[i]).toBeDefined();
       expect(dimMap[i].id()).toBeDefined();
     }
@@ -304,10 +388,25 @@ describe("typedCrossfilter", () => {
   test("group, default mapping, default reducer, no filter", () => {
     expect(payments).toBeDefined();
 
-    var quantity = payments.dimension(r => r.quantity, Int32Array);
-    var tip = payments.dimension(r => r.tip, Int32Array);
-    var type = payments.dimension(r => r.type, "enum");
-    var total = payments.dimension(r => r.total, Int32Array);
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Int32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total,
+      Int32Array
+    );
 
     _.each(
       {
@@ -331,9 +430,20 @@ describe("typedCrossfilter", () => {
     // custom mapping in groups only works for scalar types.  Enums do not
     // currently implement it.
 
-    const tip = payments.dimension(r => r.tip, Int32Array);
-    const totalX10 = payments.dimension(r => r.total * 10, Int32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Int32Array
+    );
+    const totalX10 = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total * 10,
+      Int32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     const paymentsByTip_A = tip.group();
     const paymentsByTip_B = tip.group(r => 10 * r);
@@ -370,8 +480,15 @@ describe("typedCrossfilter", () => {
   test("group, default map, custom reducer, no filters", () => {
     expect(payments).toBeDefined();
 
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total,
+      Float32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     const paymentsByTotal = total.group();
     const paymentsByType = type.group();
@@ -407,9 +524,20 @@ describe("typedCrossfilter", () => {
 
     expect(payments).toBeDefined();
 
-    const tip = payments.dimension(r => r.tip, Int32Array);
-    const total = payments.dimension(r => r.total, Int32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].tip,
+      Int32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      (i, data) => data[i].total,
+      Int32Array
+    );
+    const type = payments.dimension(
+      crossfilter.EnumDimension,
+      (i, data) => data[i].type
+    );
 
     const paymentsByTip = tip.group();
     const paymentsByTotal = total.group();

client/configuration/eslint/eslint.js

@@ -34,7 +34,8 @@ module.exports = {
     "object-curly-newline": ["error", { consistent: true }],
     "react/prop-types": [0],
     "space-before-function-paren": "off",
-    "function-paren-newline": "off"
+    "function-paren-newline": "off",
+    "prefer-destructuring": ["error", { object: true, array: false }]
   },
   overrides: [
     {

client/configuration/webpack/webpack.config.prod.js

@@ -10,7 +10,7 @@ const nodeModules = path.resolve("node_modules");
 
 const babelOptions = require("../babel/babel.prod");
 
-const publicPath = "/";
+const publicPath = "";
 
 module.exports = {
   mode: "production",

client/index_template.html

@@ -24,7 +24,7 @@
 <script type="text/javascript">
 window.CELLXGENE = {};
 window.CELLXGENE.API = {
-  prefix: "{{ prefix | safe }}",
+  prefix: window.location.href + "api/",
   version: "v0.2/"
 };
 </script>

client/package.json

@@ -1,16 +1,20 @@
 {
   "name": "cellxgene",
-  "version": "0.0.2",
+  "version": "0.7.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
   "scripts": {
+    "backend-dev": "python3.6 -m venv cellxgene && source cellxgene/bin/activate && yes | pip uninstall cellxgene || true && pip install -e .. && cellxgene launch ",
     "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
-    "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
     "clean": "rimraf build",
-    "start": "node server/development.js",
+    "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
+    "e2e": "jest e2e",
     "lint": "eslint src",
-    "test": "jest"
+    "smoke-test": "start-server-and-test start-server-for-test :5000 e2e",
+    "start": "node server/development.js",
+    "start-server-for-test": "cellxgene launch -p 5000 ../example-dataset/pbmc3k.h5ad",
+    "unit-test": "jest --testPathIgnorePatterns e2e"
   },
   "engineStrict": true,
   "engines": {
@@ -19,10 +23,9 @@
   "eslintConfig": {
     "extends": "./configuration/eslint/eslint.js"
   },
-  "nyc": {
-    "sourceMap": false,
-    "instrument": false
-  },
+  "eslintIgnore": [
+    "src/util/stateManager/matrix_generated.js"
+  ],
   "resolutions": {
     "eslint-scope": "3.7.1"
   },
@@ -33,9 +36,12 @@
     "canvas-fit": "^1.5.0",
     "d3": "^4.10.0",
     "d3-scale-chromatic": "^1.3.0",
+    "flatbuffers": "^1.10.2",
     "font-color-contrast": "^1.0.3",
+    "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
     "gl-matrix": "^2.7.1",
+    "is-number": "^7.0.0",
     "key-pressed": "0.0.1",
     "lodash": "^4.17.4",
     "memoize-one": "^4.0.0",
@@ -73,31 +79,31 @@
     "babel-eslint": "^10.0.1",
     "babel-jest": "^23.6.0",
     "babel-loader": "^8.0.0",
-    "babel-plugin-istanbul": "^5.1.0",
     "babel-preset-modern-browsers": "^12.0.0",
-    "chalk": "^2.4.1",
-    "connect-history-api-fallback": "^1.3.0",
+    "chalk": "^2.4.2",
+    "connect-history-api-fallback": "^1.6.0",
     "copy-webpack-plugin": "^4.6.0",
     "css-loader": "^1.0.1",
-    "eslint": "^5.8.0",
+    "eslint": "^5.13.0",
     "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^3.1.0",
-    "eslint-loader": "^2.1.1",
+    "eslint-config-prettier": "^4.0.0",
+    "eslint-loader": "^2.1.2",
     "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.14.0",
-    "eslint-plugin-jest": "^21.27.2",
-    "eslint-plugin-jsx-a11y": "^6.1.1",
-    "eslint-plugin-react": "^7.11.1",
+    "eslint-plugin-import": "^2.16.0",
+    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-jsx-a11y": "^6.2.1",
+    "eslint-plugin-react": "^7.12.4",
     "express": "^4.14.0",
     "file-loader": "^2.0.0",
     "html-webpack-inline-source-plugin": "0.0.10",
     "html-webpack-plugin": "^3.2.0",
-    "jest": "^23.5.0",
+    "jest": "^24.1.0",
     "json-loader": "^0.5.4",
     "mini-css-extract-plugin": "^0.4.1",
-    "nyc": "^13.0.1",
-    "rimraf": "^2.5.4",
+    "puppeteer": "^1.12.1",
+    "rimraf": "^2.6.3",
     "serve-favicon": "^2.3.0",
+    "start-server-and-test": "^1.7.11",
     "style-loader": "^0.23.1",
     "sw-precache-webpack-plugin": "^0.11.5",
     "url-loader": "^1.1.0",
@@ -109,7 +115,10 @@
     "testMatch": [
       "**/__tests__/**/?(*.)(spec|test).js?(x)"
     ],
-    "testURL": "http://localhost/"
+    "testURL": "http://localhost/",
+    "setupFiles": [
+      "./__tests__/setupMissingGlobals.js"
+    ]
   },
   "babel": {
     "env": {

client/src/actions/index.js

@@ -1,11 +1,11 @@
 // jshint esversion: 6
 import _ from "lodash";
 import * as globals from "../globals";
-import { Universe, kvCache } from "../util/stateManager";
+import { Universe } from "../util/stateManager";
 import {
   catchErrorsWrap,
   doJsonRequest,
-  rangeEncodeIndices,
+  doBinaryRequest,
   dispatchNetworkErrorMessageToUser
 } from "../util/actionHelpers";
 
@@ -13,29 +13,33 @@ import {
 Bootstrap application with the initial data loading.
   * /config - application configuration
   * /schema - schema of dataframe
-  * /annotations/obs - all metadata annotation
+  * /annotations - all metadata annotation
+  * /layout - all default layout
 */
 const doInitialDataLoad = () =>
   catchErrorsWrap(async dispatch => {
     dispatch({ type: "initial data load start" });
 
     try {
-      const requests = _([
-        "config",
-        "schema",
-        "annotations/obs",
-        "annotations/var",
-        "layout/obs"
-      ])
+      const requestJson = _(["config", "schema"])
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
         .map(url => doJsonRequest(url))
         .value();
-      const results = await Promise.all(requests);
+      const requestBinary = _([
+        "annotations/obs",
+        "annotations/var?annotation-name=name",
+        "layout/obs"
+      ])
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doBinaryRequest(url))
+        .value();
+
+      const results = await Promise.all(_.concat(requestJson, requestBinary));
 
       /* set config defaults */
       const config = { ...globals.configDefaults, ...results[0].config };
       const [, schema, obsAnno, varAnno, obsLayout] = [...results];
-      const universe = Universe.createUniverseFromRestV02Response(
+      const universe = Universe.createUniverseFromResponse(
         config,
         schema,
         obsAnno,
@@ -87,15 +91,47 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  /* helper for this function only */
+  const fetchData = async geneNames => {
+    const res = await fetch(
+      `${globals.API.prefix}${globals.API.version}data/var`,
+      {
+        method: "PUT",
+        body: JSON.stringify({
+          filter: {
+            var: {
+              annotation_value: [{ name: "name", values: geneNames }]
+            }
+          }
+        }),
+        headers: new Headers({
+          accept: "application/octet-stream",
+          "Content-Type": "application/json"
+        })
+      }
+    );
+
+    if (
+      !res.ok ||
+      res.headers.get("Content-Type") !== "application/octet-stream"
+    ) {
+      // WILL throw
+      return dispatchExpressionErrors(dispatch, res);
+    }
+
+    const data = await res.arrayBuffer();
+    return Universe.convertDataFBStoObject(universe, data);
+  };
+
   const state = getState();
   const { universe } = state.controls;
   /* preload data already in cache */
   let expressionData = _.transform(
     genes,
     (expData, g) => {
-      const data = kvCache.get(universe.varDataCache, g);
+      const data = universe.varData.col(g);
       if (data) {
-        expData[g] = data;
+        expData[g] = data.asArray();
       }
     },
     {}
@@ -108,35 +144,10 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
   /* Fetch data for any genes not in cache */
   if (genesToFetch.length) {
     try {
-      // XXX: TODO - this could be using /data/var rather than /data/obs,
-      // as that would simplify the transformation in convertExpressionRESTv02ToObject
-      const res = await fetch(
-        `${globals.API.prefix}${globals.API.version}data/obs`,
-        {
-          method: "PUT",
-          body: JSON.stringify({
-            filter: {
-              var: {
-                annotation_value: [{ name: "name", values: genesToFetch }]
-              }
-            }
-          }),
-          headers: new Headers({
-            accept: "application/json",
-            "Content-Type": "application/json"
-          })
-        }
-      );
-
-      if (!res.ok || res.headers.get("Content-Type") !== "application/json") {
-        // WILL throw
-        return dispatchExpressionErrors(dispatch, res);
-      }
-
-      const data = await res.json();
+      const newExpressionData = await fetchData(genesToFetch);
       expressionData = {
         ...expressionData,
-        ...Universe.convertExpressionRESTv02ToObject(universe, data)
+        ...newExpressionData
       };
     } catch (error) {
       dispatch({ type: "expression load error", error });
@@ -158,7 +169,7 @@ function requestSingleGeneExpressionCountsForColoringPOST(gene) {
         type: "color by expression",
         gene,
         data: {
-          [gene]: kvCache.get(world.varDataCache, gene)
+          [gene]: world.varData.col(gene).asArray()
         }
       });
     } catch (error) {
@@ -181,7 +192,7 @@ const requestUserDefinedGene = gene => async (dispatch, getState) => {
       type: "request user defined gene success",
       data: {
         genes: [gene],
-        expression: kvCache.get(world.varDataCache, gene)
+        expression: world.varData.col(gene).asArray()
       }
     });
   } catch (error) {
@@ -230,12 +241,18 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     */
     const state = getState();
     const { universe } = state.controls;
-    const set1ByIndex = rangeEncodeIndices(
-      _.map(set1, s => universe.obsNameToIndexMap[s])
-    );
-    const set2ByIndex = rangeEncodeIndices(
-      _.map(set2, s => universe.obsNameToIndexMap[s])
-    );
+
+    // Legal values are null, Array or TypedArray.  Null is initial state.
+    if (!set1) set1 = [];
+    if (!set2) set2 = [];
+
+    // These lines ensure that we convert any TypedArray to an Array.
+    // This is necessary because JSON.stringify() does some very strange
+    // things with TypedArrays (they are marshalled to JSON objects, rather
+    // than being marshalled as a JSON array).
+    set1 = Array.isArray(set1) ? set1 : Array.from(set1);
+    set2 = Array.isArray(set2) ? set2 : Array.from(set2);
+
     const res = await fetch(
       `${globals.API.prefix}${globals.API.version}diffexp/obs`,
       {
@@ -247,8 +264,8 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
         body: JSON.stringify({
           mode: "topN",
           count: num_genes,
-          set1: { filter: { obs: { index: set1ByIndex } } },
-          set2: { filter: { obs: { index: set2ByIndex } } }
+          set1: { filter: { obs: { index: set1 } } },
+          set2: { filter: { obs: { index: set2 } } }
         })
       }
     );
@@ -259,7 +276,9 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
 
     const data = await res.json();
     // result is [ [varIdx, ...], ... ]
-    const topNGenes = _.map(data, r => universe.varAnnotations[r[0]].name);
+    const topNGenes = _.map(data, r =>
+      universe.varAnnotations.at(r[0], "name")
+    );
 
     /*
     Kick off secondary action to fetch all of the expression data for the

client/src/components/brushableHistogram/index.js

@@ -10,9 +10,9 @@ import { Button, ButtonGroup, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
 import * as d3 from "d3";
 import memoize from "memoize-one";
-import { kvCache } from "../../util/stateManager";
 import * as globals from "../../globals";
 import actions from "../../actions";
+import finiteExtent from "../../util/finiteExtent";
 
 @connect(state => ({
   world: state.controls.world,
@@ -20,13 +20,12 @@ import actions from "../../actions";
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
   crossfilter: state.controls.crossfilter,
   differential: state.differential,
-  initializeRanges: _.get(state.controls.world, "summary.obs"),
   colorAccessor: state.controls.colorAccessor,
   colorScale: state.controls.colorScale,
   obsAnnotations: _.get(state.controls.world, "obsAnnotations", null)
 }))
 class HistogramBrush extends React.Component {
-  calcHistogramCache = memoize((obsAnnotations, field, ranges) => {
+  calcHistogramCache = memoize((obsAnnotations, field, rangeMin, rangeMax) => {
     const { world } = this.props;
     const histogramCache = {};
 
@@ -34,13 +33,15 @@ class HistogramBrush extends React.Component {
       .scaleLinear()
       .range([this.height - this.marginBottom, 0]);
 
-    if (obsAnnotations[0][field]) {
+    if (obsAnnotations.hasCol(field)) {
       // recalculate expensive stuff
-      const allValuesForContinuousFieldAsArray = _.map(obsAnnotations, field);
+      const allValuesForContinuousFieldAsArray = obsAnnotations
+        .col(field)
+        .asArray();
 
       histogramCache.x = d3
         .scaleLinear()
-        .domain([ranges.min, ranges.max])
+        .domain([rangeMin, rangeMax])
         .range([0, this.width]);
 
       histogramCache.bins = d3
@@ -49,14 +50,13 @@ class HistogramBrush extends React.Component {
         .thresholds(40)(allValuesForContinuousFieldAsArray);
 
       histogramCache.numValues = allValuesForContinuousFieldAsArray.length;
-    } else if (kvCache.get(world.varDataCache, field)) {
-      /* it's not in observations, so it's a gene, but let's check to make sure */
-      const varValues = kvCache.get(world.varDataCache, field);
+    } else if (world.varData.hasCol(field)) {
+      const varValues = world.varData.col(field).asArray();
 
       histogramCache.x = d3
         .scaleLinear()
         .domain(
-          d3.extent(varValues)
+          finiteExtent(varValues)
         ) /* replace this if we have ranges for genes back from server like we do for annotations on cells */
         .range([0, this.width]);
 
@@ -130,7 +130,8 @@ class HistogramBrush extends React.Component {
     const histogramCache = this.calcHistogramCache(
       obsAnnotations,
       field,
-      ranges
+      ranges.min,
+      ranges.max
     );
 
     const { x, y, bins, numValues } = histogramCache;
@@ -139,27 +140,27 @@ class HistogramBrush extends React.Component {
   }
 
   handleColorAction() {
-    const {
-      obsAnnotations,
-      dispatch,
-      field,
-      world,
-      initializeRanges
-    } = this.props;
+    const { obsAnnotations, dispatch, field, world, ranges } = this.props;
 
-    if (obsAnnotations[0][field]) {
+    if (obsAnnotations.hasCol(field)) {
       dispatch({
         type: "color by continuous metadata",
         colorAccessor: field,
-        rangeMaxForColorAccessor: initializeRanges[field].range.max
+        rangeForColorAccessor: ranges
       });
-    } else if (kvCache.get(world.varDataCache, field)) {
+    } else if (world.varData.hasCol(field)) {
       dispatch(actions.requestSingleGeneExpressionCountsForColoringPOST(field));
     }
   }
 
   removeHistogram() {
-    const { dispatch, field, colorAccessor } = this.props;
+    const {
+      dispatch,
+      field,
+      colorAccessor,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor
+    } = this.props;
     dispatch({
       type: "clear user defined gene",
       data: field
@@ -169,6 +170,18 @@ class HistogramBrush extends React.Component {
         type: "reset colorscale"
       });
     }
+    if (field === scatterplotXXaccessor) {
+      dispatch({
+        type: "set scatterplot x",
+        data: null
+      });
+    }
+    if (field === scatterplotYYaccessor) {
+      dispatch({
+        type: "set scatterplot y",
+        data: null
+      });
+    }
   }
 
   handleSetGeneAsScatterplotX() {
@@ -215,6 +228,7 @@ class HistogramBrush extends React.Component {
     d3.select(svgRef)
       .append("g")
       .attr("class", "brush")
+      .attr("data-testid", `${svgRef.id}-brush`)
       .call(
         d3
           .brushX()
@@ -259,13 +273,15 @@ class HistogramBrush extends React.Component {
     return (
       <div
         id={`histogram_${field}`}
+        data-testid={`histogram-${field}`}
+        data-testclass={isDiffExp ? `histogram-diffexp` : ""}
         style={{
           padding: globals.leftSidebarSectionPadding,
           backgroundColor: zebra ? globals.lightestGrey : "white"
         }}
       >
         <div style={{ display: "flex", justifyContent: "flex-end" }}>
-          {isDiffExp ? (
+          {isDiffExp || isUserDefined ? (
             <span>
               <span
                 style={{ marginRight: 7 }}

client/src/components/categorical/categorical.js

@@ -5,34 +5,13 @@ import { connect } from "react-redux";
 import * as globals from "../../globals";
 import Category from "./category";
 
-/* Cap the max number of displayed categories */
-const truncateCategories = options => {
-  const numOptions = _.size(options);
-  if (numOptions <= globals.maxCategoricalOptionsToDisplay) {
-    return options;
-  }
-  return _(options)
-    .map((v, k) => ({ name: k, val: v }))
-    .sortBy("val")
-    .slice(numOptions - globals.maxCategoricalOptionsToDisplay)
-    .transform((r, v) => {
-      r[v.name] = v.val;
-    }, {})
-    .value();
-};
-
 @connect(state => ({
-  ranges: _.get(state.controls.world, "summary.obs", null),
-  categorySelectionLimit: _.get(
-    state.config,
-    "parameters.max-category-items",
-    globals.configDefaults.parameters["max-category-items"]
-  )
+  categoricalSelectionState: state.controls.categoricalSelectionState
 }))
 class Categories extends React.Component {
   render() {
-    const { ranges, categorySelectionLimit } = this.props;
-    if (!ranges) return null;
+    const { categoricalSelectionState } = this.props;
+    if (!categoricalSelectionState) return null;
 
     return (
       <div
@@ -47,27 +26,9 @@ class Categories extends React.Component {
         >
           Categorical Metadata
         </p>
-        {_.map(ranges, (value, key) => {
-          const isColorField = key.includes("color") || key.includes("Color");
-          const isSelectableCategory =
-            value.options &&
-            !isColorField &&
-            key !== "name" &&
-            value.numOptions < categorySelectionLimit;
-
-          if (isSelectableCategory) {
-            const categoryOptions = truncateCategories(value.options);
-            return (
-              <Category
-                key={key}
-                metadataField={key}
-                values={categoryOptions}
-                isTruncated={categoryOptions !== value.options}
-              />
-            );
-          }
-          return undefined;
-        })}
+        {_.map(categoricalSelectionState, (catState, catName) => (
+          <Category key={catName} metadataField={catName} />
+        ))}
       </div>
     );
   }

client/src/components/categorical/category.js

@@ -2,29 +2,15 @@ import React from "react";
 import _ from "lodash";
 import { connect } from "react-redux";
 import { FaChevronRight, FaChevronDown } from "react-icons/fa";
-import memoize from "memoize-one";
-import { Button, Tooltip, Position } from "@blueprintjs/core";
+import { Button, Tooltip } from "@blueprintjs/core";
 
 import * as globals from "../../globals";
 import Value from "./value";
-import alphabeticallySortedValues from "./util";
-
-const countCategories = (values, optsAsBools) =>
-  _.reduce(
-    values,
-    (r, v, k) => {
-      r.total += 1;
-      if (optsAsBools[k]) {
-        r.on += 1;
-      }
-      return r;
-    },
-    { total: 0, on: 0 }
-  );
+import sortedCategoryValues from "./util";
 
 @connect(state => ({
   colorAccessor: state.controls.colorAccessor,
-  categoricalAsBooleansMap: state.controls.categoricalAsBooleansMap
+  categoricalSelectionState: state.controls.categoricalSelectionState
 }))
 class Category extends React.Component {
   constructor(props) {
@@ -33,24 +19,28 @@ class Category extends React.Component {
       isChecked: true,
       isExpanded: false
     };
-    this.countCategories = memoize((values, optsAsBools) =>
-      countCategories(values, optsAsBools)
-    );
   }
 
   componentDidUpdate() {
-    const { categoricalAsBooleansMap, metadataField, values } = this.props;
-    const categoryCount = this.countCategories(
-      values,
-      categoricalAsBooleansMap[metadataField]
-    );
-    if (categoryCount.on === categoryCount.total) {
+    const { categoricalSelectionState, metadataField } = this.props;
+    const cat = categoricalSelectionState[metadataField];
+    const categoryCount = {
+      // total number of categories in this dimension
+      totalCatCount: cat.numCategories,
+      // number of selected options in this category
+      selectedCatCount: _.reduce(
+        cat.categorySelected,
+        (res, cond) => (cond ? res + 1 : res),
+        0
+      )
+    };
+    if (categoryCount.selectedCatCount === categoryCount.totalCatCount) {
       /* everything is on, so not indeterminate */
       this.checkbox.indeterminate = false;
-    } else if (categoryCount.on === 0) {
+    } else if (categoryCount.selectedCatCount === 0) {
       /* nothing is on, so no */
       this.checkbox.indeterminate = false;
-    } else if (categoryCount.on < categoryCount.total) {
+    } else if (categoryCount.selectedCatCount < categoryCount.totalCatCount) {
       /* to be explicit... */
       this.checkbox.indeterminate = true;
     }
@@ -74,11 +64,10 @@ class Category extends React.Component {
   }
 
   toggleNone() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField } = this.props;
     dispatch({
       type: "categorical metadata filter none of these",
-      metadataField,
-      value
+      metadataField
     });
     this.setState({ isChecked: false });
   }
@@ -94,13 +83,16 @@ class Category extends React.Component {
   }
 
   renderCategoryItems() {
-    const { values, metadataField } = this.props;
-    return _.map(alphabeticallySortedValues(values), (v, i) => (
+    const { categoricalSelectionState, metadataField } = this.props;
+
+    const cat = categoricalSelectionState[metadataField];
+    const optTuples = sortedCategoryValues([...cat.categoryIndices]);
+    return _.map(optTuples, (tuple, i) => (
       <Value
-        key={v}
+        optTuples={optTuples}
+        key={tuple[1]}
         metadataField={metadataField}
-        count={values[v]}
-        value={v}
+        categoryIndex={tuple[1]}
         i={i}
       />
     ));
@@ -108,12 +100,15 @@ class Category extends React.Component {
 
   render() {
     const { isExpanded, isChecked } = this.state;
-    const { metadataField, colorAccessor, isTruncated } = this.props;
+    const {
+      metadataField,
+      colorAccessor,
+      categoricalSelectionState
+    } = this.props;
+    const { isTruncated } = categoricalSelectionState[metadataField];
     return (
       <div
         style={{
-          // display: "flex",
-          // alignItems: "baseline",
           maxWidth: globals.maxControlsWidth
         }}
       >

client/src/components/categorical/occupancy.js

@@ -0,0 +1,73 @@
+// jshint esversion: 6
+import React from "react";
+import _ from "lodash";
+import { connect } from "react-redux";
+import * as d3 from "d3";
+
+@connect()
+class Occupancy extends React.Component {
+  render() {
+    const {
+      occupancy,
+      colorScale,
+      categoricalSelectionState,
+      colorAccessor,
+      schema
+    } = this.props;
+    const width = 100;
+    const height = 11;
+
+    const categories = _.filter(schema.annotations.obs, {
+      name: colorAccessor
+    })[0].categories;
+
+    const x = d3
+      .scaleLinear()
+      /* get all the keys d[1] as an array, then find the sum */
+      .domain([0, d3.sum(Array.from(occupancy, d => d[1]))])
+      .range([0, width]);
+
+    let currentOffset = 0;
+
+    const stacks = categoricalSelectionState[colorAccessor].categoryValues.map(
+      d => {
+        const o = occupancy.get(d);
+
+        const scaledValue = x(o);
+
+        const stackItem = {
+          key: d,
+          value: o || 0,
+          rectWidth: o ? scaledValue : 0,
+          offset: currentOffset,
+          fill: o ? colorScale(categories.indexOf(d)) : "rgb(255,255,255)"
+        };
+        currentOffset += o ? scaledValue : 0;
+        return stackItem;
+      }
+    );
+
+    return (
+      <svg
+        style={{
+          marginRight: 5,
+          width,
+          height
+        }}
+      >
+        {stacks.map(d => (
+          <rect
+            key={d.key}
+            width={d.rectWidth}
+            height={height}
+            x={d.offset}
+            title={d.metadataField}
+            fill={d.fill}
+          />
+        ))}
+      </svg>
+    );
+  }
+}
+
+export default Occupancy;

client/src/components/categorical/util.js

@@ -1,7 +1,35 @@
 // jshint esversion: 6
-export default values =>
-  Object.keys(values).sort((a, b) => {
-    const textA = a.toUpperCase();
-    const textB = b.toUpperCase();
+
+// values is [ [optVal, optIdx], ...]
+// index is range array
+// return sorted index
+
+import isNumber from "is-number";
+import _ from "lodash";
+
+const sortedCategoryValues = values => {
+  /* this sort could be memoized for perf */
+
+  const strings = [];
+  const ints = [];
+
+  _.forEach(values, v => {
+    if (isNumber(v[0])) {
+      ints.push(v);
+    } else {
+      strings.push(v);
+    }
+  });
+
+  strings.sort((a, b) => {
+    const textA = String(a[0]).toUpperCase();
+    const textB = String(b[0]).toUpperCase();
     return textA < textB ? -1 : textA > textB ? 1 : 0;
   });
+
+  ints.sort((a, b) => +a[0] - +b[0]);
+
+  return ints.concat(strings);
+};
+
+export default sortedCategoryValues;

client/src/components/categorical/value.js

@@ -1,47 +1,81 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
+import _ from "lodash";
+import Occupancy from "./occupancy";
+import { countCategoryValues2D } from "../../util/stateManager/worldUtil";
+import * as globals from "../../globals";
 
 @connect(state => ({
-  categoricalAsBooleansMap: state.controls.categoricalAsBooleansMap,
+  categoricalSelectionState: state.controls.categoricalSelectionState,
   colorScale: state.controls.colorScale,
-  colorAccessor: state.controls.colorAccessor
+  colorAccessor: state.controls.colorAccessor,
+  schema: _.get(state.controls.world, "schema", null),
+  world: state.controls.world
 }))
 class CategoryValue extends React.Component {
   toggleOff() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter deselect",
       metadataField,
-      value
+      categoryIndex
     });
   }
 
   toggleOn() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter select",
       metadataField,
-      value
+      categoryIndex
     });
   }
 
   render() {
     const {
-      categoricalAsBooleansMap,
+      categoricalSelectionState,
       metadataField,
-      count,
-      value,
+      categoryIndex,
       colorAccessor,
       colorScale,
-      i
+      i,
+      schema,
+      world
     } = this.props;
 
-    if (!categoricalAsBooleansMap) return null;
+    if (!categoricalSelectionState) return null;
+
+    const category = categoricalSelectionState[metadataField];
+    const selected = category.categorySelected[categoryIndex];
+    const count = category.categoryCounts[categoryIndex];
+    const value = category.categoryValues[categoryIndex];
+    const displayString = String(
+      category.categoryValues[categoryIndex]
+    ).valueOf();
 
-    const selected = categoricalAsBooleansMap[metadataField][value];
     /* this is the color scale, so add swatches below */
-    const c = metadataField === colorAccessor;
+    const isColorBy = metadataField === colorAccessor;
+    let categories = null;
+    let occupancy = null;
+
+    if (isColorBy && schema) {
+      categories = _.filter(schema.annotations.obs, {
+        name: colorAccessor
+      })[0].categories;
+    }
+
+    if (
+      colorAccessor &&
+      !isColorBy &&
+      categoricalSelectionState[colorAccessor]
+    ) {
+      occupancy = countCategoryValues2D(
+        metadataField,
+        colorAccessor,
+        world.obsAnnotations
+      );
+    }
 
     return (
       <div
@@ -56,7 +90,10 @@ class CategoryValue extends React.Component {
           style={{
             margin: 0,
             padding: 0,
-            userSelect: "none"
+            userSelect: "none",
+            width: globals.leftSidebarWidth - 130,
+            display: "flex",
+            justifyContent: "space-between"
           }}
         >
           <label className="bp3-control bp3-checkbox">
@@ -68,8 +105,20 @@ class CategoryValue extends React.Component {
               type="checkbox"
             />
             <span className="bp3-control-indicator" />
-            {value}
+            {displayString}
           </label>
+          <span style={{ flexShrink: 0 }}>
+            {colorAccessor &&
+            !isColorBy &&
+            categoricalSelectionState[colorAccessor] ? (
+              <Occupancy
+                occupancy={occupancy.get(
+                  category.categoryValues[categoryIndex]
+                )}
+                {...this.props}
+              />
+            ) : null}
+          </span>
         </div>
         <span>
           <span>{count}</span>
@@ -78,7 +127,10 @@ class CategoryValue extends React.Component {
               marginLeft: 5,
               width: 11,
               height: 11,
-              backgroundColor: c ? colorScale(value) : "inherit"
+              backgroundColor:
+                isColorBy && categories
+                  ? colorScale(categories.indexOf(value))
+                  : "inherit"
             }}
           />
         </span>

client/src/components/continuous/continuous.js

@@ -9,8 +9,7 @@ import * as globals from "../../globals";
 import HistogramBrush from "../brushableHistogram";
 
 @connect(state => ({
-  ranges: _.get(state.controls.world, "summary.obs", null),
-  metadata: _.get(state.controls.world, "obsAnnotations", null),
+  obsAnnotations: _.get(state.controls.world, "obsAnnotations", null),
   colorAccessor: state.controls.colorAccessor,
   colorScale: state.controls.colorScale,
   selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null),
@@ -29,17 +28,18 @@ class Continuous extends React.Component {
 
   handleColorAction(key) {
     return () => {
-      const { dispatch, ranges } = this.props;
+      const { dispatch, obsAnnotations } = this.props;
+      const summary = obsAnnotations.col(key).summarize();
       dispatch({
         type: "color by continuous metadata",
         colorAccessor: key,
-        rangeMaxForColorAccessor: ranges[key].range.max
+        rangeForColorAccessor: summary
       });
     };
   }
 
   render() {
-    const { ranges, obsAnnotations, schema } = this.props;
+    const { obsAnnotations, schema } = this.props;
     if (schema && !this.continuousChecked) {
       this.hasContinuous = _.some(
         schema.annotations.obs,
@@ -63,24 +63,34 @@ class Continuous extends React.Component {
             Continuous metadata
           </p>
         ) : null}
-        {_.map(ranges, (value, key) => {
-          const isColorField = key.includes("color") || key.includes("Color");
-          zebra += 1;
-          if (value.range && key !== "name" && !isColorField) {
-            return (
-              <HistogramBrush
-                key={key}
-                field={key}
-                isObs
-                zebra={zebra % 2 === 0}
-                fieldValues={obsAnnotations}
-                ranges={value.range}
-                handleColorAction={this.handleColorAction(key).bind(this)}
-              />
-            );
-          }
-          return null;
-        })}
+        {obsAnnotations
+          ? _.map(obsAnnotations.colIndex.keys(), key => {
+              const summary = obsAnnotations.col(key).summarize();
+              const isColorField =
+                key.includes("color") || key.includes("Color");
+              const nonFiniteExtent =
+                summary.min === undefined || summary.max === undefined;
+              zebra += 1;
+              if (
+                !summary.categorical &&
+                key !== "name" &&
+                !isColorField &&
+                !nonFiniteExtent
+              ) {
+                return (
+                  <HistogramBrush
+                    key={key}
+                    field={key}
+                    isObs
+                    zebra={zebra % 2 === 0}
+                    ranges={summary}
+                    handleColorAction={this.handleColorAction(key).bind(this)}
+                  />
+                );
+              }
+              return null;
+            })
+          : null}
       </div>
     );
   }

client/src/components/continuousLegend/index.js

@@ -2,7 +2,7 @@
 import React from "react";
 import { connect } from "react-redux";
 import * as d3 from "d3";
-import { interpolateViridis } from "d3-scale-chromatic";
+import { interpolateViridis, interpolateCool } from "d3-scale-chromatic";
 
 // create continuous color legend
 // http://bl.ocks.org/syntagmatic/e8ccca52559796be775553b467593a9f
@@ -121,12 +121,12 @@ class ContinuousLegend extends React.Component {
         .remove();
     }
 
-    if (colorAccessor && colorScale) {
+    if (colorAccessor && colorScale && colorScale.range) {
       /* fragile! continuous range is 0 to 1, not [#fa4b2c, ...], make this a flag? */
       if (colorScale.range()[0][0] !== "#") {
         continuous(
           "#continuous_legend",
-          d3.scaleSequential(interpolateViridis).domain(colorScale.domain()),
+          d3.scaleSequential(interpolateCool).domain(colorScale.domain()),
           colorAccessor
         );
       }

client/src/components/framework/toasters.js

@@ -13,6 +13,13 @@ A "user" error - eg, bad input
 export const postUserErrorToast = message =>
   ErrorToastTopCenter.show({ message, intent: Intent.WARNING });
 
+/*
+A toast the user must dismiss manually, because they need to act on its information,
+ie., 8 bulk add genes out of 40 were bad. Manually see which ones and fix.
+*/
+export const keepAroundErrorToast = message =>
+  ErrorToastTopCenter.show({ message, timeout: 0, intent: Intent.WARNING });
+
 /*
 a hard network error
 */

client/src/components/geneExpression/cellSetButtons.js

@@ -1,8 +1,8 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
-import { Button, Tooltip } from "@blueprintjs/core";
+import { AnchorButton, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
+import { World } from "../../util/stateManager";
 
 @connect()
 class CellSetButton extends React.Component {
@@ -14,7 +14,11 @@ class CellSetButton extends React.Component {
       eitherCellSetOneOrTwo
     } = this.props;
 
-    const set = _.map(crossfilter.allFiltered(), "name");
+    // Reducer and components assume that value will be null if
+    // no selection made.  World..getSelectedByIndex() returns a
+    // zero length TypedArray when nothing is selected.
+    let set = World.getSelectedByIndex(crossfilter);
+    if (set.length === 0) set = null;
 
     if (!differential.diffExp) {
       /* diffexp needs to be cleared before we store a new set */
@@ -33,17 +37,19 @@ class CellSetButton extends React.Component {
         content="Save current selection for differential expression computation"
         position="top"
       >
-        <Button
+        <AnchorButton
           style={{ marginRight: 10 }}
           type="button"
+          disabled={differential.diffExp}
           onClick={this.set.bind(this)}
+          data-testid={`cellset-button-${eitherCellSetOneOrTwo}`}
         >
           {eitherCellSetOneOrTwo}
           {": "}
           {differential[cellListName]
             ? `${differential[cellListName].length} cells`
             : "0 cells"}
-        </Button>
+        </AnchorButton>
       </Tooltip>
     );
   }

client/src/components/geneExpression/expressionButtons.js

@@ -68,6 +68,8 @@ class Expression extends React.Component {
               style={{ marginTop: 10 }}
               disabled={!haveBothCellSets}
               intent="primary"
+              data-testid="diffexp-button"
+              loading={differential.loading}
               fill
               type="button"
               onClick={this.computeDiffExp.bind(this)}

client/src/components/geneExpression/index.js

@@ -3,28 +3,65 @@
 
 import React from "react";
 import _ from "lodash";
-import * as d3 from "d3";
+import fuzzysort from "fuzzysort";
+
 import { connect } from "react-redux";
-import { Button, Tooltip } from "@blueprintjs/core";
+import {
+  MenuItem,
+  Button,
+  FormGroup,
+  InputGroup,
+  ControlGroup
+} from "@blueprintjs/core";
+import { Suggest } from "@blueprintjs/select";
 import HistogramBrush from "../brushableHistogram";
 import * as globals from "../../globals";
 import actions from "../../actions";
-import { postUserErrorToast } from "../framework/toasters";
+import {
+  postUserErrorToast,
+  keepAroundErrorToast
+} from "../framework/toasters";
 import ExpressionButtons from "./expressionButtons";
+import finiteExtent from "../../util/finiteExtent";
+
+const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
+  if (!modifiers.matchesPredicate) {
+    return null;
+  }
+  /* the fuzzysort wraps the object with other properties, like a score */
+  const geneName = fuzzySortResult.target;
+
+  return (
+    <MenuItem
+      active={modifiers.active}
+      disabled={modifiers.disabled}
+      // Use of annotations in this way is incorrect and dataset specific.
+      // See https://github.com/chanzuckerberg/cellxgene/issues/483
+      // label={gene.n_counts}
+      key={geneName}
+      onClick={g =>
+        /* this fires when user clicks a menu item */
+        handleClick(g)
+      }
+      text={geneName}
+    />
+  );
+};
+
+const filterGenes = (query, genes) =>
+  /* fires on load, once, and then for each character typed into the input */
+  fuzzysort.go(query, genes, {
+    limit: 5,
+    threshold: -10000 // don't return bad results
+  });
 
 @connect(state => {
-  const metadata = _.get(state.controls.world, "obsAnnotations", null);
-  const ranges = _.get(state.controls.world, "summary.obs", null);
-  const initializeRanges = _.get(state.controls.world, "summary.obs");
-
   return {
-    ranges,
-    metadata,
-    initializeRanges,
+    obsAnnotations: _.get(state.controls.world, "obsAnnotations", null),
     userDefinedGenes: state.controls.userDefinedGenes,
+    userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
     world: state.controls.world,
     colorAccessor: state.controls.colorAccessor,
-    allGeneNames: state.controls.allGeneNames,
     differential: state.differential
   };
 })
@@ -32,27 +69,52 @@ class GeneExpression extends React.Component {
   constructor(props) {
     super(props);
     this.state = {
-      gene: ""
+      bulkAdd: "",
+      tab: "autosuggest"
     };
   }
 
-  keyPress(e) {
-    if (e.keyCode === 13) {
-      this.handleClick();
+  placeholderGeneNames() {
+    /*
+    return a string containing gene name suggestions for use as a user hint.
+    Eg.,    Apod, Cd74, ...
+    Will return a max of 3 genes, totalling 15 characters in length.
+    Randomly selects gene names.
+
+    NOTE: the random selection means it will re-render constantly.
+    */
+    const { world } = this.props;
+    const { varAnnotations } = world;
+    const geneNames = varAnnotations.col("name").asArray();
+    if (geneNames.length > 0) {
+      const placeholder = [];
+      let len = geneNames.length;
+      const maxGeneNameCount = 3;
+      const maxStrLength = 15;
+      len = len < maxGeneNameCount ? len : maxGeneNameCount;
+      for (let i = 0, strLen = 0; i < len && strLen < maxStrLength; i += 1) {
+        const deal = Math.floor(Math.random() * geneNames.length);
+        const geneName = geneNames[deal];
+        placeholder.push(geneName);
+        strLen += geneName.length + 2; // '2' is the length of a comma and space
+      }
+      placeholder.push("...");
+      return placeholder.join(", ");
     }
+    // default - should never happen.
+    return "Apod, Cd74, ...";
   }
 
-  handleClick() {
+  handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
-    const { gene } = this.state;
-
+    const gene = g.target;
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
     } else if (userDefinedGenes.length > 15) {
       postUserErrorToast(
         "That's too many genes, you can have at most 15 user defined genes"
       );
-    } else if (!_.find(world.varAnnotations, { name: gene })) {
+    } else if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
       postUserErrorToast("That doesn't appear to be a valid gene name.");
     } else {
       dispatch(actions.requestUserDefinedGene(gene));
@@ -60,13 +122,53 @@ class GeneExpression extends React.Component {
         type: "user defined gene",
         data: gene
       });
-      this.setState({ gene: "" });
     }
   }
 
+  handleBulkAddClick() {
+    const { world, dispatch, userDefinedGenes } = this.props;
+    const { bulkAdd } = this.state;
+
+    /*
+      test:
+      Apod,,, Cd74,,    ,,,    Foo,    Bar-2,,
+    */
+    if (bulkAdd !== "") {
+      const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");
+
+      genes.forEach(gene => {
+        if (gene.length === 0) {
+          keepAroundErrorToast("Must enter a gene name.");
+        } else if (userDefinedGenes.indexOf(gene) !== -1) {
+          keepAroundErrorToast("That gene already exists");
+        } else if (
+          world.varAnnotations.col("name").indexOf(gene) === undefined
+        ) {
+          keepAroundErrorToast(
+            `${gene} doesn't appear to be a valid gene name.`
+          );
+        } else {
+          dispatch(actions.requestUserDefinedGene(gene));
+          dispatch({
+            type: "user defined gene",
+            data: gene
+          });
+        }
+      });
+    }
+
+    this.setState({ bulkAdd: "" });
+  }
+
   render() {
-    const { world, userDefinedGenes, differential } = this.props;
-    const { gene } = this.state;
+    const {
+      world,
+      userDefinedGenes,
+      userDefinedGenesLoading,
+      differential
+    } = this.props;
+
+    const { tab, bulkAdd } = this.state;
 
     return (
       <div>
@@ -84,43 +186,120 @@ class GeneExpression extends React.Component {
             Selected Genes
           </p>
           <div
-            style={{ padding: globals.leftSidebarSectionPadding }}
-            className="bp3-control-group"
+            style={{
+              padding: globals.leftSidebarSectionPadding
+            }}
           >
-            <div className="bp3-input-group bp3-fill">
-              <input
-                onKeyDown={this.keyPress.bind(this)}
-                onChange={e => {
-                  this.setState({ gene: e.target.value });
-                }}
-                value={gene}
-                type="text"
-                className="bp3-input"
-                placeholder="Enter a gene name"
-                style={{ paddingRight: 94 }}
-              />
-            </div>
-            <Tooltip
-              content="Add a gene to see its expression levels"
-              position="bottom"
+            <Button
+              active={tab === "autosuggest"}
+              style={{ marginRight: 5 }}
+              minimal
+              small
+              onClick={() => {
+                this.setState({ tab: "autosuggest" });
+              }}
             >
-              <Button intent="primary" onClick={this.handleClick.bind(this)}>
+              Autosuggest
+            </Button>
+            <Button
+              active={tab === "bulkadd"}
+              minimal
+              small
+              onClick={() => {
+                this.setState({ tab: "bulkadd" });
+              }}
+            >
+              Bulk add genes
+            </Button>
+          </div>
+
+          {tab === "autosuggest" ? (
+            <ControlGroup
+              style={{
+                paddingLeft: globals.leftSidebarSectionPadding,
+                paddingBottom: globals.leftSidebarSectionPadding
+              }}
+            >
+              <Suggest
+                closeOnSelect
+                openOnKeyDown
+                resetOnSelect
+                itemDisabled={
+                  userDefinedGenesLoading ? () => true : () => false
+                }
+                noResults={<MenuItem disabled text="No matching genes." />}
+                onItemSelect={g => {
+                  /* this happens on 'enter' */
+                  this.handleClick(g);
+                }}
+                inputProps={{ "data-testid": "gene-search" }}
+                inputValueRenderer={g => {
+                  return "";
+                }}
+                itemListPredicate={filterGenes}
+                itemRenderer={renderGene.bind(this)}
+                items={
+                  world && world.varAnnotations
+                    ? world.varAnnotations.col("name").asArray()
+                    : ["No genes"]
+                }
+                popoverProps={{ minimal: true }}
+              />
+              <Button
+                className="bp3-button bp3-intent-primary"
+                data-testid={"add-gene"}
+                loading={userDefinedGenesLoading}
+              >
                 Add
               </Button>
-            </Tooltip>
-          </div>
+            </ControlGroup>
+          ) : null}
+          {tab === "bulkadd" ? (
+            <div style={{ paddingLeft: globals.leftSidebarSectionPadding }}>
+              <form
+                onSubmit={e => {
+                  e.preventDefault();
+                  this.handleBulkAddClick();
+                }}
+              >
+                <FormGroup
+                  helperText="Add a list of genes (comma delimited)"
+                  labelFor="text-input-bulk-add"
+                >
+                  <ControlGroup>
+                    <InputGroup
+                      onChange={e => {
+                        this.setState({ bulkAdd: e.target.value });
+                      }}
+                      id="text-input-bulk-add"
+                      placeholder={this.placeholderGeneNames()}
+                      value={bulkAdd}
+                    />
+                    <Button
+                      intent="primary"
+                      onClick={this.handleBulkAddClick.bind(this)}
+                      loading={userDefinedGenesLoading}
+                    >
+                      Add
+                    </Button>
+                  </ControlGroup>
+                </FormGroup>
+              </form>
+            </div>
+          ) : null}
           {world && userDefinedGenes.length > 0
             ? _.map(userDefinedGenes, (geneName, index) => {
-                const values = world.varDataCache[geneName];
+                const values = world.varData.col(geneName);
                 if (!values) {
                   return null;
                 }
+                const summary = values.summarize();
                 return (
                   <HistogramBrush
                     key={geneName}
                     field={geneName}
                     zebra={index % 2 === 0}
-                    ranges={d3.extent(values)}
+                    ranges={summary}
                     isUserDefined
                   />
                 );
@@ -139,18 +318,18 @@ class GeneExpression extends React.Component {
           <ExpressionButtons />
           {differential.diffExp
             ? _.map(differential.diffExp, (value, index) => {
-                const annotations = world.varAnnotations[value[0]];
-                const { name } = annotations;
-                const values = world.varDataCache[name];
+                const name = world.varAnnotations.at(value[0], "name");
+                const values = world.varData.col(name);
                 if (!values) {
                   return null;
                 }
+                const summary = values.summarize();
                 return (
                   <HistogramBrush
                     key={name}
                     field={name}
                     zebra={index % 2 === 0}
-                    ranges={d3.extent(values)}
+                    ranges={summary}
                     isDiffExp
                     logFoldChange={value[1]}
                     pval={value[2]}

client/src/components/graph/graph.js

@@ -22,7 +22,8 @@ import scaleLinear from "../../util/scaleLinear";
   responsive: state.responsive,
   colorRGB: _.get(state.controls, "colorRGB", null),
   opacityForDeselectedCells: state.controls.opacityForDeselectedCells,
-  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null),
+  resettingInterface: state.controls.resettingInterface
 }))
 class Graph extends React.Component {
   constructor(props) {
@@ -34,12 +35,13 @@ class Graph extends React.Component {
     this.graphPaddingRight = globals.leftSidebarWidth;
     this.renderCache = {
       positions: null,
-      colors: null
+      colors: null,
+      sizes: null
     };
     this.state = {
       svg: null,
       brush: null,
-      mode: "brush"
+      mode: "lasso"
     };
   }
 
@@ -82,12 +84,13 @@ class Graph extends React.Component {
   }
 
   componentDidUpdate(prevProps) {
+    const { renderCache } = this;
     const {
       world,
       crossfilter,
-      selectionUpdate,
       colorRGB,
-      responsive
+      responsive,
+      selectionUpdate
     } = this.props;
     const {
       reglRender,
@@ -108,35 +111,27 @@ class Graph extends React.Component {
 
     if (regl && world) {
       /* update the regl state */
-      const { obsLayout } = world;
-      const cellCount = crossfilter.size();
+      const { obsLayout, nObs } = world;
+      const X = obsLayout.col("X").asArray();
+      const Y = obsLayout.col("Y").asArray();
 
       // X/Y positions for each point - a cached value that only
       // changes if we have loaded entirely new cell data
       //
-      if (
-        !this.renderCache.positions ||
-        selectionUpdate !== prevProps.selectionUpdate
-      ) {
-        if (!this.renderCache.positions) {
-          this.renderCache.positions = new Float32Array(2 * cellCount);
-        }
+      if (!renderCache.positions || world !== prevProps.world) {
+        renderCache.positions = new Float32Array(2 * nObs);
 
         const glScaleX = scaleLinear([0, 1], [-1, 1]);
         const glScaleY = scaleLinear([0, 1], [1, -1]);
 
-        const offset = [d3.mean(obsLayout.X) - 0.5, d3.mean(obsLayout.Y) - 0.5];
+        const offset = [d3.mean(X) - 0.5, d3.mean(Y) - 0.5];
 
-        for (
-          let i = 0, { positions } = this.renderCache;
-          i < cellCount;
-          i += 1
-        ) {
-          positions[2 * i] = glScaleX(obsLayout.X[i] - offset[0]);
-          positions[2 * i + 1] = glScaleY(obsLayout.Y[i] - offset[1]);
+        for (let i = 0, { positions } = renderCache; i < nObs; i += 1) {
+          positions[2 * i] = glScaleX(X[i] - offset[0]);
+          positions[2 * i + 1] = glScaleY(Y[i] - offset[1]);
         }
         pointBuffer({
-          data: this.renderCache.positions,
+          data: renderCache.positions,
           dimension: 2
         });
 
@@ -151,30 +146,28 @@ class Graph extends React.Component {
       // could have changed for some other reason, but for now color is
       // the only metadata that changes client-side.  If this is problematic,
       // we could add some sort of color-specific indicator to the app state.
-      if (!this.renderCache.colors || colorRGB !== prevProps.colorRGB) {
+      if (!renderCache.colors || colorRGB !== prevProps.colorRGB) {
         const rgb = colorRGB;
-        if (!this.renderCache.colors) {
-          this.renderCache.colors = new Float32Array(3 * rgb.length);
+        if (!renderCache.colors) {
+          renderCache.colors = new Float32Array(3 * rgb.length);
         }
-        for (let i = 0, { colors } = this.renderCache; i < rgb.length; i += 1) {
+        for (let i = 0, { colors } = renderCache; i < rgb.length; i += 1) {
           colors.set(rgb[i], 3 * i);
         }
-        colorBuffer({ data: this.renderCache.colors, dimension: 3 });
+        colorBuffer({ data: renderCache.colors, dimension: 3 });
       }
 
-      // Sizes for each point - this is presumed to change each time the
-      // component receives new props.   Almost always a true assumption, as
-      // most property upates are due to changes driving a crossfilter
-      // selection set change.
-      //
-      if (!this.renderCache.sizes) {
-        this.renderCache.sizes = new Float32Array(cellCount);
+      // Sizes for each point - updates are triggered only when selected
+      // obs change
+      if (!renderCache.sizes || selectionUpdate !== prevProps.selectionUpdate) {
+        if (!renderCache.sizes) {
+          renderCache.sizes = new Float32Array(nObs);
+        }
+        crossfilter.fillByIsFiltered(renderCache.sizes, 4, 0.2);
+        sizeBuffer({ data: renderCache.sizes, dimension: 1 });
       }
 
-      crossfilter.fillByIsFiltered(this.renderCache.sizes, 4, 0.2);
-      sizeBuffer({ data: this.renderCache.sizes, dimension: 1 });
-
-      this.count = cellCount;
+      this.count = nObs;
 
       regl._refresh();
       this.reglDraw(
@@ -201,7 +194,9 @@ class Graph extends React.Component {
         this.handleBrushSelectAction.bind(this),
         this.handleBrushDeselectAction.bind(this),
         responsive,
-        this.graphPaddingRight
+        this.graphPaddingRight,
+        this.handleLassoStart.bind(this),
+        this.handleLassoEnd.bind(this)
       );
       this.setState({ svg: newSvg, brush });
     }
@@ -250,54 +245,54 @@ class Graph extends React.Component {
     });
   }
 
+  invertPoint(pin) {
+    const { responsive } = this.props;
+    const { regl, camera, offset } = this.state;
+
+    const gl = regl._gl;
+
+    // get aspect ratio
+    const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
+
+    // compute inverse view matrix
+    const inverse = mat4.invert([], camera.view());
+
+    // transform screen coordinates -> cell coordinates
+    const x = (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
+    const y = 2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
+    const pout = [
+      x * inverse[14] * aspect + inverse[12],
+      y * inverse[14] + inverse[13]
+    ];
+    return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
+  }
+
   handleBrushSelectAction() {
     /*
-    This conditional handles procedural brush deselect. Brush emits
-    an event on procedural deselect because it is move: null
+      This conditional handles procedural brush deselect. Brush emits
+      an event on procedural deselect because it is move: null
     */
-
-    const { camera, offset } = this.state;
-    const { dispatch, responsive } = this.props;
-
-    if (d3.event.sourceEvent !== null) {
-      /*
-      No idea why d3 event scope works like this
-      but apparently
-      it does
-      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
-    */
-      const s = d3.event.selection;
-      const gl = this.state.regl._gl;
-      /*
+    /*
       event describing brush position:
       @-------|
       |       |
       |       |
       |-------@
     */
+    /*
+      No idea why d3 event scope works like this
+      but apparently
+      it does
+      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
+    */
+    const { dispatch } = this.props;
 
-      // get aspect ratio
-      const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
-
-      // compute inverse view matrix
-      const inverse = mat4.invert([], camera.view());
-
-      // transform screen coordinates -> cell coordinates
-      const invert = pin => {
-        const x =
-          (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
-        const y =
-          2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
-        const pout = [
-          x * inverse[14] * aspect + inverse[12],
-          y * inverse[14] + inverse[13]
-        ];
-        return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
-      };
+    if (d3.event.sourceEvent !== null) {
+      const s = d3.event.selection;
 
       const brushCoords = {
-        northwest: invert([s[0][0], s[0][1]]),
-        southeast: invert([s[1][0], s[1][1]])
+        northwest: this.invertPoint([s[0][0], s[0][1]]),
+        southeast: this.invertPoint([s[1][0], s[1][1]])
       };
 
       dispatch({
@@ -329,6 +324,25 @@ class Graph extends React.Component {
     }
   }
 
+  handleLassoStart() {
+    const { dispatch } = this.props;
+    // reset selected points when starting a new polygon
+    // making it easier for the user to make the next selection
+    dispatch({
+      type: "lasso started"
+    });
+  }
+
+  // when a lasso is completed, filter to the points within the lasso polygon
+  handleLassoEnd(polygon) {
+    const { dispatch } = this.props;
+
+    dispatch({
+      type: "lasso selection",
+      polygon: polygon.map(xy => this.invertPoint(xy)) // transform the polygon
+    });
+  }
+
   handleOpacityRangeChange(e) {
     const { dispatch } = this.props;
     dispatch({
@@ -339,11 +353,20 @@ class Graph extends React.Component {
 
   resetInterface() {
     const { dispatch } = this.props;
+    dispatch({
+      type: "interface reset started"
+    });
     dispatch(actions.resetInterface());
   }
 
   render() {
-    const { dispatch, responsive, crossfilter } = this.props;
+    const {
+      dispatch,
+      responsive,
+      crossfilter,
+      resettingInterface
+    } = this.props;
+
     const { mode } = this.state;
     return (
       <div id="graphWrapper">
@@ -392,6 +415,7 @@ class Graph extends React.Component {
                   /* world && universe ? worldEqUniverse(world, universe) : false */
                 }
                 type="button"
+                loading={resettingInterface}
                 intent="warning"
                 style={{ marginRight: 10 }}
                 onClick={this.resetInterface.bind(this)}
@@ -401,13 +425,18 @@ class Graph extends React.Component {
             </Tooltip>
             <div>
               <div className="bp3-button-group">
-                <Tooltip content="Lasso cells" position="left">
+                <Tooltip content="Lasso selection" position="left">
                   <Button
-                    className="bp3-button bp3-icon-select"
                     type="button"
-                    active={mode === "brush"}
+                    className="bp3-button bp3-icon-polygon-filter"
+                    active={mode === "lasso"}
                     onClick={() => {
-                      this.setState({ mode: "brush" });
+                      this.handleBrushDeselectAction();
+                      // this.restartReglLoop();
+                      this.setState({ mode: "lasso" });
+                    }}
+                    style={{
+                      cursor: "pointer"
                     }}
                   />
                 </Tooltip>
@@ -440,7 +469,7 @@ class Graph extends React.Component {
         >
           <div
             style={{
-              display: mode === "brush" ? "inherit" : "none"
+              display: mode === "lasso" ? "inherit" : "none"
             }}
             id="graphAttachPoint"
           />
@@ -448,6 +477,7 @@ class Graph extends React.Component {
             <canvas
               width={responsive.width - this.graphPaddingRight}
               height={responsive.height - this.graphPaddingTop}
+              data-testid="layout"
               ref={canvas => {
                 this.reglCanvas = canvas;
               }}

client/src/components/graph/setupLasso.js

@@ -0,0 +1,127 @@
+// https://bl.ocks.org/pbeshai/8008075f9ce771ee8be39e8c38907570
+
+import * as d3 from "d3";
+
+const Lasso = () => {
+  const dispatch = d3.dispatch("start", "end");
+
+  const polygonToPath = polygon =>
+    `M${polygon.map(d => d.join(",")).join("L")}`;
+
+  const distance = (pt1, pt2) =>
+    Math.sqrt((pt2[0] - pt1[0]) ** 2 + (pt2[1] - pt1[1]) ** 2);
+
+  // distance last point has to be to first point before it auto closes when mouse is released
+  const closeDistance = 75;
+
+  const lasso = svg => {
+    let lassoPolygon;
+    let lassoPath;
+    let closePath;
+
+    const handleDragStart = () => {
+      lassoPolygon = [d3.mouse(svg.node())]; // current x y of mouse within element
+
+      if (lassoPath) {
+        lassoPath.remove();
+      }
+
+      lassoPath = g
+        .append("path")
+        .attr("fill", "#0bb")
+        .attr("fill-opacity", 0.1)
+        .attr("stroke", "#0bb")
+        .attr("stroke-dasharray", "3, 3");
+
+      closePath = g
+        .append("line")
+        .attr("x2", lassoPolygon[0][0])
+        .attr("y2", lassoPolygon[0][1])
+        .attr("stroke", "#0bb")
+        .attr("stroke-dasharray", "3, 3")
+        .attr("opacity", 0);
+
+      dispatch.call("start", lasso, lassoPolygon);
+    };
+
+    const handleDrag = () => {
+      const point = d3.mouse(svg.node());
+      lassoPolygon.push(point);
+      lassoPath.attr("d", polygonToPath(lassoPolygon));
+
+      // indicate if we are within closing distance
+      if (
+        distance(lassoPolygon[0], lassoPolygon[lassoPolygon.length - 1]) <
+        closeDistance
+      ) {
+        closePath
+          .attr("x1", point[0])
+          .attr("y1", point[1])
+          .attr("opacity", 1);
+      } else {
+        closePath.attr("opacity", 0);
+      }
+    };
+
+    const handleDragEnd = () => {
+      // remove the close path
+      closePath.remove();
+      closePath = null;
+
+      // succesfully closed
+      if (
+        distance(lassoPolygon[0], lassoPolygon[lassoPolygon.length - 1]) <
+        closeDistance
+      ) {
+        lassoPath.attr("d", `${polygonToPath(lassoPolygon)}Z`);
+        dispatch.call("end", lasso, lassoPolygon);
+
+        // otherwise cancel
+      } else {
+        lassoPath.remove();
+        lassoPath = null;
+        lassoPolygon = null;
+      }
+    };
+
+    // append a <g> with a rect
+    const g = svg.append("g").attr("class", "lasso-group");
+    const bbox = svg.node().getBoundingClientRect();
+    const area = g
+      .append("rect")
+      .attr("width", bbox.width)
+      .attr("height", bbox.height)
+      .attr("fill", "tomato")
+      .attr("opacity", 0);
+
+    const drag = d3
+      .drag()
+      .on("start", handleDragStart)
+      .on("drag", handleDrag)
+      .on("end", handleDragEnd);
+
+    area.call(drag);
+
+    lasso.reset = () => {
+      if (lassoPath) {
+        lassoPath.remove();
+        lassoPath = null;
+      }
+
+      lassoPolygon = null;
+      if (closePath) {
+        closePath.remove();
+        closePath = null;
+      }
+    };
+  };
+
+  lasso.on = (type, callback) => {
+    dispatch.on(type, callback);
+    return lasso;
+  };
+
+  return lasso;
+};
+
+export default Lasso;

client/src/components/graph/setupSVGandBrush.js

@@ -1,6 +1,7 @@
 // jshint esversion: 6
 import * as d3 from "d3";
 import styles from "./graph.css";
+import Lasso from "./setupLasso";
 
 /******************************************
 *******************************************
@@ -12,11 +13,14 @@ export default (
   handleBrushSelectAction,
   handleBrushDeselectAction,
   responsive,
-  graphPaddingRight
+  graphPaddingRight,
+  handleLassoStart,
+  handleLassoEnd
 ) => {
   const svg = d3
     .select("#graphAttachPoint")
     .append("svg")
+    .attr("data-testid", "layout-overlay")
     .attr("width", responsive.width - graphPaddingRight)
     .attr("height", responsive.height)
     .attr("class", `${styles.graphSVG}`);
@@ -32,9 +36,16 @@ export default (
     .attr("class", "graph_brush")
     .call(brush);
 
+  const lassoInstance = Lasso()
+    .on("end", handleLassoEnd)
+    .on("start", handleLassoStart);
+
+  const lasso = svg.call(lassoInstance);
+
   return {
     svg,
     brushContainer,
-    brush
+    brush,
+    lasso
   };
 };

client/src/components/leftsidebar.js

@@ -40,6 +40,7 @@ class LeftSideBar extends React.Component {
         }}
       >
         <p
+          data-testid="header"
           style={{
             position: "fixed",
             top: globals.cellxgeneTitleTopPadding,

client/src/components/scatterplot/scatterplot.js

@@ -19,7 +19,7 @@ import _drawPoints from "./drawPointsRegl";
 import scaleLinear from "../../util/scaleLinear";
 
 import { margin, width, height } from "./util";
-import { kvCache } from "../../util/stateManager";
+import finiteExtent from "../../util/finiteExtent";
 
 @connect(state => {
   const {
@@ -29,12 +29,16 @@ import { kvCache } from "../../util/stateManager";
     scatterplotYYaccessor
   } = state.controls;
   const expressionX =
-    world && scatterplotXXaccessor
-      ? kvCache.get(world.varDataCache, scatterplotXXaccessor)
+    world &&
+    scatterplotXXaccessor &&
+    world.varData.hasCol(scatterplotXXaccessor)
+      ? world.varData.col(scatterplotXXaccessor).asArray()
       : null;
   const expressionY =
-    world && scatterplotYYaccessor
-      ? kvCache.get(world.varDataCache, scatterplotYYaccessor)
+    world &&
+    scatterplotYYaccessor &&
+    world.varData.hasCol(scatterplotYYaccessor)
+      ? world.varData.col(scatterplotYYaccessor).asArray()
       : null;
 
   return {
@@ -64,12 +68,17 @@ class Scatterplot extends React.Component {
     super(props);
     this.count = 0;
     this.axes = false;
-    this.state = {
-      svg: null,
-      minimized: null,
+    this.renderCache = {
+      positions: null,
+      colors: null,
+      sizes: null,
       xScale: null,
       yScale: null
     };
+    this.state = {
+      svg: null,
+      minimized: null
+    };
   }
 
   componentDidMount() {
@@ -80,6 +89,7 @@ class Scatterplot extends React.Component {
     if (svg && expressionX && expressionY) {
       scales = Scatterplot.setupScales(expressionX, expressionY);
       this.drawAxesSVG(scales.xScale, scales.yScale, svg);
+      this.renderCache = { ...this.renderCache, ...scales };
     }
 
     const camera = _camera(this.reglCanvas, { scale: true, rotate: false });
@@ -112,8 +122,6 @@ class Scatterplot extends React.Component {
       pointBuffer,
       colorBuffer,
       svg,
-      xScale: scales ? scales.xScale : null,
-      yScale: scales ? scales.yScale : null,
       reglRender,
       camera,
       drawPoints
@@ -128,12 +136,11 @@ class Scatterplot extends React.Component {
       scatterplotYYaccessor,
       expressionX,
       expressionY,
-      colorRGB
+      colorRGB,
+      selectionUpdate
     } = this.props;
     const {
       reglRender,
-      xScale,
-      yScale,
       regl,
       pointBuffer,
       colorBuffer,
@@ -144,17 +151,12 @@ class Scatterplot extends React.Component {
     } = this.state;
 
     if (
-      world &&
-      svg &&
-      xScale &&
-      yScale &&
-      scatterplotXXaccessor &&
-      scatterplotYYaccessor &&
-      (scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
-      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor || // was CLU now FTH1 etc
-        !this.axes) // clicked off the tab and back again, rerender
+      scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
+      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor // was CLU now FTH1 etc
     ) {
-      this.drawAxesSVG(xScale, yScale, svg);
+      const scales = Scatterplot.setupScales(expressionX, expressionY);
+      this.drawAxesSVG(scales.xScale, scales.yScale, svg);
+      this.renderCache = { ...this.renderCache, ...scales };
     }
 
     if (reglRender && this.reglRenderState === "rendering") {
@@ -171,35 +173,51 @@ class Scatterplot extends React.Component {
       expressionX &&
       expressionY &&
       scatterplotXXaccessor &&
-      scatterplotYYaccessor &&
-      xScale &&
-      yScale
+      scatterplotYYaccessor
     ) {
+      const { renderCache } = this;
+      const { xScale, yScale } = this.renderCache;
       const cellCount = expressionX.length;
-      const positionsBuf = new Float32Array(2 * cellCount);
-      const colorsBuf = new Float32Array(3 * cellCount);
-      const sizesBuf = new Float32Array(cellCount);
 
-      const glScaleX = scaleLinear([0, width], [-0.95, 0.95]);
-      const glScaleY = scaleLinear([0, height], [-1, 1]);
-
-      /*
-        Construct Vectors
-      */
-      for (let i = 0; i < cellCount; i += 1) {
-        positionsBuf[2 * i] = glScaleX(xScale(expressionX[i]));
-        positionsBuf[2 * i + 1] = glScaleY(yScale(expressionY[i]));
+      // Points change when expressionX or expressionY change.
+      if (
+        !renderCache.positions ||
+        expressionX !== prevProps.expressionX ||
+        expressionY !== prevProps.expressionY
+      ) {
+        if (!renderCache.positions) {
+          renderCache.positions = new Float32Array(2 * cellCount);
+        }
+        const glScaleX = scaleLinear([0, width], [-0.95, 0.95]);
+        const glScaleY = scaleLinear([0, height], [-1, 1]);
+        for (let i = 0, { positions } = renderCache; i < cellCount; i += 1) {
+          positions[2 * i] = glScaleX(xScale(expressionX[i]));
+          positions[2 * i + 1] = glScaleY(yScale(expressionY[i]));
+        }
+        pointBuffer({ data: renderCache.positions, dimension: 2 });
       }
 
-      for (let i = 0; i < cellCount; i += 1) {
-        colorsBuf.set(colorRGB[i], 3 * i);
+      // Colors for each point - change only when props.colorsRGB change.
+      if (!renderCache.colors || colorRGB !== prevProps.colorRGB) {
+        if (!renderCache.colors) {
+          renderCache.colors = new Float32Array(3 * cellCount);
+        }
+        for (let i = 0, { colors } = renderCache; i < cellCount; i += 1) {
+          colors.set(colorRGB[i], 3 * i);
+        }
+        colorBuffer({ data: renderCache.colors, dimension: 3 });
       }
 
-      crossfilter.fillByIsFiltered(sizesBuf, 4, 0.2);
+      // Sizes for each point - updates are triggered only when selected
+      // obs change
+      if (!renderCache.sizes || selectionUpdate !== prevProps.selctionUpdate) {
+        if (!renderCache.sizes) {
+          renderCache.sizes = new Float32Array(cellCount);
+        }
+        crossfilter.fillByIsFiltered(renderCache.sizes, 4, 0.2);
+        sizeBuffer({ data: renderCache.sizes, dimension: 1 });
+      }
 
-      pointBuffer({ data: positionsBuf, dimension: 2 });
-      colorBuffer({ data: colorsBuf, dimension: 3 });
-      sizeBuffer({ data: sizesBuf, dimension: 1 });
       this.count = cellCount;
 
       regl._refresh();
@@ -212,26 +230,16 @@ class Scatterplot extends React.Component {
         camera
       );
     }
-
-    if (
-      expressionX &&
-      expressionY &&
-      (scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
-        scatterplotYYaccessor !== prevProps.scatterplotYYaccessor)
-    ) {
-      const scales = Scatterplot.setupScales(expressionX, expressionY);
-      this.setState(scales);
-    }
   }
 
   static setupScales(expressionX, expressionY) {
     const xScale = d3
       .scaleLinear()
-      .domain(d3.extent(expressionX))
+      .domain(finiteExtent(expressionX))
       .range([0, width]);
     const yScale = d3
       .scaleLinear()
-      .domain(d3.extent(expressionY))
+      .domain(finiteExtent(expressionY))
       .range([height, 0]);
 
     return {

client/src/globals.js

@@ -1,3 +1,5 @@
+import { Colors } from "@blueprintjs/core";
+
 // jshint esversion: 6
 /* these will be either (preferably) specified or inferred */
 export const categories = [
@@ -44,8 +46,8 @@ export const configDefaults = {
 };
 
 /* colors */
-export const blue = "#4a90e2";
-export const hcaBlue = "#1c7cc7";
+export const blue = Colors.BLUE3;
+export const linkBlue = Colors.BLUE5;
 export const lightestGrey = "rgb(249,249,249)";
 export const lighterGrey = "rgb(245,245,245)";
 export const lightGrey = "rgb(211,211,211)";
@@ -57,6 +59,7 @@ export const brightBlue = "#4a90e2";
 export const brightGreen = "#A2D729";
 export const darkGreen = "#448C4D";
 
+export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";
 
 /* typography constants */

client/src/middleware/updateCellColors.js

@@ -1,9 +1,10 @@
 // jshint esversion: 6
 import _ from "lodash";
 import * as d3 from "d3";
-import { interpolateViridis } from "d3-scale-chromatic";
+import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
 import * as globals from "../globals";
 import parseRGB from "../util/parseRGB";
+import finiteExtent from "../util/finiteExtent";
 
 /*
   https://medium.com/@jacobp100/you-arent-using-redux-middleware-enough-94ffe991e6
@@ -40,64 +41,98 @@ const updateCellColorsMiddleware = store => next => action => {
     action.type === "color by continuous metadata" ||
     action.type === "color by categorical metadata";
 
-  if (!filterJustChanged || !s.controls.world.obsAnnotations) {
+  const obsAnnotations = _.get(s.controls, "world.obsAnnotations", null);
+  if (!filterJustChanged || !obsAnnotations) {
     return next(
       action
     ); /* if the cells haven't loaded or the action wasn't a color change, bail */
   }
 
-  const { obsAnnotations } = s.controls.world;
   let colorScale;
-  const colorsByName = new Array(obsAnnotations.length);
   const colorsByRGB = new Array(obsAnnotations.length);
 
   /*
   in plain language...
    (a) once the cells have loaded.
    (b) each time a user changes a color control we need to update cellsMetadata colors
-  This is available to all the draw functions as world.colorName[index] or world.colorRGB[index]
+  This is available to all the draw functions as controls.colorRGB[index]
   */
 
   if (action.type === "color by categorical metadata") {
-    colorScale = d3.scaleOrdinal().range(globals.ordinalColors);
+    const { categories } = _.filter(s.controls.world.schema.annotations.obs, {
+      name: action.colorAccessor
+    })[0];
 
-    for (let i = 0; i < obsAnnotations.length; i += 1) {
-      const obs = obsAnnotations[i];
-      const c = colorScale(obs[action.colorAccessor]);
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+    colorScale = d3
+      .scaleSequential(interpolateRainbow)
+      .domain([0, categories.length]);
+
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = _.transform(categories, (acc, cat, idx) => {
+      acc[cat] = parseRGB(colorScale(idx));
+    });
+
+    const key = action.colorAccessor;
+    const col = obsAnnotations.col(key).asArray();
+    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
+      const cat = col[i];
+      colorsByRGB[i] = colors[cat];
     }
   }
 
   if (action.type === "color by continuous metadata") {
+    const colorBins = 100;
+    const { min, max } = action.rangeForColorAccessor;
     colorScale = d3
-      .scaleLinear()
-      .domain([0, action.rangeMaxForColorAccessor])
-      .range([1, 0]);
+      .scaleQuantile()
+      .domain([min, max])
+      .range(_.range(colorBins - 1, -1, -1));
 
-    for (let i = 0; i < obsAnnotations.length; i += 1) {
-      const obs = obsAnnotations[i];
-      const c = interpolateViridis(colorScale(obs[action.colorAccessor]));
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = new Array(colorBins);
+    for (let i = 0; i < colorBins; i += 1) {
+      colors[i] = parseRGB(interpolateCool(i / colorBins));
+    }
+
+    const key = action.colorAccessor;
+    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
+    const col = obsAnnotations.col(key).asArray();
+    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
+      const val = col[i];
+      if (Number.isFinite(val)) {
+        const c = colorScale(val);
+        colorsByRGB[i] = colors[c];
+      } else {
+        colorsByRGB[i] = nonFiniteColor;
+      }
     }
   }
 
   if (action.type === "color by expression") {
     const { gene, data } = action;
     const expression = data[gene]; // Float32Array
+    const colorBins = 100;
+    const [min, max] = finiteExtent(expression);
     colorScale = d3
-      .scaleLinear()
-      .domain([_.min(expression), _.max(expression)])
-      .range([
-        1,
-        0
-      ]); /* invert viridis... probably pass this scale through to others */
+      .scaleQuantile()
+      .domain([min, max])
+      .range(_.range(colorBins - 1, -1, -1));
+
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = new Array(colorBins);
+    for (let i = 0; i < colorBins; i += 1) {
+      colors[i] = parseRGB(interpolateCool(i / colorBins));
+    }
+    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
 
     for (let i = 0, len = expression.length; i < len; i += 1) {
-      const c = interpolateViridis(colorScale(expression[i]));
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+      const e = expression[i];
+      if (Number.isFinite(e)) {
+        const c = colorScale(e);
+        colorsByRGB[i] = colors[c];
+      } else {
+        colorsByRGB[i] = nonFiniteColor;
+      }
     }
   }
 
@@ -105,7 +140,7 @@ const updateCellColorsMiddleware = store => next => action => {
   append the result of all the filters to the action the user just triggered
   */
   const modifiedAction = Object.assign({}, action, {
-    colors: { name: colorsByName, rgb: colorsByRGB },
+    colors: { rgb: colorsByRGB },
     colorScale
   });
 

client/src/reducers/controls.js

@@ -1,7 +1,8 @@
 // jshint esversion: 6
 
 import _ from "lodash";
-import { World, kvCache } from "../util/stateManager";
+
+import { World, WorldUtil, ControlsHelper } from "../util/stateManager";
 import parseRGB from "../util/parseRGB";
 import Crossfilter from "../util/typedCrossfilter";
 import * as globals from "../globals";
@@ -13,40 +14,32 @@ import {
   makeContinuousDimensionName
 } from "../util/nameCreators";
 
-function createCategoricalAsBooleansMap(world) {
-  const res = {};
-  _.each(world.summary.obs, (value, key) => {
-    if (value.options && key !== "name") {
-      const optionsAsBooleans = {};
-      _.each(value.options, (_value, _key) => {
-        optionsAsBooleans[_key] = true;
-      });
-      res[key] = optionsAsBooleans;
-    }
-  });
-  return res;
-}
-
 const Controls = (
   state = {
     // data loading flag
     loading: false,
     error: null,
 
+    // configuration
+    maxCategoryItems: globals.configDefaults.parameters["max-category-items"],
+
+    // the whole big bang
     universe: null,
+    fullUniverseCache: null,
 
     // all of the data + selection state
     world: null,
-    colorName: null,
     colorRGB: null,
-    categoricalAsBooleansMap: null,
+    categoricalSelectionState: null,
     crossfilter: null,
     dimensionMap: null,
     userDefinedGenes: [],
+    userDefinedGenesLoading: false,
     diffexpGenes: [],
 
     colorAccessor: null,
     colorScale: null,
+    resettingInterface: false,
 
     opacityForDeselectedCells: 0.2,
     graphBrushSelection: null,
@@ -72,73 +65,94 @@ const Controls = (
           Initialization, World/Universe management
           and data loading.
     ******************************************************/
+    case "configuration load complete": {
+      // there are a couple of configuration items we need to retain
+      return {
+        ...state,
+        maxCategoryItems: _.get(
+          state.config,
+          "parameters.max-category-items",
+          globals.configDefaults.parameters["max-category-items"]
+        )
+      };
+    }
     case "initial data load start": {
       return { ...state, loading: true };
     }
-    case "initial data load complete (universe exists)":
-    case "reset World to eq Universe": {
-      const { userDefinedGenes, diffexpGenes } = state;
+    case "initial data load complete (universe exists)": {
       /* first light - create world & other data-driven defaults */
       const { universe } = action;
       const world = World.createWorldFromEntireUniverse(universe);
-      const colorName = new Array(universe.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
-      const categoricalAsBooleansMap = createCategoricalAsBooleansMap(world);
+      const colorRGB = new Array(universe.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
+      const categoricalSelectionState = ControlsHelper.createCategoricalSelectionState(
+        state,
+        world
+      );
       const crossfilter = Crossfilter(world.obsAnnotations);
       const dimensionMap = World.createObsDimensionMap(crossfilter, world);
-
-      const worldVarDataCache = world.varDataCache;
-
-      // dimensionMap = {
-      //     layout_X: dim-for-X,
-      //     obsAnno_name: dim for an annotation,
-      //     varData_userDefined_genename: dim for user defined expression,
-      //     varData_diffexp_genename: dim for diff-exp added gene expression
-      // }
-      /* var dimensions */
-      if (userDefinedGenes.length > 0) {
-        /*
-          verbose & slightly confusing that we also access this as an object
-          in controls rather than an array, should be abstracted into
-          util ie., createDimensionsFromBothListsOfGenes(userGenes, diffExp)
-        */
-        _.forEach(userDefinedGenes, gene => {
-          dimensionMap[
-            userDefinedDimensionName(gene)
-          ] = World.createVarDimension(
-            /* "__var__" + */
-            world,
-            worldVarDataCache,
-            crossfilter,
-            gene
-          );
-        });
-      }
-
-      if (diffexpGenes.length > 0) {
-        _.forEach(diffexpGenes, gene => {
-          dimensionMap[diffexpDimensionName(gene)] = World.createVarDimension(
-            /* "__var__" + */
-            world,
-            worldVarDataCache,
-            crossfilter,
-            gene
-          );
-        });
-      }
+      WorldUtil.clearCaches();
 
       return {
         ...state,
         loading: false,
         error: null,
         universe,
+        fullUniverseCache: { world, crossfilter, dimensionMap },
         world,
-        colorName,
         colorRGB,
-        categoricalAsBooleansMap,
+        categoricalSelectionState,
         crossfilter,
         dimensionMap,
-        colorAccessor: null
+        colorAccessor: null,
+        resettingInterface: false
+      };
+    }
+    case "reset World to eq Universe": {
+      const {
+        userDefinedGenes,
+        diffexpGenes,
+        universe,
+        fullUniverseCache
+      } = state;
+      const { world, crossfilter } = fullUniverseCache;
+      // reset all crossfilter dimensions
+      _.forEach(fullUniverseCache.dimensionMap, dim => dim.filterAll());
+      const colorRGB = new Array(universe.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
+      const categoricalSelectionState = ControlsHelper.createCategoricalSelectionState(
+        state,
+        world
+      );
+
+      /* free dimensions not in cache (otherwise they leak) */
+      _.forEach(state.dimensionMap, (dim, dimName) => {
+        if (!fullUniverseCache.dimensionMap[dimName]) {
+          dim.dispose();
+        }
+      });
+      const dimensionMap = {
+        ...fullUniverseCache.dimensionMap,
+        ...ControlsHelper.createGenesDimMap(
+          userDefinedGenes,
+          diffexpGenes,
+          world,
+          crossfilter
+        )
+      };
+      WorldUtil.clearCaches();
+
+      return {
+        ...state,
+        world,
+        colorRGB,
+        categoricalSelectionState,
+        crossfilter,
+        dimensionMap,
+        colorAccessor: null,
+        resettingInterface: false
       };
     }
     case "set World to current selection": {
@@ -150,54 +164,32 @@ const Controls = (
         action.world,
         action.crossfilter
       );
-      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
-      const categoricalAsBooleansMap = createCategoricalAsBooleansMap(world);
+      const colorRGB = new Array(world.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
+      const categoricalSelectionState = ControlsHelper.createCategoricalSelectionState(
+        state,
+        world
+      );
       const crossfilter = Crossfilter(world.obsAnnotations);
-      const dimensionMap = World.createObsDimensionMap(crossfilter, world);
-
-      const worldVarDataCache = world.varDataCache;
-      /* var dimensions */
-
-      if (userDefinedGenes.length > 0) {
-        /*
-          verbose & slightly confusing that we also access this as an object
-          in controls rather than an array, should be abstracted into
-          util ie., createDimensionsFromBothListsOfGenes(userGenes, diffExp)
-        */
-        _.forEach(userDefinedGenes, gene => {
-          dimensionMap[
-            userDefinedDimensionName(gene)
-          ] = World.createVarDimension(
-            /* "__var__" + */
-            world,
-            worldVarDataCache,
-            crossfilter,
-            gene
-          );
-        });
-      }
-
-      if (diffexpGenes.length > 0) {
-        _.forEach(diffexpGenes, gene => {
-          dimensionMap[diffexpDimensionName(gene)] = World.createVarDimension(
-            /* "__var__" + */
-            world,
-            worldVarDataCache,
-            crossfilter,
-            gene
-          );
-        });
-      }
+      const dimensionMap = {
+        ...World.createObsDimensionMap(crossfilter, world),
+        ...ControlsHelper.createGenesDimMap(
+          userDefinedGenes,
+          diffexpGenes,
+          world,
+          crossfilter
+        )
+      };
+      WorldUtil.clearCaches();
 
       return {
         ...state,
         loading: false,
         error: null,
         world,
-        colorName,
         colorRGB,
-        categoricalAsBooleansMap,
+        categoricalSelectionState,
         crossfilter,
         dimensionMap,
         colorAccessor: null
@@ -205,65 +197,108 @@ const Controls = (
     }
     case "expression load success": {
       const { world, universe } = state;
-      let universeVarDataCache = universe.varDataCache;
-      let worldVarDataCache = world.varDataCache;
+      let universeVarData = universe.varData;
+      let worldVarData = world.varData;
+
+      // Load new expression data into the varData dataframes, if
+      // not already present.
       _.forEach(action.expressionData, (val, key) => {
-        universeVarDataCache = kvCache.set(universeVarDataCache, key, val);
-        if (kvCache.get(worldVarDataCache, key) === undefined) {
-          worldVarDataCache = kvCache.set(
-            worldVarDataCache,
+        // If not already in universe.varData, save entire expression column
+        if (!universeVarData.hasCol(key)) {
+          universeVarData = universeVarData.withCol(key, val);
+        }
+
+        // If not already in world.varData, save sliced expression column
+        if (!worldVarData.hasCol(key)) {
+          // Slice if world !== universe, else just use whole column.
+          // Use the obsAnnotation index as the cut key, as we keep
+          // all world dataframes in sync.
+          let worldValSlice = val;
+          if (!World.worldEqUniverse(world, universe)) {
+            worldValSlice = universeVarData
+              .subset(world.obsAnnotations.rowIndex.keys(), [key], null)
+              .icol(0)
+              .asArray();
+          }
+
+          // Now build world's varData dataframe
+          worldVarData = worldVarData.withCol(
             key,
-            World.subsetVarData(world, universe, val)
+            worldValSlice,
+            world.obsAnnotations.rowIndex
           );
         }
       });
 
+      // Prune size of varData "cache" if getting out of hand....
+      const { userDefinedGenes, diffexpGenes } = state;
+      const allTheGenesWeNeed = _.uniq(
+        [].concat(
+          userDefinedGenes,
+          diffexpGenes,
+          Object.keys(action.expressionData)
+        )
+      );
+      universeVarData = ControlsHelper.pruneVarDataCache(
+        universeVarData,
+        allTheGenesWeNeed
+      );
+      worldVarData = ControlsHelper.pruneVarDataCache(
+        worldVarData,
+        allTheGenesWeNeed
+      );
+
       return {
         ...state,
         universe: {
           ...universe,
-          varDataCache: universeVarDataCache
+          varData: universeVarData
         },
         world: {
           ...world,
-          varDataCache: worldVarDataCache
+          varData: worldVarData
         }
       };
     }
+    case "request user defined gene started": {
+      return {
+        ...state,
+        userDefinedGenesLoading: true
+      };
+    }
+    case "request user defined gene error": {
+      return {
+        ...state,
+        userDefinedGenesLoading: false
+      };
+    }
     case "request user defined gene success": {
       const { world, crossfilter, dimensionMap, userDefinedGenes } = state;
-      const worldVarDataCache = world.varDataCache;
       const _userDefinedGenes = userDefinedGenes.slice();
       const gene = action.data.genes[0];
 
-      dimensionMap[userDefinedDimensionName(gene)] = World.createVarDimension(
-        /* "__var__" + */
-        world,
-        worldVarDataCache,
-        crossfilter,
-        gene
-      );
+      dimensionMap[
+        userDefinedDimensionName(gene)
+      ] = World.createVarDataDimension(world, crossfilter, gene);
 
       return {
         ...state,
         dimensionMap,
-        userDefinedGenes: _userDefinedGenes
+        userDefinedGenes: _userDefinedGenes,
+        userDefinedGenesLoading: false
       };
     }
     case "request differential expression success": {
       const { world, crossfilter, dimensionMap } = state;
-      const worldVarDataCache = world.varDataCache;
       const _diffexpGenes = [];
 
       action.data.forEach(d => {
-        _diffexpGenes.push(world.varAnnotations[d[0]].name);
+        _diffexpGenes.push(world.varAnnotations.at(d[0], "name"));
       });
 
       _.forEach(_diffexpGenes, gene => {
-        dimensionMap[diffexpDimensionName(gene)] = World.createVarDimension(
-          /* "__var__" + */
+        dimensionMap[diffexpDimensionName(gene)] = World.createVarDataDimension(
           world,
-          worldVarDataCache,
           crossfilter,
           gene
         );
@@ -276,13 +311,10 @@ const Controls = (
       };
     }
     case "clear differential expression": {
-      const { world, universe, dimensionMap } = state;
+      const { world, dimensionMap } = state;
       const _dimensionMap = dimensionMap;
-      const universeVarDataCache = universe.varDataCache;
-      const worldVarDataCache = world.varDataCache;
-
       _.forEach(action.diffExp, values => {
-        const { name } = world.varAnnotations[values[0]];
+        const name = world.varAnnotations.at(values[0], "name");
         // clean up crossfilter dimensions
         const dimension = dimensionMap[diffexpDimensionName(name)];
         dimension.dispose();
@@ -291,15 +323,7 @@ const Controls = (
       return {
         ...state,
         dimensionMap: _dimensionMap,
-        diffexpGenes: [],
-        universe: {
-          ...universe,
-          varDataCache: universeVarDataCache
-        },
-        world: {
-          ...world,
-          varDataCache: worldVarDataCache
-        }
+        diffexpGenes: []
       };
     }
     case "user defined gene": {
@@ -348,11 +372,11 @@ const Controls = (
     }
     case "reset colorscale": {
       const { world } = state;
-      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
+      const colorRGB = new Array(world.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
       return {
         ...state,
-        colorName,
         colorRGB,
         colorAccessor: null
       };
@@ -370,27 +394,36 @@ const Controls = (
              User Events
      *******************************/
     case "graph brush selection change": {
-      state.dimensionMap[layoutDimensionName("X")].filterRange([
-        action.brushCoords.northwest[0],
-        action.brushCoords.southeast[0]
-      ]);
-      state.dimensionMap[layoutDimensionName("Y")].filterRange([
-        action.brushCoords.southeast[1],
-        action.brushCoords.northwest[1]
-      ]);
+      state.dimensionMap[layoutDimensionName("XY")].filterWithinRect(
+        action.brushCoords.northwest,
+        action.brushCoords.southeast
+      );
       return {
         ...state,
         graphBrushSelection: action.brushCoords
       };
     }
+    case "lasso deselect":
     case "graph brush deselect": {
-      state.dimensionMap[layoutDimensionName("X")].filterAll();
-      state.dimensionMap[layoutDimensionName("Y")].filterAll();
+      state.dimensionMap[layoutDimensionName("XY")].filterAll();
       return {
         ...state,
         graphBrushSelection: null
       };
     }
+    case "lasso selection": {
+      const { polygon } = action;
+      const dXY = state.dimensionMap[layoutDimensionName("XY")];
+      if (polygon.length < 3) {
+        // single point or a line is not a polygon, and is therefore a deselect
+        dXY.filterAll();
+      } else {
+        dXY.filterWithinPolygon(polygon);
+      }
+      return {
+        ...state
+      };
+    }
     case "continuous metadata histogram brush": {
       const name = makeContinuousDimensionName(
         action.continuousNamespace,
@@ -418,87 +451,99 @@ const Controls = (
         graphRenderCounter: c
       };
     }
+    case "interface reset started": {
+      return {
+        ...state,
+        resettingInterface: true
+      };
+    }
     /*******************************
           Categorical metadata
     *******************************/
     case "categorical metadata filter select": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap,
+      const newCategorySelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].categorySelected
+      );
+      newCategorySelected[action.categoryIndex] = true;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
         [action.metadataField]: {
-          ...state.categoricalAsBooleansMap[action.metadataField],
-          [action.value]: true
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: newCategorySelected
         }
       };
-      // update the filter for the one category that changed state
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
       state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        _.filter(
-          _.map(
-            newCategoricalAsBooleansMap[action.metadataField],
-            (val, key) => (val ? key : false)
-          )
-        )
+        ControlsHelper.selectedValuesForCategory(cat)
       );
+
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
     case "categorical metadata filter deselect": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap,
+      const newCategorySelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].categorySelected
+      );
+      newCategorySelected[action.categoryIndex] = false;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
         [action.metadataField]: {
-          ...state.categoricalAsBooleansMap[action.metadataField],
-          [action.value]: false
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: newCategorySelected
         }
       };
-      // update the filter for the one category that changed state
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
       state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        _.filter(
-          _.map(
-            newCategoricalAsBooleansMap[action.metadataField],
-            (val, key) => (val ? key : false)
-          )
-        )
+        ControlsHelper.selectedValuesForCategory(cat)
       );
+
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
     case "categorical metadata filter none of these": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap
-      };
-      _.forEach(
-        newCategoricalAsBooleansMap[action.metadataField],
-        (v, k, c) => {
-          c[k] = false;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: Array.from(
+            state.categoricalSelectionState[action.metadataField]
+              .categorySelected
+          ).fill(false)
         }
-      );
+      };
       state.dimensionMap[
         obsAnnoDimensionName(action.metadataField)
       ].filterNone();
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
     case "categorical metadata filter all of these": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap
-      };
-      _.forEach(
-        newCategoricalAsBooleansMap[action.metadataField],
-        (v, k, c) => {
-          c[k] = true;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: Array.from(
+            state.categoricalSelectionState[action.metadataField]
+              .categorySelected
+          ).fill(true)
         }
-      );
+      };
       state.dimensionMap[
         obsAnnoDimensionName(action.metadataField)
       ].filterAll();
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
 
@@ -509,7 +554,6 @@ const Controls = (
     case "color by continuous metadata": {
       return {
         ...state,
-        colorName: action.colors.name,
         colorRGB: action.colors.rgb,
         colorAccessor: action.colorAccessor,
         colorScale: action.colorScale
@@ -518,7 +562,6 @@ const Controls = (
     case "color by expression": {
       return {
         ...state,
-        colorName: action.colors.name,
         colorRGB: action.colors.rgb,
         colorAccessor: action.gene,
         colorScale: action.colorScale

client/src/reducers/index.js

@@ -1,9 +1,9 @@
 // jshint esversion: 6
 import { combineReducers, createStore, applyMiddleware } from "redux";
 import thunk from "redux-thunk";
+import { composeWithDevTools } from "redux-devtools-extension";
 import updateURLMiddleware from "../middleware/updateURLMiddleware";
 import updateCellColors from "../middleware/updateCellColors";
-import { composeWithDevTools } from "redux-devtools-extension";
 
 import config from "./config";
 import differential from "./differential";

client/src/util/actionHelpers.js

@@ -25,24 +25,45 @@ export function catchErrorsWrap(fn, dispatchToUser = false) {
 }
 
 /*
-Wrapper to perform an async fetch and JSON decode response.
+Wrapper to perform async fetch with some modest error handling
+and decoding.
 */
-export const doJsonRequest = async url => {
+const doFetch = async (url, acceptType) => {
   const res = await fetch(url, {
     method: "get",
     headers: new Headers({
-      "Content-Type": "application/json"
+      Accept: acceptType
     })
   });
-  if (res.ok && res.headers.get("Content-Type") === "application/json") {
-    return res.json();
+  if (res.ok && res.headers.get("Content-Type").includes(acceptType)) {
+    return res;
   }
   // else an error
-  const msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  let msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  const body = await res.text();
+  if (body && body.length > 0) {
+    msg = `${msg} -- ${body}`;
+  }
   dispatchNetworkErrorMessageToUser(msg);
   throw new Error(msg);
 };
 
+/*
+Wrapper to perform an async fetch and JSON decode response.
+*/
+export const doJsonRequest = async url => {
+  const res = await doFetch(url, "application/json");
+  return res.json();
+};
+
+/*
+Wrapper to perform an async fetch for binary data.
+*/
+export const doBinaryRequest = async url => {
+  const res = await doFetch(url, "application/octet-stream");
+  return res.arrayBuffer();
+};
+
 /*
 This function "packs" filter index lists into the more efficient
 "range" form specified in the REST 0.2 spec.

client/src/util/dataframe/dataframe.js

@@ -0,0 +1,574 @@
+import { IdentityInt32Index, isLabelIndex } from "./labelIndex";
+// weird cross-dependency that we should clean up someday...
+import { sort } from "../typedCrossfilter/sort";
+import { isTypedArray, isArrayOrTypedArray, callOnceLazy } from "./util";
+import { summarizeContinuous, summarizeCategorical } from "./summarize";
+
+/*
+Dataframe is an immutable 2D matrix similiar to Python Pandas Dataframe,
+but (currently) without all of the surrounding support functions.
+Data is stored in column-major layout, and each column is monomorphic.
+
+It supports:
+* Relatively efficient creation, cloning and subsetting
+* Very efficient columnar access (eg, sum down a column), and access
+  to the underlying column arrays.
+* Data access by row/col offset or label.  Labels are reasonably well
+  optimized for both numeric lables and arbitrary (eg, sting) labels.
+
+It does not currently support:
+* Views on matrix subset - for currently known access patterns,
+  it is more effiicent to copy on subsetting, optimizing for access
+  speed over memory use.
+* JS iterators - they are too slow.  Use explicit iteration over
+  offest or labels.
+
+Important assumptions embedded in the API:
+* Columns are implicitly categorical if they are a JS Array and numeric
+  (aka continuous) if they are a TypedArray.
+
+There are three index types for row/col indexing:
+* IdentityInt32Index - noop index, where the index label is the offset.
+* KeyIndex - index arbitrary JS objects.
+* DenseInt32Index - integer indexing.  Optimization over KeyIndex as it uses
+  Int32Array as a back-map to offsets.  This means that the index array
+  must be sized to [minLabel, maxLabel), so this is only useful when the label
+  range is relatively close the underlying offset range [minOffset, maxOffset).
+
+All private functions/methods/fields are prefixed by '__', eg, __compile().
+Don't use them outside of this file.
+
+Simple example:
+
+  // default indexing is integer offset.
+  const df = Dataframe.create([2,2], [['a', 'b'], [0, 1]])
+  console.log(df.at(0,0));  // outputs: a
+  console.log(df.col(1).asArray());   // outputs: [0, 1]
+
+  // KeyIndex
+  const df = new Dataframe([1,2], [['a'], ['b']], null, new KeyIndex(['A', 'B']))
+  console.log(df.at(0, 'A')); // outputs: a
+  console.log(df.col('A').asArray(); // outputs: ['a']
+
+Performance tuning is primarily focused on columnar access patterns, which is the
+dominant pattern in cellxgene.
+*/
+
+/**
+Dataframe
+**/
+
+class Dataframe {
+  /**
+  Constructors & factories
+  **/
+
+  constructor(dims, columnarData, rowIndex = null, colIndex = null) {
+    /*
+    The base constructor is relatively hard to use - as an alternative,
+    see factory methods and clone/slice, below.
+
+    Parameters:
+      * dims - 2D array describing intendend dimensionality: [nRows,nCols].
+      * columnarData - JS array, nCols in length, containing array
+        or TypedArray of length nRows.
+      * rowIndex/colIndex - null (create default index using offsets as key),
+        or a caller-provided index.
+    All columns and indices must have appropriate dimensionality.
+    */
+    const [nRows, nCols] = dims;
+    if (nRows < 0 || nCols < 0) {
+      throw new RangeError("Dataframe dimensions must be positive");
+    }
+    if (!rowIndex) {
+      rowIndex = new IdentityInt32Index(nRows);
+    }
+    if (!colIndex) {
+      colIndex = new IdentityInt32Index(nCols);
+    }
+    Dataframe.__errorChecks(dims, columnarData, rowIndex, colIndex);
+
+    this.__columns = Array.from(columnarData);
+    this.dims = dims;
+    this.length = nRows; // convenience accessor for row dimension
+    this.rowIndex = rowIndex;
+    this.colIndex = colIndex;
+
+    this.__compile();
+  }
+
+  static __errorChecks(dims, columnarData, rowIndex, colIndex) {
+    const [nRows, nCols] = dims;
+
+    /* check for expected types */
+    if (!Array.isArray(columnarData)) {
+      throw new TypeError("Dataframe constructor requires array of columns");
+    }
+    if (!columnarData.every(c => isArrayOrTypedArray(c))) {
+      throw new TypeError("Dataframe columns must all be Array or TypedArray");
+    }
+    if (!isLabelIndex(rowIndex)) {
+      throw new TypeError("Dataframe rowIndex is an unsupported type.");
+    }
+    if (!isLabelIndex(colIndex)) {
+      throw new TypeError("Dataframe colIndex is an unsupported type.");
+    }
+
+    /* check for expected dimensionality / size */
+    if (
+      nCols !== columnarData.length ||
+      !columnarData.every(c => c.length === nRows)
+    ) {
+      throw new RangeError(
+        "Dataframe dimension does not match provided data shape"
+      );
+    }
+    if (nRows !== rowIndex.size()) {
+      throw new RangeError(
+        "Dataframe rowIndex must have same size as underlying data"
+      );
+    }
+    if (nCols !== colIndex.size()) {
+      throw new RangeError(
+        "Dataframe colIndex must have same size as underlying data"
+      );
+    }
+  }
+
+  __compile() {
+    /*
+    Compile data accessors for each column.
+
+    Each column accessor is a function which will lookup data by
+    index (ie, is equivalent to dataframe.get(row, col), where 'col'
+    is fixed.
+
+    In addition, each column accessor has several functions:
+
+    asArray() -- return the entire column as a native Array or TypedArray.
+      Crucially, this native array only supports label indexing.
+      Example:
+        const arr = df.col('a').asArray();
+
+    has(rlabel) -- return boolean indicating of the row label
+      is contained within the column.  Example:
+        const isInColumn = df.col('a').includes(99)
+      For the default offset indexing, this is identical to:
+        const isInColumn = (99 > 0) && (99 < df.nRows);
+
+    ihas(roffset) -- same as has(), but accepts a row offset
+      instead of a row label.
+
+    indexOf(value) -- return the label (not offset) of the first instance of
+      'value' in the column.  If you want the offset, just use the builtin JS
+      indexOf() function, available on both Array and TypedArray.
+
+    iget(offset) -- return the value at 'offset'
+
+    */
+    const { getOffset, getLabel } = this.rowIndex;
+    this.__columnsAccessor = this.__columns.map(column => {
+      const { length } = column;
+
+      /* get value by row label */
+      const get = function get(rlabel) {
+        return column[getOffset(rlabel)];
+      };
+
+      /* get value by row offset */
+      const iget = function iget(roffset) {
+        return column[roffset];
+      };
+
+      /* full column array access */
+      const asArray = function asArray() {
+        return column;
+      };
+
+      /* test for row label inclusion in column */
+      const has = function has(rlabel) {
+        const offset = getOffset(rlabel);
+        return offset >= 0 && offset < length;
+      };
+
+      const ihas = function ihas(offset) {
+        return offset >= 0 && offset < length;
+      };
+
+      /*
+      return first label (index) at which the value is found in this column,
+      or undefined if not found.
+
+      NOTE: not found return is DIFFERENT than the default Array.indexOf as
+      -1 is a plausible Dataframe row/col label.
+      */
+      const indexOf = function indexOf(value) {
+        const offset = column.indexOf(value);
+        if (offset === -1) {
+          return undefined;
+        }
+        return getLabel(offset);
+      };
+
+      /*
+      Summarize the column data. Lazy eval;
+      */
+      const summarize = callOnceLazy(() =>
+        isTypedArray(column)
+          ? summarizeContinuous(column)
+          : summarizeCategorical(column)
+      );
+
+      get.summarize = summarize;
+      get.asArray = asArray;
+      get.has = has;
+      get.ihas = ihas;
+      get.indexOf = indexOf;
+      get.iget = iget;
+      return get;
+    });
+  }
+
+  clone() {
+    /*
+    Clone this dataframe
+    */
+    return new this.constructor(
+      this.dims,
+      [...this.__columns],
+      this.rowIndex,
+      this.colIndex
+    );
+  }
+
+  withCol(label, colData, withRowIndex = null) {
+    /*
+    Create a new DF, which is `this` plus the new column. Example:
+    const newDf = df.withCol("foo", [1,2,3]);
+
+    Dimensionality of new column must match existing dataframe.
+
+    Special case: empty dataframe will accept any size column.  Example:
+    const newDf = Dataframe.empty().withCol("foo", [1,2,3]);
+
+    If `withRowIndex` specified, the provided index will become the
+    rowIndex for the newly created dataframe.   If not specified,
+    the rowIndex from `this` will be used (ie, the rowIndex is
+    unchanged).
+    */
+    let dims;
+    let rowIndex;
+    if (this.isEmpty()) {
+      dims = [colData.length, 1];
+      rowIndex = null;
+    } else {
+      dims = [this.dims[0], this.dims[1] + 1];
+      ({ rowIndex } = this);
+    }
+
+    if (withRowIndex) {
+      rowIndex = withRowIndex;
+    }
+
+    const columns = [...this.__columns];
+    columns.push(colData);
+    const colIndex = this.colIndex.withLabel(label);
+    return new this.constructor(dims, columns, rowIndex, colIndex);
+  }
+
+  dropCol(label) {
+    /*
+    Create a new dataframe, omitting one columns.
+
+    const newDf = df.dropCol("colors");
+    */
+    const dims = [this.dims[0], this.dims[1] - 1];
+    const coffset = this.colIndex.getOffset(label);
+    const columns = [...this.__columns];
+    columns.splice(coffset, 1);
+    const colIndex = this.colIndex.dropLabel(label);
+    return new this.constructor(dims, columns, this.rowIndex, colIndex);
+  }
+
+  static empty(rowIndex = null, colIndex = null) {
+    return new Dataframe([0, 0], [], rowIndex, colIndex);
+  }
+
+  static create(dims, columnarData) {
+    /*
+    Create a dataframe from raw columnar data.  All column arrays
+    must have the same length.   Identity indexing will be used.
+
+    Example:
+    const df = Dataframe.create([2,2], [new Uint32Array(2), new Float32Array(2)]);
+    */
+    return new Dataframe(dims, columnarData, null, null);
+  }
+
+  __subset(rowOffsets, colOffsets, withRowIndex) {
+    const dims = [...this.dims];
+
+    const getSortedLabelAndOffsets = (offsets, index) => {
+      /*
+      Given offsets, return both offsets and associated lables,
+      sorted by offset.
+      */
+      if (!offsets) {
+        return [null, null];
+      }
+      const sortedOffsets = sort(offsets);
+      const sortedLabels = new Array(sortedOffsets.length);
+      for (let i = 0, l = sortedOffsets.length; i < l; i += 1) {
+        sortedLabels[i] = index.getLabel(sortedOffsets[i]);
+      }
+      return [sortedLabels, sortedOffsets];
+    };
+
+    let { colIndex } = this;
+    if (colOffsets) {
+      let colLabels;
+      [colLabels, colOffsets] = getSortedLabelAndOffsets(
+        colOffsets,
+        this.colIndex
+      );
+      dims[1] = colOffsets.length;
+      colIndex = this.colIndex.subsetLabels(colLabels);
+    }
+
+    let { rowIndex } = this;
+    if (withRowIndex) rowIndex = withRowIndex;
+    if (rowOffsets) {
+      let rowLabels;
+      [rowLabels, rowOffsets] = getSortedLabelAndOffsets(
+        rowOffsets,
+        this.rowIndex
+      );
+      dims[0] = rowLabels.length;
+      if (!withRowIndex) rowIndex = this.rowIndex.subsetLabels(rowLabels);
+    }
+
+    /* subset columns */
+    let columns = this.__columns;
+    if (colOffsets) {
+      columns = new Array(colOffsets.length);
+      for (let i = 0, l = colOffsets.length; i < l; i += 1) {
+        columns[i] = this.__columns[colOffsets[i]];
+      }
+    }
+
+    /* subset rows */
+    if (rowOffsets) {
+      columns = columns.map(col => {
+        const newCol = new col.constructor(rowOffsets.length);
+        for (let i = 0, l = rowOffsets.length; i < l; i += 1) {
+          newCol[i] = col[rowOffsets[i]];
+        }
+        return newCol;
+      });
+    }
+    return new Dataframe(dims, columns, rowIndex, colIndex);
+  }
+
+  subset(rowLabels, colLabels = null, withRowIndex = null) {
+    /*
+    Subset by row/col labels.
+
+    withRowIndex allows assignment of new row index during subset operation.
+    If withRowIndex === null, it will reset the index to identity (offset)
+    indexing.  if withRowIndex is a label index object, it will be used
+    for the new dataframe.
+    */
+    const toOffsets = (labels, index) => {
+      if (!labels) {
+        return null;
+      }
+      return labels.map(label => {
+        const off = index.getOffset(label);
+        if (off === undefined) {
+          throw new RangeError(`unknown label: ${label}`);
+        }
+        return off;
+      });
+    };
+
+    const rowOffsets = toOffsets(rowLabels, this.rowIndex);
+    const colOffsets = toOffsets(colLabels, this.colIndex);
+    return this.__subset(rowOffsets, colOffsets, withRowIndex);
+  }
+
+  isubset(rowOffsets, colOffsets = null, withRowIndex = null) {
+    /*
+    Subset by row/col offset.
+
+    withRowIndex allows assignment of new row index during subset operation.
+    If withRowIndex === null, it will reset the index to identity (offset)
+    indexing.  if withRowIndex is a label index object, it will be used
+    for the new dataframe.
+    */
+    return this.__subset(rowOffsets, colOffsets, withRowIndex);
+  }
+
+  isubsetMask(rowMask, colMask = null, withRowIndex = null) {
+    /*
+    Subset on row/column based upon a truthy/falsey array (a mask).
+
+    withRowIndex allows assignment of new row index during subset operation.
+    If withRowIndex === null, it will reset the index to identity (offset)
+    indexing.  if withRowIndex is a label index object, it will be used
+    for the new dataframe.
+    */
+    const [nRows, nCols] = this.dims;
+    if (
+      (rowMask && rowMask.length !== nRows) ||
+      (colMask && colMask.length !== nCols)
+    ) {
+      throw new RangeError("boolean arrays must match row/col dimensions");
+    }
+
+    /* convert masks to lists - method wastes space, but is fast */
+    const toList = (mask, maxSize) => {
+      if (!mask) {
+        return null;
+      }
+      const list = new Int32Array(maxSize);
+      let elems = 0;
+      for (let i = 0, l = mask.length; i < l; i += 1) {
+        if (mask[i]) {
+          list[elems] = i;
+          elems += 1;
+        }
+      }
+      return new Int32Array(list.buffer, 0, elems);
+    };
+    const rowOffsets = toList(rowMask, nRows);
+    const colOffsets = toList(colMask, nCols);
+    return this.__subset(rowOffsets, colOffsets, withRowIndex);
+  }
+
+  /**
+  Data access with row/col.
+  **/
+
+  col(columnLabel) {
+    /*
+    Return accessor bound to a column.  Allows random row access
+    based upon the row indexing.  Returns undefined if the
+    columnLabel is not present in the dataframe.
+
+    Example for a dataframe with string labeled columns, and
+    default (offset) indices for rows (eg, [0, 'foo'])
+
+      const getValue = df.col('foo');
+      for (let r = 0; r < df.nRows; r += 1) {
+        console.log(r, getValue(r));
+      }
+
+    See __compile() for the functions available in a column accessor.
+    */
+    const coff = this.colIndex.getOffset(columnLabel);
+    return this.__columnsAccessor[coff];
+  }
+
+  icol(columnOffset) {
+    /*
+    Return column accessor by offset.
+    */
+    return this.__columnsAccessor[columnOffset];
+  }
+
+  at(r, c) {
+    /*
+    Access a single value, for a row/col label pair.
+
+    For performance reasons, there are no bounds or existance
+    checks on labels, and no defined behavior when these are supplied.
+    May return undefined, throw an Error, or do something else for
+    non-existant labels.  If you want predictable out-of-bounds
+    behavior, use has(), eg,
+
+    const myVal = df.has(r,l) ? df.at(r,l) : undefined;
+    */
+    const coff = this.colIndex.getOffset(c);
+    const roff = this.rowIndex.getOffset(r);
+    return this.__columns[coff][roff];
+  }
+
+  iat(r, c) {
+    /*
+    Access a single value, for a row/col offset (integer) position.
+
+    For performance reasons, there are no bounds checks on row/col offsets
+    or other well-defined behavior for out-of-bounds values.  If you want
+    well-defined bounds checking, use ihas(), eg,
+
+    const myVal = df.ihas(r, c) ? df.iat(r, c) : undefined;
+    */
+    return this.__columns[c][r];
+  }
+
+  has(r, c) {
+    /*
+    Test if row/col labels exist in the dataframe - returns true/false
+    */
+    const [nRows, nCols] = this.dims;
+    const coff = this.colIndex.getOffset(c);
+    const roff = this.rowIndex.getOffset(r);
+    return coff >= 0 && coff < nCols && roff >= 0 && roff < nRows;
+  }
+
+  ihas(r, c) {
+    /*
+    Test if row/col offset (integer) position exists in the
+    dataframe - returns true/false
+    */
+    const [nRows, nCols] = this.dims;
+    return c >= 0 && c < nCols && r >= 0 && r < nRows;
+  }
+
+  hasCol(c) {
+    /*
+    Test if col label exists - return true/false
+    */
+    return !!this.col(c);
+  }
+
+  isEmpty() {
+    /*
+    Return true if this is an empty dataframe, ie, has dimensions [0,0]
+    */
+    const [rows, cols] = this.dims;
+    return rows === 0 && cols === 0;
+  }
+
+  /****
+  Functional (map/reduce/etc) data access
+
+  XXX: not yet implemented, as there is no clear use case.   Can easily
+  add these as useful.
+  ****/
+
+  /*
+  Map & reduce of column or row
+
+  XXX TODO remainder of map/reduce functions:  mapCol, mapRow, reduceRow, ...
+  */
+  /* comment out until we have a use for this
+
+  reduceCol(clabel, callback, initialValue) {
+    const coff = this.colIndex.getOffset(clabel);
+    const column = this.__columns[coff];
+    let start = 0;
+    let acc = initialValue;
+    if (initialValue === undefined) {
+      acc = column[0];
+      start = 1;
+    }
+    for (let i = start, l = column.length; i < l; i += 1) {
+      acc = callback(acc, column[i]);
+    }
+    return acc;
+  }
+  */
+}
+
+export default Dataframe;

client/src/util/dataframe/index.js

@@ -0,0 +1,2 @@
+export { default as Dataframe } from "./dataframe";
+export { DenseInt32Index, IdentityInt32Index, KeyIndex } from "./labelIndex";

client/src/util/dataframe/labelIndex.js

@@ -0,0 +1,244 @@
+/**
+Label indexing - map a label to & from an integer offset.  See Dataframe
+for how this is used.
+**/
+
+/*
+Private utility functions
+*/
+function extent(tarr) {
+  let min = 0x7fffffff;
+  let max = ~min; // eslint-disable-line no-bitwise
+  for (let i = 0, l = tarr.length; i < l; i += 1) {
+    const v = tarr[i];
+    if (v < min) {
+      min = v;
+    }
+    if (v > max) {
+      max = v;
+    }
+  }
+  return [min, max];
+}
+
+function fillRange(arr, start = 0) {
+  const larr = arr;
+  for (let i = 0, l = larr.length; i < l; i += 1) {
+    larr[i] = i + start;
+  }
+  return larr;
+}
+
+/* eslint-disable class-methods-use-this */
+class IdentityInt32Index {
+  /*
+  identity/noop index, with small assumptions that labels are int32
+  */
+  constructor(maxOffset) {
+    this.maxOffset = maxOffset;
+  }
+
+  keys() {
+    // memoize
+    const k = fillRange(new Int32Array(this.maxOffset));
+    this.keys = function keys() {
+      return k;
+    };
+    return k;
+  }
+
+  getOffset(i) {
+    // label to offset
+    return i;
+  }
+
+  getLabel(i) {
+    // offset to label
+    return i;
+  }
+
+  size() {
+    return this.maxOffset;
+  }
+
+  __promote(labelArray) {
+    /*
+    time/space decision - based on the resulting density
+    */
+    const [minLabel, maxLabel] = extent(labelArray);
+    const labelSpaceSize = maxLabel - minLabel + 1;
+    const density = labelSpaceSize / this.maxOffset;
+    /* 0.1 is a magic number, that needs testing to optimize */
+    if (density < 0.1) {
+      return new KeyIndex(labelArray);
+    }
+    return new DenseInt32Index(labelArray, [minLabel, maxLabel]);
+  }
+
+  subsetLabels(labelArray) {
+    return this.__promote(labelArray);
+  }
+
+  withLabel(label) {
+    if (label === this.maxOffset) {
+      return new IdentityInt32Index(label + 1);
+    }
+    return this.__promote([...this.keys(), label]);
+  }
+
+  dropLabel(label) {
+    if (label === this.maxOffset - 1) {
+      return new IdentityInt32Index(label);
+    }
+    const labelArray = [...this.keys()];
+    labelArray.splice(labelArray.indexOf(label), 1);
+    return this.__promote(labelArray);
+  }
+}
+/* eslint-enable class-methods-use-this */
+
+/* eslint-disable class-methods-use-this */
+class DenseInt32Index {
+  /*
+  DenseInt32Index indexes integer labels, and uses Int32Array typed arrays
+  for both forward and reverse indexing.   This means that the min/max range
+  of the forward index labels must be known a priori (so that the index
+  array can be pre-allocated).
+  */
+  constructor(labels, labelRange = null) {
+    if (labels.constructor !== Int32Array) {
+      labels = new Int32Array(labels);
+    }
+
+    if (!labelRange) {
+      labelRange = extent(labels);
+    }
+    const [minLabel, maxLabel] = labelRange;
+    const labelSpaceSize = maxLabel - minLabel + 1;
+    const index = new Int32Array(labelSpaceSize).fill(-1);
+    for (let i = 0, l = labels.length; i < l; i += 1) {
+      const label = labels[i];
+      index[label - minLabel] = i;
+    }
+
+    this.minLabel = minLabel;
+    this.rindex = labels;
+    this.index = index;
+    this.__compile();
+  }
+
+  __compile() {
+    const { minLabel, index, rindex } = this;
+    this.getOffset = function getOffset(l) {
+      return index[l - minLabel];
+    };
+    this.getLabel = function getLabel(i) {
+      return rindex[i];
+    };
+  }
+
+  keys() {
+    return this.rindex;
+  }
+
+  size() {
+    return this.rindex.length;
+  }
+
+  __promote(labelArray) {
+    /*
+    time/space decision - if we are going to use less than 10% of the
+    dense index space, switch to a KeyIndex (which is slower, but uses
+    less memory for sparse label spaces).
+    */
+    const [minLabel, maxLabel] = extent(labelArray);
+    const labelSpaceSize = maxLabel - minLabel + 1;
+    const density = labelSpaceSize / this.rindex.length;
+    /* 0.1 is a magic number, that needs testing to optimize */
+    if (density < 0.1) {
+      return new KeyIndex(labelArray);
+    }
+    return new DenseInt32Index(labelArray, [minLabel, maxLabel]);
+  }
+
+  subsetLabels(labelArray) {
+    return this.__promote(labelArray);
+  }
+
+  withLabel(label) {
+    return this.__promote([...this.keys(), label]);
+  }
+
+  dropLabel(label) {
+    const labelArray = [...this.keys()];
+    labelArray.splice(labelArray.indexOf(label), 1);
+    return this.__promote(labelArray);
+  }
+}
+/* eslint-enable class-methods-use-this */
+
+/* eslint-disable class-methods-use-this */
+class KeyIndex {
+  /*
+  KeyIndex indexes arbitrary JS primitive types, and uses a Map()
+  as its core data structure.
+  */
+  constructor(labels) {
+    const index = new Map();
+    if (labels === undefined) {
+      labels = [];
+    }
+    const rindex = labels;
+    labels.forEach((v, i) => {
+      index.set(v, i);
+    });
+
+    this.index = index;
+    this.rindex = rindex;
+    this.__compile();
+  }
+
+  __compile() {
+    const { index, rindex } = this;
+    this.getOffset = function getOffset(k) {
+      return index.get(k);
+    };
+    this.getLabel = function getLabel(i) {
+      return rindex[i];
+    };
+  }
+
+  keys() {
+    return this.rindex;
+  }
+
+  size() {
+    return this.rindex.length;
+  }
+
+  subsetLabels(labelArray) {
+    return new KeyIndex(labelArray);
+  }
+
+  withLabel(label) {
+    return new KeyIndex([...this.rindex, label]);
+  }
+
+  dropLabel(label) {
+    const idx = this.rindex.indexOf(label);
+    const labelArray = [...this.rindex];
+    labelArray.splice(idx, 1);
+    return new KeyIndex(labelArray);
+  }
+}
+/* eslint-enable class-methods-use-this */
+
+function isLabelIndex(i) {
+  return (
+    i instanceof IdentityInt32Index ||
+    i instanceof DenseInt32Index ||
+    i instanceof KeyIndex
+  );
+}
+
+export { DenseInt32Index, IdentityInt32Index, KeyIndex, isLabelIndex };

client/src/util/dataframe/summarize.js

@@ -0,0 +1,57 @@
+/*
+Private dataframe support functions
+*/
+
+export function summarizeContinuous(col) {
+  let min;
+  let max;
+  let nan = 0;
+  let pinf = 0;
+  let ninf = 0;
+  if (col) {
+    for (let r = 0, l = col.length; r < l; r += 1) {
+      const val = Number(col[r]);
+      if (Number.isFinite(val)) {
+        if (min === undefined) {
+          min = val;
+          max = val;
+        } else {
+          min = val < min ? val : min;
+          max = val > max ? val : max;
+        }
+      } else if (Number.isNaN(val)) {
+        nan += 1;
+      } else if (val > 0) {
+        pinf += 1;
+      } else {
+        ninf += 1;
+      }
+    }
+  }
+  return {
+    categorical: false,
+    min,
+    max,
+    nan,
+    pinf,
+    ninf
+  };
+}
+
+export function summarizeCategorical(col) {
+  const categoryCounts = new Map();
+  if (col) {
+    for (let r = 0, l = col.length; r < l; r += 1) {
+      const val = col[r];
+      let curCount = categoryCounts.get(val);
+      if (curCount === undefined) curCount = 0;
+      categoryCounts.set(val, curCount + 1);
+    }
+  }
+  return {
+    categorical: true,
+    categories: [...categoryCounts.keys()],
+    categoryCounts,
+    numCategories: categoryCounts.size
+  };
+}

client/src/util/dataframe/util.js

@@ -0,0 +1,28 @@
+/*
+Private utility code for dataframe
+*/
+
+export function isTypedArray(x) {
+  return (
+    ArrayBuffer.isView(x) &&
+    Object.prototype.toString.call(x) !== "[object DataView]"
+  );
+}
+
+export function isArrayOrTypedArray(x) {
+  return Array.isArray(x) || isTypedArray(x);
+}
+
+export function callOnceLazy(f) {
+  let value;
+  let calledOnce = false;
+  const result = function result(...args) {
+    if (!calledOnce) {
+      value = f(...args);
+      calledOnce = true;
+    }
+    return value;
+  };
+
+  return result;
+}

client/src/util/finiteExtent.js

@@ -0,0 +1,33 @@
+/*
+Return the [minimum, maximum] extent, of the given typed array, ignoring
+non-finite values (ie, +Infinity, -Infinity).
+
+If undefined or empty array, or array contains only non-finite numbers,
+will return [undefined, undefined]
+*/
+
+function finiteExtent(tarr) {
+  let min;
+  let max;
+  let i;
+
+  for (i = 0; i < tarr.length; i += 1) {
+    const val = tarr[i];
+    if (Number.isFinite(val)) {
+      min = val;
+      max = val;
+      i += 1;
+      break;
+    }
+  }
+  for (; i < tarr.length; i += 1) {
+    const val = tarr[i];
+    if (Number.isFinite(val)) {
+      if (min > val) min = val;
+      if (max < val) max = val;
+    }
+  }
+  return [min, max];
+}
+
+export default finiteExtent;

client/src/util/stateManager/controlsHelpers.js

@@ -0,0 +1,164 @@
+/*
+Helper functions for the controls reducer
+*/
+
+import _ from "lodash";
+
+import * as globals from "../../globals";
+import { fillRange } from "../typedCrossfilter/util";
+import {
+  userDefinedDimensionName,
+  diffexpDimensionName
+} from "../nameCreators";
+import * as World from "./world";
+
+/*
+Selection state for categoricals are tracked in an Object that
+has two main components for each category:
+1. mapping of option value to an index
+2. array of bool selection state by index
+Remember that option values can be ANY js type, except undefined/null.
+
+  {
+    _category_name_1: {
+      // map of option value to index
+      categoryIndices: Map([
+        catval1: index,
+        ...
+      ])
+
+      // index->selection true/false state
+      categorySelected: [ true/false, true/false, ... ]
+
+      // number of options
+      numCategories: number,
+
+      // isTruncated - true if the options for selection has
+      // been truncated (ie, was too large to implement)
+    }
+  }
+*/
+function topNCategories(summary) {
+  const counts = _.map(summary.categories, cat =>
+    summary.categoryCounts.get(cat)
+  );
+  const sortIndex = fillRange(new Array(summary.numCategories)).sort(
+    (a, b) => counts[b] - counts[a]
+  );
+  const sortedCategories = _.map(sortIndex, i => summary.categories[i]);
+  const sortedCounts = _.map(sortIndex, i => counts[i]);
+  const N = globals.maxCategoricalOptionsToDisplay;
+
+  if (sortedCategories.length < N) {
+    return [sortedCategories, sortedCounts];
+  }
+  return [sortedCategories.slice(0, N), sortedCounts.slice(0, N)];
+}
+
+export function createCategoricalSelectionState(state, world) {
+  const res = {};
+  _.forEach(world.obsAnnotations.colIndex.keys(), key => {
+    const summary = world.obsAnnotations.col(key).summarize();
+    if (summary.categories) {
+      const isColorField = key.includes("color") || key.includes("Color");
+      const isSelectableCategory =
+        !isColorField &&
+        key !== "name" &&
+        summary.categories.length < state.maxCategoryItems;
+      if (isSelectableCategory) {
+        const [categoryValues, categoryCounts] = topNCategories(summary);
+        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
+        const numCategories = categoryIndices.size;
+        const categorySelected = new Array(numCategories).fill(true);
+        const isTruncated = categoryValues.length < summary.numCategories;
+        res[key] = {
+          categoryValues, // array: of natively typed category values
+          categoryIndices, // map: category value (native type) -> category index
+          categorySelected, // array: t/f selection state
+          numCategories, // number: of categories
+          isTruncated, // bool: true if list was truncated
+          categoryCounts // array: cardinality of each category
+        };
+      }
+    }
+  });
+  return res;
+}
+
+/*
+given a categoricalSelectionState, return the list of all category values
+where selection state is true (ie, they are selected).
+*/
+export function selectedValuesForCategory(categorySelectionState) {
+  const selectedValues = _([...categorySelectionState.categoryIndices])
+    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
+    .map(tuple => tuple[0])
+    .value();
+  return selectedValues;
+}
+
+/*
+build a crossfilter dimension map for all gene expression related dimensions.
+*/
+export function createGenesDimMap(
+  userDefinedGenes,
+  diffexpGenes,
+  world,
+  crossfilter
+) {
+  function _createGenesDimMap(genes, nameCreator) {
+    return genes.reduce((acc, gene) => {
+      acc[nameCreator(gene)] = World.createVarDataDimension(
+        world,
+        crossfilter,
+        gene
+      );
+      return acc;
+    }, {});
+  }
+
+  return {
+    ..._createGenesDimMap(userDefinedGenes, userDefinedDimensionName),
+    ..._createGenesDimMap(diffexpGenes, diffexpDimensionName)
+  };
+}
+
+export function pruneVarDataCache(varData, needed) {
+  /*
+  Remove any unneeded columns from the varData dataframe.  Will only
+  prune / remove if the total column count exceeds VarDataCacheLowWatermark
+
+  Note: this code leverages the fact that dataframe offsets indicate
+  the order in which the columns were added.  This crudely provides
+  LRU semantics, so we can delete "older" columns first.
+  */
+
+  /*
+  VarDataCacheLowWatermark - this cofig value sets the minimum cache size,
+  in columns, below which we don't throw away data.
+
+  The value should be high enough so we are caching the maximum which will
+  "typically" be used in the UI (currently: 10 for diffexp, and N for user-
+  specified genes), and low enough to account for memory use (any single
+  column size is 4 bytes * numObs, so a column can be multi-megabyte in common
+  use cases).
+  */
+  const VarDataCacheLowWatermark = 32;
+
+  const numOverWatermark = varData.dims[1] - VarDataCacheLowWatermark;
+  if (numOverWatermark <= 0) return varData;
+
+  const { colIndex } = varData;
+  const all = colIndex.keys();
+  const unused = _.difference(all, needed);
+  if (unused.length > 0) {
+    // sort by offset in the dataframe - ie, psuedo-LRU
+    unused.sort((a, b) => colIndex.getOffset(a) - colIndex.getOffset(b));
+    const numToDrop =
+      unused.length < numOverWatermark ? unused.length : numOverWatermark;
+    for (let i = 0; i < numToDrop; i += 1) {
+      varData = varData.dropCol(unused[i]);
+    }
+  }
+  return varData;
+}

client/src/util/stateManager/index.js

@@ -3,7 +3,7 @@
 /*
 Model manager providing an abstraction for the use of the reducer code.
 This module provides several buckets of functionality:
-  - schema and config driven tranformation of the dataframe wire protocol
+  - schema and config driven tranformation of the wire protocol
     into a format that is easy for the UI code to use.
   - manage the universe/world abstraction:
     + universe: all of the server-provided, read-only data
@@ -16,4 +16,5 @@ exists to support those concepts.
 
 export * as Universe from "./universe";
 export * as World from "./world";
-export * as kvCache from "./keyvalcache";
+export * as WorldUtil from "./worldUtil";
+export * as ControlsHelper from "./controlsHelpers";

client/src/util/stateManager/keyvalcache.js

@@ -1,122 +0,0 @@
-// jshint esversion: 6
-import _ from "lodash";
-
-/*
-Very simple key/value cache for use by World & Universe.   Cache keys must
-be a string, and values are any JS non-primitive value.
-
-  * constructor(lowWatermark, minTTL):
-      - lowWatermark defines the number of cache elements below which
-        flushing will not occur.
-      - minTTL defines minimum time in milliseconds that cache entries will live.
-        A value of -1 disables automatic flushing (flush() can still
-        be called by external user).
-  * set() - add a key/val pair.
-  * get() - get a value or undefined if not present.
-  * flush(minAgeMs) - flush cache entries in excess of lowWatermark if those
-    entries are older than minAgeMs.
-
-*/
-
-const cachePrivateKey = "__kvcachekey__";
-const defaultLowWatermark = 32;
-const defaultMinTTL = 1000;
-
-function create(lowWatermark = defaultLowWatermark, minTTL = defaultMinTTL) {
-  if (typeof minTTL !== "number" || typeof lowWatermark !== "number") {
-    throw new TypeError(
-      "minTTL and lowWatermark parameters must be a primitive number"
-    );
-  }
-  if (lowWatermark < 0 || minTTL < 0) {
-    throw new RangeError(
-      "minTTL and lowWatermark parameters must be number greater than zero"
-    );
-  }
-
-  return {
-    [cachePrivateKey]: {
-      lowWatermark,
-      minTTL
-    }
-  };
-}
-
-function get(kvcache, key) {
-  if (key === cachePrivateKey) {
-    throw new RangeError(`key parameter may not have value ${cachePrivateKey}`);
-  }
-
-  const val = kvcache[key];
-  if (val) {
-    val[cachePrivateKey] = Date.now();
-  }
-  return val;
-}
-
-function set(kvcache, key, val) {
-  if (key === cachePrivateKey) {
-    throw new RangeError(`key parameter may not have value ${cachePrivateKey}`);
-  }
-
-  const newKvCache = { ...kvcache };
-  newKvCache[key] = val;
-  val[cachePrivateKey] = Date.now();
-  flushInPlace(newKvCache);
-  return newKvCache;
-}
-
-function flush(kvcache) {
-  const newKvCache = { ...kvcache };
-  flushInPlace(newKvCache);
-  return newKvCache;
-}
-
-/*
-Flush elements from cache IF cache size is greater than lowWatermark, and
-those elements are older than minAgeMS
-*/
-function flushInPlace(kvCache) {
-  const { lowWatermark, minTTL } = kvCache[cachePrivateKey];
-  const eol = Date.now() - minTTL;
-  const allKeys = _(kvCache)
-    .keys()
-    .filter(k => k !== cachePrivateKey)
-    .sortBy([k => kvCache[k][cachePrivateKey]])
-    .value();
-
-  if (allKeys.length > lowWatermark) {
-    const keysToDelete = _(allKeys)
-      .slice(0, allKeys.length - lowWatermark)
-      .filter(k => kvCache[k][cachePrivateKey] <= eol)
-      .value();
-    _.forEach(keysToDelete, k => delete kvCache[k]);
-  }
-
-  return kvCache;
-}
-
-/*
-use to create a cache that is a transformation of another cache.
-*/
-function map(srcKvCache, cb, createOptions) {
-  const keysInSrcKvCache = _(srcKvCache)
-    .keys()
-    .filter(k => k !== cachePrivateKey)
-    .value();
-  const lowWatermark = _.get(
-    createOptions,
-    "lowWatermark",
-    defaultLowWatermark
-  );
-  const minTTL = _.get(createOptions, "minTTL", defaultMinTTL);
-  const newKvCache = create(lowWatermark, minTTL);
-  _.forEach(keysInSrcKvCache, key => {
-    const val = cb(get(srcKvCache, key), key);
-    newKvCache[key] = val;
-    val[cachePrivateKey] = Date.now();
-  });
-  return newKvCache;
-}
-
-export { create, get, set, flush, map };

client/src/util/stateManager/matrix.js

@@ -0,0 +1,75 @@
+import { flatbuffers } from "flatbuffers";
+import { NetEncoding } from "./matrix_generated";
+
+const utf8Decoder = new TextDecoder("utf-8");
+
+/*
+Matrix flatbuffer decoding support.   See fbs/matrix.fbs
+*/
+
+/*
+Decode NetEncoding.TypedArray
+*/
+function decodeTypedArray(uType, uValF, inplace = false) {
+  if (uType === NetEncoding.TypedArray.NONE) {
+    return null;
+  }
+
+  // Convert to a JS class that supports this type
+  const TypeClass = NetEncoding[NetEncoding.TypedArray[uType]];
+  // Create a TypedArray that references the underlying buffer
+  let arr = uValF(new TypeClass()).dataArray();
+  if (uType === NetEncoding.TypedArray.JSONEncodedArray) {
+    const json = utf8Decoder.decode(arr);
+    arr = JSON.parse(json);
+  } else if (!inplace) {
+    /* force copy to release underlying FBS buffer */
+    arr = new arr.constructor(arr);
+  }
+  return arr;
+}
+
+/*
+Parameter: Uint8Array or ArrayBuffer containing raw flatbuffer Matrix
+Returns: object containing decoded Matrix:
+{
+  nRows: num,
+  nCols: num,
+  columns: [
+    each column, which will be a TypedArray or Array
+  ]
+  colIdx: []|null
+}
+*/
+function decodeMatrixFBS(arrayBuffer, inplace = false) {
+  const bb = new flatbuffers.ByteBuffer(new Uint8Array(arrayBuffer));
+  const df = NetEncoding.Matrix.getRootAsMatrix(bb);
+
+  const nRows = df.nRows();
+  const nCols = df.nCols();
+
+  /* decode columns */
+  const columnsLength = df.columnsLength();
+  const columns = Array(columnsLength).fill(null);
+  for (let c = 0; c < columnsLength; c += 1) {
+    const col = df.columns(c);
+    columns[c] = decodeTypedArray(col.uType(), col.u.bind(col), inplace);
+  }
+
+  /* decode col_idx */
+  const colIdx = decodeTypedArray(
+    df.colIndexType(),
+    df.colIndex.bind(df),
+    inplace
+  );
+
+  return {
+    nRows,
+    nCols,
+    columns,
+    colIdx,
+    rowIdx: null
+  };
+}
+
+export default decodeMatrixFBS;

client/src/util/stateManager/matrix_generated.js

@@ -0,0 +1,835 @@
+// automatically generated by the FlatBuffers compiler, do not modify
+
+/**
+ * @const
+ * @namespace
+ */
+var NetEncoding = NetEncoding || {};
+
+/**
+ * @enum
+ */
+NetEncoding.TypedArray = {
+  NONE: 0, 0: 'NONE',
+  Float32Array: 1, 1: 'Float32Array',
+  Int32Array: 2, 2: 'Int32Array',
+  Uint32Array: 3, 3: 'Uint32Array',
+  Float64Array: 4, 4: 'Float64Array',
+  JSONEncodedArray: 5, 5: 'JSONEncodedArray'
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Float32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Float32Array}
+ */
+NetEncoding.Float32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Float32Array=} obj
+ * @returns {NetEncoding.Float32Array}
+ */
+NetEncoding.Float32Array.getRootAsFloat32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Float32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Float32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readFloat32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Float32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Float32Array}
+ */
+NetEncoding.Float32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Float32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Float32Array.startFloat32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Float32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addFloat32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Float32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float32Array.endFloat32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Uint32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Uint32Array}
+ */
+NetEncoding.Uint32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Uint32Array=} obj
+ * @returns {NetEncoding.Uint32Array}
+ */
+NetEncoding.Uint32Array.getRootAsUint32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Uint32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Uint32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Uint32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Uint32Array}
+ */
+NetEncoding.Uint32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Uint32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Uint32Array.startUint32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Uint32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Uint32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Uint32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Uint32Array.endUint32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Int32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Int32Array}
+ */
+NetEncoding.Int32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Int32Array=} obj
+ * @returns {NetEncoding.Int32Array}
+ */
+NetEncoding.Int32Array.getRootAsInt32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Int32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Int32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readInt32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Int32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Int32Array}
+ */
+NetEncoding.Int32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Int32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Int32Array.startInt32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Int32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Int32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Int32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Int32Array.endInt32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Float64Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Float64Array}
+ */
+NetEncoding.Float64Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Float64Array=} obj
+ * @returns {NetEncoding.Float64Array}
+ */
+NetEncoding.Float64Array.getRootAsFloat64Array = function(bb, obj) {
+  return (obj || new NetEncoding.Float64Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Float64Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readFloat64(this.bb.__vector(this.bb_pos + offset) + index * 8) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Float64Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Float64Array}
+ */
+NetEncoding.Float64Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Float64Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Float64Array.startFloat64Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Float64Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float64Array.createDataVector = function(builder, data) {
+  builder.startVector(8, data.length, 8);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addFloat64(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Float64Array.startDataVector = function(builder, numElems) {
+  builder.startVector(8, numElems, 8);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float64Array.endFloat64Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.JSONEncodedArray = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.JSONEncodedArray}
+ */
+NetEncoding.JSONEncodedArray.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.JSONEncodedArray=} obj
+ * @returns {NetEncoding.JSONEncodedArray}
+ */
+NetEncoding.JSONEncodedArray.getRootAsJSONEncodedArray = function(bb, obj) {
+  return (obj || new NetEncoding.JSONEncodedArray).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.JSONEncodedArray.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint8(this.bb.__vector(this.bb_pos + offset) + index) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.JSONEncodedArray.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Uint8Array}
+ */
+NetEncoding.JSONEncodedArray.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Uint8Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.JSONEncodedArray.startJSONEncodedArray = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.JSONEncodedArray.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.JSONEncodedArray.createDataVector = function(builder, data) {
+  builder.startVector(1, data.length, 1);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt8(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.JSONEncodedArray.startDataVector = function(builder, numElems) {
+  builder.startVector(1, numElems, 1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.JSONEncodedArray.endJSONEncodedArray = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Column = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Column.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Column=} obj
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Column.getRootAsColumn = function(bb, obj) {
+  return (obj || new NetEncoding.Column).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Column.prototype.uType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Column.prototype.u = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 6);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Column.startColumn = function(builder) {
+  builder.startObject(2);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} uType
+ */
+NetEncoding.Column.addUType = function(builder, uType) {
+  builder.addFieldInt8(0, uType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} uOffset
+ */
+NetEncoding.Column.addU = function(builder, uOffset) {
+  builder.addFieldOffset(1, uOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Column.endColumn = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Matrix = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Matrix}
+ */
+NetEncoding.Matrix.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Matrix=} obj
+ * @returns {NetEncoding.Matrix}
+ */
+NetEncoding.Matrix.getRootAsMatrix = function(bb, obj) {
+  return (obj || new NetEncoding.Matrix).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.nRows = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint32(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.nCols = function() {
+  var offset = this.bb.__offset(this.bb_pos, 6);
+  return offset ? this.bb.readUint32(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @param {number} index
+ * @param {NetEncoding.Column=} obj
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Matrix.prototype.columns = function(index, obj) {
+  var offset = this.bb.__offset(this.bb_pos, 8);
+  return offset ? (obj || new NetEncoding.Column).__init(this.bb.__indirect(this.bb.__vector(this.bb_pos + offset) + index * 4), this.bb) : null;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.columnsLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 8);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Matrix.prototype.colIndexType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 10);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Matrix.prototype.colIndex = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 12);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Matrix.prototype.rowIndexType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 14);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Matrix.prototype.rowIndex = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 16);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Matrix.startMatrix = function(builder) {
+  builder.startObject(7);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} nRows
+ */
+NetEncoding.Matrix.addNRows = function(builder, nRows) {
+  builder.addFieldInt32(0, nRows, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} nCols
+ */
+NetEncoding.Matrix.addNCols = function(builder, nCols) {
+  builder.addFieldInt32(1, nCols, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} columnsOffset
+ */
+NetEncoding.Matrix.addColumns = function(builder, columnsOffset) {
+  builder.addFieldOffset(2, columnsOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<flatbuffers.Offset>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Matrix.createColumnsVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addOffset(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Matrix.startColumnsVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} colIndexType
+ */
+NetEncoding.Matrix.addColIndexType = function(builder, colIndexType) {
+  builder.addFieldInt8(3, colIndexType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} colIndexOffset
+ */
+NetEncoding.Matrix.addColIndex = function(builder, colIndexOffset) {
+  builder.addFieldOffset(4, colIndexOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} rowIndexType
+ */
+NetEncoding.Matrix.addRowIndexType = function(builder, rowIndexType) {
+  builder.addFieldInt8(5, rowIndexType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} rowIndexOffset
+ */
+NetEncoding.Matrix.addRowIndex = function(builder, rowIndexOffset) {
+  builder.addFieldOffset(6, rowIndexOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Matrix.endMatrix = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} offset
+ */
+NetEncoding.Matrix.finishMatrixBuffer = function(builder, offset) {
+  builder.finish(offset);
+};
+
+// Exports for ECMAScript6 Modules
+export {NetEncoding};

client/src/util/stateManager/summarizeAnnotations.js

@@ -1,94 +0,0 @@
-import _ from "lodash";
-
-/*
-Build and return obs/var summary using any annotation in the schema
-
-Summary information for each annotation, keyed by annotation name.
-Value will be an object, containing summary information.
-
-For continuous annotations (int, float, etc):
-  <annotation_name>: {
-    range {
-      min: <number>,
-      max: <number>
-    }
-  }
-
-For categorical annotations (boolean, string, category):
-  <annotatoin_name>: {
-    options: {
-      <option1>: <number>,
-      ...
-    },
-    numOptions: <number>
-  }
-
-Summarize will be returned for BOTH obs and var annotations.
-
-Example:
-  {
-    "Splice_sites_Annotated": {
-      "range": {
-        "min": 26,
-        "max": 1075869
-      }
-    },
-    "Selection": {
-      numOptions, 6,
-      "options": {
-        "Astrocytes(HEPACAM)": 714,
-        "Endothelial(BSC)": 123,
-        "Oligodendrocytes(GC)": 294,
-        "Neurons(Thy1)": 685,
-        "Microglia(CD45)": 1108,
-        "Unpanned": 665
-      }
-    }
-  }
-
-NOTE: will not summarize the required 'name' annotation, as that is
-specified as unique per element.
-*/
-function summarizeDimension(schema, annotations) {
-  return _(schema)
-    .filter(v => v.name !== "name")
-    .keyBy("name")
-    .mapValues(anno => {
-      const { name, type } = anno;
-      const continuous = type === "int32" || type === "float32";
-
-      if (!continuous) {
-        const options = _.countBy(annotations, name);
-        const numOptions = _.size(options);
-        return {
-          numOptions,
-          options
-        };
-      }
-
-      if (continuous) {
-        let min = Number.POSITIVE_INFINITY;
-        let max = Number.NEGATIVE_INFINITY;
-        _.forEach(annotations, obs => {
-          const val = Number(obs[name]);
-          min = val < min ? val : min;
-          max = val > max ? val : max;
-        });
-        return { range: { min, max } };
-      }
-
-      throw new Error("incomprehensible schema");
-    })
-    .value();
-}
-
-export default function summarizeAnnotations(
-  schema,
-  obsAnnotations,
-  varAnnotations
-) {
-  return {
-    obs: summarizeDimension(schema.annotations.obs, obsAnnotations),
-    var: summarizeDimension(schema.annotations.var, varAnnotations)
-  };
-}

client/src/util/stateManager/universe.js

@@ -1,23 +1,16 @@
 // jshint esversion: 6
 
 import _ from "lodash";
-import * as kvCache from "./keyvalcache";
-import summarizeAnnotations from "./summarizeAnnotations";
+
+import decodeMatrixFBS from "./matrix";
+import * as Dataframe from "../dataframe";
 
 /*
 Private helper function - create and return a template Universe
 */
 function templateUniverse() {
   /* default universe template */
-
-  /* varDataCache config - see kvCache for semantics */
-  const VarDataCacheLowWatermark = 32; // cache element count
-  const VarDataCacheTTLMs = 1000; // min cache time in MS
-
   return {
-    api: null,
-    finalized: false, // XXX: may not be needed
-
     nObs: 0,
     nVar: 0,
     schema: {},
@@ -25,21 +18,14 @@ function templateUniverse() {
     /*
     Annotations
     */
-    obsAnnotations: [] /* all obs annotations, by obs index */,
-    varAnnotations: [] /* all var annotations, by var index */,
-    obsNameToIndexMap: {} /* reverse map 'name' to index */,
-    varNameToIndexMap: {} /* reverse map 'name' to index */,
-    summary: null /* derived data summaries XXX: consider exploding in place */,
-
-    obsLayout: { X: [], Y: [] } /* xy layout */,
+    obsAnnotations: Dataframe.Dataframe.empty(),
+    varAnnotations: Dataframe.Dataframe.empty(),
+    obsLayout: Dataframe.Dataframe.empty(),
 
     /*
-    Cache of var data (expression), by var annotation name.   Data can be
-    accesses as a POJO, but if you want caching semantics, use the kvCache
-    API (eg., kvCache.get(), kvCache.set(), ...), which will maintain the
-    LRU semantics.
+    Var data columns - subset of all
     */
-    varDataCache: kvCache.create(VarDataCacheLowWatermark, VarDataCacheTTLMs)
+    varData: Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex())
   };
 }
 
@@ -51,122 +37,63 @@ These functions are used exclusively by the actions and reducers to
 build an internal POJO for use by the rendering components.
 */
 
-/*
-generate any client-side transformations or summarization that
-is independent of REST API response formats.
-*/
-function finalize(universe) {
-  /* A bit of sanity checking! */
-  const { nObs, nVar } = universe;
-  if (
-    nObs !== universe.obsAnnotations.length ||
-    nObs !== universe.obsLayout.X.length ||
-    nObs !== universe.obsLayout.Y.length ||
-    nVar !== universe.varAnnotations.length
-  ) {
-    throw new Error("Universe dimensionality mismatch - failed to load");
-  }
-  // TODO: add more sanity checks, such as:
-  //  - all annotations in the schema
-  //  - layout has supported number of dimensions
-  //  - ...
-
+function AnnotationsFBSToDataframe(arrayBuffer) {
   /*
-  Create all derived (convenience) data structures.
+  Convert a Matrix FBS to a Dataframe.
   */
-  universe.obsNameToIndexMap = _.transform(
-    universe.obsAnnotations,
-    (acc, value, idx) => {
-      acc[value.name] = idx;
-    },
-    {}
+  const fbs = decodeMatrixFBS(arrayBuffer);
+  const df = new Dataframe.Dataframe(
+    [fbs.nRows, fbs.nCols],
+    fbs.columns,
+    null,
+    new Dataframe.KeyIndex(fbs.colIdx)
   );
-  universe.varNameToIndexMap = _.transform(
-    universe.varAnnotations,
-    (acc, value, idx) => {
-      acc[value.name] = idx;
-    },
-    {}
-  );
-  universe.finalized = true;
-  return universe;
+  return df;
 }
 
-function RESTv02AnnotationsResponseToInternal(response) {
-  /*
-  Source per the spec:
-  {
-    names: [
-      'tissue_type', 'sex', 'num_reads', 'clusters'
-    ],
-    data: [
-      [ 0, 'lung', 'F', 39844, 99 ],
-      [ 1, 'heart', 'M', 83, 1 ],
-      [ 49, 'spleen', null, 2, "unknown cluster" ],
-      // [ obsOrVarIndex, value, value, value, value ],
-      // ...
-    ]
-  }
-
-  Internal (target) format:
-  [
-    { __index__: 0, tissue_type: "lung", sex: "F", ... },
-    ...
-  ]
-  */
-  const { names, data } = response;
-  const keys = ["__index__", ...names];
-  return _(data)
-    .map(obs => _.zipObject(keys, obs))
-    .value();
+function LayoutFBSToDataframe(arrayBuffer) {
+  const fbs = decodeMatrixFBS(arrayBuffer, true);
+  const df = new Dataframe.Dataframe(
+    [fbs.nRows, fbs.nCols],
+    fbs.columns,
+    null,
+    new Dataframe.KeyIndex(["X", "Y"])
+  );
+  return df;
 }
 
-function RESTv02LayoutResponseToInternal(response) {
+function reconcileSchemaCategoriesWithSummary(universe) {
   /*
-  Source per the spec:
-  {
-    layout: {
-      ndims: 2,
-      coordinates: [
-        [ 0, 0.284483, 0.983744 ],
-        [ 1, 0.038844, 0.739444 ],
-        // [ obsOrVarIndex, X_coord, Y_coord ],
-        // ...
-      ]
+  where we treat types as (essentially) categorical metadata, update
+  the schema with data-derived categories (in addition to those in
+  the server declared schema).
+
+  For example, boolean defined fields in the schema do not contain
+  explicit declaration of categories (nor do string fields).  In these
+  cases, add a 'categories' field to the schema so it is accessible.
+  */
+
+  _.forEach(universe.schema.annotations.obs, s => {
+    if (
+      s.type === "string" ||
+      s.type === "boolean" ||
+      s.type === "categorical"
+    ) {
+      const categories = _.union(
+        _.get(s, "categories", []),
+        _.get(universe.obsAnnotations.col(s.name).summarize(), "categories", [])
+      );
+      s.categories = categories;
     }
-  }
-
-  Target (internal) format:
-  {
-    X: Float32Array(numObs),
-    Y: Float32Array(numObs)
-  }
-  In the same order as obsAnnotations
-  */
-  const { ndims, coordinates } = response.layout;
-  if (ndims !== 2) {
-    throw new Error("Unsupported layout dimensionality");
-  }
-
-  const layout = {
-    X: new Float32Array(coordinates.length),
-    Y: new Float32Array(coordinates.length)
-  };
-
-  for (let i = 0; i < coordinates.length; i += 1) {
-    const [idx, x, y] = coordinates[i];
-    layout.X[idx] = x;
-    layout.Y[idx] = y;
-  }
-  return layout;
+  });
 }
 
-export function createUniverseFromRestV02Response(
+export function createUniverseFromResponse(
   configResponse,
   schemaResponse,
   annotationsObsResponse,
   annotationsVarResponse,
-  layoutObsResponse
+  layoutFBSResponse
 ) {
   /*
   build & return universe from a REST 0.2 /config, /schema and /annotations/obs response
@@ -174,60 +101,48 @@ export function createUniverseFromRestV02Response(
   const { schema } = schemaResponse;
   const universe = templateUniverse();
 
-  /* constants */
-  universe.api = "0.2";
-
   /* schema related */
   universe.schema = schema;
   universe.nObs = schema.dataframe.nObs;
   universe.nVar = schema.dataframe.nVar;
 
   /* annotations */
-  universe.obsAnnotations = RESTv02AnnotationsResponseToInternal(
-    annotationsObsResponse
-  );
-  universe.varAnnotations = RESTv02AnnotationsResponseToInternal(
-    annotationsVarResponse
-  );
-
+  universe.obsAnnotations = AnnotationsFBSToDataframe(annotationsObsResponse);
+  universe.varAnnotations = AnnotationsFBSToDataframe(annotationsVarResponse);
   /* layout */
-  universe.obsLayout = RESTv02LayoutResponseToInternal(layoutObsResponse);
+  universe.obsLayout = LayoutFBSToDataframe(layoutFBSResponse);
 
-  universe.summary = summarizeAnnotations(
-    universe.schema,
-    universe.obsAnnotations,
-    universe.varAnnotations
-  );
-
-  return finalize(universe);
-}
-
-export function convertExpressionRESTv02ToObject(universe, response) {
-  /*
-  /data/obs response looks like:
-  {
-    var: [ varIndices fetched ],
-    obs: [
-      [ obsIndex, evalue, ... ],
-      ...
-    ]
+  /* sanity check */
+  if (
+    universe.nObs !== universe.obsLayout.length ||
+    universe.nObs !== universe.obsAnnotations.length ||
+    universe.nVar !== universe.varAnnotations.length
+  ) {
+    throw new Error("Universe dimensionality mismatch - failed to load");
   }
 
-  convert expression toa simple Float32Array, and return
-  { geneName: array, geneName: array, ... }
-  NOTE: geneName, not varIndex
+  reconcileSchemaCategoriesWithSummary(universe);
+  return universe;
+}
+
+export function convertDataFBStoObject(universe, arrayBuffer) {
+  /*
+  /data/var returns a flatbuffer (FBS) as described by cellxgene/fbs/matrix.fbs
+
+  This routine converts the binary wire encoding into a JS object:
+
+  {
+    gene: Float32Array,
+    ...
+  }
   */
-  const vars = response.var;
-  const { obs } = response;
+  const fbs = decodeMatrixFBS(arrayBuffer);
+  const { colIdx, columns } = fbs;
   const result = {};
-  // XXX TODO: could this use _.unzip and have less code?
-  for (let varIdx = 0; varIdx < vars.length; varIdx += 1) {
-    const gene = universe.varAnnotations[vars[varIdx]].name;
-    const data = new Float32Array(universe.nObs);
-    for (let obsIdx = 0; obsIdx < obs.length; obsIdx += 1) {
-      data[obsIdx] = obs[obsIdx][varIdx + 1];
-    }
-    result[gene] = data;
+
+  for (let c = 0; c < colIdx.length; c += 1) {
+    const varName = universe.varAnnotations.at(colIdx[c], "name");
+    result[varName] = columns[c];
   }
   return result;
 }

client/src/util/stateManager/world.js

@@ -1,12 +1,12 @@
 // jshint esversion: 6
 
 import _ from "lodash";
-import * as kvCache from "./keyvalcache";
-import summarizeAnnotations from "./summarizeAnnotations";
 import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
-import { sliceByIndex } from "../typedCrossfilter/util";
+import Crossfilter from "../typedCrossfilter";
+import * as Dataframe from "../dataframe";
 
 /*
+
 World is a subset of universe.   Most code should use world, and should
 (generally) not use Universe.   World contains any per-obs or per-var data
 that must be consistent acorss the app when we view/manipulate subsets
@@ -15,120 +15,76 @@ of Universe.
 Private API indicated by leading underscore in key name (eg, _foo).  Anything else
 is public.
 
-World contains several public keys, obsAnnotations, and obsLayout, which are
-arrays contianing information about an OBS in the same order/offset.  In
-other words, world.obsAnnotations[0] and world.obsLayout.X[0] refer to the same
-obs/cell.
+Notable keys in the world object:
+
+* nObs, nVar: dimensions
+
+* schema: data schema from the server
 
 * obsAnnotations:
 
-  obsAnnotations will return an array of objects.  Each object contains all annotation
-  values for a given observation/cell, keyed by annotation name, PLUS a key
-  '__cellId__', containing a REST API ID for this obs/cell (referred to as the
-  obsIndex in the REST 0.2 spec or cellIndex in the 0.1 spec.
+  Dataframe containing obs annotations.  Columns are indexed by annotation
+  name (eg, 'tissue type'), and rows are indexed by the REST API obsIndex
+  (ie, the offset into the underlying server-side dataframe).
 
-  Example:  [ { __cellId__: 99, cluster: 'blue', numReads: 93933 } ]
-
-  NOTE: world.obsAnnotation should be identical to the old state.cells value,
-  EXCEPT that
-    * __cellIndex__ renamed to __index__
-    * __x__ and __y__ are now in world.obsLayout
-    * __color__ and __colorRBG__ should be moved to controls reducer
+  This indexing means that you can access data by _either_ the server's
+  obxIndex, or the offset into the client-side column array .  Be careful
+  to know which you want and are using.
 
 * obsLayout:
 
-  obsLayout will return an object containing two arrays, containing X and Y
-  coordinates respectively.
+  A dataframe containing the X/Y layout for all obs.  Columns are named
+  'X' and 'Y', and rows are indexed in the same way as obsAnnotation.
 
-  Example: { X: [ 0.33, 0.23, ... ], Y: [ 0.8, 0.777, ... ]}
-
-* crossfilter - a crossfilter object across world.obsAnnotations
-
-* dimensionMap - an object mapping annotation names to dimensions on
-  the crossfilter
+* varData: a cache of expression columns, stored in a Dataframe.  Cache
+  managed by controls reducer.
 
 */
 
-/* varDataCache config - see kvCache for semantics */
-const VarDataCacheLowWatermark = 32; // cache element count
-const VarDataCacheTTLMs = 1000; // min cache time in MS
-
 function templateWorld() {
   return {
-    // map from universe obsIndex to world offset.
-    // Undefined / null indicates identity mapping.
-    obsIndex: null,
-    obsBackIndex: null,
-
     /* schema/version related */
-    api: null,
     schema: null,
     nObs: 0,
     nVar: 0,
 
     /* annotations */
-    obsAnnotations: null,
-    varAnnotations: null,
+    obsAnnotations: Dataframe.Dataframe.empty(),
+    varAnnotations: Dataframe.Dataframe.empty(),
 
-    /* layout of graph */
-    obsLayout: null,
+    /* layout of graph. Dataframe. */
+    obsLayout: Dataframe.Dataframe.empty(),
 
-    /* derived data summaries XXX: consider exploding in place */
-    summary: null,
-
-    varDataCache: kvCache.create(
-      VarDataCacheLowWatermark,
-      VarDataCacheTTLMs
-    ) /* cache of var data (expression) */
+    /*
+    Var data columns - subset of all data (may be empty)
+    */
+    varData: Dataframe.Dataframe.empty(null, new Dataframe.KeyIndex())
   };
 }
 
 export function createWorldFromEntireUniverse(universe) {
-  if (!universe.finalized) {
-    throw new Error("World can't be created from an partial Universe");
-  }
-
   const world = templateWorld();
 
-  // map from the universe obsIndex to our world offset.
-  // undefined/null indicates identity map.
-  // In other words   obsBackIndex[universeIdx] -> worldIdx
-  world.obsBackIndex = null;
-  // Map to the universe index for each element in world.
-  // Null indicates identity map (aka world === universe)
-  // In other wrods  obsIndex[worldIdx]  ->  universeIdx
-  world.obsIndex = null;
-
   /*
   public interface follows
   */
 
   /* Schema related */
-  world.api = universe.api;
   world.schema = universe.schema;
   world.nObs = universe.nObs;
   world.nVar = universe.nVar;
 
-  /* annotations */
+  /* annotation dataframes */
   world.obsAnnotations = universe.obsAnnotations;
   world.varAnnotations = universe.varAnnotations;
 
-  /* layout and display characteristics */
+  /* layout and display characteristics dataframe */
   world.obsLayout = universe.obsLayout;
 
-  /* derived data & summaries */
-  world.summary = summarizeAnnotations(
-    world.schema,
-    world.obsAnnotations,
-    world.varAnnotations
-  );
-
-  /* build the varDataCache */
-  world.varDataCache = kvCache.map(
-    universe.varDataCache,
-    val => subsetVarData(world, universe, val),
-    { lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
-  );
+  /*
+  Var data columns - subset of all
+  */
+  world.varData = universe.varData.clone();
 
   return world;
 }
@@ -137,54 +93,24 @@ export function createWorldFromCurrentSelection(universe, world, crossfilter) {
   const newWorld = templateWorld();
 
   /* these don't change as only OBS are selected in our current implementation */
-  newWorld.api = universe.api;
   newWorld.nVar = universe.nVar;
   newWorld.schema = universe.schema;
   newWorld.varAnnotations = universe.varAnnotations;
 
-  /* build index maps and back maps based upon current selection state */
-  const obsBackIndex = new Uint32Array(universe.nObs);
-  obsBackIndex.fill(-1); // default - aka unused
-  const notSelected = obsBackIndex[0];
-  let nObs = 0;
-  for (let i = 0; i < universe.nObs; i += 1) {
-    if (crossfilter.isElementFiltered(i)) {
-      obsBackIndex[i] = nObs;
-      nObs += 1;
-    }
+  /* now subset/cut obs */
+  const mask = crossfilter.allFilteredMask();
+  newWorld.obsAnnotations = world.obsAnnotations.isubsetMask(mask);
+  newWorld.obsLayout = world.obsLayout.isubsetMask(mask);
+  newWorld.nObs = newWorld.obsAnnotations.dims[0];
+
+  /*
+  Var data columns - subset of all
+  */
+  if (world.varData.isEmpty()) {
+    newWorld.varData = world.varData.clone();
+  } else {
+    newWorld.varData = world.varData.isubsetMask(mask);
   }
-  const obsIndex = new Uint32Array(nObs);
-  for (let i = 0; i < universe.nObs; i += 1) {
-    const worldIdx = obsBackIndex[i];
-    if (worldIdx !== notSelected) {
-      obsIndex[worldIdx] = i;
-    }
-  }
-
-  newWorld.nObs = nObs;
-  newWorld.obsIndex = obsIndex;
-  newWorld.obsBackIndex = obsBackIndex;
-
-  /* now slice */
-  newWorld.obsAnnotations = sliceByIndex(universe.obsAnnotations, obsIndex);
-  newWorld.obsLayout = {
-    X: sliceByIndex(universe.obsLayout.X, obsIndex),
-    Y: sliceByIndex(universe.obsLayout.Y, obsIndex)
-  };
-
-  /* derived data & summaries */
-  newWorld.summary = summarizeAnnotations(
-    newWorld.schema,
-    newWorld.obsAnnotations,
-    newWorld.varAnnotations
-  );
-
-  /* build the varDataCache */
-  newWorld.varDataCache = kvCache.map(
-    universe.varDataCache,
-    val => subsetVarData(newWorld, universe, val),
-    { lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
-  );
   return newWorld;
 }
 
@@ -224,14 +150,12 @@ function deduceDimensionType(attributes, fieldName) {
   when it is no longer needed
   (it will not be garbage collected without this call)
 */
-
-export function createVarDimension(
-  world,
-  _worldVarDataCache,
-  crossfilter,
-  geneName
-) {
-  return crossfilter.dimension(_worldVarDataCache[geneName], Float32Array);
+export function createVarDataDimension(world, crossfilter, name) {
+  return crossfilter.dimension(
+    Crossfilter.ScalarDimension,
+    world.varData.col(name).asArray(),
+    Float32Array
+  );
 }
 
 export function createObsDimensionMap(crossfilter, world) {
@@ -239,17 +163,23 @@ export function createObsDimensionMap(crossfilter, world) {
   create and return a crossfilter dimension for every obs annotation
   for which we have a supported type.
   */
-  const { schema, obsLayout } = world;
+  const { schema, obsLayout, obsAnnotations } = world;
 
   // Create a crossfilter dimension for all obs annotations *except* 'name'
   const dimensionMap = _(schema.annotations.obs)
     .filter(anno => anno.name !== "name")
     .transform((result, anno) => {
       const dimType = deduceDimensionType(anno, anno.name);
-      // XXX if dimtype is a scalar, we may be able to do better?
-      if (dimType) {
+      const colData = obsAnnotations.col(anno.name).asArray();
+      if (dimType === "enum") {
         result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
-          r => r[anno.name],
+          Crossfilter.EnumDimension,
+          colData
+        );
+      } else if (dimType) {
+        result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
+          Crossfilter.ScalarDimension,
+          colData,
           dimType
         );
       } // else ignore the annotation
@@ -259,13 +189,10 @@ export function createObsDimensionMap(crossfilter, world) {
   /*
   Add crossfilter dimensions allowing filtering on layout
   */
-  dimensionMap[layoutDimensionName("X")] = crossfilter.dimension(
-    obsLayout.X,
-    Float32Array
-  );
-  dimensionMap[layoutDimensionName("Y")] = crossfilter.dimension(
-    obsLayout.Y,
-    Float32Array
+  dimensionMap[layoutDimensionName("XY")] = crossfilter.dimension(
+    Crossfilter.SpatialDimension,
+    obsLayout.col("X").asArray(),
+    obsLayout.col("Y").asArray()
   );
 
   return dimensionMap;
@@ -275,10 +202,20 @@ export function worldEqUniverse(world, universe) {
   return world.obsAnnotations === universe.obsAnnotations;
 }
 
-export function subsetVarData(world, universe, varData) {
-  // If world === universe, just return the entire varData array
-  if (worldEqUniverse(world, universe)) {
-    return varData;
+export function getSelectedByIndex(crossfilter) {
+  /*
+  return array of obsIndex, containing all selected obs/cells.
+  */
+  const selected = crossfilter.allFilteredMask(); // array of bool-ish
+  const keys = crossfilter.data.rowIndex.keys(); // row keys, aka universe rowIndex
+
+  const set = new Int32Array(selected.length);
+  let numElems = 0;
+  for (let i = 0, l = selected.length; i < l; i += 1) {
+    if (selected[i]) {
+      set[numElems] = keys[i];
+      numElems += 1;
+    }
   }
-  return sliceByIndex(varData, world.obsIndex);
+  return new Int32Array(set.buffer, 0, numElems);
 }

client/src/util/stateManager/worldUtil.js

@@ -0,0 +1,75 @@
+/* eslint-disable import/prefer-default-export */
+import _ from "lodash";
+
+/*
+Various utility functions operating on World/Universe
+*/
+
+/*
+Count unique category values, binning first by dim1 then by dim2
+Return:
+
+Map {
+   dim1_val1: Map {
+      dim2_val1: number,
+      dim2_val2: number,
+      ...
+   },
+   ...
+}
+
+Parameters are:
+- dim1: dimension 1 name/label
+- dim2: dimension 2 name/label
+- df: dataframe containing dim1 and dim2 on the column axis
+
+*/
+function _countCategoryValues2D(dim1, dim2, df) {
+  const dimMap = new Map();
+  const col1 = df.col(dim1) ? df.col(dim1).asArray() : null;
+  const col2 = df.col(dim2) ? df.col(dim2).asArray() : null;
+  if (!col1 || !col2) {
+    return dimMap;
+  }
+
+  for (let r = 0, l = df.length; r < l; r += 1) {
+    const val1 = col1[r];
+    const val2 = col2[r];
+    let d2Map = dimMap.get(val1);
+    if (d2Map === undefined) {
+      d2Map = new Map();
+      dimMap.set(val1, d2Map);
+    }
+    let curCount = d2Map.get(val2);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    d2Map.set(val2, curCount + 1);
+  }
+  return dimMap;
+}
+
+let __worldUtilMemoId__ = 0;
+function _memoizedId(x) {
+  if (!x.__worldUtilMemoId__) {
+    __worldUtilMemoId__ += 1;
+    x.__worldUtilMemoId__ = __worldUtilMemoId__;
+  }
+  return x.__worldUtilMemoId__;
+}
+function _countCategoryValues2DResolver(...args) {
+  const id = args[0] + args[1] + _memoizedId(args[2]);
+  return id;
+}
+
+export const countCategoryValues2D = _.memoize(
+  _countCategoryValues2D,
+  _countCategoryValues2DResolver
+);
+
+/*
+Clear any cached data within WorldUtil caches, eg, memoized functions
+*/
+export function clearCaches() {
+  countCategoryValues2D.cache.clear();
+}

client/src/util/typedCrossfilter/bitArray.js

@@ -40,7 +40,7 @@ class BitArray {
   // Return the number of records that are selected, ie, have a one bit in
   // all allocated dimensions.
   //
-  get selectionCount() {
+  selectionCount() {
     return this.countAllOnes();
   }
 
@@ -48,16 +48,27 @@ class BitArray {
   //
   countAllOnes() {
     let count = 0;
-    const { bitarray, bitmask, length, width } = this;
-    for (let l = 0; l < length; l += 1) {
-      let dimensionsSet = 0;
-      for (let w = 0; w < width; w += 1) {
-        if (bitarray[w * length + l] === bitmask[w]) {
-          dimensionsSet += 1;
+    const { bitarray, length, width } = this;
+    if (width === 1) {
+      // special case, width === 1, for performance
+      const bitmask = this.bitmask[0];
+      for (let l = 0; l < length; l += 1) {
+        if (bitarray[l] === bitmask) {
+          count += 1;
         }
       }
-      if (dimensionsSet === width) {
-        count += 1;
+    } else {
+      const { bitmask } = this;
+      for (let l = 0; l < length; l += 1) {
+        let dimensionsSet = 0;
+        for (let w = 0; w < width; w += 1) {
+          if (bitarray[w * length + l] === bitmask[w]) {
+            dimensionsSet += 1;
+          }
+        }
+        if (dimensionsSet === width) {
+          count += 1;
+        }
       }
     }
     return count;
@@ -233,12 +244,14 @@ class BitArray {
   fillBySelection(result, selectedValue, deselectedValue) {
     // special case (width === 1) for performance
     if (this.width === 1) {
-      const bitmask = this.bitmask[0];
-      for (let i = 0, len = this.length; i < len; i += 1) {
-        result[i] =
-          bitmask && this.bitarray[i] === bitmask
-            ? selectedValue
-            : deselectedValue;
+      const { bitmask, bitarray } = this;
+      const mask = bitmask[0];
+      if (!mask) {
+        result.fill(deselectedValue);
+      } else {
+        for (let i = 0, len = this.length; i < len; i += 1) {
+          result[i] = bitarray[i] === mask ? selectedValue : deselectedValue;
+        }
       }
     } else {
       for (let i = 0, len = this.length; i < len; i += 1) {

client/src/util/typedCrossfilter/index.js

@@ -27,6 +27,8 @@ more complex API.  In a few cases, elements of that API were incorporated.
     https://github.com/square/crossfilter/
 
 */
+// XXX replace
+import { polygonContains } from "d3";
 
 import PositiveIntervals from "./positiveIntervals";
 import BitArray from "./bitArray";
@@ -37,6 +39,14 @@ import {
   upperBoundIndirect
 } from "./util";
 
+function isArrayOrTypedArray(x) {
+  return (
+    Array.isArray(x) ||
+    (ArrayBuffer.isView(x) &&
+      Object.prototype.toString.call(x) !== "[object DataView]")
+  );
+}
+
 class NotImplementedError extends Error {
   constructor(...params) {
     super(...params);
@@ -50,6 +60,11 @@ class NotImplementedError extends Error {
 
 class TypedCrossfilter {
   constructor(data) {
+    /*
+    Typically, data is one of:
+      - Array of objects/records
+      - Dataframe (util/dataframe)
+    */
     this.data = data;
 
     // filters: array of { id, dimension }
@@ -66,14 +81,22 @@ class TypedCrossfilter {
     return this.data;
   }
 
-  dimension(value, valueArrayType) {
+  /*
+  Create a crossfilter dimension, upon which filtering (subselection) can
+  be done.  Each dimension is typed, and has a particular set of filtering
+  semantics.
+    * ScalarDimension - backed by TypedArray values, supporting filtering
+      by value (within a value range, or one or more exact values)
+    * EnumDimension - backed by an enumeration (eg, strings, bools), filtering
+      by one or more enum categories.
+    * SpatialDimension - backed by 2D points, filter by containment within
+      various shapes (currently supports within Rectangle and within Polygon).
+  Call this method to create a dimension, passing arguments appropriate for
+  the dimension constructor.
+  */
+  dimension(DimensionType, ...rest) {
     const id = this.selection.allocDimension();
-    let dim;
-    if (valueArrayType === "enum") {
-      dim = new EnumDimension(value, this, id);
-    } else {
-      dim = new ScalarDimension(value, valueArrayType, this, id);
-    }
+    const dim = new DimensionType(this, id, ...rest);
     this.filters.push({ id, dim });
     dim.filterAll();
     return dim;
@@ -87,18 +110,32 @@ class TypedCrossfilter {
   // return array of all records that are selected/filtered
   // by all dimensions.
   allFiltered() {
-    const { selection } = this;
-    const res = [];
-    for (let i = 0, len = this.data.length; i < len; i += 1) {
-      if (selection.isSelected(i)) {
-        res.push(this.data[i]);
+    const { data, selection } = this;
+    if (Array.isArray(data)) {
+      const res = [];
+      for (let i = 0, len = data.length; i < len; i += 1) {
+        if (selection.isSelected(i)) {
+          res.push(data[i]);
+        }
       }
+      return res;
     }
-    return res;
+    /* else, Dataframe-like */
+    return data.isubsetMask(this.allFilteredMask());
+  }
+
+  // return Uint8array containing selection state (truthy/falsey) for each record.
+  //
+  allFilteredMask() {
+    return this.selection.fillBySelection(
+      new Uint8Array(this.data.length),
+      1,
+      0
+    );
   }
 
   countFiltered() {
-    return this.selection.selectionCount;
+    return this.selection.selectionCount();
   }
 
   isElementFiltered(i) {
@@ -115,13 +152,34 @@ class TypedCrossfilter {
   }
 }
 
-// Base dimension type - value must be a scalar type (eg, int, float),
-// and value array must be a TypedArray.
-//
-class ScalarDimension {
-  constructor(value, ValueArrayType, xfltr, id) {
+// Base dimension type - not exported.
+class _Dimension {
+  constructor(xfltr, id) {
     this.crossfilter = xfltr;
     this._id = id;
+    this.groups = [];
+  }
+
+  dispose() {
+    this.crossfilter._freeDimension(this._id);
+    return this;
+  }
+
+  id() {
+    return this._id;
+  }
+
+  _filterUpdate() {
+    this.crossfilter.updateTime += 1;
+  }
+}
+
+// Scalar dimension type - value must be a scalar type (eg, int, float),
+// and value array must be a TypedArray.
+//
+class ScalarDimension extends _Dimension {
+  constructor(xfltr, id, value, ValueArrayType) {
+    super(xfltr, id);
 
     // current selection filter, expressed as PostiveIntervals.
     this.currentFilter = [];
@@ -130,6 +188,7 @@ class ScalarDimension {
     // or a map function which will create it.
     let array;
     if (value instanceof ValueArrayType) {
+      // user has provided the final typed array - just use it
       if (value.length !== this.crossfilter.data.length) {
         throw new RangeError(
           "ScalarDimension values length must equal crossfilter data record count"
@@ -137,11 +196,18 @@ class ScalarDimension {
       }
       array = value;
     } else if (value instanceof Function) {
-      // Create value array
+      // Create value array from user-provided map function.
       array = this._createValueArray(
         value,
         new ValueArrayType(this.crossfilter.data.length)
       );
+    } else if (isArrayOrTypedArray(value)) {
+      // Create value array from user-provided array.  Typically used
+      // only by enumerated dimensions
+      array = this._createValueArray(
+        i => value[i],
+        new ValueArrayType(this.crossfilter.data.length)
+      );
     } else {
       throw new NotImplementedError(
         "dimension value must be function or value array type"
@@ -151,9 +217,6 @@ class ScalarDimension {
 
     // create sort index
     this.index = makeSortIndex(array);
-
-    // groups, if any
-    this.groups = [];
   }
 
   _createValueArray(value, array) {
@@ -162,20 +225,11 @@ class ScalarDimension {
     const len = data.length;
     const larray = array;
     for (let i = 0; i < len; i += 1) {
-      larray[i] = value(data[i]);
+      larray[i] = value(i, data);
     }
     return larray;
   }
 
-  dispose() {
-    this.crossfilter._freeDimension(this._id);
-    return this;
-  }
-
-  id() {
-    return this._id;
-  }
-
   // Argument is an array of intervals indicating records newly selected/filtered
   //
   _updateFilters(newFilter) {
@@ -209,7 +263,7 @@ class ScalarDimension {
     );
 
     this.currentFilter = cNewFilter;
-    this.crossfilter.updateTime += 1;
+    this._filterUpdate();
   }
 
   // filter by value - exact match
@@ -355,8 +409,8 @@ class ScalarDimension {
 // strings, which can be mapped into an fixed numeric range [0..n).
 //
 class EnumDimension extends ScalarDimension {
-  constructor(value, xfltr, id) {
-    super(value, Uint32Array, xfltr, id);
+  constructor(xfltr, id, value) {
+    super(xfltr, id, value, Uint32Array);
   }
 
   _createValueArray(value, array) {
@@ -368,7 +422,7 @@ class EnumDimension extends ScalarDimension {
     // and the enum.
     const s = new Set();
     for (let i = 0; i < len; i += 1) {
-      s.add(value(data[i]));
+      s.add(value(i, data));
     }
     this.enumIndex = Array.from(s);
     this.enumIndex.sort();
@@ -376,7 +430,7 @@ class EnumDimension extends ScalarDimension {
     // create dimension value array
     const enumLen = this.enumIndex.length;
     for (let i = 0; i < len; i += 1) {
-      const v = value(data[i]);
+      const v = value(i, data);
       const e = lowerBound(this.enumIndex, v, 0, enumLen);
       larray[i] = e;
     }
@@ -408,6 +462,127 @@ class EnumDimension extends ScalarDimension {
   }
 }
 
+/*
+Super simple 2D spatial dimension, supporting basic "filter within"
+operations.
+*/
+class SpatialDimension extends _Dimension {
+  constructor(xfltr, id, X, Y) {
+    super(xfltr, id);
+
+    if (X.length !== Y.length && X.length !== this.crossfilter.data.length) {
+      throw new RangeError(
+        "SpatialDimension values must have same dimensionality as crossfilter"
+      );
+    }
+    this.X = X;
+    this.Y = Y;
+
+    this.Xindex = makeSortIndex(X);
+    this.Yindex = makeSortIndex(Y);
+  }
+
+  filterAll() {
+    this.crossfilter.selection.selectAll(this._id);
+    this._filterUpdate();
+  }
+
+  filterNone() {
+    this.crossfilter.selection.deselectAll(this._id);
+    this._filterUpdate();
+  }
+
+  /*
+  this could be smarter, but we don't currently use it...
+  */
+  filterWithinRect(northwest, southeast) {
+    const [x0, y0] = northwest;
+    const [x1, y1] = southeast;
+    const { X, Y } = this;
+    const seln = this.crossfilter.selection;
+    const { _id } = this;
+    seln.deselectAll(_id);
+    for (let i = 0, l = this.X.length; i < l; i += 1) {
+      const x = X[i];
+      const y = Y[i];
+      if (x0 <= x && x < x1 && y0 <= y && y < y1) {
+        seln.selectOne(_id, i);
+      }
+    }
+    this._filterUpdate();
+  }
+
+  /*
+  Relatively brute force filter by polygon.   Polygon is array of points, where
+  each point is [x,y].   Eg, [[x0,y0], [x1,y1], ...].
+
+  Currently uses d3.polygonContains() to test for polygon inclusion, which itself
+  uses a ray casting (crossing number) algorithm.  There are a series of optimizations
+  to make this faster:
+    * first sliced by X or Y, using an index on the axis
+    * then the polygon bounding box is used for trivial rejection
+    * then the polygon test is applied
+  */
+  filterWithinPolygon(polygon) {
+    /* return bounding box of the polygon */
+    function polygonBoundingBox(pg) {
+      let minX = Number.MAX_VALUE;
+      let minY = Number.MAX_VALUE;
+      let maxX = Number.MIN_VALUE;
+      let maxY = Number.MIN_VALUE;
+      for (let i = 0, l = pg.length; i < l; i += 1) {
+        const p = pg[i];
+        const x = p[0];
+        const y = p[1];
+        if (x < minX) minX = x;
+        if (y < minY) minY = y;
+        if (x > maxX) maxX = x;
+        if (y > maxY) maxY = y;
+      }
+      return [minX, minY, maxX, maxY];
+    }
+
+    const [minX, minY, maxX, maxY] = polygonBoundingBox(polygon);
+    const { X, Y } = this;
+    let slice;
+    let index;
+    if (maxY - minY > maxX - minX) {
+      slice = [
+        lowerBoundIndirect(X, this.Xindex, minX, 0, X.length),
+        upperBoundIndirect(X, this.Xindex, maxX, 0, X.length)
+      ];
+      index = this.Xindex;
+    } else {
+      slice = [
+        lowerBoundIndirect(Y, this.Yindex, minY, 0, Y.length),
+        upperBoundIndirect(Y, this.Yindex, maxY, 0, Y.length)
+      ];
+      index = this.Yindex;
+    }
+
+    const seln = this.crossfilter.selection;
+    const { _id } = this;
+    const testWithin = polygonContains; // d3.polygonContains()
+    seln.deselectAll(_id);
+
+    for (let i = slice[0], e = slice[1]; i < e; i += 1) {
+      const rid = index[i];
+      const x = X[rid];
+      const y = Y[rid];
+      if (
+        minX <= x &&
+        x < maxX &&
+        minY <= y &&
+        y < maxY &&
+        testWithin(polygon, [x, y])
+      ) {
+        seln.selectOne(_id, rid);
+      }
+    }
+    this._filterUpdate();
+  }
+}
+
 // Groups!  Map/reduce
 //
 class ScalarGroup {
@@ -611,5 +786,6 @@ crossfilter.BitArray = BitArray;
 crossfilter.TypedCrossfilter = TypedCrossfilter;
 crossfilter.ScalarDimension = ScalarDimension;
 crossfilter.EnumDimension = EnumDimension;
+crossfilter.SpatialDimension = SpatialDimension;
 
 export default crossfilter;

dev_docs/REST_API.md

@@ -1,5 +1,11 @@
 # cellxgene REST API 0.2 specification
 
+_Note:_ this document is increasingly divergent from the code base and should be read with great suspicion. For example, it lacks any information
+about the binary encoding used by various routes, and has incorrect information about "required" routes and features. We may update it at a
+later date when the protocol stabilizes a bit.
+
+---
+
 Items marked as (_future_) are intended for future implementation, and are included in the design to round out the concept, and highlight what we would do when/if we needed more functionality. The (_future_) items are not currently used by the cellxgene web application, and may be omitted from any backend - see [Current Front-End Dependencies](#current-front-end-dependencies) for more details.
 
 _Caveat emptor, partial spec_: this is a sketch for a spec, not a full spec, and some shortcuts have been taken in the authorship. Best practices for a REST API are assumed but not documented here, such as API versioning, reasonable choices for HTTP response codes, etc. In addition, for clarity the JSON examples will not always have all required quoting (eg, on keys) - the actual implementation should use legal JSON/CSV.
@@ -568,14 +574,14 @@ If differential expression is not supported by the server, must return an HTTP 5
 
 **Response body:**
 
-- For 200 Success, differential expression statistics returned as array of arrays sorted by varindex, where each contains the following values:
+- For 200 Success, differential expression statistics returned as array of arrays, where each contains the following values:
 
   - **varIndex**: variable index for the computed results
   - **logfoldchange**: log fold-change of the average expression between the two groups. Positive values indicate that the gene is more highly expressed in the first group,
   - **pVal**: unadjusted p-value,
   - **pValAdj**: adjusted p-value
 
-  Statistics are encoded as an array of arrays, with fields ordered as:
+  Values ordered as:
 
   _varIndex_, _logfoldchange_, _pVal_, _pValAdj_
 

dev_docs/pull_request_guidelines.md

@@ -0,0 +1,35 @@
+## Creating PR
+1. Name [username]/branchname 
+    1. Branch name should be all lowercase
+    2. Words separated by “-”
+2. Code should address only one issue ideally, make a separate PR for each task
+3. Description 
+    1. Clear explanation of issues solved
+    2. Describe why and how, when appropriate
+    3. Call out specific areas you want extra attention in review (optional)
+    4. If your PR requires more than one reviewer tag those people in the description or comments and let them know that you specifically require them
+4. Ensure that the PR updates tests and documentation and adds tests where appropriate
+5. Use github’s issue keywords when PR is addressing an issue https://help.github.com/articles/closing-issues-using-keywords/
+6. Tags (add at beginning of title)
+    1. [EASY] - small non-controversial change, easy to review
+    2. [DO NOT MERGE] - PR is in progress, do not merge changes
+
+## Review
+1. Assign at least one reviewer to submitted PRs.   Reviewers should be selected based on expertise in areas affected by the PR (eg, web UI: Colin), and should include Comp Bio and PM as needed.
+2. Reviewers should approve or request changes (not just comment) and put general and line level comments where appropriate
+3. As a PR submitter respond to all comments (eg, comment, commit a change, etc)
+4. External PRs
+    1. For external PRs or PRs not from our core team, core team should assign a reviewer and make initial contact within 1 business day
+    2. Build code on local environment and run smoke tests 
+
+## Required to Merge
+1. Travis CI Build passing
+2. At least one reviewer approved
+    1. Exceptions: 
+        1. Release PRs where version is just bumped should not need review
+        2. Complex PRs which touch multiple parts of the codebase should have reviews from all relevant parties
+3. License and Security checks (SNYK) passing. If their server is down and you didn’t add any new external npm or python packages, merge is OK
+
+## Merging
+1. Use "squash and merge" option when merging
+2. If you resolved conflicts, wait until the build passes to merge

dev_docs/release_process.md

@@ -0,0 +1,66 @@
+# cellxgene release process
+
+_This document defines the release process for cellxgene_
+
+## Overview
+
+The goal of the release process is to publish an installable package
+to PyPi, with a matching tagged release on github.
+
+The release process should result in the following side-effects:
+
+- Version number bump, using semantic versioning
+- JS assets built & packaged, committed to the repo
+- Tagged github release
+- Publication to PyPi
+
+## Recipe
+
+Follow these steps to create a release.
+
+1.  Preparation:
+    - python3.6 environment, and a cellxgene clone
+    - Define the release version number, using [semantic versioning](https://semver.org/),
+      and specifying all three digits (eg, 0.3.0)
+    - Write the release title and release notes and add to
+      [release notes document](https://docs.google.com/document/d/1KnHwkYfhyWO5H8BDcMu7y3ogjvq5Yi4OwpmZ8DB6w0Y/edit)
+2.  Create a release branch, eg, `release-version`
+3.  In the release branch:
+    - Run `make release-stage-1 PART=[major | minor | patch]` where you choose major/minor/patch depending on which part of the version
+      is being bumped (eg, 0.2.9->0.3 is minor).
+4.  Commit and push the new branch
+5.  Create a PR for the release.
+    - [optional] As needed, conduct PR review.
+6.  Merge to master
+7.  Create Github release using the version number and release notes
+    ([instructions](https://help.github.com/articles/creating-releases/)).
+    - Draft new release
+    - Type version name matching release version number from (1)
+    - Select `master` as release branch (ensure you merged the release PR)
+    - Type title `Release {version num}`
+    - [optional] Check pre-release if this release is not ready for production
+    - Publish Release
+8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
+and that you have write access to the cellxgene pypi package):
+    - Build the distribution and upload to test pypi `make release-stage-2`
+    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Upload the package to real pypi using `make release-stage-final`
+    - [optional] Test the installation in a fresh virtual environment using
+      `pip install cellxgene`
+    - **Troubleshooting**:
+        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
+        if you accidentally burned a release number you want to use on prod, you have a couple options. 
+            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+             then upload to prod `make release-stage-final`. 
+            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
+            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+            PART=patch until you get to an unburned version.
+        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
+        go to the cellxgene package, click manage, then in the options drop down, click delete and 
+        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
+        and not a major regression, you can just release a patch. 
+    
+
+The optional steps are for testing purposes, and are recommended
+for publishing any major releases, and any releases that significantly
+change the packaging (e.g. new bundled files, new dependencies, etc.)

dev_docs/testing_environment_for_web_hosting.md

@@ -0,0 +1,69 @@
+### How to set up a testing environment for changes related to web hosting.
+
+We often get PRs related to someone using a server to host cellxgene externally or on a local network (ex. https://github.com/chanzuckerberg/cellxgene/pull/568 ). Here is how you can test these changes locally.
+
+We are going to run docker containers for cellxgene and an apache server running a reverse proxy on a local docker network. We run the cellxgene container without exposing any ports so that we cannot access it directly, only through the apache server. We can also update our cellxgene Dockerfile so that we can install a local build instead of having to deploy to pypi.
+
+1 Create Docker network, this allows the containers to communicate with each other.
+
+```
+docker network create cxg
+```
+
+2 Create and run cellxgene container
+
+(optional) To install cellxgene from the local codebase
+
+a Create sdist file
+`make pydist`
+
+b Update Dockerfile to install from dist
+
+```
+FROM ubuntu:bionic
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+COPY [ "dist/",    "/cellxgene/dist/" ]
+
+RUN apt-get update && \
+    apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev && \
+    pip3 install /cellxgene/dist/cellxgene-0.5.1.tar.gz
+
+ENTRYPOINT ["cellxgene"]
+```
+
+(required) Build container
+`docker build . -t cellxgene`
+
+3 Create the proxy container
+
+In a separate directory create these two files
+
+Dockerfile
+
+```
+FROM rgoyard/apache-proxy:latest
+ADD proxy.conf /conf/
+```
+
+proxy.conf
+
+```
+ProxyPass "/data/" http://cellxgene:5005/
+ProxyPassReverse "/data/" http://cellxgene:5005/
+```
+
+Build the container
+`docker build -t proxy .`
+
+4 Run containers and attach to network
+
+```
+docker run -d -p 80:80 --network cxg --name proxy proxy
+docker run -v "$PWD/example-dataset/:/data/" --name cellxgene --network cxg cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad
+```
+
+5 Go to served site
+
+http://localhost/data/

docs/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gem "github-pages", group: :jekyll_plugins

docs/_config.yml

@@ -0,0 +1,9 @@
+theme: jekyll-theme-cayman
+show_downloads: false
+baseurl: /cellxgene
+
+nav:
+  - title: Data
+    url: data.html
+  - title: FAQ
+    url: faq.html

docs/_layouts/default.html

@@ -0,0 +1,49 @@
+<!DOCTYPE html>
+<html lang="{{ site.lang | default: "en-US" }}">
+  <head>
+
+    {% if site.google_analytics %}
+      <script async src="https://www.googletagmanager.com/gtag/js?id={{ site.google_analytics }}"></script>
+      <script>
+        window.dataLayer = window.dataLayer || [];
+        function gtag(){dataLayer.push(arguments);}
+        gtag('js', new Date());
+        gtag('config', '{{ site.google_analytics }}');
+      </script>
+    {% endif %}
+    <meta charset="UTF-8">
+
+{% seo %}
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <meta name="theme-color" content="#157878">
+    <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
+    <link rel="stylesheet" href="{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}">
+  </head>
+  <body>
+
+    <header class="page-header" role="banner">
+      <h1 class="project-name">{{ site.title | default: site.github.repository_name }}</h1>
+      <h2 class="project-tagline">{{ site.description | default: site.github.project_tagline }}</h2>
+      {% if site.nav %}
+        <a href="{{ site.baseurl }}/" class="btn">Home</a>
+        {% for item in site.nav %}
+          <a href="{{ item.url }}" class="btn">{{ item.title }}</a>
+        {% endfor %}
+      {% endif %}
+      {% if site.github.is_project_page %}
+        <a href="{{ site.github.repository_url }}" class="btn" target="_blank">Code</a>
+      {% endif %}
+    </header>
+
+    <main id="content" class="main-content" role="main">
+      {{ content }}
+
+      <footer class="site-footer">
+        {% if site.github.is_project_page %}
+          <span class="site-footer-owner"><a href="{{ site.github.repository_url }}">{{ site.github.repository_name }}</a> is maintained by <a href="{{ site.github.owner_url }}">{{ site.github.owner_name }}</a>.</span>
+        {% endif %}
+        <span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a>.</span>
+      </footer>
+    </main>
+  </body>
+</html>

docs/assets/css/style.scss

@@ -0,0 +1,4 @@
+---
+---
+
+@import "{{ site.theme }}";

docs/data.md

@@ -0,0 +1,29 @@
+---
+layout: default
+title: data
+description: Data
+---
+
+
+# data vignette: how to use cellxgene prepare
+
+#### coming soon!
+
+# example datasets to use with cellxgene
+
+### Examination of single cells from primary human pancreas tissue
+cells: 2,544  
+tissue(s): pancreas  
+data: [Human Cell Atlas Data Portal](https://prod.data.humancellatlas.org/explore/projects?filter=%5B%7B%22facetName%22%3A%22organ%22%2C%22terms%22%3A%5B%22pancreas%22%5D%7D%2C%7B%22facetName%22%3A%22project%22%2C%22terms%22%3A%5B%22Single+cell+transcriptome+analysis+of+human+pancreas%22%5D%7D%5D)  
+paper: [Enge, Martin, et al.](https://www.cell.com/cell/fulltext/S0092-8674(17)31053-X?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS009286741731053X%3Fshowall%3Dtrue)
+
+### Tabula Muris
+cells: 53,800  
+tissue(s): muscle, pancreas, bone, large intestine, heart, brain, fat, mammary gland, tongue , diaphragm, bladder, spleen, thymus, lung , skin, liver, trachea, kidney  
+data: [Tabula Muris Data](https://github.com/czbiohub/tabula-muris-vignettes/tree/master/data)  
+paper: [Tabula Muris Consortium](https://www.nature.com/articles/s41586-018-0590-4)  
+
+### Transcriptional profiling of 1.3 million brain cells
+cells: 1,330,000  
+tissue(s): brain  
+data: [10x Genomics](https://community.10xgenomics.com/t5/10x-Blog/Our-1-3-million-single-cell-dataset-is-ready-to-download/ba-p/276)  

docs/faq.md

@@ -0,0 +1,94 @@
+---
+layout: default
+title: FAQ
+description: Frequently Asked Questions
+---
+
+# Data formatting
+
+#### What file formats can I use with _cellxgene_?
+
+Currently, you can go straight into `cellxgene launch` with your own analyzed data in h5ad format, after you have performed dimenstionality reduction (tsne, umap) and clustering (louvain).
+
+If your data is in a different format, and/or you still need to perform dimensionality reduction and clustering, `cellxgene` can do that for you with the `prepare` command. `cellxgene prepare` runs `scanpy` under the hood and can read in any format that is currently supported by `scanpy` (including mtx, loom, and more listed [here](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)).
+
+The output of `cellxgene prepare` is a h5ad file with your computed clusters and tsne/umap projections that can be used in `cellxgene launch`.
+
+#### I have a directory of 10X-Genomics data with _mtx_ files and I've never used _scanpy_, can I use _cellxgene_?
+
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+```
+
+Depending on the size of the dataset, this may take some time. Once it's done, call
+
+```
+cellxgene launch data-processed.h5ad --layout=umap --open
+```
+
+And your web browser should open with an interactive view of your data.
+
+#### I have extra metadata that I want to add to my dataset
+
+Currently this is not supported directly, but you should be able to do this yourself using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
+
+#### What part of the _anndata_ objects does cellxgene pull in for visualization?
+
+- `.obs` and `.var` annotations are use to extract metadata for filtering
+- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
+- `.obsm` is used for layout
+
+# Algorithms
+
+#### How are you computing and sorting differential expression results?
+
+We use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`.
+
+# Problems, errors, & bugs
+
+#### How do I create a Python 3.6 environment for _cellxgene_?
+
+If you use conda and want to create a [conda environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for _cellxgene_ you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.6
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3.6 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+#### In my _prepare_ command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
+
+Louvain clustering requires additional dependencies, so we don't include them by default. For now, you need to specify that you want these packages by using
+
+```
+pip install cellxgene[louvain]
+```
+
+#### I ran _prepare_ and I'm getting results that look unexpected
+
+You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
+```
+
+It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
+
+#### I tried to `pip install cellxgene` and got a weird error I don't understand
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.
+
+#### I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
+
+This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.

docs/index.md

@@ -0,0 +1,37 @@
+_cellxgene_ is an interactive data explorer for single-cell transcriptomics data. Whether you need to visualize one thousand cells or one million, _cellxgene_ helps you gain insight into your single-cell data.
+
+## features
+
+#### Flexible selections, coloring, and differential expression of your selected sets of cells
+<img src="diffexp.gif" width="600"/>
+
+#### Single-gene analyses (e.g. expression analysis)
+<img src="customGene.gif" width="600" />
+
+## getting started
+
+_cellxgene_ **only** supports Python 3.6. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
+
+Install the package.
+``` bash
+pip install cellxgene
+```
+
+Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
+
+``` bash
+curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
+```
+
+Launch _cellxgene_
+``` bash
+cellxgene launch pbmc3k.h5ad
+```
+
+## getting help
+
+We'd love to hear from you!
+
+For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cziscience-slack.herokuapp.com/) and say "hi!".
+
+For any errors, [report bugs on Github](https://github.com/chanzuckerberg/cellxgene/issues).

docs/release_process.md

@@ -1,48 +0,0 @@
-# cellxgene release process
-
-_This document defines the release process for cellxgene_
-
-## Overview
-
-The goal of the release process is to publish an installable package
-to PyPi, with a matching tagged release on github.
-
-The release process should result in the following side-effects:
-
-- Version number bump, using semantic versioning
-- JS assets built & packaged, committed to the repo
-- Tagged github release
-- Publication to PyPi
-
-## Process
-
-Follow these steps to create a release.
-
-1.  Preparation:
-    - Define the release version number, using [semantic versioning](https://semver.org/)
-    - Write the release title and release notes
-2.  Create a release branch, eg, `release-version`
-3.  In the release branch:
-    - run `bumpversion --config-file .bumpversion.cfg [major | minor | patch]`
-    - build the JS asserts using `bin/build-client`
-4.  Commit and push the new branch
-5.  Create a PR for the release.
-    - [optional] As needed, conduct PR review.
-6.  Create Github release using the version number and release notes ([instructions](https://help.github.com/articles/creating-releases/)).
-7.  Publish to pypi by performing the following steps
-    (assumes you have `setuptools` and `twine` installed and that you have
-    registered for pypi and have write access to the cellxgene pypi package)
-    - build the distribution by calling
-      `python setup.py sdist`
-      inside the top-level directory
-    - [optional] upload the package to test pypi
-      `twine upload --repository-url https://test.pypi.org/legacy/ dist/*`
-    - [optional] test the test installation in a fresh virtual environment using
-      `pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene`
-    - upload the package to real pypi using `twine upload dist/*`
-    - [optional] test the installation in a fresh virtual environment using
-      `pip install cellxgene`
-
-The optional steps are for testing purposes, and are recommended
-for publishing any major releases, and any releases that significantly
-change the packaging (e.g. new bundled files, new dependencies, etc.)

fbs/matrix.fbs

@@ -0,0 +1,73 @@
+
+/*
+
+  Flatbuffers schema for use in cellxgene wire-format.
+
+  Schema defines a general purpose, polymorphic, 2D matrix.  Data is
+  organized in a columnar layout.   Each column is homomorphic, and
+  several column types are supported:
+    - IEEE 32 and 64 bit floats
+    - signed and unsigned 32 bit integers
+    - JSON/UTF8 encoded array (for other types)
+
+  https://github.com/google/flatbuffers
+  http://google.github.io/flatbuffers/
+
+  NOTE: IF YOU MODIFY THIS FILE, YOU MUST RECOMPILE AND COMMIT
+  RESULTING FILES TO THE REPO:
+  * server/app/util/fbs/NetEncoding/*
+  * client/src/util/stateManager/matrix_generated.js
+
+*/
+
+namespace NetEncoding;
+
+table Float32Array {
+  data: [float32];
+}
+
+table Uint32Array {
+  data: [uint32];
+}
+
+table Int32Array {
+  data: [int32];
+}
+
+table Float64Array {
+  data: [float64];
+}
+
+table JSONEncodedArray {
+  // contains a UTF-8/JSON encoded array.  Used to store other
+  // types (or polymorphic arrays)
+  data: [uint8];
+}
+
+union TypedArray {
+  Float32Array,
+  Int32Array,
+  Uint32Array,
+  Float64Array,
+  JSONEncodedArray
+}
+
+// Extra level of indirection required because vector of union not yet supported
+table Column {
+  u: TypedArray;
+}
+
+// 2D matrix stored in columnar layout
+//
+table Matrix {
+  n_rows: uint32;     // all columns have this length
+  n_cols: uint32;     // same as columns.length
+  columns: [Column];  // length n_cols
+
+  // optional row and column index, with same length as corresponding dimension.
+  // If null, defaults to numeric index, ie, [0, n_rows) or [0, n_cols)
+  col_index: TypedArray;
+  row_index: TypedArray;
+}
+
+root_type Matrix;

makefile

@@ -0,0 +1,125 @@
+BUILDDIR := build
+CLIENTBUILD := $(BUILDDIR)/client
+SERVERBUILD := $(BUILDDIR)/server
+CLEANFILES :=  $(BUILDDIR)/ client/build dist cellxgene.egg-info
+
+PART ?= patch
+
+# BUILDING PACKAGE
+
+build : clean build-server
+	@echo "done"
+
+build-server : build-client
+	mkdir -p $(SERVERBUILD)
+	cp -r server/* $(SERVERBUILD)
+	cp -r client/build/  $(CLIENTBUILD)
+	mkdir -p $(SERVERBUILD)/app/web/static/img
+	mkdir -p $(SERVERBUILD)/app/web/templates/
+	cp $(CLIENTBUILD)/index.html $(SERVERBUILD)/app/web/templates/
+	cp -r $(CLIENTBUILD)/static $(SERVERBUILD)/app/web/
+	cp $(CLIENTBUILD)/favicon.png $(SERVERBUILD)/app/web/static/img
+	cp $(CLIENTBUILD)/service-worker.js $(SERVERBUILD)/app/web/static/js/
+	cp MANIFEST.in README.md setup.cfg setup.py $(BUILDDIR)
+
+build-client :
+	npm install --prefix client/ client
+	npm run  --prefix client build
+
+# If you are actively developing in the server folder use this, dirties the source tree
+build-for-server-dev : clean-server build-client
+	mkdir -p server/app/web/static/img
+	mkdir -p server/app/web/static/js
+	mkdir -p server/app/web/templates/
+	cp client/build/index.html server/app/web/templates/
+	cp -r client/build/static server/app/web/
+	cp client/build/favicon.png server/app/web/static/img
+	cp client/build/service-worker.js server/app/web/static/js/
+
+clean : clean-lite clean-server
+	rm -rf client/node_modules
+
+# cleaning node_modules is the longest one, so we avoid that if possible
+clean-lite :
+	rm -rf $(CLEANFILES)
+
+clean-server :
+	rm -f server/app/web/templates/index.html
+	rm -rf server/app/web/static
+
+.PHONY : build build-server build-client build-for-server-dev clean clean-lite clean-server
+
+# CREATING DISTRIBUTION RELEASE
+
+pydist : build
+	cd $(BUILDDIR); python setup.py sdist -d ../dist
+	@echo "done"
+
+.PHONY : pydist
+
+# RELEASE HELPERS
+
+# create new version to commit to master
+release-stage-1 : dev-env bump clean-lite gen-package-lock
+	@echo "Version bumped part:$(PART) and client built. Ready to commit and push"
+
+# build dist and release to dev pypi
+release-stage-2 : dev-env pydist twine
+	@echo "Dist built and uploaded to test.pypi.org"
+	@echo "Test the install `make install-release-test` and then upload to Pypi prod"
+	@echo "`make twine-prod`"
+
+release-stage-final: twine-prod
+	@echo "Release uploaded to pypi.org"
+
+# DANGER: releases directly to prod
+# use this if you accidently burned a test release version number,
+release-burned : dev-env pydist twine-prod
+	@echo "Dist built and uploaded to pypi.org"
+	@echo "Test the install `make install-release`"
+
+dev-env :
+	pip install -r server/requirements-dev.txt
+
+# give PART=[major, minor, part] as param to make bump
+bump :
+	bumpversion --config-file .bumpversion.cfg $(PART)
+
+twine :
+	twine upload --repository-url https://test.pypi.org/legacy/ dist/*
+
+twine-prod :
+	twine upload dist/*
+
+# quicker than re-building client
+gen-package-lock :
+	npm install --prefix client/ client
+
+.PHONY : release-stage-1 release-stage-2 release-stage-final release-burned dev-env bump twine twine-prod gen-package-lock
+
+# INSTALL
+
+# setup.py sucks when you have your library in a separate folder, adding these in to help setup envs
+
+# install from build directory
+install : uninstall
+	cd $(BUILDDIR); pip install -e .
+
+# install from source tree for development
+install-dev : uninstall
+	pip install -e .
+
+# install from test.pypi to test your release
+install-release-test : uninstall
+	pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
+	@echo "Installed cellxgene from test.pypi.org, now run and smoke test"
+
+# install from pypi to test your release
+install-release : uninstall
+	pip install cellxgene
+	@echo "Installed cellxgene from pypi.org"
+
+uninstall :
+	yes | pip uninstall cellxgene || true
+
+.PHONY : install install-dev install-release-test install-release uninstall

server/__main__.py

@@ -2,11 +2,14 @@
 if __package__ is None:
     import sys
     from pathlib import Path
+
     PKG_PATH = Path(__file__).parent
     sys.path.insert(0, str(PKG_PATH.parent))
-    import server
+    import server # noqa F401
+
     __package__ = PKG_PATH.name
 
 # Main thing
-from .cli.cli import cli
+from .cli.cli import cli # noqa F402
+
 cli()

server/app/app.py

@@ -4,7 +4,6 @@ from flask import Flask
 from flask_caching import Cache
 from flask_compress import Compress
 from flask_cors import CORS
-from flask_restful_swagger_2 import get_swagger_blueprint
 
 from .rest_api.rest import get_api_resources
 from .util.utils import Float32JSONEncoder
@@ -14,29 +13,19 @@ REACTIVE_LIMIT = 1_000_000
 
 app = Flask(__name__, static_folder="web/static")
 app.json_encoder = Float32JSONEncoder
-cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860000})
+cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860_000})
 Compress(app)
 CORS(app)
 
 # Config
 SECRET_KEY = os.environ.get("CXG_SECRET_KEY", default="SparkleAndShine")
 
-app.config.update(
-    SECRET_KEY=SECRET_KEY,
-)
+app.config.update(SECRET_KEY=SECRET_KEY)
 
 # Application Data
 data = None
 
-# A list of swagger document objects
-docs = []
 resources = get_api_resources()
-docs.append(resources.get_swagger_doc())
-
 app.register_blueprint(webapp.bp)
 app.register_blueprint(resources.blueprint)
-app.register_blueprint(
-    get_swagger_blueprint(docs, "/api/swagger", produces=["application/json"], title="cellxgene rest api",
-                          description="An API connecting ExpressionMatrix2 clustering algorithm to cellxgene"))
-
 app.add_url_rule("/", endpoint="index")

server/app/driver/driver.py

@@ -11,23 +11,20 @@ Sort order for methods
 
 
 class CXGDriver(metaclass=ABCMeta):
-
     def __init__(self, data, args):
         self.data = self._load_data(data)
         self.layout_method = args["layout"]
         self.diffexp_method = args["diffexp"]
         self.max_category_items = args["max_category_items"]
+        self.diffexp_lfc_cutoff = args["diffexp_lfc_cutoff"]
         self.cluster = None
 
     @property
     def features(self):
         features = {
             "cluster": {"available": False},
-            "layout": {
-                "obs": {"available": False},
-                "var": {"available": False},
-            },
-            "diffexp": {"available": False}
+            "layout": {"obs": {"available": False}, "var": {"available": False}},
+            "diffexp": {"available": False},
         }
         # TODO - Interactive limit should be generated from the actual available methods see GH issue #94
         if self.layout_method:
@@ -45,41 +42,17 @@ class CXGDriver(metaclass=ABCMeta):
         pass
 
     @abstractmethod
-    def filter_dataframe(self, filter):
-        """
-        Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
-        indexing and filtering by annotation value. Filters are combined with the and operator.
-        See REST specs for info on filter format:
-        https://github.com/chanzuckerberg/cellxgene/blob/master/docs/REST_API.md
-
-        :param filter: dictionary with filter params
-        :return: View into scanpy object with cells/genes filtered
-        """
-        pass
-
-    @abstractmethod
-    def annotation(self, filter, axis, fields=None):
+    def annotation_to_fbs_matrix(self, axis, field=None):
         """
         Gets annotation value for each observation
-        :param filter: filter: dictionary with filter params
         :param axis: string obs or var
         :param fields: list of keys for annotation to return, returns all annotation values if not set.
-        :return: dict: names - list of fields in order, data - list of lists or metadata
-        [observation ids, val1, val2...]
+        :return: flatbuffer: in fbs/matrix.fbs encoding
         """
         pass
 
     @abstractmethod
-    def data_frame(self, filter, axis):
-        """
-        Retrieves data for each variable for observations in data frame
-        :param filter: filter: dictionary with filter params
-        :param axis: string obs or var
-        :return: {
-            "var": list of variable ids,
-            "obs": [cellid, var1 expression, var2 expression, ...],
-        }
-        """
+    def data_frame_to_fbs_matrix(self, filter, axis):
         pass
 
     @abstractmethod
@@ -99,11 +72,6 @@ class CXGDriver(metaclass=ABCMeta):
         pass
 
     @abstractmethod
-    def layout(self, filter, interactive_limit=None):
-        """
-        Computes a n-d layout for cells through dimensionality reduction.
-        :param filter: filter: dictionary with filter params
-        :param interactive_limit: -- don't compute if total # genes in dataframes are larger than this
-        :return:  [cellid, x, y, ...]
-        """
+    def layout_to_fbs_matrix(self, filter):
+        """ same as layout, except returns a flatbuffer """
         pass

server/app/rest_api/rest.py

@@ -1,17 +1,21 @@
 from http import HTTPStatus
 import pkg_resources
+import warnings
 
-from flask import (
-    Blueprint, current_app, jsonify, make_response, request
+from flask import Blueprint, current_app, jsonify, make_response, request
+from flask_restful import Api, Resource
+
+from server.app.util.constants import (
+    Axis,
+    DiffExpMode,
+    JSON_NaN_to_num_warning_msg,
+)
+from server.app.util.errors import (
+    FilterError,
+    InteractiveError,
+    JSONEncodingValueError,
+    PrepareError,
 )
-from flask_restful_swagger_2 import Api, swagger, Resource
-from werkzeug.datastructures import ImmutableMultiDict
-
-from server.app.util.constants import Axis, DiffExpMode
-from server.app.util.filter import parse_filter, QueryStringError
-from server.app.util.models import FilterModel
-from server.app.util.utils import get_mime_type
-from server.app.util.errors import MimeTypeError, FilterError, InteractiveError, PrepareError
 
 """
 Sort order for routes
@@ -22,558 +26,112 @@ Sort order for routes
 
 
 class SchemaAPI(Resource):
-    @swagger.doc({
-        "summary": "get schema for dataframe and annotations",
-        "tags": ["initialize"],
-        "parameters": [],
-        "responses": {
-            "200": {
-                "description": "schema",
-                "examples": {
-                    "application/json": {
-                        "schema": {
-                            "dataframe": {
-                                "nObs": 383,
-                                "nVar": 19944,
-                                "type": "float32"
-                            },
-                            "annotations": {
-                                "obs": [
-                                    {"name": "name", "type": "string"},
-                                    {"name": "tissue_type", "type": "string"},
-                                    {"name": "num_reads", "type": "int32"},
-                                    {"name": "sample_name", "type": "string"},
-                                    {
-                                        "name": "clusters",
-                                        "type": "categorical",
-                                        "categories": [99, 1, "unknown cluster"]
-                                    },
-                                    {"name": "QScore", "type": "float32"}
-                                ],
-                                "var": [
-                                    {"name": "name", "type": "string"},
-                                    {"name": "gene", "type": "string"}
-                                ]
-                            }
-                        }
-                    }
-                }
-            }
-        }
-
-    })
     def get(self):
-        return make_response(jsonify({"schema": current_app.data.schema}), HTTPStatus.OK)
+        return make_response(
+            jsonify({"schema": current_app.data.schema}), HTTPStatus.OK
+        )
 
 
 class ConfigAPI(Resource):
-    @swagger.doc({
-        "summary": "Configuration information to assist in front-end adaptation"
-                   " to underlying engine, available functionality, interactive time limits, etc",
-        "tags": ["initialize"],
-        "parameters": [],
-        "responses": {
-            "200": {
-                "description": "schema",
-                "examples": {
-                    "application/json": {
-                        "config": {
-                            "features": [
-                                {"method": "POST", "path": "/cluster/", "available": False},
-                                {
-                                    "method": "POST",
-                                    "path": "/layout/obs",
-                                    "available": True,
-                                    "interactiveLimit": 10000
-                                },
-                                {"method": "POST", "path": "/layout/var", "available": False}
-
-                            ],
-                            "displayNames": {
-                                "engine": "ScanPy version 1.33",
-                                "dataset": "/home/joe/mouse/blorth.csv"
-                            },
-                        }
-                    }
-                }
-            }
-        }
-    })
     def get(self):
         config = {
             "config": {
                 "features": [
-                    {"method": "POST", "path": "/cluster/", **current_app.data.features["cluster"]},
-                    {"method": "POST", "path": "/layout/obs", **current_app.data.features["layout"]["obs"]},
-                    {"method": "POST", "path": "/layout/var", **current_app.data.features["layout"]["var"]},
-                    {"method": "POST", "path": "/diffexp/", **current_app.data.features["diffexp"]},
+                    {
+                        "method": "POST",
+                        "path": "/cluster/",
+                        **current_app.data.features["cluster"],
+                    },
+                    {
+                        "method": "POST",
+                        "path": "/layout/obs",
+                        **current_app.data.features["layout"]["obs"],
+                    },
+                    {
+                        "method": "POST",
+                        "path": "/layout/var",
+                        **current_app.data.features["layout"]["var"],
+                    },
+                    {
+                        "method": "POST",
+                        "path": "/diffexp/",
+                        **current_app.data.features["diffexp"],
+                    },
                 ],
                 "displayNames": {
                     "engine": f"cellxgene Scanpy engine version {pkg_resources.get_distribution('cellxgene').version}",
-                    "dataset": current_app.config["DATASET_TITLE"]
+                    "dataset": current_app.config["DATASET_TITLE"],
                 },
                 "parameters": {
                     "max_category_items": current_app.data.max_category_items
-                }
+                },
             }
         }
         return make_response(jsonify(config), HTTPStatus.OK)
 
 
 class AnnotationsObsAPI(Resource):
-    @swagger.doc({
-        "summary": "Fetch annotations (metadata) for all observations.",
-        "tags": ["annotations"],
-        "parameters": [{
-            "in": "query",
-            "name": "annotation-name",
-            "type": "string",
-            "description": "list of 1 or more annotation names"
-        }],
-        "responses": {
-            "200": {
-                "description": "annotations",
-                "examples": {
-                    "application/json": {
-                        "names": [
-                            "tissue_type", "sex", "num_reads", "clusters"
-                        ],
-                        "data": [
-                            [0, "lung", "F", 39844, 99],
-                            [1, "heart", "M", 83, 1],
-                            [49, "spleen", None, 2, "unknown cluster"],
-
-                        ]
-                    }
-
-                }
-            },
-            "400": {
-                "description": "one or more of the annotation-name identifiers were not associated with an "
-                               "annotation name"
-            }
-        }
-    })
     def get(self):
         fields = request.args.getlist("annotation-name", None)
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/octet-stream"]
+        )
         try:
-            annotation_response = current_app.data.annotation({}, "obs", fields)
+            if preferred_mimetype == "application/octet-stream":
+                return make_response(current_app.data.annotation_to_fbs_matrix("obs", fields),
+                                     HTTPStatus.OK,
+                                     {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
-
-    @swagger.doc({
-        "summary": "Fetch annotations (metadata) for filtered subset of observations.",
-        "tags": ["annotations"],
-        "parameters": [
-            {
-                "in": "query",
-                "name": "annotation-name",
-                "type": "string",
-                "description": "list of 1 or more annotation names"
-            },
-            {
-                "name": "filter",
-                "description": "Complex Filter",
-                "in": "body",
-                "schema": FilterModel
-            }
-        ],
-        "responses": {
-            "200": {
-                "description": "annotations",
-                "examples": {
-                    "application/json": {
-                        "names": [
-                            "tissue_type", "sex", "num_reads", "clusters"
-                        ],
-                        "data": [
-                            [0, "lung", "F", 39844, 99],
-                            [1, "heart", "M", 83, 1],
-                            [49, "spleen", None, 2, "unknown cluster"],
-
-                        ]
-                    }
-
-                }
-            },
-            "400": {
-                "description": "malformed filter or one or more of the annotation-name identifiers were"
-                               "not associated with an annotation name"
-            }
-        }
-    })
-    def put(self):
-        fields = request.args.getlist("annotation-name", None)
-        try:
-            annotation_response = current_app.data.annotation(request.get_json()["filter"], "obs", fields)
-        except KeyError:
-            return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        except FilterError as e:
-            return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class AnnotationsVarAPI(Resource):
-    @swagger.doc({
-        "summary": "Fetch annotations (metadata) for all variables.",
-        "tags": ["annotations"],
-        "parameters": [{
-            "in": "query",
-            "name": "annotation-name",
-            "type": "string",
-            "description": "list of 1 or more annotation names"
-        }],
-        "responses": {
-            "200": {
-                "description": "annotations",
-                "examples": {
-                    "application/json": {
-                        "names": [
-                            "name", "category"
-                        ],
-                        "data": [
-                            [0, "ATAD3C", 1],
-                            [1, "RER1", None],
-                            [49, "S100B", 6]
-                        ]
-                    }
-
-                }
-            },
-            "400": {
-                "description": "one or more of the annotation-name identifiers were not associated with an"
-                               " annotation name"
-            }
-        }
-    })
     def get(self):
         fields = request.args.getlist("annotation-name", None)
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/octet-stream"]
+        )
         try:
-            annotation_response = current_app.data.annotation({}, "var", fields)
+            if preferred_mimetype == "application/octet-stream":
+                return make_response(current_app.data.annotation_to_fbs_matrix("var", fields),
+                                     HTTPStatus.OK,
+                                     {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
-
-    @swagger.doc({
-        "summary": "Fetch annotations (metadata) for filtered subset of variables.",
-        "tags": ["annotations"],
-        "parameters": [
-            {
-                "in": "query",
-                "name": "annotation-name",
-                "type": "string",
-                "description": "list of 1 or more annotation names"
-            },
-            {
-                "name": "filter",
-                "description": "Complex Filter",
-                "in": "body",
-                "schema": FilterModel
-            }
-        ],
-        "responses": {
-            "200": {
-                "description": "annotations",
-                "examples": {
-                    "application/json": {
-                        "names": [
-                            "name", "category"
-                        ],
-                        "data": [
-                            [0, "ATAD3C", 1],
-                            [1, "RER1", None],
-                            [49, "S100B", 6]
-                        ]
-                    }
-                }
-            },
-            "400": {
-                "description": "malformed filter or one or more of the annotation-name identifiers were"
-                               "not associated with an annotation name"
-            }
-        }
-    })
-    def put(self):
-        fields = request.args.getlist("annotation-name", None)
-        try:
-            annotation_response = current_app.data.annotation(request.get_json()["filter"], "var", fields)
-        except KeyError:
-            return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        except FilterError:
-            return make_response("Malformed filter", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
-
-
-class DataObsAPI(Resource):
-    @swagger.doc({
-        "summary": "Get data (expression values) from the dataframe.",
-        "tags": ["data"],
-        "parameters": [
-            {
-                "in": "query",
-                "name": "filter",
-                "type": "string",
-                "description": "axis:key:value"
-            },
-            {
-                "in": "query",
-                "name": "accept-type",
-                "type": "string",
-                "description": "MIME type"
-            },
-        ],
-        "responses": {
-            "200": {
-                "description": "expression",
-                "examples": {
-                    "application/json": {
-                        "var": [0, 20000],
-                        "obs": [
-                            [1, 39483, 3902, 203, 0, 0, 28]
-                        ]
-                    }
-                }
-            },
-            "400": {
-                "description": "Malformed filter"
-            },
-            "406": {
-                "description": "Unacceptable MIME type"
-            },
-        }
-    })
-    def get(self):
-        accept_type = request.args.get("accept-type", None)
-        # request.args is immutable
-        args = dict(request.args)
-        args.pop("accept-type", None)
-        try:
-            filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema['annotations'])
-        except QueryStringError as e:
-            return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        # TODO support CSV
-        try:
-            # TODO store mime_type when more than one is supported
-            get_mime_type(acceptable_types=["application/json"], query_param=accept_type,
-                          header=request.accept_mimetypes)
-        except MimeTypeError as e:
-            return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
-        try:
-            return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.OBS))), HTTPStatus.OK)
-        except FilterError as e:
-            return make_response(e.message, HTTPStatus.BAD_REQUEST)
-
-    @swagger.doc({
-        "summary": "Get data (expression values) from the dataframe.",
-        "tags": ["data"],
-        "parameters": [
-            {
-                'name': 'filter',
-                'description': 'Complex Filter',
-                'in': 'body',
-                'schema': FilterModel
-            }
-        ],
-        "responses": {
-            "200": {
-                "description": "expression",
-                "examples": {
-                    "application/json": {
-                        "var": [0, 20000],
-                        "obs": [
-                            [1, 39483, 3902, 203, 0, 0, 28]
-                        ]
-                    }
-                }
-            },
-            "400": {
-                "description": "Malformed filter"
-            },
-            "406": {
-                "description": "Unacceptable MIME type"
-            },
-        }
-    })
-    def put(self):
-        if not request.accept_mimetypes.best_match(["application/json", "text/csv"]):
-            return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
-        try:
-            get_mime_type(acceptable_types=["application/json"], header=request.accept_mimetypes)
-        except MimeTypeError as e:
-            return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
-        try:
-            return make_response((jsonify(current_app.data.data_frame(request.get_json()["filter"], axis=Axis.OBS))),
-                                 HTTPStatus.OK)
-        except FilterError as e:
-            return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DataVarAPI(Resource):
-    @swagger.doc({
-        "summary": "Get data (expression values) from the dataframe.",
-        "tags": ["data"],
-        "parameters": [
-            {
-                "in": "query",
-                "name": "filter",
-                "type": "string",
-                "description": "axis:key:value"
-            },
-            {
-                "in": "query",
-                "name": "accept-type",
-                "type": "string",
-                "description": "MIME type"
-            },
-        ],
-        "responses": {
-            "200": {
-                "description": "expression",
-                "examples": {
-                    "application/json": {
-                        "obs": [0, 20000],
-                        "var": [
-                            [1, 39483, 3902, 203, 0, 0, 28]
-                        ]
-                    }
-                }
-            },
-            "400": {
-                "description": "Malformed filter"
-            },
-            "406": {
-                "description": "Unacceptable MIME type"
-            },
-        }
-    })
-    def get(self):
-        accept_type = request.args.get("accept-type", None)
-        # request.args is immutable
-        args = dict(request.args)
-        args.pop("accept-type", None)
-        try:
-            filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema['annotations'])
-        except QueryStringError as e:
-            return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        try:
-            get_mime_type(acceptable_types=["application/json"], query_param=accept_type,
-                          header=request.accept_mimetypes)
-        except MimeTypeError as e:
-            return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
-        try:
-            return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.VAR))), HTTPStatus.OK)
-        except FilterError as e:
-            return make_response(e.message, HTTPStatus.BAD_REQUEST)
-
-    @swagger.doc({
-        "summary": "Get data (expression values) from the dataframe.",
-        "tags": ["data"],
-        "parameters": [
-            {
-                'name': 'filter',
-                'description': 'Complex Filter',
-                'in': 'body',
-                'schema': FilterModel
-            }
-        ],
-        "responses": {
-            "200": {
-                "description": "expression",
-                "examples": {
-                    "application/json": {
-                        "obs": [0, 20000],
-                        "var": [
-                            [1, 39483, 3902, 203, 0, 0, 28]
-                        ]
-                    }
-                }
-            },
-            "400": {
-                "description": "Malformed filter"
-            },
-            "406": {
-                "description": "Unacceptable MIME type"
-            },
-        }
-    })
     def put(self):
-        if not request.accept_mimetypes.best_match(["application/json", "text/csv"]):
-            return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
-        # TODO support CSV
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/octet-stream"]
+        )
         try:
-            get_mime_type(acceptable_types=["application/json"], header=request.accept_mimetypes)
-        except MimeTypeError as e:
-            return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
-        try:
-            return make_response((jsonify(current_app.data.data_frame(request.get_json()["filter"], axis=Axis.VAR))),
-                                 HTTPStatus.OK)
+            if preferred_mimetype == "application/octet-stream":
+                filter_json = request.get_json()
+                filter = filter_json["filter"] if filter_json else None
+                return make_response(
+                    current_app.data.data_frame_to_fbs_matrix(
+                        filter, axis=Axis.VAR
+                    ),
+                    HTTPStatus.OK,
+                    {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DiffExpObsAPI(Resource):
-    @swagger.doc({
-        "summary": "Generate differential expression (DE) statistics for two specified subsets of data, "
-                   "as indicated by the two provided observation complex filters",
-        "tags": ["diffexp"],
-        # TODO sort out params
-        # "parameters": [
-        #     # {
-        #     #     "in": "body",
-        #     #     "name": "mode",
-        #     #     "type": "string",
-        #     #     "required": True,
-        #     #     "description": "topN or varFilter"
-        #     # },
-        #     {
-        #         "in": "query",
-        #         "name": "count",
-        #         "type": "int32",
-        #         "description": "TopN mode: how many vars to return"
-        #     },
-        #     {
-        #         "in": "body",
-        #         "name": "varFilter",
-        #         "schema": FilterModel,
-        #         "description": "varFilter: Complex filter, only var for which vars to return"
-        #     },
-        #     {
-        #         "in": "body",
-        #         "name": "set1",
-        #         "schema": FilterModel,
-        #         "required": True,
-        #         "description": "Complex filter, only obs - observations in set1"
-        #     },
-        #     {
-        #         "in": "body",
-        #         "name": "set2",
-        #         "schema": FilterModel,
-        #         "description": "Complex filter, only obs - observations in set2. If not included, inverse of set1."
-        #     },
-        # ],
-        "responses": {
-            "200": {
-                "description": "Statistics are encoded as an array of arrays, with fields ordered as: "
-                               "varIndex, logfoldchange,  pVal, pValAdj",
-                "examples": {
-                    "application/json": [
-                        [328, -2.569489, 2.655706e-63, 3.642036e-57],
-                        [1250, -2.569489, 2.655706e-63, 3.642036e-57],
-                    ]
-                }
-            },
-            "400": {
-                "description": "malformed filter"
-            },
-            "403": {
-                "description": "non-interactive request"
-            },
-            "501": {
-                "description": "diffexp is not implemented"
-            }
-        }
-    })
     def post(self):
         args = request.get_json()
         # confirm mode is present and legal
@@ -582,23 +140,35 @@ class DiffExpObsAPI(Resource):
         except KeyError:
             return make_response("Error: mode is required", HTTPStatus.BAD_REQUEST)
         except ValueError:
-            return make_response(f"Error: invalid mode option {args['mode']}", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                f"Error: invalid mode option {args['mode']}", HTTPStatus.BAD_REQUEST
+            )
         # Validate filters
         if mode == DiffExpMode.VAR_FILTER or "varFilter" in args:
             # not NOT_IMPLEMENTED
-            return make_response("mode=varfilter not implemented", HTTPStatus.NOT_IMPLEMENTED)
+            return make_response(
+                "mode=varfilter not implemented", HTTPStatus.NOT_IMPLEMENTED
+            )
         if mode == DiffExpMode.TOP_N and "count" not in args:
-            return make_response("mode=topN requires a count parameter", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                "mode=topN requires a count parameter", HTTPStatus.BAD_REQUEST
+            )
 
         if "set1" not in args:
             return make_response("set1 is required.", HTTPStatus.BAD_REQUEST)
         if Axis.VAR in args["set1"]["filter"]:
-            return make_response("Var filter not allowed for set1", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                "Var filter not allowed for set1", HTTPStatus.BAD_REQUEST
+            )
         # set2
         if "set2" not in args:
-            return make_response("Set2 as inverse of set1 is not implemented", HTTPStatus.NOT_IMPLEMENTED)
+            return make_response(
+                "Set2 as inverse of set1 is not implemented", HTTPStatus.NOT_IMPLEMENTED
+            )
         if Axis.VAR in args["set2"]["filter"]:
-            return make_response("Var filter not allowed for set2", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                "Var filter not allowed for set2", HTTPStatus.BAD_REQUEST
+            )
 
         set1_filter = args["set1"]["filter"]
         set2_filter = args.get("set2", {"filter": {}})["filter"]
@@ -608,103 +178,54 @@ class DiffExpObsAPI(Resource):
         # mode=topN
         count = args.get("count", None)
         try:
-            diffexp = current_app.data.diffexp_topN(set1_filter, set2_filter, count,
-                                                    current_app.data.features["diffexp"]["interactiveLimit"])
+            diffexp = current_app.data.diffexp_topN(
+                set1_filter,
+                set2_filter,
+                count,
+                current_app.data.features["diffexp"]["interactiveLimit"],
+            )
+            return make_response(
+                diffexp, HTTPStatus.OK, {"Content-Type": "application/json"}
+            )
         except (ValueError, FilterError) as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
         except InteractiveError:
             return make_response("Non-interactive request", HTTPStatus.FORBIDDEN)
-        return make_response(jsonify(diffexp), HTTPStatus.OK)
+        except JSONEncodingValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class LayoutObsAPI(Resource):
-    @swagger.doc({
-        "summary": "Get the default layout for all observations.",
-        "tags": ["layout"],
-        "parameters": [],
-        "responses": {
-            "200": {
-                "description": "layout",
-                "examples": {
-                    "application/json": {
-                        "layout": {
-                            "ndims": 2,
-                            "coordinates": [
-                                [0, 0.284483, 0.983744],
-                                [1, 0.038844, 0.739444]
-                            ]
-                        }
-                    }
-                }
-            },
-            "400": {
-                "description": "Data preparation error"
-            }
-        }
-    })
     def get(self):
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/octet-stream"]
+        )
         try:
-            layout = current_app.data.layout({})
+            if preferred_mimetype == "application/octet-stream":
+                return make_response(current_app.data.layout_to_fbs_matrix(),
+                                     HTTPStatus.OK,
+                                     {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except PrepareError as e:
             return make_response(e.message, HTTPStatus.INTERNAL_SERVER_ERROR)
-        return make_response((jsonify({"layout": layout})), HTTPStatus.OK)
-
-    # @swagger.doc({
-    #     "summary": "Observation layout for filtered subset.",
-    #     "tags": ["layout"],
-    #     "parameters": [
-    #         {
-    #             "name": "filter",
-    #             "description": "Complex Filter",
-    #             "in": "body",
-    #             "schema": FilterModel
-    #         }
-    #     ],
-    #     "responses": {
-    #         "200": {
-    #             "description": "layout",
-    #             "examples": {
-    #                 "application/json": {
-    #                     "layout": {
-    #                         "ndims": 2,
-    #                         "coordinates": [
-    #                             [0, 0.284483, 0.983744],
-    #                             [1, 0.038844, 0.739444]
-    #                         ]
-    #                     }
-    #                 }
-    #             }
-    #         },
-    #         "400": {
-    #             "description": "Malformed filter"
-    #         },
-    #         "403": {
-    #             "description": "Non-interactive request"
-    #         },
-    #     }
-    # })
-    # def put(self):
-    #     try:
-    #         filter = request.get_json()["filter"]
-    #         interactive_limit = current_app.data.features["layout"]["obs"]["interactiveLimit"]
-    #         layout = current_app.data.layout(filter, interactive_limit=interactive_limit)
-    #         return make_response(jsonify({"layout": layout}), HTTPStatus.OK)
-    #     except FilterError as e:
-    #         return make_response(e.message, HTTPStatus.BAD_REQUEST)
-    #     except InteractiveError:
-    #         return make_response("Non-interactive request", HTTPStatus.FORBIDDEN)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 def get_api_resources():
     bp = Blueprint("api", __name__, url_prefix="/api/v0.2")
-    api = Api(bp, add_api_spec_resource=False)
+    api = Api(bp)
     # Initialization routes
     api.add_resource(SchemaAPI, "/schema")
     api.add_resource(ConfigAPI, "/config")
     # Data routes
     api.add_resource(AnnotationsObsAPI, "/annotations/obs")
     api.add_resource(AnnotationsVarAPI, "/annotations/var")
-    api.add_resource(DataObsAPI, "/data/obs")
     api.add_resource(DataVarAPI, "/data/var")
     # Computation routes
     api.add_resource(DiffExpObsAPI, "/diffexp/obs")

server/app/scanpy_engine/diffexp.py

@@ -1,4 +1,3 @@
-
 import numpy as np
 from scipy import sparse, stats
 
@@ -10,74 +9,114 @@ def _mean_var_n(X):
     than naive methods (and same method used by numpy.var())
     https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass
     """
-    n = X.shape[0]
-    if sparse.issparse(X):
-        mean = X.mean(axis=0).A1
-        dfm = X - mean
-        sumsq = np.sum(np.multiply(dfm, dfm), axis=0).A1
-        v = sumsq / (n - 1)
-    else:
-        mean = X.mean(axis=0)
-        dfm = X - mean
-        sumsq = np.sum(np.multiply(dfm, dfm), axis=0)
-        v = sumsq / (n - 1)
+    # fp_err_occurred is a flag indicating that a floating point error
+    # occured somewhere in our compute.  Used to trigger non-finite
+    # number handling.
+    fp_err_occurred = False
 
+    def fp_err_set(err, flag):
+        nonlocal fp_err_occurred
+        fp_err_occurred = True
+
+    with np.errstate(divide="call", invalid="call", call=fp_err_set):
+        n = X.shape[0]
+        if sparse.issparse(X):
+            mean = X.mean(axis=0).A1
+            dfm = X - mean
+            sumsq = np.sum(np.multiply(dfm, dfm), axis=0).A1
+            v = sumsq / (n - 1)
+        else:
+            mean = X.mean(axis=0)
+            dfm = X - mean
+            sumsq = np.sum(np.multiply(dfm, dfm), axis=0)
+            v = sumsq / (n - 1)
+
+    if fp_err_occurred:
+        mean[np.isfinite(mean) == False] = 0    # noqa: E712
+        v[np.isfinite(v) == False] = 0          # noqa: E712
     return mean, v, n
 
 
-def diffexp_ttest(adata, maskA, maskB, top_n=8):
+def diffexp_ttest(adata, maskA, maskB, top_n=8, diffexp_lfc_cutoff=0.01):
     """
-    Return differential expression statistics for top N variables, sorted by
-    t statistic.   Implemented as a unequal variance t-test.
+    Return differential expression statistics for top N variables.
+
+    Algorithm:
+    - compute log fold change (log2(meanA/meanB))
+    - compute Welch's t-test statistic and pvalue (w/ Bonferroni correction)
+    - return top N abs(logfoldchange) where lfc > diffexp_lfc_cutoff
+
+    If there are not N which meet criteria, augment by removing the logfoldchange
+    threshold requirement.
+
+    Notes on alogrithm:
+    - Welch's ttest provides basic statistics test.
+      https://en.wikipedia.org/wiki/Welch%27s_t-test
+    - p-values adjusted with Bonferroni correction.
+      https://en.wikipedia.org/wiki/Bonferroni_correction
 
     :param adata: anndata dataframe
     :param maskA: observation selection mask for set 1
     :param maskB: observation selection mask for set 2
     :param top_n: number of variables to return stats for
+    :param diffexp_lfc_cutoff: minimum
     :return:  for top N genes, [ varindex, logfoldchange, pval, pval_adj ]
     """
 
-    # mean, variance, N
+    if top_n > adata.n_obs:
+        top_n = adata.n_obs
+
+    # mean, variance, N - calculate for both selections
     meanA, vA, nA = _mean_var_n(adata._X[maskA])
     meanB, vB, nB = _mean_var_n(adata._X[maskB])
 
     # variance / N
-    vnA = vA / nA
-    vnB = vB / nB
+    vnA = vA / min(nA, nB)  # overestimate variance, would normally be nA
+    vnB = vB / min(nA, nB)  # overestimate variance, would normally be nB
     sum_vn = vnA + vnB
 
     # degrees of freedom for Welch's t-test
-    with np.errstate(divide='ignore', invalid='ignore'):
-        dof = sum_vn**2 / (vnA**2 / (nA - 1) + vnB**2 / (nB - 1))
+    with np.errstate(divide="ignore", invalid="ignore"):
+        dof = sum_vn ** 2 / (vnA ** 2 / (nA - 1) + vnB ** 2 / (nB - 1))
     dof[np.isnan(dof)] = 1
 
     # Welch's t-test score calculation
-    with np.errstate(divide='ignore', invalid='ignore'):
+    with np.errstate(divide="ignore", invalid="ignore"):
         tscores = (meanA - meanB) / np.sqrt(sum_vn)
     tscores[np.isnan(tscores)] = 0
 
     # p-value
     pvals = stats.t.sf(np.abs(tscores), dof) * 2
     pvals_adj = pvals * adata._X.shape[1]
+    pvals_adj[pvals_adj > 1] = 1  # cap adjusted p-value at 1
 
     # logfoldchanges: log2(meanA / meanB)
     logfoldchanges = np.log2(np.abs((meanA + 1e-9) / (meanB + 1e-9)))
 
-    # top n sort
+    # find all with lfc > cutoff
+    lfc_above_cutoff_idx = np.nonzero(np.abs(logfoldchanges) > diffexp_lfc_cutoff)[0]
     stats_to_sort = np.abs(tscores)
-    partition = np.argpartition(stats_to_sort, -top_n)[-top_n:]
-    rel_sort_order = np.argsort(stats_to_sort[partition])[::-1]
-    vars_indices = np.arange(adata.n_vars, dtype=int)
-    sort_order = vars_indices[partition][rel_sort_order]
 
-    # top n slice
+    # derive sort order
+    if lfc_above_cutoff_idx.shape[0] > top_n:
+        # partition top N
+        rel_t_partition = np.argpartition(stats_to_sort[lfc_above_cutoff_idx], -top_n)[-top_n:]
+        t_partition = lfc_above_cutoff_idx[rel_t_partition]
+        # sort the top N partition
+        rel_sort_order = np.argsort(stats_to_sort[t_partition])[::-1]
+        sort_order = t_partition[rel_sort_order]
+    else:
+        # partition and sort top N, ignoring lfc cutoff
+        partition = np.argpartition(stats_to_sort, -top_n)[-top_n:]
+        rel_sort_order = np.argsort(stats_to_sort[partition])[::-1]
+        indices = np.indices(stats_to_sort.shape)[0]
+        sort_order = indices[partition][rel_sort_order]
+
+    # top n slice based upon sort order
     logfoldchanges_top_n = logfoldchanges[sort_order]
     pvals_top_n = pvals[sort_order]
     pvals_adj_top_n = pvals_adj[sort_order]
 
     # varIndex, logfoldchange, pval, pval_adj
-    result = [[sort_order[i],
-               logfoldchanges_top_n[i],
-               pvals_top_n[i],
-               pvals_adj_top_n[i]] for i in range(top_n)]
+    result = [[sort_order[i], logfoldchanges_top_n[i], pvals_top_n[i], pvals_adj_top_n[i]] for i in range(top_n)]
     return result

server/app/scanpy_engine/scanpy_engine.py

@@ -1,15 +1,20 @@
 import warnings
 
 import numpy as np
-from pandas import DataFrame
 from pandas.core.dtypes.dtypes import CategoricalDtype
 import scanpy.api as sc
-from scipy import sparse
 
 from server.app.driver.driver import CXGDriver
 from server.app.util.constants import Axis, DEFAULT_TOP_N
-from server.app.util.errors import FilterError, InteractiveError, PrepareError, ScanpyFileError
+from server.app.util.errors import (
+    FilterError,
+    JSONEncodingValueError,
+    PrepareError,
+    ScanpyFileError,
+)
+from server.app.util.utils import jsonify_scanpy
 from server.app.scanpy_engine.diffexp import diffexp_ttest
+from server.app.util.fbs.matrix import encode_matrix_fbs
 
 """
 Sort order for methods
@@ -22,7 +27,6 @@ Sort order for methods
 
 
 class ScanpyEngine(CXGDriver):
-
     def __init__(self, data, args):
         super().__init__(data, args)
         self._alias_annotation_names(Axis.OBS, args["obs_names"])
@@ -55,46 +59,73 @@ class ScanpyEngine(CXGDriver):
             df_axis.rename(inplace=True, columns={"index": "name"})
         elif name in df_axis.columns:
             if name not in df_axis.columns:
-                raise KeyError(f"Annotation name {name}, specified in --{ax_name}-name does not exist.")
+                raise KeyError(
+                    f"Annotation name {name}, specified in --{ax_name}-name does not exist."
+                )
             if not df_axis[name].is_unique:
-                raise KeyError(f"Values in -{ax_name}-name must be unique. "
-                               "Please prepare data to contain unique values.")
+                raise KeyError(
+                    f"Values in -{ax_name}-name must be unique. "
+                    "Please prepare data to contain unique values."
+                )
             # reset index to simple range; alias user-specified annotation to "name"
             df_axis.reset_index(drop=True, inplace=True)
             df_axis.rename(inplace=True, columns={name: "name"})
         else:
-            raise KeyError(f"Annotation name {name}, specified in --{ax_name}_name does not exist.")
+            raise KeyError(
+                f"Annotation name {name}, specified in --{ax_name}_name does not exist."
+            )
+
+    @staticmethod
+    def _can_cast_to_float32(ann):
+        if ann.dtype.kind == "f":
+            if not np.can_cast(ann.dtype, np.float32):
+                warnings.warn(
+                    f"Annotation {ann.name} will be converted to 32 bit float and may lose precision."
+                )
+            return True
+        return False
+
+    @staticmethod
+    def _can_cast_to_int32(ann):
+        if ann.dtype.kind in ["i", "u"]:
+            if np.can_cast(ann.dtype, np.int32):
+                return True
+            ii32 = np.iinfo(np.int32)
+            if ann.min() >= ii32.min and ann.max() <= ii32.max:
+                return True
+        return False
 
     def _create_schema(self):
         self.schema = {
             "dataframe": {
                 "nObs": self.cell_count,
                 "nVar": self.gene_count,
-                "type": str(self.data.X.dtype)
+                "type": str(self.data.X.dtype),
             },
-            "annotations": {
-                "obs": [],
-                "var": []
-            }
+            "annotations": {"obs": [], "var": []},
         }
         for ax in Axis:
             curr_axis = getattr(self.data, str(ax))
             for ann in curr_axis:
                 ann_schema = {"name": ann}
-                data_kind = curr_axis[ann].dtype.kind
-                if data_kind == "f":
+                dtype = curr_axis[ann].dtype
+                data_kind = dtype.kind
+
+                if self._can_cast_to_float32(curr_axis[ann]):
                     ann_schema["type"] = "float32"
-                elif data_kind in ["i", "u"]:
+                elif self._can_cast_to_int32(curr_axis[ann]):
                     ann_schema["type"] = "int32"
-                elif data_kind == "?":
+                elif dtype == np.bool_:
                     ann_schema["type"] = "boolean"
-                elif data_kind == "O" and curr_axis[ann].dtype == "object":
+                elif data_kind == "O" and dtype == "object":
                     ann_schema["type"] = "string"
-                elif data_kind == "O" and curr_axis[ann].dtype == "category":
+                elif data_kind == "O" and dtype == "category":
                     ann_schema["type"] = "categorical"
                     ann_schema["categories"] = curr_axis[ann].dtype.categories.tolist()
                 else:
-                    raise TypeError(f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene.")
+                    raise TypeError(
+                        f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene."
+                    )
                 self.schema["annotations"][ax].append(ann_schema)
 
     @staticmethod
@@ -105,32 +136,41 @@ class ScanpyEngine(CXGDriver):
         try:
             result = sc.read(data, cache=True)
         except ValueError:
-            raise ScanpyFileError("File must be in the .h5ad format. Please read "
-                                  "https://github.com/theislab/scanpy_usage/blob/master/170505_seurat/info_h5ad.md to "
-                                  "learn more about this format. You may be able to convert your file into this format "
-                                  "using `cellxgene prepare`, please run `cellxgene prepare --help` for more "
-                                  "information.")
+            raise ScanpyFileError(
+                "File must be in the .h5ad format. Please read "
+                "https://github.com/theislab/scanpy_usage/blob/master/170505_seurat/info_h5ad.md to "
+                "learn more about this format. You may be able to convert your file into this format "
+                "using `cellxgene prepare`, please run `cellxgene prepare --help` for more "
+                "information."
+            )
         except Exception as e:
-            raise ScanpyFileError(f"Error while loading file: {e}, File must be in the .h5ad format, please check "
-                                  f"that your input and try again.")
+            raise ScanpyFileError(
+                f"Error while loading file: {e}, File must be in the .h5ad format, please check "
+                f"that your input and try again."
+            )
         return result
 
     def _validate_data_types(self):
         if self.data.X.dtype != "float32":
-            warnings.warn(f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
-                          f"Precision may be truncated.")
+            warnings.warn(
+                f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
+                f"Precision may be truncated."
+            )
         for ax in Axis:
             curr_axis = getattr(self.data, str(ax))
             for ann in curr_axis:
                 datatype = curr_axis[ann].dtype
-                downcast_map = {"int64": "int32",
-                                "uint32": "int32",
-                                "uint64": "int32",
-                                "float64": "float32",
-                                }
+                downcast_map = {
+                    "int64": "int32",
+                    "uint32": "int32",
+                    "uint64": "int32",
+                    "float64": "float32",
+                }
                 if datatype in downcast_map:
-                    warnings.warn(f"Scanpy annotation {ax}:{ann} is in unsupported format: {datatype}. "
-                                  f"Data will be downcast to {downcast_map[datatype]}.")
+                    warnings.warn(
+                        f"Scanpy annotation {ax}:{ann} is in unsupported format: {datatype}. "
+                        f"Data will be downcast to {downcast_map[datatype]}."
+                    )
                 if isinstance(datatype, CategoricalDtype):
                     category_num = len(curr_axis[ann].dtype.categories)
                     if category_num > 500 and category_num > self.max_category_items:
@@ -138,7 +178,8 @@ class ScanpyEngine(CXGDriver):
                             f"{str(ax).title()} annotation '{ann}' has {category_num} categories, this may be "
                             f"cumbersome or slow to display. We recommend setting the "
                             f"--max-category-items option to 500, this will hide categorical "
-                            f"annotations with more than 500 categories in the UI")
+                            f"annotations with more than 500 categories in the UI"
+                        )
 
     def _validate_data_calculations(self):
         layout_key = f"X_{self.layout_method}"
@@ -150,28 +191,12 @@ class ScanpyEngine(CXGDriver):
                 f" layout may have been computed. The requested layout must be pre-calculated and saved "
                 f"back in the h5ad file. You can run "
                 f"`cellxgene prepare --layout {self.layout_method} <datafile>` "
-                f"to solve this problem. ")
-
-    def filter_dataframe(self, filter):
-        """
-         Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
-         indexing and filtering by annotation value. Filters are combined with the and operator.
-         See REST specs for info on filter format:
-         # TODO update this link to swagger when it's done
-         https://docs.google.com/document/d/1Fxjp1SKtCk7l8QP9-7KAjGXL0eldi_qEnNT0NmlGzXI/edit#heading=h.8qc9q57amldx
-
-        :param filter: dictionary with filter params
-        :return: View into scanpy object with cells/genes filtered
-        """
-        if not filter:
-            return self.data
-        obs_selector, var_selector = self._filter_to_mask(filter, use_slices=False)
-        data = self._slice(self.data, obs_selector, var_selector)
-        return data
+                f"to solve this problem. "
+            )
 
     @staticmethod
     def _annotation_filter_to_mask(filter, d_axis, count):
-        mask = np.ones((count, ), dtype=bool)
+        mask = np.ones((count,), dtype=bool)
         for v in filter:
             if d_axis[v["name"]].dtype.name in ["boolean", "category", "object"]:
                 key_idx = np.in1d(getattr(d_axis, v["name"]), v["values"])
@@ -189,24 +214,28 @@ class ScanpyEngine(CXGDriver):
 
     @staticmethod
     def _index_filter_to_mask(filter, count):
-        mask = np.zeros((count, ), dtype=bool)
+        mask = np.zeros((count,), dtype=bool)
         for i in filter:
             if type(i) == list:
-                mask[i[0]:i[1]] = True
+                mask[i[0] : i[1]] = True
             else:
                 mask[i] = True
         return mask
 
     @staticmethod
     def _axis_filter_to_mask(filter, d_axis, count):
-        mask = np.ones((count, ), dtype=bool)
+        mask = np.ones((count,), dtype=bool)
         if "index" in filter:
-            mask = np.logical_and(mask, ScanpyEngine._index_filter_to_mask(filter["index"], count))
+            mask = np.logical_and(
+                mask, ScanpyEngine._index_filter_to_mask(filter["index"], count)
+            )
         if "annotation_value" in filter:
-            mask = np.logical_and(mask,
-                                  ScanpyEngine._annotation_filter_to_mask(filter["annotation_value"],
-                                                                          d_axis,
-                                                                          count))
+            mask = np.logical_and(
+                mask,
+                ScanpyEngine._annotation_filter_to_mask(
+                    filter["annotation_value"], d_axis, count
+                ),
+            )
         return mask
 
     def _filter_to_mask(self, filter, use_slices=True):
@@ -219,142 +248,87 @@ class ScanpyEngine(CXGDriver):
 
         if filter is not None:
             if Axis.OBS in filter:
-                obs_selector = self._axis_filter_to_mask(filter["obs"], self.data.obs, self.data.n_obs)
+                obs_selector = self._axis_filter_to_mask(
+                    filter["obs"], self.data.obs, self.data.n_obs
+                )
             if Axis.VAR in filter:
-                var_selector = self._axis_filter_to_mask(filter["var"], self.data.var, self.data.n_vars)
+                var_selector = self._axis_filter_to_mask(
+                    filter["var"], self.data.var, self.data.n_vars
+                )
         return obs_selector, var_selector
 
-    @staticmethod
-    def _slice(data, obs_selector=None, vars_selector=None):
-        """
-        Slice date using any selector that the AnnData object
-        supprots for slicing.  If selector is None, will not slice
-        on that axis.
-
-        This method exists to optimize filtering/slicing sparse data that has
-        access patterns which impact slicing performance.
-
-        https://docs.scipy.org/doc/scipy/reference/sparse.html
-        """
-        prefer_row_access = sparse.isspmatrix_csr(data._X) or sparse.isspmatrix_lil(data._X) \
-            or sparse.isspmatrix_bsr(data._X)
-        if prefer_row_access:
-            # Row-major slicing
-            if obs_selector is not None:
-                data = data[obs_selector, :]
-            if vars_selector is not None:
-                data = data[:, vars_selector]
+    def annotation_to_fbs_matrix(self, axis, fields=None):
+        if axis == Axis.OBS:
+            df = self.data.obs
         else:
-            # Col-major slicing
-            if vars_selector is not None:
-                data = data[:, vars_selector]
-            if obs_selector is not None:
-                data = data[obs_selector, :]
+            df = self.data.var
+        if fields is not None and len(fields) > 0:
+            df = df[fields]
+        return encode_matrix_fbs(df, col_idx=df.columns)
 
-        return data
-
-    def annotation(self, filter, axis, fields=None):
+    def data_frame_to_fbs_matrix(self, filter, axis):
         """
-         Gets annotation value for each observation
+        Retrieves data 'X' and returns in a flatbuffer Matrix.
         :param filter: filter: dictionary with filter params
         :param axis: string obs or var
-        :param fields: list of keys for annotation to return, returns all annotation values if not set.
-        :return: dict: names - list of fields in order, data - list of lists or metadata
-        [observation ids, val1, val2...]
-        """
-        try:
-            obs_selector, var_selector = self._filter_to_mask(filter)
-        except (KeyError, IndexError) as e:
-            raise FilterError(f"Error parsing filter: {e}") from e
-        if axis == Axis.OBS:
-            obs = self.data.obs[obs_selector]
-            if not fields:
-                fields = obs.columns.tolist()
-            result = {
-                "names": fields,
-                "data": DataFrame(obs[fields]).to_records(index=True).tolist()
-            }
-        else:
-            var = self.data.var[var_selector]
-            if not fields:
-                fields = var.columns.tolist()
-            result = {
-                "names": fields,
-                "data": DataFrame(var[fields]).to_records(index=True).tolist()
-            }
-        return result
+        :return: flatbuffer Matrix
 
-    def data_frame(self, filter, axis):
-        """
-        Retrieves data for each variable for observations in data frame
-        :param filter: filter: dictionary with filter params
-        :param axis: string obs or var
-        :return: {
-            "var": list of variable ids,
-            "obs": [cellid, var1 expression, var2 expression, ...],
-        }
+        Caveats:
+        * currently only supports access on VAR axis
+        * currently only supports filtering on VAR axis
         """
+        if axis != Axis.VAR:
+            raise ValueError("Only VAR dimension access is supported")
         try:
-            obs_selector, var_selector = self._filter_to_mask(filter)
-        except (KeyError, IndexError) as e:
+            obs_selector, var_selector = self._filter_to_mask(filter, use_slices=False)
+        except (KeyError, IndexError, TypeError) as e:
             raise FilterError(f"Error parsing filter: {e}") from e
-        _X = self.data._X[obs_selector, var_selector]
-        if sparse.issparse(_X):
-            _X = _X.toarray()
-        var_index_sliced = self.data.var.index[var_selector]
-        obs_index_sliced = self.data.obs.index[obs_selector]
-        if axis == Axis.OBS:
-            result = {
-                "var": var_index_sliced.tolist(),
-                "obs": DataFrame(_X, index=obs_index_sliced).to_records(index=True).tolist()
-            }
-        else:
-            result = {
-                "obs": obs_index_sliced.tolist(),
-                "var": DataFrame(_X.T, index=var_index_sliced).to_records(index=True).tolist()
-            }
-        return result
+        if obs_selector is not None:
+            raise FilterError("filtering on obs unsupported")
+
+        # Currently only handles VAR dimension
+        X = self.data._X
+        if var_selector is not None:
+            X = X[:, var_selector]
+        return encode_matrix_fbs(X, col_idx=np.nonzero(var_selector)[0], row_idx=None)
 
     def diffexp_topN(self, obsFilterA, obsFilterB, top_n=None, interactive_limit=None):
         if Axis.VAR in obsFilterA or Axis.VAR in obsFilterB:
             raise FilterError("Observation filters may not contain vaiable conditions")
         try:
-            obs_mask_A = self._axis_filter_to_mask(obsFilterA["obs"], self.data.obs, self.data.n_obs)
-            obs_mask_B = self._axis_filter_to_mask(obsFilterB["obs"], self.data.obs, self.data.n_obs)
+            obs_mask_A = self._axis_filter_to_mask(
+                obsFilterA["obs"], self.data.obs, self.data.n_obs
+            )
+            obs_mask_B = self._axis_filter_to_mask(
+                obsFilterB["obs"], self.data.obs, self.data.n_obs
+            )
         except (KeyError, IndexError) as e:
             raise FilterError(f"Error parsing filter: {e}") from e
         if top_n is None:
             top_n = DEFAULT_TOP_N
-        result = diffexp_ttest(self.data, obs_mask_A, obs_mask_B, top_n)
-        return sorted(result, key=lambda r: r[0])
+        result = diffexp_ttest(
+            self.data, obs_mask_A, obs_mask_B, top_n, self.diffexp_lfc_cutoff
+        )
+        try:
+            return jsonify_scanpy(result)
+        except ValueError:
+            raise JSONEncodingValueError(
+                "Error encoding differential expression to JSON"
+            )
 
-    def layout(self, filter, interactive_limit=None):
+    def layout_to_fbs_matrix(self):
         """
-        Computes a n-d layout for cells through dimensionality reduction.
-        :param filter: filter: dictionary with filter params
-        :param interactive_limit: -- don't compute if total # genes in dataframes are larger than this
-        :return:  [cellid, x, y, ...]
+        Return the default 2-D layout for cells as a FBS Matrix.
+
+        Caveats:
+        * does not support filtering
+        * only returns Matrix in columnar layout
         """
         try:
-            df = self.filter_dataframe(filter)
-        except (KeyError, IndexError) as e:
-            raise FilterError(f"Error parsing filter: {e}") from e
-        if interactive_limit and len(df.obs.index) > interactive_limit:
-            raise InteractiveError("Size data is too large for interactive computation")
-        # TODO Filtering cells is fine, but filtering genes does nothing because the neighbors are
-        # calculated using the original vars (geneset) and this doesn’t get updated when you use less.
-        # Need to recalculate neighbors (long) if user requests new layout filtered by var
-        # TODO for MVP we are pushing computation of layout to preprocessing and not allowing re-layout
-        # this will probably change after user feedback
-        # getattr(sc.tl, self.layout_method)(df, random_state=123)
-        try:
-            df_layout = df.obsm[f"X_{self.layout_method}"]
+            df_layout = self.data.obsm[f"X_{self.layout_method}"]
         except ValueError as e:
-            raise PrepareError(f"Layout has not been calculated using {self.layout_method}, "
-                               f"please prepare your datafile and relaunch cellxgene") from e
-        normalized_layout = DataFrame((df_layout - df_layout.min()) / (df_layout.max() - df_layout.min()),
-                                      index=df.obs.index)
-        return {
-            "ndims": normalized_layout.shape[1],
-            "coordinates": normalized_layout.to_records(index=True).tolist()
-        }
+            raise PrepareError(
+                f"Layout has not been calculated using {self.layout_method}, "
+                f"please prepare your datafile and relaunch cellxgene") from e
+        normalized_layout = (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min())
+        return encode_matrix_fbs(normalized_layout.astype(dtype=np.float32), col_idx=None, row_idx=None)

server/app/util/constants.py

@@ -25,3 +25,8 @@ class Axis(AugmentedEnum):
 class DiffExpMode(AugmentedEnum):
     TOP_N = "topN"
     VAR_FILTER = "varFilter"
+
+
+JSON_NaN_to_num_warning_msg = (
+    "JSON encoding failure - please verify all data are finite values (no NaN or Infinities)"
+)

server/app/util/errors.py

@@ -16,6 +16,15 @@ class InteractiveError(Exception):
         self.message = message
 
 
+class JSONEncodingValueError(Exception):
+    """
+    Raised when file loaded into scanpy is misformatted
+    """
+
+    def __init__(self, message):
+        self.message = message
+
+
 class MimeTypeError(Exception):
     """
     Raised when incompatible MIME type selected

server/app/util/fbs/NetEncoding/Column.py

@@ -0,0 +1,41 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Column(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsColumn(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Column()
+        x.Init(buf, n + offset)
+        return x
+
+    # Column
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Column
+    def UType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Column
+    def U(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+def ColumnStart(builder): builder.StartObject(2)
+def ColumnAddUType(builder, uType): builder.PrependUint8Slot(0, uType, 0)
+def ColumnAddU(builder, u): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(u), 0)
+def ColumnEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Float32Array.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Float32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsFloat32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Float32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Float32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Float32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Float32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Float32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Float32Flags, o)
+        return 0
+
+    # Float32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Float32ArrayStart(builder): builder.StartObject(1)
+def Float32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Float32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Float32ArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Float64Array.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Float64Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsFloat64Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Float64Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Float64Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Float64Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Float64Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 8))
+        return 0
+
+    # Float64Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Float64Flags, o)
+        return 0
+
+    # Float64Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Float64ArrayStart(builder): builder.StartObject(1)
+def Float64ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Float64ArrayStartDataVector(builder, numElems): return builder.StartVector(8, numElems, 8)
+def Float64ArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Int32Array.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Int32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsInt32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Int32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Int32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Int32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Int32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Int32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Int32Flags, o)
+        return 0
+
+    # Int32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Int32ArrayStart(builder): builder.StartObject(1)
+def Int32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Int32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Int32ArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/JSONEncodedArray.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class JSONEncodedArray(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsJSONEncodedArray(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = JSONEncodedArray()
+        x.Init(buf, n + offset)
+        return x
+
+    # JSONEncodedArray
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # JSONEncodedArray
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 1))
+        return 0
+
+    # JSONEncodedArray
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Uint8Flags, o)
+        return 0
+
+    # JSONEncodedArray
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def JSONEncodedArrayStart(builder): builder.StartObject(1)
+def JSONEncodedArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def JSONEncodedArrayStartDataVector(builder, numElems): return builder.StartVector(1, numElems, 1)
+def JSONEncodedArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Matrix.py

@@ -0,0 +1,98 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Matrix(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsMatrix(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Matrix()
+        x.Init(buf, n + offset)
+        return x
+
+    # Matrix
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Matrix
+    def NRows(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def NCols(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def Columns(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8))
+        if o != 0:
+            x = self._tab.Vector(o)
+            x += flatbuffers.number_types.UOffsetTFlags.py_type(j) * 4
+            x = self._tab.Indirect(x)
+            from .Column import Column
+            obj = Column()
+            obj.Init(self._tab.Bytes, x)
+            return obj
+        return None
+
+    # Matrix
+    def ColumnsLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+    # Matrix
+    def ColIndexType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(10))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def ColIndex(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+    # Matrix
+    def RowIndexType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(14))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def RowIndex(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(16))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+def MatrixStart(builder): builder.StartObject(7)
+def MatrixAddNRows(builder, nRows): builder.PrependUint32Slot(0, nRows, 0)
+def MatrixAddNCols(builder, nCols): builder.PrependUint32Slot(1, nCols, 0)
+def MatrixAddColumns(builder, columns): builder.PrependUOffsetTRelativeSlot(2, flatbuffers.number_types.UOffsetTFlags.py_type(columns), 0)
+def MatrixStartColumnsVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def MatrixAddColIndexType(builder, colIndexType): builder.PrependUint8Slot(3, colIndexType, 0)
+def MatrixAddColIndex(builder, colIndex): builder.PrependUOffsetTRelativeSlot(4, flatbuffers.number_types.UOffsetTFlags.py_type(colIndex), 0)
+def MatrixAddRowIndexType(builder, rowIndexType): builder.PrependUint8Slot(5, rowIndexType, 0)
+def MatrixAddRowIndex(builder, rowIndex): builder.PrependUOffsetTRelativeSlot(6, flatbuffers.number_types.UOffsetTFlags.py_type(rowIndex), 0)
+def MatrixEnd(builder): return builder.EndObject()