bumped version (#602 )

0.6.0 was burned on pypi
bump version (#601 )
2026-09-26 21:38:11 +08:00 · 2019-02-14 10:00:16 -08:00 · 2019-02-13 16:29:14 -08:00 · 2019-02-13 15:35:45 -08:00 · 2019-02-11 09:01:37 -08:00 · 2019-02-08 14:56:16 -08:00
111 changed files with 8574 additions and 3683 deletions
@@ -1,6 +1,5 @@
 [bumpversion]
-current_version = 0.0.2
-
+current_version = 0.6.1

 [bumpversion:file:setup.py]
 search = version="{current_version}"
@@ -0,0 +1,5 @@
+bin
+client
+dist
+docs
+server
@@ -35,3 +35,7 @@ npm-debug.log
 __pycache__
 *.DS_Store*
 data
+
+# Jekyll
+docs/_site/
+docs/Gemfile.lock
@@ -8,13 +8,14 @@ cache:
 install:
  - set -eo pipefail
  - pip install flake8
-  - ./bin/build-client
-  - pip install -e .
+  - make build
+  - make install
  - pip install -r server/requirements-dev.txt
+  - docker build .
 script:
  - set -eo pipefail
-  - flake8 server/app/
-  - flake8 server/cli/
+  - flake8 server
+  - black --check
  - npm run --prefix client/ build
  - npm run --prefix client/ test
  - pytest -s server/test
@@ -0,0 +1,10 @@
+FROM ubuntu:bionic
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+RUN apt-get update && \
+    apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev && \
+    pip3 install cellxgene
+
+ENTRYPOINT ["cellxgene"]
@@ -1,3 +1,4 @@
 recursive-include server/app/web/templates *
 recursive-include server/app/web/static *

+include server/requirements.txt
@@ -1,111 +1,288 @@
 # cellxgene

-### An interactive, performant explorer for single cell transcriptomics data.
+> an interactive explorer for single-cell transcriptomics data

-<img align="right" width="350" height="218" src="./example-dataset/cellxgene-demo.gif" pad="50px">
-cellxgene is an open-source experiment in how to bring powerful tools from modern web development to visualize and explore large single-cell transcriptomics datasets.
-Started in the context of the Human Cell Atlas Consortium, cellxgene hopes to both enable scientists to explore their data and to equip developers with scalable, reusable patterns and frameworks for visualizing large scientific datasets.
+`cellxgene` is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.

-## Features
+<img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">

- **Visualization at scale:** built with [WebGL](https://www.khronos.org/webgl/), [React](https://reactjs.org/) & [Redux](https://redux.js.org/) to handle visualization of at least 1 million cells.
+## getting started

- **Interactive exploration:** select, cross-filter, and compare subsets of your data with performant indexing and data handling.
+You'll need **python 3.6** and **Google Chrome**. (_Warning_: Python 3.7 is **not** supported at this time)
+The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).

- **Flexible API:** the cellxgene client-server model is designed to support a range of existing analysis packages for backend computational tasks (eg scanpy), integrated with client-side visualization via a [REST API](https://restfulapi.net/).
+To install run

-## Getting Started
+```
+pip install cellxgene
+```

-**Requirements**
+To start exploring a dataset call
+
+```
+cellxgene launch dataset.h5ad --open
+```
+
+If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
+
+```
+cellxgene launch pbmc3k.h5ad --open
+```
+
+You should see your web browser open with the following
+
+<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+
+**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+
+There are several options available, such as:
+
+- `--layout` to specify the layout as `tsne` or `umap`
+- `--title` to show a title on the explorer
+- `--open` to automatically open the web browser after launching (OS X only)
+
+To see all options call
+
+```
+cellxgene launch --help
+```
+
+There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+
+## data formatting
+
+### assumptions
+
+The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
+
+- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
+- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
+- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_tsne` for the `tsne` layout or `X_umap` for the `umap` layout)
+- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
+
+### prepare
+
+The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
+
+To prepare from an existing `.h5ad` file use
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
+```
+
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
+
+Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
+```
+
+To see all options call
+
+```
+cellxgene prepare --help
+```
+
+**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
+
+```
+pip install cellxgene[louvain]
+```
+
+If the aforementioned optional package installation fails, you can also install these packages directly:
+
+```
+pip install python-igraph louvain>=0.6
+```
+
+## conda and virtual environments
+
+If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.6
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3.6 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+## docker
+
+We have included a dockerfile to conveniently run cellxgene from docker.
+
+1. Build the image `docker build . -t cellxgene`
+2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
+   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
+
+## FAQ
+
+<details>
+
+<summary> questions about data formatting </summary>
+  
+<hr>
+  
+> Someone sent me a directory of `10X-Genomics` data with a `mtx` file and I've never used `scanpy`, can I use `cellxgene`?
+
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+```
+
+Depending on the size of the dataset, this may take some time. Once it's done, call
+
+```
+cellxgene launch data-processed.h5ad --layout=umap --open
+```
+
+And your web browser should open with an interactive view of your data.
+
+<hr>
+
+> In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
+
+Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using
+
+```
+pip install cellxgene[louvain]
+```
+
+<hr>
+
+> I ran `prepare` and I'm getting results that look unexpected
+
+You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
+```
+
+It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
+
+<hr>
+
+> I have extra metadata that I want to add to my dataset
+
+Currently this is not supported directly, but you should be able to do this manually using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
+
+<hr>
+
+> What part of the anndata objects does cellxgene pull in for visualization?
+
+- `.obs` and `.var` annotations are use to extract metadata for filtering
+- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
+- `.obsm` is used for layout
+
+</details>
+
+<details>
+  
+<summary> questions about installing and building </summary>
+
+<hr>
+
+> I tried to `pip install cellxgene` and got a weird error about missing paths to an HDF5 library?  
+
+You probably just need to install HDF5 first. If you're on a mac, you can simply `brew install hdf5` and then try `pip install cellxgene` again.  
+
+> I tried to `pip install cellxgene` and got another weird error I don't understand  
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.  
+
+<hr>
+
+> I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
+
+This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
+
+</details>
+
+<details>
+
+<summary> questions about algorithms </summary>
+
+<hr>
+
+> How are you computing and sorting differential expression results?
+
+Currently we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
+
+</details>
+
+## developer guide
+
+This project has made a few key design choices
+
+- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
+- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
+- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
+
+Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
+
+If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine

- OS: OSX, Windows, Linux -- the developers are currently testing on OSX and Windows (via WSL using Ubuntu). It should work on other platforms but if you are using something different and need help, please let us know.
 - python 3.6
- python3 tkinter
- npm
- Google Chrome
+- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)

-**Clone project**
+Then clone the project

-    git clone https://github.com/chanzuckerberg/cellxgene.git
+```
+git clone https://github.com/chanzuckerberg/cellxgene.git
+```

-**Install client**
+Build the client web assets by calling this from inside the `cellxgene` folder

-    cd cellxgene
-    ./bin/build-client
+```
+./bin/build-client
+```

-**To use with virtual env for python**
-(optional, but recommended)
+Install all requirements (we recommend doing this inside a virtual environment)

-    ENV_NAME=cellxgene
-    python3 -m venv ${ENV_NAME}
-    source ${ENV_NAME}/bin/activate
+```
+pip install -e .
+```

-**Install server**
+You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.

-     pip install -e .
+If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.

-**Run (with demo data)**
+## development roadmap

-    cellxgene launch --title PBMC3K example-dataset/pbmc3k.h5ad
+`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact)

-**Help**
+- **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
+- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene` to render the results of these analyses when they have been performed
+- **Deploy to web** Many projects release public data browser websites alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
+- **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data

-    cellxgene --help
+## contributing

-_For help with the scanpy engine_
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!

-    cellxgene scanpy --help
+## inspiration and collaboration

-## Using your own data
+We've been heavily inspired by several other related single-cell visualization projects, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), and many others. We hope to explore collaborations where useful as this community works together on improving interactive visualization for single-cell data.

-### Scanpy
+We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.

-To prepare your data you will need to format your data into AnnData format using scanpy and calculate PCA and nearest neighbors and save in h5ad format.
+We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.

-1.  [Load data into scanpy](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)
+We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)

-    - Ensure that `obs`'s index is the cell names: `print(data.obs_names)` should show your cell indices. If it shows gene names, you may need to just call `data.transpose()`.
+## help and contact

-2.  Calculate PCA
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!

-    sc.pp.pca(data) ## sc is scanpy.api
+## reuse

-3.  Calculate nearest neighbors (depending on layout algorithm)
-
-    ```
-    # For umap layout algorithm, you need to use the "umap" method for neighbors
-    sc.pp.neighbors(data, method="umap", metric="euclidean", use_rep="X_pca")
-
-    # For tsne layout algorithm, you can use either "umap" or "gauss"; we recommend "gauss"
-    sc.pp.neighbors(data, method="gauss", metric="euclidean", use_rep="X_pca")
-    ```
-
-4.  Save file
-
-    ```
-    # cellxgene requires file to be named data.h5ad
-    data.write("data.h5ad")
-    ```
-
-## Contributing
-
-We warmly welcome contributions from the community. Please submit any bug reports and feature requests through github issues. Please submit any direct contributions via a branch + pull request.
-
-## Inspiration and collaboration
-
-We’ve been inspired by several other related efforts in this space, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), & many others; we hope to explore collaborations where useful.
-
-## Help/Contact
-
-Have questions, suggestions, or comments? You can contact us by joining [CZI Science Slack](https://cziscience.slack.com/messages/CCTA8DF1T) and posting in the #cellxgene channel. Please submit any feature requests or bugs as an issue in github. We'd love to hear from you!
-
-## Reuse
-
-This project was started with the sole goal of empowering the scientific community to explore and understand their data. As such, we whole-heartedly encourage other scientific tool builders to adopt the patterns, tools, and code from this project, and reach out to us with ideas or questions using Github Issues or Pull Requests. All code is freely available for reuse under the [MIT license](https://opensource.org/licenses/MIT).
-
-## Acknowledgements
-
-cellxgene is inspired by many innovative projects. We would like to specifically thank:
-
- Alex Wolf for the demo dataset.
- Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for API inspiration.
+This project was started with the sole goal of empowering the scientific community to explore and understand their data. As such, we encourage other scientific tool builders in academia or industry to adopt the patterns, tools, and code from this project, and reach out to us with ideas or questions. All code is freely available for reuse under the [MIT license](https://opensource.org/licenses/MIT).
@@ -1,15 +0,0 @@
-#!/bin/bash
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-CELLXGENE_DIR=$(dirname $DIR)
-
-cd $CELLXGENE_DIR
-
-npm install --prefix client/ client
-npm run  --prefix client build
-rm -rf server/app/web/static
-mkdir -p server/app/web/static/img
-cp client/build/index.html server/app/web/templates/
-cp -r client/build/static server/app/web/
-
-cp client/build/favicon.png server/app/web/static/img
-cp client/build/service-worker.js server/app/web/static/js/
@@ -1,16 +0,0 @@
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-CELLXGENE_DIR=$(dirname $DIR)
-
-echo "Uninstalling cellxgene"
-yes | pip uninstall cellxgene
-echo "removing node_modules"
-rm -rf $CELLXGENE_DIR/client/node_modules
-echo "removing client_build"
-rm -rf $CELLXGENE_DIR/client/build
-echo "removing egg-info"
-rm -rf $CELLXGENE_DIR/cellxgene.egg-info
-echo "removing static files"
-rm -f $CELLXGENE_DIR/server/app/web/templates/index.html
-rm -rf $CELLXGENE_DIR/server/app/web/static
-echo "cellxgene cleanup complete"
-
@@ -0,0 +1,9 @@
+/*
+Define globals which are present in the client, but not in node (and therefore not in
+the jest test environment).
+*/
+
+import { TextDecoder, TextEncoder } from "util";
+
+global.TextDecoder = TextDecoder;
+global.TextEncoder = TextEncoder;
@@ -1,5 +1,7 @@
 /* eslint no-bitwise: "off" */
 import _ from "lodash";
+import { flatbuffers } from "flatbuffers";
+import { NetEncoding } from "../../../src/util/stateManager/matrix_generated";

 /*
 test data mocking REST 0.2 API responses.  Used in several tests.
@@ -58,7 +60,7 @@ const aSchemaResponse = {
  }
 };

-const anAnnotationsObsResponse = {
+const anAnnotationsObsJSONResponse = {
  names: ["name", "field1", "field2", "field3", "field4"],
  data: _()
    .range(nObs)
@@ -73,7 +75,7 @@ const anAnnotationsObsResponse = {
    .value()
 };

-const anAnnotationsVarResponse = {
+const anAnnotationsVarJSONResponse = {
  names: ["fieldA", "fieldB", "fieldC", "fieldD", "name"],
  data: _()
    .range(nVar)
@@ -88,7 +90,74 @@ const anAnnotationsVarResponse = {
    .value()
 };

-const aLayoutResponse = {
+function encodeTypedArray(builder, uType, uData) {
+  const uTypeName = NetEncoding.TypedArray[uType];
+  const ArrayType = NetEncoding[uTypeName];
+  const dv = ArrayType.createDataVector(builder, uData);
+  builder.startObject(1);
+  builder.addFieldOffset(0, dv, 0);
+  return builder.endObject();
+}
+
+function encodeMatrix(columns, colIndex = undefined) {
+  const utf8Encoder = new TextEncoder("utf-8");
+  const builder = new flatbuffers.Builder(1024);
+  const cols = _.map(columns, carr => {
+    let uType;
+    let tarr;
+    if (_.every(carr, _.isNumber)) {
+      uType = NetEncoding.TypedArray.Float32Array;
+      tarr = encodeTypedArray(builder, uType, new Float32Array(carr));
+    } else {
+      uType = NetEncoding.TypedArray.JSONEncodedArray;
+      const json = JSON.stringify(carr);
+      const jsonUTF8 = utf8Encoder.encode(json);
+      tarr = encodeTypedArray(builder, uType, jsonUTF8);
+    }
+    NetEncoding.Column.startColumn(builder);
+    NetEncoding.Column.addUType(builder, uType);
+    NetEncoding.Column.addU(builder, tarr);
+    return NetEncoding.Column.endColumn(builder);
+  });
+
+  const encColumns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+  let encColIndex;
+  if (colIndex) {
+    encColIndex = encodeTypedArray(
+      builder,
+      NetEncoding.TypedArray.JSONEncodedArray,
+      utf8Encoder.encode(JSON.stringify(colIndex))
+    );
+  }
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, columns[0].length);
+  NetEncoding.Matrix.addNCols(builder, columns.length);
+  NetEncoding.Matrix.addColumns(builder, encColumns);
+  if (colIndex) {
+    NetEncoding.Matrix.addColIndexType(
+      builder,
+      NetEncoding.TypedArray.JSONEncodedArray
+    );
+    NetEncoding.Matrix.addColIndex(builder, encColIndex);
+  }
+  const root = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(root);
+  return builder.asUint8Array();
+}
+
+const anAnnotationsObsFBSResponse = (() => {
+  const columns = _.zip(...anAnnotationsObsJSONResponse.data).slice(1);
+  return encodeMatrix(columns, anAnnotationsObsJSONResponse.names);
+})();
+
+const anAnnotationsVarFBSResponse = (() => {
+  const columns = _.zip(...anAnnotationsVarJSONResponse.data).slice(1);
+  return encodeMatrix(columns, anAnnotationsVarJSONResponse.names);
+})();
+
+const aLayoutJSONResponse = {
  layout: {
    ndims: 2,
    coordinates: _()
@@ -98,6 +167,36 @@ const aLayoutResponse = {
  }
 };

+const aLayoutFBSResponse = (() => {
+  const coords = [
+    new Float32Array(nObs).fill(Math.random()),
+    new Float32Array(nObs).fill(Math.random())
+  ];
+  const builder = new flatbuffers.Builder(1024);
+
+  const cols = _.map(coords, carr => {
+    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
+    NetEncoding.Float32Array.startFloat32Array(builder);
+    NetEncoding.Float32Array.addData(builder, cdv);
+    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
+
+    NetEncoding.Column.startColumn(builder);
+    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
+    NetEncoding.Column.addU(builder, floatArr);
+    return NetEncoding.Column.endColumn(builder);
+  });
+
+  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, nObs);
+  NetEncoding.Matrix.addNCols(builder, nVar);
+  NetEncoding.Matrix.addColumns(builder, columns);
+  const matrix = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(matrix);
+  return builder.asUint8Array();
+})();
+
 const aDataObsResponse = {
  var: [2, 4, 29],
  obs: _()
@@ -107,10 +206,10 @@ const aDataObsResponse = {
 };

 export {
-  aLayoutResponse as layoutObs,
+  aLayoutFBSResponse as layoutObs,
  aDataObsResponse as dataObs,
-  anAnnotationsVarResponse as annotationsVar,
-  anAnnotationsObsResponse as annotationsObs,
+  anAnnotationsVarFBSResponse as annotationsVar,
+  anAnnotationsObsFBSResponse as annotationsObs,
  aSchemaResponse as schema,
  aConfigResponse as config
 };
@@ -0,0 +1,280 @@
+import summarizeAnnotations from "../../../src/util/stateManager/summarizeAnnotations";
+
+describe("summarizeAnnotations", () => {
+  const schema = {
+    annotations: {
+      obs: [
+        { name: "name", type: "string" },
+        { name: "nameString", type: "string" },
+        { name: "nameBoolean", type: "boolean" },
+        { name: "nameFloat32", type: "float32" },
+        { name: "nameInt32", type: "int32" },
+        {
+          name: "nameCategorical",
+          type: "categorical",
+          categories: [true, false, 1, 0, 0.00001, 4383.4833, "test", "", "0"]
+        }
+      ],
+      var: [{ name: "name", type: "string" }]
+    }
+  };
+
+  test("empty test", () => {
+    const summary = summarizeAnnotations(schema, [], []);
+    expect(summary).toEqual(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          },
+          nameFloat32: {
+            categorical: false,
+            range: {
+              max: undefined,
+              min: undefined,
+              nan: 0,
+              ninf: 0,
+              pinf: 0
+            }
+          },
+          nameInt32: {
+            categorical: false,
+            range: {
+              max: undefined,
+              min: undefined,
+              nan: 0,
+              ninf: 0,
+              pinf: 0
+            }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("simple test", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toEqual(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: ["hi"],
+            categoryCounts: new Map([["hi", 1]]),
+            numCategories: 1
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: [true],
+            categoryCounts: new Map([[true, 1]]),
+            numCategories: 1
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 39.3, max: 39.3, nan: 0, ninf: 0, pinf: 0 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99, nan: 0, ninf: 0, pinf: 0 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: [1],
+            categoryCounts: new Map([[1, 1]]),
+            numCategories: 1
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("multi test", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n0",
+        nameString: "hi",
+        nameBoolean: false,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      },
+      {
+        __index__: 1,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: false
+      },
+      {
+        __index__: 2,
+        name: "n2",
+        nameString: "bye",
+        nameBoolean: true,
+        nameFloat32: 0,
+        nameInt32: 99,
+        nameCategorical: "0"
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toMatchObject(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: expect.arrayContaining(["hi", "bye"]),
+            categoryCounts: new Map([["hi", 2], ["bye", 1]]),
+            numCategories: 2
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: expect.arrayContaining([true, false]),
+            categoryCounts: new Map([[true, 2], [false, 1]]),
+            numCategories: 2
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 0, max: 39.3, nan: 0, ninf: 0, pinf: 0 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99, nan: 0, ninf: 0, pinf: 0 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: expect.arrayContaining([1, false, "0"]),
+            categoryCounts: new Map([[1, 1], [false, 1], ["0", 1]]),
+            numCategories: 3
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("non-finite numbers", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n0",
+        nameString: "hi",
+        nameBoolean: false,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      },
+      {
+        __index__: 1,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: Number.NEGATIVE_INFINITY,
+        nameInt32: 99,
+        nameCategorical: false
+      },
+      {
+        __index__: 2,
+        name: "n2",
+        nameString: "bye",
+        nameBoolean: true,
+        nameFloat32: Number.NaN,
+        nameInt32: 99,
+        nameCategorical: "0"
+      },
+      {
+        __index__: 3,
+        name: "n2",
+        nameString: "bye",
+        nameBoolean: true,
+        nameFloat32: Number.POSITIVE_INFINITY,
+        nameInt32: 99,
+        nameCategorical: "0"
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toMatchObject(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: expect.arrayContaining(["hi", "bye"]),
+            categoryCounts: new Map([["hi", 2], ["bye", 1]]),
+            numCategories: 2
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: expect.arrayContaining([true, false]),
+            categoryCounts: new Map([[true, 2], [false, 1]]),
+            numCategories: 2
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 39.3, max: 39.3, nan: 1, ninf: 1, pinf: 1 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99, nan: 0, ninf: 0, pinf: 0 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: expect.arrayContaining([1, false, "0"]),
+            categoryCounts: new Map([[1, 1], [false, 1], ["0", 1]]),
+            numCategories: 3
+          }
+        },
+        var: {}
+      })
+    );
+  });
+});
@@ -30,7 +30,6 @@ describe("createUniverseFromRestV02Response", () => {
    create a universe from sample data nad validate its shape & contents
    */
    const { nObs, nVar } = REST.schema.schema.dataframe;
-
    const universe = Universe.createUniverseFromRestV02Response(
      REST.config,
      REST.schema,
@@ -66,56 +65,3 @@ describe("createUniverseFromRestV02Response", () => {
    expect(_.keys(universe.varNameToIndexMap)).toHaveLength(nVar);
  });
 });
-
-describe("convertExpressionRESTv02ToObject", () => {
-  /*
-  test convertExpressionRESTv02ToObject
-
-  convertExpressionRESTv02ToObject(
-    universe,
-    response) --> { geneName: Float32Array, geneName: Float32Array, ... }
-
-  reponse is a /data/obs response:
-  {
-    var: [ varIndices fetched ],
-    obs: [
-      [ obsIndex, evalue, ... ],
-      ...
-    ]
-  }
-  */
-  test("create from response data", () => {
-    const universe = Universe.createUniverseFromRestV02Response(
-      REST.config,
-      REST.schema,
-      REST.annotationsObs,
-      REST.annotationsVar,
-      REST.layoutObs
-    );
-    const expression = Universe.convertExpressionRESTv02ToObject(
-      universe,
-      REST.dataObs
-    );
-
-    /* Check that the expected keys are present */
-    const expectedGeneNames = _.map(
-      REST.dataObs.var,
-      v => REST.annotationsVar.data[v][5]
-    );
-    expect(Object.keys(expression)).toEqual(
-      expect.arrayContaining(expectedGeneNames)
-    );
-
-    const expectedExpressionValues = _.map(
-      _.unzip(REST.dataObs.obs),
-      a => new Float32Array(a)
-    );
-
-    _.forEach(REST.dataObs.var, (varIdx, idx) => {
-      const varName = universe.varAnnotations[varIdx].name;
-      expect(varName).toBeDefined();
-      expect(varIdx).toBe(universe.varNameToIndexMap[varName]);
-      expect(expression[varName]).toEqual(expectedExpressionValues[idx + 1]);
-    });
-  });
-});
@@ -168,10 +168,13 @@ describe("createObsDimensionMap", () => {
    */

    const { dimensionMap } = defaultBigBang();
-
+    const annotationNames = _.map(
+      REST.schema.schema.annotations.obs,
+      c => c.name
+    );
    const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
    expect(dimensionMap).toBeDefined();
-    REST.annotationsObs.names.forEach(name => {
+    annotationNames.forEach(name => {
      const dim = dimensionMap[obsAnnoDimensionName(name)];
      if (name === "name") {
        expect(dim).toBeUndefined();
@@ -184,11 +187,8 @@ describe("createObsDimensionMap", () => {
        }
      }
    });
-    expect(dimensionMap[layoutDimensionName("X")]).toBeInstanceOf(
-      Crossfilter.ScalarDimension
-    );
-    expect(dimensionMap[layoutDimensionName("Y")]).toBeInstanceOf(
-      Crossfilter.ScalarDimension
+    expect(dimensionMap[layoutDimensionName("XY")]).toBeInstanceOf(
+      Crossfilter.SpatialDimension
    );
  });
 });
@@ -0,0 +1,45 @@
+import {
+  countCategoryValues2D,
+  clearCaches
+} from "../../../src/util/stateManager/worldUtil";
+
+describe("WorldUtil cache management", () => {
+  test("empty", () => {
+    const count = countCategoryValues2D("a", "b", []);
+    expect(count).toMatchObject(new Map());
+  });
+
+  test("simple couts", () => {
+    const rows = [{ a: 0, b: false }, { a: 0, b: true }, { a: 1, b: false }];
+    const count = countCategoryValues2D("a", "b", rows);
+    expect(count).toMatchObject(
+      new Map([
+        [0, new Map([[true, 1], [false, 1]])],
+        [1, new Map([[false, 1]])]
+      ])
+    );
+  });
+
+  test("memo cache clear", () => {
+    clearCaches();
+    const row1 = [];
+    const row2 = [{ a: 0, b: false }, { a: 0, b: true }, { a: 1, b: false }];
+    const count1 = countCategoryValues2D("a", "b", row1);
+    const count2 = countCategoryValues2D("a", "b", row1);
+    const count3 = countCategoryValues2D("a", "b", []);
+    const count4 = countCategoryValues2D("a", "b", row2);
+
+    clearCaches();
+    const count10 = countCategoryValues2D("a", "b", row1);
+    const count11 = countCategoryValues2D("a", "b", row2);
+
+    expect(count1).toEqual(count2);
+    expect(count1).toEqual(count3);
+    expect(count1).toEqual(count10);
+    expect(count1).not.toBe(count3);
+    expect(count1).not.toBe(count10);
+
+    expect(count4).toEqual(count11);
+    expect(count4).not.toBe(count11);
+  });
+});
@@ -102,22 +102,20 @@ const someData = [
 ];

 function groupReduce(data, valueMap, valueReduce, valueInit) {
-  return _
-    .reduce(
-      data,
-      (acc, value) => {
-        const k = valueMap(value);
-        let r = _.find(acc, o => o.key === k);
-        if (!r) {
-          r = { key: k, value: valueInit() };
-          acc.push(r);
-        }
-        r.value = valueReduce(r.value, value);
-        return acc;
-      },
-      []
-    )
-    .sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
+  return _.reduce(
+    data,
+    (acc, value) => {
+      const k = valueMap(value);
+      let r = _.find(acc, o => o.key === k);
+      if (!r) {
+        r = { key: k, value: valueInit() };
+        acc.push(r);
+      }
+      r.value = valueReduce(r.value, value);
+      return acc;
+    },
+    []
+  ).sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
 }

 function groupCount(data, map) {
@@ -139,7 +137,11 @@ describe("typedCrossfilter", () => {
    expect(payments.size()).toEqual(someData.length);
    expect(payments.all()).toEqual(someData);

-    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
    expect(quantity).toBeDefined();
    expect(quantity.id()).toBeDefined();

@@ -150,10 +152,22 @@ describe("typedCrossfilter", () => {

  test("filterAll and filterNone", () => {
    expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    expect(quantity).toBeDefined();
    expect(tip).toBeDefined();
@@ -198,10 +212,22 @@ describe("typedCrossfilter", () => {

  test("filterExact", () => {
    expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    quantity.filterExact(1);
    expect(payments.countFiltered()).toEqual(
@@ -222,10 +248,22 @@ describe("typedCrossfilter", () => {

  test("filterRange", () => {
    expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    tip.filterRange([0, 91]);
    expect(payments.allFiltered()).toEqual(
@@ -251,10 +289,22 @@ describe("typedCrossfilter", () => {

  test("filterEnum", () => {
    expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    type.filterEnum(["tab", "cash"]);
    expect(payments.allFiltered()).toEqual(
@@ -274,15 +324,31 @@ describe("typedCrossfilter", () => {

  test("more than 32 dimensions", () => {
    expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    // Create a bunch of fake dimensions to ensure we can handle > 32
    let dimMap = {};
    for (let i = 0; i < 65; i++) {
-      dimMap[i] = payments.dimension(r => Math.random(), Float32Array);
+      dimMap[i] = payments.dimension(
+        crossfilter.ScalarDimension,
+        r => Math.random(),
+        Float32Array
+      );
      expect(dimMap[i]).toBeDefined();
      expect(dimMap[i].id()).toBeDefined();
    }
@@ -304,10 +370,22 @@ describe("typedCrossfilter", () => {
  test("group, default mapping, default reducer, no filter", () => {
    expect(payments).toBeDefined();

-    var quantity = payments.dimension(r => r.quantity, Int32Array);
-    var tip = payments.dimension(r => r.tip, Int32Array);
-    var type = payments.dimension(r => r.type, "enum");
-    var total = payments.dimension(r => r.total, Int32Array);
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Int32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Int32Array
+    );

    _.each(
      {
@@ -331,9 +409,17 @@ describe("typedCrossfilter", () => {
    // custom mapping in groups only works for scalar types.  Enums do not
    // currently implement it.

-    const tip = payments.dimension(r => r.tip, Int32Array);
-    const totalX10 = payments.dimension(r => r.total * 10, Int32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Int32Array
+    );
+    const totalX10 = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total * 10,
+      Int32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    const paymentsByTip_A = tip.group();
    const paymentsByTip_B = tip.group(r => 10 * r);
@@ -370,8 +456,12 @@ describe("typedCrossfilter", () => {
  test("group, default map, custom reducer, no filters", () => {
    expect(payments).toBeDefined();

-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    const paymentsByTotal = total.group();
    const paymentsByType = type.group();
@@ -407,9 +497,17 @@ describe("typedCrossfilter", () => {

    expect(payments).toBeDefined();

-    const tip = payments.dimension(r => r.tip, Int32Array);
-    const total = payments.dimension(r => r.total, Int32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Int32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Int32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);

    const paymentsByTip = tip.group();
    const paymentsByTotal = total.group();
@@ -34,7 +34,8 @@ module.exports = {
    "object-curly-newline": ["error", { consistent: true }],
    "react/prop-types": [0],
    "space-before-function-paren": "off",
-    "function-paren-newline": "off"
+    "function-paren-newline": "off",
+    "prefer-destructuring": ["error", { object: true, array: false }]
  },
  overrides: [
    {
@@ -10,7 +10,7 @@ const nodeModules = path.resolve("node_modules");

 const babelOptions = require("../babel/babel.prod");

-const publicPath = "/";
+const publicPath = "";

 module.exports = {
  mode: "production",
@@ -24,7 +24,7 @@
 <script type="text/javascript">
 window.CELLXGENE = {};
 window.CELLXGENE.API = {
-  prefix: "{{ prefix | safe }}",
+  prefix: window.location.href + "api/",
  version: "v0.2/"
 };
 </script>
@@ -1,10 +1,11 @@
 {
  "name": "cellxgene",
-  "version": "0.0.2",
+  "version": "0.6.1",
  "license": "MIT",
  "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
  "repository": "https://github.com/chanzuckerberg/cellxgene",
  "scripts": {
+    "backend-dev": "python3.6 -m venv cellxgene && source cellxgene/bin/activate && yes | pip uninstall cellxgene || true && pip install -e .. && cellxgene launch ",
    "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
    "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
    "clean": "rimraf build",
@@ -19,10 +20,9 @@
  "eslintConfig": {
    "extends": "./configuration/eslint/eslint.js"
  },
-  "nyc": {
-    "sourceMap": false,
-    "instrument": false
-  },
+  "eslintIgnore": [
+    "src/util/stateManager/matrix_generated.js"
+  ],
  "resolutions": {
    "eslint-scope": "3.7.1"
  },
@@ -33,9 +33,12 @@
    "canvas-fit": "^1.5.0",
    "d3": "^4.10.0",
    "d3-scale-chromatic": "^1.3.0",
+    "flatbuffers": "^1.10.2",
    "font-color-contrast": "^1.0.3",
+    "fuzzysort": "^1.1.4",
    "gl-mat4": "^1.1.4",
    "gl-matrix": "^2.7.1",
+    "is-number": "^7.0.0",
    "key-pressed": "0.0.1",
    "lodash": "^4.17.4",
    "memoize-one": "^4.0.0",
@@ -73,30 +76,28 @@
    "babel-eslint": "^10.0.1",
    "babel-jest": "^23.6.0",
    "babel-loader": "^8.0.0",
-    "babel-plugin-istanbul": "^5.1.0",
    "babel-preset-modern-browsers": "^12.0.0",
-    "chalk": "^2.4.1",
-    "connect-history-api-fallback": "^1.3.0",
+    "chalk": "^2.4.2",
+    "connect-history-api-fallback": "^1.6.0",
    "copy-webpack-plugin": "^4.6.0",
    "css-loader": "^1.0.1",
-    "eslint": "^5.8.0",
+    "eslint": "^5.13.0",
    "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^3.1.0",
-    "eslint-loader": "^2.1.1",
+    "eslint-config-prettier": "^4.0.0",
+    "eslint-loader": "^2.1.2",
    "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.14.0",
-    "eslint-plugin-jest": "^21.27.2",
-    "eslint-plugin-jsx-a11y": "^6.1.1",
-    "eslint-plugin-react": "^7.11.1",
+    "eslint-plugin-import": "^2.16.0",
+    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-jsx-a11y": "^6.2.1",
+    "eslint-plugin-react": "^7.12.4",
    "express": "^4.14.0",
    "file-loader": "^2.0.0",
    "html-webpack-inline-source-plugin": "0.0.10",
    "html-webpack-plugin": "^3.2.0",
-    "jest": "^23.5.0",
+    "jest": "^24.1.0",
    "json-loader": "^0.5.4",
    "mini-css-extract-plugin": "^0.4.1",
-    "nyc": "^13.0.1",
-    "rimraf": "^2.5.4",
+    "rimraf": "^2.6.3",
    "serve-favicon": "^2.3.0",
    "style-loader": "^0.23.1",
    "sw-precache-webpack-plugin": "^0.11.5",
@@ -109,7 +110,10 @@
    "testMatch": [
      "**/__tests__/**/?(*.)(spec|test).js?(x)"
    ],
-    "testURL": "http://localhost/"
+    "testURL": "http://localhost/",
+    "setupFiles": [
+      "./__tests__/setupMissingGlobals.js"
+    ]
  },
  "babel": {
    "env": {
@@ -5,6 +5,7 @@ import { Universe, kvCache } from "../util/stateManager";
 import {
  catchErrorsWrap,
  doJsonRequest,
+  doBinaryRequest,
  rangeEncodeIndices,
  dispatchNetworkErrorMessageToUser
 } from "../util/actionHelpers";
@@ -13,24 +14,28 @@ import {
 Bootstrap application with the initial data loading.
  * /config - application configuration
  * /schema - schema of dataframe
-  * /annotations/obs - all metadata annotation
+  * /annotations - all metadata annotation
+  * /layout - all default layout
 */
 const doInitialDataLoad = () =>
  catchErrorsWrap(async dispatch => {
    dispatch({ type: "initial data load start" });

    try {
-      const requests = _([
-        "config",
-        "schema",
-        "annotations/obs",
-        "annotations/var",
-        "layout/obs"
-      ])
+      const requestJson = _(["config", "schema"])
        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
        .map(url => doJsonRequest(url))
        .value();
-      const results = await Promise.all(requests);
+      const requestBinary = _([
+        "annotations/obs",
+        "annotations/var?annotation-name=name",
+        "layout/obs"
+      ])
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doBinaryRequest(url))
+        .value();
+
+      const results = await Promise.all(_.concat(requestJson, requestBinary));

      /* set config defaults */
      const config = { ...globals.configDefaults, ...results[0].config };
@@ -87,6 +92,38 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  /* helper for this function only */
+  const fetchData = async geneNames => {
+    const res = await fetch(
+      `${globals.API.prefix}${globals.API.version}data/var`,
+      {
+        method: "PUT",
+        body: JSON.stringify({
+          filter: {
+            var: {
+              annotation_value: [{ name: "name", values: geneNames }]
+            }
+          }
+        }),
+        headers: new Headers({
+          accept: "application/octet-stream",
+          "Content-Type": "application/json"
+        })
+      }
+    );
+
+    if (
+      !res.ok ||
+      res.headers.get("Content-Type") !== "application/octet-stream"
+    ) {
+      // WILL throw
+      return dispatchExpressionErrors(dispatch, res);
+    }
+
+    const data = await res.arrayBuffer();
+    return Universe.convertDataFBStoObject(universe, data);
+  };
+
  const state = getState();
  const { universe } = state.controls;
  /* preload data already in cache */
@@ -108,35 +145,10 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
  /* Fetch data for any genes not in cache */
  if (genesToFetch.length) {
    try {
-      // XXX: TODO - this could be using /data/var rather than /data/obs,
-      // as that would simplify the transformation in convertExpressionRESTv02ToObject
-      const res = await fetch(
-        `${globals.API.prefix}${globals.API.version}data/obs`,
-        {
-          method: "PUT",
-          body: JSON.stringify({
-            filter: {
-              var: {
-                annotation_value: [{ name: "name", values: genesToFetch }]
-              }
-            }
-          }),
-          headers: new Headers({
-            accept: "application/json",
-            "Content-Type": "application/json"
-          })
-        }
-      );
-
-      if (!res.ok || res.headers.get("Content-Type") !== "application/json") {
-        // WILL throw
-        return dispatchExpressionErrors(dispatch, res);
-      }
-
-      const data = await res.json();
+      const newExpressionData = await fetchData(genesToFetch);
      expressionData = {
        ...expressionData,
-        ...Universe.convertExpressionRESTv02ToObject(universe, data)
+        ...newExpressionData
      };
    } catch (error) {
      dispatch({ type: "expression load error", error });
@@ -13,6 +13,7 @@ import memoize from "memoize-one";
 import { kvCache } from "../../util/stateManager";
 import * as globals from "../../globals";
 import actions from "../../actions";
+import finiteExtent from "../../util/finiteExtent";

@connect(state => ({
  world: state.controls.world,
@@ -26,7 +27,7 @@ import actions from "../../actions";
  obsAnnotations: _.get(state.controls.world, "obsAnnotations", null)
 }))
 class HistogramBrush extends React.Component {
-  calcHistogramCache = memoize((obsAnnotations, field, ranges) => {
+  calcHistogramCache = memoize((obsAnnotations, field, rangeMin, rangeMax) => {
    const { world } = this.props;
    const histogramCache = {};

@@ -34,13 +35,13 @@ class HistogramBrush extends React.Component {
      .scaleLinear()
      .range([this.height - this.marginBottom, 0]);

-    if (obsAnnotations[0][field]) {
+    if (obsAnnotations[0][field] !== undefined) {
      // recalculate expensive stuff
      const allValuesForContinuousFieldAsArray = _.map(obsAnnotations, field);

      histogramCache.x = d3
        .scaleLinear()
-        .domain([ranges.min, ranges.max])
+        .domain([rangeMin, rangeMax])
        .range([0, this.width]);

      histogramCache.bins = d3
@@ -56,7 +57,7 @@ class HistogramBrush extends React.Component {
      histogramCache.x = d3
        .scaleLinear()
        .domain(
-          d3.extent(varValues)
+          finiteExtent(varValues)
        ) /* replace this if we have ranges for genes back from server like we do for annotations on cells */
        .range([0, this.width]);

@@ -130,7 +131,8 @@ class HistogramBrush extends React.Component {
    const histogramCache = this.calcHistogramCache(
      obsAnnotations,
      field,
-      ranges
+      ranges.min,
+      ranges.max
    );

    const { x, y, bins, numValues } = histogramCache;
@@ -159,7 +161,13 @@ class HistogramBrush extends React.Component {
  }

  removeHistogram() {
-    const { dispatch, field, colorAccessor } = this.props;
+    const {
+      dispatch,
+      field,
+      colorAccessor,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor
+    } = this.props;
    dispatch({
      type: "clear user defined gene",
      data: field
@@ -169,6 +177,18 @@ class HistogramBrush extends React.Component {
        type: "reset colorscale"
      });
    }
+    if (field === scatterplotXXaccessor) {
+      dispatch({
+        type: "set scatterplot x",
+        data: null
+      });
+    }
+    if (field === scatterplotYYaccessor) {
+      dispatch({
+        type: "set scatterplot y",
+        data: null
+      });
+    }
  }

  handleSetGeneAsScatterplotX() {
@@ -265,7 +285,7 @@ class HistogramBrush extends React.Component {
        }}
      >
        <div style={{ display: "flex", justifyContent: "flex-end" }}>
-          {isDiffExp ? (
+          {isDiffExp || isUserDefined ? (
            <span>
              <span
                style={{ marginRight: 7 }}
@@ -5,34 +5,13 @@ import { connect } from "react-redux";
 import * as globals from "../../globals";
 import Category from "./category";

-/* Cap the max number of displayed categories */
-const truncateCategories = options => {
-  const numOptions = _.size(options);
-  if (numOptions <= globals.maxCategoricalOptionsToDisplay) {
-    return options;
-  }
-  return _(options)
-    .map((v, k) => ({ name: k, val: v }))
-    .sortBy("val")
-    .slice(numOptions - globals.maxCategoricalOptionsToDisplay)
-    .transform((r, v) => {
-      r[v.name] = v.val;
-    }, {})
-    .value();
-};
-
@connect(state => ({
-  ranges: _.get(state.controls.world, "summary.obs", null),
-  categorySelectionLimit: _.get(
-    state.config,
-    "parameters.max-category-items",
-    globals.configDefaults.parameters["max-category-items"]
-  )
+  categoricalSelectionState: state.controls.categoricalSelectionState
 }))
 class Categories extends React.Component {
  render() {
-    const { ranges, categorySelectionLimit } = this.props;
-    if (!ranges) return null;
+    const { categoricalSelectionState } = this.props;
+    if (!categoricalSelectionState) return null;

    return (
      <div
@@ -47,27 +26,9 @@ class Categories extends React.Component {
        >
          Categorical Metadata
        </p>
-        {_.map(ranges, (value, key) => {
-          const isColorField = key.includes("color") || key.includes("Color");
-          const isSelectableCategory =
-            value.options &&
-            !isColorField &&
-            key !== "name" &&
-            value.numOptions < categorySelectionLimit;
-
-          if (isSelectableCategory) {
-            const categoryOptions = truncateCategories(value.options);
-            return (
-              <Category
-                key={key}
-                metadataField={key}
-                values={categoryOptions}
-                isTruncated={categoryOptions !== value.options}
-              />
-            );
-          }
-          return undefined;
-        })}
+        {_.map(categoricalSelectionState, (catState, catName) => (
+          <Category key={catName} metadataField={catName} />
+        ))}
      </div>
    );
  }
@@ -2,29 +2,15 @@ import React from "react";
 import _ from "lodash";
 import { connect } from "react-redux";
 import { FaChevronRight, FaChevronDown } from "react-icons/fa";
-import memoize from "memoize-one";
-import { Button, Tooltip, Position } from "@blueprintjs/core";
+import { Button, Tooltip } from "@blueprintjs/core";

 import * as globals from "../../globals";
 import Value from "./value";
-import alphabeticallySortedValues from "./util";
-
-const countCategories = (values, optsAsBools) =>
-  _.reduce(
-    values,
-    (r, v, k) => {
-      r.total += 1;
-      if (optsAsBools[k]) {
-        r.on += 1;
-      }
-      return r;
-    },
-    { total: 0, on: 0 }
-  );
+import sortedCategoryValues from "./util";

@connect(state => ({
  colorAccessor: state.controls.colorAccessor,
-  categoricalAsBooleansMap: state.controls.categoricalAsBooleansMap
+  categoricalSelectionState: state.controls.categoricalSelectionState
 }))
 class Category extends React.Component {
  constructor(props) {
@@ -33,24 +19,28 @@ class Category extends React.Component {
      isChecked: true,
      isExpanded: false
    };
-    this.countCategories = memoize((values, optsAsBools) =>
-      countCategories(values, optsAsBools)
-    );
  }

  componentDidUpdate() {
-    const { categoricalAsBooleansMap, metadataField, values } = this.props;
-    const categoryCount = this.countCategories(
-      values,
-      categoricalAsBooleansMap[metadataField]
-    );
-    if (categoryCount.on === categoryCount.total) {
+    const { categoricalSelectionState, metadataField } = this.props;
+    const cat = categoricalSelectionState[metadataField];
+    const categoryCount = {
+      // total number of categories in this dimension
+      totalCatCount: cat.numCategories,
+      // number of selected options in this category
+      selectedCatCount: _.reduce(
+        cat.categorySelected,
+        (res, cond) => (cond ? res + 1 : res),
+        0
+      )
+    };
+    if (categoryCount.selectedCatCount === categoryCount.totalCatCount) {
      /* everything is on, so not indeterminate */
      this.checkbox.indeterminate = false;
-    } else if (categoryCount.on === 0) {
+    } else if (categoryCount.selectedCatCount === 0) {
      /* nothing is on, so no */
      this.checkbox.indeterminate = false;
-    } else if (categoryCount.on < categoryCount.total) {
+    } else if (categoryCount.selectedCatCount < categoryCount.totalCatCount) {
      /* to be explicit... */
      this.checkbox.indeterminate = true;
    }
@@ -74,11 +64,10 @@ class Category extends React.Component {
  }

  toggleNone() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField } = this.props;
    dispatch({
      type: "categorical metadata filter none of these",
-      metadataField,
-      value
+      metadataField
    });
    this.setState({ isChecked: false });
  }
@@ -94,13 +83,16 @@ class Category extends React.Component {
  }

  renderCategoryItems() {
-    const { values, metadataField } = this.props;
-    return _.map(alphabeticallySortedValues(values), (v, i) => (
+    const { categoricalSelectionState, metadataField } = this.props;
+
+    const cat = categoricalSelectionState[metadataField];
+    const optTuples = sortedCategoryValues([...cat.categoryIndices]);
+    return _.map(optTuples, (tuple, i) => (
      <Value
-        key={v}
+        optTuples={optTuples}
+        key={tuple[1]}
        metadataField={metadataField}
-        count={values[v]}
-        value={v}
+        categoryIndex={tuple[1]}
        i={i}
      />
    ));
@@ -108,12 +100,15 @@ class Category extends React.Component {

  render() {
    const { isExpanded, isChecked } = this.state;
-    const { metadataField, colorAccessor, isTruncated } = this.props;
+    const {
+      metadataField,
+      colorAccessor,
+      categoricalSelectionState
+    } = this.props;
+    const { isTruncated } = categoricalSelectionState[metadataField];
    return (
      <div
        style={{
-          // display: "flex",
-          // alignItems: "baseline",
          maxWidth: globals.maxControlsWidth
        }}
      >
@@ -0,0 +1,73 @@
+// jshint esversion: 6
+import React from "react";
+import _ from "lodash";
+import { connect } from "react-redux";
+import * as d3 from "d3";
+
+@connect()
+class Occupancy extends React.Component {
+  render() {
+    const {
+      occupancy,
+      colorScale,
+      categoricalSelectionState,
+      colorAccessor,
+      schema
+    } = this.props;
+    const width = 100;
+    const height = 11;
+
+    const categories = _.filter(schema.annotations.obs, {
+      name: colorAccessor
+    })[0].categories;
+
+    const x = d3
+      .scaleLinear()
+      /* get all the keys d[1] as an array, then find the sum */
+      .domain([0, d3.sum(Array.from(occupancy, d => d[1]))])
+      .range([0, width]);
+
+    let currentOffset = 0;
+
+    const stacks = categoricalSelectionState[colorAccessor].categoryValues.map(
+      d => {
+        const o = occupancy.get(d);
+
+        const scaledValue = x(o);
+
+        const stackItem = {
+          key: d,
+          value: o || 0,
+          rectWidth: o ? scaledValue : 0,
+          offset: currentOffset,
+          fill: o ? colorScale(categories.indexOf(d)) : "rgb(255,255,255)"
+        };
+        currentOffset += o ? scaledValue : 0;
+        return stackItem;
+      }
+    );
+
+    return (
+      <svg
+        style={{
+          marginRight: 5,
+          width,
+          height
+        }}
+      >
+        {stacks.map(d => (
+          <rect
+            key={d.key}
+            width={d.rectWidth}
+            height={height}
+            x={d.offset}
+            title={d.metadataField}
+            fill={d.fill}
+          />
+        ))}
+      </svg>
+    );
+  }
+}
+
+export default Occupancy;
@@ -1,7 +1,35 @@
 // jshint esversion: 6
-export default values =>
-  Object.keys(values).sort((a, b) => {
-    const textA = a.toUpperCase();
-    const textB = b.toUpperCase();
+
+// values is [ [optVal, optIdx], ...]
+// index is range array
+// return sorted index
+
+import isNumber from "is-number";
+import _ from "lodash";
+
+const sortedCategoryValues = values => {
+  /* this sort could be memoized for perf */
+
+  const strings = [];
+  const ints = [];
+
+  _.forEach(values, v => {
+    if (isNumber(v[0])) {
+      ints.push(v);
+    } else {
+      strings.push(v);
+    }
+  });
+
+  strings.sort((a, b) => {
+    const textA = String(a[0]).toUpperCase();
+    const textB = String(b[0]).toUpperCase();
    return textA < textB ? -1 : textA > textB ? 1 : 0;
  });
+
+  ints.sort((a, b) => +a[0] - +b[0]);
+
+  return ints.concat(strings);
+};
+
+export default sortedCategoryValues;
@@ -1,47 +1,77 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
+import _ from "lodash";
+import Occupancy from "./occupancy";
+import { countCategoryValues2D } from "../../util/stateManager/worldUtil";
+import * as globals from "../../globals";

@connect(state => ({
-  categoricalAsBooleansMap: state.controls.categoricalAsBooleansMap,
+  categoricalSelectionState: state.controls.categoricalSelectionState,
  colorScale: state.controls.colorScale,
-  colorAccessor: state.controls.colorAccessor
+  colorAccessor: state.controls.colorAccessor,
+  schema: _.get(state.controls.world, "schema", null),
+  world: state.controls.world
 }))
 class CategoryValue extends React.Component {
  toggleOff() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
    dispatch({
      type: "categorical metadata filter deselect",
      metadataField,
-      value
+      categoryIndex
    });
  }

  toggleOn() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
    dispatch({
      type: "categorical metadata filter select",
      metadataField,
-      value
+      categoryIndex
    });
  }

  render() {
    const {
-      categoricalAsBooleansMap,
+      categoricalSelectionState,
      metadataField,
-      count,
-      value,
+      categoryIndex,
      colorAccessor,
      colorScale,
-      i
+      i,
+      schema,
+      world
    } = this.props;

-    if (!categoricalAsBooleansMap) return null;
+    if (!categoricalSelectionState) return null;
+
+    const category = categoricalSelectionState[metadataField];
+    const selected = category.categorySelected[categoryIndex];
+    const count = category.categoryCounts[categoryIndex];
+    const value = category.categoryValues[categoryIndex];
+    const displayString = String(
+      category.categoryValues[categoryIndex]
+    ).valueOf();

-    const selected = categoricalAsBooleansMap[metadataField][value];
    /* this is the color scale, so add swatches below */
-    const c = metadataField === colorAccessor;
+    const isColorBy = metadataField === colorAccessor;
+    let categories = null;
+    let occupancy = null;
+
+    if (isColorBy && schema) {
+      categories = _.filter(schema.annotations.obs, {
+        name: colorAccessor
+      })[0].categories;
+    }
+
+    if (colorAccessor && !isColorBy) {
+      occupancy = countCategoryValues2D(
+        metadataField,
+        colorAccessor,
+        world.obsAnnotations
+      );
+    }

    return (
      <div
@@ -56,7 +86,10 @@ class CategoryValue extends React.Component {
          style={{
            margin: 0,
            padding: 0,
-            userSelect: "none"
+            userSelect: "none",
+            width: globals.leftSidebarWidth - 130,
+            display: "flex",
+            justifyContent: "space-between"
          }}
        >
          <label className="bp3-control bp3-checkbox">
@@ -68,8 +101,20 @@ class CategoryValue extends React.Component {
              type="checkbox"
            />
            <span className="bp3-control-indicator" />
-            {value}
+            {displayString}
          </label>
+          <span style={{ flexShrink: 0 }}>
+            {colorAccessor &&
+            !isColorBy &&
+            categoricalSelectionState[colorAccessor] ? (
+              <Occupancy
+                occupancy={occupancy.get(
+                  category.categoryValues[categoryIndex]
+                )}
+                {...this.props}
+              />
+            ) : null}
+          </span>
        </div>
        <span>
          <span>{count}</span>
@@ -78,7 +123,10 @@ class CategoryValue extends React.Component {
              marginLeft: 5,
              width: 11,
              height: 11,
-              backgroundColor: c ? colorScale(value) : "inherit"
+              backgroundColor:
+                isColorBy && categories
+                  ? colorScale(categories.indexOf(value))
+                  : "inherit"
            }}
          />
        </span>
@@ -2,7 +2,7 @@
 import React from "react";
 import { connect } from "react-redux";
 import * as d3 from "d3";
-import { interpolateViridis } from "d3-scale-chromatic";
+import { interpolateViridis, interpolateCool } from "d3-scale-chromatic";

 // create continuous color legend
 // http://bl.ocks.org/syntagmatic/e8ccca52559796be775553b467593a9f
@@ -121,12 +121,12 @@ class ContinuousLegend extends React.Component {
        .remove();
    }

-    if (colorAccessor && colorScale) {
+    if (colorAccessor && colorScale && colorScale.range) {
      /* fragile! continuous range is 0 to 1, not [#fa4b2c, ...], make this a flag? */
      if (colorScale.range()[0][0] !== "#") {
        continuous(
          "#continuous_legend",
-          d3.scaleSequential(interpolateViridis).domain(colorScale.domain()),
+          d3.scaleSequential(interpolateCool).domain(colorScale.domain()),
          colorAccessor
        );
      }
@@ -13,6 +13,13 @@ A "user" error - eg, bad input
 export const postUserErrorToast = message =>
  ErrorToastTopCenter.show({ message, intent: Intent.WARNING });

+/*
+A toast the user must dismiss manually, because they need to act on its information,
+ie., 8 bulk add genes out of 40 were bad. Manually see which ones and fix.
+*/
+export const keepAroundErrorToast = message =>
+  ErrorToastTopCenter.show({ message, timeout: 0, intent: Intent.WARNING });
+
 /*
 a hard network error
 */
@@ -1,7 +1,7 @@
 // jshint esversion: 6
 import React from "react";
 import _ from "lodash";
-import { Button, Tooltip } from "@blueprintjs/core";
+import { AnchorButton, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";

@connect()
@@ -33,9 +33,10 @@ class CellSetButton extends React.Component {
        content="Save current selection for differential expression computation"
        position="top"
      >
-        <Button
+        <AnchorButton
          style={{ marginRight: 10 }}
          type="button"
+          disabled={differential.diffExp}
          onClick={this.set.bind(this)}
        >
          {eitherCellSetOneOrTwo}
@@ -43,7 +44,7 @@ class CellSetButton extends React.Component {
          {differential[cellListName]
            ? `${differential[cellListName].length} cells`
            : "0 cells"}
-        </Button>
+        </AnchorButton>
      </Tooltip>
    );
  }
@@ -68,6 +68,7 @@ class Expression extends React.Component {
              style={{ marginTop: 10 }}
              disabled={!haveBothCellSets}
              intent="primary"
+              loading={differential.loading}
              fill
              type="button"
              onClick={this.computeDiffExp.bind(this)}
@@ -3,14 +3,60 @@

 import React from "react";
 import _ from "lodash";
-import * as d3 from "d3";
+import fuzzysort from "fuzzysort";
+
 import { connect } from "react-redux";
-import { Button, Tooltip } from "@blueprintjs/core";
+import {
+  MenuItem,
+  Button,
+  FormGroup,
+  InputGroup,
+  ControlGroup
+} from "@blueprintjs/core";
+import { Suggest } from "@blueprintjs/select";
 import HistogramBrush from "../brushableHistogram";
 import * as globals from "../../globals";
 import actions from "../../actions";
-import { postUserErrorToast } from "../framework/toasters";
+import {
+  postUserErrorToast,
+  keepAroundErrorToast
+} from "../framework/toasters";
 import ExpressionButtons from "./expressionButtons";
+import finiteExtent from "../../util/finiteExtent";
+
+const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
+  if (!modifiers.matchesPredicate) {
+    return null;
+  }
+  /* the fuzzysort wraps the object with other properties, like a score */
+  const gene = fuzzySortResult.obj;
+  const text = gene.name;
+
+  return (
+    <MenuItem
+      active={modifiers.active}
+      disabled={modifiers.disabled}
+      // Use of annotations in this way is incorrect and dataset specific.
+      // See https://github.com/chanzuckerberg/cellxgene/issues/483
+      // label={gene.n_counts}
+      key={gene.name}
+      onClick={g => {
+        /* this fires when user clicks a menu item */
+        handleClick(g);
+      }}
+      text={text}
+    />
+  );
+};
+
+const filterGenes = (query, genes) => {
+  /* fires on load, once, and then for each character typed into the input */
+  return fuzzysort.go(query, genes, {
+    key: "name",
+    limit: 5,
+    threshold: -10000 // don't return bad results
+  });
+};

@connect(state => {
  const metadata = _.get(state.controls.world, "obsAnnotations", null);
@@ -22,6 +68,7 @@ import ExpressionButtons from "./expressionButtons";
    metadata,
    initializeRanges,
    userDefinedGenes: state.controls.userDefinedGenes,
+    userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
    world: state.controls.world,
    colorAccessor: state.controls.colorAccessor,
    allGeneNames: state.controls.allGeneNames,
@@ -32,20 +79,14 @@ class GeneExpression extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
-      gene: ""
+      bulkAdd: "",
+      tab: "autosuggest"
    };
  }

-  keyPress(e) {
-    if (e.keyCode === 13) {
-      this.handleClick();
-    }
-  }
-
-  handleClick() {
+  handleClick(g) {
    const { world, dispatch, userDefinedGenes } = this.props;
-    const { gene } = this.state;
-
+    const gene = g.target;
    if (userDefinedGenes.indexOf(gene) !== -1) {
      postUserErrorToast("That gene already exists");
    } else if (userDefinedGenes.length > 15) {
@@ -60,13 +101,49 @@ class GeneExpression extends React.Component {
        type: "user defined gene",
        data: gene
      });
-      this.setState({ gene: "" });
    }
  }

+  handleBulkAddClick() {
+    const { world, dispatch, userDefinedGenes } = this.props;
+    const { bulkAdd } = this.state;
+
+    /*
+      test:
+      Apod,,, Cd74,,    ,,,    Foo,    Bar-2,,
+    */
+    if (bulkAdd !== "") {
+      const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");
+
+      genes.forEach(gene => {
+        if (userDefinedGenes.indexOf(gene) !== -1) {
+          keepAroundErrorToast("That gene already exists");
+        } else if (!_.find(world.varAnnotations, { name: gene })) {
+          keepAroundErrorToast(
+            `${gene} doesn't appear to be a valid gene name.`
+          );
+        } else {
+          dispatch(actions.requestUserDefinedGene(gene));
+          dispatch({
+            type: "user defined gene",
+            data: gene
+          });
+        }
+      });
+    }
+
+    this.setState({ bulkAdd: "" });
+  }
+
  render() {
-    const { world, userDefinedGenes, differential } = this.props;
-    const { gene } = this.state;
+    const {
+      world,
+      userDefinedGenes,
+      userDefinedGenesLoading,
+      differential
+    } = this.props;
+
+    const { tab, bulkAdd } = this.state;

    return (
      <div>
@@ -84,31 +161,105 @@ class GeneExpression extends React.Component {
            Selected Genes
          </p>
          <div
-            style={{ padding: globals.leftSidebarSectionPadding }}
-            className="bp3-control-group"
+            style={{
+              padding: globals.leftSidebarSectionPadding
+            }}
          >
-            <div className="bp3-input-group bp3-fill">
-              <input
-                onKeyDown={this.keyPress.bind(this)}
-                onChange={e => {
-                  this.setState({ gene: e.target.value });
-                }}
-                value={gene}
-                type="text"
-                className="bp3-input"
-                placeholder="Enter a gene name"
-                style={{ paddingRight: 94 }}
-              />
-            </div>
-            <Tooltip
-              content="Add a gene to see its expression levels"
-              position="bottom"
+            <Button
+              active={tab === "autosuggest"}
+              style={{ marginRight: 5 }}
+              minimal
+              small
+              onClick={() => {
+                this.setState({ tab: "autosuggest" });
+              }}
            >
-              <Button intent="primary" onClick={this.handleClick.bind(this)}>
+              Autosuggest
+            </Button>
+            <Button
+              active={tab === "bulkadd"}
+              minimal
+              small
+              onClick={() => {
+                this.setState({ tab: "bulkadd" });
+              }}
+            >
+              Bulk add genes
+            </Button>
+          </div>
+
+          {tab === "autosuggest" ? (
+            <ControlGroup
+              style={{
+                paddingLeft: globals.leftSidebarSectionPadding,
+                paddingBottom: globals.leftSidebarSectionPadding
+              }}
+            >
+              <Suggest
+                closeOnSelect
+                openOnKeyDown
+                resetOnSelect
+                itemDisabled={
+                  userDefinedGenesLoading ? () => true : () => false
+                }
+                noResults={<MenuItem disabled text="No matching genes." />}
+                onItemSelect={g => {
+                  /* this happens on 'enter' */
+                  this.handleClick(g);
+                }}
+                inputValueRenderer={g => {
+                  return "";
+                }}
+                itemListPredicate={filterGenes}
+                itemRenderer={renderGene.bind(this)}
+                items={
+                  world && world.varAnnotations
+                    ? world.varAnnotations
+                    : [{ name: "No genes" }]
+                }
+                popoverProps={{ minimal: true }}
+              />
+              <Button
+                className="bp3-button bp3-intent-primary"
+                loading={userDefinedGenesLoading}
+              >
                Add
              </Button>
-            </Tooltip>
-          </div>
+            </ControlGroup>
+          ) : null}
+          {tab === "bulkadd" ? (
+            <div style={{ paddingLeft: globals.leftSidebarSectionPadding }}>
+              <form
+                onSubmit={e => {
+                  e.preventDefault();
+                  this.handleBulkAddClick();
+                }}
+              >
+                <FormGroup
+                  helperText="Add a list of genes (comma delimited)"
+                  labelFor="text-input-bulk-add"
+                >
+                  <ControlGroup>
+                    <InputGroup
+                      onChange={e => {
+                        this.setState({ bulkAdd: e.target.value });
+                      }}
+                      id="text-input-bulk-add"
+                      placeholder="Apod, Cd74, ..."
+                      value={bulkAdd}
+                    />
+                    <Button
+                      intent="primary"
+                      onClick={this.handleBulkAddClick.bind(this)}
+                      loading={userDefinedGenesLoading}
+                    >
+                      Add
+                    </Button>
+                  </ControlGroup>
+                </FormGroup>
+              </form>
+            </div>
+          ) : null}
          {world && userDefinedGenes.length > 0
            ? _.map(userDefinedGenes, (geneName, index) => {
                const values = world.varDataCache[geneName];
@@ -120,7 +271,7 @@ class GeneExpression extends React.Component {
                    key={geneName}
                    field={geneName}
                    zebra={index % 2 === 0}
-                    ranges={d3.extent(values)}
+                    ranges={finiteExtent(values)}
                    isUserDefined
                  />
                );
@@ -150,7 +301,7 @@ class GeneExpression extends React.Component {
                    key={name}
                    field={name}
                    zebra={index % 2 === 0}
-                    ranges={d3.extent(values)}
+                    ranges={finiteExtent(values)}
                    isDiffExp
                    logFoldChange={value[1]}
                    pval={value[2]}
@@ -22,7 +22,8 @@ import scaleLinear from "../../util/scaleLinear";
  responsive: state.responsive,
  colorRGB: _.get(state.controls, "colorRGB", null),
  opacityForDeselectedCells: state.controls.opacityForDeselectedCells,
-  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null),
+  resettingInterface: state.controls.resettingInterface
 }))
 class Graph extends React.Component {
  constructor(props) {
@@ -39,7 +40,7 @@ class Graph extends React.Component {
    this.state = {
      svg: null,
      brush: null,
-      mode: "brush"
+      mode: "lasso"
    };
  }

@@ -201,7 +202,9 @@ class Graph extends React.Component {
        this.handleBrushSelectAction.bind(this),
        this.handleBrushDeselectAction.bind(this),
        responsive,
-        this.graphPaddingRight
+        this.graphPaddingRight,
+        this.handleLassoStart.bind(this),
+        this.handleLassoEnd.bind(this)
      );
      this.setState({ svg: newSvg, brush });
    }
@@ -250,54 +253,54 @@ class Graph extends React.Component {
    });
  }

+  invertPoint(pin) {
+    const { responsive } = this.props;
+    const { regl, camera, offset } = this.state;
+
+    const gl = regl._gl;
+
+    // get aspect ratio
+    const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
+
+    // compute inverse view matrix
+    const inverse = mat4.invert([], camera.view());
+
+    // transform screen coordinates -> cell coordinates
+    const x = (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
+    const y = 2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
+    const pout = [
+      x * inverse[14] * aspect + inverse[12],
+      y * inverse[14] + inverse[13]
+    ];
+    return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
+  }
+
  handleBrushSelectAction() {
    /*
-    This conditional handles procedural brush deselect. Brush emits
-    an event on procedural deselect because it is move: null
+      This conditional handles procedural brush deselect. Brush emits
+      an event on procedural deselect because it is move: null
    */
-
-    const { camera, offset } = this.state;
-    const { dispatch, responsive } = this.props;
-
-    if (d3.event.sourceEvent !== null) {
-      /*
-      No idea why d3 event scope works like this
-      but apparently
-      it does
-      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
-    */
-      const s = d3.event.selection;
-      const gl = this.state.regl._gl;
-      /*
+    /*
      event describing brush position:
      @-------|
      |       |
      |       |
      |-------@
    */
+    /*
+      No idea why d3 event scope works like this
+      but apparently
+      it does
+      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
+    */
+    const { dispatch } = this.props;

-      // get aspect ratio
-      const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
-
-      // compute inverse view matrix
-      const inverse = mat4.invert([], camera.view());
-
-      // transform screen coordinates -> cell coordinates
-      const invert = pin => {
-        const x =
-          (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
-        const y =
-          2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
-        const pout = [
-          x * inverse[14] * aspect + inverse[12],
-          y * inverse[14] + inverse[13]
-        ];
-        return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
-      };
+    if (d3.event.sourceEvent !== null) {
+      const s = d3.event.selection;

      const brushCoords = {
-        northwest: invert([s[0][0], s[0][1]]),
-        southeast: invert([s[1][0], s[1][1]])
+        northwest: this.invertPoint([s[0][0], s[0][1]]),
+        southeast: this.invertPoint([s[1][0], s[1][1]])
      };

      dispatch({
@@ -329,6 +332,25 @@ class Graph extends React.Component {
    }
  }

+  handleLassoStart() {
+    const { dispatch } = this.props;
+    // reset selected points when starting a new polygon
+    // making it easier for the user to make the next selection
+    dispatch({
+      type: "lasso started"
+    });
+  }
+
+  // when a lasso is completed, filter to the points within the lasso polygon
+  handleLassoEnd(polygon) {
+    const { dispatch } = this.props;
+
+    dispatch({
+      type: "lasso selection",
+      polygon: polygon.map(xy => this.invertPoint(xy)) // transform the polygon
+    });
+  }
+
  handleOpacityRangeChange(e) {
    const { dispatch } = this.props;
    dispatch({
@@ -339,11 +361,20 @@ class Graph extends React.Component {

  resetInterface() {
    const { dispatch } = this.props;
+    dispatch({
+      type: "interface reset started"
+    });
    dispatch(actions.resetInterface());
  }

  render() {
-    const { dispatch, responsive, crossfilter } = this.props;
+    const {
+      dispatch,
+      responsive,
+      crossfilter,
+      resettingInterface
+    } = this.props;
+
    const { mode } = this.state;
    return (
      <div id="graphWrapper">
@@ -392,6 +423,7 @@ class Graph extends React.Component {
                  /* world && universe ? worldEqUniverse(world, universe) : false */
                }
                type="button"
+                loading={resettingInterface}
                intent="warning"
                style={{ marginRight: 10 }}
                onClick={this.resetInterface.bind(this)}
@@ -401,13 +433,18 @@ class Graph extends React.Component {
            </Tooltip>
            <div>
              <div className="bp3-button-group">
-                <Tooltip content="Lasso cells" position="left">
+                <Tooltip content="Lasso selection" position="left">
                  <Button
-                    className="bp3-button bp3-icon-select"
                    type="button"
-                    active={mode === "brush"}
+                    className="bp3-button bp3-icon-polygon-filter"
+                    active={mode === "lasso"}
                    onClick={() => {
-                      this.setState({ mode: "brush" });
+                      this.handleBrushDeselectAction();
+                      // this.restartReglLoop();
+                      this.setState({ mode: "lasso" });
+                    }}
+                    style={{
+                      cursor: "pointer"
                    }}
                  />
                </Tooltip>
@@ -440,7 +477,7 @@ class Graph extends React.Component {
        >
          <div
            style={{
-              display: mode === "brush" ? "inherit" : "none"
+              display: mode === "lasso" ? "inherit" : "none"
            }}
            id="graphAttachPoint"
          />
@@ -0,0 +1,127 @@
+// https://bl.ocks.org/pbeshai/8008075f9ce771ee8be39e8c38907570
+
+import * as d3 from "d3";
+
+const Lasso = () => {
+  const dispatch = d3.dispatch("start", "end");
+
+  const polygonToPath = polygon =>
+    `M${polygon.map(d => d.join(",")).join("L")}`;
+
+  const distance = (pt1, pt2) =>
+    Math.sqrt((pt2[0] - pt1[0]) ** 2 + (pt2[1] - pt1[1]) ** 2);
+
+  // distance last point has to be to first point before it auto closes when mouse is released
+  const closeDistance = 75;
+
+  const lasso = svg => {
+    let lassoPolygon;
+    let lassoPath;
+    let closePath;
+
+    const handleDragStart = () => {
+      lassoPolygon = [d3.mouse(svg.node())]; // current x y of mouse within element
+
+      if (lassoPath) {
+        lassoPath.remove();
+      }
+
+      lassoPath = g
+        .append("path")
+        .attr("fill", "#0bb")
+        .attr("fill-opacity", 0.1)
+        .attr("stroke", "#0bb")
+        .attr("stroke-dasharray", "3, 3");
+
+      closePath = g
+        .append("line")
+        .attr("x2", lassoPolygon[0][0])
+        .attr("y2", lassoPolygon[0][1])
+        .attr("stroke", "#0bb")
+        .attr("stroke-dasharray", "3, 3")
+        .attr("opacity", 0);
+
+      dispatch.call("start", lasso, lassoPolygon);
+    };
+
+    const handleDrag = () => {
+      const point = d3.mouse(svg.node());
+      lassoPolygon.push(point);
+      lassoPath.attr("d", polygonToPath(lassoPolygon));
+
+      // indicate if we are within closing distance
+      if (
+        distance(lassoPolygon[0], lassoPolygon[lassoPolygon.length - 1]) <
+        closeDistance
+      ) {
+        closePath
+          .attr("x1", point[0])
+          .attr("y1", point[1])
+          .attr("opacity", 1);
+      } else {
+        closePath.attr("opacity", 0);
+      }
+    };
+
+    const handleDragEnd = () => {
+      // remove the close path
+      closePath.remove();
+      closePath = null;
+
+      // succesfully closed
+      if (
+        distance(lassoPolygon[0], lassoPolygon[lassoPolygon.length - 1]) <
+        closeDistance
+      ) {
+        lassoPath.attr("d", `${polygonToPath(lassoPolygon)}Z`);
+        dispatch.call("end", lasso, lassoPolygon);
+
+        // otherwise cancel
+      } else {
+        lassoPath.remove();
+        lassoPath = null;
+        lassoPolygon = null;
+      }
+    };
+
+    // append a <g> with a rect
+    const g = svg.append("g").attr("class", "lasso-group");
+    const bbox = svg.node().getBoundingClientRect();
+    const area = g
+      .append("rect")
+      .attr("width", bbox.width)
+      .attr("height", bbox.height)
+      .attr("fill", "tomato")
+      .attr("opacity", 0);
+
+    const drag = d3
+      .drag()
+      .on("start", handleDragStart)
+      .on("drag", handleDrag)
+      .on("end", handleDragEnd);
+
+    area.call(drag);
+
+    lasso.reset = () => {
+      if (lassoPath) {
+        lassoPath.remove();
+        lassoPath = null;
+      }
+
+      lassoPolygon = null;
+      if (closePath) {
+        closePath.remove();
+        closePath = null;
+      }
+    };
+  };
+
+  lasso.on = (type, callback) => {
+    dispatch.on(type, callback);
+    return lasso;
+  };
+
+  return lasso;
+};
+
+export default Lasso;
@@ -1,6 +1,7 @@
 // jshint esversion: 6
 import * as d3 from "d3";
 import styles from "./graph.css";
+import Lasso from "./setupLasso";

 /******************************************
 *******************************************
@@ -12,7 +13,9 @@ export default (
  handleBrushSelectAction,
  handleBrushDeselectAction,
  responsive,
-  graphPaddingRight
+  graphPaddingRight,
+  handleLassoStart,
+  handleLassoEnd
 ) => {
  const svg = d3
    .select("#graphAttachPoint")
@@ -32,9 +35,16 @@ export default (
    .attr("class", "graph_brush")
    .call(brush);

+  const lassoInstance = Lasso()
+    .on("end", handleLassoEnd)
+    .on("start", handleLassoStart);
+
+  const lasso = svg.call(lassoInstance);
+
  return {
    svg,
    brushContainer,
-    brush
+    brush,
+    lasso
  };
 };
@@ -20,6 +20,7 @@ import scaleLinear from "../../util/scaleLinear";

 import { margin, width, height } from "./util";
 import { kvCache } from "../../util/stateManager";
+import finiteExtent from "../../util/finiteExtent";

@connect(state => {
  const {
@@ -227,11 +228,11 @@ class Scatterplot extends React.Component {
  static setupScales(expressionX, expressionY) {
    const xScale = d3
      .scaleLinear()
-      .domain(d3.extent(expressionX))
+      .domain(finiteExtent(expressionX))
      .range([0, width]);
    const yScale = d3
      .scaleLinear()
-      .domain(d3.extent(expressionY))
+      .domain(finiteExtent(expressionY))
      .range([height, 0]);

    return {
@@ -1,3 +1,5 @@
+import { Colors } from "@blueprintjs/core";
+
 // jshint esversion: 6
 /* these will be either (preferably) specified or inferred */
 export const categories = [
@@ -44,8 +46,8 @@ export const configDefaults = {
 };

 /* colors */
-export const blue = "#4a90e2";
-export const hcaBlue = "#1c7cc7";
+export const blue = Colors.BLUE3;
+export const linkBlue = Colors.BLUE5;
 export const lightestGrey = "rgb(249,249,249)";
 export const lighterGrey = "rgb(245,245,245)";
 export const lightGrey = "rgb(211,211,211)";
@@ -57,6 +59,7 @@ export const brightBlue = "#4a90e2";
 export const brightGreen = "#A2D729";
 export const darkGreen = "#448C4D";

+export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";

 /* typography constants */
@@ -1,9 +1,10 @@
 // jshint esversion: 6
 import _ from "lodash";
 import * as d3 from "d3";
-import { interpolateViridis } from "d3-scale-chromatic";
+import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
 import * as globals from "../globals";
 import parseRGB from "../util/parseRGB";
+import finiteExtent from "../util/finiteExtent";

 /*
  https://medium.com/@jacobp100/you-arent-using-redux-middleware-enough-94ffe991e6
@@ -48,56 +49,89 @@ const updateCellColorsMiddleware = store => next => action => {

  const { obsAnnotations } = s.controls.world;
  let colorScale;
-  const colorsByName = new Array(obsAnnotations.length);
  const colorsByRGB = new Array(obsAnnotations.length);

  /*
  in plain language...
   (a) once the cells have loaded.
   (b) each time a user changes a color control we need to update cellsMetadata colors
-  This is available to all the draw functions as world.colorName[index] or world.colorRGB[index]
+  This is available to all the draw functions as controls.colorRGB[index]
  */

  if (action.type === "color by categorical metadata") {
-    colorScale = d3.scaleOrdinal().range(globals.ordinalColors);
+    const { categories } = _.filter(s.controls.world.schema.annotations.obs, {
+      name: action.colorAccessor
+    })[0];

-    for (let i = 0; i < obsAnnotations.length; i += 1) {
+    colorScale = d3
+      .scaleSequential(interpolateRainbow)
+      .domain([0, categories.length]);
+
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = _.transform(categories, (acc, cat, idx) => {
+      acc[cat] = parseRGB(colorScale(idx));
+    });
+
+    const key = action.colorAccessor;
+    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
      const obs = obsAnnotations[i];
-      const c = colorScale(obs[action.colorAccessor]);
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+      const cat = obs[key];
+      colorsByRGB[i] = colors[cat];
    }
  }

  if (action.type === "color by continuous metadata") {
+    const colorBins = 100;
+    const [min, max] = [0, action.rangeMaxForColorAccessor];
    colorScale = d3
-      .scaleLinear()
-      .domain([0, action.rangeMaxForColorAccessor])
-      .range([1, 0]);
+      .scaleQuantile()
+      .domain([min, max])
+      .range(_.range(colorBins - 1, -1, -1));

-    for (let i = 0; i < obsAnnotations.length; i += 1) {
-      const obs = obsAnnotations[i];
-      const c = interpolateViridis(colorScale(obs[action.colorAccessor]));
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = new Array(colorBins);
+    for (let i = 0; i < colorBins; i += 1) {
+      colors[i] = parseRGB(interpolateCool(i / colorBins));
+    }
+
+    const key = action.colorAccessor;
+    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
+    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
+      const val = obsAnnotations[i][key];
+      if (Number.isFinite(val)) {
+        const c = colorScale(val);
+        colorsByRGB[i] = colors[c];
+      } else {
+        colorsByRGB[i] = nonFiniteColor;
+      }
    }
  }

  if (action.type === "color by expression") {
    const { gene, data } = action;
    const expression = data[gene]; // Float32Array
+    const colorBins = 100;
+    const [min, max] = finiteExtent(expression);
    colorScale = d3
-      .scaleLinear()
-      .domain([_.min(expression), _.max(expression)])
-      .range([
-        1,
-        0
-      ]); /* invert viridis... probably pass this scale through to others */
+      .scaleQuantile()
+      .domain([min, max])
+      .range(_.range(colorBins - 1, -1, -1));
+
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = new Array(colorBins);
+    for (let i = 0; i < colorBins; i += 1) {
+      colors[i] = parseRGB(interpolateCool(i / colorBins));
+    }
+    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);

    for (let i = 0, len = expression.length; i < len; i += 1) {
-      const c = interpolateViridis(colorScale(expression[i]));
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+      const e = expression[i];
+      if (Number.isFinite(e)) {
+        const c = colorScale(e);
+        colorsByRGB[i] = colors[c];
+      } else {
+        colorsByRGB[i] = nonFiniteColor;
+      }
    }
  }

@@ -105,7 +139,7 @@ const updateCellColorsMiddleware = store => next => action => {
  append the result of all the filters to the action the user just triggered
  */
  const modifiedAction = Object.assign({}, action, {
-    colors: { name: colorsByName, rgb: colorsByRGB },
+    colors: { rgb: colorsByRGB },
    colorScale
  });

@@ -1,7 +1,9 @@
 // jshint esversion: 6

 import _ from "lodash";
-import { World, kvCache } from "../util/stateManager";
+import { polygonContains } from "d3";
+
+import { World, kvCache, WorldUtil } from "../util/stateManager";
 import parseRGB from "../util/parseRGB";
 import Crossfilter from "../util/typedCrossfilter";
 import * as globals from "../globals";
@@ -12,41 +14,117 @@ import {
  diffexpDimensionName,
  makeContinuousDimensionName
 } from "../util/nameCreators";
+import { fillRange } from "../util/typedCrossfilter/util";

-function createCategoricalAsBooleansMap(world) {
+/*
+Selection state for categoricals are tracked in an Object that
+has two main components for each category:
+1. mapping of option value to an index
+2. array of bool selection state by index
+Remember that option values can be ANY js type, except undefined/null.
+
+  {
+    _category_name_1: {
+      // map of option value to index
+      categoryIndices: Map([
+        catval1: index,
+        ...
+      ])
+
+      // index->selection true/false state
+      categorySelected: [ true/false, true/false, ... ]
+
+      // number of options
+      numCategories: number,
+
+      // isTruncated - true if the options for selection has
+      // been truncated (ie, was too large to implement)
+    }
+  }
+*/
+function topNCategories(summary) {
+  const counts = _.map(summary.categories, cat =>
+    summary.categoryCounts.get(cat)
+  );
+  const sortIndex = fillRange(new Array(summary.numCategories)).sort(
+    (a, b) => counts[b] - counts[a]
+  );
+  const sortedCategories = _.map(sortIndex, i => summary.categories[i]);
+  const sortedCounts = _.map(sortIndex, i => counts[i]);
+  const N = globals.maxCategoricalOptionsToDisplay;
+
+  if (sortedCategories.length < N) {
+    return [sortedCategories, sortedCounts];
+  }
+  return [sortedCategories.slice(0, N), sortedCounts.slice(0, N)];
+}
+
+function createCategoricalSelectionState(state, world) {
  const res = {};
-  _.each(world.summary.obs, (value, key) => {
-    if (value.options && key !== "name") {
-      const optionsAsBooleans = {};
-      _.each(value.options, (_value, _key) => {
-        optionsAsBooleans[_key] = true;
-      });
-      res[key] = optionsAsBooleans;
+  _.forEach(world.summary.obs, (value, key) => {
+    if (value.categories) {
+      const isColorField = key.includes("color") || key.includes("Color");
+      const isSelectableCategory =
+        !isColorField &&
+        key !== "name" &&
+        value.categories.length < state.maxCategoryItems;
+      if (isSelectableCategory) {
+        const [categoryValues, categoryCounts] = topNCategories(value);
+        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
+        const numCategories = categoryIndices.size;
+        const categorySelected = new Array(numCategories).fill(true);
+        const isTruncated = categoryValues.length < value.numCategories;
+        res[key] = {
+          categoryValues, // array: of natively typed category values
+          categoryIndices, // map: category value (native type) -> category index
+          categorySelected, // array: t/f selection state
+          numCategories, // number: of categories
+          isTruncated, // bool: true if list was truncated
+          categoryCounts // array: cardinality of each category
+        };
+      }
    }
  });
  return res;
 }

+/*
+given a categoricalSelectionState, return the list of all category values
+where selection state is true (ie, they are selected).
+*/
+function selectedValuesForCategory(categorySelectionState) {
+  const selectedValues = _([...categorySelectionState.categoryIndices])
+    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
+    .map(tuple => tuple[0])
+    .value();
+  return selectedValues;
+}
+
 const Controls = (
  state = {
    // data loading flag
    loading: false,
    error: null,

+    // configuration
+    maxCategoryItems: globals.configDefaults.parameters["max-category-items"],
+
+    // the whole big bang
    universe: null,

    // all of the data + selection state
    world: null,
-    colorName: null,
    colorRGB: null,
-    categoricalAsBooleansMap: null,
+    categoricalSelectionState: null,
    crossfilter: null,
    dimensionMap: null,
    userDefinedGenes: [],
+    userDefinedGenesLoading: false,
    diffexpGenes: [],

    colorAccessor: null,
    colorScale: null,
+    resettingInterface: false,

    opacityForDeselectedCells: 0.2,
    graphBrushSelection: null,
@@ -72,6 +150,17 @@ const Controls = (
          Initialization, World/Universe management
          and data loading.
    ******************************************************/
+    case "configuration load complete": {
+      // there are a couple of configuration items we need to retain
+      return {
+        ...state,
+        maxCategoryItems: _.get(
+          state.config,
+          "parameters.max-category-items",
+          globals.configDefaults.parameters["max-category-items"]
+        )
+      };
+    }
    case "initial data load start": {
      return { ...state, loading: true };
    }
@@ -81,11 +170,16 @@ const Controls = (
      /* first light - create world & other data-driven defaults */
      const { universe } = action;
      const world = World.createWorldFromEntireUniverse(universe);
-      const colorName = new Array(universe.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
-      const categoricalAsBooleansMap = createCategoricalAsBooleansMap(world);
+      const colorRGB = new Array(universe.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
+      const categoricalSelectionState = createCategoricalSelectionState(
+        state,
+        world
+      );
      const crossfilter = Crossfilter(world.obsAnnotations);
      const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+      WorldUtil.clearCaches();

      const worldVarDataCache = world.varDataCache;

@@ -133,12 +227,12 @@ const Controls = (
        error: null,
        universe,
        world,
-        colorName,
        colorRGB,
-        categoricalAsBooleansMap,
+        categoricalSelectionState,
        crossfilter,
        dimensionMap,
-        colorAccessor: null
+        colorAccessor: null,
+        resettingInterface: false
      };
    }
    case "set World to current selection": {
@@ -150,11 +244,16 @@ const Controls = (
        action.world,
        action.crossfilter
      );
-      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
-      const categoricalAsBooleansMap = createCategoricalAsBooleansMap(world);
+      const colorRGB = new Array(world.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
+      const categoricalSelectionState = createCategoricalSelectionState(
+        state,
+        world
+      );
      const crossfilter = Crossfilter(world.obsAnnotations);
      const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+      WorldUtil.clearCaches();

      const worldVarDataCache = world.varDataCache;
      /* var dimensions */
@@ -195,9 +294,8 @@ const Controls = (
        loading: false,
        error: null,
        world,
-        colorName,
        colorRGB,
-        categoricalAsBooleansMap,
+        categoricalSelectionState,
        crossfilter,
        dimensionMap,
        colorAccessor: null
@@ -230,6 +328,18 @@ const Controls = (
        }
      };
    }
+    case "request user defined gene started": {
+      return {
+        ...state,
+        userDefinedGenesLoading: true
+      };
+    }
+    case "request user defined gene error": {
+      return {
+        ...state,
+        userDefinedGenesLoading: false
+      };
+    }
    case "request user defined gene success": {
      const { world, crossfilter, dimensionMap, userDefinedGenes } = state;
      const worldVarDataCache = world.varDataCache;
@@ -247,7 +357,8 @@ const Controls = (
      return {
        ...state,
        dimensionMap,
-        userDefinedGenes: _userDefinedGenes
+        userDefinedGenes: _userDefinedGenes,
+        userDefinedGenesLoading: false
      };
    }
    case "request differential expression success": {
@@ -348,11 +459,11 @@ const Controls = (
    }
    case "reset colorscale": {
      const { world } = state;
-      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
+      const colorRGB = new Array(world.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
      return {
        ...state,
-        colorName,
        colorRGB,
        colorAccessor: null
      };
@@ -370,27 +481,36 @@ const Controls = (
             User Events
     *******************************/
    case "graph brush selection change": {
-      state.dimensionMap[layoutDimensionName("X")].filterRange([
-        action.brushCoords.northwest[0],
-        action.brushCoords.southeast[0]
-      ]);
-      state.dimensionMap[layoutDimensionName("Y")].filterRange([
-        action.brushCoords.southeast[1],
-        action.brushCoords.northwest[1]
-      ]);
+      state.dimensionMap[layoutDimensionName("XY")].filterWithinRect(
+        action.brushCoords.northwest,
+        action.brushCoords.southeast
+      );
      return {
        ...state,
        graphBrushSelection: action.brushCoords
      };
    }
+    case "lasso deselect":
    case "graph brush deselect": {
-      state.dimensionMap[layoutDimensionName("X")].filterAll();
-      state.dimensionMap[layoutDimensionName("Y")].filterAll();
+      state.dimensionMap[layoutDimensionName("XY")].filterAll();
      return {
        ...state,
        graphBrushSelection: null
      };
    }
+    case "lasso selection": {
+      const { polygon } = action;
+      const dXY = state.dimensionMap[layoutDimensionName("XY")];
+      if (polygon.length < 3) {
+        // single point or a line is not a polygon, and is therefore a deselect
+        dXY.filterAll();
+      } else {
+        dXY.filterWithinPolygon(polygon);
+      }
+      return {
+        ...state
+      };
+    }
    case "continuous metadata histogram brush": {
      const name = makeContinuousDimensionName(
        action.continuousNamespace,
@@ -418,87 +538,99 @@ const Controls = (
        graphRenderCounter: c
      };
    }
+    case "interface reset started": {
+      return {
+        ...state,
+        resettingInterface: true
+      };
+    }
    /*******************************
          Categorical metadata
    *******************************/
    case "categorical metadata filter select": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap,
+      const newCategorySelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].categorySelected
+      );
+      newCategorySelected[action.categoryIndex] = true;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
        [action.metadataField]: {
-          ...state.categoricalAsBooleansMap[action.metadataField],
-          [action.value]: true
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: newCategorySelected
        }
      };
-      // update the filter for the one category that changed state
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
      state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        _.filter(
-          _.map(
-            newCategoricalAsBooleansMap[action.metadataField],
-            (val, key) => (val ? key : false)
-          )
-        )
+        selectedValuesForCategory(cat)
      );
+
      return {
        ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
      };
    }
    case "categorical metadata filter deselect": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap,
+      const newCategorySelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].categorySelected
+      );
+      newCategorySelected[action.categoryIndex] = false;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
        [action.metadataField]: {
-          ...state.categoricalAsBooleansMap[action.metadataField],
-          [action.value]: false
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: newCategorySelected
        }
      };
-      // update the filter for the one category that changed state
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
      state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        _.filter(
-          _.map(
-            newCategoricalAsBooleansMap[action.metadataField],
-            (val, key) => (val ? key : false)
-          )
-        )
+        selectedValuesForCategory(cat)
      );
+
      return {
        ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
      };
    }
    case "categorical metadata filter none of these": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap
-      };
-      _.forEach(
-        newCategoricalAsBooleansMap[action.metadataField],
-        (v, k, c) => {
-          c[k] = false;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: Array.from(
+            state.categoricalSelectionState[action.metadataField]
+              .categorySelected
+          ).fill(false)
        }
-      );
+      };
      state.dimensionMap[
        obsAnnoDimensionName(action.metadataField)
      ].filterNone();
      return {
        ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
      };
    }
    case "categorical metadata filter all of these": {
-      const newCategoricalAsBooleansMap = {
-        ...state.categoricalAsBooleansMap
-      };
-      _.forEach(
-        newCategoricalAsBooleansMap[action.metadataField],
-        (v, k, c) => {
-          c[k] = true;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
+        [action.metadataField]: {
+          ...state.categoricalSelectionState[action.metadataField],
+          categorySelected: Array.from(
+            state.categoricalSelectionState[action.metadataField]
+              .categorySelected
+          ).fill(true)
        }
-      );
+      };
      state.dimensionMap[
        obsAnnoDimensionName(action.metadataField)
      ].filterAll();
      return {
        ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
      };
    }

@@ -509,7 +641,6 @@ const Controls = (
    case "color by continuous metadata": {
      return {
        ...state,
-        colorName: action.colors.name,
        colorRGB: action.colors.rgb,
        colorAccessor: action.colorAccessor,
        colorScale: action.colorScale
@@ -518,7 +649,6 @@ const Controls = (
    case "color by expression": {
      return {
        ...state,
-        colorName: action.colors.name,
        colorRGB: action.colors.rgb,
        colorAccessor: action.gene,
        colorScale: action.colorScale
@@ -1,9 +1,9 @@
 // jshint esversion: 6
 import { combineReducers, createStore, applyMiddleware } from "redux";
 import thunk from "redux-thunk";
+import { composeWithDevTools } from "redux-devtools-extension";
 import updateURLMiddleware from "../middleware/updateURLMiddleware";
 import updateCellColors from "../middleware/updateCellColors";
-import { composeWithDevTools } from "redux-devtools-extension";

 import config from "./config";
 import differential from "./differential";
@@ -25,24 +25,45 @@ export function catchErrorsWrap(fn, dispatchToUser = false) {
 }

 /*
-Wrapper to perform an async fetch and JSON decode response.
+Wrapper to perform async fetch with some modest error handling
+and decoding.
 */
-export const doJsonRequest = async url => {
+const doFetch = async (url, acceptType) => {
  const res = await fetch(url, {
    method: "get",
    headers: new Headers({
-      "Content-Type": "application/json"
+      Accept: acceptType
    })
  });
-  if (res.ok && res.headers.get("Content-Type") === "application/json") {
-    return res.json();
+  if (res.ok && res.headers.get("Content-Type").includes(acceptType)) {
+    return res;
  }
  // else an error
-  const msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  let msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  const body = await res.text();
+  if (body && body.length > 0) {
+    msg = `${msg} -- ${body}`;
+  }
  dispatchNetworkErrorMessageToUser(msg);
  throw new Error(msg);
 };

+/*
+Wrapper to perform an async fetch and JSON decode response.
+*/
+export const doJsonRequest = async url => {
+  const res = await doFetch(url, "application/json");
+  return res.json();
+};
+
+/*
+Wrapper to perform an async fetch for binary data.
+*/
+export const doBinaryRequest = async url => {
+  const res = await doFetch(url, "application/octet-stream");
+  return res.arrayBuffer();
+};
+
 /*
 This function "packs" filter index lists into the more efficient
 "range" form specified in the REST 0.2 spec.
@@ -0,0 +1,33 @@
+/*
+Return the [minimum, maximum] extent, of the given typed array, ignoring
+non-finite values (ie, +Infinity, -Infinity).
+
+If undefined or empty array, or array contains only non-finite numbers,
+will return [undefined, undefined]
+*/
+
+function finiteExtent(tarr) {
+  let min;
+  let max;
+  let i;
+
+  for (i = 0; i < tarr.length; i += 1) {
+    const val = tarr[i];
+    if (Number.isFinite(val)) {
+      min = val;
+      max = val;
+      i += 1;
+      break;
+    }
+  }
+  for (; i < tarr.length; i += 1) {
+    const val = tarr[i];
+    if (Number.isFinite(val)) {
+      if (min > val) min = val;
+      if (max < val) max = val;
+    }
+  }
+  return [min, max];
+}
+
+export default finiteExtent;
@@ -3,7 +3,7 @@
 /*
 Model manager providing an abstraction for the use of the reducer code.
 This module provides several buckets of functionality:
-  - schema and config driven tranformation of the dataframe wire protocol
+  - schema and config driven tranformation of the wire protocol
    into a format that is easy for the UI code to use.
  - manage the universe/world abstraction:
    + universe: all of the server-provided, read-only data
@@ -17,3 +17,4 @@ exists to support those concepts.
 export * as Universe from "./universe";
 export * as World from "./world";
 export * as kvCache from "./keyvalcache";
+export * as WorldUtil from "./worldUtil";
@@ -0,0 +1,75 @@
+import { flatbuffers } from "flatbuffers";
+import { NetEncoding } from "./matrix_generated";
+
+const utf8Decoder = new TextDecoder("utf-8");
+
+/*
+Matrix flatbuffer decoding support.   See fbs/matrix.fbs
+*/
+
+/*
+Decode NetEncoding.TypedArray
+*/
+function decodeTypedArray(uType, uValF, inplace = false) {
+  if (uType === NetEncoding.TypedArray.NONE) {
+    return null;
+  }
+
+  // Convert to a JS class that supports this type
+  const TypeClass = NetEncoding[NetEncoding.TypedArray[uType]];
+  // Create a TypedArray that references the underlying buffer
+  let arr = uValF(new TypeClass()).dataArray();
+  if (uType === NetEncoding.TypedArray.JSONEncodedArray) {
+    const json = utf8Decoder.decode(arr);
+    arr = JSON.parse(json);
+  } else if (!inplace) {
+    /* force copy to release underlying FBS buffer */
+    arr = new arr.constructor(arr);
+  }
+  return arr;
+}
+
+/*
+Parameter: Uint8Array or ArrayBuffer containing raw flatbuffer Matrix
+Returns: object containing decoded Matrix:
+{
+  nRows: num,
+  nCols: num,
+  columns: [
+    each column, which will be a TypedArray or Array
+  ]
+  colIdx: []|null
+}
+*/
+function decodeMatrixFBS(arrayBuffer, inplace = false) {
+  const bb = new flatbuffers.ByteBuffer(new Uint8Array(arrayBuffer));
+  const df = NetEncoding.Matrix.getRootAsMatrix(bb);
+
+  const nRows = df.nRows();
+  const nCols = df.nCols();
+
+  /* decode columns */
+  const columnsLength = df.columnsLength();
+  const columns = Array(columnsLength).fill(null);
+  for (let c = 0; c < columnsLength; c += 1) {
+    const col = df.columns(c);
+    columns[c] = decodeTypedArray(col.uType(), col.u.bind(col), inplace);
+  }
+
+  /* decode col_idx */
+  const colIdx = decodeTypedArray(
+    df.colIndexType(),
+    df.colIndex.bind(df),
+    inplace
+  );
+
+  return {
+    nRows,
+    nCols,
+    columns,
+    colIdx,
+    rowIdx: null
+  };
+}
+
+export default decodeMatrixFBS;
@@ -0,0 +1,835 @@
+// automatically generated by the FlatBuffers compiler, do not modify
+
+/**
+ * @const
+ * @namespace
+ */
+var NetEncoding = NetEncoding || {};
+
+/**
+ * @enum
+ */
+NetEncoding.TypedArray = {
+  NONE: 0, 0: 'NONE',
+  Float32Array: 1, 1: 'Float32Array',
+  Int32Array: 2, 2: 'Int32Array',
+  Uint32Array: 3, 3: 'Uint32Array',
+  Float64Array: 4, 4: 'Float64Array',
+  JSONEncodedArray: 5, 5: 'JSONEncodedArray'
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Float32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Float32Array}
+ */
+NetEncoding.Float32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Float32Array=} obj
+ * @returns {NetEncoding.Float32Array}
+ */
+NetEncoding.Float32Array.getRootAsFloat32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Float32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Float32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readFloat32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Float32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Float32Array}
+ */
+NetEncoding.Float32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Float32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Float32Array.startFloat32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Float32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addFloat32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Float32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float32Array.endFloat32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Uint32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Uint32Array}
+ */
+NetEncoding.Uint32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Uint32Array=} obj
+ * @returns {NetEncoding.Uint32Array}
+ */
+NetEncoding.Uint32Array.getRootAsUint32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Uint32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Uint32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Uint32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Uint32Array}
+ */
+NetEncoding.Uint32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Uint32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Uint32Array.startUint32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Uint32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Uint32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Uint32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Uint32Array.endUint32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Int32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Int32Array}
+ */
+NetEncoding.Int32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Int32Array=} obj
+ * @returns {NetEncoding.Int32Array}
+ */
+NetEncoding.Int32Array.getRootAsInt32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Int32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Int32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readInt32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Int32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Int32Array}
+ */
+NetEncoding.Int32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Int32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Int32Array.startInt32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Int32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Int32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Int32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Int32Array.endInt32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Float64Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Float64Array}
+ */
+NetEncoding.Float64Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Float64Array=} obj
+ * @returns {NetEncoding.Float64Array}
+ */
+NetEncoding.Float64Array.getRootAsFloat64Array = function(bb, obj) {
+  return (obj || new NetEncoding.Float64Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Float64Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readFloat64(this.bb.__vector(this.bb_pos + offset) + index * 8) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Float64Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Float64Array}
+ */
+NetEncoding.Float64Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Float64Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Float64Array.startFloat64Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Float64Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float64Array.createDataVector = function(builder, data) {
+  builder.startVector(8, data.length, 8);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addFloat64(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Float64Array.startDataVector = function(builder, numElems) {
+  builder.startVector(8, numElems, 8);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float64Array.endFloat64Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.JSONEncodedArray = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.JSONEncodedArray}
+ */
+NetEncoding.JSONEncodedArray.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.JSONEncodedArray=} obj
+ * @returns {NetEncoding.JSONEncodedArray}
+ */
+NetEncoding.JSONEncodedArray.getRootAsJSONEncodedArray = function(bb, obj) {
+  return (obj || new NetEncoding.JSONEncodedArray).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.JSONEncodedArray.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint8(this.bb.__vector(this.bb_pos + offset) + index) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.JSONEncodedArray.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Uint8Array}
+ */
+NetEncoding.JSONEncodedArray.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Uint8Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.JSONEncodedArray.startJSONEncodedArray = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.JSONEncodedArray.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.JSONEncodedArray.createDataVector = function(builder, data) {
+  builder.startVector(1, data.length, 1);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt8(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.JSONEncodedArray.startDataVector = function(builder, numElems) {
+  builder.startVector(1, numElems, 1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.JSONEncodedArray.endJSONEncodedArray = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Column = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Column.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Column=} obj
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Column.getRootAsColumn = function(bb, obj) {
+  return (obj || new NetEncoding.Column).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Column.prototype.uType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Column.prototype.u = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 6);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Column.startColumn = function(builder) {
+  builder.startObject(2);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} uType
+ */
+NetEncoding.Column.addUType = function(builder, uType) {
+  builder.addFieldInt8(0, uType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} uOffset
+ */
+NetEncoding.Column.addU = function(builder, uOffset) {
+  builder.addFieldOffset(1, uOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Column.endColumn = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Matrix = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Matrix}
+ */
+NetEncoding.Matrix.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Matrix=} obj
+ * @returns {NetEncoding.Matrix}
+ */
+NetEncoding.Matrix.getRootAsMatrix = function(bb, obj) {
+  return (obj || new NetEncoding.Matrix).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.nRows = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint32(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.nCols = function() {
+  var offset = this.bb.__offset(this.bb_pos, 6);
+  return offset ? this.bb.readUint32(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @param {number} index
+ * @param {NetEncoding.Column=} obj
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Matrix.prototype.columns = function(index, obj) {
+  var offset = this.bb.__offset(this.bb_pos, 8);
+  return offset ? (obj || new NetEncoding.Column).__init(this.bb.__indirect(this.bb.__vector(this.bb_pos + offset) + index * 4), this.bb) : null;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.columnsLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 8);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Matrix.prototype.colIndexType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 10);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Matrix.prototype.colIndex = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 12);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Matrix.prototype.rowIndexType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 14);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Matrix.prototype.rowIndex = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 16);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Matrix.startMatrix = function(builder) {
+  builder.startObject(7);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} nRows
+ */
+NetEncoding.Matrix.addNRows = function(builder, nRows) {
+  builder.addFieldInt32(0, nRows, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} nCols
+ */
+NetEncoding.Matrix.addNCols = function(builder, nCols) {
+  builder.addFieldInt32(1, nCols, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} columnsOffset
+ */
+NetEncoding.Matrix.addColumns = function(builder, columnsOffset) {
+  builder.addFieldOffset(2, columnsOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<flatbuffers.Offset>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Matrix.createColumnsVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addOffset(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Matrix.startColumnsVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} colIndexType
+ */
+NetEncoding.Matrix.addColIndexType = function(builder, colIndexType) {
+  builder.addFieldInt8(3, colIndexType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} colIndexOffset
+ */
+NetEncoding.Matrix.addColIndex = function(builder, colIndexOffset) {
+  builder.addFieldOffset(4, colIndexOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} rowIndexType
+ */
+NetEncoding.Matrix.addRowIndexType = function(builder, rowIndexType) {
+  builder.addFieldInt8(5, rowIndexType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} rowIndexOffset
+ */
+NetEncoding.Matrix.addRowIndex = function(builder, rowIndexOffset) {
+  builder.addFieldOffset(6, rowIndexOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Matrix.endMatrix = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} offset
+ */
+NetEncoding.Matrix.finishMatrixBuffer = function(builder, offset) {
+  builder.finish(offset);
+};
+
+// Exports for ECMAScript6 Modules
+export {NetEncoding};
@@ -1,4 +1,5 @@
 import _ from "lodash";
+import finiteExtent from "../finiteExtent";

 /*
 Build and return obs/var summary using any annotation in the schema
@@ -8,6 +9,7 @@ Value will be an object, containing summary information.

 For continuous annotations (int, float, etc):
  <annotation_name>: {
+    categorical: false,
    range {
      min: <number>,
      max: <number>
@@ -15,12 +17,14 @@ For continuous annotations (int, float, etc):
  }

 For categorical annotations (boolean, string, category):
-  <annotatoin_name>: {
-    options: {
-      <option1>: <number>,
+  <annotation_name>: {
+    categorical: true,
+    categories: [ <category1>, <category2>, ... ]
+    categoryCounts: Map {
+      <category1>: <number>,
      ...
    },
-    numOptions: <number>
+    numCategories: <number>
  }

 Summarize will be returned for BOTH obs and var annotations.
@@ -28,19 +32,19 @@ Summarize will be returned for BOTH obs and var annotations.
 Example:
  {
    "Splice_sites_Annotated": {
-      "range": {
+      categorical: false,
+      range: {
        "min": 26,
        "max": 1075869
      }
    },
    "Selection": {
-      numOptions, 6,
-      "options": {
+      categorical: true,
+      numCategories, 3,
+      categories: [ "Astrocytes(HEPACAM)", "Endothelial(BSC)", "Unpanned" ],
+      categoryCounts: Map {
        "Astrocytes(HEPACAM)": 714,
        "Endothelial(BSC)": 123,
-        "Oligodendrocytes(GC)": 294,
-        "Neurons(Thy1)": 685,
-        "Microglia(CD45)": 1108,
        "Unpanned": 665
      }
    }
@@ -49,37 +53,61 @@ Example:
 NOTE: will not summarize the required 'name' annotation, as that is
 specified as unique per element.
 */
-function summarizeDimension(schema, annotations) {
-  return _(schema)
+function _summarizeAnnotations(_schema, annotations) {
+  const summary = _(_schema) // lodash wrapping: https://lodash.com/docs/4.17.11#lodash
    .filter(v => v.name !== "name")
    .keyBy("name")
    .mapValues(anno => {
      const { name, type } = anno;
      const continuous = type === "int32" || type === "float32";

-      if (!continuous) {
-        const options = _.countBy(annotations, name);
-        const numOptions = _.size(options);
+      if (continuous) {
+        let min;
+        let max;
+        let nan = 0;
+        let pinf = 0;
+        let ninf = 0;
+        for (let r = 0; r < annotations.length; r += 1) {
+          const val = Number(annotations[r][name]);
+          if (Number.isFinite(val)) {
+            if (min === undefined) {
+              min = val;
+              max = val;
+            } else {
+              min = val < min ? val : min;
+              max = val > max ? val : max;
+            }
+          } else if (Number.isNaN(val)) {
+            nan += 1;
+          } else if (val > 0) {
+            pinf += 1;
+          } else {
+            ninf += 1;
+          }
+        }
        return {
-          numOptions,
-          options
+          categorical: false,
+          range: { min, max, nan, pinf, ninf }
        };
      }

-      if (continuous) {
-        let min = Number.POSITIVE_INFINITY;
-        let max = Number.NEGATIVE_INFINITY;
-        _.forEach(annotations, obs => {
-          const val = Number(obs[name]);
-          min = val < min ? val : min;
-          max = val > max ? val : max;
-        });
-        return { range: { min, max } };
+      /* else categorical */
+      const categoryCounts = new Map();
+      for (let r = 0; r < annotations.length; r += 1) {
+        const val = annotations[r][name];
+        let curCount = categoryCounts.get(val);
+        if (curCount === undefined) curCount = 0;
+        categoryCounts.set(val, curCount + 1);
      }
-
-      throw new Error("incomprehensible schema");
+      return {
+        categorical: true,
+        categories: [...categoryCounts.keys()],
+        categoryCounts,
+        numCategories: categoryCounts.size
+      };
    })
    .value();
+  return summary;
 }

 export default function summarizeAnnotations(
@@ -88,7 +116,7 @@ export default function summarizeAnnotations(
  varAnnotations
 ) {
  return {
-    obs: summarizeDimension(schema.annotations.obs, obsAnnotations),
-    var: summarizeDimension(schema.annotations.var, varAnnotations)
+    obs: _summarizeAnnotations(schema.annotations.obs, obsAnnotations),
+    var: _summarizeAnnotations(schema.annotations.var, varAnnotations)
  };
 }
@@ -1,8 +1,10 @@
 // jshint esversion: 6

 import _ from "lodash";
+
 import * as kvCache from "./keyvalcache";
 import summarizeAnnotations from "./summarizeAnnotations";
+import decodeMatrixFBS from "./matrix";

 /*
 Private helper function - create and return a template Universe
@@ -92,73 +94,66 @@ function finalize(universe) {
  return universe;
 }

-function RESTv02AnnotationsResponseToInternal(response) {
+function RESTv02AnotationsFBSResponseToInternal(arrayBuffer) {
  /*
-  Source per the spec:
-  {
-    names: [
-      'tissue_type', 'sex', 'num_reads', 'clusters'
-    ],
-    data: [
-      [ 0, 'lung', 'F', 39844, 99 ],
-      [ 1, 'heart', 'M', 83, 1 ],
-      [ 49, 'spleen', null, 2, "unknown cluster" ],
-      // [ obsOrVarIndex, value, value, value, value ],
-      // ...
-    ]
-  }
+  Convert a Matrix FBS to our internal format -- row-major array of
+  observations/cells, stored as an object.  Each obs has a key for each
+  annotation, plus __index__ containing its obsIndex.

-  Internal (target) format:
+  Example:
  [
    { __index__: 0, tissue_type: "lung", sex: "F", ... },
    ...
  ]
+
+  XXX TODO: we could make use of the columns in building crossfilter
+  dimensions (they have to be recreated).  Future optimization.
  */
-  const { names, data } = response;
-  const keys = ["__index__", ...names];
-  return _(data)
-    .map(obs => _.zipObject(keys, obs))
-    .value();
+  const fbs = decodeMatrixFBS(arrayBuffer);
+  const keys = fbs.colIdx;
+  const result = Array(fbs.nRows);
+  for (let row = 0; row < fbs.nRows; row += 1) {
+    const rec = { __index__: row };
+    for (let col = 0; col < fbs.nCols; col += 1) {
+      rec[keys[col]] = fbs.columns[col][row];
+    }
+    result[row] = rec;
+  }
+  return result;
 }

-function RESTv02LayoutResponseToInternal(response) {
-  /*
-  Source per the spec:
-  {
-    layout: {
-      ndims: 2,
-      coordinates: [
-        [ 0, 0.284483, 0.983744 ],
-        [ 1, 0.038844, 0.739444 ],
-        // [ obsOrVarIndex, X_coord, Y_coord ],
-        // ...
-      ]
-    }
-  }
-
-  Target (internal) format:
-  {
-    X: Float32Array(numObs),
-    Y: Float32Array(numObs)
-  }
-  In the same order as obsAnnotations
-  */
-  const { ndims, coordinates } = response.layout;
-  if (ndims !== 2) {
-    throw new Error("Unsupported layout dimensionality");
-  }
-
-  const layout = {
-    X: new Float32Array(coordinates.length),
-    Y: new Float32Array(coordinates.length)
+function RESTv02LayoutFBSResponseToInternal(arrayBuffer) {
+  const fbs = decodeMatrixFBS(arrayBuffer, true);
+  return {
+    X: fbs.columns[0],
+    Y: fbs.columns[1]
  };
+}

-  for (let i = 0; i < coordinates.length; i += 1) {
-    const [idx, x, y] = coordinates[i];
-    layout.X[idx] = x;
-    layout.Y[idx] = y;
-  }
-  return layout;
+function reconcileSchemaCategoriesWithSummary(universe) {
+  /*
+  where we treat types as (essentially) categorical metadata, update
+  the schema with data-derived categories (in addition to those in
+  the server declared schema).
+
+  For example, boolean defined fields in the schema do not contain
+  explicit declaration of categories (nor do string fields).  In these
+  cases, add a 'categories' field to the schema so it is accessible.
+  */
+
+  _.forEach(universe.schema.annotations.obs, s => {
+    if (
+      s.type === "string" ||
+      s.type === "boolean" ||
+      s.type === "categorical"
+    ) {
+      const categories = _.union(
+        _.get(s, "categories", []),
+        _.get(universe.summary.obs[s.name], "categories", [])
+      );
+      s.categories = categories;
+    }
+  });
 }

 export function createUniverseFromRestV02Response(
@@ -166,7 +161,7 @@ export function createUniverseFromRestV02Response(
  schemaResponse,
  annotationsObsResponse,
  annotationsVarResponse,
-  layoutObsResponse
+  layoutFBSResponse
 ) {
  /*
  build & return universe from a REST 0.2 /config, /schema and /annotations/obs response
@@ -183,15 +178,15 @@ export function createUniverseFromRestV02Response(
  universe.nVar = schema.dataframe.nVar;

  /* annotations */
-  universe.obsAnnotations = RESTv02AnnotationsResponseToInternal(
+  universe.obsAnnotations = RESTv02AnotationsFBSResponseToInternal(
    annotationsObsResponse
  );
-  universe.varAnnotations = RESTv02AnnotationsResponseToInternal(
+  universe.varAnnotations = RESTv02AnotationsFBSResponseToInternal(
    annotationsVarResponse
  );

  /* layout */
-  universe.obsLayout = RESTv02LayoutResponseToInternal(layoutObsResponse);
+  universe.obsLayout = RESTv02LayoutFBSResponseToInternal(layoutFBSResponse);

  universe.summary = summarizeAnnotations(
    universe.schema,
@@ -199,35 +194,28 @@ export function createUniverseFromRestV02Response(
    universe.varAnnotations
  );

+  reconcileSchemaCategoriesWithSummary(universe);
  return finalize(universe);
 }

-export function convertExpressionRESTv02ToObject(universe, response) {
+export function convertDataFBStoObject(universe, arrayBuffer) {
  /*
-  /data/obs response looks like:
-  {
-    var: [ varIndices fetched ],
-    obs: [
-      [ obsIndex, evalue, ... ],
-      ...
-    ]
-  }
+  /data/var returns a flatbuffer (FBS) as described by cellxgene/fbs/matrix.fbs

-  convert expression toa simple Float32Array, and return
-  { geneName: array, geneName: array, ... }
-  NOTE: geneName, not varIndex
+  This routine converts the binary wire encoding into a JS object:
+
+  {
+    gene: Float32Array,
+    ...
+  }
  */
-  const vars = response.var;
-  const { obs } = response;
+  const fbs = decodeMatrixFBS(arrayBuffer);
+  const { colIdx, columns } = fbs;
  const result = {};
-  // XXX TODO: could this use _.unzip and have less code?
-  for (let varIdx = 0; varIdx < vars.length; varIdx += 1) {
-    const gene = universe.varAnnotations[vars[varIdx]].name;
-    const data = new Float32Array(universe.nObs);
-    for (let obsIdx = 0; obsIdx < obs.length; obsIdx += 1) {
-      data[obsIdx] = obs[obsIdx][varIdx + 1];
-    }
-    result[gene] = data;
+
+  for (let c = 0; c < colIdx.length; c += 1) {
+    const gene = universe.varAnnotations[colIdx[c]].name;
+    result[gene] = columns[c];
  }
  return result;
 }
@@ -4,6 +4,7 @@ import _ from "lodash";
 import * as kvCache from "./keyvalcache";
 import summarizeAnnotations from "./summarizeAnnotations";
 import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
+import Crossfilter from "../typedCrossfilter";
 import { sliceByIndex } from "../typedCrossfilter/util";

 /*
@@ -231,7 +232,12 @@ export function createVarDimension(
  crossfilter,
  geneName
 ) {
-  return crossfilter.dimension(_worldVarDataCache[geneName], Float32Array);
+  // return crossfilter.dimension(_worldVarDataCache[geneName], Float32Array);
+  return crossfilter.dimension(
+    Crossfilter.ScalarDimension,
+    _worldVarDataCache[geneName],
+    Float32Array
+  );
 }

 export function createObsDimensionMap(crossfilter, world) {
@@ -246,9 +252,14 @@ export function createObsDimensionMap(crossfilter, world) {
    .filter(anno => anno.name !== "name")
    .transform((result, anno) => {
      const dimType = deduceDimensionType(anno, anno.name);
-      // XXX if dimtype is a scalar, we may be able to do better?
-      if (dimType) {
+      if (dimType === "enum") {
        result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
+          Crossfilter.EnumDimension,
+          r => r[anno.name]
+        );
+      } else {
+        result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
+          Crossfilter.ScalarDimension,
          r => r[anno.name],
          dimType
        );
@@ -259,13 +270,10 @@ export function createObsDimensionMap(crossfilter, world) {
  /*
  Add crossfilter dimensions allowing filtering on layout
  */
-  dimensionMap[layoutDimensionName("X")] = crossfilter.dimension(
+  dimensionMap[layoutDimensionName("XY")] = crossfilter.dimension(
+    Crossfilter.SpatialDimension,
    obsLayout.X,
-    Float32Array
-  );
-  dimensionMap[layoutDimensionName("Y")] = crossfilter.dimension(
-    obsLayout.Y,
-    Float32Array
+    obsLayout.Y
  );

  return dimensionMap;
@@ -0,0 +1,65 @@
+/* eslint-disable import/prefer-default-export */
+import _ from "lodash";
+
+/*
+Various utility functions operating on World/Universe
+*/
+
+/*
+Count unique category values, binning first by dim1 then by dim2
+Return:
+
+Map {
+   dim1_val1: Map {
+      dim2_val1: number,
+      dim2_val2: number,
+      ...
+   },
+   ...
+}
+
+*/
+function _countCategoryValues2D(dim1, dim2, rows) {
+  const dimMap = new Map();
+  for (let r = 0; r < rows.length; r += 1) {
+    const row = rows[r];
+    const val1 = row[dim1];
+    const val2 = row[dim2];
+    let d2Map = dimMap.get(val1);
+    if (d2Map === undefined) {
+      d2Map = new Map();
+      dimMap.set(val1, d2Map);
+    }
+    let curCount = d2Map.get(val2);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    d2Map.set(val2, curCount + 1);
+  }
+  return dimMap;
+}
+
+let __worldUtilMemoId__ = 0;
+function _memoizedId(x) {
+  if (!x.__worldUtilMemoId__) {
+    __worldUtilMemoId__ += 1;
+    x.__worldUtilMemoId__ = __worldUtilMemoId__;
+  }
+  return x.__worldUtilMemoId__;
+}
+function _countCategoryValues2DResolver(...args) {
+  const id = args[0] + args[1] + _memoizedId(args[2]);
+  return id;
+}
+
+export const countCategoryValues2D = _.memoize(
+  _countCategoryValues2D,
+  _countCategoryValues2DResolver
+);
+
+/*
+Clear any cached data within WorldUtil caches, eg, memoized functions
+*/
+export function clearCaches() {
+  countCategoryValues2D.cache.clear();
+}
@@ -27,6 +27,8 @@ more complex API.  In a few cases, elements of that API were incorporated.
    https://github.com/square/crossfilter/

 */
+// XXX replace
+import { polygonContains } from "d3";

 import PositiveIntervals from "./positiveIntervals";
 import BitArray from "./bitArray";
@@ -66,14 +68,22 @@ class TypedCrossfilter {
    return this.data;
  }

-  dimension(value, valueArrayType) {
+  /*
+  Create a crossfilter dimension, upon which filtering (subselection) can
+  be done.  Each dimension is typed, and has a particular set of filtering
+  semantics.
+    * ScalarDimension - backed by TypedArray values, supporting filtering
+      by value (within a value range, or one or more exact values)
+    * EnumDimension - backed by an enumeration (eg, strings, bools), filtering
+      by one or more enum categories.
+    * SpatialDimension - backed by 2D points, filter by containment within
+      various shapes (currently supports within Rectangle and within Polygon).
+  Call this method to create a dimension, passing arguments appropriate for
+  the dimension constructor.
+  */
+  dimension(DimensionType, ...rest) {
    const id = this.selection.allocDimension();
-    let dim;
-    if (valueArrayType === "enum") {
-      dim = new EnumDimension(value, this, id);
-    } else {
-      dim = new ScalarDimension(value, valueArrayType, this, id);
-    }
+    const dim = new DimensionType(this, id, ...rest);
    this.filters.push({ id, dim });
    dim.filterAll();
    return dim;
@@ -115,13 +125,34 @@ class TypedCrossfilter {
  }
 }

-// Base dimension type - value must be a scalar type (eg, int, float),
-// and value array must be a TypedArray.
-//
-class ScalarDimension {
-  constructor(value, ValueArrayType, xfltr, id) {
+// Base dimension type - not exported.
+class _Dimension {
+  constructor(xfltr, id) {
    this.crossfilter = xfltr;
    this._id = id;
+    this.groups = [];
+  }
+
+  dispose() {
+    this.crossfilter._freeDimension(this._id);
+    return this;
+  }
+
+  id() {
+    return this._id;
+  }
+
+  _filterUpdate() {
+    this.crossfilter.updateTime += 1;
+  }
+}
+
+// Scalar dimension type - value must be a scalar type (eg, int, float),
+// and value array must be a TypedArray.
+//
+class ScalarDimension extends _Dimension {
+  constructor(xfltr, id, value, ValueArrayType) {
+    super(xfltr, id);

    // current selection filter, expressed as PostiveIntervals.
    this.currentFilter = [];
@@ -151,9 +182,6 @@ class ScalarDimension {

    // create sort index
    this.index = makeSortIndex(array);
-
-    // groups, if any
-    this.groups = [];
  }

  _createValueArray(value, array) {
@@ -167,15 +195,6 @@ class ScalarDimension {
    return larray;
  }

-  dispose() {
-    this.crossfilter._freeDimension(this._id);
-    return this;
-  }
-
-  id() {
-    return this._id;
-  }
-
  // Argument is an array of intervals indicating records newly selected/filtered
  //
  _updateFilters(newFilter) {
@@ -209,7 +228,7 @@ class ScalarDimension {
    );

    this.currentFilter = cNewFilter;
-    this.crossfilter.updateTime += 1;
+    this._filterUpdate();
  }

  // filter by value - exact match
@@ -355,8 +374,8 @@ class ScalarDimension {
 // strings, which can be mapped into an fixed numeric range [0..n).
 //
 class EnumDimension extends ScalarDimension {
-  constructor(value, xfltr, id) {
-    super(value, Uint32Array, xfltr, id);
+  constructor(xfltr, id, value) {
+    super(xfltr, id, value, Uint32Array);
  }

  _createValueArray(value, array) {
@@ -408,6 +427,127 @@ class EnumDimension extends ScalarDimension {
  }
 }

+/*
+Super simple 2D spatial dimension, supporting basic "filter within"
+operations.
+*/
+class SpatialDimension extends _Dimension {
+  constructor(xfltr, id, X, Y) {
+    super(xfltr, id);
+
+    if (X.length !== Y.length && X.length !== this.crossfilter.data.length) {
+      throw new RangeError(
+        "SpatialDimension values must have same dimensionality as crossfilter"
+      );
+    }
+    this.X = X;
+    this.Y = Y;
+
+    this.Xindex = makeSortIndex(X);
+    this.Yindex = makeSortIndex(Y);
+  }
+
+  filterAll() {
+    this.crossfilter.selection.selectAll(this._id);
+    this._filterUpdate();
+  }
+
+  filterNone() {
+    this.crossfilter.selection.deselectAll(this._id);
+    this._filterUpdate();
+  }
+
+  /*
+  this could be smarter, but we don't currently use it...
+  */
+  filterWithinRect(northwest, southeast) {
+    const [x0, y0] = northwest;
+    const [x1, y1] = southeast;
+    const { X, Y } = this;
+    const seln = this.crossfilter.selection;
+    const { _id } = this;
+    seln.deselectAll(_id);
+    for (let i = 0, l = this.X.length; i < l; i += 1) {
+      const x = X[i];
+      const y = Y[i];
+      if (x0 <= x && x < x1 && y0 <= y && y < y1) {
+        seln.selectOne(_id, i);
+      }
+    }
+    this._filterUpdate();
+  }
+
+  /*
+  Relatively brute force filter by polygon.   Polygon is array of points, where
+  each point is [x,y].   Eg, [[x0,y0], [x1,y1], ...].
+
+  Currently uses d3.polygonContains() to test for polygon inclusion, which itself
+  uses a ray casting (crossing number) algorithm.  There are a series of optimizations
+  to make this faster:
+    * first sliced by X or Y, using an index on the axis
+    * then the polygon bounding box is used for trivial rejection
+    * then the polygon test is applied
+  */
+  filterWithinPolygon(polygon) {
+    /* return bounding box of the polygon */
+    function polygonBoundingBox(pg) {
+      let minX = Number.MAX_VALUE;
+      let minY = Number.MAX_VALUE;
+      let maxX = Number.MIN_VALUE;
+      let maxY = Number.MIN_VALUE;
+      for (let i = 0, l = pg.length; i < l; i += 1) {
+        const p = pg[i];
+        const x = p[0];
+        const y = p[1];
+        if (x < minX) minX = x;
+        if (y < minY) minY = y;
+        if (x > maxX) maxX = x;
+        if (y > maxY) maxY = y;
+      }
+      return [minX, minY, maxX, maxY];
+    }
+
+    const [minX, minY, maxX, maxY] = polygonBoundingBox(polygon);
+    const { X, Y } = this;
+    let slice;
+    let index;
+    if (maxY - minY > maxX - minX) {
+      slice = [
+        lowerBoundIndirect(X, this.Xindex, minX, 0, X.length),
+        upperBoundIndirect(X, this.Xindex, maxX, 0, X.length)
+      ];
+      index = this.Xindex;
+    } else {
+      slice = [
+        lowerBoundIndirect(Y, this.Yindex, minY, 0, Y.length),
+        upperBoundIndirect(Y, this.Yindex, maxY, 0, Y.length)
+      ];
+      index = this.Yindex;
+    }
+
+    const seln = this.crossfilter.selection;
+    const { _id } = this;
+    const testWithin = polygonContains; // d3.polygonContains()
+    seln.deselectAll(_id);
+
+    for (let i = slice[0], e = slice[1]; i < e; i += 1) {
+      const rid = index[i];
+      const x = X[rid];
+      const y = Y[rid];
+      if (
+        minX <= x &&
+        x < maxX &&
+        minY <= y &&
+        y < maxY &&
+        testWithin(polygon, [x, y])
+      ) {
+        seln.selectOne(_id, rid);
+      }
+    }
+    this._filterUpdate();
+  }
+}
+
 // Groups!  Map/reduce
 //
 class ScalarGroup {
@@ -611,5 +751,6 @@ crossfilter.BitArray = BitArray;
 crossfilter.TypedCrossfilter = TypedCrossfilter;
 crossfilter.ScalarDimension = ScalarDimension;
 crossfilter.EnumDimension = EnumDimension;
+crossfilter.SpatialDimension = SpatialDimension;

 export default crossfilter;
@@ -1,5 +1,11 @@
 # cellxgene REST API 0.2 specification

+_Note:_ this document is increasingly divergent from the code base and should be read with great suspicion. For example, it lacks any information
+about the binary encoding used by various routes, and has incorrect information about "required" routes and features. We may update it at a
+later date when the protocol stabilizes a bit.
+
+---
+
 Items marked as (_future_) are intended for future implementation, and are included in the design to round out the concept, and highlight what we would do when/if we needed more functionality. The (_future_) items are not currently used by the cellxgene web application, and may be omitted from any backend - see [Current Front-End Dependencies](#current-front-end-dependencies) for more details.

 _Caveat emptor, partial spec_: this is a sketch for a spec, not a full spec, and some shortcuts have been taken in the authorship. Best practices for a REST API are assumed but not documented here, such as API versioning, reasonable choices for HTTP response codes, etc. In addition, for clarity the JSON examples will not always have all required quoting (eg, on keys) - the actual implementation should use legal JSON/CSV.
@@ -568,14 +574,14 @@ If differential expression is not supported by the server, must return an HTTP 5

 **Response body:**

- For 200 Success, differential expression statistics returned as array of arrays sorted by varindex, where each contains the following values:
+- For 200 Success, differential expression statistics returned as array of arrays, where each contains the following values:

  - **varIndex**: variable index for the computed results
  - **logfoldchange**: log fold-change of the average expression between the two groups. Positive values indicate that the gene is more highly expressed in the first group,
  - **pVal**: unadjusted p-value,
  - **pValAdj**: adjusted p-value

-  Statistics are encoded as an array of arrays, with fields ordered as:
+  Values ordered as:

  _varIndex_, _logfoldchange_, _pVal_, _pValAdj_

@@ -0,0 +1,35 @@
+## Creating PR
+1. Name [username]/branchname 
+    1. Branch name should be all lowercase
+    2. Words separated by “-”
+2. Code should address only one issue ideally, make a separate PR for each task
+3. Description 
+    1. Clear explanation of issues solved
+    2. Describe why and how, when appropriate
+    3. Call out specific areas you want extra attention in review (optional)
+    4. If your PR requires more than one reviewer tag those people in the description or comments and let them know that you specifically require them
+4. Ensure that the PR updates tests and documentation and adds tests where appropriate
+5. Use github’s issue keywords when PR is addressing an issue https://help.github.com/articles/closing-issues-using-keywords/
+6. Tags (add at beginning of title)
+    1. [EASY] - small non-controversial change, easy to review
+    2. [DO NOT MERGE] - PR is in progress, do not merge changes
+
+## Review
+1. Assign at least one reviewer to submitted PRs.   Reviewers should be selected based on expertise in areas affected by the PR (eg, web UI: Colin), and should include Comp Bio and PM as needed.
+2. Reviewers should approve or request changes (not just comment) and put general and line level comments where appropriate
+3. As a PR submitter respond to all comments (eg, comment, commit a change, etc)
+4. External PRs
+    1. For external PRs or PRs not from our core team, core team should assign a reviewer and make initial contact within 1 business day
+    2. Build code on local environment and run smoke tests 
+
+## Required to Merge
+1. Travis CI Build passing
+2. At least one reviewer approved
+    1. Exceptions: 
+        1. Release PRs where version is just bumped should not need review
+        2. Complex PRs which touch multiple parts of the codebase should have reviews from all relevant parties
+3. License and Security checks (SNYK) passing. If their server is down and you didn’t add any new external npm or python packages, merge is OK
+
+## Merging
+1. Use "squash and merge" option when merging
+2. If you resolved conflicts, wait until the build passes to merge
@@ -0,0 +1,66 @@
+# cellxgene release process
+
+_This document defines the release process for cellxgene_
+
+## Overview
+
+The goal of the release process is to publish an installable package
+to PyPi, with a matching tagged release on github.
+
+The release process should result in the following side-effects:
+
+- Version number bump, using semantic versioning
+- JS assets built & packaged, committed to the repo
+- Tagged github release
+- Publication to PyPi
+
+## Recipe
+
+Follow these steps to create a release.
+
+1.  Preparation:
+    - python3.6 environment, and a cellxgene clone
+    - Define the release version number, using [semantic versioning](https://semver.org/),
+      and specifying all three digits (eg, 0.3.0)
+    - Write the release title and release notes and add to
+      [release notes document](https://docs.google.com/document/d/1KnHwkYfhyWO5H8BDcMu7y3ogjvq5Yi4OwpmZ8DB6w0Y/edit)
+2.  Create a release branch, eg, `release-version`
+3.  In the release branch:
+    - Run `make release-stage-1 PART=[major | minor | patch]` where you choose major/minor/patch depending on which part of the version
+      is being bumped (eg, 0.2.9->0.3 is minor).
+4.  Commit and push the new branch
+5.  Create a PR for the release.
+    - [optional] As needed, conduct PR review.
+6.  Merge to master
+7.  Create Github release using the version number and release notes
+    ([instructions](https://help.github.com/articles/creating-releases/)).
+    - Draft new release
+    - Type version name matching release version number from (1)
+    - Select `master` as release branch (ensure you merged the release PR)
+    - Type title `Release {version num}`
+    - [optional] Check pre-release if this release is not ready for production
+    - Publish Release
+8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
+and that you have write access to the cellxgene pypi package):
+    - Build the distribution and upload to test pypi `make release-stage-2`
+    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Upload the package to real pypi using `make release-stage-final`
+    - [optional] Test the installation in a fresh virtual environment using
+      `pip install cellxgene`
+    - **Troubleshooting**:
+        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
+        if you accidentally burned a release number you want to use on prod, you have a couple options. 
+            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+             then upload to prod `make release-stage-final`. 
+            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
+            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+            PART=patch until you get to an unburned version.
+        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
+        go to the cellxgene package, click manage, then in the options drop down, click delete and 
+        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
+        and not a major regression, you can just release a patch. 
+    
+
+The optional steps are for testing purposes, and are recommended
+for publishing any major releases, and any releases that significantly
+change the packaging (e.g. new bundled files, new dependencies, etc.)
@@ -0,0 +1,69 @@
+### How to set up a testing environment for changes related to web hosting.
+
+We often get PRs related to someone using a server to host cellxgene externally or on a local network (ex. https://github.com/chanzuckerberg/cellxgene/pull/568 ). Here is how you can test these changes locally.
+
+We are going to run docker containers for cellxgene and an apache server running a reverse proxy on a local docker network. We run the cellxgene container without exposing any ports so that we cannot access it directly, only through the apache server. We can also update our cellxgene Dockerfile so that we can install a local build instead of having to deploy to pypi.
+
+1 Create Docker network, this allows the containers to communicate with each other.
+
+```
+docker network create cxg
+```
+
+2 Create and run cellxgene container
+
+(optional) To install cellxgene from the local codebase
+
+a Create sdist file
+`make pydist`
+
+b Update Dockerfile to install from dist
+
+```
+FROM ubuntu:bionic
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+COPY [ "dist/",    "/cellxgene/dist/" ]
+
+RUN apt-get update && \
+    apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev && \
+    pip3 install /cellxgene/dist/cellxgene-0.5.1.tar.gz
+
+ENTRYPOINT ["cellxgene"]
+```
+
+(required) Build container
+`docker build . -t cellxgene`
+
+3 Create the proxy container
+
+In a separate directory create these two files
+
+Dockerfile
+
+```
+FROM rgoyard/apache-proxy:latest
+ADD proxy.conf /conf/
+```
+
+proxy.conf
+
+```
+ProxyPass "/data/" http://cellxgene:5005/
+ProxyPassReverse "/data/" http://cellxgene:5005/
+```
+
+Build the container
+`docker build -t proxy .`
+
+4 Run containers and attach to network
+
+```
+docker run -d -p 80:80 --network cxg --name proxy proxy
+docker run -v "$PWD/example-dataset/:/data/" --name cellxgene --network cxg cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad
+```
+
+5 Go to served site
+
+http://localhost/data/
@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gem "github-pages", group: :jekyll_plugins
@@ -0,0 +1,9 @@
+theme: jekyll-theme-cayman
+show_downloads: false
+baseurl: /cellxgene
+
+nav:
+  - title: Data
+    url: data.html
+  - title: FAQ
+    url: faq.html
@@ -0,0 +1,49 @@
+<!DOCTYPE html>
+<html lang="{{ site.lang | default: "en-US" }}">
+  <head>
+
+    {% if site.google_analytics %}
+      <script async src="https://www.googletagmanager.com/gtag/js?id={{ site.google_analytics }}"></script>
+      <script>
+        window.dataLayer = window.dataLayer || [];
+        function gtag(){dataLayer.push(arguments);}
+        gtag('js', new Date());
+        gtag('config', '{{ site.google_analytics }}');
+      </script>
+    {% endif %}
+    <meta charset="UTF-8">
+
+{% seo %}
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <meta name="theme-color" content="#157878">
+    <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
+    <link rel="stylesheet" href="{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}">
+  </head>
+  <body>
+
+    <header class="page-header" role="banner">
+      <h1 class="project-name">{{ site.title | default: site.github.repository_name }}</h1>
+      <h2 class="project-tagline">{{ site.description | default: site.github.project_tagline }}</h2>
+      {% if site.nav %}
+        <a href="{{ site.baseurl }}/" class="btn">Home</a>
+        {% for item in site.nav %}
+          <a href="{{ item.url }}" class="btn">{{ item.title }}</a>
+        {% endfor %}
+      {% endif %}
+      {% if site.github.is_project_page %}
+        <a href="{{ site.github.repository_url }}" class="btn" target="_blank">Code</a>
+      {% endif %}
+    </header>
+
+    <main id="content" class="main-content" role="main">
+      {{ content }}
+
+      <footer class="site-footer">
+        {% if site.github.is_project_page %}
+          <span class="site-footer-owner"><a href="{{ site.github.repository_url }}">{{ site.github.repository_name }}</a> is maintained by <a href="{{ site.github.owner_url }}">{{ site.github.owner_name }}</a>.</span>
+        {% endif %}
+        <span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a>.</span>
+      </footer>
+    </main>
+  </body>
+</html>
@@ -0,0 +1,4 @@
+---
+---
+
+@import "{{ site.theme }}";
@@ -0,0 +1,29 @@
+---
+layout: default
+title: data
+description: Data
+---
+
+
+# data vignette: how to use cellxgene prepare
+
+#### coming soon!
+
+# example datasets to use with cellxgene
+
+### Examination of single cells from primary human pancreas tissue
+cells: 2,544  
+tissue(s): pancreas  
+data: [Human Cell Atlas Data Portal](https://prod.data.humancellatlas.org/explore/projects?filter=%5B%7B%22facetName%22%3A%22organ%22%2C%22terms%22%3A%5B%22pancreas%22%5D%7D%2C%7B%22facetName%22%3A%22project%22%2C%22terms%22%3A%5B%22Single+cell+transcriptome+analysis+of+human+pancreas%22%5D%7D%5D)  
+paper: [Enge, Martin, et al.](https://www.cell.com/cell/fulltext/S0092-8674(17)31053-X?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS009286741731053X%3Fshowall%3Dtrue)
+
+### Tabula Muris
+cells: 53,800  
+tissue(s): muscle, pancreas, bone, large intestine, heart, brain, fat, mammary gland, tongue , diaphragm, bladder, spleen, thymus, lung , skin, liver, trachea, kidney  
+data: [Tabula Muris Data](https://github.com/czbiohub/tabula-muris-vignettes/tree/master/data)  
+paper: [Tabula Muris Consortium](https://www.nature.com/articles/s41586-018-0590-4)  
+
+### Transcriptional profiling of 1.3 million brain cells
+cells: 1,330,000  
+tissue(s): brain  
+data: [10x Genomics](https://community.10xgenomics.com/t5/10x-Blog/Our-1-3-million-single-cell-dataset-is-ready-to-download/ba-p/276)  
@@ -0,0 +1,94 @@
+---
+layout: default
+title: FAQ
+description: Frequently Asked Questions
+---
+
+# Data formatting
+
+#### What file formats can I use with _cellxgene_?
+
+Currently, you can go straight into `cellxgene launch` with your own analyzed data in h5ad format, after you have performed dimenstionality reduction (tsne, umap) and clustering (louvain).
+
+If your data is in a different format, and/or you still need to perform dimensionality reduction and clustering, `cellxgene` can do that for you with the `prepare` command. `cellxgene prepare` runs `scanpy` under the hood and can read in any format that is currently supported by `scanpy` (including mtx, loom, and more listed [here](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)).
+
+The output of `cellxgene prepare` is a h5ad file with your computed clusters and tsne/umap projections that can be used in `cellxgene launch`.
+
+#### I have a directory of 10X-Genomics data with _mtx_ files and I've never used _scanpy_, can I use _cellxgene_?
+
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+```
+
+Depending on the size of the dataset, this may take some time. Once it's done, call
+
+```
+cellxgene launch data-processed.h5ad --layout=umap --open
+```
+
+And your web browser should open with an interactive view of your data.
+
+#### I have extra metadata that I want to add to my dataset
+
+Currently this is not supported directly, but you should be able to do this yourself using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
+
+#### What part of the _anndata_ objects does cellxgene pull in for visualization?
+
+- `.obs` and `.var` annotations are use to extract metadata for filtering
+- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
+- `.obsm` is used for layout
+
+# Algorithms
+
+#### How are you computing and sorting differential expression results?
+
+We use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`.
+
+# Problems, errors, & bugs
+
+#### How do I create a Python 3.6 environment for _cellxgene_?
+
+If you use conda and want to create a [conda environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for _cellxgene_ you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.6
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3.6 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+#### In my _prepare_ command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
+
+Louvain clustering requires additional dependencies, so we don't include them by default. For now, you need to specify that you want these packages by using
+
+```
+pip install cellxgene[louvain]
+```
+
+#### I ran _prepare_ and I'm getting results that look unexpected
+
+You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
+```
+
+It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
+
+#### I tried to `pip install cellxgene` and got a weird error I don't understand
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.
+
+#### I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
+
+This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
@@ -0,0 +1,37 @@
+_cellxgene_ is an interactive data explorer for single-cell transcriptomics data. Whether you need to visualize one thousand cells or one million, _cellxgene_ helps you gain insight into your single-cell data.
+
+## features
+
+#### Flexible selections, coloring, and differential expression of your selected sets of cells
+<img src="diffexp.gif" width="600"/>
+
+#### Single-gene analyses (e.g. expression analysis)
+<img src="customGene.gif" width="600" />
+
+## getting started
+
+_cellxgene_ **only** supports Python 3.6. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
+
+Install the package.
+``` bash
+pip install cellxgene
+```
+
+Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
+
+``` bash
+curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
+```
+
+Launch _cellxgene_
+``` bash
+cellxgene launch pbmc3k.h5ad
+```
+
+## getting help
+
+We'd love to hear from you!
+
+For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cziscience-slack.herokuapp.com/) and say "hi!".
+
+For any errors, [report bugs on Github](https://github.com/chanzuckerberg/cellxgene/issues).
@@ -1,48 +0,0 @@
-# cellxgene release process
-
-_This document defines the release process for cellxgene_
-
-## Overview
-
-The goal of the release process is to publish an installable package
-to PyPi, with a matching tagged release on github.
-
-The release process should result in the following side-effects:
-
- Version number bump, using semantic versioning
- JS assets built & packaged, committed to the repo
- Tagged github release
- Publication to PyPi
-
-## Process
-
-Follow these steps to create a release.
-
-1.  Preparation:
-    - Define the release version number, using [semantic versioning](https://semver.org/)
-    - Write the release title and release notes
-2.  Create a release branch, eg, `release-version`
-3.  In the release branch:
-    - run `bumpversion --config-file .bumpversion.cfg [major | minor | patch]`
-    - build the JS asserts using `bin/build-client`
-4.  Commit and push the new branch
-5.  Create a PR for the release.
-    - [optional] As needed, conduct PR review.
-6.  Create Github release using the version number and release notes ([instructions](https://help.github.com/articles/creating-releases/)).
-7.  Publish to pypi by performing the following steps
-    (assumes you have `setuptools` and `twine` installed and that you have
-    registered for pypi and have write access to the cellxgene pypi package)
-    - build the distribution by calling
-      `python setup.py sdist`
-      inside the top-level directory
-    - [optional] upload the package to test pypi
-      `twine upload --repository-url https://test.pypi.org/legacy/ dist/*`
-    - [optional] test the test installation in a fresh virtual environment using
-      `pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene`
-    - upload the package to real pypi using `twine upload dist/*`
-    - [optional] test the installation in a fresh virtual environment using
-      `pip install cellxgene`
-
-The optional steps are for testing purposes, and are recommended
-for publishing any major releases, and any releases that significantly
-change the packaging (e.g. new bundled files, new dependencies, etc.)
@@ -0,0 +1,73 @@
+
+/*
+
+  Flatbuffers schema for use in cellxgene wire-format.
+
+  Schema defines a general purpose, polymorphic, 2D matrix.  Data is
+  organized in a columnar layout.   Each column is homomorphic, and
+  several column types are supported:
+    - IEEE 32 and 64 bit floats
+    - signed and unsigned 32 bit integers
+    - JSON/UTF8 encoded array (for other types)
+
+  https://github.com/google/flatbuffers
+  http://google.github.io/flatbuffers/
+
+  NOTE: IF YOU MODIFY THIS FILE, YOU MUST RECOMPILE AND COMMIT
+  RESULTING FILES TO THE REPO:
+  * server/app/util/fbs/NetEncoding/*
+  * client/src/util/stateManager/matrix_generated.js
+
+*/
+
+namespace NetEncoding;
+
+table Float32Array {
+  data: [float32];
+}
+
+table Uint32Array {
+  data: [uint32];
+}
+
+table Int32Array {
+  data: [int32];
+}
+
+table Float64Array {
+  data: [float64];
+}
+
+table JSONEncodedArray {
+  // contains a UTF-8/JSON encoded array.  Used to store other
+  // types (or polymorphic arrays)
+  data: [uint8];
+}
+
+union TypedArray {
+  Float32Array,
+  Int32Array,
+  Uint32Array,
+  Float64Array,
+  JSONEncodedArray
+}
+
+// Extra level of indirection required because vector of union not yet supported
+table Column {
+  u: TypedArray;
+}
+
+// 2D matrix stored in columnar layout
+//
+table Matrix {
+  n_rows: uint32;     // all columns have this length
+  n_cols: uint32;     // same as columns.length
+  columns: [Column];  // length n_cols
+
+  // optional row and column index, with same length as corresponding dimension.
+  // If null, defaults to numeric index, ie, [0, n_rows) or [0, n_cols)
+  col_index: TypedArray;
+  row_index: TypedArray;
+}
+
+root_type Matrix;
@@ -0,0 +1,122 @@
+BUILDDIR := build
+CLIENTBUILD := $(BUILDDIR)/client
+SERVERBUILD := $(BUILDDIR)/server
+CLEANFILES :=  $(BUILDDIR)/ client/build dist cellxgene.egg-info
+
+PART ?= patch
+
+# BUILDING PACKAGE
+
+build : clean build-server
+	@echo "done"
+
+build-server : build-client
+	mkdir -p $(SERVERBUILD)
+	cp -r server/* $(SERVERBUILD)
+	cp -r client/build/  $(CLIENTBUILD)
+	mkdir -p $(SERVERBUILD)/app/web/static/img
+	cp $(CLIENTBUILD)/index.html $(SERVERBUILD)/app/web/templates/
+	cp -r $(CLIENTBUILD)/static $(SERVERBUILD)/app/web/
+	cp $(CLIENTBUILD)/favicon.png $(SERVERBUILD)/app/web/static/img
+	cp $(CLIENTBUILD)/service-worker.js $(SERVERBUILD)/app/web/static/js/
+	cp MANIFEST.in README.md setup.cfg setup.py $(BUILDDIR)
+
+build-client :
+	npm install --prefix client/ client
+	npm run  --prefix client build
+
+# If you are actively developing in the server folder use this, dirties the source tree
+build-for-server-dev : clean-server build-client
+	mkdir -p server/app/web/static/img
+	cp client/build/index.html server/app/web/templates/
+	cp -r client/build/static server/app/web/
+	cp client/build/favicon.png server/app/web/static/img
+	cp client/build/service-worker.js server/app/web/static/js/
+
+clean : clean-lite clean-server
+	rm -rf client/node_modules
+
+# cleaning node_modules is the longest one, so we avoid that if possible
+clean-lite :
+	rm -rf $(CLEANFILES)
+
+clean-server :
+	rm -f server/app/web/templates/index.html
+	rm -rf server/app/web/static
+
+.PHONY : build build-server build-client build-for-server-dev clean clean-lite clean-server
+
+# CREATING DISTRIBUTION RELEASE
+
+pydist : build
+	cd $(BUILDDIR); python setup.py sdist -d ../dist
+	@echo "done"
+
+.PHONY : pydist
+
+# RELEASE HELPERS
+
+# create new version to commit to master
+release-stage-1 : dev-env bump clean-lite gen-package-lock
+	@echo "Version bumped part:$(PART) and client built. Ready to commit and push"
+
+# build dist and release to dev pypi
+release-stage-2 : dev-env pydist twine
+	@echo "Dist built and uploaded to test.pypi.org"
+	@echo "Test the install `make install-release-test` and then upload to Pypi prod"
+	@echo "`make twine-prod`"
+
+release-stage-final: twine-prod
+	@echo "Release uploaded to pypi.org"
+
+# DANGER: releases directly to prod
+# use this if you accidently burned a test release version number,
+release-burned : dev-env pydist twine-prod
+	@echo "Dist built and uploaded to pypi.org"
+	@echo "Test the install `make install-release`"
+
+dev-env :
+	pip install -r server/requirements-dev.txt
+
+# give PART=[major, minor, part] as param to make bump
+bump :
+	bumpversion --config-file .bumpversion.cfg $(PART)
+
+twine :
+	twine upload --repository-url https://test.pypi.org/legacy/ dist/*
+
+twine-prod :
+	twine upload dist/*
+
+# quicker than re-building client
+gen-package-lock :
+	npm install --prefix client/ client
+
+.PHONY : release-stage-1 release-stage-2 release-stage-final release-burned dev-env bump twine twine-prod gen-package-lock
+
+# INSTALL
+
+# setup.py sucks when you have your library in a separate folder, adding these in to help setup envs
+
+# install from build directory
+install : uninstall
+	cd $(BUILDDIR); pip install -e .
+
+# install from source tree for development
+install-dev : uninstall
+	pip install -e .
+
+# install from test.pypi to test your release
+install-release-test : uninstall
+	pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
+	@echo "Installed cellxgene from test.pypi.org, now run and smoke test"
+
+# install from pypi to test your release
+install-release : uninstall
+	pip install cellxgene
+	@echo "Installed cellxgene from pypi.org"
+
+uninstall :
+	yes | pip uninstall cellxgene || true
+
+.PHONY : install install-dev install-release-test install-release uninstall
@@ -2,11 +2,14 @@
 if __package__ is None:
    import sys
    from pathlib import Path
+
    PKG_PATH = Path(__file__).parent
    sys.path.insert(0, str(PKG_PATH.parent))
-    import server
+    import server # noqa F401
+
    __package__ = PKG_PATH.name

 # Main thing
-from .cli.cli import cli
+from .cli.cli import cli # noqa F402
+
 cli()
@@ -14,16 +14,14 @@ REACTIVE_LIMIT = 1_000_000

 app = Flask(__name__, static_folder="web/static")
 app.json_encoder = Float32JSONEncoder
-cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860000})
+cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860_000})
 Compress(app)
 CORS(app)

 # Config
 SECRET_KEY = os.environ.get("CXG_SECRET_KEY", default="SparkleAndShine")

-app.config.update(
-    SECRET_KEY=SECRET_KEY,
-)
+app.config.update(SECRET_KEY=SECRET_KEY)

 # Application Data
 data = None
@@ -36,7 +34,13 @@ docs.append(resources.get_swagger_doc())
 app.register_blueprint(webapp.bp)
 app.register_blueprint(resources.blueprint)
 app.register_blueprint(
-    get_swagger_blueprint(docs, "/api/swagger", produces=["application/json"], title="cellxgene rest api",
-                          description="An API connecting ExpressionMatrix2 clustering algorithm to cellxgene"))
+    get_swagger_blueprint(
+        docs,
+        "/api/swagger",
+        produces=["application/json"],
+        title="cellxgene rest api",
+        description="An API connecting ExpressionMatrix2 clustering algorithm to cellxgene",
+    )
+)

 app.add_url_rule("/", endpoint="index")
@@ -11,23 +11,20 @@ Sort order for methods


 class CXGDriver(metaclass=ABCMeta):
-
    def __init__(self, data, args):
        self.data = self._load_data(data)
        self.layout_method = args["layout"]
        self.diffexp_method = args["diffexp"]
        self.max_category_items = args["max_category_items"]
+        self.diffexp_lfc_cutoff = args["diffexp_lfc_cutoff"]
        self.cluster = None

    @property
    def features(self):
        features = {
            "cluster": {"available": False},
-            "layout": {
-                "obs": {"available": False},
-                "var": {"available": False},
-            },
-            "diffexp": {"available": False}
+            "layout": {"obs": {"available": False}, "var": {"available": False}},
+            "diffexp": {"available": False},
        }
        # TODO - Interactive limit should be generated from the actual available methods see GH issue #94
        if self.layout_method:
@@ -69,6 +66,11 @@ class CXGDriver(metaclass=ABCMeta):
        """
        pass

+    @abstractmethod
+    def annotation_to_fbs_matrix(self, axis, field=None):
+        """ Same as annotation(), except returns a flatbuffer, and does not support filtering. """
+        pass
+
    @abstractmethod
    def data_frame(self, filter, axis):
        """
@@ -82,6 +84,10 @@ class CXGDriver(metaclass=ABCMeta):
        """
        pass

+    @abstractmethod
+    def data_frame_to_fbs_matrix(self, filter, axis):
+        pass
+
    @abstractmethod
    def diffexp_topN(self, obsFilter1, obsFilter2, top_n=None, interactive_limit=None):
        """
@@ -107,3 +113,8 @@ class CXGDriver(metaclass=ABCMeta):
        :return:  [cellid, x, y, ...]
        """
        pass
+
+    @abstractmethod
+    def layout_to_fbs_matrix(self, filter):
+        """ same as layout, except returns a flatbuffer """
+        pass
@@ -1,4 +1,3 @@
-
 import numpy as np
 from scipy import sparse, stats

@@ -10,74 +9,114 @@ def _mean_var_n(X):
    than naive methods (and same method used by numpy.var())
    https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass
    """
-    n = X.shape[0]
-    if sparse.issparse(X):
-        mean = X.mean(axis=0).A1
-        dfm = X - mean
-        sumsq = np.sum(np.multiply(dfm, dfm), axis=0).A1
-        v = sumsq / (n - 1)
-    else:
-        mean = X.mean(axis=0)
-        dfm = X - mean
-        sumsq = np.sum(np.multiply(dfm, dfm), axis=0)
-        v = sumsq / (n - 1)
+    # fp_err_occurred is a flag indicating that a floating point error
+    # occured somewhere in our compute.  Used to trigger non-finite
+    # number handling.
+    fp_err_occurred = False

+    def fp_err_set(err, flag):
+        nonlocal fp_err_occurred
+        fp_err_occurred = True
+
+    with np.errstate(divide="call", invalid="call", call=fp_err_set):
+        n = X.shape[0]
+        if sparse.issparse(X):
+            mean = X.mean(axis=0).A1
+            dfm = X - mean
+            sumsq = np.sum(np.multiply(dfm, dfm), axis=0).A1
+            v = sumsq / (n - 1)
+        else:
+            mean = X.mean(axis=0)
+            dfm = X - mean
+            sumsq = np.sum(np.multiply(dfm, dfm), axis=0)
+            v = sumsq / (n - 1)
+
+    if fp_err_occurred:
+        mean[np.isfinite(mean) == False] = 0    # noqa: E712
+        v[np.isfinite(v) == False] = 0          # noqa: E712
    return mean, v, n


-def diffexp_ttest(adata, maskA, maskB, top_n=8):
+def diffexp_ttest(adata, maskA, maskB, top_n=8, diffexp_lfc_cutoff=0.01):
    """
-    Return differential expression statistics for top N variables, sorted by
-    t statistic.   Implemented as a unequal variance t-test.
+    Return differential expression statistics for top N variables.
+
+    Algorithm:
+    - compute log fold change (log2(meanA/meanB))
+    - compute Welch's t-test statistic and pvalue (w/ Bonferroni correction)
+    - return top N abs(logfoldchange) where lfc > diffexp_lfc_cutoff
+
+    If there are not N which meet criteria, augment by removing the logfoldchange
+    threshold requirement.
+
+    Notes on alogrithm:
+    - Welch's ttest provides basic statistics test.
+      https://en.wikipedia.org/wiki/Welch%27s_t-test
+    - p-values adjusted with Bonferroni correction.
+      https://en.wikipedia.org/wiki/Bonferroni_correction

    :param adata: anndata dataframe
    :param maskA: observation selection mask for set 1
    :param maskB: observation selection mask for set 2
    :param top_n: number of variables to return stats for
+    :param diffexp_lfc_cutoff: minimum
    :return:  for top N genes, [ varindex, logfoldchange, pval, pval_adj ]
    """

-    # mean, variance, N
+    if top_n > adata.n_obs:
+        top_n = adata.n_obs
+
+    # mean, variance, N - calculate for both selections
    meanA, vA, nA = _mean_var_n(adata._X[maskA])
    meanB, vB, nB = _mean_var_n(adata._X[maskB])

    # variance / N
-    vnA = vA / nA
-    vnB = vB / nB
+    vnA = vA / min(nA, nB)  # overestimate variance, would normally be nA
+    vnB = vB / min(nA, nB)  # overestimate variance, would normally be nB
    sum_vn = vnA + vnB

    # degrees of freedom for Welch's t-test
-    with np.errstate(divide='ignore', invalid='ignore'):
-        dof = sum_vn**2 / (vnA**2 / (nA - 1) + vnB**2 / (nB - 1))
+    with np.errstate(divide="ignore", invalid="ignore"):
+        dof = sum_vn ** 2 / (vnA ** 2 / (nA - 1) + vnB ** 2 / (nB - 1))
    dof[np.isnan(dof)] = 1

    # Welch's t-test score calculation
-    with np.errstate(divide='ignore', invalid='ignore'):
+    with np.errstate(divide="ignore", invalid="ignore"):
        tscores = (meanA - meanB) / np.sqrt(sum_vn)
    tscores[np.isnan(tscores)] = 0

    # p-value
    pvals = stats.t.sf(np.abs(tscores), dof) * 2
    pvals_adj = pvals * adata._X.shape[1]
+    pvals_adj[pvals_adj > 1] = 1  # cap adjusted p-value at 1

    # logfoldchanges: log2(meanA / meanB)
    logfoldchanges = np.log2(np.abs((meanA + 1e-9) / (meanB + 1e-9)))

-    # top n sort
+    # find all with lfc > cutoff
+    lfc_above_cutoff_idx = np.nonzero(np.abs(logfoldchanges) > diffexp_lfc_cutoff)[0]
    stats_to_sort = np.abs(tscores)
-    partition = np.argpartition(stats_to_sort, -top_n)[-top_n:]
-    rel_sort_order = np.argsort(stats_to_sort[partition])[::-1]
-    vars_indices = np.arange(adata.n_vars, dtype=int)
-    sort_order = vars_indices[partition][rel_sort_order]

-    # top n slice
+    # derive sort order
+    if lfc_above_cutoff_idx.shape[0] > top_n:
+        # partition top N
+        rel_t_partition = np.argpartition(stats_to_sort[lfc_above_cutoff_idx], -top_n)[-top_n:]
+        t_partition = lfc_above_cutoff_idx[rel_t_partition]
+        # sort the top N partition
+        rel_sort_order = np.argsort(stats_to_sort[t_partition])[::-1]
+        sort_order = t_partition[rel_sort_order]
+    else:
+        # partition and sort top N, ignoring lfc cutoff
+        partition = np.argpartition(stats_to_sort, -top_n)[-top_n:]
+        rel_sort_order = np.argsort(stats_to_sort[partition])[::-1]
+        indices = np.indices(stats_to_sort.shape)[0]
+        sort_order = indices[partition][rel_sort_order]
+
+    # top n slice based upon sort order
    logfoldchanges_top_n = logfoldchanges[sort_order]
    pvals_top_n = pvals[sort_order]
    pvals_adj_top_n = pvals_adj[sort_order]

    # varIndex, logfoldchange, pval, pval_adj
-    result = [[sort_order[i],
-               logfoldchanges_top_n[i],
-               pvals_top_n[i],
-               pvals_adj_top_n[i]] for i in range(top_n)]
+    result = [[sort_order[i], logfoldchanges_top_n[i], pvals_top_n[i], pvals_adj_top_n[i]] for i in range(top_n)]
    return result
@@ -8,8 +8,16 @@ from scipy import sparse

 from server.app.driver.driver import CXGDriver
 from server.app.util.constants import Axis, DEFAULT_TOP_N
-from server.app.util.errors import FilterError, InteractiveError, PrepareError, ScanpyFileError
+from server.app.util.errors import (
+    FilterError,
+    InteractiveError,
+    JSONEncodingValueError,
+    PrepareError,
+    ScanpyFileError,
+)
+from server.app.util.utils import jsonify_scanpy
 from server.app.scanpy_engine.diffexp import diffexp_ttest
+from server.app.util.fbs.matrix import encode_matrix_fbs

 """
 Sort order for methods
@@ -22,7 +30,6 @@ Sort order for methods


 class ScanpyEngine(CXGDriver):
-
    def __init__(self, data, args):
        super().__init__(data, args)
        self._alias_annotation_names(Axis.OBS, args["obs_names"])
@@ -55,46 +62,73 @@ class ScanpyEngine(CXGDriver):
            df_axis.rename(inplace=True, columns={"index": "name"})
        elif name in df_axis.columns:
            if name not in df_axis.columns:
-                raise KeyError(f"Annotation name {name}, specified in --{ax_name}-name does not exist.")
+                raise KeyError(
+                    f"Annotation name {name}, specified in --{ax_name}-name does not exist."
+                )
            if not df_axis[name].is_unique:
-                raise KeyError(f"Values in -{ax_name}-name must be unique. "
-                               "Please prepare data to contain unique values.")
+                raise KeyError(
+                    f"Values in -{ax_name}-name must be unique. "
+                    "Please prepare data to contain unique values."
+                )
            # reset index to simple range; alias user-specified annotation to "name"
            df_axis.reset_index(drop=True, inplace=True)
            df_axis.rename(inplace=True, columns={name: "name"})
        else:
-            raise KeyError(f"Annotation name {name}, specified in --{ax_name}_name does not exist.")
+            raise KeyError(
+                f"Annotation name {name}, specified in --{ax_name}_name does not exist."
+            )
+
+    @staticmethod
+    def _can_cast_to_float32(ann):
+        if ann.dtype.kind == "f":
+            if not np.can_cast(ann.dtype, np.float32):
+                warnings.warn(
+                    f"Annotation {ann.name} will be converted to 32 bit float and may lose precision."
+                )
+            return True
+        return False
+
+    @staticmethod
+    def _can_cast_to_int32(ann):
+        if ann.dtype.kind in ["i", "u"]:
+            if np.can_cast(ann.dtype, np.int32):
+                return True
+            ii32 = np.iinfo(np.int32)
+            if ann.min() >= ii32.min and ann.max() <= ii32.max:
+                return True
+        return False

    def _create_schema(self):
        self.schema = {
            "dataframe": {
                "nObs": self.cell_count,
                "nVar": self.gene_count,
-                "type": str(self.data.X.dtype)
+                "type": str(self.data.X.dtype),
            },
-            "annotations": {
-                "obs": [],
-                "var": []
-            }
+            "annotations": {"obs": [], "var": []},
        }
        for ax in Axis:
            curr_axis = getattr(self.data, str(ax))
            for ann in curr_axis:
                ann_schema = {"name": ann}
-                data_kind = curr_axis[ann].dtype.kind
-                if data_kind == "f":
+                dtype = curr_axis[ann].dtype
+                data_kind = dtype.kind
+
+                if self._can_cast_to_float32(curr_axis[ann]):
                    ann_schema["type"] = "float32"
-                elif data_kind in ["i", "u"]:
+                elif self._can_cast_to_int32(curr_axis[ann]):
                    ann_schema["type"] = "int32"
-                elif data_kind == "?":
+                elif dtype == np.bool_:
                    ann_schema["type"] = "boolean"
-                elif data_kind == "O" and curr_axis[ann].dtype == "object":
+                elif data_kind == "O" and dtype == "object":
                    ann_schema["type"] = "string"
-                elif data_kind == "O" and curr_axis[ann].dtype == "category":
+                elif data_kind == "O" and dtype == "category":
                    ann_schema["type"] = "categorical"
                    ann_schema["categories"] = curr_axis[ann].dtype.categories.tolist()
                else:
-                    raise TypeError(f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene.")
+                    raise TypeError(
+                        f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene."
+                    )
                self.schema["annotations"][ax].append(ann_schema)

    @staticmethod
@@ -105,32 +139,41 @@ class ScanpyEngine(CXGDriver):
        try:
            result = sc.read(data, cache=True)
        except ValueError:
-            raise ScanpyFileError("File must be in the .h5ad format. Please read "
-                                  "https://github.com/theislab/scanpy_usage/blob/master/170505_seurat/info_h5ad.md to "
-                                  "learn more about this format. You may be able to convert your file into this format "
-                                  "using `cellxgene prepare`, please run `cellxgene prepare --help` for more "
-                                  "information.")
+            raise ScanpyFileError(
+                "File must be in the .h5ad format. Please read "
+                "https://github.com/theislab/scanpy_usage/blob/master/170505_seurat/info_h5ad.md to "
+                "learn more about this format. You may be able to convert your file into this format "
+                "using `cellxgene prepare`, please run `cellxgene prepare --help` for more "
+                "information."
+            )
        except Exception as e:
-            raise ScanpyFileError(f"Error while loading file: {e}, File must be in the .h5ad format, please check "
-                                  f"that your input and try again.")
+            raise ScanpyFileError(
+                f"Error while loading file: {e}, File must be in the .h5ad format, please check "
+                f"that your input and try again."
+            )
        return result

    def _validate_data_types(self):
        if self.data.X.dtype != "float32":
-            warnings.warn(f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
-                          f"Precision may be truncated.")
+            warnings.warn(
+                f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
+                f"Precision may be truncated."
+            )
        for ax in Axis:
            curr_axis = getattr(self.data, str(ax))
            for ann in curr_axis:
                datatype = curr_axis[ann].dtype
-                downcast_map = {"int64": "int32",
-                                "uint32": "int32",
-                                "uint64": "int32",
-                                "float64": "float32",
-                                }
+                downcast_map = {
+                    "int64": "int32",
+                    "uint32": "int32",
+                    "uint64": "int32",
+                    "float64": "float32",
+                }
                if datatype in downcast_map:
-                    warnings.warn(f"Scanpy annotation {ax}:{ann} is in unsupported format: {datatype}. "
-                                  f"Data will be downcast to {downcast_map[datatype]}.")
+                    warnings.warn(
+                        f"Scanpy annotation {ax}:{ann} is in unsupported format: {datatype}. "
+                        f"Data will be downcast to {downcast_map[datatype]}."
+                    )
                if isinstance(datatype, CategoricalDtype):
                    category_num = len(curr_axis[ann].dtype.categories)
                    if category_num > 500 and category_num > self.max_category_items:
@@ -138,7 +181,8 @@ class ScanpyEngine(CXGDriver):
                            f"{str(ax).title()} annotation '{ann}' has {category_num} categories, this may be "
                            f"cumbersome or slow to display. We recommend setting the "
                            f"--max-category-items option to 500, this will hide categorical "
-                            f"annotations with more than 500 categories in the UI")
+                            f"annotations with more than 500 categories in the UI"
+                        )

    def _validate_data_calculations(self):
        layout_key = f"X_{self.layout_method}"
@@ -150,7 +194,8 @@ class ScanpyEngine(CXGDriver):
                f" layout may have been computed. The requested layout must be pre-calculated and saved "
                f"back in the h5ad file. You can run "
                f"`cellxgene prepare --layout {self.layout_method} <datafile>` "
-                f"to solve this problem. ")
+                f"to solve this problem. "
+            )

    def filter_dataframe(self, filter):
        """
@@ -171,7 +216,7 @@ class ScanpyEngine(CXGDriver):

    @staticmethod
    def _annotation_filter_to_mask(filter, d_axis, count):
-        mask = np.ones((count, ), dtype=bool)
+        mask = np.ones((count,), dtype=bool)
        for v in filter:
            if d_axis[v["name"]].dtype.name in ["boolean", "category", "object"]:
                key_idx = np.in1d(getattr(d_axis, v["name"]), v["values"])
@@ -189,24 +234,28 @@ class ScanpyEngine(CXGDriver):

    @staticmethod
    def _index_filter_to_mask(filter, count):
-        mask = np.zeros((count, ), dtype=bool)
+        mask = np.zeros((count,), dtype=bool)
        for i in filter:
            if type(i) == list:
-                mask[i[0]:i[1]] = True
+                mask[i[0] : i[1]] = True
            else:
                mask[i] = True
        return mask

    @staticmethod
    def _axis_filter_to_mask(filter, d_axis, count):
-        mask = np.ones((count, ), dtype=bool)
+        mask = np.ones((count,), dtype=bool)
        if "index" in filter:
-            mask = np.logical_and(mask, ScanpyEngine._index_filter_to_mask(filter["index"], count))
+            mask = np.logical_and(
+                mask, ScanpyEngine._index_filter_to_mask(filter["index"], count)
+            )
        if "annotation_value" in filter:
-            mask = np.logical_and(mask,
-                                  ScanpyEngine._annotation_filter_to_mask(filter["annotation_value"],
-                                                                          d_axis,
-                                                                          count))
+            mask = np.logical_and(
+                mask,
+                ScanpyEngine._annotation_filter_to_mask(
+                    filter["annotation_value"], d_axis, count
+                ),
+            )
        return mask

    def _filter_to_mask(self, filter, use_slices=True):
@@ -219,9 +268,13 @@ class ScanpyEngine(CXGDriver):

        if filter is not None:
            if Axis.OBS in filter:
-                obs_selector = self._axis_filter_to_mask(filter["obs"], self.data.obs, self.data.n_obs)
+                obs_selector = self._axis_filter_to_mask(
+                    filter["obs"], self.data.obs, self.data.n_obs
+                )
            if Axis.VAR in filter:
-                var_selector = self._axis_filter_to_mask(filter["var"], self.data.var, self.data.n_vars)
+                var_selector = self._axis_filter_to_mask(
+                    filter["var"], self.data.var, self.data.n_vars
+                )
        return obs_selector, var_selector

    @staticmethod
@@ -236,8 +289,11 @@ class ScanpyEngine(CXGDriver):

        https://docs.scipy.org/doc/scipy/reference/sparse.html
        """
-        prefer_row_access = sparse.isspmatrix_csr(data._X) or sparse.isspmatrix_lil(data._X) \
+        prefer_row_access = (
+            sparse.isspmatrix_csr(data._X)
+            or sparse.isspmatrix_lil(data._X)
            or sparse.isspmatrix_bsr(data._X)
+        )
        if prefer_row_access:
            # Row-major slicing
            if obs_selector is not None:
@@ -272,7 +328,7 @@ class ScanpyEngine(CXGDriver):
                fields = obs.columns.tolist()
            result = {
                "names": fields,
-                "data": DataFrame(obs[fields]).to_records(index=True).tolist()
+                "data": DataFrame(obs[fields]).to_records(index=True).tolist(),
            }
        else:
            var = self.data.var[var_selector]
@@ -280,9 +336,21 @@ class ScanpyEngine(CXGDriver):
                fields = var.columns.tolist()
            result = {
                "names": fields,
-                "data": DataFrame(var[fields]).to_records(index=True).tolist()
+                "data": DataFrame(var[fields]).to_records(index=True).tolist(),
            }
-        return result
+        try:
+            return jsonify_scanpy(result)
+        except ValueError:
+            raise JSONEncodingValueError("Error encoding annotations to JSON")
+
+    def annotation_to_fbs_matrix(self, axis, fields=None):
+        if axis == Axis.OBS:
+            df = self.data.obs
+        else:
+            df = self.data.var
+        if fields is not None and len(fields) > 0:
+            df = df[fields]
+        return encode_matrix_fbs(df, col_idx=df.columns)

    def data_frame(self, filter, axis):
        """
@@ -306,27 +374,71 @@ class ScanpyEngine(CXGDriver):
        if axis == Axis.OBS:
            result = {
                "var": var_index_sliced.tolist(),
-                "obs": DataFrame(_X, index=obs_index_sliced).to_records(index=True).tolist()
+                "obs": DataFrame(_X, index=obs_index_sliced)
+                .to_records(index=True)
+                .tolist(),
            }
        else:
            result = {
                "obs": obs_index_sliced.tolist(),
-                "var": DataFrame(_X.T, index=var_index_sliced).to_records(index=True).tolist()
+                "var": DataFrame(_X.T, index=var_index_sliced)
+                .to_records(index=True)
+                .tolist(),
            }
-        return result
+        try:
+            return jsonify_scanpy(result)
+        except ValueError:
+            raise JSONEncodingValueError("Error encoding dataframe to JSON")
+
+    def data_frame_to_fbs_matrix(self, filter, axis):
+        """
+        Retrieves data 'X' and returns in a flatbuffer Matrix.
+        :param filter: filter: dictionary with filter params
+        :param axis: string obs or var
+        :return: flatbuffer Matrix
+
+        Caveats:
+        * currently only supports access on VAR axis
+        * currently only supports filtering on VAR axis
+        """
+        if axis != Axis.VAR:
+            raise ValueError("Only VAR dimension access is supported")
+        try:
+            obs_selector, var_selector = self._filter_to_mask(filter, use_slices=False)
+        except (KeyError, IndexError) as e:
+            raise FilterError(f"Error parsing filter: {e}") from e
+        if obs_selector is not None:
+            raise FilterError("filtering on obs unsupported")
+
+        # Currently only handles VAR dimension
+        X = self.data._X
+        if var_selector is not None:
+            X = X[:, var_selector]
+        return encode_matrix_fbs(X, col_idx=np.nonzero(var_selector)[0], row_idx=None)

    def diffexp_topN(self, obsFilterA, obsFilterB, top_n=None, interactive_limit=None):
        if Axis.VAR in obsFilterA or Axis.VAR in obsFilterB:
            raise FilterError("Observation filters may not contain vaiable conditions")
        try:
-            obs_mask_A = self._axis_filter_to_mask(obsFilterA["obs"], self.data.obs, self.data.n_obs)
-            obs_mask_B = self._axis_filter_to_mask(obsFilterB["obs"], self.data.obs, self.data.n_obs)
+            obs_mask_A = self._axis_filter_to_mask(
+                obsFilterA["obs"], self.data.obs, self.data.n_obs
+            )
+            obs_mask_B = self._axis_filter_to_mask(
+                obsFilterB["obs"], self.data.obs, self.data.n_obs
+            )
        except (KeyError, IndexError) as e:
            raise FilterError(f"Error parsing filter: {e}") from e
        if top_n is None:
            top_n = DEFAULT_TOP_N
-        result = diffexp_ttest(self.data, obs_mask_A, obs_mask_B, top_n)
-        return sorted(result, key=lambda r: r[0])
+        result = diffexp_ttest(
+            self.data, obs_mask_A, obs_mask_B, top_n, self.diffexp_lfc_cutoff
+        )
+        try:
+            return jsonify_scanpy(result)
+        except ValueError:
+            raise JSONEncodingValueError(
+                "Error encoding differential expression to JSON"
+            )

    def layout(self, filter, interactive_limit=None):
        """
@@ -350,11 +462,41 @@ class ScanpyEngine(CXGDriver):
        try:
            df_layout = df.obsm[f"X_{self.layout_method}"]
        except ValueError as e:
-            raise PrepareError(f"Layout has not been calculated using {self.layout_method}, "
-                               f"please prepare your datafile and relaunch cellxgene") from e
-        normalized_layout = DataFrame((df_layout - df_layout.min()) / (df_layout.max() - df_layout.min()),
-                                      index=df.obs.index)
-        return {
-            "ndims": normalized_layout.shape[1],
-            "coordinates": normalized_layout.to_records(index=True).tolist()
-        }
+            raise PrepareError(
+                f"Layout has not been calculated using {self.layout_method}, "
+                f"please prepare your datafile and relaunch cellxgene"
+            ) from e
+        normalized_layout = DataFrame(
+            (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min()),
+            index=df.obs.index,
+        )
+        try:
+            return jsonify_scanpy(
+                {
+                    "layout": {
+                        "ndims": normalized_layout.shape[1],
+                        "coordinates": normalized_layout.to_records(
+                            index=True
+                        ).tolist(),
+                    }
+                }
+            )
+        except ValueError:
+            raise JSONEncodingValueError("Error encoding layout to JSON")
+
+    def layout_to_fbs_matrix(self):
+        """
+        Return the default 2-D layout for cells as a FBS Matrix.
+
+        Caveats:
+        * does not support filtering
+        * only returns Matrix in columnar layout
+        """
+        try:
+            df_layout = self.data.obsm[f"X_{self.layout_method}"]
+        except ValueError as e:
+            raise PrepareError(
+                f"Layout has not been calculated using {self.layout_method}, "
+                f"please prepare your datafile and relaunch cellxgene") from e
+        normalized_layout = (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min())
+        return encode_matrix_fbs(normalized_layout.astype(dtype=np.float32), col_idx=None, row_idx=None)
@@ -25,3 +25,8 @@ class Axis(AugmentedEnum):
 class DiffExpMode(AugmentedEnum):
    TOP_N = "topN"
    VAR_FILTER = "varFilter"
+
+
+JSON_NaN_to_num_warning_msg = (
+    "JSON encoding failure - please verify all data are finite values (no NaN or Infinities)"
+)
@@ -16,6 +16,15 @@ class InteractiveError(Exception):
        self.message = message


+class JSONEncodingValueError(Exception):
+    """
+    Raised when file loaded into scanpy is misformatted
+    """
+
+    def __init__(self, message):
+        self.message = message
+
+
 class MimeTypeError(Exception):
    """
    Raised when incompatible MIME type selected
@@ -0,0 +1,41 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Column(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsColumn(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Column()
+        x.Init(buf, n + offset)
+        return x
+
+    # Column
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Column
+    def UType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Column
+    def U(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+def ColumnStart(builder): builder.StartObject(2)
+def ColumnAddUType(builder, uType): builder.PrependUint8Slot(0, uType, 0)
+def ColumnAddU(builder, u): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(u), 0)
+def ColumnEnd(builder): return builder.EndObject()
@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Float32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsFloat32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Float32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Float32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Float32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Float32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Float32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Float32Flags, o)
+        return 0
+
+    # Float32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Float32ArrayStart(builder): builder.StartObject(1)
+def Float32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Float32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Float32ArrayEnd(builder): return builder.EndObject()
@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Float64Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsFloat64Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Float64Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Float64Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Float64Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Float64Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 8))
+        return 0
+
+    # Float64Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Float64Flags, o)
+        return 0
+
+    # Float64Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Float64ArrayStart(builder): builder.StartObject(1)
+def Float64ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Float64ArrayStartDataVector(builder, numElems): return builder.StartVector(8, numElems, 8)
+def Float64ArrayEnd(builder): return builder.EndObject()
@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Int32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsInt32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Int32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Int32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Int32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Int32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Int32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Int32Flags, o)
+        return 0
+
+    # Int32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Int32ArrayStart(builder): builder.StartObject(1)
+def Int32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Int32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Int32ArrayEnd(builder): return builder.EndObject()
@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class JSONEncodedArray(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsJSONEncodedArray(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = JSONEncodedArray()
+        x.Init(buf, n + offset)
+        return x
+
+    # JSONEncodedArray
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # JSONEncodedArray
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 1))
+        return 0
+
+    # JSONEncodedArray
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Uint8Flags, o)
+        return 0
+
+    # JSONEncodedArray
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def JSONEncodedArrayStart(builder): builder.StartObject(1)
+def JSONEncodedArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def JSONEncodedArrayStartDataVector(builder, numElems): return builder.StartVector(1, numElems, 1)
+def JSONEncodedArrayEnd(builder): return builder.EndObject()
@@ -0,0 +1,98 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Matrix(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsMatrix(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Matrix()
+        x.Init(buf, n + offset)
+        return x
+
+    # Matrix
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Matrix
+    def NRows(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def NCols(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def Columns(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8))
+        if o != 0:
+            x = self._tab.Vector(o)
+            x += flatbuffers.number_types.UOffsetTFlags.py_type(j) * 4
+            x = self._tab.Indirect(x)
+            from .Column import Column
+            obj = Column()
+            obj.Init(self._tab.Bytes, x)
+            return obj
+        return None
+
+    # Matrix
+    def ColumnsLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+    # Matrix
+    def ColIndexType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(10))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def ColIndex(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+    # Matrix
+    def RowIndexType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(14))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def RowIndex(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(16))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+def MatrixStart(builder): builder.StartObject(7)
+def MatrixAddNRows(builder, nRows): builder.PrependUint32Slot(0, nRows, 0)
+def MatrixAddNCols(builder, nCols): builder.PrependUint32Slot(1, nCols, 0)
+def MatrixAddColumns(builder, columns): builder.PrependUOffsetTRelativeSlot(2, flatbuffers.number_types.UOffsetTFlags.py_type(columns), 0)
+def MatrixStartColumnsVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def MatrixAddColIndexType(builder, colIndexType): builder.PrependUint8Slot(3, colIndexType, 0)
+def MatrixAddColIndex(builder, colIndex): builder.PrependUOffsetTRelativeSlot(4, flatbuffers.number_types.UOffsetTFlags.py_type(colIndex), 0)
+def MatrixAddRowIndexType(builder, rowIndexType): builder.PrependUint8Slot(5, rowIndexType, 0)
+def MatrixAddRowIndex(builder, rowIndex): builder.PrependUOffsetTRelativeSlot(6, flatbuffers.number_types.UOffsetTFlags.py_type(rowIndex), 0)
+def MatrixEnd(builder): return builder.EndObject()
@@ -0,0 +1,12 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+class TypedArray(object):
+    NONE = 0
+    Float32Array = 1
+    Int32Array = 2
+    Uint32Array = 3
+    Float64Array = 4
+    JSONEncodedArray = 5
+
@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Uint32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsUint32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Uint32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Uint32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Uint32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Uint32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Uint32Flags, o)
+        return 0
+
+    # Uint32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Uint32ArrayStart(builder): builder.StartObject(1)
+def Uint32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Uint32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Uint32ArrayEnd(builder): return builder.EndObject()
@@ -0,0 +1,207 @@
+import flatbuffers
+import numpy as np
+from scipy import sparse
+import pandas as pd
+
+import server.app.util.fbs.NetEncoding.Column as Column
+import server.app.util.fbs.NetEncoding.TypedArray as TypedArray
+import server.app.util.fbs.NetEncoding.Matrix as Matrix
+
+
+# Placeholder until recent enhancements to flatbuffers Python
+# runtime are released, at which point we can use the default
+# version.  This code is a port of the head.  See:
+#
+#       https://github.com/google/flatbuffers/pull/4829
+#
+def CreateNumpyVector(builder, x):
+    """CreateNumpyVector writes a numpy array into the buffer."""
+
+    if not isinstance(x, np.ndarray):
+        raise TypeError("non-numpy-ndarray passed to CreateNumpyVector")
+
+    if x.dtype.kind not in ['b', 'i', 'u', 'f']:
+        raise TypeError("numpy-ndarray holds elements of unsupported datatype")
+
+    if x.ndim > 1:
+        raise TypeError("multidimensional-ndarray passed to CreateNumpyVector")
+
+    builder.StartVector(x.itemsize, x.size, x.dtype.alignment)
+
+    # Ensure little endian byte ordering
+    if x.dtype.str[0] == "<":
+        x_little_endian = x
+    else:
+        x_little_endian = x.byteswap(inplace=False)
+
+    # Calculate total length
+    len = int(x_little_endian.itemsize * x_little_endian.size)
+    builder.head = int(builder.Head() - len)
+
+    # tobytes ensures c_contiguous ordering
+    builder.Bytes[builder.Head():builder.Head() + len] = x_little_endian.tobytes(order='C')
+
+    return builder.EndVector(x.size)
+
+
+# Serialization helper
+def serialize_column(builder, typed_arr):
+    """ Serialize NetEncoding.Column """
+    (u_type, u_value) = typed_arr
+    Column.ColumnStart(builder)
+    Column.ColumnAddUType(builder, u_type)
+    Column.ColumnAddU(builder, u_value)
+    return Column.ColumnEnd(builder)
+
+
+# Serialization helper
+def serialize_matrix(builder, n_rows, n_cols, columns, col_idx):
+    """ Serialize NetEncoding.Matrix """
+    Matrix.MatrixStart(builder)
+    Matrix.MatrixAddNRows(builder, n_rows)
+    Matrix.MatrixAddNCols(builder, n_cols)
+    Matrix.MatrixAddColumns(builder, columns)
+    if col_idx is not None:
+        (u_type, u_val) = col_idx
+        Matrix.MatrixAddColIndexType(builder, u_type)
+        Matrix.MatrixAddColIndex(builder, u_val)
+    return Matrix.MatrixEnd(builder)
+
+
+# Serialization helper
+def serialize_typed_array(builder, source_array, encoding_info):
+    """
+    Serialize any of the various typed arrays, eg, Float32Array.   Specific
+    means of serialization and type conversion are provided by type_info.
+    """
+    arr = source_array
+    (array_type, as_type) = encoding_info(source_array)
+
+    if isinstance(arr, pd.Index):
+        arr = arr.to_series()
+
+    # convert to a simple ndarray
+    if as_type == 'json':
+        as_json = arr.to_json(orient='records')
+        arr = np.array(bytearray(as_json, 'utf-8'))
+    else:
+        if sparse.issparse(arr):
+            arr = arr.toarray()
+        elif isinstance(arr, pd.Series):
+            arr = arr.get_values()
+        if arr.dtype != as_type:
+            arr = arr.astype(as_type)
+
+    # serialize the ndarray into a vector
+    if arr.ndim == 2 and arr.shape[0] == 1:
+        arr = arr[0]
+    vec = CreateNumpyVector(builder, arr)
+
+    # serialize the typed array table
+    builder.StartObject(1)
+    builder.PrependUOffsetTRelativeSlot(0, vec, 0)
+    array_value = builder.EndObject()
+    return (array_type, array_value)
+
+
+def column_encoding(arr):
+    type_map = {
+        # dtype:  ( array_type, as_type )
+        np.float64: (TypedArray.TypedArray.Float32Array, np.float32),
+        np.float32: (TypedArray.TypedArray.Float32Array, np.float32),
+        np.float16: (TypedArray.TypedArray.Float32Array, np.float32),
+
+        np.int8: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int16: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int32: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int64: (TypedArray.TypedArray.Int32Array, np.int32),
+
+        np.uint8: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint16: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint32: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint64: (TypedArray.TypedArray.Uint32Array, np.uint32)
+    }
+    type_map_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
+    return type_map.get(arr.dtype.type, type_map_default)
+
+
+def index_encoding(arr):
+    type_map = {
+        # dtype:  ( array_type, as_type )
+        np.int32: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int64: (TypedArray.TypedArray.Int32Array, np.int32),
+
+        np.uint32: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint64: (TypedArray.TypedArray.Uint32Array, np.uint32)
+    }
+    type_map_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
+    return type_map.get(arr.dtype.type, type_map_default)
+
+
+def guess_at_mem_needed(matrix):
+    (n_rows, n_cols) = matrix.shape
+    if isinstance(matrix, np.ndarray) or sparse.issparse(matrix):
+        guess = (n_rows * n_cols * matrix.dtype.itemsize) + 1024
+    elif isinstance(matrix, pd.DataFrame):
+        # XXX TODO - DataFrame type estimate
+        guess = 1
+    else:
+        guess = 1
+
+    # round up to nearest 1024 bytes
+    guess = (guess + 0x400) & (~0x3ff)
+    return guess
+
+
+def encode_matrix_fbs(matrix, row_idx=None, col_idx=None):
+    """
+    Given a 2D DataFrame, ndarray or sparse equivalent, create and return a
+    Matrix flatbuffer.
+
+    :param matrix: 2D DataFrame, ndarray or sparse equivalent
+    :param row_idx: index for row dimension, Index or ndarray
+    :param col_idx: index for col dimension, Index or ndarray
+
+    NOTE: row indices are (currently) unsupported and must be None
+    """
+
+    if row_idx is not None:
+        raise ValueError("row indexing not supported for FBS Matrix")
+    if matrix.ndim != 2:
+        raise ValueError("FBS Matrix must be 2D")
+
+    (n_rows, n_cols) = matrix.shape
+
+    # estimate size needed, so we don't unnecessarily realloc.
+    builder = flatbuffers.Builder(guess_at_mem_needed(matrix))
+
+    if isinstance(matrix, pd.DataFrame):
+        matrix_columns = reversed(tuple(matrix[name] for name in matrix))
+    else:
+        matrix_columns = reversed(tuple(c for c in matrix.T))
+
+    columns = []
+    # for idx in reversed(np.arange(n_cols)):
+    for c in matrix_columns:
+        # serialize the typed array
+        typed_arr = serialize_typed_array(builder, c, column_encoding)
+
+        # serialize the Column union
+        columns.append(serialize_column(builder, typed_arr))
+
+    # Serialize Matrix.columns[]
+    Matrix.MatrixStartColumnsVector(builder, n_cols)
+    for c in columns:
+        builder.PrependUOffsetTRelative(c)
+    matrix_column_vec = builder.EndVector(n_cols)
+
+    # serialize the colIndex if provided
+    cidx = None
+    if col_idx is not None:
+        cidx = serialize_typed_array(builder, col_idx, index_encoding)
+
+    # Serialize Matrix
+    matrix = serialize_matrix(builder, n_rows, n_cols, matrix_column_vec, cidx)
+
+    builder.Finish(matrix)
+    return builder.Output()
@@ -7,7 +7,6 @@ from server.app.util.constants import Axis


 class QueryStringError(Exception):
-
    def __init__(self, key, message):
        self.key = key
        self.message = message
@@ -5,24 +5,13 @@ class AnnotationModel(Schema):
    type = "object"
    description = "Filter by annotation key: value"
    properties = {
-        "name": {
-            "type": "string"
-        },
+        "name": {"type": "string"},
        # TODO update to OpenAPI v3.0 when a library is available that supports it
        # Unfortunately 2.0 doesn't have a way to have a schema that accepts multiple types
        # Overloading the type key with a list seems to work ok and makes it to the page
-        "values": {
-            "type": "array",
-            "items": {
-                "type": ["float32", "string", "int32", "bool"]
-            }
-        },
-        "min": {
-            "type": ["int32", "float32"],
-        },
-        "max": {
-            "type": ["int32", "float32"],
-        }
+        "values": {"type": "array", "items": {"type": ["float32", "string", "int32", "bool"]}},
+        "min": {"type": ["int32", "float32"]},
+        "max": {"type": ["int32", "float32"]},
    }
    required = ["name"]

@@ -30,36 +19,16 @@ class AnnotationModel(Schema):
 class IndexModel(Schema):
    type = "object"
    description = "Filter by index of observation/variable ex. [0, 5, 15]"
-    properties = {
-        "index": {
-            "type": "array",
-            "items": {
-                "format": "int32",
-                "type": "integer"
-            }
-
-        }
-    }
+    properties = {"index": {"type": "array", "items": {"format": "int32", "type": "integer"}}}


 class AxisModel(Schema):
    type = "object"
    description = "Axis of data -- obs or var"
-    properties = {
-        "index": IndexModel,
-        "annotation_value": AnnotationModel.array()
-    }
+    properties = {"index": IndexModel, "annotation_value": AnnotationModel.array()}


 class FilterModel(Schema):
    type = "object"
    description = "Complex filter"
-    properties = {
-        "filter": {
-            "type": "object",
-            "properties": {
-                "obs": AxisModel,
-                "var": AxisModel
-            }
-        }
-    }
+    properties = {"filter": {"type": "object", "properties": {"obs": AxisModel, "var": AxisModel}}}
@@ -7,6 +7,17 @@ from server.app.util.errors import MimeTypeError


 class Float32JSONEncoder(json.JSONEncoder):
+    def __init__(self, *args, **kwargs):
+        """
+        NaN/Infinities are illegal in standard JSON.  Python extends JSON with
+        non-standard symbols that most JavaScript JSON parsers do not understand.
+        The `allow_nan` parameter will force Python simplejson to throw an ValueError
+        if it runs into non-finite floating point values which are unsupported by
+        standard JSON.
+        """
+        kwargs["allow_nan"] = False
+        super().__init__(*args, **kwargs)
+
    def default(self, obj):
        if isinstance(obj, float32):
            return float(obj)
@@ -15,8 +26,13 @@ class Float32JSONEncoder(json.JSONEncoder):
        return json.JSONEncoder.default(self, obj)


-def get_mime_type(default="application/json", acceptable_types=["application/json", "text/csv"], query_param=None,
-                  header=None):
+def custom_format_warning(msg, *args, **kwargs):
+    return f"[cellxgene] Warning: {msg} \n"
+
+
+def get_mime_type(
+    default="application/json", acceptable_types=["application/json", "text/csv"], query_param=None, header=None
+):
    mime_type = default
    if query_param:
        if query_param in acceptable_types:
@@ -38,3 +54,7 @@ def whole_number(value):
    if value < 0:
        raise ArgumentTypeError(f"{value} is not >= 0")
    return value
+
+
+def jsonify_scanpy(data):
+    return json.dumps(data, cls=Float32JSONEncoder, allow_nan=False)
@@ -73,7 +73,7 @@
 <script>
 	window.onload = function () {
 		const ui = SwaggerUIBundle({
-			url: window.location.origin + "/api/swagger.json",
+			url: window.location.href.replace(/\/swagger$/, "") + "/api/swagger.json",
 			dom_id: '#swagger-ui',
 			deepLinking: true,
 			presets: [
@@ -1,7 +1,5 @@
 import os
-from flask import (
-    Blueprint, render_template, send_from_directory, current_app
-)
+from flask import Blueprint, render_template, send_from_directory, current_app


 bp = Blueprint("webapp", __name__, template_folder="templates")
@@ -9,9 +7,8 @@ bp = Blueprint("webapp", __name__, template_folder="templates")

@bp.route("/")
 def index():
-    url_base = current_app.config["CXG_API_BASE"]
    dataset_title = current_app.config["DATASET_TITLE"]
-    return render_template("index.html", prefix=url_base, datasetTitle=dataset_title)
+    return render_template("index.html", datasetTitle=dataset_title)


 # renders swagger documentation
@@ -5,7 +5,7 @@ from .prepare import prepare


@click.group(name="cellxgene", context_settings=dict(max_content_width=85))
-@click.version_option(version="0.0.2", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
+@click.version_option(version="0.6.1", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
 def cli():
    pass

@@ -1,34 +1,80 @@
-import sys
-import click
 import logging
+from os import devnull
 from os.path import splitext, basename
+import sys
+import warnings
 import webbrowser

+import click
+
 from server.app.util.errors import ScanpyFileError
+from server.app.util.utils import custom_format_warning


@click.command()
@click.argument("data", metavar="<data file>", type=click.Path(exists=True, file_okay=True, dir_okay=False))
-@click.option("--layout", "-l", type=click.Choice(["umap", "tsne"]), default="umap", show_default=True,
-              help="Method for layout.")
-@click.option("--diffexp", "-d", type=click.Choice(["ttest"]), default="ttest", show_default=True,
-              help="Method for differential expression.")
+@click.option(
+    "--layout", "-l", type=click.Choice(["umap", "tsne"]), default="umap", show_default=True, help="Method for layout."
+)
+@click.option(
+    "--diffexp",
+    "-d",
+    type=click.Choice(["ttest"]),
+    default="ttest",
+    show_default=True,
+    help="Method for differential expression.",
+)
@click.option("--title", "-t", help="Title to display (if omitted will use file name).", metavar="")
-@click.option("--verbose", "-v", is_flag=True, default=False, show_default=True,
-              help="Provide verbose output, including warnings and all server requests.")
-@click.option("--debug", "-d", is_flag=True, default=False, show_default=True,
-              help="Run in debug mode.")
-@click.option("--open", "-o", "open_browser", is_flag=True, default=False, show_default=True,
-              help="Open the web browser after launch.")
+@click.option(
+    "--verbose",
+    "-v",
+    is_flag=True,
+    default=False,
+    show_default=True,
+    help="Provide verbose output, including warnings and all server requests.",
+)
+@click.option("--debug", is_flag=True, default=False, show_default=True, help="Run in debug mode.")
+@click.option(
+    "--open",
+    "-o",
+    "open_browser",
+    is_flag=True,
+    default=False,
+    show_default=True,
+    help="Open the web browser after launch.",
+)
@click.option("--port", "-p", help="Port to run server on.", metavar="", default=5005, show_default=True)
@click.option("--obs-names", default=None, metavar="", help="Name of annotation field to use for observations.")
@click.option("--var-names", default=None, metavar="", help="Name of annotation to use for variables.")
-@click.option("--listen-all", is_flag=True, default=False, show_default=True,
-              help="Bind to all interfaces (this makes the server accessible beyond this computer).")
-@click.option("--max-category-items", default=100, metavar="", show_default=True,
-              help="Limits the number of categorical annotation items displayed.")
-def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
-           open_browser, port, listen_all, max_category_items):
+@click.option("--host", default="127.0.0.1", help="Host IP address")
+@click.option(
+    "--max-category-items",
+    default=100,
+    metavar="",
+    show_default=True,
+    help="Limits the number of categorical annotation items displayed.",
+)
+@click.option(
+    "--diffexp-lfc-cutoff",
+    default=0.01,
+    show_default=True,
+    help="Relative expression cutoff used when selecting top N differentially expressed genes",
+)
+def launch(
+    data,
+    layout,
+    diffexp,
+    title,
+    verbose,
+    debug,
+    obs_names,
+    var_names,
+    open_browser,
+    port,
+    host,
+    max_category_items,
+    diffexp_lfc_cutoff,
+):
    """Launch the cellxgene data viewer.
    This web app lets you explore single-cell expression data.
    Data must be in a format that cellxgene expects, read the
@@ -43,9 +89,6 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
    # Startup message
    click.echo("[cellxgene] Starting the CLI...")

-    # Import Flask app
-    from server.app.app import app
-
    # Argument checking
    name, extension = splitext(data)
    if extension != ".h5ad":
@@ -54,6 +97,8 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
    if debug:
        verbose = True
        open_browser = False
+    else:
+        warnings.formatwarning = custom_format_warning

    if not verbose:
        sys.tracebacklimit = 0
@@ -62,19 +107,13 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
        file_parts = splitext(basename(data))
        title = file_parts[0]

-    if listen_all:
-        host = "0.0.0.0"
-    else:
-        host = "127.0.0.1"
-
    # Setup app
    cellxgene_url = f"http://{host}:{port}"
-    api_base = f"{cellxgene_url}/api/"

-    app.config.update(
-        DATASET_TITLE=title,
-        CXG_API_BASE=api_base
-    )
+    # Import Flask app
+    from server.app.app import app
+
+    app.config.update(DATASET_TITLE=title)

    if not verbose:
        log = logging.getLogger("werkzeug")
@@ -85,15 +124,17 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
    # Fix for anaconda python. matplotlib typically expects python to be installed as a framework TKAgg is usually
    # available and fixes this issue. See https://matplotlib.org/faq/virtualenv_faq.html
    import matplotlib as mpl
-    mpl.use('TkAgg')
+
+    mpl.use("TkAgg")
    from server.app.scanpy_engine.scanpy_engine import ScanpyEngine

    args = {
        "layout": layout,
        "diffexp": diffexp,
        "max_category_items": max_category_items,
+        "diffexp_lfc_cutoff": diffexp_lfc_cutoff,
        "obs_names": obs_names,
-        "var_names": var_names
+        "var_names": var_names,
    }

    try:
@@ -109,4 +150,8 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,

    click.echo("[cellxgene] Type CTRL-C at any time to exit.")

+    if not verbose:
+        f = open(devnull, "w")
+        sys.stdout = f
+
    app.run(host=host, debug=debug, port=port, threaded=True)
@@ -7,22 +7,48 @@ from scipy.sparse.csc import csc_matrix

@click.command()
@click.argument("data", nargs=1, metavar="<dataset: file or path to data>", required=True)
-@click.option("--layout", "-l", default=["umap", "tsne"], multiple=True, type=click.Choice(["umap", "tsne"]),
-              help="Layout algorithm", show_default=True)
-@click.option("--recipe", "-r", default="none", type=click.Choice(["none", "seurat", "zheng17"]),
-              help="Preprocessing to run.", show_default=True)
+@click.option(
+    "--layout",
+    "-l",
+    default=["umap", "tsne"],
+    multiple=True,
+    type=click.Choice(["umap", "tsne"]),
+    help="Layout algorithm",
+    show_default=True,
+)
+@click.option(
+    "--recipe",
+    "-r",
+    default="none",
+    type=click.Choice(["none", "seurat", "zheng17"]),
+    help="Preprocessing to run.",
+    show_default=True,
+)
@click.option("--output", "-o", default="", help="Save a new file to filename.", metavar="<filename>")
@click.option("--plotting", "-p", default=False, is_flag=True, help="Whether to generate plots.", show_default=True)
@click.option("--sparse", default=False, is_flag=True, help="Whether to force sparsity.", show_default=True)
@click.option("--overwrite", default=False, is_flag=True, help="Allow file overwriting.", show_default=True)
@click.option("--set-obs-names", default="", help="Named field to set as index for obs.", metavar="<name>")
@click.option("--set-var-names", default="", help="Named field to set as index for var.", metavar="<name>")
-@click.option("--make-obs-names-unique", default=True, is_flag=True,
-              help="Ensure obs index is unique.", show_default=True)
-@click.option("--make-var-names-unique", default=True, is_flag=True,
-              help="Ensure var index is unique.", show_default=True)
-def prepare(data, layout, recipe, output, plotting, sparse, overwrite,
-            set_obs_names, set_var_names, make_obs_names_unique, make_var_names_unique):
+@click.option(
+    "--make-obs-names-unique", default=True, is_flag=True, help="Ensure obs index is unique.", show_default=True
+)
+@click.option(
+    "--make-var-names-unique", default=True, is_flag=True, help="Ensure var index is unique.", show_default=True
+)
+def prepare(
+    data,
+    layout,
+    recipe,
+    output,
+    plotting,
+    sparse,
+    overwrite,
+    set_obs_names,
+    set_var_names,
+    make_obs_names_unique,
+    make_var_names_unique,
+):
    """Preprocesses data for use with cellxgene.

    This tool runs a series of scanpy routines for preparing a dataset
@@ -35,6 +61,7 @@ def prepare(data, layout, recipe, output, plotting, sparse, overwrite,
    # collect slow imports here to make CLI startup more responsive
    click.echo("[cellxgene] Starting CLI...")
    import matplotlib
+
    matplotlib.use("Agg")
    import scanpy.api as sc

@@ -49,8 +76,10 @@ def prepare(data, layout, recipe, output, plotting, sparse, overwrite,
    output = expanduser(output)

    if not output:
-        click.echo("Warning: No file will be saved, to save the results of cellxgene prepare include "
-                   "--output <filename> to save output to a new file")
+        click.echo(
+            "Warning: No file will be saved, to save the results of cellxgene prepare include "
+            "--output <filename> to save output to a new file"
+        )
    if isfile(output) and not overwrite:
        raise click.UsageError(f"Cannot overwrite existing file {output}, try using the flag --overwrite")

@@ -119,9 +148,11 @@ def prepare(data, layout, recipe, output, plotting, sparse, overwrite,
        try:
            sc.tl.louvain(adata)
        except ModuleNotFoundError:
-            click.echo("\nWarning: louvain module is not installed, no clusters will be calculated. "
-                       "To fix this please install cellxgene with the optional feature louvain enabled: "
-                       "`pip install cellxgene[louvain]`")
+            click.echo(
+                "\nWarning: louvain module is not installed, no clusters will be calculated. "
+                "To fix this please install cellxgene with the optional feature louvain enabled: "
+                "`pip install cellxgene[louvain]`"
+            )

    def run_layout(adata):
        if len(unique(adata.obs["louvain"].values)) < 10:
@@ -142,11 +173,11 @@ def prepare(data, layout, recipe, output, plotting, sparse, overwrite,
    def show_step(item):
        names = {
            "make_sparse": "Ensuring sparsity",
-            "run_recipe": f"Running preprocessing recipe \"{recipe}\"",
+            "run_recipe": f'Running preprocessing recipe "{recipe}"',
            "run_pca": "Running PCA",
            "run_neighbors": "Calculating neighbors",
            "run_louvain": "Calculating clusters",
-            "run_layout": "Computing layout"
+            "run_layout": "Computing layout",
        }
        if item is not None:
            return names[item.__name__]
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Charlotte Weaver	4e67c645f8	bumped version (#602 ) 0.6.0 was burned on pypi	2019-02-14 10:00:16 -08:00
Charlotte Weaver	8b28d51dfa	bump version (#601 )	2019-02-13 16:29:14 -08:00
Charlotte Weaver	40ad283107	create server testing doc (#592 )	2019-02-13 15:35:45 -08:00
Isaac Virshup	0f8d7a55de	Set API path based on access address (#568 ) * Make api paths relative * Remove request import * Set publicPath to be relative	2019-02-11 09:01:37 -08:00
Sidney Bell	b6f946ec8a	Add note about installing hdf5 to FAQ (#581 )	2019-02-08 14:56:16 -08:00
Colin Megill	dbb3a309a9	Lasso (#586 ) * lasso working * break out invert into own function * action * add spatial dimension to crossfilter, in support of polygon lasso * improve comments on new dimension API * lasso vs zoom	2019-02-08 11:48:51 -08:00
Bruce Martin	2e9525741f	doc divergence warning (#591 )	2019-02-08 11:00:51 -08:00
Charlotte Weaver	585a5808b9	check if accept type in content type string (#589 )	2019-02-08 09:41:24 -08:00
Bruce Martin	6f464f4f92	package dependency updates (#585 ) * lint * update dev-related package dependencies	2019-02-06 12:57:09 -08:00
Colin Megill	08ea7d5137	Better input validation (#580 )	2019-02-05 10:44:59 -05:00
Charlotte Weaver	ad9be3cdd7	remove build-dev from .gitignore (#583 )	2019-02-04 14:47:28 -08:00
Charlotte Weaver	f737cc4ee4	Build improvements (#577 )	2019-02-04 14:15:35 -08:00
Charlotte Weaver	1103272b95	De-dupe -d CLI option alias (#575 ) removed from debug, diffexp gets to keep it	2019-01-30 16:11:21 -08:00
Colin Megill	2df7161cd8	Bulk add genes (#567 ) * bulk add * cleanup	2019-01-29 16:26:09 -05:00
Charlotte Weaver	d31c05c970	Add backed script to package.json (#566 ) * QOL script for FE devs to get & launch the backend * ensure python3.6 * changed name to backend-dev	2019-01-29 11:41:05 -08:00
fionagriffin	07db2eb3ee	Update data.md (#544 )	2019-01-29 10:31:31 -08:00
Justin Kiggins	a6d2a2e119	updates link to scanpy recipe docs. fixes #564 (#565 )	2019-01-28 10:00:29 -08:00
Justin Kiggins	f87e4bfbd3	home page fix (#551 ) * explicit site.url in config * infra for custom css * moves home item to html layout * adds baseurl to site config	2019-01-28 09:13:09 -08:00
Charlotte Weaver	ecaa32cfb2	bump version (#563 )	2019-01-17 17:14:44 -08:00
Charlotte Weaver	10693b08cc	Add __init__ file so fbs can be imported (#562 )	2019-01-17 17:11:31 -08:00
Charlotte Weaver	b02361a6bf	bump version (#560 )	2019-01-17 16:54:25 -08:00
Colin Megill	3fa888c6d4	user defined gene sets scatterplot, remove works (#556 )	2019-01-16 03:37:08 -05:00
Colin Megill	a42c9aca1c	resetting interface spinner but locked interface (#555 )	2019-01-16 03:36:35 -05:00
Charlotte Weaver	394da40bea	Doc formatting improvements (#549 )	2019-01-10 15:09:02 -08:00
Bruce Martin	5d60505407	remove --nan-to-num CLI parameter (#548 ) * remove --nan-to-num CLI parameter * factor tests better * lint - remove unused variables	2019-01-10 15:03:03 -08:00
Colin Megill	f876a0091a	Set theme jekyll-theme-cayman	2019-01-10 12:12:58 -05:00
Colin Megill	528eb5d172	Set theme jekyll-theme-minimal	2019-01-10 12:10:11 -05:00
Bruce Martin	b90447c387	binary wire format with flatbuffers (#509 ) * first flatbuffer schema * do not lint auto-generated files * add flatbuffers package * add flatbuffer module * wire up /data/X/T route * use flatbuffers for matrix data fetc * clarity and comments * add flatbuffer layout route * clean up obsolete code * fix tests * move flake8 config to setup.cfg * add comments * lint * rework layout routes for fbs * add more type support to fbs * lint * add flatbuffer support for annotations * function name improvements * fix botched merge with master * remove unused import * route cleanup for flatbuffers * rename function for clarity * add missing globals to Jest tests * fix client JS tests * fix routes for Python tests * comments for clarity * non-finite floating point hardening * more non-finite number handling * lint * fix tests for summarizeAnnotations * harden diffexp calculation against FP errors * cleanup unused code * lint * add encoding tests for flatbuffers * application type specified as strings * fix spelling error * improve variable names * add note about documentation gap * rename FBS DataFrame to Matrix	2019-01-09 14:26:05 -08:00
Colin Megill	42e25a1a1f	Async on long running operations (#543 ) * compute differential button * add user gene loading state to reducer * create input group, consume usergeneloading * add loading state	2019-01-07 16:45:47 -05:00
Justin Kiggins	eceab377f7	cleans up landing page (#541 )	2019-01-04 14:20:23 -08:00
Colin Megill	b0daeb3a9b	disable cell set buttons (#538 )	2019-01-03 20:05:09 -05:00
fionagriffin	7499fca251	add datasets to data.md (#540 ) added links and general info for interesting example datasets to use with cellxgene	2019-01-03 16:35:37 -08:00
GenevieveHaliburton	05a5a945bb	Landing page faq comments (#536 ) * add to faq, clean up titles * h4 headings for readibility * Ordering, add section on errors	2019-01-03 14:07:38 -08:00
Colin Megill	d3c96087b2	Add GIFs to docs (#537 ) * Add gif files * rename gif for clarity * add gifs * reorder * resize	2019-01-03 15:24:33 -05:00
Bruce Martin	cfcf16aa69	[DO NOT MERGE] performance improvements in color-by feature (#535 ) performance improvements in color-by feature	2018-12-27 14:05:49 -08:00
Justin Kiggins	bc59074300	new docs (#525 ) * new docs * moves development docs to dev_docs	2018-12-20 14:28:05 -08:00
Colin Megill	56f9bc543e	Set theme jekyll-theme-minimal	2018-12-20 12:05:08 -05:00
James Taylor	f2eb2cad82	Use url_root from Flask rather than hardcoding hostname. (#520 ) Allows running in a container on a remote host, should also allow running behind a proxy with a url prefix.	2018-12-19 16:20:26 -08:00
Justin Kiggins	bd523280a4	improved quickstart (#523 )	2018-12-19 13:38:30 -08:00
Justin Kiggins	fc45fb3899	creates FAQ page in docs (#522 ) * creates FAQ page * revise index.md * tweaks	2018-12-19 13:18:16 -08:00
Charlotte Weaver	f0f7200f0b	Move jsonification to engine level (#511 )	2018-12-18 21:14:24 -08:00
fionagriffin	6af7708d62	update index.md (#519 ) ready for features	2018-12-18 11:10:29 -05:00
Colin Megill	a297ea30e2	bump version (#518 )	2018-12-17 14:30:34 -05:00
Colin Megill	0ef6c36f4c	Cluster Occupancy (n dim cube) (#513 ) * create occupancy component * occupancy working * centering, width, flex spacing * add is-number * extend sort to cover string + int * proof of concept sorted occupancy * handle undefined occupancy entry, cleanup * only render occupancy when colorby is cat * cleanup unused vars * cleanup	2018-12-17 14:19:48 -05:00
Justin Kiggins	aaa60bc303	new signup url (#516 )	2018-12-14 16:32:41 -08:00
fionagriffin	7d40d89fd8	Update index.md (#507 ) Added content from readme	2018-12-14 15:19:21 -08:00
Charlotte Weaver	46ad346df1	Clarify that we do not support python 3.7 (#512 ) * Update setup.py * Readme: no python 3.7 * specify 3.6 in venv * clarify 3.7 error we will support it eventually	2018-12-13 15:15:56 -08:00
Charlotte Weaver	83154577e4	Black -- formatter for python (#508 ) * Add black * use black to format code * Black version	2018-12-12 09:44:47 -08:00
Charlotte Weaver	a847951658	Update slack link (#506 )	2018-12-11 10:20:22 -08:00
Charlotte Weaver	24af6efbcb	Add Docker file (#505 ) * Add docker support * Add readme * minor fixes	2018-12-11 10:18:10 -08:00
Colin Megill	126cac833a	Rename README.md to index.md	2018-12-11 12:09:16 -05:00
Colin Megill	fc272dc42e	github pages test	2018-12-11 12:05:14 -05:00
Colin Megill	d3a0d66139	Set theme jekyll-theme-architect	2018-12-10 18:16:40 -05:00
Charlotte Weaver	9604231a2a	Add alternative to install [louvain] extra requires (#500 ) * louvain install by hand * reworded	2018-12-10 14:46:55 -08:00
Jeremy Freeman	6ea3b7f3cf	use collapsable details to improve FAQ formatting (#503 )	2018-12-07 21:07:57 +01:00
Charlotte Weaver	0d0a32f272	Added PR guidelines (#492 ) * Added PR guidelines Added document to cover guidelines for creating, reviewing and merging PRs. * fixed typo * another typo * grammar * Remove 5's title The subheading is clearer as a top-level	2018-12-06 13:43:10 -08:00
Bruce Martin	48e0ea542b	clarifications to release recipe (#499 ) * clarifications to release recipe * fix spelling errors	2018-12-05 16:58:53 -08:00
Bruce Martin	95fd20b405	bump version to 0.3 (#498 )	2018-12-05 14:35:17 -08:00
Charlotte Weaver	5c6cc597c8	Default warning in debug mode (#497 )	2018-12-05 13:38:34 -08:00
Bruce Martin	3bfeadc2b9	improve handling of non-finite floating point values (#490 ) * improve HTTP error reporting * generate standards-compatible JSON * add --nan-to-num work-around for non-finite floating point values * lint * update tests * correctly set Infinities to min/max * REAMDE update for --nan-to-num * define constant for repetitive warning message * clarify where NaN errors will occure	2018-12-04 14:56:16 -08:00
Bruce Martin	296ed752fa	Improved summary counts of annotation values (#478 ) * convert annotation summary to a Map * add 2d annotation count summary * add memoization on 2D annotation counting * add tests for annotation summarization * fix import/exports * rename WorldOps to WorldUtil * rename WorldOps to WorldUtil * add comment	2018-12-03 09:15:03 -08:00
Charlotte Weaver	1e66ec2b89	Update scikit learn (#487 ) They finally fixed their cloud pickle issue	2018-11-30 12:02:28 -08:00
Charlotte Weaver	4a50f0ed46	External host option (#481 ) * allow specifying host ip instead of listen all * flask serving * Limit to ip addresses only	2018-11-29 17:01:49 -08:00
Bruce Martin	af0d1f6fb2	issue #480 workaround (#484 ) * only load annotation var names * remove incorrect usage of var annotation data * temporary workaround for issue #480 * lint * issue warnings only once per item	2018-11-29 16:33:21 -08:00
fionagriffin	a83ec60308	add FAQs to readme (#482 ) Added question and answer to FAQ; added link to scanpy documentation on scientific recipes.	2018-11-29 13:59:10 -08:00
Charlotte Weaver	76c0cbf16b	loose -> lose (#479 )	2018-11-28 13:07:10 -08:00
Charlotte Weaver	455d987398	Add custom format for warnings (#477 ) [cellxgene] Warning: <message>	2018-11-28 12:47:11 -08:00
Colin Megill	00b9a8fa72	Fuzzy search on genes (#473 ) * proof of concept blueprint typeahead * poc with fuzzysearch lib * typeahead adds gene on enter * add gene on menu click * typeahead clears correctly * cleanup	2018-11-28 09:50:02 -08:00
Charlotte Weaver	28f5263c88	bump version (#475 )	2018-11-27 15:43:19 -08:00
Bruce Martin	933fac5fff	Allow float64 to down-cast to float32 (#472 ) * warn if annotations will down-cast to float32 * correctly handle falsey data values	2018-11-27 11:17:30 -08:00
Charlotte Weaver	0c26f227fe	bump version (#470 )	2018-11-26 15:38:47 -08:00
Marcus Kinsella	2b90c747f5	Fix readme images on PyPi (#467 ) But this time actually do that	2018-11-26 15:27:05 -08:00
Charlotte Weaver	43a4e087ef	Fix formatting issues (#468 ) - Fix step 8 bullets - Appropriate capitalization - Add link to release notes doc	2018-11-26 13:52:00 -08:00
Charlotte Weaver	f2612707bb	bumped version (#466 )	2018-11-26 11:42:56 -08:00
Bruce Martin	138d30909a	improve type handling for non-string annotation data (#465 ) * improve type handling for non-string annotation data * improve clarity of code	2018-11-26 10:32:29 -08:00
Charlotte Weaver	2b5094665f	version 0.2.0 (#456 )	2018-11-16 15:12:54 -08:00
Jeremy Freeman	a81258bc0d	remove printing (#455 )	2018-11-16 14:45:06 -08:00
Bruce Martin	141f802824	differential expression improvements (#452 ) * add cutoff for low expression genes in topN selection * remove debugging printfs * change cli param name for diffexp cutoff * change CLI param name * second try at diffexp - using lfc sort with pval cutoff * use lfc cutoff * update comments to match code; cap p-value adjustment to max of 1 * lint * explain diffexp in readme * add link * add diffexp-lfc-cutoff to test config * update test to match revised diffexp spec * fix latent bug in GET arg handling that was breaking tests * lint * comment cleanup * fix variance overestimation so it is symmetric * lint	2018-11-16 14:44:56 -08:00
Charlotte Weaver	3475f3f12e	Update release_process.md (#443 ) * Update release_process.md * Clarified release process	2018-11-16 14:36:43 -08:00
Jeremy Freeman	dff2526077	warm to cool (#454 )	2018-11-16 14:33:32 -08:00
Marcus Kinsella	2b4aa92f67	Use externally-reachable image urls (#453 ) This is needed for the images to show up in pypi.	2018-11-16 13:02:49 -08:00
Jeremy Freeman	3151306d7e	readme updates (#436 ) lots of updates to the readme to: improve scientific and technical clarity, reflect all recent changes to the CLI (especially the addition of prepare), reflect all recent changes to our installation, improve explanation of how to handle a few different kinds of data, and expand instructions on contributing and developing	2018-11-15 18:18:30 -08:00
Colin Megill	ef80c8df2a	Change color scales to rainbow (#449 ) * rainbow * eslint * remove comments	2018-11-15 18:17:55 -08:00
Charlotte Weaver	1a5f49239a	Release 0.0.4 (#447 ) * 0.0.3 bump * release 0.0.4	2018-11-15 16:43:59 -08:00
Charlotte Weaver	dfc04d8de5	Add requirements to manifest (#445 )	2018-11-15 11:56:22 -08:00