bump version to 0.3 (#498)

* improve HTTP error reporting

* generate standards-compatible JSON

* add --nan-to-num work-around for non-finite floating point values

* lint

* update tests

* correctly set Infinities to min/max

* REAMDE update for --nan-to-num

* define constant for repetitive warning message

* clarify where NaN errors will occure

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.0.1
+current_version = 0.3.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"

MANIFEST.in

@@ -1,3 +1,4 @@
 recursive-include server/app/web/templates *
 recursive-include server/app/web/static *
 
+include server/requirements.txt

README.md

@@ -1,111 +1,241 @@
 # cellxgene
 
-### An interactive, performant explorer for single cell transcriptomics data.
+> an interactive explorer for single-cell transcriptomics data
 
-<img align="right" width="350" height="218" src="./example-dataset/cellxgene-demo.gif" pad="50px">
+`cellxgene` is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
-cellxgene is an open-source experiment in how to bring powerful tools from modern web development to visualize and explore large single-cell transcriptomics datasets.
-Started in the context of the Human Cell Atlas Consortium, cellxgene hopes to both enable scientists to explore their data and to equip developers with scalable, reusable patterns and frameworks for visualizing large scientific datasets.
 
-## Features
+<img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
 
-- **Visualization at scale:** built with [WebGL](https://www.khronos.org/webgl/), [React](https://reactjs.org/) & [Redux](https://redux.js.org/) to handle visualization of at least 1 million cells.
+## getting started
 
-- **Interactive exploration:** select, cross-filter, and compare subsets of your data with performant indexing and data handling.
+You'll need **python 3.6** and **Google Chrome**. The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).
 
-- **Flexible API:** the cellxgene client-server model is designed to support a range of existing analysis packages for backend computational tasks (eg scanpy), integrated with client-side visualization via a [REST API](https://restfulapi.net/).
+To install run
 
-## Getting Started
+```
+pip install cellxgene
+```
 
-**Requirements**
+To start exploring a dataset call
+
+```
+cellxgene launch dataset.h5ad --open
+```
+
+If you want an example dataset download [this file](https://github.com/chanzuckerberg/cellxgene/raw/master/example-dataset/pbmc3k.h5ad) and then call
+
+```
+cellxgene launch pbmc3k.h5ad --open
+```
+
+You should see your web browser open with the following
+
+<img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
+
+**Note**: automatic opening of the browser with the `--open` flag only works on OS X, on other platforms you'll need to directly point to the provided link in your browser.
+
+There are several options available, such as:
+
+- `--layout` to specify the layout as `tsne` or `umap`
+- `--title` to show a title on the explorer
+- `--open` to automatically open the web browser after launching (OS X only)
+
+To see all options call
+
+```
+cellxgene launch --help
+```
+
+There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+
+## data formatting
+
+### assumptions
+
+The `launch` command assumes that the data is stored in the `.h5ad` format from the [`anndata`](https://anndata.readthedocs.io/en/latest/index.html) library. It also assumes that certain computations have already been performed. Briefly, the `.h5ad` format wraps a two-dimensional `ndarray` and stores additional metadata as "annotations" for either observations (referred to as `obs` and `obsm`) or variables (`var` and `varm`). `cellxgene launch` makes the following assumptions about your data (we recommend loading and inspecting your data using `scanpy` to validate these assumptions)
+
+- an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
+- a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
+- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_tsne` for the `tsne` layout or `X_umap` for the `umap` layout)
+- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
+
+### prepare
+
+The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
+
+To prepare from an existing `.h5ad` file use
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
+```
+
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://github.com/theislab/scanpy/blob/master/scanpy/preprocessing/recipes.py).
+
+Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
+
+```
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
+```
+
+To see all options call
+
+```
+cellxgene prepare --help
+```
+
+**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
+
+```
+pip install cellxgene[louvain]
+```
+
+## conda and virtual environments
+
+If you use conda and want to create a conda environment for `cellxgene` you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.6
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+## FAQ
+
+> Someone sent me a directory of `10X-Genomics` data with a `mtx` file and I've never used `scanpy`, can I use `cellxgene`?
+
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+```
+
+Depending on the size of the dataset, this may take some time. Once it's done, call
+
+```
+cellxgene launch data-processed.h5ad --layout=umap --open
+```
+
+And your web browser should open with an interactive view of your data.
+
+> In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
+
+Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using
+
+```
+pip install cellxgene[louvain]
+```
+
+> I ran `prepare` and I'm getting results that look unexpected
+
+You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
+```
+
+It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
+
+> I have extra metadata that I want to add to my dataset
+
+Currently this is not supported directly, but you should be able to do this manually using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
+
+> I tried to `pip install cellxgene` and got a weird error I don't understand
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.
+
+> How are you computing and sorting differential expression results?
+
+Currently we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
+
+> I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
+
+This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
+
+> What part of the anndata objects does cellxgene pull in for visualization?
+
+- `.obs` and `.var` annotations are use to extract metadata for filtering
+- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
+- `.obsm` is used for layout
+
+> When I start cellxgene, I get an error `Unexpected HTTP response 500, INTERNAL SERVER ERROR -- Out of range float values are not JSON compliant` in the web UI, or `Warning: JSON encoding failure - suggest trying --nan-to-num command line option` in the CLI. What can I do?
+
+At the moment, cellxgene is unable to transmit floating point NaN or Inifinty values to the web UI (due to a limitation on data serialization method in use). We expect to resolve this in a future release, but in the meantime, you can work around this issue by starting cellxgene with the `--nan-to-num` command line option, ie, `cellxgene launch data.h5ad --nan-to-num`.
+
+This option will convert all NaNs to zero, and all positive/negative infinities to the min/max of the data element within which the value was found (eg, +Infinity within an `obs` annotation will be converted to the maximum finite value in that annotation). This option will increase startup time, so we recommend only using it when the dataset contains NaN/Infinities.
+
+## developer guide
+
+This project has made a few key design choices
+
+- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
+- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
+- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
+
+Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
+
+If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
 
-- OS: OSX, Windows, Linux -- the developers are currently testing on OSX and Windows (via WSL using Ubuntu). It should work on other platforms but if you are using something different and need help, please let us know.
 - python 3.6
-- python3 tkinter
+- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
-- npm
-- Google Chrome
 
-**Clone project**
+Then clone the project
 
-    git clone https://github.com/chanzuckerberg/cellxgene.git
+```
+git clone https://github.com/chanzuckerberg/cellxgene.git
+```
 
-**Install client**
+Build the client web assets by calling this from inside the `cellxgene` folder
 
-    cd cellxgene
+```
-    ./bin/build-client
+./bin/build-client
+```
 
-**To use with virtual env for python**
+Install all requirements (we recommend doing this inside a virtual environment)
-(optional, but recommended)
 
-    ENV_NAME=cellxgene
+```
-    python3 -m venv ${ENV_NAME}
+pip install -e .
-    source ${ENV_NAME}/bin/activate
+```
 
-**Install server**
+You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
 
-     pip install -e .
+If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://cziscience.slack.com/messages/CCTA8DF1T) and posting in the `#cellxgene-dev` channel.
 
-**Run (with demo data)**
+## development roadmap
 
-    cellxgene launch --title PBMC3K example-dataset/pbmc3k.h5ad
+`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact)
 
-**Help**
+- **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
+- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene` to render the results of these analyses when they have been performed
+- **Deploy to web** Many projects release public data browser websites alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
+- **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data
 
-    cellxgene --help
+## contributing
 
-_For help with the scanpy engine_
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
 
-    cellxgene scanpy --help
+## inspiration and collaboration
 
-## Using your own data
+We've been heavily inspired by several other related single-cell visualization projects, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), and many others. We hope to explore collaborations where useful as this community works together on improving interactive visualization for single-cell data.
 
-### Scanpy
+We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.
 
-To prepare your data you will need to format your data into AnnData format using scanpy and calculate PCA and nearest neighbors and save in h5ad format.
+We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
 
-1.  [Load data into scanpy](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)
+We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
 
-    - Ensure that `obs`'s index is the cell names: `print(data.obs_names)` should show your cell indices. If it shows gene names, you may need to just call `data.transpose()`.
+## help and contact
 
-2.  Calculate PCA
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://cziscience.slack.com/messages/CCTA8DF1T) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
 
-    sc.pp.pca(data) ## sc is scanpy.api
+## reuse
 
-3.  Calculate nearest neighbors (depending on layout algorithm)
+This project was started with the sole goal of empowering the scientific community to explore and understand their data. As such, we encourage other scientific tool builders in academia or industry to adopt the patterns, tools, and code from this project, and reach out to us with ideas or questions. All code is freely available for reuse under the [MIT license](https://opensource.org/licenses/MIT).
-
-    ```
-    # For umap layout algorithm, you need to use the "umap" method for neighbors
-    sc.pp.neighbors(data, method="umap", metric="euclidean", use_rep="X_pca")
-
-    # For tsne layout algorithm, you can use either "umap" or "gauss"; we recommend "gauss"
-    sc.pp.neighbors(data, method="gauss", metric="euclidean", use_rep="X_pca")
-    ```
-
-4.  Save file
-
-    ```
-    # cellxgene requires file to be named data.h5ad
-    data.write("data.h5ad")
-    ```
-
-## Contributing
-
-We warmly welcome contributions from the community. Please submit any bug reports and feature requests through github issues. Please submit any direct contributions via a branch + pull request.
-
-## Inspiration and collaboration
-
-We’ve been inspired by several other related efforts in this space, including the [UCSC Cell Browswer](http://cells.ucsc.edu/), [Cytoscape](http://www.cytoscape.org/), [Xena](https://xena.ucsc.edu/), [ASAP](https://asap.epfl.ch/), [Gene Pattern](http://genepattern-notebook.org/), & many others; we hope to explore collaborations where useful.
-
-## Help/Contact
-
-Have questions, suggestions, or comments? You can contact us by joining [CZI Science Slack](https://cziscience.slack.com/messages/CCTA8DF1T) and posting in the #cellxgene channel. Please submit any feature requests or bugs as an issue in github. We'd love to hear from you!
-
-## Reuse
-
-This project was started with the sole goal of empowering the scientific community to explore and understand their data. As such, we whole-heartedly encourage other scientific tool builders to adopt the patterns, tools, and code from this project, and reach out to us with ideas or questions using Github Issues or Pull Requests. All code is freely available for reuse under the [MIT license](https://opensource.org/licenses/MIT).
-
-## Acknowledgements
-
-cellxgene is inspired by many innovative projects. We would like to specifically thank:
-
-- Alex Wolf for the demo dataset.
-- Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for API inspiration.

bin/clean

@@ -7,6 +7,8 @@ echo "removing node_modules"
 rm -rf $CELLXGENE_DIR/client/node_modules
 echo "removing client_build"
 rm -rf $CELLXGENE_DIR/client/build
+echo "removing dist"
+rm -rf $CELLXGENE_DIR/dist
 echo "removing egg-info"
 rm -rf $CELLXGENE_DIR/cellxgene.egg-info
 echo "removing static files"

client/__tests__/util/stateManager/summarizeAnnotations.test.js

@@ -0,0 +1,192 @@
+import summarizeAnnotations from "../../../src/util/stateManager/summarizeAnnotations";
+
+describe("summarizeAnnotations", () => {
+  const schema = {
+    annotations: {
+      obs: [
+        { name: "name", type: "string" },
+        { name: "nameString", type: "string" },
+        { name: "nameBoolean", type: "boolean" },
+        { name: "nameFloat32", type: "float32" },
+        { name: "nameInt32", type: "int32" },
+        {
+          name: "nameCategorical",
+          type: "categorical",
+          categories: [true, false, 1, 0, 0.00001, 4383.4833, "test", "", "0"]
+        }
+      ],
+      var: [{ name: "name", type: "string" }]
+    }
+  };
+
+  test("empty test", () => {
+    const summary = summarizeAnnotations(schema, [], []);
+    expect(summary).toEqual(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          },
+          nameFloat32: {
+            categorical: false,
+            range: {
+              max: Number.NEGATIVE_INFINITY,
+              min: Number.POSITIVE_INFINITY
+            }
+          },
+          nameInt32: {
+            categorical: false,
+            range: {
+              max: Number.NEGATIVE_INFINITY,
+              min: Number.POSITIVE_INFINITY
+            }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("simple test", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toEqual(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: ["hi"],
+            categoryCounts: new Map([["hi", 1]]),
+            numCategories: 1
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: [true],
+            categoryCounts: new Map([[true, 1]]),
+            numCategories: 1
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 39.3, max: 39.3 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: [1],
+            categoryCounts: new Map([[1, 1]]),
+            numCategories: 1
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("multi test", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n0",
+        nameString: "hi",
+        nameBoolean: false,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      },
+      {
+        __index__: 1,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: false
+      },
+      {
+        __index__: 2,
+        name: "n2",
+        nameString: "bye",
+        nameBoolean: true,
+        nameFloat32: 0,
+        nameInt32: 99,
+        nameCategorical: "0"
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toMatchObject(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: expect.arrayContaining(["hi", "bye"]),
+            categoryCounts: new Map([["hi", 2], ["bye", 1]]),
+            numCategories: 2
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: expect.arrayContaining([true, false]),
+            categoryCounts: new Map([[true, 2], [false, 1]]),
+            numCategories: 2
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 0, max: 39.3 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: expect.arrayContaining([1, false, "0"]),
+            categoryCounts: new Map([[1, 1], [false, 1], ["0", 1]]),
+            numCategories: 3
+          }
+        },
+        var: {}
+      })
+    );
+  });
+});

client/__tests__/util/stateManager/worldUtil.test.js

@@ -0,0 +1,45 @@
+import {
+  countCategoryValues2D,
+  clearCaches
+} from "../../../src/util/stateManager/worldUtil";
+
+describe("WorldUtil cache management", () => {
+  test("empty", () => {
+    const count = countCategoryValues2D("a", "b", []);
+    expect(count).toMatchObject(new Map());
+  });
+
+  test("simple couts", () => {
+    const rows = [{ a: 0, b: false }, { a: 0, b: true }, { a: 1, b: false }];
+    const count = countCategoryValues2D("a", "b", rows);
+    expect(count).toMatchObject(
+      new Map([
+        [0, new Map([[true, 1], [false, 1]])],
+        [1, new Map([[false, 1]])]
+      ])
+    );
+  });
+
+  test("memo cache clear", () => {
+    clearCaches();
+    const row1 = [];
+    const row2 = [{ a: 0, b: false }, { a: 0, b: true }, { a: 1, b: false }];
+    const count1 = countCategoryValues2D("a", "b", row1);
+    const count2 = countCategoryValues2D("a", "b", row1);
+    const count3 = countCategoryValues2D("a", "b", []);
+    const count4 = countCategoryValues2D("a", "b", row2);
+
+    clearCaches();
+    const count10 = countCategoryValues2D("a", "b", row1);
+    const count11 = countCategoryValues2D("a", "b", row2);
+
+    expect(count1).toEqual(count2);
+    expect(count1).toEqual(count3);
+    expect(count1).toEqual(count10);
+    expect(count1).not.toBe(count3);
+    expect(count1).not.toBe(count10);
+
+    expect(count4).toEqual(count11);
+    expect(count4).not.toBe(count11);
+  });
+});

client/configuration/eslint/eslint.js

@@ -34,7 +34,8 @@ module.exports = {
     "object-curly-newline": ["error", { consistent: true }],
     "react/prop-types": [0],
     "space-before-function-paren": "off",
-    "function-paren-newline": "off"
+    "function-paren-newline": "off",
+    "prefer-destructuring": ["error", { object: true, array: false }]
   },
   overrides: [
     {

client/package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "cellxgene",
-  "version": "0.0.1",
+  "version": "0.3.0",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {
@@ -2194,7 +2194,7 @@
     },
     "babel-plugin-syntax-object-rest-spread": {
       "version": "6.13.0",
-      "resolved": "http://registry.npmjs.org/babel-plugin-syntax-object-rest-spread/-/babel-plugin-syntax-object-rest-spread-6.13.0.tgz",
+      "resolved": "https://registry.npmjs.org/babel-plugin-syntax-object-rest-spread/-/babel-plugin-syntax-object-rest-spread-6.13.0.tgz",
       "integrity": "sha1-/WU28rzhODb/o6VFjEkDpZe7O/U=",
       "dev": true
     },
@@ -2627,7 +2627,7 @@
     },
     "browserify-aes": {
       "version": "1.2.0",
-      "resolved": "http://registry.npmjs.org/browserify-aes/-/browserify-aes-1.2.0.tgz",
+      "resolved": "https://registry.npmjs.org/browserify-aes/-/browserify-aes-1.2.0.tgz",
       "integrity": "sha512-+7CHXqGuspUn/Sl5aO7Ea0xWGAtETPXNSAjHo48JfLdPWcMng33Xe4znFvQweqc/uzk5zSOI3H52CYnjCfb5hA==",
       "dev": true,
       "requires": {
@@ -2664,7 +2664,7 @@
     },
     "browserify-rsa": {
       "version": "4.0.1",
-      "resolved": "http://registry.npmjs.org/browserify-rsa/-/browserify-rsa-4.0.1.tgz",
+      "resolved": "https://registry.npmjs.org/browserify-rsa/-/browserify-rsa-4.0.1.tgz",
       "integrity": "sha1-IeCr+vbyApzy+vsTNWenAdQTVSQ=",
       "dev": true,
       "requires": {
@@ -2718,7 +2718,7 @@
     },
     "buffer": {
       "version": "4.9.1",
-      "resolved": "http://registry.npmjs.org/buffer/-/buffer-4.9.1.tgz",
+      "resolved": "https://registry.npmjs.org/buffer/-/buffer-4.9.1.tgz",
       "integrity": "sha1-bRu2AbB6TvztlwlBMgkwJ8lbwpg=",
       "dev": true,
       "requires": {
@@ -3394,7 +3394,7 @@
     },
     "create-hash": {
       "version": "1.2.0",
-      "resolved": "http://registry.npmjs.org/create-hash/-/create-hash-1.2.0.tgz",
+      "resolved": "https://registry.npmjs.org/create-hash/-/create-hash-1.2.0.tgz",
       "integrity": "sha512-z00bCGNHDG8mHAkP7CtT1qVu+bFQUPjYq/4Iv3C3kWjTFV10zIjfSoeqXo9Asws8gwSHDGj/hl2u4OGIjapeCg==",
       "dev": true,
       "requires": {
@@ -3407,7 +3407,7 @@
     },
     "create-hmac": {
       "version": "1.1.7",
-      "resolved": "http://registry.npmjs.org/create-hmac/-/create-hmac-1.1.7.tgz",
+      "resolved": "https://registry.npmjs.org/create-hmac/-/create-hmac-1.1.7.tgz",
       "integrity": "sha512-MJG9liiZ+ogc4TzUwuvbER1JRdgvUFSB5+VR/g5h82fGaIRWMWddtKBHi7/sVhfjQZ6SehlyhvQYrcYkaUIpLg==",
       "dev": true,
       "requires": {
@@ -4090,7 +4090,7 @@
     },
     "diffie-hellman": {
       "version": "5.0.3",
-      "resolved": "http://registry.npmjs.org/diffie-hellman/-/diffie-hellman-5.0.3.tgz",
+      "resolved": "https://registry.npmjs.org/diffie-hellman/-/diffie-hellman-5.0.3.tgz",
       "integrity": "sha512-kqag/Nl+f3GwyK25fhUMYj81BUOrZ9IuJsjIcDE5icNM9FJHAVm3VcUDxdLPoQtTuUylWm6ZIknYJwwaPxsUzg==",
       "dev": true,
       "requires": {
@@ -4705,7 +4705,7 @@
         },
         "load-json-file": {
           "version": "2.0.0",
-          "resolved": "http://registry.npmjs.org/load-json-file/-/load-json-file-2.0.0.tgz",
+          "resolved": "https://registry.npmjs.org/load-json-file/-/load-json-file-2.0.0.tgz",
           "integrity": "sha1-eUfkIUmvgNaWy/eXvKq8/h/inKg=",
           "dev": true,
           "requires": {
@@ -4891,7 +4891,7 @@
     },
     "events": {
       "version": "1.1.1",
-      "resolved": "http://registry.npmjs.org/events/-/events-1.1.1.tgz",
+      "resolved": "https://registry.npmjs.org/events/-/events-1.1.1.tgz",
       "integrity": "sha1-nr23Y1rQmccNzEwqH1AEKI6L2SQ="
     },
     "evp_bytestokey": {
@@ -5206,7 +5206,7 @@
     },
     "finalhandler": {
       "version": "1.1.1",
-      "resolved": "http://registry.npmjs.org/finalhandler/-/finalhandler-1.1.1.tgz",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.1.tgz",
       "integrity": "sha512-Y1GUDo39ez4aHAw7MysnUD5JzYX+WaIj8I57kO3aEPT1fFRL4sr7mjei97FgnwhAyyzRYmQZaTHb2+9uZ1dPtg==",
       "dev": true,
       "requires": {
@@ -5401,7 +5401,8 @@
         "ansi-regex": {
           "version": "2.1.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "aproba": {
           "version": "1.2.0",
@@ -5422,12 +5423,14 @@
         "balanced-match": {
           "version": "1.0.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "brace-expansion": {
           "version": "1.1.11",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "balanced-match": "^1.0.0",
             "concat-map": "0.0.1"
@@ -5442,17 +5445,20 @@
         "code-point-at": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "concat-map": {
           "version": "0.0.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "console-control-strings": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "core-util-is": {
           "version": "1.0.2",
@@ -5569,7 +5575,8 @@
         "inherits": {
           "version": "2.0.3",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "ini": {
           "version": "1.3.5",
@@ -5581,6 +5588,7 @@
           "version": "1.0.0",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "number-is-nan": "^1.0.0"
           }
@@ -5595,6 +5603,7 @@
           "version": "3.0.4",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "brace-expansion": "^1.1.7"
           }
@@ -5602,12 +5611,14 @@
         "minimist": {
           "version": "0.0.8",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "minipass": {
           "version": "2.2.4",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "safe-buffer": "^5.1.1",
             "yallist": "^3.0.0"
@@ -5626,6 +5637,7 @@
           "version": "0.5.1",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "minimist": "0.0.8"
           }
@@ -5706,7 +5718,8 @@
         "number-is-nan": {
           "version": "1.0.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "object-assign": {
           "version": "4.1.1",
@@ -5718,6 +5731,7 @@
           "version": "1.4.0",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "wrappy": "1"
           }
@@ -5803,7 +5817,8 @@
         "safe-buffer": {
           "version": "5.1.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "safer-buffer": {
           "version": "2.1.2",
@@ -5839,6 +5854,7 @@
           "version": "1.0.2",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "code-point-at": "^1.0.0",
             "is-fullwidth-code-point": "^1.0.0",
@@ -5858,6 +5874,7 @@
           "version": "3.0.1",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "ansi-regex": "^2.0.0"
           }
@@ -5901,12 +5918,14 @@
         "wrappy": {
           "version": "1.0.2",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "yallist": {
           "version": "3.0.2",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         }
       }
     },
@@ -5922,6 +5941,11 @@
       "integrity": "sha1-GwqzvVU7Kg1jmdKcDj6gslIHgyc=",
       "dev": true
     },
+    "fuzzysort": {
+      "version": "1.1.4",
+      "resolved": "https://registry.npmjs.org/fuzzysort/-/fuzzysort-1.1.4.tgz",
+      "integrity": "sha512-JzK/lHjVZ6joAg3OnCjylwYXYVjRiwTY6Yb25LvfpJHK8bjisfnZJ5bY8aVWwTwCXgxPNgLAtmHL+Hs5q1ddLQ=="
+    },
     "get-caller-file": {
       "version": "1.0.3",
       "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-1.0.3.tgz",
@@ -5936,7 +5960,7 @@
     },
     "get-stream": {
       "version": "3.0.0",
-      "resolved": "http://registry.npmjs.org/get-stream/-/get-stream-3.0.0.tgz",
+      "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-3.0.0.tgz",
       "integrity": "sha1-jpQ9E1jcN1VQVOy+LtsFqhdO3hQ=",
       "dev": true
     },
@@ -6323,7 +6347,7 @@
     },
     "html-webpack-plugin": {
       "version": "3.2.0",
-      "resolved": "http://registry.npmjs.org/html-webpack-plugin/-/html-webpack-plugin-3.2.0.tgz",
+      "resolved": "https://registry.npmjs.org/html-webpack-plugin/-/html-webpack-plugin-3.2.0.tgz",
       "integrity": "sha1-sBq71yOsqqeze2r0SS69oD2d03s=",
       "dev": true,
       "requires": {
@@ -6399,7 +6423,7 @@
     },
     "http-errors": {
       "version": "1.6.3",
-      "resolved": "http://registry.npmjs.org/http-errors/-/http-errors-1.6.3.tgz",
+      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.6.3.tgz",
       "integrity": "sha1-i1VoC7S+KDoLW/TqLjhYC+HZMg0=",
       "dev": true,
       "requires": {
@@ -6622,7 +6646,7 @@
     },
     "is-builtin-module": {
       "version": "1.0.0",
-      "resolved": "http://registry.npmjs.org/is-builtin-module/-/is-builtin-module-1.0.0.tgz",
+      "resolved": "https://registry.npmjs.org/is-builtin-module/-/is-builtin-module-1.0.0.tgz",
       "integrity": "sha1-VAVy0096wxGfj3bDDLwbHgN6/74=",
       "dev": true,
       "requires": {
@@ -7629,7 +7653,7 @@
     },
     "json5": {
       "version": "0.5.1",
-      "resolved": "http://registry.npmjs.org/json5/-/json5-0.5.1.tgz",
+      "resolved": "https://registry.npmjs.org/json5/-/json5-0.5.1.tgz",
       "integrity": "sha1-Hq3nrMASA0rYTiOWdn6tn6VJWCE=",
       "dev": true
     },
@@ -7719,7 +7743,7 @@
     },
     "load-json-file": {
       "version": "1.1.0",
-      "resolved": "http://registry.npmjs.org/load-json-file/-/load-json-file-1.1.0.tgz",
+      "resolved": "https://registry.npmjs.org/load-json-file/-/load-json-file-1.1.0.tgz",
       "integrity": "sha1-lWkFcI1YtLq0wiYbBPWfMcmTdMA=",
       "dev": true,
       "requires": {
@@ -8043,7 +8067,7 @@
       "dependencies": {
         "minimist": {
           "version": "1.2.0",
-          "resolved": "http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
           "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ=",
           "dev": true
         }
@@ -8168,7 +8192,7 @@
     },
     "minimist": {
       "version": "0.0.8",
-      "resolved": "http://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz",
+      "resolved": "https://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz",
       "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0=",
       "dev": true
     },
@@ -8213,7 +8237,7 @@
     },
     "mkdirp": {
       "version": "0.5.1",
-      "resolved": "http://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz",
+      "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz",
       "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=",
       "dev": true,
       "requires": {
@@ -9939,7 +9963,7 @@
     },
     "parse-asn1": {
       "version": "5.1.1",
-      "resolved": "http://registry.npmjs.org/parse-asn1/-/parse-asn1-5.1.1.tgz",
+      "resolved": "https://registry.npmjs.org/parse-asn1/-/parse-asn1-5.1.1.tgz",
       "integrity": "sha512-KPx7flKXg775zZpnp9SxJlz00gTd4BmJ2yJufSc44gMCRrRQ7NSzAcSJQfifuOLgW6bEi+ftrALtsgALeB2Adw==",
       "dev": true,
       "requires": {
@@ -10647,7 +10671,7 @@
     },
     "readable-stream": {
       "version": "2.3.6",
-      "resolved": "http://registry.npmjs.org/readable-stream/-/readable-stream-2.3.6.tgz",
+      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.6.tgz",
       "integrity": "sha512-tQtKA9WIAhBF3+VLAseyMqZeBjW0AHJoxOtYqSUZNJxauErmLbVm2FW1y+J/YA9dUrAC39ITejlZWhVIwawkKw==",
       "dev": true,
       "requires": {
@@ -11669,7 +11693,7 @@
         },
         "minimist": {
           "version": "1.2.0",
-          "resolved": "http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
           "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ=",
           "dev": true
         },
@@ -11856,7 +11880,7 @@
     },
     "sha.js": {
       "version": "2.4.11",
-      "resolved": "http://registry.npmjs.org/sha.js/-/sha.js-2.4.11.tgz",
+      "resolved": "https://registry.npmjs.org/sha.js/-/sha.js-2.4.11.tgz",
       "integrity": "sha512-QMEp5B7cftE7APOjk5Y6xgrbWu+WkLVQwk8JNjZ8nKRciZaByEW6MubieAiToS7+dwvrjGhH8jRXz3MVd0AYqQ==",
       "dev": true,
       "requires": {
@@ -12310,7 +12334,7 @@
     },
     "strip-ansi": {
       "version": "3.0.1",
-      "resolved": "http://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz",
+      "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz",
       "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=",
       "dev": true,
       "requires": {
@@ -12546,7 +12570,7 @@
     },
     "through": {
       "version": "2.3.8",
-      "resolved": "http://registry.npmjs.org/through/-/through-2.3.8.tgz",
+      "resolved": "https://registry.npmjs.org/through/-/through-2.3.8.tgz",
       "integrity": "sha1-DdTJ/6q8NXlgsbckEV1+Doai4fU=",
       "dev": true
     },
@@ -13202,7 +13226,7 @@
       "dependencies": {
         "minimist": {
           "version": "1.2.0",
-          "resolved": "http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
           "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ=",
           "dev": true
         }
@@ -13865,7 +13889,7 @@
     },
     "wrap-ansi": {
       "version": "2.1.0",
-      "resolved": "http://registry.npmjs.org/wrap-ansi/-/wrap-ansi-2.1.0.tgz",
+      "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-2.1.0.tgz",
       "integrity": "sha1-2Pw9KE3QV5T+hJc8rs3Rz4JP3YU=",
       "dev": true,
       "requires": {
@@ -13973,7 +13997,7 @@
     },
     "yargs": {
       "version": "11.1.0",
-      "resolved": "http://registry.npmjs.org/yargs/-/yargs-11.1.0.tgz",
+      "resolved": "https://registry.npmjs.org/yargs/-/yargs-11.1.0.tgz",
       "integrity": "sha512-NwW69J42EsCSanF8kyn5upxvjp5ds+t3+udGBeTbFnERA+lF541DDpMawzo4z6W/QrzNM18D+BPMiOBibnFV5A==",
       "dev": true,
       "requires": {

client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cellxgene",
-  "version": "0.0.1",
+  "version": "0.3.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
@@ -34,6 +34,7 @@
     "d3": "^4.10.0",
     "d3-scale-chromatic": "^1.3.0",
     "font-color-contrast": "^1.0.3",
+    "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
     "gl-matrix": "^2.7.1",
     "key-pressed": "0.0.1",

client/src/actions/index.js

@@ -24,7 +24,7 @@ const doInitialDataLoad = () =>
         "config",
         "schema",
         "annotations/obs",
-        "annotations/var",
+        "annotations/var?annotation-name=name",
         "layout/obs"
       ])
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
@@ -90,12 +90,16 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
   const state = getState();
   const { universe } = state.controls;
   /* preload data already in cache */
-  let expressionData = _.transform(genes, (expData, g) => {
+  let expressionData = _.transform(
-    const data = kvCache.get(universe.varDataCache, g);
+    genes,
-    if (data) {
+    (expData, g) => {
-      expData[g] = data;
+      const data = kvCache.get(universe.varDataCache, g);
-    }
+      if (data) {
-  }); // --> { gene: data }
+        expData[g] = data;
+      }
+    },
+    {}
+  ); // --> { gene: data }
   /* make a list of genes for which we do not have data */
   const genesToFetch = _.filter(genes, g => expressionData[g] === undefined);
 
@@ -119,7 +123,6 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
           }),
           headers: new Headers({
             accept: "application/json",
-            "Accept-Encoding": "gzip, deflate, br",
             "Content-Type": "application/json"
           })
         }
@@ -239,7 +242,6 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
         method: "POST",
         headers: new Headers({
           Accept: "application/json",
-          "Accept-Encoding": "gzip, deflate, br",
           "Content-Type": "application/json"
         }),
         body: JSON.stringify({
@@ -297,6 +299,9 @@ const resetInterface = () => (dispatch, getState) => {
     type: "reset World to eq Universe",
     universe
   });
+  dispatch({
+    type: "increment graph render counter"
+  });
 };
 
 export default {

client/src/components/app.js

@@ -11,7 +11,8 @@ import actions from "../actions";
 
 @connect(state => ({
   loading: state.controls.loading,
-  error: state.controls.error
+  error: state.controls.error,
+  graphRenderCounter: state.controls.graphRenderCounter
 }))
 class App extends React.Component {
   constructor(props) {
@@ -54,7 +55,7 @@ class App extends React.Component {
   }
 
   render() {
-    const { loading } = this.props;
+    const { loading, error, graphRenderCounter } = this.props;
     return (
       <Container>
         <Helmet title="cellxgene" />
@@ -79,10 +80,8 @@ class App extends React.Component {
               marginLeft: 350 /* but responsive */
             }}
           >
-            {loading ? null : <Graph />}
+            {loading ? null : <Graph key={graphRenderCounter} />}
-
             <Legend />
-            {}
           </div>
         </div>
       </Container>

client/src/components/brushableHistogram/index.js

@@ -34,7 +34,7 @@ class HistogramBrush extends React.Component {
       .scaleLinear()
       .range([this.height - this.marginBottom, 0]);
 
-    if (obsAnnotations[0][field]) {
+    if (obsAnnotations[0][field] !== undefined) {
       // recalculate expensive stuff
       const allValuesForContinuousFieldAsArray = _.map(obsAnnotations, field);
 
@@ -215,7 +215,12 @@ class HistogramBrush extends React.Component {
     d3.select(svgRef)
       .append("g")
       .attr("class", "brush")
-      .call(d3.brushX().on("end", this.onBrush(field, x.invert).bind(this)));
+      .call(
+        d3
+          .brushX()
+          .on("brush", this.onBrush(field, x.invert).bind(this))
+          .on("end", this.onBrush(field, x.invert).bind(this))
+      );
 
     /* AXIS */
     d3.select(svgRef)
@@ -243,9 +248,9 @@ class HistogramBrush extends React.Component {
       colorAccessor,
       isUserDefined,
       isDiffExp,
-      avgDiff,
+      logFoldChange,
-      set1AvgExp,
+      pval,
-      set2AvgExp,
+      pvalAdj,
       scatterplotXXaccessor,
       scatterplotYYaccessor,
       zebra
@@ -332,25 +337,17 @@ class HistogramBrush extends React.Component {
             }}
           >
             <span>
-              <strong>1:</strong>
+              <strong>log fold change:</strong>
-              {` ${set1AvgExp.toPrecision(2)}`}
+              {` ${logFoldChange.toPrecision(4)}`}
             </span>
             <span
               style={{
                 marginLeft: 7,
-                backgroundColor: globals.lighterGrey,
                 padding: 2
               }}
             >
-              <strong>2:</strong>
+              <strong>p-value (adj):</strong>
-              {` ${set2AvgExp.toPrecision(2)}`}
+              {pvalAdj < 0.0001 ? " < 0.0001" : ` ${pvalAdj.toFixed(4)}`}
-            </span>
-            <span
-              style={{
-                marginLeft: 7
-              }}
-            >
-              {`Av. Diff: ${avgDiff.toFixed(2)}`}
             </span>
           </div>
         ) : null}

client/src/components/categorical/categorical.js

@@ -5,34 +5,13 @@ import { connect } from "react-redux";
 import * as globals from "../../globals";
 import Category from "./category";
 
-/* Cap the max number of displayed categories */
-const truncateCategories = options => {
-  const numOptions = _.size(options);
-  if (numOptions <= globals.maxCategoricalOptionsToDisplay) {
-    return options;
-  }
-  return _(options)
-    .map((v, k) => ({ name: k, val: v }))
-    .sortBy("val")
-    .slice(numOptions - globals.maxCategoricalOptionsToDisplay)
-    .transform((r, v) => {
-      r[v.name] = v.val;
-    }, {})
-    .value();
-};
-
 @connect(state => ({
-  ranges: _.get(state.controls.world, "summary.obs", null),
+  categoricalSelectionState: state.controls.categoricalSelectionState
-  categorySelectionLimit: _.get(
-    state.config,
-    "parameters.max-category-items",
-    globals.configDefaults.parameters["max-category-items"]
-  )
 }))
 class Categories extends React.Component {
   render() {
-    const { ranges, categorySelectionLimit } = this.props;
+    const { categoricalSelectionState } = this.props;
-    if (!ranges) return null;
+    if (!categoricalSelectionState) return null;
 
     return (
       <div
@@ -47,27 +26,9 @@ class Categories extends React.Component {
         >
           Categorical Metadata
         </p>
-        {_.map(ranges, (value, key) => {
+        {_.map(categoricalSelectionState, (catState, catName) => (
-          const isColorField = key.includes("color") || key.includes("Color");
+          <Category key={catName} metadataField={catName} />
-          const isSelectableCategory =
+        ))}
-            value.options &&
-            !isColorField &&
-            key !== "name" &&
-            value.numOptions < categorySelectionLimit;
-
-          if (isSelectableCategory) {
-            const categoryOptions = truncateCategories(value.options);
-            return (
-              <Category
-                key={key}
-                metadataField={key}
-                values={categoryOptions}
-                isTruncated={categoryOptions !== value.options}
-              />
-            );
-          }
-          return undefined;
-        })}
       </div>
     );
   }

client/src/components/categorical/category.js

@@ -2,29 +2,15 @@ import React from "react";
 import _ from "lodash";
 import { connect } from "react-redux";
 import { FaChevronRight, FaChevronDown } from "react-icons/fa";
-import memoize from "memoize-one";
+import { Button, Tooltip } from "@blueprintjs/core";
-import { Button, Tooltip, Position } from "@blueprintjs/core";
 
 import * as globals from "../../globals";
 import Value from "./value";
 import alphabeticallySortedValues from "./util";
 
-const countCategories = (values, optsAsBools) =>
-  _.reduce(
-    values,
-    (r, v, k) => {
-      r.total += 1;
-      if (optsAsBools[k]) {
-        r.on += 1;
-      }
-      return r;
-    },
-    { total: 0, on: 0 }
-  );
-
 @connect(state => ({
   colorAccessor: state.controls.colorAccessor,
-  categoricalAsBooleansMap: state.controls.categoricalAsBooleansMap
+  categoricalSelectionState: state.controls.categoricalSelectionState
 }))
 class Category extends React.Component {
   constructor(props) {
@@ -33,24 +19,28 @@ class Category extends React.Component {
       isChecked: true,
       isExpanded: false
     };
-    this.countCategories = memoize((values, optsAsBools) =>
-      countCategories(values, optsAsBools)
-    );
   }
 
   componentDidUpdate() {
-    const { categoricalAsBooleansMap, metadataField, values } = this.props;
+    const { categoricalSelectionState, metadataField } = this.props;
-    const categoryCount = this.countCategories(
+    const cat = categoricalSelectionState[metadataField];
-      values,
+    const categoryCount = {
-      categoricalAsBooleansMap[metadataField]
+      // total number of categories in this dimension
-    );
+      totalCatCount: cat.numCategories,
-    if (categoryCount.on === categoryCount.total) {
+      // number of selected options in this category
+      selectedCatCount: _.reduce(
+        cat.categorySelected,
+        (res, cond) => (cond ? res + 1 : res),
+        0
+      )
+    };
+    if (categoryCount.selectedCatCount === categoryCount.totalCatCount) {
       /* everything is on, so not indeterminate */
       this.checkbox.indeterminate = false;
-    } else if (categoryCount.on === 0) {
+    } else if (categoryCount.selectedCatCount === 0) {
       /* nothing is on, so no */
       this.checkbox.indeterminate = false;
-    } else if (categoryCount.on < categoryCount.total) {
+    } else if (categoryCount.selectedCatCount < categoryCount.totalCatCount) {
       /* to be explicit... */
       this.checkbox.indeterminate = true;
     }
@@ -74,11 +64,10 @@ class Category extends React.Component {
   }
 
   toggleNone() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField } = this.props;
     dispatch({
       type: "categorical metadata filter none of these",
-      metadataField,
+      metadataField
-      value
     });
     this.setState({ isChecked: false });
   }
@@ -94,13 +83,15 @@ class Category extends React.Component {
   }
 
   renderCategoryItems() {
-    const { values, metadataField } = this.props;
+    const { categoricalSelectionState, metadataField } = this.props;
-    return _.map(alphabeticallySortedValues(values), (v, i) => (
+
+    const cat = categoricalSelectionState[metadataField];
+    const optTuples = alphabeticallySortedValues([...cat.categoryIndices]);
+    return _.map(optTuples, (tuple, i) => (
       <Value
-        key={v}
+        key={tuple[1]}
         metadataField={metadataField}
-        count={values[v]}
+        categoryIndex={tuple[1]}
-        value={v}
         i={i}
       />
     ));
@@ -108,12 +99,15 @@ class Category extends React.Component {
 
   render() {
     const { isExpanded, isChecked } = this.state;
-    const { metadataField, colorAccessor, isTruncated } = this.props;
+    const {
+      metadataField,
+      colorAccessor,
+      categoricalSelectionState
+    } = this.props;
+    const { isTruncated } = categoricalSelectionState[metadataField];
     return (
       <div
         style={{
-          // display: "flex",
-          // alignItems: "baseline",
           maxWidth: globals.maxControlsWidth
         }}
       >

client/src/components/categorical/util.js

@@ -1,7 +1,11 @@
 // jshint esversion: 6
+
+// values is [ [optVal, optIdx], ...]
+// index is range array
+// return sorted index
 export default values =>
-  Object.keys(values).sort((a, b) => {
+  values.sort((a, b) => {
-    const textA = a.toUpperCase();
+    const textA = String(a[0]).toUpperCase();
-    const textB = b.toUpperCase();
+    const textB = String(b[0]).toUpperCase();
     return textA < textB ? -1 : textA > textB ? 1 : 0;
   });

client/src/components/categorical/value.js

@@ -1,47 +1,63 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
+import _ from "lodash";
 
 @connect(state => ({
-  categoricalAsBooleansMap: state.controls.categoricalAsBooleansMap,
+  categoricalSelectionState: state.controls.categoricalSelectionState,
   colorScale: state.controls.colorScale,
-  colorAccessor: state.controls.colorAccessor
+  colorAccessor: state.controls.colorAccessor,
+  schema: _.get(state.controls.world, "schema", null)
 }))
 class CategoryValue extends React.Component {
   toggleOff() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter deselect",
       metadataField,
-      value
+      categoryIndex
     });
   }
 
   toggleOn() {
-    const { dispatch, metadataField, value } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter select",
       metadataField,
-      value
+      categoryIndex
     });
   }
 
   render() {
     const {
-      categoricalAsBooleansMap,
+      categoricalSelectionState,
       metadataField,
-      count,
+      categoryIndex,
-      value,
       colorAccessor,
       colorScale,
-      i
+      i,
+      schema
     } = this.props;
 
-    if (!categoricalAsBooleansMap) return null;
+    if (!categoricalSelectionState) return null;
+
+    const category = categoricalSelectionState[metadataField];
+    const selected = category.categorySelected[categoryIndex];
+    const count = category.categoryCounts[categoryIndex];
+    const value = category.categoryValues[categoryIndex];
+    const displayString = String(
+      category.categoryValues[categoryIndex]
+    ).valueOf();
 
-    const selected = categoricalAsBooleansMap[metadataField][value];
     /* this is the color scale, so add swatches below */
     const c = metadataField === colorAccessor;
+    let categories = null;
+
+    if (c && schema) {
+      categories = _.filter(schema.annotations.obs, {
+        name: colorAccessor
+      })[0].categories;
+    }
 
     return (
       <div
@@ -68,7 +84,7 @@ class CategoryValue extends React.Component {
               type="checkbox"
             />
             <span className="bp3-control-indicator" />
-            {value}
+            {displayString}
           </label>
         </div>
         <span>
@@ -78,7 +94,10 @@ class CategoryValue extends React.Component {
               marginLeft: 5,
               width: 11,
               height: 11,
-              backgroundColor: c ? colorScale(value) : "inherit"
+              backgroundColor:
+                c && categories
+                  ? colorScale(categories.indexOf(value))
+                  : "inherit"
             }}
           />
         </span>

client/src/components/continuousLegend/index.js

@@ -2,7 +2,7 @@
 import React from "react";
 import { connect } from "react-redux";
 import * as d3 from "d3";
-import { interpolateViridis } from "d3-scale-chromatic";
+import { interpolateViridis, interpolateCool } from "d3-scale-chromatic";
 
 // create continuous color legend
 // http://bl.ocks.org/syntagmatic/e8ccca52559796be775553b467593a9f
@@ -121,12 +121,12 @@ class ContinuousLegend extends React.Component {
         .remove();
     }
 
-    if (colorAccessor && colorScale) {
+    if (colorAccessor && colorScale && colorScale.range) {
       /* fragile! continuous range is 0 to 1, not [#fa4b2c, ...], make this a flag? */
       if (colorScale.range()[0][0] !== "#") {
         continuous(
           "#continuous_legend",
-          d3.scaleSequential(interpolateViridis).domain(colorScale.domain()),
+          d3.scaleSequential(interpolateCool).domain(colorScale.domain()),
           colorAccessor
         );
       }

client/src/components/geneExpression/index.js

@@ -4,14 +4,51 @@
 import React from "react";
 import _ from "lodash";
 import * as d3 from "d3";
+import fuzzysort from "fuzzysort";
+
 import { connect } from "react-redux";
-import { Button, Tooltip } from "@blueprintjs/core";
+import { MenuItem } from "@blueprintjs/core";
+import { Suggest } from "@blueprintjs/select";
 import HistogramBrush from "../brushableHistogram";
 import * as globals from "../../globals";
 import actions from "../../actions";
 import { postUserErrorToast } from "../framework/toasters";
 import ExpressionButtons from "./expressionButtons";
 
+const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
+  if (!modifiers.matchesPredicate) {
+    return null;
+  }
+  /* the fuzzysort wraps the object with other properties, like a score */
+  const gene = fuzzySortResult.obj;
+  const text = gene.name;
+
+  return (
+    <MenuItem
+      active={modifiers.active}
+      disabled={modifiers.disabled}
+      // Use of annotations in this way is incorrect and dataset specific.
+      // See https://github.com/chanzuckerberg/cellxgene/issues/483
+      // label={gene.n_counts}
+      key={gene.name}
+      onClick={g => {
+        /* this fires when user clicks a menu item */
+        handleClick(g);
+      }}
+      text={text}
+    />
+  );
+};
+
+const filterGenes = (query, genes) => {
+  /* fires on load, once, and then for each character typed into the input */
+  return fuzzysort.go(query, genes, {
+    key: "name",
+    limit: 5,
+    threshold: -10000 // don't return bad results
+  });
+};
+
 @connect(state => {
   const metadata = _.get(state.controls.world, "obsAnnotations", null);
   const ranges = _.get(state.controls.world, "summary.obs", null);
@@ -29,23 +66,9 @@ import ExpressionButtons from "./expressionButtons";
   };
 })
 class GeneExpression extends React.Component {
-  constructor(props) {
+  handleClick(g) {
-    super(props);
-    this.state = {
-      gene: ""
-    };
-  }
-
-  keyPress(e) {
-    if (e.keyCode === 13) {
-      this.handleClick();
-    }
-  }
-
-  handleClick() {
     const { world, dispatch, userDefinedGenes } = this.props;
-    const { gene } = this.state;
+    const gene = g.target;
-
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
     } else if (userDefinedGenes.length > 15) {
@@ -60,13 +83,11 @@ class GeneExpression extends React.Component {
         type: "user defined gene",
         data: gene
       });
-      this.setState({ gene: "" });
     }
   }
 
   render() {
     const { world, userDefinedGenes, differential } = this.props;
-    const { gene } = this.state;
 
     return (
       <div>
@@ -87,27 +108,27 @@ class GeneExpression extends React.Component {
             style={{ padding: globals.leftSidebarSectionPadding }}
             className="bp3-control-group"
           >
-            <div className="bp3-input-group bp3-fill">
+            <Suggest
-              <input
+              closeOnSelect
-                onKeyDown={this.keyPress.bind(this)}
+              openOnKeyDown
-                onChange={e => {
+              resetOnSelect
-                  this.setState({ gene: e.target.value });
+              noResults={<MenuItem disabled text="No matching genes." />}
-                }}
+              onItemSelect={g => {
-                value={gene}
+                /* this happens on 'enter' */
-                type="text"
+                this.handleClick(g);
-                className="bp3-input"
+              }}
-                placeholder="Enter a gene name"
+              inputValueRenderer={g => {
-                style={{ paddingRight: 94 }}
+                return "";
-              />
+              }}
-            </div>
+              itemListPredicate={filterGenes}
-            <Tooltip
+              itemRenderer={renderGene.bind(this)}
-              content="Add a gene to see its expression levels"
+              items={
-              position="bottom"
+                world && world.varAnnotations
-            >
+                  ? world.varAnnotations
-              <Button intent="primary" onClick={this.handleClick.bind(this)}>
+                  : [{ name: "No genes", n_counts: "" }]
-                Add
+              }
-              </Button>
+              popoverProps={{ minimal: true }}
-            </Tooltip>
+            />
           </div>
           {world && userDefinedGenes.length > 0
             ? _.map(userDefinedGenes, (geneName, index) => {
@@ -152,9 +173,9 @@ class GeneExpression extends React.Component {
                     zebra={index % 2 === 0}
                     ranges={d3.extent(values)}
                     isDiffExp
-                    avgDiff={value[1]}
+                    logFoldChange={value[1]}
-                    set1AvgExp={value[4]}
+                    pval={value[2]}
-                    set2AvgExp={value[5]}
+                    pvalAdj={value[3]}
                   />
                 );
               })

client/src/components/graph/drawPointsRegl.js

@@ -38,7 +38,7 @@ export default function(regl) {
     uniforms: {
       distance: regl.prop("distance"),
       view: regl.prop("view"),
-      projection: () => mat4.perspective([], Math.PI / 2, 1, 0.01, 1000)
+      projection: ({viewportWidth, viewportHeight}) => mat4.perspective([], Math.PI / 2, viewportWidth / viewportHeight, 0.01, 1000)
     },
 
     count: regl.prop("count"),

client/src/components/graph/graph.js

@@ -6,8 +6,7 @@ import { connect } from "react-redux";
 import mat4 from "gl-mat4";
 import _regl from "regl";
 import { Button, AnchorButton, Tooltip } from "@blueprintjs/core";
-import { worldEqUniverse } from "../../util/stateManager/world";
+import * as globals from "../../globals";
-
 import setupSVGandBrushElements from "./setupSVGandBrush";
 import actions from "../../actions";
 import _camera from "../../util/camera";
@@ -30,9 +29,9 @@ class Graph extends React.Component {
     super(props);
     this.count = 0;
     this.inverse = mat4.identity([]);
-    this.graphPaddingTop = 100;
+    this.graphPaddingTop = 0;
     this.graphPaddingBottom = 45;
-    this.graphPaddingRight = 10;
+    this.graphPaddingRight = globals.leftSidebarWidth;
     this.renderCache = {
       positions: null,
       colors: null
@@ -126,18 +125,24 @@ class Graph extends React.Component {
         const glScaleX = scaleLinear([0, 1], [-1, 1]);
         const glScaleY = scaleLinear([0, 1], [1, -1]);
 
+        const offset = [d3.mean(obsLayout.X) - 0.5, d3.mean(obsLayout.Y) - 0.5];
+
         for (
           let i = 0, { positions } = this.renderCache;
           i < cellCount;
           i += 1
         ) {
-          positions[2 * i] = glScaleX(obsLayout.X[i]);
+          positions[2 * i] = glScaleX(obsLayout.X[i] - offset[0]);
-          positions[2 * i + 1] = glScaleY(obsLayout.Y[i]);
+          positions[2 * i + 1] = glScaleY(obsLayout.Y[i] - offset[1]);
         }
         pointBuffer({
           data: this.renderCache.positions,
           dimension: 2
         });
+
+        this.setState({
+          offset
+        });
       }
 
       // Colors for each point - a cached value that only changes when
@@ -196,7 +201,7 @@ class Graph extends React.Component {
         this.handleBrushSelectAction.bind(this),
         this.handleBrushDeselectAction.bind(this),
         responsive,
-        this.graphPaddingTop
+        this.graphPaddingRight
       );
       this.setState({ svg: newSvg, brush });
     }
@@ -251,7 +256,7 @@ class Graph extends React.Component {
     an event on procedural deselect because it is move: null
     */
 
-    const { camera } = this.state;
+    const { camera, offset } = this.state;
     const { dispatch, responsive } = this.props;
 
     if (d3.event.sourceEvent !== null) {
@@ -262,6 +267,7 @@ class Graph extends React.Component {
       https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
     */
       const s = d3.event.selection;
+      const gl = this.state.regl._gl;
       /*
       event describing brush position:
       @-------|
@@ -270,19 +276,23 @@ class Graph extends React.Component {
       |-------@
     */
 
+      // get aspect ratio
+      const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
+
       // compute inverse view matrix
       const inverse = mat4.invert([], camera.view());
 
       // transform screen coordinates -> cell coordinates
       const invert = pin => {
-        const x = (2 * pin[0]) / (responsive.height - this.graphPaddingTop) - 1;
+        const x =
+          (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
         const y =
           2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
         const pout = [
-          x * inverse[14] + inverse[12],
+          x * inverse[14] * aspect + inverse[12],
           y * inverse[14] + inverse[13]
         ];
-        return [(pout[0] + 1) / 2, (pout[1] + 1) / 2];
+        return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
       };
 
       const brushCoords = {
@@ -366,6 +376,7 @@ class Graph extends React.Component {
                 style={{ marginRight: 10 }}
                 onClick={() => {
                   dispatch(actions.regraph());
+                  dispatch({ type: "increment graph render counter" });
                 }}
               >
                 subset to current selection
@@ -421,12 +432,10 @@ class Graph extends React.Component {
         </div>
         <div
           style={{
-            marginRight: 50,
-            marginTop: 50,
             zIndex: -9999,
             position: "fixed",
-            right: this.graphPaddingRight,
+            top: 0,
-            bottom: this.graphPaddingBottom
+            right: 0
           }}
         >
           <div
@@ -437,7 +446,7 @@ class Graph extends React.Component {
           />
           <div style={{ padding: 0, margin: 0 }}>
             <canvas
-              width={responsive.height - this.graphPaddingTop}
+              width={responsive.width - this.graphPaddingRight}
               height={responsive.height - this.graphPaddingTop}
               ref={canvas => {
                 this.reglCanvas = canvas;

client/src/components/graph/setupSVGandBrush.js

@@ -12,22 +12,18 @@ export default (
   handleBrushSelectAction,
   handleBrushDeselectAction,
   responsive,
-  graphPaddingTop
+  graphPaddingRight
 ) => {
-  const side = responsive.height - graphPaddingTop;
   const svg = d3
     .select("#graphAttachPoint")
     .append("svg")
-    .attr("width", side)
+    .attr("width", responsive.width - graphPaddingRight)
-    .attr("height", side)
+    .attr("height", responsive.height)
     .attr("class", `${styles.graphSVG}`);
 
   const brush = d3
     .brush()
-    .extent([
+    .extent([[0, 0], [responsive.width - graphPaddingRight, responsive.height]])
-      [0, 0],
-      [responsive.height - graphPaddingTop, responsive.height - graphPaddingTop]
-    ])
     .on("brush", handleBrushSelectAction)
     .on("end", handleBrushDeselectAction);
 

client/src/components/scatterplot/drawPointsRegl.js

@@ -38,14 +38,7 @@ export default function(regl) {
     uniforms: {
       distance: regl.prop("distance"),
       view: regl.prop("view"),
-      projection: (context, props) =>
+      projection: () => mat4.perspective([], Math.PI / 2, 1, 0.01, 1000)
-        mat4.perspective(
-          [],
-          Math.PI / 2,
-          (context.viewportWidth * props.scale) / context.viewportHeight,
-          0.01,
-          1000
-        )
     },
 
     count: regl.prop("count"),

client/src/components/scatterplot/scatterplot.js

@@ -82,12 +82,6 @@ class Scatterplot extends React.Component {
       this.drawAxesSVG(scales.xScale, scales.yScale, svg);
     }
 
-    this.setState({
-      svg,
-      xScale: scales ? scales.xScale : null,
-      yScale: scales ? scales.yScale : null
-    });
-
     const camera = _camera(this.reglCanvas, { scale: true, rotate: false });
     const regl = _regl(this.reglCanvas);
 
@@ -98,43 +92,35 @@ class Scatterplot extends React.Component {
     const colorBuffer = regl.buffer();
     const sizeBuffer = regl.buffer();
 
-    regl.frame(({ viewportWidth, viewportHeight }) => {
+    const reglRender = regl.frame(() => {
-      regl.clear({
+      this.reglDraw(
-        depth: 1,
+        regl,
-        color: [1, 1, 1, 1]
+        drawPoints,
-      });
+        sizeBuffer,
-
+        colorBuffer,
-      drawPoints({
+        pointBuffer,
-        distance: camera.distance,
+        camera
-        color: colorBuffer,
+      );
-        position: pointBuffer,
-        size: sizeBuffer,
-        count: this.count,
-        view: camera.view(),
-        scale: viewportHeight / viewportWidth
-      });
-
       camera.tick();
     });
 
+    this.reglRenderState = "rendering";
+
     this.setState({
       regl,
       sizeBuffer,
       pointBuffer,
-      colorBuffer
+      colorBuffer,
+      svg,
+      xScale: scales ? scales.xScale : null,
+      yScale: scales ? scales.yScale : null,
+      reglRender,
+      camera,
+      drawPoints
     });
   }
 
   componentDidUpdate(prevProps) {
-    const {
-      svg,
-      xScale,
-      yScale,
-      regl,
-      pointBuffer,
-      colorBuffer,
-      sizeBuffer
-    } = this.state;
     const {
       world,
       crossfilter,
@@ -144,6 +130,18 @@ class Scatterplot extends React.Component {
       expressionY,
       colorRGB
     } = this.props;
+    const {
+      reglRender,
+      xScale,
+      yScale,
+      regl,
+      pointBuffer,
+      colorBuffer,
+      sizeBuffer,
+      svg,
+      drawPoints,
+      camera
+    } = this.state;
 
     if (
       world &&
@@ -159,6 +157,11 @@ class Scatterplot extends React.Component {
       this.drawAxesSVG(xScale, yScale, svg);
     }
 
+    if (reglRender && this.reglRenderState === "rendering") {
+      reglRender.cancel();
+      this.reglRenderState = "paused";
+    }
+
     if (
       world &&
       regl &&
@@ -198,6 +201,16 @@ class Scatterplot extends React.Component {
       colorBuffer({ data: colorsBuf, dimension: 3 });
       sizeBuffer({ data: sizesBuf, dimension: 1 });
       this.count = cellCount;
+
+      regl._refresh();
+      this.reglDraw(
+        regl,
+        drawPoints,
+        sizeBuffer,
+        colorBuffer,
+        pointBuffer,
+        camera
+      );
     }
 
     if (
@@ -227,6 +240,22 @@ class Scatterplot extends React.Component {
     };
   }
 
+  reglDraw(regl, drawPoints, sizeBuffer, colorBuffer, pointBuffer, camera) {
+    regl.clear({
+      depth: 1,
+      color: [1, 1, 1, 1]
+    });
+
+    drawPoints({
+      size: sizeBuffer,
+      distance: camera.distance,
+      color: colorBuffer,
+      position: pointBuffer,
+      count: this.count,
+      view: camera.view()
+    });
+  }
+
   drawAxesSVG(xScale, yScale, svg) {
     const { scatterplotYYaccessor, scatterplotXXaccessor } = this.props;
     svg.selectAll("*").remove();

client/src/middleware/updateCellColors.js

@@ -1,7 +1,13 @@
 // jshint esversion: 6
 import _ from "lodash";
 import * as d3 from "d3";
-import { interpolateViridis } from "d3-scale-chromatic";
+import {
+  interpolateViridis,
+  interpolateSpectral,
+  interpolateRainbow,
+  interpolateBlues,
+  interpolateCool
+} from "d3-scale-chromatic";
 import * as globals from "../globals";
 import parseRGB from "../util/parseRGB";
 
@@ -59,11 +65,17 @@ const updateCellColorsMiddleware = store => next => action => {
   */
 
   if (action.type === "color by categorical metadata") {
-    colorScale = d3.scaleOrdinal().range(globals.ordinalColors);
+    const categories = _.filter(s.controls.world.schema.annotations.obs, {
+      name: action.colorAccessor
+    })[0].categories;
+
+    colorScale = d3
+      .scaleSequential(interpolateRainbow)
+      .domain([0, categories.length]);
 
     for (let i = 0; i < obsAnnotations.length; i += 1) {
       const obs = obsAnnotations[i];
-      const c = colorScale(obs[action.colorAccessor]);
+      const c = colorScale(categories.indexOf(obs[action.colorAccessor]));
       colorsByName[i] = c;
       colorsByRGB[i] = parseRGB(c);
     }
@@ -77,7 +89,7 @@ const updateCellColorsMiddleware = store => next => action => {
 
     for (let i = 0; i < obsAnnotations.length; i += 1) {
       const obs = obsAnnotations[i];
-      const c = interpolateViridis(colorScale(obs[action.colorAccessor]));
+      const c = interpolateCool(colorScale(obs[action.colorAccessor]));
       colorsByName[i] = c;
       colorsByRGB[i] = parseRGB(c);
     }
@@ -95,7 +107,7 @@ const updateCellColorsMiddleware = store => next => action => {
       ]); /* invert viridis... probably pass this scale through to others */
 
     for (let i = 0, len = expression.length; i < len; i += 1) {
-      const c = interpolateViridis(colorScale(expression[i]));
+      const c = interpolateCool(colorScale(expression[i]));
       colorsByName[i] = c;
       colorsByRGB[i] = parseRGB(c);
     }

client/src/reducers/controls.js

@@ -1,7 +1,7 @@
 // jshint esversion: 6
 
 import _ from "lodash";
-import { World, kvCache } from "../util/stateManager";
+import { World, kvCache, WorldUtil } from "../util/stateManager";
 import parseRGB from "../util/parseRGB";
 import Crossfilter from "../util/typedCrossfilter";
 import * as globals from "../globals";
@@ -12,34 +12,109 @@ import {
   diffexpDimensionName,
   makeContinuousDimensionName
 } from "../util/nameCreators";
+import { fillRange } from "../util/typedCrossfilter/util";
 
-function createCategoricalAsBooleansMap(world) {
+/*
+Selection state for categoricals are tracked in an Object that
+has two main components for each category:
+1. mapping of option value to an index
+2. array of bool selection state by index
+Remember that option values can be ANY js type, except undefined/null.
+
+  {
+    _category_name_1: {
+      // map of option value to index
+      categoryIndices: Map([
+        catval1: index,
+        ...
+      ])
+
+      // index->selection true/false state
+      categorySelected: [ true/false, true/false, ... ]
+
+      // number of options
+      numCategories: number,
+
+      // isTruncated - true if the options for selection has
+      // been truncated (ie, was too large to implement)
+    }
+  }
+*/
+function topNCategories(summary) {
+  const counts = _.map(summary.categories, cat =>
+    summary.categoryCounts.get(cat)
+  );
+  const sortIndex = fillRange(new Array(summary.numCategories)).sort(
+    (a, b) => counts[b] - counts[a]
+  );
+  const sortedCategories = _.map(sortIndex, i => summary.categories[i]);
+  const sortedCounts = _.map(sortIndex, i => counts[i]);
+  const N = globals.maxCategoricalOptionsToDisplay;
+
+  if (sortedCategories.length < N) {
+    return [sortedCategories, sortedCounts];
+  }
+  return [sortedCategories.slice(0, N), sortedCounts.slice(0, N)];
+}
+
+function createCategoricalSelectionState(state, world) {
   const res = {};
-  _.each(world.summary.obs, (value, key) => {
+  _.forEach(world.summary.obs, (value, key) => {
-    if (value.options && key !== "name") {
+    if (value.categories) {
-      const optionsAsBooleans = {};
+      const isColorField = key.includes("color") || key.includes("Color");
-      _.each(value.options, (_value, _key) => {
+      const isSelectableCategory =
-        optionsAsBooleans[_key] = true;
+        !isColorField &&
-      });
+        key !== "name" &&
-      res[key] = optionsAsBooleans;
+        value.categories.length < state.maxCategoryItems;
+      if (isSelectableCategory) {
+        const [categoryValues, categoryCounts] = topNCategories(value);
+        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
+        const numCategories = categoryIndices.size;
+        const categorySelected = new Array(numCategories).fill(true);
+        const isTruncated = categoryValues.length < value.numCategories;
+        res[key] = {
+          categoryValues, // array: of natively typed category values
+          categoryIndices, // map: category value (native type) -> category index
+          categorySelected, // array: t/f selection state
+          numCategories, // number: of categories
+          isTruncated, // bool: true if list was truncated
+          categoryCounts // array: cardinality of each category
+        };
+      }
     }
   });
   return res;
 }
 
+/*
+given a categoricalSelectionState, return the list of all category values
+where selection state is true (ie, they are selected).
+*/
+function selectedValuesForCategory(categorySelectionState) {
+  const selectedValues = _([...categorySelectionState.categoryIndices])
+    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
+    .map(tuple => tuple[0])
+    .value();
+  return selectedValues;
+}
+
 const Controls = (
   state = {
     // data loading flag
     loading: false,
     error: null,
 
+    // configuration
+    maxCategoryItems: globals.configDefaults.parameters["max-category-items"],
+
+    // the whole big bang
     universe: null,
 
     // all of the data + selection state
     world: null,
     colorName: null,
     colorRGB: null,
-    categoricalAsBooleansMap: null,
+    categoricalSelectionState: null,
     crossfilter: null,
     dimensionMap: null,
     userDefinedGenes: [],
@@ -54,6 +129,7 @@ const Controls = (
     scatterplotXXaccessor: null, // just easier to read
     scatterplotYYaccessor: null,
     axesHaveBeenDrawn: false,
+    graphRenderCounter: 0 /* integer as <Component key={graphRenderCounter} - a change in key forces a remount */,
     __storedStateForCelllist1__: null /* will need procedural control of brush ie., brush.extent https://bl.ocks.org/micahstubbs/3cda05ca68cba260cb81 */,
     __storedStateForCelllist2__: null
   },
@@ -71,6 +147,17 @@ const Controls = (
           Initialization, World/Universe management
           and data loading.
     ******************************************************/
+    case "configuration load complete": {
+      // there are a couple of configuration items we need to retain
+      return {
+        ...state,
+        maxCategoryItems: _.get(
+          state.config,
+          "parameters.max-category-items",
+          globals.configDefaults.parameters["max-category-items"]
+        )
+      };
+    }
     case "initial data load start": {
       return { ...state, loading: true };
     }
@@ -82,9 +169,13 @@ const Controls = (
       const world = World.createWorldFromEntireUniverse(universe);
       const colorName = new Array(universe.nObs).fill(globals.defaultCellColor);
       const colorRGB = _.map(colorName, c => parseRGB(c));
-      const categoricalAsBooleansMap = createCategoricalAsBooleansMap(world);
+      const categoricalSelectionState = createCategoricalSelectionState(
+        state,
+        world
+      );
       const crossfilter = Crossfilter(world.obsAnnotations);
       const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+      WorldUtil.clearCaches();
 
       const worldVarDataCache = world.varDataCache;
 
@@ -134,7 +225,7 @@ const Controls = (
         world,
         colorName,
         colorRGB,
-        categoricalAsBooleansMap,
+        categoricalSelectionState,
         crossfilter,
         dimensionMap,
         colorAccessor: null
@@ -151,9 +242,13 @@ const Controls = (
       );
       const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
       const colorRGB = _.map(colorName, c => parseRGB(c));
-      const categoricalAsBooleansMap = createCategoricalAsBooleansMap(world);
+      const categoricalSelectionState = createCategoricalSelectionState(
+        state,
+        world
+      );
       const crossfilter = Crossfilter(world.obsAnnotations);
       const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+      WorldUtil.clearCaches();
 
       const worldVarDataCache = world.varDataCache;
       /* var dimensions */
@@ -196,7 +291,7 @@ const Controls = (
         world,
         colorName,
         colorRGB,
-        categoricalAsBooleansMap,
+        categoricalSelectionState,
         crossfilter,
         dimensionMap,
         colorAccessor: null
@@ -410,88 +505,100 @@ const Controls = (
         ...state,
         opacityForDeselectedCells: action.data
       };
-
+    case "increment graph render counter": {
+      const c = state.graphRenderCounter + 1;
+      return {
+        ...state,
+        graphRenderCounter: c
+      };
+    }
     /*******************************
           Categorical metadata
     *******************************/
     case "categorical metadata filter select": {
-      const newCategoricalAsBooleansMap = {
+      const newCategorySelected = Array.from(
-        ...state.categoricalAsBooleansMap,
+        state.categoricalSelectionState[action.metadataField].categorySelected
+      );
+      newCategorySelected[action.categoryIndex] = true;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
         [action.metadataField]: {
-          ...state.categoricalAsBooleansMap[action.metadataField],
+          ...state.categoricalSelectionState[action.metadataField],
-          [action.value]: true
+          categorySelected: newCategorySelected
         }
       };
-      // update the filter for the one category that changed state
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
       state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        _.filter(
+        selectedValuesForCategory(cat)
-          _.map(
-            newCategoricalAsBooleansMap[action.metadataField],
-            (val, key) => (val ? key : false)
-          )
-        )
       );
+
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
     case "categorical metadata filter deselect": {
-      const newCategoricalAsBooleansMap = {
+      const newCategorySelected = Array.from(
-        ...state.categoricalAsBooleansMap,
+        state.categoricalSelectionState[action.metadataField].categorySelected
+      );
+      newCategorySelected[action.categoryIndex] = false;
+      const newCategoricalSelectionState = {
+        ...state.categoricalSelectionState,
         [action.metadataField]: {
-          ...state.categoricalAsBooleansMap[action.metadataField],
+          ...state.categoricalSelectionState[action.metadataField],
-          [action.value]: false
+          categorySelected: newCategorySelected
         }
       };
-      // update the filter for the one category that changed state
+
+      // update the filter to match all selected options
+      const cat = newCategoricalSelectionState[action.metadataField];
       state.dimensionMap[obsAnnoDimensionName(action.metadataField)].filterEnum(
-        _.filter(
+        selectedValuesForCategory(cat)
-          _.map(
-            newCategoricalAsBooleansMap[action.metadataField],
-            (val, key) => (val ? key : false)
-          )
-        )
       );
+
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
     case "categorical metadata filter none of these": {
-      const newCategoricalAsBooleansMap = {
+      const newCategoricalSelectionState = {
-        ...state.categoricalAsBooleansMap
+        ...state.categoricalSelectionState,
-      };
+        [action.metadataField]: {
-      _.forEach(
+          ...state.categoricalSelectionState[action.metadataField],
-        newCategoricalAsBooleansMap[action.metadataField],
+          categorySelected: Array.from(
-        (v, k, c) => {
+            state.categoricalSelectionState[action.metadataField]
-          c[k] = false;
+              .categorySelected
+          ).fill(false)
         }
-      );
+      };
       state.dimensionMap[
         obsAnnoDimensionName(action.metadataField)
       ].filterNone();
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
     case "categorical metadata filter all of these": {
-      const newCategoricalAsBooleansMap = {
+      const newCategoricalSelectionState = {
-        ...state.categoricalAsBooleansMap
+        ...state.categoricalSelectionState,
-      };
+        [action.metadataField]: {
-      _.forEach(
+          ...state.categoricalSelectionState[action.metadataField],
-        newCategoricalAsBooleansMap[action.metadataField],
+          categorySelected: Array.from(
-        (v, k, c) => {
+            state.categoricalSelectionState[action.metadataField]
-          c[k] = true;
+              .categorySelected
+          ).fill(true)
         }
-      );
+      };
       state.dimensionMap[
         obsAnnoDimensionName(action.metadataField)
       ].filterAll();
       return {
         ...state,
-        categoricalAsBooleansMap: newCategoricalAsBooleansMap
+        categoricalSelectionState: newCategoricalSelectionState
       };
     }
 

client/src/util/actionHelpers.js

@@ -31,15 +31,18 @@ export const doJsonRequest = async url => {
   const res = await fetch(url, {
     method: "get",
     headers: new Headers({
-      "Content-Type": "application/json",
+      "Content-Type": "application/json"
-      "Accept-Encoding": "gzip, deflate, br"
     })
   });
   if (res.ok && res.headers.get("Content-Type") === "application/json") {
     return res.json();
   }
   // else an error
-  const msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  let msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  const body = await res.text();
+  if (body && body.length > 0) {
+    msg = `${msg} -- ${body}`;
+  }
   dispatchNetworkErrorMessageToUser(msg);
   throw new Error(msg);
 };

client/src/util/stateManager/index.js

@@ -17,3 +17,4 @@ exists to support those concepts.
 export * as Universe from "./universe";
 export * as World from "./world";
 export * as kvCache from "./keyvalcache";
+export * as WorldUtil from "./worldUtil";

client/src/util/stateManager/summarizeAnnotations.js

@@ -8,6 +8,7 @@ Value will be an object, containing summary information.
 
 For continuous annotations (int, float, etc):
   <annotation_name>: {
+    categorical: false,
     range {
       min: <number>,
       max: <number>
@@ -15,12 +16,14 @@ For continuous annotations (int, float, etc):
   }
 
 For categorical annotations (boolean, string, category):
-  <annotatoin_name>: {
+  <annotation_name>: {
-    options: {
+    categorical: true,
-      <option1>: <number>,
+    categories: [ <category1>, <category2>, ... ]
+    categoryCounts: Map {
+      <category1>: <number>,
       ...
     },
-    numOptions: <number>
+    numCategories: <number>
   }
 
 Summarize will be returned for BOTH obs and var annotations.
@@ -28,19 +31,19 @@ Summarize will be returned for BOTH obs and var annotations.
 Example:
   {
     "Splice_sites_Annotated": {
-      "range": {
+      categorical: false,
+      range: {
         "min": 26,
         "max": 1075869
       }
     },
     "Selection": {
-      numOptions, 6,
+      categorical: true,
-      "options": {
+      numCategories, 3,
+      categories: [ "Astrocytes(HEPACAM)", "Endothelial(BSC)", "Unpanned" ],
+      categoryCounts: Map {
         "Astrocytes(HEPACAM)": 714,
         "Endothelial(BSC)": 123,
-        "Oligodendrocytes(GC)": 294,
-        "Neurons(Thy1)": 685,
-        "Microglia(CD45)": 1108,
         "Unpanned": 665
       }
     }
@@ -49,37 +52,45 @@ Example:
 NOTE: will not summarize the required 'name' annotation, as that is
 specified as unique per element.
 */
-function summarizeDimension(schema, annotations) {
+function _summarizeAnnotations(_schema, annotations) {
-  return _(schema)
+  const summary = _(_schema) // lodash wrapping: https://lodash.com/docs/4.17.11#lodash
     .filter(v => v.name !== "name")
     .keyBy("name")
     .mapValues(anno => {
       const { name, type } = anno;
       const continuous = type === "int32" || type === "float32";
 
-      if (!continuous) {
-        const options = _.countBy(annotations, name);
-        const numOptions = _.size(options);
-        return {
-          numOptions,
-          options
-        };
-      }
-
       if (continuous) {
         let min = Number.POSITIVE_INFINITY;
         let max = Number.NEGATIVE_INFINITY;
-        _.forEach(annotations, obs => {
+        for (let r = 0; r < annotations.length; r += 1) {
-          const val = Number(obs[name]);
+          const val = Number(annotations[r][name]);
           min = val < min ? val : min;
           max = val > max ? val : max;
-        });
+        }
-        return { range: { min, max } };
+        return {
+          categorical: false,
+          range: { min, max }
+        };
       }
 
-      throw new Error("incomprehensible schema");
+      /* else categorical */
+      const categoryCounts = new Map();
+      for (let r = 0; r < annotations.length; r += 1) {
+        const val = annotations[r][name];
+        let curCount = categoryCounts.get(val);
+        if (curCount === undefined) curCount = 0;
+        categoryCounts.set(val, curCount + 1);
+      }
+      return {
+        categorical: true,
+        categories: [...categoryCounts.keys()],
+        categoryCounts,
+        numCategories: categoryCounts.size
+      };
     })
     .value();
+  return summary;
 }
 
 export default function summarizeAnnotations(
@@ -88,7 +99,7 @@ export default function summarizeAnnotations(
   varAnnotations
 ) {
   return {
-    obs: summarizeDimension(schema.annotations.obs, obsAnnotations),
+    obs: _summarizeAnnotations(schema.annotations.obs, obsAnnotations),
-    var: summarizeDimension(schema.annotations.var, varAnnotations)
+    var: _summarizeAnnotations(schema.annotations.var, varAnnotations)
   };
 }

client/src/util/stateManager/universe.js

@@ -161,6 +161,32 @@ function RESTv02LayoutResponseToInternal(response) {
   return layout;
 }
 
+function reconcileSchemaCategoriesWithSummary(universe) {
+  /*
+  where we treat types as (essentially) categorical metadata, update
+  the schema with data-derived categories (in addition to those in
+  the server declared schema).
+
+  For example, boolean defined fields in the schema do not contain
+  explicit declaration of categories (nor do string fields).  In these
+  cases, add a 'categories' field to the schema so it is accessible.
+  */
+
+  _.forEach(universe.schema.annotations.obs, s => {
+    if (
+      s.type === "string" ||
+      s.type === "boolean" ||
+      s.type === "categorical"
+    ) {
+      const categories = _.union(
+        _.get(s, "categories", []),
+        _.get(universe.summary.obs[s.name], "categories", [])
+      );
+      s.categories = categories;
+    }
+  });
+}
+
 export function createUniverseFromRestV02Response(
   configResponse,
   schemaResponse,
@@ -199,6 +225,7 @@ export function createUniverseFromRestV02Response(
     universe.varAnnotations
   );
 
+  reconcileSchemaCategoriesWithSummary(universe);
   return finalize(universe);
 }
 

client/src/util/stateManager/worldUtil.js

@@ -0,0 +1,65 @@
+/* eslint-disable import/prefer-default-export */
+import _ from "lodash";
+
+/*
+Various utility functions operating on World/Universe
+*/
+
+/*
+Count unique category values, binning first by dim1 then by dim2
+Return:
+
+Map {
+   dim1_val1: Map {
+      dim2_val1: number,
+      dim2_val2: number,
+      ...
+   },
+   ...
+}
+
+*/
+function _countCategoryValues2D(dim1, dim2, rows) {
+  const dimMap = new Map();
+  for (let r = 0; r < rows.length; r += 1) {
+    const row = rows[r];
+    const val1 = row[dim1];
+    const val2 = row[dim2];
+    let d2Map = dimMap.get(val1);
+    if (d2Map === undefined) {
+      d2Map = new Map();
+      dimMap.set(val1, d2Map);
+    }
+    let curCount = d2Map.get(val2);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    d2Map.set(val2, curCount + 1);
+  }
+  return dimMap;
+}
+
+let __worldUtilMemoId__ = 0;
+function _memoizedId(x) {
+  if (!x.__worldUtilMemoId__) {
+    __worldUtilMemoId__ += 1;
+    x.__worldUtilMemoId__ = __worldUtilMemoId__;
+  }
+  return x.__worldUtilMemoId__;
+}
+function _countCategoryValues2DResolver(...args) {
+  const id = args[0] + args[1] + _memoizedId(args[2]);
+  return id;
+}
+
+export const countCategoryValues2D = _.memoize(
+  _countCategoryValues2D,
+  _countCategoryValues2DResolver
+);
+
+/*
+Clear any cached data within WorldUtil caches, eg, memoized functions
+*/
+export function clearCaches() {
+  countCategoryValues2D.cache.clear();
+}

docs/REST_API.md

@@ -75,7 +75,7 @@ For a GET URL query parameter:
 - Annotation name is encoded as `obs:name` or `var:name`<sup>[2](#endnote-2)</sup>.
 - Enumerated values (string, categorical, boolean) are encoded as option lists, ie, `var:tissue=lung, obs:tumor=true`
 - Scalar values (int32, float32) are encoded as ranges, ie, `obs:num_reads=1000,10000` where either min or max may be replaced with an asterisk to indicate a half-open range.
-- Index filters are not be allowed within GET URL query parameter filters
+- Index filters are not allowed within GET URL query parameter filters
 - Logically, filters are ANDed, except for repeated annotation names which are ORed. For example, `?X=A&X=B&Y=1` is evaluated as `((X==A or X==B) and Y==1)`
 
 Example selection for _lung_ and _heart_ tissue with more than 1000 reads:
@@ -505,10 +505,10 @@ Generate differential expression (DE) statistics for two specified subsets of da
 
 Two modes are provided:
 
-- Return top N differentially expressed variables (genes)
+- `topN`: return top N differentially expressed variables (across all variables)
-- Return DE for caller-provided variable filter (future)
+- `varFilter`: return DE for caller-provided variable filter (_future_)
 
-Both modes perform calculations using a subset of observations, where each subset is defined by an observation filter (`set1` and `set2`).
+Both modes perform calculations using a subset of observations, where each subset is defined by an observation filter (`set1` and `set2`). These filters must not include a variable filter.
 
 If differential expression is not supported by the server, must return an HTTP 501 response. If, in the view of the server, the request will exceed a reasonable interactive time period, must immediately return HTTP 403 error (error return _before_ attempting computation).
 
@@ -568,24 +568,22 @@ If differential expression is not supported by the server, must return an HTTP 5
 
 **Response body:**
 
-- For 200 Success, differential expression statistics returned as array of arrays sorted by obs index, where each contains the following values:
+- For 200 Success, differential expression statistics returned as array of arrays, where each contains the following values:
 
   - **varIndex**: variable index for the computed results
-  - **avgDiff**: log fold-change of the average expression between the two groups. Positive values indicate that the gene is more highly expressed in the first group,
+  - **logfoldchange**: log fold-change of the average expression between the two groups. Positive values indicate that the gene is more highly expressed in the first group,
   - **pVal**: unadjusted p-value,
-  - **pValAdj**: Adjusted p-value, based on bonferroni correction using all genes in the original dataset),
+  - **pValAdj**: adjusted p-value
-  - **set1AvgExp:** average expression value for all observations in set 1,
-  - **set2AvgExp**: average expression value for all observations in set 2
 
-  Statistics are encoded as an array of arrays, with fields ordered as:
+  Values ordered as:
 
-  _varIndex_, _avgDiff_, _pVal_, _pValAdj_, _set1AvgExp_, _set2AvgExp_
+  _varIndex_, _logfoldchange_, _pVal_, _pValAdj_
 
   For example:
 
   ```
   [
-    [ 328, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9 ],
+    [ 1720, 2.4679039, 2.3124478092035228e-175, 4.250279073316075e-172 ]
     // ...
   ]
   ```
@@ -616,8 +614,8 @@ POST /diffexp/obs
 200 - Success
 {
   "diffexp": [
-    [ 328, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9 ],
+    [ 328, -2.569489, 2.655706e-63, 3.642036e-57 ],
-    // [ varIdx, avgDiff, pVal, pValAdj, set1AvgExp, set2AvgExp ],
+    // [ varIdx, logfoldchange, pVal, pValAdj ],
     // ...
   ]
 }
@@ -690,10 +688,9 @@ Routes:
 - `GET /schema`
 - `GET /annotations/obs`
 - `GET /annotations/var`
-- `GET /layout/obs`
+- `GET /layout/obs` - get the default layout
 - `PUT /data/obs` - request will contain a filter by var `name`
-- `POST /diffexp/obs` - mode `topN`, typically with a couple of 10, and two sets defined by an obs index filter (`{ filter: { obs: { index: [...] } } }`)
+- `POST /diffexp/obs` - mode `topN`, typically with a `count` of 10, and two sets defined by an obs index filter (`{ filter: { obs: { index: [...] } } }`)
-- `PUT /layout/obs` - (_coming soon_) request will contain a filter by obs index
 
 Requests include the following content negotiation headers:
 

docs/release_process.md

@@ -20,27 +20,36 @@ Follow these steps to create a release.
 
 1.  Preparation:
     - Define the release version number, using [semantic versioning](https://semver.org/)
-    - Write the release title and release notes
+    - Write the release title and release notes and add to 
+    [release notes document](https://docs.google.com/document/d/1KnHwkYfhyWO5H8BDcMu7y3ogjvq5Yi4OwpmZ8DB6w0Y/edit)
 2.  Create a release branch, eg, `release-version`
 3.  In the release branch:
-    - run `bumpversion --config-file .bumpversion.cfg [major | minor | patch]`
+    - Run `bumpversion --config-file .bumpversion.cfg [major | minor | patch]`
-    - build the JS asserts using `bin/build-client`
+    - Clean up existing environment using `bin/clean`
+    - Build the JS asserts using `bin/build-client`
 4.  Commit and push the new branch
 5.  Create a PR for the release.
     - [optional] As needed, conduct PR review.
-6.  Create Github release using the version number and release notes ([instructions](https://help.github.com/articles/creating-releases/)).
+6.  Merge to master
-7.  Publish to pypi by performing the following steps
+7.  Create Github release using the version number and release notes 
-    (assumes you have `setuptools` and `twine` installed and that you have
+([instructions](https://help.github.com/articles/creating-releases/)).
-    registered for pypi and have write access to the cellxgene pypi package)
+    - Draft new release
-    - build the distribution by calling
+    - Type version name matching release version number from (1)
-      `python setup.py sdist`
+    - Select `master` as release branch (ensure you merged the release PR)
+    - Type title `Release {version num}`
+    - [optional] Check pre-release if this release is not ready for production
+    - Publish Release
+8.  Publish to pypi by performing the following steps (assumes you have `setuptools` and `twine` installed and that you 
+have registered for pypi and have write access to the cellxgene pypi package)
+    - Build the distribution by calling
+    `python setup.py sdist`
       inside the top-level directory
-    - [optional] upload the package to test pypi
+    - [optional] Upload the package to test pypi
       `twine upload --repository-url https://test.pypi.org/legacy/ dist/*`
-    - [optional] test the test installation in a fresh virtual environment using
+    - [optional] Test the test installation in a fresh virtual environment using
-      `pip install --index-url https://test.pypi.org/simple/ cellxgene`
+      `pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene`
-    - upload the package to real pypi using `twine upload dist/*`
+    - Upload the package to real pypi using `twine upload dist/*`
-    - [optional] test the installation in a fresh virtual environment using
+    - [optional] Test the installation in a fresh virtual environment using
       `pip install cellxgene`
 
 The optional steps are for testing purposes, and are recommended

server/app/driver/driver.py

@@ -17,6 +17,7 @@ class CXGDriver(metaclass=ABCMeta):
         self.layout_method = args["layout"]
         self.diffexp_method = args["diffexp"]
         self.max_category_items = args["max_category_items"]
+        self.diffexp_lfc_cutoff = args["diffexp_lfc_cutoff"]
         self.cluster = None
 
     @property
@@ -83,16 +84,18 @@ class CXGDriver(metaclass=ABCMeta):
         pass
 
     @abstractmethod
-    def diffexp(self, filter1, filter2, top_n=None, interactive_limit=None):
+    def diffexp_topN(self, obsFilter1, obsFilter2, top_n=None, interactive_limit=None):
         """
-        Computes the top differentially expressed variables between two observation sets. If dataframes
+        Computes the top N differentially expressed variables between two observation sets. If mode
+        is "TOP_N", then stats for the top N
+        dataframes
         contain a subset of variables, then statistics for all variables will be returned, otherwise
         only the top N vars will be returned.
-        :param filter1: filter: dictionary with filter params for first set of observations
+        :param obsFilter1: filter: dictionary with filter params for first set of observations
-        :param filter2: filter: dictionary with filter params for second set of observations
+        :param obsFilter2: filter: dictionary with filter params for second set of observations
         :param top_n: Limit results to top N (Top var mode only)
         :param interactive_limit: -- don't compute if total # genes in dataframes are larger than this
-        :return: top genes, stats and expression values for variables
+        :return: top N genes and corresponding stats
         """
         pass
 

server/app/rest_api/rest.py

@@ -1,5 +1,6 @@
 from http import HTTPStatus
 import pkg_resources
+import warnings
 
 from flask import (
     Blueprint, current_app, jsonify, make_response, request
@@ -7,7 +8,7 @@ from flask import (
 from flask_restful_swagger_2 import Api, swagger, Resource
 from werkzeug.datastructures import ImmutableMultiDict
 
-from server.app.util.constants import Axis, DiffExpMode
+from server.app.util.constants import Axis, DiffExpMode, JSON_NaN_to_num_warning_msg
 from server.app.util.filter import parse_filter, QueryStringError
 from server.app.util.models import FilterModel
 from server.app.util.utils import get_mime_type
@@ -160,7 +161,12 @@ class AnnotationsObsAPI(Resource):
             annotation_response = current_app.data.annotation({}, "obs", fields)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Fetch annotations (metadata) for filtered subset of observations.",
@@ -211,7 +217,12 @@ class AnnotationsObsAPI(Resource):
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class AnnotationsVarAPI(Resource):
@@ -253,7 +264,12 @@ class AnnotationsVarAPI(Resource):
             annotation_response = current_app.data.annotation({}, "var", fields)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Fetch annotations (metadata) for filtered subset of variables.",
@@ -302,7 +318,12 @@ class AnnotationsVarAPI(Resource):
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
         except FilterError:
             return make_response("Malformed filter", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DataObsAPI(Resource):
@@ -346,7 +367,7 @@ class DataObsAPI(Resource):
     def get(self):
         accept_type = request.args.get("accept-type", None)
         # request.args is immutable
-        args = dict(request.args)
+        args = request.args.copy()
         args.pop("accept-type", None)
         try:
             filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema['annotations'])
@@ -363,6 +384,10 @@ class DataObsAPI(Resource):
             return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.OBS))), HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Get data (expression values) from the dataframe.",
@@ -407,6 +432,10 @@ class DataObsAPI(Resource):
                                  HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DataVarAPI(Resource):
@@ -450,7 +479,7 @@ class DataVarAPI(Resource):
     def get(self):
         accept_type = request.args.get("accept-type", None)
         # request.args is immutable
-        args = dict(request.args)
+        args = request.args.copy()
         args.pop("accept-type", None)
         try:
             filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema['annotations'])
@@ -465,6 +494,10 @@ class DataVarAPI(Resource):
             return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.VAR))), HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Get data (expression values) from the dataframe.",
@@ -510,6 +543,10 @@ class DataVarAPI(Resource):
                                  HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DiffExpObsAPI(Resource):
@@ -555,11 +592,11 @@ class DiffExpObsAPI(Resource):
         "responses": {
             "200": {
                 "description": "Statistics are encoded as an array of arrays, with fields ordered as: "
-                               "varIndex, avgDiff,  pVal, pValAdj, set1AvgExp, set2AvgExp",
+                               "varIndex, logfoldchange,  pVal, pValAdj",
                 "examples": {
                     "application/json": [
-                        [328, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9],
+                        [328, -2.569489, 2.655706e-63, 3.642036e-57],
-                        [1250, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9],
+                        [1250, -2.569489, 2.655706e-63, 3.642036e-57],
                     ]
                 }
             },
@@ -584,11 +621,12 @@ class DiffExpObsAPI(Resource):
         except ValueError:
             return make_response(f"Error: invalid mode option {args['mode']}", HTTPStatus.BAD_REQUEST)
         # Validate filters
-        if mode == DiffExpMode.VAR_FILTER:
+        if mode == DiffExpMode.VAR_FILTER or "varFilter" in args:
-            if "varFilter" not in args:
+            # not NOT_IMPLEMENTED
-                return make_response("varFilter is required when mode is set to varFilter ", HTTPStatus.BAD_REQUEST)
+            return make_response("mode=varfilter not implemented", HTTPStatus.NOT_IMPLEMENTED)
-            if Axis.OBS in args["varFilter"]["filter"]:
+        if mode == DiffExpMode.TOP_N and "count" not in args:
-                return make_response("Obs filter not allowed in varFilter", HTTPStatus.BAD_REQUEST)
+            return make_response("mode=topN requires a count parameter", HTTPStatus.BAD_REQUEST)
+
         if "set1" not in args:
             return make_response("set1 is required.", HTTPStatus.BAD_REQUEST)
         if Axis.VAR in args["set1"]["filter"]:
@@ -598,21 +636,27 @@ class DiffExpObsAPI(Resource):
             return make_response("Set2 as inverse of set1 is not implemented", HTTPStatus.NOT_IMPLEMENTED)
         if Axis.VAR in args["set2"]["filter"]:
             return make_response("Var filter not allowed for set2", HTTPStatus.BAD_REQUEST)
+
         set1_filter = args["set1"]["filter"]
         set2_filter = args.get("set2", {"filter": {}})["filter"]
-        if "varFilter" in args:
+
-            set1_filter[Axis.VAR] = args["varFilter"]["filter"][Axis.VAR]
+        # TODO: implement varfilter mode
-            set2_filter[Axis.VAR] = args["varFilter"]["filter"][Axis.VAR]
+
-        # mode
+        # mode=topN
         count = args.get("count", None)
         try:
-            diffexp = current_app.data.diffexp(set1_filter, set2_filter, count,
+            diffexp = current_app.data.diffexp_topN(set1_filter, set2_filter, count,
-                                               current_app.data.features["diffexp"]["interactiveLimit"])
+                                                    current_app.data.features["diffexp"]["interactiveLimit"])
         except (ValueError, FilterError) as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
         except InteractiveError:
             return make_response("Non-interactive request", HTTPStatus.FORBIDDEN)
-        return make_response(jsonify(diffexp), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(diffexp), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class LayoutObsAPI(Resource):
@@ -645,7 +689,12 @@ class LayoutObsAPI(Resource):
             layout = current_app.data.layout({})
         except PrepareError as e:
             return make_response(e.message, HTTPStatus.INTERNAL_SERVER_ERROR)
-        return make_response((jsonify({"layout": layout})), HTTPStatus.OK)
+        try:
+            return make_response((jsonify({"layout": layout})), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     # @swagger.doc({
     #     "summary": "Observation layout for filtered subset.",

server/app/scanpy_engine/diffexp.py

@@ -0,0 +1,113 @@
+
+import numpy as np
+from scipy import sparse, stats
+
+
+# Convenience function which handles sparse data
+def _mean_var_n(X):
+    """
+    Two-pass variance calculation.  Numerically (more) stable
+    than naive methods (and same method used by numpy.var())
+    https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass
+    """
+    n = X.shape[0]
+    if sparse.issparse(X):
+        mean = X.mean(axis=0).A1
+        dfm = X - mean
+        sumsq = np.sum(np.multiply(dfm, dfm), axis=0).A1
+        v = sumsq / (n - 1)
+    else:
+        mean = X.mean(axis=0)
+        dfm = X - mean
+        sumsq = np.sum(np.multiply(dfm, dfm), axis=0)
+        v = sumsq / (n - 1)
+
+    return mean, v, n
+
+
+def diffexp_ttest(adata, maskA, maskB, top_n=8, diffexp_lfc_cutoff=0.01):
+    """
+    Return differential expression statistics for top N variables.
+
+    Algorithm:
+    - compute log fold change (log2(meanA/meanB))
+    - compute Welch's t-test statistic and pvalue (w/ Bonferroni correction)
+    - return top N abs(logfoldchange) where lfc > diffexp_lfc_cutoff
+
+    If there are not N which meet criteria, augment by removing the logfoldchange
+    threshold requirement.
+
+    Notes on alogrithm:
+    - Welch's ttest provides basic statistics test.
+      https://en.wikipedia.org/wiki/Welch%27s_t-test
+    - p-values adjusted with Bonferroni correction.
+      https://en.wikipedia.org/wiki/Bonferroni_correction
+
+    :param adata: anndata dataframe
+    :param maskA: observation selection mask for set 1
+    :param maskB: observation selection mask for set 2
+    :param top_n: number of variables to return stats for
+    :param diffexp_lfc_cutoff: minimum
+    :return:  for top N genes, [ varindex, logfoldchange, pval, pval_adj ]
+    """
+
+    if top_n > adata.n_obs:
+        top_n = adata.n_obs
+
+    # mean, variance, N - calculate for both selections
+    meanA, vA, nA = _mean_var_n(adata._X[maskA])
+    meanB, vB, nB = _mean_var_n(adata._X[maskB])
+
+    # variance / N
+    vnA = vA / min(nA, nB)  # overestimate variance, would normally be nA
+    vnB = vB / min(nA, nB)  # overestimate variance, would normally be nB
+    sum_vn = vnA + vnB
+
+    # degrees of freedom for Welch's t-test
+    with np.errstate(divide='ignore', invalid='ignore'):
+        dof = sum_vn**2 / (vnA**2 / (nA - 1) + vnB**2 / (nB - 1))
+    dof[np.isnan(dof)] = 1
+
+    # Welch's t-test score calculation
+    with np.errstate(divide='ignore', invalid='ignore'):
+        tscores = (meanA - meanB) / np.sqrt(sum_vn)
+    tscores[np.isnan(tscores)] = 0
+
+    # p-value
+    pvals = stats.t.sf(np.abs(tscores), dof) * 2
+    pvals_adj = pvals * adata._X.shape[1]
+    pvals_adj[pvals_adj > 1] = 1        # cap adjusted p-value at 1
+
+    # logfoldchanges: log2(meanA / meanB)
+    logfoldchanges = np.log2(np.abs((meanA + 1e-9) / (meanB + 1e-9)))
+
+    # find all with lfc > cutoff
+    lfc_above_cutoff_idx = np.nonzero(np.abs(logfoldchanges) > diffexp_lfc_cutoff)[0]
+    stats_to_sort = np.abs(tscores)
+
+    # derive sort order
+    if lfc_above_cutoff_idx.shape[0] > top_n:
+        # partition top N
+        rel_t_partition = np.argpartition(stats_to_sort[lfc_above_cutoff_idx], -top_n)[-top_n:]
+        t_partition = lfc_above_cutoff_idx[rel_t_partition]
+        # sort the top N partition
+        rel_sort_order = np.argsort(stats_to_sort[t_partition])[::-1]
+        sort_order = t_partition[rel_sort_order]
+    else:
+        # partition and sort top N, ignoring lfc cutoff
+        partition = np.argpartition(stats_to_sort, -top_n)[-top_n:]
+        rel_sort_order = np.argsort(stats_to_sort[partition])[::-1]
+        indices = np.indices(stats_to_sort.shape)[0]
+        sort_order = indices[partition][rel_sort_order]
+
+    # top n slice based upon sort order
+    logfoldchanges_top_n = logfoldchanges[sort_order]
+    pvals_top_n = pvals[sort_order]
+    pvals_adj_top_n = pvals_adj[sort_order]
+
+    # varIndex, logfoldchange, pval, pval_adj
+    result = [[sort_order[i],
+               logfoldchanges_top_n[i],
+               pvals_top_n[i],
+               pvals_adj_top_n[i]] for i in range(top_n)]
+    return result

server/app/scanpy_engine/scanpy_engine.py

@@ -4,11 +4,12 @@ import numpy as np
 from pandas import DataFrame
 from pandas.core.dtypes.dtypes import CategoricalDtype
 import scanpy.api as sc
-from scipy import stats, sparse
+from scipy import sparse
 
 from server.app.driver.driver import CXGDriver
-from server.app.util.constants import Axis, DEFAULT_TOP_N, DiffExpMode
+from server.app.util.constants import Axis, DEFAULT_TOP_N
 from server.app.util.errors import FilterError, InteractiveError, PrepareError, ScanpyFileError
+from server.app.scanpy_engine.diffexp import diffexp_ttest
 
 """
 Sort order for methods
@@ -34,6 +35,10 @@ class ScanpyEngine(CXGDriver):
         self.diffexp_options = ["ttest"]
         self._create_schema()
 
+        # TODO: temporary work-arounds
+        if args['nan_to_num']:
+            self._IEEE754_special_values_workaround()
+
     def _alias_annotation_names(self, axis, name):
         """
         Do all user-specified annotation aliasing.
@@ -64,6 +69,24 @@ class ScanpyEngine(CXGDriver):
         else:
             raise KeyError(f"Annotation name {name}, specified in --{ax_name}_name does not exist.")
 
+    @staticmethod
+    def _can_cast_to_float32(ann):
+        if ann.dtype.kind == "f":
+            if not np.can_cast(ann.dtype, np.float32):
+                warnings.warn(f"Annotation {ann.name} will be converted to 32 bit float and may lose precision.")
+            return True
+        return False
+
+    @staticmethod
+    def _can_cast_to_int32(ann):
+        if ann.dtype.kind in ["i", "u"]:
+            if np.can_cast(ann.dtype, np.int32):
+                return True
+            ii32 = np.iinfo(np.int32)
+            if ann.min() >= ii32.min and ann.max() <= ii32.max:
+                return True
+        return False
+
     def _create_schema(self):
         self.schema = {
             "dataframe": {
@@ -80,16 +103,18 @@ class ScanpyEngine(CXGDriver):
             curr_axis = getattr(self.data, str(ax))
             for ann in curr_axis:
                 ann_schema = {"name": ann}
-                data_kind = curr_axis[ann].dtype.kind
+                dtype = curr_axis[ann].dtype
-                if data_kind == "f":
+                data_kind = dtype.kind
+
+                if self._can_cast_to_float32(curr_axis[ann]):
                     ann_schema["type"] = "float32"
-                elif data_kind in ["i", "u"]:
+                elif self._can_cast_to_int32(curr_axis[ann]):
                     ann_schema["type"] = "int32"
-                elif data_kind == "?":
+                elif dtype == np.bool_:
                     ann_schema["type"] = "boolean"
-                elif data_kind == "O" and curr_axis[ann].dtype == "object":
+                elif data_kind == "O" and dtype == "object":
                     ann_schema["type"] = "string"
-                elif data_kind == "O" and curr_axis[ann].dtype == "category":
+                elif data_kind == "O" and dtype == "category":
                     ann_schema["type"] = "categorical"
                     ann_schema["categories"] = curr_axis[ann].dtype.categories.tolist()
                 else:
@@ -114,35 +139,6 @@ class ScanpyEngine(CXGDriver):
                                   f"that your input and try again.")
         return result
 
-    @staticmethod
-    def _top_sort(values, sort_order, top_n=None):
-        """
-        Sorts an iterable in sort order limited by top_n
-        :param values: iterable of values to sort
-        :param sort_order: ndarray order to sort in
-        :param top_n: cutoff number to return
-        :return: values sorted by sort_order limited by top_n
-        """
-        return values[sort_order][:top_n]
-
-    @staticmethod
-    def _nan_to_one(values):
-        """
-        Replaces NaN values with 1
-        :param values: numpy ndarray
-        :return: ndarray
-        """
-        return np.where(np.isnan(values), 1, values)
-
-    @staticmethod
-    def _nan_to_zero(values):
-        """
-        Replaces NaN values with 0
-        :param values: numpy ndarray
-        :return: ndarray
-        """
-        return np.where(np.isnan(values), 0, values)
-
     def _validate_data_types(self):
         if self.data.X.dtype != "float32":
             warnings.warn(f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
@@ -180,7 +176,68 @@ class ScanpyEngine(CXGDriver):
                 f"`cellxgene prepare --layout {self.layout_method} <datafile>` "
                 f"to solve this problem. ")
 
-    def filter_dataframe(self, filter, include_uns=False):
+    def _IEEE754_special_values_workaround(self):
+        """
+        TODO: temporary workaround
+
+        Because all floating point data is serialized to JSON, and JSON has no means of representing
+        non-finite, floating point special values (NaN, +/-Infinity, etc), we include this temporary
+        work-around.
+
+        This will likely be removed in the future, contingent upon improved marshalling.
+
+        Where non-finite floating point is present in obs, var or X:
+        * issue a warning to the user that these values will be convert to finite numbers.
+        * set NaN to zero, and Infinities to min/max of the element.
+        """
+
+        # annotations
+        for ax in Axis:
+            curr_axis = getattr(self.data, str(ax))
+            for ann in curr_axis:
+                dtype = curr_axis[ann].dtype
+                if dtype.kind == 'f':
+                    finite_idx = np.isfinite(curr_axis[ann])
+                    if not finite_idx.all():
+                        curr_axis.loc[np.isnan(curr_axis[ann]), ann] = 0
+                        curr_axis.loc[np.isneginf(curr_axis[ann]), ann] = curr_axis[ann][finite_idx].min()
+                        curr_axis.loc[np.isposinf(curr_axis[ann]), ann] = curr_axis[ann][finite_idx].max()
+                        warnings.warn(
+                            f"{str(ax).title()} annotation '{ann}' contains floating point NaN or Infinities. "
+                            f"These will be converted to finite values."
+                        )
+
+        # X
+        non_finite_X_found = False
+        if sparse.issparse(self.data._X):
+            coo = self.data._X.tocoo()
+            finite_idx = np.isfinite(coo.data)
+            if not finite_idx.all():
+                non_finite_X_found = True
+                coo.data[np.isnan(coo.data)] = 0
+                coo.data[np.isneginf(coo.data)] = np.min(coo.data[finite_idx])
+                coo.data[np.isposinf(coo.data)] = np.max(coo.data[finite_idx])
+                coo.eliminate_zeros()
+                _X = coo.asformat(self.data._X.getformat())
+                self.data._X = _X
+        else:
+            _X = self.data._X
+            finite_idx = np.isfinite(_X.flat)
+            if not finite_idx.all():
+                non_finite_X_found = True
+                min_X = _X.flat[finite_idx].min()
+                max_X = _X.flat[finite_idx].max()
+                _X[np.isnan(_X)] = 0
+                _X[np.isneginf(_X)] = min_X
+                _X[np.isposinf(_X)] = max_X
+
+        if non_finite_X_found:
+            warnings.warn(
+                "Dataframe X contains floating point NaN or Infinities. "
+                "These will be converted to finite values."
+            )
+
+    def filter_dataframe(self, filter):
         """
          Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
          indexing and filtering by annotation value. Filters are combined with the and operator.
@@ -189,70 +246,68 @@ class ScanpyEngine(CXGDriver):
          https://docs.google.com/document/d/1Fxjp1SKtCk7l8QP9-7KAjGXL0eldi_qEnNT0NmlGzXI/edit#heading=h.8qc9q57amldx
 
         :param filter: dictionary with filter params
-        :param include_uns: bool, include unstructured annotations
         :return: View into scanpy object with cells/genes filtered
         """
         if not filter:
             return self.data
-        cells_idx = np.ones((self.cell_count,), dtype=bool)
+        obs_selector, var_selector = self._filter_to_mask(filter, use_slices=False)
-        genes_idx = np.ones((self.gene_count,), dtype=bool)
+        data = self._slice(self.data, obs_selector, var_selector)
-        if Axis.OBS in filter:
-            if "index" in filter["obs"]:
-                cells_idx = self._filter_index(filter["obs"]["index"], cells_idx, Axis.OBS)
-            if "annotation_value" in filter["obs"]:
-                cells_idx = self._filter_annotation(filter["obs"]["annotation_value"], cells_idx, Axis.OBS)
-        if Axis.VAR in filter:
-            if "index" in filter["var"]:
-                genes_idx = self._filter_index(filter["var"]["index"], genes_idx, Axis.VAR)
-            if "annotation_value" in filter["var"]:
-                genes_idx = self._filter_annotation(filter["var"]["annotation_value"], genes_idx, Axis.VAR)
-
-        data = self._slice(self.data, cells_idx, genes_idx)
         return data
 
-    def _filter_index(self, filter, index, axis):
+    @staticmethod
-        """
+    def _annotation_filter_to_mask(filter, d_axis, count):
-        Filter data based on index. ex. [1, 3, [111:200]]
+        mask = np.ones((count, ), dtype=bool)
-        :param filter: subset of filter dict for obs/var:index
-        :param index: np logical vector containing true for passing false for failing filter
-        :param axis: string obs or var
-        :return: np logical vector for whether the data passes the filter
-        """
-        if axis == Axis.OBS:
-            count_ = self.cell_count
-        elif axis == Axis.VAR:
-            count_ = self.gene_count
-        idx_filter = np.zeros((count_,), dtype=bool)
-        for i in filter:
-            if type(i) == list:
-                idx_filter[i[0]:i[1]] = True
-            else:
-                idx_filter[i] = True
-        return np.logical_and(index, idx_filter)
-
-    def _filter_annotation(self, filter, index, axis):
-        """
-        Filter data based on annotation value
-        :param filter: subset of filter dict for obs/var:annotation_value
-        :param index: np logical vector containing true for passing false for failing filter
-        :param axis: string obs or var
-        :return: np logical vector for whether the data passes the filter
-        """
-        d_axis = getattr(self.data, axis.value)
         for v in filter:
             if d_axis[v["name"]].dtype.name in ["boolean", "category", "object"]:
                 key_idx = np.in1d(getattr(d_axis, v["name"]), v["values"])
-                index = np.logical_and(index, key_idx)
+                mask = np.logical_and(mask, key_idx)
             else:
                 min_ = v.get("min", None)
                 max_ = v.get("max", None)
                 if min_ is not None:
                     key_idx = (getattr(d_axis, v["name"]) >= min_).ravel()
-                    index = np.logical_and(index, key_idx)
+                    mask = np.logical_and(mask, key_idx)
                 if max_ is not None:
                     key_idx = (getattr(d_axis, v["name"]) <= max_).ravel()
-                    index = np.logical_and(index, key_idx)
+                    mask = np.logical_and(mask, key_idx)
-        return index
+        return mask
+
+    @staticmethod
+    def _index_filter_to_mask(filter, count):
+        mask = np.zeros((count, ), dtype=bool)
+        for i in filter:
+            if type(i) == list:
+                mask[i[0]:i[1]] = True
+            else:
+                mask[i] = True
+        return mask
+
+    @staticmethod
+    def _axis_filter_to_mask(filter, d_axis, count):
+        mask = np.ones((count, ), dtype=bool)
+        if "index" in filter:
+            mask = np.logical_and(mask, ScanpyEngine._index_filter_to_mask(filter["index"], count))
+        if "annotation_value" in filter:
+            mask = np.logical_and(mask,
+                                  ScanpyEngine._annotation_filter_to_mask(filter["annotation_value"],
+                                                                          d_axis,
+                                                                          count))
+        return mask
+
+    def _filter_to_mask(self, filter, use_slices=True):
+        if use_slices:
+            obs_selector = slice(0, self.data.n_obs)
+            var_selector = slice(0, self.data.n_vars)
+        else:
+            obs_selector = None
+            var_selector = None
+
+        if filter is not None:
+            if Axis.OBS in filter:
+                obs_selector = self._axis_filter_to_mask(filter["obs"], self.data.obs, self.data.n_obs)
+            if Axis.VAR in filter:
+                var_selector = self._axis_filter_to_mask(filter["var"], self.data.var, self.data.n_vars)
+        return obs_selector, var_selector
 
     @staticmethod
     def _slice(data, obs_selector=None, vars_selector=None):
@@ -293,16 +348,25 @@ class ScanpyEngine(CXGDriver):
         [observation ids, val1, val2...]
         """
         try:
-            df = self.filter_dataframe(filter)
+            obs_selector, var_selector = self._filter_to_mask(filter)
-        except KeyError as e:
+        except (KeyError, IndexError) as e:
             raise FilterError(f"Error parsing filter: {e}") from e
-        df_axis = getattr(df, axis)
+        if axis == Axis.OBS:
-        if not fields:
+            obs = self.data.obs[obs_selector]
-            fields = df_axis.columns.tolist()
+            if not fields:
-        result = {
+                fields = obs.columns.tolist()
-            "names": fields,
+            result = {
-            "data": DataFrame(df_axis[fields]).to_records(index=True).tolist()
+                "names": fields,
-        }
+                "data": DataFrame(obs[fields]).to_records(index=True).tolist()
+            }
+        else:
+            var = self.data.var[var_selector]
+            if not fields:
+                fields = var.columns.tolist()
+            result = {
+                "names": fields,
+                "data": DataFrame(var[fields]).to_records(index=True).tolist()
+            }
         return result
 
     def data_frame(self, filter, axis):
@@ -316,85 +380,38 @@ class ScanpyEngine(CXGDriver):
         }
         """
         try:
-            slice = self.filter_dataframe(filter)
+            obs_selector, var_selector = self._filter_to_mask(filter)
-        except KeyError as e:
+        except (KeyError, IndexError) as e:
             raise FilterError(f"Error parsing filter: {e}") from e
-        # convert sparse slice to dense
+        _X = self.data._X[obs_selector, var_selector]
-        X = slice._X.toarray() if sparse.issparse(slice._X) else slice._X
+        if sparse.issparse(_X):
+            _X = _X.toarray()
+        var_index_sliced = self.data.var.index[var_selector]
+        obs_index_sliced = self.data.obs.index[obs_selector]
         if axis == Axis.OBS:
             result = {
-                "var": slice.var.index.tolist(),
+                "var": var_index_sliced.tolist(),
-                "obs": DataFrame(X, index=slice.obs.index).to_records(index=True).tolist()
+                "obs": DataFrame(_X, index=obs_index_sliced).to_records(index=True).tolist()
             }
         else:
             result = {
-                "obs": slice.obs.index.tolist(),
+                "obs": obs_index_sliced.tolist(),
-                "var": DataFrame(X.T, index=slice.var.index).to_records(index=True).tolist()
+                "var": DataFrame(_X.T, index=var_index_sliced).to_records(index=True).tolist()
             }
         return result
 
-    def diffexp(self, filter1, filter2, top_n=None, interactive_limit=None):
+    def diffexp_topN(self, obsFilterA, obsFilterB, top_n=None, interactive_limit=None):
-        """
+        if Axis.VAR in obsFilterA or Axis.VAR in obsFilterB:
-        Computes the top differentially expressed variables between two observation sets. If dataframes
+            raise FilterError("Observation filters may not contain vaiable conditions")
-        contain a subset of variables, then statistics for all variables will be returned, otherwise
-        only the top N vars will be returned.
-        :param filter1: filter: dictionary with filter params for first set of observations
-        :param filter2: filter: dictionary with filter params for second set of observations
-        :param top_n: Limit results to top N (Top var mode only)
-        :param interactive_limit: -- don't compute if total # genes in dataframes are larger than this
-        :return: top genes, stats and expression values for variables
-        """
         try:
-            df1 = self.filter_dataframe(filter1)
+            obs_mask_A = self._axis_filter_to_mask(obsFilterA["obs"], self.data.obs, self.data.n_obs)
-        except KeyError as e:
+            obs_mask_B = self._axis_filter_to_mask(obsFilterB["obs"], self.data.obs, self.data.n_obs)
-            raise FilterError(f"Error parsing filter for set 1: {e}") from e
+        except (KeyError, IndexError) as e:
-        # TODO df2 should be inverse if not filter2 provided
+            raise FilterError(f"Error parsing filter: {e}") from e
-        try:
+        if top_n is None:
-            df2 = self.filter_dataframe(filter2)
+            top_n = DEFAULT_TOP_N
-        except KeyError as e:
+        result = diffexp_ttest(self.data, obs_mask_A, obs_mask_B, top_n, self.diffexp_lfc_cutoff)
-            raise FilterError(f"Error parsing filter for set 2: {e}") from e
+        return result
-        # If not the same genes, test is wrong!
-        if np.any(df1.var.index != df2.var.index):
-            raise ValueError("Variables ares not the same in set1 and set2")
-        if interactive_limit and df1.shape[0] + df2.shape[0] > interactive_limit:
-            raise InteractiveError("Size of set 1 and 2 is too large for interactive computation")
-        # If not all genes, they used a var filter
-        if df1.var.shape[0] < self.gene_count:
-            mode = DiffExpMode.VAR_FILTER
-            if top_n:
-                raise Warning("Top N was specified but will not be used in 'Var Filter' mode")
-        else:
-            mode = DiffExpMode.TOP_N
-            if not top_n:
-                top_n = DEFAULT_TOP_N
-
-        genes_idx = df1.var.index
-        # ensure we are using a dense ndarray
-        X1 = df1._X.toarray() if sparse.issparse(df1._X) else df1._X
-        X2 = df2._X.toarray() if sparse.issparse(df2._X) else df2._X
-        diffexp_result = stats.ttest_ind(X1, X2)
-        tstats = self._nan_to_zero(diffexp_result.statistic)
-        pval = self._nan_to_one(diffexp_result.pvalue)
-        bonferroni_pval = 1 - (1 - pval) ** self.gene_count
-        ave_exp_set1 = np.mean(X1, axis=0)
-        ave_exp_set2 = np.mean(X2, axis=0)
-        ave_diff = ave_exp_set1 - ave_exp_set2
-        if mode == DiffExpMode.TOP_N:
-            sort_order = np.argsort(np.abs(tstats))[::-1]
-            # If top_n > length it will just return length
-            genes = self._top_sort(genes_idx, sort_order, top_n)
-            pval = self._top_sort(pval, sort_order, top_n)
-            bonferroni_pval = self._top_sort(bonferroni_pval, sort_order, top_n)
-            ave_exp_set1 = self._top_sort(ave_exp_set1, sort_order, top_n)
-            ave_exp_set2 = self._top_sort(ave_exp_set2, sort_order, top_n)
-            ave_diff = self._top_sort(ave_diff, sort_order, top_n)
-
-        # varIndex, avgDiff,  pVal, pValAdj, set1AvgExp, set2AvgExp
-        result = []
-        for i in range(len(genes)):
-            result.append([genes[i], ave_diff[i], pval[i], bonferroni_pval[i], ave_exp_set1[i], ave_exp_set2[i]])
-        # Results need to be returned in var index order
-        return sorted(result, key=lambda gene: gene[0])
 
     def layout(self, filter, interactive_limit=None):
         """
@@ -404,8 +421,8 @@ class ScanpyEngine(CXGDriver):
         :return:  [cellid, x, y, ...]
         """
         try:
-            df = self.filter_dataframe(filter, include_uns=True)
+            df = self.filter_dataframe(filter)
-        except KeyError as e:
+        except (KeyError, IndexError) as e:
             raise FilterError(f"Error parsing filter: {e}") from e
         if interactive_limit and len(df.obs.index) > interactive_limit:
             raise InteractiveError("Size data is too large for interactive computation")

server/app/util/constants.py

@@ -25,3 +25,6 @@ class Axis(AugmentedEnum):
 class DiffExpMode(AugmentedEnum):
     TOP_N = "topN"
     VAR_FILTER = "varFilter"
+
+
+JSON_NaN_to_num_warning_msg = "JSON encoding failure - suggest trying --nan-to-num command line option"

server/app/util/utils.py

@@ -7,6 +7,17 @@ from server.app.util.errors import MimeTypeError
 
 
 class Float32JSONEncoder(json.JSONEncoder):
+    def __init__(self, *args, **kwargs):
+        """
+        NaN/Infinities are illegal in standard JSON.  Python extends JSON with
+        non-standard symbols that most JavaScript JSON parsers do not understand.
+        The `allow_nan` parameter will force Python simplejson to throw an ValueError
+        if it runs into non-finite floating point values which are unsupported by
+        standard JSON.
+        """
+        kwargs['allow_nan'] = False
+        super().__init__(*args, **kwargs)
+
     def default(self, obj):
         if isinstance(obj, float32):
             return float(obj)
@@ -15,6 +26,10 @@ class Float32JSONEncoder(json.JSONEncoder):
         return json.JSONEncoder.default(self, obj)
 
 
+def custom_format_warning(msg, *args, **kwargs):
+    return f"[cellxgene] Warning: {msg} \n"
+
+
 def get_mime_type(default="application/json", acceptable_types=["application/json", "text/csv"], query_param=None,
                   header=None):
     mime_type = default

server/cli/cli.py

@@ -5,7 +5,7 @@ from .prepare import prepare
 
 
 @click.group(name="cellxgene", context_settings=dict(max_content_width=85))
-@click.version_option(version="0.0.1", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
+@click.version_option(version="0.3.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
 def cli():
     pass
 

server/cli/launch.py

@@ -1,10 +1,14 @@
-import sys
-import click
 import logging
+from os import devnull
 from os.path import splitext, basename
+import sys
+import warnings
 import webbrowser
 
+import click
+
 from server.app.util.errors import ScanpyFileError
+from server.app.util.utils import custom_format_warning
 
 
 @click.command()
@@ -23,12 +27,16 @@ from server.app.util.errors import ScanpyFileError
 @click.option("--port", "-p", help="Port to run server on.", metavar="", default=5005, show_default=True)
 @click.option("--obs-names", default=None, metavar="", help="Name of annotation field to use for observations.")
 @click.option("--var-names", default=None, metavar="", help="Name of annotation to use for variables.")
-@click.option("--listen-all", is_flag=True, default=False, show_default=True,
+@click.option("--host", default="127.0.0.1", help="Host IP address")
-              help="Bind to all interfaces (this makes the server accessible beyond this computer).")
 @click.option("--max-category-items", default=100, metavar="", show_default=True,
               help="Limits the number of categorical annotation items displayed.")
+@click.option("--diffexp-lfc-cutoff", default=0.01, show_default=True,
+              help="Relative expression cutoff used when selecting top N differentially expressed genes")
+@click.option("--nan-to-num", is_flag=True, default=False, show_default=True,
+              help="Replace all floating point NaN with zero, and infinities with finite numbers")
 def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
-           open_browser, port, listen_all, max_category_items):
+           open_browser, port, host, max_category_items, diffexp_lfc_cutoff,
+           nan_to_num):
     """Launch the cellxgene data viewer.
     This web app lets you explore single-cell expression data.
     Data must be in a format that cellxgene expects, read the
@@ -43,9 +51,6 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
     # Startup message
     click.echo("[cellxgene] Starting the CLI...")
 
-    # Import Flask app
-    from server.app.app import app
-
     # Argument checking
     name, extension = splitext(data)
     if extension != ".h5ad":
@@ -54,6 +59,8 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
     if debug:
         verbose = True
         open_browser = False
+    else:
+        warnings.formatwarning = custom_format_warning
 
     if not verbose:
         sys.tracebacklimit = 0
@@ -62,15 +69,13 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
         file_parts = splitext(basename(data))
         title = file_parts[0]
 
-    if listen_all:
-        host = "0.0.0.0"
-    else:
-        host = "127.0.0.1"
-
     # Setup app
     cellxgene_url = f"http://{host}:{port}"
     api_base = f"{cellxgene_url}/api/"
 
+    # Import Flask app
+    from server.app.app import app
+
     app.config.update(
         DATASET_TITLE=title,
         CXG_API_BASE=api_base
@@ -92,8 +97,10 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
         "layout": layout,
         "diffexp": diffexp,
         "max_category_items": max_category_items,
+        "diffexp_lfc_cutoff": diffexp_lfc_cutoff,
         "obs_names": obs_names,
-        "var_names": var_names
+        "var_names": var_names,
+        "nan_to_num": nan_to_num
     }
 
     try:
@@ -109,4 +116,8 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
 
     click.echo("[cellxgene] Type CTRL-C at any time to exit.")
 
+    if not verbose:
+        f = open(devnull, 'w')
+        sys.stdout = f
+
     app.run(host=host, debug=debug, port=port, threaded=True)

server/requirements.txt

@@ -1,4 +1,4 @@
-anndata>=0.6.12
+anndata>=0.6.13
 click>=6.7
 Flask>=1.0.2
 Flask-Caching>=1.4.0
@@ -11,4 +11,4 @@ numpy>=1.14.5
 pandas>=0.23.1
 scanpy>=1.3.2
 scipy>=1.1.0
-scikit-learn==0.19.1
+scikit-learn>=0.20.1

server/test/test_api.py

@@ -198,6 +198,7 @@ class EndPoints(unittest.TestCase):
         url = f"{URL_BASE}{endpoint}"
         params = {
             "mode": "topN",
+            "count": 10,
             "set1": {
                 "filter": {
                     "obs": {

server/test/test_scanpy_engine.py

@@ -14,7 +14,8 @@ from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
 class UtilTest(unittest.TestCase):
     def setUp(self):
         args = {'layout': 'umap', 'diffexp': 'ttest', 'max_category_items': 100,
-                'obs_names': None, 'var_names': None}
+                'obs_names': None, 'var_names': None, 'diffexp_lfc_cutoff': 0.01,
+                'nan_to_num': True}
 
         self.data = ScanpyEngine("example-dataset/pbmc3k.h5ad", args)
         self.data._create_schema()
@@ -85,7 +86,7 @@ class UtilTest(unittest.TestCase):
                 }
             }
         }
-        data = self.data.filter_dataframe(filter_["filter"], include_uns=False)
+        data = self.data.filter_dataframe(filter_["filter"])
         self.assertEqual(data.shape[1], 1)
 
     def test_filter_complex(self):
@@ -184,7 +185,7 @@ class UtilTest(unittest.TestCase):
         layout = self.data.layout(filter_["filter"])
         self.assertEqual(len(layout["coordinates"]), 497)
 
-    def test_diffexp(self):
+    def test_diffexp_topN(self):
         f1 = {
             "filter": {
                 "obs": {
@@ -199,11 +200,9 @@ class UtilTest(unittest.TestCase):
                 }
             }
         }
-        result = self.data.diffexp(f1["filter"], f2["filter"])
+        result = self.data.diffexp_topN(f1["filter"], f2["filter"])
         self.assertEqual(len(result), 10)
-        var_idx = [i[0] for i in result]
+        result = self.data.diffexp_topN(f1["filter"], f2["filter"], 20)
-        self.assertEqual(var_idx, sorted(var_idx))
-        result = self.data.diffexp(f1["filter"], f2["filter"], 20)
         self.assertEqual(len(result), 20)
 
     def test_data_frame(self):

setup.py

@@ -1,14 +1,14 @@
 from setuptools import setup, find_packages
 
-with open("README.md", "r") as fh:
+with open("README.md", "rb") as fh:
-    long_description = fh.read()
+    long_description = fh.read().decode()
 
 with open("server/requirements.txt") as fh:
     requirements = fh.read().splitlines()
 
 setup(
     name="cellxgene",
-    version="0.0.1",
+    version="0.3.0",
     packages=find_packages(),
     url="https://github.com/chanzuckerberg/cellxgene",
     license="MIT",
@@ -16,6 +16,7 @@ setup(
     author_email="cweaver@chanzuckerberg.com",
     description="Web application for exploration of large scale scRNA-seq datasets",
     long_description=long_description,
+    long_description_content_type='text/markdown',
     install_requires=requirements,
     include_package_data=True,
     zip_safe=False,