bump version to 0.3 (#498)

* improve HTTP error reporting

* generate standards-compatible JSON

* add --nan-to-num work-around for non-finite floating point values

* lint

* update tests

* correctly set Infinities to min/max

* REAMDE update for --nan-to-num

* define constant for repetitive warning message

* clarify where NaN errors will occure

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.2.3
+current_version = 0.3.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"

README.md

@@ -27,6 +27,7 @@ If you want an example dataset download [this file](https://github.com/chanzucke
 ```
 cellxgene launch pbmc3k.h5ad --open
 ```
+
 You should see your web browser open with the following
 
 <img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
@@ -35,7 +36,7 @@ You should see your web browser open with the following
 
 There are several options available, such as:
 
-- `--layout` to specify the layout as  `tsne` or `umap`
+- `--layout` to specify the layout as `tsne` or `umap`
 - `--title` to show a title on the explorer
 - `--open` to automatically open the web browser after launching (OS X only)
 
@@ -56,11 +57,11 @@ The `launch` command assumes that the data is stored in the `.h5ad` format from
 - an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
 - a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
 - an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_tsne` for the `tsne` layout or `X_umap` for the `umap` layout)
-- any additional  `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments) 
+- any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
 
 ### prepare
 
-The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`. 
+The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
 
 To prepare from an existing `.h5ad` file use
 
@@ -68,7 +69,7 @@ To prepare from an existing `.h5ad` file use
 cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
 ```
 
-This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. 
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://github.com/theislab/scanpy/blob/master/scanpy/preprocessing/recipes.py).
 
 Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
 
@@ -127,7 +128,7 @@ And your web browser should open with an interactive view of your data.
 
 > In my `prepare` command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
 
-Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using 
+Louvain clustering requires additional dependencies that are somewhat complex, so we don't include them by default. For now, you need to specify that you want these packages by using
 
 ```
 pip install cellxgene[louvain]
@@ -153,12 +154,24 @@ This may happen, especially as we work out bugs in our installation process! Ple
 
 > How are you computing and sorting differential expression results?
 
-Currently we use a [Welch's *t*-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
+Currently we use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`. We can explore adding support for other test types in the future.
 
 > I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
 
 This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
 
+> What part of the anndata objects does cellxgene pull in for visualization?
+
+- `.obs` and `.var` annotations are use to extract metadata for filtering
+- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
+- `.obsm` is used for layout
+
+> When I start cellxgene, I get an error `Unexpected HTTP response 500, INTERNAL SERVER ERROR -- Out of range float values are not JSON compliant` in the web UI, or `Warning: JSON encoding failure - suggest trying --nan-to-num command line option` in the CLI. What can I do?
+
+At the moment, cellxgene is unable to transmit floating point NaN or Inifinty values to the web UI (due to a limitation on data serialization method in use). We expect to resolve this in a future release, but in the meantime, you can work around this issue by starting cellxgene with the `--nan-to-num` command line option, ie, `cellxgene launch data.h5ad --nan-to-num`.
+
+This option will convert all NaNs to zero, and all positive/negative infinities to the min/max of the data element within which the value was found (eg, +Infinity within an `obs` annotation will be converted to the maximum finite value in that annotation). This option will increase startup time, so we recommend only using it when the dataset contains NaN/Infinities.
+
 ## developer guide
 
 This project has made a few key design choices
@@ -198,11 +211,11 @@ If you have any questions about developing or contributing, come hang out with u
 
 ## development roadmap
 
-`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact) 
+`cellxgene` is still very much in development, and we've love to include the community as we plan new features to work on. We are thinking about working on the following features over the next 3-12 months. If you are interested in updates, want to give feedback, want to contribute, or have ideas about other features we should work on, please [contact us](#help-and-contact)
 
 - **Visualizaling spatial metadata** Image-based transcriptomics methods also generate large cell by gene matrices, alongside rich metadata about spatial location; we would like to render this information in `cellxgene`
-- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene ` to render the results of these analyses when they have been performed
-- **Deploy to web**  Many projects release public data browser websites  alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
+- **Visualizing trajectories** Trajectory analyses infer progression along some ordering or pseudotime; we would like `cellxgene` to render the results of these analyses when they have been performed
+- **Deploy to web** Many projects release public data browser websites alongside their publicatons; we would like to make it easy for anyone to deploy `cellxgene` to a custom URL with their own dataset that they own and operate
 - **HCA Integration** The [Human Cell Atlas](https://humancellatlas.org) is generating a large corpus of single-cell expression data and will make it available through the Data Coordination Platform; we would like `cellxgene` to be one of several different portals for browsing these data
 
 ## contributing
@@ -215,7 +228,7 @@ We've been heavily inspired by several other related single-cell visualization p
 
 We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.
 
-We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset. 
+We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
 
 We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
 

client/__tests__/util/stateManager/summarizeAnnotations.test.js

@@ -0,0 +1,192 @@
+import summarizeAnnotations from "../../../src/util/stateManager/summarizeAnnotations";
+
+describe("summarizeAnnotations", () => {
+  const schema = {
+    annotations: {
+      obs: [
+        { name: "name", type: "string" },
+        { name: "nameString", type: "string" },
+        { name: "nameBoolean", type: "boolean" },
+        { name: "nameFloat32", type: "float32" },
+        { name: "nameInt32", type: "int32" },
+        {
+          name: "nameCategorical",
+          type: "categorical",
+          categories: [true, false, 1, 0, 0.00001, 4383.4833, "test", "", "0"]
+        }
+      ],
+      var: [{ name: "name", type: "string" }]
+    }
+  };
+
+  test("empty test", () => {
+    const summary = summarizeAnnotations(schema, [], []);
+    expect(summary).toEqual(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          },
+          nameFloat32: {
+            categorical: false,
+            range: {
+              max: Number.NEGATIVE_INFINITY,
+              min: Number.POSITIVE_INFINITY
+            }
+          },
+          nameInt32: {
+            categorical: false,
+            range: {
+              max: Number.NEGATIVE_INFINITY,
+              min: Number.POSITIVE_INFINITY
+            }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: [],
+            categoryCounts: new Map(),
+            numCategories: 0
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("simple test", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toEqual(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: ["hi"],
+            categoryCounts: new Map([["hi", 1]]),
+            numCategories: 1
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: [true],
+            categoryCounts: new Map([[true, 1]]),
+            numCategories: 1
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 39.3, max: 39.3 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: [1],
+            categoryCounts: new Map([[1, 1]]),
+            numCategories: 1
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("multi test", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n0",
+        nameString: "hi",
+        nameBoolean: false,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      },
+      {
+        __index__: 1,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: false
+      },
+      {
+        __index__: 2,
+        name: "n2",
+        nameString: "bye",
+        nameBoolean: true,
+        nameFloat32: 0,
+        nameInt32: 99,
+        nameCategorical: "0"
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toMatchObject(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: expect.arrayContaining(["hi", "bye"]),
+            categoryCounts: new Map([["hi", 2], ["bye", 1]]),
+            numCategories: 2
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: expect.arrayContaining([true, false]),
+            categoryCounts: new Map([[true, 2], [false, 1]]),
+            numCategories: 2
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 0, max: 39.3 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: expect.arrayContaining([1, false, "0"]),
+            categoryCounts: new Map([[1, 1], [false, 1], ["0", 1]]),
+            numCategories: 3
+          }
+        },
+        var: {}
+      })
+    );
+  });
+});

client/__tests__/util/stateManager/worldUtil.test.js

@@ -0,0 +1,45 @@
+import {
+  countCategoryValues2D,
+  clearCaches
+} from "../../../src/util/stateManager/worldUtil";
+
+describe("WorldUtil cache management", () => {
+  test("empty", () => {
+    const count = countCategoryValues2D("a", "b", []);
+    expect(count).toMatchObject(new Map());
+  });
+
+  test("simple couts", () => {
+    const rows = [{ a: 0, b: false }, { a: 0, b: true }, { a: 1, b: false }];
+    const count = countCategoryValues2D("a", "b", rows);
+    expect(count).toMatchObject(
+      new Map([
+        [0, new Map([[true, 1], [false, 1]])],
+        [1, new Map([[false, 1]])]
+      ])
+    );
+  });
+
+  test("memo cache clear", () => {
+    clearCaches();
+    const row1 = [];
+    const row2 = [{ a: 0, b: false }, { a: 0, b: true }, { a: 1, b: false }];
+    const count1 = countCategoryValues2D("a", "b", row1);
+    const count2 = countCategoryValues2D("a", "b", row1);
+    const count3 = countCategoryValues2D("a", "b", []);
+    const count4 = countCategoryValues2D("a", "b", row2);
+
+    clearCaches();
+    const count10 = countCategoryValues2D("a", "b", row1);
+    const count11 = countCategoryValues2D("a", "b", row2);
+
+    expect(count1).toEqual(count2);
+    expect(count1).toEqual(count3);
+    expect(count1).toEqual(count10);
+    expect(count1).not.toBe(count3);
+    expect(count1).not.toBe(count10);
+
+    expect(count4).toEqual(count11);
+    expect(count4).not.toBe(count11);
+  });
+});

client/package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "cellxgene",
-  "version": "0.2.3",
+  "version": "0.3.0",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {
@@ -2194,7 +2194,7 @@
     },
     "babel-plugin-syntax-object-rest-spread": {
       "version": "6.13.0",
-      "resolved": "http://registry.npmjs.org/babel-plugin-syntax-object-rest-spread/-/babel-plugin-syntax-object-rest-spread-6.13.0.tgz",
+      "resolved": "https://registry.npmjs.org/babel-plugin-syntax-object-rest-spread/-/babel-plugin-syntax-object-rest-spread-6.13.0.tgz",
       "integrity": "sha1-/WU28rzhODb/o6VFjEkDpZe7O/U=",
       "dev": true
     },
@@ -2627,7 +2627,7 @@
     },
     "browserify-aes": {
       "version": "1.2.0",
-      "resolved": "http://registry.npmjs.org/browserify-aes/-/browserify-aes-1.2.0.tgz",
+      "resolved": "https://registry.npmjs.org/browserify-aes/-/browserify-aes-1.2.0.tgz",
       "integrity": "sha512-+7CHXqGuspUn/Sl5aO7Ea0xWGAtETPXNSAjHo48JfLdPWcMng33Xe4znFvQweqc/uzk5zSOI3H52CYnjCfb5hA==",
       "dev": true,
       "requires": {
@@ -2664,7 +2664,7 @@
     },
     "browserify-rsa": {
       "version": "4.0.1",
-      "resolved": "http://registry.npmjs.org/browserify-rsa/-/browserify-rsa-4.0.1.tgz",
+      "resolved": "https://registry.npmjs.org/browserify-rsa/-/browserify-rsa-4.0.1.tgz",
       "integrity": "sha1-IeCr+vbyApzy+vsTNWenAdQTVSQ=",
       "dev": true,
       "requires": {
@@ -2718,7 +2718,7 @@
     },
     "buffer": {
       "version": "4.9.1",
-      "resolved": "http://registry.npmjs.org/buffer/-/buffer-4.9.1.tgz",
+      "resolved": "https://registry.npmjs.org/buffer/-/buffer-4.9.1.tgz",
       "integrity": "sha1-bRu2AbB6TvztlwlBMgkwJ8lbwpg=",
       "dev": true,
       "requires": {
@@ -3394,7 +3394,7 @@
     },
     "create-hash": {
       "version": "1.2.0",
-      "resolved": "http://registry.npmjs.org/create-hash/-/create-hash-1.2.0.tgz",
+      "resolved": "https://registry.npmjs.org/create-hash/-/create-hash-1.2.0.tgz",
       "integrity": "sha512-z00bCGNHDG8mHAkP7CtT1qVu+bFQUPjYq/4Iv3C3kWjTFV10zIjfSoeqXo9Asws8gwSHDGj/hl2u4OGIjapeCg==",
       "dev": true,
       "requires": {
@@ -3407,7 +3407,7 @@
     },
     "create-hmac": {
       "version": "1.1.7",
-      "resolved": "http://registry.npmjs.org/create-hmac/-/create-hmac-1.1.7.tgz",
+      "resolved": "https://registry.npmjs.org/create-hmac/-/create-hmac-1.1.7.tgz",
       "integrity": "sha512-MJG9liiZ+ogc4TzUwuvbER1JRdgvUFSB5+VR/g5h82fGaIRWMWddtKBHi7/sVhfjQZ6SehlyhvQYrcYkaUIpLg==",
       "dev": true,
       "requires": {
@@ -4090,7 +4090,7 @@
     },
     "diffie-hellman": {
       "version": "5.0.3",
-      "resolved": "http://registry.npmjs.org/diffie-hellman/-/diffie-hellman-5.0.3.tgz",
+      "resolved": "https://registry.npmjs.org/diffie-hellman/-/diffie-hellman-5.0.3.tgz",
       "integrity": "sha512-kqag/Nl+f3GwyK25fhUMYj81BUOrZ9IuJsjIcDE5icNM9FJHAVm3VcUDxdLPoQtTuUylWm6ZIknYJwwaPxsUzg==",
       "dev": true,
       "requires": {
@@ -4705,7 +4705,7 @@
         },
         "load-json-file": {
           "version": "2.0.0",
-          "resolved": "http://registry.npmjs.org/load-json-file/-/load-json-file-2.0.0.tgz",
+          "resolved": "https://registry.npmjs.org/load-json-file/-/load-json-file-2.0.0.tgz",
           "integrity": "sha1-eUfkIUmvgNaWy/eXvKq8/h/inKg=",
           "dev": true,
           "requires": {
@@ -4891,7 +4891,7 @@
     },
     "events": {
       "version": "1.1.1",
-      "resolved": "http://registry.npmjs.org/events/-/events-1.1.1.tgz",
+      "resolved": "https://registry.npmjs.org/events/-/events-1.1.1.tgz",
       "integrity": "sha1-nr23Y1rQmccNzEwqH1AEKI6L2SQ="
     },
     "evp_bytestokey": {
@@ -5206,7 +5206,7 @@
     },
     "finalhandler": {
       "version": "1.1.1",
-      "resolved": "http://registry.npmjs.org/finalhandler/-/finalhandler-1.1.1.tgz",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.1.tgz",
       "integrity": "sha512-Y1GUDo39ez4aHAw7MysnUD5JzYX+WaIj8I57kO3aEPT1fFRL4sr7mjei97FgnwhAyyzRYmQZaTHb2+9uZ1dPtg==",
       "dev": true,
       "requires": {
@@ -5401,7 +5401,8 @@
         "ansi-regex": {
           "version": "2.1.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "aproba": {
           "version": "1.2.0",
@@ -5422,12 +5423,14 @@
         "balanced-match": {
           "version": "1.0.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "brace-expansion": {
           "version": "1.1.11",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "balanced-match": "^1.0.0",
             "concat-map": "0.0.1"
@@ -5442,17 +5445,20 @@
         "code-point-at": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "concat-map": {
           "version": "0.0.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "console-control-strings": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "core-util-is": {
           "version": "1.0.2",
@@ -5569,7 +5575,8 @@
         "inherits": {
           "version": "2.0.3",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "ini": {
           "version": "1.3.5",
@@ -5581,6 +5588,7 @@
           "version": "1.0.0",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "number-is-nan": "^1.0.0"
           }
@@ -5595,6 +5603,7 @@
           "version": "3.0.4",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "brace-expansion": "^1.1.7"
           }
@@ -5602,12 +5611,14 @@
         "minimist": {
           "version": "0.0.8",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "minipass": {
           "version": "2.2.4",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "safe-buffer": "^5.1.1",
             "yallist": "^3.0.0"
@@ -5626,6 +5637,7 @@
           "version": "0.5.1",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "minimist": "0.0.8"
           }
@@ -5706,7 +5718,8 @@
         "number-is-nan": {
           "version": "1.0.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "object-assign": {
           "version": "4.1.1",
@@ -5718,6 +5731,7 @@
           "version": "1.4.0",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "wrappy": "1"
           }
@@ -5803,7 +5817,8 @@
         "safe-buffer": {
           "version": "5.1.1",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "safer-buffer": {
           "version": "2.1.2",
@@ -5839,6 +5854,7 @@
           "version": "1.0.2",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "code-point-at": "^1.0.0",
             "is-fullwidth-code-point": "^1.0.0",
@@ -5858,6 +5874,7 @@
           "version": "3.0.1",
           "bundled": true,
           "dev": true,
+          "optional": true,
           "requires": {
             "ansi-regex": "^2.0.0"
           }
@@ -5901,12 +5918,14 @@
         "wrappy": {
           "version": "1.0.2",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         },
         "yallist": {
           "version": "3.0.2",
           "bundled": true,
-          "dev": true
+          "dev": true,
+          "optional": true
         }
       }
     },
@@ -5922,6 +5941,11 @@
       "integrity": "sha1-GwqzvVU7Kg1jmdKcDj6gslIHgyc=",
       "dev": true
     },
+    "fuzzysort": {
+      "version": "1.1.4",
+      "resolved": "https://registry.npmjs.org/fuzzysort/-/fuzzysort-1.1.4.tgz",
+      "integrity": "sha512-JzK/lHjVZ6joAg3OnCjylwYXYVjRiwTY6Yb25LvfpJHK8bjisfnZJ5bY8aVWwTwCXgxPNgLAtmHL+Hs5q1ddLQ=="
+    },
     "get-caller-file": {
       "version": "1.0.3",
       "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-1.0.3.tgz",
@@ -5936,7 +5960,7 @@
     },
     "get-stream": {
       "version": "3.0.0",
-      "resolved": "http://registry.npmjs.org/get-stream/-/get-stream-3.0.0.tgz",
+      "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-3.0.0.tgz",
       "integrity": "sha1-jpQ9E1jcN1VQVOy+LtsFqhdO3hQ=",
       "dev": true
     },
@@ -6323,7 +6347,7 @@
     },
     "html-webpack-plugin": {
       "version": "3.2.0",
-      "resolved": "http://registry.npmjs.org/html-webpack-plugin/-/html-webpack-plugin-3.2.0.tgz",
+      "resolved": "https://registry.npmjs.org/html-webpack-plugin/-/html-webpack-plugin-3.2.0.tgz",
       "integrity": "sha1-sBq71yOsqqeze2r0SS69oD2d03s=",
       "dev": true,
       "requires": {
@@ -6399,7 +6423,7 @@
     },
     "http-errors": {
       "version": "1.6.3",
-      "resolved": "http://registry.npmjs.org/http-errors/-/http-errors-1.6.3.tgz",
+      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.6.3.tgz",
       "integrity": "sha1-i1VoC7S+KDoLW/TqLjhYC+HZMg0=",
       "dev": true,
       "requires": {
@@ -6622,7 +6646,7 @@
     },
     "is-builtin-module": {
       "version": "1.0.0",
-      "resolved": "http://registry.npmjs.org/is-builtin-module/-/is-builtin-module-1.0.0.tgz",
+      "resolved": "https://registry.npmjs.org/is-builtin-module/-/is-builtin-module-1.0.0.tgz",
       "integrity": "sha1-VAVy0096wxGfj3bDDLwbHgN6/74=",
       "dev": true,
       "requires": {
@@ -7629,7 +7653,7 @@
     },
     "json5": {
       "version": "0.5.1",
-      "resolved": "http://registry.npmjs.org/json5/-/json5-0.5.1.tgz",
+      "resolved": "https://registry.npmjs.org/json5/-/json5-0.5.1.tgz",
       "integrity": "sha1-Hq3nrMASA0rYTiOWdn6tn6VJWCE=",
       "dev": true
     },
@@ -7719,7 +7743,7 @@
     },
     "load-json-file": {
       "version": "1.1.0",
-      "resolved": "http://registry.npmjs.org/load-json-file/-/load-json-file-1.1.0.tgz",
+      "resolved": "https://registry.npmjs.org/load-json-file/-/load-json-file-1.1.0.tgz",
       "integrity": "sha1-lWkFcI1YtLq0wiYbBPWfMcmTdMA=",
       "dev": true,
       "requires": {
@@ -8043,7 +8067,7 @@
       "dependencies": {
         "minimist": {
           "version": "1.2.0",
-          "resolved": "http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
           "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ=",
           "dev": true
         }
@@ -8168,7 +8192,7 @@
     },
     "minimist": {
       "version": "0.0.8",
-      "resolved": "http://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz",
+      "resolved": "https://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz",
       "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0=",
       "dev": true
     },
@@ -8213,7 +8237,7 @@
     },
     "mkdirp": {
       "version": "0.5.1",
-      "resolved": "http://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz",
+      "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz",
       "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=",
       "dev": true,
       "requires": {
@@ -9939,7 +9963,7 @@
     },
     "parse-asn1": {
       "version": "5.1.1",
-      "resolved": "http://registry.npmjs.org/parse-asn1/-/parse-asn1-5.1.1.tgz",
+      "resolved": "https://registry.npmjs.org/parse-asn1/-/parse-asn1-5.1.1.tgz",
       "integrity": "sha512-KPx7flKXg775zZpnp9SxJlz00gTd4BmJ2yJufSc44gMCRrRQ7NSzAcSJQfifuOLgW6bEi+ftrALtsgALeB2Adw==",
       "dev": true,
       "requires": {
@@ -10647,7 +10671,7 @@
     },
     "readable-stream": {
       "version": "2.3.6",
-      "resolved": "http://registry.npmjs.org/readable-stream/-/readable-stream-2.3.6.tgz",
+      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.6.tgz",
       "integrity": "sha512-tQtKA9WIAhBF3+VLAseyMqZeBjW0AHJoxOtYqSUZNJxauErmLbVm2FW1y+J/YA9dUrAC39ITejlZWhVIwawkKw==",
       "dev": true,
       "requires": {
@@ -11669,7 +11693,7 @@
         },
         "minimist": {
           "version": "1.2.0",
-          "resolved": "http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
           "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ=",
           "dev": true
         },
@@ -11856,7 +11880,7 @@
     },
     "sha.js": {
       "version": "2.4.11",
-      "resolved": "http://registry.npmjs.org/sha.js/-/sha.js-2.4.11.tgz",
+      "resolved": "https://registry.npmjs.org/sha.js/-/sha.js-2.4.11.tgz",
       "integrity": "sha512-QMEp5B7cftE7APOjk5Y6xgrbWu+WkLVQwk8JNjZ8nKRciZaByEW6MubieAiToS7+dwvrjGhH8jRXz3MVd0AYqQ==",
       "dev": true,
       "requires": {
@@ -12310,7 +12334,7 @@
     },
     "strip-ansi": {
       "version": "3.0.1",
-      "resolved": "http://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz",
+      "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz",
       "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=",
       "dev": true,
       "requires": {
@@ -12546,7 +12570,7 @@
     },
     "through": {
       "version": "2.3.8",
-      "resolved": "http://registry.npmjs.org/through/-/through-2.3.8.tgz",
+      "resolved": "https://registry.npmjs.org/through/-/through-2.3.8.tgz",
       "integrity": "sha1-DdTJ/6q8NXlgsbckEV1+Doai4fU=",
       "dev": true
     },
@@ -13202,7 +13226,7 @@
       "dependencies": {
         "minimist": {
           "version": "1.2.0",
-          "resolved": "http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
           "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ=",
           "dev": true
         }
@@ -13865,7 +13889,7 @@
     },
     "wrap-ansi": {
       "version": "2.1.0",
-      "resolved": "http://registry.npmjs.org/wrap-ansi/-/wrap-ansi-2.1.0.tgz",
+      "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-2.1.0.tgz",
       "integrity": "sha1-2Pw9KE3QV5T+hJc8rs3Rz4JP3YU=",
       "dev": true,
       "requires": {
@@ -13973,7 +13997,7 @@
     },
     "yargs": {
       "version": "11.1.0",
-      "resolved": "http://registry.npmjs.org/yargs/-/yargs-11.1.0.tgz",
+      "resolved": "https://registry.npmjs.org/yargs/-/yargs-11.1.0.tgz",
       "integrity": "sha512-NwW69J42EsCSanF8kyn5upxvjp5ds+t3+udGBeTbFnERA+lF541DDpMawzo4z6W/QrzNM18D+BPMiOBibnFV5A==",
       "dev": true,
       "requires": {

client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cellxgene",
-  "version": "0.2.3",
+  "version": "0.3.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
@@ -34,6 +34,7 @@
     "d3": "^4.10.0",
     "d3-scale-chromatic": "^1.3.0",
     "font-color-contrast": "^1.0.3",
+    "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
     "gl-matrix": "^2.7.1",
     "key-pressed": "0.0.1",

client/src/actions/index.js

@@ -24,7 +24,7 @@ const doInitialDataLoad = () =>
         "config",
         "schema",
         "annotations/obs",
-        "annotations/var",
+        "annotations/var?annotation-name=name",
         "layout/obs"
       ])
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)

client/src/components/categorical/category.js

@@ -25,24 +25,22 @@ class Category extends React.Component {
     const { categoricalSelectionState, metadataField } = this.props;
     const cat = categoricalSelectionState[metadataField];
     const categoryCount = {
-      // total number of options in this category
-      totalOptionCount: cat.numOptions,
+      // total number of categories in this dimension
+      totalCatCount: cat.numCategories,
       // number of selected options in this category
-      selectedOptionCount: _.reduce(
-        cat.optionSelected,
+      selectedCatCount: _.reduce(
+        cat.categorySelected,
         (res, cond) => (cond ? res + 1 : res),
         0
       )
     };
-    if (categoryCount.selectedOptionCount === categoryCount.totalOptionCount) {
+    if (categoryCount.selectedCatCount === categoryCount.totalCatCount) {
       /* everything is on, so not indeterminate */
       this.checkbox.indeterminate = false;
-    } else if (categoryCount.selectedOptionCount === 0) {
+    } else if (categoryCount.selectedCatCount === 0) {
       /* nothing is on, so no */
       this.checkbox.indeterminate = false;
-    } else if (
-      categoryCount.selectedOptionCount < categoryCount.totalOptionCount
-    ) {
+    } else if (categoryCount.selectedCatCount < categoryCount.totalCatCount) {
       /* to be explicit... */
       this.checkbox.indeterminate = true;
     }
@@ -88,12 +86,12 @@ class Category extends React.Component {
     const { categoricalSelectionState, metadataField } = this.props;
 
     const cat = categoricalSelectionState[metadataField];
-    const optTuples = alphabeticallySortedValues([...cat.optionIndex]);
+    const optTuples = alphabeticallySortedValues([...cat.categoryIndices]);
     return _.map(optTuples, (tuple, i) => (
       <Value
         key={tuple[1]}
         metadataField={metadataField}
-        optionIndex={tuple[1]}
+        categoryIndex={tuple[1]}
         i={i}
       />
     ));

client/src/components/categorical/value.js

@@ -11,20 +11,20 @@ import _ from "lodash";
 }))
 class CategoryValue extends React.Component {
   toggleOff() {
-    const { dispatch, metadataField, optionIndex } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter deselect",
       metadataField,
-      optionIndex
+      categoryIndex
     });
   }
 
   toggleOn() {
-    const { dispatch, metadataField, optionIndex } = this.props;
+    const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter select",
       metadataField,
-      optionIndex
+      categoryIndex
     });
   }
 
@@ -32,7 +32,7 @@ class CategoryValue extends React.Component {
     const {
       categoricalSelectionState,
       metadataField,
-      optionIndex,
+      categoryIndex,
       colorAccessor,
       colorScale,
       i,
@@ -42,10 +42,12 @@ class CategoryValue extends React.Component {
     if (!categoricalSelectionState) return null;
 
     const category = categoricalSelectionState[metadataField];
-    const selected = category.optionSelected[optionIndex];
-    const count = category.optionCount[optionIndex];
-    const value = category.optionValue[optionIndex];
-    const displayString = String(category.optionValue[optionIndex]).valueOf();
+    const selected = category.categorySelected[categoryIndex];
+    const count = category.categoryCounts[categoryIndex];
+    const value = category.categoryValues[categoryIndex];
+    const displayString = String(
+      category.categoryValues[categoryIndex]
+    ).valueOf();
 
     /* this is the color scale, so add swatches below */
     const c = metadataField === colorAccessor;

client/src/components/geneExpression/index.js

@@ -4,14 +4,51 @@
 import React from "react";
 import _ from "lodash";
 import * as d3 from "d3";
+import fuzzysort from "fuzzysort";
+
 import { connect } from "react-redux";
-import { Button, Tooltip } from "@blueprintjs/core";
+import { MenuItem } from "@blueprintjs/core";
+import { Suggest } from "@blueprintjs/select";
 import HistogramBrush from "../brushableHistogram";
 import * as globals from "../../globals";
 import actions from "../../actions";
 import { postUserErrorToast } from "../framework/toasters";
 import ExpressionButtons from "./expressionButtons";
 
+const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
+  if (!modifiers.matchesPredicate) {
+    return null;
+  }
+  /* the fuzzysort wraps the object with other properties, like a score */
+  const gene = fuzzySortResult.obj;
+  const text = gene.name;
+
+  return (
+    <MenuItem
+      active={modifiers.active}
+      disabled={modifiers.disabled}
+      // Use of annotations in this way is incorrect and dataset specific.
+      // See https://github.com/chanzuckerberg/cellxgene/issues/483
+      // label={gene.n_counts}
+      key={gene.name}
+      onClick={g => {
+        /* this fires when user clicks a menu item */
+        handleClick(g);
+      }}
+      text={text}
+    />
+  );
+};
+
+const filterGenes = (query, genes) => {
+  /* fires on load, once, and then for each character typed into the input */
+  return fuzzysort.go(query, genes, {
+    key: "name",
+    limit: 5,
+    threshold: -10000 // don't return bad results
+  });
+};
+
 @connect(state => {
   const metadata = _.get(state.controls.world, "obsAnnotations", null);
   const ranges = _.get(state.controls.world, "summary.obs", null);
@@ -29,23 +66,9 @@ import ExpressionButtons from "./expressionButtons";
   };
 })
 class GeneExpression extends React.Component {
-  constructor(props) {
-    super(props);
-    this.state = {
-      gene: ""
-    };
-  }
-
-  keyPress(e) {
-    if (e.keyCode === 13) {
-      this.handleClick();
-    }
-  }
-
-  handleClick() {
+  handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
-    const { gene } = this.state;
-
+    const gene = g.target;
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
     } else if (userDefinedGenes.length > 15) {
@@ -60,13 +83,11 @@ class GeneExpression extends React.Component {
         type: "user defined gene",
         data: gene
       });
-      this.setState({ gene: "" });
     }
   }
 
   render() {
     const { world, userDefinedGenes, differential } = this.props;
-    const { gene } = this.state;
 
     return (
       <div>
@@ -87,27 +108,27 @@ class GeneExpression extends React.Component {
             style={{ padding: globals.leftSidebarSectionPadding }}
             className="bp3-control-group"
           >
-            <div className="bp3-input-group bp3-fill">
-              <input
-                onKeyDown={this.keyPress.bind(this)}
-                onChange={e => {
-                  this.setState({ gene: e.target.value });
-                }}
-                value={gene}
-                type="text"
-                className="bp3-input"
-                placeholder="Enter a gene name"
-                style={{ paddingRight: 94 }}
-              />
-            </div>
-            <Tooltip
-              content="Add a gene to see its expression levels"
-              position="bottom"
-            >
-              <Button intent="primary" onClick={this.handleClick.bind(this)}>
-                Add
-              </Button>
-            </Tooltip>
+            <Suggest
+              closeOnSelect
+              openOnKeyDown
+              resetOnSelect
+              noResults={<MenuItem disabled text="No matching genes." />}
+              onItemSelect={g => {
+                /* this happens on 'enter' */
+                this.handleClick(g);
+              }}
+              inputValueRenderer={g => {
+                return "";
+              }}
+              itemListPredicate={filterGenes}
+              itemRenderer={renderGene.bind(this)}
+              items={
+                world && world.varAnnotations
+                  ? world.varAnnotations
+                  : [{ name: "No genes", n_counts: "" }]
+              }
+              popoverProps={{ minimal: true }}
+            />
           </div>
           {world && userDefinedGenes.length > 0
             ? _.map(userDefinedGenes, (geneName, index) => {

client/src/reducers/controls.js

@@ -1,7 +1,7 @@
 // jshint esversion: 6
 
 import _ from "lodash";
-import { World, kvCache } from "../util/stateManager";
+import { World, kvCache, WorldUtil } from "../util/stateManager";
 import parseRGB from "../util/parseRGB";
 import Crossfilter from "../util/typedCrossfilter";
 import * as globals from "../globals";
@@ -24,25 +24,27 @@ Remember that option values can be ANY js type, except undefined/null.
   {
     _category_name_1: {
       // map of option value to index
-      optionIndex: Map([
-        optval1: index,
+      categoryIndices: Map([
+        catval1: index,
         ...
       ])
 
       // index->selection true/false state
-      optionSelected: [ true/false, true/false, ... ]
+      categorySelected: [ true/false, true/false, ... ]
 
       // number of options
-      numOptions: number,
+      numCategories: number,
 
       // isTruncated - true if the options for selection has
       // been truncated (ie, was too large to implement)
     }
   }
 */
-function topNoptions(summary) {
-  const counts = _.map(summary.categories, cat => summary.options[cat]);
-  const sortIndex = fillRange(new Array(summary.numOptions)).sort(
+function topNCategories(summary) {
+  const counts = _.map(summary.categories, cat =>
+    summary.categoryCounts.get(cat)
+  );
+  const sortIndex = fillRange(new Array(summary.numCategories)).sort(
     (a, b) => counts[b] - counts[a]
   );
   const sortedCategories = _.map(sortIndex, i => summary.categories[i]);
@@ -65,20 +67,18 @@ function createCategoricalSelectionState(state, world) {
         key !== "name" &&
         value.categories.length < state.maxCategoryItems;
       if (isSelectableCategory) {
-        const [optionValue, optionCount] = topNoptions(value);
-        // const optionCount = Object.values(value.options);
-
-        const optionIndex = new Map(optionValue.map((v, i) => [v, i]));
-        const numOptions = optionIndex.size;
-        const optionSelected = new Array(numOptions).fill(true);
-        const isTruncated = optionValue.length < value.numOptions;
+        const [categoryValues, categoryCounts] = topNCategories(value);
+        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
+        const numCategories = categoryIndices.size;
+        const categorySelected = new Array(numCategories).fill(true);
+        const isTruncated = categoryValues.length < value.numCategories;
         res[key] = {
-          optionValue, // array: of natively typed option values
-          optionIndex, // map: option value (native type) -> option index
-          optionSelected, // array: t/f selection state
-          numOptions, // number: of options
+          categoryValues, // array: of natively typed category values
+          categoryIndices, // map: category value (native type) -> category index
+          categorySelected, // array: t/f selection state
+          numCategories, // number: of categories
           isTruncated, // bool: true if list was truncated
-          optionCount // array: cardinality of each option
+          categoryCounts // array: cardinality of each category
         };
       }
     }
@@ -87,12 +87,12 @@ function createCategoricalSelectionState(state, world) {
 }
 
 /*
-given a categoricalSelectionState, return the list of all option values
+given a categoricalSelectionState, return the list of all category values
 where selection state is true (ie, they are selected).
 */
 function selectedValuesForCategory(categorySelectionState) {
-  const selectedValues = _([...categorySelectionState.optionIndex])
-    .filter(tuple => categorySelectionState.optionSelected[tuple[1]])
+  const selectedValues = _([...categorySelectionState.categoryIndices])
+    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
     .map(tuple => tuple[0])
     .value();
   return selectedValues;
@@ -175,6 +175,7 @@ const Controls = (
       );
       const crossfilter = Crossfilter(world.obsAnnotations);
       const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+      WorldUtil.clearCaches();
 
       const worldVarDataCache = world.varDataCache;
 
@@ -247,6 +248,7 @@ const Controls = (
       );
       const crossfilter = Crossfilter(world.obsAnnotations);
       const dimensionMap = World.createObsDimensionMap(crossfilter, world);
+      WorldUtil.clearCaches();
 
       const worldVarDataCache = world.varDataCache;
       /* var dimensions */
@@ -514,15 +516,15 @@ const Controls = (
           Categorical metadata
     *******************************/
     case "categorical metadata filter select": {
-      const newOptionSelected = Array.from(
-        state.categoricalSelectionState[action.metadataField].optionSelected
+      const newCategorySelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].categorySelected
       );
-      newOptionSelected[action.optionIndex] = true;
+      newCategorySelected[action.categoryIndex] = true;
       const newCategoricalSelectionState = {
         ...state.categoricalSelectionState,
         [action.metadataField]: {
           ...state.categoricalSelectionState[action.metadataField],
-          optionSelected: newOptionSelected
+          categorySelected: newCategorySelected
         }
       };
 
@@ -538,15 +540,15 @@ const Controls = (
       };
     }
     case "categorical metadata filter deselect": {
-      const newOptionSelected = Array.from(
-        state.categoricalSelectionState[action.metadataField].optionSelected
+      const newCategorySelected = Array.from(
+        state.categoricalSelectionState[action.metadataField].categorySelected
       );
-      newOptionSelected[action.optionIndex] = false;
+      newCategorySelected[action.categoryIndex] = false;
       const newCategoricalSelectionState = {
         ...state.categoricalSelectionState,
         [action.metadataField]: {
           ...state.categoricalSelectionState[action.metadataField],
-          optionSelected: newOptionSelected
+          categorySelected: newCategorySelected
         }
       };
 
@@ -566,8 +568,9 @@ const Controls = (
         ...state.categoricalSelectionState,
         [action.metadataField]: {
           ...state.categoricalSelectionState[action.metadataField],
-          optionSelected: Array.from(
-            state.categoricalSelectionState[action.metadataField].optionSelected
+          categorySelected: Array.from(
+            state.categoricalSelectionState[action.metadataField]
+              .categorySelected
           ).fill(false)
         }
       };
@@ -584,8 +587,9 @@ const Controls = (
         ...state.categoricalSelectionState,
         [action.metadataField]: {
           ...state.categoricalSelectionState[action.metadataField],
-          optionSelected: Array.from(
-            state.categoricalSelectionState[action.metadataField].optionSelected
+          categorySelected: Array.from(
+            state.categoricalSelectionState[action.metadataField]
+              .categorySelected
           ).fill(true)
         }
       };

client/src/util/actionHelpers.js

@@ -38,7 +38,11 @@ export const doJsonRequest = async url => {
     return res.json();
   }
   // else an error
-  const msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  let msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
+  const body = await res.text();
+  if (body && body.length > 0) {
+    msg = `${msg} -- ${body}`;
+  }
   dispatchNetworkErrorMessageToUser(msg);
   throw new Error(msg);
 };

client/src/util/stateManager/index.js

@@ -17,3 +17,4 @@ exists to support those concepts.
 export * as Universe from "./universe";
 export * as World from "./world";
 export * as kvCache from "./keyvalcache";
+export * as WorldUtil from "./worldUtil";

client/src/util/stateManager/summarizeAnnotations.js

@@ -8,6 +8,7 @@ Value will be an object, containing summary information.
 
 For continuous annotations (int, float, etc):
   <annotation_name>: {
+    categorical: false,
     range {
       min: <number>,
       max: <number>
@@ -15,12 +16,14 @@ For continuous annotations (int, float, etc):
   }
 
 For categorical annotations (boolean, string, category):
-  <annotatoin_name>: {
-    options: {
-      <option1>: <number>,
+  <annotation_name>: {
+    categorical: true,
+    categories: [ <category1>, <category2>, ... ]
+    categoryCounts: Map {
+      <category1>: <number>,
       ...
     },
-    numOptions: <number>
+    numCategories: <number>
   }
 
 Summarize will be returned for BOTH obs and var annotations.
@@ -28,19 +31,19 @@ Summarize will be returned for BOTH obs and var annotations.
 Example:
   {
     "Splice_sites_Annotated": {
-      "range": {
+      categorical: false,
+      range: {
         "min": 26,
         "max": 1075869
       }
     },
     "Selection": {
-      numOptions, 6,
-      "options": {
+      categorical: true,
+      numCategories, 3,
+      categories: [ "Astrocytes(HEPACAM)", "Endothelial(BSC)", "Unpanned" ],
+      categoryCounts: Map {
         "Astrocytes(HEPACAM)": 714,
         "Endothelial(BSC)": 123,
-        "Oligodendrocytes(GC)": 294,
-        "Neurons(Thy1)": 685,
-        "Microglia(CD45)": 1108,
         "Unpanned": 665
       }
     }
@@ -48,46 +51,46 @@ Example:
 
 NOTE: will not summarize the required 'name' annotation, as that is
 specified as unique per element.
-
-TODO: XXX - this data structure coerces all metadata categories into a string
-(ie, stores values as an Object property in the `options` field).   This looses
-information (eg, type) for category types which are not strings.  Consider an
-alterative data structure that does not use the object property for non-string
-data types (and does not use _.countBy to summarize).
 */
-function summarizeDimension(schema, annotations) {
-  return _(schema)
+function _summarizeAnnotations(_schema, annotations) {
+  const summary = _(_schema) // lodash wrapping: https://lodash.com/docs/4.17.11#lodash
     .filter(v => v.name !== "name")
     .keyBy("name")
     .mapValues(anno => {
       const { name, type } = anno;
       const continuous = type === "int32" || type === "float32";
 
-      if (!continuous) {
-        const categories = _.uniq(_.flatMap(annotations, name));
-        const options = _.countBy(annotations, name);
-        const numOptions = _.size(options);
-        return {
-          numOptions,
-          options,
-          categories
-        };
-      }
-
       if (continuous) {
         let min = Number.POSITIVE_INFINITY;
         let max = Number.NEGATIVE_INFINITY;
-        _.forEach(annotations, obs => {
-          const val = Number(obs[name]);
+        for (let r = 0; r < annotations.length; r += 1) {
+          const val = Number(annotations[r][name]);
           min = val < min ? val : min;
           max = val > max ? val : max;
-        });
-        return { range: { min, max } };
+        }
+        return {
+          categorical: false,
+          range: { min, max }
+        };
       }
 
-      throw new Error("incomprehensible schema");
+      /* else categorical */
+      const categoryCounts = new Map();
+      for (let r = 0; r < annotations.length; r += 1) {
+        const val = annotations[r][name];
+        let curCount = categoryCounts.get(val);
+        if (curCount === undefined) curCount = 0;
+        categoryCounts.set(val, curCount + 1);
+      }
+      return {
+        categorical: true,
+        categories: [...categoryCounts.keys()],
+        categoryCounts,
+        numCategories: categoryCounts.size
+      };
     })
     .value();
+  return summary;
 }
 
 export default function summarizeAnnotations(
@@ -96,7 +99,7 @@ export default function summarizeAnnotations(
   varAnnotations
 ) {
   return {
-    obs: summarizeDimension(schema.annotations.obs, obsAnnotations),
-    var: summarizeDimension(schema.annotations.var, varAnnotations)
+    obs: _summarizeAnnotations(schema.annotations.obs, obsAnnotations),
+    var: _summarizeAnnotations(schema.annotations.var, varAnnotations)
   };
 }

client/src/util/stateManager/worldUtil.js

@@ -0,0 +1,65 @@
+/* eslint-disable import/prefer-default-export */
+import _ from "lodash";
+
+/*
+Various utility functions operating on World/Universe
+*/
+
+/*
+Count unique category values, binning first by dim1 then by dim2
+Return:
+
+Map {
+   dim1_val1: Map {
+      dim2_val1: number,
+      dim2_val2: number,
+      ...
+   },
+   ...
+}
+
+*/
+function _countCategoryValues2D(dim1, dim2, rows) {
+  const dimMap = new Map();
+  for (let r = 0; r < rows.length; r += 1) {
+    const row = rows[r];
+    const val1 = row[dim1];
+    const val2 = row[dim2];
+    let d2Map = dimMap.get(val1);
+    if (d2Map === undefined) {
+      d2Map = new Map();
+      dimMap.set(val1, d2Map);
+    }
+    let curCount = d2Map.get(val2);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    d2Map.set(val2, curCount + 1);
+  }
+  return dimMap;
+}
+
+let __worldUtilMemoId__ = 0;
+function _memoizedId(x) {
+  if (!x.__worldUtilMemoId__) {
+    __worldUtilMemoId__ += 1;
+    x.__worldUtilMemoId__ = __worldUtilMemoId__;
+  }
+  return x.__worldUtilMemoId__;
+}
+function _countCategoryValues2DResolver(...args) {
+  const id = args[0] + args[1] + _memoizedId(args[2]);
+  return id;
+}
+
+export const countCategoryValues2D = _.memoize(
+  _countCategoryValues2D,
+  _countCategoryValues2DResolver
+);
+
+/*
+Clear any cached data within WorldUtil caches, eg, memoized functions
+*/
+export function clearCaches() {
+  countCategoryValues2D.cache.clear();
+}

server/app/rest_api/rest.py

@@ -1,5 +1,6 @@
 from http import HTTPStatus
 import pkg_resources
+import warnings
 
 from flask import (
     Blueprint, current_app, jsonify, make_response, request
@@ -7,7 +8,7 @@ from flask import (
 from flask_restful_swagger_2 import Api, swagger, Resource
 from werkzeug.datastructures import ImmutableMultiDict
 
-from server.app.util.constants import Axis, DiffExpMode
+from server.app.util.constants import Axis, DiffExpMode, JSON_NaN_to_num_warning_msg
 from server.app.util.filter import parse_filter, QueryStringError
 from server.app.util.models import FilterModel
 from server.app.util.utils import get_mime_type
@@ -160,7 +161,12 @@ class AnnotationsObsAPI(Resource):
             annotation_response = current_app.data.annotation({}, "obs", fields)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Fetch annotations (metadata) for filtered subset of observations.",
@@ -211,7 +217,12 @@ class AnnotationsObsAPI(Resource):
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class AnnotationsVarAPI(Resource):
@@ -253,7 +264,12 @@ class AnnotationsVarAPI(Resource):
             annotation_response = current_app.data.annotation({}, "var", fields)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Fetch annotations (metadata) for filtered subset of variables.",
@@ -302,7 +318,12 @@ class AnnotationsVarAPI(Resource):
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
         except FilterError:
             return make_response("Malformed filter", HTTPStatus.BAD_REQUEST)
-        return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(annotation_response), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DataObsAPI(Resource):
@@ -363,6 +384,10 @@ class DataObsAPI(Resource):
             return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.OBS))), HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Get data (expression values) from the dataframe.",
@@ -407,6 +432,10 @@ class DataObsAPI(Resource):
                                  HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DataVarAPI(Resource):
@@ -465,6 +494,10 @@ class DataVarAPI(Resource):
             return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.VAR))), HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc({
         "summary": "Get data (expression values) from the dataframe.",
@@ -510,6 +543,10 @@ class DataVarAPI(Resource):
                                  HTTPStatus.OK)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DiffExpObsAPI(Resource):
@@ -614,7 +651,12 @@ class DiffExpObsAPI(Resource):
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
         except InteractiveError:
             return make_response("Non-interactive request", HTTPStatus.FORBIDDEN)
-        return make_response(jsonify(diffexp), HTTPStatus.OK)
+        try:
+            return make_response(jsonify(diffexp), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class LayoutObsAPI(Resource):
@@ -647,7 +689,12 @@ class LayoutObsAPI(Resource):
             layout = current_app.data.layout({})
         except PrepareError as e:
             return make_response(e.message, HTTPStatus.INTERNAL_SERVER_ERROR)
-        return make_response((jsonify({"layout": layout})), HTTPStatus.OK)
+        try:
+            return make_response((jsonify({"layout": layout})), HTTPStatus.OK)
+        except ValueError as e:
+            # JSON encoding failure, usually due to bad data
+            warnings.warn(JSON_NaN_to_num_warning_msg)
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     # @swagger.doc({
     #     "summary": "Observation layout for filtered subset.",

server/app/scanpy_engine/scanpy_engine.py

@@ -35,6 +35,10 @@ class ScanpyEngine(CXGDriver):
         self.diffexp_options = ["ttest"]
         self._create_schema()
 
+        # TODO: temporary work-arounds
+        if args['nan_to_num']:
+            self._IEEE754_special_values_workaround()
+
     def _alias_annotation_names(self, axis, name):
         """
         Do all user-specified annotation aliasing.
@@ -69,7 +73,7 @@ class ScanpyEngine(CXGDriver):
     def _can_cast_to_float32(ann):
         if ann.dtype.kind == "f":
             if not np.can_cast(ann.dtype, np.float32):
-                warnings.warn(f"Annotation {ann.name} will be converted to 32 bit float and may loose precision.")
+                warnings.warn(f"Annotation {ann.name} will be converted to 32 bit float and may lose precision.")
             return True
         return False
 
@@ -172,6 +176,67 @@ class ScanpyEngine(CXGDriver):
                 f"`cellxgene prepare --layout {self.layout_method} <datafile>` "
                 f"to solve this problem. ")
 
+    def _IEEE754_special_values_workaround(self):
+        """
+        TODO: temporary workaround
+
+        Because all floating point data is serialized to JSON, and JSON has no means of representing
+        non-finite, floating point special values (NaN, +/-Infinity, etc), we include this temporary
+        work-around.
+
+        This will likely be removed in the future, contingent upon improved marshalling.
+
+        Where non-finite floating point is present in obs, var or X:
+        * issue a warning to the user that these values will be convert to finite numbers.
+        * set NaN to zero, and Infinities to min/max of the element.
+        """
+
+        # annotations
+        for ax in Axis:
+            curr_axis = getattr(self.data, str(ax))
+            for ann in curr_axis:
+                dtype = curr_axis[ann].dtype
+                if dtype.kind == 'f':
+                    finite_idx = np.isfinite(curr_axis[ann])
+                    if not finite_idx.all():
+                        curr_axis.loc[np.isnan(curr_axis[ann]), ann] = 0
+                        curr_axis.loc[np.isneginf(curr_axis[ann]), ann] = curr_axis[ann][finite_idx].min()
+                        curr_axis.loc[np.isposinf(curr_axis[ann]), ann] = curr_axis[ann][finite_idx].max()
+                        warnings.warn(
+                            f"{str(ax).title()} annotation '{ann}' contains floating point NaN or Infinities. "
+                            f"These will be converted to finite values."
+                        )
+
+        # X
+        non_finite_X_found = False
+        if sparse.issparse(self.data._X):
+            coo = self.data._X.tocoo()
+            finite_idx = np.isfinite(coo.data)
+            if not finite_idx.all():
+                non_finite_X_found = True
+                coo.data[np.isnan(coo.data)] = 0
+                coo.data[np.isneginf(coo.data)] = np.min(coo.data[finite_idx])
+                coo.data[np.isposinf(coo.data)] = np.max(coo.data[finite_idx])
+                coo.eliminate_zeros()
+                _X = coo.asformat(self.data._X.getformat())
+                self.data._X = _X
+        else:
+            _X = self.data._X
+            finite_idx = np.isfinite(_X.flat)
+            if not finite_idx.all():
+                non_finite_X_found = True
+                min_X = _X.flat[finite_idx].min()
+                max_X = _X.flat[finite_idx].max()
+                _X[np.isnan(_X)] = 0
+                _X[np.isneginf(_X)] = min_X
+                _X[np.isposinf(_X)] = max_X
+
+        if non_finite_X_found:
+            warnings.warn(
+                "Dataframe X contains floating point NaN or Infinities. "
+                "These will be converted to finite values."
+            )
+
     def filter_dataframe(self, filter):
         """
          Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with

server/app/util/constants.py

@@ -25,3 +25,6 @@ class Axis(AugmentedEnum):
 class DiffExpMode(AugmentedEnum):
     TOP_N = "topN"
     VAR_FILTER = "varFilter"
+
+
+JSON_NaN_to_num_warning_msg = "JSON encoding failure - suggest trying --nan-to-num command line option"

server/app/util/utils.py

@@ -7,6 +7,17 @@ from server.app.util.errors import MimeTypeError
 
 
 class Float32JSONEncoder(json.JSONEncoder):
+    def __init__(self, *args, **kwargs):
+        """
+        NaN/Infinities are illegal in standard JSON.  Python extends JSON with
+        non-standard symbols that most JavaScript JSON parsers do not understand.
+        The `allow_nan` parameter will force Python simplejson to throw an ValueError
+        if it runs into non-finite floating point values which are unsupported by
+        standard JSON.
+        """
+        kwargs['allow_nan'] = False
+        super().__init__(*args, **kwargs)
+
     def default(self, obj):
         if isinstance(obj, float32):
             return float(obj)
@@ -15,6 +26,10 @@ class Float32JSONEncoder(json.JSONEncoder):
         return json.JSONEncoder.default(self, obj)
 
 
+def custom_format_warning(msg, *args, **kwargs):
+    return f"[cellxgene] Warning: {msg} \n"
+
+
 def get_mime_type(default="application/json", acceptable_types=["application/json", "text/csv"], query_param=None,
                   header=None):
     mime_type = default

server/cli/cli.py

@@ -5,7 +5,7 @@ from .prepare import prepare
 
 
 @click.group(name="cellxgene", context_settings=dict(max_content_width=85))
-@click.version_option(version="0.2.3", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
+@click.version_option(version="0.3.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
 def cli():
     pass
 

server/cli/launch.py

@@ -1,11 +1,14 @@
-import sys
-import click
 import logging
 from os import devnull
 from os.path import splitext, basename
+import sys
+import warnings
 import webbrowser
 
+import click
+
 from server.app.util.errors import ScanpyFileError
+from server.app.util.utils import custom_format_warning
 
 
 @click.command()
@@ -24,14 +27,16 @@ from server.app.util.errors import ScanpyFileError
 @click.option("--port", "-p", help="Port to run server on.", metavar="", default=5005, show_default=True)
 @click.option("--obs-names", default=None, metavar="", help="Name of annotation field to use for observations.")
 @click.option("--var-names", default=None, metavar="", help="Name of annotation to use for variables.")
-@click.option("--listen-all", is_flag=True, default=False, show_default=True,
-              help="Bind to all interfaces (this makes the server accessible beyond this computer).")
+@click.option("--host", default="127.0.0.1", help="Host IP address")
 @click.option("--max-category-items", default=100, metavar="", show_default=True,
               help="Limits the number of categorical annotation items displayed.")
 @click.option("--diffexp-lfc-cutoff", default=0.01, show_default=True,
               help="Relative expression cutoff used when selecting top N differentially expressed genes")
+@click.option("--nan-to-num", is_flag=True, default=False, show_default=True,
+              help="Replace all floating point NaN with zero, and infinities with finite numbers")
 def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
-           open_browser, port, listen_all, max_category_items, diffexp_lfc_cutoff):
+           open_browser, port, host, max_category_items, diffexp_lfc_cutoff,
+           nan_to_num):
     """Launch the cellxgene data viewer.
     This web app lets you explore single-cell expression data.
     Data must be in a format that cellxgene expects, read the
@@ -46,9 +51,6 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
     # Startup message
     click.echo("[cellxgene] Starting the CLI...")
 
-    # Import Flask app
-    from server.app.app import app
-
     # Argument checking
     name, extension = splitext(data)
     if extension != ".h5ad":
@@ -57,6 +59,8 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
     if debug:
         verbose = True
         open_browser = False
+    else:
+        warnings.formatwarning = custom_format_warning
 
     if not verbose:
         sys.tracebacklimit = 0
@@ -65,15 +69,13 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
         file_parts = splitext(basename(data))
         title = file_parts[0]
 
-    if listen_all:
-        host = "0.0.0.0"
-    else:
-        host = "127.0.0.1"
-
     # Setup app
     cellxgene_url = f"http://{host}:{port}"
     api_base = f"{cellxgene_url}/api/"
 
+    # Import Flask app
+    from server.app.app import app
+
     app.config.update(
         DATASET_TITLE=title,
         CXG_API_BASE=api_base
@@ -97,7 +99,8 @@ def launch(data, layout, diffexp, title, verbose, debug, obs_names, var_names,
         "max_category_items": max_category_items,
         "diffexp_lfc_cutoff": diffexp_lfc_cutoff,
         "obs_names": obs_names,
-        "var_names": var_names
+        "var_names": var_names,
+        "nan_to_num": nan_to_num
     }
 
     try:

server/requirements.txt

@@ -11,4 +11,4 @@ numpy>=1.14.5
 pandas>=0.23.1
 scanpy>=1.3.2
 scipy>=1.1.0
-scikit-learn==0.19.1
+scikit-learn>=0.20.1

server/test/test_scanpy_engine.py

@@ -14,7 +14,8 @@ from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
 class UtilTest(unittest.TestCase):
     def setUp(self):
         args = {'layout': 'umap', 'diffexp': 'ttest', 'max_category_items': 100,
-                'obs_names': None, 'var_names': None, 'diffexp_lfc_cutoff': 0.01}
+                'obs_names': None, 'var_names': None, 'diffexp_lfc_cutoff': 0.01,
+                'nan_to_num': True}
 
         self.data = ScanpyEngine("example-dataset/pbmc3k.h5ad", args)
         self.data._create_schema()

setup.py

@@ -8,7 +8,7 @@ with open("server/requirements.txt") as fh:
 
 setup(
     name="cellxgene",
-    version="0.2.3",
+    version="0.3.0",
     packages=find_packages(),
     url="https://github.com/chanzuckerberg/cellxgene",
     license="MIT",