bump version (#601)

* Make api paths relative

* Remove request import

* Set publicPath to be relative

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.4.0
+current_version = 0.6.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"

.gitignore

@@ -35,3 +35,7 @@ npm-debug.log
 __pycache__
 *.DS_Store*
 data
+
+# Jekyll
+docs/_site/
+docs/Gemfile.lock

.travis.yml

@@ -8,8 +8,8 @@ cache:
 install:
   - set -eo pipefail
   - pip install flake8
-  - ./bin/build-client
-  - pip install -e .
+  - make build
+  - make install
   - pip install -r server/requirements-dev.txt
   - docker build .
 script:

README.md

@@ -8,10 +8,9 @@
 
 ## getting started
 
-You'll need **python 3.6** and **Google Chrome**. (*Warning*: Python 3.7 is **not** supported at this time)
+You'll need **python 3.6** and **Google Chrome**. (_Warning_: Python 3.7 is **not** supported at this time)
 The web UI is tested on OSX and Windows using Chrome, and the python CLI is tested on OSX and Ubuntu (via WSL/Windows). It should work on other platforms, but if you run into trouble let us know (see [help](#help-and-contact) below).
 
-
 To install run
 
 ```
@@ -71,7 +70,7 @@ To prepare from an existing `.h5ad` file use
 cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
 ```
 
-This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://github.com/theislab/scanpy/blob/master/scanpy/preprocessing/recipes.py).
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
 
 Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
 
@@ -118,12 +117,11 @@ pip install cellxgene
 
 ## docker
 
-We have included a dockerfile to conveniently run cellxgene from docker. 
+We have included a dockerfile to conveniently run cellxgene from docker.
 
-1. Build the  image `docker build . -t cellxgene`
+1. Build the image `docker build . -t cellxgene`
 2. Run the container and mount data `docker run -v "$PWD/example-dataset/:/data/" -p 5005:5005 cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad`
-    * You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
-
+   - You will need to use --host 0.0.0.0 to have the container listen to incoming requests from the browser
 
 ## FAQ
 
@@ -185,14 +183,6 @@ Currently this is not supported directly, but you should be able to do this manu
 - `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
 - `.obsm` is used for layout
 
-<hr>
-
-> When I start cellxgene, I get an error `Unexpected HTTP response 500, INTERNAL SERVER ERROR -- Out of range float values are not JSON compliant` in the web UI, or `Warning: JSON encoding failure - suggest trying --nan-to-num command line option` in the CLI. What can I do?
-
-At the moment, cellxgene is unable to transmit floating point NaN or Inifinty values to the web UI (due to a limitation on data serialization method in use). We expect to resolve this in a future release, but in the meantime, you can work around this issue by starting cellxgene with the `--nan-to-num` command line option, ie, `cellxgene launch data.h5ad --nan-to-num`.
-
-This option will convert all NaNs to zero, and all positive/negative infinities to the min/max of the data element within which the value was found (eg, +Infinity within an `obs` annotation will be converted to the maximum finite value in that annotation). This option will increase startup time, so we recommend only using it when the dataset contains NaN/Infinities.
-
 </details>
 
 <details>
@@ -201,9 +191,13 @@ This option will convert all NaNs to zero, and all positive/negative infinities
 
 <hr>
 
-> I tried to `pip install cellxgene` and got a weird error I don't understand
+> I tried to `pip install cellxgene` and got a weird error about missing paths to an HDF5 library?  
 
-This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.
+You probably just need to install HDF5 first. If you're on a mac, you can simply `brew install hdf5` and then try `pip install cellxgene` again.  
+
+> I tried to `pip install cellxgene` and got another weird error I don't understand  
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.  
 
 <hr>
 

bin/build-client

@@ -1,15 +0,0 @@
-#!/bin/bash
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-CELLXGENE_DIR=$(dirname $DIR)
-
-cd $CELLXGENE_DIR
-
-npm install --prefix client/ client
-npm run  --prefix client build
-rm -rf server/app/web/static
-mkdir -p server/app/web/static/img
-cp client/build/index.html server/app/web/templates/
-cp -r client/build/static server/app/web/
-
-cp client/build/favicon.png server/app/web/static/img
-cp client/build/service-worker.js server/app/web/static/js/

bin/clean

@@ -1,18 +0,0 @@
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-CELLXGENE_DIR=$(dirname $DIR)
-
-echo "Uninstalling cellxgene"
-yes | pip uninstall cellxgene
-echo "removing node_modules"
-rm -rf $CELLXGENE_DIR/client/node_modules
-echo "removing client_build"
-rm -rf $CELLXGENE_DIR/client/build
-echo "removing dist"
-rm -rf $CELLXGENE_DIR/dist
-echo "removing egg-info"
-rm -rf $CELLXGENE_DIR/cellxgene.egg-info
-echo "removing static files"
-rm -f $CELLXGENE_DIR/server/app/web/templates/index.html
-rm -rf $CELLXGENE_DIR/server/app/web/static
-echo "cellxgene cleanup complete"
-

client/__tests__/setupMissingGlobals.js

@@ -0,0 +1,9 @@
+/*
+Define globals which are present in the client, but not in node (and therefore not in
+the jest test environment).
+*/
+
+import { TextDecoder, TextEncoder } from "util";
+
+global.TextDecoder = TextDecoder;
+global.TextEncoder = TextEncoder;

client/__tests__/util/stateManager/sampleResponses.js

@@ -1,5 +1,7 @@
 /* eslint no-bitwise: "off" */
 import _ from "lodash";
+import { flatbuffers } from "flatbuffers";
+import { NetEncoding } from "../../../src/util/stateManager/matrix_generated";
 
 /*
 test data mocking REST 0.2 API responses.  Used in several tests.
@@ -58,7 +60,7 @@ const aSchemaResponse = {
   }
 };
 
-const anAnnotationsObsResponse = {
+const anAnnotationsObsJSONResponse = {
   names: ["name", "field1", "field2", "field3", "field4"],
   data: _()
     .range(nObs)
@@ -73,7 +75,7 @@ const anAnnotationsObsResponse = {
     .value()
 };
 
-const anAnnotationsVarResponse = {
+const anAnnotationsVarJSONResponse = {
   names: ["fieldA", "fieldB", "fieldC", "fieldD", "name"],
   data: _()
     .range(nVar)
@@ -88,7 +90,74 @@ const anAnnotationsVarResponse = {
     .value()
 };
 
-const aLayoutResponse = {
+function encodeTypedArray(builder, uType, uData) {
+  const uTypeName = NetEncoding.TypedArray[uType];
+  const ArrayType = NetEncoding[uTypeName];
+  const dv = ArrayType.createDataVector(builder, uData);
+  builder.startObject(1);
+  builder.addFieldOffset(0, dv, 0);
+  return builder.endObject();
+}
+
+function encodeMatrix(columns, colIndex = undefined) {
+  const utf8Encoder = new TextEncoder("utf-8");
+  const builder = new flatbuffers.Builder(1024);
+  const cols = _.map(columns, carr => {
+    let uType;
+    let tarr;
+    if (_.every(carr, _.isNumber)) {
+      uType = NetEncoding.TypedArray.Float32Array;
+      tarr = encodeTypedArray(builder, uType, new Float32Array(carr));
+    } else {
+      uType = NetEncoding.TypedArray.JSONEncodedArray;
+      const json = JSON.stringify(carr);
+      const jsonUTF8 = utf8Encoder.encode(json);
+      tarr = encodeTypedArray(builder, uType, jsonUTF8);
+    }
+    NetEncoding.Column.startColumn(builder);
+    NetEncoding.Column.addUType(builder, uType);
+    NetEncoding.Column.addU(builder, tarr);
+    return NetEncoding.Column.endColumn(builder);
+  });
+
+  const encColumns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+  let encColIndex;
+  if (colIndex) {
+    encColIndex = encodeTypedArray(
+      builder,
+      NetEncoding.TypedArray.JSONEncodedArray,
+      utf8Encoder.encode(JSON.stringify(colIndex))
+    );
+  }
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, columns[0].length);
+  NetEncoding.Matrix.addNCols(builder, columns.length);
+  NetEncoding.Matrix.addColumns(builder, encColumns);
+  if (colIndex) {
+    NetEncoding.Matrix.addColIndexType(
+      builder,
+      NetEncoding.TypedArray.JSONEncodedArray
+    );
+    NetEncoding.Matrix.addColIndex(builder, encColIndex);
+  }
+  const root = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(root);
+  return builder.asUint8Array();
+}
+
+const anAnnotationsObsFBSResponse = (() => {
+  const columns = _.zip(...anAnnotationsObsJSONResponse.data).slice(1);
+  return encodeMatrix(columns, anAnnotationsObsJSONResponse.names);
+})();
+
+const anAnnotationsVarFBSResponse = (() => {
+  const columns = _.zip(...anAnnotationsVarJSONResponse.data).slice(1);
+  return encodeMatrix(columns, anAnnotationsVarJSONResponse.names);
+})();
+
+const aLayoutJSONResponse = {
   layout: {
     ndims: 2,
     coordinates: _()
@@ -98,6 +167,36 @@ const aLayoutResponse = {
   }
 };
 
+const aLayoutFBSResponse = (() => {
+  const coords = [
+    new Float32Array(nObs).fill(Math.random()),
+    new Float32Array(nObs).fill(Math.random())
+  ];
+  const builder = new flatbuffers.Builder(1024);
+
+  const cols = _.map(coords, carr => {
+    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
+    NetEncoding.Float32Array.startFloat32Array(builder);
+    NetEncoding.Float32Array.addData(builder, cdv);
+    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
+
+    NetEncoding.Column.startColumn(builder);
+    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
+    NetEncoding.Column.addU(builder, floatArr);
+    return NetEncoding.Column.endColumn(builder);
+  });
+
+  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, nObs);
+  NetEncoding.Matrix.addNCols(builder, nVar);
+  NetEncoding.Matrix.addColumns(builder, columns);
+  const matrix = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(matrix);
+  return builder.asUint8Array();
+})();
+
 const aDataObsResponse = {
   var: [2, 4, 29],
   obs: _()
@@ -107,10 +206,10 @@ const aDataObsResponse = {
 };
 
 export {
-  aLayoutResponse as layoutObs,
+  aLayoutFBSResponse as layoutObs,
   aDataObsResponse as dataObs,
-  anAnnotationsVarResponse as annotationsVar,
-  anAnnotationsObsResponse as annotationsObs,
+  anAnnotationsVarFBSResponse as annotationsVar,
+  anAnnotationsObsFBSResponse as annotationsObs,
   aSchemaResponse as schema,
   aConfigResponse as config
 };

client/__tests__/util/stateManager/summarizeAnnotations.test.js

@@ -39,15 +39,21 @@ describe("summarizeAnnotations", () => {
           nameFloat32: {
             categorical: false,
             range: {
-              max: Number.NEGATIVE_INFINITY,
-              min: Number.POSITIVE_INFINITY
+              max: undefined,
+              min: undefined,
+              nan: 0,
+              ninf: 0,
+              pinf: 0
             }
           },
           nameInt32: {
             categorical: false,
             range: {
-              max: Number.NEGATIVE_INFINITY,
-              min: Number.POSITIVE_INFINITY
+              max: undefined,
+              min: undefined,
+              nan: 0,
+              ninf: 0,
+              pinf: 0
             }
           },
           nameCategorical: {
@@ -99,11 +105,11 @@ describe("summarizeAnnotations", () => {
           },
           nameFloat32: {
             categorical: false,
-            range: { min: 39.3, max: 39.3 }
+            range: { min: 39.3, max: 39.3, nan: 0, ninf: 0, pinf: 0 }
           },
           nameInt32: {
             categorical: false,
-            range: { min: 99, max: 99 }
+            range: { min: 99, max: 99, nan: 0, ninf: 0, pinf: 0 }
           },
           nameCategorical: {
             categorical: true,
@@ -172,11 +178,93 @@ describe("summarizeAnnotations", () => {
           },
           nameFloat32: {
             categorical: false,
-            range: { min: 0, max: 39.3 }
+            range: { min: 0, max: 39.3, nan: 0, ninf: 0, pinf: 0 }
           },
           nameInt32: {
             categorical: false,
-            range: { min: 99, max: 99 }
+            range: { min: 99, max: 99, nan: 0, ninf: 0, pinf: 0 }
+          },
+          nameCategorical: {
+            categorical: true,
+            categories: expect.arrayContaining([1, false, "0"]),
+            categoryCounts: new Map([[1, 1], [false, 1], ["0", 1]]),
+            numCategories: 3
+          }
+        },
+        var: {}
+      })
+    );
+  });
+
+  test("non-finite numbers", () => {
+    const obsAnnotations = [
+      {
+        __index__: 0,
+        name: "n0",
+        nameString: "hi",
+        nameBoolean: false,
+        nameFloat32: 39.3,
+        nameInt32: 99,
+        nameCategorical: 1
+      },
+      {
+        __index__: 1,
+        name: "n1",
+        nameString: "hi",
+        nameBoolean: true,
+        nameFloat32: Number.NEGATIVE_INFINITY,
+        nameInt32: 99,
+        nameCategorical: false
+      },
+      {
+        __index__: 2,
+        name: "n2",
+        nameString: "bye",
+        nameBoolean: true,
+        nameFloat32: Number.NaN,
+        nameInt32: 99,
+        nameCategorical: "0"
+      },
+      {
+        __index__: 3,
+        name: "n2",
+        nameString: "bye",
+        nameBoolean: true,
+        nameFloat32: Number.POSITIVE_INFINITY,
+        nameInt32: 99,
+        nameCategorical: "0"
+      }
+    ];
+    const varAnnotations = [];
+
+    const summary = summarizeAnnotations(
+      schema,
+      obsAnnotations,
+      varAnnotations
+    );
+
+    expect(summary).toMatchObject(
+      expect.objectContaining({
+        obs: {
+          nameString: {
+            categorical: true,
+            categories: expect.arrayContaining(["hi", "bye"]),
+            categoryCounts: new Map([["hi", 2], ["bye", 1]]),
+            numCategories: 2
+          },
+          nameBoolean: {
+            categorical: true,
+            categories: expect.arrayContaining([true, false]),
+            categoryCounts: new Map([[true, 2], [false, 1]]),
+            numCategories: 2
+          },
+          nameFloat32: {
+            categorical: false,
+            range: { min: 39.3, max: 39.3, nan: 1, ninf: 1, pinf: 1 }
+          },
+          nameInt32: {
+            categorical: false,
+            range: { min: 99, max: 99, nan: 0, ninf: 0, pinf: 0 }
           },
           nameCategorical: {
             categorical: true,

client/__tests__/util/stateManager/universe.test.js

@@ -30,7 +30,6 @@ describe("createUniverseFromRestV02Response", () => {
     create a universe from sample data nad validate its shape & contents
     */
     const { nObs, nVar } = REST.schema.schema.dataframe;
-
     const universe = Universe.createUniverseFromRestV02Response(
       REST.config,
       REST.schema,
@@ -66,56 +65,3 @@ describe("createUniverseFromRestV02Response", () => {
     expect(_.keys(universe.varNameToIndexMap)).toHaveLength(nVar);
   });
 });
-
-describe("convertExpressionRESTv02ToObject", () => {
-  /*
-  test convertExpressionRESTv02ToObject
-
-  convertExpressionRESTv02ToObject(
-    universe,
-    response) --> { geneName: Float32Array, geneName: Float32Array, ... }
-
-  reponse is a /data/obs response:
-  {
-    var: [ varIndices fetched ],
-    obs: [
-      [ obsIndex, evalue, ... ],
-      ...
-    ]
-  }
-  */
-  test("create from response data", () => {
-    const universe = Universe.createUniverseFromRestV02Response(
-      REST.config,
-      REST.schema,
-      REST.annotationsObs,
-      REST.annotationsVar,
-      REST.layoutObs
-    );
-    const expression = Universe.convertExpressionRESTv02ToObject(
-      universe,
-      REST.dataObs
-    );
-
-    /* Check that the expected keys are present */
-    const expectedGeneNames = _.map(
-      REST.dataObs.var,
-      v => REST.annotationsVar.data[v][5]
-    );
-    expect(Object.keys(expression)).toEqual(
-      expect.arrayContaining(expectedGeneNames)
-    );
-
-    const expectedExpressionValues = _.map(
-      _.unzip(REST.dataObs.obs),
-      a => new Float32Array(a)
-    );
-
-    _.forEach(REST.dataObs.var, (varIdx, idx) => {
-      const varName = universe.varAnnotations[varIdx].name;
-      expect(varName).toBeDefined();
-      expect(varIdx).toBe(universe.varNameToIndexMap[varName]);
-      expect(expression[varName]).toEqual(expectedExpressionValues[idx + 1]);
-    });
-  });
-});

client/__tests__/util/stateManager/world.test.js

@@ -168,10 +168,13 @@ describe("createObsDimensionMap", () => {
     */
 
     const { dimensionMap } = defaultBigBang();
-
+    const annotationNames = _.map(
+      REST.schema.schema.annotations.obs,
+      c => c.name
+    );
     const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
     expect(dimensionMap).toBeDefined();
-    REST.annotationsObs.names.forEach(name => {
+    annotationNames.forEach(name => {
       const dim = dimensionMap[obsAnnoDimensionName(name)];
       if (name === "name") {
         expect(dim).toBeUndefined();
@@ -184,11 +187,8 @@ describe("createObsDimensionMap", () => {
         }
       }
     });
-    expect(dimensionMap[layoutDimensionName("X")]).toBeInstanceOf(
-      Crossfilter.ScalarDimension
-    );
-    expect(dimensionMap[layoutDimensionName("Y")]).toBeInstanceOf(
-      Crossfilter.ScalarDimension
+    expect(dimensionMap[layoutDimensionName("XY")]).toBeInstanceOf(
+      Crossfilter.SpatialDimension
     );
   });
 });

client/__tests__/util/typedCrossfilter/typedCrossfilter.test.js

@@ -102,22 +102,20 @@ const someData = [
 ];
 
 function groupReduce(data, valueMap, valueReduce, valueInit) {
-  return _
-    .reduce(
-      data,
-      (acc, value) => {
-        const k = valueMap(value);
-        let r = _.find(acc, o => o.key === k);
-        if (!r) {
-          r = { key: k, value: valueInit() };
-          acc.push(r);
-        }
-        r.value = valueReduce(r.value, value);
-        return acc;
-      },
-      []
-    )
-    .sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
+  return _.reduce(
+    data,
+    (acc, value) => {
+      const k = valueMap(value);
+      let r = _.find(acc, o => o.key === k);
+      if (!r) {
+        r = { key: k, value: valueInit() };
+        acc.push(r);
+      }
+      r.value = valueReduce(r.value, value);
+      return acc;
+    },
+    []
+  ).sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0));
 }
 
 function groupCount(data, map) {
@@ -139,7 +137,11 @@ describe("typedCrossfilter", () => {
     expect(payments.size()).toEqual(someData.length);
     expect(payments.all()).toEqual(someData);
 
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
     expect(quantity).toBeDefined();
     expect(quantity.id()).toBeDefined();
 
@@ -150,10 +152,22 @@ describe("typedCrossfilter", () => {
 
   test("filterAll and filterNone", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     expect(quantity).toBeDefined();
     expect(tip).toBeDefined();
@@ -198,10 +212,22 @@ describe("typedCrossfilter", () => {
 
   test("filterExact", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     quantity.filterExact(1);
     expect(payments.countFiltered()).toEqual(
@@ -222,10 +248,22 @@ describe("typedCrossfilter", () => {
 
   test("filterRange", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     tip.filterRange([0, 91]);
     expect(payments.allFiltered()).toEqual(
@@ -251,10 +289,22 @@ describe("typedCrossfilter", () => {
 
   test("filterEnum", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     type.filterEnum(["tab", "cash"]);
     expect(payments.allFiltered()).toEqual(
@@ -274,15 +324,31 @@ describe("typedCrossfilter", () => {
 
   test("more than 32 dimensions", () => {
     expect(payments).toBeDefined();
-    const quantity = payments.dimension(r => r.quantity, Int32Array);
-    const tip = payments.dimension(r => r.tip, Float32Array);
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Float32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     // Create a bunch of fake dimensions to ensure we can handle > 32
     let dimMap = {};
     for (let i = 0; i < 65; i++) {
-      dimMap[i] = payments.dimension(r => Math.random(), Float32Array);
+      dimMap[i] = payments.dimension(
+        crossfilter.ScalarDimension,
+        r => Math.random(),
+        Float32Array
+      );
       expect(dimMap[i]).toBeDefined();
       expect(dimMap[i].id()).toBeDefined();
     }
@@ -304,10 +370,22 @@ describe("typedCrossfilter", () => {
   test("group, default mapping, default reducer, no filter", () => {
     expect(payments).toBeDefined();
 
-    var quantity = payments.dimension(r => r.quantity, Int32Array);
-    var tip = payments.dimension(r => r.tip, Int32Array);
-    var type = payments.dimension(r => r.type, "enum");
-    var total = payments.dimension(r => r.total, Int32Array);
+    const quantity = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.quantity,
+      Int32Array
+    );
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Int32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Int32Array
+    );
 
     _.each(
       {
@@ -331,9 +409,17 @@ describe("typedCrossfilter", () => {
     // custom mapping in groups only works for scalar types.  Enums do not
     // currently implement it.
 
-    const tip = payments.dimension(r => r.tip, Int32Array);
-    const totalX10 = payments.dimension(r => r.total * 10, Int32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Int32Array
+    );
+    const totalX10 = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total * 10,
+      Int32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     const paymentsByTip_A = tip.group();
     const paymentsByTip_B = tip.group(r => 10 * r);
@@ -370,8 +456,12 @@ describe("typedCrossfilter", () => {
   test("group, default map, custom reducer, no filters", () => {
     expect(payments).toBeDefined();
 
-    const total = payments.dimension(r => r.total, Float32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Float32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     const paymentsByTotal = total.group();
     const paymentsByType = type.group();
@@ -407,9 +497,17 @@ describe("typedCrossfilter", () => {
 
     expect(payments).toBeDefined();
 
-    const tip = payments.dimension(r => r.tip, Int32Array);
-    const total = payments.dimension(r => r.total, Int32Array);
-    const type = payments.dimension(r => r.type, "enum");
+    const tip = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.tip,
+      Int32Array
+    );
+    const total = payments.dimension(
+      crossfilter.ScalarDimension,
+      r => r.total,
+      Int32Array
+    );
+    const type = payments.dimension(crossfilter.EnumDimension, r => r.type);
 
     const paymentsByTip = tip.group();
     const paymentsByTotal = total.group();

client/configuration/webpack/webpack.config.prod.js

@@ -10,7 +10,7 @@ const nodeModules = path.resolve("node_modules");
 
 const babelOptions = require("../babel/babel.prod");
 
-const publicPath = "/";
+const publicPath = "";
 
 module.exports = {
   mode: "production",

client/index_template.html

@@ -24,7 +24,7 @@
 <script type="text/javascript">
 window.CELLXGENE = {};
 window.CELLXGENE.API = {
-  prefix: "{{ prefix | safe }}",
+  prefix: window.location.href + "api/",
   version: "v0.2/"
 };
 </script>

client/package.json

@@ -1,10 +1,11 @@
 {
   "name": "cellxgene",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
   "scripts": {
+    "backend-dev": "python3.6 -m venv cellxgene && source cellxgene/bin/activate && yes | pip uninstall cellxgene || true && pip install -e .. && cellxgene launch ",
     "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
     "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
     "clean": "rimraf build",
@@ -19,10 +20,9 @@
   "eslintConfig": {
     "extends": "./configuration/eslint/eslint.js"
   },
-  "nyc": {
-    "sourceMap": false,
-    "instrument": false
-  },
+  "eslintIgnore": [
+    "src/util/stateManager/matrix_generated.js"
+  ],
   "resolutions": {
     "eslint-scope": "3.7.1"
   },
@@ -33,6 +33,7 @@
     "canvas-fit": "^1.5.0",
     "d3": "^4.10.0",
     "d3-scale-chromatic": "^1.3.0",
+    "flatbuffers": "^1.10.2",
     "font-color-contrast": "^1.0.3",
     "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
@@ -75,30 +76,28 @@
     "babel-eslint": "^10.0.1",
     "babel-jest": "^23.6.0",
     "babel-loader": "^8.0.0",
-    "babel-plugin-istanbul": "^5.1.0",
     "babel-preset-modern-browsers": "^12.0.0",
-    "chalk": "^2.4.1",
-    "connect-history-api-fallback": "^1.3.0",
+    "chalk": "^2.4.2",
+    "connect-history-api-fallback": "^1.6.0",
     "copy-webpack-plugin": "^4.6.0",
     "css-loader": "^1.0.1",
-    "eslint": "^5.8.0",
+    "eslint": "^5.13.0",
     "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^3.1.0",
-    "eslint-loader": "^2.1.1",
+    "eslint-config-prettier": "^4.0.0",
+    "eslint-loader": "^2.1.2",
     "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.14.0",
-    "eslint-plugin-jest": "^21.27.2",
-    "eslint-plugin-jsx-a11y": "^6.1.1",
-    "eslint-plugin-react": "^7.11.1",
+    "eslint-plugin-import": "^2.16.0",
+    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-jsx-a11y": "^6.2.1",
+    "eslint-plugin-react": "^7.12.4",
     "express": "^4.14.0",
     "file-loader": "^2.0.0",
     "html-webpack-inline-source-plugin": "0.0.10",
     "html-webpack-plugin": "^3.2.0",
-    "jest": "^23.5.0",
+    "jest": "^24.1.0",
     "json-loader": "^0.5.4",
     "mini-css-extract-plugin": "^0.4.1",
-    "nyc": "^13.0.1",
-    "rimraf": "^2.5.4",
+    "rimraf": "^2.6.3",
     "serve-favicon": "^2.3.0",
     "style-loader": "^0.23.1",
     "sw-precache-webpack-plugin": "^0.11.5",
@@ -111,7 +110,10 @@
     "testMatch": [
       "**/__tests__/**/?(*.)(spec|test).js?(x)"
     ],
-    "testURL": "http://localhost/"
+    "testURL": "http://localhost/",
+    "setupFiles": [
+      "./__tests__/setupMissingGlobals.js"
+    ]
   },
   "babel": {
     "env": {

client/src/actions/index.js

@@ -5,6 +5,7 @@ import { Universe, kvCache } from "../util/stateManager";
 import {
   catchErrorsWrap,
   doJsonRequest,
+  doBinaryRequest,
   rangeEncodeIndices,
   dispatchNetworkErrorMessageToUser
 } from "../util/actionHelpers";
@@ -13,24 +14,28 @@ import {
 Bootstrap application with the initial data loading.
   * /config - application configuration
   * /schema - schema of dataframe
-  * /annotations/obs - all metadata annotation
+  * /annotations - all metadata annotation
+  * /layout - all default layout
 */
 const doInitialDataLoad = () =>
   catchErrorsWrap(async dispatch => {
     dispatch({ type: "initial data load start" });
 
     try {
-      const requests = _([
-        "config",
-        "schema",
+      const requestJson = _(["config", "schema"])
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doJsonRequest(url))
+        .value();
+      const requestBinary = _([
         "annotations/obs",
         "annotations/var?annotation-name=name",
         "layout/obs"
       ])
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doJsonRequest(url))
+        .map(url => doBinaryRequest(url))
         .value();
-      const results = await Promise.all(requests);
+
+      const results = await Promise.all(_.concat(requestJson, requestBinary));
 
       /* set config defaults */
       const config = { ...globals.configDefaults, ...results[0].config };
@@ -87,6 +92,38 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  /* helper for this function only */
+  const fetchData = async geneNames => {
+    const res = await fetch(
+      `${globals.API.prefix}${globals.API.version}data/var`,
+      {
+        method: "PUT",
+        body: JSON.stringify({
+          filter: {
+            var: {
+              annotation_value: [{ name: "name", values: geneNames }]
+            }
+          }
+        }),
+        headers: new Headers({
+          accept: "application/octet-stream",
+          "Content-Type": "application/json"
+        })
+      }
+    );
+
+    if (
+      !res.ok ||
+      res.headers.get("Content-Type") !== "application/octet-stream"
+    ) {
+      // WILL throw
+      return dispatchExpressionErrors(dispatch, res);
+    }
+
+    const data = await res.arrayBuffer();
+    return Universe.convertDataFBStoObject(universe, data);
+  };
+
   const state = getState();
   const { universe } = state.controls;
   /* preload data already in cache */
@@ -108,35 +145,10 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
   /* Fetch data for any genes not in cache */
   if (genesToFetch.length) {
     try {
-      // XXX: TODO - this could be using /data/var rather than /data/obs,
-      // as that would simplify the transformation in convertExpressionRESTv02ToObject
-      const res = await fetch(
-        `${globals.API.prefix}${globals.API.version}data/obs`,
-        {
-          method: "PUT",
-          body: JSON.stringify({
-            filter: {
-              var: {
-                annotation_value: [{ name: "name", values: genesToFetch }]
-              }
-            }
-          }),
-          headers: new Headers({
-            accept: "application/json",
-            "Content-Type": "application/json"
-          })
-        }
-      );
-
-      if (!res.ok || res.headers.get("Content-Type") !== "application/json") {
-        // WILL throw
-        return dispatchExpressionErrors(dispatch, res);
-      }
-
-      const data = await res.json();
+      const newExpressionData = await fetchData(genesToFetch);
       expressionData = {
         ...expressionData,
-        ...Universe.convertExpressionRESTv02ToObject(universe, data)
+        ...newExpressionData
       };
     } catch (error) {
       dispatch({ type: "expression load error", error });

client/src/components/brushableHistogram/index.js

@@ -13,6 +13,7 @@ import memoize from "memoize-one";
 import { kvCache } from "../../util/stateManager";
 import * as globals from "../../globals";
 import actions from "../../actions";
+import finiteExtent from "../../util/finiteExtent";
 
 @connect(state => ({
   world: state.controls.world,
@@ -26,7 +27,7 @@ import actions from "../../actions";
   obsAnnotations: _.get(state.controls.world, "obsAnnotations", null)
 }))
 class HistogramBrush extends React.Component {
-  calcHistogramCache = memoize((obsAnnotations, field, ranges) => {
+  calcHistogramCache = memoize((obsAnnotations, field, rangeMin, rangeMax) => {
     const { world } = this.props;
     const histogramCache = {};
 
@@ -40,7 +41,7 @@ class HistogramBrush extends React.Component {
 
       histogramCache.x = d3
         .scaleLinear()
-        .domain([ranges.min, ranges.max])
+        .domain([rangeMin, rangeMax])
         .range([0, this.width]);
 
       histogramCache.bins = d3
@@ -56,7 +57,7 @@ class HistogramBrush extends React.Component {
       histogramCache.x = d3
         .scaleLinear()
         .domain(
-          d3.extent(varValues)
+          finiteExtent(varValues)
         ) /* replace this if we have ranges for genes back from server like we do for annotations on cells */
         .range([0, this.width]);
 
@@ -130,7 +131,8 @@ class HistogramBrush extends React.Component {
     const histogramCache = this.calcHistogramCache(
       obsAnnotations,
       field,
-      ranges
+      ranges.min,
+      ranges.max
     );
 
     const { x, y, bins, numValues } = histogramCache;
@@ -159,7 +161,13 @@ class HistogramBrush extends React.Component {
   }
 
   removeHistogram() {
-    const { dispatch, field, colorAccessor } = this.props;
+    const {
+      dispatch,
+      field,
+      colorAccessor,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor
+    } = this.props;
     dispatch({
       type: "clear user defined gene",
       data: field
@@ -169,6 +177,18 @@ class HistogramBrush extends React.Component {
         type: "reset colorscale"
       });
     }
+    if (field === scatterplotXXaccessor) {
+      dispatch({
+        type: "set scatterplot x",
+        data: null
+      });
+    }
+    if (field === scatterplotYYaccessor) {
+      dispatch({
+        type: "set scatterplot y",
+        data: null
+      });
+    }
   }
 
   handleSetGeneAsScatterplotX() {
@@ -265,7 +285,7 @@ class HistogramBrush extends React.Component {
         }}
       >
         <div style={{ display: "flex", justifyContent: "flex-end" }}>
-          {isDiffExp ? (
+          {isDiffExp || isUserDefined ? (
             <span>
               <span
                 style={{ marginRight: 7 }}

client/src/components/framework/toasters.js

@@ -13,6 +13,13 @@ A "user" error - eg, bad input
 export const postUserErrorToast = message =>
   ErrorToastTopCenter.show({ message, intent: Intent.WARNING });
 
+/*
+A toast the user must dismiss manually, because they need to act on its information,
+ie., 8 bulk add genes out of 40 were bad. Manually see which ones and fix.
+*/
+export const keepAroundErrorToast = message =>
+  ErrorToastTopCenter.show({ message, timeout: 0, intent: Intent.WARNING });
+
 /*
 a hard network error
 */

client/src/components/geneExpression/cellSetButtons.js

@@ -1,7 +1,7 @@
 // jshint esversion: 6
 import React from "react";
 import _ from "lodash";
-import { Button, Tooltip } from "@blueprintjs/core";
+import { AnchorButton, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
 
 @connect()
@@ -33,9 +33,10 @@ class CellSetButton extends React.Component {
         content="Save current selection for differential expression computation"
         position="top"
       >
-        <Button
+        <AnchorButton
           style={{ marginRight: 10 }}
           type="button"
+          disabled={differential.diffExp}
           onClick={this.set.bind(this)}
         >
           {eitherCellSetOneOrTwo}
@@ -43,7 +44,7 @@ class CellSetButton extends React.Component {
           {differential[cellListName]
             ? `${differential[cellListName].length} cells`
             : "0 cells"}
-        </Button>
+        </AnchorButton>
       </Tooltip>
     );
   }

client/src/components/geneExpression/expressionButtons.js

@@ -68,6 +68,7 @@ class Expression extends React.Component {
               style={{ marginTop: 10 }}
               disabled={!haveBothCellSets}
               intent="primary"
+              loading={differential.loading}
               fill
               type="button"
               onClick={this.computeDiffExp.bind(this)}

client/src/components/geneExpression/index.js

@@ -3,17 +3,26 @@
 
 import React from "react";
 import _ from "lodash";
-import * as d3 from "d3";
 import fuzzysort from "fuzzysort";
 
 import { connect } from "react-redux";
-import { MenuItem } from "@blueprintjs/core";
+import {
+  MenuItem,
+  Button,
+  FormGroup,
+  InputGroup,
+  ControlGroup
+} from "@blueprintjs/core";
 import { Suggest } from "@blueprintjs/select";
 import HistogramBrush from "../brushableHistogram";
 import * as globals from "../../globals";
 import actions from "../../actions";
-import { postUserErrorToast } from "../framework/toasters";
+import {
+  postUserErrorToast,
+  keepAroundErrorToast
+} from "../framework/toasters";
 import ExpressionButtons from "./expressionButtons";
+import finiteExtent from "../../util/finiteExtent";
 
 const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
   if (!modifiers.matchesPredicate) {
@@ -59,6 +68,7 @@ const filterGenes = (query, genes) => {
     metadata,
     initializeRanges,
     userDefinedGenes: state.controls.userDefinedGenes,
+    userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
     world: state.controls.world,
     colorAccessor: state.controls.colorAccessor,
     allGeneNames: state.controls.allGeneNames,
@@ -66,6 +76,14 @@ const filterGenes = (query, genes) => {
   };
 })
 class GeneExpression extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      bulkAdd: "",
+      tab: "autosuggest"
+    };
+  }
+
   handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
     const gene = g.target;
@@ -86,8 +104,46 @@ class GeneExpression extends React.Component {
     }
   }
 
+  handleBulkAddClick() {
+    const { world, dispatch, userDefinedGenes } = this.props;
+    const { bulkAdd } = this.state;
+
+    /*
+      test:
+      Apod,,, Cd74,,    ,,,    Foo,    Bar-2,,
+    */
+    if (bulkAdd !== "") {
+      const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");
+
+      genes.forEach(gene => {
+        if (userDefinedGenes.indexOf(gene) !== -1) {
+          keepAroundErrorToast("That gene already exists");
+        } else if (!_.find(world.varAnnotations, { name: gene })) {
+          keepAroundErrorToast(
+            `${gene} doesn't appear to be a valid gene name.`
+          );
+        } else {
+          dispatch(actions.requestUserDefinedGene(gene));
+          dispatch({
+            type: "user defined gene",
+            data: gene
+          });
+        }
+      });
+    }
+
+    this.setState({ bulkAdd: "" });
+  }
+
   render() {
-    const { world, userDefinedGenes, differential } = this.props;
+    const {
+      world,
+      userDefinedGenes,
+      userDefinedGenesLoading,
+      differential
+    } = this.props;
+
+    const { tab, bulkAdd } = this.state;
 
     return (
       <div>
@@ -105,31 +161,105 @@ class GeneExpression extends React.Component {
             Selected Genes
           </p>
           <div
-            style={{ padding: globals.leftSidebarSectionPadding }}
-            className="bp3-control-group"
+            style={{
+              padding: globals.leftSidebarSectionPadding
+            }}
           >
-            <Suggest
-              closeOnSelect
-              openOnKeyDown
-              resetOnSelect
-              noResults={<MenuItem disabled text="No matching genes." />}
-              onItemSelect={g => {
-                /* this happens on 'enter' */
-                this.handleClick(g);
+            <Button
+              active={tab === "autosuggest"}
+              style={{ marginRight: 5 }}
+              minimal
+              small
+              onClick={() => {
+                this.setState({ tab: "autosuggest" });
               }}
-              inputValueRenderer={g => {
-                return "";
+            >
+              Autosuggest
+            </Button>
+            <Button
+              active={tab === "bulkadd"}
+              minimal
+              small
+              onClick={() => {
+                this.setState({ tab: "bulkadd" });
               }}
-              itemListPredicate={filterGenes}
-              itemRenderer={renderGene.bind(this)}
-              items={
-                world && world.varAnnotations
-                  ? world.varAnnotations
-                  : [{ name: "No genes", n_counts: "" }]
-              }
-              popoverProps={{ minimal: true }}
-            />
+            >
+              Bulk add genes
+            </Button>
           </div>
+
+          {tab === "autosuggest" ? (
+            <ControlGroup
+              style={{
+                paddingLeft: globals.leftSidebarSectionPadding,
+                paddingBottom: globals.leftSidebarSectionPadding
+              }}
+            >
+              <Suggest
+                closeOnSelect
+                openOnKeyDown
+                resetOnSelect
+                itemDisabled={
+                  userDefinedGenesLoading ? () => true : () => false
+                }
+                noResults={<MenuItem disabled text="No matching genes." />}
+                onItemSelect={g => {
+                  /* this happens on 'enter' */
+                  this.handleClick(g);
+                }}
+                inputValueRenderer={g => {
+                  return "";
+                }}
+                itemListPredicate={filterGenes}
+                itemRenderer={renderGene.bind(this)}
+                items={
+                  world && world.varAnnotations
+                    ? world.varAnnotations
+                    : [{ name: "No genes" }]
+                }
+                popoverProps={{ minimal: true }}
+              />
+              <Button
+                className="bp3-button bp3-intent-primary"
+                loading={userDefinedGenesLoading}
+              >
+                Add
+              </Button>
+            </ControlGroup>
+          ) : null}
+          {tab === "bulkadd" ? (
+            <div style={{ paddingLeft: globals.leftSidebarSectionPadding }}>
+              <form
+                onSubmit={e => {
+                  e.preventDefault();
+                  this.handleBulkAddClick();
+                }}
+              >
+                <FormGroup
+                  helperText="Add a list of genes (comma delimited)"
+                  labelFor="text-input-bulk-add"
+                >
+                  <ControlGroup>
+                    <InputGroup
+                      onChange={e => {
+                        this.setState({ bulkAdd: e.target.value });
+                      }}
+                      id="text-input-bulk-add"
+                      placeholder="Apod, Cd74, ..."
+                      value={bulkAdd}
+                    />
+                    <Button
+                      intent="primary"
+                      onClick={this.handleBulkAddClick.bind(this)}
+                      loading={userDefinedGenesLoading}
+                    >
+                      Add
+                    </Button>
+                  </ControlGroup>
+                </FormGroup>
+              </form>
+            </div>
+          ) : null}
           {world && userDefinedGenes.length > 0
             ? _.map(userDefinedGenes, (geneName, index) => {
                 const values = world.varDataCache[geneName];
@@ -141,7 +271,7 @@ class GeneExpression extends React.Component {
                     key={geneName}
                     field={geneName}
                     zebra={index % 2 === 0}
-                    ranges={d3.extent(values)}
+                    ranges={finiteExtent(values)}
                     isUserDefined
                   />
                 );
@@ -171,7 +301,7 @@ class GeneExpression extends React.Component {
                     key={name}
                     field={name}
                     zebra={index % 2 === 0}
-                    ranges={d3.extent(values)}
+                    ranges={finiteExtent(values)}
                     isDiffExp
                     logFoldChange={value[1]}
                     pval={value[2]}

client/src/components/graph/graph.js

@@ -22,7 +22,8 @@ import scaleLinear from "../../util/scaleLinear";
   responsive: state.responsive,
   colorRGB: _.get(state.controls, "colorRGB", null),
   opacityForDeselectedCells: state.controls.opacityForDeselectedCells,
-  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null),
+  resettingInterface: state.controls.resettingInterface
 }))
 class Graph extends React.Component {
   constructor(props) {
@@ -39,7 +40,7 @@ class Graph extends React.Component {
     this.state = {
       svg: null,
       brush: null,
-      mode: "brush"
+      mode: "lasso"
     };
   }
 
@@ -201,7 +202,9 @@ class Graph extends React.Component {
         this.handleBrushSelectAction.bind(this),
         this.handleBrushDeselectAction.bind(this),
         responsive,
-        this.graphPaddingRight
+        this.graphPaddingRight,
+        this.handleLassoStart.bind(this),
+        this.handleLassoEnd.bind(this)
       );
       this.setState({ svg: newSvg, brush });
     }
@@ -250,54 +253,54 @@ class Graph extends React.Component {
     });
   }
 
+  invertPoint(pin) {
+    const { responsive } = this.props;
+    const { regl, camera, offset } = this.state;
+
+    const gl = regl._gl;
+
+    // get aspect ratio
+    const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
+
+    // compute inverse view matrix
+    const inverse = mat4.invert([], camera.view());
+
+    // transform screen coordinates -> cell coordinates
+    const x = (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
+    const y = 2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
+    const pout = [
+      x * inverse[14] * aspect + inverse[12],
+      y * inverse[14] + inverse[13]
+    ];
+    return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
+  }
+
   handleBrushSelectAction() {
     /*
-    This conditional handles procedural brush deselect. Brush emits
-    an event on procedural deselect because it is move: null
+      This conditional handles procedural brush deselect. Brush emits
+      an event on procedural deselect because it is move: null
     */
-
-    const { camera, offset } = this.state;
-    const { dispatch, responsive } = this.props;
-
-    if (d3.event.sourceEvent !== null) {
-      /*
-      No idea why d3 event scope works like this
-      but apparently
-      it does
-      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
-    */
-      const s = d3.event.selection;
-      const gl = this.state.regl._gl;
-      /*
+    /*
       event describing brush position:
       @-------|
       |       |
       |       |
       |-------@
     */
+    /*
+      No idea why d3 event scope works like this
+      but apparently
+      it does
+      https://bl.ocks.org/EfratVil/0e542f5fc426065dd1d4b6daaa345a9f
+    */
+    const { dispatch } = this.props;
 
-      // get aspect ratio
-      const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
-
-      // compute inverse view matrix
-      const inverse = mat4.invert([], camera.view());
-
-      // transform screen coordinates -> cell coordinates
-      const invert = pin => {
-        const x =
-          (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
-        const y =
-          2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
-        const pout = [
-          x * inverse[14] * aspect + inverse[12],
-          y * inverse[14] + inverse[13]
-        ];
-        return [(pout[0] + 1) / 2 + offset[0], (pout[1] + 1) / 2 + offset[1]];
-      };
+    if (d3.event.sourceEvent !== null) {
+      const s = d3.event.selection;
 
       const brushCoords = {
-        northwest: invert([s[0][0], s[0][1]]),
-        southeast: invert([s[1][0], s[1][1]])
+        northwest: this.invertPoint([s[0][0], s[0][1]]),
+        southeast: this.invertPoint([s[1][0], s[1][1]])
       };
 
       dispatch({
@@ -329,6 +332,25 @@ class Graph extends React.Component {
     }
   }
 
+  handleLassoStart() {
+    const { dispatch } = this.props;
+    // reset selected points when starting a new polygon
+    // making it easier for the user to make the next selection
+    dispatch({
+      type: "lasso started"
+    });
+  }
+
+  // when a lasso is completed, filter to the points within the lasso polygon
+  handleLassoEnd(polygon) {
+    const { dispatch } = this.props;
+
+    dispatch({
+      type: "lasso selection",
+      polygon: polygon.map(xy => this.invertPoint(xy)) // transform the polygon
+    });
+  }
+
   handleOpacityRangeChange(e) {
     const { dispatch } = this.props;
     dispatch({
@@ -339,11 +361,20 @@ class Graph extends React.Component {
 
   resetInterface() {
     const { dispatch } = this.props;
+    dispatch({
+      type: "interface reset started"
+    });
     dispatch(actions.resetInterface());
   }
 
   render() {
-    const { dispatch, responsive, crossfilter } = this.props;
+    const {
+      dispatch,
+      responsive,
+      crossfilter,
+      resettingInterface
+    } = this.props;
+
     const { mode } = this.state;
     return (
       <div id="graphWrapper">
@@ -392,6 +423,7 @@ class Graph extends React.Component {
                   /* world && universe ? worldEqUniverse(world, universe) : false */
                 }
                 type="button"
+                loading={resettingInterface}
                 intent="warning"
                 style={{ marginRight: 10 }}
                 onClick={this.resetInterface.bind(this)}
@@ -401,13 +433,18 @@ class Graph extends React.Component {
             </Tooltip>
             <div>
               <div className="bp3-button-group">
-                <Tooltip content="Lasso cells" position="left">
+                <Tooltip content="Lasso selection" position="left">
                   <Button
-                    className="bp3-button bp3-icon-select"
                     type="button"
-                    active={mode === "brush"}
+                    className="bp3-button bp3-icon-polygon-filter"
+                    active={mode === "lasso"}
                     onClick={() => {
-                      this.setState({ mode: "brush" });
+                      this.handleBrushDeselectAction();
+                      // this.restartReglLoop();
+                      this.setState({ mode: "lasso" });
+                    }}
+                    style={{
+                      cursor: "pointer"
                     }}
                   />
                 </Tooltip>
@@ -440,7 +477,7 @@ class Graph extends React.Component {
         >
           <div
             style={{
-              display: mode === "brush" ? "inherit" : "none"
+              display: mode === "lasso" ? "inherit" : "none"
             }}
             id="graphAttachPoint"
           />

client/src/components/graph/setupLasso.js

@@ -0,0 +1,127 @@
+// https://bl.ocks.org/pbeshai/8008075f9ce771ee8be39e8c38907570
+
+import * as d3 from "d3";
+
+const Lasso = () => {
+  const dispatch = d3.dispatch("start", "end");
+
+  const polygonToPath = polygon =>
+    `M${polygon.map(d => d.join(",")).join("L")}`;
+
+  const distance = (pt1, pt2) =>
+    Math.sqrt((pt2[0] - pt1[0]) ** 2 + (pt2[1] - pt1[1]) ** 2);
+
+  // distance last point has to be to first point before it auto closes when mouse is released
+  const closeDistance = 75;
+
+  const lasso = svg => {
+    let lassoPolygon;
+    let lassoPath;
+    let closePath;
+
+    const handleDragStart = () => {
+      lassoPolygon = [d3.mouse(svg.node())]; // current x y of mouse within element
+
+      if (lassoPath) {
+        lassoPath.remove();
+      }
+
+      lassoPath = g
+        .append("path")
+        .attr("fill", "#0bb")
+        .attr("fill-opacity", 0.1)
+        .attr("stroke", "#0bb")
+        .attr("stroke-dasharray", "3, 3");
+
+      closePath = g
+        .append("line")
+        .attr("x2", lassoPolygon[0][0])
+        .attr("y2", lassoPolygon[0][1])
+        .attr("stroke", "#0bb")
+        .attr("stroke-dasharray", "3, 3")
+        .attr("opacity", 0);
+
+      dispatch.call("start", lasso, lassoPolygon);
+    };
+
+    const handleDrag = () => {
+      const point = d3.mouse(svg.node());
+      lassoPolygon.push(point);
+      lassoPath.attr("d", polygonToPath(lassoPolygon));
+
+      // indicate if we are within closing distance
+      if (
+        distance(lassoPolygon[0], lassoPolygon[lassoPolygon.length - 1]) <
+        closeDistance
+      ) {
+        closePath
+          .attr("x1", point[0])
+          .attr("y1", point[1])
+          .attr("opacity", 1);
+      } else {
+        closePath.attr("opacity", 0);
+      }
+    };
+
+    const handleDragEnd = () => {
+      // remove the close path
+      closePath.remove();
+      closePath = null;
+
+      // succesfully closed
+      if (
+        distance(lassoPolygon[0], lassoPolygon[lassoPolygon.length - 1]) <
+        closeDistance
+      ) {
+        lassoPath.attr("d", `${polygonToPath(lassoPolygon)}Z`);
+        dispatch.call("end", lasso, lassoPolygon);
+
+        // otherwise cancel
+      } else {
+        lassoPath.remove();
+        lassoPath = null;
+        lassoPolygon = null;
+      }
+    };
+
+    // append a <g> with a rect
+    const g = svg.append("g").attr("class", "lasso-group");
+    const bbox = svg.node().getBoundingClientRect();
+    const area = g
+      .append("rect")
+      .attr("width", bbox.width)
+      .attr("height", bbox.height)
+      .attr("fill", "tomato")
+      .attr("opacity", 0);
+
+    const drag = d3
+      .drag()
+      .on("start", handleDragStart)
+      .on("drag", handleDrag)
+      .on("end", handleDragEnd);
+
+    area.call(drag);
+
+    lasso.reset = () => {
+      if (lassoPath) {
+        lassoPath.remove();
+        lassoPath = null;
+      }
+
+      lassoPolygon = null;
+      if (closePath) {
+        closePath.remove();
+        closePath = null;
+      }
+    };
+  };
+
+  lasso.on = (type, callback) => {
+    dispatch.on(type, callback);
+    return lasso;
+  };
+
+  return lasso;
+};
+
+export default Lasso;

client/src/components/graph/setupSVGandBrush.js

@@ -1,6 +1,7 @@
 // jshint esversion: 6
 import * as d3 from "d3";
 import styles from "./graph.css";
+import Lasso from "./setupLasso";
 
 /******************************************
 *******************************************
@@ -12,7 +13,9 @@ export default (
   handleBrushSelectAction,
   handleBrushDeselectAction,
   responsive,
-  graphPaddingRight
+  graphPaddingRight,
+  handleLassoStart,
+  handleLassoEnd
 ) => {
   const svg = d3
     .select("#graphAttachPoint")
@@ -32,9 +35,16 @@ export default (
     .attr("class", "graph_brush")
     .call(brush);
 
+  const lassoInstance = Lasso()
+    .on("end", handleLassoEnd)
+    .on("start", handleLassoStart);
+
+  const lasso = svg.call(lassoInstance);
+
   return {
     svg,
     brushContainer,
-    brush
+    brush,
+    lasso
   };
 };

client/src/components/scatterplot/scatterplot.js

@@ -20,6 +20,7 @@ import scaleLinear from "../../util/scaleLinear";
 
 import { margin, width, height } from "./util";
 import { kvCache } from "../../util/stateManager";
+import finiteExtent from "../../util/finiteExtent";
 
 @connect(state => {
   const {
@@ -227,11 +228,11 @@ class Scatterplot extends React.Component {
   static setupScales(expressionX, expressionY) {
     const xScale = d3
       .scaleLinear()
-      .domain(d3.extent(expressionX))
+      .domain(finiteExtent(expressionX))
       .range([0, width]);
     const yScale = d3
       .scaleLinear()
-      .domain(d3.extent(expressionY))
+      .domain(finiteExtent(expressionY))
       .range([height, 0]);
 
     return {

client/src/globals.js

@@ -1,3 +1,5 @@
+import { Colors } from "@blueprintjs/core";
+
 // jshint esversion: 6
 /* these will be either (preferably) specified or inferred */
 export const categories = [
@@ -44,8 +46,8 @@ export const configDefaults = {
 };
 
 /* colors */
-export const blue = "#4a90e2";
-export const hcaBlue = "#1c7cc7";
+export const blue = Colors.BLUE3;
+export const linkBlue = Colors.BLUE5;
 export const lightestGrey = "rgb(249,249,249)";
 export const lighterGrey = "rgb(245,245,245)";
 export const lightGrey = "rgb(211,211,211)";
@@ -57,6 +59,7 @@ export const brightBlue = "#4a90e2";
 export const brightGreen = "#A2D729";
 export const darkGreen = "#448C4D";
 
+export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";
 
 /* typography constants */

client/src/middleware/updateCellColors.js

@@ -1,15 +1,10 @@
 // jshint esversion: 6
 import _ from "lodash";
 import * as d3 from "d3";
-import {
-  interpolateViridis,
-  interpolateSpectral,
-  interpolateRainbow,
-  interpolateBlues,
-  interpolateCool
-} from "d3-scale-chromatic";
+import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
 import * as globals from "../globals";
 import parseRGB from "../util/parseRGB";
+import finiteExtent from "../util/finiteExtent";
 
 /*
   https://medium.com/@jacobp100/you-arent-using-redux-middleware-enough-94ffe991e6
@@ -54,62 +49,89 @@ const updateCellColorsMiddleware = store => next => action => {
 
   const { obsAnnotations } = s.controls.world;
   let colorScale;
-  const colorsByName = new Array(obsAnnotations.length);
   const colorsByRGB = new Array(obsAnnotations.length);
 
   /*
   in plain language...
    (a) once the cells have loaded.
    (b) each time a user changes a color control we need to update cellsMetadata colors
-  This is available to all the draw functions as world.colorName[index] or world.colorRGB[index]
+  This is available to all the draw functions as controls.colorRGB[index]
   */
 
   if (action.type === "color by categorical metadata") {
-    const categories = _.filter(s.controls.world.schema.annotations.obs, {
+    const { categories } = _.filter(s.controls.world.schema.annotations.obs, {
       name: action.colorAccessor
-    })[0].categories;
+    })[0];
 
     colorScale = d3
       .scaleSequential(interpolateRainbow)
       .domain([0, categories.length]);
 
-    for (let i = 0; i < obsAnnotations.length; i += 1) {
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = _.transform(categories, (acc, cat, idx) => {
+      acc[cat] = parseRGB(colorScale(idx));
+    });
+
+    const key = action.colorAccessor;
+    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
       const obs = obsAnnotations[i];
-      const c = colorScale(categories.indexOf(obs[action.colorAccessor]));
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+      const cat = obs[key];
+      colorsByRGB[i] = colors[cat];
     }
   }
 
   if (action.type === "color by continuous metadata") {
+    const colorBins = 100;
+    const [min, max] = [0, action.rangeMaxForColorAccessor];
     colorScale = d3
-      .scaleLinear()
-      .domain([0, action.rangeMaxForColorAccessor])
-      .range([1, 0]);
+      .scaleQuantile()
+      .domain([min, max])
+      .range(_.range(colorBins - 1, -1, -1));
 
-    for (let i = 0; i < obsAnnotations.length; i += 1) {
-      const obs = obsAnnotations[i];
-      const c = interpolateCool(colorScale(obs[action.colorAccessor]));
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = new Array(colorBins);
+    for (let i = 0; i < colorBins; i += 1) {
+      colors[i] = parseRGB(interpolateCool(i / colorBins));
+    }
+
+    const key = action.colorAccessor;
+    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
+    for (let i = 0, len = obsAnnotations.length; i < len; i += 1) {
+      const val = obsAnnotations[i][key];
+      if (Number.isFinite(val)) {
+        const c = colorScale(val);
+        colorsByRGB[i] = colors[c];
+      } else {
+        colorsByRGB[i] = nonFiniteColor;
+      }
     }
   }
 
   if (action.type === "color by expression") {
     const { gene, data } = action;
     const expression = data[gene]; // Float32Array
+    const colorBins = 100;
+    const [min, max] = finiteExtent(expression);
     colorScale = d3
-      .scaleLinear()
-      .domain([_.min(expression), _.max(expression)])
-      .range([
-        1,
-        0
-      ]); /* invert viridis... probably pass this scale through to others */
+      .scaleQuantile()
+      .domain([min, max])
+      .range(_.range(colorBins - 1, -1, -1));
+
+    /* pre-create colors - much faster than doing it for each obs */
+    const colors = new Array(colorBins);
+    for (let i = 0; i < colorBins; i += 1) {
+      colors[i] = parseRGB(interpolateCool(i / colorBins));
+    }
+    const nonFiniteColor = parseRGB(globals.nonFiniteCellColor);
 
     for (let i = 0, len = expression.length; i < len; i += 1) {
-      const c = interpolateCool(colorScale(expression[i]));
-      colorsByName[i] = c;
-      colorsByRGB[i] = parseRGB(c);
+      const e = expression[i];
+      if (Number.isFinite(e)) {
+        const c = colorScale(e);
+        colorsByRGB[i] = colors[c];
+      } else {
+        colorsByRGB[i] = nonFiniteColor;
+      }
     }
   }
 
@@ -117,7 +139,7 @@ const updateCellColorsMiddleware = store => next => action => {
   append the result of all the filters to the action the user just triggered
   */
   const modifiedAction = Object.assign({}, action, {
-    colors: { name: colorsByName, rgb: colorsByRGB },
+    colors: { rgb: colorsByRGB },
     colorScale
   });
 

client/src/reducers/controls.js

@@ -1,6 +1,8 @@
 // jshint esversion: 6
 
 import _ from "lodash";
+import { polygonContains } from "d3";
+
 import { World, kvCache, WorldUtil } from "../util/stateManager";
 import parseRGB from "../util/parseRGB";
 import Crossfilter from "../util/typedCrossfilter";
@@ -112,16 +114,17 @@ const Controls = (
 
     // all of the data + selection state
     world: null,
-    colorName: null,
     colorRGB: null,
     categoricalSelectionState: null,
     crossfilter: null,
     dimensionMap: null,
     userDefinedGenes: [],
+    userDefinedGenesLoading: false,
     diffexpGenes: [],
 
     colorAccessor: null,
     colorScale: null,
+    resettingInterface: false,
 
     opacityForDeselectedCells: 0.2,
     graphBrushSelection: null,
@@ -167,8 +170,9 @@ const Controls = (
       /* first light - create world & other data-driven defaults */
       const { universe } = action;
       const world = World.createWorldFromEntireUniverse(universe);
-      const colorName = new Array(universe.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
+      const colorRGB = new Array(universe.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
       const categoricalSelectionState = createCategoricalSelectionState(
         state,
         world
@@ -223,12 +227,12 @@ const Controls = (
         error: null,
         universe,
         world,
-        colorName,
         colorRGB,
         categoricalSelectionState,
         crossfilter,
         dimensionMap,
-        colorAccessor: null
+        colorAccessor: null,
+        resettingInterface: false
       };
     }
     case "set World to current selection": {
@@ -240,8 +244,9 @@ const Controls = (
         action.world,
         action.crossfilter
       );
-      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
+      const colorRGB = new Array(world.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
       const categoricalSelectionState = createCategoricalSelectionState(
         state,
         world
@@ -289,7 +294,6 @@ const Controls = (
         loading: false,
         error: null,
         world,
-        colorName,
         colorRGB,
         categoricalSelectionState,
         crossfilter,
@@ -324,6 +328,18 @@ const Controls = (
         }
       };
     }
+    case "request user defined gene started": {
+      return {
+        ...state,
+        userDefinedGenesLoading: true
+      };
+    }
+    case "request user defined gene error": {
+      return {
+        ...state,
+        userDefinedGenesLoading: false
+      };
+    }
     case "request user defined gene success": {
       const { world, crossfilter, dimensionMap, userDefinedGenes } = state;
       const worldVarDataCache = world.varDataCache;
@@ -341,7 +357,8 @@ const Controls = (
       return {
         ...state,
         dimensionMap,
-        userDefinedGenes: _userDefinedGenes
+        userDefinedGenes: _userDefinedGenes,
+        userDefinedGenesLoading: false
       };
     }
     case "request differential expression success": {
@@ -442,11 +459,11 @@ const Controls = (
     }
     case "reset colorscale": {
       const { world } = state;
-      const colorName = new Array(world.nObs).fill(globals.defaultCellColor);
-      const colorRGB = _.map(colorName, c => parseRGB(c));
+      const colorRGB = new Array(world.nObs).fill(
+        parseRGB(globals.defaultCellColor)
+      );
       return {
         ...state,
-        colorName,
         colorRGB,
         colorAccessor: null
       };
@@ -464,27 +481,36 @@ const Controls = (
              User Events
      *******************************/
     case "graph brush selection change": {
-      state.dimensionMap[layoutDimensionName("X")].filterRange([
-        action.brushCoords.northwest[0],
-        action.brushCoords.southeast[0]
-      ]);
-      state.dimensionMap[layoutDimensionName("Y")].filterRange([
-        action.brushCoords.southeast[1],
-        action.brushCoords.northwest[1]
-      ]);
+      state.dimensionMap[layoutDimensionName("XY")].filterWithinRect(
+        action.brushCoords.northwest,
+        action.brushCoords.southeast
+      );
       return {
         ...state,
         graphBrushSelection: action.brushCoords
       };
     }
+    case "lasso deselect":
     case "graph brush deselect": {
-      state.dimensionMap[layoutDimensionName("X")].filterAll();
-      state.dimensionMap[layoutDimensionName("Y")].filterAll();
+      state.dimensionMap[layoutDimensionName("XY")].filterAll();
       return {
         ...state,
         graphBrushSelection: null
       };
     }
+    case "lasso selection": {
+      const { polygon } = action;
+      const dXY = state.dimensionMap[layoutDimensionName("XY")];
+      if (polygon.length < 3) {
+        // single point or a line is not a polygon, and is therefore a deselect
+        dXY.filterAll();
+      } else {
+        dXY.filterWithinPolygon(polygon);
+      }
+      return {
+        ...state
+      };
+    }
     case "continuous metadata histogram brush": {
       const name = makeContinuousDimensionName(
         action.continuousNamespace,
@@ -512,6 +538,12 @@ const Controls = (
         graphRenderCounter: c
       };
     }
+    case "interface reset started": {
+      return {
+        ...state,
+        resettingInterface: true
+      };
+    }
     /*******************************
           Categorical metadata
     *******************************/
@@ -609,7 +641,6 @@ const Controls = (
     case "color by continuous metadata": {
       return {
         ...state,
-        colorName: action.colors.name,
         colorRGB: action.colors.rgb,
         colorAccessor: action.colorAccessor,
         colorScale: action.colorScale
@@ -618,7 +649,6 @@ const Controls = (
     case "color by expression": {
       return {
         ...state,
-        colorName: action.colors.name,
         colorRGB: action.colors.rgb,
         colorAccessor: action.gene,
         colorScale: action.colorScale

client/src/reducers/index.js

@@ -1,9 +1,9 @@
 // jshint esversion: 6
 import { combineReducers, createStore, applyMiddleware } from "redux";
 import thunk from "redux-thunk";
+import { composeWithDevTools } from "redux-devtools-extension";
 import updateURLMiddleware from "../middleware/updateURLMiddleware";
 import updateCellColors from "../middleware/updateCellColors";
-import { composeWithDevTools } from "redux-devtools-extension";
 
 import config from "./config";
 import differential from "./differential";

client/src/util/actionHelpers.js

@@ -25,17 +25,18 @@ export function catchErrorsWrap(fn, dispatchToUser = false) {
 }
 
 /*
-Wrapper to perform an async fetch and JSON decode response.
+Wrapper to perform async fetch with some modest error handling
+and decoding.
 */
-export const doJsonRequest = async url => {
+const doFetch = async (url, acceptType) => {
   const res = await fetch(url, {
     method: "get",
     headers: new Headers({
-      "Content-Type": "application/json"
+      Accept: acceptType
     })
   });
-  if (res.ok && res.headers.get("Content-Type") === "application/json") {
-    return res.json();
+  if (res.ok && res.headers.get("Content-Type").includes(acceptType)) {
+    return res;
   }
   // else an error
   let msg = `Unexpected HTTP response ${res.status}, ${res.statusText}`;
@@ -47,6 +48,22 @@ export const doJsonRequest = async url => {
   throw new Error(msg);
 };
 
+/*
+Wrapper to perform an async fetch and JSON decode response.
+*/
+export const doJsonRequest = async url => {
+  const res = await doFetch(url, "application/json");
+  return res.json();
+};
+
+/*
+Wrapper to perform an async fetch for binary data.
+*/
+export const doBinaryRequest = async url => {
+  const res = await doFetch(url, "application/octet-stream");
+  return res.arrayBuffer();
+};
+
 /*
 This function "packs" filter index lists into the more efficient
 "range" form specified in the REST 0.2 spec.

client/src/util/finiteExtent.js

@@ -0,0 +1,33 @@
+/*
+Return the [minimum, maximum] extent, of the given typed array, ignoring
+non-finite values (ie, +Infinity, -Infinity).
+
+If undefined or empty array, or array contains only non-finite numbers,
+will return [undefined, undefined]
+*/
+
+function finiteExtent(tarr) {
+  let min;
+  let max;
+  let i;
+
+  for (i = 0; i < tarr.length; i += 1) {
+    const val = tarr[i];
+    if (Number.isFinite(val)) {
+      min = val;
+      max = val;
+      i += 1;
+      break;
+    }
+  }
+  for (; i < tarr.length; i += 1) {
+    const val = tarr[i];
+    if (Number.isFinite(val)) {
+      if (min > val) min = val;
+      if (max < val) max = val;
+    }
+  }
+  return [min, max];
+}
+
+export default finiteExtent;

client/src/util/stateManager/index.js

@@ -3,7 +3,7 @@
 /*
 Model manager providing an abstraction for the use of the reducer code.
 This module provides several buckets of functionality:
-  - schema and config driven tranformation of the dataframe wire protocol
+  - schema and config driven tranformation of the wire protocol
     into a format that is easy for the UI code to use.
   - manage the universe/world abstraction:
     + universe: all of the server-provided, read-only data

client/src/util/stateManager/matrix.js

@@ -0,0 +1,75 @@
+import { flatbuffers } from "flatbuffers";
+import { NetEncoding } from "./matrix_generated";
+
+const utf8Decoder = new TextDecoder("utf-8");
+
+/*
+Matrix flatbuffer decoding support.   See fbs/matrix.fbs
+*/
+
+/*
+Decode NetEncoding.TypedArray
+*/
+function decodeTypedArray(uType, uValF, inplace = false) {
+  if (uType === NetEncoding.TypedArray.NONE) {
+    return null;
+  }
+
+  // Convert to a JS class that supports this type
+  const TypeClass = NetEncoding[NetEncoding.TypedArray[uType]];
+  // Create a TypedArray that references the underlying buffer
+  let arr = uValF(new TypeClass()).dataArray();
+  if (uType === NetEncoding.TypedArray.JSONEncodedArray) {
+    const json = utf8Decoder.decode(arr);
+    arr = JSON.parse(json);
+  } else if (!inplace) {
+    /* force copy to release underlying FBS buffer */
+    arr = new arr.constructor(arr);
+  }
+  return arr;
+}
+
+/*
+Parameter: Uint8Array or ArrayBuffer containing raw flatbuffer Matrix
+Returns: object containing decoded Matrix:
+{
+  nRows: num,
+  nCols: num,
+  columns: [
+    each column, which will be a TypedArray or Array
+  ]
+  colIdx: []|null
+}
+*/
+function decodeMatrixFBS(arrayBuffer, inplace = false) {
+  const bb = new flatbuffers.ByteBuffer(new Uint8Array(arrayBuffer));
+  const df = NetEncoding.Matrix.getRootAsMatrix(bb);
+
+  const nRows = df.nRows();
+  const nCols = df.nCols();
+
+  /* decode columns */
+  const columnsLength = df.columnsLength();
+  const columns = Array(columnsLength).fill(null);
+  for (let c = 0; c < columnsLength; c += 1) {
+    const col = df.columns(c);
+    columns[c] = decodeTypedArray(col.uType(), col.u.bind(col), inplace);
+  }
+
+  /* decode col_idx */
+  const colIdx = decodeTypedArray(
+    df.colIndexType(),
+    df.colIndex.bind(df),
+    inplace
+  );
+
+  return {
+    nRows,
+    nCols,
+    columns,
+    colIdx,
+    rowIdx: null
+  };
+}
+
+export default decodeMatrixFBS;

client/src/util/stateManager/matrix_generated.js

@@ -0,0 +1,835 @@
+// automatically generated by the FlatBuffers compiler, do not modify
+
+/**
+ * @const
+ * @namespace
+ */
+var NetEncoding = NetEncoding || {};
+
+/**
+ * @enum
+ */
+NetEncoding.TypedArray = {
+  NONE: 0, 0: 'NONE',
+  Float32Array: 1, 1: 'Float32Array',
+  Int32Array: 2, 2: 'Int32Array',
+  Uint32Array: 3, 3: 'Uint32Array',
+  Float64Array: 4, 4: 'Float64Array',
+  JSONEncodedArray: 5, 5: 'JSONEncodedArray'
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Float32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Float32Array}
+ */
+NetEncoding.Float32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Float32Array=} obj
+ * @returns {NetEncoding.Float32Array}
+ */
+NetEncoding.Float32Array.getRootAsFloat32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Float32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Float32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readFloat32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Float32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Float32Array}
+ */
+NetEncoding.Float32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Float32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Float32Array.startFloat32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Float32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addFloat32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Float32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float32Array.endFloat32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Uint32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Uint32Array}
+ */
+NetEncoding.Uint32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Uint32Array=} obj
+ * @returns {NetEncoding.Uint32Array}
+ */
+NetEncoding.Uint32Array.getRootAsUint32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Uint32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Uint32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Uint32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Uint32Array}
+ */
+NetEncoding.Uint32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Uint32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Uint32Array.startUint32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Uint32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Uint32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Uint32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Uint32Array.endUint32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Int32Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Int32Array}
+ */
+NetEncoding.Int32Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Int32Array=} obj
+ * @returns {NetEncoding.Int32Array}
+ */
+NetEncoding.Int32Array.getRootAsInt32Array = function(bb, obj) {
+  return (obj || new NetEncoding.Int32Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Int32Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readInt32(this.bb.__vector(this.bb_pos + offset) + index * 4) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Int32Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Int32Array}
+ */
+NetEncoding.Int32Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Int32Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Int32Array.startInt32Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Int32Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Int32Array.createDataVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt32(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Int32Array.startDataVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Int32Array.endInt32Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Float64Array = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Float64Array}
+ */
+NetEncoding.Float64Array.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Float64Array=} obj
+ * @returns {NetEncoding.Float64Array}
+ */
+NetEncoding.Float64Array.getRootAsFloat64Array = function(bb, obj) {
+  return (obj || new NetEncoding.Float64Array).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.Float64Array.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readFloat64(this.bb.__vector(this.bb_pos + offset) + index * 8) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Float64Array.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Float64Array}
+ */
+NetEncoding.Float64Array.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Float64Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Float64Array.startFloat64Array = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.Float64Array.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float64Array.createDataVector = function(builder, data) {
+  builder.startVector(8, data.length, 8);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addFloat64(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Float64Array.startDataVector = function(builder, numElems) {
+  builder.startVector(8, numElems, 8);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Float64Array.endFloat64Array = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.JSONEncodedArray = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.JSONEncodedArray}
+ */
+NetEncoding.JSONEncodedArray.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.JSONEncodedArray=} obj
+ * @returns {NetEncoding.JSONEncodedArray}
+ */
+NetEncoding.JSONEncodedArray.getRootAsJSONEncodedArray = function(bb, obj) {
+  return (obj || new NetEncoding.JSONEncodedArray).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @param {number} index
+ * @returns {number}
+ */
+NetEncoding.JSONEncodedArray.prototype.data = function(index) {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint8(this.bb.__vector(this.bb_pos + offset) + index) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.JSONEncodedArray.prototype.dataLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {Uint8Array}
+ */
+NetEncoding.JSONEncodedArray.prototype.dataArray = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? new Uint8Array(this.bb.bytes().buffer, this.bb.bytes().byteOffset + this.bb.__vector(this.bb_pos + offset), this.bb.__vector_len(this.bb_pos + offset)) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.JSONEncodedArray.startJSONEncodedArray = function(builder) {
+  builder.startObject(1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} dataOffset
+ */
+NetEncoding.JSONEncodedArray.addData = function(builder, dataOffset) {
+  builder.addFieldOffset(0, dataOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<number>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.JSONEncodedArray.createDataVector = function(builder, data) {
+  builder.startVector(1, data.length, 1);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addInt8(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.JSONEncodedArray.startDataVector = function(builder, numElems) {
+  builder.startVector(1, numElems, 1);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.JSONEncodedArray.endJSONEncodedArray = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Column = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Column.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Column=} obj
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Column.getRootAsColumn = function(bb, obj) {
+  return (obj || new NetEncoding.Column).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Column.prototype.uType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Column.prototype.u = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 6);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Column.startColumn = function(builder) {
+  builder.startObject(2);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} uType
+ */
+NetEncoding.Column.addUType = function(builder, uType) {
+  builder.addFieldInt8(0, uType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} uOffset
+ */
+NetEncoding.Column.addU = function(builder, uOffset) {
+  builder.addFieldOffset(1, uOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Column.endColumn = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @constructor
+ */
+NetEncoding.Matrix = function() {
+  /**
+   * @type {flatbuffers.ByteBuffer}
+   */
+  this.bb = null;
+
+  /**
+   * @type {number}
+   */
+  this.bb_pos = 0;
+};
+
+/**
+ * @param {number} i
+ * @param {flatbuffers.ByteBuffer} bb
+ * @returns {NetEncoding.Matrix}
+ */
+NetEncoding.Matrix.prototype.__init = function(i, bb) {
+  this.bb_pos = i;
+  this.bb = bb;
+  return this;
+};
+
+/**
+ * @param {flatbuffers.ByteBuffer} bb
+ * @param {NetEncoding.Matrix=} obj
+ * @returns {NetEncoding.Matrix}
+ */
+NetEncoding.Matrix.getRootAsMatrix = function(bb, obj) {
+  return (obj || new NetEncoding.Matrix).__init(bb.readInt32(bb.position()) + bb.position(), bb);
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.nRows = function() {
+  var offset = this.bb.__offset(this.bb_pos, 4);
+  return offset ? this.bb.readUint32(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.nCols = function() {
+  var offset = this.bb.__offset(this.bb_pos, 6);
+  return offset ? this.bb.readUint32(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @param {number} index
+ * @param {NetEncoding.Column=} obj
+ * @returns {NetEncoding.Column}
+ */
+NetEncoding.Matrix.prototype.columns = function(index, obj) {
+  var offset = this.bb.__offset(this.bb_pos, 8);
+  return offset ? (obj || new NetEncoding.Column).__init(this.bb.__indirect(this.bb.__vector(this.bb_pos + offset) + index * 4), this.bb) : null;
+};
+
+/**
+ * @returns {number}
+ */
+NetEncoding.Matrix.prototype.columnsLength = function() {
+  var offset = this.bb.__offset(this.bb_pos, 8);
+  return offset ? this.bb.__vector_len(this.bb_pos + offset) : 0;
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Matrix.prototype.colIndexType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 10);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Matrix.prototype.colIndex = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 12);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @returns {NetEncoding.TypedArray}
+ */
+NetEncoding.Matrix.prototype.rowIndexType = function() {
+  var offset = this.bb.__offset(this.bb_pos, 14);
+  return offset ? /** @type {NetEncoding.TypedArray} */ (this.bb.readUint8(this.bb_pos + offset)) : NetEncoding.TypedArray.NONE;
+};
+
+/**
+ * @param {flatbuffers.Table} obj
+ * @returns {?flatbuffers.Table}
+ */
+NetEncoding.Matrix.prototype.rowIndex = function(obj) {
+  var offset = this.bb.__offset(this.bb_pos, 16);
+  return offset ? this.bb.__union(obj, this.bb_pos + offset) : null;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ */
+NetEncoding.Matrix.startMatrix = function(builder) {
+  builder.startObject(7);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} nRows
+ */
+NetEncoding.Matrix.addNRows = function(builder, nRows) {
+  builder.addFieldInt32(0, nRows, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} nCols
+ */
+NetEncoding.Matrix.addNCols = function(builder, nCols) {
+  builder.addFieldInt32(1, nCols, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} columnsOffset
+ */
+NetEncoding.Matrix.addColumns = function(builder, columnsOffset) {
+  builder.addFieldOffset(2, columnsOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {Array.<flatbuffers.Offset>} data
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Matrix.createColumnsVector = function(builder, data) {
+  builder.startVector(4, data.length, 4);
+  for (var i = data.length - 1; i >= 0; i--) {
+    builder.addOffset(data[i]);
+  }
+  return builder.endVector();
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {number} numElems
+ */
+NetEncoding.Matrix.startColumnsVector = function(builder, numElems) {
+  builder.startVector(4, numElems, 4);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} colIndexType
+ */
+NetEncoding.Matrix.addColIndexType = function(builder, colIndexType) {
+  builder.addFieldInt8(3, colIndexType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} colIndexOffset
+ */
+NetEncoding.Matrix.addColIndex = function(builder, colIndexOffset) {
+  builder.addFieldOffset(4, colIndexOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {NetEncoding.TypedArray} rowIndexType
+ */
+NetEncoding.Matrix.addRowIndexType = function(builder, rowIndexType) {
+  builder.addFieldInt8(5, rowIndexType, NetEncoding.TypedArray.NONE);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} rowIndexOffset
+ */
+NetEncoding.Matrix.addRowIndex = function(builder, rowIndexOffset) {
+  builder.addFieldOffset(6, rowIndexOffset, 0);
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @returns {flatbuffers.Offset}
+ */
+NetEncoding.Matrix.endMatrix = function(builder) {
+  var offset = builder.endObject();
+  return offset;
+};
+
+/**
+ * @param {flatbuffers.Builder} builder
+ * @param {flatbuffers.Offset} offset
+ */
+NetEncoding.Matrix.finishMatrixBuffer = function(builder, offset) {
+  builder.finish(offset);
+};
+
+// Exports for ECMAScript6 Modules
+export {NetEncoding};

client/src/util/stateManager/summarizeAnnotations.js

@@ -1,4 +1,5 @@
 import _ from "lodash";
+import finiteExtent from "../finiteExtent";
 
 /*
 Build and return obs/var summary using any annotation in the schema
@@ -61,16 +62,32 @@ function _summarizeAnnotations(_schema, annotations) {
       const continuous = type === "int32" || type === "float32";
 
       if (continuous) {
-        let min = Number.POSITIVE_INFINITY;
-        let max = Number.NEGATIVE_INFINITY;
+        let min;
+        let max;
+        let nan = 0;
+        let pinf = 0;
+        let ninf = 0;
         for (let r = 0; r < annotations.length; r += 1) {
           const val = Number(annotations[r][name]);
-          min = val < min ? val : min;
-          max = val > max ? val : max;
+          if (Number.isFinite(val)) {
+            if (min === undefined) {
+              min = val;
+              max = val;
+            } else {
+              min = val < min ? val : min;
+              max = val > max ? val : max;
+            }
+          } else if (Number.isNaN(val)) {
+            nan += 1;
+          } else if (val > 0) {
+            pinf += 1;
+          } else {
+            ninf += 1;
+          }
         }
         return {
           categorical: false,
-          range: { min, max }
+          range: { min, max, nan, pinf, ninf }
         };
       }
 

client/src/util/stateManager/universe.js

@@ -1,8 +1,10 @@
 // jshint esversion: 6
 
 import _ from "lodash";
+
 import * as kvCache from "./keyvalcache";
 import summarizeAnnotations from "./summarizeAnnotations";
+import decodeMatrixFBS from "./matrix";
 
 /*
 Private helper function - create and return a template Universe
@@ -92,73 +94,40 @@ function finalize(universe) {
   return universe;
 }
 
-function RESTv02AnnotationsResponseToInternal(response) {
+function RESTv02AnotationsFBSResponseToInternal(arrayBuffer) {
   /*
-  Source per the spec:
-  {
-    names: [
-      'tissue_type', 'sex', 'num_reads', 'clusters'
-    ],
-    data: [
-      [ 0, 'lung', 'F', 39844, 99 ],
-      [ 1, 'heart', 'M', 83, 1 ],
-      [ 49, 'spleen', null, 2, "unknown cluster" ],
-      // [ obsOrVarIndex, value, value, value, value ],
-      // ...
-    ]
-  }
+  Convert a Matrix FBS to our internal format -- row-major array of
+  observations/cells, stored as an object.  Each obs has a key for each
+  annotation, plus __index__ containing its obsIndex.
 
-  Internal (target) format:
+  Example:
   [
     { __index__: 0, tissue_type: "lung", sex: "F", ... },
     ...
   ]
+
+  XXX TODO: we could make use of the columns in building crossfilter
+  dimensions (they have to be recreated).  Future optimization.
   */
-  const { names, data } = response;
-  const keys = ["__index__", ...names];
-  return _(data)
-    .map(obs => _.zipObject(keys, obs))
-    .value();
+  const fbs = decodeMatrixFBS(arrayBuffer);
+  const keys = fbs.colIdx;
+  const result = Array(fbs.nRows);
+  for (let row = 0; row < fbs.nRows; row += 1) {
+    const rec = { __index__: row };
+    for (let col = 0; col < fbs.nCols; col += 1) {
+      rec[keys[col]] = fbs.columns[col][row];
+    }
+    result[row] = rec;
+  }
+  return result;
 }
 
-function RESTv02LayoutResponseToInternal(response) {
-  /*
-  Source per the spec:
-  {
-    layout: {
-      ndims: 2,
-      coordinates: [
-        [ 0, 0.284483, 0.983744 ],
-        [ 1, 0.038844, 0.739444 ],
-        // [ obsOrVarIndex, X_coord, Y_coord ],
-        // ...
-      ]
-    }
-  }
-
-  Target (internal) format:
-  {
-    X: Float32Array(numObs),
-    Y: Float32Array(numObs)
-  }
-  In the same order as obsAnnotations
-  */
-  const { ndims, coordinates } = response.layout;
-  if (ndims !== 2) {
-    throw new Error("Unsupported layout dimensionality");
-  }
-
-  const layout = {
-    X: new Float32Array(coordinates.length),
-    Y: new Float32Array(coordinates.length)
+function RESTv02LayoutFBSResponseToInternal(arrayBuffer) {
+  const fbs = decodeMatrixFBS(arrayBuffer, true);
+  return {
+    X: fbs.columns[0],
+    Y: fbs.columns[1]
   };
-
-  for (let i = 0; i < coordinates.length; i += 1) {
-    const [idx, x, y] = coordinates[i];
-    layout.X[idx] = x;
-    layout.Y[idx] = y;
-  }
-  return layout;
 }
 
 function reconcileSchemaCategoriesWithSummary(universe) {
@@ -192,7 +161,7 @@ export function createUniverseFromRestV02Response(
   schemaResponse,
   annotationsObsResponse,
   annotationsVarResponse,
-  layoutObsResponse
+  layoutFBSResponse
 ) {
   /*
   build & return universe from a REST 0.2 /config, /schema and /annotations/obs response
@@ -209,15 +178,15 @@ export function createUniverseFromRestV02Response(
   universe.nVar = schema.dataframe.nVar;
 
   /* annotations */
-  universe.obsAnnotations = RESTv02AnnotationsResponseToInternal(
+  universe.obsAnnotations = RESTv02AnotationsFBSResponseToInternal(
     annotationsObsResponse
   );
-  universe.varAnnotations = RESTv02AnnotationsResponseToInternal(
+  universe.varAnnotations = RESTv02AnotationsFBSResponseToInternal(
     annotationsVarResponse
   );
 
   /* layout */
-  universe.obsLayout = RESTv02LayoutResponseToInternal(layoutObsResponse);
+  universe.obsLayout = RESTv02LayoutFBSResponseToInternal(layoutFBSResponse);
 
   universe.summary = summarizeAnnotations(
     universe.schema,
@@ -229,32 +198,24 @@ export function createUniverseFromRestV02Response(
   return finalize(universe);
 }
 
-export function convertExpressionRESTv02ToObject(universe, response) {
+export function convertDataFBStoObject(universe, arrayBuffer) {
   /*
-  /data/obs response looks like:
-  {
-    var: [ varIndices fetched ],
-    obs: [
-      [ obsIndex, evalue, ... ],
-      ...
-    ]
-  }
+  /data/var returns a flatbuffer (FBS) as described by cellxgene/fbs/matrix.fbs
 
-  convert expression toa simple Float32Array, and return
-  { geneName: array, geneName: array, ... }
-  NOTE: geneName, not varIndex
+  This routine converts the binary wire encoding into a JS object:
+
+  {
+    gene: Float32Array,
+    ...
+  }
   */
-  const vars = response.var;
-  const { obs } = response;
+  const fbs = decodeMatrixFBS(arrayBuffer);
+  const { colIdx, columns } = fbs;
   const result = {};
-  // XXX TODO: could this use _.unzip and have less code?
-  for (let varIdx = 0; varIdx < vars.length; varIdx += 1) {
-    const gene = universe.varAnnotations[vars[varIdx]].name;
-    const data = new Float32Array(universe.nObs);
-    for (let obsIdx = 0; obsIdx < obs.length; obsIdx += 1) {
-      data[obsIdx] = obs[obsIdx][varIdx + 1];
-    }
-    result[gene] = data;
+
+  for (let c = 0; c < colIdx.length; c += 1) {
+    const gene = universe.varAnnotations[colIdx[c]].name;
+    result[gene] = columns[c];
   }
   return result;
 }

client/src/util/stateManager/world.js

@@ -4,6 +4,7 @@ import _ from "lodash";
 import * as kvCache from "./keyvalcache";
 import summarizeAnnotations from "./summarizeAnnotations";
 import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
+import Crossfilter from "../typedCrossfilter";
 import { sliceByIndex } from "../typedCrossfilter/util";
 
 /*
@@ -231,7 +232,12 @@ export function createVarDimension(
   crossfilter,
   geneName
 ) {
-  return crossfilter.dimension(_worldVarDataCache[geneName], Float32Array);
+  // return crossfilter.dimension(_worldVarDataCache[geneName], Float32Array);
+  return crossfilter.dimension(
+    Crossfilter.ScalarDimension,
+    _worldVarDataCache[geneName],
+    Float32Array
+  );
 }
 
 export function createObsDimensionMap(crossfilter, world) {
@@ -246,9 +252,14 @@ export function createObsDimensionMap(crossfilter, world) {
     .filter(anno => anno.name !== "name")
     .transform((result, anno) => {
       const dimType = deduceDimensionType(anno, anno.name);
-      // XXX if dimtype is a scalar, we may be able to do better?
-      if (dimType) {
+      if (dimType === "enum") {
         result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
+          Crossfilter.EnumDimension,
+          r => r[anno.name]
+        );
+      } else {
+        result[obsAnnoDimensionName(anno.name)] = crossfilter.dimension(
+          Crossfilter.ScalarDimension,
           r => r[anno.name],
           dimType
         );
@@ -259,13 +270,10 @@ export function createObsDimensionMap(crossfilter, world) {
   /*
   Add crossfilter dimensions allowing filtering on layout
   */
-  dimensionMap[layoutDimensionName("X")] = crossfilter.dimension(
+  dimensionMap[layoutDimensionName("XY")] = crossfilter.dimension(
+    Crossfilter.SpatialDimension,
     obsLayout.X,
-    Float32Array
-  );
-  dimensionMap[layoutDimensionName("Y")] = crossfilter.dimension(
-    obsLayout.Y,
-    Float32Array
+    obsLayout.Y
   );
 
   return dimensionMap;

client/src/util/typedCrossfilter/index.js

@@ -27,6 +27,8 @@ more complex API.  In a few cases, elements of that API were incorporated.
     https://github.com/square/crossfilter/
 
 */
+// XXX replace
+import { polygonContains } from "d3";
 
 import PositiveIntervals from "./positiveIntervals";
 import BitArray from "./bitArray";
@@ -66,14 +68,22 @@ class TypedCrossfilter {
     return this.data;
   }
 
-  dimension(value, valueArrayType) {
+  /*
+  Create a crossfilter dimension, upon which filtering (subselection) can
+  be done.  Each dimension is typed, and has a particular set of filtering
+  semantics.
+    * ScalarDimension - backed by TypedArray values, supporting filtering
+      by value (within a value range, or one or more exact values)
+    * EnumDimension - backed by an enumeration (eg, strings, bools), filtering
+      by one or more enum categories.
+    * SpatialDimension - backed by 2D points, filter by containment within
+      various shapes (currently supports within Rectangle and within Polygon).
+  Call this method to create a dimension, passing arguments appropriate for
+  the dimension constructor.
+  */
+  dimension(DimensionType, ...rest) {
     const id = this.selection.allocDimension();
-    let dim;
-    if (valueArrayType === "enum") {
-      dim = new EnumDimension(value, this, id);
-    } else {
-      dim = new ScalarDimension(value, valueArrayType, this, id);
-    }
+    const dim = new DimensionType(this, id, ...rest);
     this.filters.push({ id, dim });
     dim.filterAll();
     return dim;
@@ -115,13 +125,34 @@ class TypedCrossfilter {
   }
 }
 
-// Base dimension type - value must be a scalar type (eg, int, float),
-// and value array must be a TypedArray.
-//
-class ScalarDimension {
-  constructor(value, ValueArrayType, xfltr, id) {
+// Base dimension type - not exported.
+class _Dimension {
+  constructor(xfltr, id) {
     this.crossfilter = xfltr;
     this._id = id;
+    this.groups = [];
+  }
+
+  dispose() {
+    this.crossfilter._freeDimension(this._id);
+    return this;
+  }
+
+  id() {
+    return this._id;
+  }
+
+  _filterUpdate() {
+    this.crossfilter.updateTime += 1;
+  }
+}
+
+// Scalar dimension type - value must be a scalar type (eg, int, float),
+// and value array must be a TypedArray.
+//
+class ScalarDimension extends _Dimension {
+  constructor(xfltr, id, value, ValueArrayType) {
+    super(xfltr, id);
 
     // current selection filter, expressed as PostiveIntervals.
     this.currentFilter = [];
@@ -151,9 +182,6 @@ class ScalarDimension {
 
     // create sort index
     this.index = makeSortIndex(array);
-
-    // groups, if any
-    this.groups = [];
   }
 
   _createValueArray(value, array) {
@@ -167,15 +195,6 @@ class ScalarDimension {
     return larray;
   }
 
-  dispose() {
-    this.crossfilter._freeDimension(this._id);
-    return this;
-  }
-
-  id() {
-    return this._id;
-  }
-
   // Argument is an array of intervals indicating records newly selected/filtered
   //
   _updateFilters(newFilter) {
@@ -209,7 +228,7 @@ class ScalarDimension {
     );
 
     this.currentFilter = cNewFilter;
-    this.crossfilter.updateTime += 1;
+    this._filterUpdate();
   }
 
   // filter by value - exact match
@@ -355,8 +374,8 @@ class ScalarDimension {
 // strings, which can be mapped into an fixed numeric range [0..n).
 //
 class EnumDimension extends ScalarDimension {
-  constructor(value, xfltr, id) {
-    super(value, Uint32Array, xfltr, id);
+  constructor(xfltr, id, value) {
+    super(xfltr, id, value, Uint32Array);
   }
 
   _createValueArray(value, array) {
@@ -408,6 +427,127 @@ class EnumDimension extends ScalarDimension {
   }
 }
 
+/*
+Super simple 2D spatial dimension, supporting basic "filter within"
+operations.
+*/
+class SpatialDimension extends _Dimension {
+  constructor(xfltr, id, X, Y) {
+    super(xfltr, id);
+
+    if (X.length !== Y.length && X.length !== this.crossfilter.data.length) {
+      throw new RangeError(
+        "SpatialDimension values must have same dimensionality as crossfilter"
+      );
+    }
+    this.X = X;
+    this.Y = Y;
+
+    this.Xindex = makeSortIndex(X);
+    this.Yindex = makeSortIndex(Y);
+  }
+
+  filterAll() {
+    this.crossfilter.selection.selectAll(this._id);
+    this._filterUpdate();
+  }
+
+  filterNone() {
+    this.crossfilter.selection.deselectAll(this._id);
+    this._filterUpdate();
+  }
+
+  /*
+  this could be smarter, but we don't currently use it...
+  */
+  filterWithinRect(northwest, southeast) {
+    const [x0, y0] = northwest;
+    const [x1, y1] = southeast;
+    const { X, Y } = this;
+    const seln = this.crossfilter.selection;
+    const { _id } = this;
+    seln.deselectAll(_id);
+    for (let i = 0, l = this.X.length; i < l; i += 1) {
+      const x = X[i];
+      const y = Y[i];
+      if (x0 <= x && x < x1 && y0 <= y && y < y1) {
+        seln.selectOne(_id, i);
+      }
+    }
+    this._filterUpdate();
+  }
+
+  /*
+  Relatively brute force filter by polygon.   Polygon is array of points, where
+  each point is [x,y].   Eg, [[x0,y0], [x1,y1], ...].
+
+  Currently uses d3.polygonContains() to test for polygon inclusion, which itself
+  uses a ray casting (crossing number) algorithm.  There are a series of optimizations
+  to make this faster:
+    * first sliced by X or Y, using an index on the axis
+    * then the polygon bounding box is used for trivial rejection
+    * then the polygon test is applied
+  */
+  filterWithinPolygon(polygon) {
+    /* return bounding box of the polygon */
+    function polygonBoundingBox(pg) {
+      let minX = Number.MAX_VALUE;
+      let minY = Number.MAX_VALUE;
+      let maxX = Number.MIN_VALUE;
+      let maxY = Number.MIN_VALUE;
+      for (let i = 0, l = pg.length; i < l; i += 1) {
+        const p = pg[i];
+        const x = p[0];
+        const y = p[1];
+        if (x < minX) minX = x;
+        if (y < minY) minY = y;
+        if (x > maxX) maxX = x;
+        if (y > maxY) maxY = y;
+      }
+      return [minX, minY, maxX, maxY];
+    }
+
+    const [minX, minY, maxX, maxY] = polygonBoundingBox(polygon);
+    const { X, Y } = this;
+    let slice;
+    let index;
+    if (maxY - minY > maxX - minX) {
+      slice = [
+        lowerBoundIndirect(X, this.Xindex, minX, 0, X.length),
+        upperBoundIndirect(X, this.Xindex, maxX, 0, X.length)
+      ];
+      index = this.Xindex;
+    } else {
+      slice = [
+        lowerBoundIndirect(Y, this.Yindex, minY, 0, Y.length),
+        upperBoundIndirect(Y, this.Yindex, maxY, 0, Y.length)
+      ];
+      index = this.Yindex;
+    }
+
+    const seln = this.crossfilter.selection;
+    const { _id } = this;
+    const testWithin = polygonContains; // d3.polygonContains()
+    seln.deselectAll(_id);
+
+    for (let i = slice[0], e = slice[1]; i < e; i += 1) {
+      const rid = index[i];
+      const x = X[rid];
+      const y = Y[rid];
+      if (
+        minX <= x &&
+        x < maxX &&
+        minY <= y &&
+        y < maxY &&
+        testWithin(polygon, [x, y])
+      ) {
+        seln.selectOne(_id, rid);
+      }
+    }
+    this._filterUpdate();
+  }
+}
+
 // Groups!  Map/reduce
 //
 class ScalarGroup {
@@ -611,5 +751,6 @@ crossfilter.BitArray = BitArray;
 crossfilter.TypedCrossfilter = TypedCrossfilter;
 crossfilter.ScalarDimension = ScalarDimension;
 crossfilter.EnumDimension = EnumDimension;
+crossfilter.SpatialDimension = SpatialDimension;
 
 export default crossfilter;

dev_docs/REST_API.md

@@ -1,5 +1,11 @@
 # cellxgene REST API 0.2 specification
 
+_Note:_ this document is increasingly divergent from the code base and should be read with great suspicion. For example, it lacks any information
+about the binary encoding used by various routes, and has incorrect information about "required" routes and features. We may update it at a
+later date when the protocol stabilizes a bit.
+
+---
+
 Items marked as (_future_) are intended for future implementation, and are included in the design to round out the concept, and highlight what we would do when/if we needed more functionality. The (_future_) items are not currently used by the cellxgene web application, and may be omitted from any backend - see [Current Front-End Dependencies](#current-front-end-dependencies) for more details.
 
 _Caveat emptor, partial spec_: this is a sketch for a spec, not a full spec, and some shortcuts have been taken in the authorship. Best practices for a REST API are assumed but not documented here, such as API versioning, reasonable choices for HTTP response codes, etc. In addition, for clarity the JSON examples will not always have all required quoting (eg, on keys) - the actual implementation should use legal JSON/CSV.

dev_docs/release_process.md

@@ -20,18 +20,14 @@ Follow these steps to create a release.
 
 1.  Preparation:
     - python3.6 environment, and a cellxgene clone
-    - install required tools: `pip install -r requirements-dev.txt`
     - Define the release version number, using [semantic versioning](https://semver.org/),
       and specifying all three digits (eg, 0.3.0)
     - Write the release title and release notes and add to
       [release notes document](https://docs.google.com/document/d/1KnHwkYfhyWO5H8BDcMu7y3ogjvq5Yi4OwpmZ8DB6w0Y/edit)
 2.  Create a release branch, eg, `release-version`
 3.  In the release branch:
-    - Run `bumpversion --config-file .bumpversion.cfg [major | minor | patch]`,
-      where you choose major/minor/patch depending on which part of the version
+    - Run `make release-stage-1 PART=[major | minor | patch]` where you choose major/minor/patch depending on which part of the version
       is being bumped (eg, 0.2.9->0.3 is minor).
-    - Clean up existing environment using `bin/clean`
-    - Build the JS asserts using `bin/build-client`
 4.  Commit and push the new branch
 5.  Create a PR for the release.
     - [optional] As needed, conduct PR review.
@@ -44,18 +40,26 @@ Follow these steps to create a release.
     - Type title `Release {version num}`
     - [optional] Check pre-release if this release is not ready for production
     - Publish Release
-8.  Publish to pypi by performing the following steps (assumes you have `setuptools`
-    and `twine` installed, that you have registered for pypi, and that you have
-    write access to the cellxgene pypi package):
-    - Build the distribution by calling `python setup.py sdist`
-      inside the top-level directory
-    - [optional] Upload the package to test pypi
-      `twine upload --repository-url https://test.pypi.org/legacy/ dist/*`
-    - [optional] Test the test installation in a fresh virtual environment using
-      `pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene`
-    - Upload the package to real pypi using `twine upload dist/*`
+8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
+and that you have write access to the cellxgene pypi package):
+    - Build the distribution and upload to test pypi `make release-stage-2`
+    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Upload the package to real pypi using `make release-stage-final`
     - [optional] Test the installation in a fresh virtual environment using
       `pip install cellxgene`
+    - **Troubleshooting**:
+        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
+        if you accidentally burned a release number you want to use on prod, you have a couple options. 
+            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+             then upload to prod `make release-stage-final`. 
+            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
+            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+            PART=patch until you get to an unburned version.
+        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
+        go to the cellxgene package, click manage, then in the options drop down, click delete and 
+        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
+        and not a major regression, you can just release a patch. 
+    
 
 The optional steps are for testing purposes, and are recommended
 for publishing any major releases, and any releases that significantly

dev_docs/testing_environment_for_web_hosting.md

@@ -0,0 +1,69 @@
+### How to set up a testing environment for changes related to web hosting.
+
+We often get PRs related to someone using a server to host cellxgene externally or on a local network (ex. https://github.com/chanzuckerberg/cellxgene/pull/568 ). Here is how you can test these changes locally.
+
+We are going to run docker containers for cellxgene and an apache server running a reverse proxy on a local docker network. We run the cellxgene container without exposing any ports so that we cannot access it directly, only through the apache server. We can also update our cellxgene Dockerfile so that we can install a local build instead of having to deploy to pypi.
+
+1 Create Docker network, this allows the containers to communicate with each other.
+
+```
+docker network create cxg
+```
+
+2 Create and run cellxgene container
+
+(optional) To install cellxgene from the local codebase
+
+a Create sdist file
+`make pydist`
+
+b Update Dockerfile to install from dist
+
+```
+FROM ubuntu:bionic
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+COPY [ "dist/",    "/cellxgene/dist/" ]
+
+RUN apt-get update && \
+    apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev && \
+    pip3 install /cellxgene/dist/cellxgene-0.5.1.tar.gz
+
+ENTRYPOINT ["cellxgene"]
+```
+
+(required) Build container
+`docker build . -t cellxgene`
+
+3 Create the proxy container
+
+In a separate directory create these two files
+
+Dockerfile
+
+```
+FROM rgoyard/apache-proxy:latest
+ADD proxy.conf /conf/
+```
+
+proxy.conf
+
+```
+ProxyPass "/data/" http://cellxgene:5005/
+ProxyPassReverse "/data/" http://cellxgene:5005/
+```
+
+Build the container
+`docker build -t proxy .`
+
+4 Run containers and attach to network
+
+```
+docker run -d -p 80:80 --network cxg --name proxy proxy
+docker run -v "$PWD/example-dataset/:/data/" --name cellxgene --network cxg cellxgene launch --host 0.0.0.0 data/pbmc3k.h5ad
+```
+
+5 Go to served site
+
+http://localhost/data/

docs/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gem "github-pages", group: :jekyll_plugins

docs/_config.yml

@@ -1 +1,9 @@
-theme: jekyll-theme-architect
+theme: jekyll-theme-cayman
+show_downloads: false
+baseurl: /cellxgene
+
+nav:
+  - title: Data
+    url: data.html
+  - title: FAQ
+    url: faq.html

docs/_layouts/default.html

@@ -0,0 +1,49 @@
+<!DOCTYPE html>
+<html lang="{{ site.lang | default: "en-US" }}">
+  <head>
+
+    {% if site.google_analytics %}
+      <script async src="https://www.googletagmanager.com/gtag/js?id={{ site.google_analytics }}"></script>
+      <script>
+        window.dataLayer = window.dataLayer || [];
+        function gtag(){dataLayer.push(arguments);}
+        gtag('js', new Date());
+        gtag('config', '{{ site.google_analytics }}');
+      </script>
+    {% endif %}
+    <meta charset="UTF-8">
+
+{% seo %}
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <meta name="theme-color" content="#157878">
+    <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
+    <link rel="stylesheet" href="{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}">
+  </head>
+  <body>
+
+    <header class="page-header" role="banner">
+      <h1 class="project-name">{{ site.title | default: site.github.repository_name }}</h1>
+      <h2 class="project-tagline">{{ site.description | default: site.github.project_tagline }}</h2>
+      {% if site.nav %}
+        <a href="{{ site.baseurl }}/" class="btn">Home</a>
+        {% for item in site.nav %}
+          <a href="{{ item.url }}" class="btn">{{ item.title }}</a>
+        {% endfor %}
+      {% endif %}
+      {% if site.github.is_project_page %}
+        <a href="{{ site.github.repository_url }}" class="btn" target="_blank">Code</a>
+      {% endif %}
+    </header>
+
+    <main id="content" class="main-content" role="main">
+      {{ content }}
+
+      <footer class="site-footer">
+        {% if site.github.is_project_page %}
+          <span class="site-footer-owner"><a href="{{ site.github.repository_url }}">{{ site.github.repository_name }}</a> is maintained by <a href="{{ site.github.owner_url }}">{{ site.github.owner_name }}</a>.</span>
+        {% endif %}
+        <span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a>.</span>
+      </footer>
+    </main>
+  </body>
+</html>

docs/assets/css/style.scss

@@ -0,0 +1,4 @@
+---
+---
+
+@import "{{ site.theme }}";

docs/data.md

@@ -0,0 +1,29 @@
+---
+layout: default
+title: data
+description: Data
+---
+
+
+# data vignette: how to use cellxgene prepare
+
+#### coming soon!
+
+# example datasets to use with cellxgene
+
+### Examination of single cells from primary human pancreas tissue
+cells: 2,544  
+tissue(s): pancreas  
+data: [Human Cell Atlas Data Portal](https://prod.data.humancellatlas.org/explore/projects?filter=%5B%7B%22facetName%22%3A%22organ%22%2C%22terms%22%3A%5B%22pancreas%22%5D%7D%2C%7B%22facetName%22%3A%22project%22%2C%22terms%22%3A%5B%22Single+cell+transcriptome+analysis+of+human+pancreas%22%5D%7D%5D)  
+paper: [Enge, Martin, et al.](https://www.cell.com/cell/fulltext/S0092-8674(17)31053-X?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS009286741731053X%3Fshowall%3Dtrue)
+
+### Tabula Muris
+cells: 53,800  
+tissue(s): muscle, pancreas, bone, large intestine, heart, brain, fat, mammary gland, tongue , diaphragm, bladder, spleen, thymus, lung , skin, liver, trachea, kidney  
+data: [Tabula Muris Data](https://github.com/czbiohub/tabula-muris-vignettes/tree/master/data)  
+paper: [Tabula Muris Consortium](https://www.nature.com/articles/s41586-018-0590-4)  
+
+### Transcriptional profiling of 1.3 million brain cells
+cells: 1,330,000  
+tissue(s): brain  
+data: [10x Genomics](https://community.10xgenomics.com/t5/10x-Blog/Our-1-3-million-single-cell-dataset-is-ready-to-download/ba-p/276)  

docs/faq.md

@@ -0,0 +1,94 @@
+---
+layout: default
+title: FAQ
+description: Frequently Asked Questions
+---
+
+# Data formatting
+
+#### What file formats can I use with _cellxgene_?
+
+Currently, you can go straight into `cellxgene launch` with your own analyzed data in h5ad format, after you have performed dimenstionality reduction (tsne, umap) and clustering (louvain).
+
+If your data is in a different format, and/or you still need to perform dimensionality reduction and clustering, `cellxgene` can do that for you with the `prepare` command. `cellxgene prepare` runs `scanpy` under the hood and can read in any format that is currently supported by `scanpy` (including mtx, loom, and more listed [here](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)).
+
+The output of `cellxgene prepare` is a h5ad file with your computed clusters and tsne/umap projections that can be used in `cellxgene launch`.
+
+#### I have a directory of 10X-Genomics data with _mtx_ files and I've never used _scanpy_, can I use _cellxgene_?
+
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+```
+
+Depending on the size of the dataset, this may take some time. Once it's done, call
+
+```
+cellxgene launch data-processed.h5ad --layout=umap --open
+```
+
+And your web browser should open with an interactive view of your data.
+
+#### I have extra metadata that I want to add to my dataset
+
+Currently this is not supported directly, but you should be able to do this yourself using `scanpy`. For example, this [notebook](https://github.com/falexwolf/fun-analyses/blob/master/tabula_muris/tabula_muris.ipynb) shows adding the contents of a `csv` file with metadata to an `anndata` object. For now, you could do this manually on your data in the same way and then save out the result before loading into `cellxgene`.
+
+#### What part of the _anndata_ objects does cellxgene pull in for visualization?
+
+- `.obs` and `.var` annotations are use to extract metadata for filtering
+- `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
+- `.obsm` is used for layout
+
+# Algorithms
+
+#### How are you computing and sorting differential expression results?
+
+We use a [Welch's _t_-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) implementation including the same variance overestimation correction as used in `scanpy`. We sort the `tscore` to identify the top N genes, and then filter to remove any that fall below a cutoff log fold change value, which can help remove spurious test results. The default threshold is `0.01` and can be changed using the option `--diffexp-lfc-cutoff`.
+
+# Problems, errors, & bugs
+
+#### How do I create a Python 3.6 environment for _cellxgene_?
+
+If you use conda and want to create a [conda environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for _cellxgene_ you can use the following commands
+
+```
+conda create --yes -n cellxgene python=3.6
+conda activate cellxgene
+pip install cellxgene
+```
+
+Or you can create a virtual environment by using
+
+```
+ENV_NAME=cellxgene
+python3.6 -m venv ${ENV_NAME}
+source ${ENV_NAME}/bin/activate
+pip install cellxgene
+```
+
+#### In my _prepare_ command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
+
+Louvain clustering requires additional dependencies, so we don't include them by default. For now, you need to specify that you want these packages by using
+
+```
+pip install cellxgene[louvain]
+```
+
+#### I ran _prepare_ and I'm getting results that look unexpected
+
+You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
+
+```
+cellxgene prepare data/ --output=data-processed.h5ad --recipe=zheng17
+```
+
+It should be easy to run `prepare` then call `cellxgene launch` a few times with different settings to explore different behaviors. We may explore adding other preprocessing options in the future.
+
+#### I tried to `pip install cellxgene` and got a weird error I don't understand
+
+This may happen, especially as we work out bugs in our installation process! Please create a new [Github issue](https://github.com/chanzuckerberg/cellxgene/issues), explain what you did, and include all the error messages you saw. It'd also be super helpful if you call `pip freeze` and include the full output alongside your issue.
+
+#### I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
+
+This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.

docs/index.md

@@ -1,37 +1,37 @@
-# cellxgene
-
-cellxgene is an interactive data explorer for single-cell transcriptomics data designed to handle large datasets (1 million cells or more) and integrate with your favorite analysis tools
-
-## getting started
-
-install the package
-> `> pip install cellxgene`
-
-preprocess the data for use with cellxgene (optional)
-> `> cellxgene --prepare dataset.h5ad -o processed.h5ad`
-
-launch the web app
-> `> cellxgene --launch processed.h5ad`
+_cellxgene_ is an interactive data explorer for single-cell transcriptomics data. Whether you need to visualize one thousand cells or one million, _cellxgene_ helps you gain insight into your single-cell data.
 
 ## features
 
+#### Flexible selections, coloring, and differential expression of your selected sets of cells
+<img src="diffexp.gif" width="600"/>
 
+#### Single-gene analyses (e.g. expression analysis)
+<img src="customGene.gif" width="600" />
 
-### inspiration and collaboration
+## getting started
 
-We've been heavily inspired by several other related single-cell visualization projects:
-* [UCSC Cell Browswer](http://cells.ucsc.edu/)
-* [Cytoscape](http://www.cytoscape.org/)
-* [Xena](https://xena.ucsc.edu/)
-* [ASAP](https://asap.epfl.ch/)
-* [Gene Pattern](http://genepattern-notebook.org/)
+_cellxgene_ **only** supports Python 3.6. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
 
-We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.
+Install the package.
+``` bash
+pip install cellxgene
+```
 
-We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
+Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
 
-We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
+``` bash
+curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
+```
 
-### help and contact
+Launch _cellxgene_
+``` bash
+cellxgene launch pbmc3k.h5ad
+```
 
-Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cziscience-slack.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+## getting help
+
+We'd love to hear from you!
+
+For questions, suggestions, or accolades, [join the `#cellxgene-users` channel on the CZI Science Slack](https://join-cziscience-slack.herokuapp.com/) and say "hi!".
+
+For any errors, [report bugs on Github](https://github.com/chanzuckerberg/cellxgene/issues).

fbs/matrix.fbs

@@ -0,0 +1,73 @@
+
+/*
+
+  Flatbuffers schema for use in cellxgene wire-format.
+
+  Schema defines a general purpose, polymorphic, 2D matrix.  Data is
+  organized in a columnar layout.   Each column is homomorphic, and
+  several column types are supported:
+    - IEEE 32 and 64 bit floats
+    - signed and unsigned 32 bit integers
+    - JSON/UTF8 encoded array (for other types)
+
+  https://github.com/google/flatbuffers
+  http://google.github.io/flatbuffers/
+
+  NOTE: IF YOU MODIFY THIS FILE, YOU MUST RECOMPILE AND COMMIT
+  RESULTING FILES TO THE REPO:
+  * server/app/util/fbs/NetEncoding/*
+  * client/src/util/stateManager/matrix_generated.js
+
+*/
+
+namespace NetEncoding;
+
+table Float32Array {
+  data: [float32];
+}
+
+table Uint32Array {
+  data: [uint32];
+}
+
+table Int32Array {
+  data: [int32];
+}
+
+table Float64Array {
+  data: [float64];
+}
+
+table JSONEncodedArray {
+  // contains a UTF-8/JSON encoded array.  Used to store other
+  // types (or polymorphic arrays)
+  data: [uint8];
+}
+
+union TypedArray {
+  Float32Array,
+  Int32Array,
+  Uint32Array,
+  Float64Array,
+  JSONEncodedArray
+}
+
+// Extra level of indirection required because vector of union not yet supported
+table Column {
+  u: TypedArray;
+}
+
+// 2D matrix stored in columnar layout
+//
+table Matrix {
+  n_rows: uint32;     // all columns have this length
+  n_cols: uint32;     // same as columns.length
+  columns: [Column];  // length n_cols
+
+  // optional row and column index, with same length as corresponding dimension.
+  // If null, defaults to numeric index, ie, [0, n_rows) or [0, n_cols)
+  col_index: TypedArray;
+  row_index: TypedArray;
+}
+
+root_type Matrix;

makefile

@@ -0,0 +1,122 @@
+BUILDDIR := build
+CLIENTBUILD := $(BUILDDIR)/client
+SERVERBUILD := $(BUILDDIR)/server
+CLEANFILES :=  $(BUILDDIR)/ client/build dist cellxgene.egg-info
+
+PART ?= patch
+
+# BUILDING PACKAGE
+
+build : clean build-server
+	@echo "done"
+
+build-server : build-client
+	mkdir -p $(SERVERBUILD)
+	cp -r server/* $(SERVERBUILD)
+	cp -r client/build/  $(CLIENTBUILD)
+	mkdir -p $(SERVERBUILD)/app/web/static/img
+	cp $(CLIENTBUILD)/index.html $(SERVERBUILD)/app/web/templates/
+	cp -r $(CLIENTBUILD)/static $(SERVERBUILD)/app/web/
+	cp $(CLIENTBUILD)/favicon.png $(SERVERBUILD)/app/web/static/img
+	cp $(CLIENTBUILD)/service-worker.js $(SERVERBUILD)/app/web/static/js/
+	cp MANIFEST.in README.md setup.cfg setup.py $(BUILDDIR)
+
+build-client :
+	npm install --prefix client/ client
+	npm run  --prefix client build
+
+# If you are actively developing in the server folder use this, dirties the source tree
+build-for-server-dev : clean-server build-client
+	mkdir -p server/app/web/static/img
+	cp client/build/index.html server/app/web/templates/
+	cp -r client/build/static server/app/web/
+	cp client/build/favicon.png server/app/web/static/img
+	cp client/build/service-worker.js server/app/web/static/js/
+
+clean : clean-lite clean-server
+	rm -rf client/node_modules
+
+# cleaning node_modules is the longest one, so we avoid that if possible
+clean-lite :
+	rm -rf $(CLEANFILES)
+
+clean-server :
+	rm -f server/app/web/templates/index.html
+	rm -rf server/app/web/static
+
+.PHONY : build build-server build-client build-for-server-dev clean clean-lite clean-server
+
+# CREATING DISTRIBUTION RELEASE
+
+pydist : build
+	cd $(BUILDDIR); python setup.py sdist -d ../dist
+	@echo "done"
+
+.PHONY : pydist
+
+# RELEASE HELPERS
+
+# create new version to commit to master
+release-stage-1 : dev-env bump clean-lite gen-package-lock
+	@echo "Version bumped part:$(PART) and client built. Ready to commit and push"
+
+# build dist and release to dev pypi
+release-stage-2 : dev-env pydist twine
+	@echo "Dist built and uploaded to test.pypi.org"
+	@echo "Test the install `make install-release-test` and then upload to Pypi prod"
+	@echo "`make twine-prod`"
+
+release-stage-final: twine-prod
+	@echo "Release uploaded to pypi.org"
+
+# DANGER: releases directly to prod
+# use this if you accidently burned a test release version number,
+release-burned : dev-env pydist twine-prod
+	@echo "Dist built and uploaded to pypi.org"
+	@echo "Test the install `make install-release`"
+
+dev-env :
+	pip install -r server/requirements-dev.txt
+
+# give PART=[major, minor, part] as param to make bump
+bump :
+	bumpversion --config-file .bumpversion.cfg $(PART)
+
+twine :
+	twine upload --repository-url https://test.pypi.org/legacy/ dist/*
+
+twine-prod :
+	twine upload dist/*
+
+# quicker than re-building client
+gen-package-lock :
+	npm install --prefix client/ client
+
+.PHONY : release-stage-1 release-stage-2 release-stage-final release-burned dev-env bump twine twine-prod gen-package-lock
+
+# INSTALL
+
+# setup.py sucks when you have your library in a separate folder, adding these in to help setup envs
+
+# install from build directory
+install : uninstall
+	cd $(BUILDDIR); pip install -e .
+
+# install from source tree for development
+install-dev : uninstall
+	pip install -e .
+
+# install from test.pypi to test your release
+install-release-test : uninstall
+	pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
+	@echo "Installed cellxgene from test.pypi.org, now run and smoke test"
+
+# install from pypi to test your release
+install-release : uninstall
+	pip install cellxgene
+	@echo "Installed cellxgene from pypi.org"
+
+uninstall :
+	yes | pip uninstall cellxgene || true
+
+.PHONY : install install-dev install-release-test install-release uninstall

server/app/driver/driver.py

@@ -66,6 +66,11 @@ class CXGDriver(metaclass=ABCMeta):
         """
         pass
 
+    @abstractmethod
+    def annotation_to_fbs_matrix(self, axis, field=None):
+        """ Same as annotation(), except returns a flatbuffer, and does not support filtering. """
+        pass
+
     @abstractmethod
     def data_frame(self, filter, axis):
         """
@@ -79,6 +84,10 @@ class CXGDriver(metaclass=ABCMeta):
         """
         pass
 
+    @abstractmethod
+    def data_frame_to_fbs_matrix(self, filter, axis):
+        pass
+
     @abstractmethod
     def diffexp_topN(self, obsFilter1, obsFilter2, top_n=None, interactive_limit=None):
         """
@@ -104,3 +113,8 @@ class CXGDriver(metaclass=ABCMeta):
         :return:  [cellid, x, y, ...]
         """
         pass
+
+    @abstractmethod
+    def layout_to_fbs_matrix(self, filter):
+        """ same as layout, except returns a flatbuffer """
+        pass

server/app/rest_api/rest.py

@@ -6,11 +6,21 @@ from flask import Blueprint, current_app, jsonify, make_response, request
 from flask_restful_swagger_2 import Api, swagger, Resource
 from werkzeug.datastructures import ImmutableMultiDict
 
-from server.app.util.constants import Axis, DiffExpMode, JSON_NaN_to_num_warning_msg
+from server.app.util.constants import (
+    Axis,
+    DiffExpMode,
+    JSON_NaN_to_num_warning_msg,
+)
 from server.app.util.filter import parse_filter, QueryStringError
 from server.app.util.models import FilterModel
 from server.app.util.utils import get_mime_type
-from server.app.util.errors import MimeTypeError, FilterError, InteractiveError, PrepareError
+from server.app.util.errors import (
+    FilterError,
+    InteractiveError,
+    JSONEncodingValueError,
+    MimeTypeError,
+    PrepareError,
+)
 
 """
 Sort order for routes
@@ -32,7 +42,11 @@ class SchemaAPI(Resource):
                     "examples": {
                         "application/json": {
                             "schema": {
-                                "dataframe": {"nObs": 383, "nVar": 19944, "type": "float32"},
+                                "dataframe": {
+                                    "nObs": 383,
+                                    "nVar": 19944,
+                                    "type": "float32",
+                                },
                                 "annotations": {
                                     "obs": [
                                         {"name": "name", "type": "string"},
@@ -46,7 +60,10 @@ class SchemaAPI(Resource):
                                         },
                                         {"name": "QScore", "type": "float32"},
                                     ],
-                                    "var": [{"name": "name", "type": "string"}, {"name": "gene", "type": "string"}],
+                                    "var": [
+                                        {"name": "name", "type": "string"},
+                                        {"name": "gene", "type": "string"},
+                                    ],
                                 },
                             }
                         }
@@ -56,7 +73,9 @@ class SchemaAPI(Resource):
         }
     )
     def get(self):
-        return make_response(jsonify({"schema": current_app.data.schema}), HTTPStatus.OK)
+        return make_response(
+            jsonify({"schema": current_app.data.schema}), HTTPStatus.OK
+        )
 
 
 class ConfigAPI(Resource):
@@ -73,14 +92,22 @@ class ConfigAPI(Resource):
                         "application/json": {
                             "config": {
                                 "features": [
-                                    {"method": "POST", "path": "/cluster/", "available": False},
+                                    {
+                                        "method": "POST",
+                                        "path": "/cluster/",
+                                        "available": False,
+                                    },
                                     {
                                         "method": "POST",
                                         "path": "/layout/obs",
                                         "available": True,
                                         "interactiveLimit": 10000,
                                     },
-                                    {"method": "POST", "path": "/layout/var", "available": False},
+                                    {
+                                        "method": "POST",
+                                        "path": "/layout/var",
+                                        "available": False,
+                                    },
                                 ],
                                 "displayNames": {
                                     "engine": "ScanPy version 1.33",
@@ -97,16 +124,34 @@ class ConfigAPI(Resource):
         config = {
             "config": {
                 "features": [
-                    {"method": "POST", "path": "/cluster/", **current_app.data.features["cluster"]},
-                    {"method": "POST", "path": "/layout/obs", **current_app.data.features["layout"]["obs"]},
-                    {"method": "POST", "path": "/layout/var", **current_app.data.features["layout"]["var"]},
-                    {"method": "POST", "path": "/diffexp/", **current_app.data.features["diffexp"]},
+                    {
+                        "method": "POST",
+                        "path": "/cluster/",
+                        **current_app.data.features["cluster"],
+                    },
+                    {
+                        "method": "POST",
+                        "path": "/layout/obs",
+                        **current_app.data.features["layout"]["obs"],
+                    },
+                    {
+                        "method": "POST",
+                        "path": "/layout/var",
+                        **current_app.data.features["layout"]["var"],
+                    },
+                    {
+                        "method": "POST",
+                        "path": "/diffexp/",
+                        **current_app.data.features["diffexp"],
+                    },
                 ],
                 "displayNames": {
                     "engine": f"cellxgene Scanpy engine version {pkg_resources.get_distribution('cellxgene').version}",
                     "dataset": current_app.config["DATASET_TITLE"],
                 },
-                "parameters": {"max_category_items": current_app.data.max_category_items},
+                "parameters": {
+                    "max_category_items": current_app.data.max_category_items
+                },
             }
         }
         return make_response(jsonify(config), HTTPStatus.OK)
@@ -148,16 +193,29 @@ class AnnotationsObsAPI(Resource):
     )
     def get(self):
         fields = request.args.getlist("annotation-name", None)
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/json", "application/octet-stream"],
+            "application/json"
+        )
         try:
-            annotation_response = current_app.data.annotation({}, "obs", fields)
+            if preferred_mimetype == "application/json":
+                return make_response(
+                    current_app.data.annotation({}, "obs", fields), HTTPStatus.OK, {"Content-Type": "application/json"}
+                )
+            elif preferred_mimetype == "application/octet-stream":
+                return make_response(current_app.data.annotation_to_fbs_matrix("obs", fields),
+                                     HTTPStatus.OK,
+                                     {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        try:
-            return make_response(jsonify(annotation_response), HTTPStatus.OK)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc(
         {
@@ -170,7 +228,12 @@ class AnnotationsObsAPI(Resource):
                     "type": "string",
                     "description": "list of 1 or more annotation names",
                 },
-                {"name": "filter", "description": "Complex Filter", "in": "body", "schema": FilterModel},
+                {
+                    "name": "filter",
+                    "description": "Complex Filter",
+                    "in": "body",
+                    "schema": FilterModel,
+                },
             ],
             "responses": {
                 "200": {
@@ -196,17 +259,22 @@ class AnnotationsObsAPI(Resource):
     def put(self):
         fields = request.args.getlist("annotation-name", None)
         try:
-            annotation_response = current_app.data.annotation(request.get_json()["filter"], "obs", fields)
+            annotation_response = current_app.data.annotation(
+                request.get_json()["filter"], "obs", fields
+            )
+            return make_response(
+                annotation_response, HTTPStatus.OK, {"Content-Type": "application/json"}
+            )
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        try:
-            return make_response(jsonify(annotation_response), HTTPStatus.OK)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class AnnotationsVarAPI(Resource):
@@ -228,7 +296,11 @@ class AnnotationsVarAPI(Resource):
                     "examples": {
                         "application/json": {
                             "names": ["name", "category"],
-                            "data": [[0, "ATAD3C", 1], [1, "RER1", None], [49, "S100B", 6]],
+                            "data": [
+                                [0, "ATAD3C", 1],
+                                [1, "RER1", None],
+                                [49, "S100B", 6],
+                            ],
                         }
                     },
                 },
@@ -241,16 +313,29 @@ class AnnotationsVarAPI(Resource):
     )
     def get(self):
         fields = request.args.getlist("annotation-name", None)
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/json", "application/octet-stream"],
+            "application/json"
+        )
         try:
-            annotation_response = current_app.data.annotation({}, "var", fields)
+            if preferred_mimetype == "application/json":
+                return make_response(current_app.data.annotation({}, "var", fields),
+                                     HTTPStatus.OK,
+                                     {"Content-Type": "application/json"})
+            elif preferred_mimetype == "application/octet-stream":
+                return make_response(current_app.data.annotation_to_fbs_matrix("var", fields),
+                                     HTTPStatus.OK,
+                                     {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
-        try:
-            return make_response(jsonify(annotation_response), HTTPStatus.OK)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc(
         {
@@ -263,7 +348,12 @@ class AnnotationsVarAPI(Resource):
                     "type": "string",
                     "description": "list of 1 or more annotation names",
                 },
-                {"name": "filter", "description": "Complex Filter", "in": "body", "schema": FilterModel},
+                {
+                    "name": "filter",
+                    "description": "Complex Filter",
+                    "in": "body",
+                    "schema": FilterModel,
+                },
             ],
             "responses": {
                 "200": {
@@ -271,7 +361,11 @@ class AnnotationsVarAPI(Resource):
                     "examples": {
                         "application/json": {
                             "names": ["name", "category"],
-                            "data": [[0, "ATAD3C", 1], [1, "RER1", None], [49, "S100B", 6]],
+                            "data": [
+                                [0, "ATAD3C", 1],
+                                [1, "RER1", None],
+                                [49, "S100B", 6],
+                            ],
                         }
                     },
                 },
@@ -285,17 +379,22 @@ class AnnotationsVarAPI(Resource):
     def put(self):
         fields = request.args.getlist("annotation-name", None)
         try:
-            annotation_response = current_app.data.annotation(request.get_json()["filter"], "var", fields)
+            annotation_response = current_app.data.annotation(
+                request.get_json()["filter"], "var", fields
+            )
+            return make_response(
+                annotation_response, HTTPStatus.OK, {"Content-Type": "application/json"}
+            )
         except KeyError:
             return make_response(f"Error bad key in {fields}", HTTPStatus.BAD_REQUEST)
         except FilterError:
             return make_response("Malformed filter", HTTPStatus.BAD_REQUEST)
-        try:
-            return make_response(jsonify(annotation_response), HTTPStatus.OK)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DataObsAPI(Resource):
@@ -304,13 +403,28 @@ class DataObsAPI(Resource):
             "summary": "Get data (expression values) from the dataframe.",
             "tags": ["data"],
             "parameters": [
-                {"in": "query", "name": "filter", "type": "string", "description": "axis:key:value"},
-                {"in": "query", "name": "accept-type", "type": "string", "description": "MIME type"},
+                {
+                    "in": "query",
+                    "name": "filter",
+                    "type": "string",
+                    "description": "axis:key:value",
+                },
+                {
+                    "in": "query",
+                    "name": "accept-type",
+                    "type": "string",
+                    "description": "MIME type",
+                },
             ],
             "responses": {
                 "200": {
                     "description": "expression",
-                    "examples": {"application/json": {"var": [0, 20000], "obs": [[1, 39483, 3902, 203, 0, 0, 28]]}},
+                    "examples": {
+                        "application/json": {
+                            "var": [0, 20000],
+                            "obs": [[1, 39483, 3902, 203, 0, 0, 28]],
+                        }
+                    },
                 },
                 "400": {"description": "Malformed filter"},
                 "406": {"description": "Unacceptable MIME type"},
@@ -323,35 +437,57 @@ class DataObsAPI(Resource):
         args = request.args.copy()
         args.pop("accept-type", None)
         try:
-            filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema["annotations"])
+            filter_ = parse_filter(
+                ImmutableMultiDict(args), current_app.data.schema["annotations"]
+            )
         except QueryStringError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
         # TODO support CSV
         try:
             # TODO store mime_type when more than one is supported
             get_mime_type(
-                acceptable_types=["application/json"], query_param=accept_type, header=request.accept_mimetypes
+                acceptable_types=["application/json"],
+                query_param=accept_type,
+                header=request.accept_mimetypes,
             )
         except MimeTypeError as e:
             return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
         try:
-            return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.OBS))), HTTPStatus.OK)
+            return make_response(
+                current_app.data.data_frame(filter_, axis=Axis.OBS),
+                HTTPStatus.OK,
+                {"Content-Type": "application/json"},
+            )
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc(
         {
             "summary": "Get data (expression values) from the dataframe.",
             "tags": ["data"],
-            "parameters": [{"name": "filter", "description": "Complex Filter", "in": "body", "schema": FilterModel}],
+            "parameters": [
+                {
+                    "name": "filter",
+                    "description": "Complex Filter",
+                    "in": "body",
+                    "schema": FilterModel,
+                }
+            ],
             "responses": {
                 "200": {
                     "description": "expression",
-                    "examples": {"application/json": {"var": [0, 20000], "obs": [[1, 39483, 3902, 203, 0, 0, 28]]}},
+                    "examples": {
+                        "application/json": {
+                            "var": [0, 20000],
+                            "obs": [[1, 39483, 3902, 203, 0, 0, 28]],
+                        }
+                    },
                 },
                 "400": {"description": "Malformed filter"},
                 "406": {"description": "Unacceptable MIME type"},
@@ -360,21 +496,34 @@ class DataObsAPI(Resource):
     )
     def put(self):
         if not request.accept_mimetypes.best_match(["application/json", "text/csv"]):
-            return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
+            return make_response(
+                f"Unsupported MIME type '{request.accept_mimetypes}'",
+                HTTPStatus.NOT_ACCEPTABLE,
+            )
         try:
-            get_mime_type(acceptable_types=["application/json"], header=request.accept_mimetypes)
+            get_mime_type(
+                acceptable_types=["application/json"], header=request.accept_mimetypes
+            )
         except MimeTypeError as e:
             return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
         try:
             return make_response(
-                (jsonify(current_app.data.data_frame(request.get_json()["filter"], axis=Axis.OBS))), HTTPStatus.OK
+                (
+                    current_app.data.data_frame(
+                        request.get_json()["filter"], axis=Axis.OBS
+                    )
+                ),
+                HTTPStatus.OK,
+                {"Content-Type": "application/json"},
             )
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DataVarAPI(Resource):
@@ -383,13 +532,28 @@ class DataVarAPI(Resource):
             "summary": "Get data (expression values) from the dataframe.",
             "tags": ["data"],
             "parameters": [
-                {"in": "query", "name": "filter", "type": "string", "description": "axis:key:value"},
-                {"in": "query", "name": "accept-type", "type": "string", "description": "MIME type"},
+                {
+                    "in": "query",
+                    "name": "filter",
+                    "type": "string",
+                    "description": "axis:key:value",
+                },
+                {
+                    "in": "query",
+                    "name": "accept-type",
+                    "type": "string",
+                    "description": "MIME type",
+                },
             ],
             "responses": {
                 "200": {
                     "description": "expression",
-                    "examples": {"application/json": {"obs": [0, 20000], "var": [[1, 39483, 3902, 203, 0, 0, 28]]}},
+                    "examples": {
+                        "application/json": {
+                            "obs": [0, 20000],
+                            "var": [[1, 39483, 3902, 203, 0, 0, 28]],
+                        }
+                    },
                 },
                 "400": {"description": "Malformed filter"},
                 "406": {"description": "Unacceptable MIME type"},
@@ -402,33 +566,55 @@ class DataVarAPI(Resource):
         args = request.args.copy()
         args.pop("accept-type", None)
         try:
-            filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema["annotations"])
+            filter_ = parse_filter(
+                ImmutableMultiDict(args), current_app.data.schema["annotations"]
+            )
         except QueryStringError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
         try:
             get_mime_type(
-                acceptable_types=["application/json"], query_param=accept_type, header=request.accept_mimetypes
+                acceptable_types=["application/json"],
+                query_param=accept_type,
+                header=request.accept_mimetypes,
             )
         except MimeTypeError as e:
             return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
         try:
-            return make_response((jsonify(current_app.data.data_frame(filter_, axis=Axis.VAR))), HTTPStatus.OK)
+            return make_response(
+                current_app.data.data_frame(filter_, axis=Axis.VAR),
+                HTTPStatus.OK,
+                {"Content-Type": "application/json"},
+            )
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     @swagger.doc(
         {
             "summary": "Get data (expression values) from the dataframe.",
             "tags": ["data"],
-            "parameters": [{"name": "filter", "description": "Complex Filter", "in": "body", "schema": FilterModel}],
+            "parameters": [
+                {
+                    "name": "filter",
+                    "description": "Complex Filter",
+                    "in": "body",
+                    "schema": FilterModel,
+                }
+            ],
             "responses": {
                 "200": {
                     "description": "expression",
-                    "examples": {"application/json": {"obs": [0, 20000], "var": [[1, 39483, 3902, 203, 0, 0, 28]]}},
+                    "examples": {
+                        "application/json": {
+                            "obs": [0, 20000],
+                            "var": [[1, 39483, 3902, 203, 0, 0, 28]],
+                        }
+                    },
                 },
                 "400": {"description": "Malformed filter"},
                 "406": {"description": "Unacceptable MIME type"},
@@ -436,23 +622,38 @@ class DataVarAPI(Resource):
         }
     )
     def put(self):
-        if not request.accept_mimetypes.best_match(["application/json", "text/csv"]):
-            return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
-        # TODO support CSV
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/json", "application/octet-stream"],
+            "application/json"
+        )
         try:
-            get_mime_type(acceptable_types=["application/json"], header=request.accept_mimetypes)
-        except MimeTypeError as e:
-            return make_response(e.message, HTTPStatus.NOT_ACCEPTABLE)
-        try:
-            return make_response(
-                (jsonify(current_app.data.data_frame(request.get_json()["filter"], axis=Axis.VAR))), HTTPStatus.OK
-            )
+            if preferred_mimetype == "application/json":
+                return make_response(
+                    (
+                        current_app.data.data_frame(
+                            request.get_json()["filter"], axis=Axis.VAR
+                        )
+                    ),
+                    HTTPStatus.OK,
+                    {"Content-Type": "application/json"},
+                )
+            elif preferred_mimetype == "application/octet-stream":
+                return make_response(
+                    current_app.data.data_frame_to_fbs_matrix(
+                        request.get_json()["filter"], axis=Axis.VAR
+                    ),
+                    HTTPStatus.OK,
+                    {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except FilterError as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class DiffExpObsAPI(Resource):
@@ -522,23 +723,35 @@ class DiffExpObsAPI(Resource):
         except KeyError:
             return make_response("Error: mode is required", HTTPStatus.BAD_REQUEST)
         except ValueError:
-            return make_response(f"Error: invalid mode option {args['mode']}", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                f"Error: invalid mode option {args['mode']}", HTTPStatus.BAD_REQUEST
+            )
         # Validate filters
         if mode == DiffExpMode.VAR_FILTER or "varFilter" in args:
             # not NOT_IMPLEMENTED
-            return make_response("mode=varfilter not implemented", HTTPStatus.NOT_IMPLEMENTED)
+            return make_response(
+                "mode=varfilter not implemented", HTTPStatus.NOT_IMPLEMENTED
+            )
         if mode == DiffExpMode.TOP_N and "count" not in args:
-            return make_response("mode=topN requires a count parameter", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                "mode=topN requires a count parameter", HTTPStatus.BAD_REQUEST
+            )
 
         if "set1" not in args:
             return make_response("set1 is required.", HTTPStatus.BAD_REQUEST)
         if Axis.VAR in args["set1"]["filter"]:
-            return make_response("Var filter not allowed for set1", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                "Var filter not allowed for set1", HTTPStatus.BAD_REQUEST
+            )
         # set2
         if "set2" not in args:
-            return make_response("Set2 as inverse of set1 is not implemented", HTTPStatus.NOT_IMPLEMENTED)
+            return make_response(
+                "Set2 as inverse of set1 is not implemented", HTTPStatus.NOT_IMPLEMENTED
+            )
         if Axis.VAR in args["set2"]["filter"]:
-            return make_response("Var filter not allowed for set2", HTTPStatus.BAD_REQUEST)
+            return make_response(
+                "Var filter not allowed for set2", HTTPStatus.BAD_REQUEST
+            )
 
         set1_filter = args["set1"]["filter"]
         set2_filter = args.get("set2", {"filter": {}})["filter"]
@@ -549,18 +762,24 @@ class DiffExpObsAPI(Resource):
         count = args.get("count", None)
         try:
             diffexp = current_app.data.diffexp_topN(
-                set1_filter, set2_filter, count, current_app.data.features["diffexp"]["interactiveLimit"]
+                set1_filter,
+                set2_filter,
+                count,
+                current_app.data.features["diffexp"]["interactiveLimit"],
+            )
+            return make_response(
+                diffexp, HTTPStatus.OK, {"Content-Type": "application/json"}
             )
         except (ValueError, FilterError) as e:
             return make_response(e.message, HTTPStatus.BAD_REQUEST)
         except InteractiveError:
             return make_response("Non-interactive request", HTTPStatus.FORBIDDEN)
-        try:
-            return make_response(jsonify(diffexp), HTTPStatus.OK)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
 
 class LayoutObsAPI(Resource):
@@ -576,7 +795,10 @@ class LayoutObsAPI(Resource):
                         "application/json": {
                             "layout": {
                                 "ndims": 2,
-                                "coordinates": [[0, 0.284_483, 0.983_744], [1, 0.038_844, 0.739_444]],
+                                "coordinates": [
+                                    [0, 0.284_483, 0.983_744],
+                                    [1, 0.038_844, 0.739_444],
+                                ],
                             }
                         }
                     },
@@ -586,16 +808,28 @@ class LayoutObsAPI(Resource):
         }
     )
     def get(self):
+        preferred_mimetype = request.accept_mimetypes.best_match(
+            ["application/json", "application/octet-stream"],
+            "application/json"
+        )
         try:
-            layout = current_app.data.layout({})
+            if preferred_mimetype == "application/json":
+                return make_response(current_app.data.layout({}), HTTPStatus.OK, {"Content-Type": "application/json"})
+
+            elif preferred_mimetype == "application/octet-stream":
+                return make_response(current_app.data.layout_to_fbs_matrix(),
+                                     HTTPStatus.OK,
+                                     {"Content-Type": "application/octet-stream"})
+            else:
+                return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
         except PrepareError as e:
             return make_response(e.message, HTTPStatus.INTERNAL_SERVER_ERROR)
-        try:
-            return make_response((jsonify({"layout": layout})), HTTPStatus.OK)
-        except ValueError as e:
+        except JSONEncodingValueError as e:
             # JSON encoding failure, usually due to bad data
             warnings.warn(JSON_NaN_to_num_warning_msg)
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+        except ValueError as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
     # @swagger.doc({
     #     "summary": "Observation layout for filtered subset.",
@@ -636,7 +870,7 @@ class LayoutObsAPI(Resource):
     #         filter = request.get_json()["filter"]
     #         interactive_limit = current_app.data.features["layout"]["obs"]["interactiveLimit"]
     #         layout = current_app.data.layout(filter, interactive_limit=interactive_limit)
-    #         return make_response(jsonify({"layout": layout}), HTTPStatus.OK)
+    #         return make_response(layout, HTTPStatus.OK, {"Content-Type": content_type})
     #     except FilterError as e:
     #         return make_response(e.message, HTTPStatus.BAD_REQUEST)
     #     except InteractiveError:

server/app/scanpy_engine/diffexp.py

@@ -9,18 +9,31 @@ def _mean_var_n(X):
     than naive methods (and same method used by numpy.var())
     https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass
     """
-    n = X.shape[0]
-    if sparse.issparse(X):
-        mean = X.mean(axis=0).A1
-        dfm = X - mean
-        sumsq = np.sum(np.multiply(dfm, dfm), axis=0).A1
-        v = sumsq / (n - 1)
-    else:
-        mean = X.mean(axis=0)
-        dfm = X - mean
-        sumsq = np.sum(np.multiply(dfm, dfm), axis=0)
-        v = sumsq / (n - 1)
+    # fp_err_occurred is a flag indicating that a floating point error
+    # occured somewhere in our compute.  Used to trigger non-finite
+    # number handling.
+    fp_err_occurred = False
 
+    def fp_err_set(err, flag):
+        nonlocal fp_err_occurred
+        fp_err_occurred = True
+
+    with np.errstate(divide="call", invalid="call", call=fp_err_set):
+        n = X.shape[0]
+        if sparse.issparse(X):
+            mean = X.mean(axis=0).A1
+            dfm = X - mean
+            sumsq = np.sum(np.multiply(dfm, dfm), axis=0).A1
+            v = sumsq / (n - 1)
+        else:
+            mean = X.mean(axis=0)
+            dfm = X - mean
+            sumsq = np.sum(np.multiply(dfm, dfm), axis=0)
+            v = sumsq / (n - 1)
+
+    if fp_err_occurred:
+        mean[np.isfinite(mean) == False] = 0    # noqa: E712
+        v[np.isfinite(v) == False] = 0          # noqa: E712
     return mean, v, n
 
 

server/app/scanpy_engine/scanpy_engine.py

@@ -8,8 +8,16 @@ from scipy import sparse
 
 from server.app.driver.driver import CXGDriver
 from server.app.util.constants import Axis, DEFAULT_TOP_N
-from server.app.util.errors import FilterError, InteractiveError, PrepareError, ScanpyFileError
+from server.app.util.errors import (
+    FilterError,
+    InteractiveError,
+    JSONEncodingValueError,
+    PrepareError,
+    ScanpyFileError,
+)
+from server.app.util.utils import jsonify_scanpy
 from server.app.scanpy_engine.diffexp import diffexp_ttest
+from server.app.util.fbs.matrix import encode_matrix_fbs
 
 """
 Sort order for methods
@@ -34,10 +42,6 @@ class ScanpyEngine(CXGDriver):
         self.diffexp_options = ["ttest"]
         self._create_schema()
 
-        # TODO: temporary work-arounds
-        if args["nan_to_num"]:
-            self._IEEE754_special_values_workaround()
-
     def _alias_annotation_names(self, axis, name):
         """
         Do all user-specified annotation aliasing.
@@ -58,22 +62,29 @@ class ScanpyEngine(CXGDriver):
             df_axis.rename(inplace=True, columns={"index": "name"})
         elif name in df_axis.columns:
             if name not in df_axis.columns:
-                raise KeyError(f"Annotation name {name}, specified in --{ax_name}-name does not exist.")
+                raise KeyError(
+                    f"Annotation name {name}, specified in --{ax_name}-name does not exist."
+                )
             if not df_axis[name].is_unique:
                 raise KeyError(
-                    f"Values in -{ax_name}-name must be unique. " "Please prepare data to contain unique values."
+                    f"Values in -{ax_name}-name must be unique. "
+                    "Please prepare data to contain unique values."
                 )
             # reset index to simple range; alias user-specified annotation to "name"
             df_axis.reset_index(drop=True, inplace=True)
             df_axis.rename(inplace=True, columns={name: "name"})
         else:
-            raise KeyError(f"Annotation name {name}, specified in --{ax_name}_name does not exist.")
+            raise KeyError(
+                f"Annotation name {name}, specified in --{ax_name}_name does not exist."
+            )
 
     @staticmethod
     def _can_cast_to_float32(ann):
         if ann.dtype.kind == "f":
             if not np.can_cast(ann.dtype, np.float32):
-                warnings.warn(f"Annotation {ann.name} will be converted to 32 bit float and may lose precision.")
+                warnings.warn(
+                    f"Annotation {ann.name} will be converted to 32 bit float and may lose precision."
+                )
             return True
         return False
 
@@ -89,7 +100,11 @@ class ScanpyEngine(CXGDriver):
 
     def _create_schema(self):
         self.schema = {
-            "dataframe": {"nObs": self.cell_count, "nVar": self.gene_count, "type": str(self.data.X.dtype)},
+            "dataframe": {
+                "nObs": self.cell_count,
+                "nVar": self.gene_count,
+                "type": str(self.data.X.dtype),
+            },
             "annotations": {"obs": [], "var": []},
         }
         for ax in Axis:
@@ -111,7 +126,9 @@ class ScanpyEngine(CXGDriver):
                     ann_schema["type"] = "categorical"
                     ann_schema["categories"] = curr_axis[ann].dtype.categories.tolist()
                 else:
-                    raise TypeError(f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene.")
+                    raise TypeError(
+                        f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene."
+                    )
                 self.schema["annotations"][ax].append(ann_schema)
 
     @staticmethod
@@ -139,13 +156,19 @@ class ScanpyEngine(CXGDriver):
     def _validate_data_types(self):
         if self.data.X.dtype != "float32":
             warnings.warn(
-                f"Scanpy data matrix is in {self.data.X.dtype} format not float32. " f"Precision may be truncated."
+                f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
+                f"Precision may be truncated."
             )
         for ax in Axis:
             curr_axis = getattr(self.data, str(ax))
             for ann in curr_axis:
                 datatype = curr_axis[ann].dtype
-                downcast_map = {"int64": "int32", "uint32": "int32", "uint64": "int32", "float64": "float32"}
+                downcast_map = {
+                    "int64": "int32",
+                    "uint32": "int32",
+                    "uint64": "int32",
+                    "float64": "float32",
+                }
                 if datatype in downcast_map:
                     warnings.warn(
                         f"Scanpy annotation {ax}:{ann} is in unsupported format: {datatype}. "
@@ -174,66 +197,6 @@ class ScanpyEngine(CXGDriver):
                 f"to solve this problem. "
             )
 
-    def _IEEE754_special_values_workaround(self):
-        """
-        TODO: temporary workaround
-
-        Because all floating point data is serialized to JSON, and JSON has no means of representing
-        non-finite, floating point special values (NaN, +/-Infinity, etc), we include this temporary
-        work-around.
-
-        This will likely be removed in the future, contingent upon improved marshalling.
-
-        Where non-finite floating point is present in obs, var or X:
-        * issue a warning to the user that these values will be convert to finite numbers.
-        * set NaN to zero, and Infinities to min/max of the element.
-        """
-
-        # annotations
-        for ax in Axis:
-            curr_axis = getattr(self.data, str(ax))
-            for ann in curr_axis:
-                dtype = curr_axis[ann].dtype
-                if dtype.kind == "f":
-                    finite_idx = np.isfinite(curr_axis[ann])
-                    if not finite_idx.all():
-                        curr_axis.loc[np.isnan(curr_axis[ann]), ann] = 0
-                        curr_axis.loc[np.isneginf(curr_axis[ann]), ann] = curr_axis[ann][finite_idx].min()
-                        curr_axis.loc[np.isposinf(curr_axis[ann]), ann] = curr_axis[ann][finite_idx].max()
-                        warnings.warn(
-                            f"{str(ax).title()} annotation '{ann}' contains floating point NaN or Infinities. "
-                            f"These will be converted to finite values."
-                        )
-
-        # X
-        non_finite_X_found = False
-        if sparse.issparse(self.data._X):
-            coo = self.data._X.tocoo()
-            finite_idx = np.isfinite(coo.data)
-            if not finite_idx.all():
-                non_finite_X_found = True
-                coo.data[np.isnan(coo.data)] = 0
-                coo.data[np.isneginf(coo.data)] = np.min(coo.data[finite_idx])
-                coo.data[np.isposinf(coo.data)] = np.max(coo.data[finite_idx])
-                coo.eliminate_zeros()
-                _X = coo.asformat(self.data._X.getformat())
-                self.data._X = _X
-        else:
-            _X = self.data._X
-            finite_idx = np.isfinite(_X.flat)
-            if not finite_idx.all():
-                non_finite_X_found = True
-                min_X = _X.flat[finite_idx].min()
-                max_X = _X.flat[finite_idx].max()
-                _X[np.isnan(_X)] = 0
-                _X[np.isneginf(_X)] = min_X
-                _X[np.isposinf(_X)] = max_X
-
-        if non_finite_X_found:
-            warnings.warn(
-                "Dataframe X contains floating point NaN or Infinities. " "These will be converted to finite values."
-            )
-
     def filter_dataframe(self, filter):
         """
          Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
@@ -283,10 +246,15 @@ class ScanpyEngine(CXGDriver):
     def _axis_filter_to_mask(filter, d_axis, count):
         mask = np.ones((count,), dtype=bool)
         if "index" in filter:
-            mask = np.logical_and(mask, ScanpyEngine._index_filter_to_mask(filter["index"], count))
+            mask = np.logical_and(
+                mask, ScanpyEngine._index_filter_to_mask(filter["index"], count)
+            )
         if "annotation_value" in filter:
             mask = np.logical_and(
-                mask, ScanpyEngine._annotation_filter_to_mask(filter["annotation_value"], d_axis, count)
+                mask,
+                ScanpyEngine._annotation_filter_to_mask(
+                    filter["annotation_value"], d_axis, count
+                ),
             )
         return mask
 
@@ -300,9 +268,13 @@ class ScanpyEngine(CXGDriver):
 
         if filter is not None:
             if Axis.OBS in filter:
-                obs_selector = self._axis_filter_to_mask(filter["obs"], self.data.obs, self.data.n_obs)
+                obs_selector = self._axis_filter_to_mask(
+                    filter["obs"], self.data.obs, self.data.n_obs
+                )
             if Axis.VAR in filter:
-                var_selector = self._axis_filter_to_mask(filter["var"], self.data.var, self.data.n_vars)
+                var_selector = self._axis_filter_to_mask(
+                    filter["var"], self.data.var, self.data.n_vars
+                )
         return obs_selector, var_selector
 
     @staticmethod
@@ -318,7 +290,9 @@ class ScanpyEngine(CXGDriver):
         https://docs.scipy.org/doc/scipy/reference/sparse.html
         """
         prefer_row_access = (
-            sparse.isspmatrix_csr(data._X) or sparse.isspmatrix_lil(data._X) or sparse.isspmatrix_bsr(data._X)
+            sparse.isspmatrix_csr(data._X)
+            or sparse.isspmatrix_lil(data._X)
+            or sparse.isspmatrix_bsr(data._X)
         )
         if prefer_row_access:
             # Row-major slicing
@@ -352,13 +326,31 @@ class ScanpyEngine(CXGDriver):
             obs = self.data.obs[obs_selector]
             if not fields:
                 fields = obs.columns.tolist()
-            result = {"names": fields, "data": DataFrame(obs[fields]).to_records(index=True).tolist()}
+            result = {
+                "names": fields,
+                "data": DataFrame(obs[fields]).to_records(index=True).tolist(),
+            }
         else:
             var = self.data.var[var_selector]
             if not fields:
                 fields = var.columns.tolist()
-            result = {"names": fields, "data": DataFrame(var[fields]).to_records(index=True).tolist()}
-        return result
+            result = {
+                "names": fields,
+                "data": DataFrame(var[fields]).to_records(index=True).tolist(),
+            }
+        try:
+            return jsonify_scanpy(result)
+        except ValueError:
+            raise JSONEncodingValueError("Error encoding annotations to JSON")
+
+    def annotation_to_fbs_matrix(self, axis, fields=None):
+        if axis == Axis.OBS:
+            df = self.data.obs
+        else:
+            df = self.data.var
+        if fields is not None and len(fields) > 0:
+            df = df[fields]
+        return encode_matrix_fbs(df, col_idx=df.columns)
 
     def data_frame(self, filter, axis):
         """
@@ -382,27 +374,71 @@ class ScanpyEngine(CXGDriver):
         if axis == Axis.OBS:
             result = {
                 "var": var_index_sliced.tolist(),
-                "obs": DataFrame(_X, index=obs_index_sliced).to_records(index=True).tolist(),
+                "obs": DataFrame(_X, index=obs_index_sliced)
+                .to_records(index=True)
+                .tolist(),
             }
         else:
             result = {
                 "obs": obs_index_sliced.tolist(),
-                "var": DataFrame(_X.T, index=var_index_sliced).to_records(index=True).tolist(),
+                "var": DataFrame(_X.T, index=var_index_sliced)
+                .to_records(index=True)
+                .tolist(),
             }
-        return result
+        try:
+            return jsonify_scanpy(result)
+        except ValueError:
+            raise JSONEncodingValueError("Error encoding dataframe to JSON")
+
+    def data_frame_to_fbs_matrix(self, filter, axis):
+        """
+        Retrieves data 'X' and returns in a flatbuffer Matrix.
+        :param filter: filter: dictionary with filter params
+        :param axis: string obs or var
+        :return: flatbuffer Matrix
+
+        Caveats:
+        * currently only supports access on VAR axis
+        * currently only supports filtering on VAR axis
+        """
+        if axis != Axis.VAR:
+            raise ValueError("Only VAR dimension access is supported")
+        try:
+            obs_selector, var_selector = self._filter_to_mask(filter, use_slices=False)
+        except (KeyError, IndexError) as e:
+            raise FilterError(f"Error parsing filter: {e}") from e
+        if obs_selector is not None:
+            raise FilterError("filtering on obs unsupported")
+
+        # Currently only handles VAR dimension
+        X = self.data._X
+        if var_selector is not None:
+            X = X[:, var_selector]
+        return encode_matrix_fbs(X, col_idx=np.nonzero(var_selector)[0], row_idx=None)
 
     def diffexp_topN(self, obsFilterA, obsFilterB, top_n=None, interactive_limit=None):
         if Axis.VAR in obsFilterA or Axis.VAR in obsFilterB:
             raise FilterError("Observation filters may not contain vaiable conditions")
         try:
-            obs_mask_A = self._axis_filter_to_mask(obsFilterA["obs"], self.data.obs, self.data.n_obs)
-            obs_mask_B = self._axis_filter_to_mask(obsFilterB["obs"], self.data.obs, self.data.n_obs)
+            obs_mask_A = self._axis_filter_to_mask(
+                obsFilterA["obs"], self.data.obs, self.data.n_obs
+            )
+            obs_mask_B = self._axis_filter_to_mask(
+                obsFilterB["obs"], self.data.obs, self.data.n_obs
+            )
         except (KeyError, IndexError) as e:
             raise FilterError(f"Error parsing filter: {e}") from e
         if top_n is None:
             top_n = DEFAULT_TOP_N
-        result = diffexp_ttest(self.data, obs_mask_A, obs_mask_B, top_n, self.diffexp_lfc_cutoff)
-        return result
+        result = diffexp_ttest(
+            self.data, obs_mask_A, obs_mask_B, top_n, self.diffexp_lfc_cutoff
+        )
+        try:
+            return jsonify_scanpy(result)
+        except ValueError:
+            raise JSONEncodingValueError(
+                "Error encoding differential expression to JSON"
+            )
 
     def layout(self, filter, interactive_limit=None):
         """
@@ -431,6 +467,36 @@ class ScanpyEngine(CXGDriver):
                 f"please prepare your datafile and relaunch cellxgene"
             ) from e
         normalized_layout = DataFrame(
-            (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min()), index=df.obs.index
+            (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min()),
+            index=df.obs.index,
         )
-        return {"ndims": normalized_layout.shape[1], "coordinates": normalized_layout.to_records(index=True).tolist()}
+        try:
+            return jsonify_scanpy(
+                {
+                    "layout": {
+                        "ndims": normalized_layout.shape[1],
+                        "coordinates": normalized_layout.to_records(
+                            index=True
+                        ).tolist(),
+                    }
+                }
+            )
+        except ValueError:
+            raise JSONEncodingValueError("Error encoding layout to JSON")
+
+    def layout_to_fbs_matrix(self):
+        """
+        Return the default 2-D layout for cells as a FBS Matrix.
+
+        Caveats:
+        * does not support filtering
+        * only returns Matrix in columnar layout
+        """
+        try:
+            df_layout = self.data.obsm[f"X_{self.layout_method}"]
+        except ValueError as e:
+            raise PrepareError(
+                f"Layout has not been calculated using {self.layout_method}, "
+                f"please prepare your datafile and relaunch cellxgene") from e
+        normalized_layout = (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min())
+        return encode_matrix_fbs(normalized_layout.astype(dtype=np.float32), col_idx=None, row_idx=None)

server/app/util/constants.py

@@ -27,4 +27,6 @@ class DiffExpMode(AugmentedEnum):
     VAR_FILTER = "varFilter"
 
 
-JSON_NaN_to_num_warning_msg = "JSON encoding failure - suggest trying --nan-to-num command line option"
+JSON_NaN_to_num_warning_msg = (
+    "JSON encoding failure - please verify all data are finite values (no NaN or Infinities)"
+)

server/app/util/errors.py

@@ -16,6 +16,15 @@ class InteractiveError(Exception):
         self.message = message
 
 
+class JSONEncodingValueError(Exception):
+    """
+    Raised when file loaded into scanpy is misformatted
+    """
+
+    def __init__(self, message):
+        self.message = message
+
+
 class MimeTypeError(Exception):
     """
     Raised when incompatible MIME type selected

server/app/util/fbs/NetEncoding/Column.py

@@ -0,0 +1,41 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Column(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsColumn(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Column()
+        x.Init(buf, n + offset)
+        return x
+
+    # Column
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Column
+    def UType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Column
+    def U(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+def ColumnStart(builder): builder.StartObject(2)
+def ColumnAddUType(builder, uType): builder.PrependUint8Slot(0, uType, 0)
+def ColumnAddU(builder, u): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(u), 0)
+def ColumnEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Float32Array.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Float32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsFloat32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Float32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Float32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Float32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Float32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Float32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Float32Flags, o)
+        return 0
+
+    # Float32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Float32ArrayStart(builder): builder.StartObject(1)
+def Float32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Float32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Float32ArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Float64Array.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Float64Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsFloat64Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Float64Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Float64Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Float64Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Float64Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 8))
+        return 0
+
+    # Float64Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Float64Flags, o)
+        return 0
+
+    # Float64Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Float64ArrayStart(builder): builder.StartObject(1)
+def Float64ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Float64ArrayStartDataVector(builder, numElems): return builder.StartVector(8, numElems, 8)
+def Float64ArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Int32Array.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Int32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsInt32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Int32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Int32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Int32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Int32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Int32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Int32Flags, o)
+        return 0
+
+    # Int32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Int32ArrayStart(builder): builder.StartObject(1)
+def Int32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Int32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Int32ArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/JSONEncodedArray.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class JSONEncodedArray(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsJSONEncodedArray(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = JSONEncodedArray()
+        x.Init(buf, n + offset)
+        return x
+
+    # JSONEncodedArray
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # JSONEncodedArray
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 1))
+        return 0
+
+    # JSONEncodedArray
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Uint8Flags, o)
+        return 0
+
+    # JSONEncodedArray
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def JSONEncodedArrayStart(builder): builder.StartObject(1)
+def JSONEncodedArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def JSONEncodedArrayStartDataVector(builder, numElems): return builder.StartVector(1, numElems, 1)
+def JSONEncodedArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/Matrix.py

@@ -0,0 +1,98 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Matrix(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsMatrix(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Matrix()
+        x.Init(buf, n + offset)
+        return x
+
+    # Matrix
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Matrix
+    def NRows(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def NCols(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def Columns(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8))
+        if o != 0:
+            x = self._tab.Vector(o)
+            x += flatbuffers.number_types.UOffsetTFlags.py_type(j) * 4
+            x = self._tab.Indirect(x)
+            from .Column import Column
+            obj = Column()
+            obj.Init(self._tab.Bytes, x)
+            return obj
+        return None
+
+    # Matrix
+    def ColumnsLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+    # Matrix
+    def ColIndexType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(10))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def ColIndex(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+    # Matrix
+    def RowIndexType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(14))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos)
+        return 0
+
+    # Matrix
+    def RowIndex(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(16))
+        if o != 0:
+            from flatbuffers.table import Table
+            obj = Table(bytearray(), 0)
+            self._tab.Union(obj, o)
+            return obj
+        return None
+
+def MatrixStart(builder): builder.StartObject(7)
+def MatrixAddNRows(builder, nRows): builder.PrependUint32Slot(0, nRows, 0)
+def MatrixAddNCols(builder, nCols): builder.PrependUint32Slot(1, nCols, 0)
+def MatrixAddColumns(builder, columns): builder.PrependUOffsetTRelativeSlot(2, flatbuffers.number_types.UOffsetTFlags.py_type(columns), 0)
+def MatrixStartColumnsVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def MatrixAddColIndexType(builder, colIndexType): builder.PrependUint8Slot(3, colIndexType, 0)
+def MatrixAddColIndex(builder, colIndex): builder.PrependUOffsetTRelativeSlot(4, flatbuffers.number_types.UOffsetTFlags.py_type(colIndex), 0)
+def MatrixAddRowIndexType(builder, rowIndexType): builder.PrependUint8Slot(5, rowIndexType, 0)
+def MatrixAddRowIndex(builder, rowIndex): builder.PrependUOffsetTRelativeSlot(6, flatbuffers.number_types.UOffsetTFlags.py_type(rowIndex), 0)
+def MatrixEnd(builder): return builder.EndObject()

server/app/util/fbs/NetEncoding/TypedArray.py

@@ -0,0 +1,12 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+class TypedArray(object):
+    NONE = 0
+    Float32Array = 1
+    Int32Array = 2
+    Uint32Array = 3
+    Float64Array = 4
+    JSONEncodedArray = 5
+

server/app/util/fbs/NetEncoding/Uint32Array.py

@@ -0,0 +1,46 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: NetEncoding
+
+import flatbuffers
+
+class Uint32Array(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAsUint32Array(cls, buf, offset):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = Uint32Array()
+        x.Init(buf, n + offset)
+        return x
+
+    # Uint32Array
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Uint32Array
+    def Data(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            a = self._tab.Vector(o)
+            return self._tab.Get(flatbuffers.number_types.Uint32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4))
+        return 0
+
+    # Uint32Array
+    def DataAsNumpy(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Uint32Flags, o)
+        return 0
+
+    # Uint32Array
+    def DataLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+def Uint32ArrayStart(builder): builder.StartObject(1)
+def Uint32ArrayAddData(builder, data): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0)
+def Uint32ArrayStartDataVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def Uint32ArrayEnd(builder): return builder.EndObject()

server/app/util/fbs/matrix.py

@@ -0,0 +1,207 @@
+import flatbuffers
+import numpy as np
+from scipy import sparse
+import pandas as pd
+
+import server.app.util.fbs.NetEncoding.Column as Column
+import server.app.util.fbs.NetEncoding.TypedArray as TypedArray
+import server.app.util.fbs.NetEncoding.Matrix as Matrix
+
+
+# Placeholder until recent enhancements to flatbuffers Python
+# runtime are released, at which point we can use the default
+# version.  This code is a port of the head.  See:
+#
+#       https://github.com/google/flatbuffers/pull/4829
+#
+def CreateNumpyVector(builder, x):
+    """CreateNumpyVector writes a numpy array into the buffer."""
+
+    if not isinstance(x, np.ndarray):
+        raise TypeError("non-numpy-ndarray passed to CreateNumpyVector")
+
+    if x.dtype.kind not in ['b', 'i', 'u', 'f']:
+        raise TypeError("numpy-ndarray holds elements of unsupported datatype")
+
+    if x.ndim > 1:
+        raise TypeError("multidimensional-ndarray passed to CreateNumpyVector")
+
+    builder.StartVector(x.itemsize, x.size, x.dtype.alignment)
+
+    # Ensure little endian byte ordering
+    if x.dtype.str[0] == "<":
+        x_little_endian = x
+    else:
+        x_little_endian = x.byteswap(inplace=False)
+
+    # Calculate total length
+    len = int(x_little_endian.itemsize * x_little_endian.size)
+    builder.head = int(builder.Head() - len)
+
+    # tobytes ensures c_contiguous ordering
+    builder.Bytes[builder.Head():builder.Head() + len] = x_little_endian.tobytes(order='C')
+
+    return builder.EndVector(x.size)
+
+
+# Serialization helper
+def serialize_column(builder, typed_arr):
+    """ Serialize NetEncoding.Column """
+    (u_type, u_value) = typed_arr
+    Column.ColumnStart(builder)
+    Column.ColumnAddUType(builder, u_type)
+    Column.ColumnAddU(builder, u_value)
+    return Column.ColumnEnd(builder)
+
+
+# Serialization helper
+def serialize_matrix(builder, n_rows, n_cols, columns, col_idx):
+    """ Serialize NetEncoding.Matrix """
+    Matrix.MatrixStart(builder)
+    Matrix.MatrixAddNRows(builder, n_rows)
+    Matrix.MatrixAddNCols(builder, n_cols)
+    Matrix.MatrixAddColumns(builder, columns)
+    if col_idx is not None:
+        (u_type, u_val) = col_idx
+        Matrix.MatrixAddColIndexType(builder, u_type)
+        Matrix.MatrixAddColIndex(builder, u_val)
+    return Matrix.MatrixEnd(builder)
+
+
+# Serialization helper
+def serialize_typed_array(builder, source_array, encoding_info):
+    """
+    Serialize any of the various typed arrays, eg, Float32Array.   Specific
+    means of serialization and type conversion are provided by type_info.
+    """
+    arr = source_array
+    (array_type, as_type) = encoding_info(source_array)
+
+    if isinstance(arr, pd.Index):
+        arr = arr.to_series()
+
+    # convert to a simple ndarray
+    if as_type == 'json':
+        as_json = arr.to_json(orient='records')
+        arr = np.array(bytearray(as_json, 'utf-8'))
+    else:
+        if sparse.issparse(arr):
+            arr = arr.toarray()
+        elif isinstance(arr, pd.Series):
+            arr = arr.get_values()
+        if arr.dtype != as_type:
+            arr = arr.astype(as_type)
+
+    # serialize the ndarray into a vector
+    if arr.ndim == 2 and arr.shape[0] == 1:
+        arr = arr[0]
+    vec = CreateNumpyVector(builder, arr)
+
+    # serialize the typed array table
+    builder.StartObject(1)
+    builder.PrependUOffsetTRelativeSlot(0, vec, 0)
+    array_value = builder.EndObject()
+    return (array_type, array_value)
+
+
+def column_encoding(arr):
+    type_map = {
+        # dtype:  ( array_type, as_type )
+        np.float64: (TypedArray.TypedArray.Float32Array, np.float32),
+        np.float32: (TypedArray.TypedArray.Float32Array, np.float32),
+        np.float16: (TypedArray.TypedArray.Float32Array, np.float32),
+
+        np.int8: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int16: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int32: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int64: (TypedArray.TypedArray.Int32Array, np.int32),
+
+        np.uint8: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint16: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint32: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint64: (TypedArray.TypedArray.Uint32Array, np.uint32)
+    }
+    type_map_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
+    return type_map.get(arr.dtype.type, type_map_default)
+
+
+def index_encoding(arr):
+    type_map = {
+        # dtype:  ( array_type, as_type )
+        np.int32: (TypedArray.TypedArray.Int32Array, np.int32),
+        np.int64: (TypedArray.TypedArray.Int32Array, np.int32),
+
+        np.uint32: (TypedArray.TypedArray.Uint32Array, np.uint32),
+        np.uint64: (TypedArray.TypedArray.Uint32Array, np.uint32)
+    }
+    type_map_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
+    return type_map.get(arr.dtype.type, type_map_default)
+
+
+def guess_at_mem_needed(matrix):
+    (n_rows, n_cols) = matrix.shape
+    if isinstance(matrix, np.ndarray) or sparse.issparse(matrix):
+        guess = (n_rows * n_cols * matrix.dtype.itemsize) + 1024
+    elif isinstance(matrix, pd.DataFrame):
+        # XXX TODO - DataFrame type estimate
+        guess = 1
+    else:
+        guess = 1
+
+    # round up to nearest 1024 bytes
+    guess = (guess + 0x400) & (~0x3ff)
+    return guess
+
+
+def encode_matrix_fbs(matrix, row_idx=None, col_idx=None):
+    """
+    Given a 2D DataFrame, ndarray or sparse equivalent, create and return a
+    Matrix flatbuffer.
+
+    :param matrix: 2D DataFrame, ndarray or sparse equivalent
+    :param row_idx: index for row dimension, Index or ndarray
+    :param col_idx: index for col dimension, Index or ndarray
+
+    NOTE: row indices are (currently) unsupported and must be None
+    """
+
+    if row_idx is not None:
+        raise ValueError("row indexing not supported for FBS Matrix")
+    if matrix.ndim != 2:
+        raise ValueError("FBS Matrix must be 2D")
+
+    (n_rows, n_cols) = matrix.shape
+
+    # estimate size needed, so we don't unnecessarily realloc.
+    builder = flatbuffers.Builder(guess_at_mem_needed(matrix))
+
+    if isinstance(matrix, pd.DataFrame):
+        matrix_columns = reversed(tuple(matrix[name] for name in matrix))
+    else:
+        matrix_columns = reversed(tuple(c for c in matrix.T))
+
+    columns = []
+    # for idx in reversed(np.arange(n_cols)):
+    for c in matrix_columns:
+        # serialize the typed array
+        typed_arr = serialize_typed_array(builder, c, column_encoding)
+
+        # serialize the Column union
+        columns.append(serialize_column(builder, typed_arr))
+
+    # Serialize Matrix.columns[]
+    Matrix.MatrixStartColumnsVector(builder, n_cols)
+    for c in columns:
+        builder.PrependUOffsetTRelative(c)
+    matrix_column_vec = builder.EndVector(n_cols)
+
+    # serialize the colIndex if provided
+    cidx = None
+    if col_idx is not None:
+        cidx = serialize_typed_array(builder, col_idx, index_encoding)
+
+    # Serialize Matrix
+    matrix = serialize_matrix(builder, n_rows, n_cols, matrix_column_vec, cidx)
+
+    builder.Finish(matrix)
+    return builder.Output()

server/app/util/utils.py

@@ -54,3 +54,7 @@ def whole_number(value):
     if value < 0:
         raise ArgumentTypeError(f"{value} is not >= 0")
     return value
+
+
+def jsonify_scanpy(data):
+    return json.dumps(data, cls=Float32JSONEncoder, allow_nan=False)

server/app/web/templates/swagger.html

@@ -73,7 +73,7 @@
 <script>
 	window.onload = function () {
 		const ui = SwaggerUIBundle({
-			url: window.location.origin + "/api/swagger.json",
+			url: window.location.href.replace(/\/swagger$/, "") + "/api/swagger.json",
 			dom_id: '#swagger-ui',
 			deepLinking: true,
 			presets: [

server/app/web/webapp.py

@@ -7,9 +7,8 @@ bp = Blueprint("webapp", __name__, template_folder="templates")
 
 @bp.route("/")
 def index():
-    url_base = current_app.config["CXG_API_BASE"]
     dataset_title = current_app.config["DATASET_TITLE"]
-    return render_template("index.html", prefix=url_base, datasetTitle=dataset_title)
+    return render_template("index.html", datasetTitle=dataset_title)
 
 
 # renders swagger documentation

server/cli/cli.py

@@ -5,7 +5,7 @@ from .prepare import prepare
 
 
 @click.group(name="cellxgene", context_settings=dict(max_content_width=85))
-@click.version_option(version="0.4.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
+@click.version_option(version="0.6.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
 def cli():
     pass
 

server/cli/launch.py

@@ -33,7 +33,7 @@ from server.app.util.utils import custom_format_warning
     show_default=True,
     help="Provide verbose output, including warnings and all server requests.",
 )
-@click.option("--debug", "-d", is_flag=True, default=False, show_default=True, help="Run in debug mode.")
+@click.option("--debug", is_flag=True, default=False, show_default=True, help="Run in debug mode.")
 @click.option(
     "--open",
     "-o",
@@ -60,13 +60,6 @@ from server.app.util.utils import custom_format_warning
     show_default=True,
     help="Relative expression cutoff used when selecting top N differentially expressed genes",
 )
-@click.option(
-    "--nan-to-num",
-    is_flag=True,
-    default=False,
-    show_default=True,
-    help="Replace all floating point NaN with zero, and infinities with finite numbers",
-)
 def launch(
     data,
     layout,
@@ -81,7 +74,6 @@ def launch(
     host,
     max_category_items,
     diffexp_lfc_cutoff,
-    nan_to_num,
 ):
     """Launch the cellxgene data viewer.
     This web app lets you explore single-cell expression data.
@@ -117,12 +109,11 @@ def launch(
 
     # Setup app
     cellxgene_url = f"http://{host}:{port}"
-    api_base = f"{cellxgene_url}/api/"
 
     # Import Flask app
     from server.app.app import app
 
-    app.config.update(DATASET_TITLE=title, CXG_API_BASE=api_base)
+    app.config.update(DATASET_TITLE=title)
 
     if not verbose:
         log = logging.getLogger("werkzeug")
@@ -144,7 +135,6 @@ def launch(
         "diffexp_lfc_cutoff": diffexp_lfc_cutoff,
         "obs_names": obs_names,
         "var_names": var_names,
-        "nan_to_num": nan_to_num,
     }
 
     try:

server/requirements.txt

@@ -6,6 +6,7 @@ Flask-Compress>=1.4.0
 Flask-Cors>=3.0.6
 Flask-RESTful>=0.3.6
 flask-restful-swagger-2>=0.35
+flatbuffers>=1.10.0
 matplotlib>=2.2
 numpy>=1.14.5
 pandas>=0.23.1

server/test/decode_fbs.py

@@ -0,0 +1,69 @@
+
+"""
+Code to decode, for testing purposes, the flatbuffer encoded blobs.
+This code will need to be updated if fbs/matrix.fbs changes.
+
+For more information, see fbs/matrix.fbs and server/app/util/fbs/
+"""
+import json
+
+import server.app.util.fbs.NetEncoding.TypedArray as TypedArray
+import server.app.util.fbs.NetEncoding.Matrix as Matrix
+import server.app.util.fbs.NetEncoding.Int32Array as Int32Array
+import server.app.util.fbs.NetEncoding.Uint32Array as Uint32Array
+import server.app.util.fbs.NetEncoding.Float32Array as Float32Array
+import server.app.util.fbs.NetEncoding.Float64Array as Float64Array
+import server.app.util.fbs.NetEncoding.JSONEncodedArray as JSONEncodedArray
+
+
+def decode_typed_array(tarr):
+    type_map = {
+        TypedArray.TypedArray.Uint32Array: Uint32Array.Uint32Array,
+        TypedArray.TypedArray.Int32Array: Int32Array.Int32Array,
+        TypedArray.TypedArray.Float32Array: Float32Array.Float32Array,
+        TypedArray.TypedArray.Float64Array: Float64Array.Float64Array,
+        TypedArray.TypedArray.JSONEncodedArray: JSONEncodedArray.JSONEncodedArray
+    }
+    (u_type, u) = tarr
+    if u_type == TypedArray.TypedArray.NONE:
+        return None
+
+    TarType = type_map.get(u_type, None)
+    assert(TarType is not None)
+
+    arr = TarType()
+    arr.Init(u.Bytes, u.Pos)
+    narr = arr.DataAsNumpy()
+    if u_type == TypedArray.TypedArray.JSONEncodedArray:
+        narr = json.loads(narr.tostring().decode('utf-8'))
+    return narr
+
+
+def decode_matrix_FBS(buf):
+    """
+    Given a FBS Matrix, return an decoded Python dict containing
+    same info in native format.
+
+    NOTE / TODO: row_idx not currently implemented
+    """
+    df = Matrix.Matrix.GetRootAsMatrix(buf, 0)
+    n_rows = df.NRows()
+    n_cols = df.NCols()
+
+    columns_length = df.ColumnsLength()
+
+    decoded_columns = []
+    for col_idx in range(0, columns_length):
+        col = df.Columns(col_idx)
+        tarr = (col.UType(), col.U())
+        decoded_columns.append(decode_typed_array(tarr))
+
+    cidx = decode_typed_array((df.ColIndexType(), df.ColIndex()))
+
+    return {
+        "n_rows": n_rows,
+        "n_cols": n_cols,
+        "columns": decoded_columns,
+        "col_idx": cidx,
+        "row_idx": None
+    }

server/test/test_api.py

@@ -5,6 +5,8 @@ import time
 
 import requests
 
+import decode_fbs
+
 LOCAL_URL = "http://127.0.0.1:5005/"
 VERSION = "v0.2"
 URL_BASE = f"{LOCAL_URL}api/{VERSION}/"
@@ -40,6 +42,7 @@ class EndPoints(unittest.TestCase):
         url = f"{URL_BASE}{endpoint}"
         result = self.session.get(url)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["schema"]["dataframe"]["nObs"], 2638)
         self.assertEqual(len(result_data["schema"]["annotations"]["obs"]), 5)
@@ -49,6 +52,7 @@ class EndPoints(unittest.TestCase):
         url = f"{URL_BASE}{endpoint}"
         result = self.session.get(url)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["config"]["displayNames"]["dataset"], "pbmc3k")
         self.assertEqual(len(result_data["config"]["features"]), 4)
@@ -58,10 +62,26 @@ class EndPoints(unittest.TestCase):
         url = f"{URL_BASE}{endpoint}"
         result = self.session.get(url)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["layout"]["ndims"], 2)
         self.assertEqual(len(result_data["layout"]["coordinates"]), 2638)
 
+    def test_get_layout_fbs(self):
+        endpoint = "layout/obs"
+        url = f"{URL_BASE}{endpoint}"
+        header = {"Accept": "application/octet-stream"}
+        result = self.session.get(url, headers=header)
+        self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/octet-stream")
+        df = decode_fbs.decode_matrix_FBS(result.content)
+        self.assertEqual(df['n_rows'], 2638)
+        self.assertEqual(df['n_cols'], 2)
+        self.assertIsNotNone(df['columns'])
+        self.assertIsNone(df['col_idx'])
+        self.assertIsNone(df['row_idx'])
+        self.assertEqual(len(df['columns']), df['n_cols'])
+
     # def test_put_layout(self):
     #     endpoint = "layout/obs"
     #     url = f"{URL_BASE}{endpoint}"
@@ -93,6 +113,7 @@ class EndPoints(unittest.TestCase):
         url = f"{URL_BASE}{endpoint}"
         result = self.session.get(url)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["names"], ["name", "n_genes", "percent_mito", "n_counts", "louvain"])
         self.assertEqual(len(result_data["data"]), 2638)
@@ -103,11 +124,28 @@ class EndPoints(unittest.TestCase):
         query = "annotation-name=n_genes&annotation-name=percent_mito"
         url = f"{URL_BASE}{endpoint}?{query}"
         result = self.session.get(url)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         self.assertEqual(result.status_code, HTTPStatus.OK)
         result_data = result.json()
         self.assertEqual(result_data["names"], ["n_genes", "percent_mito"])
         self.assertEqual(len(result_data["data"][0]), 3)
 
+    def test_get_annotations_obs_fbs(self):
+        endpoint = "annotations/obs"
+        url = f"{URL_BASE}{endpoint}"
+        header = {"Accept": "application/octet-stream"}
+        result = self.session.get(url, headers=header)
+        self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/octet-stream")
+        df = decode_fbs.decode_matrix_FBS(result.content)
+        self.assertEqual(df['n_rows'], 2638)
+        self.assertEqual(df['n_cols'], 5)
+        self.assertIsNotNone(df['columns'])
+        self.assertIsNotNone(df['col_idx'])
+        self.assertIsNone(df['row_idx'])
+        self.assertEqual(len(df['columns']), df['n_cols'])
+        self.assertListEqual(df['col_idx'], ['name', 'n_genes', 'percent_mito', 'n_counts', 'louvain'])
+
     def test_get_annotations_obs_error(self):
         endpoint = "annotations/obs"
         query = "annotation-name=notakey"
@@ -131,6 +169,7 @@ class EndPoints(unittest.TestCase):
         }
         result = self.session.put(url, json=obs_filter)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["names"], ["name", "n_genes", "percent_mito", "n_counts", "louvain"])
         self.assertEqual(len(result_data["data"]), 15)
@@ -152,6 +191,7 @@ class EndPoints(unittest.TestCase):
         }
         result = self.session.put(url, json=obs_filter)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["names"], ["n_genes", "percent_mito"])
         self.assertEqual(len(result_data["data"][0]), 3)
@@ -168,6 +208,7 @@ class EndPoints(unittest.TestCase):
         }
         result = self.session.post(url, json=params)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(len(result_data), 7)
 
@@ -182,6 +223,7 @@ class EndPoints(unittest.TestCase):
         }
         result = self.session.post(url, json=params)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(len(result_data), 10)
 
@@ -190,6 +232,7 @@ class EndPoints(unittest.TestCase):
         url = f"{URL_BASE}{endpoint}"
         result = self.session.get(url)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["names"], ["name", "n_cells"])
         self.assertEqual(len(result_data["data"]), 1838)
@@ -201,10 +244,27 @@ class EndPoints(unittest.TestCase):
         url = f"{URL_BASE}{endpoint}?{query}"
         result = self.session.get(url)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["names"], ["n_cells"])
         self.assertEqual(len(result_data["data"][0]), 2)
 
+    def test_get_annotations_var_fbs(self):
+        endpoint = "annotations/var"
+        url = f"{URL_BASE}{endpoint}"
+        header = {"Accept": "application/octet-stream"}
+        result = self.session.get(url, headers=header)
+        self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/octet-stream")
+        df = decode_fbs.decode_matrix_FBS(result.content)
+        self.assertEqual(df['n_rows'], 1838)
+        self.assertEqual(df['n_cols'], 2)
+        self.assertIsNotNone(df['columns'])
+        self.assertIsNotNone(df['col_idx'])
+        self.assertIsNone(df['row_idx'])
+        self.assertEqual(len(df['columns']), df['n_cols'])
+        self.assertListEqual(df['col_idx'], ['name', 'n_cells'])
+
     def test_get_annotations_var_error(self):
         endpoint = "annotations/var"
         query = "annotation-name=notakey"
@@ -218,6 +278,7 @@ class EndPoints(unittest.TestCase):
         var_filter = {"filter": {"var": {"annotation_value": [{"name": "name", "values": ["ATAD3C", "RER1"]}]}}}
         result = self.session.put(url, json=var_filter)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["names"], ["name", "n_cells"])
         self.assertEqual(len(result_data["data"]), 2)
@@ -229,6 +290,7 @@ class EndPoints(unittest.TestCase):
         var_filter = {"filter": {"var": {"annotation_value": [{"name": "name", "values": ["ATAD3C", "RER1"]}]}}}
         result = self.session.put(url, json=var_filter)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data = result.json()
         self.assertEqual(result_data["names"], ["n_cells"])
         self.assertEqual(len(result_data["data"][0]), 2)
@@ -241,6 +303,7 @@ class EndPoints(unittest.TestCase):
             url = f"{URL_BASE}{endpoint}?{query}"
             result = self.session.get(url)
             self.assertEqual(result.status_code, HTTPStatus.OK)
+            self.assertEqual(result.headers["Content-Type"], "application/json")
             result_data = result.json()
             self.assertEqual(len(result_data["obs"]), 2638)
 
@@ -262,6 +325,7 @@ class EndPoints(unittest.TestCase):
             url = f"{URL_BASE}{endpoint}"
             result = self.session.get(url)
             self.assertEqual(result.status_code, HTTPStatus.OK)
+            self.assertEqual(result.headers["Content-Type"], "application/json")
 
     def test_data_filter(self):
         for axis in ["obs", "var"]:
@@ -270,10 +334,11 @@ class EndPoints(unittest.TestCase):
             url = f"{URL_BASE}{endpoint}?{query}"
             result = self.session.get(url)
             self.assertEqual(result.status_code, HTTPStatus.OK)
+            self.assertEqual(result.headers["Content-Type"], "application/json")
             result_data = result.json()
             self.assertEqual(len(result_data["obs"]), 38)
 
-    def test_data_put(self):
+    def test_data_json_put(self):
         for axis in ["obs", "var"]:
             endpoint = f"data/{axis}"
             url = f"{URL_BASE}{endpoint}"
@@ -291,9 +356,33 @@ class EndPoints(unittest.TestCase):
             }
             result = self.session.put(url, headers=header, json=obs_filter)
             self.assertEqual(result.status_code, HTTPStatus.OK)
+            self.assertEqual(result.headers["Content-Type"], "application/json")
             result_data = result.json()
             self.assertEqual(len(result_data["obs"]), 15)
 
+    def test_data_put_fbs(self):
+        endpoint = f"data/var"
+        url = f"{URL_BASE}{endpoint}"
+        header = {"Accept": "application/octet-stream"}
+        filter = {
+            "filter": {
+                "var": {
+                    "index": [0, 1, 4]
+                }
+            }
+        }
+        result = self.session.put(url, headers=header, json=filter)
+        self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/octet-stream")
+        df = decode_fbs.decode_matrix_FBS(result.content)
+        self.assertEqual(df['n_rows'], 2638)
+        self.assertEqual(df['n_cols'], 3)
+        self.assertIsNotNone(df['columns'])
+        self.assertIsNotNone(df['col_idx'])
+        self.assertIsNone(df['row_idx'])
+        self.assertEqual(len(df['columns']), df['n_cols'])
+        self.assertListEqual(df['col_idx'].tolist(), [0, 1, 4])
+
     def test_data_put_single_var(self):
         for axis in ["obs", "var"]:
             endpoint = f"data/{axis}"
@@ -302,6 +391,7 @@ class EndPoints(unittest.TestCase):
             var_filter = {"filter": {"var": {"annotation_value": [{"name": "name", "values": ["RER1"]}]}}}
             result = self.session.put(url, headers=header, json=var_filter)
             self.assertEqual(result.status_code, HTTPStatus.OK)
+            self.assertEqual(result.headers["Content-Type"], "application/json")
             result_data = result.json()
             if axis == "obs":
                 self.assertEqual(len(result_data["obs"][0]), 2)
@@ -338,6 +428,7 @@ class EndPoints(unittest.TestCase):
         }
         result = self.session.put(url, json=f1)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data1 = result.json()
         f2 = {
             "filter": {
@@ -353,6 +444,7 @@ class EndPoints(unittest.TestCase):
         }
         result = self.session.put(url, json=f2)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data2 = result.json()
         self.assertNotEqual(result_data1, result_data2)
 
@@ -377,6 +469,7 @@ class EndPoints(unittest.TestCase):
         }
         result = self.session.put(url, json=f2)
         self.assertEqual(result.status_code, HTTPStatus.OK)
+        self.assertEqual(result.headers["Content-Type"], "application/json")
         result_data2 = result.json()
         self.assertNotEqual(result_data1, result_data2)
 

server/test/test_nan_rest.py

@@ -0,0 +1,51 @@
+from http import HTTPStatus
+from subprocess import Popen
+import unittest
+import time
+
+import requests
+
+LOCAL_URL = "http://127.0.0.1:5005/"
+VERSION = "v0.2"
+URL_BASE = f"{LOCAL_URL}api/{VERSION}/"
+
+BAD_FILTER = {"filter": {"obs": {"annotation_value": [{"name": "xyz"}]}}}
+
+
+class WithNaNs(unittest.TestCase):
+    """Test Case for endpoints"""
+
+    @classmethod
+    def setUpClass(cls):
+        cls.ps = Popen(
+            ["cellxgene", "launch", "server/test/test_datasets/nan.h5ad", "--debug"]
+        )
+        session = requests.Session()
+        for i in range(90):
+            try:
+                session.get(f"{URL_BASE}schema")
+            except requests.exceptions.ConnectionError:
+                time.sleep(1)
+
+    @classmethod
+    def tearDownClass(cls):
+        try:
+            cls.ps.terminate()
+        except ProcessLookupError:
+            pass
+
+    def setUp(self):
+        self.session = requests.Session()
+
+    def test_initialize(self):
+        endpoint = "schema"
+        url = f"{URL_BASE}{endpoint}"
+        result = self.session.get(url)
+        self.assertEqual(result.status_code, HTTPStatus.OK)
+
+    def test_errors(self):
+        endpoints = ["annotations/obs", "annotations/var", "data/obs", "data/var"]
+        for endpoint in endpoints:
+            url = f"{URL_BASE}{endpoint}"
+            result = self.session.get(url)
+            self.assertEqual(result.status_code, HTTPStatus.INTERNAL_SERVER_ERROR)

server/test/test_nan_scanpy_engine.py

@@ -0,0 +1,72 @@
+import json
+import pytest
+import unittest
+import warnings
+import math
+
+import decode_fbs
+
+from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
+from server.app.util.errors import JSONEncodingValueError
+
+
+class NaNTest(unittest.TestCase):
+    def setUp(self):
+        self.args = {
+            "layout": "umap",
+            "diffexp": "ttest",
+            "max_category_items": 100,
+            "obs_names": None,
+            "var_names": None,
+            "diffexp_lfc_cutoff": 0.01,
+        }
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UserWarning)
+            self.data = ScanpyEngine("server/test/test_datasets/nan.h5ad", self.args)
+            self.data._create_schema()
+
+    def test_load(self):
+        with self.assertWarns(UserWarning):
+            ScanpyEngine("server/test/test_datasets/nan.h5ad", self.args)
+
+    def test_init(self):
+        self.assertEqual(self.data.cell_count, 100)
+        self.assertEqual(self.data.gene_count, 100)
+        epsilon = 0.000_005
+        self.assertTrue(self.data.data.X[0, 0] - -0.171_469_51 < epsilon)
+
+    def test_dataframe(self):
+        data_frame_var = decode_fbs.decode_matrix_FBS(self.data.data_frame_to_fbs_matrix(None, "var"))
+        self.assertIsNotNone(data_frame_var)
+        self.assertEqual(data_frame_var["n_rows"], 100)
+        self.assertEqual(data_frame_var["n_cols"], 100)
+        self.assertTrue(math.isnan(data_frame_var["columns"][3][3]))
+
+        with pytest.raises(JSONEncodingValueError):
+            json.loads(self.data.data_frame(None, "obs"))
+        with pytest.raises(JSONEncodingValueError):
+            json.loads(self.data.data_frame(None, "var"))
+
+    def test_dataframe_obs_not_implemented(self):
+        with self.assertRaises(ValueError) as cm:
+            decode_fbs.decode_matrix_FBS(self.data.data_frame_to_fbs_matrix(None, "obs"))
+        self.assertIsNotNone(cm.exception)
+
+    def test_annotation(self):
+        annotations = decode_fbs.decode_matrix_FBS(self.data.annotation_to_fbs_matrix("obs"))
+        self.assertEqual(
+            annotations["col_idx"],
+            ["name", "n_genes", "percent_mito", "n_counts", "louvain"]
+        )
+        self.assertEqual(annotations["n_rows"], 100)
+        self.assertTrue(math.isnan(annotations["columns"][2][0]))
+
+        annotations = decode_fbs.decode_matrix_FBS(self.data.annotation_to_fbs_matrix("var"))
+        self.assertEqual(annotations["col_idx"], ["name", "n_cells", "var_with_nans"])
+        self.assertEqual(annotations["n_rows"], 100)
+        self.assertTrue(math.isnan(annotations["columns"][2][0]))
+
+        with pytest.raises(JSONEncodingValueError):
+            json.loads(self.data.annotation(None, "obs"))
+        with pytest.raises(JSONEncodingValueError):
+            json.loads(self.data.annotation(None, "var"))

server/test/test_scanpy_engine.py

@@ -19,7 +19,6 @@ class UtilTest(unittest.TestCase):
             "obs_names": None,
             "var_names": None,
             "diffexp_lfc_cutoff": 0.01,
-            "nan_to_num": True,
         }
 
         self.data = ScanpyEngine("example-dataset/pbmc3k.h5ad", args)
@@ -44,22 +43,39 @@ class UtilTest(unittest.TestCase):
             self.data._validate_data_types()
 
     def test_filter_idx(self):
-        filter_ = {"filter": {"var": {"index": [1, 99, [200, 300]]}, "obs": {"index": [1, 99, [1000, 2000]]}}}
+        filter_ = {
+            "filter": {
+                "var": {"index": [1, 99, [200, 300]]},
+                "obs": {"index": [1, 99, [1000, 2000]]},
+            }
+        }
         data = self.data.filter_dataframe(filter_["filter"])
         self.assertEqual(data.shape, (1002, 102))
 
     def test_filter_annotation(self):
         filter_ = {
-            "filter": {"obs": {"annotation_value": [{"name": "louvain", "values": ["NK cells", "CD8 T cells"]}]}}
+            "filter": {
+                "obs": {
+                    "annotation_value": [
+                        {"name": "louvain", "values": ["NK cells", "CD8 T cells"]}
+                    ]
+                }
+            }
         }
         data = self.data.filter_dataframe(filter_["filter"])
         self.assertEqual(data.shape, (470, 1838))
-        filter_ = {"filter": {"obs": {"annotation_value": [{"name": "n_counts", "min": 3000}]}}}
+        filter_ = {
+            "filter": {"obs": {"annotation_value": [{"name": "n_counts", "min": 3000}]}}
+        }
         data = self.data.filter_dataframe(filter_["filter"])
         self.assertEqual(data.shape, (497, 1838))
 
     def test_filter_annotation_no_uns(self):
-        filter_ = {"filter": {"var": {"annotation_value": [{"name": "name", "values": ["RER1"]}]}}}
+        filter_ = {
+            "filter": {
+                "var": {"annotation_value": [{"name": "name", "values": ["RER1"]}]}
+            }
+        }
         data = self.data.filter_dataframe(filter_["filter"])
         self.assertEqual(data.shape[1], 1)
 
@@ -90,36 +106,45 @@ class UtilTest(unittest.TestCase):
 
     def test_schema_produces_error(self):
         self.data.data.obs["time"] = Series(
-            list([time.time() for i in range(self.data.cell_count)]), dtype="datetime64[ns]"
+            list([time.time() for i in range(self.data.cell_count)]),
+            dtype="datetime64[ns]",
         )
         with pytest.raises(TypeError):
             self.data._create_schema()
 
     def test_config(self):
-        self.assertEqual(self.data.features["layout"]["obs"], {"available": True, "interactiveLimit": 50000})
+        self.assertEqual(
+            self.data.features["layout"]["obs"],
+            {"available": True, "interactiveLimit": 50000},
+        )
 
     def test_layout(self):
-        layout = self.data.layout(None)
-        self.assertEqual(layout["ndims"], 2)
-        self.assertEqual(len(layout["coordinates"]), 2638)
-        self.assertEqual(layout["coordinates"][0][0], 0)
-        for idx, val in enumerate(layout["coordinates"]):
+        layout = json.loads(self.data.layout(None))
+        self.assertEqual(layout["layout"]["ndims"], 2)
+        self.assertEqual(len(layout["layout"]["coordinates"]), 2638)
+        self.assertEqual(layout["layout"]["coordinates"][0][0], 0)
+        for idx, val in enumerate(layout["layout"]["coordinates"]):
             self.assertLessEqual(val[1], 1)
             self.assertLessEqual(val[2], 1)
 
     def test_annotations(self):
-        annotations = self.data.annotation(None, "obs")
-        self.assertEqual(annotations["names"], ["name", "n_genes", "percent_mito", "n_counts", "louvain"])
+        annotations = json.loads(self.data.annotation(None, "obs"))
+        self.assertEqual(
+            annotations["names"],
+            ["name", "n_genes", "percent_mito", "n_counts", "louvain"],
+        )
         self.assertEqual(len(annotations["data"]), 2638)
-        annotations = self.data.annotation(None, "var")
+        annotations = json.loads(self.data.annotation(None, "var"))
         self.assertEqual(annotations["names"], ["name", "n_cells"])
         self.assertEqual(len(annotations["data"]), 1838)
 
     def test_annotation_fields(self):
-        annotations = self.data.annotation(None, "obs", ["n_genes", "n_counts"])
+        annotations = json.loads(
+            self.data.annotation(None, "obs", ["n_genes", "n_counts"])
+        )
         self.assertEqual(annotations["names"], ["n_genes", "n_counts"])
         self.assertEqual(len(annotations["data"]), 2638)
-        annotations = self.data.annotation(None, "var", ["name"])
+        annotations = json.loads(self.data.annotation(None, "var", ["name"]))
         self.assertEqual(annotations["names"], ["name"])
         self.assertEqual(len(annotations["data"]), 1838)
 
@@ -127,45 +152,54 @@ class UtilTest(unittest.TestCase):
         filter_ = {
             "filter": {
                 "obs": {"annotation_value": [{"name": "n_counts", "min": 3000}]},
-                "var": {"annotation_value": [{"name": "name", "values": ["ATAD3C", "RER1"]}]},
+                "var": {
+                    "annotation_value": [{"name": "name", "values": ["ATAD3C", "RER1"]}]
+                },
             }
         }
-        annotations = self.data.annotation(filter_["filter"], "obs")
-        self.assertEqual(annotations["names"], ["name", "n_genes", "percent_mito", "n_counts", "louvain"])
+        annotations = json.loads(self.data.annotation(filter_["filter"], "obs"))
+        self.assertEqual(
+            annotations["names"],
+            ["name", "n_genes", "percent_mito", "n_counts", "louvain"],
+        )
         self.assertEqual(len(annotations["data"]), 497)
-        annotations = self.data.annotation(filter_["filter"], "var")
+        annotations = json.loads(self.data.annotation(filter_["filter"], "var"))
         self.assertEqual(annotations["names"], ["name", "n_cells"])
         self.assertEqual(len(annotations["data"]), 2)
 
     def test_filtered_layout(self):
-        filter_ = {"filter": {"obs": {"annotation_value": [{"name": "n_counts", "min": 3000}]}}}
-        layout = self.data.layout(filter_["filter"])
-        self.assertEqual(len(layout["coordinates"]), 497)
+        filter_ = {
+            "filter": {"obs": {"annotation_value": [{"name": "n_counts", "min": 3000}]}}
+        }
+        layout = json.loads(self.data.layout(filter_["filter"]))
+        self.assertEqual(len(layout["layout"]["coordinates"]), 497)
 
     def test_diffexp_topN(self):
         f1 = {"filter": {"obs": {"index": [[0, 500]]}}}
         f2 = {"filter": {"obs": {"index": [[500, 1000]]}}}
-        result = self.data.diffexp_topN(f1["filter"], f2["filter"])
+        result = json.loads(self.data.diffexp_topN(f1["filter"], f2["filter"]))
         self.assertEqual(len(result), 10)
-        result = self.data.diffexp_topN(f1["filter"], f2["filter"], 20)
+        result = json.loads(self.data.diffexp_topN(f1["filter"], f2["filter"], 20))
         self.assertEqual(len(result), 20)
 
     def test_data_frame(self):
-        data_frame_obs = self.data.data_frame(None, "obs")
+        data_frame_obs = json.loads(self.data.data_frame(None, "obs"))
         self.assertEqual(len(data_frame_obs["var"]), 1838)
         self.assertEqual(len(data_frame_obs["obs"]), 2638)
-        data_frame_var = self.data.data_frame(None, "var")
+        data_frame_var = json.loads(self.data.data_frame(None, "var"))
         self.assertEqual(len(data_frame_var["var"]), 1838)
         self.assertEqual(len(data_frame_var["obs"]), 2638)
 
     def test_filtered_data_frame(self):
-        filter_ = {"filter": {"obs": {"annotation_value": [{"name": "n_counts", "min": 3000}]}}}
-        data_frame_obs = self.data.data_frame(filter_["filter"], "obs")
+        filter_ = {
+            "filter": {"obs": {"annotation_value": [{"name": "n_counts", "min": 3000}]}}
+        }
+        data_frame_obs = json.loads(self.data.data_frame(filter_["filter"], "obs"))
         self.assertEqual(len(data_frame_obs["var"]), 1838)
         self.assertEqual(len(data_frame_obs["obs"]), 497)
         self.assertIsInstance(data_frame_obs["obs"][0], (list, tuple))
         self.assertEqual(type(data_frame_obs["var"][0]), int)
-        data_frame_var = self.data.data_frame(filter_["filter"], "var")
+        data_frame_var = json.loads(self.data.data_frame(filter_["filter"], "var"))
         self.assertEqual(len(data_frame_var["var"]), 1838)
         self.assertEqual(len(data_frame_var["obs"]), 497)
         self.assertIsInstance(data_frame_var["var"][0], (list, tuple))
@@ -173,8 +207,12 @@ class UtilTest(unittest.TestCase):
 
     def test_data_single_gene(self):
         for axis in ["obs", "var"]:
-            filter_ = {"filter": {"var": {"annotation_value": [{"name": "name", "values": ["RER1"]}]}}}
-            data_frame_var = self.data.data_frame(filter_["filter"], axis)
+            filter_ = {
+                "filter": {
+                    "var": {"annotation_value": [{"name": "name", "values": ["RER1"]}]}
+                }
+            }
+            data_frame_var = json.loads(self.data.data_frame(filter_["filter"], axis))
             if axis == "obs":
                 self.assertEqual(type(data_frame_var["var"][0]), int)
                 self.assertIsInstance(data_frame_var["obs"][0], (list, tuple))

setup.cfg

@@ -1,3 +1,4 @@
 [flake8]
 max-line-length = 120
-ignore = E203
+ignore = E203, W503
+exclude = server/app/util/fbs/NetEncoding/

setup.py

@@ -17,7 +17,7 @@ with open("server/requirements.txt") as fh:
 
 setup(
     name="cellxgene",
-    version="0.4.0",
+    version="0.6.0",
     packages=find_packages(),
     url="https://github.com/chanzuckerberg/cellxgene",
     license="MIT",