release 0.12.0 (#981)

* initial cut at backed mode

* make flask multithreading conditional on debug flag

* update X access to support backed mode

* lint

* improve help message for backed mode

* fix tests

* add MatrixProxy to normalize supported matrix types

* add FAQ entry for --backed

* remove use of matrix.T

* clean up

* add ability to disable diffexp from CLI; add hueristic to detect likely slow diffexp calculation, and warn user

* fix tests

* do not print diffexp speed warning if diffexp is disabled

* tweak wording of diffexp speed messages

* add FAQ entry on --disable-diffexp

* revise heuristic for warning about slow diffexp

* use quick tooltip delay on diffexp button

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.11.0
+current_version = 0.12.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"
@@ -14,6 +14,6 @@ search = "version": "{current_version}"
 replace = "version": "{new_version}"
 
 [bumpversion:file:server/__init__.py]
-search = "__version__ = "{current_version}"
-replace = "__version__ = "{new_version}"
+search = __version__ = "{current_version}"
+replace = __version__ = "{new_version}"
 

MANIFEST.in

@@ -1,4 +1,5 @@
 recursive-include server/app/web/templates *
 recursive-include server/app/web/static *
 
-include server/requirements.txt
+include server/requirements.txt
+include server/requirements-prepare.txt

README.md

@@ -1,38 +1,41 @@
-# cellxgene
+<img src="./docs/cellxgene-logo.svg" width="300">
 
-> an interactive explorer for single-cell transcriptomics data
+_an interactive explorer for single-cell transcriptomics data_
 
-[![DOI](https://zenodo.org/badge/105615409.svg)](https://zenodo.org/badge/latestdoi/105615409)
+[![DOI](https://zenodo.org/badge/105615409.svg)](https://zenodo.org/badge/latestdoi/105615409) [![PyPI](https://img.shields.io/pypi/v/cellxgene)](https://pypi.org/project/cellxgene/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/cellxgene)](https://pypistats.org/packages/cellxgene) [![GitHub last commit](https://img.shields.io/github/last-commit/chanzuckerberg/cellxgene)](https://github.com/chanzuckerberg/cellxgene/pulse)
 
-_cellxgene_ (pronounced "sell-by-jean") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
+cellxgene (pronounced "cell-by-gene") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data.
 
 <img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
 
 - Want to install and use cellxgene? Visit the [cellxgene docs](https://chanzuckerberg.github.io/cellxgene/).
-- Want to see where we are going? Check out [our roadmap](ROADMAP.md).
-- Want to contribute? See our [contributors guide](CONTRIBUTING.md).
+- Want to see where we are going? Check out [our roadmap](https://github.com/chanzuckerberg/cellxgene/blob/master/ROADMAP.md).
+- Want to contribute? See our [contributors guide](https://github.com/chanzuckerberg/cellxgene/blob/master/CONTRIBUTING.md).
 
 ## quick start
 
-To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](https://chanzuckerberg.github.io/cellxgene/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
+To install cellxgene you need Python 3.6+. We recommend [installing cellxgene into a conda or virtual environment.](https://chanzuckerberg.github.io/cellxgene/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
 
 Install the package.
-``` bash
+
+```bash
 pip install cellxgene
 ```
 
 Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
 
-``` bash
-curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
+```bash
+curl -O https://cellxgene-example-data.czi.technology/pbmc3k.h5ad.zip
+unzip pbmc3k.h5ad
 ```
 
-Launch _cellxgene_
-``` bash
+Launch cellxgene
+
+```bash
 cellxgene launch pbmc3k.h5ad --open
 ```
 
-To learn more about what you can do with _cellxgene_, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-started.html) guide.
+To learn more about what you can do with cellxgene, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-started.html) guide.
 
 ## get in touch
 
@@ -40,10 +43,24 @@ Have questions, suggestions, or comments? You can come hang out with us by joini
 
 ## contributing
 
-We warmly welcome contributions from the community! Please see our [contributing guide](CONTRIBUTING.md) and don't hesitate to open an issue or send a pull request to improve cellxgene.
+We warmly welcome contributions from the community! Please see our [contributing guide](https://github.com/chanzuckerberg/cellxgene/blob/master/CONTRIBUTING.md) and don't hesitate to open an issue or send a pull request to improve cellxgene.
 
 This project adheres to the Contributor Covenant [code of conduct](https://github.com/chanzuckerberg/.github/blob/master/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.
 
+## core team
+
+The current core team:
+
+- Colin Megill, frontend & product design
+- Bruce Martin, software engineer
+- Sidney Bell, computational biologist
+- Lia Prins, designer
+- Severiano Badajoz, software engineer
+
+We would also like to gratefully acknowledge contributions from past core team members:
+
+- Charlotte Weaver, software engineer
+
 ## where we are going
 
 Our goal is to enable teams of computational and experimental
@@ -56,21 +73,7 @@ There are 4 key features we plan to implement in the near term.
 - Toggle embeddings
 - Gene information
 
-For more detail on these features and where we are going, see [our roadmap](ROADMAP.md).
-
-## risks of hosting cellxgene
-
-_cellxgene_ is built on standard web technologies, but is currently designed as a single-user desktop application.
-We've done this so we can prioritize the features on [our roadmap](ROADMAP.md).
-
-Some of our users have experimented with hosting _cellxgene_ either for their lab or for public use, but please note that the _cellxgene_ team does not officially support, troubleshoot, or maintain any web deployments at this time.
-
-If do you choose setup _cellxgene_ as a hosted service, you should be aware of the following risks:
-
-- `$ cellxgene launch` uses Flask's development server, which is not recommended for hosted deployment (see the [Flask documentation](http://flask.pocoo.org/docs/1.0/tutorial/deploy/#run-with-a-production-server))
-- We have no testing or official support for deployments where multiple users are accessing the same _cellxgene_ instance.
-- Your _cellxgene_ instance is likely to hang or crash if too many people access it at the same time, especially if they using functions that call the Python backend (such as differential expression, updating the layout, or coloring by gene).
-- _cellxgene_ only supports one instance per dataset
+For more detail on these features and where we are going, see [our roadmap](https://github.com/chanzuckerberg/cellxgene/blob/master/ROADMAP.md).
 
 ## inspiration
 
@@ -78,17 +81,9 @@ We've been heavily inspired by several other related single-cell visualization p
 
 We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.
 
-We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
+We have been working closely with the [scanpy](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
 
-We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
-
-## core team
-
-- Colin Megill, frontend & product design
-- Charlotte Weaver, software engineer
-- Bruce Martin, software engineer
-- Sidney Bell, computational biologist
-- Justin Kiggins, product manager
+We are eager to explore integrations with other computational backends such as [Seurat](https://github.com/satijalab/seurat) or [Bioconductor](https://github.com/Bioconductor)
 
 ## reuse
 

app.json

@@ -0,0 +1,21 @@
+{
+  "name": "cellxgene",
+  "description": "An interactive explorer for single-cell transcriptomics data",
+  "repository": "https://github.com/chanzuckerberg/cellxgene/",
+  "logo": "https://cellxgene-example-data.czi.technology/favicon.png",
+  "keywords": [
+    "scientific",
+    "visualization",
+    "scrna-seq",
+    "transcriptomics",
+    "dataviz"
+  ],
+  "stack": "container",
+  "env": {
+    "DATASET": {
+      "description": "Link to dataset",
+      "value": "https://cellxgene-example-data.czi.technology/pbmc3k.h5ad",
+      "required": "true"
+    }
+  }
+}

client/__tests__/e2e/data.js

@@ -27,7 +27,7 @@ export const datasets = {
       lasso: [
         {
           "coordinates-as-percent": { x1: 0.05, y1: 0.25, x2: 0.15, y2: 0.35 },
-          count: "71"
+          count: "88"
         }
       ],
       categorical: [
@@ -87,12 +87,18 @@ export const datasets = {
       categorical: {
         louvain: {
           "B cells": "342",
-          Megakaryocytes: "15"
+          "CD14+ Monocytes": "0",
+          "CD4 T cells": "0",
+          "CD8 T cells": "0",
+          "Dendritic cells": "0",
+          "FCGR3A+ Monocytes": "0",
+          Megakaryocytes: "15",
+          "NK cells": "0"
         }
       },
       lasso: {
         "coordinates-as-percent": { x1: 0.45, y1: 0.05, x2: 0.65, y2: 0.15 },
-        count: "36"
+        count: "39"
       }
     },
     scatter: {

client/__tests__/util/dataframe/dataframe.test.js

@@ -452,6 +452,59 @@ describe("dataframe factories", () => {
     });
   });
 
+  describe("withColsFrom", () => {
+    test("error conditions", () => {
+      /*
+      make sure we catch common errors:
+      - duplicate column names
+      - dimensionality difference
+      */
+      const dfA = new Dataframe.Dataframe(
+        [2, 3],
+        [["red", "blue"], [true, false], [1, 0]],
+        null,
+        new Dataframe.KeyIndex(["colors", "bools", "numbers"])
+      );
+
+      /* different dimensionality should throw error */
+      const dfB = new Dataframe.Dataframe(
+        [3, 1],
+        [["red", "blue", "green"]],
+        null,
+        new Dataframe.KeyIndex(["colorsA"])
+      );
+      expect(() => dfA.withColsFrom(dfB)).toThrow(RangeError);
+
+      /* duplicate labels should throw an error */
+      expect(() => dfA.withColsFrom(dfA)).toThrow(Error);
+    });
+
+    test("simple", () => {
+      /* simple test that it works as expected in common case */
+      const dfA = new Dataframe.Dataframe(
+        [2, 1],
+        [["red", "blue"]],
+        null,
+        new Dataframe.KeyIndex(["colors"])
+      );
+      const dfB = new Dataframe.Dataframe(
+        [2, 1],
+        [[true, false]],
+        null,
+        new Dataframe.KeyIndex(["bools"])
+      );
+
+      const dfC = dfA.withColsFrom(dfB);
+      expect(dfC).toBeDefined();
+      expect(dfC.dims).toEqual([2, 2]);
+      expect(dfC.colIndex.keys()).toEqual(["colors", "bools"]);
+      expect(dfC.rowIndex).toEqual(dfA.rowIndex);
+      expect(dfC.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+      expect(dfC.col("colors").asArray()).toEqual(["red", "blue"]);
+      expect(dfC.col("bools").asArray()).toEqual([true, false]);
+    });
+  });
+
   describe("dropCol", () => {
     test("KeyIndex", () => {
       const df = new Dataframe.Dataframe(
@@ -567,6 +620,32 @@ describe("dataframe factories", () => {
       expect(dfB.iat(0, 1)).toEqual(1);
       expect(dfB.iat(0, 2)).toEqual(1);
     });
+
+    test("columns", () => {
+      const df = Dataframe.Dataframe.create(
+        [3, 3],
+        [new Array(3).fill(0), new Array(3).fill(0), new Array(3).fill(0)]
+      );
+
+      expect(df).toBeDefined();
+      expect(df.columns()).toHaveLength(3);
+      expect(df.columns()[0]).toEqual(df.icol(0));
+      expect(df.columns()[2]).toEqual(df.icol(2));
+    });
+
+    test("renameCol", () => {
+      const dfA = new Dataframe.Dataframe(
+        [2, 2],
+        [[true, false], [1, 0]],
+        null,
+        new Dataframe.KeyIndex(["A", "B"])
+      );
+      const dfB = dfA.renameCol("B", "C");
+      expect(dfA.colIndex.keys()).toEqual(["A", "B"]);
+      expect(dfB.colIndex.keys()).toEqual(["A", "C"]);
+      expect(dfA.dims).toMatchObject(dfB.dims);
+      expect(dfA.columns()).toMatchObject(dfB.columns());
+    });
   });
 });
 

client/__tests__/util/dataframe/histogram.test.js

@@ -66,4 +66,28 @@ describe("Dataframe column histogram", () => {
     // memoized?
     expect(df.col("value").histogram(3, [0, 2])).toMatchObject(h1);
   });
+
+  test("continuous thesholds correct", () => {
+    const vals = [0, 1, 9, 10, 11, 20, 99, 100];
+    const df = new Dataframe.Dataframe(
+      [8, 2],
+      [new Int32Array(vals), new Float32Array(vals)]
+    );
+
+    expect(df.col(0).histogram(5, [0, 100])).toEqual([5, 1, 0, 0, 2]);
+    expect(df.col(1).histogram(5, [0, 100])).toEqual([5, 1, 0, 0, 2]);
+    expect(df.col(0).histogram(2, [0, 10])).toEqual([2, 2]);
+    expect(df.col(0).histogram(10, [0, 100])).toEqual([
+      3,
+      2,
+      1,
+      0,
+      0,
+      0,
+      0,
+      0,
+      0,
+      2
+    ]);
+  });
 });

client/__tests__/util/stateManager/fbs.test.js

@@ -0,0 +1,34 @@
+/*
+test FBS encode/decode API
+*/
+import { Dataframe, KeyIndex } from "../../../src/util/dataframe";
+import {
+	decodeMatrixFBS,
+	encodeMatrixFBS
+} from "../../../src/util/stateManager/matrix";
+
+describe("encode/decode", () => {
+	test("round trip", () => {
+		const columns = [
+			["red", "green", "blue"],
+			new Int32Array(3).fill(0),
+			new Uint32Array(3).fill(1),
+			new Float32Array(3).fill(2)
+		];
+
+		const dfNoColIdx = new Dataframe([3, 4], columns);
+		const dfA = decodeMatrixFBS(encodeMatrixFBS(dfNoColIdx));
+		expect([dfA.nRows, dfA.nCols]).toEqual(dfNoColIdx.dims);
+		expect(dfA.colIdx).toBeNull();
+		expect(dfA.rowIdx).toBeNull();
+		expect(dfA.columns).toEqual(columns);
+
+		const colIndex = new KeyIndex(["a", "b", "c", "d"]);
+		const dfWithColIdx = new Dataframe([3, 4], columns, null, colIndex);
+		const dfB = decodeMatrixFBS(encodeMatrixFBS(dfWithColIdx));
+		expect([dfB.nRows, dfB.nCols]).toEqual(dfWithColIdx.dims);
+		expect(dfB.colIdx).toEqual(colIndex.keys());
+		expect(dfB.rowIdx).toBeNull();
+		expect(dfB.columns).toEqual(columns);
+	});
+});

client/__tests__/util/stateManager/sampleResponses.js

@@ -110,6 +110,14 @@ function encodeTypedArray(builder, uType, uData) {
 }
 
 function encodeMatrix(columns, colIndex = undefined) {
+  /*
+  IMPORTANT: this is not a general purpose encoder.  in particular,
+  it doesn't correctly handle all column index types, nor does it 
+  handle all column typedarray types.
+
+  encodeMatrixFBS in matrix.py is more general.  This is used only
+  as a testing santity check (alt implementation).
+  */
   const utf8Encoder = new TextEncoder("utf-8");
   const builder = new flatbuffers.Builder(1024);
   const cols = _.map(columns, carr => {

client/__tests__/util/stateManager/world.test.js

@@ -18,11 +18,11 @@ const defaultBigBang = () => {
   /* create unverse, world, crossfilter and dimensionMap */
   /* create universe */
   const universe = Universe.createUniverseFromResponse(
-    REST.config,
-    REST.schema,
-    REST.annotationsObs,
-    REST.annotationsVar,
-    REST.layoutObs
+    _.cloneDeep(REST.config),
+    _.cloneDeep(REST.schema),
+    _.cloneDeep(REST.annotationsObs),
+    _.cloneDeep(REST.annotationsVar),
+    _.cloneDeep(REST.layoutObs)
   );
   /* create world */
   const world = World.createWorldFromEntireUniverse(universe);
@@ -43,11 +43,11 @@ const defaultBigBang = () => {
 describe("createWorldFromEntireUniverse", () => {
   test("create from REST sample", () => {
     const universe = Universe.createUniverseFromResponse(
-      REST.config,
-      REST.schema,
-      REST.annotationsObs,
-      REST.annotationsVar,
-      REST.layoutObs
+      _.cloneDeep(REST.config),
+      _.cloneDeep(REST.schema),
+      _.cloneDeep(REST.annotationsObs),
+      _.cloneDeep(REST.annotationsVar),
+      _.cloneDeep(REST.layoutObs)
     );
     expect(universe).toBeDefined();
 

client/__tests__/util/stateManager/worldUtil.test.js

@@ -1,62 +0,0 @@
-import {
-  countCategoryValues2D,
-  clearCaches
-} from "../../../src/util/stateManager/worldUtil";
-import * as Dataframe from "../../../src/util/dataframe";
-
-describe("WorldUtil cache management", () => {
-  test("empty", () => {
-    const count = countCategoryValues2D(
-      "a",
-      "b",
-      new Dataframe.Dataframe([0, 0], [])
-    );
-    expect(count).toMatchObject(new Map());
-    expect(count.size).toBe(0);
-  });
-
-  test("simple couts", () => {
-    const df = new Dataframe.Dataframe(
-      [3, 2],
-      [[0, 0, 1], [false, true, false]],
-      null,
-      new Dataframe.KeyIndex(["a", "b"])
-    );
-    const count = countCategoryValues2D("a", "b", df);
-    expect(count).toMatchObject(
-      new Map([
-        [0, new Map([[true, 1], [false, 1]])],
-        [1, new Map([[false, 1]])]
-      ])
-    );
-  });
-
-  test("memo cache clear", () => {
-    clearCaches();
-    const df1 = new Dataframe.Dataframe([0, 0], []);
-    const df2 = new Dataframe.Dataframe(
-      [3, 2],
-      [[0, 0, 1], [false, true, false]],
-      null,
-      new Dataframe.KeyIndex(["a", "b"])
-    );
-
-    const count1 = countCategoryValues2D("a", "b", df1);
-    const count2 = countCategoryValues2D("a", "b", df1);
-    const count3 = countCategoryValues2D("a", "b", df1.clone());
-    const count4 = countCategoryValues2D("a", "b", df2);
-
-    clearCaches();
-    const count10 = countCategoryValues2D("a", "b", df1);
-    const count11 = countCategoryValues2D("a", "b", df2);
-
-    expect(count1).toEqual(count2);
-    expect(count1).toEqual(count3);
-    expect(count1).toEqual(count10);
-    expect(count1).not.toBe(count3);
-    expect(count1).not.toBe(count10);
-
-    expect(count4).toEqual(count11);
-    expect(count4).not.toBe(count11);
-  });
-});

client/__tests__/util/typedCrossfilter/sort.test.js

@@ -180,6 +180,7 @@ describe("lowerBound", () => {
     expect(lowerBound([0, 1, 2, 3], 1, 0, 4)).toEqual(1);
     expect(lowerBound([0, 1, 2, 3], 3, 0, 4)).toEqual(3);
     expect(lowerBound([0, 1, 2, 3], 4, 0, 4)).toEqual(4);
+    expect(lowerBound([0, 1, 2, 3], 4, 0, 3)).toEqual(3);
 
     expect(lowerBound([0, 1, 2, 3, 4], -1, 0, 5)).toEqual(0);
     expect(lowerBound([0, 1, 2, 3, 4], 0, 0, 5)).toEqual(0);
@@ -204,6 +205,7 @@ describe("lowerBound", () => {
     expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
     expect(lowerBound(new Float32Array([0, 1, 2, 3]), 3, 0, 4)).toEqual(3);
     expect(lowerBound(new Float32Array([0, 1, 2, 3]), 4, 0, 4)).toEqual(4);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 4, 0, 3)).toEqual(3);
 
     expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), -1, 0, 5)).toEqual(0);
     expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 0, 0, 5)).toEqual(0);

client/configuration/webpack/webpack.config.dev.js

@@ -37,8 +37,9 @@ module.exports = {
           {
             loader: "css-loader",
             options: {
-              modules: true,
-              localIdentName: "[name]__[local]___[hash:base64:5]"
+              modules: {
+                localIdentName: "[name]__[local]___[hash:base64:5]"
+              }
             }
           }
         ]

client/configuration/webpack/webpack.config.prod.js

@@ -40,9 +40,10 @@ module.exports = {
           {
             loader: "css-loader",
             options: {
-              modules: true,
-              importLoaders: 1,
-              localIdentName: "[name]__[local]___[hash:base64:5]"
+              modules: {
+                localIdentName: "[name]__[local]___[hash:base64:5]"
+              },
+              importLoaders: 1
             }
           }
         ]

client/package.json

@@ -1,11 +1,12 @@
 {
   "name": "cellxgene",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
   "scripts": {
     "backend-dev": "python3.6 -m venv cellxgene && source cellxgene/bin/activate && yes | pip uninstall cellxgene || true && pip install -e .. && cellxgene launch ",
+    "backend-dev-anno": "python3.6 -m venv cellxgene && source cellxgene/bin/activate && yes | pip uninstall cellxgene || true && pip install -e .. && cellxgene launch --experimental-label-file labels.csv ",
     "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
     "clean": "rimraf build",
     "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
@@ -31,87 +32,78 @@
     "eslint-scope": "3.7.1"
   },
   "dependencies": {
-    "@blueprintjs/core": "^3.15.1",
-    "@blueprintjs/icons": "^3.8.0",
-    "@blueprintjs/select": "^3.8.0",
-    "canvas-fit": "^1.5.0",
+    "@blueprintjs/core": "^3.18.1",
+    "@blueprintjs/icons": "^3.10.0",
+    "@blueprintjs/select": "^3.10.0",
     "d3": "^4.10.0",
-    "d3-scale-chromatic": "^1.3.0",
-    "flatbuffers": "^1.10.2",
-    "font-color-contrast": "^1.0.3",
+    "d3-scale-chromatic": "^1.5.0",
+    "flatbuffers": "^1.11.0",
     "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
-    "gl-matrix": "^3.0.0",
+    "gl-matrix": "^3.1.0",
     "gl-vec3": "^1.1.3",
     "is-number": "^7.0.0",
-    "key-pressed": "0.0.1",
-    "lodash": "^4.17.14",
-    "memoize-one": "^5.0.4",
-    "mouse-position": "^2.0.1",
-    "mouse-pressed": "^1.0.0",
-    "orbit-camera": "^1.0.0",
-    "query-string": "^6.5.0",
-    "react": "^16.8.6",
+    "lodash": "^4.17.15",
+    "memoize-one": "^5.1.1",
+    "react": "^16.9.0",
     "react-autocomplete": "^1.7.2",
-    "react-dom": "^16.8.6",
+    "react-dom": "^16.9.0",
     "react-helmet": "^5.2.1",
     "react-icons": "^3.7.0",
-    "react-redux": "^7.0.3",
-    "redux": "^4.0.1",
+    "react-redux": "^7.1.1",
+    "redux": "^4.0.4",
     "redux-thunk": "^2.2.0",
-    "regl": "^1.3.11",
-    "scroll-speed": "^1.0.0",
-    "urijs": "^1.19.0"
+    "regl": "^1.3.13"
   },
   "devDependencies": {
-    "@babel/core": "^7.4.4",
-    "@babel/plugin-proposal-class-properties": "^7.4.4",
-    "@babel/plugin-proposal-decorators": "^7.4.4",
-    "@babel/plugin-proposal-export-namespace-from": "^7.2.0",
+    "@babel/core": "^7.6.0",
+    "@babel/plugin-proposal-class-properties": "^7.5.5",
+    "@babel/plugin-proposal-decorators": "^7.6.0",
+    "@babel/plugin-proposal-export-namespace-from": "^7.5.2",
     "@babel/plugin-proposal-function-bind": "^7.2.0",
     "@babel/plugin-proposal-nullish-coalescing-operator": "^7.4.4",
-    "@babel/plugin-proposal-optional-chaining": "^7.2.0",
-    "@babel/plugin-transform-react-constant-elements": "^7.2.0",
-    "@babel/plugin-transform-runtime": "^7.4.4",
-    "@babel/preset-env": "^7.4.4",
+    "@babel/plugin-proposal-optional-chaining": "^7.6.0",
+    "@babel/plugin-transform-react-constant-elements": "^7.6.0",
+    "@babel/plugin-transform-runtime": "^7.6.0",
+    "@babel/preset-env": "^7.6.0",
     "@babel/preset-react": "^7.0.0",
-    "@babel/register": "^7.4.4",
-    "@babel/runtime": "^7.4.4",
-    "babel-eslint": "^10.0.1",
-    "babel-jest": "^24.8.0",
+    "@babel/register": "^7.6.0",
+    "@babel/runtime": "^7.6.0",
+    "babel-eslint": "^10.0.3",
+    "babel-jest": "^24.9.0",
     "babel-loader": "^8.0.6",
     "babel-preset-modern-browsers": "^14.0.0",
     "chalk": "^2.4.2",
     "connect-history-api-fallback": "^1.6.0",
-    "copy-webpack-plugin": "^5.0.3",
-    "css-loader": "^2.1.1",
-    "eslint": "^5.16.0",
-    "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^4.2.0",
-    "eslint-loader": "^2.1.2",
+    "copy-webpack-plugin": "^5.0.4",
+    "css-loader": "^3.2.0",
+    "eslint": "^6.4.0",
+    "eslint-config-airbnb": "^18.0.1",
+    "eslint-config-prettier": "^6.3.0",
+    "eslint-loader": "^3.0.0",
     "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.17.2",
-    "eslint-plugin-jest": "^22.5.1",
-    "eslint-plugin-jsx-a11y": "^6.2.1",
-    "eslint-plugin-react": "^7.13.0",
-    "express": "^4.14.0",
-    "file-loader": "^3.0.1",
+    "eslint-plugin-import": "^2.18.2",
+    "eslint-plugin-jest": "^22.17.0",
+    "eslint-plugin-jsx-a11y": "^6.2.3",
+    "eslint-plugin-react": "^7.14.3",
+    "express": "^4.17.1",
+    "file-loader": "^4.2.0",
     "html-webpack-inline-source-plugin": "0.0.10",
     "html-webpack-plugin": "^3.2.0",
-    "jest": "^24.8.0",
-    "jest-puppeteer": "^4.1.1",
+    "jest": "^24.9.0",
+    "jest-puppeteer": "^4.3.0",
     "json-loader": "^0.5.4",
-    "mini-css-extract-plugin": "^0.6.0",
-    "puppeteer": "^1.16.0",
-    "rimraf": "^2.6.3",
+    "mini-css-extract-plugin": "^0.8.0",
+    "puppeteer": "^1.20.0",
+    "rimraf": "^3.0.0",
     "serve-favicon": "^2.3.0",
-    "start-server-and-test": "^1.9.0",
-    "style-loader": "^0.23.1",
+    "start-server-and-test": "^1.10.2",
+    "style-loader": "^1.0.0",
     "sw-precache-webpack-plugin": "^0.11.5",
-    "url-loader": "^1.1.0",
-    "webpack": "^4.31.0",
-    "webpack-cli": "^3.3.2",
-    "webpack-dev-middleware": "^3.6.2"
+    "url-loader": "^2.1.0",
+    "webpack": "^4.40.2",
+    "webpack-cli": "^3.3.8",
+    "webpack-dev-middleware": "^3.7.1"
   },
   "jest": {
     "testMatch": [

client/src/actions/index.js

@@ -1,7 +1,7 @@
 // jshint esversion: 6
 import _ from "lodash";
 import * as globals from "../globals";
-import { Universe } from "../util/stateManager";
+import { Universe, MatrixFBS } from "../util/stateManager";
 import {
   catchErrorsWrap,
   doJsonRequest,
@@ -340,11 +340,57 @@ const resetInterface = () => (dispatch, getState) => {
   });
 };
 
+const saveObsAnnotations = () => async (dispatch, getState) => {
+  const { universe } = getState();
+  const { obsAnnotations, schema } = universe;
+
+  dispatch({
+    type: "writable obs annotations - save started"
+  });
+
+  const writableAnnotations = schema.annotations.obs.columns
+    .filter(s => s.writable)
+    .map(s => s.name);
+  const df = obsAnnotations.subset(null, writableAnnotations);
+  const matrix = MatrixFBS.encodeMatrixFBS(df);
+  try {
+    const res = await fetch(
+      `${globals.API.prefix}${globals.API.version}annotations/obs`,
+      {
+        method: "PUT",
+        body: matrix,
+        headers: new Headers({
+          "Content-Type": "application/octet-stream"
+        })
+      }
+    );
+    if (res.ok) {
+      dispatch({
+        type: "writable obs annotations - save complete",
+        obsAnnotations
+      });
+    } else {
+      dispatch({
+        type: "writable obs annotations - save error",
+        message: `HTTP error ${res.status} - ${res.statusText}`,
+        res
+      });
+    }
+  } catch (error) {
+    dispatch({
+      type: "writable obs annotations - save error",
+      message: error.toString(),
+      error
+    });
+  }
+};
+
 export default {
   regraph,
   resetInterface,
   requestSingleGeneExpressionCountsForColoringPOST,
   requestDifferentialExpression,
   requestUserDefinedGene,
-  doInitialDataLoad
+  doInitialDataLoad,
+  saveObsAnnotations
 };

client/src/components/app.js

@@ -8,6 +8,7 @@ import LeftSideBar from "./leftSidebar";
 import Legend from "./continuousLegend";
 import Graph from "./graph/graph";
 import MenuBar from "./menubar";
+import Autosave from "./autosave";
 
 import actions from "../actions";
 
@@ -89,6 +90,7 @@ class App extends React.Component {
           {loading ? null : <LeftSideBar />}
           {loading ? null : <MenuBar />}
           {loading ? null : <Graph key={graphRenderCounter} />}
+          {loading ? null : <Autosave />}
           <Legend />
         </div>
       </Container>

client/src/components/autosave/index.js

@@ -0,0 +1,81 @@
+import React from "react";
+import { connect } from "react-redux";
+
+import actions from "../../actions";
+
+@connect(state => ({
+  universe: state.universe,
+  obsAnnotations: state.universe.obsAnnotations,
+  saveInProgress: state.autosave?.saveInProgress ?? false,
+  lastSavedObsAnnotations: state.autosave?.lastSavedObsAnnotations,
+  error: state.autosave?.error,
+  writableCategoriesEnabled: state.config?.parameters?.["label_file"] ?? false
+}))
+class Autosave extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      timer: null
+    };
+  }
+
+  componentDidMount() {
+    const { writableCategoriesEnabled } = this.props;
+
+    let { timer } = this.state;
+    if (timer) clearInterval(timer);
+    if (writableCategoriesEnabled) {
+      timer = setInterval(this.tick, 2500);
+    } else {
+      timer = null;
+    }
+    this.setState({ timer });
+  }
+
+  componentWillUnmount() {
+    const { timer } = this.state;
+    if (timer) this.clearInterval(timer);
+  }
+
+  tick = () => {
+    const { dispatch, saveInProgress } = this.props;
+    if (this.needToSave() && !saveInProgress) {
+      dispatch(actions.saveObsAnnotations());
+    }
+  };
+
+  needToSave = () => {
+    /* return true if we need to save, false if we don't */
+    const { obsAnnotations, lastSavedObsAnnotations } = this.props;
+    return (
+      lastSavedObsAnnotations && obsAnnotations !== lastSavedObsAnnotations
+    );
+  };
+
+  statusMessage() {
+    const { error } = this.props;
+    if (error) {
+      return `Autosave error: ${error}`;
+    }
+    return this.needToSave() ? "Unsaved" : "All saved";
+  }
+
+  render() {
+    const { writableCategoriesEnabled } = this.props;
+    return writableCategoriesEnabled ? (
+      <div
+        id="autosave"
+        style={{
+          position: "fixed",
+          display: "inherit",
+          right: 5,
+          bottom: 5
+        }}
+      >
+        {this.statusMessage()}
+      </div>
+    ) : null;
+  }
+}
+
+export default Autosave;

client/src/components/brushableHistogram/index.js

@@ -13,19 +13,25 @@ import * as globals from "../../globals";
 import actions from "../../actions";
 import { makeContinuousDimensionName } from "../../util/nameCreators";
 
-@connect(state => ({
-  world: state.world,
-  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
-  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
-  continuousSelection: state.continuousSelection,
-  differential: state.differential,
-  colorAccessor: state.colors.colorAccessor
-}))
-class HistogramBrush extends React.Component {
+@connect((state, ownProps) => {
+  const { isObs, isUserDefined, isDiffExp, field } = ownProps;
+  const myName = makeContinuousDimensionName(
+    { isObs, isUserDefined, isDiffExp },
+    field
+  );
+  return {
+    world: state.world,
+    scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+    scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+    continuousSelectionRange: state.continuousSelection[myName],
+    colorAccessor: state.colors.colorAccessor
+  };
+})
+class HistogramBrush extends React.PureComponent {
   static getColumn(world, field, clipped = true) {
     /*
-    Return the underlying Dataframe column for our field.   By default, 
-    returns the clipped column.   If clipped===false, will return the 
+    Return the underlying Dataframe column for our field.   By default,
+    returns the clipped column.   If clipped===false, will return the
     unclipped column.
     */
     const obsAnnotations = clipped
@@ -84,7 +90,7 @@ class HistogramBrush extends React.Component {
   }
 
   componentDidUpdate(prevProps) {
-    const { field, world, continuousSelection } = this.props;
+    const { field, world } = this.props;
     const { x, y, bins, svgRef } = this._histogram;
     let { brushXselection, brushX } = this.state;
     let forceBrushUpdate = false;
@@ -112,16 +118,8 @@ class HistogramBrush extends React.Component {
     if the selection has changed, ensure that the brush correctly reflects
     the underlying selection.
     */
-    if (
-      forceBrushUpdate ||
-      continuousSelection !== prevProps.continuousSelection
-    ) {
-      const { isObs, isUserDefined, isDiffExp } = this.props;
-      const myName = makeContinuousDimensionName(
-        { isObs, isUserDefined, isDiffExp },
-        field
-      );
-      const range = continuousSelection[myName];
+    const { continuousSelectionRange: range } = this.props;
+    if (forceBrushUpdate || range !== prevProps.continuousSelectionRange) {
       if (brushXselection) {
         const selection = d3.brushSelection(brushXselection.node());
         if (!range && selection) {
@@ -357,14 +355,24 @@ class HistogramBrush extends React.Component {
       .append("g")
       .attr("class", "axis axis--x")
       .attr("transform", `translate(0,${this.height - this.marginBottom})`)
-      .call(d3.axisBottom(x).ticks(5));
+      .call(
+        d3
+          .axisBottom(x)
+          .ticks(5)
+          .tickFormat(d3.format(".0s"))
+      );
 
     /* Y AXIS */
     svg
       .append("g")
       .attr("class", "axis axis--y")
       .attr("transform", `translate(${this.width - this.marginRight},0)`)
-      .call(d3.axisRight(y).ticks(3));
+      .call(
+        d3
+          .axisRight(y)
+          .ticks(3)
+          .tickFormat(d3.format(".0s"))
+      );
 
     /* axis style */
     svg.selectAll(".axis text").style("fill", "rgb(80,80,80)");
@@ -461,7 +469,11 @@ class HistogramBrush extends React.Component {
               remove
             </Button>
           ) : null}
-          <Tooltip content="Use as color scale" position="bottom">
+          <Tooltip
+            content="Use as color scale"
+            position="bottom"
+            hoverOpenDelay={globals.tooltipHoverOpenDelay}
+          >
             <Button
               onClick={this.handleColorAction.bind(this)}
               active={colorAccessor === field}

client/src/components/categorical/categorical.js

@@ -1,27 +1,176 @@
 // jshint esversion: 6
 import React from "react";
 import _ from "lodash";
+import {
+  Button,
+  Tooltip,
+  InputGroup,
+  Dialog,
+  Classes,
+  MenuItem
+} from "@blueprintjs/core";
+import { Select } from "@blueprintjs/select";
 import { connect } from "react-redux";
 import * as globals from "../../globals";
 import Category from "./category";
 
 @connect(state => ({
-  categoricalSelection: state.categoricalSelection
+  categoricalSelection: state.categoricalSelection,
+  writableCategoriesEnabled: state.config?.parameters?.["label_file"] ?? false,
+  schema: state.world?.schema
 }))
 class Categories extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      createAnnoModeActive: false,
+      newCategoryText: "",
+      categoryToDuplicate: null
+    };
+  }
+
+  handleCreateUserAnno = () => {
+    const { dispatch } = this.props;
+    const { newCategoryText, categoryToDuplicate } = this.state;
+    dispatch({
+      type: "annotation: create category",
+      data: newCategoryText,
+      categoryToDuplicate
+    });
+    this.setState({
+      createAnnoModeActive: false,
+      categoryToDuplicate: null,
+      newCategoryText: ""
+    });
+  };
+
+  handleEnableAnnoMode = () => {
+    this.setState({ createAnnoModeActive: true });
+  };
+
+  handleDisableAnnoMode = () => {
+    this.setState({ createAnnoModeActive: false });
+  };
+
+  handleModalDuplicateCategorySelection = d => {
+    this.setState({ categoryToDuplicate: d });
+  };
+
   render() {
-    const { categoricalSelection } = this.props;
+    const { createAnnoModeActive, categoryToDuplicate } = this.state;
+    const {
+      categoricalSelection,
+      writableCategoriesEnabled,
+      schema
+    } = this.props;
     if (!categoricalSelection) return null;
 
+    /* all names, sorted in display order.  Will be rendered in this order */
+    const allCategoryNames = Object.keys(categoricalSelection).sort();
+
     return (
       <div
         style={{
           padding: globals.leftSidebarSectionPadding
         }}
       >
-        {_.map(categoricalSelection, (catState, catName) => (
-          <Category key={catName} metadataField={catName} />
-        ))}
+        {/* READ ONLY CATEGORICAL FIELDS */}
+        {/* this is duplicative but flat, could be abstracted */}
+        {_.map(allCategoryNames, catName =>
+          !schema.annotations.obsByName[catName].writable ? (
+            <Category
+              key={catName}
+              metadataField={catName}
+              createAnnoModeActive={createAnnoModeActive}
+              isUserAnno={false}
+            />
+          ) : null
+        )}
+        {/* WRITEABLE FIELDS */}
+        {_.map(allCategoryNames, catName =>
+          schema.annotations.obsByName[catName].writable ? (
+            <Category
+              key={catName}
+              metadataField={catName}
+              createAnnoModeActive={createAnnoModeActive}
+              isUserAnno
+            />
+          ) : null
+        )}
+        {writableCategoriesEnabled ? (
+          <div>
+            <Dialog
+              icon="tag"
+              title="Create new category"
+              isOpen={createAnnoModeActive}
+              onClose={this.handleDisableAnnoMode}
+            >
+              <form
+                onSubmit={e => {
+                  e.preventDefault();
+                  this.handleCreateUserAnno();
+                }}
+              >
+                <div className={Classes.DIALOG_BODY}>
+                  <div style={{ marginBottom: 20 }}>
+                    <p>New, unique category name:</p>
+                    <InputGroup
+                      autoFocus
+                      onChange={e =>
+                        this.setState({ newCategoryText: e.target.value })
+                      }
+                      leftIcon="tag"
+                    />
+                  </div>
+                  <p>
+                    Optionally duplicate all labels & cell assignments from
+                    existing category into new category:
+                  </p>
+                  <Select
+                    items={allCategoryNames}
+                    filterable={false}
+                    itemRenderer={(d, { handleClick }) => {
+                      return (
+                        <MenuItem onClick={handleClick} key={d} text={d} />
+                      );
+                    }}
+                    noResults={<MenuItem disabled text="No results." />}
+                    onItemSelect={d => {
+                      this.handleModalDuplicateCategorySelection(d);
+                    }}
+                  >
+                    {/* children become the popover target; render value here */}
+                    <Button
+                      text={
+                        categoryToDuplicate || "None (all cells 'unassigned')"
+                      }
+                      rightIcon="double-caret-vertical"
+                    />
+                  </Select>
+                </div>
+                <div className={Classes.DIALOG_FOOTER}>
+                  <div className={Classes.DIALOG_FOOTER_ACTIONS}>
+                    <Tooltip content="Close this dialog without creating a category.">
+                      <Button onClick={this.handleDisableAnnoMode}>
+                        Cancel
+                      </Button>
+                    </Tooltip>
+                    <Button
+                      onClick={this.handleCreateUserAnno}
+                      intent="primary"
+                      type="submit"
+                    >
+                      Create new category
+                    </Button>
+                  </div>
+                </div>
+              </form>
+            </Dialog>
+            <Button onClick={this.handleEnableAnnoMode} intent="primary">
+              Create new category
+            </Button>
+          </div>
+        ) : null}
       </div>
     );
   }

client/src/components/categorical/category.js

@@ -2,7 +2,19 @@ import React from "react";
 import _ from "lodash";
 import { connect } from "react-redux";
 import { FaChevronRight, FaChevronDown } from "react-icons/fa";
-import { Button, Tooltip } from "@blueprintjs/core";
+import {
+  Button,
+  Tooltip,
+  InputGroup,
+  Menu,
+  Dialog,
+  MenuItem,
+  Popover,
+  Classes,
+  Icon,
+  Position,
+  PopoverInteractionKind
+} from "@blueprintjs/core";
 
 import * as globals from "../../globals";
 import Value from "./value";
@@ -10,14 +22,18 @@ import sortedCategoryValues from "./util";
 
 @connect(state => ({
   colorAccessor: state.colors.colorAccessor,
-  categoricalSelection: state.categoricalSelection
+  categoricalSelection: state.categoricalSelection,
+  annotations: state.annotations,
+  universe: state.universe
 }))
 class Category extends React.Component {
   constructor(props) {
     super(props);
     this.state = {
       isChecked: true,
-      isExpanded: false
+      isExpanded: false,
+      newCategoryText: "",
+      newLabelText: ""
     };
   }
 
@@ -51,6 +67,74 @@ class Category extends React.Component {
     }
   }
 
+  activateAddNewLabelMode = () => {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "annotation: activate add new label mode",
+      data: metadataField
+    });
+  };
+
+  disableAddNewLabelMode = () => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "annotation: disable add new label mode"
+    });
+  };
+
+  handleAddNewLabelToCategory = () => {
+    const { dispatch, metadataField } = this.props;
+    const { newLabelText } = this.state;
+    /*
+    XXX TODO - temporary code generates random label string.   Remove
+    when the label creation UI is implemented.
+
+    const { newLabelText } = this.state;
+    */
+    // const newLabelText = `label${Math.random()}`;
+    dispatch({
+      type: "annotation: add new label to category",
+      metadataField,
+      newLabelText
+    });
+    this.setState({ newLabelText: "" });
+  };
+
+  activateEditCategoryMode = () => {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "annotation: activate category edit mode",
+      data: metadataField
+    });
+  };
+
+  disableEditCategoryMode = () => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "annotation: disable category edit mode"
+    });
+  };
+
+  handleEditCategory = () => {
+    const { dispatch, metadataField } = this.props;
+    const { newCategoryText } = this.state;
+
+    dispatch({
+      type: "annotation: category edited",
+      metadataField,
+      newCategoryText,
+      data: newCategoryText
+    });
+  };
+
+  handleDeleteCategory = () => {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "annotation: delete category",
+      metadataField
+    });
+  };
+
   handleColorChange = () => {
     const { dispatch, metadataField } = this.props;
     dispatch({
@@ -88,12 +172,13 @@ class Category extends React.Component {
   }
 
   renderCategoryItems() {
-    const { categoricalSelection, metadataField } = this.props;
+    const { categoricalSelection, metadataField, isUserAnno } = this.props;
 
     const cat = categoricalSelection[metadataField];
     const optTuples = sortedCategoryValues([...cat.categoryValueIndices]);
     return _.map(optTuples, (tuple, i) => (
       <Value
+        isUserAnno={isUserAnno}
         optTuples={optTuples}
         key={tuple[1]}
         metadataField={metadataField}
@@ -104,9 +189,17 @@ class Category extends React.Component {
   }
 
   render() {
-    const { isExpanded, isChecked } = this.state;
-    const { metadataField, colorAccessor, categoricalSelection } = this.props;
+    const { isExpanded, isChecked, newLabelText, newCategoryText } = this.state;
+    const {
+      metadataField,
+      colorAccessor,
+      categoricalSelection,
+      isUserAnno,
+      annotations,
+      universe
+    } = this.props;
     const { isTruncated } = categoricalSelection[metadataField];
+
     return (
       <div
         style={{
@@ -144,7 +237,6 @@ class Category extends React.Component {
               <span className="bp3-control-indicator" />
               {""}
             </label>
-
             <span
               data-testid={`category-expand-${metadataField}`}
               style={{
@@ -152,10 +244,58 @@ class Category extends React.Component {
                 display: "inline-block"
               }}
               onClick={() => {
-                this.setState({ isExpanded: !isExpanded });
+                const editingCategory =
+                  annotations.isEditingCategoryName &&
+                  annotations.categoryBeingEdited === metadataField;
+                if (!editingCategory) {
+                  this.setState({ isExpanded: !isExpanded });
+                }
               }}
             >
-              {metadataField}
+              {isUserAnno ? (
+                <Icon style={{ marginRight: 5 }} icon="tag" iconSize={16} />
+              ) : null}
+
+              {annotations.isEditingCategoryName &&
+              annotations.categoryBeingEdited === metadataField ? (
+                <form
+                  style={{ display: "inline-block" }}
+                  onSubmit={e => {
+                    e.preventDefault();
+                    this.handleEditCategory();
+                  }}
+                >
+                  <InputGroup
+                    style={{ position: "relative", top: -1 }}
+                    ref={input => {
+                      this.editableCategoryInput = input;
+                    }}
+                    small
+                    autoFocus
+                    onChange={e => {
+                      this.setState({
+                        newCategoryText: e.target.value
+                      });
+                    }}
+                    defaultValue={metadataField}
+                    rightElement={
+                      <Button
+                        minimal
+                        disabled={newCategoryText.length === 0}
+                        style={{ position: "relative", top: -1 }}
+                        type="button"
+                        icon="small-tick"
+                        data-testclass="submitCategoryNameEdit"
+                        data-testid="submitCategoryNameEdit"
+                        onClick={this.handleEditCategory}
+                      />
+                    }
+                  />
+                </form>
+              ) : (
+                metadataField
+              )}
+
               {isExpanded ? (
                 <FaChevronDown
                   data-testclass="category-expand-is-expanded"
@@ -169,16 +309,113 @@ class Category extends React.Component {
               )}
             </span>
           </div>
-          <Tooltip content="Use as color scale" position="bottom">
-            <Button
-              data-testclass="colorby"
-              data-testid={`colorby-${metadataField}`}
-              onClick={this.handleColorChange}
-              active={colorAccessor === metadataField}
-              intent={colorAccessor === metadataField ? "primary" : "none"}
-              icon="tint"
-            />
-          </Tooltip>
+          <div>
+            {isUserAnno ? (
+              <>
+                <Dialog
+                  icon="tag"
+                  title="Add new label"
+                  isOpen={annotations.isAddingNewLabel}
+                  onClose={this.disableAddNewLabelMode}
+                >
+                  <form
+                    onSubmit={e => {
+                      e.preventDefault();
+                      this.handleAddNewLabelToCategory();
+                    }}
+                  >
+                    <div className={Classes.DIALOG_BODY}>
+                      <div style={{ marginBottom: 20 }}>
+                        <p>New, unique label name:</p>
+                        <InputGroup
+                          autoFocus
+                          onChange={e =>
+                            this.setState({ newLabelText: e.target.value })
+                          }
+                          leftIcon="tag"
+                        />
+                      </div>
+                    </div>
+                    <div className={Classes.DIALOG_FOOTER}>
+                      <div className={Classes.DIALOG_FOOTER_ACTIONS}>
+                        <Tooltip content="Close this dialog without adding a label.">
+                          <Button onClick={this.disableAddNewLabelMode}>
+                            Cancel
+                          </Button>
+                        </Tooltip>
+                        <Button
+                          disabled={
+                            newLabelText.length === 0 ||
+                            universe.schema.annotations.obsByName[
+                              metadataField
+                            ].categories.indexOf(newLabelText) !== -1
+                          }
+                          onClick={this.handleAddNewLabelToCategory}
+                          intent="primary"
+                          type="submit"
+                        >
+                          Add new label to category
+                        </Button>
+                      </div>
+                    </div>
+                  </form>
+                </Dialog>
+                <Popover
+                  interactionKind={PopoverInteractionKind.HOVER}
+                  boundary="window"
+                  position={Position.RIGHT_TOP}
+                  content={
+                    <Menu>
+                      <MenuItem
+                        icon="tag"
+                        data-testclass="handleAddNewLabelToCategory"
+                        data-testid={`handleAddNewLabelToCategory-${metadataField}`}
+                        onClick={this.activateAddNewLabelMode}
+                        text="Add a new label to this category"
+                      />
+                      <MenuItem
+                        icon="edit"
+                        data-testclass="activateEditCategoryMode"
+                        data-testid={`activateEditCategoryMode-${metadataField}`}
+                        onClick={this.activateEditCategoryMode}
+                        text="Edit this category's name"
+                      />
+                      <MenuItem
+                        icon="delete"
+                        intent="danger"
+                        data-testclass="handleDeleteCategory"
+                        data-testid={`handleDeleteCategory-${metadataField}`}
+                        onClick={this.handleDeleteCategory}
+                        text="Delete this category, all associated labels, and remove all cell assignments"
+                      />
+                    </Menu>
+                  }
+                >
+                  <Button
+                    style={{ marginLeft: 0 }}
+                    data-testclass="seeActions"
+                    data-testid={`seeActions-${metadataField}`}
+                    icon="more"
+                    minimal
+                  />
+                </Popover>
+              </>
+            ) : null}
+            <Tooltip
+              content="Use as color scale"
+              position="bottom"
+              hoverOpenDelay={globals.tooltipHoverOpenDelay}
+            >
+              <Button
+                data-testclass="colorby"
+                data-testid={`colorby-${metadataField}`}
+                onClick={this.handleColorChange}
+                active={colorAccessor === metadataField}
+                intent={colorAccessor === metadataField ? "primary" : "none"}
+                icon="tint"
+              />
+            </Tooltip>
+          </div>
         </div>
         <div style={{ marginLeft: 26 }}>
           {isExpanded ? this.renderCategoryItems() : null}

client/src/components/categorical/occupancy.js

@@ -45,7 +45,10 @@ class Occupancy extends React.Component {
       groupBy
     ); /* Because the signature changes we really need different names for histogram to differentiate signatures  */
 
-    const bins = histogramMap.get(category.categoryValues[categoryIndex]);
+    const categoryValue = category.categoryValues[categoryIndex];
+    const bins = histogramMap.has(categoryValue)
+      ? histogramMap.get(categoryValue)
+      : new Array(50).fill(0);
 
     const xScale = d3
       .scaleLinear()
@@ -102,30 +105,34 @@ class Occupancy extends React.Component {
 
     const occupancy = occupancyMap.get(category.categoryValues[categoryIndex]);
 
-    const x = d3
-      .scaleLinear()
-      /* get all the keys d[1] as an array, then find the sum */
-      .domain([0, d3.sum(Array.from(occupancy.values()))])
-      .range([0, this._WIDTH]);
-    const categories = schema.annotations.obsByName[colorAccessor]?.categories;
+    if (occupancy && occupancy.size > 0) {
+      // not all categories have occupancy, so occupancy may be undefined.
+      const x = d3
+        .scaleLinear()
+        /* get all the keys d[1] as an array, then find the sum */
+        .domain([0, d3.sum(Array.from(occupancy.values()))])
+        .range([0, this._WIDTH]);
+      const categories =
+        schema.annotations.obsByName[colorAccessor]?.categories;
 
-    let currentOffset = 0;
-    const dfColumn = world.obsAnnotations.col(colorAccessor);
-    const categoryValues = dfColumn.summarize().categories;
+      let currentOffset = 0;
+      const dfColumn = world.obsAnnotations.col(colorAccessor);
+      const categoryValues = dfColumn.summarize().categories;
 
-    let o;
-    let scaledValue;
-    let value;
+      let o;
+      let scaledValue;
+      let value;
 
-    for (let i = 0, { length } = categoryValues; i < length; i += 1) {
-      value = categoryValues[i];
-      o = occupancy.get(value);
-      scaledValue = x(o);
-      ctx.fillStyle = o
-        ? colorScale(categories.indexOf(value))
-        : "rgb(255,255,255)";
-      ctx.fillRect(currentOffset, 0, o ? scaledValue : 0, this._HEIGHT);
-      currentOffset += o ? scaledValue : 0;
+      for (let i = 0, { length } = categoryValues; i < length; i += 1) {
+        value = categoryValues[i];
+        o = occupancy.get(value);
+        scaledValue = x(o);
+        ctx.fillStyle = o
+          ? colorScale(categories.indexOf(value))
+          : "rgb(255,255,255)";
+        ctx.fillRect(currentOffset, 0, o ? scaledValue : 0, this._HEIGHT);
+        currentOffset += o ? scaledValue : 0;
+      }
     }
   };
 

client/src/components/categorical/value.js

@@ -1,18 +1,109 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
+
+import {
+  Button,
+  InputGroup,
+  Menu,
+  MenuItem,
+  Popover,
+  Position,
+  Icon,
+  PopoverInteractionKind
+} from "@blueprintjs/core";
 import Occupancy from "./occupancy";
 import * as globals from "../../globals";
 import styles from "./categorical.css";
+import { Tooltip } from "@blueprintjs/core";
 
 @connect(state => ({
   categoricalSelection: state.categoricalSelection,
+  annotations: state.annotations,
   colorScale: state.colors.scale,
   colorAccessor: state.colors.colorAccessor,
   schema: state.world?.schema,
   world: state.world
 }))
 class CategoryValue extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      editedLabelText: ""
+    };
+  }
+
+  handleDeleteValue = () => {
+    const {
+      dispatch,
+      metadataField,
+      categoryIndex,
+      categoricalSelection
+    } = this.props;
+    const category = categoricalSelection[metadataField];
+    const label = category.categoryValues[categoryIndex];
+    dispatch({
+      type: "annotation: delete label",
+      metadataField,
+      label
+    });
+  };
+
+  handleAddCurrentSelectionToThisLabel = () => {
+    const {
+      dispatch,
+      metadataField,
+      categoryIndex,
+      categoricalSelection
+    } = this.props;
+    const category = categoricalSelection[metadataField];
+    const label = category.categoryValues[categoryIndex];
+    dispatch({
+      type: "annotation: label current cell selection",
+      metadataField,
+      categoryIndex,
+      label
+    });
+  };
+
+  handleEditValue = () => {
+    const {
+      dispatch,
+      metadataField,
+      categoryIndex,
+      categoricalSelection
+    } = this.props;
+    const { editedLabelText } = this.state;
+    const category = categoricalSelection[metadataField];
+    const label = category.categoryValues[categoryIndex];
+    dispatch({
+      type: "annotation: label edited",
+      editedLabel: editedLabelText,
+      metadataField,
+      categoryIndex,
+      label
+    });
+    this.setState({ editedLabelText: "" });
+  };
+
+  activateEditLabelMode = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "annotation: activate edit label mode",
+      metadataField,
+      categoryIndex
+    });
+  };
+
+  cancelEdit = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "annotation: cancel edit label mode",
+      metadataField,
+      categoryIndex
+    });
+  };
+
   toggleOff = () => {
     const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
@@ -23,12 +114,12 @@ class CategoryValue extends React.Component {
   };
 
   shouldComponentUpdate = nextProps => {
-    /* 
+    /*
     Checks to see if at least one of the following changed:
     * world state
     * the color accessor (what is currently being colored by)
     * if this catagorical value's selection status has changed
-    
+
     If and only if true, update the component
     */
     const { props } = this;
@@ -45,8 +136,14 @@ class CategoryValue extends React.Component {
 
     const worldChange = props.world !== nextProps.world;
     const colorAccessorChange = props.colorAccessor !== nextProps.colorAccessor;
+    const annotationsChange = props.annotations !== nextProps.annotations;
 
-    return valueSelectionChange || worldChange || colorAccessorChange;
+    return (
+      valueSelectionChange ||
+      worldChange ||
+      colorAccessorChange ||
+      annotationsChange
+    );
   };
 
   toggleOn = () => {
@@ -84,7 +181,9 @@ class CategoryValue extends React.Component {
       colorAccessor,
       colorScale,
       i,
-      schema
+      schema,
+      isUserAnno,
+      annotations
     } = this.props;
 
     if (!categoricalSelection) return null;
@@ -105,6 +204,30 @@ class CategoryValue extends React.Component {
       categories = schema.annotations.obsByName[colorAccessor]?.categories;
     }
 
+    let truncatedString = null;
+
+    if (
+      colorAccessor &&
+      !isColorBy &&
+      displayString.length > globals.categoryLabelDisplayStringShortLength
+    ) {
+      truncatedString = `${displayString.slice(
+        0,
+        globals.categoryLabelDisplayStringShortLength / 2
+      )}…${displayString.slice(
+        -globals.categoryLabelDisplayStringShortLength / 2
+      )}`;
+    } else if (
+      displayString.length > globals.categoryLabelDisplayStringLongLength
+    ) {
+      truncatedString = `${displayString.slice(
+        0,
+        globals.categoryLabelDisplayStringLongLength / 2
+      )}…${displayString.slice(
+        -globals.categoryLabelDisplayStringLongLength / 2
+      )}`;
+    }
+
     return (
       <div
         key={i}
@@ -131,7 +254,7 @@ class CategoryValue extends React.Component {
             justifyContent: "space-between"
           }}
         >
-          <div style={{ display: "flex" }}>
+          <div style={{ display: "flex", alignItems: "baseline" }}>
             <label className="bp3-control bp3-checkbox" style={{ margin: 0 }}>
               <input
                 onChange={selected ? this.toggleOff : this.toggleOn}
@@ -146,16 +269,88 @@ class CategoryValue extends React.Component {
                 onMouseLeave={this.handleMouseEnter}
               />
             </label>
-            <span
-              data-testid={`categorical-value-${metadataField}-${displayString}`}
-              data-testclass="categorical-value"
-              style={{ wordBreak: "break-all" }}
+            <Tooltip
+              content={displayString}
+              disabled={truncatedString === null}
+              hoverOpenDelay={globals.tooltipHoverOpenDelayQuick}
             >
-              {displayString}
-            </span>
+              <span
+                data-testid={`categorical-value-${metadataField}-${displayString}`}
+                data-testclass="categorical-value"
+                style={{
+                  color:
+                    displayString === globals.unassignedCategoryLabel
+                      ? "#ababab"
+                      : "black",
+                  fontStyle:
+                    displayString === globals.unassignedCategoryLabel
+                      ? "italic"
+                      : "normal",
+                  display: "inline-block",
+                  overflow: "hidden",
+                  lineHeight: "1.1em",
+                  height: "1.1em",
+                  wordBreak: "break-all",
+                  verticalAlign: "middle"
+                }}
+              >
+                {annotations.isEditingLabelName &&
+                annotations.labelEditable.category === metadataField &&
+                annotations.labelEditable.label === categoryIndex
+                  ? null
+                  : truncatedString || displayString}
+              </span>
+            </Tooltip>
+            {isUserAnno &&
+            annotations.labelEditable.category === metadataField &&
+            annotations.isEditingLabelName &&
+            annotations.labelEditable.label === categoryIndex ? (
+              <form
+                onSubmit={e => {
+                  e.preventDefault();
+                  this.handleEditValue();
+                }}
+              >
+                <InputGroup
+                  style={{ position: "relative", top: -1 }}
+                  ref={input => {
+                    this.editableInput = input;
+                  }}
+                  small
+                  autoFocus
+                  onChange={e => {
+                    this.setState({ editedLabelText: e.target.value });
+                  }}
+                  defaultValue={displayString}
+                  rightElement={
+                    <Button
+                      minimal
+                      style={{ position: "relative", top: -1 }}
+                      type="button"
+                      icon="small-tick"
+                      data-testclass="submitEdit"
+                      data-testid="submitEdit"
+                      onClick={this.handleEditValue}
+                    />
+                  }
+                />
+              </form>
+            ) : null}
+            {/*
+            CANCEL IT, WITH BUTTON, ESCAPE KEY, CLICK OUT, UNDO?
+
+            <Button
+            minimal
+            style={{ position: "relative", top: -1 }}
+            type="button"
+            icon="cross"
+            data-testclass="submitEdit"
+            data-testid="submitEdit"
+            onClick={this.cancelEdit}
+          /> */}
           </div>
           <span style={{ flexShrink: 0 }}>
-            {colorAccessor && !isColorBy ? (
+            {colorAccessor && !isColorBy && !annotations.isEditingLabelName ? (
               <Occupancy category={category} {...this.props} />
             ) : null}
           </span>
@@ -164,10 +359,22 @@ class CategoryValue extends React.Component {
           <span
             data-testclass="categorical-value-count"
             data-testid={`categorical-value-count-${metadataField}-${displayString}`}
+            style={{
+              color:
+                displayString === globals.unassignedCategoryLabel
+                  ? "#ababab"
+                  : "black",
+              fontStyle:
+                displayString === globals.unassignedCategoryLabel
+                  ? "italic"
+                  : "auto"
+            }}
           >
             {count}
           </span>
+
           <svg
+            display={isColorBy && categories ? "auto" : "none"}
             style={{
               marginLeft: 5,
               width: 11,
@@ -178,6 +385,57 @@ class CategoryValue extends React.Component {
                   : "inherit"
             }}
           />
+          {isUserAnno ? (
+            <span
+              onMouseEnter={this.handleMouseExit}
+              onMouseLeave={this.handleMouseEnter}
+            >
+              <Popover
+                interactionKind={PopoverInteractionKind.HOVER}
+                boundary="window"
+                position={Position.RIGHT_TOP}
+                content={
+                  <Menu>
+                    <MenuItem
+                      icon="plus"
+                      data-testclass="handleAddCurrentSelectionToThisLabel"
+                      data-testid={`handleAddCurrentSelectionToThisLabel-${metadataField}`}
+                      onClick={this.handleAddCurrentSelectionToThisLabel}
+                      text={`Re-label currently selected cells as ${displayString}`}
+                    />
+                    {displayString !== globals.unassignedCategoryLabel ? (
+                      <MenuItem
+                        icon="edit"
+                        text="Edit this label's name"
+                        data-testclass="handleEditValue"
+                        data-testid={`handleEditValue-${metadataField}`}
+                        onClick={this.activateEditLabelMode}
+                      />
+                    ) : null}
+                    {displayString !== globals.unassignedCategoryLabel ? (
+                      <MenuItem
+                        icon="delete"
+                        intent="danger"
+                        data-testclass="handleDeleteValue"
+                        data-testid={`handleDeleteValue-${metadataField}`}
+                        onClick={this.handleDeleteValue}
+                        text="Delete this label, and reassign all cells to type 'unknown'"
+                      />
+                    ) : null}
+                  </Menu>
+                }
+              >
+                <Button
+                  style={{ marginLeft: 0, position: "relative", top: -1 }}
+                  data-testclass="seeActions"
+                  data-testid={`seeActions-${metadataField}`}
+                  icon="more"
+                  small
+                  minimal
+                />
+              </Popover>
+            </span>
+          ) : null}
         </span>
       </div>
     );

client/src/components/continuous/drawAxes.js

@@ -1,83 +0,0 @@
-// jshint esversion: 6
-import * as d3 from "d3";
-import styles from "./parallelCoordinates.css";
-import { yAxis, brushstart } from "./util";
-
-const drawAxes = (
-  svg,
-  ctx,
-  dimensions,
-  xscale,
-  height,
-  width,
-  handleBrushAction,
-  handleColorAction
-) => {
-  /*****************************************
-  ******************************************
-  Handles a brush event, toggling the display of foreground lines.
-  ******************************************
-  ******************************************/
-
-  function brush() {
-    const actives = [];
-    svg
-      .selectAll(".parcoords_axis .parcoords_brush")
-      .filter(() => d3.brushSelection(this))
-      .each(d => {
-        actives.push({
-          dimension: d,
-          extent: d3.brushSelection(this)
-        });
-      });
-    /* fire action, with selected dimensions & their values */
-    handleBrushAction(actives);
-  }
-
-  const axes = svg
-    .selectAll(".parcoords_axis")
-    .data(dimensions)
-    .enter()
-    .append("g")
-    .attr("class", `${styles.axis} parcoords_axis`)
-    .attr("transform", (d, i) => `translate(${xscale(i)})`);
-
-  axes
-    .append("g")
-    .each(d => {
-      const renderAxis =
-        "axis" in d
-          ? d.axis.scale(d.scale) // custom axis
-          : yAxis.scale(d.scale); // default axis
-      d3.select(this).call(renderAxis);
-    })
-    .append("text")
-    .on("click", d => {
-      handleColorAction(d.key);
-    })
-    .attr("class", styles.title)
-    .attr("text-anchor", "start")
-    .text(d => ("description" in d ? `${d.description}  🖌️` : `${d.key}  🖌️`));
-
-  // Add and store a brush for each axis.
-  axes
-    .append("g")
-    .attr("class", `${styles.brush} parcoords_brush`)
-    .each(d => {
-      d3.select(this).call(
-        (d.brush = d3
-          .brushY()
-          .extent([[-10, 0], [10, height]])
-          .on("start", brushstart)
-          .on("brush", brush)
-          .on("end", brush))
-      );
-    })
-    .selectAll("rect")
-    .attr("x", -8)
-    .attr("width", 16);
-
-  return axes;
-};
-
-export default drawAxes;

client/src/components/framework/custom-icons.js

@@ -0,0 +1,16 @@
+/* https://github.com/palantir/blueprint/issues/2348 */
+
+<defs>
+<clipPath id="clip0">
+<rect width="16" height="16" fill="white"/>
+</clipPath>
+</defs>
+<g clip-path="url(#clip0)">
+<rect width="16" height="16" fill="white"/>
+<path d="M1.33415 8.75877C0.939491 8.36411 0.727699 7.82249 0.749957 7.2648L0.926361 2.84501C0.967947 1.80308 1.80308 0.967947 2.84501 0.926361L7.2648 0.749958C7.82249 0.727699 8.36411 0.939492 8.75877 1.33415L14.3595 6.93485C15.1405 7.7159 15.1405 8.98223 14.3595 9.76328L9.76328 14.3595C8.98223 15.1405 7.7159 15.1405 6.93485 14.3595L1.33415 8.75877Z" fill="black"/>
+<circle cx="4.5" cy="4.5" r="1.5" fill="white"/>
+<circle cx="4.5" cy="11.5" r="3.75" stroke="white" stroke-width="0.5"/>
+<circle cx="4.5" cy="11.5" r="3.5" fill="black"/>
+<line x1="4.5" y1="10" x2="4.5" y2="13" stroke="white"/>
+<line x1="3" y1="11.5" x2="6" y2="11.5" stroke="white"/>
+</g>

client/src/components/geneExpression/index.js

@@ -22,6 +22,8 @@ import {
   keepAroundErrorToast
 } from "../framework/toasters";
 
+import { memoize } from "../../util/dataframe/util";
+
 const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
   if (!modifiers.matchesPredicate) {
     return null;
@@ -69,10 +71,24 @@ class GeneExpression extends React.Component {
     super(props);
     this.state = {
       bulkAdd: "",
-      tab: "autosuggest"
+      tab: "autosuggest",
+      activeItem: null
     };
   }
 
+  _genesToUpper = listGenes => {
+    // Has to be a Map to preserve index
+    const upperGenes = new Map();
+    for (let i = 0, { length } = listGenes; i < length; i += 1) {
+      upperGenes.set(listGenes[i].toUpperCase(), i);
+    }
+
+    return upperGenes;
+  };
+
+  // eslint-disable-next-line react/sort-comp
+  _memoGenesToUpper = memoize(this._genesToUpper, arr => arr);
+
   placeholderGeneNames() {
     /*
     return a string containing gene name suggestions for use as a user hint.
@@ -108,6 +124,7 @@ class GeneExpression extends React.Component {
   handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
     const varIndexName = world.schema.annotations.var.index;
+    if (!g) return;
     const gene = g.target;
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
@@ -139,24 +156,37 @@ class GeneExpression extends React.Component {
     */
     if (bulkAdd !== "") {
       const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");
+      if (genes.length === 0) {
+        return keepAroundErrorToast("Must enter a gene name.");
+      }
+      const worldGenes = world.varAnnotations.col(varIndexName).asArray();
+
+      // These gene lists are unique enough where memoization is useless
+      const upperGenes = this._genesToUpper(genes);
+      const upperUserDefinedGenes = this._genesToUpper(userDefinedGenes);
+
+      const upperWorldGenes = this._memoGenesToUpper(worldGenes);
 
       dispatch({ type: "bulk user defined gene start" });
+
       Promise.all(
-        genes.map(gene => {
-          if (gene.length === 0) {
-            return keepAroundErrorToast("Must enter a gene name.");
-          }
-          if (userDefinedGenes.indexOf(gene) !== -1) {
+        [...upperGenes.keys()].map(upperGene => {
+          if (upperUserDefinedGenes.get(upperGene) !== undefined) {
             return keepAroundErrorToast("That gene already exists");
           }
-          if (
-            world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
-          ) {
+
+          const indexOfGene = upperWorldGenes.get(upperGene);
+
+          if (indexOfGene === undefined) {
             return keepAroundErrorToast(
-              `${gene} doesn't appear to be a valid gene name.`
+              `${
+                genes[upperGenes.get(upperGene)]
+              } doesn't appear to be a valid gene name.`
             );
           }
-          return dispatch(actions.requestUserDefinedGene(gene));
+          return dispatch(
+            actions.requestUserDefinedGene(worldGenes[indexOfGene])
+          );
         })
       ).then(
         () => dispatch({ type: "bulk user defined gene complete" }),
@@ -175,7 +205,7 @@ class GeneExpression extends React.Component {
       differential
     } = this.props;
     const varIndexName = world?.schema?.annotations?.var?.index;
-    const { tab, bulkAdd } = this.state;
+    const { tab, bulkAdd, activeItem } = this.state;
 
     return (
       <div>
@@ -229,11 +259,13 @@ class GeneExpression extends React.Component {
                   /* this happens on 'enter' */
                   this.handleClick(g);
                 }}
+                initialContent={<MenuItem disabled text="Enter a gene…" />}
                 inputProps={{ "data-testid": "gene-search" }}
                 inputValueRenderer={g => {
                   return "";
                 }}
                 itemListPredicate={filterGenes}
+                onActiveItemChange={item => this.setState({ activeItem: item })}
                 itemRenderer={renderGene.bind(this)}
                 items={
                   world && world.varAnnotations
@@ -246,6 +278,7 @@ class GeneExpression extends React.Component {
                 className="bp3-button bp3-intent-primary"
                 data-testid={"add-gene"}
                 loading={userDefinedGenesLoading}
+                onClick={() => this.handleClick(activeItem)}
               >
                 Add gene
               </Button>

client/src/components/graph/drawPointsRegl.js

@@ -1,58 +1,67 @@
-// jshint esversion: 6
-const mat4 = require("gl-mat4");
-const vec3 = require("gl-vec3");
-
-// opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js
+import { glPointFlags, glPointSize } from "../../util/glHelpers";
 
 export default function(regl) {
   return regl({
     vert: `
     precision mediump float;
+
     attribute vec2 position;
     attribute vec3 color;
-    attribute float size;
+    attribute float flag;
+
     uniform float distance;
-    uniform mat4 projection, view;
-    varying vec3 fragColor;
+    uniform mat3 projView;
+    uniform float nPoints;
+    uniform float minViewportDimension;
+
+    varying vec4 fragColor;
+
+    const float zBottom = 0.99;
+    const float zMiddle = 0.;
+    const float zTop = -1.;
+
+    // import getFlags()
+    ${glPointFlags}
+
+    // get pointSize()
+    ${glPointSize}
+
     void main() {
-      gl_PointSize = 7.0 / pow(distance, 2.5) + size;
-      gl_Position = projection * view * vec4(position.x, -position.y, 0, 1);
-      fragColor = color;
+      bool isNaN, isSelected, isHighlight;
+      getFlags(flag, isNaN, isSelected, isHighlight);
+
+      float size = pointSize(nPoints, minViewportDimension, isSelected, isHighlight);
+      gl_PointSize = size * pow(distance, 0.5);
+
+      float z = isNaN ? zBottom : (isHighlight ? zTop : zMiddle);
+      vec3 xy = projView * vec3(position, 1.);
+      gl_Position = vec4(xy.xy, z, 1.);
+
+      float alpha = isNaN ? 0.9 : 1.0;
+      fragColor = vec4(color, alpha);
     }`,
 
     frag: `
     precision mediump float;
-    varying vec3 fragColor;
+    varying vec4 fragColor;
     void main() {
       if (length(gl_PointCoord.xy - 0.5) > 0.5) {
         discard;
       }
-      gl_FragColor = vec4(fragColor, 1);
+      gl_FragColor = fragColor;
     }`,
 
     attributes: {
       position: regl.prop("position"),
       color: regl.prop("color"),
-      size: regl.prop("size")
+      flag: regl.prop("flag")
     },
 
     uniforms: {
       distance: regl.prop("distance"),
-      view: regl.prop("view"),
-      projection: ({ viewportWidth, viewportHeight }) => {
-        const aspectRatio = viewportWidth / viewportHeight;
-        let m = mat4.perspective(
-          [],
-          Math.PI / 2,
-          viewportWidth / viewportHeight,
-          0.01,
-          1000
-        );
-        if (aspectRatio < 1) {
-          m = mat4.scale(m, m, vec3.fromValues(1, 1, 1 / aspectRatio));
-        }
-        return m;
-      }
+      projView: regl.prop("projView"),
+      nPoints: regl.prop("nPoints"),
+      minViewportDimension: regl.prop("minViewportDimension")
     },
 
     count: regl.prop("count"),

client/src/components/graph/graph.js

@@ -2,21 +2,76 @@
 import React from "react";
 import * as d3 from "d3";
 import { connect } from "react-redux";
-import mat4 from "gl-mat4";
+import { mat3, vec2 } from "gl-matrix";
 import _regl from "regl";
 import memoize from "memoize-one";
 
 import * as globals from "../../globals";
 import setupSVGandBrushElements from "./setupSVGandBrush";
 import setupCentroidSVG from "./setupCentroidSVG";
-import actions from "../../actions";
 import _camera from "../../util/camera";
 import _drawPoints from "./drawPointsRegl";
-import scaleLinear from "../../util/scaleLinear";
+import { isTypedArray } from "../../util/typeHelpers";
 
-/* https://bl.ocks.org/mbostock/9078690 - quadtree for onClick / hover selections */
+/*
+Simple 2D transforms control all point painting.  There are three:
+  * model - convert from underlying per-point coordinate to a layout.
+    Currently used to move from data to webgl coordinate system.
+  * camera - apply a 2D camera transformation (pan, zoom)
+  * projection - apply any transformation required for screen size and layout
+*/
+
+function createProjectionTF(viewportWidth, viewportHeight) {
+  /*
+  the projection transform accounts for the screen size & other layout
+  */
+  const fractionToUse = 0.95; // fraction of min dimension to use
+  const topGutterSizePx = 32; // toolbar box height
+  const heightMinusGutter = viewportHeight - topGutterSizePx;
+  const minDim = Math.min(viewportWidth, heightMinusGutter);
+  const aspectScale = [
+    (fractionToUse * minDim) / viewportWidth,
+    (fractionToUse * minDim) / viewportHeight
+  ];
+  const m = mat3.create();
+  mat3.fromTranslation(m, [
+    0,
+    -topGutterSizePx / viewportHeight / aspectScale[1]
+  ]);
+  mat3.scale(m, m, aspectScale);
+  return m;
+}
+
+function createModelTF() {
+  /*
+  preallocate coordinate system transformation between data and gl.
+  Data arrives in a [0,1] range, and we operate elsewhere in [-1,1].
+  */
+  const m = mat3.fromScaling(mat3.create(), [2, 2]);
+  mat3.translate(m, m, [-0.5, -0.5]);
+  return m;
+}
+
+function renderThrottle(callback) {
+  /*
+  This wraps a call to requestAnimationFrame(), enforcing a single
+  render callback at any given time (ie, you can call this any number
+  of times, and it will coallesce multiple inter-frame calls into a
+  single render).
+  */
+  let rafCurrentlyInProgress = null;
+  return function f() {
+    if (rafCurrentlyInProgress) return;
+    const context = this;
+    rafCurrentlyInProgress = window.requestAnimationFrame(() => {
+      callback.apply(context);
+      rafCurrentlyInProgress = null;
+    });
+  };
+}
 
 @connect(state => ({
+  universe: state.universe,
   world: state.world,
   crossfilter: state.crossfilter,
   responsive: state.responsive,
@@ -28,15 +83,17 @@ import scaleLinear from "../../util/scaleLinear";
   graphInteractionMode: state.controls.graphInteractionMode,
   colorAccessor: state.colors.colorAccessor
 }))
-class Graph extends React.Component {
-  computePointPositions = memoize((X, Y, scaleX, scaleY) => {
+class Graph extends React.PureComponent {
+  computePointPositions = memoize((X, Y, modelTF) => {
     /*
-    compute webgl coordinate buffer for each point
+    compute the model coordinate for each point
     */
     const positions = new Float32Array(2 * X.length);
     for (let i = 0, len = X.length; i < len; i += 1) {
-      positions[2 * i] = scaleX(X[i]);
-      positions[2 * i + 1] = scaleY(Y[i]);
+      const p = vec2.fromValues(X[i], Y[i]);
+      vec2.transformMat3(p, p, modelTF);
+      positions[2 * i] = p[0];
+      positions[2 * i + 1] = p[1];
     }
     return positions;
   });
@@ -52,38 +109,65 @@ class Graph extends React.Component {
     return colors;
   });
 
-  computePointSizesFromCrossfilter = memoize((len, crossfilter) => {
-    const sizes = new Float32Array(len);
-    crossfilter.fillByIsSelected(sizes, 4, 0.2);
-
-    return sizes;
-  });
-
-  computePointSizes = memoize(
-    (len, crossfilter, metadataField, categoryField) => {
-      /*
-    compute webgl dot size for each point
-    */
-
-      const selectionSizes = this.computePointSizesFromCrossfilter(
-        len,
-        crossfilter
+  computeSelectedFlags = memoize(
+    (crossfilter, flagSelected, flagUnselected) => {
+      const x = crossfilter.fillByIsSelected(
+        new Float32Array(crossfilter.size()),
+        flagSelected,
+        flagUnselected
       );
-      let sizes;
+      return x;
+    }
+  );
 
-      if (metadataField && categoryField) {
-        sizes = selectionSizes.slice();
-        const valuesArr = crossfilter.data.col(metadataField).asArray();
+  computePointFlags = memoize(
+    (world, crossfilter, colorAccessor, centroidLabel) => {
+      /*
+      We communicate with the shader using three flags:
+      - isNaN -- the value is a NaN. Only makes sense when we have a colorAccessor
+      - isSelected -- the value is selected
+      - isHightlighted -- the value is highlighted in the UI (orthogonal from selection highlighting)
 
-        for (let i = 0; i < len; i += 1) {
-          if (valuesArr[i] === categoryField) {
-            sizes[i] = 10;
+      Due to constraints in webgl vertex shader attributes, these are encoded in a float, "kinda"
+      like bitmasks.
+
+      We also have separate code paths for generating flags for categorical and
+      continuous metadata, as they rely on different tests, and some of the flags
+      (eg, isNaN) are meaningless in the face of categorical metadata.
+      */
+
+      const flagSelected = 1;
+      const flagNaN = 2;
+      const flagHighlight = 4;
+
+      const flags = this.computeSelectedFlags(
+        crossfilter,
+        flagSelected,
+        0
+      ).slice();
+
+      const { metadataField, categoryField } = centroidLabel;
+      const highlightData = metadataField
+        ? world.obsAnnotations.col(metadataField)?.asArray()
+        : null;
+      const colorByColumn = colorAccessor
+        ? world.obsAnnotations.col(colorAccessor)?.asArray() ||
+          world.varData.col(colorAccessor)?.asArray()
+        : null;
+      const colorByData =
+        colorByColumn && isTypedArray(colorByColumn) ? colorByColumn : null;
+
+      if (colorByData || highlightData) {
+        for (let i = 0, len = flags.length; i < len; i += 1) {
+          if (highlightData) {
+            flags[i] += highlightData[i] === categoryField ? flagHighlight : 0;
+          }
+          if (colorByData) {
+            flags[i] += Number.isFinite(colorByData[i]) ? 0 : flagNaN;
           }
         }
-      } else {
-        sizes = selectionSizes;
       }
-      return sizes;
+      return flags;
     }
   );
 
@@ -97,7 +181,8 @@ class Graph extends React.Component {
       Y: null,
       positions: null,
       colors: null,
-      sizes: null
+      sizes: null,
+      flags: null
     };
     this.state = {
       toolSVG: null,
@@ -108,55 +193,44 @@ class Graph extends React.Component {
   }
 
   componentDidMount() {
-    // setup canvas and camera
-    const camera = _camera(this.reglCanvas, { scale: true, rotate: false });
+    // setup canvas, webgl draw function and camera
+    const camera = _camera(this.reglCanvas);
     const regl = _regl(this.reglCanvas);
-
     const drawPoints = _drawPoints(regl);
 
-    // preallocate buffers
+    // preallocate webgl buffers
     const pointBuffer = regl.buffer();
     const colorBuffer = regl.buffer();
-    const sizeBuffer = regl.buffer();
+    const flagBuffer = regl.buffer();
 
-    // preallocate coordinate system transformation between data and gl
-    const fractionToUse = 0.93; // fraction of dimension to use
-    const shiftForMenuBar = 0.05;
-    const transform = {
-      glScaleX: scaleLinear([0, 1], [-1 * fractionToUse, 1 * fractionToUse]),
-      glScaleY: scaleLinear(
-        [0, 1],
-        [
-          (1 + shiftForMenuBar) * fractionToUse,
-          (-1 + shiftForMenuBar) * fractionToUse
-        ]
-      )
-    };
+    // create all default rendering transformations
+    const modelTF = createModelTF();
+    const projectionTF = createProjectionTF(
+      this.reglCanvas.width,
+      this.reglCanvas.height
+    );
 
-    /* first time, but this duplicates above function, should be possile to avoid this */
-    const reglRender = regl.frame(() => {
-      this.reglDraw(
-        regl,
-        drawPoints,
-        sizeBuffer,
-        colorBuffer,
-        pointBuffer,
-        camera
-      );
-      camera.tick();
-    });
-
-    this.reglRenderState = "rendering";
+    // initial draw to canvas
+    this.renderPoints(
+      regl,
+      drawPoints,
+      colorBuffer,
+      pointBuffer,
+      flagBuffer,
+      camera,
+      projectionTF
+    );
 
     this.setState({
       regl,
       drawPoints,
       pointBuffer,
       colorBuffer,
-      sizeBuffer,
+      flagBuffer,
       camera,
-      reglRender,
-      transform
+      modelTF,
+      modelInvTF: mat3.invert([], modelTF),
+      projectionTF
     });
   }
 
@@ -174,50 +248,47 @@ class Graph extends React.Component {
       colorAccessor,
       centroidLabel
     } = this.props;
-    const { reglRender, mode, regl, toolSVG, centroidSVG } = this.state;
+    const { regl, toolSVG, centroidSVG } = this.state;
     let stateChanges = {};
 
-    if (reglRender) {
-      if (
-        // If it IS RENDERING and it is NOT IN ZOOM mode, stop rendering.
-        this.reglRenderState === "rendering" &&
-        graphInteractionMode !== "zoom"
-      ) {
-        reglRender.cancel();
-        this.reglRenderState = "paused";
-      }
-
-      if (
-        // If it is NOT RENDERING and it IS IN ZOOM mode, start rendering
-        this.reglRenderState !== "rendering" &&
-        graphInteractionMode === "zoom"
-      ) {
-        this.restartReglLoop();
-        this.reglRenderState = "rendering";
-      }
-    }
-
     if (regl && world) {
-      /* update the regl state */
+      /* update the regl and point rendering state */
       const { obsLayout, nObs } = world;
       const {
         drawPoints,
-        transform,
         camera,
         pointBuffer,
         colorBuffer,
-        sizeBuffer
+        flagBuffer,
+        modelTF
       } = this.state;
+      let { projectionTF } = this.state;
+      let needsRepaint = false;
+
+      if (
+        prevProps.responsive.height !== responsive.height ||
+        prevProps.responsive.width !== responsive.width
+      ) {
+        projectionTF = createProjectionTF(
+          this.reglCanvas.width,
+          this.reglCanvas.height
+        );
+        needsRepaint = true;
+        stateChanges = {
+          ...stateChanges,
+          projectionTF
+        };
+      }
 
       /* coordinates for each point */
-      const { glScaleX, glScaleY } = transform;
       const X = obsLayout.col(layoutChoice.currentDimNames[0]).asArray();
       const Y = obsLayout.col(layoutChoice.currentDimNames[1]).asArray();
-      const newPositions = this.computePointPositions(X, Y, glScaleX, glScaleY);
+      const newPositions = this.computePointPositions(X, Y, modelTF);
       if (renderCache.positions !== newPositions) {
         /* update our cache & GL if the buffer changes */
         renderCache.positions = newPositions;
         pointBuffer({ data: newPositions, dimension: 2 });
+        needsRepaint = true;
       }
 
       /* colors for each point */
@@ -226,105 +297,52 @@ class Graph extends React.Component {
         /* update our cache & GL if the buffer changes */
         renderCache.colors = newColors;
         colorBuffer({ data: newColors, dimension: 3 });
+        needsRepaint = true;
       }
 
-      /* sizes for each point */
-      const { metadataField, categoryField } = centroidLabel;
-      const newSizes = this.computePointSizes(
-        nObs,
+      /* flags for each point */
+      const newFlags = this.computePointFlags(
+        world,
         crossfilter,
-        metadataField,
-        categoryField
+        colorAccessor,
+        centroidLabel
       );
-      if (renderCache.sizes !== newSizes) {
-        /* update our cache & GL if the buffer changes */
-        renderCache.size = newSizes;
-        sizeBuffer({ data: newSizes, dimension: 1 });
+      if (renderCache.flags !== newFlags) {
+        renderCache.flags = newFlags;
+        flagBuffer({ data: newFlags, dimension: 1 });
+        needsRepaint = true;
       }
 
       this.count = nObs;
 
-      regl._refresh();
-      this.reglDraw(
-        regl,
-        drawPoints,
-        sizeBuffer,
-        colorBuffer,
-        pointBuffer,
-        camera
-      );
+      if (needsRepaint) {
+        this.renderPoints(
+          regl,
+          drawPoints,
+          colorBuffer,
+          pointBuffer,
+          flagBuffer,
+          camera,
+          projectionTF
+        );
+      }
     }
 
-    const createToolSVG = () => {
-      /* clear out whatever was on the div, even if nothing, but usually the brushes etc */
-      d3.select("#graphAttachPoint")
-        .select("#tool")
-        .remove();
-
-      let handleStart;
-      let handleDrag;
-      let handleEnd;
-      let handleCancel;
-      if (selectionTool === "brush") {
-        handleStart = this.handleBrushStartAction.bind(this);
-        handleDrag = this.handleBrushDragAction.bind(this);
-        handleEnd = this.handleBrushEndAction.bind(this);
-      } else {
-        handleStart = this.handleLassoStart.bind(this);
-        handleEnd = this.handleLassoEnd.bind(this);
-        handleCancel = this.handleLassoCancel.bind(this);
-      }
-
-      const { svg: newToolSVG, tool, container } = setupSVGandBrushElements(
-        selectionTool,
-        handleStart,
-        handleDrag,
-        handleEnd,
-        handleCancel,
-        responsive,
-        this.graphPaddingRight,
-        graphInteractionMode
-      );
-
-      stateChanges = { ...stateChanges, toolSVG: newToolSVG, tool, container };
-    };
-
-    const createCentroidSVG = () => {
-      d3.select("#graphAttachPoint")
-        .select("#centroid-container")
-        .remove();
-
-      if (centroidLabel.metadataField === "" || !centroidLabel.centroidXY) {
-        return;
-      }
-
-      const centroidScreen = this.mapPointToScreen(centroidLabel.centroidXY);
-
-      const newCentroidSVG = setupCentroidSVG(
-        responsive,
-        this.graphPaddingRight,
-        centroidScreen,
-        centroidLabel.categoryField,
-        colorAccessor
-      );
-
-      stateChanges = { ...stateChanges, centroidSVG: newCentroidSVG };
-    };
-
-    // Centroid SVG creation is disabled for now but should go into the first and third cases if enabled
+    // Centroid SVG creation is disabled for now but should go into the
+    // first and third cases if enabled
     if (
       prevProps.responsive.height !== responsive.height ||
       prevProps.responsive.width !== responsive.width
     ) {
       // If the window size has changed we want to recreate all SVGs
-      createToolSVG();
+      stateChanges = { ...stateChanges, ...this.createToolSVG() };
     } else if (
       (responsive.height && responsive.width && !toolSVG) ||
       selectionTool !== prevProps.selectionTool ||
       prevProps.graphInteractionMode !== graphInteractionMode
     ) {
       // first time or change of selection tool6
-      createToolSVG();
+      stateChanges = { ...stateChanges, ...this.createToolSVG() };
     } else if (
       centroidLabel !== prevProps.centroidLabel ||
       (responsive.height && responsive.width && !centroidSVG)
@@ -353,6 +371,83 @@ class Graph extends React.Component {
     }
   }
 
+  handleCanvasEvent = e => {
+    const { camera, projectionTF } = this.state;
+    if (e.type !== "wheel") e.preventDefault();
+    if (camera.handleEvent(e, projectionTF)) {
+      this.renderCanvas();
+    }
+  };
+
+  createToolSVG() {
+    /*
+    Called from componentDidUpdate. Create the tool SVG, and return any
+    state changes that should be passed to setState().
+    */
+    const { responsive, selectionTool, graphInteractionMode } = this.props;
+
+    /* clear out whatever was on the div, even if nothing, but usually the brushes etc */
+    d3.select("#graphAttachPoint")
+      .select("#tool")
+      .remove();
+
+    let handleStart;
+    let handleDrag;
+    let handleEnd;
+    let handleCancel;
+    if (selectionTool === "brush") {
+      handleStart = this.handleBrushStartAction.bind(this);
+      handleDrag = this.handleBrushDragAction.bind(this);
+      handleEnd = this.handleBrushEndAction.bind(this);
+    } else {
+      handleStart = this.handleLassoStart.bind(this);
+      handleEnd = this.handleLassoEnd.bind(this);
+      handleCancel = this.handleLassoCancel.bind(this);
+    }
+
+    const { svg: newToolSVG, tool, container } = setupSVGandBrushElements(
+      selectionTool,
+      handleStart,
+      handleDrag,
+      handleEnd,
+      handleCancel,
+      responsive,
+      this.graphPaddingRight,
+      graphInteractionMode
+    );
+
+    return { toolSVG: newToolSVG, tool, container };
+  }
+
+  createCentroidSVG() {
+    /*
+    Called from componentDidUpdate. Create the centroid SVG, and return any
+    state changes that should be passed to setState().
+
+    CURRENTLY UNUSED
+    */
+    const { responsive, centroidLabel, colorAccessor } = this.props;
+    d3.select("#graphAttachPoint")
+      .select("#centroid-container")
+      .remove();
+
+    if (centroidLabel.metadataField === "" || !centroidLabel.centroidXY) {
+      return {};
+    }
+
+    const centroidScreen = this.mapPointToScreen(centroidLabel.centroidXY);
+
+    const newCentroidSVG = setupCentroidSVG(
+      responsive,
+      this.graphPaddingRight,
+      centroidScreen,
+      centroidLabel.categoryField,
+      colorAccessor
+    );
+
+    return { centroidSVG: newCentroidSVG };
+  }
+
   brushToolUpdate(tool, container) {
     /*
     this is called from componentDidUpdate(), so be very careful using
@@ -433,76 +528,25 @@ class Graph extends React.Component {
     }
   }
 
-  reglDraw(regl, drawPoints, sizeBuffer, colorBuffer, pointBuffer, camera) {
-    regl.clear({
-      depth: 1,
-      color: [1, 1, 1, 1]
-    });
-    drawPoints({
-      size: sizeBuffer,
-      distance: camera.distance,
-      color: colorBuffer,
-      position: pointBuffer,
-      count: this.count,
-      view: camera.view()
-    });
-  }
-
-  restartReglLoop() {
-    const {
-      regl,
-      drawPoints,
-      sizeBuffer,
-      colorBuffer,
-      pointBuffer,
-      camera
-    } = this.state;
-    const reglRender = regl.frame(() => {
-      this.reglDraw(
-        regl,
-        drawPoints,
-        sizeBuffer,
-        colorBuffer,
-        pointBuffer,
-        camera
-      );
-      camera.tick();
-    });
-
-    this.reglRenderState = "rendering";
-
-    this.setState({
-      reglRender
-    });
-  }
-
   mapScreenToPoint(pin) {
     /*
     Map an XY coordinates from screen domain to cell/point range,
     accounting for current pan/zoom camera.
     */
+
     const { responsive } = this.props;
-    const { regl, camera, transform } = this.state;
-    const { glScaleX, glScaleY } = transform;
+    const { camera, projectionTF, modelInvTF } = this.state;
+    const cameraInvTF = camera.invView();
 
-    const gl = regl._gl;
-
-    // get aspect ratio
-    const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
-    const scale = aspect < 1 ? 1 / aspect : 1;
-
-    // compute inverse view matrix
-    const inverse = mat4.invert([], camera.view());
-
-    // transform screen coordinates -> cell coordinates
+    /* screen -> gl */
     const x = (2 * pin[0]) / (responsive.width - this.graphPaddingRight) - 1;
     const y = 2 * (1 - pin[1] / (responsive.height - this.graphPaddingTop)) - 1;
-    const pout = [
-      x * inverse[14] * aspect * scale + inverse[12],
-      -(y * inverse[14] * scale + inverse[13])
-    ];
 
-    const xy = [glScaleX.invert(pout[0]), glScaleY.invert(pout[1])];
+    const xy = vec2.fromValues(x, y);
+    const projectionInvTF = mat3.invert(mat3.create(), projectionTF);
+    vec2.transformMat3(xy, xy, projectionInvTF);
+    vec2.transformMat3(xy, xy, cameraInvTF);
+    vec2.transformMat3(xy, xy, modelInvTF);
     return xy;
   }
 
@@ -511,29 +555,21 @@ class Graph extends React.Component {
     Map an XY coordinate from cell/point domain to screen range.  Inverse
     of mapScreenToPoint()
     */
+
     const { responsive } = this.props;
-    const { regl, camera, transform } = this.state;
-    const { glScaleX, glScaleY } = transform;
+    const { camera, projectionTF, modelTF } = this.state;
+    const cameraTF = camera.view();
 
-    const gl = regl._gl;
-
-    // get aspect ratio
-    const aspect = gl.drawingBufferWidth / gl.drawingBufferHeight;
-    const scale = aspect < 1 ? 1 / aspect : 1;
-
-    // compute inverse view matrix
-    const inverse = mat4.invert([], camera.view());
-
-    // variable names are choosen to reflect inverse of those used
-    // in mapScreenToPoint().
-    const pout = [glScaleX(xyCell[0]), glScaleY(xyCell[1])];
-    const x = (pout[0] - inverse[12]) / aspect / scale / inverse[14];
-    const y = (-pout[1] - inverse[13]) / scale / inverse[14];
+    const xy = vec2.transformMat3(vec2.create(), xyCell, modelTF);
+    vec2.transformMat3(xy, xy, cameraTF);
+    vec2.transformMat3(xy, xy, projectionTF);
 
     const pin = [
-      Math.round(((x + 1) * (responsive.width - this.graphPaddingRight)) / 2),
       Math.round(
-        -((y + 1) / 2 - 1) * (responsive.height - this.graphPaddingTop)
+        ((xy[0] + 1) * (responsive.width - this.graphPaddingRight)) / 2
+      ),
+      Math.round(
+        -((xy[1] + 1) / 2 - 1) * (responsive.height - this.graphPaddingTop)
       )
     ];
     return pin;
@@ -654,8 +690,61 @@ class Graph extends React.Component {
     });
   }
 
+  renderPoints(
+    regl,
+    drawPoints,
+    colorBuffer,
+    pointBuffer,
+    flagBuffer,
+    camera,
+    projectionTF
+  ) {
+    const { universe } = this.props;
+    if (!this.reglCanvas || !universe) return;
+    const cameraTF = camera.view();
+    const projView = mat3.multiply(mat3.create(), projectionTF, cameraTF);
+    const { width, height } = this.reglCanvas;
+    regl.poll();
+    regl.clear({
+      depth: 1,
+      color: [1, 1, 1, 1]
+    });
+    drawPoints({
+      distance: camera.distance(),
+      color: colorBuffer,
+      position: pointBuffer,
+      flag: flagBuffer,
+      count: this.count,
+      projView,
+      nPoints: universe.nObs,
+      minViewportDimension: Math.min(width || 800, height || 600)
+    });
+    regl._gl.flush();
+  }
+
+  renderCanvas = renderThrottle(() => {
+    const {
+      regl,
+      drawPoints,
+      colorBuffer,
+      pointBuffer,
+      flagBuffer,
+      camera,
+      projectionTF
+    } = this.state;
+    this.renderPoints(
+      regl,
+      drawPoints,
+      colorBuffer,
+      pointBuffer,
+      flagBuffer,
+      camera,
+      projectionTF
+    );
+  });
+
   render() {
-    const { responsive, graphInteractionMode } = this.props;
+    const { responsive } = this.props;
 
     return (
       <div id="graphWrapper">
@@ -676,6 +765,11 @@ class Graph extends React.Component {
               ref={canvas => {
                 this.reglCanvas = canvas;
               }}
+              onMouseDown={this.handleCanvasEvent}
+              onMouseUp={this.handleCanvasEvent}
+              onMouseMove={this.handleCanvasEvent}
+              onDoubleClick={this.handleCanvasEvent}
+              onWheel={this.handleCanvasEvent}
             />
           </div>
         </div>

client/src/components/leftSidebar/topLeftLogoAndTitle.js

@@ -7,15 +7,26 @@ import Logo from "../framework/logo";
 @connect(state => ({
   responsive: state.responsive,
   datasetTitle: state.config?.displayNames?.dataset ?? "",
+  aboutURL: state.config?.links?.["about-dataset"],
   scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor
 }))
 class LeftSideBar extends React.Component {
   render() {
-    const { datasetTitle } = this.props;
+    const { datasetTitle, aboutURL } = this.props;
 
     const paddingToAvoidScrollBar = 15;
 
+    const displayTitle =
+      datasetTitle.length > globals.datasetTitleMaxCharacterCount
+        ? `${datasetTitle.substring(
+            0,
+            Math.floor(globals.datasetTitleMaxCharacterCount / 2)
+          )}…${datasetTitle.slice(
+            -Math.floor(globals.datasetTitleMaxCharacterCount / 2)
+          )}`
+        : datasetTitle;
+
     return (
       <div
         style={{
@@ -41,7 +52,8 @@ class LeftSideBar extends React.Component {
             userSelect: "none"
           }}
         >
-          cell<span
+          cell
+          <span
             style={{
               position: "relative",
               top: 1,
@@ -50,19 +62,26 @@ class LeftSideBar extends React.Component {
             }}
           >
             ×
-          </span>gene
+          </span>
+          gene
         </span>
-        <span
+        <div
           data-testid="header"
           style={{
             fontSize: 14,
             position: "relative",
-            marginLeft: 7,
-            top: -8
+            top: -6,
+            display: "inline-block",
+            width: "190px",
+            marginLeft: "7px",
+            height: "1.1em",
+            overflow: "hidden",
+            wordBreak: "break-all"
           }}
+          title={datasetTitle}
         >
-          {datasetTitle}
-        </span>
+          {aboutURL ? <a href={aboutURL}>{displayTitle}</a> : displayTitle}
+        </div>
       </div>
     );
   }

client/src/components/menubar/cellSetButtons.js

@@ -3,9 +3,10 @@ import React from "react";
 import { AnchorButton, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
 import { World } from "../../util/stateManager";
+import { tooltipHoverOpenDelay } from "../../globals";
 
 @connect()
-class CellSetButton extends React.Component {
+class CellSetButton extends React.PureComponent {
   set() {
     const {
       differential,
@@ -38,7 +39,8 @@ class CellSetButton extends React.Component {
     return (
       <Tooltip
         content="Save current selection for differential expression computation"
-        position="top"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
       >
         <AnchorButton
           type="button"

client/src/components/menubar/clip.js

@@ -5,8 +5,10 @@ import {
   Button,
   Popover,
   NumericInput,
-  Icon
+  Icon,
+  Tooltip
 } from "@blueprintjs/core";
+import { tooltipHoverOpenDelay } from "../../globals";
 
 function Clip(props) {
   const {
@@ -40,14 +42,20 @@ function Clip(props) {
     >
       <Popover
         target={
-          <Button
-            type="button"
-            data-testid="visualization-settings"
-            className={`bp3-button bp3-icon-timeline-bar-chart ${activeClipClass}`}
-            style={{
-              cursor: "pointer"
-            }}
-          />
+          <Tooltip
+            content="Clip all continuous values to a percentile range"
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelay}
+          >
+            <Button
+              type="button"
+              data-testid="visualization-settings"
+              className={`bp3-button bp3-icon-timeline-bar-chart ${activeClipClass}`}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
         }
         position={Position.BOTTOM_RIGHT}
         onOpening={handleClipOpening}

client/src/components/menubar/index.js

@@ -16,6 +16,10 @@ import CellSetButton from "./cellSetButtons";
 import InformationMenu from "./infoMenu";
 import UndoRedoReset from "./undoRedoReset";
 import Clip from "./clip";
+import {
+  tooltipHoverOpenDelay,
+  tooltipHoverOpenDelayQuick
+} from "../../globals";
 
 @connect(state => ({
   universe: state.universe,
@@ -37,7 +41,10 @@ import Clip from "./clip";
   celllist2: state.differential.celllist2,
   libraryVersions: state.config?.library_versions, // eslint-disable-line camelcase
   undoDisabled: state["@@undoable/past"].length === 0,
-  redoDisabled: state["@@undoable/future"].length === 0
+  redoDisabled: state["@@undoable/future"].length === 0,
+  aboutLink: state.config?.links?.["about-dataset"],
+  disableDiffexp: state.config?.parameters?.["disable-diffexp"] ?? false,
+  diffexpMayBeSlow: state.config?.parameters?.["diffexp-may-be-slow"] ?? false
 }))
 class MenuBar extends React.Component {
   static isValidDigitKeyEvent(e) {
@@ -253,6 +260,65 @@ class MenuBar extends React.Component {
     });
   };
 
+  renderDiffExp() {
+    /* diffexp-related buttons may be disabled */
+    const { disableDiffexp, differential, diffexpMayBeSlow } = this.props;
+    if (disableDiffexp) return null;
+
+    const haveBothCellSets =
+      !!differential.celllist1 && !!differential.celllist2;
+
+    const tipMessage =
+      "See top 10 differentially expressed genes" +
+      (diffexpMayBeSlow
+        ? " (CAUTION: large dataset - may take longer or fail)"
+        : "");
+
+    return (
+      <div className="bp3-button-group" style={{ marginRight: 10 }}>
+        <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
+        <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
+        {!differential.diffExp ? (
+          <Tooltip
+            content={tipMessage}
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelayQuick}
+          >
+            <AnchorButton
+              disabled={!haveBothCellSets}
+              intent="primary"
+              data-testid="diffexp-button"
+              loading={differential.loading}
+              icon="left-join"
+              fill
+              type="button"
+              onClick={this.computeDiffExp}
+            >
+              Compute Differential Expression
+            </AnchorButton>
+          </Tooltip>
+        ) : null}
+
+        {differential.diffExp ? (
+          <Tooltip
+            content="Remove differentially expressed gene list and clear cell selections"
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelayQuick}
+          >
+            <Button
+              type="button"
+              fill
+              intent="warning"
+              onClick={this.clearDifferentialExpression}
+            >
+              Clear Differential Expression
+            </Button>
+          </Tooltip>
+        ) : null}
+      </div>
+    );
+  }
+
   render() {
     const {
       dispatch,
@@ -266,13 +332,11 @@ class MenuBar extends React.Component {
       clipPercentileMin,
       clipPercentileMax,
       layoutChoice,
-      graphInteractionMode
+      graphInteractionMode,
+      aboutLink
     } = this.props;
     const { pendingClipPercentiles } = this.state;
 
-    const haveBothCellSets =
-      !!differential.celllist1 && !!differential.celllist2;
-
     // constants used to create selection tool button
     let selectionTooltip;
     let selectionButtonClass;
@@ -292,48 +356,11 @@ class MenuBar extends React.Component {
           top: 8
         }}
       >
-        <div className="bp3-button-group" style={{ marginRight: 10 }}>
-          <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
-          <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
-          {!differential.diffExp ? (
-            <Tooltip
-              content="Add two cells selections, see the top 15 differentially expressed genes between them"
-              position="bottom"
-            >
-              <AnchorButton
-                disabled={!haveBothCellSets}
-                intent="primary"
-                data-testid="diffexp-button"
-                loading={differential.loading}
-                icon="left-join"
-                fill
-                type="button"
-                onClick={this.computeDiffExp}
-              >
-                Compute Differential Expression
-              </AnchorButton>
-            </Tooltip>
-          ) : null}
-
-          {differential.diffExp ? (
-            <Tooltip
-              content="Remove differentially expressed gene list and clear cell selections"
-              position="bottom"
-            >
-              <Button
-                type="button"
-                fill
-                intent="warning"
-                onClick={this.clearDifferentialExpression}
-              >
-                Clear Differential Expression
-              </Button>
-            </Tooltip>
-          ) : null}
-        </div>
+        {this.renderDiffExp()}
         <Tooltip
           content="Show only metadata and cells which are currently selected"
-          position="left"
+          position="bottom"
+          hoverOpenDelay={tooltipHoverOpenDelay}
         >
           <AnchorButton
             type="button"
@@ -355,7 +382,11 @@ class MenuBar extends React.Component {
           </AnchorButton>
         </Tooltip>
         <div className="bp3-button-group">
-          <Tooltip content={selectionTooltip} position="left">
+          <Tooltip
+            content={selectionTooltip}
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelay}
+          >
             <Button
               type="button"
               data-testid="mode-lasso"
@@ -372,7 +403,11 @@ class MenuBar extends React.Component {
               }}
             />
           </Tooltip>
-          <Tooltip content="Pan and zoom" position="left">
+          <Tooltip
+            content="Drag to pan, scroll to zoom"
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelay}
+          >
             <Button
               type="button"
               data-testid="mode-pan-zoom"
@@ -398,14 +433,20 @@ class MenuBar extends React.Component {
         >
           <Popover
             target={
-              <Button
-                type="button"
-                data-testid="layout-choice"
-                className="bp3-button bp3-icon-heatmap"
-                style={{
-                  cursor: "pointer"
-                }}
-              />
+              <Tooltip
+                content="Select embedding for visualization"
+                position="bottom"
+                hoverOpenDelay={tooltipHoverOpenDelay}
+              >
+                <Button
+                  type="button"
+                  data-testid="layout-choice"
+                  className="bp3-button bp3-icon-heatmap"
+                  style={{
+                    cursor: "pointer"
+                  }}
+                />
+              </Tooltip>
             }
             position={Position.BOTTOM_RIGHT}
             content={
@@ -419,7 +460,7 @@ class MenuBar extends React.Component {
                 }}
               >
                 <RadioGroup
-                  label="Layout Choice"
+                  label="Embedding Choice"
                   onChange={this.handleLayoutChoiceChange}
                   selectedValue={layoutChoice.current}
                 >
@@ -455,7 +496,10 @@ class MenuBar extends React.Component {
           undoDisabled={undoDisabled}
           redoDisabled={redoDisabled}
         />
-        <InformationMenu libraryVersions={libraryVersions} />
+        <InformationMenu
+          libraryVersions={libraryVersions}
+          aboutLink={aboutLink}
+        />
       </div>
     );
   }

client/src/components/menubar/infoMenu.js

@@ -3,12 +3,23 @@ import React from "react";
 import { Button, Popover, Menu, MenuItem, Position } from "@blueprintjs/core";
 
 function InformationMenu(props) {
-  const { libraryVersions } = props;
+  const { libraryVersions, aboutLink } = props;
   return (
     <div style={{ marginLeft: 10 }} className="bp3-button-group">
       <Popover
         content={
           <Menu>
+            {aboutLink ? (
+              <MenuItem
+                href={aboutLink}
+                target="_blank"
+                icon="document-open"
+                text="About this dataset"
+              />
+            ) : (
+              ""
+            )}
+
             <MenuItem
               href="https://chanzuckerberg.github.io/cellxgene/faq.html"
               target="_blank"

client/src/components/menubar/undoRedoReset.js

@@ -1,6 +1,7 @@
 // jshint esversion: 6
 import React from "react";
 import { AnchorButton, Tooltip } from "@blueprintjs/core";
+import { tooltipHoverOpenDelay } from "../../globals";
 
 function InformationMenu(props) {
   const {
@@ -13,7 +14,11 @@ function InformationMenu(props) {
   } = props;
   return (
     <div style={{ marginLeft: 10 }} className="bp3-button-group">
-      <Tooltip content="Undo" position="left">
+      <Tooltip
+        content="Undo"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
+      >
         <AnchorButton
           type="button"
           className="bp3-button bp3-icon-undo"
@@ -26,7 +31,11 @@ function InformationMenu(props) {
           }}
         />
       </Tooltip>
-      <Tooltip content="Redo" position="left">
+      <Tooltip
+        content="Redo"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
+      >
         <AnchorButton
           type="button"
           className="bp3-button bp3-icon-redo"
@@ -41,7 +50,8 @@ function InformationMenu(props) {
       </Tooltip>
       <Tooltip
         content="Reset cellxgene, clearing all selections"
-        position="left"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
       >
         <AnchorButton
           disabled={isResetDisabled()}

client/src/components/scatterplot/drawPointsRegl.js

@@ -1,44 +1,64 @@
-// jshint esversion: 6
-const mat4 = require("gl-mat4");
-
-// opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js
+import { glPointFlags, glPointSize } from "../../util/glHelpers";
 
 export default function(regl) {
   return regl({
     vert: `
     precision mediump float;
+
     attribute vec2 position;
     attribute vec3 color;
-    attribute float size;
-    uniform float distance;
-    uniform mat4 projection, view;
-    varying vec3 fragColor;
+    attribute float flag;
+
+    uniform mat3 projection;
+    uniform float nPoints;
+    uniform float minViewportDimension;
+
+    varying vec4 fragColor;
+
+    const float zBottom = 0.99;
+    const float zMiddle = 0.;
+    const float zTop = -1.;
+
+    // import getFlags()
+    ${glPointFlags}
+
+    // get pointSize()
+    ${glPointSize}
+
     void main() {
-      gl_PointSize = 7.0 / pow(distance, 2.5) + size;
-      gl_Position = projection * view * vec4(position.x, -position.y, 0, 1);
-      fragColor = color;
+      bool isNaN, isSelected, isHighlight;
+      getFlags(flag, isNaN, isSelected, isHighlight);
+
+      gl_PointSize = pointSize(nPoints, minViewportDimension, isSelected, isHighlight);
+
+      float z = isNaN ? zBottom : (isHighlight ? zTop : zMiddle);
+      vec3 xy = projection * vec3(position, 1.);
+      gl_Position = vec4(xy.xy, z, 1.);
+
+      float alpha = isNaN ? 0.9 : 1.0;
+      fragColor = vec4(color, alpha);
     }`,
 
     frag: `
     precision mediump float;
-    varying vec3 fragColor;
+    varying vec4 fragColor;
     void main() {
       if (length(gl_PointCoord.xy - 0.5) > 0.5) {
         discard;
       }
-      gl_FragColor = vec4(fragColor, 1);
+      gl_FragColor = fragColor;
     }`,
 
     attributes: {
       position: regl.prop("position"),
       color: regl.prop("color"),
-      size: regl.prop("size")
+      flag: regl.prop("flag")
     },
 
     uniforms: {
-      distance: regl.prop("distance"),
-      view: regl.prop("view"),
-      projection: () => mat4.perspective([], Math.PI / 2, 1, 0.01, 1000)
+      projection: regl.prop("projection"),
+      nPoints: regl.prop("nPoints"),
+      minViewportDimension: regl.prop("minViewportDimension")
     },
 
     count: regl.prop("count"),

client/src/components/scatterplot/scatterplot.js

@@ -1,48 +1,47 @@
-// jshint esversion: 6
-// https://bl.ocks.org/Jverma/076377dd0125b1a508621441752735fc
-// https://peterbeshai.com/scatterplot-in-d3-with-voronoi-interaction.html
-
 import React from "react";
 import { connect } from "react-redux";
 import { Button, ButtonGroup } from "@blueprintjs/core";
 import _regl from "regl";
 import * as d3 from "d3";
+import { mat3 } from "gl-matrix";
+import memoize from "memoize-one";
+import { isTypedArray } from "../../util/typeHelpers";
+
 import * as globals from "../../globals";
-
-import _camera from "../../util/camera";
-
 import setupScatterplot from "./setupScatterplot";
 import styles from "./scatterplot.css";
-
 import _drawPoints from "./drawPointsRegl";
-import scaleLinear from "../../util/scaleLinear";
-
 import { margin, width, height } from "./util";
 import finiteExtent from "../../util/finiteExtent";
 
+function createProjectionTF(viewportWidth, viewportHeight) {
+  /*
+  the projection transform accounts for the screen size & other layout
+  */
+  const m = mat3.create();
+  return mat3.projection(m, viewportWidth, viewportHeight);
+}
+
 @connect(state => {
-  const { world, crossfilter } = state;
+  const { world, crossfilter, universe } = state;
   const { scatterplotXXaccessor, scatterplotYYaccessor } = state.controls;
-  const expressionX =
-    world &&
-    scatterplotXXaccessor &&
-    world.varData.hasCol(scatterplotXXaccessor)
-      ? world.varData.col(scatterplotXXaccessor).asArray()
-      : null;
-  const expressionY =
-    world &&
-    scatterplotYYaccessor &&
-    world.varData.hasCol(scatterplotYYaccessor)
-      ? world.varData.col(scatterplotYYaccessor).asArray()
-      : null;
+  const expressionX = scatterplotXXaccessor
+    ? world.varData.col(scatterplotXXaccessor)?.asArray()
+    : null;
+  const expressionY = scatterplotYYaccessor
+    ? world.varData.col(scatterplotYYaccessor)?.asArray()
+    : null;
 
   return {
     world,
+    universe,
 
     colorRGB: state.colors.rgb,
     colorScale: state.colors.scale,
     colorAccessor: state.colors.colorAccessor,
 
+    centroidLabel: state.centroidLabel,
+
     // Accessors are var/gene names (strings)
     scatterplotXXaccessor,
     scatterplotYYaccessor,
@@ -53,10 +52,80 @@ import finiteExtent from "../../util/finiteExtent";
     expressionX,
     expressionY,
 
-    crossfilter
+    crossfilter,
+
+    responsive: state.responsive
   };
 })
-class Scatterplot extends React.Component {
+class Scatterplot extends React.PureComponent {
+  computePointPositions = memoize((X, Y, xScale, yScale) => {
+    const positions = new Float32Array(2 * X.length);
+    for (let i = 0, len = X.length; i < len; i += 1) {
+      positions[2 * i] = xScale(X[i]);
+      positions[2 * i + 1] = yScale(Y[i]);
+    }
+    return positions;
+  });
+
+  computePointColors = memoize(rgb => {
+    /*
+    compute webgl colors for each point
+    */
+    const colors = new Float32Array(3 * rgb.length);
+    for (let i = 0, len = rgb.length; i < len; i += 1) {
+      colors.set(rgb[i], 3 * i);
+    }
+    return colors;
+  });
+
+  computeSelectedFlags = memoize(
+    (crossfilter, flagSelected, flagUnselected) => {
+      const x = crossfilter.fillByIsSelected(
+        new Float32Array(crossfilter.size()),
+        flagSelected,
+        flagUnselected
+      );
+      return x;
+    }
+  );
+
+  computePointFlags = memoize(
+    (world, crossfilter, colorAccessor, centroidLabel) => {
+      const flagSelected = 1;
+      const flagNaN = 2;
+      const flagHighlight = 4;
+
+      const flags = this.computeSelectedFlags(
+        crossfilter,
+        flagSelected,
+        0
+      ).slice();
+
+      const { metadataField, categoryField } = centroidLabel;
+      const highlightData = metadataField
+        ? world.obsAnnotations.col(metadataField)?.asArray()
+        : null;
+      const colorByColumn = colorAccessor
+        ? world.obsAnnotations.col(colorAccessor)?.asArray() ||
+          world.varData.col(colorAccessor)?.asArray()
+        : null;
+      const colorByData =
+        colorByColumn && isTypedArray(colorByColumn) ? colorByColumn : null;
+
+      if (colorByData || highlightData) {
+        for (let i = 0, len = flags.length; i < len; i += 1) {
+          if (highlightData) {
+            flags[i] += highlightData[i] === categoryField ? flagHighlight : 0;
+          }
+          if (colorByData) {
+            flags[i] += Number.isFinite(colorByData[i]) ? 0 : flagNaN;
+          }
+        }
+      }
+      return flags;
+    }
+  );
+
   constructor(props) {
     super(props);
     this.count = 0;
@@ -64,7 +133,7 @@ class Scatterplot extends React.Component {
     this.renderCache = {
       positions: null,
       colors: null,
-      sizes: null,
+      flags: null,
       xScale: null,
       yScale: null
     };
@@ -85,39 +154,37 @@ class Scatterplot extends React.Component {
       this.renderCache = { ...this.renderCache, ...scales };
     }
 
-    const camera = _camera(this.reglCanvas, { scale: true, rotate: false });
     const regl = _regl(this.reglCanvas);
-
     const drawPoints = _drawPoints(regl);
 
+    // Create render transform
+    const projectionTF = createProjectionTF(
+      this.reglCanvas.width,
+      this.reglCanvas.height
+    );
+
     // preallocate buffers
     const pointBuffer = regl.buffer();
     const colorBuffer = regl.buffer();
-    const sizeBuffer = regl.buffer();
+    const flagBuffer = regl.buffer();
 
-    const reglRender = regl.frame(() => {
-      this.reglDraw(
-        regl,
-        drawPoints,
-        sizeBuffer,
-        colorBuffer,
-        pointBuffer,
-        camera
-      );
-      camera.tick();
-    });
-
-    this.reglRenderState = "rendering";
+    this.renderPoints(
+      regl,
+      drawPoints,
+      flagBuffer,
+      colorBuffer,
+      pointBuffer,
+      projectionTF
+    );
 
     this.setState({
       regl,
-      sizeBuffer,
+      flagBuffer,
       pointBuffer,
       colorBuffer,
       svg,
-      reglRender,
-      camera,
-      drawPoints
+      drawPoints,
+      projectionTF
     });
   }
 
@@ -129,22 +196,23 @@ class Scatterplot extends React.Component {
       scatterplotYYaccessor,
       expressionX,
       expressionY,
-      colorRGB
+      colorRGB,
+      colorAccessor,
+      centroidLabel
     } = this.props;
     const {
-      reglRender,
       regl,
       pointBuffer,
       colorBuffer,
-      sizeBuffer,
+      flagBuffer,
       svg,
       drawPoints,
-      camera
+      projectionTF
     } = this.state;
 
     if (
-      scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
-      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor || // was CLU now FTH1 etc
+      scatterplotXXaccessor !== prevProps.scatterplotXXaccessor ||
+      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor ||
       world !== prevProps.world // shape or clip of world changed
     ) {
       const scales = Scatterplot.setupScales(expressionX, expressionY);
@@ -152,76 +220,55 @@ class Scatterplot extends React.Component {
       this.renderCache = { ...this.renderCache, ...scales };
     }
 
-    if (reglRender && this.reglRenderState === "rendering") {
-      reglRender.cancel();
-      this.reglRenderState = "paused";
-    }
-
-    if (
-      world &&
-      regl &&
-      pointBuffer &&
-      colorBuffer &&
-      sizeBuffer &&
-      expressionX &&
-      expressionY &&
-      scatterplotXXaccessor &&
-      scatterplotYYaccessor
-    ) {
+    if (world && regl) {
       const { renderCache } = this;
       const { xScale, yScale } = this.renderCache;
-      const cellCount = expressionX.length;
+      let needsRepaint = false;
 
-      // Points change when expressionX or expressionY change.
-      if (
-        !renderCache.positions ||
-        expressionX !== prevProps.expressionX ||
-        expressionY !== prevProps.expressionY
-      ) {
-        if (!renderCache.positions) {
-          renderCache.positions = new Float32Array(2 * cellCount);
-        }
-        const glScaleX = scaleLinear([0, width], [-0.95, 0.95]);
-        const glScaleY = scaleLinear([0, height], [-1, 1]);
-        for (let i = 0, { positions } = renderCache; i < cellCount; i += 1) {
-          positions[2 * i] = glScaleX(xScale(expressionX[i]));
-          positions[2 * i + 1] = glScaleY(yScale(expressionY[i]));
-        }
-        pointBuffer({ data: renderCache.positions, dimension: 2 });
-      }
-
-      // Colors for each point - change only when props.colorsRGB change.
-      if (!renderCache.colors || colorRGB !== prevProps.colorRGB) {
-        if (!renderCache.colors) {
-          renderCache.colors = new Float32Array(3 * cellCount);
-        }
-        for (let i = 0, { colors } = renderCache; i < cellCount; i += 1) {
-          colors.set(colorRGB[i], 3 * i);
-        }
-        colorBuffer({ data: renderCache.colors, dimension: 3 });
-      }
-
-      // Sizes for each point - updates are triggered only when selected
-      // obs change
-      if (!renderCache.sizes || crossfilter !== prevProps.crossfilter) {
-        if (!renderCache.sizes) {
-          renderCache.sizes = new Float32Array(cellCount);
-        }
-        crossfilter.fillByIsSelected(renderCache.sizes, 4, 0.2);
-        sizeBuffer({ data: renderCache.sizes, dimension: 1 });
-      }
-
-      this.count = cellCount;
-
-      regl._refresh();
-      this.reglDraw(
-        regl,
-        drawPoints,
-        sizeBuffer,
-        colorBuffer,
-        pointBuffer,
-        camera
+      const newPositions = this.computePointPositions(
+        expressionX,
+        expressionY,
+        xScale,
+        yScale
       );
+      if (renderCache.positions !== newPositions) {
+        renderCache.positions = newPositions;
+        pointBuffer({ data: renderCache.positions, dimension: 2 });
+        needsRepaint = true;
+      }
+
+      /* colors for each point */
+      const newColors = this.computePointColors(colorRGB);
+      if (renderCache.colors !== newColors) {
+        renderCache.colors = newColors;
+        colorBuffer({ data: renderCache.colors, dimension: 3 });
+        needsRepaint = true;
+      }
+
+      const newFlags = this.computePointFlags(
+        world,
+        crossfilter,
+        colorAccessor,
+        centroidLabel
+      );
+      if (renderCache.flags !== newFlags) {
+        renderCache.flags = newFlags;
+        flagBuffer({ data: renderCache.flags, dimension: 1 });
+        needsRepaint = true;
+      }
+
+      this.count = expressionX.length;
+
+      if (needsRepaint) {
+        this.renderPoints(
+          regl,
+          drawPoints,
+          flagBuffer,
+          colorBuffer,
+          pointBuffer,
+          projectionTF
+        );
+      }
     }
   }
 
@@ -241,22 +288,6 @@ class Scatterplot extends React.Component {
     };
   }
 
-  reglDraw(regl, drawPoints, sizeBuffer, colorBuffer, pointBuffer, camera) {
-    regl.clear({
-      depth: 1,
-      color: [1, 1, 1, 1]
-    });
-
-    drawPoints({
-      size: sizeBuffer,
-      distance: camera.distance,
-      color: colorBuffer,
-      position: pointBuffer,
-      count: this.count,
-      view: camera.view()
-    });
-  }
-
   drawAxesSVG(xScale, yScale, svg) {
     const { scatterplotYYaccessor, scatterplotXXaccessor } = this.props;
     svg.selectAll("*").remove();
@@ -307,6 +338,41 @@ class Scatterplot extends React.Component {
       .text(scatterplotXXaccessor);
   }
 
+  renderPoints(
+    regl,
+    drawPoints,
+    flagBuffer,
+    colorBuffer,
+    pointBuffer,
+    projectionTF
+  ) {
+    if (!this.reglCanvas) return;
+    const { universe, responsive } = this.props;
+    // The viewport dimension is used to scale points, so we want to pass
+    // the dimension of the MAIN viewport, not the scatterplot viewport.
+    // Slightly hacky, but we want all points to scale uniformly.  Perhaps
+    // this should move to the redux state and be shared?
+    const { width: cvWidth, height: cvHeight } = responsive;
+    regl.poll();
+    regl.clear({
+      depth: 1,
+      color: [1, 1, 1, 1]
+    });
+    drawPoints({
+      flag: flagBuffer,
+      color: colorBuffer,
+      position: pointBuffer,
+      projection: projectionTF,
+      count: this.count,
+      nPoints: universe.nObs,
+      minViewportDimension: Math.min(
+        cvWidth - globals.leftSidebarWidth || width,
+        cvHeight || height
+      )
+    });
+    regl._gl.flush();
+  }
+
   render() {
     const { dispatch } = this.props;
     const { minimized } = this.state;
@@ -367,7 +433,7 @@ class Scatterplot extends React.Component {
             height={height}
             data-testid="scatterplot"
             style={{
-              marginLeft: margin.left - 7,
+              marginLeft: margin.left,
               marginTop: margin.top
             }}
             ref={canvas => {

client/src/globals.js

@@ -3,6 +3,9 @@ import { Colors } from "@blueprintjs/core";
 /* if a categorical metadata field has more options than this, truncate */
 export const maxCategoricalOptionsToDisplay = 100;
 
+/* default "unassigned" value for user-created categorical metadata */
+export const unassignedCategoryLabel = "unassigned";
+
 /*
 these are default values for configuration  the CLI may supply.
 See the REST API and CLI specs for more info.
@@ -11,8 +14,11 @@ export const configDefaults = {
   features: {},
   displayNames: {},
   parameters: {
-    "max-category-items": 1000
-  }
+    "max-category-items": 1000,
+    "disable-diffexp": false,
+    "diffexp-may-be-slow": false
+  },
+  links: {}
 };
 
 /* colors */
@@ -47,6 +53,8 @@ export const maxParagraphWidth = 600;
 export const cellxgeneTitleLeftPadding = 14;
 export const cellxgeneTitleTopPadding = 7;
 
+export const datasetTitleMaxCharacterCount = 25;
+
 export const maxControlsWidth = 800;
 
 export const graphMargin = { top: 20, right: 10, bottom: 30, left: 40 };
@@ -62,6 +70,12 @@ export const leftSidebarSectionHeading = {
   letterSpacing: ".05em"
 };
 export const leftSidebarSectionPadding = 10;
+export const categoryLabelDisplayStringLongLength = 35;
+export const categoryLabelDisplayStringShortLength = 15;
+
+/* various timing-related behaviors */
+export const tooltipHoverOpenDelay = 1000; /* ms delay before a tooltip displays */
+export const tooltipHoverOpenDelayQuick = 500;
 
 let _API = {
   // prefix: "http://api.clustering.czi.technology/api/",

client/src/reducers/annotations.js

@@ -0,0 +1,79 @@
+/*
+Reducers for annotation UI-state.
+*/
+const Annotations = (
+  state = {
+    isEditingCategoryName: false,
+    isEditingLabelName: false,
+    categoryBeingEdited: false,
+    labelEditable: { category: null, label: null }
+  },
+  action
+) => {
+  switch (action.type) {
+    /* CATEGORY */
+    case "annotation: activate add new label mode":
+      return {
+        ...state,
+        isAddingNewLabel: true,
+        categoryAddingNewLabel: action.data
+      };
+    case "annotation: disable add new label mode":
+      return {
+        ...state,
+        isAddingNewLabel: false,
+        categoryAddingNewLabel: null
+      };
+    case "annotation: add new label to category":
+      return {
+        ...state,
+        isAddingNewLabel: false,
+        categoryAddingNewLabel: null
+      };
+    case "annotation: activate category edit mode":
+      return {
+        ...state,
+        isEditingCategoryName: true,
+        categoryBeingEdited: action.data
+      };
+    case "annotation: disable category edit mode":
+      return {
+        ...state,
+        isEditingCategoryName: false,
+        categoryBeingEdited: null
+      };
+    case "annotation: category edited":
+      return {
+        ...state,
+        isEditingCategoryName: true,
+        categoryBeingEdited: null
+      };
+
+    /* LABEL */
+    case "annotation: activate edit label mode":
+      return {
+        ...state,
+        isEditingLabelName: true,
+        labelEditable: {
+          category: action.metadataField,
+          label: action.categoryIndex
+        }
+      };
+    case "annotation: cancel edit label mode":
+      return {
+        ...state,
+        isEditingLabelName: false,
+        labelEditable: { category: null, label: null }
+      };
+    case "annotation: label edited":
+      return {
+        ...state,
+        isEditingLabelName: false,
+        labelEditable: { category: null, label: null }
+      };
+    default:
+      return state;
+  }
+};
+
+export default Annotations;

client/src/reducers/autosave.js

@@ -0,0 +1,53 @@
+const Autosave = (
+	state = {
+		saveInProgress: false,
+		error: false,
+		lastSavedObsAnnotations: null
+	},
+	action,
+	nextSharedState
+) => {
+	switch (action.type) {
+		case "initial data load complete (universe exists)": {
+			/* don't save on init */
+			const { universe } = nextSharedState;
+			return {
+				...state,
+				error: false,
+				saveInProgress: false,
+				lastSavedObsAnnotations: universe.obsAnnotations
+			};
+		}
+
+		case "writable obs annotations - save started": {
+			return {
+				...state,
+				saveInProgress: true
+			};
+		}
+
+		case "writable obs annotations - save error": {
+			const { message } = action;
+			return {
+				...state,
+				error: message,
+				saveInProgress: false
+			};
+		}
+
+		case "writable obs annotations - save complete": {
+			const lastSavedObsAnnotations = action.obsAnnotations;
+			return {
+				...state,
+				saveInProgress: false,
+				error: false,
+				lastSavedObsAnnotations
+			};
+		}
+
+		default:
+			return { ...state };
+	}
+};
+
+export default Autosave;

client/src/reducers/categoricalSelection.js

@@ -1,4 +1,4 @@
-import { ControlsHelpers } from "../util/stateManager";
+import { ControlsHelpers as CH } from "../util/stateManager";
 import * as globals from "../globals";
 
 function maxCategoryItems(state) {
@@ -20,10 +20,11 @@ const CategoricalSelection = (
     case "reset World to eq Universe":
     case "set clip quantiles": {
       const { world } = nextSharedState;
-      return ControlsHelpers.createCategoricalSelection(
-        maxCategoryItems(prevSharedState),
-        world
+      const newState = CH.createCategoricalSelection(
+        world,
+        CH.selectableCategoryNames(world, maxCategoryItems(prevSharedState))
       );
+      return newState;
     }
 
     case "categorical metadata filter select": {
@@ -96,6 +97,43 @@ const CategoricalSelection = (
       return newCategoricalSelection;
     }
 
+    case "annotation: create category": {
+      const { world } = nextSharedState;
+      const name = action.data;
+      return {
+        ...state,
+        ...CH.createCategoricalSelection(world, [name])
+      };
+    }
+
+    case "annotation: category edited": {
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      const { [name]: catSeln, ...newState } = state;
+      newState[newName] = catSeln;
+      return newState;
+    }
+
+    case "annotation: delete category": {
+      const name = action.metadataField;
+      const { [name]: _, ...newState } = state;
+      return newState;
+    }
+
+    case "annotation: label current cell selection":
+    case "annotation: add new label to category":
+    case "annotation: label edited":
+    case "annotation: delete label": {
+      /* need to rebuild the state for this annotation */
+      const { world } = nextSharedState;
+      const name = action.metadataField;
+      const { [name]: _, ...partialState } = state;
+      return {
+        ...partialState,
+        ...CH.createCategoricalSelection(world, [name])
+      };
+    }
+
     default: {
       return state;
     }

client/src/reducers/colors.js

@@ -108,6 +108,27 @@ const ColorsReducer = (
       };
     }
 
+    case "annotation: add new label to category":
+    case "annotation: label current cell selection":
+    case "annotation: delete label": {
+      const { world } = nextSharedState;
+      const { colorMode, colorAccessor } = state;
+      const { metadataField } = action;
+      if (
+        colorMode !== "color by categorical metadata" ||
+        colorAccessor !== metadataField
+      )
+        return state;
+
+      /* else, we need to rebuild colors as labels have changed! */
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
+      return { ...state, rgb, scale };
+    }
+
     case "clear differential expression": {
       const { world: prevWorld, controls: prevControls } = prevSharedState;
       const resetColorState = ColorHelpers.checkIfColorByDiffexpAndResetColors(

client/src/reducers/controls.js

@@ -2,12 +2,10 @@
 
 import _ from "lodash";
 
-import { WorldUtil } from "../util/stateManager";
-
 const Controls = (
   state = {
     // data loading flag
-    loading: false,
+    loading: true,
     error: null,
 
     // all of the data + selection state
@@ -45,7 +43,6 @@ const Controls = (
     }
     case "initial data load complete (universe exists)": {
       /* first light - create world & other data-driven defaults */
-      WorldUtil.clearCaches();
       return {
         ...state,
         loading: false,
@@ -54,14 +51,12 @@ const Controls = (
       };
     }
     case "reset World to eq Universe": {
-      WorldUtil.clearCaches();
       return {
         ...state,
         resettingInterface: false
       };
     }
     case "set World to current selection": {
-      WorldUtil.clearCaches();
       return {
         ...state,
         loading: false,

client/src/reducers/crossfilter.js

@@ -1,7 +1,11 @@
 import _ from "lodash";
 
 import Crossfilter from "../util/typedCrossfilter";
-import { World, ControlsHelpers } from "../util/stateManager";
+import {
+  World,
+  ControlsHelpers as CH,
+  AnnotationsHelpers as AH
+} from "../util/stateManager";
 import {
   layoutDimensionName,
   obsAnnoDimensionName,
@@ -12,7 +16,7 @@ import {
 
 const XYDimName = layoutDimensionName("XY");
 
-const CrossfilterReducer = (
+const CrossfilterReducerBase = (
   state = null,
   action,
   nextSharedState,
@@ -32,12 +36,14 @@ const CrossfilterReducer = (
     case "reset World to eq Universe": {
       const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
       const { world } = nextSharedState;
-      const crossfilter = ControlsHelpers.createGeneDimensions(
+      let { crossfilter } = prevSharedState.resetCache;
+      crossfilter = CH.createGeneDimensions(
         userDefinedGenes,
         diffexpGenes,
         world,
-        prevSharedState.resetCache.crossfilter
+        crossfilter
       );
+      crossfilter = AH.createWritableAnnotationDimensions(world, crossfilter);
       return crossfilter;
     }
 
@@ -51,7 +57,7 @@ const CrossfilterReducer = (
         world,
         layoutChoice.currentDimNames
       );
-      crossfilter = ControlsHelpers.createGeneDimensions(
+      crossfilter = CH.createGeneDimensions(
         userDefinedGenes,
         diffexpGenes,
         world,
@@ -136,6 +142,37 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "annotation: create category": {
+      const name = action.data;
+      const { world } = nextSharedState;
+      const colData = world.obsAnnotations.col(name).asArray();
+      return state.addDimension(obsAnnoDimensionName(name), "enum", colData);
+    }
+
+    case "annotation: category edited": {
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      return state.renameDimension(
+        obsAnnoDimensionName(name),
+        obsAnnoDimensionName(newName)
+      );
+    }
+
+    case "annotation: delete category": {
+      return state.delDimension(obsAnnoDimensionName(action.metadataField));
+    }
+
+    case "annotation: label current cell selection":
+    case "annotation: label edited":
+    case "annotation: delete label": {
+      /* we need to reindex the dimension.  For now, just drop it and add another */
+      const name = action.metadataField;
+      const dimName = obsAnnoDimensionName(name);
+      const { world } = nextSharedState;
+      const colData = world.obsAnnotations.col(name).asArray();
+      return state.delDimension(dimName).addDimension(dimName, "enum", colData);
+    }
+
     case "graph brush end":
     case "graph brush change": {
       const [minX, maxY] = action.brushCoords.northwest;
@@ -185,12 +222,12 @@ const CrossfilterReducer = (
     case "categorical metadata filter select":
     case "categorical metadata filter deselect": {
       const { categoricalSelection } = nextSharedState;
-      const { world } = prevSharedState;
       const cat = categoricalSelection[action.metadataField];
-      const col = world.obsAnnotations.col(action.metadataField);
+      const { categoryValues, categoryValueSelected } = cat;
+      const values = categoryValues.filter((v, i) => categoryValueSelected[i]);
       return state.select(obsAnnoDimensionName(action.metadataField), {
         mode: "exact",
-        values: ControlsHelpers.selectedValuesForCategory(cat, col)
+        values
       });
     }
 
@@ -212,4 +249,29 @@ const CrossfilterReducer = (
   }
 };
 
+/*
+  IMPORTANT: the system assumes that crossfilter.data() will point at the
+  same value as world.obsAnnotations.  For actions handled in this reducer,
+  make sure that this remains true.
+
+  This wrapper performs only this function.
+*/
+const CrossfilterReducer = (
+  state,
+  action,
+  nextSharedState,
+  prevSharedState
+) => {
+  const nextState = CrossfilterReducerBase(
+    state,
+    action,
+    nextSharedState,
+    prevSharedState
+  );
+  if (!nextState || nextState.all() === nextSharedState.world.obsAnnotations) {
+    return nextState;
+  }
+  return nextState.setData(nextSharedState.world.obsAnnotations);
+};
+
 export default CrossfilterReducer;

client/src/reducers/index.js

@@ -17,38 +17,44 @@ import responsive from "./responsive";
 import controls from "./controls";
 import resetCache from "./resetCache";
 import centroidLabel from "./centroidLabel";
+import annotations from "./annotations";
+import autosave from "./autosave";
 
 import undoableConfig from "./undoableConfig";
 
 const Reducer = undoable(
-  cascadeReducers([
-    ["config", config],
-    ["universe", universe],
-    ["world", world],
-    ["layoutChoice", layoutChoice],
-    ["categoricalSelection", categoricalSelection],
-    ["continuousSelection", continuousSelection],
-    ["graphSelection", graphSelection],
-    ["crossfilter", crossfilter],
-    ["colors", colors],
-    ["controls", controls],
-    ["differential", differential],
-    ["responsive", responsive],
-    ["centroidLabel", centroidLabel],
-    ["resetCache", resetCache]
-  ]),
-  [
-    "world",
-    "categoricalSelection",
-    "continuousSelection",
-    "graphSelection",
-    "crossfilter",
-    "colors",
-    "controls",
-    "differential",
-    "layoutChoice"
-  ],
-  undoableConfig
+    cascadeReducers([
+        ["config", config],
+        ["universe", universe],
+        ["world", world],
+        ["annotations", annotations],
+        ["layoutChoice", layoutChoice],
+        ["categoricalSelection", categoricalSelection],
+        ["continuousSelection", continuousSelection],
+        ["graphSelection", graphSelection],
+        ["crossfilter", crossfilter],
+        ["colors", colors],
+        ["controls", controls],
+        ["differential", differential],
+        ["responsive", responsive],
+        ["centroidLabel", centroidLabel],
+        ["autosave", autosave],
+        ["resetCache", resetCache]
+    ]),
+    [
+        "universe",
+        "world",
+        "categoricalSelection",
+        "continuousSelection",
+        "graphSelection",
+        "crossfilter",
+        "colors",
+        "controls",
+        "differential",
+        "layoutChoice",
+        "annotations"
+    ],
+    undoableConfig
 );
 
 const store = createStore(Reducer, applyMiddleware(thunk));

client/src/reducers/resetCache.js

@@ -1,13 +1,13 @@
 /*
 Reducer which caches derived state to be used in a reset or other 
-recomputation.
+recomputation.  Add stuff here you want stashed at init time (or whenever),
+for later use.
 
-Currently this only caches the baseline (full universe) world & crossfilter, 
-for use in a Reset.
+Currently this only caches the baseline (full universe) crossfilter, 
+which improves Reset UI performance.
 */
 const ResetCacheReducer = (
   state = {
-    world: null,
     crossfilter: null
   },
   action,
@@ -15,10 +15,9 @@ const ResetCacheReducer = (
 ) => {
   switch (action.type) {
     case "initial data load complete (universe exists)": {
-      const { world, crossfilter } = nextSharedState;
+      const { crossfilter } = nextSharedState;
       return {
         ...state,
-        world,
         crossfilter
       };
     }

client/src/reducers/undoableConfig.js

@@ -78,7 +78,15 @@ const saveOnActions = new Set([
   "set clip quantiles",
 
   "set layout choice",
-  "change graph interaction mode"
+  "change graph interaction mode",
+
+  // user editable annotations
+  "annotation: create category",
+  "annotation: add new label to category",
+  "annotation: delete category",
+  "annotation: label edited",
+  "annotation: label current cell selection",
+  "annotation: delete label"
 ]);
 
 /**

client/src/reducers/universe.js

@@ -1,4 +1,9 @@
-import { ControlsHelpers } from "../util/stateManager";
+import { unassignedCategoryLabel } from "../globals";
+import {
+  World,
+  ControlsHelpers as CH,
+  AnnotationsHelpers as AH
+} from "../util/stateManager";
 
 const Universe = (state = null, action, nextSharedState, prevSharedState) => {
   switch (action.type) {
@@ -30,12 +35,186 @@ const Universe = (state = null, action, nextSharedState, prevSharedState) => {
           Object.keys(action.expressionData)
         )
       ];
-      varData = ControlsHelpers.pruneVarDataCache(varData, allTheGenesWeNeed);
+      varData = CH.pruneVarDataCache(varData, allTheGenesWeNeed);
+      return { ...state, varData };
+    }
 
-      return {
-        ...state,
-        varData
+    case "annotation: create category": {
+      /* create a new annotation category, with all values set to 'unassigned' */
+      const name = action.data;
+      const { categoryToDuplicate } = action;
+
+      /* name must be a string,  non-zero length */
+      if (typeof name !== "string" || name.length === 0)
+        throw new Error("user annotations require string name");
+
+      /* ensure the name isn't already in use! */
+      if (state.obsAnnotations.hasCol(name))
+        throw new Error("name collision on annotation category create");
+
+      /* ensure the duplicate col exists */
+      if (
+        categoryToDuplicate &&
+        !state.obsAnnotations.hasCol(categoryToDuplicate)
+      )
+        throw new Error("categoryToDuplicate does not exist");
+
+      let schema;
+      let data;
+      if (categoryToDuplicate) {
+        /* duplicate the named annotation */
+        schema = AH.dupObsAnnoSchema(state.schema, categoryToDuplicate, name, {
+          writable: true
+        });
+        /* if we are duplicating a non-writable annotation, it may not have an unassigned category */
+        const s = schema.annotations.obsByName[categoryToDuplicate];
+        if (s.categories.indexOf(unassignedCategoryLabel) === -1) {
+          s.categories = s.categories.concat(unassignedCategoryLabel);
+        }
+        data = state.obsAnnotations.col(categoryToDuplicate).asArray();
+      } else {
+        /* else, all are unassined */
+        const categories = [unassignedCategoryLabel];
+        schema = AH.addObsAnnoSchema(state.schema, name, {
+          name,
+          categories,
+          type: "categorical",
+          writable: true
+        });
+        data = new Array(state.nObs).fill(unassignedCategoryLabel);
+      }
+
+      const obsAnnotations = state.obsAnnotations.withCol(name, data);
+      return { ...state, obsAnnotations, schema };
+    }
+
+    case "annotation: category edited": {
+      /* change the name of an obs annotation category */
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      if (!AH.isUserAnnotation(state, name))
+        throw new Error("unable to edit read-only annotation");
+      if (typeof newName !== "string" || newName.length === 0)
+        throw new Error("user annotations require string name");
+
+      const colSchema = {
+        ...state.schema.annotations.obsByName[name],
+        name: newName
       };
+      const schema = AH.addObsAnnoSchema(
+        AH.removeObsAnnoSchema(state.schema, name),
+        newName,
+        colSchema
+      );
+      const obsAnnotations = state.obsAnnotations.renameCol(name, newName);
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: delete category": {
+      /* delete annotation category from schema and obsAnnotations */
+      const name = action.metadataField;
+      if (!AH.isUserAnnotation(state, name))
+        throw new Error("unable to delete read-only annotation");
+
+      const schema = AH.removeObsAnnoSchema(state.schema, name);
+      const obsAnnotations = state.obsAnnotations.dropCol(name);
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: add new label to category": {
+      const annotationName = action.metadataField;
+      const newLabelName = action.newLabelText;
+      if (!AH.isUserAnnotation(state, annotationName))
+        throw new Error("unable to modify read-only annotation");
+      if (typeof newLabelName !== "string" || newLabelName.length === 0)
+        throw new Error(
+          "user annotations require a non-zero length string name"
+        );
+
+      /* add the new label to the annotation */
+      const schema = AH.addObsAnnoCategory(
+        state.schema,
+        annotationName,
+        newLabelName
+      );
+      return { ...state, schema };
+    }
+
+    case "annotation: label edited": {
+      const annotationName = action.metadataField;
+      const oldLabelName = action.label;
+      const newLabelName = action.editedLabel;
+      if (!AH.isUserAnnotation(state, annotationName))
+        throw new Error("unable to modify read-only annotation");
+      if (typeof newLabelName !== "string" || newLabelName.length === 0)
+        throw new Error(
+          "user annotations require a non-zero length string name"
+        );
+
+      /* remove old label, add new label */
+      const schema = AH.addObsAnnoCategory(
+        AH.removeObsAnnoCategory(state.schema, annotationName, oldLabelName),
+        annotationName,
+        newLabelName
+      );
+
+      /* change all values in obsAnnotation */
+      const obsAnnotations = AH.setLabelByValue(
+        state.obsAnnotations,
+        annotationName,
+        oldLabelName,
+        newLabelName
+      );
+
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: delete label": {
+      /* delete the label from the annotation, and set all cells with this value to unassigned */
+      const annotationName = action.metadataField;
+      const labelName = action.label;
+      if (!AH.isUserAnnotation(state, annotationName))
+        throw new Error("unable to modify read-only annotation");
+      if (labelName === unassignedCategoryLabel)
+        throw new Error("may not remove the unassigned label");
+
+      /* remove the category from the schema */
+      const schema = AH.removeObsAnnoCategory(
+        state.schema,
+        annotationName,
+        labelName
+      );
+
+      /* set all values to unassigned in obsAnnotations */
+      const obsAnnotations = AH.setLabelByValue(
+        state.obsAnnotations,
+        annotationName,
+        labelName,
+        unassignedCategoryLabel
+      );
+
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: label current cell selection": {
+      const { metadataField, label } = action;
+      const { world, crossfilter } = prevSharedState;
+
+      /*
+      selection state is relative to world.  We need to convert it
+      to a mask for Universe before applying it.
+      */
+      const worldMask = crossfilter.allSelectedMask();
+      const mask = World.worldEqUniverse(world, state)
+        ? worldMask
+        : AH.worldToUniverseMask(worldMask, world.obsAnnotations, state.nObs);
+      const obsAnnotations = AH.setLabelByMask(
+        state.obsAnnotations,
+        metadataField,
+        mask,
+        label
+      );
+      return { ...state, obsAnnotations };
     }
 
     default: {

client/src/reducers/world.js

@@ -1,7 +1,22 @@
-import { World, ControlsHelpers } from "../util/stateManager";
+import { unassignedCategoryLabel } from "../globals";
+import {
+  World,
+  ControlsHelpers as CH,
+  AnnotationsHelpers as AH
+} from "../util/stateManager";
 import clip from "../util/clip";
 import quantile from "../util/quantile";
 
+/*
+important note: much of this code assumes that wriable (user) annotations
+will NOT contain scalar data (ie, will only contain categorical labelled
+data), and therefore will never need to be clipped.  Put another way, it
+assumes that for these annotations, the clipped & unclipped data is equal.
+
+If we ever start allowing user editable scalar data, this assumption will
+need to be revisited.
+*/
+
 const WorldReducer = (
   state = null,
   action,
@@ -9,16 +24,13 @@ const WorldReducer = (
   prevSharedState
 ) => {
   switch (action.type) {
-    case "initial data load complete (universe exists)": {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
       const { universe } = nextSharedState;
       const world = World.createWorldFromEntireUniverse(universe);
       return world;
     }
 
-    case "reset World to eq Universe": {
-      return prevSharedState.resetCache.world;
-    }
-
     case "set World to current selection": {
       /* Set viewable world to be the currently selected data */
       const world = World.createWorldBySelection(
@@ -80,7 +92,7 @@ const WorldReducer = (
           Object.keys(action.expressionData)
         )
       ];
-      unclippedVarData = ControlsHelpers.pruneVarDataCache(
+      unclippedVarData = CH.pruneVarDataCache(
         unclippedVarData,
         allTheGenesWeNeed
       );
@@ -122,6 +134,135 @@ const WorldReducer = (
       };
     }
 
+    case "annotation: create category": {
+      const name = action.data;
+      const { universe } = nextSharedState;
+      const { schema } = universe;
+
+      /*
+      if world !== universe, we have to subset the newly created annotation,
+      else, just use it as is.
+      */
+      let newAnnotation = null;
+      if (!World.worldEqUniverse(state, universe)) {
+        newAnnotation = universe.obsAnnotations
+          .subset(state.obsAnnotations.rowIndex.keys(), [name], null)
+          .icol(0)
+          .asArray();
+      } else {
+        newAnnotation = universe.obsAnnotations.col(name).asArray();
+      }
+      const obsAnnotations = state.obsAnnotations.withCol(
+        name,
+        newAnnotation,
+        state.obsAnnotations.rowIndex
+      );
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: state.unclipped.obsAnnotations.withCol(
+          name,
+          newAnnotation,
+          state.unclipped.obsAnnotations.rowIndex
+        )
+      };
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: category edited": {
+      /* change the name of an obs annotation */
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      const { schema } = nextSharedState.universe;
+      const obsAnnotations = state.obsAnnotations.renameCol(name, newName);
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: state.unclipped.obsAnnotations.renameCol(name, newName)
+      };
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: delete category": {
+      /* remove a category from obs annotation */
+      const { schema } = nextSharedState.universe;
+      const name = action.metadataField;
+      const obsAnnotations = state.obsAnnotations.dropCol(name);
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: state.unclipped.obsAnnotations.dropCol(name)
+      };
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: add new label to category": {
+      /* add a new label to the schema - schema updated by universe reducer, we just need to note it */
+      const { schema } = nextSharedState.universe;
+      return { ...state, schema };
+    }
+
+    case "annotation: label edited": {
+      const { schema } = nextSharedState.universe;
+      const { metadataField } = action;
+      const oldLabelName = action.label;
+      const newLabelName = action.editedLabel;
+
+      /* set all values to to new label */
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: AH.setLabelByValue(
+          state.unclipped.obsAnnotations,
+          metadataField,
+          oldLabelName,
+          newLabelName
+        )
+      };
+      const obsAnnotations = state.obsAnnotations.replaceColData(
+        metadataField,
+        unclipped.obsAnnotations.col(metadataField).asArray()
+      );
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: delete label": {
+      const { schema } = nextSharedState.universe;
+      const { label, metadataField } = action;
+
+      /* set all values to unassigned in obsAnnotations */
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: AH.setLabelByValue(
+          state.unclipped.obsAnnotations,
+          metadataField,
+          label,
+          unassignedCategoryLabel
+        )
+      };
+      const obsAnnotations = state.obsAnnotations.replaceColData(
+        metadataField,
+        unclipped.obsAnnotations.col(metadataField).asArray()
+      );
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: label current cell selection": {
+      const { metadataField, label } = action;
+      const { crossfilter } = prevSharedState;
+      const mask = crossfilter.allSelectedMask();
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: AH.setLabelByMask(
+          state.unclipped.obsAnnotations,
+          metadataField,
+          mask,
+          label
+        )
+      };
+      const obsAnnotations = state.obsAnnotations.replaceColData(
+        metadataField,
+        unclipped.obsAnnotations.col(metadataField).asArray()
+      );
+      return { ...state, obsAnnotations, unclipped };
+    }
+
     default: {
       return state;
     }

client/src/util/camera.js

@@ -1,66 +1,181 @@
-// jshint esversion: 6
-const createCamera = require("orbit-camera");
-const createScroll = require("scroll-speed");
-const mp = require("mouse-position");
-const mb = require("mouse-pressed");
-const key = require("key-pressed");
+import { vec2, mat3 } from "gl-matrix";
+
+const EPSILON = 0.000001;
 
-const panSpeed = 1.0; // changed from 0.4 to 1.0 per issue #722
 const scaleSpeed = 0.5;
-const scaleMax = 3;
-const scaleMin = 1.03;
+const scaleMax = 3.0;
+const scaleMin = 0.5;
+const panBound = 0.8;
 
-function attachCamera(canvas, opts) {
-  opts = opts || {};
-  opts.pan = opts.pan !== false;
-  opts.scale = opts.scale !== false;
-  opts.rotate = opts.rotate !== false;
+// private
+const scratch0 = new Float32Array(16);
+const scratch1 = new Float32Array(16);
 
-  const scroll = createScroll(canvas, opts.scale);
-  const mbut = mb(canvas, opts.rotate);
-  const mpos = mp(canvas);
-  const camera = createCamera([0, 0, 1], [0, 0, -1], [0, 1, 0]);
+function clamp(val, rng) {
+  return Math.max(Math.min(val, rng[1]), rng[0]);
+}
 
-  camera.tick = tick;
+class Camera {
+  constructor(canvas) {
+    this.prevEvent = {
+      clientX: 0,
+      clientY: 0,
+      type: 0
+    };
+    this.canvas = canvas;
+    this.viewMatrix = mat3.create();
+    this.viewMatrixInv = mat3.create();
+  }
 
-  return camera;
+  view() {
+    return this.viewMatrix;
+  }
 
-  function tick() {
-    const ctrl = key("<control>") || key("<alt>");
-    const alt = key("<shift>");
-    const { height, width } = canvas;
+  invView() {
+    return this.viewMatrixInv;
+  }
 
-    if (opts.rotate && mbut.left && ctrl && !alt) {
-      camera.rotate(
-        [mpos.x / width - 0.5, mpos.y / height - 0.5],
-        [mpos.prevX / width - 0.5, mpos.prevY / height - 0.5]
-      );
+  distance() {
+    return this.viewMatrix[0];
+  }
+
+  pan(dx, dy) {
+    const m = this.viewMatrix;
+    const dyRange = [
+      -panBound - (m[7] + 1) / m[4],
+      panBound - (m[7] - 1) / m[4]
+    ];
+    const dxRange = [
+      -panBound - (m[6] + 1) / m[0],
+      panBound - (m[6] - 1) / m[0]
+    ];
+
+    const dxClamped = clamp(dx, dxRange);
+    const dyClamped = clamp(dy, dyRange);
+    if (Math.abs(dxClamped) <= EPSILON && Math.abs(dyClamped) <= EPSILON)
+      return;
+
+    mat3.translate(m, m, [dxClamped, dyClamped]);
+    mat3.invert(this.viewMatrixInv, m);
+  }
+
+  zoomAt(d, x = 0, y = 0) {
+    /*
+    Camera zoom at [x,y]
+    */
+    const m = this.viewMatrix;
+    const bounds = [-panBound, panBound];
+    x = clamp(x, bounds);
+    y = clamp(y, bounds);
+
+    const dClamped = clamp(d * m[0], [scaleMin, scaleMax]) / m[0];
+    if (Math.abs(1 - dClamped) <= EPSILON) return; // noop request
+
+    mat3.translate(m, m, [x, y]);
+    mat3.scale(m, m, [dClamped, dClamped]);
+    mat3.translate(m, m, [-x, -y]);
+
+    mat3.invert(this.viewMatrixInv, m);
+  }
+
+  /*
+  Event handling
+  */
+
+  flush(e) {
+    this.prevEvent.type = e.type;
+    this.prevEvent.clientX = e.clientX;
+    this.prevEvent.clientY = e.clientY;
+  }
+
+  localPosition(target, canvasX, canvasY, projectionInvTF) {
+    /*
+    Convert mouse position to local
+    */
+    const { height, width } = target;
+    const targetRect = target.getBoundingClientRect();
+    canvasX -= targetRect.left;
+    canvasY -= targetRect.top;
+
+    const pos = vec2.fromValues(
+      2 * (canvasX / width) - 1,
+      -2 * (canvasY / height) + 1
+    );
+    if (projectionInvTF) {
+      vec2.transformMat3(pos, pos, projectionInvTF);
     }
+    vec2.transformMat3(pos, pos, this.invView());
+    return pos;
+  }
 
-    if ((opts.pan && mbut.right) || (mbut.left && !ctrl && !alt)) {
-      camera.pan([
-        ((panSpeed * (mpos[0] - mpos.prev[0])) / width) * camera.distance,
-        ((panSpeed * (mpos[1] - mpos.prev[1])) / height) * camera.distance
-      ]);
+  mousePan(e, projectionTF) {
+    const projectionInvTF = mat3.invert(scratch0, projectionTF);
+    const pos = this.localPosition(
+      this.canvas,
+      e.clientX,
+      e.clientY,
+      projectionInvTF
+    );
+    const prev = this.localPosition(
+      this.canvas,
+      this.prevEvent.clientX,
+      this.prevEvent.clientY,
+      projectionInvTF
+    );
+
+    const delta = vec2.sub(scratch1, pos, prev);
+    this.pan(delta[0], delta[1]);
+    return true;
+  }
+
+  wheelZoom(e, projectionTF) {
+    const { height } = this.canvas;
+    const { deltaY, deltaMode, clientX, clientY } = e;
+    const scale = scaleSpeed * (deltaMode === 1 ? 12 : 1) * (deltaY || 0);
+
+    const projectionInvTF = mat3.invert(scratch0, projectionTF);
+    const pos = this.localPosition(
+      this.canvas,
+      clientX,
+      clientY,
+      projectionInvTF
+    );
+    this.zoomAt(1 / Math.exp(scale / height), pos[0], pos[1]);
+    return true;
+  }
+
+  handleEvent(e, projectionTF) {
+    /*
+    process the event, and return true if camera view changed
+    */
+    let viewChanged = false;
+    switch (e.type) {
+      case "mousemove": {
+        /* eslint-disable no-bitwise */
+        if (e.buttons & 0x1) {
+          viewChanged = this.mousePan(e, projectionTF);
+        }
+        /* eslint-enable no-bitwise */
+        this.flush(e);
+        break;
+      }
+
+      case "wheel": {
+        viewChanged = this.wheelZoom(e, projectionTF);
+        this.flush(e);
+        break;
+      }
+
+      default:
+        // noop
+        break;
     }
-
-    if (opts.scale && scroll[1]) {
-      camera.distance *= Math.exp((scroll[1] * scaleSpeed) / height);
-    }
-
-    if (opts.scale && (mbut.middle || (mbut.left && !ctrl && alt))) {
-      const d = mpos.y - mpos.prevY;
-      if (!d) return;
-
-      camera.distance *= Math.exp(d / height);
-    }
-
-    if (camera.distance > scaleMax) camera.distance = scaleMax;
-    if (camera.distance < scaleMin) camera.distance = scaleMin;
-
-    scroll.flush();
-    mpos.flush();
+    return viewChanged;
   }
 }
 
+function attachCamera(canvas) {
+  return new Camera(canvas);
+}
+
 export default attachCamera;

client/src/util/dataframe/dataframe.js

@@ -127,6 +127,7 @@ class Dataframe {
     this.__id = Dataframe.__getId();
 
     this.__compile(__columnsAccessor);
+    Object.freeze(this);
   }
 
   static __errorChecks(dims, columnarData, rowIndex, colIndex) {
@@ -269,6 +270,7 @@ class Dataframe {
     get.iget = iget;
     get.__id = __id;
 
+    Object.freeze(get);
     return get;
   }
 
@@ -288,6 +290,7 @@ class Dataframe {
       }
       return Dataframe.__compileColumn(column, getRowByOffset, getRowByLabel);
     });
+    Object.freeze(this.__columnsAccessor);
   }
 
   clone() {
@@ -345,12 +348,50 @@ class Dataframe {
     );
   }
 
+  withColsFrom(dataframe) {
+    /*
+    return a new dataframe containing all columns from both `this` and the
+    provided dataframe.
+
+    The row index from `this` will be used.  Both dataframes must have identical
+    dimensionality, and no overlapping columns labels.
+     */
+    const dims = [this.dims[0], this.dims[1] + dataframe.dims[1]];
+    const { rowIndex } = this;
+    const columns = [...this.__columns, ...dataframe.__columns];
+    const colIndex = this.colIndex.withLabels(dataframe.colIndex.keys());
+    const columnsAccessor = [
+      ...this.__columnsAccessor,
+      ...dataframe.__columnsAccessor
+    ];
+    return new this.constructor(
+      dims,
+      columns,
+      rowIndex,
+      colIndex,
+      columnsAccessor
+    );
+  }
+
   dropCol(label) {
     /*
     Create a new dataframe, omitting one columns.
 
     const newDf = df.dropCol("colors");
+
+    Corner case to manage: if dropping the last column, return an empty dataframe.
     */
+    if (!this.hasCol(label)) {
+      throw new RangeError(`unknown label: ${label}`);
+    }
+
+    /* 
+    Corner case to manage: if dropping the last column, return an empty dataframe. 
+    */
+    if (this.dims[1] === 1) {
+      return Dataframe.empty();
+    }
+
     const dims = [this.dims[0], this.dims[1] - 1];
     const coffset = this.colIndex.getOffset(label);
     const columns = [...this.__columns];
@@ -367,6 +408,50 @@ class Dataframe {
     );
   }
 
+  renameCol(oldLabel, newLabel) {
+    /*
+    Accelerator for dropping a column and then adding it again with a new label
+    */
+    const coffset = this.colIndex.getOffset(oldLabel);
+    const colIndex = this.colIndex.dropLabel(oldLabel).withLabel(newLabel);
+
+    const columns = [...this.__columns];
+    columns.push(columns[coffset]);
+    columns.splice(coffset, 1);
+
+    const columnsAccessor = [...this.__columnsAccessor];
+    columnsAccessor.push(columnsAccessor[coffset]);
+    columnsAccessor.splice(coffset, 1);
+
+    return new this.constructor(
+      this.dims,
+      columns,
+      this.rowIndex,
+      colIndex,
+      columnsAccessor
+    );
+  }
+
+  replaceColData(label, newColData) {
+    /*
+    Accelerator for dropping a column then adding it again with same
+    label and different values.
+    */
+    const coffset = this.colIndex.getOffset(label);
+    const columns = [...this.__columns];
+    columns[coffset] = newColData;
+    const columnsAccessor = [...this.__columnsAccessor];
+    columnsAccessor[coffset] = null;
+
+    return new this.constructor(
+      this.dims,
+      columns,
+      this.rowIndex,
+      this.colIndex,
+      columnsAccessor
+    );
+  }
+
   static empty(rowIndex = null, colIndex = null) {
     return new Dataframe([0, 0], [], rowIndex, colIndex);
   }
@@ -443,6 +528,8 @@ class Dataframe {
         return newCol;
       });
     }
+
+    if (dims[0] === 0 || dims[1] === 0) return Dataframe.empty();
     return new Dataframe(dims, columns, rowIndex, colIndex);
   }
 
@@ -526,6 +613,11 @@ class Dataframe {
   Data access with row/col.
   **/
 
+  columns() {
+    /* return all column accessors as an array, in offset order */
+    return [...this.__columnsAccessor];
+  }
+
   col(columnLabel) {
     /*
     Return accessor bound to a column.  Allows random row access

client/src/util/dataframe/histogram.js

@@ -8,13 +8,13 @@ function _histogramContinuous(column, bins, min, max) {
   if (!column) {
     return valBins;
   }
-  const binWidth = (max - min) / (bins - 1);
+  const binWidth = (max - min) / bins;
   const colArray = column.asArray();
   for (let r = 0, len = colArray.length; r < len; r += 1) {
     const val = colArray[r];
     if (val <= max && val >= min) {
       // ensure test excludes NaN values
-      const valBin = (val - min) / binWidth;
+      const valBin = Math.min(Math.floor((val - min) / binWidth), bins - 1);
       valBins[valBin] += 1;
     }
   }
@@ -26,7 +26,7 @@ function _histogramContinuousBy(column, bins, min, max, by) {
   if (!column || !by) {
     return byMap;
   }
-  const binWidth = (max - min) / (bins - 1);
+  const binWidth = (max - min) / bins;
   const byArray = by.asArray();
   const colArray = column.asArray();
   for (let r = 0, len = colArray.length; r < len; r += 1) {
@@ -39,8 +39,8 @@ function _histogramContinuousBy(column, bins, min, max, by) {
     const val = colArray[r];
     if (val <= max && val >= min) {
       // ensure test excludes NaN values
-      const valBin = (val - min) / binWidth;
-      valBins[Math.floor(valBin)] += 1;
+      const valBin = Math.min(Math.floor((val - min) / binWidth), bins - 1);
+      valBins[valBin] += 1;
     }
   }
   return byMap;

client/src/util/dataframe/labelIndex.js

@@ -80,6 +80,10 @@ class IdentityInt32Index {
     return this.__promote([...this.keys(), label]);
   }
 
+  withLabels(labels) {
+    return this.__promote([...this.keys(), ...labels]);
+  }
+
   dropLabel(label) {
     if (label === this.maxOffset - 1) {
       return new IdentityInt32Index(label);
@@ -163,6 +167,10 @@ class DenseInt32Index {
     return this.__promote([...this.keys(), label]);
   }
 
+  withLabels(labels) {
+    return this.__promote([...this.keys(), ...labels]);
+  }
+
   dropLabel(label) {
     const labelArray = [...this.keys()];
     labelArray.splice(labelArray.indexOf(label), 1);
@@ -187,6 +195,11 @@ class KeyIndex {
       index.set(v, i);
     });
 
+    if (index.size !== rindex.length) {
+      /* if true, there was a duplicate in the keys */
+      throw new Error("duplicate label provided to KeyIndex");
+    }
+
     this.index = index;
     this.rindex = rindex;
     this.__compile();
@@ -218,6 +231,10 @@ class KeyIndex {
     return new KeyIndex([...this.rindex, label]);
   }
 
+  withLabels(labels) {
+    return new KeyIndex([...this.rindex, ...labels]);
+  }
+
   dropLabel(label) {
     const idx = this.rindex.indexOf(label);
     const labelArray = [...this.rindex];

client/src/util/dataframe/util.js

@@ -21,7 +21,7 @@ export function callOnceLazy(f) {
   return result;
 }
 
-export function memoize(fn, hashFn) {
+export function memoize(fn, hashFn, maxResultsCached = -1) {
   /* 
   function memoization, with user-provided hash.  hashFn must return a
   key which will be unique as a Map key (ie, obeys "sameValueZero" algorithm
@@ -36,7 +36,19 @@ export function memoize(fn, hashFn) {
     }
     const result = fn(...args);
     cache.set(key, result);
+
+    if (maxResultsCached > -1 && cache.size > maxResultsCached) {
+      /* Least recent insertion deletion */
+      cache.delete(cache.keys().next().value);
+    }
+
     return result;
   };
+
+  wrap.clear = function clear() {
+    /* clear memoization cache */
+    cache.clear();
+  };
+
   return wrap;
 }

client/src/util/glHelpers.js

@@ -0,0 +1,81 @@
+/*
+Utility code for WebGL shaders
+*/
+
+/*
+PointFlags:
+  Point flags are used in graph & scatter plots.
+
+  We want a bitmask-like flag structure, but due to webgl limitations
+  must emulate it with floats.
+*/
+
+// for JS
+export const flagSelected = 1;
+export const flagNaN = 2;
+export const flagHighlight = 4;
+
+// for GLSL
+export const glPointFlags = `
+
+  const float flagSelected = 1.;
+  const float flagNaN = 2.;
+  const float flagHighlight = 4.;
+
+  bool isLowBitSet(float f) {
+    f = mod(f, 2.);
+    return (f > 0.9 && f <= 1.1);
+  }
+
+  float shiftRightOne(float f) {
+    return floor(f / 2.);
+  }
+
+  void getFlags(in float flag,
+                out bool isNaN,
+                out bool isSelected,
+                out bool isHighlight) {
+    isSelected = isLowBitSet(flag);
+    flag = shiftRightOne(flag);
+    isNaN = isLowBitSet(flag);
+    flag = shiftRightOne(flag);
+    isHighlight = isLowBitSet(flag);
+  }
+
+`;
+
+/*
+Point Size:
+  Calculate point size for scatter plot based upon pseudo density.
+
+  Current approach: linear scaling of point size, clamped to [1,10],
+  between two points that are based on empirical testing.
+
+    - 1M points on a 500x500 canvas: 1M/(500*500) -> 0.5
+    - 1000 points on a 1440x1440 canvas:  1000/(1440*1440) -> 5
+
+  The domain is pseudo density (numPoints / minViewportDimension^2)
+  The range is web gl point size.
+*/
+
+// configuration
+const domain = [1000000 / (500 * 500), 1000 / (1440 * 1440)];
+const range = [0.5, 5];
+
+// derived from configuration
+const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+const offset = scale * -domain[0] + range[0];
+
+export const glPointSize = `
+  float pointSize(float nPoints, float minViewportDimension, bool isSelected, bool isHighlight) {
+    float density = nPoints / (minViewportDimension * minViewportDimension);
+    float pointSize = (${scale.toFixed(4)}*density) + ${offset.toFixed(4)};
+    pointSize = clamp(pointSize, 
+      ${range[0].toFixed(4)}, 
+      ${range[1].toFixed(4)});
+
+    if (isHighlight) return 2. * pointSize;
+    if (isSelected) return pointSize;
+    return pointSize / 3.;
+  }
+`;

client/src/util/stateManager/annotationsHelpers.js

@@ -0,0 +1,168 @@
+/*
+Helper functions for user-editable nnotations state management.
+See also reducers/annotations.js
+*/
+import { unassignedCategoryLabel } from "../../globals";
+import * as SchemaHelpers from "./schemaHelpers";
+import { obsAnnoDimensionName } from "../nameCreators";
+
+/*
+There are a number of state constraints assumed throughout the
+application:
+	- all obs annotations are in {world|universe}.obsAnnotations,
+		regardless of whether or not they are user editable.
+	- the {world|universe}.schema is always up to date and matches
+		the data
+	- the schema flag `writable` correctly indicates whether
+	  the annotation is editable/mutable.
+
+In addition, the current state management only allows for
+categorical annotations to be writable.
+*/
+
+export function isCategoricalAnnotation(schema, name) {
+	/* we treat any string, categorical or boolean as a categorical */
+	const { type } = schema.annotations.obsByName[name];
+	return type === "string" || type === "boolean" || type === "categorical";
+}
+
+export function isContinuousAnnotation(schema, name) {
+	return !isCategoricalAnnotation(schema, name);
+}
+
+function _isUserAnnotation(schema, name) {
+	return schema.annotations.obsByName[name]?.writable;
+}
+
+export function isUserAnnotation(worldOrUniverse, name) {
+	return _isUserAnnotation(worldOrUniverse.schema, name);
+}
+
+export function removeObsAnnoSchema(schema, name) {
+	/*
+	remove named annotation from obs annotation schema
+	*/
+
+	/* only remove if it exists and is a user annotation */
+	if (!_isUserAnnotation(schema, name))
+		throw new Error("removing non-user-defined schema");
+	return SchemaHelpers.removeObsAnnoColumn(schema, name);
+}
+
+export function addObsAnnoSchema(schema, name, colSchema) {
+	/*
+	add a categorical type to the obs annotation schema 
+	*/
+
+	/* collision detection */
+	if (schema.annotations.obs.columns.some(v => v.name === name))
+		throw Error("annotations may not contain duplicate category names");
+	if (name !== colSchema.name) throw Error("column schema does not match");
+	return SchemaHelpers.addObsAnnoColumn(schema, name, colSchema);
+}
+
+export function dupObsAnnoSchema(schema, sourceName, dupName, defaultSchema) {
+	/*
+	duplicate the obs annotation `sourceName` schema, but with the name `dupName`
+	*/
+	const colSchema = {
+		...schema.annotations.obsByName[sourceName],
+		...defaultSchema,
+		name: dupName
+	};
+	/* existance check */
+	if (!colSchema) throw Error("source annotation does not exist");
+	/* collision detection */
+	if (schema.annotations.obs.columns.some(v => v.name === dupName))
+		throw Error("annotations may not contain duplicate category names");
+	return SchemaHelpers.addObsAnnoColumn(schema, dupName, colSchema);
+}
+
+export function removeObsAnnoCategory(schema, name, category) {
+	/* don't allow deletion of unassigned category on writable annotations */
+
+	if (!_isUserAnnotation(schema, name))
+		throw new Error("unable to modify read-only schema");
+	if (category === unassignedCategoryLabel)
+		throw new Error("may not remove unassigned category label");
+
+	return SchemaHelpers.removeObsAnnoCategory(schema, name, category);
+}
+
+export function addObsAnnoCategory(schema, name, category) {
+	if (!_isUserAnnotation(schema, name))
+		throw new Error("unable to modify read-only schema");
+
+	return SchemaHelpers.addObsAnnoCategory(schema, name, category);
+}
+
+export function setLabelByValue(df, colName, fromLabel, toLabel) {
+	/* 
+	in the dataframe column `colName`, set any value of `fromLabel` to `toLabel`
+	*/
+	const keys = df.colIndex.keys();
+	const ndf = df.mapColumns((col, colIdx) => {
+		if (colName !== keys[colIdx]) return col;
+
+		/* clone data and return it. */
+		const newCol = col.slice();
+		for (let i = 0, l = newCol.length; i < l; i += 1) {
+			if (newCol[i] === fromLabel) newCol[i] = toLabel;
+		}
+		return newCol;
+	});
+	return ndf;
+}
+
+export function setLabelByMask(df, colName, mask, label) {
+	/* 
+	in the dataframe column `colName`, set the masked rows to 'label'
+	*/
+	const keys = df.colIndex.keys();
+	const ndf = df.mapColumns((col, colIdx) => {
+		if (colName !== keys[colIdx]) return col;
+
+		/* clone data and return it. */
+		const newCol = col.slice();
+		for (let i = 0, l = newCol.length; i < l; i += 1) {
+			if (mask[i]) newCol[i] = label;
+		}
+		return newCol;
+	});
+	return ndf;
+}
+
+export function worldToUniverseMask(worldMask, worldObsAnnotations, nObs) {
+	/*
+	given world seleciton mask, return a selection mask for entire universe 
+	that has same selection state.
+	*/
+	const mask = new Uint8Array(nObs);
+	const { rowIndex } = worldObsAnnotations;
+
+	for (let i = 0, l = worldMask.length; i < l; i += 1) {
+		if (worldMask[i]) {
+			const label = rowIndex.getLabel(i);
+			mask[label] = 1;
+		}
+	}
+	return mask;
+}
+
+export function createWritableAnnotationDimensions(world, crossfilter) {
+	const { obsAnnotations, schema } = world;
+	const writableAnnotations = schema.annotations.obs.columns
+		.filter(s => s.writable)
+		.map(s => s.name);
+
+	crossfilter = writableAnnotations.reduce((xflt, anno) => {
+		const dimName = obsAnnoDimensionName(anno);
+		if (xflt.hasDimension(dimName)) xflt = xflt.delDimension(dimName);
+		return xflt.addDimension(
+			dimName,
+			"enum",
+			obsAnnotations.col(anno).asArray()
+		);
+	}, crossfilter);
+	return crossfilter;
+}

client/src/util/stateManager/colorHelpers.js

@@ -50,8 +50,9 @@ function createColorsByCategoricalMetadata(world, accessor) {
   }, {});
 
   const rgb = new Array(world.nObs);
-  const data = world.obsAnnotations.col(accessor).asArray();
-  for (let i = 0, len = world.obsAnnotations.length; i < len; i += 1) {
+  const df = world.obsAnnotations;
+  const data = df.col(accessor).asArray();
+  for (let i = 0, len = df.length; i < len; i += 1) {
     const cat = data[i];
     rgb[i] = colors[cat];
   }

client/src/util/stateManager/controlsHelpers.js

@@ -37,16 +37,14 @@ Remember that option values can be ANY js type, except undefined/null.
     }
   }
 */
-function topNCategories(summary) {
-  const counts = _.map(summary.categories, cat =>
-    summary.categoryCounts.get(cat)
-  );
-  const sortIndex = fillRange(new Array(summary.numCategories)).sort(
+function topNCategories(colSchema, summary, N) {
+  const { categories } = colSchema;
+  const counts = _.map(categories, cat => summary.categoryCounts.get(cat) ?? 0);
+  const sortIndex = fillRange(new Array(categories.length)).sort(
     (a, b) => counts[b] - counts[a]
   );
-  const sortedCategories = _.map(sortIndex, i => summary.categories[i]);
+  const sortedCategories = _.map(sortIndex, i => categories[i]);
   const sortedCounts = _.map(sortIndex, i => counts[i]);
-  const N = globals.maxCategoricalOptionsToDisplay;
 
   if (sortedCategories.length < N) {
     return [sortedCategories, sortedCounts];
@@ -54,64 +52,56 @@ function topNCategories(summary) {
   return [sortedCategories.slice(0, N), sortedCounts.slice(0, N)];
 }
 
-export function createCategoricalSelection(maxCategoryItems, world) {
-  const res = {};
-  const obsIndexName = world.schema.annotations.obs.index;
-  _.forEach(world.obsAnnotations.colIndex.keys(), key => {
-    const summary = world.obsAnnotations.col(key).summarize();
-    if (summary.categories) {
-      const isColorField = key.includes("color") || key.includes("Color");
-      const isSelectableCategory =
-        !isColorField &&
-        key !== obsIndexName &&
-        summary.categories.length < maxCategoryItems;
-      if (isSelectableCategory) {
-        const [categoryValues, categoryValueCounts] = topNCategories(summary);
-        const categoryValueIndices = new Map(
-          categoryValues.map((v, i) => [v, i])
-        );
-        const numCategoryValues = categoryValueIndices.size;
-        const categoryValueSelected = new Array(numCategoryValues).fill(true);
-        const isTruncated = categoryValues.length < summary.numCategories;
-        res[key] = {
-          categoryValues, // array: of natively typed category values
-          categoryValueIndices, // map: category value (native type) -> category index
-          categoryValueSelected, // array: t/f selection state
-          numCategoryValues, // number: of values in the category
-          isTruncated, // bool: true if list was truncated
-          categoryValueCounts, // array: cardinality of each category,
-          categorySelected: true // bool - default state for entire category
-        };
-      }
-    }
-  });
-  return res;
+export function selectableCategoryNames(world, maxCategoryItems) {
+  const { schema } = world;
+  const { index, columns } = schema.annotations.obs;
+  return columns
+    .filter(colSchema => {
+      const { name, categories } = colSchema;
+      return (
+        categories && categories.length < maxCategoryItems && name !== index
+      );
+    })
+    .map(v => v.name);
 }
 
-/*
-given a categoricalSelection, return the list of all category values
-where selection state is true (ie, they are selected).
-*/
-export function selectedValuesForCategory(categorySelectionState, dfColumn) {
-  const {
-    categorySelected,
-    categoryValueSelected,
-    categoryValueIndices
-  } = categorySelectionState;
-  let selectedValues;
-  if (categorySelected) {
-    selectedValues = new Set(dfColumn.summarize().categories);
-  } else {
-    selectedValues = new Set();
-  }
-  categoryValueIndices.forEach((catIndex, catValue) => {
-    if (!categoryValueSelected[catIndex]) {
-      selectedValues.delete(catValue);
-    } else {
-      selectedValues.add(catValue);
-    }
-  });
-  return [...selectedValues.values()];
+export function createCategoricalSelection(world, names) {
+  const N = globals.maxCategoricalOptionsToDisplay;
+  const { obsAnnotations, schema } = world;
+
+  const res = names.reduce((acc, name) => {
+    const colSchema = schema.annotations.obsByName[name];
+    const { writable: isUserAnno } = colSchema;
+
+    /*
+    Summarize the annotation data currently in world.  Must return categoryValues
+    in sorted order, and must include all category values even if they are not
+    actively used in the current world.
+    */
+    const summary = obsAnnotations.col(name).summarize();
+    const [categoryValues, categoryValueCounts] = topNCategories(
+      colSchema,
+      summary,
+      N
+    );
+    const categoryValueIndices = new Map(categoryValues.map((v, i) => [v, i]));
+    const numCategoryValues = categoryValueIndices.size;
+    const categoryValueSelected = new Array(numCategoryValues).fill(true);
+    const isTruncated = categoryValues.length < summary.numCategories;
+
+    acc[name] = {
+      categoryValues, // array: of natively typed category values
+      categoryValueIndices, // map: category value (native type) -> category index
+      categoryValueSelected, // array: t/f selection state
+      numCategoryValues, // number: of values in the category
+      isTruncated, // bool: true if list was truncated
+      categoryValueCounts, // array: cardinality of each category,
+      categorySelected: true, // bool - default state for entire category
+      isUserAnno // bool
+    };
+    return acc;
+  }, {});
+  return res;
 }
 
 /*

client/src/util/stateManager/index.js

@@ -17,5 +17,7 @@ exists to support those concepts.
 export * as ColorHelpers from "./colorHelpers";
 export * as Universe from "./universe";
 export * as World from "./world";
-export * as WorldUtil from "./worldUtil";
 export * as ControlsHelpers from "./controlsHelpers";
+export * as AnnotationsHelpers from "./annotationsHelpers";
+export * as SchemaHelpers from "./schemaHelpers";
+export * as MatrixFBS from "./matrix";

client/src/util/stateManager/matrix.js

@@ -1,5 +1,7 @@
 import { flatbuffers } from "flatbuffers";
 import { NetEncoding } from "./matrix_generated";
+import { isTypedArray } from "../typeHelpers";
+import { IdentityInt32Index, DenseInt32Index, KeyIndex } from "../dataframe";
 
 const utf8Decoder = new TextDecoder("utf-8");
 
@@ -41,25 +43,25 @@ Returns: object containing decoded Matrix:
   colIdx: []|null
 }
 */
-function decodeMatrixFBS(arrayBuffer, inplace = false) {
+export function decodeMatrixFBS(arrayBuffer, inplace = false) {
   const bb = new flatbuffers.ByteBuffer(new Uint8Array(arrayBuffer));
-  const df = NetEncoding.Matrix.getRootAsMatrix(bb);
+  const matrix = NetEncoding.Matrix.getRootAsMatrix(bb);
 
-  const nRows = df.nRows();
-  const nCols = df.nCols();
+  const nRows = matrix.nRows();
+  const nCols = matrix.nCols();
 
   /* decode columns */
-  const columnsLength = df.columnsLength();
+  const columnsLength = matrix.columnsLength();
   const columns = Array(columnsLength).fill(null);
   for (let c = 0; c < columnsLength; c += 1) {
-    const col = df.columns(c);
+    const col = matrix.columns(c);
     columns[c] = decodeTypedArray(col.uType(), col.u.bind(col), inplace);
   }
 
   /* decode col_idx */
   const colIdx = decodeTypedArray(
-    df.colIndexType(),
-    df.colIndex.bind(df),
+    matrix.colIndexType(),
+    matrix.colIndex.bind(matrix),
     inplace
   );
 
@@ -72,4 +74,91 @@ function decodeMatrixFBS(arrayBuffer, inplace = false) {
   };
 }
 
-export default decodeMatrixFBS;
+function encodeTypedArray(builder, uType, uData) {
+  const uTypeName = NetEncoding.TypedArray[uType];
+  const ArrayType = NetEncoding[uTypeName];
+  const dv = ArrayType.createDataVector(builder, uData);
+  builder.startObject(1);
+  builder.addFieldOffset(0, dv, 0);
+  return builder.endObject();
+}
+
+export function encodeMatrixFBS(df) {
+  /*
+  encode the dataframe as an FBS Matrix
+  */
+
+  /* row indexing not supported currently */
+  if (df.rowIndex.constructor !== IdentityInt32Index) {
+    throw new Error("FBS does not support row index encoding at this time");
+  }
+
+  const shape = df.dims;
+  const utf8Encoder = new TextEncoder("utf-8");
+  const builder = new flatbuffers.Builder(1024);
+
+  let encColIndex;
+  let encColIndexUType;
+  let encColumns;
+
+  if (shape[0] > 0 && shape[1] > 0) {
+    const columns = df.columns().map(col => col.asArray());
+
+    const cols = columns.map(carr => {
+      let uType;
+      let tarr;
+      if (isTypedArray(carr)) {
+        uType = NetEncoding.TypedArray[carr.constructor.name];
+        tarr = encodeTypedArray(builder, uType, carr);
+      } else {
+        uType = NetEncoding.TypedArray.JSONEncodedArray;
+        const json = JSON.stringify(carr);
+        const jsonUTF8 = utf8Encoder.encode(json);
+        tarr = encodeTypedArray(builder, uType, jsonUTF8);
+      }
+      NetEncoding.Column.startColumn(builder);
+      NetEncoding.Column.addUType(builder, uType);
+      NetEncoding.Column.addU(builder, tarr);
+      return NetEncoding.Column.endColumn(builder);
+    });
+
+    encColumns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+    if (df.colIndex && shape[1] > 0) {
+      const colIndexType = df.colIndex.constructor;
+      if (colIndexType === IdentityInt32Index) {
+        encColIndex = undefined;
+      } else if (colIndexType === DenseInt32Index) {
+        encColIndexUType = NetEncoding.TypedArray.Int32Array;
+        encColIndex = encodeTypedArray(
+          builder,
+          encColIndexUType,
+          df.colIndex.keys()
+        );
+      } else if (colIndexType === KeyIndex) {
+        encColIndexUType = NetEncoding.TypedArray.JSONEncodedArray;
+        encColIndex = encodeTypedArray(
+          builder,
+          encColIndexUType,
+          utf8Encoder.encode(JSON.stringify(df.colIndex.keys()))
+        );
+      } else {
+        throw new Error("Index type FBS encoding unsupported");
+      }
+    }
+  }
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, shape[0]);
+  NetEncoding.Matrix.addNCols(builder, shape[1]);
+  if (encColumns) {
+    NetEncoding.Matrix.addColumns(builder, encColumns);
+  }
+  if (encColIndexUType) {
+    NetEncoding.Matrix.addColIndexType(builder, encColIndexUType);
+    NetEncoding.Matrix.addColIndex(builder, encColIndex);
+  }
+  const root = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(root);
+  return builder.asUint8Array();
+}

client/src/util/stateManager/schemaHelpers.js

@@ -0,0 +1,96 @@
+/*
+Helpers for schema management
+*/
+import _ from "lodash";
+
+import fromEntries from "../fromEntries";
+
+/*
+System wide schema assumptions:
+	- schema and data wil be consistent (eg, for user-created annotations)
+	- schema will be internally self-consistent (eg, index matches columns)
+	- world & universe schema are same - only data is subset
+*/
+
+export function indexEntireSchema(schema) {
+	/* Index schema for ease of use */
+	schema.annotations.obsByName = fromEntries(
+		schema.annotations.obs.columns.map(v => [v.name, v])
+	);
+	schema.annotations.varByName = fromEntries(
+		schema.annotations.var.columns.map(v => [v.name, v])
+	);
+	schema.layout.obsByName = fromEntries(
+		schema.layout.obs.map(v => [v.name, v])
+	);
+	schema.layout.varByName = fromEntries(
+		schema.layout.var.map(v => [v.name, v])
+	);
+
+	return schema;
+}
+
+function _copy(schema) {
+	/* redux copy conventions - WARNING, only for modifyign obs annotations */
+	return {
+		...schema,
+		annotations: {
+			...schema.annotations,
+			obs: _.cloneDeep(schema.annotations.obs)
+		}
+	};
+}
+
+function _reindex(schema) {
+	/* reindex obs annotations ONLY */
+	schema.annotations.obsByName = fromEntries(
+		schema.annotations.obs.columns.map(v => [v.name, v])
+	);
+	return schema;
+}
+
+export function removeObsAnnoColumn(schema, name) {
+	const newSchema = _copy(schema);
+	newSchema.annotations.obs.columns = schema.annotations.obs.columns.filter(
+		v => v.name !== name
+	);
+	return _reindex(newSchema);
+}
+
+export function addObsAnnoColumn(schema, name, defn) {
+	const newSchema = _copy(schema);
+	newSchema.annotations.obs.columns.push(defn);
+	return _reindex(newSchema);
+}
+
+export function removeObsAnnoCategory(schema, name, category) {
+	/* remove a category from a categorical annotation */
+	const categories = schema.annotations.obsByName[name]?.categories;
+	if (!categories)
+		throw new Error("column does not exist or is not categorical");
+
+	const idx = categories.indexOf(category);
+	if (idx === -1) throw new Error("category does not exist");
+
+	const newSchema = _reindex(_copy(schema));
+
+	/* remove category */
+	newSchema.annotations.obsByName[name].categories.splice(idx, 1);
+	return newSchema;
+}
+
+export function addObsAnnoCategory(schema, name, category) {
+	/* add a category to a categorical annotation */
+	const categories = schema.annotations.obsByName[name]?.categories;
+	if (!categories)
+		throw new Error("column does not exist or is not categorical");
+
+	const idx = categories.indexOf(category);
+	if (idx !== -1) throw new Error("category already exists");
+
+	const newSchema = _reindex(_copy(schema));
+
+	/* remove category */
+	newSchema.annotations.obsByName[name].categories.push(category);
+	return newSchema;
+}

client/src/util/stateManager/universe.js

@@ -1,11 +1,11 @@
-// jshint esversion: 6
-
 import _ from "lodash";
 
-import decodeMatrixFBS from "./matrix";
+import { unassignedCategoryLabel } from "../../globals";
+import { decodeMatrixFBS } from "./matrix";
 import * as Dataframe from "../dataframe";
-import fromEntries from "../fromEntries";
 import { isFpTypedArray } from "../typeHelpers";
+import { indexEntireSchema } from "./schemaHelpers";
+import { isCategoricalAnnotation } from "./annotationsHelpers";
 
 /*
 Private helper function - create and return a template Universe
@@ -18,10 +18,13 @@ function templateUniverse() {
     schema: {},
 
     /*
-    Annotations
+    annotations
     */
     obsAnnotations: Dataframe.Dataframe.empty(),
     varAnnotations: Dataframe.Dataframe.empty(),
+    /*
+    layout
+    */
     obsLayout: Dataframe.Dataframe.empty(),
 
     /*
@@ -122,6 +125,10 @@ function reconcileSchemaCategoriesWithSummary(universe) {
   For example, boolean defined fields in the schema do not contain
   explicit declaration of categories (nor do string fields).  In these
   cases, add a 'categories' field to the schema so it is accessible.
+
+  In addition, we have a client-side convention (UI) that all writable
+  annotations must have an 'unassigned' category, even if it is not currently
+  in use.
   */
 
   universe.schema.annotations.obs.columns.forEach(s => {
@@ -136,6 +143,10 @@ function reconcileSchemaCategoriesWithSummary(universe) {
       );
       s.categories = categories;
     }
+
+    if (s.writable && s.categories.indexOf(unassignedCategoryLabel) === -1) {
+      s.categories = s.categories.concat(unassignedCategoryLabel);
+    }
   });
 }
 
@@ -166,7 +177,7 @@ export function createUniverseFromResponse(
   /* layout */
   universe.obsLayout = LayoutFBSToDataframe(layoutFBSResponse);
 
-  /* sanity check */
+  /* sanity checks */
   if (
     universe.nObs !== universe.obsLayout.length ||
     universe.nObs !== universe.obsAnnotations.length ||
@@ -176,20 +187,19 @@ export function createUniverseFromResponse(
   }
 
   reconcileSchemaCategoriesWithSummary(universe);
+  indexEntireSchema(universe.schema);
+
+  /* sanity checks */
+  if (
+    schema.annotations.obs.columns.some(
+      s => s.writable && !isCategoricalAnnotation(schema, s.name)
+    )
+  ) {
+    throw new Error(
+      "Writable continuous obs annotations are not supproted - failed to laod"
+    );
+  }
 
-  /* Index schema for ease of use */
-  universe.schema.annotations.obsByName = fromEntries(
-    universe.schema.annotations.obs.columns.map(v => [v.name, v])
-  );
-  universe.schema.annotations.varByName = fromEntries(
-    universe.schema.annotations.var.columns.map(v => [v.name, v])
-  );
-  universe.schema.layout.obsByName = fromEntries(
-    universe.schema.layout.obs.map(v => [v.name, v])
-  );
-  universe.schema.layout.varByName = fromEntries(
-    universe.schema.layout.var.map(v => [v.name, v])
-  );
   return universe;
 }
 

client/src/util/stateManager/world.js

@@ -1,14 +1,7 @@
-// jshint esversion: 6
-
 import clip from "../clip";
-import {
-  layoutDimensionName,
-  obsAnnoDimensionName,
-  diffexpDimensionName,
-  userDefinedDimensionName
-} from "../nameCreators";
+import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
 import * as Dataframe from "../dataframe";
-import ImmutableTypedCrossfilter from "../typedCrossfilter/crossfilter";
+import { isContinuousAnnotation } from "./annotationsHelpers";
 
 /*
 
@@ -158,7 +151,7 @@ and world.varData.
 function setClippedDataframes(world) {
   const { schema } = world;
   const isContinuousObsAnnotation = (df, idx, label) =>
-    deduceDimensionType(schema.annotations.obsByName[label], label) !== "enum";
+    isContinuousAnnotation(schema, label);
   const obsQuantile = (label, q) =>
     world.unclipped.obsAnnotations.col(label).summarize().percentiles[100 * q];
   world.obsAnnotations = clipDataframe(
@@ -183,7 +176,7 @@ function setClippedDataframes(world) {
 /*
 Subset the current world based upon the current selection, maintaining any existing
 clip.  Returns new world.  Parameters:
-  * unvierse
+  * universe
   * world - the current world
   * crossfilter - the selection state
 */

client/src/util/stateManager/worldUtil.js

@@ -1,75 +0,0 @@
-/* eslint-disable import/prefer-default-export */
-import _ from "lodash";
-
-/*
-Various utility functions operating on World/Universe
-*/
-
-/*
-Count unique category values, binning first by dim1 then by dim2
-Return:
-
-Map {
-   dim1_val1: Map {
-      dim2_val1: number,
-      dim2_val2: number,
-      ...
-   },
-   ...
-}
-
-Parameters are:
-- dim1: dimension 1 name/label
-- dim2: dimension 2 name/label
-- df: dataframe containing dim1 and dim2 on the column axis
-
-*/
-function _countCategoryValues2D(dim1, dim2, df) {
-  const dimMap = new Map();
-  const col1 = df.col(dim1) ? df.col(dim1).asArray() : null;
-  const col2 = df.col(dim2) ? df.col(dim2).asArray() : null;
-  if (!col1 || !col2) {
-    return dimMap;
-  }
-
-  for (let r = 0, l = df.length; r < l; r += 1) {
-    const val1 = col1[r];
-    const val2 = col2[r];
-    let d2Map = dimMap.get(val1);
-    if (d2Map === undefined) {
-      d2Map = new Map();
-      dimMap.set(val1, d2Map);
-    }
-    let curCount = d2Map.get(val2);
-    if (curCount === undefined) {
-      curCount = 0;
-    }
-    d2Map.set(val2, curCount + 1);
-  }
-  return dimMap;
-}
-
-let __worldUtilMemoId__ = 0;
-function _memoizedId(x) {
-  if (!x.__worldUtilMemoId__) {
-    __worldUtilMemoId__ += 1;
-    x.__worldUtilMemoId__ = __worldUtilMemoId__;
-  }
-  return x.__worldUtilMemoId__;
-}
-function _countCategoryValues2DResolver(...args) {
-  const id = args[0] + args[1] + _memoizedId(args[2]);
-  return id;
-}
-
-export const countCategoryValues2D = _.memoize(
-  _countCategoryValues2D,
-  _countCategoryValues2DResolver
-);
-
-/*
-Clear any cached data within WorldUtil caches, eg, memoized functions
-*/
-export function clearCaches() {
-  countCategoryValues2D.cache.clear();
-}

client/src/util/typedCrossfilter/bitArray.js

@@ -35,6 +35,7 @@ class BitArray {
 
     this.bitmask = new Int32Array(this.width); // dimension allocation mask
     this.bitarray = new Int32Array(this.width * this.length);
+    Object.seal(this);
   }
 
   // Return the number of records that are selected, ie, have a one bit in

client/src/util/typedCrossfilter/crossfilter.js

@@ -5,6 +5,7 @@ import BitArray from "./bitArray";
 import {
   sortArray,
   lowerBound,
+  binarySearch,
   lowerBoundIndirect,
   upperBoundIndirect
 } from "./sort";
@@ -46,6 +47,7 @@ export default class ImmutableTypedCrossfilter {
     this.data = data;
     this.selectionCache = selectionCache; /* BitArray */
     this.dimensions = dimensions; /* name: { id, dim, name, selection } */
+    Object.preventExtensions(this);
   }
 
   size() {
@@ -56,6 +58,14 @@ export default class ImmutableTypedCrossfilter {
     return this.data;
   }
 
+  setData(data) {
+    return new ImmutableTypedCrossfilter(
+      data,
+      this.dimensions,
+      this.selectionCache
+    );
+  }
+
   dimensionNames() {
     /* return array of all dimensions (by name) */
     return Object.keys(this.dimensions);
@@ -85,6 +95,7 @@ export default class ImmutableTypedCrossfilter {
     }
     const DimensionType = DimTypes[type];
     const dim = new DimensionType(name, data, ...rest);
+    Object.freeze(dim);
     const dimensions = {
       ...this.dimensions,
       [name]: {
@@ -113,6 +124,23 @@ export default class ImmutableTypedCrossfilter {
     return new ImmutableTypedCrossfilter(data, dimensions, selectionCache);
   }
 
+  renameDimension(oldName, newName) {
+    /*
+    rename a dimension
+    */
+    const { [oldName]: dim, ...dimensions } = this.dimensions;
+    const { data, selectionCache } = this;
+    dim.dim.rename(newName);
+    return new ImmutableTypedCrossfilter(
+      data,
+      {
+        ...dimensions,
+        [newName]: dim
+      },
+      selectionCache
+    );
+  }
+
   select(name, spec) {
     /*
     select on named dimension, as indicated by `spec`.   Spec is an object
@@ -288,6 +316,10 @@ class _ImmutableBaseDimension {
     this.name = name;
   }
 
+  rename(name) {
+    this.name = name;
+  }
+
   select(spec) {
     const { mode } = spec;
     if (mode === undefined) {
@@ -436,7 +468,7 @@ class ImmutableEnumDimension extends ImmutableScalarDimension {
     const { values } = spec;
     return super.selectExact({
       mode: spec.mode,
-      values: values.map(v => lowerBound(enumIndex, v, 0, enumIndex.length))
+      values: values.map(v => binarySearch(enumIndex, v, 0, enumIndex.length))
     });
   }
 

client/src/util/typedCrossfilter/sort.js

@@ -413,3 +413,16 @@ export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
   }
   return upperBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
 }
+
+// Search for `value` in the sorted array `arr`, in the range [first, last).
+// Return the first index where arr[index] == value, OR if value not present,
+// return `last`
+//
+// The same semantics/behavior as:
+//    C++: binary_search()
+//
+export function binarySearch(valueArray, value, first, last) {
+  const index = lowerBound(valueArray, value, first, last);
+  if (index !== last && value === valueArray[index]) return index;
+  return last;
+}

docs/_layouts/default.html

@@ -17,6 +17,7 @@
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta name="theme-color" content="#157878">
     <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
+    <link rel="shortcut icon" type="image/x-icon" href="cellxgene-favicon.png">
     <link rel="stylesheet" href="{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}">
   </head>
   <body>

docs/cellxgene-logo.svg

@@ -0,0 +1,52 @@
+<svg width="315" height="106" viewBox="0 0 315 106" fill="none" xmlns="http://www.w3.org/2000/svg">
+<rect width="315" height="106" fill="white"/>
+<path d="M93.9375 63.375C95.0469 63.375 95.9062 63.0781 96.5156 62.4844C97.125 61.875 97.4219 61.0312 97.4062 59.9531H104.812C104.812 62.7344 103.812 65.0156 101.812 66.7969C99.8281 68.5781 97.2656 69.4688 94.125 69.4688C90.4375 69.4688 87.5312 68.3125 85.4062 66C83.2812 63.6875 82.2188 60.4844 82.2188 56.3906V56.0625C82.2188 53.5 82.6875 51.2422 83.625 49.2891C84.5781 47.3203 85.9453 45.8125 87.7266 44.7656C89.5078 43.7031 91.6172 43.1719 94.0547 43.1719C97.3516 43.1719 99.9688 44.0859 101.906 45.9141C103.844 47.7422 104.812 50.2188 104.812 53.3438H97.4062C97.4062 52.0312 97.0859 51.0234 96.4453 50.3203C95.8047 49.6172 94.9453 49.2656 93.8672 49.2656C91.8203 49.2656 90.625 50.5703 90.2812 53.1797C90.1719 54.0078 90.1172 55.1484 90.1172 56.6016C90.1172 59.1484 90.4219 60.9141 91.0312 61.8984C91.6406 62.8828 92.6094 63.375 93.9375 63.375ZM120.422 69.4688C116.531 69.4688 113.383 68.3125 110.977 66C108.57 63.6719 107.367 60.6484 107.367 56.9297V56.2734C107.367 53.6797 107.844 51.3906 108.797 49.4062C109.766 47.4219 111.172 45.8906 113.016 44.8125C114.859 43.7188 117.047 43.1719 119.578 43.1719C123.141 43.1719 125.953 44.2812 128.016 46.5C130.078 48.7031 131.109 51.7812 131.109 55.7344V58.8047H115.406C115.688 60.2266 116.305 61.3438 117.258 62.1562C118.211 62.9688 119.445 63.375 120.961 63.375C123.461 63.375 125.414 62.5 126.82 60.75L130.43 65.0156C129.445 66.375 128.047 67.4609 126.234 68.2734C124.438 69.0703 122.5 69.4688 120.422 69.4688ZM119.531 49.2656C117.219 49.2656 115.844 50.7969 115.406 53.8594H123.375V53.25C123.406 51.9844 123.086 51.0078 122.414 50.3203C121.742 49.6172 120.781 49.2656 119.531 49.2656ZM142.641 69H134.719V33H142.641V69ZM155.859 69H147.938V33H155.859V69ZM185.414 56.1328C185.414 53.5703 185.836 51.3047 186.68 49.3359C187.523 47.3672 188.734 45.8516 190.312 44.7891C191.891 43.7109 193.727 43.1719 195.82 43.1719C198.461 43.1719 200.523 44.0625 202.008 45.8438L202.289 43.6406H209.461V68.0391C209.461 70.2734 208.938 72.2109 207.891 73.8516C206.859 75.5078 205.359 76.7734 203.391 77.6484C201.422 78.5391 199.141 78.9844 196.547 78.9844C194.688 78.9844 192.883 78.6328 191.133 77.9297C189.398 77.2266 188.07 76.3125 187.148 75.1875L190.453 70.5469C191.922 72.2812 193.859 73.1484 196.266 73.1484C199.766 73.1484 201.516 71.3516 201.516 67.7578V66.9609C200 68.6328 198.086 69.4688 195.773 69.4688C192.68 69.4688 190.18 68.2891 188.273 65.9297C186.367 63.5547 185.414 60.3828 185.414 56.4141V56.1328ZM193.336 56.625C193.336 58.7031 193.727 60.3516 194.508 61.5703C195.289 62.7734 196.383 63.375 197.789 63.375C199.492 63.375 200.734 62.8047 201.516 61.6641V51C200.75 49.8438 199.523 49.2656 197.836 49.2656C196.43 49.2656 195.328 49.8984 194.531 51.1641C193.734 52.4141 193.336 54.2344 193.336 56.625ZM226.219 69.4688C222.328 69.4688 219.18 68.3125 216.773 66C214.367 63.6719 213.164 60.6484 213.164 56.9297V56.2734C213.164 53.6797 213.641 51.3906 214.594 49.4062C215.562 47.4219 216.969 45.8906 218.812 44.8125C220.656 43.7188 222.844 43.1719 225.375 43.1719C228.938 43.1719 231.75 44.2812 233.812 46.5C235.875 48.7031 236.906 51.7812 236.906 55.7344V58.8047H221.203C221.484 60.2266 222.102 61.3438 223.055 62.1562C224.008 62.9688 225.242 63.375 226.758 63.375C229.258 63.375 231.211 62.5 232.617 60.75L236.227 65.0156C235.242 66.375 233.844 67.4609 232.031 68.2734C230.234 69.0703 228.297 69.4688 226.219 69.4688ZM225.328 49.2656C223.016 49.2656 221.641 50.7969 221.203 53.8594H229.172V53.25C229.203 51.9844 228.883 51.0078 228.211 50.3203C227.539 49.6172 226.578 49.2656 225.328 49.2656ZM247.359 43.6406L247.617 46.6172C249.367 44.3203 251.781 43.1719 254.859 43.1719C257.5 43.1719 259.469 43.9609 260.766 45.5391C262.078 47.1172 262.758 49.4922 262.805 52.6641V69H254.883V52.9922C254.883 51.7109 254.625 50.7734 254.109 50.1797C253.594 49.5703 252.656 49.2656 251.297 49.2656C249.75 49.2656 248.602 49.875 247.852 51.0938V69H239.953V43.6406H247.359ZM279.469 69.4688C275.578 69.4688 272.43 68.3125 270.023 66C267.617 63.6719 266.414 60.6484 266.414 56.9297V56.2734C266.414 53.6797 266.891 51.3906 267.844 49.4062C268.812 47.4219 270.219 45.8906 272.062 44.8125C273.906 43.7188 276.094 43.1719 278.625 43.1719C282.188 43.1719 285 44.2812 287.062 46.5C289.125 48.7031 290.156 51.7812 290.156 55.7344V58.8047H274.453C274.734 60.2266 275.352 61.3438 276.305 62.1562C277.258 62.9688 278.492 63.375 280.008 63.375C282.508 63.375 284.461 62.5 285.867 60.75L289.477 65.0156C288.492 66.375 287.094 67.4609 285.281 68.2734C283.484 69.0703 281.547 69.4688 279.469 69.4688ZM278.578 49.2656C276.266 49.2656 274.891 50.7969 274.453 53.8594H282.422V53.25C282.453 51.9844 282.133 51.0078 281.461 50.3203C280.789 49.6172 279.828 49.2656 278.578 49.2656Z" fill="#E9429A" fill-opacity="0.5"/>
+<path d="M160.617 61.2891L168.352 53.4141L160.664 45.5625L163.453 42.6797L171.164 50.5312L178.875 42.6797L181.664 45.5625L173.977 53.4141L181.711 61.2891L178.922 64.1719L171.164 56.2734L163.406 64.1719L160.617 61.2891Z" fill="white" fill-opacity="0.5"/>
+<g filter="url(#filter0_d)">
+<rect x="22" y="27" width="48" height="48" rx="3" fill="#E9429A"/>
+<rect x="22.5" y="27.5" width="47" height="47" rx="2.5" stroke="white"/>
+</g>
+<mask id="mask0" mask-type="alpha" maskUnits="userSpaceOnUse" x="22" y="27" width="48" height="48">
+<rect x="22" y="27" width="48" height="48" rx="3" fill="white"/>
+</mask>
+<g mask="url(#mask0)">
+<rect x="37" y="42" width="28" height="28" fill="white"/>
+<rect x="42" y="47" width="18" height="18" fill="#E9429A"/>
+</g>
+<rect x="27" y="42" width="6" height="28" fill="white"/>
+<rect x="37" y="32" width="28" height="6" fill="white"/>
+<g filter="url(#filter1_d)">
+<rect x="22" y="27" width="48" height="48" rx="3" fill="#E9429A"/>
+<rect x="23" y="28" width="46" height="46" rx="2" stroke="white" stroke-width="2"/>
+</g>
+<mask id="mask1" mask-type="alpha" maskUnits="userSpaceOnUse" x="22" y="27" width="48" height="48">
+<rect x="22" y="27" width="48" height="48" rx="3" fill="white"/>
+</mask>
+<g mask="url(#mask1)">
+<rect x="41" y="46" width="22" height="22" fill="white"/>
+<rect x="46" y="51" width="12" height="12" fill="#E9429A"/>
+</g>
+<rect x="29" y="46" width="7" height="22" fill="white"/>
+<rect x="41" y="34" width="22" height="7" fill="white"/>
+<path d="M160.586 64.2891L168.32 56.4141L160.633 48.5625L163.422 45.6797L171.133 53.5312L178.844 45.6797L181.633 48.5625L173.945 56.4141L181.68 64.2891L178.891 67.1719L171.133 59.2734L163.375 67.1719L160.586 64.2891Z" fill="#E9429A" fill-opacity="0.5"/>
+<defs>
+<filter id="filter0_d" x="18" y="23" width="60" height="60" filterUnits="userSpaceOnUse" color-interpolation-filters="sRGB">
+<feFlood flood-opacity="0" result="BackgroundImageFix"/>
+<feColorMatrix in="SourceAlpha" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 127 0"/>
+<feOffset dx="2" dy="2"/>
+<feGaussianBlur stdDeviation="3"/>
+<feColorMatrix type="matrix" values="0 0 0 0 0.572052 0 0 0 0 0.572052 0 0 0 0 0.572052 0 0 0 0.25 0"/>
+<feBlend mode="normal" in2="BackgroundImageFix" result="effect1_dropShadow"/>
+<feBlend mode="normal" in="SourceGraphic" in2="effect1_dropShadow" result="shape"/>
+</filter>
+<filter id="filter1_d" x="18" y="23" width="60" height="60" filterUnits="userSpaceOnUse" color-interpolation-filters="sRGB">
+<feFlood flood-opacity="0" result="BackgroundImageFix"/>
+<feColorMatrix in="SourceAlpha" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 127 0"/>
+<feOffset dx="2" dy="2"/>
+<feGaussianBlur stdDeviation="3"/>
+<feColorMatrix type="matrix" values="0 0 0 0 0.572052 0 0 0 0 0.572052 0 0 0 0 0.572052 0 0 0 0.25 0"/>
+<feBlend mode="normal" in2="BackgroundImageFix" result="effect1_dropShadow"/>
+<feBlend mode="normal" in="SourceGraphic" in2="effect1_dropShadow" result="shape"/>
+</filter>
+</defs>
+</svg>

docs/data.md

@@ -14,6 +14,14 @@ description: Data
 
 `cellxgene prepare` is not meant as a way to formally process or analyze your data. It's simply a utility for quickly wrangling your data into cellxgene-compatible format and computing a "vanilla" embedding so you can try out `cellxgene` and get a general sense of a dataset.
 
+#### How do I install `cellxgene prepare`?
+
+The `cellxgene prepare` command is an optional install that you can install alongside `cellxgene launch` by running
+
+```
+pip install cellxgene[prepare]
+```
+
 #### What input formats does it accept?
 
 Currently, we accept `h5ad` and `loom` files, as well as `10x` directories, and are hoping to accept more formats in the future.
@@ -38,8 +46,15 @@ As a quick example, let's construct a command to use `prepare` to take a raw exp
 
 We'll start off using the raw data from the pbmc3k dataset. This dataset is described [here](https://icb-scanpy.readthedocs-hosted.com/en/stable/api/scanpy.datasets.pbmc3k.html), and is available as part of the scanpy API. For this example, we'll assume this raw data is stored in a file called `pbmc3k-raw.h5ad`.
 
-Our `prepare` compose our command looks like this:  
-<img src="prepare-cmd-example.jpg" width="700" />
+Our `prepare` compose our command looks like this:
+
+```
+cellxgene prepare pbmc3k-raw.h5ad \
+	--run-qc \                                  # (A)
+	--recipe seurat \                           # (B)
+	--layout tsne --layout umap \               # (C)
+	--output pbmc3k-prepared.h5ad               # (D)
+```
 
 Let's look at what `prepare` is doing to our data, and how each step relates to the command above. You can see a walkthrough of what's going on under the hood for this example in [this notebook](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-prepare-example.ipynb).
 
@@ -52,10 +67,9 @@ Let's look at what `prepare` is doing to our data, and how each step relates to
 
 # Example datasets to use with cellxgene
 
-
-**To download and use these datasets, run:**  
-`curl -O [URL]`  
-`unzip [filename.zip]`  
+**To download and use these datasets, run:**
+`curl -O [URL]`
+`unzip [filename.zip]`
 `cellxgene launch [filename.h5ad] --open`
 
 ### Peripheral blood mononuclear cells

docs/faq.md

@@ -12,20 +12,22 @@ Currently, you can go straight into `cellxgene launch` with your own analyzed da
 
 If your data is in a different format, and/or you still need to perform dimensionality reduction and clustering, `cellxgene` can do that for you with the `prepare` command. `cellxgene prepare` runs `scanpy` under the hood and can read in any format that is currently supported by `scanpy` (including mtx, loom, and more listed [here](https://scanpy.readthedocs.io/en/latest/api/index.html#reading)).
 
+To add `cellxgene prepare` to your cellxgene installation run `pip install cellxgene[prepare]`.
+
 The output of `cellxgene prepare` is a h5ad file with your computed clusters and tsne/umap projections that can be used in `cellxgene launch`.
 
 #### I have a directory of 10X-Genomics data with _mtx_ files and I've never used _scanpy_, can I use _cellxgene_?
 
-Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `louvain` packages as described above. Just run
+Yep! This should only take a couple steps. We'll assume your data is in a folder called `data/` and you've successfully installed `cellxgene` with the `prepare` packages as described above. Just run
 
 ```
-cellxgene prepare data/ --output=data-processed.h5ad --layout=umap
+cellxgene prepare data/ --output=data-processed.h5ad --embedding=umap
 ```
 
 Depending on the size of the dataset, this may take some time. Once it's done, call
 
 ```
-cellxgene launch data-processed.h5ad --layout=umap --open
+cellxgene launch data-processed.h5ad --embedding=umap --open
 ```
 
 And your web browser should open with an interactive view of your data.
@@ -38,7 +40,7 @@ Currently this is not supported directly, but you should be able to do this your
 
 - `.obs` and `.var` annotations are use to extract metadata for filtering
 - `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
-- `.obsm` is used for layout. If an embedding has more than two components, the first two will be used for visualization.
+- `.obsm` is used for embedding(s). If an embedding has more than two components, the first two will be used for visualization.
 
 #### I have a BIG dataset - how can I make cellxgene run as fast as possible?
 
@@ -75,14 +77,6 @@ source ${ENV_NAME}/bin/activate
 pip install cellxgene
 ```
 
-#### In my _prepare_ command I received the following error `Warning: louvain module is not installed, no clusters will be calculated. To fix this please install cellxgene with the optional feature louvain enabled`
-
-Louvain clustering requires additional dependencies, so we don't include them by default. For now, you need to specify that you want these packages by using
-
-```
-pip install cellxgene[louvain]
-```
-
 #### I ran _prepare_ and I'm getting results that look unexpected
 
 You might want to try running one of the preprocessing recipes included with `scanpy` (read more about them [here](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes)). You can specify this with the `--recipe` option, such as
@@ -100,3 +94,37 @@ This may happen, especially as we work out bugs in our installation process! Ple
 #### I'm following the developer instructions and get an error about "missing files and directories” when trying to build the client
 
 This is likely because you do not have node and npm installed, we recommend using [nvm](https://github.com/creationix/nvm) if you're new to using these tools.
+
+# Data access
+
+#### Can I use a _s3:_ or _gs:_ URL with `cellxgene launch`?
+
+Yes. Support for S3 and GCS is not enabled by default. If you wish to directly access S3 or GFS, install one or both of the following packages using `pip`:
+
+- [s3fs](https://s3fs.readthedocs.io/en/latest/) for S3 support
+- [gcsfs](https://gcsfs.readthedocs.io/en/latest/) for GCS support
+
+For example:
+
+```
+pip install s3fs
+cellxgene launch s3://mybucket.s3-us-west-2.amazonaws.com/mydata.h5ad
+```
+
+#### What does the command line option `--backed` do?
+
+The `--backed` option instructs `cellxgene launch` to read the H5AD file in "backed" mode (for more information, see the
+[anndata.read_h5ad() documentation](https://anndata.readthedocs.io/en/latest/anndata.read_h5ad.html#anndata.read_h5ad)).
+
+By default, cellxgene will read the entire H5AD will be into memory at startup, improving application speed and performance.
+Very large datasets may not fit in memory. The "--backed" mode will read the file incrementally, reducing memory
+use, and for large files, improving startup speed. _However_, this option will also significantly slow
+down access to gene expression histograms, and may render differential expression calculations too slow
+to use (see `--disable-diffexp` for an option to disable this feature).
+
+#### What does the command line option `--disable-diffexp` do?
+
+The `--disable-diffexp` option will disable and hide the `Compute Differential Expression` feature.
+For large datasets, or datasets loaded with the `--backed` option, computing differential expression may
+be extremely slow or use excessive reources on the host computer (eg, memory thrasing).
+Disabling the feature will ensure that the end-user does not accidentally initiate this computation.

docs/getting-started.md

@@ -21,6 +21,12 @@ If you want an example dataset download [this file](https://github.com/chanzucke
 cellxgene launch pbmc3k.h5ad --open
 ```
 
+You can also directly specify URLs as a data source, and the data will be downloaded during launch
+
+```
+cellxgene launch https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
+```
+
 On Mac OS and Ubuntu, you should see your web browser open with the following
 
 <img width="450" src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-opening-screenshot.png" pad="50px">
@@ -29,7 +35,7 @@ On Mac OS and Ubuntu, you should see your web browser open with the following
 
 There are several options available, such as:
 
-- `--layout` to specify the layout as `tsne`, `umap`, `diffmap`, `phate`, `draw_graph_fa`, or `draw_graph_fr`
+- `--embedding` to restrict available emdeddings in the UI, eg, `tsne`, `umap`, `diffmap`, `phate`, `draw_graph_fa`, or `draw_graph_fr`
 - `--title` to show a title on the explorer
 - `--open` to automatically open the web browser after launching (OS X only)
 
@@ -39,7 +45,7 @@ To see all options call
 cellxgene launch --help
 ```
 
-There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](##data-formatting) for more info on `prepare`).
+There is an additional subcommand called `cellxgene prepare` that takes an existing dataset in one of several formats and applies minimal preprocessing and reformatting so that `launch` can use it (see [the next section](#data-formatting) for more info on `prepare`).
 
 ## data formatting
 
@@ -49,25 +55,37 @@ The `launch` command assumes that the data is stored in the `.h5ad` format from
 
 - an `obs` field has a unique identifier for every cell (you can specify which field to use with the `--obs-names` option, by default it will use the value of `data.obs_names`)
 - a `var` field has a unique identifier for every gene (you can specify which field to use with the `--var-names` option, by default it will use the value of `data.var_names`)
-- an `obsm` field contains the two-dimensional coordinates for the layout that you want to render (e.g. `X_umap` for the `umap` layout)
+- an `obsm` field contains the two-dimensional coordinates for the embedding that you want to render (e.g. `X_umap` for the `umap` embedding)
 - any additional `obs` fields will be rendered as per-cell continuous or categorical metadata by the app (e.g. `louvain` cluster assignments)
 
 ### prepare
 
 The `prepare` command is included to help you format your data. It uses `scanpy` under the hood. This is especially useful if you are starting with raw unanalyzed data and are unfamiliar with `scanpy`.
 
+To install `cellxgene prepare` alongside `cellxgene launch`
+
+```
+pip install cellxgene[prepare]
+```
+
+If the aforementioned optional package installation fails, you can also install these packages directly:
+
+```
+pip install scanpy>=1.3.7 python-igraph louvain>=0.6
+```
+
 To prepare from an existing `.h5ad` file use
 
 ```
 cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad
 ```
 
-This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` layouts and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
+This will load the input data, perform PCA and nearest neighbor calculations, compute `umap` and `tsne` embeddings and `louvain` cluster assignments, and save the results in a new file called `dataset-processed.h5ad` that can be loaded using `cellxgene launch`. Data can be loaded from several formats, including `.h5ad` `.loom` and a `10-Genomics-formatted` `mtx` directory. Several options are available, including running one of the preprocessing `recipes` included with `scanpy`, which include steps like cell filtering and gene selection. To learn more about the `recipes` please see the `scanpy` [documentation](https://scanpy.readthedocs.io/en/latest/api/index.html#recipes).
 
-Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the layout for `umap`, using a call like this
+Depending on the options chosen, `prepare` can take a long time to run (a few minutes for datasets with 10-100k cells, up to an hour or more for datasets with >100k cells). If you want `prepare` to run faster we recommend using the `sparse` option and only computing the embedding for `umap`, using a call like this
 
 ```
-cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --layout=umap --sparse
+cellxgene prepare dataset.h5ad --output=dataset-processed.h5ad --embedding=umap --sparse
 ```
 
 To see all options call
@@ -76,18 +94,6 @@ To see all options call
 cellxgene prepare --help
 ```
 
-**Note**: `cellxgene prepare` will only perform `louvain` clustering if you have the `python-igraph` and `louvain` packages installed. To make sure they are installed alongside `cellxgene` use
-
-```
-pip install cellxgene[louvain]
-```
-
-If the aforementioned optional package installation fails, you can also install these packages directly:
-
-```
-pip install python-igraph louvain>=0.6
-```
-
 ## conda and virtual environments
 
 If you use conda and want to create a conda environment for `cellxgene` you can use the following commands

docs/index.md

@@ -1,5 +1,8 @@
-_cellxgene_ is an interactive data explorer for single-cell transcriptomics data. Whether you need to visualize one thousand cells or one million, _cellxgene_ helps you gain insight into your single-cell data.
+<img src="cellxgene-logo.svg" width="300">
 
+_an interactive explorer for single-cell transcriptomics data_
+
+Whether you need to visualize one thousand cells or one million, cellxgene helps you gain insight into your single-cell data.
 ## features
 
 #### flexible selections, coloring, and differential expression of your selected sets of cells
@@ -10,7 +13,7 @@ _cellxgene_ is an interactive data explorer for single-cell transcriptomics data
 
 ## quick start
 
-To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-environment-for-cellxgene)
+To install cellxgene you need Python 3.6+. We recommend [installing cellxgene into a conda or virtual environment.](/faq.html#how-do-i-create-a-python-environment-for-cellxgene)
 
 Install the package.
 ``` bash
@@ -23,14 +26,14 @@ Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
 curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
 ```
 
-Launch _cellxgene_
+Launch cellxgene
 ``` bash
 cellxgene launch pbmc3k.h5ad --open
 ```
 
-To explore more datasets already formatted for _cellxgene_, see [Data](data) or
+To explore more datasets already formatted for cellxgene, see [Data](data) or
 visit [Getting Started](getting-started) to learn more about formatting your own
-data for _cellxgene_.
+data for cellxgene.
 
 ## getting help
 

experiments/heroku/Dockerfile

@@ -0,0 +1,7 @@
+FROM python:3.7
+
+WORKDIR /usr/src/app
+
+RUN pip3 install cellxgene
+
+expose 5005

experiments/heroku/README.md

@@ -0,0 +1,58 @@
+# cellxgene cloud deployment with Heroku
+
+## Quickstart
+
+Clicking on the following button will forward you to Heroku to begin the deployment process:
+
+<a href="https://heroku.com/deploy?template=https://github.com/chanzuckerberg/cellxgene/tree/heroku">
+ <img src="https://www.herokucdn.com/deploy/button.svg" alt="Deploy">
+</a>
+
+If not already logged in to Heroku, there you will be prompted to log in or sign up for an account.
+
+Once logged in you will be sent to the setup page. Here you can set some of the basic settings for the app:
+
+#### Default settings
+
+- `App name`: the unique name for your deployment
+- This will also serve as the default URL (e.g. https://cellxgene.herokapp.com/)
+- `App owner`: Who will own this app. Either you personally or an organization/team
+- `Region`: Location of the server where the app will be deployed (EU or US)
+
+#### Configuration
+
+- `DATASET`: A _publicly_ accessible URL pointing to a .h5ad file to view
+- This defaults to pbm3k.h5ad
+
+After filling out the settings and pressing the `Deploy app` button Heroku will begin building your deployment. This process will take a few minutes, but once completed you will have a personal free hosted version of cellxgene!
+
+## What is Heroku?
+
+Heroku is a quick and easy way to host applications on the cloud.
+
+A Heroku deployment of cellxgene means that the app is not running on your local machine. Instead, the app is installed, configured, and ran on the Heroku servers (read: cloud).
+
+On Heroku's servers, applications run on a [dyno](https://www.heroku.com/dynos) which are Heroku's implementation and abstraction of containers.
+
+Heroku is one of many options available for hosting instances of cellxgene on the web.
+Some other options include: Amazon Web Services, Google Cloud Platform, Digital Ocean, and Microsoft Azure.
+
+## Why use Heroku to deploy cellxgene?
+
+What Heroku enables is a quick, non-technical method of setting up a cellxgene instance. No command line knowledge needed. This also allows machines to access the instance via the internet, so sharing a visualized dataset is as simple as sharing a link.
+
+Because cellxgene currently heavily relies on its Python backend for providing the viewer with the necessary data and tooling, it is currently not possible to host cellxgene as a static webpage.
+
+This is a good option if you want to quickly deploy an instance of cellxgene to the web. Heroku deployments are free for small datasets up to around 250MBs in size. See below regarding larger datasets.
+
+## When should I not deploy with Heroku?
+
+- The default free dyno offered by Heroku is limited in memory to 512 MBs
+  - The amount of memory needed for the dyno is roughly the same size as the h5ad file
+  - Heroku offers tiered paid dynos. More can be found [here](https://www.heroku.com/pricing)
+  - Note that this can get _very_ expensive for larger datasets (\$25+ a month)
+- On the free dyno, after 30 minutes of inactivity, Heroku will put your app into a hibernation mode. On the next access, Heroku will need time to boot the dyno back online.
+- Having multiple simultaneous users requires more memory. This means that the free container size is easily overwhelmed by multiple users, even with small datasets; this can be addressed by purchasing a larger container size
+- For this facilitated Heroku deployment to work, your dataset must be hosted on a publicly accessible URL
+- By default, Heroku publically shares your instance to anyone with the URL.
+  - There are many ways of securing your instance. One quick and simple way is by installing [wwwhisper](https://elements.heroku.com/addons/wwwhisper), a Heroku addon

heroku.yml

@@ -0,0 +1,5 @@
+build:
+  docker:
+    web: experiments/heroku/Dockerfile
+run:
+  web: curl -o dataset.h5ad $DATASET && cellxgene launch dataset.h5ad --host 0.0.0.0 --port $PORT

server/__init__.py

@@ -1 +1 @@
-__version__ = "0.11.0"
+__version__ = "0.12.0"

server/app/app.py

@@ -33,6 +33,6 @@ class Server:
         self.app.register_blueprint(resources.blueprint)
         self.app.add_url_rule("/", endpoint="index")
 
-    def attach_data(self, data, title="Demo"):
-        self.app.config.update(DATASET_TITLE=title)
+    def attach_data(self, data, title="Demo", about=""):
+        self.app.config.update(DATASET_TITLE=title, ABOUT_DATASET=about)
         self.app.data = data

server/app/driver/driver.py

@@ -29,7 +29,9 @@ class CXGDriver(metaclass=ABCMeta):
         return {
             "layout": None,
             "max_category_items": None,
-            "diffexp_lfc_cutoff": None
+            "diffexp_lfc_cutoff": None,
+            "disable_diffexp": False,
+            "diffexp_may_be_slow": False
         }
 
     @property
@@ -46,7 +48,14 @@ class CXGDriver(metaclass=ABCMeta):
         return features
 
     @abstractmethod
-    def _load_data(self, data):
+    def get_schema(self):
+        """
+        Return current schema
+        """
+        pass
+
+    @abstractmethod
+    def _load_data(self, data_locator):
         pass
 
     @abstractmethod
@@ -59,6 +68,13 @@ class CXGDriver(metaclass=ABCMeta):
         """
         pass
 
+    @abstractmethod
+    def annotation_put_fbs(self, axis, fbs):
+        """
+        Put/save FBS as user-defined labels
+        """
+        pass
+
     @abstractmethod
     def data_frame_to_fbs_matrix(self, filter, axis):
         pass

server/app/rest_api/rest.py

@@ -1,5 +1,6 @@
 from http import HTTPStatus
 import warnings
+from os.path import basename
 
 from flask import Blueprint, current_app, jsonify, make_response, request
 from flask_restful import Api, Resource
@@ -16,6 +17,7 @@ from server.app.util.errors import (
     InteractiveError,
     JSONEncodingValueError,
     PrepareError,
+    DisabledFeatureError,
 )
 
 """
@@ -29,7 +31,7 @@ Sort order for routes
 class SchemaAPI(Resource):
     def get(self):
         return make_response(
-            jsonify({"schema": current_app.data.schema}), HTTPStatus.OK
+            jsonify({"schema": current_app.data.get_schema()}), HTTPStatus.OK
         )
 
 
@@ -63,8 +65,13 @@ class ConfigAPI(Resource):
                     "engine": f"cellxgene Scanpy engine version ",
                     "dataset": current_app.config["DATASET_TITLE"],
                 },
+                "links": {
+                    "about-dataset": current_app.config["ABOUT_DATASET"]
+                },
                 "parameters": {
-                    "max-category-items": current_app.data.config["max_category_items"]
+                    "max-category-items": current_app.data.config["max_category_items"],
+                    "disable-diffexp": current_app.data.config["disable_diffexp"],
+                    "diffexp-may-be-slow": current_app.data.config["diffexp_may_be_slow"]
                 },
                 "library_versions": {
                     "cellxgene": cellxgene_version,
@@ -72,6 +79,11 @@ class ConfigAPI(Resource):
                 }
             }
         }
+
+        label_file = current_app.data.config["label_file"]
+        if label_file:
+            config["config"]["parameters"]["label_file"] = basename(label_file)
+
         return make_response(jsonify(config), HTTPStatus.OK)
 
 
@@ -93,6 +105,18 @@ class AnnotationsObsAPI(Resource):
         except ValueError as e:
             return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
 
+    def put(self):
+        try:
+            fbs = request.get_data()
+            res = current_app.data.annotation_put_fbs("obs", fbs)
+            return make_response(
+                res, HTTPStatus.OK, {"Content-Type": "application/json"}
+            )
+        except (ValueError, DisabledFeatureError, KeyError) as e:
+            return make_response(str(e), HTTPStatus.BAD_REQUEST)
+        except Exception as e:
+            return make_response(str(e), HTTPStatus.INTERNAL_SERVER_ERROR)
+
 
 class AnnotationsVarAPI(Resource):
     def get(self):

server/app/scanpy_engine/diffexp.py

@@ -62,13 +62,12 @@ def diffexp_ttest(adata, maskA, maskB, top_n=8, diffexp_lfc_cutoff=0.01):
     :param diffexp_lfc_cutoff: minimum
     :return:  for top N genes, [ varindex, logfoldchange, pval, pval_adj ]
     """
-
     if top_n > adata.n_obs:
         top_n = adata.n_obs
 
     # mean, variance, N - calculate for both selections
-    meanA, vA, nA = _mean_var_n(adata._X[maskA])
-    meanB, vB, nB = _mean_var_n(adata._X[maskB])
+    meanA, vA, nA = _mean_var_n(adata.X[maskA, :])
+    meanB, vB, nB = _mean_var_n(adata.X[maskB, :])
 
     # variance / N
     vnA = vA / min(nA, nB)  # overestimate variance, would normally be nA
@@ -87,7 +86,7 @@ def diffexp_ttest(adata, maskA, maskB, top_n=8, diffexp_lfc_cutoff=0.01):
 
     # p-value
     pvals = stats.t.sf(np.abs(tscores), dof) * 2
-    pvals_adj = pvals * adata._X.shape[1]
+    pvals_adj = pvals * adata.X.shape[1]
     pvals_adj[pvals_adj > 1] = 1  # cap adjusted p-value at 1
 
     # logfoldchanges: log2(meanA / meanB)

server/app/scanpy_engine/labels.py

@@ -0,0 +1,49 @@
+"""
+Helpers for user annotations / label_file parameter
+"""
+from os.path import exists, splitext, getsize
+from os import remove, rename
+import pandas as pd
+
+
+def read_labels(fname):
+    if exists(fname) and getsize(fname) > 0:
+        return pd.read_csv(fname, dtype='category', index_col=0)
+    else:
+        return pd.DataFrame()
+
+
+def write_labels(fname, df):
+    rotate_fname(fname)
+    if not df.empty:
+        df.to_csv(fname)
+    else:
+        open(fname, 'a').close()
+
+
+def rotate_fname(fname):
+    """
+    save N backups of file.
+        fname -> fname-0
+        fname-0 -> fname->1
+        ...
+        fname-(N-1) -> fname-N
+    """
+
+    def rotate(src, dst):
+        if exists(src):
+            if exists(dst):
+                remove(dst)
+            rename(src, dst)
+
+    rotation_size = 9  # rotation size
+    name, ext = splitext(fname)
+
+    # rotate existing files
+    for i in range(rotation_size - 1, 0, -1):
+        src = f"{name}-{i}{ext}"
+        tgt = f"{name}-{i+1}{ext}"
+        rotate(src, tgt)
+
+    tgt = f"{name}-1{ext}"
+    rotate(fname, tgt)

server/app/scanpy_engine/matrix_proxy.py

@@ -0,0 +1,42 @@
+
+from server.app.util.matrix_proxy import MatrixProxyView, ArrayProxyView
+
+"""
+AnnData/h5py are inconsistent in the API supported by various types of
+X matrices.  Sometimes you get a fully ndarray, sometims a Scipy sparse
+matrix, sometimes h5py proxies with a subset of our needed interfaces.
+
+This glue code paves over all of that, providing the core set of methods
+that the cellxgene ScanPy driver assumes are in existance.  Put another
+way, all of the non-portable assumptions are here.
+"""
+
+
+class ArrayProxyView_anndata_h5py(ArrayProxyView):
+    """
+    override to handle sparse getitem semantics, which differ
+    from numpy.
+    """
+    def toarray(self):
+        """ sadly, sparse indexing doesn't drop dimensions like numpy! """
+        arr = self.m[self._index[0], self._index[1]]
+        if self._vdim == 0:
+            arr = arr.transpose()
+        return arr.toarray()[0]
+
+
+class MatrixProxy_anndata_h5py(MatrixProxyView):
+    """
+    AnnData sparse array stored in H5AD, or proxies for backed data.
+    None of these handle indexing very well, so we plop a proxy on top.
+    """
+    @classmethod
+    def __supports__(cls):
+        return ("anndata.h5py.h5sparse.SparseDataset",
+                "anndata.h5py.h5sparse.backed_csc_matrix",
+                "anndata.h5py.h5sparse.backed_csr_matrix",
+                "h5py._hl.dataset.Dataset")
+
+    @classmethod
+    def create_array(cls, *args, **kwargs):
+        return ArrayProxyView_anndata_h5py(*args, **kwargs)

server/app/scanpy_engine/scanpy_engine.py

@@ -1,4 +1,6 @@
 import warnings
+import copy
+import threading
 
 import numpy as np
 import pandas
@@ -13,24 +15,27 @@ from server.app.util.errors import (
     JSONEncodingValueError,
     PrepareError,
     ScanpyFileError,
+    DisabledFeatureError,
 )
 from server.app.util.utils import jsonify_scanpy, requires_data
 from server.app.scanpy_engine.diffexp import diffexp_ttest
-from server.app.util.fbs.matrix import encode_matrix_fbs
+from server.app.util.fbs.matrix import encode_matrix_fbs, decode_matrix_fbs
+from server.app.scanpy_engine.labels import read_labels, write_labels
+import server.app.scanpy_engine.matrix_proxy  # noqa: F401
+from server.app.util.matrix_proxy import MatrixProxy
 
-"""
-Sort order for methods
-1. Initialize
-2. Helper
-3. Filter
-4. Data & Metadata
-5. Computation
-"""
+
+def has_method(o, name):
+    """ return True if `o` has callable method `name` """
+    op = getattr(o, name, None)
+    return op is not None and callable(op)
 
 
 class ScanpyEngine(CXGDriver):
     def __init__(self, data=None, args={}):
         super().__init__(data, args)
+        # lock used to protect label file write ops
+        self.label_lock = threading.Lock()
         if self.data:
             self._validate_and_initialize()
 
@@ -47,6 +52,10 @@ class ScanpyEngine(CXGDriver):
             "obs_names": None,
             "var_names": None,
             "diffexp_lfc_cutoff": 0.01,
+            "label_file": None,
+            "backed": False,
+            "disable_diffexp": False,
+            "diffexp_may_be_slow": False
         }
 
     @staticmethod
@@ -74,6 +83,8 @@ class ScanpyEngine(CXGDriver):
         index column name to the front-end via the obs_names and var_names config
         (which is incorporated into the schema).
         """
+        self.original_obs_index = self.data.obs.index
+
         for (ax_name, config_name) in ((Axis.OBS, "obs_names"), (Axis.VAR, "var_names")):
             name = self.config[config_name]
             df_axis = getattr(self.data, str(ax_name))
@@ -125,6 +136,29 @@ class ScanpyEngine(CXGDriver):
                 return True
         return False
 
+    @staticmethod
+    def _get_col_type(col):
+        dtype = col.dtype
+        data_kind = dtype.kind
+        schema = {}
+
+        if ScanpyEngine._can_cast_to_float32(col):
+            schema["type"] = "float32"
+        elif ScanpyEngine._can_cast_to_int32(col):
+            schema["type"] = "int32"
+        elif dtype == np.bool_:
+            schema["type"] = "boolean"
+        elif data_kind == "O" and dtype == "object":
+            schema["type"] = "string"
+        elif data_kind == "O" and dtype == "category":
+            schema["type"] = "categorical"
+            schema["categories"] = dtype.categories.tolist()
+        else:
+            raise TypeError(
+                f"Annotations of type {dtype} are unsupported by cellxgene."
+            )
+        return schema
+
     @requires_data
     def _create_schema(self):
         self.schema = {
@@ -148,25 +182,8 @@ class ScanpyEngine(CXGDriver):
         for ax in Axis:
             curr_axis = getattr(self.data, str(ax))
             for ann in curr_axis:
-                ann_schema = {"name": ann}
-                dtype = curr_axis[ann].dtype
-                data_kind = dtype.kind
-
-                if self._can_cast_to_float32(curr_axis[ann]):
-                    ann_schema["type"] = "float32"
-                elif self._can_cast_to_int32(curr_axis[ann]):
-                    ann_schema["type"] = "int32"
-                elif dtype == np.bool_:
-                    ann_schema["type"] = "boolean"
-                elif data_kind == "O" and dtype == "object":
-                    ann_schema["type"] = "string"
-                elif data_kind == "O" and dtype == "category":
-                    ann_schema["type"] = "categorical"
-                    ann_schema["categories"] = curr_axis[ann].dtype.categories.tolist()
-                else:
-                    raise TypeError(
-                        f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene."
-                    )
+                ann_schema = {"name": ann, "writable": False}
+                ann_schema.update(self._get_col_type(curr_axis[ann]))
                 self.schema["annotations"][ax]["columns"].append(ann_schema)
 
         for layout in self.config['layout']:
@@ -177,11 +194,34 @@ class ScanpyEngine(CXGDriver):
             }
             self.schema["layout"]["obs"].append(layout_schema)
 
-    def _load_data(self, data):
+    @requires_data
+    def get_schema(self):
+        schema = self.schema  # base schema
+        # add label obs annotations as needed
+        if self.labels is not None:
+            schema = copy.deepcopy(schema)
+            for col in self.labels.columns:
+                col_schema = {
+                    "name": col,
+                    "writable": True,
+                }
+                col_schema.update(self._get_col_type(self.labels[col]))
+                schema["annotations"]["obs"]["columns"].append(col_schema)
+        return schema
+
+    def _load_data(self, data_locator):
         # as of AnnData 0.6.19, backed mode performs initial load fast, but at the
         # cost of significantly slower access to X data.
         try:
-            self.data = anndata.read_h5ad(data)
+            # there is no guarantee data_locator indicates a local file.  The AnnData
+            # API will only consume local file objects.  If we get a non-local object,
+            # make a copy in tmp, and delete it after we load into memory.
+            with data_locator.local_handle() as lh:
+                # as of AnnData 0.6.19, backed mode performs initial load fast, but at the
+                # cost of significantly slower access to X data.
+                backed = 'r' if self.config['backed'] else None
+                self.data = anndata.read_h5ad(lh, backed=backed)
+
         except ValueError:
             raise ScanpyFileError(
                 "File must be in the .h5ad format. Please read "
@@ -191,14 +231,24 @@ class ScanpyEngine(CXGDriver):
                 "information."
             )
         except MemoryError:
-            raise ScanpyFileError("Error while loading file: out of memory, file is too large"
-                                  " for memory available")
+            raise ScanpyFileError("Out of memory - file is too large for available memory.")
         except Exception as e:
             raise ScanpyFileError(
-                f"Error while loading file: {e}, File must be in the .h5ad format, please check "
-                f"that your input and try again."
+                f"{e} - file not found or is inaccessible.  File must be an .h5ad object.  "
+                f"Please check your input and try again."
             )
 
+        if self.config["label_file"]:
+            try:
+                self.labels = read_labels(self.config["label_file"])
+            except Exception as e:
+                raise ScanpyFileError(
+                    f"Error while loading label file: {e}, File must be in the .csv format, please check "
+                    f"your input and try again."
+                )
+        else:
+            self.labels = None
+
     @requires_data
     def _validate_and_initialize(self):
         # var and obs column names must be unique
@@ -210,8 +260,14 @@ class ScanpyEngine(CXGDriver):
         self.cell_count = self.data.shape[0]
         self.gene_count = self.data.shape[1]
         self._default_and_validate_layouts()
+        self._validate_label_data()
         self._create_schema()
 
+        # heuristic
+        n_values = self.data.shape[0] * self.data.shape[1]
+        if (n_values > 1e8 and self.config['backed'] is True) or (n_values > 5e8):
+            self.config.update({"diffexp_may_be_slow": True})
+
     @requires_data
     def _default_and_validate_layouts(self):
         """ function:
@@ -293,6 +349,37 @@ class ScanpyEngine(CXGDriver):
                             f"annotations with more than 500 categories in the UI"
                         )
 
+    @requires_data
+    def _validate_label_data(self):
+        """
+        labels is None if disabled, empty if enabled by no data
+        """
+        if self.labels is None or self.labels.empty:
+            return
+
+        # all lables must have a name, which must be unique and not used in obs column names
+        if not self.labels.columns.is_unique:
+            raise KeyError(f"All column names specified in {self.config['label_file']} must be unique.")
+
+        # the label index must be unique, and must have same values the anndata obs index
+        if not self.labels.index.is_unique:
+            raise KeyError(f"All row index values specified in the label file "
+                           f"`{self.config['label_file']}` must be unique.")
+
+        if not self.labels.index.equals(self.original_obs_index):
+            raise KeyError("Label file row index does not match H5AD file index. "
+                           "Please ensure that column zero (0) in the label file contain the same "
+                           "index values as the H5AD file.")
+
+        duplicate_columns = list(set(self.labels.columns) & set(self.data.obs.columns))
+        if len(duplicate_columns) > 0:
+            raise KeyError(f"Labels file may not contain column names which overlap "
+                           f"with h5ad obs columns {duplicate_columns}")
+
+        # labels must have same count as obs annotations
+        if self.labels.shape[0] != self.data.obs.shape[0]:
+            raise ValueError("Labels file must have same number of rows as h5ad file.")
+
     @staticmethod
     def _annotation_filter_to_mask(filter, d_axis, count):
         mask = np.ones((count,), dtype=bool)
@@ -360,28 +447,42 @@ class ScanpyEngine(CXGDriver):
     @requires_data
     def annotation_to_fbs_matrix(self, axis, fields=None):
         if axis == Axis.OBS:
-            df = self.data.obs
+            if self.labels is not None and not self.labels.empty:
+                df = self.data.obs.join(self.labels, self.config['obs_names'])
+            else:
+                df = self.data.obs
         else:
             df = self.data.var
         if fields is not None and len(fields) > 0:
             df = df[fields]
         return encode_matrix_fbs(df, col_idx=df.columns)
 
-    @staticmethod
-    def slice_columns(X, var_mask):
-        """
-        Slice columns from the matrix X, as specified by the mask
-        Semantically equivalent to X[:, var_mask], but handles sparse
-        matrices in a more performant manner.
-        """
-        if var_mask is None:    # noop
-            return X
-        if sparse.issparse(X):  # use tuned getcol/hstack for performance
-            indices = np.nonzero(var_mask)[0]
-            cols = [X.getcol(i) for i in indices]
-            return sparse.hstack(cols, format="csc")
-        else:   # else, just use standard slicing, which is fine for dense arrays
-            return X[:, var_mask]
+    @requires_data
+    def annotation_put_fbs(self, axis, fbs):
+        fname = self.config["label_file"]
+        if not fname or self.labels is None:
+            raise DisabledFeatureError("Writable annotations are not enabled")
+
+        if axis != Axis.OBS:
+            raise ValueError("Only OBS dimension access is supported")
+
+        new_label_df = decode_matrix_fbs(fbs)
+        new_label_df.index = self.original_obs_index
+        self._validate_label_data()  # paranoia
+
+        # if any of the new column labels overlap with our existing labels, raise error
+        duplicate_columns = list(set(new_label_df.columns) & set(self.data.obs.columns))
+        if not new_label_df.columns.is_unique or len(duplicate_columns) > 0:
+            raise KeyError(f"Labels file may not contain column names which overlap "
+                           f"with h5ad obs columns {duplicate_columns}")
+
+        # update our internal state and save it.  Multi-threading often enabled,
+        # so treat this as a critical section critical section.
+        with self.label_lock:
+            self.labels = new_label_df
+            write_labels(fname, self.labels)
+
+        return jsonify_scanpy({"status": "OK"})
 
     @requires_data
     def data_frame_to_fbs_matrix(self, filter, axis):
@@ -405,7 +506,8 @@ class ScanpyEngine(CXGDriver):
             raise FilterError("filtering on obs unsupported")
 
         # Currently only handles VAR dimension
-        X = self.slice_columns(self.data._X, var_selector)
+        X = MatrixProxy.create(self.data.X if var_selector is None
+                               else self.data.X[:, var_selector])
         return encode_matrix_fbs(X, col_idx=np.nonzero(var_selector)[0], row_idx=None)
 
     @requires_data

server/app/util/data_locator.py

@@ -0,0 +1,84 @@
+import os
+import tempfile
+import fsspec
+
+
+class DataLocator():
+    """
+    DataLocator is a simple wrapper around fsspec functionality, and provides a
+    set of functions to encapsulate a data location (URI or path), interogate
+    metadata about the object at that location (size, existance, etc) and
+    access the underlying data.
+
+    https://filesystem-spec.readthedocs.io/en/latest/index.html
+
+    Example:
+        dl = DataLocator("/tmp/foo.h5ad")
+        if dl.exists():
+            print(dl.size())
+            with dl.open() as f:
+                thecontents = f.read()
+
+    DataLocator will accept a URI or native path.  Error handling is as defined
+    in fsspec.
+
+    """
+
+    def __init__(self, uri_or_path):
+        self.uri_or_path = uri_or_path
+        self.protocol, self.path = DataLocator._get_protocol_and_path(uri_or_path)
+        # work-around for LocalFileSystem not treating file: and None as the same scheme/protocol
+        self.cname = self.path if self.protocol == 'file' else self.uri_or_path
+        # will throw RuntimeError if the protocol is unsupported
+        self.fs = fsspec.filesystem(self.protocol)
+
+    @staticmethod
+    def _get_protocol_and_path(uri_or_path):
+        if "://" in uri_or_path:
+            protocol, path = uri_or_path.split("://", 1)
+            # windows!!!  Ignore single letter drive identifiers,
+            # eg, G:\foo.txt
+            if len(protocol) > 1:
+                return protocol, path
+        return None, uri_or_path
+
+    def exists(self):
+        return self.fs.exists(self.cname)
+
+    def size(self):
+        return self.fs.size(self.cname)
+
+    def isfile(self):
+        return self.fs.isfile(self.cname)
+
+    def open(self, *args):
+        return self.fs.open(self.uri_or_path, *args)
+
+    def islocal(self):
+        return self.protocol is None or self.protocol == 'file'
+
+    def local_handle(self):
+        if self.islocal():
+            return LocalFilePath(self.path)
+
+        # if not local, create a tmp file system object to contain the data,
+        # and clean it up when done.
+        with self.open() as src, tempfile.NamedTemporaryFile(prefix="cellxgene_", delete=False) as tmp:
+            tmp.write(src.read())
+            tmp.close()
+            src.close()
+            tmp_path = tmp.name
+            return LocalFilePath(tmp_path, delete=True)
+
+
+class LocalFilePath():
+    def __init__(self, tmp_path, delete=False):
+        self.tmp_path = tmp_path
+        self.delete = delete
+
+    def __enter__(self):
+        return self.tmp_path
+
+    def __exit__(self, *args):
+        if self.delete:
+            os.unlink(self.tmp_path)

server/app/util/errors.py

@@ -59,3 +59,12 @@ class DriverError(Exception):
 
     def __init__(self, message):
         self.message = message
+
+
+class DisabledFeatureError(Exception):
+    """
+    Raised when an attempt to use a disabled feature occurs
+    """
+
+    def __init__(self, message):
+        self.message = message

server/app/util/fbs/matrix.py

@@ -2,10 +2,17 @@ import flatbuffers
 import numpy as np
 from scipy import sparse
 import pandas as pd
+import json
 
 import server.app.util.fbs.NetEncoding.Column as Column
 import server.app.util.fbs.NetEncoding.TypedArray as TypedArray
 import server.app.util.fbs.NetEncoding.Matrix as Matrix
+import server.app.util.fbs.NetEncoding.Int32Array as Int32Array
+import server.app.util.fbs.NetEncoding.Uint32Array as Uint32Array
+import server.app.util.fbs.NetEncoding.Float32Array as Float32Array
+import server.app.util.fbs.NetEncoding.Float64Array as Float64Array
+import server.app.util.fbs.NetEncoding.JSONEncodedArray as JSONEncodedArray
+from server.app.util.matrix_proxy import MatrixProxy
 
 
 # Placeholder until recent enhancements to flatbuffers Python
@@ -18,7 +25,7 @@ def CreateNumpyVector(builder, x):
     """CreateNumpyVector writes a numpy array into the buffer."""
 
     if not isinstance(x, np.ndarray):
-        raise TypeError("non-numpy-ndarray passed to CreateNumpyVector")
+        raise TypeError(f"non-numpy-ndarray passed to CreateNumpyVector ({type(x)}")
 
     if x.dtype.kind not in ['b', 'i', 'u', 'f']:
         raise TypeError("numpy-ndarray holds elements of unsupported datatype")
@@ -85,7 +92,7 @@ def serialize_typed_array(builder, source_array, encoding_info):
         as_json = arr.to_json(orient='records')
         arr = np.array(bytearray(as_json, 'utf-8'))
     else:
-        if sparse.issparse(arr):
+        if MatrixProxy.ismatrixproxy(arr) or sparse.issparse(arr):
             arr = arr.toarray()
         elif isinstance(arr, pd.Series):
             arr = arr.get_values()
@@ -93,8 +100,11 @@ def serialize_typed_array(builder, source_array, encoding_info):
             arr = arr.astype(as_type)
 
     # serialize the ndarray into a vector
-    if arr.ndim == 2 and arr.shape[0] == 1:
-        arr = arr[0]
+    if arr.ndim == 2:
+        if arr.shape[0] == 1:
+            arr = arr[0]
+        elif arr.shape[1] == 1:
+            arr = arr.T[0]
     vec = CreateNumpyVector(builder, arr)
 
     # serialize the typed array table
@@ -104,38 +114,42 @@ def serialize_typed_array(builder, source_array, encoding_info):
     return (array_type, array_value)
 
 
+column_encoding_type_map = {
+    # array protocol string:  ( array_type, as_type )
+    np.dtype(np.float64).str: (TypedArray.TypedArray.Float32Array, np.float32),
+    np.dtype(np.float32).str: (TypedArray.TypedArray.Float32Array, np.float32),
+    np.dtype(np.float16).str: (TypedArray.TypedArray.Float32Array, np.float32),
+
+    np.dtype(np.int8).str: (TypedArray.TypedArray.Int32Array, np.int32),
+    np.dtype(np.int16).str: (TypedArray.TypedArray.Int32Array, np.int32),
+    np.dtype(np.int32).str: (TypedArray.TypedArray.Int32Array, np.int32),
+    np.dtype(np.int64).str: (TypedArray.TypedArray.Int32Array, np.int32),
+
+    np.dtype(np.uint8).str: (TypedArray.TypedArray.Uint32Array, np.uint32),
+    np.dtype(np.uint16).str: (TypedArray.TypedArray.Uint32Array, np.uint32),
+    np.dtype(np.uint32).str: (TypedArray.TypedArray.Uint32Array, np.uint32),
+    np.dtype(np.uint64).str: (TypedArray.TypedArray.Uint32Array, np.uint32)
+}
+column_encoding_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
+
+
 def column_encoding(arr):
-    type_map = {
-        # dtype:  ( array_type, as_type )
-        np.float64: (TypedArray.TypedArray.Float32Array, np.float32),
-        np.float32: (TypedArray.TypedArray.Float32Array, np.float32),
-        np.float16: (TypedArray.TypedArray.Float32Array, np.float32),
+    return column_encoding_type_map.get(arr.dtype.str, column_encoding_default)
 
-        np.int8: (TypedArray.TypedArray.Int32Array, np.int32),
-        np.int16: (TypedArray.TypedArray.Int32Array, np.int32),
-        np.int32: (TypedArray.TypedArray.Int32Array, np.int32),
-        np.int64: (TypedArray.TypedArray.Int32Array, np.int32),
 
-        np.uint8: (TypedArray.TypedArray.Uint32Array, np.uint32),
-        np.uint16: (TypedArray.TypedArray.Uint32Array, np.uint32),
-        np.uint32: (TypedArray.TypedArray.Uint32Array, np.uint32),
-        np.uint64: (TypedArray.TypedArray.Uint32Array, np.uint32)
-    }
-    type_map_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
-    return type_map.get(arr.dtype.type, type_map_default)
+index_encoding_type_map = {
+    # array protocol string:  ( array_type, as_type )
+    np.dtype(np.int32).str: (TypedArray.TypedArray.Int32Array, np.int32),
+    np.dtype(np.int64).str: (TypedArray.TypedArray.Int32Array, np.int32),
+
+    np.dtype(np.uint32).str: (TypedArray.TypedArray.Uint32Array, np.uint32),
+    np.dtype(np.uint64).str: (TypedArray.TypedArray.Uint32Array, np.uint32)
+}
+index_encoding_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
 
 
 def index_encoding(arr):
-    type_map = {
-        # dtype:  ( array_type, as_type )
-        np.int32: (TypedArray.TypedArray.Int32Array, np.int32),
-        np.int64: (TypedArray.TypedArray.Int32Array, np.int32),
-
-        np.uint32: (TypedArray.TypedArray.Uint32Array, np.uint32),
-        np.uint64: (TypedArray.TypedArray.Uint32Array, np.uint32)
-    }
-    type_map_default = (TypedArray.TypedArray.JSONEncodedArray, 'json')
-    return type_map.get(arr.dtype.type, type_map_default)
+    return index_encoding_type_map.get(arr.dtype.str, index_encoding_default)
 
 
 def guess_at_mem_needed(matrix):
@@ -175,16 +189,11 @@ def encode_matrix_fbs(matrix, row_idx=None, col_idx=None):
     # estimate size needed, so we don't unnecessarily realloc.
     builder = flatbuffers.Builder(guess_at_mem_needed(matrix))
 
-    if isinstance(matrix, pd.DataFrame):
-        matrix_columns = reversed(tuple(matrix[name] for name in matrix))
-    else:
-        matrix_columns = reversed(tuple(c for c in matrix.T))
-
     columns = []
-    # for idx in reversed(np.arange(n_cols)):
-    for c in matrix_columns:
+    for cidx in range(n_cols - 1, -1, -1):
         # serialize the typed array
-        typed_arr = serialize_typed_array(builder, c, column_encoding)
+        col = matrix.iloc[:, cidx] if isinstance(matrix, pd.DataFrame) else matrix[:, cidx]
+        typed_arr = serialize_typed_array(builder, col, column_encoding)
 
         # serialize the Column union
         columns.append(serialize_column(builder, typed_arr))
@@ -205,3 +214,73 @@ def encode_matrix_fbs(matrix, row_idx=None, col_idx=None):
 
     builder.Finish(matrix)
     return builder.Output()
+
+
+def deserialize_typed_array(tarr):
+    type_map = {
+        TypedArray.TypedArray.NONE: None,
+        TypedArray.TypedArray.Uint32Array: Uint32Array.Uint32Array,
+        TypedArray.TypedArray.Int32Array: Int32Array.Int32Array,
+        TypedArray.TypedArray.Float32Array: Float32Array.Float32Array,
+        TypedArray.TypedArray.Float64Array: Float64Array.Float64Array,
+        TypedArray.TypedArray.JSONEncodedArray: JSONEncodedArray.JSONEncodedArray
+    }
+    (u_type, u) = tarr
+    if u_type is TypedArray.TypedArray.NONE:
+        return None
+
+    TarType = type_map.get(u_type, None)
+    if TarType is None:
+        raise TypeError(f"FBS contains unknown data type: {u_type}")
+
+    arr = TarType()
+    arr.Init(u.Bytes, u.Pos)
+    narr = arr.DataAsNumpy()
+    if u_type == TypedArray.TypedArray.JSONEncodedArray:
+        narr = json.loads(narr.tostring().decode('utf-8'))
+    return narr
+
+
+def decode_matrix_fbs(fbs):
+    """
+    Given an FBS-encoded Matrix, return a Pandas DataFrame the contains the data
+    and indices.
+    """
+    matrix = Matrix.Matrix.GetRootAsMatrix(fbs, 0)
+    n_rows = matrix.NRows()
+    n_cols = matrix.NCols()
+    if n_rows == 0 or n_cols == 0:
+        return pd.DataFrame()
+
+    if matrix.RowIndexType() is not TypedArray.TypedArray.NONE:
+        raise ValueError("row indexing not supported for FBS Matrix")
+
+    columns_length = matrix.ColumnsLength()
+
+    columns_index = deserialize_typed_array((matrix.ColIndexType(), matrix.ColIndex()))
+    if columns_index is None:
+        columns_index = range(0, n_cols)
+
+    # sanity checks
+    if len(columns_index) != n_cols or columns_length != n_cols:
+        raise ValueError("FBS column count does not match number of columns in underlying matrix")
+
+    columns_data = {}
+    columns_type = {}
+    for col_idx in range(0, columns_length):
+        col = matrix.Columns(col_idx)
+        tarr = (col.UType(), col.U())
+        data = deserialize_typed_array(tarr)
+        columns_data[columns_index[col_idx]] = data
+        if len(data) != n_rows:
+            raise ValueError("FBS column length does not match number of rows")
+        if col.UType() is TypedArray.TypedArray.JSONEncodedArray:
+            columns_type[columns_index[col_idx]] = "category"
+
+    df = pd.DataFrame.from_dict(data=columns_data).astype(columns_type, copy=False)
+
+    # more sanity checks
+    if not df.columns.is_unique or len(df.columns) != n_cols:
+        raise KeyError("FBS column indices are not unique")
+
+    return df

server/app/util/matrix_proxy.py

@@ -0,0 +1,425 @@
+import abc
+from itertools import zip_longest
+from copy import copy
+import numpy as np
+
+"""
+cellxgene deals with a variety of matrix data types, many of which do
+not support a consistent API.  This framework allows proxies to be created
+to pave over some of this.  Most significantly, AnnData.X does not guarantee
+that much of its API (eg. .X.T) will work.
+"""
+
+INT_TYPES = (int, np.integer)
+
+
+class _ArrayProxyBase(abc.ABC):
+    """
+    Private base class for array or matrix proxy.  This summarizes
+    the interface used by the rest of cellxgene.
+    """
+    @property
+    @abc.abstractmethod
+    def dtype(self):
+        raise NotImplementedError()
+
+    @property
+    @abc.abstractmethod
+    def ndim(self):
+        raise NotImplementedError()
+
+    @property
+    @abc.abstractmethod
+    def shape(self):
+        raise NotImplementedError()
+
+    @property
+    @abc.abstractmethod
+    def T(self):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def __iter__(self):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def __getitem__(self, args):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def toarray():
+        raise NotImplementedError()
+
+
+class MatrixProxy(_ArrayProxyBase):
+    """
+    Abstract class - all interfaces we need, plus a factory method
+    to create a proxy based upon actual matrix type.
+
+    This class primarily provides the factory method and related support.
+    All other functionality is delegated to subclasses.
+    """
+
+    """
+    Registry of types to proxy class, where values are:
+    * None: unsupported
+    * True: self-supported
+    * string: proxy class
+    Sub-classes automatically register.
+    """
+    base_proxy_registry = {
+        'pandas.core.frame.DataFrame': True,
+        'numpy.ndarray': True,
+        'scipy.sparse.csc.csc_matrix': True,
+        'scipy.sparse.csr.csr_matrix': True,
+    }
+    proxy_registry = None
+    last_cache_token = None
+
+    @staticmethod
+    def _register_subclasses(subclasses, registry):
+        for c in subclasses:
+            names = c.__supports__()
+            for name in names:
+                registry[name] = c
+            MatrixProxy._register_subclasses(c.__subclasses__(), registry)
+
+    @classmethod
+    def build_proxy_registry(cls):
+        if cls.proxy_registry and abc.get_cache_token() == cls.last_cache_token:
+            return
+
+        cls.last_cache_token = abc.get_cache_token()
+        registry = copy(cls.base_proxy_registry)
+        MatrixProxy._register_subclasses(cls.__subclasses__(), registry)
+        cls.proxy_registry = registry
+
+    @classmethod
+    def create(cls, matrix):
+        """
+        Factory - call with a matrix and it will create a proxy if needed.
+        If the type already supports the necessary API, it is just returned
+        directly.
+        """
+        cls.build_proxy_registry()
+        t = type(matrix)
+        fqtn = t.__module__ + '.' + t.__name__
+        proxy_cls = cls.proxy_registry.get(fqtn, None)
+        if proxy_cls is None:
+            raise Exception(f"Matrix format `{fqtn}` is unsupported by proxy.")
+        if proxy_cls is True:
+            return matrix
+        return proxy_cls(matrix)
+
+    def __init__(self, m):
+        self.m = m
+
+    @classmethod
+    @abc.abstractmethod
+    def __supports__(cls):
+        raise NotImplementedError()
+
+    @classmethod
+    def ismatrixproxy(cls, m):
+        return isinstance(m, _ArrayProxyBase)
+
+
+class MatrixProxyView(MatrixProxy):
+    """
+    2D matrix view to a 2D matrix
+    """
+    def __init__(self, arg1, shape=None, index=(),
+                 transposed=False, copy=False):
+        if not copy:
+            m = arg1
+            super().__init__(m)
+
+            if shape is None:
+                shape = m.shape
+            assert(len(shape) == 2)
+
+            index = tuple(
+                map(lambda s_i:
+                    slice(0, s_i[0], 1) if s_i[1] is None else s_i[1],
+                    zip_longest(shape, index))
+            )
+
+            self._shape = shape
+            self._index = index
+            self.transposed = transposed
+
+        else:  # copy mode
+            super().__init__(arg1.m)
+            self._shape = arg1._shape
+            self._index = arg1._index
+            self.transposed = arg1.transposed
+
+    @classmethod
+    def create_array(cls, *args, **kwargs):
+        """ override if you use a different 1D array proxy """
+        return ArrayProxyView(*args, **kwargs)
+
+    def copy(self):
+        """ override if you need additional behaviors """
+        return self.__class__(self, copy=True)
+
+    @classmethod
+    def __supports__(cls):
+        return ()
+
+    def _swap(self, x):
+        return (x[1], x[0]) if self.transposed else x
+
+    @property
+    def dtype(self):
+        return self.m.dtype
+
+    @property
+    def ndim(self):
+        return len(self._shape)
+
+    @property
+    def shape(self):
+        return self._swap(self._shape)
+
+    @property
+    def T(self):
+        m = self.copy()
+        m.transposed = not m.transposed
+        return m
+
+    def __iter__(self):
+        M = self._swap(self._shape)[0]
+        s = self._swap((self._index))[0]
+        start, stop, step = s.indices(M)
+        for d in range(start, stop, step):
+            yield self[d]
+
+    def __getitem__(self, args):
+        """
+        decompose into the indexing patterns we use, throw for the rest.
+        Subclassses implement specialized access.
+        """
+        row, col = self._swap(_unpack_index(args, self.shape))
+        M, N = self._shape
+        allM, allN = self.m.shape
+
+        if isinstance(row, INT_TYPES):
+            row += self._index[0].start
+        elif isinstance(row, slice):
+            row = _slice_slice(self._index[0], allM, row, M)
+        if isinstance(col, INT_TYPES):
+            col += self._index[1].start
+        elif isinstance(col, slice):
+            col = _slice_slice(self._index[1], allN, col, N)
+
+        if isinstance(row, INT_TYPES):
+            if isinstance(col, INT_TYPES):
+                return self._getitem_intXint(row, col)
+            elif isinstance(col, slice):
+                return self._getitem_intXslice(row, col)
+        elif isinstance(row, slice):
+            if isinstance(col, INT_TYPES):
+                return self._getitem_sliceXint(row, col)
+            elif isinstance(col, slice):
+                return self._getitem_sliceXslice(row, col)
+
+        raise IndexError("unsupported column index types")
+
+    """
+    These getitem signatures are separate so that they may be
+    overridden by subclasses as necessary.  We don't do much
+    with them by default other than the obvious sub-slicing.
+
+    NOTE: these follow the numpy rules for dimensionality reduction
+    when an integer index is specified.
+    """
+    def _getitem_intXint(self, row, col):
+        return self.m[row, col]
+
+    def _getitem_intXslice(self, row, col):
+        shape = (_slice_length(col, self.m.shape[1]), )
+        return self.__class__.create_array(self.m, shape=shape, index=(row, col))
+
+    def _getitem_sliceXint(self, row, col):
+        shape = (_slice_length(row, self.m.shape[0]), )
+        return self.__class__.create_array(self.m, shape=shape, index=(row, col))
+
+    def _getitem_sliceXslice(self, row, col):
+        shape = (_slice_length(row, self.m.shape[0]),
+                 _slice_length(col, self.m.shape[1]))
+        return self.__class__(self.m, shape=shape, index=(row, col), transposed=self.transposed)
+
+    def toarray(self):
+        arr = self.m[self._index]
+        if self.transposed:
+            arr = arr.transpose()
+        return arr
+
+
+class ArrayProxyView(_ArrayProxyBase):
+    """
+    1D array view to a 2D matrix
+    """
+    def __init__(self, arg1, shape=None, index=None, copy=False):
+        super().__init__()
+        if not copy:
+            m = arg1
+
+            # one index MUST be an integer and the other MUST be a slice
+            assert(len(index) == 2)
+            assert(all(isinstance(idx, INT_TYPES + (slice, )) for idx in index))
+            assert(isinstance(index[0], INT_TYPES) != isinstance(index[1], INT_TYPES))
+
+            if shape is None:
+                if isinstance(index[0], INT_TYPES):
+                    shape = (m.shape[0], )
+                else:
+                    shape = (m.shape[1], )
+            assert(len(shape) == 1)
+
+            self._shape = shape
+            self.m = m
+            self._index = index
+            self._vdim = 1 if isinstance(index[0], INT_TYPES) else 0
+
+        else:
+            self.m = arg1.m
+            self._shape = arg1._shape
+            self._index = arg1._index
+            self.fixed = arg1.fixed
+
+    def copy(self):
+        return self.__class__(self, copy=True)
+
+    @property
+    def dtype(self):
+        return self.m.dtype
+
+    @property
+    def ndim(self):
+        return len(self._shape)
+
+    @property
+    def shape(self):
+        return self._shape
+
+    @property
+    def T(self):
+        return self.copy()
+
+    def __iter__(self):
+        _vdim = self._vdim
+        M = self._shape[0]
+        for d in range(*self._index[_vdim].indices(M)):
+            yield self[d]
+
+    def __getitem__(self, args):
+        _vdim = self._vdim
+        index = _unpack_index(args, self.shape)[0]
+        M = self._shape[0]
+        allM = self.m.shape[_vdim]
+
+        if isinstance(index, INT_TYPES):
+            index += self._index[_vdim].start
+        elif isinstance(index, slice):
+            index = _slice_slice(self._index[_vdim], allM, index, M)
+
+        if _vdim == 0:
+            row, col = index, self._index[1]
+        else:
+            row, col = self._index[0], index
+
+        if isinstance(row, INT_TYPES):
+            if isinstance(col, INT_TYPES):
+                return self._getitem_intXint(row, col)
+            elif isinstance(col, slice):
+                return self._getitem_intXslice(row, col)
+        elif isinstance(row, slice):
+            assert(isinstance(col, INT_TYPES))
+            return self._getitem_sliceXint(row, col)
+
+        raise IndexError("unsupported column index types")
+
+    def _getitem_intXint(self, row, col):
+        return self.m[row, col]
+
+    def _getitem_intXslice(self, row, col):
+        shape = (_slice_length(col, self.m.shape[1]), )
+        return self.__class__(self.m, shape=shape, index=(row, col))
+
+    def _getitem_sliceXint(self, row, col):
+        shape = (_slice_length(row, self.m.shape[0]), )
+        return self.__class__(self.m, shape=shape, index=(row, col))
+
+    def toarray(self):
+        return self.m[self._index]
+
+
+def _unpack_index(index, shape):
+    if not isinstance(index, tuple):
+        index = (index, )
+    if len(shape) < len(index):
+        raise IndexError("invalid index dimensionality - must be 2")
+
+    unpacked = ()
+    for shp, idx in zip_longest(shape, index):
+        idx = slice(None) if idx is None else idx
+        idx = _slice_defaults(idx, shp) if isinstance(idx, slice) else idx
+        unpacked += (idx, )
+
+    return unpacked
+
+
+def _slice_slice(outer, outer_len, inner, inner_len):
+    """
+    slice a slice - we take advantage of Python 3 range's support
+    for indexing.
+    """
+    assert(outer_len >= inner_len)
+    outer_rng = range(*outer.indices(outer_len))
+    rng = outer_rng[inner]
+    start, stop, step = rng.start, rng.stop, rng.step
+    if step < 0 and stop < 0:
+        stop = None
+    return slice(start, stop, step)
+
+
+def _range_length(start, stop, step):
+    """ return length of range """
+    assert(step != 0)
+    assert(start is not None and stop is not None and step is not None)
+    if step > 0 and start < stop:
+        return 1 + (stop - 1 - start) // step
+    elif step < 0 and start > stop:
+        return 1 + (start - 1 - stop) // -step
+    else:
+        return 0
+
+
+def _slice_length(s, length):
+    """ return slice length """
+    return _range_length(*s.indices(length))
+
+
+def _slice_defaults(s, length):
+    """ apply slice defaulting conventions """
+    assert(length >= 0)
+
+    step = 1 if s.step is None else s.step
+
+    if s.start is not None:
+        start = s.start
+        if start < 0:
+            start += length
+    else:
+        start = 0 if step > 0 else (length - 1)
+
+    if s.stop is not None:
+        stop = s.stop
+        if stop < 0:
+            stop += length
+    else:
+        stop = length if step > 0 else -length - 1
+
+    return slice(start, stop, step)

server/cli/cli.py

@@ -5,7 +5,7 @@ from .prepare import prepare
 
 
 @click.group(name="cellxgene", context_settings=dict(max_content_width=85))
-@click.version_option(version="0.11.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
+@click.version_option(version="0.12.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
 def cli():
     pass
 

server/cli/launch.py

@@ -2,10 +2,11 @@ import errno
 import functools
 import logging
 from os import devnull
-from os.path import splitext, basename, getsize
+from os.path import splitext, basename
 import sys
 import warnings
 import webbrowser
+from urllib.parse import urlparse
 
 import click
 
@@ -13,6 +14,7 @@ from server.app.app import Server
 from server.app.util.errors import ScanpyFileError
 from server.app.util.utils import custom_format_warning
 from server.utils.utils import find_available_port, is_port_available
+from server.app.util.data_locator import DataLocator
 
 # anything bigger than this will generate a special message
 BIG_FILE_SIZE_THRESHOLD = 100 * 2 ** 20  # 100MB
@@ -22,14 +24,18 @@ def common_args(func):
     """
     Decorator to contain CLI args that will be common to both CLI and GUI: title and engine args.
     """
+
     @click.option("--title", "-t", help="Title to display (if omitted will use file name).")
+    @click.option("--about",
+                  help="A URL to more information about the dataset."
+                       "(This must be an absolute URL including HTTP(S) protocol)")
     @click.option(
-        "--layout",
-        "-l",
+        "--embedding",
+        "-e",
         default=[],
         multiple=True,
-        show_default=True,
-        help="Layout name, eg, 'umap'."
+        show_default=False,
+        help="Embedding name, eg, 'umap'. Repeat option for multiple embeddings. Defaults to all."
     )
     @click.option("--obs-names", default=None, metavar="", help="Name of annotation field to use for observations.")
     @click.option("--var-names", default=None, metavar="", help="Name of annotation to use for variables.")
@@ -46,24 +52,52 @@ def common_args(func):
         show_default=True,
         help="Relative expression cutoff used when selecting top N differentially expressed genes",
     )
+    @click.option(
+        "--experimental-label-file",
+        default=None,
+        show_default=True,
+        multiple=False,
+        metavar="<user labels CSV file>",
+        help="CSV file containing user annotations; will be overwritten.  Created if does not exist.",
+    )
+    @click.option(
+        "--backed",
+        is_flag=True,
+        default=False,
+        show_default=False,
+        help="Load data in file-backed mode, which may save memory, but result in slower overall performance."
+    )
+    @click.option(
+        "--disable-diffexp",
+        is_flag=True,
+        default=False,
+        show_default=False,
+        help="Disable on-demand differential expression."
+    )
     @functools.wraps(func)
     def wrapper(*args, **kwargs):
         return func(*args, **kwargs)
+
     return wrapper
 
 
-def parse_engine_args(layout, obs_names, var_names, max_category_items, diffexp_lfc_cutoff):
+def parse_engine_args(embedding, obs_names, var_names, max_category_items,
+                      diffexp_lfc_cutoff, experimental_label_file, backed,
+                      disable_diffexp):
     return {
-        "layout": layout,
+        "layout": embedding,
         "max_category_items": max_category_items,
         "diffexp_lfc_cutoff": diffexp_lfc_cutoff,
         "obs_names": obs_names,
         "var_names": var_names,
+        "label_file": experimental_label_file,
+        "backed": backed,
+        "disable_diffexp": disable_diffexp
     }
 
 
 @click.command()
-@click.argument("data", metavar="<data file>", type=click.Path(exists=True, file_okay=True, dir_okay=False))
+@click.argument("data", nargs=1, metavar="<data file>", required=True)
 @click.option(
     "--verbose",
     "-v",
@@ -100,13 +134,17 @@ def launch(
         open_browser,
         port,
         host,
-        layout,
+        embedding,
         obs_names,
         var_names,
         max_category_items,
         diffexp_lfc_cutoff,
         title,
-        scripts
+        scripts,
+        about,
+        experimental_label_file,
+        backed,
+        disable_diffexp
 ):
     """Launch the cellxgene data viewer.
     This web app lets you explore single-cell expression data.
@@ -117,16 +155,34 @@ def launch(
 
     > cellxgene launch example_dataset/pbmc3k.h5ad --title pbmc3k
 
-    > cellxgene launch <your data file> --title <your title>"""
+    > cellxgene launch <your data file> --title <your title>
+
+    > cellxgene launch <url>"""
+
+    e_args = parse_engine_args(embedding, obs_names, var_names, max_category_items,
+                               diffexp_lfc_cutoff, experimental_label_file, backed,
+                               disable_diffexp)
+    try:
+        data_locator = DataLocator(data)
+    except RuntimeError as re:
+        raise click.ClickException(f"Unable to access data at {data}.  {str(re)}")
 
-    e_args = parse_engine_args(layout, obs_names, var_names, max_category_items, diffexp_lfc_cutoff)
     # Startup message
     click.echo("[cellxgene] Starting the CLI...")
 
     # Argument checking
-    name, extension = splitext(data)
-    if extension != ".h5ad":
-        raise click.FileError(basename(data), hint="file type must be .h5ad")
+    if data_locator.islocal():
+        # if data locator is local, apply file system conventions and other "cheap"
+        # validation checks.  If a URI, defer until we actually fetch the data and
+        # try to read it.  Many of these tests don't make sense for URIs (eg, extension-
+        # based typing).
+        if not data_locator.exists():
+            raise click.FileError(data, hint="file does not exist")
+        if not data_locator.isfile():
+            raise click.FileError(data, hint="data is not a file")
+        name, extension = splitext(data)
+        if extension != ".h5ad":
+            raise click.FileError(basename(data), hint="file type must be .h5ad")
 
     if debug:
         verbose = True
@@ -164,6 +220,25 @@ def launch(
     else:
         port = find_available_port(host)
 
+    if experimental_label_file:
+        lf_name, lf_ext = splitext(experimental_label_file)
+        if lf_ext and lf_ext != ".csv":
+            raise click.FileError(basename(experimental_label_file), hint="label file type must be .csv")
+
+    if about:
+        def url_check(url):
+            try:
+                result = urlparse(url)
+                if all([result.scheme, result.netloc]):
+                    return True
+                else:
+                    return False
+            except ValueError:
+                return False
+
+        if not url_check(about):
+            raise click.ClickException("Must provide an absolute URL for --about. (Example format: http://example.com)")
+
     # Setup app
     cellxgene_url = f"http://{host}:{port}"
 
@@ -177,26 +252,25 @@ def launch(
         log = logging.getLogger("werkzeug")
         log.setLevel(logging.ERROR)
 
-    file_size = getsize(data)
+    file_size = data_locator.size() if data_locator.islocal() else 0
 
     # if a big file, let the user know it may take a while to load.
     if file_size > BIG_FILE_SIZE_THRESHOLD:
-        click.echo(f"[cellxgene] Loading data from {basename(data)}, this may take awhile...")
+        click.echo(f"[cellxgene] Loading data from {basename(data)}, this may take a while...")
     else:
         click.echo(f"[cellxgene] Loading data from {basename(data)}.")
 
-    # Fix for anaconda python. matplotlib typically expects python to be installed as a framework TKAgg is usually
-    # available and fixes this issue. See https://matplotlib.org/faq/virtualenv_faq.html
-    import matplotlib as mpl
-
-    mpl.use("TkAgg")
     from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
 
     try:
-        server.attach_data(ScanpyEngine(data, e_args), title=title)
+        server.attach_data(ScanpyEngine(data_locator, e_args), title=title, about=about)
     except ScanpyFileError as e:
         raise click.ClickException(f"{e}")
 
+    if not disable_diffexp and server.app.data.config['diffexp_may_be_slow']:
+        click.echo(f"[cellxgene] CAUTION: due to the size of your dataset, "
+                   f"running differential expression may take longer or fail.")
+
     if open_browser:
         click.echo(f"[cellxgene] Launching! Opening your browser to {cellxgene_url} now.")
         webbrowser.open(cellxgene_url)
@@ -210,7 +284,7 @@ def launch(
         sys.stdout = f
 
     try:
-        server.app.run(host=host, debug=debug, port=port, threaded=True, use_debugger=False)
+        server.app.run(host=host, debug=debug, port=port, threaded=False if debug else True, use_debugger=False)
     except OSError as e:
         if e.errno == errno.EADDRINUSE:
             raise click.ClickException("Port is in use, please specify an open port using the --port flag.") from e

server/cli/prepare.py

@@ -8,12 +8,12 @@ from scipy.sparse.csc import csc_matrix
 @click.command()
 @click.argument("data", nargs=1, metavar="<dataset: file or path to data>", required=True)
 @click.option(
-    "--layout",
-    "-l",
+    "--embedding",
+    "-e",
     default=["umap", "tsne"],
     multiple=True,
     type=click.Choice(["umap", "tsne"]),
-    help="Layout algorithm",
+    help="Embedding algorithm",
     show_default=True,
 )
 @click.option(
@@ -41,34 +41,40 @@ from scipy.sparse.csc import csc_matrix
     "--make-var-names-unique", default=True, is_flag=True, help="Ensure var index is unique.", show_default=True
 )
 def prepare(
-    data,
-    layout,
-    recipe,
-    output,
-    plotting,
-    sparse,
-    overwrite,
-    set_obs_names,
-    set_var_names,
-    run_qc,
-    make_obs_names_unique,
-    make_var_names_unique,
+        data,
+        embedding,
+        recipe,
+        output,
+        plotting,
+        sparse,
+        overwrite,
+        set_obs_names,
+        set_var_names,
+        run_qc,
+        make_obs_names_unique,
+        make_var_names_unique,
 ):
     """Preprocesses data for use with cellxgene.
 
     This tool runs a series of scanpy routines for preparing a dataset
     for use with cellxgene. It loads data from different formats
     (h5ad, loom, or a 10x directory), runs dimensionality reduction,
-    computes nearest neighbors, computes a layout, performs clustering,
+    computes nearest neighbors, computes an embedding, performs clustering,
     and saves the results. Includes additional options for naming
     annotations, ensuring sparsity, and plotting results."""
 
     # collect slow imports here to make CLI startup more responsive
     click.echo("[cellxgene] Starting CLI...")
-    import matplotlib
+    try:
+        import matplotlib
 
-    matplotlib.use("Agg")
-    import scanpy as sc
+        matplotlib.use("Agg")
+        import scanpy as sc
+    except ImportError:
+        raise click.ClickException(
+            "[cellxgene] cellxgene prepare has not been installed. Please run `pip install cellxgene[prepare]` "
+            "to install the necessary requirements."
+        )
 
     # scanpy settings
     sc.settings.verbosity = 0
@@ -154,27 +160,20 @@ def prepare(
         sc.pp.neighbors(adata)
 
     def run_louvain(adata):
-        try:
-            sc.tl.louvain(adata)
-        except ModuleNotFoundError:
-            click.echo(
-                "\nWarning: louvain module is not installed, no clusters will be calculated. "
-                "To fix this please install cellxgene with the optional feature louvain enabled: "
-                "`pip install cellxgene[louvain]`"
-            )
+        sc.tl.louvain(adata)
 
-    def run_layout(adata):
+    def run_embedding(adata):
         if len(unique(adata.obs["louvain"].values)) < 10:
             palette = "tab10"
         else:
             palette = "tab20"
 
-        if "umap" in layout:
+        if "umap" in embedding:
             sc.tl.umap(adata)
             if plotting:
                 sc.pl.umap(adata, color="louvain", palette=palette, save="_louvain")
 
-        if "tsne" in layout:
+        if "tsne" in embedding:
             sc.tl.tsne(adata)
             if plotting:
                 sc.pl.tsne(adata, color="louvain", palette=palette, save="_louvain")
@@ -191,12 +190,12 @@ def prepare(
             "run_pca": "Running PCA",
             "run_neighbors": "Calculating neighbors",
             "run_louvain": "Calculating clusters",
-            "run_layout": "Computing layout",
+            "run_embedding": "Computing embedding",
         }
         if item is not None:
             return names[item.__name__]
 
-    steps = [calculate_qc_metrics, make_sparse, run_recipe, run_pca, run_neighbors, run_louvain, run_layout]
+    steps = [calculate_qc_metrics, make_sparse, run_recipe, run_pca, run_neighbors, run_louvain, run_embedding]
 
     click.echo(f"[cellxgene] Loading data from {data}, please wait...")
     adata = load_data(data)

server/requirements-dev.txt

@@ -1,5 +1,6 @@
 black
 bumpversion>=0.5
+parameterized>=0.7.0
 pytest>=3.6.3
 requests>=2.18.4
 twine>=1.12.1

server/requirements-prepare.txt

@@ -0,0 +1,3 @@
+scanpy>=1.3.7
+python-igraph
+louvain>=0.6

server/requirements.txt

@@ -6,10 +6,10 @@ Flask-Compress>=1.4.0
 Flask-Cors>=3.0.6
 Flask-RESTful>=0.3.6
 flatbuffers>=1.10.0
-matplotlib>=2.2
+fsspec>=0.4.4
 numpy>=1.15.2
 pandas>=0.23.1
-scanpy>=1.3.7
-scipy>=1.1.0
-scikit-learn>=0.19.1,!=0.20.0
+scipy>=1.3.0
 tables==3.5.1
+# TEMP workaround for https://github.com/theislab/scanpy/issues/832 aka h5py regression
+h5py==2.9.0

server/test/schema.json

@@ -10,19 +10,23 @@
       "columns": [
         {
           "name": "name_0",
-          "type": "string"
+          "type": "string",
+          "writable": false
         },
         {
           "name": "n_genes",
-          "type": "int32"
+          "type": "int32",
+          "writable": false
         },
         {
           "name": "percent_mito",
-          "type": "float32"
+          "type": "float32",
+          "writable": false
         },
         {
           "name": "n_counts",
-          "type": "float32"
+          "type": "float32",
+          "writable": false
         },
         {
           "name": "louvain",
@@ -36,7 +40,8 @@
             "FCGR3A+ Monocytes",
             "Dendritic cells",
             "Megakaryocytes"
-          ]
+          ],
+          "writable": false
         }
       ]
     },
@@ -45,11 +50,13 @@
       "columns": [
         {
           "name": "name_0",
-          "type": "string"
+          "type": "string",
+          "writable": false
         },
         {
           "name": "n_cells",
-          "type": "int32"
+          "type": "int32",
+          "writable": false
         }
       ]
     }

server/test/test_fbs.py

@@ -0,0 +1,93 @@
+import unittest
+import pandas as pd
+import numpy as np
+from scipy import sparse
+
+import decode_fbs
+from server.app.util.fbs.matrix import encode_matrix_fbs, decode_matrix_fbs
+
+
+class FbsTests(unittest.TestCase):
+    """Test Case for Matrix FBS data encode/decode """
+
+    def test_encode_boundary(self):
+        """ test various boundary checks """
+
+        # row indexing is unsupported
+        with self.assertRaises(ValueError):
+            encode_matrix_fbs(matrix=pd.DataFrame(), row_idx=[])
+
+        # matrix must be 2D
+        with self.assertRaises(ValueError):
+            encode_matrix_fbs(matrix=np.zeros((3, 2, 1)))
+        with self.assertRaises(ValueError):
+            encode_matrix_fbs(matrix=np.ones((10,)))
+
+    def fbs_checks(self, fbs, dims, expected_types, expected_column_idx):
+        d = decode_fbs.decode_matrix_FBS(fbs)
+        self.assertEqual(d["n_rows"], dims[0])
+        self.assertEqual(d["n_cols"], dims[1])
+        self.assertIsNone(d["row_idx"])
+        self.assertEqual(len(d["columns"]), dims[1])
+        for i in range(0, len(d["columns"])):
+            self.assertEqual(len(d["columns"][i]), dims[0])
+            self.assertIsInstance(d["columns"][i], expected_types[i][0])
+            if (expected_types[i][1] is not None):
+                self.assertEqual(d["columns"][i].dtype, expected_types[i][1])
+        if expected_column_idx is not None:
+            self.assertSetEqual(set(expected_column_idx), set(d["col_idx"]))
+
+    def test_encode_DataFrame(self):
+        df = pd.DataFrame(
+            data={
+                'a': np.zeros((10,), dtype=np.float32),
+                'b': np.ones((10,), dtype=np.int64),
+                'c': np.array([i for i in range(0, 10)], dtype=np.uint16),
+                'd': pd.Series(['x', 'y', 'z', 'x', 'y', 'z', 'a', 'x', 'y', 'z'], dtype='category')
+            })
+        expected_types = (
+            (np.ndarray, np.float32),
+            (np.ndarray, np.int32),
+            (np.ndarray, np.uint32),
+            (list, None)
+        )
+        fbs = encode_matrix_fbs(matrix=df, row_idx=None, col_idx=df.columns)
+        self.fbs_checks(fbs, (10, 4), expected_types, ['a', 'b', 'c', 'd'])
+
+    def test_encode_ndarray(self):
+        arr = np.zeros((3, 2), dtype=np.float32)
+        expected_types = (
+            (np.ndarray, np.float32),
+            (np.ndarray, np.float32),
+            (np.ndarray, np.float32)
+        )
+        fbs = encode_matrix_fbs(matrix=arr, row_idx=None, col_idx=None)
+        self.fbs_checks(fbs, (3, 2), expected_types, None)
+
+    def test_encode_sparse(self):
+        csc = sparse.csc_matrix(np.array([[0, 1, 2], [3, 0, 4]]))
+        expected_types = (
+            (np.ndarray, np.int32),
+            (np.ndarray, np.int32),
+            (np.ndarray, np.int32)
+        )
+        fbs = encode_matrix_fbs(matrix=csc, row_idx=None, col_idx=None)
+        self.fbs_checks(fbs, (2, 3), expected_types, None)
+
+    def test_roundtrip(self):
+        dfSrc = pd.DataFrame(
+            data={
+                'a': np.zeros((10,), dtype=np.float32),
+                'b': np.ones((10,), dtype=np.int64),
+                'c': np.array([i for i in range(0, 10)], dtype=np.uint16),
+                'd': pd.Series(['x', 'y', 'z', 'x', 'y', 'z', 'a', 'x', 'y', 'z'], dtype='category')
+            })
+        dfDst = decode_matrix_fbs(encode_matrix_fbs(matrix=dfSrc, col_idx=dfSrc.columns))
+        self.assertEqual(dfSrc.shape, dfDst.shape)
+        self.assertEqual(set(dfSrc.columns), set(dfDst.columns))
+        for c in dfSrc.columns:
+            self.assertTrue(c in dfDst.columns)
+            if isinstance(dfSrc[c], pd.Series):
+                self.assertTrue(np.all(dfSrc[c] == dfDst[c]))
+            else:
+                self.assertEqual(dfSrc[c], dfDst[c])