release 0.12.0 (#981)

* initial cut at backed mode

* make flask multithreading conditional on debug flag

* update X access to support backed mode

* lint

* improve help message for backed mode

* fix tests

* add MatrixProxy to normalize supported matrix types

* add FAQ entry for --backed

* remove use of matrix.T

* clean up

* add ability to disable diffexp from CLI; add hueristic to detect likely slow diffexp calculation, and warn user

* fix tests

* do not print diffexp speed warning if diffexp is disabled

* tweak wording of diffexp speed messages

* add FAQ entry on --disable-diffexp

* revise heuristic for warning about slow diffexp

* use quick tooltip delay on diffexp button

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.9.1
+current_version = 0.12.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"
@@ -13,3 +13,7 @@ replace = version="{new_version}"
 search = "version": "{current_version}"
 replace = "version": "{new_version}"
 
+[bumpversion:file:server/__init__.py]
+search = __version__ = "{current_version}"
+replace = __version__ = "{new_version}"
+

.travis.yml

@@ -9,8 +9,8 @@ cache:
 install:
   - set -eo pipefail
   - pip install flake8
-  - make build
+  - make pydist
-  - make install
+  - make install-dist
   - pip install -r server/requirements-dev.txt
 
 jobs:
@@ -27,6 +27,5 @@ jobs:
     script: docker build .
   - name: "Smoke Tests"
     python: "3.6"
-    if: branch = master AND type = cron
     script:
       - npm run --prefix client/ smoke-test

CONTRIBUTING.md

@@ -0,0 +1,49 @@
+# contributing to cellxgene
+
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress.
+
+All code contributions and dependencies must be compatible with the project's [open-source license (MIT)](LICENSE.txt).
+
+This project adheres to the Contributor Covenant
+[code of conduct](https://github.com/chanzuckerberg/.github/tree/master/CODE_OF_CONDUCT.md).
+By participating, you are expected to uphold this code. Please report unacceptable behavior
+to opensource@chanzuckerberg.com.
+
+And finally, if you have any questions about any of this stuff, just ask! :)
+
+## developer guide
+
+This project has made a few key design choices
+
+- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
+- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
+- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
+
+Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
+
+If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
+
+- python 3.6+
+- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
+
+Then clone the project
+
+```
+git clone https://github.com/chanzuckerberg/cellxgene.git
+```
+
+Build the client web assets by calling `make` from inside the `cellxgene` folder
+
+```
+make
+```
+
+Install all requirements (we recommend doing this inside a virtual environment)
+
+```
+pip install -e .
+```
+
+You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
+
+If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.

MANIFEST.in

@@ -1,4 +1,5 @@
 recursive-include server/app/web/templates *
 recursive-include server/app/web/static *
 
-include server/requirements.txt
+include server/requirements.txt
+include server/requirements-prepare.txt

README.md

@@ -1,40 +1,65 @@
-# cellxgene
+<img src="./docs/cellxgene-logo.svg" width="300">
 
-> an interactive explorer for single-cell transcriptomics data
+_an interactive explorer for single-cell transcriptomics data_
 
-_cellxgene_ (pronounced "sell-by-jean") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
+[![DOI](https://zenodo.org/badge/105615409.svg)](https://zenodo.org/badge/latestdoi/105615409) [![PyPI](https://img.shields.io/pypi/v/cellxgene)](https://pypi.org/project/cellxgene/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/cellxgene)](https://pypistats.org/packages/cellxgene) [![GitHub last commit](https://img.shields.io/github/last-commit/chanzuckerberg/cellxgene)](https://github.com/chanzuckerberg/cellxgene/pulse)
+
+cellxgene (pronounced "cell-by-gene") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data.
 
 <img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
 
 - Want to install and use cellxgene? Visit the [cellxgene docs](https://chanzuckerberg.github.io/cellxgene/).
-- Want to see where we are going? Check out [our roadmap](ROADMAP.md).
+- Want to see where we are going? Check out [our roadmap](https://github.com/chanzuckerberg/cellxgene/blob/master/ROADMAP.md).
-- Want to contribute? See our [contributors guide](#Contributing)
+- Want to contribute? See our [contributors guide](https://github.com/chanzuckerberg/cellxgene/blob/master/CONTRIBUTING.md).
 
 ## quick start
 
-To install _cellxgene_ you need Python 3.6+. We recommend [installing _cellxgene_ into a conda or virtual environment.](https://chanzuckerberg.github.io/cellxgene/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
+To install cellxgene you need Python 3.6+. We recommend [installing cellxgene into a conda or virtual environment.](https://chanzuckerberg.github.io/cellxgene/faq.html#how-do-i-create-a-python-36-environment-for-cellxgene)
 
 Install the package.
-``` bash
+
+```bash
 pip install cellxgene
 ```
 
 Download an example [anndata](https://anndata.readthedocs.io/en/latest/) file
 
-``` bash
+```bash
-curl -o pbmc3k.h5ad https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad
+curl -O https://cellxgene-example-data.czi.technology/pbmc3k.h5ad.zip
+unzip pbmc3k.h5ad
 ```
 
-Launch _cellxgene_
+Launch cellxgene
-``` bash
+
+```bash
 cellxgene launch pbmc3k.h5ad --open
 ```
 
-To learn more about what you can do with _cellxgene_, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-stared/) guide.
+To learn more about what you can do with cellxgene, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-started.html) guide.
 
 ## get in touch
 
-Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. Have feature requests or bugs? Please submit these as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+
+## contributing
+
+We warmly welcome contributions from the community! Please see our [contributing guide](https://github.com/chanzuckerberg/cellxgene/blob/master/CONTRIBUTING.md) and don't hesitate to open an issue or send a pull request to improve cellxgene.
+
+This project adheres to the Contributor Covenant [code of conduct](https://github.com/chanzuckerberg/.github/blob/master/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.
+
+## core team
+
+The current core team:
+
+- Colin Megill, frontend & product design
+- Bruce Martin, software engineer
+- Sidney Bell, computational biologist
+- Lia Prins, designer
+- Severiano Badajoz, software engineer
+
+We would also like to gratefully acknowledge contributions from past core team members:
+
+- Charlotte Weaver, software engineer
 
 ## where we are going
 
@@ -48,48 +73,7 @@ There are 4 key features we plan to implement in the near term.
 - Toggle embeddings
 - Gene information
 
-For more detail on these features and where we are going, see [our roadmap](ROADMAP.md).
+For more detail on these features and where we are going, see [our roadmap](https://github.com/chanzuckerberg/cellxgene/blob/master/ROADMAP.md).
-
-## contributing
-
-We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
-
-### developer guide
-
-This project has made a few key design choices
-
-- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
-- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
-- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
-
-Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
-
-If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
-
-- python 3.6+
-- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
-
-Then clone the project
-
-```
-git clone https://github.com/chanzuckerberg/cellxgene.git
-```
-
-Build the client web assets by calling `make` from inside the `cellxgene` folder
-
-```
-make
-```
-
-Install all requirements (we recommend doing this inside a virtual environment)
-
-```
-pip install -e .
-```
-
-You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
-
-If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.
 
 ## inspiration
 
@@ -97,17 +81,9 @@ We've been heavily inspired by several other related single-cell visualization p
 
 We were inspired by Mike Bostock and the [crossfilter](https://github.com/crossfilter) team for the design of our filtering implementation.
 
-We have been working closely with the [`scanpy`](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
+We have been working closely with the [scanpy](https://github.com/theislab/scanpy) team to integrate with their awesome analysis tools. Special thanks to Alex Wolf, Fabian Theis, and the rest of the team for their help during development and for providing an example dataset.
 
-We are eager to explore integrations with other computational backends such as [`Seurat`](https://github.com/satijalab/seurat) or [`Bioconductor`](https://github.com/Bioconductor)
+We are eager to explore integrations with other computational backends such as [Seurat](https://github.com/satijalab/seurat) or [Bioconductor](https://github.com/Bioconductor)
-
-## core team
-
-- Colin Megill, frontend & product design
-- Charlotte Weaver, software engineer
-- Bruce Martin, software engineer
-- Sidney Bell, computational biologist
-- Justin Kiggins, product manager
 
 ## reuse
 

ROADMAP.md

@@ -1,54 +1,46 @@
 # cellxgene roadmap
 
-We are very exited for _cellxgene_ to become a valuable tool in collaborations
+cellxgene makes it easier for biologists to collaboratively explore and understand their single-cell RNA-seq data. 
-between computational biologists and experimental biologists working on
+In the near term, we are focused on continuing to enable fast, interactive exploration of single-cell data, supporting collaborative workflows in single-cell analysis, and improving user support. 
-single-cell transcriptomics data. _cellxgene_ is in active development, and we
+If you have questions or feedback about this roadmap, please submit an issue on GitHub.
-would love to include the community as we plan new features to work on. If you
-have questions of feedback about this roadmap, please submit an issue on
-GitHub.
-
 Please note: this roadmap is subject to change.
 
-*Last updated: April 11, 2019*
+Last updated: June 25, 2019
 
-## what we are building now
+## Fast, interactive exploration of single-cell data
 
-In the near term, our goal is to enable teams of computational and experimental
+### Exposing Relationships Between Metadata and Data
-biologists to collaboratively explore and annotate their single-cell RNA-seq data.
+Biologists need to understand how variables (stored in metadata) are associated with one another and how they relate to changes in gene expression. 
+Building upon visualization features that reveal categorical metadata relationships (cluster occupancy) and gene expression relationships (scatterplot), we plan to add exploratory visualization components that enable investigation of relationships between metadata and gene expression. 
+See [issue #616](https://github.com/chanzuckerberg/cellxgene/issues/616) for more details.
 
-There are 4 key features we plan to implement in the near term.
+### Contextualizing Genes
+While exploring a transcriptomics dataset, scientists need to understand the biological context of genes.
+This context may be provided by user-defined gene metadata or publicly available gene databases. 
+We plan to support augmenting gene names with additional information that is useful to biologists. 
+See [issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for more detail.
 
-- Click install and launch
+## Support collaborative workflows in single-cell analysis
-- Manual annotation workflows
-- Toggle embeddings
-- Gene information
 
-### simple install and launch
+### Manual Annotations
+cellxgene offers exploratory visualizations that are critical for manual annotation workflows, especially in collaborative environments. 
+We plan to support manually annotating cells with labels (i.e., cell type or QC flags), and their easy export for downstream analysis. 
+See [issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524) for more details.
 
-The command line interface for installing and launching cellxgene is a barrier
+### Simple Click to Launch
-for users who are not used to Python or using the command line. We plan to
-support installation and launch of cellxgene on Mac and Windows. See
-[Issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for more details.
 
-### manual annotation workflows
+Many biologists prefer not to interact with the command line and need an OS-native experience when using cellxgene. 
+We plan to implement a point-and-click installation and launch experience so that users can easily load data into cellxgene. 
+See [issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for details.
 
-The exploratory visualization that cellxgene offers is critical for manual
+### Python API
-annotation workflows, especially in collaborative environments. We plan to
+For computational biologists, saving h5ad files then loading them into cellxgene is a point of friction.  
-support manually annotate cells with labels (i.e., cell type or QC flags) for
+We plan to support importing cellxgene as a Python package so that users can launch cellxgene directly from an interactive environment (such as Jupyter, IPython, or Spyder), and pass data to and from the cellxgene UI. 
-downstream analysis. See [Issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524)
-for more details.
 
-### toggle embeddings
+## Improving user support
 
-While a single dataset may have multiple embeddings calculated (tSNE, umap, in
+### Improved documentation
-situ coordinates, trajectories, etc), cellxgene currently requires the user to select the
+cellxgene has some specific expectations about how data is stored. 
-embedding to use in the main layout at launch. We plan to support letting users
+We want to ensure that new users can get started easily and learn how to use cellxgene with their own data. 
-toggle between any embedding present in a file from the cellxgene interface.
+We plan to improve documentation on getting started, installation, data, and contributing. 
-See [Issue #594](https://github.com/chanzuckerberg/cellxgene/issues/594) for details.
+See [issue #533](https://github.com/chanzuckerberg/cellxgene/issues/533) for more details.
-
-### gene information
-
-Differential expression returns only the names of genes, but no additional information
-about gene metadata, function, or known associations. We plan to help users learn
-more about genes they discover by exposing additional gene metadata. See
-[Issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for details.

app.json

@@ -0,0 +1,21 @@
+{
+  "name": "cellxgene",
+  "description": "An interactive explorer for single-cell transcriptomics data",
+  "repository": "https://github.com/chanzuckerberg/cellxgene/",
+  "logo": "https://cellxgene-example-data.czi.technology/favicon.png",
+  "keywords": [
+    "scientific",
+    "visualization",
+    "scrna-seq",
+    "transcriptomics",
+    "dataviz"
+  ],
+  "stack": "container",
+  "env": {
+    "DATASET": {
+      "description": "Link to dataset",
+      "value": "https://cellxgene-example-data.czi.technology/pbmc3k.h5ad",
+      "required": "true"
+    }
+  }
+}

cellxgene-osx.spec

@@ -0,0 +1,41 @@
+# -*- mode: python ; coding: utf-8 -*-
+
+block_cipher = None
+
+
+a = Analysis(['server/gui/main.py'],
+             pathex=['/Users/charlotteweaver/Documents/Git/cellxgene'],
+             binaries=[('/System/Library/Frameworks/Tk.framework/Tk', 'tk'), ('/System/Library/Frameworks/Tcl.framework/Tcl', 'tcl')],
+             datas=[('server/app/web/templates/', 'server/app/web/templates/'), ('server/app/web/static/', 'server/app/web/static/')],
+             hiddenimports=['sklearn', 'sklearn.utils._cython_blas', 'sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree', 'sklearn.tree._utils'],
+             hookspath=['server/gui/'],
+             runtime_hooks=[],
+             excludes=[],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+exe = EXE(pyz,
+          a.scripts,
+          [],
+          exclude_binaries=True,
+          name='cellxgene',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          console=False , icon='server/gui/images/cxg_icons.icns')
+coll = COLLECT(exe,
+               a.binaries,
+               a.zipfiles,
+               a.datas,
+               strip=False,
+               upx=True,
+               upx_exclude=[],
+               name='cellxgene')
+app = BUNDLE(coll,
+             name='cellxgene.app',
+             icon='server/gui/images/cxg_icons.icns',
+             bundle_identifier=None)

cellxgene-windows.spec

@@ -0,0 +1,36 @@
+# -*- mode: python -*-
+
+block_cipher = None
+
+
+a = Analysis(['server\\gui\\main.py'],
+             pathex=['C:\\Users\\Charlotte\\Documents\\git\\cellxgene'],
+             binaries=[],
+             datas=[('server/app/web/templates/', 'server/app/web/templates'), ('server/app/web/static/', 'server/app/web/static')],
+             hiddenimports=[],
+             hookspath=['server/gui/'],
+             runtime_hooks=[],
+             excludes=[],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+exe = EXE(pyz,
+          a.scripts,
+          [],
+          exclude_binaries=True,
+          name='cellxgene',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          console=False , icon='server\\gui\\images\\icon.ico')
+coll = COLLECT(exe,
+               a.binaries,
+               a.zipfiles,
+               a.datas,
+               strip=False,
+               upx=True,
+               name='cellxgene')

client/__tests__/e2e/data.js

@@ -1,6 +1,6 @@
 export const datasets = {
   pbmc3k: {
-    title: "cellxgene: pbmc3k",
+    title: "pbmc3k",
     dataframe: {
       nObs: "2638",
       nVar: "1838",
@@ -26,8 +26,8 @@ export const datasets = {
     cellsets: {
       lasso: [
         {
-          "coordinates-as-percent": { x1: 0.25, y1: 0.25, x2: 0.35, y2: 0.35 },
+          "coordinates-as-percent": { x1: 0.05, y1: 0.25, x2: 0.15, y2: 0.35 },
-          count: "26"
+          count: "88"
         }
       ],
       categorical: [
@@ -87,12 +87,18 @@ export const datasets = {
       categorical: {
         louvain: {
           "B cells": "342",
-          Megakaryocytes: "15"
+          "CD14+ Monocytes": "0",
+          "CD4 T cells": "0",
+          "CD8 T cells": "0",
+          "Dendritic cells": "0",
+          "FCGR3A+ Monocytes": "0",
+          Megakaryocytes: "15",
+          "NK cells": "0"
         }
       },
       lasso: {
-        "coordinates-as-percent": { x1: 0.45, y1: 0.45, x2: 0.5, y2: 0.5 },
+        "coordinates-as-percent": { x1: 0.45, y1: 0.05, x2: 0.65, y2: 0.15 },
-        count: "67"
+        count: "39"
       }
     },
     scatter: {
@@ -108,6 +114,15 @@ export const datasets = {
           count: "24"
         }
       }
+    },
+    clip: {
+      min: "30",
+      max: "70",
+      metadata: "n_genes",
+      gene: "S100A8",
+      "coordinates-as-percent": { x1: 0.25, y1: 0.5, x2: 0.55, y2: 0.5 },
+      count: "392",
+      "gene-cell-count": "421"
     }
   }
 };

client/__tests__/e2e/e2e.test.js

@@ -54,14 +54,14 @@ afterAll(() => {
   }
 });
 
-describe("did launch", async () => {
+describe("did launch", () => {
   test("page launched", async () => {
     let el = await utils.getOneElementInnerHTML("[data-testid='header']");
     expect(el).toBe(data.title);
   });
 });
 
-describe("metadata loads", async () => {
+describe("metadata loads", () => {
   test("categories and values from dataset appear", async () => {
     for (const label in data.categorical) {
       await utils.waitByID(`category-${label}`);
@@ -87,7 +87,7 @@ describe("metadata loads", async () => {
   });
 });
 
-describe("cell selection", async () => {
+describe("cell selection", () => {
   test("selects all cells cellset 1", async () => {
     const cellCount = await cxgActions.cellSet(1);
     expect(cellCount).toBe(data.dataframe.nObs);
@@ -138,7 +138,7 @@ describe("cell selection", async () => {
   });
 });
 
-describe("gene entry", async () => {
+describe("gene entry", () => {
   test("search for single gene", async () => {
     // blueprint's  typeahead is treating typing weird, clicking & waiting first solves this
     await utils.typeInto("gene-search", data.genes.search);
@@ -154,14 +154,17 @@ describe("gene entry", async () => {
     await utils.clickOn("section-bulk-add");
     await utils.typeInto("input-bulk-add", testGenes.join(","));
     await page.keyboard.press("Enter");
-    const userGeneHist = await cxgActions.getAllHistograms(
+
-      "histogram-user-gene"
+    const allHistograms = await cxgActions.getAllHistograms(
+      "histogram-user-gene",
+      testGenes
     );
-    expect(userGeneHist).toEqual(expect.arrayContaining(testGenes));
+    expect(allHistograms).toEqual(expect.arrayContaining(testGenes));
+    expect(allHistograms.length).toEqual(testGenes.length);
   });
 });
 
-describe("diffexp", async () => {
+describe("diffexp", () => {
   test("selects cells, saves them and performs diffexp", async () => {
     for (const select of data.diffexp.cellset1) {
       if (select.kind === "categorical") {
@@ -176,15 +179,18 @@ describe("diffexp", async () => {
     }
     await cxgActions.cellSet(2);
     await utils.clickOn("diffexp-button");
-    const diffExpHists = await cxgActions.getAllHistograms("histogram-diffexp");
+    const allHistograms = await cxgActions.getAllHistograms(
-    expect(diffExpHists).toEqual(
+      "histogram-diffexp",
+      data.diffexp["gene-results"]
+    );
+    expect(allHistograms).toEqual(
       expect.arrayContaining(data.diffexp["gene-results"])
     );
+    expect(allHistograms.length).toEqual(data.diffexp["gene-results"].length);
   });
 });
-//
 
-describe("subset/reset", async () => {
+describe("subset/reset", () => {
   test("subset - cell count matches", async () => {
     for (const select of data.subset.cellset1) {
       if (select.kind === "categorical") {
@@ -258,7 +264,7 @@ describe("subset/reset", async () => {
   });
 });
 
-describe("scatter plot", async () => {
+describe("scatter plot", () => {
   test("scatter plot appears", async () => {
     await cxgActions.reset();
     const testGenes = data.scatter.genes;
@@ -271,8 +277,37 @@ describe("scatter plot", async () => {
   });
 });
 
+describe("clipping", () => {
+  test("clip continuous", async () => {
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.metadata}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip.count);
+  });
+
+  test("clip gene", async () => {
+    await utils.typeInto("gene-search", data.clip.gene);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(`[data-testid='histogram-${data.clip.gene}']`);
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.gene}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip["gene-cell-count"]);
+  });
+});
+
 // interact with UI elements just that they do not break
-describe("ui elements don't error", async () => {
+describe("ui elements don't error", () => {
   test("color by", async () => {
     for (const label in data.categorical) {
       await utils.clickOn(`colorby-${label}`);

client/__tests__/e2e/puppeteerUtils.js

@@ -13,13 +13,48 @@ export const puppeteerUtils = puppeteerPage => ({
     );
   },
 
+  async waitForAllByIds(testids, props = {}) {
+    await Promise.all(
+      testids.map(testid =>
+        puppeteerPage.waitForSelector(`[data-testid='${testid}']`)
+      )
+    );
+  },
+
+  async getAllByClass(testclass, props = {}) {
+    const elements = await puppeteerPage.$$eval(
+      `[data-testclass=${testclass}]`,
+      els => {
+        return els.map(el => {
+          return el.dataset.testid;
+        });
+      }
+    );
+    return elements;
+  },
+
   async typeInto(testid, text) {
     // only works for text without special characters
     await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
     // type ahead can be annoying if you don't pause before you type
-    await puppeteerPage.click(`[data-testid='${testid}']`);
+    await puppeteerPage.click(selector);
     await puppeteerPage.waitFor(200);
-    await puppeteerPage.type(`[data-testid='${testid}']`, text);
+    await puppeteerPage.type(selector, text);
+  },
+
+  async clearInputAndTypeInto(testid, text) {
+    await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
+    // only works for text without special characters
+    // type ahead can be annoying if you don't pause before you type
+    await puppeteerPage.click(selector);
+    await puppeteerPage.waitFor(200);
+    // select all
+
+    await puppeteerPage.click(selector, { clickCount: 3 });
+    await puppeteerPage.keyboard.type("Backspace");
+    await puppeteerPage.type(selector, text);
   },
 
   async clickOn(testid) {
@@ -29,11 +64,13 @@ export const puppeteerUtils = puppeteerPage => ({
   },
 
   async getOneElementInnerHTML(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerHTML);
     return text;
   },
 
   async getOneElementInnerText(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerText);
     return text;
   }
@@ -60,20 +97,16 @@ export const cellxgeneActions = puppeteerPage => ({
     await puppeteerPage.mouse.up();
   },
 
-  async getAllHistograms(testclass) {
+  async getAllHistograms(testclass, testids) {
-    await puppeteerUtils(puppeteerPage).waitByClass(testclass);
+    const histTestIds = testids.map(tid => `histogram-${tid}`);
-    const histograms = await puppeteerPage.$$eval(
+    // these load asynchronously, so we need to wait for each histogram individually
-      `[data-testclass=${testclass}]`,
+    await puppeteerUtils(puppeteerPage).waitForAllByIds(histTestIds);
-      els => {
+    const allHistograms = await puppeteerUtils(puppeteerPage).getAllByClass(
-        return els.map(el => {
+      testclass
-          return el.dataset.testid.substring(
+    );
-            "histogram_".length,
+    return allHistograms.map(hist =>
-            el.dataset.testid.length
+      hist.substr("histogram_".length, hist.length)
-          );
-        });
-      }
     );
-    return histograms;
   },
 
   async getAllCategoriesAndCounts(category) {
@@ -161,5 +194,18 @@ export const cellxgeneActions = puppeteerPage => ({
     await puppeteerUtils(puppeteerPage).clickOn("reset");
     // loading state never actually happens, reset is too fast
     await page.waitFor(200);
+  },
+
+  async clip(min = 0, max = 100) {
+    await puppeteerUtils(puppeteerPage).clickOn("visualization-settings");
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto(
+      "clip-min-input",
+      min
+    );
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto(
+      "clip-max-input",
+      max
+    );
+    await puppeteerUtils(puppeteerPage).clickOn("clip-commit");
   }
 });

client/__tests__/util/dataframe/dataframe.test.js

@@ -452,6 +452,59 @@ describe("dataframe factories", () => {
     });
   });
 
+  describe("withColsFrom", () => {
+    test("error conditions", () => {
+      /*
+      make sure we catch common errors:
+      - duplicate column names
+      - dimensionality difference
+      */
+      const dfA = new Dataframe.Dataframe(
+        [2, 3],
+        [["red", "blue"], [true, false], [1, 0]],
+        null,
+        new Dataframe.KeyIndex(["colors", "bools", "numbers"])
+      );
+
+      /* different dimensionality should throw error */
+      const dfB = new Dataframe.Dataframe(
+        [3, 1],
+        [["red", "blue", "green"]],
+        null,
+        new Dataframe.KeyIndex(["colorsA"])
+      );
+      expect(() => dfA.withColsFrom(dfB)).toThrow(RangeError);
+
+      /* duplicate labels should throw an error */
+      expect(() => dfA.withColsFrom(dfA)).toThrow(Error);
+    });
+
+    test("simple", () => {
+      /* simple test that it works as expected in common case */
+      const dfA = new Dataframe.Dataframe(
+        [2, 1],
+        [["red", "blue"]],
+        null,
+        new Dataframe.KeyIndex(["colors"])
+      );
+      const dfB = new Dataframe.Dataframe(
+        [2, 1],
+        [[true, false]],
+        null,
+        new Dataframe.KeyIndex(["bools"])
+      );
+
+      const dfC = dfA.withColsFrom(dfB);
+      expect(dfC).toBeDefined();
+      expect(dfC.dims).toEqual([2, 2]);
+      expect(dfC.colIndex.keys()).toEqual(["colors", "bools"]);
+      expect(dfC.rowIndex).toEqual(dfA.rowIndex);
+      expect(dfC.rowIndex.keys()).toEqual(dfA.rowIndex.keys());
+      expect(dfC.col("colors").asArray()).toEqual(["red", "blue"]);
+      expect(dfC.col("bools").asArray()).toEqual([true, false]);
+    });
+  });
+
   describe("dropCol", () => {
     test("KeyIndex", () => {
       const df = new Dataframe.Dataframe(
@@ -567,6 +620,32 @@ describe("dataframe factories", () => {
       expect(dfB.iat(0, 1)).toEqual(1);
       expect(dfB.iat(0, 2)).toEqual(1);
     });
+
+    test("columns", () => {
+      const df = Dataframe.Dataframe.create(
+        [3, 3],
+        [new Array(3).fill(0), new Array(3).fill(0), new Array(3).fill(0)]
+      );
+
+      expect(df).toBeDefined();
+      expect(df.columns()).toHaveLength(3);
+      expect(df.columns()[0]).toEqual(df.icol(0));
+      expect(df.columns()[2]).toEqual(df.icol(2));
+    });
+
+    test("renameCol", () => {
+      const dfA = new Dataframe.Dataframe(
+        [2, 2],
+        [[true, false], [1, 0]],
+        null,
+        new Dataframe.KeyIndex(["A", "B"])
+      );
+      const dfB = dfA.renameCol("B", "C");
+      expect(dfA.colIndex.keys()).toEqual(["A", "B"]);
+      expect(dfB.colIndex.keys()).toEqual(["A", "C"]);
+      expect(dfA.dims).toMatchObject(dfB.dims);
+      expect(dfA.columns()).toMatchObject(dfB.columns());
+    });
   });
 });
 

client/__tests__/util/dataframe/histogram.test.js

@@ -0,0 +1,93 @@
+import * as Dataframe from "../../../src/util/dataframe";
+
+describe("Dataframe column histogram", () => {
+  test("categorical by categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("cat").histogram(df.col("name"));
+    expect(h1).toMatchObject(
+      new Map([
+        ["n1", new Map([["c1", 1]])],
+        ["n2", new Map([["c2", 1]])],
+        ["n3", new Map([["c3", 1]])]
+      ])
+    );
+    // memoized?
+    expect(df.col("cat").histogram(df.col("name"))).toMatchObject(h1);
+  });
+
+  test("continuous by categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("value").histogram(3, [0, 2], df.col("name"));
+    expect(h1).toMatchObject(
+      new Map([["n1", [1, 0, 0]], ["n2", [0, 1, 0]], ["n3", [0, 0, 1]]])
+    );
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2], df.col("name"))).toMatchObject(
+      h1
+    );
+  });
+
+  test("categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("cat").histogram();
+    expect(h1).toMatchObject(new Map([["c1", 1], ["c2", 1], ["c3", 1]]));
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2])).toMatchObject(h1);
+  });
+
+  test("continuous", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("value").histogram(3, [0, 2]);
+    expect(h1).toMatchObject([1, 1, 1]);
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2])).toMatchObject(h1);
+  });
+
+  test("continuous thesholds correct", () => {
+    const vals = [0, 1, 9, 10, 11, 20, 99, 100];
+    const df = new Dataframe.Dataframe(
+      [8, 2],
+      [new Int32Array(vals), new Float32Array(vals)]
+    );
+
+    expect(df.col(0).histogram(5, [0, 100])).toEqual([5, 1, 0, 0, 2]);
+    expect(df.col(1).histogram(5, [0, 100])).toEqual([5, 1, 0, 0, 2]);
+    expect(df.col(0).histogram(2, [0, 10])).toEqual([2, 2]);
+    expect(df.col(0).histogram(10, [0, 100])).toEqual([
+      3,
+      2,
+      1,
+      0,
+      0,
+      0,
+      0,
+      0,
+      0,
+      2
+    ]);
+  });
+});

client/__tests__/util/range.test.js

@@ -0,0 +1,42 @@
+import { range, rangeFill } from "../../src/util/range";
+
+describe("range", () => {
+	test("no defaults", () => {
+		expect(range(0, 3, 1)).toMatchObject([0, 1, 2]);
+	});
+
+	test("range(stop)", () => {
+		expect(range(3)).toMatchObject([0, 1, 2]);
+		expect(range(0)).toMatchObject([]);
+		expect(range(1)).toMatchObject([0]);
+	});
+
+	test("range(start,stop)", () => {
+		expect(range(0, 0)).toMatchObject([]);
+		expect(range(0, 2)).toMatchObject([0, 1]);
+		expect(range(4, 8)).toMatchObject([4, 5, 6, 7]);
+	});
+
+	test("range(start, stop, step", () => {
+		expect(range(4, 0, -1)).toMatchObject([4, 3, 2, 1]);
+		expect(range(0, 4, 2)).toMatchObject([0, 2]);
+	});
+});
+
+describe("rangefill", () => {
+	test("rangeFill(arr)", () => {
+		expect(rangeFill(new Int32Array(3))).toMatchObject(
+			new Int32Array([0, 1, 2])
+		);
+	});
+	test("rangeFill(arr, start)", () => {
+		expect(rangeFill(new Int32Array(2), 1)).toMatchObject(
+			new Int32Array([1, 2])
+		);
+	});
+	test("rangeFill(arr, start, step)", () => {
+		expect(rangeFill(new Int32Array(3), 2, -1)).toMatchObject(
+			new Int32Array([2, 1, 0])
+		);
+	});
+});

client/__tests__/util/stateManager/fbs.test.js

@@ -0,0 +1,34 @@
+/*
+test FBS encode/decode API
+*/
+import { Dataframe, KeyIndex } from "../../../src/util/dataframe";
+import {
+	decodeMatrixFBS,
+	encodeMatrixFBS
+} from "../../../src/util/stateManager/matrix";
+
+describe("encode/decode", () => {
+	test("round trip", () => {
+		const columns = [
+			["red", "green", "blue"],
+			new Int32Array(3).fill(0),
+			new Uint32Array(3).fill(1),
+			new Float32Array(3).fill(2)
+		];
+
+		const dfNoColIdx = new Dataframe([3, 4], columns);
+		const dfA = decodeMatrixFBS(encodeMatrixFBS(dfNoColIdx));
+		expect([dfA.nRows, dfA.nCols]).toEqual(dfNoColIdx.dims);
+		expect(dfA.colIdx).toBeNull();
+		expect(dfA.rowIdx).toBeNull();
+		expect(dfA.columns).toEqual(columns);
+
+		const colIndex = new KeyIndex(["a", "b", "c", "d"]);
+		const dfWithColIdx = new Dataframe([3, 4], columns, null, colIndex);
+		const dfB = decodeMatrixFBS(encodeMatrixFBS(dfWithColIdx));
+		expect([dfB.nRows, dfB.nCols]).toEqual(dfWithColIdx.dims);
+		expect(dfB.colIdx).toEqual(colIndex.keys());
+		expect(dfB.rowIdx).toBeNull();
+		expect(dfB.columns).toEqual(columns);
+	});
+});

client/__tests__/util/stateManager/sampleResponses.js

@@ -34,28 +34,38 @@ const aSchemaResponse = {
       type: "float32"
     },
     annotations: {
-      obs: [
+      obs: {
-        { name: "name", type: "string" },
+        index: "name",
-        { name: "field1", type: "int32" },
+        columns: [
-        { name: "field2", type: "float32" },
+          { name: "name", type: "string" },
-        { name: "field3", type: "boolean" },
+          { name: "field1", type: "int32" },
-        {
+          { name: "field2", type: "float32" },
-          name: "field4",
+          { name: "field3", type: "boolean" },
-          type: "categorical",
+          {
-          categories: field4Categories
+            name: "field4",
-        }
+            type: "categorical",
-      ],
+            categories: field4Categories
-      var: [
+          }
-        { name: "name", type: "string" },
+        ]
-        { name: "fieldA", type: "int32" },
+      },
-        { name: "fieldB", type: "float32" },
+      var: {
-        { name: "fieldC", type: "boolean" },
+        index: "name",
-        {
+        columns: [
-          name: "fieldD",
+          { name: "name", type: "string" },
-          type: "categorical",
+          { name: "fieldA", type: "int32" },
-          categories: fieldDCategories
+          { name: "fieldB", type: "float32" },
-        }
+          { name: "fieldC", type: "boolean" },
-      ]
+          {
+            name: "fieldD",
+            type: "categorical",
+            categories: fieldDCategories
+          }
+        ]
+      }
+    },
+    layout: {
+      obs: [{ name: "umap", type: "float32", dims: ["umap_0", "umap_1"] }],
+      var: []
     }
   }
 };
@@ -100,6 +110,14 @@ function encodeTypedArray(builder, uType, uData) {
 }
 
 function encodeMatrix(columns, colIndex = undefined) {
+  /*
+  IMPORTANT: this is not a general purpose encoder.  in particular,
+  it doesn't correctly handle all column index types, nor does it 
+  handle all column typedarray types.
+
+  encodeMatrixFBS in matrix.py is more general.  This is used only
+  as a testing santity check (alt implementation).
+  */
   const utf8Encoder = new TextEncoder("utf-8");
   const builder = new flatbuffers.Builder(1024);
   const cols = _.map(columns, carr => {
@@ -162,29 +180,7 @@ const aLayoutFBSResponse = (() => {
     new Float32Array(nObs).fill(Math.random()),
     new Float32Array(nObs).fill(Math.random())
   ];
-  const builder = new flatbuffers.Builder(1024);
+  return encodeMatrix(coords, ["umap_0", "umap_1"]);
-
-  const cols = _.map(coords, carr => {
-    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
-    NetEncoding.Float32Array.startFloat32Array(builder);
-    NetEncoding.Float32Array.addData(builder, cdv);
-    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
-
-    NetEncoding.Column.startColumn(builder);
-    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
-    NetEncoding.Column.addU(builder, floatArr);
-    return NetEncoding.Column.endColumn(builder);
-  });
-
-  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
-
-  NetEncoding.Matrix.startMatrix(builder);
-  NetEncoding.Matrix.addNRows(builder, nObs);
-  NetEncoding.Matrix.addNCols(builder, coords.length);
-  NetEncoding.Matrix.addColumns(builder, columns);
-  const matrix = NetEncoding.Matrix.endMatrix(builder);
-  builder.finish(matrix);
-  return builder.asUint8Array();
 })();
 
 const aDataObsResponse = {

client/__tests__/util/stateManager/universe.test.js

@@ -53,13 +53,15 @@ describe("createUniverseFromResponse", () => {
 
     expect(universe.obsAnnotations.dims).toEqual([
       nObs,
-      REST.schema.schema.annotations.obs.length
+      REST.schema.schema.annotations.obs.columns.length
     ]);
     expect(universe.obsLayout.dims).toEqual([nObs, 2]);
-    expect(universe.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(universe.obsLayout.colIndex.keys()).toEqual(
+      universe.schema.layout.obs[0].dims
+    );
     expect(universe.varAnnotations.dims).toEqual([
       nVar,
-      REST.schema.schema.annotations.var.length
+      REST.schema.schema.annotations.var.columns.length
     ]);
     expect(universe.varData.isEmpty()).toBeTruthy();
   });

client/__tests__/util/stateManager/world.test.js

@@ -18,18 +18,19 @@ const defaultBigBang = () => {
   /* create unverse, world, crossfilter and dimensionMap */
   /* create universe */
   const universe = Universe.createUniverseFromResponse(
-    REST.config,
+    _.cloneDeep(REST.config),
-    REST.schema,
+    _.cloneDeep(REST.schema),
-    REST.annotationsObs,
+    _.cloneDeep(REST.annotationsObs),
-    REST.annotationsVar,
+    _.cloneDeep(REST.annotationsVar),
-    REST.layoutObs
+    _.cloneDeep(REST.layoutObs)
   );
   /* create world */
   const world = World.createWorldFromEntireUniverse(universe);
   /* create crossfilter */
   const crossfilter = World.createObsDimensions(
     new Crossfilter(world.obsAnnotations),
-    world
+    world,
+    REST.schema.schema.layout.obs[0].dims
   );
 
   return {
@@ -42,11 +43,11 @@ const defaultBigBang = () => {
 describe("createWorldFromEntireUniverse", () => {
   test("create from REST sample", () => {
     const universe = Universe.createUniverseFromResponse(
-      REST.config,
+      _.cloneDeep(REST.config),
-      REST.schema,
+      _.cloneDeep(REST.schema),
-      REST.annotationsObs,
+      _.cloneDeep(REST.annotationsObs),
-      REST.annotationsVar,
+      _.cloneDeep(REST.annotationsVar),
-      REST.layoutObs
+      _.cloneDeep(REST.layoutObs)
     );
     expect(universe).toBeDefined();
 
@@ -138,7 +139,9 @@ describe("createWorldFromCurrentSelection", () => {
     expect(world.obsLayout.rowIndex.keys()).toEqual(
       new Int32Array(matchingIndices)
     );
-    expect(world.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(world.obsLayout.colIndex.keys()).toEqual(
+      world.schema.layout.obs[0].dims
+    );
   });
 });
 
@@ -152,14 +155,18 @@ describe("createObsDimensionMap", () => {
 
     const { crossfilter } = defaultBigBang();
     const annotationNames = _.map(
-      REST.schema.schema.annotations.obs,
+      REST.schema.schema.annotations.obs.columns,
       c => c.name
     );
-    const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
+    const obsIndexColName = REST.schema.schema.annotations.obs.index;
+    const schemaByObsName = _.keyBy(
+      REST.schema.schema.annotations.obs.columns,
+      "name"
+    );
     expect(crossfilter).toBeDefined();
     annotationNames.forEach(name => {
       const dim = crossfilter.dimensions[obsAnnoDimensionName(name)];
-      if (name === "name") {
+      if (name === obsIndexColName) {
         expect(dim).toBeUndefined();
       } else {
         const { type } = schemaByObsName[name];

client/__tests__/util/stateManager/worldUtil.test.js

@@ -1,62 +0,0 @@
-import {
-  countCategoryValues2D,
-  clearCaches
-} from "../../../src/util/stateManager/worldUtil";
-import * as Dataframe from "../../../src/util/dataframe";
-
-describe("WorldUtil cache management", () => {
-  test("empty", () => {
-    const count = countCategoryValues2D(
-      "a",
-      "b",
-      new Dataframe.Dataframe([0, 0], [])
-    );
-    expect(count).toMatchObject(new Map());
-    expect(count.size).toBe(0);
-  });
-
-  test("simple couts", () => {
-    const df = new Dataframe.Dataframe(
-      [3, 2],
-      [[0, 0, 1], [false, true, false]],
-      null,
-      new Dataframe.KeyIndex(["a", "b"])
-    );
-    const count = countCategoryValues2D("a", "b", df);
-    expect(count).toMatchObject(
-      new Map([
-        [0, new Map([[true, 1], [false, 1]])],
-        [1, new Map([[false, 1]])]
-      ])
-    );
-  });
-
-  test("memo cache clear", () => {
-    clearCaches();
-    const df1 = new Dataframe.Dataframe([0, 0], []);
-    const df2 = new Dataframe.Dataframe(
-      [3, 2],
-      [[0, 0, 1], [false, true, false]],
-      null,
-      new Dataframe.KeyIndex(["a", "b"])
-    );
-
-    const count1 = countCategoryValues2D("a", "b", df1);
-    const count2 = countCategoryValues2D("a", "b", df1);
-    const count3 = countCategoryValues2D("a", "b", df1.clone());
-    const count4 = countCategoryValues2D("a", "b", df2);
-
-    clearCaches();
-    const count10 = countCategoryValues2D("a", "b", df1);
-    const count11 = countCategoryValues2D("a", "b", df2);
-
-    expect(count1).toEqual(count2);
-    expect(count1).toEqual(count3);
-    expect(count1).toEqual(count10);
-    expect(count1).not.toBe(count3);
-    expect(count1).not.toBe(count10);
-
-    expect(count4).toEqual(count11);
-    expect(count4).not.toBe(count11);
-  });
-});

client/__tests__/util/typedCrossfilter/sort.test.js

@@ -180,6 +180,7 @@ describe("lowerBound", () => {
     expect(lowerBound([0, 1, 2, 3], 1, 0, 4)).toEqual(1);
     expect(lowerBound([0, 1, 2, 3], 3, 0, 4)).toEqual(3);
     expect(lowerBound([0, 1, 2, 3], 4, 0, 4)).toEqual(4);
+    expect(lowerBound([0, 1, 2, 3], 4, 0, 3)).toEqual(3);
 
     expect(lowerBound([0, 1, 2, 3, 4], -1, 0, 5)).toEqual(0);
     expect(lowerBound([0, 1, 2, 3, 4], 0, 0, 5)).toEqual(0);
@@ -204,6 +205,7 @@ describe("lowerBound", () => {
     expect(lowerBound(new Float32Array([0, 1, 2, 3]), 1, 0, 4)).toEqual(1);
     expect(lowerBound(new Float32Array([0, 1, 2, 3]), 3, 0, 4)).toEqual(3);
     expect(lowerBound(new Float32Array([0, 1, 2, 3]), 4, 0, 4)).toEqual(4);
+    expect(lowerBound(new Float32Array([0, 1, 2, 3]), 4, 0, 3)).toEqual(3);
 
     expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), -1, 0, 5)).toEqual(0);
     expect(lowerBound(new Float32Array([0, 1, 2, 3, 4]), 0, 0, 5)).toEqual(0);

client/__tests__/util/typedCrossfilter/util.test.js

@@ -1,8 +1,8 @@
 import {
-  fillRange,
   sliceByIndex,
   makeSortIndex
 } from "../../../src/util/typedCrossfilter/util";
+import { rangeFill as fillRange } from "../../../src/util/range";
 
 describe("fillRange", () => {
   test("Array", () => {

client/configuration/babel/babel.dev.js

@@ -7,8 +7,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-proposal-optional-chaining",
     "@babel/plugin-proposal-nullish-coalescing-operator"

client/configuration/babel/babel.prod.js

@@ -6,8 +6,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-transform-react-constant-elements",
     "@babel/plugin-transform-runtime",

client/configuration/webpack/webpack.config.dev.js

@@ -37,8 +37,9 @@ module.exports = {
           {
             loader: "css-loader",
             options: {
-              modules: true,
+              modules: {
-              localIdentName: "[name]__[local]___[hash:base64:5]"
+                localIdentName: "[name]__[local]___[hash:base64:5]"
+              }
             }
           }
         ]

client/configuration/webpack/webpack.config.prod.js

@@ -40,9 +40,10 @@ module.exports = {
           {
             loader: "css-loader",
             options: {
-              modules: true,
+              modules: {
-              importLoaders: 1,
+                localIdentName: "[name]__[local]___[hash:base64:5]"
-              localIdentName: "[name]__[local]___[hash:base64:5]"
+              },
+              importLoaders: 1
             }
           }
         ]

client/index.html

@@ -3,7 +3,7 @@
   <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>cellxgene</title>
+    <title>cell&times;gene</title>
     <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
       html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {

client/index_template.html

@@ -3,7 +3,7 @@
 <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>cellxgene</title>
+    <title>cell&times;gene</title>
     <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
         html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {

client/package.json

@@ -1,21 +1,22 @@
 {
   "name": "cellxgene",
-  "version": "0.9.1",
+  "version": "0.12.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
   "scripts": {
     "backend-dev": "python3.6 -m venv cellxgene && source cellxgene/bin/activate && yes | pip uninstall cellxgene || true && pip install -e .. && cellxgene launch ",
+    "backend-dev-anno": "python3.6 -m venv cellxgene && source cellxgene/bin/activate && yes | pip uninstall cellxgene || true && pip install -e .. && cellxgene launch --experimental-label-file labels.csv ",
     "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
     "clean": "rimraf build",
     "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
-    "e2e": "jest --verbose false --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
+    "e2e": "node node_modules/jest/bin/jest.js --verbose false --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
     "lint": "eslint src",
     "smoke-test": "start-server-and-test start-server-for-test :5000 e2e",
     "start": "node server/development.js",
     "start-server-for-test": "cellxgene launch -p 5000 ../example-dataset/pbmc3k.h5ad",
-    "test": "jest",
+    "test": "node node_modules/jest/bin/jest.js",
-    "unit-test": "jest --testPathIgnorePatterns e2e"
+    "unit-test": "node node_modules/jest/bin/jest.js --testPathIgnorePatterns e2e"
   },
   "engineStrict": true,
   "engines": {
@@ -31,89 +32,78 @@
     "eslint-scope": "3.7.1"
   },
   "dependencies": {
-    "@blueprintjs/core": "^3.15.0",
+    "@blueprintjs/core": "^3.18.1",
-    "@blueprintjs/icons": "^3.3.0",
+    "@blueprintjs/icons": "^3.10.0",
-    "@blueprintjs/select": "^3.8.0",
+    "@blueprintjs/select": "^3.10.0",
-    "canvas-fit": "^1.5.0",
     "d3": "^4.10.0",
-    "d3-scale-chromatic": "^1.3.0",
+    "d3-scale-chromatic": "^1.5.0",
-    "flatbuffers": "^1.10.2",
+    "flatbuffers": "^1.11.0",
-    "font-color-contrast": "^1.0.3",
     "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
-    "gl-matrix": "^2.7.1",
+    "gl-matrix": "^3.1.0",
+    "gl-vec3": "^1.1.3",
     "is-number": "^7.0.0",
-    "key-pressed": "0.0.1",
+    "lodash": "^4.17.15",
-    "lodash": "^4.17.4",
+    "memoize-one": "^5.1.1",
-    "memoize-one": "^4.0.0",
+    "react": "^16.9.0",
-    "mouse-position": "^2.0.1",
-    "mouse-pressed": "^1.0.0",
-    "normalize.css": "^8.0.0",
-    "orbit-camera": "^1.0.0",
-    "query-string": "^6.1.0",
-    "react": "^16.6.0",
     "react-autocomplete": "^1.7.2",
-    "react-dom": "^16.6.0",
+    "react-dom": "^16.9.0",
-    "react-helmet": "^5.2.0",
+    "react-helmet": "^5.2.1",
-    "react-icons": "^3.2.2",
+    "react-icons": "^3.7.0",
-    "react-redux": "^5.1.0",
+    "react-redux": "^7.1.1",
-    "redux": "^4.0.1",
+    "redux": "^4.0.4",
-    "redux-devtools-extension": "^2.13.5",
     "redux-thunk": "^2.2.0",
-    "regl": "^1.3.9",
+    "regl": "^1.3.13"
-    "scroll-speed": "^1.0.0",
-    "urijs": "^1.19.0"
   },
   "devDependencies": {
-    "@babel/core": "^7.1.5",
+    "@babel/core": "^7.6.0",
-    "@babel/plugin-proposal-class-properties": "^7.0.0",
+    "@babel/plugin-proposal-class-properties": "^7.5.5",
-    "@babel/plugin-proposal-decorators": "^7.0.0",
+    "@babel/plugin-proposal-decorators": "^7.6.0",
-    "@babel/plugin-proposal-export-namespace-from": "^7.0.0",
+    "@babel/plugin-proposal-export-namespace-from": "^7.5.2",
-    "@babel/plugin-proposal-function-bind": "^7.0.0",
+    "@babel/plugin-proposal-function-bind": "^7.2.0",
-    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.2.0",
+    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.4.4",
-    "@babel/plugin-proposal-optional-chaining": "^7.2.0",
+    "@babel/plugin-proposal-optional-chaining": "^7.6.0",
-    "@babel/plugin-transform-react-constant-elements": "^7.0.0",
+    "@babel/plugin-transform-react-constant-elements": "^7.6.0",
-    "@babel/plugin-transform-runtime": "^7.1.0",
+    "@babel/plugin-transform-runtime": "^7.6.0",
-    "@babel/preset-env": "^7.1.5",
+    "@babel/preset-env": "^7.6.0",
     "@babel/preset-react": "^7.0.0",
-    "@babel/register": "^7.0.0",
+    "@babel/register": "^7.6.0",
-    "@babel/runtime": "^7.1.5",
+    "@babel/runtime": "^7.6.0",
-    "babel-core": "^7.0.0-bridge.0",
+    "babel-eslint": "^10.0.3",
-    "babel-eslint": "^10.0.1",
+    "babel-jest": "^24.9.0",
-    "babel-jest": "^23.6.0",
+    "babel-loader": "^8.0.6",
-    "babel-loader": "^8.0.0",
+    "babel-preset-modern-browsers": "^14.0.0",
-    "babel-preset-modern-browsers": "^12.0.0",
     "chalk": "^2.4.2",
     "connect-history-api-fallback": "^1.6.0",
-    "copy-webpack-plugin": "^4.6.0",
+    "copy-webpack-plugin": "^5.0.4",
-    "css-loader": "^1.0.1",
+    "css-loader": "^3.2.0",
-    "eslint": "^5.13.0",
+    "eslint": "^6.4.0",
-    "eslint-config-airbnb": "^17.1.0",
+    "eslint-config-airbnb": "^18.0.1",
-    "eslint-config-prettier": "^4.0.0",
+    "eslint-config-prettier": "^6.3.0",
-    "eslint-loader": "^2.1.2",
+    "eslint-loader": "^3.0.0",
     "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.16.0",
+    "eslint-plugin-import": "^2.18.2",
-    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-jest": "^22.17.0",
-    "eslint-plugin-jsx-a11y": "^6.2.1",
+    "eslint-plugin-jsx-a11y": "^6.2.3",
-    "eslint-plugin-react": "^7.12.4",
+    "eslint-plugin-react": "^7.14.3",
-    "express": "^4.14.0",
+    "express": "^4.17.1",
-    "file-loader": "^2.0.0",
+    "file-loader": "^4.2.0",
     "html-webpack-inline-source-plugin": "0.0.10",
     "html-webpack-plugin": "^3.2.0",
-    "jest": "^24.1.0",
+    "jest": "^24.9.0",
-    "jest-puppeteer": "^4.1.0",
+    "jest-puppeteer": "^4.3.0",
     "json-loader": "^0.5.4",
-    "mini-css-extract-plugin": "^0.4.1",
+    "mini-css-extract-plugin": "^0.8.0",
-    "puppeteer": "^1.12.1",
+    "puppeteer": "^1.20.0",
-    "rimraf": "^2.6.3",
+    "rimraf": "^3.0.0",
     "serve-favicon": "^2.3.0",
-    "start-server-and-test": "^1.7.11",
+    "start-server-and-test": "^1.10.2",
-    "style-loader": "^0.23.1",
+    "style-loader": "^1.0.0",
     "sw-precache-webpack-plugin": "^0.11.5",
-    "url-loader": "^1.1.0",
+    "url-loader": "^2.1.0",
-    "webpack": "^4.25.1",
+    "webpack": "^4.40.2",
-    "webpack-cli": "^3.1.0",
+    "webpack-cli": "^3.3.8",
-    "webpack-dev-middleware": "^3.1.3"
+    "webpack-dev-middleware": "^3.7.1"
   },
   "jest": {
     "testMatch": [
@@ -133,16 +123,23 @@
         ],
         "plugins": [
           "@babel/plugin-proposal-function-bind",
-          "@babel/plugin-proposal-class-properties",
           [
             "@babel/plugin-proposal-decorators",
             {
               "legacy": true
             }
           ],
+          [
+            "@babel/plugin-proposal-class-properties",
+            {
+              "loose": true
+            }
+          ],
           "@babel/plugin-proposal-export-namespace-from",
           "@babel/plugin-transform-react-constant-elements",
-          "@babel/plugin-transform-runtime"
+          "@babel/plugin-transform-runtime",
+          "@babel/plugin-proposal-optional-chaining",
+          "@babel/plugin-proposal-nullish-coalescing-operator"
         ]
       }
     }

client/src/actions/index.js

@@ -1,7 +1,7 @@
 // jshint esversion: 6
 import _ from "lodash";
 import * as globals from "../globals";
-import { Universe } from "../util/stateManager";
+import { Universe, MatrixFBS } from "../util/stateManager";
 import {
   catchErrorsWrap,
   doJsonRequest,
@@ -21,24 +21,32 @@ const doInitialDataLoad = () =>
     dispatch({ type: "initial data load start" });
 
     try {
-      const requestJson = _(["config", "schema"])
+      /*
+      Step 1 - config & schema, all JSON
+      */
+      const requestJson = ["config", "schema"]
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doJsonRequest(url))
+        .map(url => doJsonRequest(url));
-        .value();
+      const stepOneResults = await Promise.all(requestJson);
-      const requestBinary = _([
-        "annotations/obs",
-        "annotations/var?annotation-name=name",
-        "layout/obs"
-      ])
-        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doBinaryRequest(url))
-        .value();
-
-      const results = await Promise.all(_.concat(requestJson, requestBinary));
-
       /* set config defaults */
-      const config = { ...globals.configDefaults, ...results[0].config };
+      const config = { ...globals.configDefaults, ...stepOneResults[0].config };
-      const [, schema, obsAnno, varAnno, obsLayout] = [...results];
+      const schema = stepOneResults[1];
+
+      /*
+      Step 2 - dataframes, all binary.  NOTE: uses results of step 1.
+      */
+      /* only load names for var annotations, if possible*/
+      const varIndexName = schema?.schema?.annotations?.var?.index;
+      const varAnnotationsQuery = varIndexName
+        ? `?annotation-name=${varIndexName}`
+        : "";
+      const varAnnotationsURL = `annotations/var${varAnnotationsQuery}`;
+      const requestBinary = ["annotations/obs", varAnnotationsURL, "layout/obs"]
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doBinaryRequest(url));
+      const stepTwoResults = await Promise.all(requestBinary);
+      const [obsAnno, varAnno, obsLayout] = [...stepTwoResults];
+
       const universe = Universe.createUniverseFromResponse(
         config,
         schema,
@@ -91,6 +99,10 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  const state = getState();
+  const { universe } = state;
+  const varIndexName = universe.schema.annotations.var.index;
+
   /* helper for this function only */
   const fetchData = async geneNames => {
     const res = await fetch(
@@ -100,7 +112,7 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
         body: JSON.stringify({
           filter: {
             var: {
-              annotation_value: [{ name: "name", values: geneNames }]
+              annotation_value: [{ name: varIndexName, values: geneNames }]
             }
           }
         }),
@@ -123,8 +135,6 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
     return Universe.convertDataFBStoObject(universe, data);
   };
 
-  const state = getState();
-  const { universe } = state;
   /* preload data already in cache */
   let expressionData = _.transform(
     genes,
@@ -241,6 +251,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     */
     const state = getState();
     const { universe } = state;
+    const varIndexName = universe.schema.annotations.var.index;
 
     // Legal values are null, Array or TypedArray.  Null is initial state.
     if (!set1) set1 = [];
@@ -277,7 +288,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     const data = await res.json();
     // result is [ [varIdx, ...], ... ]
     const topNGenes = _.map(data, r =>
-      universe.varAnnotations.at(r[0], "name")
+      universe.varAnnotations.at(r[0], varIndexName)
     );
 
     /*
@@ -329,11 +340,57 @@ const resetInterface = () => (dispatch, getState) => {
   });
 };
 
+const saveObsAnnotations = () => async (dispatch, getState) => {
+  const { universe } = getState();
+  const { obsAnnotations, schema } = universe;
+
+  dispatch({
+    type: "writable obs annotations - save started"
+  });
+
+  const writableAnnotations = schema.annotations.obs.columns
+    .filter(s => s.writable)
+    .map(s => s.name);
+  const df = obsAnnotations.subset(null, writableAnnotations);
+  const matrix = MatrixFBS.encodeMatrixFBS(df);
+  try {
+    const res = await fetch(
+      `${globals.API.prefix}${globals.API.version}annotations/obs`,
+      {
+        method: "PUT",
+        body: matrix,
+        headers: new Headers({
+          "Content-Type": "application/octet-stream"
+        })
+      }
+    );
+    if (res.ok) {
+      dispatch({
+        type: "writable obs annotations - save complete",
+        obsAnnotations
+      });
+    } else {
+      dispatch({
+        type: "writable obs annotations - save error",
+        message: `HTTP error ${res.status} - ${res.statusText}`,
+        res
+      });
+    }
+  } catch (error) {
+    dispatch({
+      type: "writable obs annotations - save error",
+      message: error.toString(),
+      error
+    });
+  }
+};
+
 export default {
   regraph,
   resetInterface,
   requestSingleGeneExpressionCountsForColoringPOST,
   requestDifferentialExpression,
   requestUserDefinedGene,
-  doInitialDataLoad
+  doInitialDataLoad,
+  saveObsAnnotations
 };

client/src/components/app.js

@@ -4,9 +4,12 @@ import Helmet from "react-helmet";
 import { connect } from "react-redux";
 
 import Container from "./framework/container";
-import LeftSideBar from "./leftsidebar";
+import LeftSideBar from "./leftSidebar";
 import Legend from "./continuousLegend";
 import Graph from "./graph/graph";
+import MenuBar from "./menubar";
+import Autosave from "./autosave";
+
 import actions from "../actions";
 
 @connect(state => ({
@@ -71,18 +74,24 @@ class App extends React.Component {
             loading cellxgene
           </div>
         ) : null}
-        <div>
+        {error ? (
-          {loading ? null : <LeftSideBar />}
           <div
             style={{
-              padding: 15,
+              position: "fixed",
-              width: 1440 - 410 /* but responsive */,
+              fontWeight: 500,
-              marginLeft: 350 /* but responsive */
+              top: window.innerHeight / 2,
+              left: window.innerWidth / 2 - 50
             }}
           >
-            {loading ? null : <Graph key={graphRenderCounter} />}
+            error loading
-            <Legend />
           </div>
+        ) : null}
+        <div>
+          {loading ? null : <LeftSideBar />}
+          {loading ? null : <MenuBar />}
+          {loading ? null : <Graph key={graphRenderCounter} />}
+          {loading ? null : <Autosave />}
+          <Legend />
         </div>
       </Container>
     );

client/src/components/autosave/index.js

@@ -0,0 +1,81 @@
+import React from "react";
+import { connect } from "react-redux";
+
+import actions from "../../actions";
+
+@connect(state => ({
+  universe: state.universe,
+  obsAnnotations: state.universe.obsAnnotations,
+  saveInProgress: state.autosave?.saveInProgress ?? false,
+  lastSavedObsAnnotations: state.autosave?.lastSavedObsAnnotations,
+  error: state.autosave?.error,
+  writableCategoriesEnabled: state.config?.parameters?.["label_file"] ?? false
+}))
+class Autosave extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      timer: null
+    };
+  }
+
+  componentDidMount() {
+    const { writableCategoriesEnabled } = this.props;
+
+    let { timer } = this.state;
+    if (timer) clearInterval(timer);
+    if (writableCategoriesEnabled) {
+      timer = setInterval(this.tick, 2500);
+    } else {
+      timer = null;
+    }
+    this.setState({ timer });
+  }
+
+  componentWillUnmount() {
+    const { timer } = this.state;
+    if (timer) this.clearInterval(timer);
+  }
+
+  tick = () => {
+    const { dispatch, saveInProgress } = this.props;
+    if (this.needToSave() && !saveInProgress) {
+      dispatch(actions.saveObsAnnotations());
+    }
+  };
+
+  needToSave = () => {
+    /* return true if we need to save, false if we don't */
+    const { obsAnnotations, lastSavedObsAnnotations } = this.props;
+    return (
+      lastSavedObsAnnotations && obsAnnotations !== lastSavedObsAnnotations
+    );
+  };
+
+  statusMessage() {
+    const { error } = this.props;
+    if (error) {
+      return `Autosave error: ${error}`;
+    }
+    return this.needToSave() ? "Unsaved" : "All saved";
+  }
+
+  render() {
+    const { writableCategoriesEnabled } = this.props;
+    return writableCategoriesEnabled ? (
+      <div
+        id="autosave"
+        style={{
+          position: "fixed",
+          display: "inherit",
+          right: 5,
+          bottom: 5
+        }}
+      >
+        {this.statusMessage()}
+      </div>
+    ) : null;
+  }
+}
+
+export default Autosave;

client/src/components/brushableHistogram/index.js

@@ -13,19 +13,25 @@ import * as globals from "../../globals";
 import actions from "../../actions";
 import { makeContinuousDimensionName } from "../../util/nameCreators";
 
-@connect(state => ({
+@connect((state, ownProps) => {
-  world: state.world,
+  const { isObs, isUserDefined, isDiffExp, field } = ownProps;
-  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  const myName = makeContinuousDimensionName(
-  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+    { isObs, isUserDefined, isDiffExp },
-  continuousSelection: state.continuousSelection,
+    field
-  differential: state.differential,
+  );
-  colorAccessor: state.colors.colorAccessor
+  return {
-}))
+    world: state.world,
-class HistogramBrush extends React.Component {
+    scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+    scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+    continuousSelectionRange: state.continuousSelection[myName],
+    colorAccessor: state.colors.colorAccessor
+  };
+})
+class HistogramBrush extends React.PureComponent {
   static getColumn(world, field, clipped = true) {
     /*
-    Return the underlying Dataframe column for our field.   By default, 
+    Return the underlying Dataframe column for our field.   By default,
-    returns the clipped column.   If clipped===false, will return the 
+    returns the clipped column.   If clipped===false, will return the
     unclipped column.
     */
     const obsAnnotations = clipped
@@ -84,7 +90,7 @@ class HistogramBrush extends React.Component {
   }
 
   componentDidUpdate(prevProps) {
-    const { field, world, continuousSelection } = this.props;
+    const { field, world } = this.props;
     const { x, y, bins, svgRef } = this._histogram;
     let { brushXselection, brushX } = this.state;
     let forceBrushUpdate = false;
@@ -112,16 +118,8 @@ class HistogramBrush extends React.Component {
     if the selection has changed, ensure that the brush correctly reflects
     the underlying selection.
     */
-    if (
+    const { continuousSelectionRange: range } = this.props;
-      forceBrushUpdate ||
+    if (forceBrushUpdate || range !== prevProps.continuousSelectionRange) {
-      continuousSelection !== prevProps.continuousSelection
-    ) {
-      const { isObs, isUserDefined, isDiffExp } = this.props;
-      const myName = makeContinuousDimensionName(
-        { isObs, isUserDefined, isDiffExp },
-        field
-      );
-      const range = continuousSelection[myName];
       if (brushXselection) {
         const selection = d3.brushSelection(brushXselection.node());
         if (!range && selection) {
@@ -357,14 +355,24 @@ class HistogramBrush extends React.Component {
       .append("g")
       .attr("class", "axis axis--x")
       .attr("transform", `translate(0,${this.height - this.marginBottom})`)
-      .call(d3.axisBottom(x).ticks(5));
+      .call(
+        d3
+          .axisBottom(x)
+          .ticks(5)
+          .tickFormat(d3.format(".0s"))
+      );
 
     /* Y AXIS */
     svg
       .append("g")
       .attr("class", "axis axis--y")
       .attr("transform", `translate(${this.width - this.marginRight},0)`)
-      .call(d3.axisRight(y).ticks(3));
+      .call(
+        d3
+          .axisRight(y)
+          .ticks(3)
+          .tickFormat(d3.format(".0s"))
+      );
 
     /* axis style */
     svg.selectAll(".axis text").style("fill", "rgb(80,80,80)");
@@ -461,7 +469,11 @@ class HistogramBrush extends React.Component {
               remove
             </Button>
           ) : null}
-          <Tooltip content="Use as color scale" position="bottom">
+          <Tooltip
+            content="Use as color scale"
+            position="bottom"
+            hoverOpenDelay={globals.tooltipHoverOpenDelay}
+          >
             <Button
               onClick={this.handleColorAction.bind(this)}
               active={colorAccessor === field}

client/src/components/categorical/categorical.css

@@ -0,0 +1,3 @@
+:local(.value):hover {
+  background: rgba(167, 182, 194, 0.3);
+}

client/src/components/categorical/categorical.js

@@ -1,34 +1,176 @@
 // jshint esversion: 6
 import React from "react";
 import _ from "lodash";
+import {
+  Button,
+  Tooltip,
+  InputGroup,
+  Dialog,
+  Classes,
+  MenuItem
+} from "@blueprintjs/core";
+import { Select } from "@blueprintjs/select";
 import { connect } from "react-redux";
 import * as globals from "../../globals";
 import Category from "./category";
 
 @connect(state => ({
-  categoricalSelection: state.categoricalSelection
+  categoricalSelection: state.categoricalSelection,
+  writableCategoriesEnabled: state.config?.parameters?.["label_file"] ?? false,
+  schema: state.world?.schema
 }))
 class Categories extends React.Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      createAnnoModeActive: false,
+      newCategoryText: "",
+      categoryToDuplicate: null
+    };
+  }
+
+  handleCreateUserAnno = () => {
+    const { dispatch } = this.props;
+    const { newCategoryText, categoryToDuplicate } = this.state;
+    dispatch({
+      type: "annotation: create category",
+      data: newCategoryText,
+      categoryToDuplicate
+    });
+    this.setState({
+      createAnnoModeActive: false,
+      categoryToDuplicate: null,
+      newCategoryText: ""
+    });
+  };
+
+  handleEnableAnnoMode = () => {
+    this.setState({ createAnnoModeActive: true });
+  };
+
+  handleDisableAnnoMode = () => {
+    this.setState({ createAnnoModeActive: false });
+  };
+
+  handleModalDuplicateCategorySelection = d => {
+    this.setState({ categoryToDuplicate: d });
+  };
+
   render() {
-    const { categoricalSelection } = this.props;
+    const { createAnnoModeActive, categoryToDuplicate } = this.state;
+    const {
+      categoricalSelection,
+      writableCategoriesEnabled,
+      schema
+    } = this.props;
     if (!categoricalSelection) return null;
 
+    /* all names, sorted in display order.  Will be rendered in this order */
+    const allCategoryNames = Object.keys(categoricalSelection).sort();
+
     return (
       <div
         style={{
           padding: globals.leftSidebarSectionPadding
         }}
       >
-        <p
+        {/* READ ONLY CATEGORICAL FIELDS */}
-          style={Object.assign({}, globals.leftSidebarSectionHeading, {
+        {/* this is duplicative but flat, could be abstracted */}
-            marginTop: 4
+        {_.map(allCategoryNames, catName =>
-          })}
+          !schema.annotations.obsByName[catName].writable ? (
-        >
+            <Category
-          Categorical Metadata
+              key={catName}
-        </p>
+              metadataField={catName}
-        {_.map(categoricalSelection, (catState, catName) => (
+              createAnnoModeActive={createAnnoModeActive}
-          <Category key={catName} metadataField={catName} />
+              isUserAnno={false}
-        ))}
+            />
+          ) : null
+        )}
+        {/* WRITEABLE FIELDS */}
+        {_.map(allCategoryNames, catName =>
+          schema.annotations.obsByName[catName].writable ? (
+            <Category
+              key={catName}
+              metadataField={catName}
+              createAnnoModeActive={createAnnoModeActive}
+              isUserAnno
+            />
+          ) : null
+        )}
+        {writableCategoriesEnabled ? (
+          <div>
+            <Dialog
+              icon="tag"
+              title="Create new category"
+              isOpen={createAnnoModeActive}
+              onClose={this.handleDisableAnnoMode}
+            >
+              <form
+                onSubmit={e => {
+                  e.preventDefault();
+                  this.handleCreateUserAnno();
+                }}
+              >
+                <div className={Classes.DIALOG_BODY}>
+                  <div style={{ marginBottom: 20 }}>
+                    <p>New, unique category name:</p>
+                    <InputGroup
+                      autoFocus
+                      onChange={e =>
+                        this.setState({ newCategoryText: e.target.value })
+                      }
+                      leftIcon="tag"
+                    />
+                  </div>
+                  <p>
+                    Optionally duplicate all labels & cell assignments from
+                    existing category into new category:
+                  </p>
+                  <Select
+                    items={allCategoryNames}
+                    filterable={false}
+                    itemRenderer={(d, { handleClick }) => {
+                      return (
+                        <MenuItem onClick={handleClick} key={d} text={d} />
+                      );
+                    }}
+                    noResults={<MenuItem disabled text="No results." />}
+                    onItemSelect={d => {
+                      this.handleModalDuplicateCategorySelection(d);
+                    }}
+                  >
+                    {/* children become the popover target; render value here */}
+                    <Button
+                      text={
+                        categoryToDuplicate || "None (all cells 'unassigned')"
+                      }
+                      rightIcon="double-caret-vertical"
+                    />
+                  </Select>
+                </div>
+                <div className={Classes.DIALOG_FOOTER}>
+                  <div className={Classes.DIALOG_FOOTER_ACTIONS}>
+                    <Tooltip content="Close this dialog without creating a category.">
+                      <Button onClick={this.handleDisableAnnoMode}>
+                        Cancel
+                      </Button>
+                    </Tooltip>
+                    <Button
+                      onClick={this.handleCreateUserAnno}
+                      intent="primary"
+                      type="submit"
+                    >
+                      Create new category
+                    </Button>
+                  </div>
+                </div>
+              </form>
+            </Dialog>
+            <Button onClick={this.handleEnableAnnoMode} intent="primary">
+              Create new category
+            </Button>
+          </div>
+        ) : null}
       </div>
     );
   }

client/src/components/categorical/category.js

@@ -2,7 +2,19 @@ import React from "react";
 import _ from "lodash";
 import { connect } from "react-redux";
 import { FaChevronRight, FaChevronDown } from "react-icons/fa";
-import { Button, Tooltip } from "@blueprintjs/core";
+import {
+  Button,
+  Tooltip,
+  InputGroup,
+  Menu,
+  Dialog,
+  MenuItem,
+  Popover,
+  Classes,
+  Icon,
+  Position,
+  PopoverInteractionKind
+} from "@blueprintjs/core";
 
 import * as globals from "../../globals";
 import Value from "./value";
@@ -10,14 +22,18 @@ import sortedCategoryValues from "./util";
 
 @connect(state => ({
   colorAccessor: state.colors.colorAccessor,
-  categoricalSelection: state.categoricalSelection
+  categoricalSelection: state.categoricalSelection,
+  annotations: state.annotations,
+  universe: state.universe
 }))
 class Category extends React.Component {
   constructor(props) {
     super(props);
     this.state = {
       isChecked: true,
-      isExpanded: false
+      isExpanded: false,
+      newCategoryText: "",
+      newLabelText: ""
     };
   }
 
@@ -27,10 +43,10 @@ class Category extends React.Component {
       const cat = categoricalSelection[metadataField];
       const categoryCount = {
         // total number of categories in this dimension
-        totalCatCount: cat.numCategories,
+        totalCatCount: cat.numCategoryValues,
         // number of selected options in this category
         selectedCatCount: _.reduce(
-          cat.categorySelected,
+          cat.categoryValueSelected,
           (res, cond) => (cond ? res + 1 : res),
           0
         )
@@ -51,6 +67,74 @@ class Category extends React.Component {
     }
   }
 
+  activateAddNewLabelMode = () => {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "annotation: activate add new label mode",
+      data: metadataField
+    });
+  };
+
+  disableAddNewLabelMode = () => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "annotation: disable add new label mode"
+    });
+  };
+
+  handleAddNewLabelToCategory = () => {
+    const { dispatch, metadataField } = this.props;
+    const { newLabelText } = this.state;
+    /*
+    XXX TODO - temporary code generates random label string.   Remove
+    when the label creation UI is implemented.
+
+    const { newLabelText } = this.state;
+    */
+    // const newLabelText = `label${Math.random()}`;
+    dispatch({
+      type: "annotation: add new label to category",
+      metadataField,
+      newLabelText
+    });
+    this.setState({ newLabelText: "" });
+  };
+
+  activateEditCategoryMode = () => {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "annotation: activate category edit mode",
+      data: metadataField
+    });
+  };
+
+  disableEditCategoryMode = () => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "annotation: disable category edit mode"
+    });
+  };
+
+  handleEditCategory = () => {
+    const { dispatch, metadataField } = this.props;
+    const { newCategoryText } = this.state;
+
+    dispatch({
+      type: "annotation: category edited",
+      metadataField,
+      newCategoryText,
+      data: newCategoryText
+    });
+  };
+
+  handleDeleteCategory = () => {
+    const { dispatch, metadataField } = this.props;
+    dispatch({
+      type: "annotation: delete category",
+      metadataField
+    });
+  };
+
   handleColorChange = () => {
     const { dispatch, metadataField } = this.props;
     dispatch({
@@ -88,12 +172,13 @@ class Category extends React.Component {
   }
 
   renderCategoryItems() {
-    const { categoricalSelection, metadataField } = this.props;
+    const { categoricalSelection, metadataField, isUserAnno } = this.props;
 
     const cat = categoricalSelection[metadataField];
-    const optTuples = sortedCategoryValues([...cat.categoryIndices]);
+    const optTuples = sortedCategoryValues([...cat.categoryValueIndices]);
     return _.map(optTuples, (tuple, i) => (
       <Value
+        isUserAnno={isUserAnno}
         optTuples={optTuples}
         key={tuple[1]}
         metadataField={metadataField}
@@ -104,9 +189,17 @@ class Category extends React.Component {
   }
 
   render() {
-    const { isExpanded, isChecked } = this.state;
+    const { isExpanded, isChecked, newLabelText, newCategoryText } = this.state;
-    const { metadataField, colorAccessor, categoricalSelection } = this.props;
+    const {
+      metadataField,
+      colorAccessor,
+      categoricalSelection,
+      isUserAnno,
+      annotations,
+      universe
+    } = this.props;
     const { isTruncated } = categoricalSelection[metadataField];
+
     return (
       <div
         style={{
@@ -144,7 +237,6 @@ class Category extends React.Component {
               <span className="bp3-control-indicator" />
               {""}
             </label>
-
             <span
               data-testid={`category-expand-${metadataField}`}
               style={{
@@ -152,10 +244,58 @@ class Category extends React.Component {
                 display: "inline-block"
               }}
               onClick={() => {
-                this.setState({ isExpanded: !isExpanded });
+                const editingCategory =
+                  annotations.isEditingCategoryName &&
+                  annotations.categoryBeingEdited === metadataField;
+                if (!editingCategory) {
+                  this.setState({ isExpanded: !isExpanded });
+                }
               }}
             >
-              {metadataField}
+              {isUserAnno ? (
+                <Icon style={{ marginRight: 5 }} icon="tag" iconSize={16} />
+              ) : null}
+
+              {annotations.isEditingCategoryName &&
+              annotations.categoryBeingEdited === metadataField ? (
+                <form
+                  style={{ display: "inline-block" }}
+                  onSubmit={e => {
+                    e.preventDefault();
+                    this.handleEditCategory();
+                  }}
+                >
+                  <InputGroup
+                    style={{ position: "relative", top: -1 }}
+                    ref={input => {
+                      this.editableCategoryInput = input;
+                    }}
+                    small
+                    autoFocus
+                    onChange={e => {
+                      this.setState({
+                        newCategoryText: e.target.value
+                      });
+                    }}
+                    defaultValue={metadataField}
+                    rightElement={
+                      <Button
+                        minimal
+                        disabled={newCategoryText.length === 0}
+                        style={{ position: "relative", top: -1 }}
+                        type="button"
+                        icon="small-tick"
+                        data-testclass="submitCategoryNameEdit"
+                        data-testid="submitCategoryNameEdit"
+                        onClick={this.handleEditCategory}
+                      />
+                    }
+                  />
+                </form>
+              ) : (
+                metadataField
+              )}
+
               {isExpanded ? (
                 <FaChevronDown
                   data-testclass="category-expand-is-expanded"
@@ -169,16 +309,113 @@ class Category extends React.Component {
               )}
             </span>
           </div>
-          <Tooltip content="Use as color scale" position="bottom">
+          <div>
-            <Button
+            {isUserAnno ? (
-              data-testclass="colorby"
+              <>
-              data-testid={`colorby-${metadataField}`}
+                <Dialog
-              onClick={this.handleColorChange}
+                  icon="tag"
-              active={colorAccessor === metadataField}
+                  title="Add new label"
-              intent={colorAccessor === metadataField ? "primary" : "none"}
+                  isOpen={annotations.isAddingNewLabel}
-              icon="tint"
+                  onClose={this.disableAddNewLabelMode}
-            />
+                >
-          </Tooltip>
+                  <form
+                    onSubmit={e => {
+                      e.preventDefault();
+                      this.handleAddNewLabelToCategory();
+                    }}
+                  >
+                    <div className={Classes.DIALOG_BODY}>
+                      <div style={{ marginBottom: 20 }}>
+                        <p>New, unique label name:</p>
+                        <InputGroup
+                          autoFocus
+                          onChange={e =>
+                            this.setState({ newLabelText: e.target.value })
+                          }
+                          leftIcon="tag"
+                        />
+                      </div>
+                    </div>
+                    <div className={Classes.DIALOG_FOOTER}>
+                      <div className={Classes.DIALOG_FOOTER_ACTIONS}>
+                        <Tooltip content="Close this dialog without adding a label.">
+                          <Button onClick={this.disableAddNewLabelMode}>
+                            Cancel
+                          </Button>
+                        </Tooltip>
+                        <Button
+                          disabled={
+                            newLabelText.length === 0 ||
+                            universe.schema.annotations.obsByName[
+                              metadataField
+                            ].categories.indexOf(newLabelText) !== -1
+                          }
+                          onClick={this.handleAddNewLabelToCategory}
+                          intent="primary"
+                          type="submit"
+                        >
+                          Add new label to category
+                        </Button>
+                      </div>
+                    </div>
+                  </form>
+                </Dialog>
+                <Popover
+                  interactionKind={PopoverInteractionKind.HOVER}
+                  boundary="window"
+                  position={Position.RIGHT_TOP}
+                  content={
+                    <Menu>
+                      <MenuItem
+                        icon="tag"
+                        data-testclass="handleAddNewLabelToCategory"
+                        data-testid={`handleAddNewLabelToCategory-${metadataField}`}
+                        onClick={this.activateAddNewLabelMode}
+                        text="Add a new label to this category"
+                      />
+                      <MenuItem
+                        icon="edit"
+                        data-testclass="activateEditCategoryMode"
+                        data-testid={`activateEditCategoryMode-${metadataField}`}
+                        onClick={this.activateEditCategoryMode}
+                        text="Edit this category's name"
+                      />
+                      <MenuItem
+                        icon="delete"
+                        intent="danger"
+                        data-testclass="handleDeleteCategory"
+                        data-testid={`handleDeleteCategory-${metadataField}`}
+                        onClick={this.handleDeleteCategory}
+                        text="Delete this category, all associated labels, and remove all cell assignments"
+                      />
+                    </Menu>
+                  }
+                >
+                  <Button
+                    style={{ marginLeft: 0 }}
+                    data-testclass="seeActions"
+                    data-testid={`seeActions-${metadataField}`}
+                    icon="more"
+                    minimal
+                  />
+                </Popover>
+              </>
+            ) : null}
+            <Tooltip
+              content="Use as color scale"
+              position="bottom"
+              hoverOpenDelay={globals.tooltipHoverOpenDelay}
+            >
+              <Button
+                data-testclass="colorby"
+                data-testid={`colorby-${metadataField}`}
+                onClick={this.handleColorChange}
+                active={colorAccessor === metadataField}
+                intent={colorAccessor === metadataField ? "primary" : "none"}
+                icon="tint"
+              />
+            </Tooltip>
+          </div>
         </div>
         <div style={{ marginLeft: 26 }}>
           {isExpanded ? this.renderCategoryItems() : null}

client/src/components/categorical/occupancy.js

@@ -1,69 +1,199 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import * as d3 from "d3";
+import {
+  Popover,
+  PopoverInteractionKind,
+  Position,
+  Classes
+} from "@blueprintjs/core";
 
 @connect()
 class Occupancy extends React.Component {
+  _WIDTH = 100;
+
+  _HEIGHT = 11;
+
+  createHistogram = () => {
+    /* 
+      Knowing that colorScale is based off continous data, 
+      createHistogram fetches the continous data in relation to the cells releveant to the catagory value.
+      It then seperates that data into 50 bins for drawing the mini-histogram
+    */
+    const {
+      world,
+      metadataField,
+      colorAccessor,
+      category,
+      categoryIndex
+    } = this.props;
+
+    if (!this.canvas) return;
+
+    const groupBy = world.obsAnnotations.col(metadataField);
+
+    const col =
+      world.obsAnnotations.col(colorAccessor) ||
+      world.varData.col(colorAccessor);
+
+    const range = col.summarize();
+
+    const histogramMap = col.histogram(
+      50,
+      [range.min, range.max],
+      groupBy
+    ); /* Because the signature changes we really need different names for histogram to differentiate signatures  */
+
+    const categoryValue = category.categoryValues[categoryIndex];
+    const bins = histogramMap.has(categoryValue)
+      ? histogramMap.get(categoryValue)
+      : new Array(50).fill(0);
+
+    const xScale = d3
+      .scaleLinear()
+      .domain([0, bins.length])
+      .range([0, this._WIDTH]);
+
+    const largestBin = Math.max(...bins);
+
+    const yScale = d3
+      .scaleLinear()
+      .domain([0, largestBin])
+      .range([0, this._HEIGHT]);
+
+    const ctx = this.canvas.getContext("2d");
+
+    ctx.fillStyle = "#000";
+
+    let x;
+    let y;
+
+    const rectWidth = this._WIDTH / bins.length;
+
+    for (let i = 0, { length } = bins; i < length; i += 1) {
+      x = xScale(i);
+      y = yScale(bins[i]);
+      ctx.fillRect(x, this._HEIGHT - y, rectWidth, y);
+    }
+  };
+
+  createOccupancyStack = () => {
+    /* 
+      Knowing that the color scale is based off of catagorical data, 
+      createOccupancyStack obtains a map showing the number if cells per colored value
+      Using the colorScale a stack of colored bars is drawn representing the map
+     */
+    const {
+      world,
+      metadataField,
+      colorAccessor,
+      category,
+      categoryIndex,
+      schema,
+      colorScale
+    } = this.props;
+
+    const ctx = this.canvas?.getContext("2d");
+
+    if (!ctx) return;
+
+    const groupBy = world.obsAnnotations.col(metadataField);
+    const occupancyMap = world.obsAnnotations
+      .col(colorAccessor)
+      .histogram(groupBy);
+
+    const occupancy = occupancyMap.get(category.categoryValues[categoryIndex]);
+
+    if (occupancy && occupancy.size > 0) {
+      // not all categories have occupancy, so occupancy may be undefined.
+      const x = d3
+        .scaleLinear()
+        /* get all the keys d[1] as an array, then find the sum */
+        .domain([0, d3.sum(Array.from(occupancy.values()))])
+        .range([0, this._WIDTH]);
+      const categories =
+        schema.annotations.obsByName[colorAccessor]?.categories;
+
+      let currentOffset = 0;
+      const dfColumn = world.obsAnnotations.col(colorAccessor);
+      const categoryValues = dfColumn.summarize().categories;
+
+      let o;
+      let scaledValue;
+      let value;
+
+      for (let i = 0, { length } = categoryValues; i < length; i += 1) {
+        value = categoryValues[i];
+        o = occupancy.get(value);
+        scaledValue = x(o);
+        ctx.fillStyle = o
+          ? colorScale(categories.indexOf(value))
+          : "rgb(255,255,255)";
+        ctx.fillRect(currentOffset, 0, o ? scaledValue : 0, this._HEIGHT);
+        currentOffset += o ? scaledValue : 0;
+      }
+    }
+  };
+
   render() {
     const {
-      occupancy,
-      colorScale,
-      categoricalSelection,
       colorAccessor,
-      schema
+      categoricalSelection,
+      category,
+      categoryIndex
     } = this.props;
-    const width = 100;
-    const height = 11;
 
-    const categories = _.filter(schema.annotations.obs, {
+    this.canvas?.getContext("2d").clearRect(0, 0, this._WIDTH, this._HEIGHT);
-      name: colorAccessor
-    })[0].categories;
 
-    const x = d3
+    const colorByIsCatagoricalData = !!categoricalSelection[colorAccessor];
-      .scaleLinear()
-      /* get all the keys d[1] as an array, then find the sum */
-      .domain([0, d3.sum(Array.from(occupancy, d => d[1]))])
-      .range([0, width]);
-
-    let currentOffset = 0;
-
-    const stacks = categoricalSelection[colorAccessor].categoryValues.map(d => {
-      const o = occupancy.get(d);
-
-      const scaledValue = x(o);
-
-      const stackItem = {
-        key: d,
-        value: o || 0,
-        rectWidth: o ? scaledValue : 0,
-        offset: currentOffset,
-        fill: o ? colorScale(categories.indexOf(d)) : "rgb(255,255,255)"
-      };
-      currentOffset += o ? scaledValue : 0;
-      return stackItem;
-    });
 
     return (
-      <svg
+      <Popover
-        style={{
+        interactionKind={PopoverInteractionKind.HOVER_TARGET_ONLY}
-          marginRight: 5,
+        hoverOpenDelay={1500}
-          width,
+        hoverCloseDelay={200}
-          height
+        position={Position.LEFT}
+        modifiers={{
+          preventOverflow: { enabled: false },
+          hide: { enabled: false }
         }}
+        lazy
+        usePortal
+        disabled={colorByIsCatagoricalData}
+        popoverClassName={Classes.POPOVER_CONTENT_SIZING}
       >
-        {stacks.map(d => (
+        <canvas
-          <rect
+          className="bp3-popover-targer"
-            key={d.key}
+          style={{
-            width={d.rectWidth}
+            marginRight: 5,
-            height={height}
+            width: this._WIDTH,
-            x={d.offset}
+            height: this._HEIGHT,
-            title={d.metadataField}
+            borderBottom: colorByIsCatagoricalData
-            fill={d.fill}
+              ? ""
-          />
+              : "solid rgb(230, 230, 230) 0.25px"
-        ))}
+          }}
-      </svg>
+          width={this._WIDTH}
+          height={this._HEIGHT}
+          ref={ref => {
+            this.canvas = ref;
+            if (colorByIsCatagoricalData) this.createOccupancyStack();
+            else this.createHistogram();
+          }}
+        />
+        <div key="text" style={{ fontFamily: "Roboto", fontSize: "14px" }}>
+          <p style={{ margin: "0" }}>
+            This histograms shows the distribution of{" "}
+            <strong>{colorAccessor}</strong> within{" "}
+            <strong>{category.categoryValues[categoryIndex]}</strong>.
+            <br />
+            <br />
+            The x axis is the same for each histogram, while the y axis is
+            scaled to the largest bin within this histogram instead of the
+            largest bin within the whole category.
+          </p>
+        </div>
+      </Popover>
     );
   }
 }

client/src/components/categorical/util.js

@@ -5,7 +5,6 @@
 // return sorted index
 
 import isNumber from "is-number";
-import _ from "lodash";
 
 const sortedCategoryValues = values => {
   /* this sort could be memoized for perf */
@@ -13,7 +12,7 @@ const sortedCategoryValues = values => {
   const strings = [];
   const ints = [];
 
-  _.forEach(values, v => {
+  values.forEach(v => {
     if (isNumber(v[0])) {
       ints.push(v);
     } else {

client/src/components/categorical/value.js

@@ -1,36 +1,177 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
-import _ from "lodash";
+
+import {
+  Button,
+  InputGroup,
+  Menu,
+  MenuItem,
+  Popover,
+  Position,
+  Icon,
+  PopoverInteractionKind
+} from "@blueprintjs/core";
 import Occupancy from "./occupancy";
-import { countCategoryValues2D } from "../../util/stateManager/worldUtil";
 import * as globals from "../../globals";
+import styles from "./categorical.css";
+import { Tooltip } from "@blueprintjs/core";
 
 @connect(state => ({
   categoricalSelection: state.categoricalSelection,
+  annotations: state.annotations,
   colorScale: state.colors.scale,
   colorAccessor: state.colors.colorAccessor,
-  schema: _.get(state.world, "schema", null),
+  schema: state.world?.schema,
   world: state.world
 }))
 class CategoryValue extends React.Component {
-  toggleOff() {
+  constructor(props) {
+    super(props);
+    this.state = {
+      editedLabelText: ""
+    };
+  }
+
+  handleDeleteValue = () => {
+    const {
+      dispatch,
+      metadataField,
+      categoryIndex,
+      categoricalSelection
+    } = this.props;
+    const category = categoricalSelection[metadataField];
+    const label = category.categoryValues[categoryIndex];
+    dispatch({
+      type: "annotation: delete label",
+      metadataField,
+      label
+    });
+  };
+
+  handleAddCurrentSelectionToThisLabel = () => {
+    const {
+      dispatch,
+      metadataField,
+      categoryIndex,
+      categoricalSelection
+    } = this.props;
+    const category = categoricalSelection[metadataField];
+    const label = category.categoryValues[categoryIndex];
+    dispatch({
+      type: "annotation: label current cell selection",
+      metadataField,
+      categoryIndex,
+      label
+    });
+  };
+
+  handleEditValue = () => {
+    const {
+      dispatch,
+      metadataField,
+      categoryIndex,
+      categoricalSelection
+    } = this.props;
+    const { editedLabelText } = this.state;
+    const category = categoricalSelection[metadataField];
+    const label = category.categoryValues[categoryIndex];
+    dispatch({
+      type: "annotation: label edited",
+      editedLabel: editedLabelText,
+      metadataField,
+      categoryIndex,
+      label
+    });
+    this.setState({ editedLabelText: "" });
+  };
+
+  activateEditLabelMode = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "annotation: activate edit label mode",
+      metadataField,
+      categoryIndex
+    });
+  };
+
+  cancelEdit = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "annotation: cancel edit label mode",
+      metadataField,
+      categoryIndex
+    });
+  };
+
+  toggleOff = () => {
     const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter deselect",
       metadataField,
       categoryIndex
     });
-  }
+  };
 
-  toggleOn() {
+  shouldComponentUpdate = nextProps => {
+    /*
+    Checks to see if at least one of the following changed:
+    * world state
+    * the color accessor (what is currently being colored by)
+    * if this catagorical value's selection status has changed
+
+    If and only if true, update the component
+    */
+    const { props } = this;
+    const { metadataField, categoryIndex, categoricalSelection } = props;
+    const { categoricalSelection: newCategoricalSelection } = nextProps;
+
+    const valueSelectionChange =
+      categoricalSelection[metadataField].categoryValueSelected[
+        categoryIndex
+      ] !==
+      newCategoricalSelection[metadataField].categoryValueSelected[
+        categoryIndex
+      ];
+
+    const worldChange = props.world !== nextProps.world;
+    const colorAccessorChange = props.colorAccessor !== nextProps.colorAccessor;
+    const annotationsChange = props.annotations !== nextProps.annotations;
+
+    return (
+      valueSelectionChange ||
+      worldChange ||
+      colorAccessorChange ||
+      annotationsChange
+    );
+  };
+
+  toggleOn = () => {
     const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter select",
       metadataField,
       categoryIndex
     });
-  }
+  };
+
+  handleMouseEnter = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "category value mouse hover start",
+      metadataField,
+      categoryIndex
+    });
+  };
+
+  handleMouseExit = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "category value mouse hover end",
+      metadataField,
+      categoryIndex
+    });
+  };
 
   render() {
     const {
@@ -41,14 +182,15 @@ class CategoryValue extends React.Component {
       colorScale,
       i,
       schema,
-      world
+      isUserAnno,
+      annotations
     } = this.props;
 
     if (!categoricalSelection) return null;
 
     const category = categoricalSelection[metadataField];
-    const selected = category.categorySelected[categoryIndex];
+    const selected = category.categoryValueSelected[categoryIndex];
-    const count = category.categoryCounts[categoryIndex];
+    const count = category.categoryValueCounts[categoryIndex];
     const value = category.categoryValues[categoryIndex];
     const displayString = String(
       category.categoryValues[categoryIndex]
@@ -57,31 +199,50 @@ class CategoryValue extends React.Component {
     /* this is the color scale, so add swatches below */
     const isColorBy = metadataField === colorAccessor;
     let categories = null;
-    let occupancy = null;
 
     if (isColorBy && schema) {
-      categories = _.filter(schema.annotations.obs, {
+      categories = schema.annotations.obsByName[colorAccessor]?.categories;
-        name: colorAccessor
-      })[0].categories;
     }
 
-    if (colorAccessor && !isColorBy && categoricalSelection[colorAccessor]) {
+    let truncatedString = null;
-      occupancy = countCategoryValues2D(
+
-        metadataField,
+    if (
-        colorAccessor,
+      colorAccessor &&
-        world.obsAnnotations
+      !isColorBy &&
-      );
+      displayString.length > globals.categoryLabelDisplayStringShortLength
+    ) {
+      truncatedString = `${displayString.slice(
+        0,
+        globals.categoryLabelDisplayStringShortLength / 2
+      )}…${displayString.slice(
+        -globals.categoryLabelDisplayStringShortLength / 2
+      )}`;
+    } else if (
+      displayString.length > globals.categoryLabelDisplayStringLongLength
+    ) {
+      truncatedString = `${displayString.slice(
+        0,
+        globals.categoryLabelDisplayStringLongLength / 2
+      )}…${displayString.slice(
+        -globals.categoryLabelDisplayStringLongLength / 2
+      )}`;
     }
 
     return (
       <div
         key={i}
+        className={styles.value}
+        data-testclass="categorical-row"
         style={{
+          padding: "4px 7px",
           display: "flex",
           alignItems: "baseline",
-          justifyContent: "space-between"
+          justifyContent: "space-between",
+          marginBottom: "2px",
+          borderRadius: "2px"
         }}
-        data-testclass="categorical-row"
+        onMouseEnter={this.handleMouseEnter}
+        onMouseLeave={this.handleMouseExit}
       >
         <div
           style={{
@@ -93,34 +254,104 @@ class CategoryValue extends React.Component {
             justifyContent: "space-between"
           }}
         >
-          <label className="bp3-control bp3-checkbox">
+          <div style={{ display: "flex", alignItems: "baseline" }}>
-            <input
+            <label className="bp3-control bp3-checkbox" style={{ margin: 0 }}>
-              onChange={
+              <input
-                selected ? this.toggleOff.bind(this) : this.toggleOn.bind(this)
+                onChange={selected ? this.toggleOff : this.toggleOn}
-              }
+                data-testclass="categorical-value-select"
-              data-testclass="categorical-value-select"
+                data-testid={`categorical-value-select-${metadataField}-${displayString}`}
-              data-testid={`categorical-value-select-${metadataField}-${displayString}`}
+                checked={selected}
-              checked={selected}
+                type="checkbox"
-              type="checkbox"
-            />
-            <span className="bp3-control-indicator" />
-            <span
-              data-testid={`categorical-value-${metadataField}-${displayString}`}
-              data-testclass="categorical-value"
-            >
-              {displayString}
-            </span>
-          </label>
-          <span style={{ flexShrink: 0 }}>
-            {colorAccessor &&
-            !isColorBy &&
-            categoricalSelection[colorAccessor] ? (
-              <Occupancy
-                occupancy={occupancy.get(
-                  category.categoryValues[categoryIndex]
-                )}
-                {...this.props}
               />
+              <span
+                className="bp3-control-indicator"
+                onMouseEnter={this.handleMouseExit}
+                onMouseLeave={this.handleMouseEnter}
+              />
+            </label>
+            <Tooltip
+              content={displayString}
+              disabled={truncatedString === null}
+              hoverOpenDelay={globals.tooltipHoverOpenDelayQuick}
+            >
+              <span
+                data-testid={`categorical-value-${metadataField}-${displayString}`}
+                data-testclass="categorical-value"
+                style={{
+                  color:
+                    displayString === globals.unassignedCategoryLabel
+                      ? "#ababab"
+                      : "black",
+                  fontStyle:
+                    displayString === globals.unassignedCategoryLabel
+                      ? "italic"
+                      : "normal",
+                  display: "inline-block",
+                  overflow: "hidden",
+                  lineHeight: "1.1em",
+                  height: "1.1em",
+                  wordBreak: "break-all",
+                  verticalAlign: "middle"
+                }}
+              >
+                {annotations.isEditingLabelName &&
+                annotations.labelEditable.category === metadataField &&
+                annotations.labelEditable.label === categoryIndex
+                  ? null
+                  : truncatedString || displayString}
+              </span>
+            </Tooltip>
+            {isUserAnno &&
+            annotations.labelEditable.category === metadataField &&
+            annotations.isEditingLabelName &&
+            annotations.labelEditable.label === categoryIndex ? (
+              <form
+                onSubmit={e => {
+                  e.preventDefault();
+                  this.handleEditValue();
+                }}
+              >
+                <InputGroup
+                  style={{ position: "relative", top: -1 }}
+                  ref={input => {
+                    this.editableInput = input;
+                  }}
+                  small
+                  autoFocus
+                  onChange={e => {
+                    this.setState({ editedLabelText: e.target.value });
+                  }}
+                  defaultValue={displayString}
+                  rightElement={
+                    <Button
+                      minimal
+                      style={{ position: "relative", top: -1 }}
+                      type="button"
+                      icon="small-tick"
+                      data-testclass="submitEdit"
+                      data-testid="submitEdit"
+                      onClick={this.handleEditValue}
+                    />
+                  }
+                />
+              </form>
+            ) : null}
+            {/*
+            CANCEL IT, WITH BUTTON, ESCAPE KEY, CLICK OUT, UNDO?
+
+            <Button
+            minimal
+            style={{ position: "relative", top: -1 }}
+            type="button"
+            icon="cross"
+            data-testclass="submitEdit"
+            data-testid="submitEdit"
+            onClick={this.cancelEdit}
+          /> */}
+          </div>
+          <span style={{ flexShrink: 0 }}>
+            {colorAccessor && !isColorBy && !annotations.isEditingLabelName ? (
+              <Occupancy category={category} {...this.props} />
             ) : null}
           </span>
         </div>
@@ -128,10 +359,22 @@ class CategoryValue extends React.Component {
           <span
             data-testclass="categorical-value-count"
             data-testid={`categorical-value-count-${metadataField}-${displayString}`}
+            style={{
+              color:
+                displayString === globals.unassignedCategoryLabel
+                  ? "#ababab"
+                  : "black",
+              fontStyle:
+                displayString === globals.unassignedCategoryLabel
+                  ? "italic"
+                  : "auto"
+            }}
           >
             {count}
           </span>
+
           <svg
+            display={isColorBy && categories ? "auto" : "none"}
             style={{
               marginLeft: 5,
               width: 11,
@@ -142,6 +385,57 @@ class CategoryValue extends React.Component {
                   : "inherit"
             }}
           />
+          {isUserAnno ? (
+            <span
+              onMouseEnter={this.handleMouseExit}
+              onMouseLeave={this.handleMouseEnter}
+            >
+              <Popover
+                interactionKind={PopoverInteractionKind.HOVER}
+                boundary="window"
+                position={Position.RIGHT_TOP}
+                content={
+                  <Menu>
+                    <MenuItem
+                      icon="plus"
+                      data-testclass="handleAddCurrentSelectionToThisLabel"
+                      data-testid={`handleAddCurrentSelectionToThisLabel-${metadataField}`}
+                      onClick={this.handleAddCurrentSelectionToThisLabel}
+                      text={`Re-label currently selected cells as ${displayString}`}
+                    />
+                    {displayString !== globals.unassignedCategoryLabel ? (
+                      <MenuItem
+                        icon="edit"
+                        text="Edit this label's name"
+                        data-testclass="handleEditValue"
+                        data-testid={`handleEditValue-${metadataField}`}
+                        onClick={this.activateEditLabelMode}
+                      />
+                    ) : null}
+                    {displayString !== globals.unassignedCategoryLabel ? (
+                      <MenuItem
+                        icon="delete"
+                        intent="danger"
+                        data-testclass="handleDeleteValue"
+                        data-testid={`handleDeleteValue-${metadataField}`}
+                        onClick={this.handleDeleteValue}
+                        text="Delete this label, and reassign all cells to type 'unknown'"
+                      />
+                    ) : null}
+                  </Menu>
+                }
+              >
+                <Button
+                  style={{ marginLeft: 0, position: "relative", top: -1 }}
+                  data-testclass="seeActions"
+                  data-testid={`seeActions-${metadataField}`}
+                  icon="more"
+                  small
+                  minimal
+                />
+              </Popover>
+            </span>
+          ) : null}
         </span>
       </div>
     );

client/src/components/continuous/continuous.js

@@ -9,10 +9,10 @@ import * as globals from "../../globals";
 import HistogramBrush from "../brushableHistogram";
 
 @connect(state => ({
-  obsAnnotations: _.get(state.world, "obsAnnotations", null),
+  obsAnnotations: state.world?.obsAnnotations,
   colorAccessor: state.colors.colorAccessor,
   colorScale: state.colors.scale,
-  schema: _.get(state.world, "schema", null)
+  schema: state.world?.schema
 }))
 class Continuous extends React.Component {
   constructor(props) {
@@ -48,7 +48,7 @@ class Continuous extends React.Component {
     }
 
     /* initial value for iterator to simulate index, ranges is an object */
-    let zebra = -1;
+    let zebra = 0;
 
     return (
       <div>
@@ -66,11 +66,15 @@ class Continuous extends React.Component {
           ? _.map(obsAnnotations.colIndex.keys(), key => {
               const isColorField =
                 key.includes("color") || key.includes("Color");
-              if (key === "name" || isColorField) return null;
+              if (key === schema.annotations.obs.index || isColorField)
+                return null;
 
               const summary = obsAnnotations.col(key).summarize();
               const nonFiniteExtent =
-                summary.min === undefined || summary.max === undefined;
+                summary.min === undefined ||
+                summary.max === undefined ||
+                Number.isNaN(summary.min) ||
+                Number.isNaN(summary.max);
               if (!summary.categorical && !nonFiniteExtent) {
                 zebra += 1;
                 return (

client/src/components/continuous/drawAxes.js

@@ -1,83 +0,0 @@
-// jshint esversion: 6
-import * as d3 from "d3";
-import styles from "./parallelCoordinates.css";
-import { yAxis, brushstart } from "./util";
-
-const drawAxes = (
-  svg,
-  ctx,
-  dimensions,
-  xscale,
-  height,
-  width,
-  handleBrushAction,
-  handleColorAction
-) => {
-  /*****************************************
-  ******************************************
-  Handles a brush event, toggling the display of foreground lines.
-  ******************************************
-  ******************************************/
-
-  function brush() {
-    const actives = [];
-    svg
-      .selectAll(".parcoords_axis .parcoords_brush")
-      .filter(() => d3.brushSelection(this))
-      .each(d => {
-        actives.push({
-          dimension: d,
-          extent: d3.brushSelection(this)
-        });
-      });
-    /* fire action, with selected dimensions & their values */
-    handleBrushAction(actives);
-  }
-
-  const axes = svg
-    .selectAll(".parcoords_axis")
-    .data(dimensions)
-    .enter()
-    .append("g")
-    .attr("class", `${styles.axis} parcoords_axis`)
-    .attr("transform", (d, i) => `translate(${xscale(i)})`);
-
-  axes
-    .append("g")
-    .each(d => {
-      const renderAxis =
-        "axis" in d
-          ? d.axis.scale(d.scale) // custom axis
-          : yAxis.scale(d.scale); // default axis
-      d3.select(this).call(renderAxis);
-    })
-    .append("text")
-    .on("click", d => {
-      handleColorAction(d.key);
-    })
-    .attr("class", styles.title)
-    .attr("text-anchor", "start")
-    .text(d => ("description" in d ? `${d.description}  🖌️` : `${d.key}  🖌️`));
-
-  // Add and store a brush for each axis.
-  axes
-    .append("g")
-    .attr("class", `${styles.brush} parcoords_brush`)
-    .each(d => {
-      d3.select(this).call(
-        (d.brush = d3
-          .brushY()
-          .extent([[-10, 0], [10, height]])
-          .on("start", brushstart)
-          .on("brush", brush)
-          .on("end", brush))
-      );
-    })
-    .selectAll("rect")
-    .attr("x", -8)
-    .attr("width", 16);
-
-  return axes;
-};
-
-export default drawAxes;

client/src/components/continuousLegend/index.js

@@ -2,7 +2,7 @@
 import React from "react";
 import { connect } from "react-redux";
 import * as d3 from "d3";
-import { interpolateViridis, interpolateCool } from "d3-scale-chromatic";
+import { interpolateCool } from "d3-scale-chromatic";
 
 // create continuous color legend
 // http://bl.ocks.org/syntagmatic/e8ccca52559796be775553b467593a9f

client/src/components/framework/custom-icons.js

@@ -0,0 +1,16 @@
+/* https://github.com/palantir/blueprint/issues/2348 */
+
+<defs>
+<clipPath id="clip0">
+<rect width="16" height="16" fill="white"/>
+</clipPath>
+</defs>
+<g clip-path="url(#clip0)">
+<rect width="16" height="16" fill="white"/>
+<path d="M1.33415 8.75877C0.939491 8.36411 0.727699 7.82249 0.749957 7.2648L0.926361 2.84501C0.967947 1.80308 1.80308 0.967947 2.84501 0.926361L7.2648 0.749958C7.82249 0.727699 8.36411 0.939492 8.75877 1.33415L14.3595 6.93485C15.1405 7.7159 15.1405 8.98223 14.3595 9.76328L9.76328 14.3595C8.98223 15.1405 7.7159 15.1405 6.93485 14.3595L1.33415 8.75877Z" fill="black"/>
+<circle cx="4.5" cy="4.5" r="1.5" fill="white"/>
+<circle cx="4.5" cy="11.5" r="3.75" stroke="white" stroke-width="0.5"/>
+<circle cx="4.5" cy="11.5" r="3.5" fill="black"/>
+<line x1="4.5" y1="10" x2="4.5" y2="13" stroke="white"/>
+<line x1="3" y1="11.5" x2="6" y2="11.5" stroke="white"/>
+</g>

client/src/components/framework/logo.js

@@ -0,0 +1,18 @@
+import React from "react";
+import * as globals from "../../globals";
+
+const Logo = props => {
+  const { size } = props;
+  return (
+    <svg width={size} height={size} viewBox="0 0 48 48" fill="none">
+      <rect width="48" height="48" fill="white" />
+      <rect width="48" height="48" fill={globals.logoColor} />
+      <rect x="19" y="19" width="22" height="22" fill="white" />
+      <rect x="24" y="24" width="12" height="12" fill={globals.logoColor} />
+      <rect x="7" y="19" width="7" height="22" fill="white" />
+      <rect x="19" y="7" width="22" height="7" fill="white" />
+    </svg>
+  );
+};
+
+export default Logo;

client/src/components/geneExpression/expressionButtons.js

@@ -1,101 +0,0 @@
-// jshint esversion: 6
-import React from "react";
-import _ from "lodash";
-import { Button, AnchorButton, Tooltip } from "@blueprintjs/core";
-import { connect } from "react-redux";
-import * as globals from "../../globals";
-import actions from "../../actions";
-import CellSetButton from "./cellSetButtons";
-
-@connect(state => ({
-  differential: state.differential,
-  world: state.world,
-  crossfilter: state.crossfilter
-}))
-class Expression extends React.Component {
-  constructor(props) {
-    super(props);
-    this.state = {};
-  }
-
-  computeDiffExp() {
-    const { dispatch, differential } = this.props;
-    if (differential.celllist1 && differential.celllist2) {
-      dispatch(
-        actions.requestDifferentialExpression(
-          differential.celllist1,
-          differential.celllist2
-        )
-      );
-    }
-  }
-
-  clearDifferentialExpression() {
-    const { dispatch, differential } = this.props;
-    dispatch({
-      type: "clear differential expression",
-      diffExp: differential.diffExp
-    });
-    dispatch({
-      type: "clear scatterplot"
-    });
-  }
-
-  render() {
-    const { differential } = this.props;
-    if (!differential) {
-      return null;
-    }
-    const haveBothCellSets =
-      !!differential.celllist1 && !!differential.celllist2;
-    return (
-      <div
-        style={{
-          marginRight: 10,
-          marginBottom: 10,
-          paddingLeft: globals.leftSidebarSectionPadding
-        }}
-      >
-        <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
-        <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
-        {!differential.diffExp ? (
-          <Tooltip
-            content="Add two cells selections, see the top 15 differentially expressed genes between them"
-            position="bottom"
-          >
-            <AnchorButton
-              style={{ marginTop: 10 }}
-              disabled={!haveBothCellSets}
-              intent="primary"
-              data-testid="diffexp-button"
-              loading={differential.loading}
-              fill
-              type="button"
-              onClick={this.computeDiffExp.bind(this)}
-            >
-              Compute Differential Expression
-            </AnchorButton>
-          </Tooltip>
-        ) : null}
-        {differential.diffExp ? (
-          <Tooltip
-            content="Remove differentially expressed gene list and clear cell selections"
-            position="bottom"
-          >
-            <Button
-              type="button"
-              fill
-              style={{ marginTop: 10 }}
-              intent="warning"
-              onClick={this.clearDifferentialExpression.bind(this)}
-            >
-              Clear Differential Expression
-            </Button>
-          </Tooltip>
-        ) : null}
-      </div>
-    );
-  }
-}
-
-export default Expression;

client/src/components/geneExpression/index.js

@@ -21,7 +21,8 @@ import {
   postUserErrorToast,
   keepAroundErrorToast
 } from "../framework/toasters";
-import ExpressionButtons from "./expressionButtons";
+
+import { memoize } from "../../util/dataframe/util";
 
 const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
   if (!modifiers.matchesPredicate) {
@@ -57,7 +58,7 @@ const filterGenes = (query, genes) =>
 
 @connect(state => {
   return {
-    obsAnnotations: _.get(state.world, "obsAnnotations", null),
+    obsAnnotations: state.world?.obsAnnotations,
     userDefinedGenes: state.controls.userDefinedGenes,
     userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
     world: state.world,
@@ -70,10 +71,24 @@ class GeneExpression extends React.Component {
     super(props);
     this.state = {
       bulkAdd: "",
-      tab: "autosuggest"
+      tab: "autosuggest",
+      activeItem: null
     };
   }
 
+  _genesToUpper = listGenes => {
+    // Has to be a Map to preserve index
+    const upperGenes = new Map();
+    for (let i = 0, { length } = listGenes; i < length; i += 1) {
+      upperGenes.set(listGenes[i].toUpperCase(), i);
+    }
+
+    return upperGenes;
+  };
+
+  // eslint-disable-next-line react/sort-comp
+  _memoGenesToUpper = memoize(this._genesToUpper, arr => arr);
+
   placeholderGeneNames() {
     /*
     return a string containing gene name suggestions for use as a user hint.
@@ -85,7 +100,8 @@ class GeneExpression extends React.Component {
     */
     const { world } = this.props;
     const { varAnnotations } = world;
-    const geneNames = varAnnotations.col("name").asArray();
+    const varIndexName = world.schema.annotations.var.index;
+    const geneNames = varAnnotations.col(varIndexName).asArray();
     if (geneNames.length > 0) {
       const placeholder = [];
       let len = geneNames.length;
@@ -107,6 +123,8 @@ class GeneExpression extends React.Component {
 
   handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
+    if (!g) return;
     const gene = g.target;
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
@@ -114,7 +132,9 @@ class GeneExpression extends React.Component {
       postUserErrorToast(
         "That's too many genes, you can have at most 15 user defined genes"
       );
-    } else if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
+    } else if (
+      world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+    ) {
       postUserErrorToast("That doesn't appear to be a valid gene name.");
     } else {
       dispatch({ type: "single user defined gene start" });
@@ -127,6 +147,7 @@ class GeneExpression extends React.Component {
 
   handleBulkAddClick() {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
     const { bulkAdd } = this.state;
 
     /*
@@ -135,22 +156,37 @@ class GeneExpression extends React.Component {
     */
     if (bulkAdd !== "") {
       const genes = _.pull(_.uniq(bulkAdd.split(/[ ,]+/)), "");
+      if (genes.length === 0) {
+        return keepAroundErrorToast("Must enter a gene name.");
+      }
+      const worldGenes = world.varAnnotations.col(varIndexName).asArray();
+
+      // These gene lists are unique enough where memoization is useless
+      const upperGenes = this._genesToUpper(genes);
+      const upperUserDefinedGenes = this._genesToUpper(userDefinedGenes);
+
+      const upperWorldGenes = this._memoGenesToUpper(worldGenes);
 
       dispatch({ type: "bulk user defined gene start" });
+
       Promise.all(
-        genes.map(gene => {
+        [...upperGenes.keys()].map(upperGene => {
-          if (gene.length === 0) {
+          if (upperUserDefinedGenes.get(upperGene) !== undefined) {
-            return keepAroundErrorToast("Must enter a gene name.");
-          }
-          if (userDefinedGenes.indexOf(gene) !== -1) {
             return keepAroundErrorToast("That gene already exists");
           }
-          if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
+
+          const indexOfGene = upperWorldGenes.get(upperGene);
+
+          if (indexOfGene === undefined) {
             return keepAroundErrorToast(
-              `${gene} doesn't appear to be a valid gene name.`
+              `${
+                genes[upperGenes.get(upperGene)]
+              } doesn't appear to be a valid gene name.`
             );
           }
-          return dispatch(actions.requestUserDefinedGene(gene));
+          return dispatch(
+            actions.requestUserDefinedGene(worldGenes[indexOfGene])
+          );
         })
       ).then(
         () => dispatch({ type: "bulk user defined gene complete" }),
@@ -168,24 +204,12 @@ class GeneExpression extends React.Component {
       userDefinedGenesLoading,
       differential
     } = this.props;
-
+    const varIndexName = world?.schema?.annotations?.var?.index;
-    const { tab, bulkAdd } = this.state;
+    const { tab, bulkAdd, activeItem } = this.state;
 
     return (
       <div>
-        <div
+        <div>
-          style={{
-            marginTop: 30
-          }}
-        >
-          <p
-            style={Object.assign({}, globals.leftSidebarSectionHeading, {
-              paddingLeft: globals.leftSidebarSectionPadding,
-              margin: 0
-            })}
-          >
-            Selected Genes
-          </p>
           <div
             style={{
               padding: globals.leftSidebarSectionPadding
@@ -201,7 +225,7 @@ class GeneExpression extends React.Component {
                 this.setState({ tab: "autosuggest" });
               }}
             >
-              Autosuggest
+              Autosuggest genes
             </Button>
             <Button
               active={tab === "bulkadd"}
@@ -235,15 +259,17 @@ class GeneExpression extends React.Component {
                   /* this happens on 'enter' */
                   this.handleClick(g);
                 }}
+                initialContent={<MenuItem disabled text="Enter a gene…" />}
                 inputProps={{ "data-testid": "gene-search" }}
                 inputValueRenderer={g => {
                   return "";
                 }}
                 itemListPredicate={filterGenes}
+                onActiveItemChange={item => this.setState({ activeItem: item })}
                 itemRenderer={renderGene.bind(this)}
                 items={
                   world && world.varAnnotations
-                    ? world.varAnnotations.col("name").asArray()
+                    ? world.varAnnotations.col(varIndexName).asArray()
                     : ["No genes"]
                 }
                 popoverProps={{ minimal: true }}
@@ -252,8 +278,9 @@ class GeneExpression extends React.Component {
                 className="bp3-button bp3-intent-primary"
                 data-testid={"add-gene"}
                 loading={userDefinedGenesLoading}
+                onClick={() => this.handleClick(activeItem)}
               >
-                Add
+                Add gene
               </Button>
             </ControlGroup>
           ) : null}
@@ -284,7 +311,7 @@ class GeneExpression extends React.Component {
                       onClick={this.handleBulkAddClick.bind(this)}
                       loading={userDefinedGenesLoading}
                     >
-                      Add
+                      Add genes
                     </Button>
                   </ControlGroup>
                 </FormGroup>
@@ -311,18 +338,9 @@ class GeneExpression extends React.Component {
             : null}
         </div>
         <div>
-          <p
-            style={Object.assign({}, globals.leftSidebarSectionHeading, {
-              marginTop: 40,
-              paddingLeft: globals.leftSidebarSectionPadding
-            })}
-          >
-            Differentially Expressed Genes
-          </p>
-          <ExpressionButtons />
           {differential.diffExp
             ? _.map(differential.diffExp, (value, index) => {
-                const name = world.varAnnotations.at(value[0], "name");
+                const name = world.varAnnotations.at(value[0], varIndexName);
                 const values = world.varData.col(name);
                 if (!values) {
                   return null;

client/src/components/graph/drawPointsRegl.js

@@ -1,44 +1,67 @@
-// jshint esversion: 6
+import { glPointFlags, glPointSize } from "../../util/glHelpers";
-const mat4 = require("gl-mat4");
-
-// opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js
 
 export default function(regl) {
   return regl({
     vert: `
     precision mediump float;
+
     attribute vec2 position;
     attribute vec3 color;
-    attribute float size;
+    attribute float flag;
+
     uniform float distance;
-    uniform mat4 projection, view;
+    uniform mat3 projView;
-    varying vec3 fragColor;
+    uniform float nPoints;
+    uniform float minViewportDimension;
+
+    varying vec4 fragColor;
+
+    const float zBottom = 0.99;
+    const float zMiddle = 0.;
+    const float zTop = -1.;
+
+    // import getFlags()
+    ${glPointFlags}
+
+    // get pointSize()
+    ${glPointSize}
+
     void main() {
-      gl_PointSize = 7.0 / pow(distance, 2.5) + size;
+      bool isNaN, isSelected, isHighlight;
-      gl_Position = projection * view * vec4(position.x, -position.y, 0, 1);
+      getFlags(flag, isNaN, isSelected, isHighlight);
-      fragColor = color;
+
+      float size = pointSize(nPoints, minViewportDimension, isSelected, isHighlight);
+      gl_PointSize = size * pow(distance, 0.5);
+
+      float z = isNaN ? zBottom : (isHighlight ? zTop : zMiddle);
+      vec3 xy = projView * vec3(position, 1.);
+      gl_Position = vec4(xy.xy, z, 1.);
+
+      float alpha = isNaN ? 0.9 : 1.0;
+      fragColor = vec4(color, alpha);
     }`,
 
     frag: `
     precision mediump float;
-    varying vec3 fragColor;
+    varying vec4 fragColor;
     void main() {
       if (length(gl_PointCoord.xy - 0.5) > 0.5) {
         discard;
       }
-      gl_FragColor = vec4(fragColor, 1);
+      gl_FragColor = fragColor;
     }`,
 
     attributes: {
       position: regl.prop("position"),
       color: regl.prop("color"),
-      size: regl.prop("size")
+      flag: regl.prop("flag")
     },
 
     uniforms: {
       distance: regl.prop("distance"),
-      view: regl.prop("view"),
+      projView: regl.prop("projView"),
-      projection: ({viewportWidth, viewportHeight}) => mat4.perspective([], Math.PI / 2, viewportWidth / viewportHeight, 0.01, 1000)
+      nPoints: regl.prop("nPoints"),
+      minViewportDimension: regl.prop("minViewportDimension")
     },
 
     count: regl.prop("count"),

client/src/components/graph/setupCentroidSVG.js

@@ -0,0 +1,34 @@
+import * as d3 from "d3";
+import styles from "./graph.css";
+
+export default (responsive, graphPaddingRight, xy, text, colorBy) => {
+  const containerWidth = responsive.width - graphPaddingRight;
+
+  const svg = d3
+    .select("#graphAttachPoint")
+    .append("svg")
+    .attr("id", "centroid-container")
+    .attr("data-testid", "centroid-overlay")
+    .attr("width", containerWidth)
+    .attr("height", responsive.height)
+    .attr("class", `${styles.graphSVG}`)
+    .style("z-index", 998)
+    .style("pointer-events", "none");
+  //  TODO: Create own styles, ask Colin for an explanation on the css
+  // For now I'm going to put centroid z-index at 998 and lasso on 999
+
+  const label = svg
+    .append("g")
+    .attr("transform", `translate(${xy[0]}, ${xy[1]})`);
+
+  label
+    .append("text")
+    .attr("text-anchor", "middle")
+    .text(text)
+    .style("font-family", "Roboto Condensed")
+    .style("font-size", "18px")
+    .style("font-weight", "700")
+    .style("fill", colorBy ? "black" : "rgb(32, 178, 212)");
+
+  return svg;
+};

client/src/components/graph/setupSVGandBrush.js

@@ -16,15 +16,19 @@ export default (
   handleEndAction,
   handleCancelAction,
   responsive,
-  graphPaddingRight
+  graphPaddingRight,
+  graphInteractionMode
 ) => {
   const svg = d3
     .select("#graphAttachPoint")
     .append("svg")
+    .attr("id", "tool")
     .attr("data-testid", "layout-overlay")
     .attr("width", responsive.width - graphPaddingRight)
     .attr("height", responsive.height)
-    .attr("class", `${styles.graphSVG}`);
+    .attr("class", `${styles.graphSVG}`)
+    .style("z-index", 999)
+    .style("display", graphInteractionMode === "select" ? "inherit" : "none");
 
   if (selectionToolType === "brush") {
     const brush = d3

client/src/components/leftSidebar/index.js

@@ -1,16 +1,15 @@
 // jshint esversion: 6
-import _ from "lodash";
 import React from "react";
 import { connect } from "react-redux";
-import Categorical from "./categorical/categorical";
+import Categorical from "../categorical/categorical";
-import Continuous from "./continuous/continuous";
+import Continuous from "../continuous/continuous";
-import GeneExpression from "./geneExpression";
+import GeneExpression from "../geneExpression";
-import * as globals from "../globals";
+import * as globals from "../../globals";
-import DynamicScatterplot from "./scatterplot/scatterplot";
+import DynamicScatterplot from "../scatterplot/scatterplot";
+import TopLeftLogoAndTitle from "./topLeftLogoAndTitle";
 
 @connect(state => ({
   responsive: state.responsive,
-  datasetTitle: _.get(state.config, "displayNames.dataset"),
   scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor
 }))
@@ -18,7 +17,6 @@ class LeftSideBar extends React.Component {
   render() {
     const {
       responsive,
-      datasetTitle,
       scatterplotXXaccessor,
       scatterplotYYaccessor
     } = this.props;
@@ -27,8 +25,7 @@ class LeftSideBar extends React.Component {
     this magic number should be made less fragile,
     if cellxgene logo or tabs change, this must as well
     */
-    const metadataSectionPadding = 0;
+    const logoRelatedPadding = 50;
-    // scatterplotXXaccessor && scatterplotYYaccessor ? 450 : 0;
 
     return (
       <div
@@ -36,27 +33,14 @@ class LeftSideBar extends React.Component {
           position: "fixed",
           backgroundColor: "white",
           /* x y blur spread color */
-          boxShadow: "1px 0px 6px 2px rgba(153,153,153,0.4)"
+          boxShadow: "-3px 0px 6px 2px rgba(153,153,153,0.4)"
         }}
       >
-        <p
+        <TopLeftLogoAndTitle />
-          data-testid="header"
-          style={{
-            position: "fixed",
-            top: globals.cellxgeneTitleTopPadding,
-            left: globals.leftSidebarWidth + globals.cellxgeneTitleLeftPadding,
-            margin: 0,
-            fontSize: globals.largestFontSize,
-            color: globals.darkerGrey,
-            width: "100%"
-          }}
-        >
-          cellxgene: {datasetTitle}
-        </p>
-
         <div
           style={{
-            height: responsive.height - metadataSectionPadding,
+            height: responsive.height - logoRelatedPadding,
+            marginTop: logoRelatedPadding,
             width: globals.leftSidebarWidth,
             overflowY: "auto",
             overflowX: "hidden"

client/src/components/leftSidebar/topLeftLogoAndTitle.js

@@ -0,0 +1,90 @@
+// jshint esversion: 6
+import React from "react";
+import { connect } from "react-redux";
+import * as globals from "../../globals";
+import Logo from "../framework/logo";
+
+@connect(state => ({
+  responsive: state.responsive,
+  datasetTitle: state.config?.displayNames?.dataset ?? "",
+  aboutURL: state.config?.links?.["about-dataset"],
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor
+}))
+class LeftSideBar extends React.Component {
+  render() {
+    const { datasetTitle, aboutURL } = this.props;
+
+    const paddingToAvoidScrollBar = 15;
+
+    const displayTitle =
+      datasetTitle.length > globals.datasetTitleMaxCharacterCount
+        ? `${datasetTitle.substring(
+            0,
+            Math.floor(globals.datasetTitleMaxCharacterCount / 2)
+          )}…${datasetTitle.slice(
+            -Math.floor(globals.datasetTitleMaxCharacterCount / 2)
+          )}`
+        : datasetTitle;
+
+    return (
+      <div
+        style={{
+          paddingLeft: 8,
+          paddingTop: 8,
+          width: globals.leftSidebarWidth - paddingToAvoidScrollBar,
+          position: "absolute",
+          backgroundColor: "white",
+          zIndex: 8888
+          /* x y blur spread color */
+          // boxShadow: "-5px -1px 4px 2px rgba(225,225,225,0.4)"
+        }}
+      >
+        <Logo size={30} />
+        <span
+          style={{
+            fontSize: 28,
+            position: "relative",
+            top: -6,
+            fontWeight: "bold",
+            marginLeft: 5,
+            color: globals.logoColor,
+            userSelect: "none"
+          }}
+        >
+          cell
+          <span
+            style={{
+              position: "relative",
+              top: 1,
+              fontWeight: 300,
+              fontSize: 24
+            }}
+          >
+            ×
+          </span>
+          gene
+        </span>
+        <div
+          data-testid="header"
+          style={{
+            fontSize: 14,
+            position: "relative",
+            top: -6,
+            display: "inline-block",
+            width: "190px",
+            marginLeft: "7px",
+            height: "1.1em",
+            overflow: "hidden",
+            wordBreak: "break-all"
+          }}
+          title={datasetTitle}
+        >
+          {aboutURL ? <a href={aboutURL}>{displayTitle}</a> : displayTitle}
+        </div>
+      </div>
+    );
+  }
+}
+
+export default LeftSideBar;

client/src/components/menubar/cellSetButtons.js

@@ -3,9 +3,10 @@ import React from "react";
 import { AnchorButton, Tooltip } from "@blueprintjs/core";
 import { connect } from "react-redux";
 import { World } from "../../util/stateManager";
+import { tooltipHoverOpenDelay } from "../../globals";
 
 @connect()
-class CellSetButton extends React.Component {
+class CellSetButton extends React.PureComponent {
   set() {
     const {
       differential,
@@ -32,16 +33,16 @@ class CellSetButton extends React.Component {
   render() {
     const { differential, eitherCellSetOneOrTwo } = this.props;
     const cellListName = `celllist${eitherCellSetOneOrTwo}`;
-    let cells_selected = differential[cellListName]
+    const cellsSelected = differential[cellListName]
       ? differential[cellListName].length
       : 0;
     return (
       <Tooltip
         content="Save current selection for differential expression computation"
-        position="top"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
       >
         <AnchorButton
-          style={{ marginRight: 10 }}
           type="button"
           disabled={differential.diffExp}
           onClick={this.set.bind(this)}
@@ -50,7 +51,7 @@ class CellSetButton extends React.Component {
           {eitherCellSetOneOrTwo}
           {": "}
           <span data-testid={`cellset-count-${eitherCellSetOneOrTwo}`}>
-            {cells_selected}
+            {cellsSelected}
           </span>
           {" cells"}
         </AnchorButton>

client/src/components/menubar/clip.js

@@ -0,0 +1,138 @@
+// jshint esversion: 6
+import React from "react";
+import {
+  Position,
+  Button,
+  Popover,
+  NumericInput,
+  Icon,
+  Tooltip
+} from "@blueprintjs/core";
+import { tooltipHoverOpenDelay } from "../../globals";
+
+function Clip(props) {
+  const {
+    pendingClipPercentiles,
+    clipPercentileMin,
+    clipPercentileMax,
+    handleClipOpening,
+    handleClipClosing,
+    handleClipCommit,
+    isClipDisabled,
+    handleClipOnKeyPress,
+    handleClipPercentileMaxValueChange,
+    handleClipPercentileMinValueChange
+  } = props;
+
+  const clipMin =
+    pendingClipPercentiles?.clipPercentileMin ?? clipPercentileMin;
+  const clipMax =
+    pendingClipPercentiles?.clipPercentileMax ?? clipPercentileMax;
+  const activeClipClass =
+    clipPercentileMin > 0 || clipPercentileMax < 100
+      ? " bp3-intent-warning"
+      : "";
+
+  return (
+    <div
+      className="bp3-button-group"
+      style={{
+        marginLeft: 10
+      }}
+    >
+      <Popover
+        target={
+          <Tooltip
+            content="Clip all continuous values to a percentile range"
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelay}
+          >
+            <Button
+              type="button"
+              data-testid="visualization-settings"
+              className={`bp3-button bp3-icon-timeline-bar-chart ${activeClipClass}`}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
+        }
+        position={Position.BOTTOM_RIGHT}
+        onOpening={handleClipOpening}
+        onClosing={handleClipClosing}
+        content={
+          <div
+            style={{
+              display: "flex",
+              justifyContent: "flex-start",
+              alignItems: "flex-start",
+              flexDirection: "column",
+              padding: 10
+            }}
+          >
+            <div>Clip all continuous values to percentile range</div>
+            <div
+              style={{
+                display: "flex",
+                justifyContent: "space-between",
+                alignItems: "center",
+                paddingTop: 5,
+                paddingBottom: 5
+              }}
+            >
+              <NumericInput
+                style={{ width: 50 }}
+                data-testid="clip-min-input"
+                onValueChange={handleClipPercentileMinValueChange}
+                onKeyPress={handleClipOnKeyPress}
+                value={clipMin}
+                min={0}
+                max={100}
+                fill={false}
+                minorStepSize={null}
+                rightElement={
+                  <div style={{ padding: "4px 2px" }}>
+                    <Icon icon="percentage" intent="primary" iconSize={14} />
+                  </div>
+                }
+              />
+              <span style={{ marginRight: 5, marginLeft: 5 }}> - </span>
+              <NumericInput
+                style={{ width: 50 }}
+                data-testid="clip-max-input"
+                onValueChange={handleClipPercentileMaxValueChange}
+                onKeyPress={handleClipOnKeyPress}
+                value={clipMax}
+                min={0}
+                max={100}
+                fill={false}
+                minorStepSize={null}
+                rightElement={
+                  <div style={{ padding: "4px 2px" }}>
+                    <Icon icon="percentage" intent="primary" iconSize={14} />
+                  </div>
+                }
+              />
+              <Button
+                type="button"
+                data-testid="clip-commit"
+                className="bp3-button"
+                disabled={isClipDisabled()}
+                style={{
+                  cursor: "pointer",
+                  marginRight: 5,
+                  marginLeft: 5
+                }}
+                onClick={handleClipCommit}
+              >
+                Clip
+              </Button>
+            </div>
+          </div>
+        }
+      />
+    </div>
+  );
+}
+
+export default Clip;

client/src/components/menubar/index.js

@@ -0,0 +1,508 @@
+// jshint esversion: 6
+import React from "react";
+import { connect } from "react-redux";
+import {
+  Button,
+  AnchorButton,
+  Tooltip,
+  Popover,
+  Position,
+  RadioGroup,
+  Radio
+} from "@blueprintjs/core";
+import { World } from "../../util/stateManager";
+import actions from "../../actions";
+import CellSetButton from "./cellSetButtons";
+import InformationMenu from "./infoMenu";
+import UndoRedoReset from "./undoRedoReset";
+import Clip from "./clip";
+import {
+  tooltipHoverOpenDelay,
+  tooltipHoverOpenDelayQuick
+} from "../../globals";
+
+@connect(state => ({
+  universe: state.universe,
+  world: state.world,
+  loading: state.controls.loading,
+  crossfilter: state.crossfilter,
+  differential: state.differential,
+  resettingInterface: state.controls.resettingInterface,
+  layoutChoice: state.layoutChoice,
+  graphInteractionMode: state.controls.graphInteractionMode,
+  clipPercentileMin: Math.round(100 * (state.world?.clipQuantiles?.min ?? 0)),
+  clipPercentileMax: Math.round(100 * (state.world?.clipQuantiles?.max ?? 1)),
+  userDefinedGenes: state.controls.userDefinedGenes,
+  diffexpGenes: state.controls.diffexpGenes,
+  colorAccessor: state.colors.colorAccessor,
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+  celllist1: state.differential.celllist1,
+  celllist2: state.differential.celllist2,
+  libraryVersions: state.config?.library_versions, // eslint-disable-line camelcase
+  undoDisabled: state["@@undoable/past"].length === 0,
+  redoDisabled: state["@@undoable/future"].length === 0,
+  aboutLink: state.config?.links?.["about-dataset"],
+  disableDiffexp: state.config?.parameters?.["disable-diffexp"] ?? false,
+  diffexpMayBeSlow: state.config?.parameters?.["diffexp-may-be-slow"] ?? false
+}))
+class MenuBar extends React.Component {
+  static isValidDigitKeyEvent(e) {
+    /*
+    Return true if this event is necessary to enter a percent number input.
+    Return false if not.
+
+    Returns true for events with keys: backspace, control, alt, meta, [0-9],
+    or events that don't have a key.
+    */
+    if (e.key === null) return true;
+    if (e.ctrlKey || e.altKey || e.metaKey) return true;
+
+    // concept borrowed from blueprint's numericInputUtils:
+    // keys that print a single character when pressed have a `key` name of
+    // length 1. every other key has a longer `key` name (e.g. "Backspace",
+    // "ArrowUp", "Shift"). since none of those keys can print a character
+    // to the field--and since they may have important native behaviors
+    // beyond printing a character--we don't want to disable their effects.
+    const isSingleCharKey = e.key.length === 1;
+    if (!isSingleCharKey) return true;
+
+    const key = e.key.charCodeAt(0) - 48; /* "0" */
+    return key >= 0 && key <= 9;
+  }
+
+  constructor(props) {
+    super(props);
+    this.state = {
+      pendingClipPercentiles: null
+    };
+  }
+
+  isClipDisabled = () => {
+    /*
+    return true if clip button should be disabled.
+    */
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMin = pendingClipPercentiles?.clipPercentileMin;
+    const clipPercentileMax = pendingClipPercentiles?.clipPercentileMax;
+
+    const { world } = this.props;
+    const currentClipMin = 100 * world?.clipQuantiles?.min;
+    const currentClipMax = 100 * world?.clipQuantiles?.max;
+
+    // if you change this test, be careful with logic around
+    // comparisons between undefined / NaN handling.
+    const isDisabled =
+      !(clipPercentileMin < clipPercentileMax) ||
+      (clipPercentileMin === currentClipMin &&
+        clipPercentileMax === currentClipMax);
+
+    return isDisabled;
+  };
+
+  isResetDisabled = () => {
+    /*
+    Reset should be disabled when all of the following are true:
+      * nothing is selected in the crossfilter
+      * world EQ universe
+      * nothing is colored by
+      * there are no userDefinedGenes or diffexpGenes displayed
+      * scatterplot is not displayed
+      * nothing in cellset1 or cellset2
+      * clip percentiles are [0,100]
+    */
+    const {
+      crossfilter,
+      world,
+      universe,
+      userDefinedGenes,
+      diffexpGenes,
+      colorAccessor,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor,
+      celllist1,
+      celllist2,
+      clipPercentileMin,
+      clipPercentileMax
+    } = this.props;
+
+    if (!crossfilter || !world || !universe) {
+      return false;
+    }
+    const nothingSelected = crossfilter.countSelected() === crossfilter.size();
+    const nothingColoredBy = !colorAccessor;
+    const noGenes = userDefinedGenes.length === 0 && diffexpGenes.length === 0;
+    const scatterNotDpl = !scatterplotXXaccessor || !scatterplotYYaccessor;
+    const nothingInCellsets = !celllist1 && !celllist2;
+
+    return (
+      nothingSelected &&
+      World.worldEqUniverse(world, universe) &&
+      nothingColoredBy &&
+      noGenes &&
+      scatterNotDpl &&
+      nothingInCellsets &&
+      clipPercentileMax === 100 &&
+      clipPercentileMin === 0
+    );
+  };
+
+  resetInterface = () => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "interface reset started"
+    });
+    dispatch(actions.resetInterface());
+  };
+
+  handleClipOnKeyPress = e => {
+    /*
+    allow only numbers, plus other critical keys which
+    may be required to make a number
+    */
+    if (!MenuBar.isValidDigitKeyEvent(e)) {
+      e.preventDefault();
+    }
+  };
+
+  handleClipPercentileMinValueChange = v => {
+    /*
+    Ignore anything that isn't a legit number
+    */
+    if (!Number.isFinite(v)) return;
+
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMax = pendingClipPercentiles?.clipPercentileMax;
+
+    /*
+    clamp to [0, currentClipPercentileMax]
+    */
+    if (v <= 0) v = 0;
+    if (v > 100) v = 100;
+    const clipPercentileMin = Math.round(v); // paranoia
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipPercentileMaxValueChange = v => {
+    /*
+    Ignore anything that isn't a legit number
+    */
+    if (!Number.isFinite(v)) return;
+
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMin = pendingClipPercentiles?.clipPercentileMin;
+
+    /*
+    clamp to [0, 100]
+    */
+    if (v < 0) v = 0;
+    if (v > 100) v = 100;
+    const clipPercentileMax = Math.round(v); // paranoia
+
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipCommit = () => {
+    const { dispatch } = this.props;
+    const { pendingClipPercentiles } = this.state;
+    const { clipPercentileMin, clipPercentileMax } = pendingClipPercentiles;
+    const min = clipPercentileMin / 100;
+    const max = clipPercentileMax / 100;
+    dispatch({
+      type: "set clip quantiles",
+      clipQuantiles: { min, max }
+    });
+  };
+
+  handleClipOpening = () => {
+    const { clipPercentileMin, clipPercentileMax } = this.props;
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipClosing = () => {
+    this.setState({ pendingClipPercentiles: null });
+  };
+
+  handleLayoutChoiceChange = e => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "set layout choice",
+      layoutChoice: e.currentTarget.value
+    });
+  };
+
+  computeDiffExp = () => {
+    const { dispatch, differential } = this.props;
+    if (differential.celllist1 && differential.celllist2) {
+      dispatch(
+        actions.requestDifferentialExpression(
+          differential.celllist1,
+          differential.celllist2
+        )
+      );
+    }
+  };
+
+  clearDifferentialExpression = () => {
+    const { dispatch, differential } = this.props;
+    dispatch({
+      type: "clear differential expression",
+      diffExp: differential.diffExp
+    });
+    dispatch({
+      type: "clear scatterplot"
+    });
+  };
+
+  renderDiffExp() {
+    /* diffexp-related buttons may be disabled */
+    const { disableDiffexp, differential, diffexpMayBeSlow } = this.props;
+    if (disableDiffexp) return null;
+
+    const haveBothCellSets =
+      !!differential.celllist1 && !!differential.celllist2;
+
+    const tipMessage =
+      "See top 10 differentially expressed genes" +
+      (diffexpMayBeSlow
+        ? " (CAUTION: large dataset - may take longer or fail)"
+        : "");
+
+    return (
+      <div className="bp3-button-group" style={{ marginRight: 10 }}>
+        <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
+        <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
+        {!differential.diffExp ? (
+          <Tooltip
+            content={tipMessage}
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelayQuick}
+          >
+            <AnchorButton
+              disabled={!haveBothCellSets}
+              intent="primary"
+              data-testid="diffexp-button"
+              loading={differential.loading}
+              icon="left-join"
+              fill
+              type="button"
+              onClick={this.computeDiffExp}
+            >
+              Compute Differential Expression
+            </AnchorButton>
+          </Tooltip>
+        ) : null}
+
+        {differential.diffExp ? (
+          <Tooltip
+            content="Remove differentially expressed gene list and clear cell selections"
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelayQuick}
+          >
+            <Button
+              type="button"
+              fill
+              intent="warning"
+              onClick={this.clearDifferentialExpression}
+            >
+              Clear Differential Expression
+            </Button>
+          </Tooltip>
+        ) : null}
+      </div>
+    );
+  }
+
+  render() {
+    const {
+      dispatch,
+      differential,
+      crossfilter,
+      resettingInterface,
+      libraryVersions,
+      undoDisabled,
+      redoDisabled,
+      selectionTool,
+      clipPercentileMin,
+      clipPercentileMax,
+      layoutChoice,
+      graphInteractionMode,
+      aboutLink
+    } = this.props;
+    const { pendingClipPercentiles } = this.state;
+
+    // constants used to create selection tool button
+    let selectionTooltip;
+    let selectionButtonClass;
+    if (selectionTool === "brush") {
+      selectionTooltip = "Brush selection";
+      selectionButtonClass = "bp3-icon-select";
+    } else {
+      selectionTooltip = "Lasso selection";
+      selectionButtonClass = "bp3-icon-polygon-filter";
+    }
+
+    return (
+      <div
+        style={{
+          position: "fixed",
+          right: 8,
+          top: 8
+        }}
+      >
+        {this.renderDiffExp()}
+        <Tooltip
+          content="Show only metadata and cells which are currently selected"
+          position="bottom"
+          hoverOpenDelay={tooltipHoverOpenDelay}
+        >
+          <AnchorButton
+            type="button"
+            data-testid="subset-button"
+            disabled={
+              crossfilter &&
+              (crossfilter.countSelected() === 0 ||
+                crossfilter.countSelected() === crossfilter.size())
+            }
+            style={{
+              marginRight: 10
+            }}
+            onClick={() => {
+              dispatch(actions.regraph());
+              dispatch({ type: "increment graph render counter" });
+            }}
+          >
+            subset to current selection
+          </AnchorButton>
+        </Tooltip>
+        <div className="bp3-button-group">
+          <Tooltip
+            content={selectionTooltip}
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelay}
+          >
+            <Button
+              type="button"
+              data-testid="mode-lasso"
+              className={`bp3-button ${selectionButtonClass}`}
+              active={graphInteractionMode === "select"}
+              onClick={() => {
+                dispatch({
+                  type: "change graph interaction mode",
+                  data: "select"
+                });
+              }}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
+          <Tooltip
+            content="Drag to pan, scroll to zoom"
+            position="bottom"
+            hoverOpenDelay={tooltipHoverOpenDelay}
+          >
+            <Button
+              type="button"
+              data-testid="mode-pan-zoom"
+              className="bp3-button bp3-icon-zoom-in"
+              active={graphInteractionMode === "zoom"}
+              onClick={() => {
+                dispatch({
+                  type: "change graph interaction mode",
+                  data: "zoom"
+                });
+              }}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
+        </div>
+        <div
+          className="bp3-button-group"
+          style={{
+            marginLeft: 10
+          }}
+        >
+          <Popover
+            target={
+              <Tooltip
+                content="Select embedding for visualization"
+                position="bottom"
+                hoverOpenDelay={tooltipHoverOpenDelay}
+              >
+                <Button
+                  type="button"
+                  data-testid="layout-choice"
+                  className="bp3-button bp3-icon-heatmap"
+                  style={{
+                    cursor: "pointer"
+                  }}
+                />
+              </Tooltip>
+            }
+            position={Position.BOTTOM_RIGHT}
+            content={
+              <div
+                style={{
+                  display: "flex",
+                  justifyContent: "flex-start",
+                  alignItems: "flex-start",
+                  flexDirection: "column",
+                  padding: 10
+                }}
+              >
+                <RadioGroup
+                  label="Embedding Choice"
+                  onChange={this.handleLayoutChoiceChange}
+                  selectedValue={layoutChoice.current}
+                >
+                  {layoutChoice.available.map(name => (
+                    <Radio label={name} value={name} key={name} />
+                  ))}
+                </RadioGroup>
+              </div>
+            }
+          />
+        </div>
+        <Clip
+          pendingClipPercentiles={pendingClipPercentiles}
+          clipPercentileMin={clipPercentileMin}
+          clipPercentileMax={clipPercentileMax}
+          handleClipOpening={this.handleClipOpening}
+          handleClipClosing={this.handleClipClosing}
+          handleClipCommit={this.handleClipCommit}
+          isClipDisabled={this.isClipDisabled}
+          handleClipOnKeyPress={this.handleClipOnKeyPress}
+          handleClipPercentileMaxValueChange={
+            this.handleClipPercentileMaxValueChange
+          }
+          handleClipPercentileMinValueChange={
+            this.handleClipPercentileMinValueChange
+          }
+        />
+        <UndoRedoReset
+          dispatch={dispatch}
+          isResetDisabled={this.isResetDisabled}
+          resetInterface={this.resetInterface}
+          resettingInterface={resettingInterface}
+          undoDisabled={undoDisabled}
+          redoDisabled={redoDisabled}
+        />
+        <InformationMenu
+          libraryVersions={libraryVersions}
+          aboutLink={aboutLink}
+        />
+      </div>
+    );
+  }
+}
+
+export default MenuBar;

client/src/components/menubar/infoMenu.js

@@ -0,0 +1,72 @@
+// jshint esversion: 6
+import React from "react";
+import { Button, Popover, Menu, MenuItem, Position } from "@blueprintjs/core";
+
+function InformationMenu(props) {
+  const { libraryVersions, aboutLink } = props;
+  return (
+    <div style={{ marginLeft: 10 }} className="bp3-button-group">
+      <Popover
+        content={
+          <Menu>
+            {aboutLink ? (
+              <MenuItem
+                href={aboutLink}
+                target="_blank"
+                icon="document-open"
+                text="About this dataset"
+              />
+            ) : (
+              ""
+            )}
+
+            <MenuItem
+              href="https://chanzuckerberg.github.io/cellxgene/faq.html"
+              target="_blank"
+              icon="help"
+              text="FAQ"
+            />
+            <MenuItem
+              href="https://join-cellxgene-users.herokuapp.com/"
+              target="_blank"
+              icon="chat"
+              text="Chat"
+            />
+            <MenuItem
+              href="https://chanzuckerberg.github.io/cellxgene/"
+              target="_blank"
+              icon="book"
+              text="Docs"
+            />
+            <MenuItem
+              href="https://github.com/chanzuckerberg/cellxgene"
+              target="_blank"
+              icon="git-branch"
+              text="Github"
+            />
+            <MenuItem
+              target="_blank"
+              text={`cellxgene v${
+                libraryVersions && libraryVersions.cellxgene
+                  ? libraryVersions.cellxgene
+                  : null
+              }`}
+            />
+            <MenuItem text="MIT License" />
+          </Menu>
+        }
+        position={Position.BOTTOM_RIGHT}
+      >
+        <Button
+          type="button"
+          className="bp3-button bp3-icon-info-sign"
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Popover>
+    </div>
+  );
+}
+
+export default InformationMenu;

client/src/components/menubar/undoRedoReset.js

@@ -0,0 +1,74 @@
+// jshint esversion: 6
+import React from "react";
+import { AnchorButton, Tooltip } from "@blueprintjs/core";
+import { tooltipHoverOpenDelay } from "../../globals";
+
+function InformationMenu(props) {
+  const {
+    resettingInterface,
+    undoDisabled,
+    redoDisabled,
+    resetInterface,
+    isResetDisabled,
+    dispatch
+  } = props;
+  return (
+    <div style={{ marginLeft: 10 }} className="bp3-button-group">
+      <Tooltip
+        content="Undo"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
+      >
+        <AnchorButton
+          type="button"
+          className="bp3-button bp3-icon-undo"
+          disabled={undoDisabled}
+          onClick={() => {
+            dispatch({ type: "@@undoable/undo" });
+          }}
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Tooltip>
+      <Tooltip
+        content="Redo"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
+      >
+        <AnchorButton
+          type="button"
+          className="bp3-button bp3-icon-redo"
+          disabled={redoDisabled}
+          onClick={() => {
+            dispatch({ type: "@@undoable/redo" });
+          }}
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Tooltip>
+      <Tooltip
+        content="Reset cellxgene, clearing all selections"
+        position="bottom"
+        hoverOpenDelay={tooltipHoverOpenDelay}
+      >
+        <AnchorButton
+          disabled={isResetDisabled()}
+          style={{ marginLeft: 10 }}
+          type="button"
+          loading={resettingInterface}
+          intent="none"
+          icon="refresh"
+          onClick={resetInterface}
+          data-testid="reset"
+          data-testclass={`resetting-${resettingInterface}`}
+        >
+          reset
+        </AnchorButton>
+      </Tooltip>
+    </div>
+  );
+}
+
+export default InformationMenu;

client/src/components/scatterplot/drawPointsRegl.js

@@ -1,44 +1,64 @@
-// jshint esversion: 6
+import { glPointFlags, glPointSize } from "../../util/glHelpers";
-const mat4 = require("gl-mat4");
-
-// opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js
 
 export default function(regl) {
   return regl({
     vert: `
     precision mediump float;
+
     attribute vec2 position;
     attribute vec3 color;
-    attribute float size;
+    attribute float flag;
-    uniform float distance;
+
-    uniform mat4 projection, view;
+    uniform mat3 projection;
-    varying vec3 fragColor;
+    uniform float nPoints;
+    uniform float minViewportDimension;
+
+    varying vec4 fragColor;
+
+    const float zBottom = 0.99;
+    const float zMiddle = 0.;
+    const float zTop = -1.;
+
+    // import getFlags()
+    ${glPointFlags}
+
+    // get pointSize()
+    ${glPointSize}
+
     void main() {
-      gl_PointSize = 7.0 / pow(distance, 2.5) + size;
+      bool isNaN, isSelected, isHighlight;
-      gl_Position = projection * view * vec4(position.x, -position.y, 0, 1);
+      getFlags(flag, isNaN, isSelected, isHighlight);
-      fragColor = color;
+
+      gl_PointSize = pointSize(nPoints, minViewportDimension, isSelected, isHighlight);
+
+      float z = isNaN ? zBottom : (isHighlight ? zTop : zMiddle);
+      vec3 xy = projection * vec3(position, 1.);
+      gl_Position = vec4(xy.xy, z, 1.);
+
+      float alpha = isNaN ? 0.9 : 1.0;
+      fragColor = vec4(color, alpha);
     }`,
 
     frag: `
     precision mediump float;
-    varying vec3 fragColor;
+    varying vec4 fragColor;
     void main() {
       if (length(gl_PointCoord.xy - 0.5) > 0.5) {
         discard;
       }
-      gl_FragColor = vec4(fragColor, 1);
+      gl_FragColor = fragColor;
     }`,
 
     attributes: {
       position: regl.prop("position"),
       color: regl.prop("color"),
-      size: regl.prop("size")
+      flag: regl.prop("flag")
     },
 
     uniforms: {
-      distance: regl.prop("distance"),
+      projection: regl.prop("projection"),
-      view: regl.prop("view"),
+      nPoints: regl.prop("nPoints"),
-      projection: () => mat4.perspective([], Math.PI / 2, 1, 0.01, 1000)
+      minViewportDimension: regl.prop("minViewportDimension")
     },
 
     count: regl.prop("count"),

client/src/components/scatterplot/scatterplot.js

@@ -1,49 +1,47 @@
-// jshint esversion: 6
-// https://bl.ocks.org/Jverma/076377dd0125b1a508621441752735fc
-// https://peterbeshai.com/scatterplot-in-d3-with-voronoi-interaction.html
-
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import { Button, ButtonGroup } from "@blueprintjs/core";
 import _regl from "regl";
 import * as d3 from "d3";
+import { mat3 } from "gl-matrix";
+import memoize from "memoize-one";
+import { isTypedArray } from "../../util/typeHelpers";
+
 import * as globals from "../../globals";
-
-import _camera from "../../util/camera";
-
 import setupScatterplot from "./setupScatterplot";
 import styles from "./scatterplot.css";
-
 import _drawPoints from "./drawPointsRegl";
-import scaleLinear from "../../util/scaleLinear";
-
 import { margin, width, height } from "./util";
 import finiteExtent from "../../util/finiteExtent";
 
+function createProjectionTF(viewportWidth, viewportHeight) {
+  /*
+  the projection transform accounts for the screen size & other layout
+  */
+  const m = mat3.create();
+  return mat3.projection(m, viewportWidth, viewportHeight);
+}
+
 @connect(state => {
-  const { world, crossfilter } = state;
+  const { world, crossfilter, universe } = state;
   const { scatterplotXXaccessor, scatterplotYYaccessor } = state.controls;
-  const expressionX =
+  const expressionX = scatterplotXXaccessor
-    world &&
+    ? world.varData.col(scatterplotXXaccessor)?.asArray()
-    scatterplotXXaccessor &&
+    : null;
-    world.varData.hasCol(scatterplotXXaccessor)
+  const expressionY = scatterplotYYaccessor
-      ? world.varData.col(scatterplotXXaccessor).asArray()
+    ? world.varData.col(scatterplotYYaccessor)?.asArray()
-      : null;
+    : null;
-  const expressionY =
-    world &&
-    scatterplotYYaccessor &&
-    world.varData.hasCol(scatterplotYYaccessor)
-      ? world.varData.col(scatterplotYYaccessor).asArray()
-      : null;
 
   return {
     world,
+    universe,
 
     colorRGB: state.colors.rgb,
     colorScale: state.colors.scale,
     colorAccessor: state.colors.colorAccessor,
 
+    centroidLabel: state.centroidLabel,
+
     // Accessors are var/gene names (strings)
     scatterplotXXaccessor,
     scatterplotYYaccessor,
@@ -54,10 +52,80 @@ import finiteExtent from "../../util/finiteExtent";
     expressionX,
     expressionY,
 
-    crossfilter
+    crossfilter,
+
+    responsive: state.responsive
   };
 })
-class Scatterplot extends React.Component {
+class Scatterplot extends React.PureComponent {
+  computePointPositions = memoize((X, Y, xScale, yScale) => {
+    const positions = new Float32Array(2 * X.length);
+    for (let i = 0, len = X.length; i < len; i += 1) {
+      positions[2 * i] = xScale(X[i]);
+      positions[2 * i + 1] = yScale(Y[i]);
+    }
+    return positions;
+  });
+
+  computePointColors = memoize(rgb => {
+    /*
+    compute webgl colors for each point
+    */
+    const colors = new Float32Array(3 * rgb.length);
+    for (let i = 0, len = rgb.length; i < len; i += 1) {
+      colors.set(rgb[i], 3 * i);
+    }
+    return colors;
+  });
+
+  computeSelectedFlags = memoize(
+    (crossfilter, flagSelected, flagUnselected) => {
+      const x = crossfilter.fillByIsSelected(
+        new Float32Array(crossfilter.size()),
+        flagSelected,
+        flagUnselected
+      );
+      return x;
+    }
+  );
+
+  computePointFlags = memoize(
+    (world, crossfilter, colorAccessor, centroidLabel) => {
+      const flagSelected = 1;
+      const flagNaN = 2;
+      const flagHighlight = 4;
+
+      const flags = this.computeSelectedFlags(
+        crossfilter,
+        flagSelected,
+        0
+      ).slice();
+
+      const { metadataField, categoryField } = centroidLabel;
+      const highlightData = metadataField
+        ? world.obsAnnotations.col(metadataField)?.asArray()
+        : null;
+      const colorByColumn = colorAccessor
+        ? world.obsAnnotations.col(colorAccessor)?.asArray() ||
+          world.varData.col(colorAccessor)?.asArray()
+        : null;
+      const colorByData =
+        colorByColumn && isTypedArray(colorByColumn) ? colorByColumn : null;
+
+      if (colorByData || highlightData) {
+        for (let i = 0, len = flags.length; i < len; i += 1) {
+          if (highlightData) {
+            flags[i] += highlightData[i] === categoryField ? flagHighlight : 0;
+          }
+          if (colorByData) {
+            flags[i] += Number.isFinite(colorByData[i]) ? 0 : flagNaN;
+          }
+        }
+      }
+      return flags;
+    }
+  );
+
   constructor(props) {
     super(props);
     this.count = 0;
@@ -65,7 +133,7 @@ class Scatterplot extends React.Component {
     this.renderCache = {
       positions: null,
       colors: null,
-      sizes: null,
+      flags: null,
       xScale: null,
       yScale: null
     };
@@ -86,39 +154,37 @@ class Scatterplot extends React.Component {
       this.renderCache = { ...this.renderCache, ...scales };
     }
 
-    const camera = _camera(this.reglCanvas, { scale: true, rotate: false });
     const regl = _regl(this.reglCanvas);
-
     const drawPoints = _drawPoints(regl);
 
+    // Create render transform
+    const projectionTF = createProjectionTF(
+      this.reglCanvas.width,
+      this.reglCanvas.height
+    );
+
     // preallocate buffers
     const pointBuffer = regl.buffer();
     const colorBuffer = regl.buffer();
-    const sizeBuffer = regl.buffer();
+    const flagBuffer = regl.buffer();
 
-    const reglRender = regl.frame(() => {
+    this.renderPoints(
-      this.reglDraw(
+      regl,
-        regl,
+      drawPoints,
-        drawPoints,
+      flagBuffer,
-        sizeBuffer,
+      colorBuffer,
-        colorBuffer,
+      pointBuffer,
-        pointBuffer,
+      projectionTF
-        camera
+    );
-      );
-      camera.tick();
-    });
-
-    this.reglRenderState = "rendering";
 
     this.setState({
       regl,
-      sizeBuffer,
+      flagBuffer,
       pointBuffer,
       colorBuffer,
       svg,
-      reglRender,
+      drawPoints,
-      camera,
+      projectionTF
-      drawPoints
     });
   }
 
@@ -130,22 +196,23 @@ class Scatterplot extends React.Component {
       scatterplotYYaccessor,
       expressionX,
       expressionY,
-      colorRGB
+      colorRGB,
+      colorAccessor,
+      centroidLabel
     } = this.props;
     const {
-      reglRender,
       regl,
       pointBuffer,
       colorBuffer,
-      sizeBuffer,
+      flagBuffer,
       svg,
       drawPoints,
-      camera
+      projectionTF
     } = this.state;
 
     if (
-      scatterplotXXaccessor !== prevProps.scatterplotXXaccessor || // was CLU now FTH1 etc
+      scatterplotXXaccessor !== prevProps.scatterplotXXaccessor ||
-      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor || // was CLU now FTH1 etc
+      scatterplotYYaccessor !== prevProps.scatterplotYYaccessor ||
       world !== prevProps.world // shape or clip of world changed
     ) {
       const scales = Scatterplot.setupScales(expressionX, expressionY);
@@ -153,76 +220,55 @@ class Scatterplot extends React.Component {
       this.renderCache = { ...this.renderCache, ...scales };
     }
 
-    if (reglRender && this.reglRenderState === "rendering") {
+    if (world && regl) {
-      reglRender.cancel();
-      this.reglRenderState = "paused";
-    }
-
-    if (
-      world &&
-      regl &&
-      pointBuffer &&
-      colorBuffer &&
-      sizeBuffer &&
-      expressionX &&
-      expressionY &&
-      scatterplotXXaccessor &&
-      scatterplotYYaccessor
-    ) {
       const { renderCache } = this;
       const { xScale, yScale } = this.renderCache;
-      const cellCount = expressionX.length;
+      let needsRepaint = false;
 
-      // Points change when expressionX or expressionY change.
+      const newPositions = this.computePointPositions(
-      if (
+        expressionX,
-        !renderCache.positions ||
+        expressionY,
-        expressionX !== prevProps.expressionX ||
+        xScale,
-        expressionY !== prevProps.expressionY
+        yScale
-      ) {
-        if (!renderCache.positions) {
-          renderCache.positions = new Float32Array(2 * cellCount);
-        }
-        const glScaleX = scaleLinear([0, width], [-0.95, 0.95]);
-        const glScaleY = scaleLinear([0, height], [-1, 1]);
-        for (let i = 0, { positions } = renderCache; i < cellCount; i += 1) {
-          positions[2 * i] = glScaleX(xScale(expressionX[i]));
-          positions[2 * i + 1] = glScaleY(yScale(expressionY[i]));
-        }
-        pointBuffer({ data: renderCache.positions, dimension: 2 });
-      }
-
-      // Colors for each point - change only when props.colorsRGB change.
-      if (!renderCache.colors || colorRGB !== prevProps.colorRGB) {
-        if (!renderCache.colors) {
-          renderCache.colors = new Float32Array(3 * cellCount);
-        }
-        for (let i = 0, { colors } = renderCache; i < cellCount; i += 1) {
-          colors.set(colorRGB[i], 3 * i);
-        }
-        colorBuffer({ data: renderCache.colors, dimension: 3 });
-      }
-
-      // Sizes for each point - updates are triggered only when selected
-      // obs change
-      if (!renderCache.sizes || crossfilter !== prevProps.crossfilter) {
-        if (!renderCache.sizes) {
-          renderCache.sizes = new Float32Array(cellCount);
-        }
-        crossfilter.fillByIsSelected(renderCache.sizes, 4, 0.2);
-        sizeBuffer({ data: renderCache.sizes, dimension: 1 });
-      }
-
-      this.count = cellCount;
-
-      regl._refresh();
-      this.reglDraw(
-        regl,
-        drawPoints,
-        sizeBuffer,
-        colorBuffer,
-        pointBuffer,
-        camera
       );
+      if (renderCache.positions !== newPositions) {
+        renderCache.positions = newPositions;
+        pointBuffer({ data: renderCache.positions, dimension: 2 });
+        needsRepaint = true;
+      }
+
+      /* colors for each point */
+      const newColors = this.computePointColors(colorRGB);
+      if (renderCache.colors !== newColors) {
+        renderCache.colors = newColors;
+        colorBuffer({ data: renderCache.colors, dimension: 3 });
+        needsRepaint = true;
+      }
+
+      const newFlags = this.computePointFlags(
+        world,
+        crossfilter,
+        colorAccessor,
+        centroidLabel
+      );
+      if (renderCache.flags !== newFlags) {
+        renderCache.flags = newFlags;
+        flagBuffer({ data: renderCache.flags, dimension: 1 });
+        needsRepaint = true;
+      }
+
+      this.count = expressionX.length;
+
+      if (needsRepaint) {
+        this.renderPoints(
+          regl,
+          drawPoints,
+          flagBuffer,
+          colorBuffer,
+          pointBuffer,
+          projectionTF
+        );
+      }
     }
   }
 
@@ -242,22 +288,6 @@ class Scatterplot extends React.Component {
     };
   }
 
-  reglDraw(regl, drawPoints, sizeBuffer, colorBuffer, pointBuffer, camera) {
-    regl.clear({
-      depth: 1,
-      color: [1, 1, 1, 1]
-    });
-
-    drawPoints({
-      size: sizeBuffer,
-      distance: camera.distance,
-      color: colorBuffer,
-      position: pointBuffer,
-      count: this.count,
-      view: camera.view()
-    });
-  }
-
   drawAxesSVG(xScale, yScale, svg) {
     const { scatterplotYYaccessor, scatterplotXXaccessor } = this.props;
     svg.selectAll("*").remove();
@@ -308,6 +338,41 @@ class Scatterplot extends React.Component {
       .text(scatterplotXXaccessor);
   }
 
+  renderPoints(
+    regl,
+    drawPoints,
+    flagBuffer,
+    colorBuffer,
+    pointBuffer,
+    projectionTF
+  ) {
+    if (!this.reglCanvas) return;
+    const { universe, responsive } = this.props;
+    // The viewport dimension is used to scale points, so we want to pass
+    // the dimension of the MAIN viewport, not the scatterplot viewport.
+    // Slightly hacky, but we want all points to scale uniformly.  Perhaps
+    // this should move to the redux state and be shared?
+    const { width: cvWidth, height: cvHeight } = responsive;
+    regl.poll();
+    regl.clear({
+      depth: 1,
+      color: [1, 1, 1, 1]
+    });
+    drawPoints({
+      flag: flagBuffer,
+      color: colorBuffer,
+      position: pointBuffer,
+      projection: projectionTF,
+      count: this.count,
+      nPoints: universe.nObs,
+      minViewportDimension: Math.min(
+        cvWidth - globals.leftSidebarWidth || width,
+        cvHeight || height
+      )
+    });
+    regl._gl.flush();
+  }
+
   render() {
     const { dispatch } = this.props;
     const { minimized } = this.state;
@@ -368,7 +433,7 @@ class Scatterplot extends React.Component {
             height={height}
             data-testid="scatterplot"
             style={{
-              marginLeft: margin.left - 7,
+              marginLeft: margin.left,
               marginTop: margin.top
             }}
             ref={canvas => {

client/src/globals.js

@@ -3,6 +3,9 @@ import { Colors } from "@blueprintjs/core";
 /* if a categorical metadata field has more options than this, truncate */
 export const maxCategoricalOptionsToDisplay = 100;
 
+/* default "unassigned" value for user-created categorical metadata */
+export const unassignedCategoryLabel = "unassigned";
+
 /*
 these are default values for configuration  the CLI may supply.
 See the REST API and CLI specs for more info.
@@ -11,8 +14,11 @@ export const configDefaults = {
   features: {},
   displayNames: {},
   parameters: {
-    "max-category-items": 1000
+    "max-category-items": 1000,
-  }
+    "disable-diffexp": false,
+    "diffexp-may-be-slow": false
+  },
+  links: {}
 };
 
 /* colors */
@@ -31,6 +37,7 @@ export const darkGreen = "#448C4D";
 
 export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";
+export const logoColor = "black"; /* logo pink: "#E9429A" */
 
 /* typography constants */
 
@@ -46,6 +53,8 @@ export const maxParagraphWidth = 600;
 export const cellxgeneTitleLeftPadding = 14;
 export const cellxgeneTitleTopPadding = 7;
 
+export const datasetTitleMaxCharacterCount = 25;
+
 export const maxControlsWidth = 800;
 
 export const graphMargin = { top: 20, right: 10, bottom: 30, left: 40 };
@@ -61,6 +70,12 @@ export const leftSidebarSectionHeading = {
   letterSpacing: ".05em"
 };
 export const leftSidebarSectionPadding = 10;
+export const categoryLabelDisplayStringLongLength = 35;
+export const categoryLabelDisplayStringShortLength = 15;
+
+/* various timing-related behaviors */
+export const tooltipHoverOpenDelay = 1000; /* ms delay before a tooltip displays */
+export const tooltipHoverOpenDelayQuick = 500;
 
 let _API = {
   // prefix: "http://api.clustering.czi.technology/api/",
@@ -72,114 +87,3 @@ let _API = {
 
 if (window.CELLXGENE && window.CELLXGENE.API) _API = window.CELLXGENE.API;
 export const API = _API;
-
-export const ordinalColors = [
-  "#0ac115",
-  "#c10ab6",
-  "#c1710a",
-  "#0a5ac1",
-  "#c1150a",
-  "#0ab6c1",
-  "#5ac10a",
-  "#710ac1",
-  "#0ac171",
-  "#c10a5a",
-  "#b6c10a",
-  "#150ac1",
-  "#b2ffb7",
-  "#ffb2fa",
-  "#ffddb2",
-  "#b2d4ff",
-  "#ffb7b2",
-  "#b2faff",
-  "#d4ffb2",
-  "#ddb2ff",
-  "#b2ffdd",
-  "#ffb2d4",
-  "#faffb2",
-  "#b7b2ff",
-  "#27a908",
-  "#8b08a9",
-  "#a93a08",
-  "#0877a9",
-  "#a90827",
-  "#08a98b",
-  "#77a908",
-  "#3a08a9",
-  "#08a93a",
-  "#a90877",
-  "#a98b08",
-  "#0827a9",
-  "#00ff0f",
-  "#ff00ef",
-  "#ff8e00",
-  "#0070ff",
-  "#ff0f00",
-  "#00efff",
-  "#70ff00",
-  "#8e00ff",
-  "#00ff8e",
-  "#ff0070",
-  "#efff00",
-  "#0f00ff",
-  "#006606",
-  "#66005f",
-  "#663900",
-  "#002c66",
-  "#660600",
-  "#005f66",
-  "#2c6600",
-  "#390066",
-  "#006639",
-  "#66002c",
-  "#5f6600",
-  "#060066",
-  "#83ff65",
-  "#e165ff",
-  "#ff9565",
-  "#65cfff",
-  "#ff6583",
-  "#65ffe1",
-  "#cfff65",
-  "#9565ff",
-  "#65ff95",
-  "#ff65cf",
-  "#ffe165",
-  "#6583ff",
-  "#009909",
-  "#99008f",
-  "#995500",
-  "#004399",
-  "#990900",
-  "#008f99",
-  "#439900",
-  "#550099",
-  "#009955",
-  "#990043",
-  "#8f9900",
-  "#090099",
-  "#d9fecc",
-  "#f1ccfe",
-  "#fed7cc",
-  "#ccf3fe",
-  "#feccd9",
-  "#ccfef1",
-  "#f3fecc",
-  "#d7ccfe",
-  "#ccfed7",
-  "#feccf3",
-  "#fef1cc",
-  "#ccd9fe",
-  "#47ea51",
-  "#ea47e0",
-  "#eaa247",
-  "#478fea",
-  "#ea5147",
-  "#47e0ea",
-  "#8fea47",
-  "#a247ea",
-  "#47eaa2",
-  "#ea478f",
-  "#e0ea47",
-  "#5147ea"
-];

client/src/reducers/annotations.js

@@ -0,0 +1,79 @@
+/*
+Reducers for annotation UI-state.
+*/
+const Annotations = (
+  state = {
+    isEditingCategoryName: false,
+    isEditingLabelName: false,
+    categoryBeingEdited: false,
+    labelEditable: { category: null, label: null }
+  },
+  action
+) => {
+  switch (action.type) {
+    /* CATEGORY */
+    case "annotation: activate add new label mode":
+      return {
+        ...state,
+        isAddingNewLabel: true,
+        categoryAddingNewLabel: action.data
+      };
+    case "annotation: disable add new label mode":
+      return {
+        ...state,
+        isAddingNewLabel: false,
+        categoryAddingNewLabel: null
+      };
+    case "annotation: add new label to category":
+      return {
+        ...state,
+        isAddingNewLabel: false,
+        categoryAddingNewLabel: null
+      };
+    case "annotation: activate category edit mode":
+      return {
+        ...state,
+        isEditingCategoryName: true,
+        categoryBeingEdited: action.data
+      };
+    case "annotation: disable category edit mode":
+      return {
+        ...state,
+        isEditingCategoryName: false,
+        categoryBeingEdited: null
+      };
+    case "annotation: category edited":
+      return {
+        ...state,
+        isEditingCategoryName: true,
+        categoryBeingEdited: null
+      };
+
+    /* LABEL */
+    case "annotation: activate edit label mode":
+      return {
+        ...state,
+        isEditingLabelName: true,
+        labelEditable: {
+          category: action.metadataField,
+          label: action.categoryIndex
+        }
+      };
+    case "annotation: cancel edit label mode":
+      return {
+        ...state,
+        isEditingLabelName: false,
+        labelEditable: { category: null, label: null }
+      };
+    case "annotation: label edited":
+      return {
+        ...state,
+        isEditingLabelName: false,
+        labelEditable: { category: null, label: null }
+      };
+    default:
+      return state;
+  }
+};
+
+export default Annotations;

client/src/reducers/autosave.js

@@ -0,0 +1,53 @@
+const Autosave = (
+	state = {
+		saveInProgress: false,
+		error: false,
+		lastSavedObsAnnotations: null
+	},
+	action,
+	nextSharedState
+) => {
+	switch (action.type) {
+		case "initial data load complete (universe exists)": {
+			/* don't save on init */
+			const { universe } = nextSharedState;
+			return {
+				...state,
+				error: false,
+				saveInProgress: false,
+				lastSavedObsAnnotations: universe.obsAnnotations
+			};
+		}
+
+		case "writable obs annotations - save started": {
+			return {
+				...state,
+				saveInProgress: true
+			};
+		}
+
+		case "writable obs annotations - save error": {
+			const { message } = action;
+			return {
+				...state,
+				error: message,
+				saveInProgress: false
+			};
+		}
+
+		case "writable obs annotations - save complete": {
+			const lastSavedObsAnnotations = action.obsAnnotations;
+			return {
+				...state,
+				saveInProgress: false,
+				error: false,
+				lastSavedObsAnnotations
+			};
+		}
+
+		default:
+			return { ...state };
+	}
+};
+
+export default Autosave;

client/src/reducers/categoricalSelection.js

@@ -1,12 +1,9 @@
-import _ from "lodash";
+import { ControlsHelpers as CH } from "../util/stateManager";
-
-import { ControlsHelpers } from "../util/stateManager";
 import * as globals from "../globals";
 
 function maxCategoryItems(state) {
-  return _.get(
+  return (
-    state.config,
+    state.config.parameters?.["max-category-items"] ??
-    "parameters.max-category-items",
     globals.configDefaults.parameters["max-category-items"]
   );
 }
@@ -23,25 +20,26 @@ const CategoricalSelection = (
     case "reset World to eq Universe":
     case "set clip quantiles": {
       const { world } = nextSharedState;
-      return ControlsHelpers.createCategoricalSelection(
+      const newState = CH.createCategoricalSelection(
-        maxCategoryItems(prevSharedState),
+        world,
-        world
+        CH.selectableCategoryNames(world, maxCategoryItems(prevSharedState))
       );
+      return newState;
     }
 
     case "categorical metadata filter select": {
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
+      const newCategoryValueSelected = Array.from(
-        state[action.metadataField].categorySelected
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = true;
+      newCategoryValueSelected[action.categoryIndex] = true;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -51,15 +49,15 @@ const CategoricalSelection = (
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
+      const newCategoryValueSelected = Array.from(
-        state[action.metadataField].categorySelected
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = false;
+      newCategoryValueSelected[action.categoryIndex] = false;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -73,8 +71,9 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
+          categorySelected: false,
-            state[action.metadataField].categorySelected
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(false)
         }
       };
@@ -89,14 +88,52 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
+          categorySelected: true,
-            state[action.metadataField].categorySelected
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(true)
         }
       };
       return newCategoricalSelection;
     }
 
+    case "annotation: create category": {
+      const { world } = nextSharedState;
+      const name = action.data;
+      return {
+        ...state,
+        ...CH.createCategoricalSelection(world, [name])
+      };
+    }
+
+    case "annotation: category edited": {
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      const { [name]: catSeln, ...newState } = state;
+      newState[newName] = catSeln;
+      return newState;
+    }
+
+    case "annotation: delete category": {
+      const name = action.metadataField;
+      const { [name]: _, ...newState } = state;
+      return newState;
+    }
+
+    case "annotation: label current cell selection":
+    case "annotation: add new label to category":
+    case "annotation: label edited":
+    case "annotation: delete label": {
+      /* need to rebuild the state for this annotation */
+      const { world } = nextSharedState;
+      const name = action.metadataField;
+      const { [name]: _, ...partialState } = state;
+      return {
+        ...partialState,
+        ...CH.createCategoricalSelection(world, [name])
+      };
+    }
+
     default: {
       return state;
     }

client/src/reducers/centroidLabel.js

@@ -0,0 +1,44 @@
+import calcCentroid from "../util/centroid";
+
+const initialState = {
+  metadataField: "",
+  categoryIndex: -1,
+  categoryField: "",
+  centroidXY: [-1, -1]
+};
+
+const CentroidLabel = (state = initialState, action, sharedNextState) => {
+  const { categoricalSelection, world, layoutChoice } = sharedNextState;
+  const { metadataField, categoryIndex } = action;
+  const categoryField =
+    categoricalSelection?.[metadataField]?.categoryValues[categoryIndex];
+  switch (action.type) {
+    case "category value mouse hover start":
+      return {
+        ...state,
+        metadataField,
+        categoryIndex,
+        categoryField,
+        centroidXY: null /* calcCentroid(  This function call is computationally heavy and also leading to large GC. Before reimplementation, look into optimization and memoization
+          world,
+          metadataField,
+          categoryField,
+          layoutChoice.currentDimNames
+        ) */
+      };
+
+    case "category value mouse hover end":
+      if (
+        metadataField === state.metadataField &&
+        categoryIndex === state.categoryIndex
+      ) {
+        return initialState;
+      }
+      return state;
+
+    default:
+      return state;
+  }
+};
+
+export default CentroidLabel;

client/src/reducers/colors.js

@@ -1,4 +1,4 @@
-import { createColors } from "../util/stateManager";
+import { ColorHelpers } from "../util/stateManager";
 
 const ColorsReducer = (
   state = {
@@ -17,7 +17,7 @@ const ColorsReducer = (
       const { world } = nextSharedState;
       const colorMode = null;
       const colorAccessor = null;
-      const { rgb, scale } = createColors(world, colorMode);
+      const { rgb, scale } = ColorHelpers.createColors(world, colorMode);
       return {
         ...state,
         colorAccessor,
@@ -29,9 +29,23 @@ const ColorsReducer = (
 
     case "set clip quantiles":
     case "set World to current selection": {
+      const { world: prevWorld, controls: prevControls } = prevSharedState;
+      const resetColorState = ColorHelpers.checkIfColorByDiffexpAndResetColors(
+        prevControls,
+        state,
+        prevWorld
+      );
+      if (resetColorState) {
+        return resetColorState;
+      }
+
       const { colorMode, colorAccessor } = state;
       const { world } = nextSharedState;
-      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
         rgb,
@@ -40,14 +54,9 @@ const ColorsReducer = (
     }
 
     case "reset colorscale": {
-      const { world } = prevSharedState;
-      const { rgb, scale } = createColors(world);
       return {
         ...state,
-        colorMode: null,
+        ...ColorHelpers.resetColors(prevSharedState.world)
-        colorAccessor: null,
-        rgb,
-        scale
       };
     }
 
@@ -62,7 +71,11 @@ const ColorsReducer = (
       const colorMode = !resetCurrent ? action.type : null;
       const colorAccessor = !resetCurrent ? action.colorAccessor : null;
 
-      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
         colorMode,
@@ -81,7 +94,11 @@ const ColorsReducer = (
       const colorMode = !resetCurrent ? action.type : null;
       const colorAccessor = !resetCurrent ? action.gene : null;
 
-      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
         colorMode,
@@ -91,6 +108,40 @@ const ColorsReducer = (
       };
     }
 
+    case "annotation: add new label to category":
+    case "annotation: label current cell selection":
+    case "annotation: delete label": {
+      const { world } = nextSharedState;
+      const { colorMode, colorAccessor } = state;
+      const { metadataField } = action;
+      if (
+        colorMode !== "color by categorical metadata" ||
+        colorAccessor !== metadataField
+      )
+        return state;
+
+      /* else, we need to rebuild colors as labels have changed! */
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
+      return { ...state, rgb, scale };
+    }
+
+    case "clear differential expression": {
+      const { world: prevWorld, controls: prevControls } = prevSharedState;
+      const resetColorState = ColorHelpers.checkIfColorByDiffexpAndResetColors(
+        prevControls,
+        state,
+        prevWorld
+      );
+      if (resetColorState) {
+        return resetColorState;
+      }
+      return state;
+    }
+
     default: {
       return state;
     }

client/src/reducers/controls.js

@@ -2,12 +2,10 @@
 
 import _ from "lodash";
 
-import { WorldUtil } from "../util/stateManager";
-
 const Controls = (
   state = {
     // data loading flag
-    loading: false,
+    loading: true,
     error: null,
 
     // all of the data + selection state
@@ -16,7 +14,7 @@ const Controls = (
     diffexpGenes: [],
 
     resettingInterface: false,
-
+    graphInteractionMode: "select",
     opacityForDeselectedCells: 0.2,
     scatterplotXXaccessor: null, // just easier to read
     scatterplotYYaccessor: null,
@@ -45,7 +43,6 @@ const Controls = (
     }
     case "initial data load complete (universe exists)": {
       /* first light - create world & other data-driven defaults */
-      WorldUtil.clearCaches();
       return {
         ...state,
         loading: false,
@@ -54,14 +51,12 @@ const Controls = (
       };
     }
     case "reset World to eq Universe": {
-      WorldUtil.clearCaches();
       return {
         ...state,
         resettingInterface: false
       };
     }
     case "set World to current selection": {
-      WorldUtil.clearCaches();
       return {
         ...state,
         loading: false,
@@ -93,9 +88,10 @@ const Controls = (
     }
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const _diffexpGenes = [];
       action.data.forEach(d => {
-        _diffexpGenes.push(world.varAnnotations.at(d[0], "name"));
+        _diffexpGenes.push(world.varAnnotations.at(d[0], varIndexName));
       });
       return {
         ...state,
@@ -137,6 +133,11 @@ const Controls = (
     /*******************************
              User Events
      *******************************/
+    case "change graph interaction mode":
+      return {
+        ...state,
+        graphInteractionMode: action.data
+      };
     case "change opacity deselected cells in 2d graph background":
       return {
         ...state,

client/src/reducers/crossfilter.js

@@ -1,7 +1,11 @@
 import _ from "lodash";
 
 import Crossfilter from "../util/typedCrossfilter";
-import { World, ControlsHelpers } from "../util/stateManager";
+import {
+  World,
+  ControlsHelpers as CH,
+  AnnotationsHelpers as AH
+} from "../util/stateManager";
 import {
   layoutDimensionName,
   obsAnnoDimensionName,
@@ -12,7 +16,7 @@ import {
 
 const XYDimName = layoutDimensionName("XY");
 
-const CrossfilterReducer = (
+const CrossfilterReducerBase = (
   state = null,
   action,
   nextSharedState,
@@ -20,10 +24,11 @@ const CrossfilterReducer = (
 ) => {
   switch (action.type) {
     case "initial data load complete (universe exists)": {
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       const crossfilter = World.createObsDimensions(
         new Crossfilter(world.obsAnnotations),
-        world
+        world,
+        layoutChoice.currentDimNames
       );
       return crossfilter;
     }
@@ -31,22 +36,28 @@ const CrossfilterReducer = (
     case "reset World to eq Universe": {
       const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
       const { world } = nextSharedState;
-      const crossfilter = ControlsHelpers.createGeneDimensions(
+      let { crossfilter } = prevSharedState.resetCache;
+      crossfilter = CH.createGeneDimensions(
         userDefinedGenes,
         diffexpGenes,
         world,
-        prevSharedState.resetCache.crossfilter
+        crossfilter
       );
+      crossfilter = AH.createWritableAnnotationDimensions(world, crossfilter);
       return crossfilter;
     }
 
     case "set clip quantiles":
     case "set World to current selection": {
       const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       let crossfilter = new Crossfilter(world.obsAnnotations);
-      crossfilter = World.createObsDimensions(crossfilter, world);
+      crossfilter = World.createObsDimensions(
-      crossfilter = ControlsHelpers.createGeneDimensions(
+        crossfilter,
+        world,
+        layoutChoice.currentDimNames
+      );
+      crossfilter = CH.createGeneDimensions(
         userDefinedGenes,
         diffexpGenes,
         world,
@@ -55,6 +66,23 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "set layout choice": {
+      /*
+      when switching layouts:
+      - delete the existing XY index
+      - add the new XY index (which implicitly selects all on it)
+      */
+      const { world, layoutChoice } = nextSharedState;
+      return state
+        .delDimension(layoutDimensionName("XY"))
+        .addDimension(
+          layoutDimensionName("XY"),
+          "spatial",
+          world.obsLayout.col(layoutChoice.currentDimNames[0]).asArray(),
+          world.obsLayout.col(layoutChoice.currentDimNames[1]).asArray()
+        );
+    }
+
     case "request user defined gene success": {
       const { world } = prevSharedState;
       const gene = action.data.genes[0];
@@ -68,8 +96,9 @@ const CrossfilterReducer = (
 
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const genes = _.map(action.data, d =>
-        world.varAnnotations.at(d[0], "name")
+        world.varAnnotations.at(d[0], varIndexName)
       );
       const crossfilter = _.reduce(
         genes,
@@ -87,10 +116,11 @@ const CrossfilterReducer = (
 
     case "clear differential expression": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const crossfilter = _.reduce(
         action.diffExp,
         (xfltr, values) => {
-          const name = world.varAnnotations.at(values[0], "name");
+          const name = world.varAnnotations.at(values[0], varIndexName);
           return xfltr.delDimension(diffexpDimensionName(name));
         },
         state
@@ -112,6 +142,37 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "annotation: create category": {
+      const name = action.data;
+      const { world } = nextSharedState;
+      const colData = world.obsAnnotations.col(name).asArray();
+      return state.addDimension(obsAnnoDimensionName(name), "enum", colData);
+    }
+
+    case "annotation: category edited": {
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      return state.renameDimension(
+        obsAnnoDimensionName(name),
+        obsAnnoDimensionName(newName)
+      );
+    }
+
+    case "annotation: delete category": {
+      return state.delDimension(obsAnnoDimensionName(action.metadataField));
+    }
+
+    case "annotation: label current cell selection":
+    case "annotation: label edited":
+    case "annotation: delete label": {
+      /* we need to reindex the dimension.  For now, just drop it and add another */
+      const name = action.metadataField;
+      const dimName = obsAnnoDimensionName(name);
+      const { world } = nextSharedState;
+      const colData = world.obsAnnotations.col(name).asArray();
+      return state.delDimension(dimName).addDimension(dimName, "enum", colData);
+    }
+
     case "graph brush end":
     case "graph brush change": {
       const [minX, maxY] = action.brushCoords.northwest;
@@ -162,9 +223,11 @@ const CrossfilterReducer = (
     case "categorical metadata filter deselect": {
       const { categoricalSelection } = nextSharedState;
       const cat = categoricalSelection[action.metadataField];
+      const { categoryValues, categoryValueSelected } = cat;
+      const values = categoryValues.filter((v, i) => categoryValueSelected[i]);
       return state.select(obsAnnoDimensionName(action.metadataField), {
         mode: "exact",
-        values: ControlsHelpers.selectedValuesForCategory(cat)
+        values
       });
     }
 
@@ -186,4 +249,29 @@ const CrossfilterReducer = (
   }
 };
 
+/*
+  IMPORTANT: the system assumes that crossfilter.data() will point at the
+  same value as world.obsAnnotations.  For actions handled in this reducer,
+  make sure that this remains true.
+
+  This wrapper performs only this function.
+*/
+const CrossfilterReducer = (
+  state,
+  action,
+  nextSharedState,
+  prevSharedState
+) => {
+  const nextState = CrossfilterReducerBase(
+    state,
+    action,
+    nextSharedState,
+    prevSharedState
+  );
+  if (!nextState || nextState.all() === nextSharedState.world.obsAnnotations) {
+    return nextState;
+  }
+  return nextState.setData(nextSharedState.world.obsAnnotations);
+};
+
 export default CrossfilterReducer;

client/src/reducers/graphSelection.js

@@ -7,7 +7,8 @@ const GraphSelection = (
 ) => {
   switch (action.type) {
     case "set clip quantiles":
-    case "reset World to eq Universe": {
+    case "reset World to eq Universe":
+    case "set layout choice": {
       return {
         ...state,
         selection: {

client/src/reducers/index.js

@@ -12,38 +12,49 @@ import graphSelection from "./graphSelection";
 import crossfilter from "./crossfilter";
 import colors from "./colors";
 import differential from "./differential";
+import layoutChoice from "./layoutChoice";
 import responsive from "./responsive";
 import controls from "./controls";
 import resetCache from "./resetCache";
+import centroidLabel from "./centroidLabel";
+import annotations from "./annotations";
+import autosave from "./autosave";
 
 import undoableConfig from "./undoableConfig";
 
 const Reducer = undoable(
-  cascadeReducers([
+    cascadeReducers([
-    ["config", config],
+        ["config", config],
-    ["universe", universe],
+        ["universe", universe],
-    ["world", world],
+        ["world", world],
-    ["categoricalSelection", categoricalSelection],
+        ["annotations", annotations],
-    ["continuousSelection", continuousSelection],
+        ["layoutChoice", layoutChoice],
-    ["graphSelection", graphSelection],
+        ["categoricalSelection", categoricalSelection],
-    ["crossfilter", crossfilter],
+        ["continuousSelection", continuousSelection],
-    ["colors", colors],
+        ["graphSelection", graphSelection],
-    ["controls", controls],
+        ["crossfilter", crossfilter],
-    ["differential", differential],
+        ["colors", colors],
-    ["responsive", responsive],
+        ["controls", controls],
-    ["resetCache", resetCache]
+        ["differential", differential],
-  ]),
+        ["responsive", responsive],
-  [
+        ["centroidLabel", centroidLabel],
-    "world",
+        ["autosave", autosave],
-    "categoricalSelection",
+        ["resetCache", resetCache]
-    "continuousSelection",
+    ]),
-    "graphSelection",
+    [
-    "crossfilter",
+        "universe",
-    "colors",
+        "world",
-    "controls",
+        "categoricalSelection",
-    "differential"
+        "continuousSelection",
-  ],
+        "graphSelection",
-  undoableConfig
+        "crossfilter",
+        "colors",
+        "controls",
+        "differential",
+        "layoutChoice",
+        "annotations"
+    ],
+    undoableConfig
 );
 
 const store = createStore(Reducer, applyMiddleware(thunk));

client/src/reducers/layoutChoice.js

@@ -0,0 +1,50 @@
+/*
+we have a UI heuristic to pick the default layout, based on assumptions
+about commonly used names.  Preferentially, pick in the following order:
+
+  1. "umap"
+  2. "tsne"
+  3. "pca"
+  4. give up, use the first available
+*/
+function bestDefaultLayout(layouts) {
+  const preferredNames = ["umap", "tsne", "pca"];
+  const idx = preferredNames.findIndex(name => layouts.indexOf(name) !== -1);
+  if (idx !== -1) return preferredNames[idx];
+  return layouts[0];
+}
+
+const LayoutChoice = (
+  state = {
+    available: [], // all available choices
+    current: undefined, // name of the current layout, eg, 'umap'
+    currentDimNames: [] // dimension name
+  },
+  action,
+  nextSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
+      // set default to default
+      const { schema } = nextSharedState.world;
+      const available = schema.layout.obs.map(v => v.name);
+      const current = bestDefaultLayout(available);
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { available, current, currentDimNames };
+    }
+
+    case "set layout choice": {
+      const { schema } = nextSharedState.world;
+      const current = action.layoutChoice;
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { ...state, current, currentDimNames };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default LayoutChoice;

client/src/reducers/resetCache.js

@@ -1,13 +1,13 @@
 /*
 Reducer which caches derived state to be used in a reset or other 
-recomputation.
+recomputation.  Add stuff here you want stashed at init time (or whenever),
+for later use.
 
-Currently this only caches the baseline (full universe) world & crossfilter, 
+Currently this only caches the baseline (full universe) crossfilter, 
-for use in a Reset.
+which improves Reset UI performance.
 */
 const ResetCacheReducer = (
   state = {
-    world: null,
     crossfilter: null
   },
   action,
@@ -15,10 +15,9 @@ const ResetCacheReducer = (
 ) => {
   switch (action.type) {
     case "initial data load complete (universe exists)": {
-      const { world, crossfilter } = nextSharedState;
+      const { crossfilter } = nextSharedState;
       return {
         ...state,
-        world,
         crossfilter
       };
     }

client/src/reducers/undoableConfig.js

@@ -29,7 +29,10 @@ const skipOnActions = new Set([
   "clear all user defined genes",
 
   "get single gene expression for coloring started",
-  "get single gene expression for coloring error"
+  "get single gene expression for coloring error",
+
+  "category value mouse hover start",
+  "category value mouse hover end"
 ]);
 
 /*
@@ -59,7 +62,7 @@ const saveOnActions = new Set([
   "categorical metadata filter select",
   "categorical metadata filter deselect",
   "categorical metadata filter all of these",
-  "categorical metadata none of these",
+  "categorical metadata filter none of these",
 
   "color by categorical metadata",
   "color by continuous metadata",
@@ -72,7 +75,18 @@ const saveOnActions = new Set([
   "store current cell selection as differential set 2",
 
   "set World to current selection",
-  "set clip quantiles"
+  "set clip quantiles",
+
+  "set layout choice",
+  "change graph interaction mode",
+
+  // user editable annotations
+  "annotation: create category",
+  "annotation: add new label to category",
+  "annotation: delete category",
+  "annotation: label edited",
+  "annotation: label current cell selection",
+  "annotation: delete label"
 ]);
 
 /**
@@ -98,9 +112,15 @@ const applyPending = () => ({
   [actionKey]: "applyPending",
   [stateKey]: { fsm: null }
 });
-const skip = fsm => ({ [actionKey]: "skip", [stateKey]: { fsm } });
+const skip = (fsm, transition) => ({
+  [actionKey]: "skip",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
 const clear = () => ({ [actionKey]: "clear", [stateKey]: { fsm: null } });
-const save = fsm => ({ [actionKey]: "save", [stateKey]: { fsm } });
+const save = (fsm, transition) => ({
+  [actionKey]: "save",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
 
 /*
 Error handler for state transitions that are unexpected.  Called by

client/src/reducers/universe.js

@@ -1,4 +1,9 @@
-import { ControlsHelpers } from "../util/stateManager";
+import { unassignedCategoryLabel } from "../globals";
+import {
+  World,
+  ControlsHelpers as CH,
+  AnnotationsHelpers as AH
+} from "../util/stateManager";
 
 const Universe = (state = null, action, nextSharedState, prevSharedState) => {
   switch (action.type) {
@@ -30,12 +35,186 @@ const Universe = (state = null, action, nextSharedState, prevSharedState) => {
           Object.keys(action.expressionData)
         )
       ];
-      varData = ControlsHelpers.pruneVarDataCache(varData, allTheGenesWeNeed);
+      varData = CH.pruneVarDataCache(varData, allTheGenesWeNeed);
+      return { ...state, varData };
+    }
 
-      return {
+    case "annotation: create category": {
-        ...state,
+      /* create a new annotation category, with all values set to 'unassigned' */
-        varData
+      const name = action.data;
+      const { categoryToDuplicate } = action;
+
+      /* name must be a string,  non-zero length */
+      if (typeof name !== "string" || name.length === 0)
+        throw new Error("user annotations require string name");
+
+      /* ensure the name isn't already in use! */
+      if (state.obsAnnotations.hasCol(name))
+        throw new Error("name collision on annotation category create");
+
+      /* ensure the duplicate col exists */
+      if (
+        categoryToDuplicate &&
+        !state.obsAnnotations.hasCol(categoryToDuplicate)
+      )
+        throw new Error("categoryToDuplicate does not exist");
+
+      let schema;
+      let data;
+      if (categoryToDuplicate) {
+        /* duplicate the named annotation */
+        schema = AH.dupObsAnnoSchema(state.schema, categoryToDuplicate, name, {
+          writable: true
+        });
+        /* if we are duplicating a non-writable annotation, it may not have an unassigned category */
+        const s = schema.annotations.obsByName[categoryToDuplicate];
+        if (s.categories.indexOf(unassignedCategoryLabel) === -1) {
+          s.categories = s.categories.concat(unassignedCategoryLabel);
+        }
+        data = state.obsAnnotations.col(categoryToDuplicate).asArray();
+      } else {
+        /* else, all are unassined */
+        const categories = [unassignedCategoryLabel];
+        schema = AH.addObsAnnoSchema(state.schema, name, {
+          name,
+          categories,
+          type: "categorical",
+          writable: true
+        });
+        data = new Array(state.nObs).fill(unassignedCategoryLabel);
+      }
+
+      const obsAnnotations = state.obsAnnotations.withCol(name, data);
+      return { ...state, obsAnnotations, schema };
+    }
+
+    case "annotation: category edited": {
+      /* change the name of an obs annotation category */
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      if (!AH.isUserAnnotation(state, name))
+        throw new Error("unable to edit read-only annotation");
+      if (typeof newName !== "string" || newName.length === 0)
+        throw new Error("user annotations require string name");
+
+      const colSchema = {
+        ...state.schema.annotations.obsByName[name],
+        name: newName
       };
+      const schema = AH.addObsAnnoSchema(
+        AH.removeObsAnnoSchema(state.schema, name),
+        newName,
+        colSchema
+      );
+      const obsAnnotations = state.obsAnnotations.renameCol(name, newName);
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: delete category": {
+      /* delete annotation category from schema and obsAnnotations */
+      const name = action.metadataField;
+      if (!AH.isUserAnnotation(state, name))
+        throw new Error("unable to delete read-only annotation");
+
+      const schema = AH.removeObsAnnoSchema(state.schema, name);
+      const obsAnnotations = state.obsAnnotations.dropCol(name);
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: add new label to category": {
+      const annotationName = action.metadataField;
+      const newLabelName = action.newLabelText;
+      if (!AH.isUserAnnotation(state, annotationName))
+        throw new Error("unable to modify read-only annotation");
+      if (typeof newLabelName !== "string" || newLabelName.length === 0)
+        throw new Error(
+          "user annotations require a non-zero length string name"
+        );
+
+      /* add the new label to the annotation */
+      const schema = AH.addObsAnnoCategory(
+        state.schema,
+        annotationName,
+        newLabelName
+      );
+      return { ...state, schema };
+    }
+
+    case "annotation: label edited": {
+      const annotationName = action.metadataField;
+      const oldLabelName = action.label;
+      const newLabelName = action.editedLabel;
+      if (!AH.isUserAnnotation(state, annotationName))
+        throw new Error("unable to modify read-only annotation");
+      if (typeof newLabelName !== "string" || newLabelName.length === 0)
+        throw new Error(
+          "user annotations require a non-zero length string name"
+        );
+
+      /* remove old label, add new label */
+      const schema = AH.addObsAnnoCategory(
+        AH.removeObsAnnoCategory(state.schema, annotationName, oldLabelName),
+        annotationName,
+        newLabelName
+      );
+
+      /* change all values in obsAnnotation */
+      const obsAnnotations = AH.setLabelByValue(
+        state.obsAnnotations,
+        annotationName,
+        oldLabelName,
+        newLabelName
+      );
+
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: delete label": {
+      /* delete the label from the annotation, and set all cells with this value to unassigned */
+      const annotationName = action.metadataField;
+      const labelName = action.label;
+      if (!AH.isUserAnnotation(state, annotationName))
+        throw new Error("unable to modify read-only annotation");
+      if (labelName === unassignedCategoryLabel)
+        throw new Error("may not remove the unassigned label");
+
+      /* remove the category from the schema */
+      const schema = AH.removeObsAnnoCategory(
+        state.schema,
+        annotationName,
+        labelName
+      );
+
+      /* set all values to unassigned in obsAnnotations */
+      const obsAnnotations = AH.setLabelByValue(
+        state.obsAnnotations,
+        annotationName,
+        labelName,
+        unassignedCategoryLabel
+      );
+
+      return { ...state, schema, obsAnnotations };
+    }
+
+    case "annotation: label current cell selection": {
+      const { metadataField, label } = action;
+      const { world, crossfilter } = prevSharedState;
+
+      /*
+      selection state is relative to world.  We need to convert it
+      to a mask for Universe before applying it.
+      */
+      const worldMask = crossfilter.allSelectedMask();
+      const mask = World.worldEqUniverse(world, state)
+        ? worldMask
+        : AH.worldToUniverseMask(worldMask, world.obsAnnotations, state.nObs);
+      const obsAnnotations = AH.setLabelByMask(
+        state.obsAnnotations,
+        metadataField,
+        mask,
+        label
+      );
+      return { ...state, obsAnnotations };
     }
 
     default: {

client/src/reducers/world.js

@@ -1,7 +1,22 @@
-import { World, ControlsHelpers } from "../util/stateManager";
+import { unassignedCategoryLabel } from "../globals";
+import {
+  World,
+  ControlsHelpers as CH,
+  AnnotationsHelpers as AH
+} from "../util/stateManager";
 import clip from "../util/clip";
 import quantile from "../util/quantile";
 
+/*
+important note: much of this code assumes that wriable (user) annotations
+will NOT contain scalar data (ie, will only contain categorical labelled
+data), and therefore will never need to be clipped.  Put another way, it
+assumes that for these annotations, the clipped & unclipped data is equal.
+
+If we ever start allowing user editable scalar data, this assumption will
+need to be revisited.
+*/
+
 const WorldReducer = (
   state = null,
   action,
@@ -9,16 +24,13 @@ const WorldReducer = (
   prevSharedState
 ) => {
   switch (action.type) {
-    case "initial data load complete (universe exists)": {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
       const { universe } = nextSharedState;
       const world = World.createWorldFromEntireUniverse(universe);
       return world;
     }
 
-    case "reset World to eq Universe": {
-      return prevSharedState.resetCache.world;
-    }
-
     case "set World to current selection": {
       /* Set viewable world to be the currently selected data */
       const world = World.createWorldBySelection(
@@ -80,7 +92,7 @@ const WorldReducer = (
           Object.keys(action.expressionData)
         )
       ];
-      unclippedVarData = ControlsHelpers.pruneVarDataCache(
+      unclippedVarData = CH.pruneVarDataCache(
         unclippedVarData,
         allTheGenesWeNeed
       );
@@ -122,6 +134,135 @@ const WorldReducer = (
       };
     }
 
+    case "annotation: create category": {
+      const name = action.data;
+      const { universe } = nextSharedState;
+      const { schema } = universe;
+
+      /*
+      if world !== universe, we have to subset the newly created annotation,
+      else, just use it as is.
+      */
+      let newAnnotation = null;
+      if (!World.worldEqUniverse(state, universe)) {
+        newAnnotation = universe.obsAnnotations
+          .subset(state.obsAnnotations.rowIndex.keys(), [name], null)
+          .icol(0)
+          .asArray();
+      } else {
+        newAnnotation = universe.obsAnnotations.col(name).asArray();
+      }
+      const obsAnnotations = state.obsAnnotations.withCol(
+        name,
+        newAnnotation,
+        state.obsAnnotations.rowIndex
+      );
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: state.unclipped.obsAnnotations.withCol(
+          name,
+          newAnnotation,
+          state.unclipped.obsAnnotations.rowIndex
+        )
+      };
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: category edited": {
+      /* change the name of an obs annotation */
+      const name = action.metadataField;
+      const newName = action.newCategoryText;
+      const { schema } = nextSharedState.universe;
+      const obsAnnotations = state.obsAnnotations.renameCol(name, newName);
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: state.unclipped.obsAnnotations.renameCol(name, newName)
+      };
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: delete category": {
+      /* remove a category from obs annotation */
+      const { schema } = nextSharedState.universe;
+      const name = action.metadataField;
+      const obsAnnotations = state.obsAnnotations.dropCol(name);
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: state.unclipped.obsAnnotations.dropCol(name)
+      };
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: add new label to category": {
+      /* add a new label to the schema - schema updated by universe reducer, we just need to note it */
+      const { schema } = nextSharedState.universe;
+      return { ...state, schema };
+    }
+
+    case "annotation: label edited": {
+      const { schema } = nextSharedState.universe;
+      const { metadataField } = action;
+      const oldLabelName = action.label;
+      const newLabelName = action.editedLabel;
+
+      /* set all values to to new label */
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: AH.setLabelByValue(
+          state.unclipped.obsAnnotations,
+          metadataField,
+          oldLabelName,
+          newLabelName
+        )
+      };
+      const obsAnnotations = state.obsAnnotations.replaceColData(
+        metadataField,
+        unclipped.obsAnnotations.col(metadataField).asArray()
+      );
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: delete label": {
+      const { schema } = nextSharedState.universe;
+      const { label, metadataField } = action;
+
+      /* set all values to unassigned in obsAnnotations */
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: AH.setLabelByValue(
+          state.unclipped.obsAnnotations,
+          metadataField,
+          label,
+          unassignedCategoryLabel
+        )
+      };
+      const obsAnnotations = state.obsAnnotations.replaceColData(
+        metadataField,
+        unclipped.obsAnnotations.col(metadataField).asArray()
+      );
+      return { ...state, schema, obsAnnotations, unclipped };
+    }
+
+    case "annotation: label current cell selection": {
+      const { metadataField, label } = action;
+      const { crossfilter } = prevSharedState;
+      const mask = crossfilter.allSelectedMask();
+      const unclipped = {
+        ...state.unclipped,
+        obsAnnotations: AH.setLabelByMask(
+          state.unclipped.obsAnnotations,
+          metadataField,
+          mask,
+          label
+        )
+      };
+      const obsAnnotations = state.obsAnnotations.replaceColData(
+        metadataField,
+        unclipped.obsAnnotations.col(metadataField).asArray()
+      );
+      return { ...state, obsAnnotations, unclipped };
+    }
+
     default: {
       return state;
     }

client/src/util/camera.js

@@ -1,67 +1,181 @@
-// jshint esversion: 6
+import { vec2, mat3 } from "gl-matrix";
-const createCamera = require("orbit-camera");
+
-const createScroll = require("scroll-speed");
+const EPSILON = 0.000001;
-const mp = require("mouse-position");
-const mb = require("mouse-pressed");
-const key = require("key-pressed");
 
-const panSpeed = 0.4;
 const scaleSpeed = 0.5;
-const scaleMax = 3;
+const scaleMax = 3.0;
-// const scaleMin = 1.15
+const scaleMin = 0.5;
-const scaleMin = 1.03;
+const panBound = 0.8;
 
-function attachCamera(canvas, opts) {
+// private
-  opts = opts || {};
+const scratch0 = new Float32Array(16);
-  opts.pan = opts.pan !== false;
+const scratch1 = new Float32Array(16);
-  opts.scale = opts.scale !== false;
-  opts.rotate = opts.rotate !== false;
 
-  const scroll = createScroll(canvas, opts.scale);
+function clamp(val, rng) {
-  const mbut = mb(canvas, opts.rotate);
+  return Math.max(Math.min(val, rng[1]), rng[0]);
-  const mpos = mp(canvas);
+}
-  const camera = createCamera([0, 0, 1], [0, 0, -1], [0, 1, 0]);
 
-  camera.tick = tick;
+class Camera {
+  constructor(canvas) {
+    this.prevEvent = {
+      clientX: 0,
+      clientY: 0,
+      type: 0
+    };
+    this.canvas = canvas;
+    this.viewMatrix = mat3.create();
+    this.viewMatrixInv = mat3.create();
+  }
 
-  return camera;
+  view() {
+    return this.viewMatrix;
+  }
 
-  function tick() {
+  invView() {
-    const ctrl = key("<control>") || key("<alt>");
+    return this.viewMatrixInv;
-    const alt = key("<shift>");
+  }
-    const { height, width } = canvas;
 
-    if (opts.rotate && mbut.left && ctrl && !alt) {
+  distance() {
-      camera.rotate(
+    return this.viewMatrix[0];
-        [mpos.x / width - 0.5, mpos.y / height - 0.5],
+  }
-        [mpos.prevX / width - 0.5, mpos.prevY / height - 0.5]
+
-      );
+  pan(dx, dy) {
+    const m = this.viewMatrix;
+    const dyRange = [
+      -panBound - (m[7] + 1) / m[4],
+      panBound - (m[7] - 1) / m[4]
+    ];
+    const dxRange = [
+      -panBound - (m[6] + 1) / m[0],
+      panBound - (m[6] - 1) / m[0]
+    ];
+
+    const dxClamped = clamp(dx, dxRange);
+    const dyClamped = clamp(dy, dyRange);
+    if (Math.abs(dxClamped) <= EPSILON && Math.abs(dyClamped) <= EPSILON)
+      return;
+
+    mat3.translate(m, m, [dxClamped, dyClamped]);
+    mat3.invert(this.viewMatrixInv, m);
+  }
+
+  zoomAt(d, x = 0, y = 0) {
+    /*
+    Camera zoom at [x,y]
+    */
+    const m = this.viewMatrix;
+    const bounds = [-panBound, panBound];
+    x = clamp(x, bounds);
+    y = clamp(y, bounds);
+
+    const dClamped = clamp(d * m[0], [scaleMin, scaleMax]) / m[0];
+    if (Math.abs(1 - dClamped) <= EPSILON) return; // noop request
+
+    mat3.translate(m, m, [x, y]);
+    mat3.scale(m, m, [dClamped, dClamped]);
+    mat3.translate(m, m, [-x, -y]);
+
+    mat3.invert(this.viewMatrixInv, m);
+  }
+
+  /*
+  Event handling
+  */
+
+  flush(e) {
+    this.prevEvent.type = e.type;
+    this.prevEvent.clientX = e.clientX;
+    this.prevEvent.clientY = e.clientY;
+  }
+
+  localPosition(target, canvasX, canvasY, projectionInvTF) {
+    /*
+    Convert mouse position to local
+    */
+    const { height, width } = target;
+    const targetRect = target.getBoundingClientRect();
+    canvasX -= targetRect.left;
+    canvasY -= targetRect.top;
+
+    const pos = vec2.fromValues(
+      2 * (canvasX / width) - 1,
+      -2 * (canvasY / height) + 1
+    );
+    if (projectionInvTF) {
+      vec2.transformMat3(pos, pos, projectionInvTF);
     }
+    vec2.transformMat3(pos, pos, this.invView());
+    return pos;
+  }
 
-    if ((opts.pan && mbut.right) || (mbut.left && !ctrl && !alt)) {
+  mousePan(e, projectionTF) {
-      camera.pan([
+    const projectionInvTF = mat3.invert(scratch0, projectionTF);
-        ((panSpeed * (mpos[0] - mpos.prev[0])) / width) * camera.distance,
+    const pos = this.localPosition(
-        ((panSpeed * (mpos[1] - mpos.prev[1])) / height) * camera.distance
+      this.canvas,
-      ]);
+      e.clientX,
+      e.clientY,
+      projectionInvTF
+    );
+    const prev = this.localPosition(
+      this.canvas,
+      this.prevEvent.clientX,
+      this.prevEvent.clientY,
+      projectionInvTF
+    );
+
+    const delta = vec2.sub(scratch1, pos, prev);
+    this.pan(delta[0], delta[1]);
+    return true;
+  }
+
+  wheelZoom(e, projectionTF) {
+    const { height } = this.canvas;
+    const { deltaY, deltaMode, clientX, clientY } = e;
+    const scale = scaleSpeed * (deltaMode === 1 ? 12 : 1) * (deltaY || 0);
+
+    const projectionInvTF = mat3.invert(scratch0, projectionTF);
+    const pos = this.localPosition(
+      this.canvas,
+      clientX,
+      clientY,
+      projectionInvTF
+    );
+    this.zoomAt(1 / Math.exp(scale / height), pos[0], pos[1]);
+    return true;
+  }
+
+  handleEvent(e, projectionTF) {
+    /*
+    process the event, and return true if camera view changed
+    */
+    let viewChanged = false;
+    switch (e.type) {
+      case "mousemove": {
+        /* eslint-disable no-bitwise */
+        if (e.buttons & 0x1) {
+          viewChanged = this.mousePan(e, projectionTF);
+        }
+        /* eslint-enable no-bitwise */
+        this.flush(e);
+        break;
+      }
+
+      case "wheel": {
+        viewChanged = this.wheelZoom(e, projectionTF);
+        this.flush(e);
+        break;
+      }
+
+      default:
+        // noop
+        break;
     }
-
+    return viewChanged;
-    if (opts.scale && scroll[1]) {
-      camera.distance *= Math.exp((scroll[1] * scaleSpeed) / height);
-    }
-
-    if (opts.scale && (mbut.middle || (mbut.left && !ctrl && alt))) {
-      const d = mpos.y - mpos.prevY;
-      if (!d) return;
-
-      camera.distance *= Math.exp(d / height);
-    }
-
-    if (camera.distance > scaleMax) camera.distance = scaleMax;
-    if (camera.distance < scaleMin) camera.distance = scaleMin;
-
-    scroll.flush();
-    mpos.flush();
   }
 }
 
+function attachCamera(canvas) {
+  return new Camera(canvas);
+}
+
 export default attachCamera;

client/src/util/centroid.js

@@ -0,0 +1,58 @@
+import quantile from "./quantile";
+
+/*
+  Centroid coordinate calculation
+*/
+const calcMeanCentroid = (world, annoName, annoValue, layoutDimNames) => {
+  const centroid = { x: 0, y: 0, size: 0 };
+  const annoArray = world.obsAnnotations.col(annoName).asArray();
+  const layoutXArray = world.obsLayout.col(layoutDimNames[0]).asArray();
+  const layoutYArray = world.obsLayout.col(layoutDimNames[1]).asArray();
+
+  for (let i = 0, len = annoArray.length; i < len; i += 1) {
+    if (annoArray[i] === annoValue) {
+      centroid.x += layoutXArray[i];
+      centroid.y += layoutYArray[i];
+      centroid.size += 1;
+    }
+  }
+
+  if (centroid[2] !== 0) {
+    centroid.x /= centroid.size;
+    centroid.y /= centroid.size;
+  }
+
+  return [centroid.x, centroid.y];
+};
+
+const calcMedianCentroid = (world, annoName, annoValue, layoutDimNames) => {
+  const centroidX = [];
+  const centroidY = [];
+  let hasFinite = false;
+
+  const annoArray = world.obsAnnotations.col(annoName).asArray();
+  const layoutXArray = world.obsLayout.col(layoutDimNames[0]).asArray();
+  const layoutYArray = world.obsLayout.col(layoutDimNames[1]).asArray();
+
+  for (let i = 0, len = annoArray.length; i < len; i += 1) {
+    if (annoArray[i] === annoValue) {
+      hasFinite =
+        Number.isFinite(layoutXArray[i]) || Number.isFinite(layoutYArray[i])
+          ? true
+          : hasFinite;
+      centroidX.push(layoutXArray[i]);
+      centroidY.push(layoutYArray[i]);
+    }
+  }
+
+  if (hasFinite) {
+    const medianX = quantile([0.5], Float64Array.from(centroidX));
+    const medianY = quantile([0.5], Float64Array.from(centroidY));
+
+    return [medianX, medianY];
+  }
+
+  return null;
+};
+
+export default calcMedianCentroid;

client/src/util/dataframe/dataframe.js

@@ -1,8 +1,19 @@
 import { IdentityInt32Index, isLabelIndex } from "./labelIndex";
 // weird cross-dependency that we should clean up someday...
 import { sortArray } from "../typedCrossfilter/sort";
-import { isTypedArray, isArrayOrTypedArray, callOnceLazy } from "./util";
+import {
+  isTypedArray,
+  isArrayOrTypedArray,
+  callOnceLazy,
+  memoize
+} from "./util";
 import { summarizeContinuous, summarizeCategorical } from "./summarize";
+import {
+  histogramCategorical,
+  hashCategorical,
+  histogramContinuous,
+  hashContinuous
+} from "./histogram";
 
 /*
 Dataframe is an immutable 2D matrix similiar to Python Pandas Dataframe,
@@ -59,6 +70,17 @@ Dataframe
 **/
 
 class Dataframe {
+  /**
+  memoization helpers.
+  **/
+  static __DataframeId__ = 0;
+
+  static __getId() {
+    const id = Dataframe.__DataframeId__;
+    Dataframe.__DataframeId__ += 1;
+    return id;
+  }
+
   /**
   Constructors & factories
   **/
@@ -102,8 +124,10 @@ class Dataframe {
     this.length = nRows; // convenience accessor for row dimension
     this.rowIndex = rowIndex;
     this.colIndex = colIndex;
+    this.__id = Dataframe.__getId();
 
     this.__compile(__columnsAccessor);
+    Object.freeze(this);
   }
 
   static __errorChecks(dims, columnarData, rowIndex, colIndex) {
@@ -144,7 +168,7 @@ class Dataframe {
     }
   }
 
-  static __compileColumn(column, getOffset, getLabel) {
+  static __compileColumn(column, getRowByOffset, getRowByLabel) {
     /*
       Each column accessor is a function which will lookup data by
       index (ie, is equivalent to dataframe.get(row, col), where 'col'
@@ -172,12 +196,15 @@ class Dataframe {
 
       iget(offset) -- return the value at 'offset'
 
+      ... and more ...
+
     */
     const { length } = column;
+    const __id = Dataframe.__getId();
 
     /* get value by row label */
     const get = function get(rlabel) {
-      return column[getOffset(rlabel)];
+      return column[getRowByOffset(rlabel)];
     };
 
     /* get value by row offset */
@@ -192,7 +219,7 @@ class Dataframe {
 
     /* test for row label inclusion in column */
     const has = function has(rlabel) {
-      const offset = getOffset(rlabel);
+      const offset = getRowByOffset(rlabel);
       return offset >= 0 && offset < length;
     };
 
@@ -212,7 +239,7 @@ class Dataframe {
       if (offset === -1) {
         return undefined;
       }
-      return getLabel(offset);
+      return getRowByLabel(offset);
     };
 
     /*
@@ -224,12 +251,26 @@ class Dataframe {
         : summarizeCategorical(column)
     );
 
+    /*
+    Create histogram bins for this column.  Memoized.
+    */
+    if (isTypedArray(column)) {
+      const mFn = memoize(histogramContinuous, hashContinuous);
+      get.histogram = (bins, domain, by) => mFn(get, bins, domain, by);
+    } else {
+      const mFn = memoize(histogramCategorical, hashCategorical);
+      get.histogram = by => mFn(get, by);
+    }
+
     get.summarize = summarize;
     get.asArray = asArray;
     get.has = has;
     get.ihas = ihas;
     get.indexOf = indexOf;
     get.iget = iget;
+    get.__id = __id;
+
+    Object.freeze(get);
     return get;
   }
 
@@ -239,13 +280,17 @@ class Dataframe {
 
     Use an existing accessor if provided, else compile a new one.
     */
-    const { getOffset, getLabel } = this.rowIndex;
+    const {
+      getOffset: getRowByOffset,
+      getLabel: getRowByLabel
+    } = this.rowIndex;
     this.__columnsAccessor = this.__columns.map((column, idx) => {
       if (accessors[idx]) {
         return accessors[idx];
       }
-      return Dataframe.__compileColumn(column, getOffset, getLabel);
+      return Dataframe.__compileColumn(column, getRowByOffset, getRowByLabel);
     });
+    Object.freeze(this.__columnsAccessor);
   }
 
   clone() {
@@ -303,12 +348,50 @@ class Dataframe {
     );
   }
 
+  withColsFrom(dataframe) {
+    /*
+    return a new dataframe containing all columns from both `this` and the
+    provided dataframe.
+
+    The row index from `this` will be used.  Both dataframes must have identical
+    dimensionality, and no overlapping columns labels.
+     */
+    const dims = [this.dims[0], this.dims[1] + dataframe.dims[1]];
+    const { rowIndex } = this;
+    const columns = [...this.__columns, ...dataframe.__columns];
+    const colIndex = this.colIndex.withLabels(dataframe.colIndex.keys());
+    const columnsAccessor = [
+      ...this.__columnsAccessor,
+      ...dataframe.__columnsAccessor
+    ];
+    return new this.constructor(
+      dims,
+      columns,
+      rowIndex,
+      colIndex,
+      columnsAccessor
+    );
+  }
+
   dropCol(label) {
     /*
     Create a new dataframe, omitting one columns.
 
     const newDf = df.dropCol("colors");
+
+    Corner case to manage: if dropping the last column, return an empty dataframe.
     */
+    if (!this.hasCol(label)) {
+      throw new RangeError(`unknown label: ${label}`);
+    }
+
+    /* 
+    Corner case to manage: if dropping the last column, return an empty dataframe. 
+    */
+    if (this.dims[1] === 1) {
+      return Dataframe.empty();
+    }
+
     const dims = [this.dims[0], this.dims[1] - 1];
     const coffset = this.colIndex.getOffset(label);
     const columns = [...this.__columns];
@@ -325,6 +408,50 @@ class Dataframe {
     );
   }
 
+  renameCol(oldLabel, newLabel) {
+    /*
+    Accelerator for dropping a column and then adding it again with a new label
+    */
+    const coffset = this.colIndex.getOffset(oldLabel);
+    const colIndex = this.colIndex.dropLabel(oldLabel).withLabel(newLabel);
+
+    const columns = [...this.__columns];
+    columns.push(columns[coffset]);
+    columns.splice(coffset, 1);
+
+    const columnsAccessor = [...this.__columnsAccessor];
+    columnsAccessor.push(columnsAccessor[coffset]);
+    columnsAccessor.splice(coffset, 1);
+
+    return new this.constructor(
+      this.dims,
+      columns,
+      this.rowIndex,
+      colIndex,
+      columnsAccessor
+    );
+  }
+
+  replaceColData(label, newColData) {
+    /*
+    Accelerator for dropping a column then adding it again with same
+    label and different values.
+    */
+    const coffset = this.colIndex.getOffset(label);
+    const columns = [...this.__columns];
+    columns[coffset] = newColData;
+    const columnsAccessor = [...this.__columnsAccessor];
+    columnsAccessor[coffset] = null;
+
+    return new this.constructor(
+      this.dims,
+      columns,
+      this.rowIndex,
+      this.colIndex,
+      columnsAccessor
+    );
+  }
+
   static empty(rowIndex = null, colIndex = null) {
     return new Dataframe([0, 0], [], rowIndex, colIndex);
   }
@@ -401,6 +528,8 @@ class Dataframe {
         return newCol;
       });
     }
+
+    if (dims[0] === 0 || dims[1] === 0) return Dataframe.empty();
     return new Dataframe(dims, columns, rowIndex, colIndex);
   }
 
@@ -484,6 +613,11 @@ class Dataframe {
   Data access with row/col.
   **/
 
+  columns() {
+    /* return all column accessors as an array, in offset order */
+    return [...this.__columnsAccessor];
+  }
+
   col(columnLabel) {
     /*
     Return accessor bound to a column.  Allows random row access

client/src/util/dataframe/histogram.js

@@ -0,0 +1,135 @@
+/*
+Dataframe histogram
+*/
+import { isTypedArray } from "./util";
+
+function _histogramContinuous(column, bins, min, max) {
+  const valBins = new Array(bins).fill(0);
+  if (!column) {
+    return valBins;
+  }
+  const binWidth = (max - min) / bins;
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const val = colArray[r];
+    if (val <= max && val >= min) {
+      // ensure test excludes NaN values
+      const valBin = Math.min(Math.floor((val - min) / binWidth), bins - 1);
+      valBins[valBin] += 1;
+    }
+  }
+  return valBins;
+}
+
+function _histogramContinuousBy(column, bins, min, max, by) {
+  const byMap = new Map();
+  if (!column || !by) {
+    return byMap;
+  }
+  const binWidth = (max - min) / bins;
+  const byArray = by.asArray();
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const byBin = byArray[r];
+    let valBins = byMap.get(byBin);
+    if (valBins === undefined) {
+      valBins = new Array(bins).fill(0);
+      byMap.set(byBin, valBins);
+    }
+    const val = colArray[r];
+    if (val <= max && val >= min) {
+      // ensure test excludes NaN values
+      const valBin = Math.min(Math.floor((val - min) / binWidth), bins - 1);
+      valBins[valBin] += 1;
+    }
+  }
+  return byMap;
+}
+
+function _histogramCategorical(column) {
+  const valMap = new Map();
+  if (!column) {
+    return valMap;
+  }
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const valBin = colArray[r];
+    let curCount = valMap.get(valBin);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    valMap.set(valBin, curCount + 1);
+  }
+  return valMap;
+}
+
+function _histogramCategoricalBy(column, by) {
+  const byMap = new Map();
+  if (!column || !by) {
+    return byMap;
+  }
+  const byArray = by.asArray();
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const byBin = byArray[r];
+    let valMap = byMap.get(byBin);
+    if (valMap === undefined) {
+      valMap = new Map();
+      byMap.set(byBin, valMap);
+    }
+    const valBin = colArray[r];
+    let curCount = valMap.get(valBin);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    valMap.set(valBin, curCount + 1);
+  }
+  return byMap;
+}
+
+/*
+Count category occupancy.  Optional group-by category.
+*/
+export function histogramCategorical(column, by) {
+  if (by && isTypedArray(by)) {
+    throw new Error("Group by column must be categorical");
+  }
+  return by
+    ? _histogramCategoricalBy(column, by)
+    : _histogramCategorical(column);
+}
+
+/*
+Memoization hash for histogramCategorical()
+*/
+export function hashCategorical(column, by) {
+  if (by) {
+    return `${column.__id}:${by.__id}`;
+  }
+  return `${column.__id}:`;
+}
+
+/*
+Bin counts for continuous/scalar values, with optional group-by category.
+Values outside domain are ignored.
+*/
+export function histogramContinuous(column, bins = 40, domain = [0, 1], by) {
+  if (by && isTypedArray(by)) {
+    throw new Error("Group by column must be categorical");
+  }
+  const [min, max] = domain;
+  return by
+    ? _histogramContinuousBy(column, bins, min, max, by)
+    : _histogramContinuous(column, bins, min, max);
+}
+
+/*
+Memoization hash for histogramContinuous
+*/
+export function hashContinuous(column, bins = "", domain = [0, 0], by) {
+  const [min, max] = domain;
+  if (by) {
+    return `${column.__id}:${bins}:${min}:${max}:${by.__id}`;
+  }
+  return `${column.__id}::${bins}:${min}:${max}`;
+}

client/src/util/dataframe/labelIndex.js

@@ -3,6 +3,8 @@ Label indexing - map a label to & from an integer offset.  See Dataframe
 for how this is used.
 **/
 
+import { rangeFill as fillRange } from "../range";
+
 /*
 Private utility functions
 */
@@ -21,14 +23,6 @@ function extent(tarr) {
   return [min, max];
 }
 
-function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, l = larr.length; i < l; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 /* eslint-disable class-methods-use-this */
 class IdentityInt32Index {
   /*
@@ -86,6 +80,10 @@ class IdentityInt32Index {
     return this.__promote([...this.keys(), label]);
   }
 
+  withLabels(labels) {
+    return this.__promote([...this.keys(), ...labels]);
+  }
+
   dropLabel(label) {
     if (label === this.maxOffset - 1) {
       return new IdentityInt32Index(label);
@@ -169,6 +167,10 @@ class DenseInt32Index {
     return this.__promote([...this.keys(), label]);
   }
 
+  withLabels(labels) {
+    return this.__promote([...this.keys(), ...labels]);
+  }
+
   dropLabel(label) {
     const labelArray = [...this.keys()];
     labelArray.splice(labelArray.indexOf(label), 1);
@@ -193,6 +195,11 @@ class KeyIndex {
       index.set(v, i);
     });
 
+    if (index.size !== rindex.length) {
+      /* if true, there was a duplicate in the keys */
+      throw new Error("duplicate label provided to KeyIndex");
+    }
+
     this.index = index;
     this.rindex = rindex;
     this.__compile();
@@ -224,6 +231,10 @@ class KeyIndex {
     return new KeyIndex([...this.rindex, label]);
   }
 
+  withLabels(labels) {
+    return new KeyIndex([...this.rindex, ...labels]);
+  }
+
   dropLabel(label) {
     const idx = this.rindex.indexOf(label);
     const labelArray = [...this.rindex];

client/src/util/dataframe/util.js

@@ -5,6 +5,10 @@ Private utility code for dataframe
 export { isTypedArray, isArrayOrTypedArray } from "../typeHelpers";
 
 export function callOnceLazy(f) {
+  /*
+  call function once, and save the result, regardless of arguments (this is not
+  the same as typical memoization).
+  */
   let value;
   let calledOnce = false;
   const result = function result(...args) {
@@ -14,6 +18,37 @@ export function callOnceLazy(f) {
     }
     return value;
   };
-
   return result;
 }
+
+export function memoize(fn, hashFn, maxResultsCached = -1) {
+  /* 
+  function memoization, with user-provided hash.  hashFn must return a
+  key which will be unique as a Map key (ie, obeys "sameValueZero" algorithm
+  as defined in the JS spec).  For more info on hash key, see:
+  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map#Key_equality
+  */
+  const cache = new Map();
+  const wrap = function wrap(...args) {
+    const key = hashFn(...args);
+    if (cache.has(key)) {
+      return cache.get(key);
+    }
+    const result = fn(...args);
+    cache.set(key, result);
+
+    if (maxResultsCached > -1 && cache.size > maxResultsCached) {
+      /* Least recent insertion deletion */
+      cache.delete(cache.keys().next().value);
+    }
+
+    return result;
+  };
+
+  wrap.clear = function clear() {
+    /* clear memoization cache */
+    cache.clear();
+  };
+
+  return wrap;
+}

client/src/util/glHelpers.js

@@ -0,0 +1,81 @@
+/*
+Utility code for WebGL shaders
+*/
+
+/*
+PointFlags:
+  Point flags are used in graph & scatter plots.
+
+  We want a bitmask-like flag structure, but due to webgl limitations
+  must emulate it with floats.
+*/
+
+// for JS
+export const flagSelected = 1;
+export const flagNaN = 2;
+export const flagHighlight = 4;
+
+// for GLSL
+export const glPointFlags = `
+
+  const float flagSelected = 1.;
+  const float flagNaN = 2.;
+  const float flagHighlight = 4.;
+
+  bool isLowBitSet(float f) {
+    f = mod(f, 2.);
+    return (f > 0.9 && f <= 1.1);
+  }
+
+  float shiftRightOne(float f) {
+    return floor(f / 2.);
+  }
+
+  void getFlags(in float flag,
+                out bool isNaN,
+                out bool isSelected,
+                out bool isHighlight) {
+    isSelected = isLowBitSet(flag);
+    flag = shiftRightOne(flag);
+    isNaN = isLowBitSet(flag);
+    flag = shiftRightOne(flag);
+    isHighlight = isLowBitSet(flag);
+  }
+
+`;
+
+/*
+Point Size:
+  Calculate point size for scatter plot based upon pseudo density.
+
+  Current approach: linear scaling of point size, clamped to [1,10],
+  between two points that are based on empirical testing.
+
+    - 1M points on a 500x500 canvas: 1M/(500*500) -> 0.5
+    - 1000 points on a 1440x1440 canvas:  1000/(1440*1440) -> 5
+
+  The domain is pseudo density (numPoints / minViewportDimension^2)
+  The range is web gl point size.
+*/
+
+// configuration
+const domain = [1000000 / (500 * 500), 1000 / (1440 * 1440)];
+const range = [0.5, 5];
+
+// derived from configuration
+const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+const offset = scale * -domain[0] + range[0];
+
+export const glPointSize = `
+  float pointSize(float nPoints, float minViewportDimension, bool isSelected, bool isHighlight) {
+    float density = nPoints / (minViewportDimension * minViewportDimension);
+    float pointSize = (${scale.toFixed(4)}*density) + ${offset.toFixed(4)};
+    pointSize = clamp(pointSize, 
+      ${range[0].toFixed(4)}, 
+      ${range[1].toFixed(4)});
+
+    if (isHighlight) return 2. * pointSize;
+    if (isSelected) return pointSize;
+    return pointSize / 3.;
+  }
+`;

client/src/util/range.js

@@ -0,0 +1,45 @@
+/*
+Array range creation
+
+range(start, stop, step) -> Array
+	This is identical to https://docs.python.org/3/library/functions.html#func-range
+	Returns new array filled with a range of numbers.
+
+	Usage:
+
+		range(stop) - start defaults to zero, step defaults to 1
+		range(start, stop, [step]) - step defaults to 1
+
+	Examples:
+		range(3) -> [0, 1, 2]
+		range(1, 3) -> [1, 2]
+		range(1, 5, 2) -> [1, 3]
+
+
+rangeFill(array, start, step) -> array
+	Fill entire array with values, from start, by step.  Returns first array.
+	start defaults to zero, step defaults to one.
+
+*/
+
+function _doFill(arr, start, step, count) {
+	for (let idx = 0, val = start; idx < count; idx += 1, val += step) {
+		arr[idx] = val;
+	}
+	return arr;
+}
+
+export function rangeFill(arr, start = 0, step = 1) {
+	return _doFill(arr, start, step, arr.length);
+}
+
+export function range(start, stop, step) {
+	if (start === undefined) return [];
+	if (stop === undefined) {
+		stop = start;
+		start = 0;
+	}
+	step = step || 1; // catch undefind and zero
+	const len = Math.max(Math.ceil((stop - start) / step), 0);
+	return _doFill(new Array(len), start, step, len);
+}

client/src/util/scaleLinear.js

@@ -9,8 +9,14 @@
 // this is is equivalent to d3.scaleLinear().domain([0,1]).range([-1,1])
 
 export default (domain, range) => {
-  const domainStart = domain[0];
+	const domainStart = domain[0];
-  const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+	const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
-  const rangeStart = range[0];
+	const invScale = 1 / scale;
-  return value => (value - domainStart) * scale + rangeStart;
+	const rangeStart = range[0];
+	const f = value => (value - domainStart) * scale + rangeStart;
+
+	// inverter
+	f.invert = value => (value - rangeStart) * invScale + domainStart;
+
+	return f;
 };

client/src/util/stateManager/annotationsHelpers.js

@@ -0,0 +1,168 @@
+/*
+Helper functions for user-editable nnotations state management.
+See also reducers/annotations.js
+*/
+import { unassignedCategoryLabel } from "../../globals";
+import * as SchemaHelpers from "./schemaHelpers";
+import { obsAnnoDimensionName } from "../nameCreators";
+
+/*
+There are a number of state constraints assumed throughout the
+application:
+	- all obs annotations are in {world|universe}.obsAnnotations,
+		regardless of whether or not they are user editable.
+	- the {world|universe}.schema is always up to date and matches
+		the data
+	- the schema flag `writable` correctly indicates whether
+	  the annotation is editable/mutable.
+
+In addition, the current state management only allows for
+categorical annotations to be writable.
+*/
+
+export function isCategoricalAnnotation(schema, name) {
+	/* we treat any string, categorical or boolean as a categorical */
+	const { type } = schema.annotations.obsByName[name];
+	return type === "string" || type === "boolean" || type === "categorical";
+}
+
+export function isContinuousAnnotation(schema, name) {
+	return !isCategoricalAnnotation(schema, name);
+}
+
+function _isUserAnnotation(schema, name) {
+	return schema.annotations.obsByName[name]?.writable;
+}
+
+export function isUserAnnotation(worldOrUniverse, name) {
+	return _isUserAnnotation(worldOrUniverse.schema, name);
+}
+
+export function removeObsAnnoSchema(schema, name) {
+	/*
+	remove named annotation from obs annotation schema
+	*/
+
+	/* only remove if it exists and is a user annotation */
+	if (!_isUserAnnotation(schema, name))
+		throw new Error("removing non-user-defined schema");
+	return SchemaHelpers.removeObsAnnoColumn(schema, name);
+}
+
+export function addObsAnnoSchema(schema, name, colSchema) {
+	/*
+	add a categorical type to the obs annotation schema 
+	*/
+
+	/* collision detection */
+	if (schema.annotations.obs.columns.some(v => v.name === name))
+		throw Error("annotations may not contain duplicate category names");
+	if (name !== colSchema.name) throw Error("column schema does not match");
+	return SchemaHelpers.addObsAnnoColumn(schema, name, colSchema);
+}
+
+export function dupObsAnnoSchema(schema, sourceName, dupName, defaultSchema) {
+	/*
+	duplicate the obs annotation `sourceName` schema, but with the name `dupName`
+	*/
+	const colSchema = {
+		...schema.annotations.obsByName[sourceName],
+		...defaultSchema,
+		name: dupName
+	};
+	/* existance check */
+	if (!colSchema) throw Error("source annotation does not exist");
+	/* collision detection */
+	if (schema.annotations.obs.columns.some(v => v.name === dupName))
+		throw Error("annotations may not contain duplicate category names");
+	return SchemaHelpers.addObsAnnoColumn(schema, dupName, colSchema);
+}
+
+export function removeObsAnnoCategory(schema, name, category) {
+	/* don't allow deletion of unassigned category on writable annotations */
+
+	if (!_isUserAnnotation(schema, name))
+		throw new Error("unable to modify read-only schema");
+	if (category === unassignedCategoryLabel)
+		throw new Error("may not remove unassigned category label");
+
+	return SchemaHelpers.removeObsAnnoCategory(schema, name, category);
+}
+
+export function addObsAnnoCategory(schema, name, category) {
+	if (!_isUserAnnotation(schema, name))
+		throw new Error("unable to modify read-only schema");
+
+	return SchemaHelpers.addObsAnnoCategory(schema, name, category);
+}
+
+export function setLabelByValue(df, colName, fromLabel, toLabel) {
+	/* 
+	in the dataframe column `colName`, set any value of `fromLabel` to `toLabel`
+	*/
+	const keys = df.colIndex.keys();
+	const ndf = df.mapColumns((col, colIdx) => {
+		if (colName !== keys[colIdx]) return col;
+
+		/* clone data and return it. */
+		const newCol = col.slice();
+		for (let i = 0, l = newCol.length; i < l; i += 1) {
+			if (newCol[i] === fromLabel) newCol[i] = toLabel;
+		}
+		return newCol;
+	});
+	return ndf;
+}
+
+export function setLabelByMask(df, colName, mask, label) {
+	/* 
+	in the dataframe column `colName`, set the masked rows to 'label'
+	*/
+	const keys = df.colIndex.keys();
+	const ndf = df.mapColumns((col, colIdx) => {
+		if (colName !== keys[colIdx]) return col;
+
+		/* clone data and return it. */
+		const newCol = col.slice();
+		for (let i = 0, l = newCol.length; i < l; i += 1) {
+			if (mask[i]) newCol[i] = label;
+		}
+		return newCol;
+	});
+	return ndf;
+}
+
+export function worldToUniverseMask(worldMask, worldObsAnnotations, nObs) {
+	/*
+	given world seleciton mask, return a selection mask for entire universe 
+	that has same selection state.
+	*/
+	const mask = new Uint8Array(nObs);
+	const { rowIndex } = worldObsAnnotations;
+
+	for (let i = 0, l = worldMask.length; i < l; i += 1) {
+		if (worldMask[i]) {
+			const label = rowIndex.getLabel(i);
+			mask[label] = 1;
+		}
+	}
+	return mask;
+}
+
+export function createWritableAnnotationDimensions(world, crossfilter) {
+	const { obsAnnotations, schema } = world;
+	const writableAnnotations = schema.annotations.obs.columns
+		.filter(s => s.writable)
+		.map(s => s.name);
+
+	crossfilter = writableAnnotations.reduce((xflt, anno) => {
+		const dimName = obsAnnoDimensionName(anno);
+		if (xflt.hasDimension(dimName)) xflt = xflt.delDimension(dimName);
+		return xflt.addDimension(
+			dimName,
+			"enum",
+			obsAnnotations.col(anno).asArray()
+		);
+	}, crossfilter);
+	return crossfilter;
+}

client/src/util/stateManager/colorHelpers.js

@@ -1,12 +1,12 @@
 /*
 Helper functions for the embedded graph colors
 */
-import _ from "lodash";
 import * as d3 from "d3";
 import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
 import * as globals from "../../globals";
 import parseRGB from "../parseRGB";
 import finiteExtent from "../finiteExtent";
+import { range } from "../range";
 
 /*
 create new colors state object.   Paramters:
@@ -15,7 +15,7 @@ create new colors state object.   Paramters:
     "color by continuous metadata", "color by categorical metadata"
   -
 */
-function createColors(world, colorMode = null, colorAccessor = null) {
+export function createColors(world, colorMode = null, colorAccessor = null) {
   switch (colorMode) {
     case "color by categorical metadata": {
       return createColorsByCategoricalMetadata(world, colorAccessor);
@@ -37,9 +37,7 @@ function createColors(world, colorMode = null, colorAccessor = null) {
 }
 
 function createColorsByCategoricalMetadata(world, accessor) {
-  const { categories } = _.filter(world.schema.annotations.obs, {
+  const { categories } = world.schema.annotations.obsByName[accessor];
-    name: accessor
-  })[0];
 
   const scale = d3
     .scaleSequential(interpolateRainbow)
@@ -52,8 +50,9 @@ function createColorsByCategoricalMetadata(world, accessor) {
   }, {});
 
   const rgb = new Array(world.nObs);
-  const data = world.obsAnnotations.col(accessor).asArray();
+  const df = world.obsAnnotations;
-  for (let i = 0, len = world.obsAnnotations.length; i < len; i += 1) {
+  const data = df.col(accessor).asArray();
+  for (let i = 0, len = df.length; i < len; i += 1) {
     const cat = data[i];
     rgb[i] = colors[cat];
   }
@@ -67,7 +66,7 @@ function createColorsByContinuousMetadata(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);
@@ -97,7 +96,7 @@ function createColorsByExpression(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);
@@ -119,4 +118,26 @@ function createColorsByExpression(world, accessor) {
   return { rgb, scale };
 }
 
-export default createColors;
+export const resetColors = world => {
+  const { rgb, scale } = createColors(world);
+  return {
+    colorMode: null,
+    colorAccessor: null,
+    rgb,
+    scale
+  };
+};
+
+export const checkIfColorByDiffexpAndResetColors = (
+  prevControls,
+  state,
+  prevWorld
+) => {
+  if (prevControls.diffexpGenes.includes(state.colorAccessor)) {
+    return {
+      ...state,
+      ...resetColors(prevWorld)
+    };
+  }
+  return null;
+};

client/src/util/stateManager/controlsHelpers.js

@@ -5,7 +5,7 @@ Helper functions for the controls reducer
 import _ from "lodash";
 
 import * as globals from "../../globals";
-import { fillRange } from "../typedCrossfilter/util";
+import { rangeFill as fillRange } from "../range";
 import {
   userDefinedDimensionName,
   diffexpDimensionName
@@ -21,32 +21,30 @@ Remember that option values can be ANY js type, except undefined/null.
   {
     _category_name_1: {
       // map of option value to index
-      categoryIndices: Map([
+      categoryValueIndices: Map([
         catval1: index,
         ...
       ])
 
       // index->selection true/false state
-      categorySelected: [ true/false, true/false, ... ]
+      categoryValueSelected: [ true/false, true/false, ... ]
 
       // number of options
-      numCategories: number,
+      numCategoryValues: number,
 
       // isTruncated - true if the options for selection has
       // been truncated (ie, was too large to implement)
     }
   }
 */
-function topNCategories(summary) {
+function topNCategories(colSchema, summary, N) {
-  const counts = _.map(summary.categories, cat =>
+  const { categories } = colSchema;
-    summary.categoryCounts.get(cat)
+  const counts = _.map(categories, cat => summary.categoryCounts.get(cat) ?? 0);
-  );
+  const sortIndex = fillRange(new Array(categories.length)).sort(
-  const sortIndex = fillRange(new Array(summary.numCategories)).sort(
     (a, b) => counts[b] - counts[a]
   );
-  const sortedCategories = _.map(sortIndex, i => summary.categories[i]);
+  const sortedCategories = _.map(sortIndex, i => categories[i]);
   const sortedCounts = _.map(sortIndex, i => counts[i]);
-  const N = globals.maxCategoricalOptionsToDisplay;
 
   if (sortedCategories.length < N) {
     return [sortedCategories, sortedCounts];
@@ -54,46 +52,56 @@ function topNCategories(summary) {
   return [sortedCategories.slice(0, N), sortedCounts.slice(0, N)];
 }
 
-export function createCategoricalSelection(maxCategoryItems, world) {
+export function selectableCategoryNames(world, maxCategoryItems) {
-  const res = {};
+  const { schema } = world;
-  _.forEach(world.obsAnnotations.colIndex.keys(), key => {
+  const { index, columns } = schema.annotations.obs;
-    const summary = world.obsAnnotations.col(key).summarize();
+  return columns
-    if (summary.categories) {
+    .filter(colSchema => {
-      const isColorField = key.includes("color") || key.includes("Color");
+      const { name, categories } = colSchema;
-      const isSelectableCategory =
+      return (
-        !isColorField &&
+        categories && categories.length < maxCategoryItems && name !== index
-        key !== "name" &&
+      );
-        summary.categories.length < maxCategoryItems;
+    })
-      if (isSelectableCategory) {
+    .map(v => v.name);
-        const [categoryValues, categoryCounts] = topNCategories(summary);
-        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
-        const numCategories = categoryIndices.size;
-        const categorySelected = new Array(numCategories).fill(true);
-        const isTruncated = categoryValues.length < summary.numCategories;
-        res[key] = {
-          categoryValues, // array: of natively typed category values
-          categoryIndices, // map: category value (native type) -> category index
-          categorySelected, // array: t/f selection state
-          numCategories, // number: of categories
-          isTruncated, // bool: true if list was truncated
-          categoryCounts // array: cardinality of each category
-        };
-      }
-    }
-  });
-  return res;
 }
 
-/*
+export function createCategoricalSelection(world, names) {
-given a categoricalSelection, return the list of all category values
+  const N = globals.maxCategoricalOptionsToDisplay;
-where selection state is true (ie, they are selected).
+  const { obsAnnotations, schema } = world;
-*/
+
-export function selectedValuesForCategory(categorySelectionState) {
+  const res = names.reduce((acc, name) => {
-  const selectedValues = _([...categorySelectionState.categoryIndices])
+    const colSchema = schema.annotations.obsByName[name];
-    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
+    const { writable: isUserAnno } = colSchema;
-    .map(tuple => tuple[0])
+
-    .value();
+    /*
-  return selectedValues;
+    Summarize the annotation data currently in world.  Must return categoryValues
+    in sorted order, and must include all category values even if they are not
+    actively used in the current world.
+    */
+    const summary = obsAnnotations.col(name).summarize();
+    const [categoryValues, categoryValueCounts] = topNCategories(
+      colSchema,
+      summary,
+      N
+    );
+    const categoryValueIndices = new Map(categoryValues.map((v, i) => [v, i]));
+    const numCategoryValues = categoryValueIndices.size;
+    const categoryValueSelected = new Array(numCategoryValues).fill(true);
+    const isTruncated = categoryValues.length < summary.numCategories;
+
+    acc[name] = {
+      categoryValues, // array: of natively typed category values
+      categoryValueIndices, // map: category value (native type) -> category index
+      categoryValueSelected, // array: t/f selection state
+      numCategoryValues, // number: of values in the category
+      isTruncated, // bool: true if list was truncated
+      categoryValueCounts, // array: cardinality of each category,
+      categorySelected: true, // bool - default state for entire category
+      isUserAnno // bool
+    };
+    return acc;
+  }, {});
+  return res;
 }
 
 /*

client/src/util/stateManager/index.js

@@ -14,8 +14,10 @@ This is all VERY tightly integrated with reducers and actions, and
 exists to support those concepts.
 */
 
-export { default as createColors } from "./colorHelpers";
+export * as ColorHelpers from "./colorHelpers";
 export * as Universe from "./universe";
 export * as World from "./world";
-export * as WorldUtil from "./worldUtil";
 export * as ControlsHelpers from "./controlsHelpers";
+export * as AnnotationsHelpers from "./annotationsHelpers";
+export * as SchemaHelpers from "./schemaHelpers";
+export * as MatrixFBS from "./matrix";

client/src/util/stateManager/matrix.js

@@ -1,5 +1,7 @@
 import { flatbuffers } from "flatbuffers";
 import { NetEncoding } from "./matrix_generated";
+import { isTypedArray } from "../typeHelpers";
+import { IdentityInt32Index, DenseInt32Index, KeyIndex } from "../dataframe";
 
 const utf8Decoder = new TextDecoder("utf-8");
 
@@ -41,25 +43,25 @@ Returns: object containing decoded Matrix:
   colIdx: []|null
 }
 */
-function decodeMatrixFBS(arrayBuffer, inplace = false) {
+export function decodeMatrixFBS(arrayBuffer, inplace = false) {
   const bb = new flatbuffers.ByteBuffer(new Uint8Array(arrayBuffer));
-  const df = NetEncoding.Matrix.getRootAsMatrix(bb);
+  const matrix = NetEncoding.Matrix.getRootAsMatrix(bb);
 
-  const nRows = df.nRows();
+  const nRows = matrix.nRows();
-  const nCols = df.nCols();
+  const nCols = matrix.nCols();
 
   /* decode columns */
-  const columnsLength = df.columnsLength();
+  const columnsLength = matrix.columnsLength();
   const columns = Array(columnsLength).fill(null);
   for (let c = 0; c < columnsLength; c += 1) {
-    const col = df.columns(c);
+    const col = matrix.columns(c);
     columns[c] = decodeTypedArray(col.uType(), col.u.bind(col), inplace);
   }
 
   /* decode col_idx */
   const colIdx = decodeTypedArray(
-    df.colIndexType(),
+    matrix.colIndexType(),
-    df.colIndex.bind(df),
+    matrix.colIndex.bind(matrix),
     inplace
   );
 
@@ -72,4 +74,91 @@ function decodeMatrixFBS(arrayBuffer, inplace = false) {
   };
 }
 
-export default decodeMatrixFBS;
+function encodeTypedArray(builder, uType, uData) {
+  const uTypeName = NetEncoding.TypedArray[uType];
+  const ArrayType = NetEncoding[uTypeName];
+  const dv = ArrayType.createDataVector(builder, uData);
+  builder.startObject(1);
+  builder.addFieldOffset(0, dv, 0);
+  return builder.endObject();
+}
+
+export function encodeMatrixFBS(df) {
+  /*
+  encode the dataframe as an FBS Matrix
+  */
+
+  /* row indexing not supported currently */
+  if (df.rowIndex.constructor !== IdentityInt32Index) {
+    throw new Error("FBS does not support row index encoding at this time");
+  }
+
+  const shape = df.dims;
+  const utf8Encoder = new TextEncoder("utf-8");
+  const builder = new flatbuffers.Builder(1024);
+
+  let encColIndex;
+  let encColIndexUType;
+  let encColumns;
+
+  if (shape[0] > 0 && shape[1] > 0) {
+    const columns = df.columns().map(col => col.asArray());
+
+    const cols = columns.map(carr => {
+      let uType;
+      let tarr;
+      if (isTypedArray(carr)) {
+        uType = NetEncoding.TypedArray[carr.constructor.name];
+        tarr = encodeTypedArray(builder, uType, carr);
+      } else {
+        uType = NetEncoding.TypedArray.JSONEncodedArray;
+        const json = JSON.stringify(carr);
+        const jsonUTF8 = utf8Encoder.encode(json);
+        tarr = encodeTypedArray(builder, uType, jsonUTF8);
+      }
+      NetEncoding.Column.startColumn(builder);
+      NetEncoding.Column.addUType(builder, uType);
+      NetEncoding.Column.addU(builder, tarr);
+      return NetEncoding.Column.endColumn(builder);
+    });
+
+    encColumns = NetEncoding.Matrix.createColumnsVector(builder, cols);
+
+    if (df.colIndex && shape[1] > 0) {
+      const colIndexType = df.colIndex.constructor;
+      if (colIndexType === IdentityInt32Index) {
+        encColIndex = undefined;
+      } else if (colIndexType === DenseInt32Index) {
+        encColIndexUType = NetEncoding.TypedArray.Int32Array;
+        encColIndex = encodeTypedArray(
+          builder,
+          encColIndexUType,
+          df.colIndex.keys()
+        );
+      } else if (colIndexType === KeyIndex) {
+        encColIndexUType = NetEncoding.TypedArray.JSONEncodedArray;
+        encColIndex = encodeTypedArray(
+          builder,
+          encColIndexUType,
+          utf8Encoder.encode(JSON.stringify(df.colIndex.keys()))
+        );
+      } else {
+        throw new Error("Index type FBS encoding unsupported");
+      }
+    }
+  }
+
+  NetEncoding.Matrix.startMatrix(builder);
+  NetEncoding.Matrix.addNRows(builder, shape[0]);
+  NetEncoding.Matrix.addNCols(builder, shape[1]);
+  if (encColumns) {
+    NetEncoding.Matrix.addColumns(builder, encColumns);
+  }
+  if (encColIndexUType) {
+    NetEncoding.Matrix.addColIndexType(builder, encColIndexUType);
+    NetEncoding.Matrix.addColIndex(builder, encColIndex);
+  }
+  const root = NetEncoding.Matrix.endMatrix(builder);
+  builder.finish(root);
+  return builder.asUint8Array();
+}

client/src/util/stateManager/schemaHelpers.js

@@ -0,0 +1,96 @@
+/*
+Helpers for schema management
+*/
+import _ from "lodash";
+
+import fromEntries from "../fromEntries";
+
+/*
+System wide schema assumptions:
+	- schema and data wil be consistent (eg, for user-created annotations)
+	- schema will be internally self-consistent (eg, index matches columns)
+	- world & universe schema are same - only data is subset
+*/
+
+export function indexEntireSchema(schema) {
+	/* Index schema for ease of use */
+	schema.annotations.obsByName = fromEntries(
+		schema.annotations.obs.columns.map(v => [v.name, v])
+	);
+	schema.annotations.varByName = fromEntries(
+		schema.annotations.var.columns.map(v => [v.name, v])
+	);
+	schema.layout.obsByName = fromEntries(
+		schema.layout.obs.map(v => [v.name, v])
+	);
+	schema.layout.varByName = fromEntries(
+		schema.layout.var.map(v => [v.name, v])
+	);
+
+	return schema;
+}
+
+function _copy(schema) {
+	/* redux copy conventions - WARNING, only for modifyign obs annotations */
+	return {
+		...schema,
+		annotations: {
+			...schema.annotations,
+			obs: _.cloneDeep(schema.annotations.obs)
+		}
+	};
+}
+
+function _reindex(schema) {
+	/* reindex obs annotations ONLY */
+	schema.annotations.obsByName = fromEntries(
+		schema.annotations.obs.columns.map(v => [v.name, v])
+	);
+	return schema;
+}
+
+export function removeObsAnnoColumn(schema, name) {
+	const newSchema = _copy(schema);
+	newSchema.annotations.obs.columns = schema.annotations.obs.columns.filter(
+		v => v.name !== name
+	);
+	return _reindex(newSchema);
+}
+
+export function addObsAnnoColumn(schema, name, defn) {
+	const newSchema = _copy(schema);
+	newSchema.annotations.obs.columns.push(defn);
+	return _reindex(newSchema);
+}
+
+export function removeObsAnnoCategory(schema, name, category) {
+	/* remove a category from a categorical annotation */
+	const categories = schema.annotations.obsByName[name]?.categories;
+	if (!categories)
+		throw new Error("column does not exist or is not categorical");
+
+	const idx = categories.indexOf(category);
+	if (idx === -1) throw new Error("category does not exist");
+
+	const newSchema = _reindex(_copy(schema));
+
+	/* remove category */
+	newSchema.annotations.obsByName[name].categories.splice(idx, 1);
+	return newSchema;
+}
+
+export function addObsAnnoCategory(schema, name, category) {
+	/* add a category to a categorical annotation */
+	const categories = schema.annotations.obsByName[name]?.categories;
+	if (!categories)
+		throw new Error("column does not exist or is not categorical");
+
+	const idx = categories.indexOf(category);
+	if (idx !== -1) throw new Error("category already exists");
+
+	const newSchema = _reindex(_copy(schema));
+
+	/* remove category */
+	newSchema.annotations.obsByName[name].categories.push(category);
+	return newSchema;
+}

client/src/util/stateManager/universe.js

@@ -1,11 +1,11 @@
-// jshint esversion: 6
-
 import _ from "lodash";
 
-import decodeMatrixFBS from "./matrix";
+import { unassignedCategoryLabel } from "../../globals";
+import { decodeMatrixFBS } from "./matrix";
 import * as Dataframe from "../dataframe";
-import fromEntries from "../fromEntries";
 import { isFpTypedArray } from "../typeHelpers";
+import { indexEntireSchema } from "./schemaHelpers";
+import { isCategoricalAnnotation } from "./annotationsHelpers";
 
 /*
 Private helper function - create and return a template Universe
@@ -18,10 +18,13 @@ function templateUniverse() {
     schema: {},
 
     /*
-    Annotations
+    annotations
     */
     obsAnnotations: Dataframe.Dataframe.empty(),
     varAnnotations: Dataframe.Dataframe.empty(),
+    /*
+    layout
+    */
     obsLayout: Dataframe.Dataframe.empty(),
 
     /*
@@ -78,6 +81,7 @@ function AnnotationsFBSToDataframe(arrayBuffer) {
   The application has strong assumptions that all scalar data will be
   stored as a float32 or float64 (regardless of underlying data types).
   For example, clipping of value ranges (eg, user-selected percentiles)
+  depends on the ability to use NaN in any numeric type.
 
   All float data from the server is left as is.  All non-float is promoted
   to an appropriate float.
@@ -98,15 +102,16 @@ function AnnotationsFBSToDataframe(arrayBuffer) {
 
 function LayoutFBSToDataframe(arrayBuffer) {
   const fbs = decodeMatrixFBS(arrayBuffer, true);
-  if (fbs.columns.length !== 2 || !fbs.columns.every(isFpTypedArray)) {
+  if (fbs.columns.length < 2 || !fbs.columns.every(isFpTypedArray)) {
     // We have strong assumptions about the shape & type of layout data.
     throw new Error("Unexpected layout data type returned from server");
   }
+
   const df = new Dataframe.Dataframe(
     [fbs.nRows, fbs.nCols],
     fbs.columns,
     null,
-    new Dataframe.KeyIndex(["X", "Y"])
+    new Dataframe.KeyIndex(fbs.colIdx)
   );
   return df;
 }
@@ -120,20 +125,28 @@ function reconcileSchemaCategoriesWithSummary(universe) {
   For example, boolean defined fields in the schema do not contain
   explicit declaration of categories (nor do string fields).  In these
   cases, add a 'categories' field to the schema so it is accessible.
+
+  In addition, we have a client-side convention (UI) that all writable
+  annotations must have an 'unassigned' category, even if it is not currently
+  in use.
   */
 
-  _.forEach(universe.schema.annotations.obs, s => {
+  universe.schema.annotations.obs.columns.forEach(s => {
     if (
       s.type === "string" ||
       s.type === "boolean" ||
       s.type === "categorical"
     ) {
       const categories = _.union(
-        _.get(s, "categories", []),
+        s.categories ?? [],
-        _.get(universe.obsAnnotations.col(s.name).summarize(), "categories", [])
+        universe.obsAnnotations.col(s.name).summarize().categories ?? []
       );
       s.categories = categories;
     }
+
+    if (s.writable && s.categories.indexOf(unassignedCategoryLabel) === -1) {
+      s.categories = s.categories.concat(unassignedCategoryLabel);
+    }
   });
 }
 
@@ -154,6 +167,9 @@ export function createUniverseFromResponse(
   universe.schema = schema;
   universe.nObs = schema.dataframe.nObs;
   universe.nVar = schema.dataframe.nVar;
+  /* add defaults, as we can't assume back-end will fully populate schema */
+  if (!schema.layout.var) schema.layout.var = [];
+  if (!schema.layout.obs) schema.layout.obs = [];
 
   /* annotations */
   universe.obsAnnotations = AnnotationsFBSToDataframe(annotationsObsResponse);
@@ -161,7 +177,7 @@ export function createUniverseFromResponse(
   /* layout */
   universe.obsLayout = LayoutFBSToDataframe(layoutFBSResponse);
 
-  /* sanity check */
+  /* sanity checks */
   if (
     universe.nObs !== universe.obsLayout.length ||
     universe.nObs !== universe.obsAnnotations.length ||
@@ -171,14 +187,19 @@ export function createUniverseFromResponse(
   }
 
   reconcileSchemaCategoriesWithSummary(universe);
+  indexEntireSchema(universe.schema);
+
+  /* sanity checks */
+  if (
+    schema.annotations.obs.columns.some(
+      s => s.writable && !isCategoricalAnnotation(schema, s.name)
+    )
+  ) {
+    throw new Error(
+      "Writable continuous obs annotations are not supproted - failed to laod"
+    );
+  }
 
-  /* Index schema for ease of use */
-  universe.schema.annotations.obsByName = fromEntries(
-    universe.schema.annotations.obs.map(v => [v.name, v])
-  );
-  universe.schema.annotations.varByName = fromEntries(
-    universe.schema.annotations.var.map(v => [v.name, v])
-  );
   return universe;
 }
 
@@ -202,8 +223,9 @@ export function convertDataFBStoObject(universe, arrayBuffer) {
     throw new Error("Unexpected non-floating point response from server.");
   }
 
+  const varIndexName = universe.schema.annotations.var.index;
   for (let c = 0; c < colIdx.length; c += 1) {
-    const varName = universe.varAnnotations.at(colIdx[c], "name");
+    const varName = universe.varAnnotations.at(colIdx[c], varIndexName);
     result[varName] = columns[c];
   }
   return result;

client/src/util/stateManager/world.js

@@ -1,14 +1,7 @@
-// jshint esversion: 6
-
 import clip from "../clip";
-import {
+import { layoutDimensionName, obsAnnoDimensionName } from "../nameCreators";
-  layoutDimensionName,
-  obsAnnoDimensionName,
-  diffexpDimensionName,
-  userDefinedDimensionName
-} from "../nameCreators";
 import * as Dataframe from "../dataframe";
-import ImmutableTypedCrossfilter from "../typedCrossfilter/crossfilter";
+import { isContinuousAnnotation } from "./annotationsHelpers";
 
 /*
 
@@ -158,7 +151,7 @@ and world.varData.
 function setClippedDataframes(world) {
   const { schema } = world;
   const isContinuousObsAnnotation = (df, idx, label) =>
-    deduceDimensionType(schema.annotations.obsByName[label], label) !== "enum";
+    isContinuousAnnotation(schema, label);
   const obsQuantile = (label, q) =>
     world.unclipped.obsAnnotations.col(label).summarize().percentiles[100 * q];
   world.obsAnnotations = clipDataframe(
@@ -183,7 +176,7 @@ function setClippedDataframes(world) {
 /*
 Subset the current world based upon the current selection, maintaining any existing
 clip.  Returns new world.  Parameters:
-  * unvierse
+  * universe
   * world - the current world
   * crossfilter - the selection state
 */
@@ -260,13 +253,17 @@ function deduceDimensionType(attributes, fieldName) {
   return dimensionType;
 }
 
-export function createObsDimensions(crossfilter, world) {
+export function createObsDimensions(crossfilter, world, XYdimNames) {
   /*
   create and return a crossfilter with a dimension for every obs annotation
-  for which we have a supported type, *except* 'name'
+  for which we have a supported type, *except* for the index column, indicated
+  by schema.annotations.obs.index.
   */
   const { schema, obsLayout, obsAnnotations } = world;
-  const annoList = schema.annotations.obs.filter(anno => anno.name !== "name");
+  const indexName = schema.annotations.obs.index;
+  const annoList = schema.annotations.obs.columns.filter(
+    anno => anno.name !== indexName
+  );
   crossfilter = annoList.reduce((xfltr, anno) => {
     const dimType = deduceDimensionType(anno, anno.name);
     const colData = obsAnnotations.col(anno.name).asArray();
@@ -283,8 +280,8 @@ export function createObsDimensions(crossfilter, world) {
   return crossfilter.addDimension(
     layoutDimensionName("XY"),
     "spatial",
-    obsLayout.col("X").asArray(),
+    obsLayout.col(XYdimNames[0]).asArray(),
-    obsLayout.col("Y").asArray()
+    obsLayout.col(XYdimNames[1]).asArray()
   );
 }
 

client/src/util/stateManager/worldUtil.js

@@ -1,75 +0,0 @@
-/* eslint-disable import/prefer-default-export */
-import _ from "lodash";
-
-/*
-Various utility functions operating on World/Universe
-*/
-
-/*
-Count unique category values, binning first by dim1 then by dim2
-Return:
-
-Map {
-   dim1_val1: Map {
-      dim2_val1: number,
-      dim2_val2: number,
-      ...
-   },
-   ...
-}
-
-Parameters are:
-- dim1: dimension 1 name/label
-- dim2: dimension 2 name/label
-- df: dataframe containing dim1 and dim2 on the column axis
-
-*/
-function _countCategoryValues2D(dim1, dim2, df) {
-  const dimMap = new Map();
-  const col1 = df.col(dim1) ? df.col(dim1).asArray() : null;
-  const col2 = df.col(dim2) ? df.col(dim2).asArray() : null;
-  if (!col1 || !col2) {
-    return dimMap;
-  }
-
-  for (let r = 0, l = df.length; r < l; r += 1) {
-    const val1 = col1[r];
-    const val2 = col2[r];
-    let d2Map = dimMap.get(val1);
-    if (d2Map === undefined) {
-      d2Map = new Map();
-      dimMap.set(val1, d2Map);
-    }
-    let curCount = d2Map.get(val2);
-    if (curCount === undefined) {
-      curCount = 0;
-    }
-    d2Map.set(val2, curCount + 1);
-  }
-  return dimMap;
-}
-
-let __worldUtilMemoId__ = 0;
-function _memoizedId(x) {
-  if (!x.__worldUtilMemoId__) {
-    __worldUtilMemoId__ += 1;
-    x.__worldUtilMemoId__ = __worldUtilMemoId__;
-  }
-  return x.__worldUtilMemoId__;
-}
-function _countCategoryValues2DResolver(...args) {
-  const id = args[0] + args[1] + _memoizedId(args[2]);
-  return id;
-}
-
-export const countCategoryValues2D = _.memoize(
-  _countCategoryValues2D,
-  _countCategoryValues2DResolver
-);
-
-/*
-Clear any cached data within WorldUtil caches, eg, memoized functions
-*/
-export function clearCaches() {
-  countCategoryValues2D.cache.clear();
-}

client/src/util/typedCrossfilter/bitArray.js

@@ -35,6 +35,7 @@ class BitArray {
 
     this.bitmask = new Int32Array(this.width); // dimension allocation mask
     this.bitarray = new Int32Array(this.width * this.length);
+    Object.seal(this);
   }
 
   // Return the number of records that are selected, ie, have a one bit in

client/src/util/typedCrossfilter/crossfilter.js

@@ -5,6 +5,7 @@ import BitArray from "./bitArray";
 import {
   sortArray,
   lowerBound,
+  binarySearch,
   lowerBoundIndirect,
   upperBoundIndirect
 } from "./sort";
@@ -46,6 +47,7 @@ export default class ImmutableTypedCrossfilter {
     this.data = data;
     this.selectionCache = selectionCache; /* BitArray */
     this.dimensions = dimensions; /* name: { id, dim, name, selection } */
+    Object.preventExtensions(this);
   }
 
   size() {
@@ -56,6 +58,14 @@ export default class ImmutableTypedCrossfilter {
     return this.data;
   }
 
+  setData(data) {
+    return new ImmutableTypedCrossfilter(
+      data,
+      this.dimensions,
+      this.selectionCache
+    );
+  }
+
   dimensionNames() {
     /* return array of all dimensions (by name) */
     return Object.keys(this.dimensions);
@@ -85,6 +95,7 @@ export default class ImmutableTypedCrossfilter {
     }
     const DimensionType = DimTypes[type];
     const dim = new DimensionType(name, data, ...rest);
+    Object.freeze(dim);
     const dimensions = {
       ...this.dimensions,
       [name]: {
@@ -113,6 +124,23 @@ export default class ImmutableTypedCrossfilter {
     return new ImmutableTypedCrossfilter(data, dimensions, selectionCache);
   }
 
+  renameDimension(oldName, newName) {
+    /*
+    rename a dimension
+    */
+    const { [oldName]: dim, ...dimensions } = this.dimensions;
+    const { data, selectionCache } = this;
+    dim.dim.rename(newName);
+    return new ImmutableTypedCrossfilter(
+      data,
+      {
+        ...dimensions,
+        [newName]: dim
+      },
+      selectionCache
+    );
+  }
+
   select(name, spec) {
     /*
     select on named dimension, as indicated by `spec`.   Spec is an object
@@ -288,6 +316,10 @@ class _ImmutableBaseDimension {
     this.name = name;
   }
 
+  rename(name) {
+    this.name = name;
+  }
+
   select(spec) {
     const { mode } = spec;
     if (mode === undefined) {
@@ -436,7 +468,7 @@ class ImmutableEnumDimension extends ImmutableScalarDimension {
     const { values } = spec;
     return super.selectExact({
       mode: spec.mode,
-      values: values.map(v => lowerBound(enumIndex, v, 0, enumIndex.length))
+      values: values.map(v => binarySearch(enumIndex, v, 0, enumIndex.length))
     });
   }
 

client/src/util/typedCrossfilter/sort.js

@@ -413,3 +413,16 @@ export function upperBoundIndirect(valueArray, indexArray, value, first, last) {
   }
   return upperBoundNonFloatIndirect(valueArray, indexArray, value, first, last);
 }
+
+// Search for `value` in the sorted array `arr`, in the range [first, last).
+// Return the first index where arr[index] == value, OR if value not present,
+// return `last`
+//
+// The same semantics/behavior as:
+//    C++: binary_search()
+//
+export function binarySearch(valueArray, value, first, last) {
+  const index = lowerBound(valueArray, value, first, last);
+  if (index !== last && value === valueArray[index]) return index;
+  return last;
+}

client/src/util/typedCrossfilter/util.js

@@ -1,22 +1,12 @@
 // jshint esversion: 6
 
 import { sortIndex } from "./sort";
+import { rangeFill as fillRange } from "../range";
 
 /*
     Utility functions, private to this module.
 */
 
-// fill an array or typedarray with a sequential range of numbers,
-// starting with `start`
-//
-export function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, len = larr.length; i < len; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 // slice out of one array into another, using an index array
 //
 export function sliceByIndex(src, index) {

dev_docs/developer_guidelines.md

@@ -0,0 +1,91 @@
+# Developer guidelines
+
+### Requirements
+- npm
+- Python 3.6+
+- Chrome
+
+[See dev section of README](../README.md)
+
+**All instructions are expected to be run from the top level cellxgene directory unless otherwise specified.**
+
+## Server dev
+### Install
+* Build the client and put static files in place: `make build-for-server-dev`
+* Install from local files: `make install-dev`
+
+### Launch
+* `cellxgene launch [options] <datafile>`
+
+### Reloading
+If you install cellxgene using `make install-dev` the server will be restarted every time you make changes on the server code. If changes affects the client, the browser must be reloaded.
+
+### Linter
+We use `flake8` to lint code. Travis CI runs `flake8 server`.
+
+### Test
+1. Install development requirements `pip install -r server/requirements-dev.txt`
+2. Run tests `pytest server/test`
+
+### Tips
+* Install in a virtualenv
+* May need to rebuild/reinstall when you make client changes
+
+## Client dev
+### Install
+1. Install prereqs for client: `npm install --prefix client/ client`
+2. Install cellxgene server: `pip install -e .` Caveat: this will not build the production client package - you must use the [server install](#install) instructions above to serve web assets.
+
+### Launch
+To launch with hot reloading you need to launch the server and the client separately. Node's hot reloading starts the client on its own node server and auto-refreshes when changes are made.
+1. Launch server (the client relies on the REST API being available): `cellxgene launch [options] <datafile>`
+2. Launch client: in `client/` directory run `npm run start`
+3. Client will be served on localhost:3000
+
+### Build
+To build only the client: `make build-client`
+
+### Linter
+We use `eslint` to lint the code and `prettier` as our code formatter.
+
+### Test
+In `client/` directory run `npm run unit-test`
+
+### Tips
+* You can also install/launch the server side code from npm scrips (requires python3.6 with virtualenv) in `client/` directory run `npm run backend-dev`
+
+## Running tests
+Client and server tests run on Travis CI for every push, PR, and commit to master on github. End to end tests run nightly on master only. 
+
+### Server unit tests
+Install development requirements `pip install -r server/requirements-dev.txt`
+Run tests `pytest server/test`
+
+### Client unit tests
+In `client/` directory run `npm run unit-test`
+
+### End to end tests
+
+End to end tests use two env variables:
+* `JEST_ENV` - environment to run end to end tests. Default `dev`
+    * `prod` - run headless with no slowdown, chromium will not open.
+    * `dev` - opens chromimum, runs tests with minimal slowdown, close on exit.
+    * `debug` - opens chromium, runs tests with 100ms slowdown, dev tools open, chrome stays open on exit.
+* `JEST_CXG_PORT` - port that end to end tests are being run on. Default `3000` (client hosted port).
+
+On CI the end to end tests are run with `JEST_ENV` set to `prod` using the `smoke-test` npm script
+
+To run end to end tests as they will be run on CI
+1. cellxgene should be built and installed as [specified in server dev](#install)
+2. `export JEST_ENV='prod'`
+3. `export JEST_CXG_PORT='5000'`
+4. Run `npm run --prefix client/ smoke-test`
+
+Run end to end tests interactively during development
+1. cellxgene should be installed as [specified in client dev](#install-1)
+2. Follow [launch](#launch-1) instructions for client dev with dataset `example-dataset/pbmc3k`
+3. Run `npm run --prefix client/ e2e`
+4. To debug a failing test `export JEST_ENV='debug'` and re-run.
+
+
+

dev_docs/release_process.md

@@ -43,24 +43,53 @@ Follow these steps to create a release.
 8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
 and that you have write access to the cellxgene pypi package):
     - Build the distribution and upload to test pypi `make release-stage-2`
-    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Test the test installation in a fresh virtual environment using `make install-release-test`
     - Upload the package to real pypi using `make release-stage-final`
-    - [optional] Test the installation in a fresh virtual environment using
+    - Test the installation in a fresh virtual environment using
       `pip install cellxgene`
-    - **Troubleshooting**:
-        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
-        if you accidentally burned a release number you want to use on prod, you have a couple options. 
-            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
-             then upload to prod `make release-stage-final`. 
-            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
-            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
-            PART=patch until you get to an unburned version.
-        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
-        go to the cellxgene package, click manage, then in the options drop down, click delete and 
-        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
-        and not a major regression, you can just release a patch. 
-    
 
 The optional steps are for testing purposes, and are recommended
 for publishing any major releases, and any releases that significantly
 change the packaging (e.g. new bundled files, new dependencies, etc.)
+
+## Troubleshooting
+### Fails to upload to test.pypi
+ 
+ _PyPi doesn't allow you to reupload a release with the same version number_ 
+If you accidentally burned a release number you want to use on prod, you have a few options:
+1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+ then upload to prod `make release-stage-final`. 
+2) OPTION 2: (DANGER) release directly to prod: `make release-directly-to-prod`.  
+3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+PART=patch until you get to an unburned version.
+    
+### The release doesn't install or fails your tests when you install it
+ 
+Delete it from pypi - Go to pypi.org -> sign in -> go to the cellxgene package -> click manage -> then in the options drop down click delete -> follow the instructions. You will not be able to use that release number again. If it is a minor bug and not a major regression, you can just release a patch. 
+### If you need to run stage final on a different computer than stage 2
+ If you run stage final without running stage 2 first, the dist will not have been build on the computer running stage final. The solution is to run `make release-directly-to-prod`. This both builds the distribution files and then releases directly to prod pypi.org. 
+
+## Stage Details
+### Stage 1 - `make release-stage-1`
+    1. Pip installs requirements-dev
+    2. Bumps version by [PART]
+    3. Deletes build directory, client/build, dist and cellxgene.egg-info
+    4. Creates the package-lock.json
+
+### Stage 2 - `make release-stage-2`
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to test.pypi.org
+
+### Stage final - `make release-stage-final`
+    ** Does not build distribution **
+    1. Uploads to pypi.org
+    
+### (DANGER) Release directly to prod `make release-directly-to-prod`
+    ** builds distribution and uploads directly to prod **
+    Only use this if you are directed to by the troubleshooting guide
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to pypi.org