bump version to 0.11.0 (#850)

* bump version to 0.11.0

* manually fix version number

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.9.1
+current_version = 0.11.0
 
 [bumpversion:file:setup.py]
 search = version="{current_version}"
@@ -13,3 +13,7 @@ replace = version="{new_version}"
 search = "version": "{current_version}"
 replace = "version": "{new_version}"
 
+[bumpversion:file:server/__init__.py]
+search = "__version__ = "{current_version}"
+replace = "__version__ = "{new_version}"
+

.travis.yml

@@ -9,8 +9,8 @@ cache:
 install:
   - set -eo pipefail
   - pip install flake8
-  - make build
-  - make install
+  - make pydist
+  - make install-dist
   - pip install -r server/requirements-dev.txt
 
 jobs:
@@ -27,6 +27,5 @@ jobs:
     script: docker build .
   - name: "Smoke Tests"
     python: "3.6"
-    if: branch = master AND type = cron
     script:
       - npm run --prefix client/ smoke-test

CONTRIBUTING.md

@@ -0,0 +1,49 @@
+# contributing to cellxgene
+
+We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress.
+
+All code contributions and dependencies must be compatible with the project's [open-source license (MIT)](LICENSE.txt).
+
+This project adheres to the Contributor Covenant
+[code of conduct](https://github.com/chanzuckerberg/.github/tree/master/CODE_OF_CONDUCT.md).
+By participating, you are expected to uphold this code. Please report unacceptable behavior
+to opensource@chanzuckerberg.com.
+
+And finally, if you have any questions about any of this stuff, just ask! :)
+
+## developer guide
+
+This project has made a few key design choices
+
+- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
+- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
+- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
+
+Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
+
+If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
+
+- python 3.6+
+- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
+
+Then clone the project
+
+```
+git clone https://github.com/chanzuckerberg/cellxgene.git
+```
+
+Build the client web assets by calling `make` from inside the `cellxgene` folder
+
+```
+make
+```
+
+Install all requirements (we recommend doing this inside a virtual environment)
+
+```
+pip install -e .
+```
+
+You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
+
+If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.

README.md

@@ -2,13 +2,15 @@
 
 > an interactive explorer for single-cell transcriptomics data
 
+[![DOI](https://zenodo.org/badge/105615409.svg)](https://zenodo.org/badge/latestdoi/105615409)
+
 _cellxgene_ (pronounced "sell-by-jean") is an interactive data explorer for single-cell transcriptomics datasets, such as those coming from the [Human Cell Atlas](https://humancellatlas.org). Leveraging modern web development techniques to enable fast visualizations of at least 1 million cells, we hope to enable biologists and computational researchers to explore their data, and to demonstrate general, scalable, and reusable patterns for scientific data visualization.
 
 <img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-1.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-2.gif" width="200" height="200" hspace="30"><img src="https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/docs/cellxgene-demo-3.gif" width="200" height="200" hspace="30">
 
 - Want to install and use cellxgene? Visit the [cellxgene docs](https://chanzuckerberg.github.io/cellxgene/).
 - Want to see where we are going? Check out [our roadmap](ROADMAP.md).
-- Want to contribute? See our [contributors guide](#Contributing)
+- Want to contribute? See our [contributors guide](CONTRIBUTING.md).
 
 ## quick start
 
@@ -30,11 +32,17 @@ Launch _cellxgene_
 cellxgene launch pbmc3k.h5ad --open
 ```
 
-To learn more about what you can do with _cellxgene_, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-stared/) guide.
+To learn more about what you can do with _cellxgene_, see the [Getting Started](https://chanzuckerberg.github.io/cellxgene/getting-started.html) guide.
 
 ## get in touch
 
-Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. As mentioned above, please submit any feature requests or bugs as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+Have questions, suggestions, or comments? You can come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-users` channel. Have feature requests or bugs? Please submit these as [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). We'd love to hear from you!
+
+## contributing
+
+We warmly welcome contributions from the community! Please see our [contributing guide](CONTRIBUTING.md) and don't hesitate to open an issue or send a pull request to improve cellxgene.
+
+This project adheres to the Contributor Covenant [code of conduct](https://github.com/chanzuckerberg/.github/blob/master/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.
 
 ## where we are going
 
@@ -50,46 +58,19 @@ There are 4 key features we plan to implement in the near term.
 
 For more detail on these features and where we are going, see [our roadmap](ROADMAP.md).
 
-## contributing
+## risks of hosting cellxgene
 
-We warmly welcome contributions from the community! Please submit any bug reports and feature requests through [Github issues](https://github.com/chanzuckerberg/cellxgene/issues). Please submit any direct contributions by forking the repository, creating a branch, and submitting a Pull Request. It'd be great for PRs to include test cases and documentation updates where relevant, though we know the core test suite is itself still a work in progress. And all code contributions and dependencies must be compatible with the project's open-source license (MIT). If you have any questions about this stuff, just ask!
+_cellxgene_ is built on standard web technologies, but is currently designed as a single-user desktop application.
+We've done this so we can prioritize the features on [our roadmap](ROADMAP.md).
 
-### developer guide
+Some of our users have experimented with hosting _cellxgene_ either for their lab or for public use, but please note that the _cellxgene_ team does not officially support, troubleshoot, or maintain any web deployments at this time.
 
-This project has made a few key design choices
+If do you choose setup _cellxgene_ as a hosted service, you should be aware of the following risks:
 
-- The front-end is built with [`regl`](https://github.com/regl-project/regl) (a webgl library), [`react`](https://reactjs.org/), [`redux`](https://redux.js.org/), [`d3`](https://github.com/d3/d3), and [`blueprint`](https://blueprintjs.com/docs/#core) to handle rendering large numbers of cells with lots of complex interactivity
-- The app is designed with a client-server model that can support a range of existing analysis packages for backend computational tasks (currently built for [scanpy](https://github.com/theislab/scanpy))
-- The client uses fast cross-filtering to handle selections and comparisons across subsets of data
-
-Depending on your background and interests, you might want to contribute to the frontend, or backend, or both!
-
-If you are interested in working on `cellxgene` development, we recommend cloning the project from Gitub. First you'll need the following installed on your machine
-
-- python 3.6+
-- node and npm (we recommend using [nvm](https://github.com/creationix/nvm) if this is your first time with node)
-
-Then clone the project
-
-```
-git clone https://github.com/chanzuckerberg/cellxgene.git
-```
-
-Build the client web assets by calling `make` from inside the `cellxgene` folder
-
-```
-make
-```
-
-Install all requirements (we recommend doing this inside a virtual environment)
-
-```
-pip install -e .
-```
-
-You can start the app while developing either by calling `cellxgene` or by calling `python -m server`. We recommend using the `--debug` flag to see more output, which you can include when reporting bugs.
-
-If you have any questions about developing or contributing, come hang out with us by joining the [CZI Science Slack](https://join-cellxgene-users.herokuapp.com/) and posting in the `#cellxgene-dev` channel.
+- `$ cellxgene launch` uses Flask's development server, which is not recommended for hosted deployment (see the [Flask documentation](http://flask.pocoo.org/docs/1.0/tutorial/deploy/#run-with-a-production-server))
+- We have no testing or official support for deployments where multiple users are accessing the same _cellxgene_ instance.
+- Your _cellxgene_ instance is likely to hang or crash if too many people access it at the same time, especially if they using functions that call the Python backend (such as differential expression, updating the layout, or coloring by gene).
+- _cellxgene_ only supports one instance per dataset
 
 ## inspiration
 

ROADMAP.md

@@ -1,54 +1,46 @@
 # cellxgene roadmap
 
-We are very exited for _cellxgene_ to become a valuable tool in collaborations
-between computational biologists and experimental biologists working on
-single-cell transcriptomics data. _cellxgene_ is in active development, and we
-would love to include the community as we plan new features to work on. If you
-have questions of feedback about this roadmap, please submit an issue on
-GitHub.
-
+cellxgene makes it easier for biologists to collaboratively explore and understand their single-cell RNA-seq data. 
+In the near term, we are focused on continuing to enable fast, interactive exploration of single-cell data, supporting collaborative workflows in single-cell analysis, and improving user support. 
+If you have questions or feedback about this roadmap, please submit an issue on GitHub.
 Please note: this roadmap is subject to change.
 
-*Last updated: April 11, 2019*
+Last updated: June 25, 2019
 
-## what we are building now
+## Fast, interactive exploration of single-cell data
 
-In the near term, our goal is to enable teams of computational and experimental
-biologists to collaboratively explore and annotate their single-cell RNA-seq data.
+### Exposing Relationships Between Metadata and Data
+Biologists need to understand how variables (stored in metadata) are associated with one another and how they relate to changes in gene expression. 
+Building upon visualization features that reveal categorical metadata relationships (cluster occupancy) and gene expression relationships (scatterplot), we plan to add exploratory visualization components that enable investigation of relationships between metadata and gene expression. 
+See [issue #616](https://github.com/chanzuckerberg/cellxgene/issues/616) for more details.
 
-There are 4 key features we plan to implement in the near term.
+### Contextualizing Genes
+While exploring a transcriptomics dataset, scientists need to understand the biological context of genes.
+This context may be provided by user-defined gene metadata or publicly available gene databases. 
+We plan to support augmenting gene names with additional information that is useful to biologists. 
+See [issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for more detail.
 
-- Click install and launch
-- Manual annotation workflows
-- Toggle embeddings
-- Gene information
+## Support collaborative workflows in single-cell analysis
 
-### simple install and launch
+### Manual Annotations
+cellxgene offers exploratory visualizations that are critical for manual annotation workflows, especially in collaborative environments. 
+We plan to support manually annotating cells with labels (i.e., cell type or QC flags), and their easy export for downstream analysis. 
+See [issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524) for more details.
 
-The command line interface for installing and launching cellxgene is a barrier
-for users who are not used to Python or using the command line. We plan to
-support installation and launch of cellxgene on Mac and Windows. See
-[Issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for more details.
+### Simple Click to Launch
 
-### manual annotation workflows
+Many biologists prefer not to interact with the command line and need an OS-native experience when using cellxgene. 
+We plan to implement a point-and-click installation and launch experience so that users can easily load data into cellxgene. 
+See [issue #687](https://github.com/chanzuckerberg/cellxgene/issues/687) for details.
 
-The exploratory visualization that cellxgene offers is critical for manual
-annotation workflows, especially in collaborative environments. We plan to
-support manually annotate cells with labels (i.e., cell type or QC flags) for
-downstream analysis. See [Issue #524](https://github.com/chanzuckerberg/cellxgene/issues/524)
-for more details.
+### Python API
+For computational biologists, saving h5ad files then loading them into cellxgene is a point of friction.  
+We plan to support importing cellxgene as a Python package so that users can launch cellxgene directly from an interactive environment (such as Jupyter, IPython, or Spyder), and pass data to and from the cellxgene UI. 
 
-### toggle embeddings
+## Improving user support
 
-While a single dataset may have multiple embeddings calculated (tSNE, umap, in
-situ coordinates, trajectories, etc), cellxgene currently requires the user to select the
-embedding to use in the main layout at launch. We plan to support letting users
-toggle between any embedding present in a file from the cellxgene interface.
-See [Issue #594](https://github.com/chanzuckerberg/cellxgene/issues/594) for details.
-
-### gene information
-
-Differential expression returns only the names of genes, but no additional information
-about gene metadata, function, or known associations. We plan to help users learn
-more about genes they discover by exposing additional gene metadata. See
-[Issue #96](https://github.com/chanzuckerberg/cellxgene/issues/96) for details.
+### Improved documentation
+cellxgene has some specific expectations about how data is stored. 
+We want to ensure that new users can get started easily and learn how to use cellxgene with their own data. 
+We plan to improve documentation on getting started, installation, data, and contributing. 
+See [issue #533](https://github.com/chanzuckerberg/cellxgene/issues/533) for more details.

cellxgene-osx.spec

@@ -0,0 +1,41 @@
+# -*- mode: python ; coding: utf-8 -*-
+
+block_cipher = None
+
+
+a = Analysis(['server/gui/main.py'],
+             pathex=['/Users/charlotteweaver/Documents/Git/cellxgene'],
+             binaries=[('/System/Library/Frameworks/Tk.framework/Tk', 'tk'), ('/System/Library/Frameworks/Tcl.framework/Tcl', 'tcl')],
+             datas=[('server/app/web/templates/', 'server/app/web/templates/'), ('server/app/web/static/', 'server/app/web/static/')],
+             hiddenimports=['sklearn', 'sklearn.utils._cython_blas', 'sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree', 'sklearn.tree._utils'],
+             hookspath=['server/gui/'],
+             runtime_hooks=[],
+             excludes=[],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+exe = EXE(pyz,
+          a.scripts,
+          [],
+          exclude_binaries=True,
+          name='cellxgene',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          console=False , icon='server/gui/images/cxg_icons.icns')
+coll = COLLECT(exe,
+               a.binaries,
+               a.zipfiles,
+               a.datas,
+               strip=False,
+               upx=True,
+               upx_exclude=[],
+               name='cellxgene')
+app = BUNDLE(coll,
+             name='cellxgene.app',
+             icon='server/gui/images/cxg_icons.icns',
+             bundle_identifier=None)

cellxgene-windows.spec

@@ -0,0 +1,36 @@
+# -*- mode: python -*-
+
+block_cipher = None
+
+
+a = Analysis(['server\\gui\\main.py'],
+             pathex=['C:\\Users\\Charlotte\\Documents\\git\\cellxgene'],
+             binaries=[],
+             datas=[('server/app/web/templates/', 'server/app/web/templates'), ('server/app/web/static/', 'server/app/web/static')],
+             hiddenimports=[],
+             hookspath=['server/gui/'],
+             runtime_hooks=[],
+             excludes=[],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+exe = EXE(pyz,
+          a.scripts,
+          [],
+          exclude_binaries=True,
+          name='cellxgene',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          console=False , icon='server\\gui\\images\\icon.ico')
+coll = COLLECT(exe,
+               a.binaries,
+               a.zipfiles,
+               a.datas,
+               strip=False,
+               upx=True,
+               name='cellxgene')

client/__tests__/e2e/data.js

@@ -1,6 +1,6 @@
 export const datasets = {
   pbmc3k: {
-    title: "cellxgene: pbmc3k",
+    title: "pbmc3k",
     dataframe: {
       nObs: "2638",
       nVar: "1838",
@@ -26,8 +26,8 @@ export const datasets = {
     cellsets: {
       lasso: [
         {
-          "coordinates-as-percent": { x1: 0.25, y1: 0.25, x2: 0.35, y2: 0.35 },
-          count: "26"
+          "coordinates-as-percent": { x1: 0.05, y1: 0.25, x2: 0.15, y2: 0.35 },
+          count: "71"
         }
       ],
       categorical: [
@@ -91,8 +91,8 @@ export const datasets = {
         }
       },
       lasso: {
-        "coordinates-as-percent": { x1: 0.45, y1: 0.45, x2: 0.5, y2: 0.5 },
-        count: "67"
+        "coordinates-as-percent": { x1: 0.45, y1: 0.05, x2: 0.65, y2: 0.15 },
+        count: "36"
       }
     },
     scatter: {
@@ -108,6 +108,15 @@ export const datasets = {
           count: "24"
         }
       }
+    },
+    clip: {
+      min: "30",
+      max: "70",
+      metadata: "n_genes",
+      gene: "S100A8",
+      "coordinates-as-percent": { x1: 0.25, y1: 0.5, x2: 0.55, y2: 0.5 },
+      count: "392",
+      "gene-cell-count": "421"
     }
   }
 };

client/__tests__/e2e/e2e.test.js

@@ -54,14 +54,14 @@ afterAll(() => {
   }
 });
 
-describe("did launch", async () => {
+describe("did launch", () => {
   test("page launched", async () => {
     let el = await utils.getOneElementInnerHTML("[data-testid='header']");
     expect(el).toBe(data.title);
   });
 });
 
-describe("metadata loads", async () => {
+describe("metadata loads", () => {
   test("categories and values from dataset appear", async () => {
     for (const label in data.categorical) {
       await utils.waitByID(`category-${label}`);
@@ -87,7 +87,7 @@ describe("metadata loads", async () => {
   });
 });
 
-describe("cell selection", async () => {
+describe("cell selection", () => {
   test("selects all cells cellset 1", async () => {
     const cellCount = await cxgActions.cellSet(1);
     expect(cellCount).toBe(data.dataframe.nObs);
@@ -138,7 +138,7 @@ describe("cell selection", async () => {
   });
 });
 
-describe("gene entry", async () => {
+describe("gene entry", () => {
   test("search for single gene", async () => {
     // blueprint's  typeahead is treating typing weird, clicking & waiting first solves this
     await utils.typeInto("gene-search", data.genes.search);
@@ -154,14 +154,17 @@ describe("gene entry", async () => {
     await utils.clickOn("section-bulk-add");
     await utils.typeInto("input-bulk-add", testGenes.join(","));
     await page.keyboard.press("Enter");
-    const userGeneHist = await cxgActions.getAllHistograms(
-      "histogram-user-gene"
+
+    const allHistograms = await cxgActions.getAllHistograms(
+      "histogram-user-gene",
+      testGenes
     );
-    expect(userGeneHist).toEqual(expect.arrayContaining(testGenes));
+    expect(allHistograms).toEqual(expect.arrayContaining(testGenes));
+    expect(allHistograms.length).toEqual(testGenes.length);
   });
 });
 
-describe("diffexp", async () => {
+describe("diffexp", () => {
   test("selects cells, saves them and performs diffexp", async () => {
     for (const select of data.diffexp.cellset1) {
       if (select.kind === "categorical") {
@@ -176,15 +179,18 @@ describe("diffexp", async () => {
     }
     await cxgActions.cellSet(2);
     await utils.clickOn("diffexp-button");
-    const diffExpHists = await cxgActions.getAllHistograms("histogram-diffexp");
-    expect(diffExpHists).toEqual(
+    const allHistograms = await cxgActions.getAllHistograms(
+      "histogram-diffexp",
+      data.diffexp["gene-results"]
+    );
+    expect(allHistograms).toEqual(
       expect.arrayContaining(data.diffexp["gene-results"])
     );
+    expect(allHistograms.length).toEqual(data.diffexp["gene-results"].length);
   });
 });
-//
 
-describe("subset/reset", async () => {
+describe("subset/reset", () => {
   test("subset - cell count matches", async () => {
     for (const select of data.subset.cellset1) {
       if (select.kind === "categorical") {
@@ -258,7 +264,7 @@ describe("subset/reset", async () => {
   });
 });
 
-describe("scatter plot", async () => {
+describe("scatter plot", () => {
   test("scatter plot appears", async () => {
     await cxgActions.reset();
     const testGenes = data.scatter.genes;
@@ -271,8 +277,37 @@ describe("scatter plot", async () => {
   });
 });
 
+describe("clipping", () => {
+  test("clip continuous", async () => {
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.metadata}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip.count);
+  });
+
+  test("clip gene", async () => {
+    await utils.typeInto("gene-search", data.clip.gene);
+    await page.keyboard.press("Enter");
+    await page.waitForSelector(`[data-testid='histogram-${data.clip.gene}']`);
+    await cxgActions.clip(data.clip.min, data.clip.max);
+    const histId = `histogram-${data.clip.gene}-plot-brush`;
+    const coords = await cxgActions.calcDragCoordinates(
+      histId,
+      data.clip["coordinates-as-percent"]
+    );
+    await cxgActions.drag(histId, coords.start, coords.end);
+    const cellCount = await cxgActions.cellSet(1);
+    expect(cellCount).toBe(data.clip["gene-cell-count"]);
+  });
+});
+
 // interact with UI elements just that they do not break
-describe("ui elements don't error", async () => {
+describe("ui elements don't error", () => {
   test("color by", async () => {
     for (const label in data.categorical) {
       await utils.clickOn(`colorby-${label}`);

client/__tests__/e2e/puppeteerUtils.js

@@ -13,13 +13,48 @@ export const puppeteerUtils = puppeteerPage => ({
     );
   },
 
+  async waitForAllByIds(testids, props = {}) {
+    await Promise.all(
+      testids.map(testid =>
+        puppeteerPage.waitForSelector(`[data-testid='${testid}']`)
+      )
+    );
+  },
+
+  async getAllByClass(testclass, props = {}) {
+    const elements = await puppeteerPage.$$eval(
+      `[data-testclass=${testclass}]`,
+      els => {
+        return els.map(el => {
+          return el.dataset.testid;
+        });
+      }
+    );
+    return elements;
+  },
+
   async typeInto(testid, text) {
     // only works for text without special characters
     await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
     // type ahead can be annoying if you don't pause before you type
-    await puppeteerPage.click(`[data-testid='${testid}']`);
+    await puppeteerPage.click(selector);
     await puppeteerPage.waitFor(200);
-    await puppeteerPage.type(`[data-testid='${testid}']`, text);
+    await puppeteerPage.type(selector, text);
+  },
+
+  async clearInputAndTypeInto(testid, text) {
+    await this.waitByID(testid);
+    const selector = `[data-testid='${testid}']`;
+    // only works for text without special characters
+    // type ahead can be annoying if you don't pause before you type
+    await puppeteerPage.click(selector);
+    await puppeteerPage.waitFor(200);
+    // select all
+
+    await puppeteerPage.click(selector, { clickCount: 3 });
+    await puppeteerPage.keyboard.type("Backspace");
+    await puppeteerPage.type(selector, text);
   },
 
   async clickOn(testid) {
@@ -29,11 +64,13 @@ export const puppeteerUtils = puppeteerPage => ({
   },
 
   async getOneElementInnerHTML(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerHTML);
     return text;
   },
 
   async getOneElementInnerText(selector) {
+    await puppeteerPage.waitForSelector(selector);
     let text = await puppeteerPage.$eval(selector, el => el.innerText);
     return text;
   }
@@ -60,20 +97,16 @@ export const cellxgeneActions = puppeteerPage => ({
     await puppeteerPage.mouse.up();
   },
 
-  async getAllHistograms(testclass) {
-    await puppeteerUtils(puppeteerPage).waitByClass(testclass);
-    const histograms = await puppeteerPage.$$eval(
-      `[data-testclass=${testclass}]`,
-      els => {
-        return els.map(el => {
-          return el.dataset.testid.substring(
-            "histogram_".length,
-            el.dataset.testid.length
-          );
-        });
-      }
+  async getAllHistograms(testclass, testids) {
+    const histTestIds = testids.map(tid => `histogram-${tid}`);
+    // these load asynchronously, so we need to wait for each histogram individually
+    await puppeteerUtils(puppeteerPage).waitForAllByIds(histTestIds);
+    const allHistograms = await puppeteerUtils(puppeteerPage).getAllByClass(
+      testclass
+    );
+    return allHistograms.map(hist =>
+      hist.substr("histogram_".length, hist.length)
     );
-    return histograms;
   },
 
   async getAllCategoriesAndCounts(category) {
@@ -161,5 +194,18 @@ export const cellxgeneActions = puppeteerPage => ({
     await puppeteerUtils(puppeteerPage).clickOn("reset");
     // loading state never actually happens, reset is too fast
     await page.waitFor(200);
+  },
+
+  async clip(min = 0, max = 100) {
+    await puppeteerUtils(puppeteerPage).clickOn("visualization-settings");
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto(
+      "clip-min-input",
+      min
+    );
+    await puppeteerUtils(puppeteerPage).clearInputAndTypeInto(
+      "clip-max-input",
+      max
+    );
+    await puppeteerUtils(puppeteerPage).clickOn("clip-commit");
   }
 });

client/__tests__/util/dataframe/histogram.test.js

@@ -0,0 +1,69 @@
+import * as Dataframe from "../../../src/util/dataframe";
+
+describe("Dataframe column histogram", () => {
+  test("categorical by categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("cat").histogram(df.col("name"));
+    expect(h1).toMatchObject(
+      new Map([
+        ["n1", new Map([["c1", 1]])],
+        ["n2", new Map([["c2", 1]])],
+        ["n3", new Map([["c3", 1]])]
+      ])
+    );
+    // memoized?
+    expect(df.col("cat").histogram(df.col("name"))).toMatchObject(h1);
+  });
+
+  test("continuous by categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("value").histogram(3, [0, 2], df.col("name"));
+    expect(h1).toMatchObject(
+      new Map([["n1", [1, 0, 0]], ["n2", [0, 1, 0]], ["n3", [0, 0, 1]]])
+    );
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2], df.col("name"))).toMatchObject(
+      h1
+    );
+  });
+
+  test("categorical", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("cat").histogram();
+    expect(h1).toMatchObject(new Map([["c1", 1], ["c2", 1], ["c3", 1]]));
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2])).toMatchObject(h1);
+  });
+
+  test("continuous", () => {
+    const df = new Dataframe.Dataframe(
+      [3, 3],
+      [["n1", "n2", "n3"], ["c1", "c2", "c3"], new Int32Array([0, 1, 2])],
+      null,
+      new Dataframe.KeyIndex(["name", "cat", "value"])
+    );
+
+    const h1 = df.col("value").histogram(3, [0, 2]);
+    expect(h1).toMatchObject([1, 1, 1]);
+    // memoized?
+    expect(df.col("value").histogram(3, [0, 2])).toMatchObject(h1);
+  });
+});

client/__tests__/util/range.test.js

@@ -0,0 +1,42 @@
+import { range, rangeFill } from "../../src/util/range";
+
+describe("range", () => {
+	test("no defaults", () => {
+		expect(range(0, 3, 1)).toMatchObject([0, 1, 2]);
+	});
+
+	test("range(stop)", () => {
+		expect(range(3)).toMatchObject([0, 1, 2]);
+		expect(range(0)).toMatchObject([]);
+		expect(range(1)).toMatchObject([0]);
+	});
+
+	test("range(start,stop)", () => {
+		expect(range(0, 0)).toMatchObject([]);
+		expect(range(0, 2)).toMatchObject([0, 1]);
+		expect(range(4, 8)).toMatchObject([4, 5, 6, 7]);
+	});
+
+	test("range(start, stop, step", () => {
+		expect(range(4, 0, -1)).toMatchObject([4, 3, 2, 1]);
+		expect(range(0, 4, 2)).toMatchObject([0, 2]);
+	});
+});
+
+describe("rangefill", () => {
+	test("rangeFill(arr)", () => {
+		expect(rangeFill(new Int32Array(3))).toMatchObject(
+			new Int32Array([0, 1, 2])
+		);
+	});
+	test("rangeFill(arr, start)", () => {
+		expect(rangeFill(new Int32Array(2), 1)).toMatchObject(
+			new Int32Array([1, 2])
+		);
+	});
+	test("rangeFill(arr, start, step)", () => {
+		expect(rangeFill(new Int32Array(3), 2, -1)).toMatchObject(
+			new Int32Array([2, 1, 0])
+		);
+	});
+});

client/__tests__/util/stateManager/sampleResponses.js

@@ -34,28 +34,38 @@ const aSchemaResponse = {
       type: "float32"
     },
     annotations: {
-      obs: [
-        { name: "name", type: "string" },
-        { name: "field1", type: "int32" },
-        { name: "field2", type: "float32" },
-        { name: "field3", type: "boolean" },
-        {
-          name: "field4",
-          type: "categorical",
-          categories: field4Categories
-        }
-      ],
-      var: [
-        { name: "name", type: "string" },
-        { name: "fieldA", type: "int32" },
-        { name: "fieldB", type: "float32" },
-        { name: "fieldC", type: "boolean" },
-        {
-          name: "fieldD",
-          type: "categorical",
-          categories: fieldDCategories
-        }
-      ]
+      obs: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "field1", type: "int32" },
+          { name: "field2", type: "float32" },
+          { name: "field3", type: "boolean" },
+          {
+            name: "field4",
+            type: "categorical",
+            categories: field4Categories
+          }
+        ]
+      },
+      var: {
+        index: "name",
+        columns: [
+          { name: "name", type: "string" },
+          { name: "fieldA", type: "int32" },
+          { name: "fieldB", type: "float32" },
+          { name: "fieldC", type: "boolean" },
+          {
+            name: "fieldD",
+            type: "categorical",
+            categories: fieldDCategories
+          }
+        ]
+      }
+    },
+    layout: {
+      obs: [{ name: "umap", type: "float32", dims: ["umap_0", "umap_1"] }],
+      var: []
     }
   }
 };
@@ -162,29 +172,7 @@ const aLayoutFBSResponse = (() => {
     new Float32Array(nObs).fill(Math.random()),
     new Float32Array(nObs).fill(Math.random())
   ];
-  const builder = new flatbuffers.Builder(1024);
-
-  const cols = _.map(coords, carr => {
-    const cdv = NetEncoding.Float32Array.createDataVector(builder, carr);
-    NetEncoding.Float32Array.startFloat32Array(builder);
-    NetEncoding.Float32Array.addData(builder, cdv);
-    const floatArr = NetEncoding.Float32Array.endFloat32Array(builder);
-
-    NetEncoding.Column.startColumn(builder);
-    NetEncoding.Column.addUType(builder, NetEncoding.TypedArray.Float32Array);
-    NetEncoding.Column.addU(builder, floatArr);
-    return NetEncoding.Column.endColumn(builder);
-  });
-
-  const columns = NetEncoding.Matrix.createColumnsVector(builder, cols);
-
-  NetEncoding.Matrix.startMatrix(builder);
-  NetEncoding.Matrix.addNRows(builder, nObs);
-  NetEncoding.Matrix.addNCols(builder, coords.length);
-  NetEncoding.Matrix.addColumns(builder, columns);
-  const matrix = NetEncoding.Matrix.endMatrix(builder);
-  builder.finish(matrix);
-  return builder.asUint8Array();
+  return encodeMatrix(coords, ["umap_0", "umap_1"]);
 })();
 
 const aDataObsResponse = {

client/__tests__/util/stateManager/universe.test.js

@@ -53,13 +53,15 @@ describe("createUniverseFromResponse", () => {
 
     expect(universe.obsAnnotations.dims).toEqual([
       nObs,
-      REST.schema.schema.annotations.obs.length
+      REST.schema.schema.annotations.obs.columns.length
     ]);
     expect(universe.obsLayout.dims).toEqual([nObs, 2]);
-    expect(universe.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(universe.obsLayout.colIndex.keys()).toEqual(
+      universe.schema.layout.obs[0].dims
+    );
     expect(universe.varAnnotations.dims).toEqual([
       nVar,
-      REST.schema.schema.annotations.var.length
+      REST.schema.schema.annotations.var.columns.length
     ]);
     expect(universe.varData.isEmpty()).toBeTruthy();
   });

client/__tests__/util/stateManager/world.test.js

@@ -29,7 +29,8 @@ const defaultBigBang = () => {
   /* create crossfilter */
   const crossfilter = World.createObsDimensions(
     new Crossfilter(world.obsAnnotations),
-    world
+    world,
+    REST.schema.schema.layout.obs[0].dims
   );
 
   return {
@@ -138,7 +139,9 @@ describe("createWorldFromCurrentSelection", () => {
     expect(world.obsLayout.rowIndex.keys()).toEqual(
       new Int32Array(matchingIndices)
     );
-    expect(world.obsLayout.colIndex.keys()).toEqual(["X", "Y"]);
+    expect(world.obsLayout.colIndex.keys()).toEqual(
+      world.schema.layout.obs[0].dims
+    );
   });
 });
 
@@ -152,14 +155,18 @@ describe("createObsDimensionMap", () => {
 
     const { crossfilter } = defaultBigBang();
     const annotationNames = _.map(
-      REST.schema.schema.annotations.obs,
+      REST.schema.schema.annotations.obs.columns,
       c => c.name
     );
-    const schemaByObsName = _.keyBy(REST.schema.schema.annotations.obs, "name");
+    const obsIndexColName = REST.schema.schema.annotations.obs.index;
+    const schemaByObsName = _.keyBy(
+      REST.schema.schema.annotations.obs.columns,
+      "name"
+    );
     expect(crossfilter).toBeDefined();
     annotationNames.forEach(name => {
       const dim = crossfilter.dimensions[obsAnnoDimensionName(name)];
-      if (name === "name") {
+      if (name === obsIndexColName) {
         expect(dim).toBeUndefined();
       } else {
         const { type } = schemaByObsName[name];

client/__tests__/util/typedCrossfilter/util.test.js

@@ -1,8 +1,8 @@
 import {
-  fillRange,
   sliceByIndex,
   makeSortIndex
 } from "../../../src/util/typedCrossfilter/util";
+import { rangeFill as fillRange } from "../../../src/util/range";
 
 describe("fillRange", () => {
   test("Array", () => {

client/configuration/babel/babel.dev.js

@@ -7,8 +7,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-proposal-optional-chaining",
     "@babel/plugin-proposal-nullish-coalescing-operator"

client/configuration/babel/babel.prod.js

@@ -6,8 +6,8 @@ module.exports = {
   ],
   plugins: [
     "@babel/plugin-proposal-function-bind",
-    "@babel/plugin-proposal-class-properties",
     ["@babel/plugin-proposal-decorators", { legacy: true }],
+    ["@babel/plugin-proposal-class-properties", { loose: true }],
     "@babel/plugin-proposal-export-namespace-from",
     "@babel/plugin-transform-react-constant-elements",
     "@babel/plugin-transform-runtime",

client/index.html

@@ -3,7 +3,7 @@
   <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>cellxgene</title>
+    <title>cell&times;gene</title>
     <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
       html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {

client/index_template.html

@@ -3,7 +3,7 @@
 <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>cellxgene</title>
+    <title>cell&times;gene</title>
     <link href="https://fonts.googleapis.com/css?family=Roboto+Condensed:400,400i,700" rel="stylesheet">
     <style>
         html, body, p, h1, h2, h3, h4, h5, h6, span, button, input, label, text, div {

client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cellxgene",
-  "version": "0.9.1",
+  "version": "0.11.0",
   "license": "MIT",
   "description": "cellxgene is a web application for the interactive exploration of single cell sequence data.",
   "repository": "https://github.com/chanzuckerberg/cellxgene",
@@ -9,13 +9,13 @@
     "build": "npm run clean && webpack --config configuration/webpack/webpack.config.prod.js",
     "clean": "rimraf build",
     "dev": "npm run clean && webpack --config configuration/webpack/webpack.config.dev.js",
-    "e2e": "jest --verbose false --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
+    "e2e": "node node_modules/jest/bin/jest.js --verbose false --config __tests__/e2e/e2eJestConfig.json e2e/e2e.test.js",
     "lint": "eslint src",
     "smoke-test": "start-server-and-test start-server-for-test :5000 e2e",
     "start": "node server/development.js",
     "start-server-for-test": "cellxgene launch -p 5000 ../example-dataset/pbmc3k.h5ad",
-    "test": "jest",
-    "unit-test": "jest --testPathIgnorePatterns e2e"
+    "test": "node node_modules/jest/bin/jest.js",
+    "unit-test": "node node_modules/jest/bin/jest.js --testPathIgnorePatterns e2e"
   },
   "engineStrict": true,
   "engines": {
@@ -31,8 +31,8 @@
     "eslint-scope": "3.7.1"
   },
   "dependencies": {
-    "@blueprintjs/core": "^3.15.0",
-    "@blueprintjs/icons": "^3.3.0",
+    "@blueprintjs/core": "^3.15.1",
+    "@blueprintjs/icons": "^3.8.0",
     "@blueprintjs/select": "^3.8.0",
     "canvas-fit": "^1.5.0",
     "d3": "^4.10.0",
@@ -41,79 +41,77 @@
     "font-color-contrast": "^1.0.3",
     "fuzzysort": "^1.1.4",
     "gl-mat4": "^1.1.4",
-    "gl-matrix": "^2.7.1",
+    "gl-matrix": "^3.0.0",
+    "gl-vec3": "^1.1.3",
     "is-number": "^7.0.0",
     "key-pressed": "0.0.1",
-    "lodash": "^4.17.4",
-    "memoize-one": "^4.0.0",
+    "lodash": "^4.17.14",
+    "memoize-one": "^5.0.4",
     "mouse-position": "^2.0.1",
     "mouse-pressed": "^1.0.0",
-    "normalize.css": "^8.0.0",
     "orbit-camera": "^1.0.0",
-    "query-string": "^6.1.0",
-    "react": "^16.6.0",
+    "query-string": "^6.5.0",
+    "react": "^16.8.6",
     "react-autocomplete": "^1.7.2",
-    "react-dom": "^16.6.0",
-    "react-helmet": "^5.2.0",
-    "react-icons": "^3.2.2",
-    "react-redux": "^5.1.0",
+    "react-dom": "^16.8.6",
+    "react-helmet": "^5.2.1",
+    "react-icons": "^3.7.0",
+    "react-redux": "^7.0.3",
     "redux": "^4.0.1",
-    "redux-devtools-extension": "^2.13.5",
     "redux-thunk": "^2.2.0",
-    "regl": "^1.3.9",
+    "regl": "^1.3.11",
     "scroll-speed": "^1.0.0",
     "urijs": "^1.19.0"
   },
   "devDependencies": {
-    "@babel/core": "^7.1.5",
-    "@babel/plugin-proposal-class-properties": "^7.0.0",
-    "@babel/plugin-proposal-decorators": "^7.0.0",
-    "@babel/plugin-proposal-export-namespace-from": "^7.0.0",
-    "@babel/plugin-proposal-function-bind": "^7.0.0",
-    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.2.0",
+    "@babel/core": "^7.4.4",
+    "@babel/plugin-proposal-class-properties": "^7.4.4",
+    "@babel/plugin-proposal-decorators": "^7.4.4",
+    "@babel/plugin-proposal-export-namespace-from": "^7.2.0",
+    "@babel/plugin-proposal-function-bind": "^7.2.0",
+    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.4.4",
     "@babel/plugin-proposal-optional-chaining": "^7.2.0",
-    "@babel/plugin-transform-react-constant-elements": "^7.0.0",
-    "@babel/plugin-transform-runtime": "^7.1.0",
-    "@babel/preset-env": "^7.1.5",
+    "@babel/plugin-transform-react-constant-elements": "^7.2.0",
+    "@babel/plugin-transform-runtime": "^7.4.4",
+    "@babel/preset-env": "^7.4.4",
     "@babel/preset-react": "^7.0.0",
-    "@babel/register": "^7.0.0",
-    "@babel/runtime": "^7.1.5",
-    "babel-core": "^7.0.0-bridge.0",
+    "@babel/register": "^7.4.4",
+    "@babel/runtime": "^7.4.4",
     "babel-eslint": "^10.0.1",
-    "babel-jest": "^23.6.0",
-    "babel-loader": "^8.0.0",
-    "babel-preset-modern-browsers": "^12.0.0",
+    "babel-jest": "^24.8.0",
+    "babel-loader": "^8.0.6",
+    "babel-preset-modern-browsers": "^14.0.0",
     "chalk": "^2.4.2",
     "connect-history-api-fallback": "^1.6.0",
-    "copy-webpack-plugin": "^4.6.0",
-    "css-loader": "^1.0.1",
-    "eslint": "^5.13.0",
+    "copy-webpack-plugin": "^5.0.3",
+    "css-loader": "^2.1.1",
+    "eslint": "^5.16.0",
     "eslint-config-airbnb": "^17.1.0",
-    "eslint-config-prettier": "^4.0.0",
+    "eslint-config-prettier": "^4.2.0",
     "eslint-loader": "^2.1.2",
     "eslint-plugin-filenames": "^1.3.2",
-    "eslint-plugin-import": "^2.16.0",
-    "eslint-plugin-jest": "^22.2.2",
+    "eslint-plugin-import": "^2.17.2",
+    "eslint-plugin-jest": "^22.5.1",
     "eslint-plugin-jsx-a11y": "^6.2.1",
-    "eslint-plugin-react": "^7.12.4",
+    "eslint-plugin-react": "^7.13.0",
     "express": "^4.14.0",
-    "file-loader": "^2.0.0",
+    "file-loader": "^3.0.1",
     "html-webpack-inline-source-plugin": "0.0.10",
     "html-webpack-plugin": "^3.2.0",
-    "jest": "^24.1.0",
-    "jest-puppeteer": "^4.1.0",
+    "jest": "^24.8.0",
+    "jest-puppeteer": "^4.1.1",
     "json-loader": "^0.5.4",
-    "mini-css-extract-plugin": "^0.4.1",
-    "puppeteer": "^1.12.1",
+    "mini-css-extract-plugin": "^0.6.0",
+    "puppeteer": "^1.16.0",
     "rimraf": "^2.6.3",
     "serve-favicon": "^2.3.0",
-    "start-server-and-test": "^1.7.11",
+    "start-server-and-test": "^1.9.0",
     "style-loader": "^0.23.1",
     "sw-precache-webpack-plugin": "^0.11.5",
     "url-loader": "^1.1.0",
-    "webpack": "^4.25.1",
-    "webpack-cli": "^3.1.0",
-    "webpack-dev-middleware": "^3.1.3"
+    "webpack": "^4.31.0",
+    "webpack-cli": "^3.3.2",
+    "webpack-dev-middleware": "^3.6.2"
   },
   "jest": {
     "testMatch": [
@@ -133,16 +131,23 @@
         ],
         "plugins": [
           "@babel/plugin-proposal-function-bind",
-          "@babel/plugin-proposal-class-properties",
           [
             "@babel/plugin-proposal-decorators",
             {
               "legacy": true
             }
           ],
+          [
+            "@babel/plugin-proposal-class-properties",
+            {
+              "loose": true
+            }
+          ],
           "@babel/plugin-proposal-export-namespace-from",
           "@babel/plugin-transform-react-constant-elements",
-          "@babel/plugin-transform-runtime"
+          "@babel/plugin-transform-runtime",
+          "@babel/plugin-proposal-optional-chaining",
+          "@babel/plugin-proposal-nullish-coalescing-operator"
         ]
       }
     }

client/src/actions/index.js

@@ -21,24 +21,32 @@ const doInitialDataLoad = () =>
     dispatch({ type: "initial data load start" });
 
     try {
-      const requestJson = _(["config", "schema"])
+      /*
+      Step 1 - config & schema, all JSON
+      */
+      const requestJson = ["config", "schema"]
         .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doJsonRequest(url))
-        .value();
-      const requestBinary = _([
-        "annotations/obs",
-        "annotations/var?annotation-name=name",
-        "layout/obs"
-      ])
-        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
-        .map(url => doBinaryRequest(url))
-        .value();
-
-      const results = await Promise.all(_.concat(requestJson, requestBinary));
-
+        .map(url => doJsonRequest(url));
+      const stepOneResults = await Promise.all(requestJson);
       /* set config defaults */
-      const config = { ...globals.configDefaults, ...results[0].config };
-      const [, schema, obsAnno, varAnno, obsLayout] = [...results];
+      const config = { ...globals.configDefaults, ...stepOneResults[0].config };
+      const schema = stepOneResults[1];
+
+      /*
+      Step 2 - dataframes, all binary.  NOTE: uses results of step 1.
+      */
+      /* only load names for var annotations, if possible*/
+      const varIndexName = schema?.schema?.annotations?.var?.index;
+      const varAnnotationsQuery = varIndexName
+        ? `?annotation-name=${varIndexName}`
+        : "";
+      const varAnnotationsURL = `annotations/var${varAnnotationsQuery}`;
+      const requestBinary = ["annotations/obs", varAnnotationsURL, "layout/obs"]
+        .map(r => `${globals.API.prefix}${globals.API.version}${r}`)
+        .map(url => doBinaryRequest(url));
+      const stepTwoResults = await Promise.all(requestBinary);
+      const [obsAnno, varAnno, obsLayout] = [...stepTwoResults];
+
       const universe = Universe.createUniverseFromResponse(
         config,
         schema,
@@ -91,6 +99,10 @@ needs expression data.
 Transparently utilizes cached data if it is already present.
 */
 async function _doRequestExpressionData(dispatch, getState, genes) {
+  const state = getState();
+  const { universe } = state;
+  const varIndexName = universe.schema.annotations.var.index;
+
   /* helper for this function only */
   const fetchData = async geneNames => {
     const res = await fetch(
@@ -100,7 +112,7 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
         body: JSON.stringify({
           filter: {
             var: {
-              annotation_value: [{ name: "name", values: geneNames }]
+              annotation_value: [{ name: varIndexName, values: geneNames }]
             }
           }
         }),
@@ -123,8 +135,6 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
     return Universe.convertDataFBStoObject(universe, data);
   };
 
-  const state = getState();
-  const { universe } = state;
   /* preload data already in cache */
   let expressionData = _.transform(
     genes,
@@ -241,6 +251,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     */
     const state = getState();
     const { universe } = state;
+    const varIndexName = universe.schema.annotations.var.index;
 
     // Legal values are null, Array or TypedArray.  Null is initial state.
     if (!set1) set1 = [];
@@ -277,7 +288,7 @@ const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
     const data = await res.json();
     // result is [ [varIdx, ...], ... ]
     const topNGenes = _.map(data, r =>
-      universe.varAnnotations.at(r[0], "name")
+      universe.varAnnotations.at(r[0], varIndexName)
     );
 
     /*

client/src/components/app.js

@@ -4,9 +4,11 @@ import Helmet from "react-helmet";
 import { connect } from "react-redux";
 
 import Container from "./framework/container";
-import LeftSideBar from "./leftsidebar";
+import LeftSideBar from "./leftSidebar";
 import Legend from "./continuousLegend";
 import Graph from "./graph/graph";
+import MenuBar from "./menubar";
+
 import actions from "../actions";
 
 @connect(state => ({
@@ -71,18 +73,23 @@ class App extends React.Component {
             loading cellxgene
           </div>
         ) : null}
-        <div>
-          {loading ? null : <LeftSideBar />}
+        {error ? (
           <div
             style={{
-              padding: 15,
-              width: 1440 - 410 /* but responsive */,
-              marginLeft: 350 /* but responsive */
+              position: "fixed",
+              fontWeight: 500,
+              top: window.innerHeight / 2,
+              left: window.innerWidth / 2 - 50
             }}
           >
-            {loading ? null : <Graph key={graphRenderCounter} />}
-            <Legend />
+            error loading
           </div>
+        ) : null}
+        <div>
+          {loading ? null : <LeftSideBar />}
+          {loading ? null : <MenuBar />}
+          {loading ? null : <Graph key={graphRenderCounter} />}
+          <Legend />
         </div>
       </Container>
     );

client/src/components/categorical/categorical.css

@@ -0,0 +1,3 @@
+:local(.value):hover {
+  background: rgba(167, 182, 194, 0.3);
+}

client/src/components/categorical/categorical.js

@@ -19,13 +19,6 @@ class Categories extends React.Component {
           padding: globals.leftSidebarSectionPadding
         }}
       >
-        <p
-          style={Object.assign({}, globals.leftSidebarSectionHeading, {
-            marginTop: 4
-          })}
-        >
-          Categorical Metadata
-        </p>
         {_.map(categoricalSelection, (catState, catName) => (
           <Category key={catName} metadataField={catName} />
         ))}

client/src/components/categorical/category.js

@@ -27,10 +27,10 @@ class Category extends React.Component {
       const cat = categoricalSelection[metadataField];
       const categoryCount = {
         // total number of categories in this dimension
-        totalCatCount: cat.numCategories,
+        totalCatCount: cat.numCategoryValues,
         // number of selected options in this category
         selectedCatCount: _.reduce(
-          cat.categorySelected,
+          cat.categoryValueSelected,
           (res, cond) => (cond ? res + 1 : res),
           0
         )
@@ -91,7 +91,7 @@ class Category extends React.Component {
     const { categoricalSelection, metadataField } = this.props;
 
     const cat = categoricalSelection[metadataField];
-    const optTuples = sortedCategoryValues([...cat.categoryIndices]);
+    const optTuples = sortedCategoryValues([...cat.categoryValueIndices]);
     return _.map(optTuples, (tuple, i) => (
       <Value
         optTuples={optTuples}

client/src/components/categorical/occupancy.js

@@ -1,69 +1,192 @@
 // jshint esversion: 6
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import * as d3 from "d3";
+import {
+  Popover,
+  PopoverInteractionKind,
+  Position,
+  Classes
+} from "@blueprintjs/core";
 
 @connect()
 class Occupancy extends React.Component {
-  render() {
-    const {
-      occupancy,
-      colorScale,
-      categoricalSelection,
-      colorAccessor,
-      schema
-    } = this.props;
-    const width = 100;
-    const height = 11;
+  _WIDTH = 100;
 
-    const categories = _.filter(schema.annotations.obs, {
-      name: colorAccessor
-    })[0].categories;
+  _HEIGHT = 11;
+
+  createHistogram = () => {
+    /* 
+      Knowing that colorScale is based off continous data, 
+      createHistogram fetches the continous data in relation to the cells releveant to the catagory value.
+      It then seperates that data into 50 bins for drawing the mini-histogram
+    */
+    const {
+      world,
+      metadataField,
+      colorAccessor,
+      category,
+      categoryIndex
+    } = this.props;
+
+    if (!this.canvas) return;
+
+    const groupBy = world.obsAnnotations.col(metadataField);
+
+    const col =
+      world.obsAnnotations.col(colorAccessor) ||
+      world.varData.col(colorAccessor);
+
+    const range = col.summarize();
+
+    const histogramMap = col.histogram(
+      50,
+      [range.min, range.max],
+      groupBy
+    ); /* Because the signature changes we really need different names for histogram to differentiate signatures  */
+
+    const bins = histogramMap.get(category.categoryValues[categoryIndex]);
+
+    const xScale = d3
+      .scaleLinear()
+      .domain([0, bins.length])
+      .range([0, this._WIDTH]);
+
+    const largestBin = Math.max(...bins);
+
+    const yScale = d3
+      .scaleLinear()
+      .domain([0, largestBin])
+      .range([0, this._HEIGHT]);
+
+    const ctx = this.canvas.getContext("2d");
+
+    ctx.fillStyle = "#000";
+
+    let x;
+    let y;
+
+    const rectWidth = this._WIDTH / bins.length;
+
+    for (let i = 0, { length } = bins; i < length; i += 1) {
+      x = xScale(i);
+      y = yScale(bins[i]);
+      ctx.fillRect(x, this._HEIGHT - y, rectWidth, y);
+    }
+  };
+
+  createOccupancyStack = () => {
+    /* 
+      Knowing that the color scale is based off of catagorical data, 
+      createOccupancyStack obtains a map showing the number if cells per colored value
+      Using the colorScale a stack of colored bars is drawn representing the map
+     */
+    const {
+      world,
+      metadataField,
+      colorAccessor,
+      category,
+      categoryIndex,
+      schema,
+      colorScale
+    } = this.props;
+
+    const ctx = this.canvas?.getContext("2d");
+
+    if (!ctx) return;
+
+    const groupBy = world.obsAnnotations.col(metadataField);
+    const occupancyMap = world.obsAnnotations
+      .col(colorAccessor)
+      .histogram(groupBy);
+
+    const occupancy = occupancyMap.get(category.categoryValues[categoryIndex]);
 
     const x = d3
       .scaleLinear()
       /* get all the keys d[1] as an array, then find the sum */
-      .domain([0, d3.sum(Array.from(occupancy, d => d[1]))])
-      .range([0, width]);
+      .domain([0, d3.sum(Array.from(occupancy.values()))])
+      .range([0, this._WIDTH]);
+    const categories = schema.annotations.obsByName[colorAccessor]?.categories;
 
     let currentOffset = 0;
+    const dfColumn = world.obsAnnotations.col(colorAccessor);
+    const categoryValues = dfColumn.summarize().categories;
 
-    const stacks = categoricalSelection[colorAccessor].categoryValues.map(d => {
-      const o = occupancy.get(d);
+    let o;
+    let scaledValue;
+    let value;
 
-      const scaledValue = x(o);
-
-      const stackItem = {
-        key: d,
-        value: o || 0,
-        rectWidth: o ? scaledValue : 0,
-        offset: currentOffset,
-        fill: o ? colorScale(categories.indexOf(d)) : "rgb(255,255,255)"
-      };
+    for (let i = 0, { length } = categoryValues; i < length; i += 1) {
+      value = categoryValues[i];
+      o = occupancy.get(value);
+      scaledValue = x(o);
+      ctx.fillStyle = o
+        ? colorScale(categories.indexOf(value))
+        : "rgb(255,255,255)";
+      ctx.fillRect(currentOffset, 0, o ? scaledValue : 0, this._HEIGHT);
       currentOffset += o ? scaledValue : 0;
-      return stackItem;
-    });
+    }
+  };
+
+  render() {
+    const {
+      colorAccessor,
+      categoricalSelection,
+      category,
+      categoryIndex
+    } = this.props;
+
+    this.canvas?.getContext("2d").clearRect(0, 0, this._WIDTH, this._HEIGHT);
+
+    const colorByIsCatagoricalData = !!categoricalSelection[colorAccessor];
 
     return (
-      <svg
-        style={{
-          marginRight: 5,
-          width,
-          height
+      <Popover
+        interactionKind={PopoverInteractionKind.HOVER_TARGET_ONLY}
+        hoverOpenDelay={1500}
+        hoverCloseDelay={200}
+        position={Position.LEFT}
+        modifiers={{
+          preventOverflow: { enabled: false },
+          hide: { enabled: false }
         }}
+        lazy
+        usePortal
+        disabled={colorByIsCatagoricalData}
+        popoverClassName={Classes.POPOVER_CONTENT_SIZING}
       >
-        {stacks.map(d => (
-          <rect
-            key={d.key}
-            width={d.rectWidth}
-            height={height}
-            x={d.offset}
-            title={d.metadataField}
-            fill={d.fill}
-          />
-        ))}
-      </svg>
+        <canvas
+          className="bp3-popover-targer"
+          style={{
+            marginRight: 5,
+            width: this._WIDTH,
+            height: this._HEIGHT,
+            borderBottom: colorByIsCatagoricalData
+              ? ""
+              : "solid rgb(230, 230, 230) 0.25px"
+          }}
+          width={this._WIDTH}
+          height={this._HEIGHT}
+          ref={ref => {
+            this.canvas = ref;
+            if (colorByIsCatagoricalData) this.createOccupancyStack();
+            else this.createHistogram();
+          }}
+        />
+        <div key="text" style={{ fontFamily: "Roboto", fontSize: "14px" }}>
+          <p style={{ margin: "0" }}>
+            This histograms shows the distribution of{" "}
+            <strong>{colorAccessor}</strong> within{" "}
+            <strong>{category.categoryValues[categoryIndex]}</strong>.
+            <br />
+            <br />
+            The x axis is the same for each histogram, while the y axis is
+            scaled to the largest bin within this histogram instead of the
+            largest bin within the whole category.
+          </p>
+        </div>
+      </Popover>
     );
   }
 }

client/src/components/categorical/util.js

@@ -5,7 +5,6 @@
 // return sorted index
 
 import isNumber from "is-number";
-import _ from "lodash";
 
 const sortedCategoryValues = values => {
   /* this sort could be memoized for perf */
@@ -13,7 +12,7 @@ const sortedCategoryValues = values => {
   const strings = [];
   const ints = [];
 
-  _.forEach(values, v => {
+  values.forEach(v => {
     if (isNumber(v[0])) {
       ints.push(v);
     } else {

client/src/components/categorical/value.js

@@ -1,36 +1,80 @@
 // jshint esversion: 6
 import { connect } from "react-redux";
 import React from "react";
-import _ from "lodash";
 import Occupancy from "./occupancy";
-import { countCategoryValues2D } from "../../util/stateManager/worldUtil";
 import * as globals from "../../globals";
+import styles from "./categorical.css";
 
 @connect(state => ({
   categoricalSelection: state.categoricalSelection,
   colorScale: state.colors.scale,
   colorAccessor: state.colors.colorAccessor,
-  schema: _.get(state.world, "schema", null),
+  schema: state.world?.schema,
   world: state.world
 }))
 class CategoryValue extends React.Component {
-  toggleOff() {
+  toggleOff = () => {
     const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter deselect",
       metadataField,
       categoryIndex
     });
-  }
+  };
 
-  toggleOn() {
+  shouldComponentUpdate = nextProps => {
+    /* 
+    Checks to see if at least one of the following changed:
+    * world state
+    * the color accessor (what is currently being colored by)
+    * if this catagorical value's selection status has changed
+    
+    If and only if true, update the component
+    */
+    const { props } = this;
+    const { metadataField, categoryIndex, categoricalSelection } = props;
+    const { categoricalSelection: newCategoricalSelection } = nextProps;
+
+    const valueSelectionChange =
+      categoricalSelection[metadataField].categoryValueSelected[
+        categoryIndex
+      ] !==
+      newCategoricalSelection[metadataField].categoryValueSelected[
+        categoryIndex
+      ];
+
+    const worldChange = props.world !== nextProps.world;
+    const colorAccessorChange = props.colorAccessor !== nextProps.colorAccessor;
+
+    return valueSelectionChange || worldChange || colorAccessorChange;
+  };
+
+  toggleOn = () => {
     const { dispatch, metadataField, categoryIndex } = this.props;
     dispatch({
       type: "categorical metadata filter select",
       metadataField,
       categoryIndex
     });
-  }
+  };
+
+  handleMouseEnter = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "category value mouse hover start",
+      metadataField,
+      categoryIndex
+    });
+  };
+
+  handleMouseExit = () => {
+    const { dispatch, metadataField, categoryIndex } = this.props;
+    dispatch({
+      type: "category value mouse hover end",
+      metadataField,
+      categoryIndex
+    });
+  };
 
   render() {
     const {
@@ -40,15 +84,14 @@ class CategoryValue extends React.Component {
       colorAccessor,
       colorScale,
       i,
-      schema,
-      world
+      schema
     } = this.props;
 
     if (!categoricalSelection) return null;
 
     const category = categoricalSelection[metadataField];
-    const selected = category.categorySelected[categoryIndex];
-    const count = category.categoryCounts[categoryIndex];
+    const selected = category.categoryValueSelected[categoryIndex];
+    const count = category.categoryValueCounts[categoryIndex];
     const value = category.categoryValues[categoryIndex];
     const displayString = String(
       category.categoryValues[categoryIndex]
@@ -57,31 +100,26 @@ class CategoryValue extends React.Component {
     /* this is the color scale, so add swatches below */
     const isColorBy = metadataField === colorAccessor;
     let categories = null;
-    let occupancy = null;
 
     if (isColorBy && schema) {
-      categories = _.filter(schema.annotations.obs, {
-        name: colorAccessor
-      })[0].categories;
-    }
-
-    if (colorAccessor && !isColorBy && categoricalSelection[colorAccessor]) {
-      occupancy = countCategoryValues2D(
-        metadataField,
-        colorAccessor,
-        world.obsAnnotations
-      );
+      categories = schema.annotations.obsByName[colorAccessor]?.categories;
     }
 
     return (
       <div
         key={i}
+        className={styles.value}
+        data-testclass="categorical-row"
         style={{
+          padding: "4px 7px",
           display: "flex",
           alignItems: "baseline",
-          justifyContent: "space-between"
+          justifyContent: "space-between",
+          marginBottom: "2px",
+          borderRadius: "2px"
         }}
-        data-testclass="categorical-row"
+        onMouseEnter={this.handleMouseEnter}
+        onMouseLeave={this.handleMouseExit}
       >
         <div
           style={{
@@ -93,34 +131,32 @@ class CategoryValue extends React.Component {
             justifyContent: "space-between"
           }}
         >
-          <label className="bp3-control bp3-checkbox">
-            <input
-              onChange={
-                selected ? this.toggleOff.bind(this) : this.toggleOn.bind(this)
-              }
-              data-testclass="categorical-value-select"
-              data-testid={`categorical-value-select-${metadataField}-${displayString}`}
-              checked={selected}
-              type="checkbox"
-            />
-            <span className="bp3-control-indicator" />
+          <div style={{ display: "flex" }}>
+            <label className="bp3-control bp3-checkbox" style={{ margin: 0 }}>
+              <input
+                onChange={selected ? this.toggleOff : this.toggleOn}
+                data-testclass="categorical-value-select"
+                data-testid={`categorical-value-select-${metadataField}-${displayString}`}
+                checked={selected}
+                type="checkbox"
+              />
+              <span
+                className="bp3-control-indicator"
+                onMouseEnter={this.handleMouseExit}
+                onMouseLeave={this.handleMouseEnter}
+              />
+            </label>
             <span
               data-testid={`categorical-value-${metadataField}-${displayString}`}
               data-testclass="categorical-value"
+              style={{ wordBreak: "break-all" }}
             >
               {displayString}
             </span>
-          </label>
+          </div>
           <span style={{ flexShrink: 0 }}>
-            {colorAccessor &&
-            !isColorBy &&
-            categoricalSelection[colorAccessor] ? (
-              <Occupancy
-                occupancy={occupancy.get(
-                  category.categoryValues[categoryIndex]
-                )}
-                {...this.props}
-              />
+            {colorAccessor && !isColorBy ? (
+              <Occupancy category={category} {...this.props} />
             ) : null}
           </span>
         </div>

client/src/components/continuous/continuous.js

@@ -9,10 +9,10 @@ import * as globals from "../../globals";
 import HistogramBrush from "../brushableHistogram";
 
 @connect(state => ({
-  obsAnnotations: _.get(state.world, "obsAnnotations", null),
+  obsAnnotations: state.world?.obsAnnotations,
   colorAccessor: state.colors.colorAccessor,
   colorScale: state.colors.scale,
-  schema: _.get(state.world, "schema", null)
+  schema: state.world?.schema
 }))
 class Continuous extends React.Component {
   constructor(props) {
@@ -48,7 +48,7 @@ class Continuous extends React.Component {
     }
 
     /* initial value for iterator to simulate index, ranges is an object */
-    let zebra = -1;
+    let zebra = 0;
 
     return (
       <div>
@@ -66,11 +66,15 @@ class Continuous extends React.Component {
           ? _.map(obsAnnotations.colIndex.keys(), key => {
               const isColorField =
                 key.includes("color") || key.includes("Color");
-              if (key === "name" || isColorField) return null;
+              if (key === schema.annotations.obs.index || isColorField)
+                return null;
 
               const summary = obsAnnotations.col(key).summarize();
               const nonFiniteExtent =
-                summary.min === undefined || summary.max === undefined;
+                summary.min === undefined ||
+                summary.max === undefined ||
+                Number.isNaN(summary.min) ||
+                Number.isNaN(summary.max);
               if (!summary.categorical && !nonFiniteExtent) {
                 zebra += 1;
                 return (

client/src/components/continuousLegend/index.js

@@ -2,7 +2,7 @@
 import React from "react";
 import { connect } from "react-redux";
 import * as d3 from "d3";
-import { interpolateViridis, interpolateCool } from "d3-scale-chromatic";
+import { interpolateCool } from "d3-scale-chromatic";
 
 // create continuous color legend
 // http://bl.ocks.org/syntagmatic/e8ccca52559796be775553b467593a9f

client/src/components/framework/logo.js

@@ -0,0 +1,18 @@
+import React from "react";
+import * as globals from "../../globals";
+
+const Logo = props => {
+  const { size } = props;
+  return (
+    <svg width={size} height={size} viewBox="0 0 48 48" fill="none">
+      <rect width="48" height="48" fill="white" />
+      <rect width="48" height="48" fill={globals.logoColor} />
+      <rect x="19" y="19" width="22" height="22" fill="white" />
+      <rect x="24" y="24" width="12" height="12" fill={globals.logoColor} />
+      <rect x="7" y="19" width="7" height="22" fill="white" />
+      <rect x="19" y="7" width="22" height="7" fill="white" />
+    </svg>
+  );
+};
+
+export default Logo;

client/src/components/geneExpression/expressionButtons.js

@@ -1,101 +0,0 @@
-// jshint esversion: 6
-import React from "react";
-import _ from "lodash";
-import { Button, AnchorButton, Tooltip } from "@blueprintjs/core";
-import { connect } from "react-redux";
-import * as globals from "../../globals";
-import actions from "../../actions";
-import CellSetButton from "./cellSetButtons";
-
-@connect(state => ({
-  differential: state.differential,
-  world: state.world,
-  crossfilter: state.crossfilter
-}))
-class Expression extends React.Component {
-  constructor(props) {
-    super(props);
-    this.state = {};
-  }
-
-  computeDiffExp() {
-    const { dispatch, differential } = this.props;
-    if (differential.celllist1 && differential.celllist2) {
-      dispatch(
-        actions.requestDifferentialExpression(
-          differential.celllist1,
-          differential.celllist2
-        )
-      );
-    }
-  }
-
-  clearDifferentialExpression() {
-    const { dispatch, differential } = this.props;
-    dispatch({
-      type: "clear differential expression",
-      diffExp: differential.diffExp
-    });
-    dispatch({
-      type: "clear scatterplot"
-    });
-  }
-
-  render() {
-    const { differential } = this.props;
-    if (!differential) {
-      return null;
-    }
-    const haveBothCellSets =
-      !!differential.celllist1 && !!differential.celllist2;
-    return (
-      <div
-        style={{
-          marginRight: 10,
-          marginBottom: 10,
-          paddingLeft: globals.leftSidebarSectionPadding
-        }}
-      >
-        <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
-        <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
-        {!differential.diffExp ? (
-          <Tooltip
-            content="Add two cells selections, see the top 15 differentially expressed genes between them"
-            position="bottom"
-          >
-            <AnchorButton
-              style={{ marginTop: 10 }}
-              disabled={!haveBothCellSets}
-              intent="primary"
-              data-testid="diffexp-button"
-              loading={differential.loading}
-              fill
-              type="button"
-              onClick={this.computeDiffExp.bind(this)}
-            >
-              Compute Differential Expression
-            </AnchorButton>
-          </Tooltip>
-        ) : null}
-        {differential.diffExp ? (
-          <Tooltip
-            content="Remove differentially expressed gene list and clear cell selections"
-            position="bottom"
-          >
-            <Button
-              type="button"
-              fill
-              style={{ marginTop: 10 }}
-              intent="warning"
-              onClick={this.clearDifferentialExpression.bind(this)}
-            >
-              Clear Differential Expression
-            </Button>
-          </Tooltip>
-        ) : null}
-      </div>
-    );
-  }
-}
-
-export default Expression;

client/src/components/geneExpression/index.js

@@ -21,7 +21,6 @@ import {
   postUserErrorToast,
   keepAroundErrorToast
 } from "../framework/toasters";
-import ExpressionButtons from "./expressionButtons";
 
 const renderGene = (fuzzySortResult, { handleClick, modifiers, query }) => {
   if (!modifiers.matchesPredicate) {
@@ -57,7 +56,7 @@ const filterGenes = (query, genes) =>
 
 @connect(state => {
   return {
-    obsAnnotations: _.get(state.world, "obsAnnotations", null),
+    obsAnnotations: state.world?.obsAnnotations,
     userDefinedGenes: state.controls.userDefinedGenes,
     userDefinedGenesLoading: state.controls.userDefinedGenesLoading,
     world: state.world,
@@ -85,7 +84,8 @@ class GeneExpression extends React.Component {
     */
     const { world } = this.props;
     const { varAnnotations } = world;
-    const geneNames = varAnnotations.col("name").asArray();
+    const varIndexName = world.schema.annotations.var.index;
+    const geneNames = varAnnotations.col(varIndexName).asArray();
     if (geneNames.length > 0) {
       const placeholder = [];
       let len = geneNames.length;
@@ -107,6 +107,7 @@ class GeneExpression extends React.Component {
 
   handleClick(g) {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
     const gene = g.target;
     if (userDefinedGenes.indexOf(gene) !== -1) {
       postUserErrorToast("That gene already exists");
@@ -114,7 +115,9 @@ class GeneExpression extends React.Component {
       postUserErrorToast(
         "That's too many genes, you can have at most 15 user defined genes"
       );
-    } else if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
+    } else if (
+      world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+    ) {
       postUserErrorToast("That doesn't appear to be a valid gene name.");
     } else {
       dispatch({ type: "single user defined gene start" });
@@ -127,6 +130,7 @@ class GeneExpression extends React.Component {
 
   handleBulkAddClick() {
     const { world, dispatch, userDefinedGenes } = this.props;
+    const varIndexName = world.schema.annotations.var.index;
     const { bulkAdd } = this.state;
 
     /*
@@ -145,7 +149,9 @@ class GeneExpression extends React.Component {
           if (userDefinedGenes.indexOf(gene) !== -1) {
             return keepAroundErrorToast("That gene already exists");
           }
-          if (world.varAnnotations.col("name").indexOf(gene) === undefined) {
+          if (
+            world.varAnnotations.col(varIndexName).indexOf(gene) === undefined
+          ) {
             return keepAroundErrorToast(
               `${gene} doesn't appear to be a valid gene name.`
             );
@@ -168,24 +174,12 @@ class GeneExpression extends React.Component {
       userDefinedGenesLoading,
       differential
     } = this.props;
-
+    const varIndexName = world?.schema?.annotations?.var?.index;
     const { tab, bulkAdd } = this.state;
 
     return (
       <div>
-        <div
-          style={{
-            marginTop: 30
-          }}
-        >
-          <p
-            style={Object.assign({}, globals.leftSidebarSectionHeading, {
-              paddingLeft: globals.leftSidebarSectionPadding,
-              margin: 0
-            })}
-          >
-            Selected Genes
-          </p>
+        <div>
           <div
             style={{
               padding: globals.leftSidebarSectionPadding
@@ -201,7 +195,7 @@ class GeneExpression extends React.Component {
                 this.setState({ tab: "autosuggest" });
               }}
             >
-              Autosuggest
+              Autosuggest genes
             </Button>
             <Button
               active={tab === "bulkadd"}
@@ -243,7 +237,7 @@ class GeneExpression extends React.Component {
                 itemRenderer={renderGene.bind(this)}
                 items={
                   world && world.varAnnotations
-                    ? world.varAnnotations.col("name").asArray()
+                    ? world.varAnnotations.col(varIndexName).asArray()
                     : ["No genes"]
                 }
                 popoverProps={{ minimal: true }}
@@ -253,7 +247,7 @@ class GeneExpression extends React.Component {
                 data-testid={"add-gene"}
                 loading={userDefinedGenesLoading}
               >
-                Add
+                Add gene
               </Button>
             </ControlGroup>
           ) : null}
@@ -284,7 +278,7 @@ class GeneExpression extends React.Component {
                       onClick={this.handleBulkAddClick.bind(this)}
                       loading={userDefinedGenesLoading}
                     >
-                      Add
+                      Add genes
                     </Button>
                   </ControlGroup>
                 </FormGroup>
@@ -311,18 +305,9 @@ class GeneExpression extends React.Component {
             : null}
         </div>
         <div>
-          <p
-            style={Object.assign({}, globals.leftSidebarSectionHeading, {
-              marginTop: 40,
-              paddingLeft: globals.leftSidebarSectionPadding
-            })}
-          >
-            Differentially Expressed Genes
-          </p>
-          <ExpressionButtons />
           {differential.diffExp
             ? _.map(differential.diffExp, (value, index) => {
-                const name = world.varAnnotations.at(value[0], "name");
+                const name = world.varAnnotations.at(value[0], varIndexName);
                 const values = world.varData.col(name);
                 if (!values) {
                   return null;

client/src/components/graph/drawPointsRegl.js

@@ -1,5 +1,6 @@
 // jshint esversion: 6
 const mat4 = require("gl-mat4");
+const vec3 = require("gl-vec3");
 
 // opacity: https://github.com/spacetx/starfish/blob/master/viz/draw/regions.js
 
@@ -38,7 +39,20 @@ export default function(regl) {
     uniforms: {
       distance: regl.prop("distance"),
       view: regl.prop("view"),
-      projection: ({viewportWidth, viewportHeight}) => mat4.perspective([], Math.PI / 2, viewportWidth / viewportHeight, 0.01, 1000)
+      projection: ({ viewportWidth, viewportHeight }) => {
+        const aspectRatio = viewportWidth / viewportHeight;
+        let m = mat4.perspective(
+          [],
+          Math.PI / 2,
+          viewportWidth / viewportHeight,
+          0.01,
+          1000
+        );
+        if (aspectRatio < 1) {
+          m = mat4.scale(m, m, vec3.fromValues(1, 1, 1 / aspectRatio));
+        }
+        return m;
+      }
     },
 
     count: regl.prop("count"),

client/src/components/graph/setupCentroidSVG.js

@@ -0,0 +1,34 @@
+import * as d3 from "d3";
+import styles from "./graph.css";
+
+export default (responsive, graphPaddingRight, xy, text, colorBy) => {
+  const containerWidth = responsive.width - graphPaddingRight;
+
+  const svg = d3
+    .select("#graphAttachPoint")
+    .append("svg")
+    .attr("id", "centroid-container")
+    .attr("data-testid", "centroid-overlay")
+    .attr("width", containerWidth)
+    .attr("height", responsive.height)
+    .attr("class", `${styles.graphSVG}`)
+    .style("z-index", 998)
+    .style("pointer-events", "none");
+  //  TODO: Create own styles, ask Colin for an explanation on the css
+  // For now I'm going to put centroid z-index at 998 and lasso on 999
+
+  const label = svg
+    .append("g")
+    .attr("transform", `translate(${xy[0]}, ${xy[1]})`);
+
+  label
+    .append("text")
+    .attr("text-anchor", "middle")
+    .text(text)
+    .style("font-family", "Roboto Condensed")
+    .style("font-size", "18px")
+    .style("font-weight", "700")
+    .style("fill", colorBy ? "black" : "rgb(32, 178, 212)");
+
+  return svg;
+};

client/src/components/graph/setupSVGandBrush.js

@@ -16,15 +16,19 @@ export default (
   handleEndAction,
   handleCancelAction,
   responsive,
-  graphPaddingRight
+  graphPaddingRight,
+  graphInteractionMode
 ) => {
   const svg = d3
     .select("#graphAttachPoint")
     .append("svg")
+    .attr("id", "tool")
     .attr("data-testid", "layout-overlay")
     .attr("width", responsive.width - graphPaddingRight)
     .attr("height", responsive.height)
-    .attr("class", `${styles.graphSVG}`);
+    .attr("class", `${styles.graphSVG}`)
+    .style("z-index", 999)
+    .style("display", graphInteractionMode === "select" ? "inherit" : "none");
 
   if (selectionToolType === "brush") {
     const brush = d3

client/src/components/leftSidebar/index.js

@@ -1,16 +1,15 @@
 // jshint esversion: 6
-import _ from "lodash";
 import React from "react";
 import { connect } from "react-redux";
-import Categorical from "./categorical/categorical";
-import Continuous from "./continuous/continuous";
-import GeneExpression from "./geneExpression";
-import * as globals from "../globals";
-import DynamicScatterplot from "./scatterplot/scatterplot";
+import Categorical from "../categorical/categorical";
+import Continuous from "../continuous/continuous";
+import GeneExpression from "../geneExpression";
+import * as globals from "../../globals";
+import DynamicScatterplot from "../scatterplot/scatterplot";
+import TopLeftLogoAndTitle from "./topLeftLogoAndTitle";
 
 @connect(state => ({
   responsive: state.responsive,
-  datasetTitle: _.get(state.config, "displayNames.dataset"),
   scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
   scatterplotYYaccessor: state.controls.scatterplotYYaccessor
 }))
@@ -18,7 +17,6 @@ class LeftSideBar extends React.Component {
   render() {
     const {
       responsive,
-      datasetTitle,
       scatterplotXXaccessor,
       scatterplotYYaccessor
     } = this.props;
@@ -27,8 +25,7 @@ class LeftSideBar extends React.Component {
     this magic number should be made less fragile,
     if cellxgene logo or tabs change, this must as well
     */
-    const metadataSectionPadding = 0;
-    // scatterplotXXaccessor && scatterplotYYaccessor ? 450 : 0;
+    const logoRelatedPadding = 50;
 
     return (
       <div
@@ -36,27 +33,14 @@ class LeftSideBar extends React.Component {
           position: "fixed",
           backgroundColor: "white",
           /* x y blur spread color */
-          boxShadow: "1px 0px 6px 2px rgba(153,153,153,0.4)"
+          boxShadow: "-3px 0px 6px 2px rgba(153,153,153,0.4)"
         }}
       >
-        <p
-          data-testid="header"
-          style={{
-            position: "fixed",
-            top: globals.cellxgeneTitleTopPadding,
-            left: globals.leftSidebarWidth + globals.cellxgeneTitleLeftPadding,
-            margin: 0,
-            fontSize: globals.largestFontSize,
-            color: globals.darkerGrey,
-            width: "100%"
-          }}
-        >
-          cellxgene: {datasetTitle}
-        </p>
-
+        <TopLeftLogoAndTitle />
         <div
           style={{
-            height: responsive.height - metadataSectionPadding,
+            height: responsive.height - logoRelatedPadding,
+            marginTop: logoRelatedPadding,
             width: globals.leftSidebarWidth,
             overflowY: "auto",
             overflowX: "hidden"

client/src/components/leftSidebar/topLeftLogoAndTitle.js

@@ -0,0 +1,71 @@
+// jshint esversion: 6
+import React from "react";
+import { connect } from "react-redux";
+import * as globals from "../../globals";
+import Logo from "../framework/logo";
+
+@connect(state => ({
+  responsive: state.responsive,
+  datasetTitle: state.config?.displayNames?.dataset ?? "",
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor
+}))
+class LeftSideBar extends React.Component {
+  render() {
+    const { datasetTitle } = this.props;
+
+    const paddingToAvoidScrollBar = 15;
+
+    return (
+      <div
+        style={{
+          paddingLeft: 8,
+          paddingTop: 8,
+          width: globals.leftSidebarWidth - paddingToAvoidScrollBar,
+          position: "absolute",
+          backgroundColor: "white",
+          zIndex: 8888
+          /* x y blur spread color */
+          // boxShadow: "-5px -1px 4px 2px rgba(225,225,225,0.4)"
+        }}
+      >
+        <Logo size={30} />
+        <span
+          style={{
+            fontSize: 28,
+            position: "relative",
+            top: -6,
+            fontWeight: "bold",
+            marginLeft: 5,
+            color: globals.logoColor,
+            userSelect: "none"
+          }}
+        >
+          cell<span
+            style={{
+              position: "relative",
+              top: 1,
+              fontWeight: 300,
+              fontSize: 24
+            }}
+          >
+            ×
+          </span>gene
+        </span>
+        <span
+          data-testid="header"
+          style={{
+            fontSize: 14,
+            position: "relative",
+            marginLeft: 7,
+            top: -8
+          }}
+        >
+          {datasetTitle}
+        </span>
+      </div>
+    );
+  }
+}
+
+export default LeftSideBar;

client/src/components/menubar/cellSetButtons.js

@@ -32,7 +32,7 @@ class CellSetButton extends React.Component {
   render() {
     const { differential, eitherCellSetOneOrTwo } = this.props;
     const cellListName = `celllist${eitherCellSetOneOrTwo}`;
-    let cells_selected = differential[cellListName]
+    const cellsSelected = differential[cellListName]
       ? differential[cellListName].length
       : 0;
     return (
@@ -41,7 +41,6 @@ class CellSetButton extends React.Component {
         position="top"
       >
         <AnchorButton
-          style={{ marginRight: 10 }}
           type="button"
           disabled={differential.diffExp}
           onClick={this.set.bind(this)}
@@ -50,7 +49,7 @@ class CellSetButton extends React.Component {
           {eitherCellSetOneOrTwo}
           {": "}
           <span data-testid={`cellset-count-${eitherCellSetOneOrTwo}`}>
-            {cells_selected}
+            {cellsSelected}
           </span>
           {" cells"}
         </AnchorButton>

client/src/components/menubar/clip.js

@@ -0,0 +1,130 @@
+// jshint esversion: 6
+import React from "react";
+import {
+  Position,
+  Button,
+  Popover,
+  NumericInput,
+  Icon
+} from "@blueprintjs/core";
+
+function Clip(props) {
+  const {
+    pendingClipPercentiles,
+    clipPercentileMin,
+    clipPercentileMax,
+    handleClipOpening,
+    handleClipClosing,
+    handleClipCommit,
+    isClipDisabled,
+    handleClipOnKeyPress,
+    handleClipPercentileMaxValueChange,
+    handleClipPercentileMinValueChange
+  } = props;
+
+  const clipMin =
+    pendingClipPercentiles?.clipPercentileMin ?? clipPercentileMin;
+  const clipMax =
+    pendingClipPercentiles?.clipPercentileMax ?? clipPercentileMax;
+  const activeClipClass =
+    clipPercentileMin > 0 || clipPercentileMax < 100
+      ? " bp3-intent-warning"
+      : "";
+
+  return (
+    <div
+      className="bp3-button-group"
+      style={{
+        marginLeft: 10
+      }}
+    >
+      <Popover
+        target={
+          <Button
+            type="button"
+            data-testid="visualization-settings"
+            className={`bp3-button bp3-icon-timeline-bar-chart ${activeClipClass}`}
+            style={{
+              cursor: "pointer"
+            }}
+          />
+        }
+        position={Position.BOTTOM_RIGHT}
+        onOpening={handleClipOpening}
+        onClosing={handleClipClosing}
+        content={
+          <div
+            style={{
+              display: "flex",
+              justifyContent: "flex-start",
+              alignItems: "flex-start",
+              flexDirection: "column",
+              padding: 10
+            }}
+          >
+            <div>Clip all continuous values to percentile range</div>
+            <div
+              style={{
+                display: "flex",
+                justifyContent: "space-between",
+                alignItems: "center",
+                paddingTop: 5,
+                paddingBottom: 5
+              }}
+            >
+              <NumericInput
+                style={{ width: 50 }}
+                data-testid="clip-min-input"
+                onValueChange={handleClipPercentileMinValueChange}
+                onKeyPress={handleClipOnKeyPress}
+                value={clipMin}
+                min={0}
+                max={100}
+                fill={false}
+                minorStepSize={null}
+                rightElement={
+                  <div style={{ padding: "4px 2px" }}>
+                    <Icon icon="percentage" intent="primary" iconSize={14} />
+                  </div>
+                }
+              />
+              <span style={{ marginRight: 5, marginLeft: 5 }}> - </span>
+              <NumericInput
+                style={{ width: 50 }}
+                data-testid="clip-max-input"
+                onValueChange={handleClipPercentileMaxValueChange}
+                onKeyPress={handleClipOnKeyPress}
+                value={clipMax}
+                min={0}
+                max={100}
+                fill={false}
+                minorStepSize={null}
+                rightElement={
+                  <div style={{ padding: "4px 2px" }}>
+                    <Icon icon="percentage" intent="primary" iconSize={14} />
+                  </div>
+                }
+              />
+              <Button
+                type="button"
+                data-testid="clip-commit"
+                className="bp3-button"
+                disabled={isClipDisabled()}
+                style={{
+                  cursor: "pointer",
+                  marginRight: 5,
+                  marginLeft: 5
+                }}
+                onClick={handleClipCommit}
+              >
+                Clip
+              </Button>
+            </div>
+          </div>
+        }
+      />
+    </div>
+  );
+}
+
+export default Clip;

client/src/components/menubar/index.js

@@ -0,0 +1,464 @@
+// jshint esversion: 6
+import React from "react";
+import { connect } from "react-redux";
+import {
+  Button,
+  AnchorButton,
+  Tooltip,
+  Popover,
+  Position,
+  RadioGroup,
+  Radio
+} from "@blueprintjs/core";
+import { World } from "../../util/stateManager";
+import actions from "../../actions";
+import CellSetButton from "./cellSetButtons";
+import InformationMenu from "./infoMenu";
+import UndoRedoReset from "./undoRedoReset";
+import Clip from "./clip";
+
+@connect(state => ({
+  universe: state.universe,
+  world: state.world,
+  loading: state.controls.loading,
+  crossfilter: state.crossfilter,
+  differential: state.differential,
+  resettingInterface: state.controls.resettingInterface,
+  layoutChoice: state.layoutChoice,
+  graphInteractionMode: state.controls.graphInteractionMode,
+  clipPercentileMin: Math.round(100 * (state.world?.clipQuantiles?.min ?? 0)),
+  clipPercentileMax: Math.round(100 * (state.world?.clipQuantiles?.max ?? 1)),
+  userDefinedGenes: state.controls.userDefinedGenes,
+  diffexpGenes: state.controls.diffexpGenes,
+  colorAccessor: state.colors.colorAccessor,
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+  celllist1: state.differential.celllist1,
+  celllist2: state.differential.celllist2,
+  libraryVersions: state.config?.library_versions, // eslint-disable-line camelcase
+  undoDisabled: state["@@undoable/past"].length === 0,
+  redoDisabled: state["@@undoable/future"].length === 0
+}))
+class MenuBar extends React.Component {
+  static isValidDigitKeyEvent(e) {
+    /*
+    Return true if this event is necessary to enter a percent number input.
+    Return false if not.
+
+    Returns true for events with keys: backspace, control, alt, meta, [0-9],
+    or events that don't have a key.
+    */
+    if (e.key === null) return true;
+    if (e.ctrlKey || e.altKey || e.metaKey) return true;
+
+    // concept borrowed from blueprint's numericInputUtils:
+    // keys that print a single character when pressed have a `key` name of
+    // length 1. every other key has a longer `key` name (e.g. "Backspace",
+    // "ArrowUp", "Shift"). since none of those keys can print a character
+    // to the field--and since they may have important native behaviors
+    // beyond printing a character--we don't want to disable their effects.
+    const isSingleCharKey = e.key.length === 1;
+    if (!isSingleCharKey) return true;
+
+    const key = e.key.charCodeAt(0) - 48; /* "0" */
+    return key >= 0 && key <= 9;
+  }
+
+  constructor(props) {
+    super(props);
+    this.state = {
+      pendingClipPercentiles: null
+    };
+  }
+
+  isClipDisabled = () => {
+    /*
+    return true if clip button should be disabled.
+    */
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMin = pendingClipPercentiles?.clipPercentileMin;
+    const clipPercentileMax = pendingClipPercentiles?.clipPercentileMax;
+
+    const { world } = this.props;
+    const currentClipMin = 100 * world?.clipQuantiles?.min;
+    const currentClipMax = 100 * world?.clipQuantiles?.max;
+
+    // if you change this test, be careful with logic around
+    // comparisons between undefined / NaN handling.
+    const isDisabled =
+      !(clipPercentileMin < clipPercentileMax) ||
+      (clipPercentileMin === currentClipMin &&
+        clipPercentileMax === currentClipMax);
+
+    return isDisabled;
+  };
+
+  isResetDisabled = () => {
+    /*
+    Reset should be disabled when all of the following are true:
+      * nothing is selected in the crossfilter
+      * world EQ universe
+      * nothing is colored by
+      * there are no userDefinedGenes or diffexpGenes displayed
+      * scatterplot is not displayed
+      * nothing in cellset1 or cellset2
+      * clip percentiles are [0,100]
+    */
+    const {
+      crossfilter,
+      world,
+      universe,
+      userDefinedGenes,
+      diffexpGenes,
+      colorAccessor,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor,
+      celllist1,
+      celllist2,
+      clipPercentileMin,
+      clipPercentileMax
+    } = this.props;
+
+    if (!crossfilter || !world || !universe) {
+      return false;
+    }
+    const nothingSelected = crossfilter.countSelected() === crossfilter.size();
+    const nothingColoredBy = !colorAccessor;
+    const noGenes = userDefinedGenes.length === 0 && diffexpGenes.length === 0;
+    const scatterNotDpl = !scatterplotXXaccessor || !scatterplotYYaccessor;
+    const nothingInCellsets = !celllist1 && !celllist2;
+
+    return (
+      nothingSelected &&
+      World.worldEqUniverse(world, universe) &&
+      nothingColoredBy &&
+      noGenes &&
+      scatterNotDpl &&
+      nothingInCellsets &&
+      clipPercentileMax === 100 &&
+      clipPercentileMin === 0
+    );
+  };
+
+  resetInterface = () => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "interface reset started"
+    });
+    dispatch(actions.resetInterface());
+  };
+
+  handleClipOnKeyPress = e => {
+    /*
+    allow only numbers, plus other critical keys which
+    may be required to make a number
+    */
+    if (!MenuBar.isValidDigitKeyEvent(e)) {
+      e.preventDefault();
+    }
+  };
+
+  handleClipPercentileMinValueChange = v => {
+    /*
+    Ignore anything that isn't a legit number
+    */
+    if (!Number.isFinite(v)) return;
+
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMax = pendingClipPercentiles?.clipPercentileMax;
+
+    /*
+    clamp to [0, currentClipPercentileMax]
+    */
+    if (v <= 0) v = 0;
+    if (v > 100) v = 100;
+    const clipPercentileMin = Math.round(v); // paranoia
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipPercentileMaxValueChange = v => {
+    /*
+    Ignore anything that isn't a legit number
+    */
+    if (!Number.isFinite(v)) return;
+
+    const { pendingClipPercentiles } = this.state;
+    const clipPercentileMin = pendingClipPercentiles?.clipPercentileMin;
+
+    /*
+    clamp to [0, 100]
+    */
+    if (v < 0) v = 0;
+    if (v > 100) v = 100;
+    const clipPercentileMax = Math.round(v); // paranoia
+
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipCommit = () => {
+    const { dispatch } = this.props;
+    const { pendingClipPercentiles } = this.state;
+    const { clipPercentileMin, clipPercentileMax } = pendingClipPercentiles;
+    const min = clipPercentileMin / 100;
+    const max = clipPercentileMax / 100;
+    dispatch({
+      type: "set clip quantiles",
+      clipQuantiles: { min, max }
+    });
+  };
+
+  handleClipOpening = () => {
+    const { clipPercentileMin, clipPercentileMax } = this.props;
+    this.setState({
+      pendingClipPercentiles: { clipPercentileMin, clipPercentileMax }
+    });
+  };
+
+  handleClipClosing = () => {
+    this.setState({ pendingClipPercentiles: null });
+  };
+
+  handleLayoutChoiceChange = e => {
+    const { dispatch } = this.props;
+    dispatch({
+      type: "set layout choice",
+      layoutChoice: e.currentTarget.value
+    });
+  };
+
+  computeDiffExp = () => {
+    const { dispatch, differential } = this.props;
+    if (differential.celllist1 && differential.celllist2) {
+      dispatch(
+        actions.requestDifferentialExpression(
+          differential.celllist1,
+          differential.celllist2
+        )
+      );
+    }
+  };
+
+  clearDifferentialExpression = () => {
+    const { dispatch, differential } = this.props;
+    dispatch({
+      type: "clear differential expression",
+      diffExp: differential.diffExp
+    });
+    dispatch({
+      type: "clear scatterplot"
+    });
+  };
+
+  render() {
+    const {
+      dispatch,
+      differential,
+      crossfilter,
+      resettingInterface,
+      libraryVersions,
+      undoDisabled,
+      redoDisabled,
+      selectionTool,
+      clipPercentileMin,
+      clipPercentileMax,
+      layoutChoice,
+      graphInteractionMode
+    } = this.props;
+    const { pendingClipPercentiles } = this.state;
+
+    const haveBothCellSets =
+      !!differential.celllist1 && !!differential.celllist2;
+
+    // constants used to create selection tool button
+    let selectionTooltip;
+    let selectionButtonClass;
+    if (selectionTool === "brush") {
+      selectionTooltip = "Brush selection";
+      selectionButtonClass = "bp3-icon-select";
+    } else {
+      selectionTooltip = "Lasso selection";
+      selectionButtonClass = "bp3-icon-polygon-filter";
+    }
+
+    return (
+      <div
+        style={{
+          position: "fixed",
+          right: 8,
+          top: 8
+        }}
+      >
+        <div className="bp3-button-group" style={{ marginRight: 10 }}>
+          <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
+          <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
+          {!differential.diffExp ? (
+            <Tooltip
+              content="Add two cells selections, see the top 15 differentially expressed genes between them"
+              position="bottom"
+            >
+              <AnchorButton
+                disabled={!haveBothCellSets}
+                intent="primary"
+                data-testid="diffexp-button"
+                loading={differential.loading}
+                icon="left-join"
+                fill
+                type="button"
+                onClick={this.computeDiffExp}
+              >
+                Compute Differential Expression
+              </AnchorButton>
+            </Tooltip>
+          ) : null}
+
+          {differential.diffExp ? (
+            <Tooltip
+              content="Remove differentially expressed gene list and clear cell selections"
+              position="bottom"
+            >
+              <Button
+                type="button"
+                fill
+                intent="warning"
+                onClick={this.clearDifferentialExpression}
+              >
+                Clear Differential Expression
+              </Button>
+            </Tooltip>
+          ) : null}
+        </div>
+        <Tooltip
+          content="Show only metadata and cells which are currently selected"
+          position="left"
+        >
+          <AnchorButton
+            type="button"
+            data-testid="subset-button"
+            disabled={
+              crossfilter &&
+              (crossfilter.countSelected() === 0 ||
+                crossfilter.countSelected() === crossfilter.size())
+            }
+            style={{
+              marginRight: 10
+            }}
+            onClick={() => {
+              dispatch(actions.regraph());
+              dispatch({ type: "increment graph render counter" });
+            }}
+          >
+            subset to current selection
+          </AnchorButton>
+        </Tooltip>
+        <div className="bp3-button-group">
+          <Tooltip content={selectionTooltip} position="left">
+            <Button
+              type="button"
+              data-testid="mode-lasso"
+              className={`bp3-button ${selectionButtonClass}`}
+              active={graphInteractionMode === "select"}
+              onClick={() => {
+                dispatch({
+                  type: "change graph interaction mode",
+                  data: "select"
+                });
+              }}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
+          <Tooltip content="Pan and zoom" position="left">
+            <Button
+              type="button"
+              data-testid="mode-pan-zoom"
+              className="bp3-button bp3-icon-zoom-in"
+              active={graphInteractionMode === "zoom"}
+              onClick={() => {
+                dispatch({
+                  type: "change graph interaction mode",
+                  data: "zoom"
+                });
+              }}
+              style={{
+                cursor: "pointer"
+              }}
+            />
+          </Tooltip>
+        </div>
+        <div
+          className="bp3-button-group"
+          style={{
+            marginLeft: 10
+          }}
+        >
+          <Popover
+            target={
+              <Button
+                type="button"
+                data-testid="layout-choice"
+                className="bp3-button bp3-icon-heatmap"
+                style={{
+                  cursor: "pointer"
+                }}
+              />
+            }
+            position={Position.BOTTOM_RIGHT}
+            content={
+              <div
+                style={{
+                  display: "flex",
+                  justifyContent: "flex-start",
+                  alignItems: "flex-start",
+                  flexDirection: "column",
+                  padding: 10
+                }}
+              >
+                <RadioGroup
+                  label="Layout Choice"
+                  onChange={this.handleLayoutChoiceChange}
+                  selectedValue={layoutChoice.current}
+                >
+                  {layoutChoice.available.map(name => (
+                    <Radio label={name} value={name} key={name} />
+                  ))}
+                </RadioGroup>
+              </div>
+            }
+          />
+        </div>
+        <Clip
+          pendingClipPercentiles={pendingClipPercentiles}
+          clipPercentileMin={clipPercentileMin}
+          clipPercentileMax={clipPercentileMax}
+          handleClipOpening={this.handleClipOpening}
+          handleClipClosing={this.handleClipClosing}
+          handleClipCommit={this.handleClipCommit}
+          isClipDisabled={this.isClipDisabled}
+          handleClipOnKeyPress={this.handleClipOnKeyPress}
+          handleClipPercentileMaxValueChange={
+            this.handleClipPercentileMaxValueChange
+          }
+          handleClipPercentileMinValueChange={
+            this.handleClipPercentileMinValueChange
+          }
+        />
+        <UndoRedoReset
+          dispatch={dispatch}
+          isResetDisabled={this.isResetDisabled}
+          resetInterface={this.resetInterface}
+          resettingInterface={resettingInterface}
+          undoDisabled={undoDisabled}
+          redoDisabled={redoDisabled}
+        />
+        <InformationMenu libraryVersions={libraryVersions} />
+      </div>
+    );
+  }
+}
+
+export default MenuBar;

client/src/components/menubar/infoMenu.js

@@ -0,0 +1,61 @@
+// jshint esversion: 6
+import React from "react";
+import { Button, Popover, Menu, MenuItem, Position } from "@blueprintjs/core";
+
+function InformationMenu(props) {
+  const { libraryVersions } = props;
+  return (
+    <div style={{ marginLeft: 10 }} className="bp3-button-group">
+      <Popover
+        content={
+          <Menu>
+            <MenuItem
+              href="https://chanzuckerberg.github.io/cellxgene/faq.html"
+              target="_blank"
+              icon="help"
+              text="FAQ"
+            />
+            <MenuItem
+              href="https://join-cellxgene-users.herokuapp.com/"
+              target="_blank"
+              icon="chat"
+              text="Chat"
+            />
+            <MenuItem
+              href="https://chanzuckerberg.github.io/cellxgene/"
+              target="_blank"
+              icon="book"
+              text="Docs"
+            />
+            <MenuItem
+              href="https://github.com/chanzuckerberg/cellxgene"
+              target="_blank"
+              icon="git-branch"
+              text="Github"
+            />
+            <MenuItem
+              target="_blank"
+              text={`cellxgene v${
+                libraryVersions && libraryVersions.cellxgene
+                  ? libraryVersions.cellxgene
+                  : null
+              }`}
+            />
+            <MenuItem text="MIT License" />
+          </Menu>
+        }
+        position={Position.BOTTOM_RIGHT}
+      >
+        <Button
+          type="button"
+          className="bp3-button bp3-icon-info-sign"
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Popover>
+    </div>
+  );
+}
+
+export default InformationMenu;

client/src/components/menubar/undoRedoReset.js

@@ -0,0 +1,64 @@
+// jshint esversion: 6
+import React from "react";
+import { AnchorButton, Tooltip } from "@blueprintjs/core";
+
+function InformationMenu(props) {
+  const {
+    resettingInterface,
+    undoDisabled,
+    redoDisabled,
+    resetInterface,
+    isResetDisabled,
+    dispatch
+  } = props;
+  return (
+    <div style={{ marginLeft: 10 }} className="bp3-button-group">
+      <Tooltip content="Undo" position="left">
+        <AnchorButton
+          type="button"
+          className="bp3-button bp3-icon-undo"
+          disabled={undoDisabled}
+          onClick={() => {
+            dispatch({ type: "@@undoable/undo" });
+          }}
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Tooltip>
+      <Tooltip content="Redo" position="left">
+        <AnchorButton
+          type="button"
+          className="bp3-button bp3-icon-redo"
+          disabled={redoDisabled}
+          onClick={() => {
+            dispatch({ type: "@@undoable/redo" });
+          }}
+          style={{
+            cursor: "pointer"
+          }}
+        />
+      </Tooltip>
+      <Tooltip
+        content="Reset cellxgene, clearing all selections"
+        position="left"
+      >
+        <AnchorButton
+          disabled={isResetDisabled()}
+          style={{ marginLeft: 10 }}
+          type="button"
+          loading={resettingInterface}
+          intent="none"
+          icon="refresh"
+          onClick={resetInterface}
+          data-testid="reset"
+          data-testclass={`resetting-${resettingInterface}`}
+        >
+          reset
+        </AnchorButton>
+      </Tooltip>
+    </div>
+  );
+}
+
+export default InformationMenu;

client/src/components/scatterplot/scatterplot.js

@@ -3,7 +3,6 @@
 // https://peterbeshai.com/scatterplot-in-d3-with-voronoi-interaction.html
 
 import React from "react";
-import _ from "lodash";
 import { connect } from "react-redux";
 import { Button, ButtonGroup } from "@blueprintjs/core";
 import _regl from "regl";

client/src/globals.js

@@ -31,6 +31,7 @@ export const darkGreen = "#448C4D";
 
 export const nonFiniteCellColor = lightGrey;
 export const defaultCellColor = "rgb(0,0,0,1)";
+export const logoColor = "black"; /* logo pink: "#E9429A" */
 
 /* typography constants */
 
@@ -72,114 +73,3 @@ let _API = {
 
 if (window.CELLXGENE && window.CELLXGENE.API) _API = window.CELLXGENE.API;
 export const API = _API;
-
-export const ordinalColors = [
-  "#0ac115",
-  "#c10ab6",
-  "#c1710a",
-  "#0a5ac1",
-  "#c1150a",
-  "#0ab6c1",
-  "#5ac10a",
-  "#710ac1",
-  "#0ac171",
-  "#c10a5a",
-  "#b6c10a",
-  "#150ac1",
-  "#b2ffb7",
-  "#ffb2fa",
-  "#ffddb2",
-  "#b2d4ff",
-  "#ffb7b2",
-  "#b2faff",
-  "#d4ffb2",
-  "#ddb2ff",
-  "#b2ffdd",
-  "#ffb2d4",
-  "#faffb2",
-  "#b7b2ff",
-  "#27a908",
-  "#8b08a9",
-  "#a93a08",
-  "#0877a9",
-  "#a90827",
-  "#08a98b",
-  "#77a908",
-  "#3a08a9",
-  "#08a93a",
-  "#a90877",
-  "#a98b08",
-  "#0827a9",
-  "#00ff0f",
-  "#ff00ef",
-  "#ff8e00",
-  "#0070ff",
-  "#ff0f00",
-  "#00efff",
-  "#70ff00",
-  "#8e00ff",
-  "#00ff8e",
-  "#ff0070",
-  "#efff00",
-  "#0f00ff",
-  "#006606",
-  "#66005f",
-  "#663900",
-  "#002c66",
-  "#660600",
-  "#005f66",
-  "#2c6600",
-  "#390066",
-  "#006639",
-  "#66002c",
-  "#5f6600",
-  "#060066",
-  "#83ff65",
-  "#e165ff",
-  "#ff9565",
-  "#65cfff",
-  "#ff6583",
-  "#65ffe1",
-  "#cfff65",
-  "#9565ff",
-  "#65ff95",
-  "#ff65cf",
-  "#ffe165",
-  "#6583ff",
-  "#009909",
-  "#99008f",
-  "#995500",
-  "#004399",
-  "#990900",
-  "#008f99",
-  "#439900",
-  "#550099",
-  "#009955",
-  "#990043",
-  "#8f9900",
-  "#090099",
-  "#d9fecc",
-  "#f1ccfe",
-  "#fed7cc",
-  "#ccf3fe",
-  "#feccd9",
-  "#ccfef1",
-  "#f3fecc",
-  "#d7ccfe",
-  "#ccfed7",
-  "#feccf3",
-  "#fef1cc",
-  "#ccd9fe",
-  "#47ea51",
-  "#ea47e0",
-  "#eaa247",
-  "#478fea",
-  "#ea5147",
-  "#47e0ea",
-  "#8fea47",
-  "#a247ea",
-  "#47eaa2",
-  "#ea478f",
-  "#e0ea47",
-  "#5147ea"
-];

client/src/reducers/categoricalSelection.js

@@ -1,12 +1,9 @@
-import _ from "lodash";
-
 import { ControlsHelpers } from "../util/stateManager";
 import * as globals from "../globals";
 
 function maxCategoryItems(state) {
-  return _.get(
-    state.config,
-    "parameters.max-category-items",
+  return (
+    state.config.parameters?.["max-category-items"] ??
     globals.configDefaults.parameters["max-category-items"]
   );
 }
@@ -33,15 +30,15 @@ const CategoricalSelection = (
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
-        state[action.metadataField].categorySelected
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = true;
+      newCategoryValueSelected[action.categoryIndex] = true;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -51,15 +48,15 @@ const CategoricalSelection = (
       /*
       Set the specific category in this field to false
       */
-      const newCategorySelected = Array.from(
-        state[action.metadataField].categorySelected
+      const newCategoryValueSelected = Array.from(
+        state[action.metadataField].categoryValueSelected
       );
-      newCategorySelected[action.categoryIndex] = false;
+      newCategoryValueSelected[action.categoryIndex] = false;
       const newCategoricalSelection = {
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: newCategorySelected
+          categoryValueSelected: newCategoryValueSelected
         }
       };
       return newCategoricalSelection;
@@ -73,8 +70,9 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
-            state[action.metadataField].categorySelected
+          categorySelected: false,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(false)
         }
       };
@@ -89,8 +87,9 @@ const CategoricalSelection = (
         ...state,
         [action.metadataField]: {
           ...state[action.metadataField],
-          categorySelected: Array.from(
-            state[action.metadataField].categorySelected
+          categorySelected: true,
+          categoryValueSelected: Array.from(
+            state[action.metadataField].categoryValueSelected
           ).fill(true)
         }
       };

client/src/reducers/centroidLabel.js

@@ -0,0 +1,44 @@
+import calcCentroid from "../util/centroid";
+
+const initialState = {
+  metadataField: "",
+  categoryIndex: -1,
+  categoryField: "",
+  centroidXY: [-1, -1]
+};
+
+const CentroidLabel = (state = initialState, action, sharedNextState) => {
+  const { categoricalSelection, world, layoutChoice } = sharedNextState;
+  const { metadataField, categoryIndex } = action;
+  const categoryField =
+    categoricalSelection?.[metadataField]?.categoryValues[categoryIndex];
+  switch (action.type) {
+    case "category value mouse hover start":
+      return {
+        ...state,
+        metadataField,
+        categoryIndex,
+        categoryField,
+        centroidXY: null /* calcCentroid(  This function call is computationally heavy and also leading to large GC. Before reimplementation, look into optimization and memoization
+          world,
+          metadataField,
+          categoryField,
+          layoutChoice.currentDimNames
+        ) */
+      };
+
+    case "category value mouse hover end":
+      if (
+        metadataField === state.metadataField &&
+        categoryIndex === state.categoryIndex
+      ) {
+        return initialState;
+      }
+      return state;
+
+    default:
+      return state;
+  }
+};
+
+export default CentroidLabel;

client/src/reducers/colors.js

@@ -1,4 +1,4 @@
-import { createColors } from "../util/stateManager";
+import { ColorHelpers } from "../util/stateManager";
 
 const ColorsReducer = (
   state = {
@@ -17,7 +17,7 @@ const ColorsReducer = (
       const { world } = nextSharedState;
       const colorMode = null;
       const colorAccessor = null;
-      const { rgb, scale } = createColors(world, colorMode);
+      const { rgb, scale } = ColorHelpers.createColors(world, colorMode);
       return {
         ...state,
         colorAccessor,
@@ -29,9 +29,23 @@ const ColorsReducer = (
 
     case "set clip quantiles":
     case "set World to current selection": {
+      const { world: prevWorld, controls: prevControls } = prevSharedState;
+      const resetColorState = ColorHelpers.checkIfColorByDiffexpAndResetColors(
+        prevControls,
+        state,
+        prevWorld
+      );
+      if (resetColorState) {
+        return resetColorState;
+      }
+
       const { colorMode, colorAccessor } = state;
       const { world } = nextSharedState;
-      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
         rgb,
@@ -40,14 +54,9 @@ const ColorsReducer = (
     }
 
     case "reset colorscale": {
-      const { world } = prevSharedState;
-      const { rgb, scale } = createColors(world);
       return {
         ...state,
-        colorMode: null,
-        colorAccessor: null,
-        rgb,
-        scale
+        ...ColorHelpers.resetColors(prevSharedState.world)
       };
     }
 
@@ -62,7 +71,11 @@ const ColorsReducer = (
       const colorMode = !resetCurrent ? action.type : null;
       const colorAccessor = !resetCurrent ? action.colorAccessor : null;
 
-      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
         colorMode,
@@ -81,7 +94,11 @@ const ColorsReducer = (
       const colorMode = !resetCurrent ? action.type : null;
       const colorAccessor = !resetCurrent ? action.gene : null;
 
-      const { rgb, scale } = createColors(world, colorMode, colorAccessor);
+      const { rgb, scale } = ColorHelpers.createColors(
+        world,
+        colorMode,
+        colorAccessor
+      );
       return {
         ...state,
         colorMode,
@@ -91,6 +108,19 @@ const ColorsReducer = (
       };
     }
 
+    case "clear differential expression": {
+      const { world: prevWorld, controls: prevControls } = prevSharedState;
+      const resetColorState = ColorHelpers.checkIfColorByDiffexpAndResetColors(
+        prevControls,
+        state,
+        prevWorld
+      );
+      if (resetColorState) {
+        return resetColorState;
+      }
+      return state;
+    }
+
     default: {
       return state;
     }

client/src/reducers/controls.js

@@ -16,7 +16,7 @@ const Controls = (
     diffexpGenes: [],
 
     resettingInterface: false,
-
+    graphInteractionMode: "select",
     opacityForDeselectedCells: 0.2,
     scatterplotXXaccessor: null, // just easier to read
     scatterplotYYaccessor: null,
@@ -93,9 +93,10 @@ const Controls = (
     }
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const _diffexpGenes = [];
       action.data.forEach(d => {
-        _diffexpGenes.push(world.varAnnotations.at(d[0], "name"));
+        _diffexpGenes.push(world.varAnnotations.at(d[0], varIndexName));
       });
       return {
         ...state,
@@ -137,6 +138,11 @@ const Controls = (
     /*******************************
              User Events
      *******************************/
+    case "change graph interaction mode":
+      return {
+        ...state,
+        graphInteractionMode: action.data
+      };
     case "change opacity deselected cells in 2d graph background":
       return {
         ...state,

client/src/reducers/crossfilter.js

@@ -20,10 +20,11 @@ const CrossfilterReducer = (
 ) => {
   switch (action.type) {
     case "initial data load complete (universe exists)": {
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       const crossfilter = World.createObsDimensions(
         new Crossfilter(world.obsAnnotations),
-        world
+        world,
+        layoutChoice.currentDimNames
       );
       return crossfilter;
     }
@@ -43,9 +44,13 @@ const CrossfilterReducer = (
     case "set clip quantiles":
     case "set World to current selection": {
       const { userDefinedGenes, diffexpGenes } = prevSharedState.controls;
-      const { world } = nextSharedState;
+      const { world, layoutChoice } = nextSharedState;
       let crossfilter = new Crossfilter(world.obsAnnotations);
-      crossfilter = World.createObsDimensions(crossfilter, world);
+      crossfilter = World.createObsDimensions(
+        crossfilter,
+        world,
+        layoutChoice.currentDimNames
+      );
       crossfilter = ControlsHelpers.createGeneDimensions(
         userDefinedGenes,
         diffexpGenes,
@@ -55,6 +60,23 @@ const CrossfilterReducer = (
       return crossfilter;
     }
 
+    case "set layout choice": {
+      /*
+      when switching layouts:
+      - delete the existing XY index
+      - add the new XY index (which implicitly selects all on it)
+      */
+      const { world, layoutChoice } = nextSharedState;
+      return state
+        .delDimension(layoutDimensionName("XY"))
+        .addDimension(
+          layoutDimensionName("XY"),
+          "spatial",
+          world.obsLayout.col(layoutChoice.currentDimNames[0]).asArray(),
+          world.obsLayout.col(layoutChoice.currentDimNames[1]).asArray()
+        );
+    }
+
     case "request user defined gene success": {
       const { world } = prevSharedState;
       const gene = action.data.genes[0];
@@ -68,8 +90,9 @@ const CrossfilterReducer = (
 
     case "request differential expression success": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const genes = _.map(action.data, d =>
-        world.varAnnotations.at(d[0], "name")
+        world.varAnnotations.at(d[0], varIndexName)
       );
       const crossfilter = _.reduce(
         genes,
@@ -87,10 +110,11 @@ const CrossfilterReducer = (
 
     case "clear differential expression": {
       const { world } = prevSharedState;
+      const varIndexName = world.schema.annotations.var.index;
       const crossfilter = _.reduce(
         action.diffExp,
         (xfltr, values) => {
-          const name = world.varAnnotations.at(values[0], "name");
+          const name = world.varAnnotations.at(values[0], varIndexName);
           return xfltr.delDimension(diffexpDimensionName(name));
         },
         state
@@ -161,10 +185,12 @@ const CrossfilterReducer = (
     case "categorical metadata filter select":
     case "categorical metadata filter deselect": {
       const { categoricalSelection } = nextSharedState;
+      const { world } = prevSharedState;
       const cat = categoricalSelection[action.metadataField];
+      const col = world.obsAnnotations.col(action.metadataField);
       return state.select(obsAnnoDimensionName(action.metadataField), {
         mode: "exact",
-        values: ControlsHelpers.selectedValuesForCategory(cat)
+        values: ControlsHelpers.selectedValuesForCategory(cat, col)
       });
     }
 

client/src/reducers/graphSelection.js

@@ -7,7 +7,8 @@ const GraphSelection = (
 ) => {
   switch (action.type) {
     case "set clip quantiles":
-    case "reset World to eq Universe": {
+    case "reset World to eq Universe":
+    case "set layout choice": {
       return {
         ...state,
         selection: {

client/src/reducers/index.js

@@ -12,9 +12,11 @@ import graphSelection from "./graphSelection";
 import crossfilter from "./crossfilter";
 import colors from "./colors";
 import differential from "./differential";
+import layoutChoice from "./layoutChoice";
 import responsive from "./responsive";
 import controls from "./controls";
 import resetCache from "./resetCache";
+import centroidLabel from "./centroidLabel";
 
 import undoableConfig from "./undoableConfig";
 
@@ -23,6 +25,7 @@ const Reducer = undoable(
     ["config", config],
     ["universe", universe],
     ["world", world],
+    ["layoutChoice", layoutChoice],
     ["categoricalSelection", categoricalSelection],
     ["continuousSelection", continuousSelection],
     ["graphSelection", graphSelection],
@@ -31,6 +34,7 @@ const Reducer = undoable(
     ["controls", controls],
     ["differential", differential],
     ["responsive", responsive],
+    ["centroidLabel", centroidLabel],
     ["resetCache", resetCache]
   ]),
   [
@@ -41,7 +45,8 @@ const Reducer = undoable(
     "crossfilter",
     "colors",
     "controls",
-    "differential"
+    "differential",
+    "layoutChoice"
   ],
   undoableConfig
 );

client/src/reducers/layoutChoice.js

@@ -0,0 +1,50 @@
+/*
+we have a UI heuristic to pick the default layout, based on assumptions
+about commonly used names.  Preferentially, pick in the following order:
+
+  1. "umap"
+  2. "tsne"
+  3. "pca"
+  4. give up, use the first available
+*/
+function bestDefaultLayout(layouts) {
+  const preferredNames = ["umap", "tsne", "pca"];
+  const idx = preferredNames.findIndex(name => layouts.indexOf(name) !== -1);
+  if (idx !== -1) return preferredNames[idx];
+  return layouts[0];
+}
+
+const LayoutChoice = (
+  state = {
+    available: [], // all available choices
+    current: undefined, // name of the current layout, eg, 'umap'
+    currentDimNames: [] // dimension name
+  },
+  action,
+  nextSharedState
+) => {
+  switch (action.type) {
+    case "initial data load complete (universe exists)":
+    case "reset World to eq Universe": {
+      // set default to default
+      const { schema } = nextSharedState.world;
+      const available = schema.layout.obs.map(v => v.name);
+      const current = bestDefaultLayout(available);
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { available, current, currentDimNames };
+    }
+
+    case "set layout choice": {
+      const { schema } = nextSharedState.world;
+      const current = action.layoutChoice;
+      const currentDimNames = schema.layout.obsByName[current].dims;
+      return { ...state, current, currentDimNames };
+    }
+
+    default: {
+      return state;
+    }
+  }
+};
+
+export default LayoutChoice;

client/src/reducers/undoableConfig.js

@@ -29,7 +29,10 @@ const skipOnActions = new Set([
   "clear all user defined genes",
 
   "get single gene expression for coloring started",
-  "get single gene expression for coloring error"
+  "get single gene expression for coloring error",
+
+  "category value mouse hover start",
+  "category value mouse hover end"
 ]);
 
 /*
@@ -59,7 +62,7 @@ const saveOnActions = new Set([
   "categorical metadata filter select",
   "categorical metadata filter deselect",
   "categorical metadata filter all of these",
-  "categorical metadata none of these",
+  "categorical metadata filter none of these",
 
   "color by categorical metadata",
   "color by continuous metadata",
@@ -72,7 +75,10 @@ const saveOnActions = new Set([
   "store current cell selection as differential set 2",
 
   "set World to current selection",
-  "set clip quantiles"
+  "set clip quantiles",
+
+  "set layout choice",
+  "change graph interaction mode"
 ]);
 
 /**
@@ -98,9 +104,15 @@ const applyPending = () => ({
   [actionKey]: "applyPending",
   [stateKey]: { fsm: null }
 });
-const skip = fsm => ({ [actionKey]: "skip", [stateKey]: { fsm } });
+const skip = (fsm, transition) => ({
+  [actionKey]: "skip",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
 const clear = () => ({ [actionKey]: "clear", [stateKey]: { fsm: null } });
-const save = fsm => ({ [actionKey]: "save", [stateKey]: { fsm } });
+const save = (fsm, transition) => ({
+  [actionKey]: "save",
+  [stateKey]: { fsm: transition.to !== "done" ? fsm : null }
+});
 
 /*
 Error handler for state transitions that are unexpected.  Called by

client/src/util/camera.js

@@ -5,10 +5,9 @@ const mp = require("mouse-position");
 const mb = require("mouse-pressed");
 const key = require("key-pressed");
 
-const panSpeed = 0.4;
+const panSpeed = 1.0; // changed from 0.4 to 1.0 per issue #722
 const scaleSpeed = 0.5;
 const scaleMax = 3;
-// const scaleMin = 1.15
 const scaleMin = 1.03;
 
 function attachCamera(canvas, opts) {

client/src/util/centroid.js

@@ -0,0 +1,58 @@
+import quantile from "./quantile";
+
+/*
+  Centroid coordinate calculation
+*/
+const calcMeanCentroid = (world, annoName, annoValue, layoutDimNames) => {
+  const centroid = { x: 0, y: 0, size: 0 };
+  const annoArray = world.obsAnnotations.col(annoName).asArray();
+  const layoutXArray = world.obsLayout.col(layoutDimNames[0]).asArray();
+  const layoutYArray = world.obsLayout.col(layoutDimNames[1]).asArray();
+
+  for (let i = 0, len = annoArray.length; i < len; i += 1) {
+    if (annoArray[i] === annoValue) {
+      centroid.x += layoutXArray[i];
+      centroid.y += layoutYArray[i];
+      centroid.size += 1;
+    }
+  }
+
+  if (centroid[2] !== 0) {
+    centroid.x /= centroid.size;
+    centroid.y /= centroid.size;
+  }
+
+  return [centroid.x, centroid.y];
+};
+
+const calcMedianCentroid = (world, annoName, annoValue, layoutDimNames) => {
+  const centroidX = [];
+  const centroidY = [];
+  let hasFinite = false;
+
+  const annoArray = world.obsAnnotations.col(annoName).asArray();
+  const layoutXArray = world.obsLayout.col(layoutDimNames[0]).asArray();
+  const layoutYArray = world.obsLayout.col(layoutDimNames[1]).asArray();
+
+  for (let i = 0, len = annoArray.length; i < len; i += 1) {
+    if (annoArray[i] === annoValue) {
+      hasFinite =
+        Number.isFinite(layoutXArray[i]) || Number.isFinite(layoutYArray[i])
+          ? true
+          : hasFinite;
+      centroidX.push(layoutXArray[i]);
+      centroidY.push(layoutYArray[i]);
+    }
+  }
+
+  if (hasFinite) {
+    const medianX = quantile([0.5], Float64Array.from(centroidX));
+    const medianY = quantile([0.5], Float64Array.from(centroidY));
+
+    return [medianX, medianY];
+  }
+
+  return null;
+};
+
+export default calcMedianCentroid;

client/src/util/dataframe/dataframe.js

@@ -1,8 +1,19 @@
 import { IdentityInt32Index, isLabelIndex } from "./labelIndex";
 // weird cross-dependency that we should clean up someday...
 import { sortArray } from "../typedCrossfilter/sort";
-import { isTypedArray, isArrayOrTypedArray, callOnceLazy } from "./util";
+import {
+  isTypedArray,
+  isArrayOrTypedArray,
+  callOnceLazy,
+  memoize
+} from "./util";
 import { summarizeContinuous, summarizeCategorical } from "./summarize";
+import {
+  histogramCategorical,
+  hashCategorical,
+  histogramContinuous,
+  hashContinuous
+} from "./histogram";
 
 /*
 Dataframe is an immutable 2D matrix similiar to Python Pandas Dataframe,
@@ -59,6 +70,17 @@ Dataframe
 **/
 
 class Dataframe {
+  /**
+  memoization helpers.
+  **/
+  static __DataframeId__ = 0;
+
+  static __getId() {
+    const id = Dataframe.__DataframeId__;
+    Dataframe.__DataframeId__ += 1;
+    return id;
+  }
+
   /**
   Constructors & factories
   **/
@@ -102,6 +124,7 @@ class Dataframe {
     this.length = nRows; // convenience accessor for row dimension
     this.rowIndex = rowIndex;
     this.colIndex = colIndex;
+    this.__id = Dataframe.__getId();
 
     this.__compile(__columnsAccessor);
   }
@@ -144,7 +167,7 @@ class Dataframe {
     }
   }
 
-  static __compileColumn(column, getOffset, getLabel) {
+  static __compileColumn(column, getRowByOffset, getRowByLabel) {
     /*
       Each column accessor is a function which will lookup data by
       index (ie, is equivalent to dataframe.get(row, col), where 'col'
@@ -172,12 +195,15 @@ class Dataframe {
 
       iget(offset) -- return the value at 'offset'
 
+      ... and more ...
+
     */
     const { length } = column;
+    const __id = Dataframe.__getId();
 
     /* get value by row label */
     const get = function get(rlabel) {
-      return column[getOffset(rlabel)];
+      return column[getRowByOffset(rlabel)];
     };
 
     /* get value by row offset */
@@ -192,7 +218,7 @@ class Dataframe {
 
     /* test for row label inclusion in column */
     const has = function has(rlabel) {
-      const offset = getOffset(rlabel);
+      const offset = getRowByOffset(rlabel);
       return offset >= 0 && offset < length;
     };
 
@@ -212,7 +238,7 @@ class Dataframe {
       if (offset === -1) {
         return undefined;
       }
-      return getLabel(offset);
+      return getRowByLabel(offset);
     };
 
     /*
@@ -224,12 +250,25 @@ class Dataframe {
         : summarizeCategorical(column)
     );
 
+    /*
+    Create histogram bins for this column.  Memoized.
+    */
+    if (isTypedArray(column)) {
+      const mFn = memoize(histogramContinuous, hashContinuous);
+      get.histogram = (bins, domain, by) => mFn(get, bins, domain, by);
+    } else {
+      const mFn = memoize(histogramCategorical, hashCategorical);
+      get.histogram = by => mFn(get, by);
+    }
+
     get.summarize = summarize;
     get.asArray = asArray;
     get.has = has;
     get.ihas = ihas;
     get.indexOf = indexOf;
     get.iget = iget;
+    get.__id = __id;
+
     return get;
   }
 
@@ -239,12 +278,15 @@ class Dataframe {
 
     Use an existing accessor if provided, else compile a new one.
     */
-    const { getOffset, getLabel } = this.rowIndex;
+    const {
+      getOffset: getRowByOffset,
+      getLabel: getRowByLabel
+    } = this.rowIndex;
     this.__columnsAccessor = this.__columns.map((column, idx) => {
       if (accessors[idx]) {
         return accessors[idx];
       }
-      return Dataframe.__compileColumn(column, getOffset, getLabel);
+      return Dataframe.__compileColumn(column, getRowByOffset, getRowByLabel);
     });
   }
 

client/src/util/dataframe/histogram.js

@@ -0,0 +1,135 @@
+/*
+Dataframe histogram
+*/
+import { isTypedArray } from "./util";
+
+function _histogramContinuous(column, bins, min, max) {
+  const valBins = new Array(bins).fill(0);
+  if (!column) {
+    return valBins;
+  }
+  const binWidth = (max - min) / (bins - 1);
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const val = colArray[r];
+    if (val <= max && val >= min) {
+      // ensure test excludes NaN values
+      const valBin = (val - min) / binWidth;
+      valBins[valBin] += 1;
+    }
+  }
+  return valBins;
+}
+
+function _histogramContinuousBy(column, bins, min, max, by) {
+  const byMap = new Map();
+  if (!column || !by) {
+    return byMap;
+  }
+  const binWidth = (max - min) / (bins - 1);
+  const byArray = by.asArray();
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const byBin = byArray[r];
+    let valBins = byMap.get(byBin);
+    if (valBins === undefined) {
+      valBins = new Array(bins).fill(0);
+      byMap.set(byBin, valBins);
+    }
+    const val = colArray[r];
+    if (val <= max && val >= min) {
+      // ensure test excludes NaN values
+      const valBin = (val - min) / binWidth;
+      valBins[Math.floor(valBin)] += 1;
+    }
+  }
+  return byMap;
+}
+
+function _histogramCategorical(column) {
+  const valMap = new Map();
+  if (!column) {
+    return valMap;
+  }
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const valBin = colArray[r];
+    let curCount = valMap.get(valBin);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    valMap.set(valBin, curCount + 1);
+  }
+  return valMap;
+}
+
+function _histogramCategoricalBy(column, by) {
+  const byMap = new Map();
+  if (!column || !by) {
+    return byMap;
+  }
+  const byArray = by.asArray();
+  const colArray = column.asArray();
+  for (let r = 0, len = colArray.length; r < len; r += 1) {
+    const byBin = byArray[r];
+    let valMap = byMap.get(byBin);
+    if (valMap === undefined) {
+      valMap = new Map();
+      byMap.set(byBin, valMap);
+    }
+    const valBin = colArray[r];
+    let curCount = valMap.get(valBin);
+    if (curCount === undefined) {
+      curCount = 0;
+    }
+    valMap.set(valBin, curCount + 1);
+  }
+  return byMap;
+}
+
+/*
+Count category occupancy.  Optional group-by category.
+*/
+export function histogramCategorical(column, by) {
+  if (by && isTypedArray(by)) {
+    throw new Error("Group by column must be categorical");
+  }
+  return by
+    ? _histogramCategoricalBy(column, by)
+    : _histogramCategorical(column);
+}
+
+/*
+Memoization hash for histogramCategorical()
+*/
+export function hashCategorical(column, by) {
+  if (by) {
+    return `${column.__id}:${by.__id}`;
+  }
+  return `${column.__id}:`;
+}
+
+/*
+Bin counts for continuous/scalar values, with optional group-by category.
+Values outside domain are ignored.
+*/
+export function histogramContinuous(column, bins = 40, domain = [0, 1], by) {
+  if (by && isTypedArray(by)) {
+    throw new Error("Group by column must be categorical");
+  }
+  const [min, max] = domain;
+  return by
+    ? _histogramContinuousBy(column, bins, min, max, by)
+    : _histogramContinuous(column, bins, min, max);
+}
+
+/*
+Memoization hash for histogramContinuous
+*/
+export function hashContinuous(column, bins = "", domain = [0, 0], by) {
+  const [min, max] = domain;
+  if (by) {
+    return `${column.__id}:${bins}:${min}:${max}:${by.__id}`;
+  }
+  return `${column.__id}::${bins}:${min}:${max}`;
+}

client/src/util/dataframe/labelIndex.js

@@ -3,6 +3,8 @@ Label indexing - map a label to & from an integer offset.  See Dataframe
 for how this is used.
 **/
 
+import { rangeFill as fillRange } from "../range";
+
 /*
 Private utility functions
 */
@@ -21,14 +23,6 @@ function extent(tarr) {
   return [min, max];
 }
 
-function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, l = larr.length; i < l; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 /* eslint-disable class-methods-use-this */
 class IdentityInt32Index {
   /*

client/src/util/dataframe/util.js

@@ -5,6 +5,10 @@ Private utility code for dataframe
 export { isTypedArray, isArrayOrTypedArray } from "../typeHelpers";
 
 export function callOnceLazy(f) {
+  /*
+  call function once, and save the result, regardless of arguments (this is not
+  the same as typical memoization).
+  */
   let value;
   let calledOnce = false;
   const result = function result(...args) {
@@ -14,6 +18,25 @@ export function callOnceLazy(f) {
     }
     return value;
   };
-
   return result;
 }
+
+export function memoize(fn, hashFn) {
+  /* 
+  function memoization, with user-provided hash.  hashFn must return a
+  key which will be unique as a Map key (ie, obeys "sameValueZero" algorithm
+  as defined in the JS spec).  For more info on hash key, see:
+  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map#Key_equality
+  */
+  const cache = new Map();
+  const wrap = function wrap(...args) {
+    const key = hashFn(...args);
+    if (cache.has(key)) {
+      return cache.get(key);
+    }
+    const result = fn(...args);
+    cache.set(key, result);
+    return result;
+  };
+  return wrap;
+}

client/src/util/range.js

@@ -0,0 +1,45 @@
+/*
+Array range creation
+
+range(start, stop, step) -> Array
+	This is identical to https://docs.python.org/3/library/functions.html#func-range
+	Returns new array filled with a range of numbers.
+
+	Usage:
+
+		range(stop) - start defaults to zero, step defaults to 1
+		range(start, stop, [step]) - step defaults to 1
+
+	Examples:
+		range(3) -> [0, 1, 2]
+		range(1, 3) -> [1, 2]
+		range(1, 5, 2) -> [1, 3]
+
+
+rangeFill(array, start, step) -> array
+	Fill entire array with values, from start, by step.  Returns first array.
+	start defaults to zero, step defaults to one.
+
+*/
+
+function _doFill(arr, start, step, count) {
+	for (let idx = 0, val = start; idx < count; idx += 1, val += step) {
+		arr[idx] = val;
+	}
+	return arr;
+}
+
+export function rangeFill(arr, start = 0, step = 1) {
+	return _doFill(arr, start, step, arr.length);
+}
+
+export function range(start, stop, step) {
+	if (start === undefined) return [];
+	if (stop === undefined) {
+		stop = start;
+		start = 0;
+	}
+	step = step || 1; // catch undefind and zero
+	const len = Math.max(Math.ceil((stop - start) / step), 0);
+	return _doFill(new Array(len), start, step, len);
+}

client/src/util/scaleLinear.js

@@ -9,8 +9,14 @@
 // this is is equivalent to d3.scaleLinear().domain([0,1]).range([-1,1])
 
 export default (domain, range) => {
-  const domainStart = domain[0];
-  const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
-  const rangeStart = range[0];
-  return value => (value - domainStart) * scale + rangeStart;
+	const domainStart = domain[0];
+	const scale = (range[1] - range[0]) / (domain[1] - domain[0]);
+	const invScale = 1 / scale;
+	const rangeStart = range[0];
+	const f = value => (value - domainStart) * scale + rangeStart;
+
+	// inverter
+	f.invert = value => (value - rangeStart) * invScale + domainStart;
+
+	return f;
 };

client/src/util/stateManager/colorHelpers.js

@@ -1,12 +1,12 @@
 /*
 Helper functions for the embedded graph colors
 */
-import _ from "lodash";
 import * as d3 from "d3";
 import { interpolateRainbow, interpolateCool } from "d3-scale-chromatic";
 import * as globals from "../../globals";
 import parseRGB from "../parseRGB";
 import finiteExtent from "../finiteExtent";
+import { range } from "../range";
 
 /*
 create new colors state object.   Paramters:
@@ -15,7 +15,7 @@ create new colors state object.   Paramters:
     "color by continuous metadata", "color by categorical metadata"
   -
 */
-function createColors(world, colorMode = null, colorAccessor = null) {
+export function createColors(world, colorMode = null, colorAccessor = null) {
   switch (colorMode) {
     case "color by categorical metadata": {
       return createColorsByCategoricalMetadata(world, colorAccessor);
@@ -37,9 +37,7 @@ function createColors(world, colorMode = null, colorAccessor = null) {
 }
 
 function createColorsByCategoricalMetadata(world, accessor) {
-  const { categories } = _.filter(world.schema.annotations.obs, {
-    name: accessor
-  })[0];
+  const { categories } = world.schema.annotations.obsByName[accessor];
 
   const scale = d3
     .scaleSequential(interpolateRainbow)
@@ -67,7 +65,7 @@ function createColorsByContinuousMetadata(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);
@@ -97,7 +95,7 @@ function createColorsByExpression(world, accessor) {
   const scale = d3
     .scaleQuantile()
     .domain([min, max])
-    .range(_.range(colorBins - 1, -1, -1));
+    .range(range(colorBins - 1, -1, -1));
 
   /* pre-create colors - much faster than doing it for each obs */
   const colors = new Array(colorBins);
@@ -119,4 +117,26 @@ function createColorsByExpression(world, accessor) {
   return { rgb, scale };
 }
 
-export default createColors;
+export const resetColors = world => {
+  const { rgb, scale } = createColors(world);
+  return {
+    colorMode: null,
+    colorAccessor: null,
+    rgb,
+    scale
+  };
+};
+
+export const checkIfColorByDiffexpAndResetColors = (
+  prevControls,
+  state,
+  prevWorld
+) => {
+  if (prevControls.diffexpGenes.includes(state.colorAccessor)) {
+    return {
+      ...state,
+      ...resetColors(prevWorld)
+    };
+  }
+  return null;
+};

client/src/util/stateManager/controlsHelpers.js

@@ -5,7 +5,7 @@ Helper functions for the controls reducer
 import _ from "lodash";
 
 import * as globals from "../../globals";
-import { fillRange } from "../typedCrossfilter/util";
+import { rangeFill as fillRange } from "../range";
 import {
   userDefinedDimensionName,
   diffexpDimensionName
@@ -21,16 +21,16 @@ Remember that option values can be ANY js type, except undefined/null.
   {
     _category_name_1: {
       // map of option value to index
-      categoryIndices: Map([
+      categoryValueIndices: Map([
         catval1: index,
         ...
       ])
 
       // index->selection true/false state
-      categorySelected: [ true/false, true/false, ... ]
+      categoryValueSelected: [ true/false, true/false, ... ]
 
       // number of options
-      numCategories: number,
+      numCategoryValues: number,
 
       // isTruncated - true if the options for selection has
       // been truncated (ie, was too large to implement)
@@ -56,27 +56,31 @@ function topNCategories(summary) {
 
 export function createCategoricalSelection(maxCategoryItems, world) {
   const res = {};
+  const obsIndexName = world.schema.annotations.obs.index;
   _.forEach(world.obsAnnotations.colIndex.keys(), key => {
     const summary = world.obsAnnotations.col(key).summarize();
     if (summary.categories) {
       const isColorField = key.includes("color") || key.includes("Color");
       const isSelectableCategory =
         !isColorField &&
-        key !== "name" &&
+        key !== obsIndexName &&
         summary.categories.length < maxCategoryItems;
       if (isSelectableCategory) {
-        const [categoryValues, categoryCounts] = topNCategories(summary);
-        const categoryIndices = new Map(categoryValues.map((v, i) => [v, i]));
-        const numCategories = categoryIndices.size;
-        const categorySelected = new Array(numCategories).fill(true);
+        const [categoryValues, categoryValueCounts] = topNCategories(summary);
+        const categoryValueIndices = new Map(
+          categoryValues.map((v, i) => [v, i])
+        );
+        const numCategoryValues = categoryValueIndices.size;
+        const categoryValueSelected = new Array(numCategoryValues).fill(true);
         const isTruncated = categoryValues.length < summary.numCategories;
         res[key] = {
           categoryValues, // array: of natively typed category values
-          categoryIndices, // map: category value (native type) -> category index
-          categorySelected, // array: t/f selection state
-          numCategories, // number: of categories
+          categoryValueIndices, // map: category value (native type) -> category index
+          categoryValueSelected, // array: t/f selection state
+          numCategoryValues, // number: of values in the category
           isTruncated, // bool: true if list was truncated
-          categoryCounts // array: cardinality of each category
+          categoryValueCounts, // array: cardinality of each category,
+          categorySelected: true // bool - default state for entire category
         };
       }
     }
@@ -88,12 +92,26 @@ export function createCategoricalSelection(maxCategoryItems, world) {
 given a categoricalSelection, return the list of all category values
 where selection state is true (ie, they are selected).
 */
-export function selectedValuesForCategory(categorySelectionState) {
-  const selectedValues = _([...categorySelectionState.categoryIndices])
-    .filter(tuple => categorySelectionState.categorySelected[tuple[1]])
-    .map(tuple => tuple[0])
-    .value();
-  return selectedValues;
+export function selectedValuesForCategory(categorySelectionState, dfColumn) {
+  const {
+    categorySelected,
+    categoryValueSelected,
+    categoryValueIndices
+  } = categorySelectionState;
+  let selectedValues;
+  if (categorySelected) {
+    selectedValues = new Set(dfColumn.summarize().categories);
+  } else {
+    selectedValues = new Set();
+  }
+  categoryValueIndices.forEach((catIndex, catValue) => {
+    if (!categoryValueSelected[catIndex]) {
+      selectedValues.delete(catValue);
+    } else {
+      selectedValues.add(catValue);
+    }
+  });
+  return [...selectedValues.values()];
 }
 
 /*

client/src/util/stateManager/index.js

@@ -14,7 +14,7 @@ This is all VERY tightly integrated with reducers and actions, and
 exists to support those concepts.
 */
 
-export { default as createColors } from "./colorHelpers";
+export * as ColorHelpers from "./colorHelpers";
 export * as Universe from "./universe";
 export * as World from "./world";
 export * as WorldUtil from "./worldUtil";

client/src/util/stateManager/universe.js

@@ -78,6 +78,7 @@ function AnnotationsFBSToDataframe(arrayBuffer) {
   The application has strong assumptions that all scalar data will be
   stored as a float32 or float64 (regardless of underlying data types).
   For example, clipping of value ranges (eg, user-selected percentiles)
+  depends on the ability to use NaN in any numeric type.
 
   All float data from the server is left as is.  All non-float is promoted
   to an appropriate float.
@@ -98,15 +99,16 @@ function AnnotationsFBSToDataframe(arrayBuffer) {
 
 function LayoutFBSToDataframe(arrayBuffer) {
   const fbs = decodeMatrixFBS(arrayBuffer, true);
-  if (fbs.columns.length !== 2 || !fbs.columns.every(isFpTypedArray)) {
+  if (fbs.columns.length < 2 || !fbs.columns.every(isFpTypedArray)) {
     // We have strong assumptions about the shape & type of layout data.
     throw new Error("Unexpected layout data type returned from server");
   }
+
   const df = new Dataframe.Dataframe(
     [fbs.nRows, fbs.nCols],
     fbs.columns,
     null,
-    new Dataframe.KeyIndex(["X", "Y"])
+    new Dataframe.KeyIndex(fbs.colIdx)
   );
   return df;
 }
@@ -122,15 +124,15 @@ function reconcileSchemaCategoriesWithSummary(universe) {
   cases, add a 'categories' field to the schema so it is accessible.
   */
 
-  _.forEach(universe.schema.annotations.obs, s => {
+  universe.schema.annotations.obs.columns.forEach(s => {
     if (
       s.type === "string" ||
       s.type === "boolean" ||
       s.type === "categorical"
     ) {
       const categories = _.union(
-        _.get(s, "categories", []),
-        _.get(universe.obsAnnotations.col(s.name).summarize(), "categories", [])
+        s.categories ?? [],
+        universe.obsAnnotations.col(s.name).summarize().categories ?? []
       );
       s.categories = categories;
     }
@@ -154,6 +156,9 @@ export function createUniverseFromResponse(
   universe.schema = schema;
   universe.nObs = schema.dataframe.nObs;
   universe.nVar = schema.dataframe.nVar;
+  /* add defaults, as we can't assume back-end will fully populate schema */
+  if (!schema.layout.var) schema.layout.var = [];
+  if (!schema.layout.obs) schema.layout.obs = [];
 
   /* annotations */
   universe.obsAnnotations = AnnotationsFBSToDataframe(annotationsObsResponse);
@@ -174,10 +179,16 @@ export function createUniverseFromResponse(
 
   /* Index schema for ease of use */
   universe.schema.annotations.obsByName = fromEntries(
-    universe.schema.annotations.obs.map(v => [v.name, v])
+    universe.schema.annotations.obs.columns.map(v => [v.name, v])
   );
   universe.schema.annotations.varByName = fromEntries(
-    universe.schema.annotations.var.map(v => [v.name, v])
+    universe.schema.annotations.var.columns.map(v => [v.name, v])
+  );
+  universe.schema.layout.obsByName = fromEntries(
+    universe.schema.layout.obs.map(v => [v.name, v])
+  );
+  universe.schema.layout.varByName = fromEntries(
+    universe.schema.layout.var.map(v => [v.name, v])
   );
   return universe;
 }
@@ -202,8 +213,9 @@ export function convertDataFBStoObject(universe, arrayBuffer) {
     throw new Error("Unexpected non-floating point response from server.");
   }
 
+  const varIndexName = universe.schema.annotations.var.index;
   for (let c = 0; c < colIdx.length; c += 1) {
-    const varName = universe.varAnnotations.at(colIdx[c], "name");
+    const varName = universe.varAnnotations.at(colIdx[c], varIndexName);
     result[varName] = columns[c];
   }
   return result;

client/src/util/stateManager/world.js

@@ -260,13 +260,17 @@ function deduceDimensionType(attributes, fieldName) {
   return dimensionType;
 }
 
-export function createObsDimensions(crossfilter, world) {
+export function createObsDimensions(crossfilter, world, XYdimNames) {
   /*
   create and return a crossfilter with a dimension for every obs annotation
-  for which we have a supported type, *except* 'name'
+  for which we have a supported type, *except* for the index column, indicated
+  by schema.annotations.obs.index.
   */
   const { schema, obsLayout, obsAnnotations } = world;
-  const annoList = schema.annotations.obs.filter(anno => anno.name !== "name");
+  const indexName = schema.annotations.obs.index;
+  const annoList = schema.annotations.obs.columns.filter(
+    anno => anno.name !== indexName
+  );
   crossfilter = annoList.reduce((xfltr, anno) => {
     const dimType = deduceDimensionType(anno, anno.name);
     const colData = obsAnnotations.col(anno.name).asArray();
@@ -283,8 +287,8 @@ export function createObsDimensions(crossfilter, world) {
   return crossfilter.addDimension(
     layoutDimensionName("XY"),
     "spatial",
-    obsLayout.col("X").asArray(),
-    obsLayout.col("Y").asArray()
+    obsLayout.col(XYdimNames[0]).asArray(),
+    obsLayout.col(XYdimNames[1]).asArray()
   );
 }
 

client/src/util/typedCrossfilter/util.js

@@ -1,22 +1,12 @@
 // jshint esversion: 6
 
 import { sortIndex } from "./sort";
+import { rangeFill as fillRange } from "../range";
 
 /*
     Utility functions, private to this module.
 */
 
-// fill an array or typedarray with a sequential range of numbers,
-// starting with `start`
-//
-export function fillRange(arr, start = 0) {
-  const larr = arr;
-  for (let i = 0, len = larr.length; i < len; i += 1) {
-    larr[i] = i + start;
-  }
-  return larr;
-}
-
 // slice out of one array into another, using an index array
 //
 export function sliceByIndex(src, index) {

dev_docs/developer_guidelines.md

@@ -0,0 +1,91 @@
+# Developer guidelines
+
+### Requirements
+- npm
+- Python 3.6+
+- Chrome
+
+[See dev section of README](../README.md)
+
+**All instructions are expected to be run from the top level cellxgene directory unless otherwise specified.**
+
+## Server dev
+### Install
+* Build the client and put static files in place: `make build-for-server-dev`
+* Install from local files: `make install-dev`
+
+### Launch
+* `cellxgene launch [options] <datafile>`
+
+### Reloading
+If you install cellxgene using `make install-dev` the server will be restarted every time you make changes on the server code. If changes affects the client, the browser must be reloaded.
+
+### Linter
+We use `flake8` to lint code. Travis CI runs `flake8 server`.
+
+### Test
+1. Install development requirements `pip install -r server/requirements-dev.txt`
+2. Run tests `pytest server/test`
+
+### Tips
+* Install in a virtualenv
+* May need to rebuild/reinstall when you make client changes
+
+## Client dev
+### Install
+1. Install prereqs for client: `npm install --prefix client/ client`
+2. Install cellxgene server: `pip install -e .` Caveat: this will not build the production client package - you must use the [server install](#install) instructions above to serve web assets.
+
+### Launch
+To launch with hot reloading you need to launch the server and the client separately. Node's hot reloading starts the client on its own node server and auto-refreshes when changes are made.
+1. Launch server (the client relies on the REST API being available): `cellxgene launch [options] <datafile>`
+2. Launch client: in `client/` directory run `npm run start`
+3. Client will be served on localhost:3000
+
+### Build
+To build only the client: `make build-client`
+
+### Linter
+We use `eslint` to lint the code and `prettier` as our code formatter.
+
+### Test
+In `client/` directory run `npm run unit-test`
+
+### Tips
+* You can also install/launch the server side code from npm scrips (requires python3.6 with virtualenv) in `client/` directory run `npm run backend-dev`
+
+## Running tests
+Client and server tests run on Travis CI for every push, PR, and commit to master on github. End to end tests run nightly on master only. 
+
+### Server unit tests
+Install development requirements `pip install -r server/requirements-dev.txt`
+Run tests `pytest server/test`
+
+### Client unit tests
+In `client/` directory run `npm run unit-test`
+
+### End to end tests
+
+End to end tests use two env variables:
+* `JEST_ENV` - environment to run end to end tests. Default `dev`
+    * `prod` - run headless with no slowdown, chromium will not open.
+    * `dev` - opens chromimum, runs tests with minimal slowdown, close on exit.
+    * `debug` - opens chromium, runs tests with 100ms slowdown, dev tools open, chrome stays open on exit.
+* `JEST_CXG_PORT` - port that end to end tests are being run on. Default `3000` (client hosted port).
+
+On CI the end to end tests are run with `JEST_ENV` set to `prod` using the `smoke-test` npm script
+
+To run end to end tests as they will be run on CI
+1. cellxgene should be built and installed as [specified in server dev](#install)
+2. `export JEST_ENV='prod'`
+3. `export JEST_CXG_PORT='5000'`
+4. Run `npm run --prefix client/ smoke-test`
+
+Run end to end tests interactively during development
+1. cellxgene should be installed as [specified in client dev](#install-1)
+2. Follow [launch](#launch-1) instructions for client dev with dataset `example-dataset/pbmc3k`
+3. Run `npm run --prefix client/ e2e`
+4. To debug a failing test `export JEST_ENV='debug'` and re-run.
+
+
+

dev_docs/release_process.md

@@ -43,24 +43,53 @@ Follow these steps to create a release.
 8. Publish to pypi by performing the following steps (assumes you that you have registered for pypi, 
 and that you have write access to the cellxgene pypi package):
     - Build the distribution and upload to test pypi `make release-stage-2`
-    - [optional] Test the test installation in a fresh virtual environment using `make install-release-test`
+    - Test the test installation in a fresh virtual environment using `make install-release-test`
     - Upload the package to real pypi using `make release-stage-final`
-    - [optional] Test the installation in a fresh virtual environment using
+    - Test the installation in a fresh virtual environment using
       `pip install cellxgene`
-    - **Troubleshooting**:
-        - Fails to upload to test.pypi: pypi doesn't allow you to reupload a release with the same version number, 
-        if you accidentally burned a release number you want to use on prod, you have a couple options. 
-            1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
-             then upload to prod `make release-stage-final`. 
-            2) OPTION 2: (DANGER) release directly to prod: `make release-burned`.  
-            3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
-            PART=patch until you get to an unburned version.
-        - The release doesn't install or fails your tests when you install it: Delete it from pypi - Go to pypi.org, sign in, 
-        go to the cellxgene package, click manage, then in the options drop down, click delete and 
-        follow the instructions. You will not be able to use that release number again. If it is a minor bug 
-        and not a major regression, you can just release a patch. 
-    
 
 The optional steps are for testing purposes, and are recommended
 for publishing any major releases, and any releases that significantly
 change the packaging (e.g. new bundled files, new dependencies, etc.)
+
+## Troubleshooting
+### Fails to upload to test.pypi
+ 
+ _PyPi doesn't allow you to reupload a release with the same version number_ 
+If you accidentally burned a release number you want to use on prod, you have a few options:
+1) OPTION 1: Create distribution `make pydist`; test release locally `pip install dist/<release tarball>`;
+ then upload to prod `make release-stage-final`. 
+2) OPTION 2: (DANGER) release directly to prod: `make release-directly-to-prod`.  
+3) OPTION 3: If the release was burned on prod as well run from Step 3 again with option
+PART=patch until you get to an unburned version.
+    
+### The release doesn't install or fails your tests when you install it
+ 
+Delete it from pypi - Go to pypi.org -> sign in -> go to the cellxgene package -> click manage -> then in the options drop down click delete -> follow the instructions. You will not be able to use that release number again. If it is a minor bug and not a major regression, you can just release a patch. 
+### If you need to run stage final on a different computer than stage 2
+ If you run stage final without running stage 2 first, the dist will not have been build on the computer running stage final. The solution is to run `make release-directly-to-prod`. This both builds the distribution files and then releases directly to prod pypi.org. 
+
+## Stage Details
+### Stage 1 - `make release-stage-1`
+    1. Pip installs requirements-dev
+    2. Bumps version by [PART]
+    3. Deletes build directory, client/build, dist and cellxgene.egg-info
+    4. Creates the package-lock.json
+
+### Stage 2 - `make release-stage-2`
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to test.pypi.org
+
+### Stage final - `make release-stage-final`
+    ** Does not build distribution **
+    1. Uploads to pypi.org
+    
+### (DANGER) Release directly to prod `make release-directly-to-prod`
+    ** builds distribution and uploads directly to prod **
+    Only use this if you are directed to by the troubleshooting guide
+    1. Pip installs requirements-dev
+    2. Builds client and server
+    3. Creates distribution release (sdist)
+    4. Uploads to pypi.org

docs/data.md

@@ -1,29 +1,94 @@
----
-layout: default
-title: data
-description: Data
----
+---
+layout: default
+title: data
+description: Data
+---
+
+# Using `cellxgene prepare`
+
+#### What is `cellxgene prepare`?
+
+`prepare` offers an easy command line interface (CLI) to preliminarily wrangle your data into the required format for previewing it with `cellxgene`.
+
+#### What is `cellxgene prepare` _not_?
+
+`cellxgene prepare` is not meant as a way to formally process or analyze your data. It's simply a utility for quickly wrangling your data into cellxgene-compatible format and computing a "vanilla" embedding so you can try out `cellxgene` and get a general sense of a dataset.
+
+#### What input formats does it accept?
+
+Currently, we accept `h5ad` and `loom` files, as well as `10x` directories, and are hoping to accept more formats in the future.
+
+While we'd like to support quick conversion from seurat and bioconductor, these packages don't currently output a python-parseable intermediate file type. In the meantime, you might check out the [converters](https://satijalab.org/seurat/v3.0/conversion_vignette.html) that are under early development.
+
+#### What can `cellxgene prepare` do?
+
+`prepare` uses scanpy to:
+
+- Handle simple data normalization (from a [recipe](https://www.pydoc.io/pypi/scanpy-0.2.3/autoapi/preprocessing/recipes/index.html))
+- Do basic preprocessing to run PCA and compute the neighbor graph
+- Infer clusters
+- Reduce dimensionality to generate embeddings.  
+  You can control which steps to run and their methods (when applicable), via the CLI. The CLI also includes options for computing QC metrics, enforcing matrix sparcity, specifying index names, and plotting output.
+
+**To see a full list of available arguments and options, run `cellxgene prepare --help`.**
+
+#### How do I use `cellxgene prepare`?
+
+As a quick example, let's construct a command to use `prepare` to take a raw expression matrix and generate a processed `h5ad` ready to visualize with cellxgene.
+
+We'll start off using the raw data from the pbmc3k dataset. This dataset is described [here](https://icb-scanpy.readthedocs-hosted.com/en/stable/api/scanpy.datasets.pbmc3k.html), and is available as part of the scanpy API. For this example, we'll assume this raw data is stored in a file called `pbmc3k-raw.h5ad`.
+
+Our `prepare` compose our command looks like this:  
+<img src="prepare-cmd-example.jpg" width="700" />
+
+Let's look at what `prepare` is doing to our data, and how each step relates to the command above. You can see a walkthrough of what's going on under the hood for this example in [this notebook](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-prepare-example.ipynb).
+
+**1 - Compute quality control metrics and store this in our `AnnData` object for later inspection (A)**  
+**2 - Normalize the expression matrix using a basic preprocessing recipe (B)**  
+**3 - Do some preprocessing to run PCA and compute the neighbor graph (auto)**  
+**4 - Infer clusters with the Louvain algorithm and store these labels to visualize later (auto)**  
+**5 - Compute and store umap and tsne embeddings (C)**  
+**6 - Write results to file (D)**
+
+# Example datasets to use with cellxgene
+
 
-
-# data vignette: how to use cellxgene prepare
-
-#### coming soon!
-
-# example datasets to use with cellxgene
-
-### Examination of single cells from primary human pancreas tissue
-cells: 2,544  
-tissue(s): pancreas  
-data: [Human Cell Atlas Data Portal](https://prod.data.humancellatlas.org/explore/projects?filter=%5B%7B%22facetName%22%3A%22organ%22%2C%22terms%22%3A%5B%22pancreas%22%5D%7D%2C%7B%22facetName%22%3A%22project%22%2C%22terms%22%3A%5B%22Single+cell+transcriptome+analysis+of+human+pancreas%22%5D%7D%5D)  
-paper: [Enge, Martin, et al.](https://www.cell.com/cell/fulltext/S0092-8674(17)31053-X?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS009286741731053X%3Fshowall%3Dtrue)
-
-### Tabula Muris
-cells: 53,800  
-tissue(s): muscle, pancreas, bone, large intestine, heart, brain, fat, mammary gland, tongue , diaphragm, bladder, spleen, thymus, lung , skin, liver, trachea, kidney  
-data: [Tabula Muris Data](https://github.com/czbiohub/tabula-muris-vignettes/tree/master/data)  
-paper: [Tabula Muris Consortium](https://www.nature.com/articles/s41586-018-0590-4)  
-
-### Transcriptional profiling of 1.3 million brain cells
-cells: 1,330,000  
-tissue(s): brain  
-data: [10x Genomics](https://community.10xgenomics.com/t5/10x-Blog/Our-1-3-million-single-cell-dataset-is-ready-to-download/ba-p/276)  
+**To download and use these datasets, run:**  
+`curl -O [URL]`  
+`unzip [filename.zip]`  
+`cellxgene launch [filename.h5ad] --open`
+
+### Peripheral blood mononuclear cells
+
+Healthy human PBMCs (10X).
+
+- Source: [10X genomics](https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k)
+- Cells: 2,638
+- File size: 19MB
+- [Raw data](http://cf.10xgenomics.com/samples/cell-exp/1.1.0/pbmc3k/pbmc3k_filtered_gene_bc_matrices.tar.gz)
+- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-processing.ipynb)
+- Download: `curl -O https://cellxgene-example-data.czi.technology/pbmc3k.h5ad.zip`
+
+### Tabula muris
+
+20 organs and tissues from healthy mice (Smart-Seq2).  
+Rich metadata and annotations.
+
+- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/237446v2)
+- Cells: 45,423
+- File size: 174MB
+- [Raw data](https://figshare.com/projects/Tabula_Muris_Transcriptomic_characterization_of_20_organs_and_tissues_from_Mus_musculus_at_single_cell_resolution/27733)
+- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/tabula-muris-processing.ipynb)
+- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris.h5ad.zip`
+
+### Tabula muris senis
+
+22 organs and tissues from healthy mice at ages 3mo, 18mo, 21mo, and 24mo (Smart-Seq2).  
+Rich metadata and annotations.
+
+- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/661728v1)
+- Cells: 81,478
+- File size: 3.9GB
+- Raw data [geo link coming soon!]
+- [Processing](https://www.biorxiv.org/content/10.1101/661728v1)
+- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris-senis.h5ad.zip`

docs/faq.md

@@ -38,7 +38,7 @@ Currently this is not supported directly, but you should be able to do this your
 
 - `.obs` and `.var` annotations are use to extract metadata for filtering
 - `.X` is used to display expression (histograms, scatterplot & colorscale) and to compute differential expression
-- `.obsm` is used for layout
+- `.obsm` is used for layout. If an embedding has more than two components, the first two will be used for visualization.
 
 #### I have a BIG dataset - how can I make cellxgene run as fast as possible?
 

makefile

@@ -1,7 +1,7 @@
 BUILDDIR := build
 CLIENTBUILD := $(BUILDDIR)/client
 SERVERBUILD := $(BUILDDIR)/server
-CLEANFILES :=  $(BUILDDIR)/ client/build dist cellxgene.egg-info
+CLEANFILES :=  $(BUILDDIR)/ client/build build dist cellxgene.egg-info
 
 PART ?= patch
 
@@ -76,7 +76,7 @@ release-stage-final: twine-prod
 
 # DANGER: releases directly to prod
 # use this if you accidently burned a test release version number,
-release-burned : dev-env pydist twine-prod
+release-directly-to-prod : dev-env pydist twine-prod
 	@echo "Dist built and uploaded to pypi.org"
 	@echo "Test the install:"
 	@echo "    make install-release"
@@ -84,6 +84,9 @@ release-burned : dev-env pydist twine-prod
 dev-env :
 	pip install -r server/requirements-dev.txt
 
+gui-env : dev-env
+	pip install -r server/requirements-gui.txt
+
 # give PART=[major, minor, part] as param to make bump
 bump :
 	bumpversion --config-file .bumpversion.cfg $(PART)
@@ -114,15 +117,43 @@ install-dev : uninstall
 
 # install from test.pypi to test your release
 install-release-test : uninstall
-	pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
+	pip install --no-cache-dir --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene
 	@echo "Installed cellxgene from test.pypi.org, now run and smoke test"
 
 # install from pypi to test your release
 install-release : uninstall
-	pip install cellxgene
+	pip install --no-cache-dir cellxgene
 	@echo "Installed cellxgene from pypi.org"
 
+# install from dist
+install-dist : uninstall
+	pip install dist/cellxgene*.tar.gz
+
 uninstall :
 	pip uninstall -y cellxgene || :
 
 .PHONY : install install-dev install-release-test install-release uninstall
+
+# GUI
+
+build-assets :
+	pyside2-rcc server/gui/cellxgene.qrc -o server/gui/cellxgene_rc.py
+
+gui-spec-osx : clean-lite gui-env
+	pip install -e .[gui]
+	pyi-makespec -D -w --additional-hooks-dir server/gui/ -n cellxgene  --add-binary='/System/Library/Frameworks/Tk.framework/Tk':'tk' --add-binary='/System/Library/Frameworks/Tcl.framework/Tcl':'tcl'  --add-data server/app/web/templates/:server/app/web/templates/ --add-data server/app/web/static/:server/app/web/static/ --icon server/gui/images/cxg_icons.icns server/gui/main.py
+	mv cellxgene.spec cellxgene-osx.spec
+
+gui-spec-windows : clean-lite dev-env
+	pip install -e .[gui]
+	pyi-makespec -D -w --additional-hooks-dir server/gui/ -n cellxgene --add-data server/app/web/templates;server/app/web/templates --add-data server/app/web/static;server/app/web/static --icon server/gui/images/icon.ico server/gui/main.py
+	mv cellxgene.spec cellxgene-windows.spec
+
+gui-build-osx : clean-lite
+	pyinstaller --clean cellxgene-osx.spec
+
+gui-build-windows : clean-lite
+	pyinstaller --clean cellxgene-windows.spec
+
+.PHONY : build-assets gui-build-osx gui-build-windows gui-build-osx gui-build-windows
+

server/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.11.0"

server/app/driver/driver.py

@@ -28,7 +28,6 @@ class CXGDriver(metaclass=ABCMeta):
     def _get_default_config():
         return {
             "layout": None,
-            "diffexp": None,
             "max_category_items": None,
             "diffexp_lfc_cutoff": None
         }
@@ -38,14 +37,12 @@ class CXGDriver(metaclass=ABCMeta):
         features = {
             "cluster": {"available": False},
             "layout": {"obs": {"available": False}, "var": {"available": False}},
-            "diffexp": {"available": False},
+            "diffexp": {"available": True, "interactiveLimit": 50000}
         }
         # TODO - Interactive limit should be generated from the actual available methods see GH issue #94
         if self.config["layout"]:
             # TODO handle "var" when gene layout becomes available
             features["layout"]["obs"] = {"available": True, "interactiveLimit": 50000}
-        if self.config["diffexp"]:
-            features["diffexp"] = {"available": True, "interactiveLimit": 50000}
         return features
 
     @abstractmethod

server/app/rest_api/rest.py

@@ -1,9 +1,10 @@
 from http import HTTPStatus
-import pkg_resources
 import warnings
 
 from flask import Blueprint, current_app, jsonify, make_response, request
 from flask_restful import Api, Resource
+from server import __version__ as cellxgene_version
+from anndata import __version__ as anndata_version
 
 from server.app.util.constants import (
     Axis,
@@ -59,16 +60,15 @@ class ConfigAPI(Resource):
                     },
                 ],
                 "displayNames": {
-                    "engine": f"cellxgene Scanpy engine version {pkg_resources.get_distribution('cellxgene').version}",
+                    "engine": f"cellxgene Scanpy engine version ",
                     "dataset": current_app.config["DATASET_TITLE"],
                 },
                 "parameters": {
-                    "max_category_items": current_app.data.config["max_category_items"]
+                    "max-category-items": current_app.data.config["max_category_items"]
                 },
                 "library_versions": {
-                    "scanpy": pkg_resources.get_distribution("scanpy").version,
-                    "cellxgene": pkg_resources.get_distribution("cellxgene").version,
-                    "anndata": pkg_resources.get_distribution("cellxgene").version
+                    "cellxgene": cellxgene_version,
+                    "anndata": anndata_version
                 }
             }
         }

server/app/scanpy_engine/scanpy_engine.py

@@ -1,12 +1,13 @@
 import warnings
 
 import numpy as np
+import pandas
 from pandas.core.dtypes.dtypes import CategoricalDtype
-import scanpy as sc
+import anndata
 from scipy import sparse
 
 from server.app.driver.driver import CXGDriver
-from server.app.util.constants import Axis, DEFAULT_TOP_N
+from server.app.util.constants import Axis, DEFAULT_TOP_N, MAX_LAYOUTS
 from server.app.util.errors import (
     FilterError,
     JSONEncodingValueError,
@@ -41,49 +42,68 @@ class ScanpyEngine(CXGDriver):
     @staticmethod
     def _get_default_config():
         return {
-            "layout": "umap",
-            "diffexp": "ttest",
+            "layout": [],
             "max_category_items": 100,
             "obs_names": None,
             "var_names": None,
             "diffexp_lfc_cutoff": 0.01,
         }
 
-    def _alias_annotation_names(self, axis, name):
-        """
-        Do all user-specified annotation aliasing.
+    @staticmethod
+    def _create_unique_column_name(df, col_name_prefix):
+        """ given the columns of a dataframe, and a name prefix, return a column name which
+            does not exist in the dataframe, AND which is prefixed by `prefix`
 
-        As a *critical* side-effect, ensure the indices are simple number ranges
-        (accomplished by calling pandas.DataFrame.reset_index())
+            The approach is to append a numeric suffix, starting at zero and increasing by
+            one, until an unused name is found (eg, prefix_0, prefix_1, ...).
         """
-        if name == "name":
-            # a noop, so skip it
-            return
+        suffix = 0
+        while f"{col_name_prefix}{suffix}" in df:
+            suffix += 1
+        return f"{col_name_prefix}{suffix}"
 
-        ax_name = str(axis)
-        df_axis = getattr(self.data, ax_name)
-        if name is None:
-            # reset index to simple range; alias "name" to point at the
-            # previously specified index.
-            df_axis.reset_index(inplace=True)
-            df_axis.rename(inplace=True, columns={"index": "name"})
-        elif name in df_axis.columns:
-            if name not in df_axis.columns:
+    def _alias_annotation_names(self):
+        """
+        The front-end relies on the existance of a unique, human-readable
+        index for obs & var (eg, var is typically gene name, obs the cell name).
+        The user can specify these via the --obs-names and --var-names config.
+        If they are not specified, use the existing index to create them, giving
+        the resulting column a unique name (eg, "name").
+
+        In both cases, enforce that the result is unique, and communicate the
+        index column name to the front-end via the obs_names and var_names config
+        (which is incorporated into the schema).
+        """
+        for (ax_name, config_name) in ((Axis.OBS, "obs_names"), (Axis.VAR, "var_names")):
+            name = self.config[config_name]
+            df_axis = getattr(self.data, str(ax_name))
+            if name is None:
+                # Default: create unique names from index
+                if not df_axis.index.is_unique:
+                    raise KeyError(
+                        f"Values in {ax_name}.index must be unique. "
+                        "Please prepare data to contain unique index values, or specify an "
+                        "alternative with --{ax_name}-name."
+                    )
+                name = self._create_unique_column_name(df_axis.columns, "name_")
+                self.config[config_name] = name
+                # reset index to simple range; alias name to point at the
+                # previously specified index.
+                df_axis.rename_axis(name, inplace=True)
+                df_axis.reset_index(inplace=True)
+            elif name in df_axis.columns:
+                # User has specified alternative column for unique names, and it exists
+                if not df_axis[name].is_unique:
+                    raise KeyError(
+                        f"Values in {ax_name}.{name} must be unique. "
+                        "Please prepare data to contain unique values."
+                    )
+                df_axis.reset_index(drop=True, inplace=True)
+            else:
+                # user specified a non-existent column name
                 raise KeyError(
                     f"Annotation name {name}, specified in --{ax_name}-name does not exist."
                 )
-            if not df_axis[name].is_unique:
-                raise KeyError(
-                    f"Values in -{ax_name}-name must be unique. "
-                    "Please prepare data to contain unique values."
-                )
-            # reset index to simple range; alias user-specified annotation to "name"
-            df_axis.reset_index(drop=True, inplace=True)
-            df_axis.rename(inplace=True, columns={name: "name"})
-        else:
-            raise KeyError(
-                f"Annotation name {name}, specified in --{ax_name}_name does not exist."
-            )
 
     @staticmethod
     def _can_cast_to_float32(ann):
@@ -113,7 +133,17 @@ class ScanpyEngine(CXGDriver):
                 "nVar": self.gene_count,
                 "type": str(self.data.X.dtype),
             },
-            "annotations": {"obs": [], "var": []},
+            "annotations": {
+                "obs": {
+                    "index": self.config["obs_names"],
+                    "columns": []
+                },
+                "var": {
+                    "index": self.config["var_names"],
+                    "columns": []
+                }
+            },
+            "layout": {"obs": []}
         }
         for ax in Axis:
             curr_axis = getattr(self.data, str(ax))
@@ -137,14 +167,21 @@ class ScanpyEngine(CXGDriver):
                     raise TypeError(
                         f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene."
                     )
-                self.schema["annotations"][ax].append(ann_schema)
+                self.schema["annotations"][ax]["columns"].append(ann_schema)
+
+        for layout in self.config['layout']:
+            layout_schema = {
+                "name": layout,
+                "type": "float32",
+                "dims": [f"{layout}_0", f"{layout}_1"]
+            }
+            self.schema["layout"]["obs"].append(layout_schema)
 
     def _load_data(self, data):
-        # Based on benchmarking, cache=True has no impact on perf.
-        # Note: as of current scanpy/anndata release, setting backed='r' will
-        # result in an error.  https://github.com/theislab/anndata/issues/79
+        # as of AnnData 0.6.19, backed mode performs initial load fast, but at the
+        # cost of significantly slower access to X data.
         try:
-            self.data = sc.read(data, cache=True)
+            self.data = anndata.read_h5ad(data)
         except ValueError:
             raise ScanpyFileError(
                 "File must be in the .h5ad format. Please read "
@@ -164,16 +201,68 @@ class ScanpyEngine(CXGDriver):
 
     @requires_data
     def _validate_and_initialize(self):
-        self._alias_annotation_names(Axis.OBS, self.config["obs_names"])
-        self._alias_annotation_names(Axis.VAR, self.config["var_names"])
+        # var and obs column names must be unique
+        if not self.data.obs.columns.is_unique or not self.data.var.columns.is_unique:
+            raise KeyError(f"All annotation column names must be unique.")
+
+        self._alias_annotation_names()
         self._validate_data_types()
-        self._validate_data_calculations()
         self.cell_count = self.data.shape[0]
         self.gene_count = self.data.shape[1]
+        self._default_and_validate_layouts()
         self._create_schema()
 
+    @requires_data
+    def _default_and_validate_layouts(self):
+        """ function:
+            a) generate list of default layouts, if not already user specified
+            b) validate layouts are legal.  remove/warn on any that are not
+            c) cap total list of layouts at global const MAX_LAYOUTS
+        """
+        layouts = self.config['layout']
+        # handle default
+        if layouts is None or len(layouts) == 0:
+            # load default layouts from the data.
+            layouts = [key[2:] for key in self.data.obsm_keys() if type(key) == str and key.startswith("X_")]
+            if len(layouts) == 0:
+                raise PrepareError(f"Unable to find any precomputed layouts within the dataset.")
+
+        # remove invalid layouts
+        valid_layouts = []
+        obsm_keys = self.data.obsm_keys()
+        for layout in layouts:
+            layout_name = f"X_{layout}"
+            if layout_name not in obsm_keys:
+                warnings.warn(f"Ignoring unknown layout name: {layout}.")
+            elif not self._is_valid_layout(self.data.obsm[layout_name]):
+                warnings.warn(f"Ignoring layout due to malformed shape or data type: {layout}")
+            else:
+                valid_layouts.append(layout)
+
+        if len(valid_layouts) == 0:
+            raise PrepareError(f"No valid layout data.")
+
+        # cap layouts to MAX_LAYOUTS
+        self.config['layout'] = valid_layouts[0:MAX_LAYOUTS]
+
+    @requires_data
+    def _is_valid_layout(self, arr):
+        """ return True if this layout data is a valid array for front-end presentation:
+            * ndarray, with shape (n_obs, >= 2), dtype float/int/uint
+            * contains only finite values
+        """
+        is_valid = type(arr) == np.ndarray and arr.dtype.kind in "fiu"
+        is_valid = is_valid and arr.shape[0] == self.data.n_obs and arr.shape[1] >= 2
+        is_valid = is_valid and np.all(np.isfinite(arr))
+        return is_valid
+
     @requires_data
     def _validate_data_types(self):
+        if sparse.isspmatrix(self.data.X) and not sparse.isspmatrix_csc(self.data.X):
+            warnings.warn(
+                f"Scanpy data matrix is sparse, but not a CSC (columnar) matrix.  "
+                f"Performance may be improved by using CSC."
+            )
         if self.data.X.dtype != "float32":
             warnings.warn(
                 f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
@@ -204,20 +293,6 @@ class ScanpyEngine(CXGDriver):
                             f"annotations with more than 500 categories in the UI"
                         )
 
-    @requires_data
-    def _validate_data_calculations(self):
-        layout_key = f"X_{self.config['layout']}"
-        try:
-            assert layout_key in self.data.obsm_keys()
-        except AssertionError:
-            raise PrepareError(
-                f"Cannot find a field with coordinates for the {self.config['layout']} layout requested. A different"
-                f" layout may have been computed. The requested layout must be pre-calculated and saved "
-                f"back in the h5ad file. You can run "
-                f"`cellxgene prepare --layout {self.config['layout']} <datafile>` "
-                f"to solve this problem. "
-            )
-
     @staticmethod
     def _annotation_filter_to_mask(filter, d_axis, count):
         mask = np.ones((count,), dtype=bool)
@@ -304,7 +379,7 @@ class ScanpyEngine(CXGDriver):
         if sparse.issparse(X):  # use tuned getcol/hstack for performance
             indices = np.nonzero(var_mask)[0]
             cols = [X.getcol(i) for i in indices]
-            return sparse.hstack(cols)
+            return sparse.hstack(cols, format="csc")
         else:   # else, just use standard slicing, which is fine for dense arrays
             return X[:, var_mask]
 
@@ -366,17 +441,33 @@ class ScanpyEngine(CXGDriver):
         Caveats:
         * does not support filtering
         * only returns Matrix in columnar layout
+
+        All embeddings must be individually centered & scaled (isotropically)
+        to a [0, 1] range.
         """
         try:
-            full_embedding = self.data.obsm[f"X_{self.config['layout']}"]
-            if full_embedding.shape[1] > 2:
-                warnings.warn(f"Warning: found {full_embedding.shape[1]} \
-                components of embedding. Using the first two for layout display.")
-            df_layout = full_embedding[:, :2]
+            layout_data = []
+            for layout in self.config["layout"]:
+                full_embedding = self.data.obsm[f"X_{layout}"]
+                embedding = full_embedding[:, :2]
+
+                # scale isotropically
+                min = embedding.min(axis=0)
+                max = embedding.max(axis=0)
+                scale = np.amax(max - min)
+                normalized_layout = (embedding - min) / scale
+
+                # translate to center on both axis
+                translate = 0.5 - ((max - min) / scale / 2)
+                normalized_layout = normalized_layout + translate
+
+                normalized_layout = normalized_layout.astype(dtype=np.float32)
+                layout_data.append(pandas.DataFrame(normalized_layout, columns=[f"{layout}_0", f"{layout}_1"]))
+
         except ValueError as e:
             raise PrepareError(
                 f"Layout has not been calculated using {self.config['layout']}, "
                 f"please prepare your datafile and relaunch cellxgene") from e
 
-        normalized_layout = (df_layout - df_layout.min()) / (df_layout.max() - df_layout.min())
-        return encode_matrix_fbs(normalized_layout.astype(dtype=np.float32), col_idx=None, row_idx=None)
+        df = pandas.concat(layout_data, axis=1, copy=False)
+        return encode_matrix_fbs(df, col_idx=df.columns, row_idx=None)

server/app/util/constants.py

@@ -31,3 +31,5 @@ JSON_NaN_to_num_warning_msg = (
     "JSON encoding failure - please verify all data are finite values (no NaN or Infinities)"
 )
 REACTIVE_LIMIT = 1_000_000
+
+MAX_LAYOUTS = 30

server/cli/cli.py

@@ -5,7 +5,7 @@ from .prepare import prepare
 
 
 @click.group(name="cellxgene", context_settings=dict(max_content_width=85))
-@click.version_option(version="0.9.1", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
+@click.version_option(version="0.11.0", prog_name="cellxgene", message="[%(prog)s] Version %(version)s")
 def cli():
     pass
 

server/cli/launch.py

@@ -1,6 +1,8 @@
+import errno
+import functools
 import logging
 from os import devnull
-from os.path import splitext, basename
+from os.path import splitext, basename, getsize
 import sys
 import warnings
 import webbrowser
@@ -10,28 +12,58 @@ import click
 from server.app.app import Server
 from server.app.util.errors import ScanpyFileError
 from server.app.util.utils import custom_format_warning
-from server.utils.constants import MODES
+from server.utils.utils import find_available_port, is_port_available
+
+# anything bigger than this will generate a special message
+BIG_FILE_SIZE_THRESHOLD = 100 * 2 ** 20  # 100MB
+
+
+def common_args(func):
+    """
+    Decorator to contain CLI args that will be common to both CLI and GUI: title and engine args.
+    """
+    @click.option("--title", "-t", help="Title to display (if omitted will use file name).")
+    @click.option(
+        "--layout",
+        "-l",
+        default=[],
+        multiple=True,
+        show_default=True,
+        help="Layout name, eg, 'umap'."
+    )
+    @click.option("--obs-names", default=None, metavar="", help="Name of annotation field to use for observations.")
+    @click.option("--var-names", default=None, metavar="", help="Name of annotation to use for variables.")
+    @click.option(
+        "--max-category-items",
+        default=1000,
+        metavar="",
+        show_default=True,
+        help="Categories with more distinct values than this will not be displayed.",
+    )
+    @click.option(
+        "--diffexp-lfc-cutoff",
+        default=0.01,
+        show_default=True,
+        help="Relative expression cutoff used when selecting top N differentially expressed genes",
+    )
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        return func(*args, **kwargs)
+    return wrapper
+
+
+def parse_engine_args(layout, obs_names, var_names, max_category_items, diffexp_lfc_cutoff):
+    return {
+        "layout": layout,
+        "max_category_items": max_category_items,
+        "diffexp_lfc_cutoff": diffexp_lfc_cutoff,
+        "obs_names": obs_names,
+        "var_names": var_names,
+    }
 
 
 @click.command()
 @click.argument("data", metavar="<data file>", type=click.Path(exists=True, file_okay=True, dir_okay=False))
-@click.option(
-    "--layout",
-    "-l",
-    type=click.Choice(MODES),
-    default="umap",
-    show_default=True,
-    help="Method for layout."
-)
-@click.option(
-    "--diffexp",
-    "-d",
-    type=click.Choice(["ttest"]),
-    default="ttest",
-    show_default=True,
-    help="Method for differential expression.",
-)
-@click.option("--title", "-t", help="Title to display (if omitted will use file name).", metavar="")
 @click.option(
     "--verbose",
     "-v",
@@ -50,23 +82,9 @@ from server.utils.constants import MODES
     show_default=True,
     help="Open the web browser after launch.",
 )
-@click.option("--port", "-p", help="Port to run server on.", metavar="", default=5005, show_default=True)
-@click.option("--obs-names", default=None, metavar="", help="Name of annotation field to use for observations.")
-@click.option("--var-names", default=None, metavar="", help="Name of annotation to use for variables.")
+@click.option("--port", "-p", help="Port to run server on, if not specified cellxgene will find an available port.",
+              metavar="", show_default=True)
 @click.option("--host", default="127.0.0.1", help="Host IP address")
-@click.option(
-    "--max-category-items",
-    default=100,
-    metavar="",
-    show_default=True,
-    help="Limits the number of categorical annotation items displayed.",
-)
-@click.option(
-    "--diffexp-lfc-cutoff",
-    default=0.01,
-    show_default=True,
-    help="Relative expression cutoff used when selecting top N differentially expressed genes",
-)
 @click.option(
     "--scripts",
     default=[],
@@ -74,21 +92,21 @@ from server.utils.constants import MODES
     help="Additional script files to include in html page",
     show_default=True,
 )
+@common_args
 def launch(
         data,
-        layout,
-        diffexp,
-        title,
         verbose,
         debug,
-        obs_names,
-        var_names,
         open_browser,
         port,
         host,
+        layout,
+        obs_names,
+        var_names,
         max_category_items,
         diffexp_lfc_cutoff,
-        scripts,
+        title,
+        scripts
 ):
     """Launch the cellxgene data viewer.
     This web app lets you explore single-cell expression data.
@@ -101,6 +119,7 @@ def launch(
 
     > cellxgene launch <your data file> --title <your title>"""
 
+    e_args = parse_engine_args(layout, obs_names, var_names, max_category_items, diffexp_lfc_cutoff)
     # Startup message
     click.echo("[cellxgene] Starting the CLI...")
 
@@ -115,26 +134,36 @@ def launch(
     else:
         warnings.formatwarning = custom_format_warning
 
-    if scripts:
-        click.echo(r"""
-/ / /\ \ \__ _ _ __ _ __ (_)_ __   __ _
-\ \/  \/ / _` | '__| '_ \| | '_ \ / _` |
- \  /\  / (_| | |  | | | | | | | | (_| |
-  \/  \/ \__,_|_|  |_| |_|_|_| |_|\__, |
-                                  |___/
-The --scripts flag is intended for developers to include google analytics etc. You could be opening yourself to a
-security risk by including the --scripts flag. Make sure you trust the scripts that you are including.
-        """)
-        scripts_pretty = ", ".join(scripts)
-        click.confirm(f"Are you sure you want to inject these scripts: {scripts_pretty}?", abort=True)
-
     if not verbose:
         sys.tracebacklimit = 0
 
+    if scripts:
+        click.echo(r"""
+    / / /\ \ \__ _ _ __ _ __ (_)_ __   __ _
+    \ \/  \/ / _` | '__| '_ \| | '_ \ / _` |
+     \  /\  / (_| | |  | | | | | | | | (_| |
+      \/  \/ \__,_|_|  |_| |_|_|_| |_|\__, |
+                                      |___/
+    The --scripts flag is intended for developers to include google analytics etc. You could be opening yourself to a
+    security risk by including the --scripts flag. Make sure you trust the scripts that you are including.
+            """)
+        scripts_pretty = ", ".join(scripts)
+        click.confirm(f"Are you sure you want to inject these scripts: {scripts_pretty}?", abort=True)
+
     if not title:
         file_parts = splitext(basename(data))
         title = file_parts[0]
 
+    if port:
+        if debug:
+            raise click.ClickException("--port and --debug may not be used together (try --verbose for error logging).")
+        if not is_port_available(host, int(port)):
+            raise click.ClickException(
+                f"The port selected {port} is in use, please specify an open port using the --port flag."
+            )
+    else:
+        port = find_available_port(host)
+
     # Setup app
     cellxgene_url = f"http://{host}:{port}"
 
@@ -148,7 +177,13 @@ security risk by including the --scripts flag. Make sure you trust the scripts t
         log = logging.getLogger("werkzeug")
         log.setLevel(logging.ERROR)
 
-    click.echo(f"[cellxgene] Loading data from {basename(data)}, this may take awhile...")
+    file_size = getsize(data)
+
+    # if a big file, let the user know it may take a while to load.
+    if file_size > BIG_FILE_SIZE_THRESHOLD:
+        click.echo(f"[cellxgene] Loading data from {basename(data)}, this may take awhile...")
+    else:
+        click.echo(f"[cellxgene] Loading data from {basename(data)}.")
 
     # Fix for anaconda python. matplotlib typically expects python to be installed as a framework TKAgg is usually
     # available and fixes this issue. See https://matplotlib.org/faq/virtualenv_faq.html
@@ -157,17 +192,8 @@ security risk by including the --scripts flag. Make sure you trust the scripts t
     mpl.use("TkAgg")
     from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
 
-    args = {
-        "layout": layout,
-        "diffexp": diffexp,
-        "max_category_items": max_category_items,
-        "diffexp_lfc_cutoff": diffexp_lfc_cutoff,
-        "obs_names": obs_names,
-        "var_names": var_names,
-    }
-
     try:
-        server.attach_data(ScanpyEngine(data, args), title=title)
+        server.attach_data(ScanpyEngine(data, e_args), title=title)
     except ScanpyFileError as e:
         raise click.ClickException(f"{e}")
 
@@ -183,4 +209,9 @@ security risk by including the --scripts flag. Make sure you trust the scripts t
         f = open(devnull, "w")
         sys.stdout = f
 
-    server.app.run(host=host, debug=debug, port=port, threaded=True)
+    try:
+        server.app.run(host=host, debug=debug, port=port, threaded=True, use_debugger=False)
+    except OSError as e:
+        if e.errno == errno.EADDRINUSE:
+            raise click.ClickException("Port is in use, please specify an open port using the --port flag.") from e
+        raise

server/gui/browser.py

@@ -11,10 +11,6 @@ WindowUtils = cef.WindowUtils()
 # OS differences
 # noinspection PyUnresolvedReferences
 CefWidgetParent = QWidget
-if LINUX:
-    # noinspection PyUnresolvedReferences
-    CefWidgetParent = QX11EmbedContainer
-
 
 class CefWidget(CefWidgetParent):
     def __init__(self, parent=None):

server/gui/cellxgene.qrc

@@ -0,0 +1,8 @@
+<!DOCTYPE RCC><RCC version="1.0">
+<qresource>
+    <file alias="logo.png">images/cellxgene_logo.png</file>
+    <file alias="collapsed.svg">images/properties_contract.svg</file>
+    <file alias="expanded.svg">images/properties_expand.svg</file>
+    <file alias="icon.png">images/properties_expand.svg</file>
+</qresource>
+</RCC>

server/gui/cellxgene.spec

@@ -0,0 +1,29 @@
+# -*- mode: python -*-
+
+block_cipher = None
+
+
+a = Analysis(['main.py'],
+             pathex=['/Users/charlotteweaver/Documents/Git/cellxgene/server/gui'],
+             hookspath=["/Users/charlotteweaver/Documents/Git/cellxgene/server/gui/"],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+
+exe = EXE(pyz,
+          a.scripts,
+          a.binaries,
+          a.zipfiles,
+          a.datas,
+          [],
+          name='cellxgene',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          runtime_tmpdir=None,
+          console=True )

server/gui/cellxgene_rc.py

@@ -0,0 +1,440 @@
+# -*- coding: utf-8 -*-
+
+# Resource object code
+#
+# Created: Wed Jun 19 15:02:04 2019
+#      by: The Resource Compiler for PySide2 (Qt v5.12.3)
+#
+# WARNING! All changes made in this file will be lost!
+
+from PySide2 import QtCore
+
+qt_resource_data = b"\
+\x00\x00\x0d\xde\
+\x89\
+PNG\x0d\x0a\x1a\x0a\x00\x00\x00\x0dIHDR\x00\
+\x00\x01;\x00\x00\x00j\x08\x06\x00\x00\x00\xc3y\xf6!\
+\x00\x00\x00\x09pHYs\x00\x00\x0b\x13\x00\x00\x0b\x13\
+\x01\x00\x9a\x9c\x18\x00\x00\x00\x01sRGB\x00\xae\xce\
+\x1c\xe9\x00\x00\x00\x04gAMA\x00\x00\xb1\x8f\x0b\xfc\
+a\x05\x00\x00\x0dsIDATx\x01\xed\xdd\x7fn\
+\x1b\xc7\x15\x07\xf07C\xdbQ\x93\x00f\xfe\x88e7\
+\x05L\x9d J/PJ9@\xe4^@\x12\x90H\
+(\x0aT\xce\x09,\x9d\xc0\xf2\x1fE \xb9\x80\xd9\x0b\
+\xd4\xf2\x01,2'\xb0|\x023@\x0bG\xea\x1fQ\
+\xff\x09\x04\xd1;\xaf\xef-\xc9X\xa48\xb3?\xb8\xfc\
+e~?\x80b\x85\xbb\xab%g\xdf\xbe\x9d\x9d\x99\x1d\
+\x12\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\xc0\x9c24\x22\xa7\xab\x87L3l\xf1xk\
+de\x03\x00\xe3g\x09\x00`\x0e \xd9\x01\xc0\x5c@\
+\xb2\x03\x80\xb9\x80d\x07\x00s\x01\xc9\x0e\x00\xe6\xc2\x0d\
+\x02\x18\xe0m\xf5i\xd5Z\xae\xfb\x96KW{\xed\xee\
+\xf1\xd6f\xef6?T\xac-\xbd!\xbf\xa6\xf4r/\
+\x11\xc0\x04\xccD\xb2\xbb\xf3\xf2;\x1a\x85\xb3\xaf\x9f\x12\
+\x00\xcc\x07\xdc\xc6\x02\xc0\x5c@\xb2\x03\x80\xb9\x80d\x07\
+\x00s\x01\xc9\x0e\x00\xe6\x02\x92\x1d\x00\xcc\x05$;\x00\
+\x98\x0b31\xf4\x04CD\x00`X\xa8\xd9\x01\xc0\x5c\
+\xc0\x13\x14c\xf0\xa6\xfa\xac\xfcI\xa9\xb5\xee\x98\x96\xad\
+\xa1\x0a3W\x88\xcc\xb91t.\xbf\xcb\xbf\xa6\x11E\
+\xd1\x8b{\x8d\xbf4i\x08\xfa\xd4\x831\xbc&?\xf7\
+\xe5\xefW\x88\xb8,\x7f\xbb\xa9\xcb\x9c\xa3\xd7\xcc\xe6\xe8\
+^\xe3\xbb\x06\xc1o\xbaef-}\xa9\xc7E\xcbk\
+\xd019\xad\x1e\xac9k\xca\x83\xfeF\xc9\xb5N\xee\
+4\xfez\x92f\x7f\x1a\x0b\xb7JQ\xb5\xe4\x5c\xb5\xff\
+8\xe9~\x1d\xd1kr\xb6\x91\xf68\xbd]=\xdc\xf0\
+.tQ\xe3jL\xf5\xc6\x07-wc0b\xfeQ\
+\x02\xa46L\xfc%\xc5^dm\xe3\x8b\x97\xdf\x1e\xd1\
+\x04a\xf2N\x8f\x22&\xef\x8c\x1f\x9f2v\x97\x8cY\
+O\xb3\xbe\x04\x9e\x9e`\x9bY\x82NO\x9eO\xed\xbb\
+\x1d&~(\x05^N\xb1I\xd3\x92\xa9}~\xfc\xdd\
+^h\xa5\x0f\xfdq\xb1\xff|}\xb0v\x83\xe9q\xfb\
+\xc4\xf4\x93\xb2\xda\xd5\xb2\x92x~Eq\x82\xb8\xce\x11\
+\xef\xdd;\xde\xde\x0d\xfd\x9d<\xc7\x89$v\x16_~\
+\xfb\xcf\xd0J\xa1\xf3L\xf6\xf5\xfd\xdd\xe3\xed\xfd\xf6\xb1\
+t\xcf\x92?+\xef\x7f~\xbc\xfd=e\xf0\xdf\xd5\xa7\
+\x8f\x8a\x8e\xbdQ\xc1m\xec\x88\x9c}}\xb8S\xb2\xa5\
+Wi\x13\x9db\xa6\xaa&\x0b\x0d\xa04\xebkr\xf9\
+\xd8^\xbe\x92\x93m7e\xb0\xa9\x8a\xae/'\xc9\x1b\
+\xdd\x9e\xa6\x88&\x04\xca)\xcb\xb6g\xabO\x1f\xdf`\
+\xf3<\xe9\xe4WZVg\xab\x87\xcf\x0cS\xe2\xba>\
+\x9al>\xb1\xad7Y\x8f\x13\xb1\xab\xe9\xbe\xf3\x96\x8b\
+\xeeK\xe3\xb0}\xd12\x95\xa4\xf5\x1d\x99\x87\xb2~\x9d\
+R\xd0\xd8\xd1\x0b\xc0,\xc5\xdeDncC\xcf\xba\x0e\
+\xea\x8c\x98\xb5gcO\xf5j\xc7\xbcK9i@H\
+\xc2\xa3\xd0\x15\xf0\xac\xfa\xf7e\xb2\xa5z\x86@\xeb\xa7\
+\xb5\xb0\xba\x04\xdd\xca\xb0\xb7\xcfE\xd0\x93\x92\xb9\xb5{\
+V=\x5c\xb9\xd3\xd8JuK\xd8\xd5\xaeQ^\xd6%\
+14\xee\xf4\xd56\xfb\x9dvj\x22\x94\x81\x94\xf1F\
+\xde{\xa0\xd3\xea?\xd6\xc9\xbaZ\xde\xdb\x1c\xdd\xf7\xc7\
+\xb6\xb5,\x09oe\xa9\xb1y\x9ee[\xa9E\xads\
+\xc6$\xad\x17\xdc\x9fW\x0f\x1ej\x8d\xd0\xb7N\xbb\xbc\
+\xad&\xc5\x0a\xe53\x91\xd8C\xcd\xae`oW\x7f\xd8\
+\x90\x90\xd9\xa5!i\xc2\x93`\xa8\x0e\xdc\x87\x04\x1b\xdb\
+\x1b\xcf\x87Ht]\x1at\xcfi\xc24!\xc8I\xa6\
+'W\x99-\xd5%\xe1-\xa7\xdd\xf6\xfd\x89g*\x9a\
+\x18B\xb5\xe2\xa2\x8eM\xfa\xf7\xf6\xb4\xaa\x89\x8e\x86\xb7\
+\xfc\xa9\xbdLU\xdb\xefS\xa1\x1c\x0c\x19\x7f\x19^)\
+o\x1a\x8e\xc6\xde\xf10\xb5\xf9\xac\x90\xec\x0a\x14\x07\x02\
+\xd9<A9P\xa9T\x1a\xf8\xb7\xe2v@2\x15*\
+\xc6\xf2\xe9\xea\xc1.MH\xbb\x86\xda\x93\x10R'\xbc\
+A'\x9e^$\xb4=n\xd0\xfaE\x1e\x9b4\xda\xed\
+d\xc5\xd0[L\xdf\xc5o\x04\xca\xbe}\x15\x1c{K\
+\xd2\x0c\x93\xa9\x96=\x0c$\xbb\x22Y\xaa\x86\x02A\xda\
+}\xce\xb51\x9b\x1c?\xd0\x1f\xc7\x1cl|\xd6[\x8a\
+\xfe\xa0\x8bk\x0b\x09\xed\x80\xb2]\xc3Q\xb4\xe9\x5c\xb4\
+\x92f?\xf2\xcev\xc6y\x85\xbdJ{1\xe32\xe9\
+\x95\x98\xf0\xbc5\x0c\xf9\xac_\xbc\xdc\xbe\xd6\xeb\xd7\xae\
+\xd5\xf5\xad\xdb\xe7\xea\xf1\xd1\xb2\x8b\x7fgnR\x0ei\
+\xf6w\xf58\xc9\xcf\xa6\xdc\xb3\x06o\xdf}\x17\xbf$\
+\xda\x99\x94-\x1e\xe4\x9d[{\xad\xec\xe3v\xb6\xa46\
+h\xf9\x0c\xd2\xd1\xf60\xed\xbe\x98\xcc\xdf\xc6\x15{\x18\
+zR\xa0\xc4\x9a\x03\xb7V\xee\xd5{\x86(\x1c\xbd]\
+=\xd0\x1e*\xefvrBW\xe5\x9f\xc6o\xffo\xdc\
+F\xb8\x13\x9d\x8f\xee\xd6\xb7\x1f\xf4\xbd\x98\xb4\x9f\xf2\x82\
+mim\xa8F\x13\xa0=\x99\xf2\xfe\xa8\xef\xfdu\x13\
+\xde\xb56\xbcP\xa2[\xacoo\x0c\xdaG\x89K\xeb\
+\x1c,6jF\x1c\xc9\xf1\xe9iCj\xc8\xbejr\
+\x5c\xebr\x92W(\x83\xe4Z$\xef\xc9q\xda\xed{\
+\xb1\xf6\xf3\xca\xe1\xbe$\x8b\x9d\x81[\xb4/~\x95,\
+\xed\x5c\xce\x91\xf4\xeeo\xd5\xfa^>\x92\xda\xfcm)\
+\xbf5\xff\x96\xe6\xcb\xfeW\xda\xb5\xba\x10>Z\xcc\x18\
+{rH\xca\xbf\xb3\x97\x1b\xf2\xab\xb7\x8d\xb0(3[\
+\xb3\xd3\xce\x85,?\xa3\xf6\xef\xb8\x16b*\xbe\xe5\xcc\
+\xee\xc5\xa0\xb1Xz\xa2k\x8d\xc2\xbf\x1d\xf7^a\x13\
+\xae\xac\xce\xb9\x81C\x07\xdaC#\xfc\xb5\x14\x09\xba?\
+\xd1\x04\xe9\xfbKS\xc3\xd3Z\xc0\xe0DG'\xbeD\
+\xf7\x8bl#\x89\xaeJ!lw\x07%\x11}\xcd\x19\
+J\xac\x09]\x95\x14\x0b\x9aX\x17=CU>\xe2\x9b\
+\xbb\x14 5\xae5J\xaf9 \xd1\xc5$N\x9e\x84\
+6\xd4\xf1\xa0\xfd\xafI[\xde7\xa1m\xc2\xb1\xe7\xad\
+\xb5\x1a\x09\xf2\xb1\xc4\x1enc\x0br\xc3r5\xb8\x82\
+1\x0d\xdf\xa2\x88\xf8\x85o\x99\x9c\xd8\xb7\xbb\xbf\xc7\xb7\
+\xb0\x01\x9aPCW}\xc7\xf4\xa3o\x99$\xbb*M\
+X(\xe1i\x8dF\x13\x9d\xf4L\x0eLt\xbf\xf2\xcd\
+\x15\xdf\xdf\xbd\xa0w\xe1\xf6?M>\x8do3%\xb4\
+\x90\xa4X`\xc3\xdeD\xf3Y\xdc\xe3\x1a\xbau\xbe^\
+\xe3\xca\xa9\x99ee\x8d=\xb9`xo7u\x8ch\
+8\xf6\xf8\xb5\x7f[[\xd4g\x0a\xc2mlA\xe4\x16\
+c\xd9\x04n\x93\xd8\xb9\x93\xc0\xc2\x9a\xd4\x1e\x1a\x83\x16\
+9\xb2\xbf\xd5\xfa\x8cu\xcb\xa1[Xc\xccIh\xfc\
+\x925\xe66\xf9Uh\x0a\xf8niu\xa8\xc2\xc7\xd4\
+\xd2\xb2\xe8M\x5c\x9dD\x17\x1a\x96\x91TnL\xce{\
+\x22\xe6!5\xa0`ME\xda\xa9\xce\xf3\x8e3\xb3L\
+\xa9{\xaa\x0be\xdfU\xa41\xc0\xbbX\x9f\x92\x18&\
+\xf6\xf4B\x96uhMVHv\x05)\x91\xb9\xcf\xc1\
+\xe5%\xef\x81\x94+b\x83\xd2`S\x09\x8f\xf72\x8f\
+$)\xe4\xeeq\x1cG\xc0\xa5\xe1Ix\x95k+\xa6\
+Ht\xed\xd5\xa8\x1c.6\x7f\xad;\x0fi\x96(\x87\
+\xda\x07\xe5v\xea\x19\xd9\x12\xe5\x11\xaa]\x8d\x96\xad\x84\
+\x96j;\xa31\xa5\x1d\xcai\x81.\xf4s\x8d4\xf6\
+p\x1b;&\x11EC\x1fH&\x1ei\xa0w\x02n\
+*xni\xdfK\x99\xe8:\xebV\xc2\x8bM\xa1'\
+\xd9\xe4\x12\xd2\xe8\x1863\xff\x99\x90\xec`j]\xb8\
+[\xfb\xbe\xce\x1by}s\x1aj\xa1\x83\xf1\x07\x97\xec\
+\xf8\x03\xf8LHv0\xb5>\xb5\x97\xcf|\xb5$\xe9\
+\xb4x\x9e\xb6\xdd\xcb\x14\x5cs\x83\xd9\x846\xbb\x82H\
+\x8f\xeaO6\xd0\xa0&mvC_\x19\x13OZ\xe6\
+\xa6\xac\xd3\xa4\x9c.haj\x92\x82>\x00\xef\x88B\
+\xc3,R?_)\xbd\x9f\xe7&\xd4\xb1Sp\xadE\
+n\xf9\x9al\x02\xb7\xce:\xf06g\xfb\x94td5\
+i\x12\x92\xf6;T\xec1\x8f#\xf6\x90\xec\x0a\x92\x94\
+\x88$\x11\xde'\x0a\x8f\x90O\xde\x89\x0eI\x08\xf5*\
+\x9a\x17\x8b\xf5\xad\x874\xe34\xd1\xc5\x0f\xdf_\xc5:\
+\xe5\x11\x9f\xf4\x0d\x84M\x99\xf0\x5c3\xd4\x93X\xe0p\
+\x8eX\xd2\x85O\x12\xd6\x93{\xc7\x83\xc7\xbfM\xafp\
+\x19\xb2t\xf2,&L\xc20i\xb8\x8d-\x08\x9b(\
+\x98\xc8\x22\xeb\xbe\xf2-{\xbbrP\xd3y\xc9\x06\xfd\
+\x5c\x9dr'8|\x85\xe2\x1e\xb1\xf5\xb4\x8f\xdeL\xea\
+\xf1\xb0$\xbeD\xe78ZY<\xde~@\xd7\x1f?\
+\xea&\xbc\x8a\xefoF\xae\x14.7\xa9A\x16Y\x1e\
+r\x1c\xc2\x8f}\xc9q\xa2Y\xe3\x06\x0f\x8d\xea\xcaR\
+\x86\x93\x8a=$\xbb\x82,\xb8\x85\xa3\xd0rc\xfcc\
+\xaf\xacI\xf7\xf4\x82\x0eQ\x09=m!\xcaif\xc7\
+h\xcf\x83\xd7\xaa\xa7\x9d7o\x5cB\x89\xae[s\x8b\
+\x9f\x92\xc8\x98\xf0\xfe\xd0\xd8:I*\xb7\xd0\x03\xe9Y\
+{\x22?r\xb7j\xa1\xe5\xfa\xd8W\x9a\xb2o\xcf\x19\
+w\xf0f\x8c\x13\x00x\xb5\xcb?\xf8\x9cp\xf9\x13\xdb\
+zL\x09\xba\x03\xc3'\x11{Hv\x05\xd1\x91\xefr\
+B5|\xcb}\x01~\x1a\xbff*\xbe\xed\xe2)\xb3\
+\xaf\xfe\x7f`\xf4\xbd\x8a'`\x94\xa41\xe8\xc4\xd7@\
+\xd3\x07\xd4;s\x91-\xc7\x13)\xea|kS M\
+\xa2\xeb\xca\x93\xf0\xa4\xfd/i2\x84G:\x8f[\xff\
+\xab:\x83\x8a\xcdX\x13K\x8a\x85\xf6\xfb\x89\xe7,|\
+4\xa8\x96\xa3\xaf\xe9\xb2x\xf2W2\xf14\x5c\xd30\
+\xd1jR\x19\xea\xf1\x93\xe4\xfc<\x14{:\xd9,M\
+(\xf6f\xb6\xcdnT\x13z\x0e#\xe2h\xcf\x9aR\
+\xd5\xb7\xbc3K\xeb\x86\x84\xc5\x89\x8e\xed\x92+\xcdZ\
+b\x97\xbe\xeb\x9d\x0fm\xc1\xdd\xdao\x99\xd6Nh,\
+\x97\x06\x9d\x9c \x1br\x0b\xdcp\xdcnX\xb6:\xd6\
+\xcc\xb4\x96\xb9\xbf\xa3\xc4\xba\xfd\xb3\xea\xe1\xeb\xac\x13f\
+\x16)K\xa2\xeb\xd2\x84w\xbar\xd0\xff\xac\xb0\xb7\x0d\
+\x8f9:J\x1a\xf4*\xed\xae\x8f\xe5\xf8\xecpg\xe2\
+\x05\xe9\xb8\xa8\xea@\xee\xd0`q\x9f\xa4XP\x1a\x0f\
+R\x1b\xd2\xd9\x81O\xba\xc7\xc9\xc4O\x88\xb4$\x19\xf4\
+(O\xc3D\xab\x1a{\x97\xe6r=<)\x82Y\x93\
+\xf7\xba\x962\xf6\x1e\x8f3\xf6P\xb3+P\xe76\xb3\
+\x91\xb0ZE\x03B\x82z#i\xf2M\x9d\x9a\xa7?\
+\xb8\xb5\xd6\xc0\xcc\xa9\x1a\x82\xb56\xa9\xfb\x89\xf7e\xa8\
+\xea\xd9_9\xcb0\x8e\xa2\xc5\xfb\xe5\xbe^\xd7\x84D\
+\xd7\xe5\xab\xe1\x19\xba\xfe\xb0|\xcac\xd3\xd9\xbe]f\
+\xc1\x87\xf9\x13\xe8\xfe\xa4\xfc\x9f$\xad\xa7\xc7\xe4\xeaq\
+\x22\xf2>\x0e6\xf1\x89V5\xf6\x1c\xbb\x22c\xef3\
+\x89\xbd\x7f\x8d+\xf6\x90\xec\x0a&W\xf4\xcd\xbcs\xa0\
+\xf5\x90\x13\x9e]4\xf0\x09\x82\xc5\xc6\xf6Q\xf0\xe9\x82\
+\xec\xfb\x9a\x988\xa1qk\xe5\xca{I\x95\xe8\xba4\
+\xe1\xf5&1\xde\xbb\xdb\x18<\xa5\xb8\x1e\x9b\x84\xb6\xbb\
+B\xdd\xd5\x9e\xf1\x14s\xc7\xa5\xc6T\x9et\xc7\x92&\
+\xf1Y\x8d=$\xbb\x82\xc5S\x02\xb1[\x19*\xe1\xc9\
+\x09o\xb8\xf5 t\xc2'>N\x95a_Y\x92\xcb\
+(\xc4\x13x\xb6'\xaf\xcc\xf5^n\xf2\xcd\x07\xed\x89\
+/yo1\xf0-_\xdd\xc4:\xce\x84\xa7\xc98M\
+\x0d/\x91|>-\x9biyv\xb9\xa0\xd8{#q\
+\xfe\xe7q\xc5\x1e\x92\xdd\x08t\x13^\xca\xdb\xa6\x1e\xf1\
+\xec\xb5\x12\xd4i\xbe\x874\x0e:\x9d\x116gb\xd5\
+\x93\xf0W\xbe\xf9\xd54|\xe1\x8e\xbc\x87\xdab}k\
+)\xcf{\xd1\xdb+\xd9\xf6\xab\xc5\x84\xaf3Tq\xb9\
+jM2m\x99i\x0d{\xc8IM\xb5\x86\xd7N\xe6\
+\xd9\x8fSw\xf6d\xfd|\xd3p\x9c\xba:\xb1\xb7\x94\
+;\xf6\xa4L%\xf6\xfe\x98\xf6\xfbv\x8b0\x91\x0e\x8a\
+\xac\x93i\x8ec\xf2\xcd\xa2u\x02sE\xda#6J\
+&\x9e%\xb7\x1aZ_\x93\x5cd\xf9\xc9\x17\xc7\xdbG\
+\x94Ag\xc6\x94\xa5\xb4\xfb\xd1\x93\xe7\x9d\xf4\xaa\x19\x8e\
+\xf6\xa7\xe9\xe4\x19'=\xc1\xb4\xb1_\xae\xf5\xbbV'\
+-\x1d\xd0\xe0\xae\xe5\xa4=\xdf\x17|k\x7f\xc1\x5c>\
+4y\xbf^\xacC\x93\xb9\xfcSK{\x9ct\x98\x87\
+\xf6~\xea\xfe\x97\xea\xd3\xf9\x0cp'~\xf2\xc4\xdeQ\
+\xea\x99~\x0a4\xdc\x11\x0c\xc0\x97d\xf7\xd2\xb6\x96\x05\
+\xbaX\x96\xbat\xe5\xfdt9\xaei\x9d=\xff\x95n\
+5\x8a\xbc=\x89\xc7e\xc9~$\x00oG\xcc\x9f\x95\
+\x8c\xf9E\xda\xab\xfe\xa7\x03C\xe75\xc1\x85t\xcbK\
+\x8f\x8b\x96\x15G\xfc\xd3\xd5c\x22\xb1\xac\x1d\x03\xdeG\
+\xd7$)mf}\x22\x22\x14\x0f\x8e\xdc\xc9\xac\x1e'\
+-KK\xb6lJ\xe6\xfe\xb4\xc5\x1e\x92\x9dG\xd1\xc9\
+\x0e\xa6\x8b~\xabY\xda[(\x1d\xd8\x1b\xea\x99\xd5\xa6\
+\x84I\xd4T \x1b\xb4\xd9\xc1\xdc\xd1D\xc7\xf6\xe6\xab\
+4\xa3\xf8\x93\x06}\x8bs$\xba\xd9\x80\x89\x00`\xae\
+\xb4\xbf`\xdc\xc6\xe3\xd5tP\xef\xd9\xea\xe1Z\xcb\xd8\
+\xbd\xcb\xa8\xd4\xd3\x94\xa0\xdf\xb9\xd0\xfe&7\x0e\x7fm\
+%Q\xa66V\x98\x1c$;\x98+\xfd_\xf2,\xc9\
+j\xf9\x06\xbb\xe77\xac\xd3\xdb\xd5\xa6\xbe&\x9d\x11\xe5\
+\xf6\x93-\xc9-\x19\xbe\xb1\x900}\x90\xec`nt\
+\xbe\xb8:PS3\x15\xfdo\xfa\xc6f\xdeC\x87\xcf\
+\xec\x18Y\xb2C\x03?L\x1bv\xb6\x5c\x5c+ux\
+\x003L\x1ftP\xc0\xdc\xf8}c{\x7f\xd8\x91\xff\
+:VLnq\xbfG\xa2\x9b=\xa8}\xc1\xdc\x89\x1f\
+<\x97\xb6;kL\xea)\x860\x18{\xf6!\xd9\xc1\
+\xdc\xea\xcc\xb6Q\xb5\xd6~\xa3S9\xc9\xc9Py?\
+u\x167\xe5\xb5\xf3\x88\xe8Gc\xa2\x93\x0b\xb7p4\
+\xbd\xdff\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x01\xff\x07\xb4<2M\x93'\x06\
+\xcb\x00\x00\x00\x00IEND\xaeB`\x82\
+\x00\x00\x03X\
+<\
+?xml version=\x221.\
+0\x22 encoding=\x22utf\
+-8\x22?>\x0a<!-- Gener\
+ator: Adobe Illu\
+strator 23.0.1, \
+SVG Export Plug-\
+In . SVG Version\
+: 6.00 Build 0) \
+ -->\x0a<svg versio\
+n=\x221.1\x22 id=\x22Laye\
+r_1\x22 xmlns=\x22http\
+://www.w3.org/20\
+00/svg\x22 xmlns:xl\
+ink=\x22http://www.\
+w3.org/1999/xlin\
+k\x22 x=\x220px\x22 y=\x220p\
+x\x22\x0a\x09 viewBox=\x220 \
+0 100 100\x22 style\
+=\x22enable-backgro\
+und:new 0 0 100 \
+100\x22 xml:space=\x22\
+preserve\x22>\x0a<styl\
+e type=\x22text/css\
+\x22>\x0a    path{fill\
+:rgb(150, 146, 1\
+44)}\x0a    polygon\
+{fill:rgb(150, 1\
+46, 144)}\x0a    ci\
+rcle{fill:rgb(15\
+0, 146, 144)}\x0a  \
+  rect{fill:rgb(\
+150, 146, 144)}\x0a\
+</style><path d=\
+\x22M51.3,75.9c-1.9\
+,0-3.8-0.8-5-2.4\
+L18.1,38.4c-2.2-\
+2.8-1.8-6.9,1-9.\
+1c2.8-2.2,6.9-1.\
+8,9.1,1l28.2,35.\
+1c2.2,2.8,1.8,6.\
+9-1,9.1\x0a\x09C54.2,7\
+5.5,52.7,75.9,51\
+.3,75.9z\x22/>\x0a<pat\
+h d=\x22M51.3,75.9c\
+-1.4,0-2.9-0.5-4\
+-1.4c-2.8-2.2-3.\
+2-6.3-1-9.1l28.2\
+-35.1c2.2-2.8,6.\
+3-3.2,9.1-1c2.8,\
+2.2,3.2,6.3,1,9.\
+1L56.4,73.5\x0a\x09C55\
+.1,75.1,53.2,75.\
+9,51.3,75.9z\x22/>\x0a\
+</svg>\x0a\
+\x00\x00\x03Y\
+<\
+?xml version=\x221.\
+0\x22 encoding=\x22utf\
+-8\x22?>\x0a<!-- Gener\
+ator: Adobe Illu\
+strator 23.0.1, \
+SVG Export Plug-\
+In . SVG Version\
+: 6.00 Build 0) \
+ -->\x0a<svg versio\
+n=\x221.1\x22 id=\x22Laye\
+r_1\x22 xmlns=\x22http\
+://www.w3.org/20\
+00/svg\x22 xmlns:xl\
+ink=\x22http://www.\
+w3.org/1999/xlin\
+k\x22 x=\x220px\x22 y=\x220p\
+x\x22\x0a\x09 viewBox=\x220 \
+0 100 100\x22 style\
+=\x22enable-backgro\
+und:new 0 0 100 \
+100;\x22 xml:space=\
+\x22preserve\x22>\x0a<sty\
+le type=\x22text/cs\
+s\x22>\x0a    path{fil\
+l:rgb(150, 146, \
+144)}\x0a    polygo\
+n{fill:rgb(150, \
+146, 144)}\x0a    c\
+ircle{fill:rgb(1\
+50, 146, 144)}\x0a \
+   rect{fill:rgb\
+(150, 146, 144)}\
+\x0a</style><path d\
+=\x22M31.8,56.4c-1.\
+9,0-3.8-0.8-5-2.\
+4c-2.2-2.8-1.8-6\
+.9,1-9.1l35.1-28\
+.2c2.8-2.2,6.9-1\
+.8,9.1,1c2.2,2.8\
+,1.8,6.9-1,9.1L3\
+5.8,54.9\x0a\x09C34.6,\
+55.9,33.2,56.4,3\
+1.8,56.4z\x22/>\x0a<pa\
+th d=\x22M66.9,84.6\
+c-1.4,0-2.9-0.5-\
+4-1.4L27.7,54.9c\
+-2.8-2.2-3.2-6.3\
+-1-9.1c2.2-2.8,6\
+.3-3.2,9.1-1l35.\
+1,28.2c2.8,2.2,3\
+.2,6.3,1,9.1\x0a\x09C7\
+0.6,83.8,68.8,84\
+.6,66.9,84.6z\x22/>\
+\x0a</svg>\x0a\
+\x00\x00\x03X\
+<\
+?xml version=\x221.\
+0\x22 encoding=\x22utf\
+-8\x22?>\x0a<!-- Gener\
+ator: Adobe Illu\
+strator 23.0.1, \
+SVG Export Plug-\
+In . SVG Version\
+: 6.00 Build 0) \
+ -->\x0a<svg versio\
+n=\x221.1\x22 id=\x22Laye\
+r_1\x22 xmlns=\x22http\
+://www.w3.org/20\
+00/svg\x22 xmlns:xl\
+ink=\x22http://www.\
+w3.org/1999/xlin\
+k\x22 x=\x220px\x22 y=\x220p\
+x\x22\x0a\x09 viewBox=\x220 \
+0 100 100\x22 style\
+=\x22enable-backgro\
+und:new 0 0 100 \
+100\x22 xml:space=\x22\
+preserve\x22>\x0a<styl\
+e type=\x22text/css\
+\x22>\x0a    path{fill\
+:rgb(150, 146, 1\
+44)}\x0a    polygon\
+{fill:rgb(150, 1\
+46, 144)}\x0a    ci\
+rcle{fill:rgb(15\
+0, 146, 144)}\x0a  \
+  rect{fill:rgb(\
+150, 146, 144)}\x0a\
+</style><path d=\
+\x22M51.3,75.9c-1.9\
+,0-3.8-0.8-5-2.4\
+L18.1,38.4c-2.2-\
+2.8-1.8-6.9,1-9.\
+1c2.8-2.2,6.9-1.\
+8,9.1,1l28.2,35.\
+1c2.2,2.8,1.8,6.\
+9-1,9.1\x0a\x09C54.2,7\
+5.5,52.7,75.9,51\
+.3,75.9z\x22/>\x0a<pat\
+h d=\x22M51.3,75.9c\
+-1.4,0-2.9-0.5-4\
+-1.4c-2.8-2.2-3.\
+2-6.3-1-9.1l28.2\
+-35.1c2.2-2.8,6.\
+3-3.2,9.1-1c2.8,\
+2.2,3.2,6.3,1,9.\
+1L56.4,73.5\x0a\x09C55\
+.1,75.1,53.2,75.\
+9,51.3,75.9z\x22/>\x0a\
+</svg>\x0a\
+"
+
+qt_resource_name = b"\
+\x00\x08\
+\x05\xe2Y'\
+\x00l\
+\x00o\x00g\x00o\x00.\x00p\x00n\x00g\
+\x00\x08\
+\x0aaZ\xa7\
+\x00i\
+\x00c\x00o\x00n\x00.\x00p\x00n\x00g\
+\x00\x0d\
+\x0dq\x0b\x87\
+\x00c\
+\x00o\x00l\x00l\x00a\x00p\x00s\x00e\x00d\x00.\x00s\x00v\x00g\
+\x00\x0c\
+\x07)\x8aG\
+\x00e\
+\x00x\x00p\x00a\x00n\x00d\x00e\x00d\x00.\x00s\x00v\x00g\
+"
+
+qt_resource_struct = b"\
+\x00\x00\x00\x00\x00\x02\x00\x00\x00\x04\x00\x00\x00\x01\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\
+\x00\x00\x00L\x00\x00\x00\x00\x00\x01\x00\x00\x14\x9b\
+\x00\x00\x00\x16\x00\x00\x00\x00\x00\x01\x00\x00\x0d\xe2\
+\x00\x00\x00,\x00\x00\x00\x00\x00\x01\x00\x00\x11>\
+"
+
+def qInitResources():
+    QtCore.qRegisterResourceData(0x01, qt_resource_struct, qt_resource_name, qt_resource_data)
+
+def qCleanupResources():
+    QtCore.qUnregisterResourceData(0x01, qt_resource_struct, qt_resource_name, qt_resource_data)
+
+qInitResources()

server/gui/hook-cefpython3.py

@@ -0,0 +1,237 @@
+"""
+This is PyInstaller hook file for CEF Python. This file
+helps PyInstaller find CEF Python dependencies that are
+required to run final executable.
+
+See PyInstaller docs for hooks:
+https://pyinstaller.readthedocs.io/en/stable/hooks.html
+"""
+
+import glob
+import os
+import platform
+import re
+import sys
+import PyInstaller
+from PyInstaller.utils.hooks import is_module_satisfies, get_package_paths
+from PyInstaller.compat import is_win, is_darwin, is_linux, is_py2
+from PyInstaller import log as logging
+
+# Constants
+CEFPYTHON_MIN_VERSION = "57.0"
+PYINSTALLER_MIN_VERSION = "3.2.1"
+
+# Makes assumption that using "python.exe" and not "pyinstaller.exe"
+# TODO: use this code to work cross-platform:
+# from PyInstaller.utils.hooks import get_package_paths
+# get_package_paths("cefpython3")
+
+CEFPYTHON3_DIR = get_package_paths("cefpython3")[1]
+
+CYTHON_MODULE_EXT = ".pyd" if is_win else ".so"
+
+# Globals
+logger = logging.getLogger(__name__)
+
+
+# Functions
+def check_platforms():
+    if not is_win and not is_darwin and not is_linux:
+        raise SystemExit("Error: Currently only Windows, Linux and Darwin "
+                         "platforms are  supported, see Issue #135.")
+
+
+def check_pyinstaller_version():
+    """Using is_module_satisfies() for pyinstaller fails when
+    installed using 'pip install develop.zip' command
+    (PyInstaller Issue #2802)."""
+    # Example version string for dev version of pyinstaller:
+    # > 3.3.dev0+g5dc9557c
+    version = PyInstaller.__version__
+    match = re.search(r"^\d+\.\d+(\.\d+)?", version)
+    if not (match.group(0) >= PYINSTALLER_MIN_VERSION):
+        raise SystemExit("Error: pyinstaller %s or higher is required"
+                         % PYINSTALLER_MIN_VERSION)
+
+
+def check_cefpython3_version():
+    if not is_module_satisfies("cefpython3 >= %s" % CEFPYTHON_MIN_VERSION):
+        raise SystemExit("Error: cefpython3 %s or higher is required"
+                         % CEFPYTHON_MIN_VERSION)
+
+
+def get_cefpython_modules():
+    """Get all cefpython Cython modules in the cefpython3 package.
+    It returns a list of names without file extension. Eg.
+    'cefpython_py27'. """
+    pyds = glob.glob(os.path.join(CEFPYTHON3_DIR,
+                                  "cefpython_py*" + CYTHON_MODULE_EXT))
+    assert len(pyds) > 1, "Missing cefpython3 Cython modules"
+    modules = []
+    for path in pyds:
+        filename = os.path.basename(path)
+        mod = filename.replace(CYTHON_MODULE_EXT, "")
+        modules.append(mod)
+    return modules
+
+
+def get_excluded_cefpython_modules():
+    """CEF Python package includes Cython modules for various Python
+       versions. When using Python 2.7 pyinstaller should not
+       bundle modules for eg. Python 3.6, otherwise it will
+       cause to include Python 3 dll dependencies. Returns a list
+       of fully qualified names eg. 'cefpython3.cefpython_py27'."""
+    pyver = "".join(map(str, sys.version_info[:2]))
+    pyver_string = "py%s" % pyver
+    modules = get_cefpython_modules()
+    excluded = []
+    for mod in modules:
+        if pyver_string in mod:
+            continue
+        excluded.append("cefpython3.%s" % mod)
+        logger.info("Exclude cefpython3 module: %s" % excluded[-1])
+    return excluded
+
+
+def get_cefpython3_datas():
+    """Returning almost all of cefpython binaries as DATAS (see exception
+    below), because pyinstaller does strange things and fails if these are
+    returned as BINARIES. It first updates manifest in .dll files:
+    >> Updating manifest in chrome_elf.dll
+
+    And then because of that it fails to load the library:
+    >> hsrc = win32api.LoadLibraryEx(filename, 0, LOAD_LIBRARY_AS_DATAFILE)
+    >> pywintypes.error: (5, 'LoadLibraryEx', 'Access is denied.')
+
+    It is not required for pyinstaller to modify in any way
+    CEF binaries or to look for its dependencies. CEF binaries
+    does not have any external dependencies like MSVCR or similar.
+
+    The .pak .dat and .bin files cannot be marked as BINARIES
+    as pyinstaller would fail to find binary depdendencies on
+    these files.
+
+    One exception is subprocess (subprocess.exe on Windows) executable
+    file, which is passed to pyinstaller as BINARIES in order to collect
+    its dependecies.
+
+    DATAS are in format: tuple(full_path, dest_subdir).
+    """
+    ret = list()
+
+    if is_win:
+        cefdatadir = "."
+    elif is_darwin or is_linux:
+        cefdatadir = "."
+    else:
+        assert False, "Unsupported system {}".format(platform.system())
+
+    # Binaries, licenses and readmes in the cefpython3/ directory
+    for filename in os.listdir(CEFPYTHON3_DIR):
+        # Ignore Cython modules which are already handled by
+        # pyinstaller automatically.
+        if filename[:-len(CYTHON_MODULE_EXT)] in get_cefpython_modules():
+            continue
+
+        # CEF binaries and datas
+        extension = os.path.splitext(filename)[1]
+        if extension in \
+            [".exe", ".dll", ".pak", ".dat", ".bin", ".txt", ".so", ".plist"] \
+                or filename.lower().startswith("license"):
+            logger.info("Include cefpython3 data: {}".format(filename))
+            ret.append((os.path.join(CEFPYTHON3_DIR, filename), cefdatadir))
+
+    if is_darwin:
+        # "Chromium Embedded Framework.framework/Resources" with subdirectories
+        # is required. Contain .pak files and locales (each locale in separate
+        # subdirectory).
+        resources_subdir = \
+            os.path.join("Chromium Embedded Framework.framework", "Resources")
+        base_path = os.path.join(CEFPYTHON3_DIR, resources_subdir)
+        assert os.path.exists(base_path), \
+            "{} dir not found in cefpython3".format(resources_subdir)
+        for path, dirs, files in os.walk(base_path):
+            for file in files:
+                absolute_file_path = os.path.join(path, file)
+                dest_path = os.path.relpath(path, CEFPYTHON3_DIR)
+                ret.append((absolute_file_path, dest_path))
+                logger.info("Include cefpython3 data: {}/{}".format(dest_path, file))
+    elif is_win or is_linux:
+        # The .pak files in cefpython3/locales/ directory
+        locales_dir = os.path.join(CEFPYTHON3_DIR, "locales")
+        assert os.path.exists(locales_dir), \
+            "locales/ dir not found in cefpython3"
+        for filename in os.listdir(locales_dir):
+            logger.info("Include cefpython3 data: {}/{}".format(
+                os.path.basename(locales_dir), filename))
+            ret.append((os.path.join(locales_dir, filename),
+                        os.path.join(cefdatadir, "locales")))
+
+        # Optional .so/.dll files in cefpython3/swiftshader/ directory
+        swiftshader_dir = os.path.join(CEFPYTHON3_DIR, "swiftshader")
+        if os.path.isdir(swiftshader_dir):
+            for filename in os.listdir(swiftshader_dir):
+                logger.info("Include cefpython3 data: {}/{}".format(
+                    os.path.basename(swiftshader_dir), filename))
+                ret.append((os.path.join(swiftshader_dir, filename),
+                            os.path.join(cefdatadir, "swiftshader")))
+    return ret
+
+
+# ----------------------------------------------------------------------------
+# Main
+# ----------------------------------------------------------------------------
+
+# Checks
+check_platforms()
+check_pyinstaller_version()
+check_cefpython3_version()
+
+# Info
+logger.info("CEF Python package directory: %s" % CEFPYTHON3_DIR)
+
+# Hidden imports.
+# PyInstaller has no way on detecting imports made by Cython
+# modules, so all pure Python imports made in cefpython .pyx
+# files need to be manually entered here.
+# TODO: Write a tool script that would find such imports in
+#       .pyx files automatically.
+hiddenimports = [
+    "codecs",
+    "copy",
+    "datetime",
+    "inspect",
+    "json",
+    "os",
+    "platform",
+    "random",
+    "re",
+    "sys",
+    "time",
+    "traceback",
+    "types",
+    "urllib",
+    "weakref",
+]
+if is_py2:
+    hiddenimports += [
+        "urlparse",
+    ]
+
+# Excluded modules
+excludedimports = get_excluded_cefpython_modules()
+
+# Include binaries requiring to collect its dependencies
+if is_darwin or is_linux:
+    binaries = [(os.path.join(CEFPYTHON3_DIR, "subprocess"), ".")]
+elif is_win:
+    binaries = [(os.path.join(CEFPYTHON3_DIR, "subprocess.exe"), ".")]
+else:
+    binaries = []
+
+# Include datas
+datas = get_cefpython3_datas()
+
+# Notify pyinstaller.spec code that this hook was executed
+# and that it succeeded.
+os.environ["PYINSTALLER_CEFPYTHON3_HOOK_SUCCEEDED"] = "1"

server/gui/images/properties_contract.svg

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 23.0.1, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 100 100" style="enable-background:new 0 0 100 100;" xml:space="preserve">
+<style type="text/css">
+    path{fill:rgb(150, 146, 144)}
+    polygon{fill:rgb(150, 146, 144)}
+    circle{fill:rgb(150, 146, 144)}
+    rect{fill:rgb(150, 146, 144)}
+</style><path d="M31.8,56.4c-1.9,0-3.8-0.8-5-2.4c-2.2-2.8-1.8-6.9,1-9.1l35.1-28.2c2.8-2.2,6.9-1.8,9.1,1c2.2,2.8,1.8,6.9-1,9.1L35.8,54.9
+	C34.6,55.9,33.2,56.4,31.8,56.4z"/>
+<path d="M66.9,84.6c-1.4,0-2.9-0.5-4-1.4L27.7,54.9c-2.8-2.2-3.2-6.3-1-9.1c2.2-2.8,6.3-3.2,9.1-1l35.1,28.2c2.8,2.2,3.2,6.3,1,9.1
+	C70.6,83.8,68.8,84.6,66.9,84.6z"/>
+</svg>

server/gui/images/properties_expand.svg

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 23.0.1, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 100 100" style="enable-background:new 0 0 100 100" xml:space="preserve">
+<style type="text/css">
+    path{fill:rgb(150, 146, 144)}
+    polygon{fill:rgb(150, 146, 144)}
+    circle{fill:rgb(150, 146, 144)}
+    rect{fill:rgb(150, 146, 144)}
+</style><path d="M51.3,75.9c-1.9,0-3.8-0.8-5-2.4L18.1,38.4c-2.2-2.8-1.8-6.9,1-9.1c2.8-2.2,6.9-1.8,9.1,1l28.2,35.1c2.2,2.8,1.8,6.9-1,9.1
+	C54.2,75.5,52.7,75.9,51.3,75.9z"/>
+<path d="M51.3,75.9c-1.4,0-2.9-0.5-4-1.4c-2.8-2.2-3.2-6.3-1-9.1l28.2-35.1c2.2-2.8,6.3-3.2,9.1-1c2.8,2.2,3.2,6.3,1,9.1L56.4,73.5
+	C55.1,75.1,53.2,75.9,51.3,75.9z"/>
+</svg>

server/gui/main.py

@@ -1,33 +1,49 @@
 # flake8: noqa F403, F405
-from os.path import splitext, basename
+from functools import partialmethod
+from multiprocessing import Pipe, Process, freeze_support
+from os import environ
+from os.path import splitext, basename, dirname, join
 import sys
 import threading
 
 from cefpython3 import cefpython as cef
+import PySide2
+from PySide2.QtGui import *
 from PySide2.QtCore import *
 from PySide2.QtWidgets import *
 
-from server.app.app import Server
+import server.gui.cellxgene_rc
 from server.gui.browser import CefWidget, CefApplication
-from server.gui.workers import DataLoadWorker, ServerRunWorker
-from server.gui.utils import WINDOWS, LINUX, MAC, FileLoadSignals
-from server.utils.constants import MODES
+from server.gui.workers import Worker, SiteReadyWorker
+from server.gui.utils import WINDOWS, LINUX, MAC, FileLoadSignals, Emitter, WorkerSignals, FileChanged
+from server.utils.utils import find_available_port
 
+if WINDOWS or LINUX:
+    dirname = dirname(PySide2.__file__)
+    plugin_path = join(dirname, 'plugins', 'platforms')
+    environ['QT_QPA_PLATFORM_PLUGIN_PATH'] = plugin_path
 
 # Configuration
 # TODO remember this or calculate it?
-WIDTH = 1024
-HEIGHT = 768
+WIDTH = 1300
+HEIGHT = 800
+MAX_CONTENT_WIDTH = 700
+GUI_PORT = find_available_port("localhost")
+BROWSER_INDEX = 0
+LOAD_INDEX = 1
+
 
-# noinspection PyUnresolvedReferences
 class MainWindow(QMainWindow):
     def __init__(self):
         super(MainWindow, self).__init__(None)
         self.cef_widget = None
         self.data_widget = None
-        self.server = Server()
-        self.server.create_app()
-        self.runServer()
+        self.stacked_layout = None
+        self.parent_conn, self.child_conn = None, None
+        self.load_emitter = None
+        self.emitter_thread = None
+        self.worker = None
+        self.url = f"http://localhost:{GUI_PORT}/"
         self.setWindowTitle("cellxgene")
 
         # Strong focus - accepts focus by tab & click
@@ -35,13 +51,26 @@ class MainWindow(QMainWindow):
         self.setupLayout()
         self.setupMenu()
 
+    def showBrowser(self):
+        self.stacked_layout.setCurrentIndex(BROWSER_INDEX)
+
+    def restartOnError(self):
+        self.window().shutdownServer()
+        # close emitter on error/finished
+        self.parent_conn, self.child_conn = Pipe()
+        self.load_emitter = Emitter(self.parent_conn, WorkerSignals)
+        self.emitter_thread = threading.Thread(target=self.load_emitter.run, daemon=True)
+        self.emitter_thread.start()
+        # send to load with error message?
+
     def setupLayout(self):
         self.resize(WIDTH, HEIGHT)
         self.cef_widget = CefWidget(self)
+        self.cef_widget.setSizePolicy(QSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.MinimumExpanding))
         self.data_widget = LoadWidget(self)
         self.stacked_layout = QStackedLayout()
-        self.stacked_layout.addWidget(self.data_widget)
         self.stacked_layout.addWidget(self.cef_widget)
+        self.stacked_layout.addWidget(self.data_widget)
         main_layout = QVBoxLayout()
         main_layout.setContentsMargins(0, 0, 0, 0)
         main_layout.setSpacing(0)
@@ -67,9 +96,26 @@ class MainWindow(QMainWindow):
             # cef widget in the layout with the container.
             self.container = QWidget.createWindowContainer(
                 self.cef_widget.hidden_window, parent=self)
-            stacked_layout.addWidget(self.container, 1, 0)
+            self.stacked_layout.replaceWidget(self.cef_widget, self.container)
+        self.stacked_layout.setCurrentIndex(LOAD_INDEX)
+
+    def setupServer(self):
+        self.shutdownServer()
+        # close emitter on error/finished
+        self.parent_conn, self.child_conn = Pipe()
+        self.load_emitter = Emitter(self.parent_conn, WorkerSignals)
+        self.emitter_thread = threading.Thread(target=self.load_emitter.run, daemon=True)
+        self.emitter_thread.start()
+        # send to load with error message?
+
+    def shutdownServer(self):
+        if self.worker:
+            self.worker.terminate()
+        if self.parent_conn:
+            self.parent_conn.close()
 
     def setupMenu(self):
+        # TODO add communication to subprocess on reload
         main_menu = self.menuBar()
         file_menu = main_menu.addMenu('File')
         load_action = QAction("Load file...", self)
@@ -79,7 +125,11 @@ class MainWindow(QMainWindow):
         file_menu.addAction(load_action)
 
     def showLoad(self):
-        self.stacked_layout.setCurrentIndex(0)
+        self.clearMessages()
+        self.stacked_layout.setCurrentIndex(LOAD_INDEX)
+
+    def clearMessages(self):
+        self.data_widget.reset()
 
     def closeEvent(self, event):
         # Close browser (force=True) and free CEF reference
@@ -87,11 +137,6 @@ class MainWindow(QMainWindow):
             self.cef_widget.browser.CloseBrowser(True)
             self.clearBrowserReferences()
 
-    def runServer(self):
-        worker = ServerRunWorker(self.server.app, host="127.0.0.1", port=8000)
-        self.httpd = threading.Thread(target=worker.run, daemon=True)
-        self.httpd.start()
-
     def clearBrowserReferences(self):
         # Clear browser references that you keep anywhere in your
         # code. All references must be cleared for CEF to shutdown cleanly.
@@ -102,93 +147,217 @@ class LoadWidget(QFrame):
     def __init__(self, parent):
         super(LoadWidget, self).__init__(parent=parent)
         # Init layout
-        self.MAX_CONTENT_WIDTH = 500
         load_ui_layout = QVBoxLayout()
-        h_margin = (WIDTH - self.MAX_CONTENT_WIDTH) // 2
+        h_margin = (WIDTH - MAX_CONTENT_WIDTH) // 2
         if h_margin < 10:
             h_margin = 10
         load_ui_layout.setContentsMargins(h_margin, 20, h_margin, 20)
         logo_layout = QHBoxLayout()
         logo_layout.setContentsMargins(0, 0, 0, 20)
 
-        load_layout = QGridLayout()
-        load_layout.setContentsMargins(0, 0, 0, 0)
-        load_layout.setSpacing(0)
+        file_layout = QVBoxLayout()
+
         message_layout = QHBoxLayout()
         message_layout.setContentsMargins(0, 0, 0, 0)
 
-        self.title = ""
-        self.label = QLabel("cellxgene")
+        self.serverError = False
+        self.file_name = FilePath()
+
+        self.label = QLabel()
+        logo = QPixmap(":/logo.png")
+        self.label.setPixmap(logo)
+        self.label.setContentsMargins(100, 0, 100, 0)
         logo_layout.addWidget(self.label)
 
         # UI section
-        # TODO add load spinner
         # TODO add cancel button to send back to browser (if available)
-        self.embedding_label = QLabel("embedding: ")
-        load_layout.addWidget(self.embedding_label, 0, 0)
-        self.file_label = QLabel("file: ")
-        load_layout.addWidget(self.file_label, 0, 1)
-        self.embeddings = QComboBox(self)
-        self.embeddings.currentIndexChanged.connect(self.updateEmbedding)
-        self.embeddings.addItems(MODES)
-        self.embedding_selection = MODES[0]
-        load_layout.addWidget(self.embeddings, 1, 0)
 
-        self.load = QPushButton("Open...")
-        self.load.clicked.connect(self.onLoad)
-        load_layout.addWidget(self.load, 1, 1)
+        self.file_area = FileArea(self)
+        self.file_name.signals.changed.connect(self.updatePath)
+
+        self.launch_widget = QLabel("Select a file to launch cellxgene")
+        # self.launch_widget.setEnabled(False)
+        # self.launch_widget.clicked.connect(self.onLoad)
+
+        self.progress = QProgressBar()
+        self.progress.setTextVisible(False)
+
+        file_layout.addWidget(self.file_area)
+        self.loading_layout = QStackedLayout()
+        self.loading_layout.addWidget(self.launch_widget)
+        self.loading_layout.addWidget(self.progress)
+        file_layout.addLayout(self.loading_layout)
+        file_layout.setStretch(0, 10)
 
         # Error section
         self.error_label = QLabel("")
         self.error_label.setWordWrap(True)
-        self.error_label.setFixedWidth(self.MAX_CONTENT_WIDTH)
-        message_layout.addWidget(self.error_label, alignment=Qt.AlignTop)
+        self.error_label.setFixedWidth(MAX_CONTENT_WIDTH)
+        message_layout.addWidget(self.error_label)
+
+        # Options Form
 
         # Layout
-        for l in [logo_layout, load_layout, message_layout ]:
+        for l in [logo_layout, file_layout, message_layout]:
             load_ui_layout.addLayout(l)
 
-        load_ui_layout.setStretch(2, 10)
+        #TODO remove magic number
+        load_ui_layout.setStretch(1, 10)
         self.setLayout(load_ui_layout)
 
+        self.timer = QTimer()
+        self.timer.setInterval(100)
+        self.timer.timeout.connect(self.updateProgress)
+
         self.signals = FileLoadSignals()
         self.signals.selectedFile.connect(self.createScanpyEngine)
+        self.signals.error.connect(self.onError)
 
-    def updateEmbedding(self, idx):
-        self.embedding_selection = MODES[idx]
+    def updatePath(self):
+        file_name = self.file_name.value
+        if file_name:
+            self.file_area.label.setText("File: " + file_name)
+        else:
+            self.file_area.label.setText("")
+        # self.launch_widget.setEnabled(bool(file_name))
+
+    def reset(self):
+        self.loading_layout.setCurrentIndex(0)
+        self.timer.stop()
+        self.error_label.setText("")
+        self.file_name.updateValue(None)
+
+    def updateProgress(self):
+        curr_val = self.progress.value()
+        next_val = (curr_val + 1) % 100
+        self.progress.setValue(next_val)
+
+    def resetProgress(self):
+        self.progress.setValue(0)
+        self.loading_layout.setCurrentIndex(0)
+        self.timer.stop()
 
     def createScanpyEngine(self, file_name):
-        worker = DataLoadWorker(file_name, self.embedding_selection)
-        worker.signals.result.connect(self.onDataSuccess)
-        worker.signals.error.connect(self.onDataError)
-        self.load_worker = threading.Thread(target=worker.run, daemon=True)
-        self.load_worker.start()
+        title = splitext(basename(file_name))[0]
+        self.window().setupServer()
+        worker = Worker(self.window().parent_conn, self.window().child_conn, file_name, host="127.0.0.1",
+                        port=GUI_PORT, title=title, engine_options={})
+        self.window().load_emitter.signals.ready.connect(self.onDataReady)
+        self.window().load_emitter.signals.engine_error.connect(self.onServerError)
+        self.window().load_emitter.signals.server_error.connect(self.onServerError)
+        # Error is generic error from emitter
+        self.window().load_emitter.signals.error.connect(self.onServerError)
+        self.window().worker = Process(target=worker.run, daemon=True)
+        self.window().worker.start()
+        self.window().child_conn.close()
 
     def onLoad(self):
+        if self.file_name.value:
+            # Reset error on reload
+            self.serverError = False
+            self.loading_layout.setCurrentIndex(1)
+            self.timer.start()
+            self.signals.selectedFile.emit(self.file_name.value)
+        else:
+            self.signals.error.emit("Please select a file before launching.")
+
+    def onDataReady(self):
+        self.site_ready_worker = SiteReadyWorker(self.window().url)
+        self.site_ready_worker.signals.ready.connect(self.onServerReady)
+        self.site_ready_worker.signals.error.connect(self.onServerError)
+
+        srw_thread = threading.Thread(target=self.site_ready_worker.run, daemon=True)
+        srw_thread.start()
+
+    def onServerReady(self):
+        if not self.serverError:
+            self.resetProgress()
+            self.window().cef_widget.browser.Navigate(self.window().url)
+            self.window().showBrowser()
+
+    def onError(self, err, server_error=False):
+        # Restart worker
+        if server_error:
+            self.serverError = True
+            # Report error and switch to load screen
+        self.window().shutdownServer()
+        self.resetProgress()
+        self.window().stacked_layout.setCurrentIndex(LOAD_INDEX)
+        self.error_label.setText(f"Error: {err}")
+        self.error_label.resize(MAX_CONTENT_WIDTH, self.error_label.height())
+        self.window().repaint()
+
+    onServerError = partialmethod(onError, server_error=True)
+
+class FilePath(QObject):
+    def __init__(self):
+        super(FilePath, self).__init__()
+        self.value = ""
+        self.signals = FileChanged()
+
+    def updateValue(self, path=None):
+        self.value = path
+        self.signals.changed.emit(self.value != path)
+
+
+class FileArea(QFrame):
+    def __init__(self, parent):
+        super(FileArea, self).__init__()
+        self.setFrameShape(QFrame.Box)
+        self.setMinimumHeight(100)
+        self.setFixedWidth(MAX_CONTENT_WIDTH)
+        self.setAcceptDrops(True)
+        self.instructions = QLabel(self)
+        self.instructions.setText("Drag & Drop a h5ad file to load or open")
+        self.instructions.setGeometry(10, 10, MAX_CONTENT_WIDTH, self.instructions.height())
+        self.loadButton = QPushButton("Open...", parent=self)
+        x_pos = (MAX_CONTENT_WIDTH - self.loadButton.width()) / 2
+        self.loadButton.setGeometry(x_pos, 50, self.loadButton.width(), self.loadButton.height())
+        self.loadButton.clicked.connect(self.fileBrowse)
+        self.label = QLabel(self)
+        self.label.setGeometry(10, 75, MAX_CONTENT_WIDTH, self.label.height())
+
+    def fileBrowse(self):
         options = QFileDialog.Options()
         # options |= QFileDialog.DontUseNativeDialog
         file_name, _ = QFileDialog.getOpenFileName(self,
-                                                  "Open H5AD File", "", "H5AD Files (*.h5ad)", options=options)
-        self.title = splitext(basename(file_name))[0]
+                                                   "Open H5AD File", "", "H5AD Files (*.h5ad)", options=options)
         if file_name:
-            self.signals.selectedFile.emit(file_name)
+            self.parent().file_name.updateValue(file_name)
+            self.parent().onLoad()
 
+    def dragEnterEvent(self, e):
+        if e.mimeData().hasUrls:
+            e.accept()
+        else:
+            e.ignore()
 
-    def onDataSuccess(self, data):
-        self.window().server.attach_data(data, self.title)
-        self.navigateToLocation()
-        # Reveal browser
-        self.window().stacked_layout.setCurrentIndex(1)
+    def dragMoveEvent(self, e):
+        if e.mimeData().hasUrls:
+            e.accept()
+        else:
+            e.ignore()
 
-    def onDataError(self, err):
-        self.error_label.setText(f"Error: {err}")
-        self.error_label.resize(self.MAX_CONTENT_WIDTH, self.error_label.height())
-
-    def navigateToLocation(self, location="http://localhost:8000/"):
-        self.window().cef_widget.browser.Navigate(location)
+    def dropEvent(self, e):
+        """
+        Drop files directly onto the widget
+        File locations are stored in fname
+        :param e:
+        :return:
+        """
+        if e.mimeData().hasUrls:
+            e.setDropAction(Qt.CopyAction)
+            e.accept()
+            for url in e.mimeData().urls():
+                file_name = str(url.toLocalFile())
+            self.parent().file_name.updateValue(file_name)
+            self.parent().onLoad()
+        else:
+            e.ignore()
 
 
 def main():
+    freeze_support()
     # This generates an error.log file on error
     sys.excepthook = cef.ExceptHook  # To shutdown all CEF processes on error
     settings = {}
@@ -200,19 +369,26 @@ def main():
     cef.Initialize(settings)
     app = CefApplication(sys.argv)
     main_window = MainWindow()
+    main_window.setWindowTitle("cellxgene")
+    main_window.setUnifiedTitleAndToolBarOnMac(True)
+    main_window.setWindowIcon(QIcon(":icon.png"))
     main_window.show()
     main_window.activateWindow()
     main_window.raise_()
-    app.exec_()
+    try:
+        app.exec_()
+    except Exception as e:
+        raise
+    finally:
+        # Clean up on close
+        if not cef.GetAppSetting("external_message_pump"):
+            app.stopTimer()
 
-    # Clean up on close
-    if not cef.GetAppSetting("external_message_pump"):
-        app.stopTimer()
-    # TODO clean up threads when we switch threading model
-    del main_window  # Just to be safe, similarly to "del app"
-    del app  # Must destroy app object before calling Shutdown
-    cef.Shutdown()
-    sys.exit(0)
+        main_window.shutdownServer()
+        del main_window  # Just to be safe, similarly to "del app"
+        del app  # Must destroy app object before calling Shutdown
+        cef.Shutdown()
+        sys.exit(0)
 
 
 if __name__ == '__main__':

server/gui/utils.py

@@ -1,3 +1,4 @@
+import errno
 import platform
 
 from PySide2.QtCore import QObject, Signal
@@ -13,13 +14,66 @@ class WorkerSignals(QObject):
     Defines the signals available from a running worker thread.
     Supported signals are:
         finished
+        ready
         error - `str` error message
         result - `object` data returned from processing, anything
     """
     finished = Signal()
+    engine_error = Signal(str)
+    server_error = Signal(str)
     error = Signal(str)
     result = Signal(object)
+    ready = Signal()
+
+
+class SiteReadySignals(QObject):
+    """
+    Defines the signals available from a running worker thread.
+    Supported signals are:
+        timeout
+        ready
+        error - `str` error message
+    """
+    ready = Signal()
+    timeout = Signal()
+    error = Signal(str)
 
 
 class FileLoadSignals(QObject):
     selectedFile = Signal(str)
+    error = Signal(str)
+
+
+class FileChanged(QObject):
+    changed = Signal(bool)
+
+
+class Emitter:
+    def __init__(self, transport, signals):
+        self.transport = transport
+        self.signals = signals()
+
+    def _emit(self, signature, args=None):
+        if args is None:
+            getattr(self.signals, signature).emit()
+        else:
+            getattr(self.signals, signature).emit(args)
+
+    def run(self):
+        while True:
+            try:
+                signature = self.transport.recv()
+            except EOFError:
+                # Server done
+                break
+            except OSError as e:
+                if e.errno == errno.EBADF:
+                    break
+                else:
+                    self.signals.error.emit(str(e))
+                    break
+            except Exception as e:
+                self.signals.error.emit(str(e))
+                break
+            else:
+                self._emit(*signature)

server/gui/workers.py

@@ -1,47 +1,89 @@
-import traceback
+from multiprocessing import Process
+import time
 
-from server.gui.utils import WorkerSignals
+import requests
+
+from server.gui.utils import SiteReadySignals
 
 
-class DataLoadWorker():
-    def __init__(self, data_file, layout="umap", *args, **kwargs):
-        super(DataLoadWorker, self).__init__()
-        self.data_file = data_file
-        self.layout = layout
-        self.signals = WorkerSignals()
+class EmittingProcess(Process):
+    def __init__(self, parent_conn, child_conn, *arg, **kwargs):
+        super(EmittingProcess, self).__init__()
+        self.parent_conn = parent_conn
+        self.child_conn = child_conn
 
     def run(self):
-        if not self.data_file:
-            self.signals.finished.emit()
-            return
+        self.parent_conn.close()
 
-        # delayed import to speed load
-        from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
-        args = {
-            "layout": self.layout,
-            "diffexp": "ttest",
-            "max_category_items": 100,
-            "diffexp_lfc_cutoff": 0.01,
-            "obs_names": None,
-            "var_names": None,
-        }
-        try:
-            data_results = ScanpyEngine(self.data_file, args)
-        except Exception as e:
-            traceback.print_exc()
-            self.signals.error.emit(str(e))
-        else:
-            self.signals.result.emit(data_results)
-        finally:
-            self.signals.finished.emit()
+    def emit(self, signal_name, *args):
+        message = (signal_name, *args)
+        self.child_conn.send(message)
 
 
-class ServerRunWorker():
-    def __init__(self, app, host, port, *args, **kwargs):
-        super(ServerRunWorker, self).__init__()
-        self.app = app
+class Worker(EmittingProcess):
+    def __init__(self, parent_conn, child_conn, data_file, host, port, title, engine_options, *args, **kwargs):
+        super(Worker, self).__init__(parent_conn, child_conn)
+        self.data_file = data_file
         self.host = host
         self.port = port
+        self.title = title
+        self.engine_options = engine_options
 
     def run(self):
-        self.app.run(host=self.host, debug=False, port=self.port, threaded=True)
+        super(Worker, self).run()
+        if not self.data_file:
+            self.emit("finished")
+            return
+        from server.app.app import Server
+        from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
+        # create server
+        try:
+            server = Server()
+            server.create_app()
+        except Exception as e:
+            self.emit("server_error", str(e))
+            self.emit("finished")
+            return
+        # load data
+        try:
+            args = {
+                "max_category_items": 100,
+                "diffexp_lfc_cutoff": 0.01,
+                "obs_names": None,
+                "var_names": None,
+            }
+            args.update(self.engine_options)
+            data = ScanpyEngine(self.data_file, args)
+            server.attach_data(data, self.title)
+            self.emit("ready")
+        except Exception as e:
+            self.emit("engine_error", str(e))
+            self.emit("finished")
+            return
+        # launch server
+        try:
+            server.app.run(host=self.host, debug=False, port=self.port, threaded=True)
+        except Exception as e:
+            self.emit("server_error", str(e))
+        finally:
+            self.emit("finished")
+
+
+class SiteReadyWorker:
+    def __init__(self, location):
+        super(SiteReadyWorker, self).__init__()
+        self.signals = SiteReadySignals()
+        self.location = location
+
+    def run(self):
+        session = requests.Session()
+        for i in range(90):
+            try:
+                session.head(self.location)
+                self.signals.ready.emit()
+                break
+            except requests.exceptions.ConnectionError:
+                time.sleep(1)
+            except Exception as e:
+                self.signals.error.emit(str(e))
+        self.signals.timeout.emit()

server/requirements-gui.txt

@@ -0,0 +1,4 @@
+cefpython3>=66
+requests
+PyInstaller>=3.4
+PySide2>=5.12.3