Restv2 feature branch merge to master (#284)

Move to new REST v0.2 communication between front and back-end.   This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc.    Protocol spec is in docs directory.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Fix merge errors

- import warnings was improperly deleted
- scanpy engine tests were totally wrong

* Fix merge error with driver

* PUT /annotations (#235)

* Add query param for annotation name

* fix descriptions, eliminate else clause

* first cut at initial data load on rest 0.2 api

* Annotation var (#248)

* Fix bug strings are always objects in pandas

* Add axis to annotation method

* Add /annotation/var to REST api

* Csweaver/expressiondata (#242)

* Refactor expression method for REST v2

* Add message to QueryStringError

* Fix range filters

* Add GET route for /data

* /data PUT route

* rename expression to data_frame

* clarification of error

* Improve accept type handling

* support all schema types for 0.2 REST API

* remove REST 0.1 code; connect var annotations loading

* config reducer; use config to set data set title; remove obsolete templating code for data set title

* REST 0.2 expression conversion support

* partial port of expression to REST 0.2

*  diffexp (#273)

* Add diffexp method to scanpy

and test

* Minor tweaks to diffexp

Get a minimal working version to unblock FE development

* Fixing things git deleted

* cleanup print statements

* Add index test

* additional, partial REST 0.2 bring up of diffexp

* Ignore unstructured annotations for data (#275)

This is a temp hack, need to figure out how to include data.uns if there is only one gene

* diffexp REST 0.2 port finish

* ignore unstructured annotaitons on all routes except layout

* correctly use varDataCache; maintain state during world rebuild

* correct varDataCache use

* temporarily disable all memoization

* refinements to expression data caching

* clear cell sets upon regraph/reset

* update version of REST to 0.2

* Travis build fixes

- comment out cache import
- fix duplicate test name

* Remove dependency from travis

* clarify semantics of config variables

* move generic action helpers into util

client/src/components/expression/diffExpHeatmap.js

@@ -3,15 +3,13 @@ import React from "react";
 import _ from "lodash";
 import memoize from "memoize-one";
 import { connect } from "react-redux";
-import * as globals from "../../globals";
-import styles from "./expression.css";
-import SectionHeader from "../framework/sectionHeader";
-import actions from "../../actions";
 import ReactAutocomplete from "react-autocomplete"; /* http://emilebres.github.io/react-virtualized-checkbox/ */
 import getContrast from "font-color-contrast"; // https://www.npmjs.com/package/font-color-contrast
 import FaPaintBrush from "react-icons/lib/fa/paint-brush";
 import * as d3 from "d3";
 import { interpolateGreys } from "d3-scale-chromatic";
+import * as globals from "../../globals";
+import actions from "../../actions";
 
 class HeatmapSquare extends React.Component {
   constructor(props) {
@@ -22,9 +20,10 @@ class HeatmapSquare extends React.Component {
   }
 
   render() {
+    const { backgroundColor, text } = this.props;
     const contrastColor = getContrast(
-      this.props.backgroundColor
-        .substring(4, this.props.backgroundColor.length - 1)
+      backgroundColor
+        .substring(4, backgroundColor.length - 1)
         .replace(/ /g, "")
         .split(",")
     );
@@ -38,10 +37,10 @@ class HeatmapSquare extends React.Component {
           flexShrink: 0,
           fontSize: 12,
           margin: 0,
-          backgroundColor: this.props.backgroundColor
+          backgroundColor
         }}
       >
-        {this.props.text}
+        {text}
       </p>
     );
   }
@@ -54,13 +53,11 @@ class HeatmapSquare extends React.Component {
 ***********************************
 ***********************************
 **********************************/
-@connect(state => {
-  return {
-    scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
-    scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
-    colorAccessor: state.controls.colorAccessor
-  };
-})
+@connect(state => ({
+  scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
+  scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
+  colorAccessor: state.controls.colorAccessor
+}))
 class HeatmapRow extends React.Component {
   constructor(props) {
     super(props);
@@ -69,35 +66,44 @@ class HeatmapRow extends React.Component {
     };
   }
 
-  handleGeneColorScaleClick(gene) {
+  handleGeneColorScaleClick() {
     return () => {
-      this.props.dispatch(
-        actions.requestSingleGeneExpressionCountsForColoringPOST(
-          this.props.gene
-        )
-      );
+      const { dispatch, gene } = this.props;
+      dispatch(actions.requestSingleGeneExpressionCountsForColoringPOST(gene));
     };
   }
 
-  handleSetGeneAsScatterplotX(gene) {
+  handleSetGeneAsScatterplotX() {
     return () => {
-      this.props.dispatch({
+      const { dispatch, gene } = this.props;
+      dispatch({
         type: "set scatterplot x",
-        data: this.props.gene
+        data: gene
       });
     };
   }
 
-  handleSetGeneAsScatterplotY(gene) {
+  handleSetGeneAsScatterplotY() {
     return () => {
-      this.props.dispatch({
+      const { dispatch, gene } = this.props;
+      dispatch({
         type: "set scatterplot y",
-        data: this.props.gene
+        data: gene
       });
     };
   }
 
   render() {
+    const {
+      gene,
+      aveDiff,
+      set1exp,
+      set2exp,
+      greyColorScale,
+      scatterplotXXaccessor,
+      scatterplotYYaccessor,
+      colorAccessor
+    } = this.props;
     return (
       <div
         style={{
@@ -113,35 +119,33 @@ class HeatmapRow extends React.Component {
               fontSize: 14
             }}
           >
-            {this.props.gene}
+            {gene}
           </span>
         </div>
         <HeatmapSquare
-          backgroundColor={this.props.greyColorScale(this.props.set1exp)}
-          text={this.props.set1exp}
+          backgroundColor={greyColorScale(set1exp)}
+          text={set1exp.toPrecision(2)}
         />
         <HeatmapSquare
-          backgroundColor={this.props.greyColorScale(this.props.set2exp)}
-          text={this.props.set2exp}
+          backgroundColor={greyColorScale(set2exp)}
+          text={set2exp.toPrecision(2)}
         />
         <span
-          title={this.props.aveDiff}
+          title={aveDiff}
           style={{
             fontSize: 12,
             marginLeft: 10,
             marginRight: 10
           }}
         >
-          {this.props.aveDiff.toFixed(2)}
+          {aveDiff.toFixed(2)}
         </span>
         <span
-          onClick={this.handleSetGeneAsScatterplotX(this.props.gene).bind(this)}
+          onClick={this.handleSetGeneAsScatterplotX(gene).bind(this)}
           style={{
             fontSize: 16,
             color:
-              this.props.scatterplotXXaccessor === this.props.gene
-                ? "white"
-                : globals.brightBlue,
+              scatterplotXXaccessor === gene ? "white" : globals.brightBlue,
             cursor: "pointer",
             position: "relative",
             top: 1,
@@ -150,21 +154,17 @@ class HeatmapRow extends React.Component {
             borderRadius: 3,
             padding: "2px 3px",
             backgroundColor:
-              this.props.scatterplotXXaccessor === this.props.gene
-                ? globals.brightBlue
-                : "inherit"
+              scatterplotXXaccessor === gene ? globals.brightBlue : "inherit"
           }}
         >
           X
         </span>
         <span
-          onClick={this.handleSetGeneAsScatterplotY(this.props.gene).bind(this)}
+          onClick={this.handleSetGeneAsScatterplotY(gene).bind(this)}
           style={{
             fontSize: 16,
             color:
-              this.props.scatterplotYYaccessor === this.props.gene
-                ? "white"
-                : globals.brightBlue,
+              scatterplotYYaccessor === gene ? "white" : globals.brightBlue,
             cursor: "pointer",
             position: "relative",
             top: 1,
@@ -173,15 +173,13 @@ class HeatmapRow extends React.Component {
             borderRadius: 3,
             padding: "2px 3px",
             backgroundColor:
-              this.props.scatterplotYYaccessor === this.props.gene
-                ? globals.brightBlue
-                : "inherit"
+              scatterplotYYaccessor === gene ? globals.brightBlue : "inherit"
           }}
         >
           Y
         </span>
         <span
-          onClick={this.handleGeneColorScaleClick(this.props.gene).bind(this)}
+          onClick={this.handleGeneColorScaleClick(gene).bind(this)}
           style={{
             fontSize: 16,
             cursor: "pointer",
@@ -189,14 +187,9 @@ class HeatmapRow extends React.Component {
             marginRight: 6,
             borderRadius: 3,
             padding: "0px 2px 2px 2px",
-            color:
-              this.props.colorAccessor === this.props.gene
-                ? "white"
-                : "inherit",
+            color: colorAccessor === gene ? "white" : "inherit",
             backgroundColor:
-              this.props.colorAccessor === this.props.gene
-                ? globals.brightBlue
-                : "inherit"
+              colorAccessor === gene ? globals.brightBlue : "inherit"
           }}
         >
           <FaPaintBrush style={{ display: "inline-block" }} />
@@ -214,12 +207,10 @@ class HeatmapRow extends React.Component {
 ***********************************
 **********************************/
 
-@connect(state => {
-  return {
-    differential: state.differential,
-    world: state.controls.world
-  };
-})
+@connect(state => ({
+  differential: state.differential,
+  world: state.controls.world
+}))
 class Heatmap extends React.Component {
   constructor(props) {
     super(props);
@@ -228,24 +219,31 @@ class Heatmap extends React.Component {
     };
   }
 
-  getAllGeneNames = memoize(world =>
-    _.map(this.props.world.varAnnotations, "name")
-  );
+  // XXX TODO unused at the moment
+  // getAllGeneNames = memoize(world =>
+  //   _.map(this.props.world.varAnnotations, "name")
+  // );
 
   render() {
-    if (!this.props.differential.diffExp)
+    const { world, differential } = this.props;
+    if (!differential.diffExp) {
       return <p>Select cells & compute differential to see heatmap</p>;
+    }
 
-    const topGenesForCellSet1 = this.props.differential.diffExp.data.celllist1;
-    const topGenesForCellSet2 = this.props.differential.diffExp.data.celllist2;
-    // const allGeneNames = this.getAllGeneNames(this.props.world);
+    // summarize the information for display.
+    const topGenes = _.map(differential.diffExp, val => ({
+      varIndex: val[0],
+      geneName: world.varAnnotations[val[0]].name,
+      avgDiff: val[1],
+      set1AvgExp: val[4],
+      set2AvgExp: val[5]
+    }));
 
+    // average expression extent
     const extent = d3.extent(
-      _.union(
-        topGenesForCellSet1.mean_expression_cellset1,
-        topGenesForCellSet1.mean_expression_cellset2,
-        topGenesForCellSet2.mean_expression_cellset1,
-        topGenesForCellSet2.mean_expression_cellset2
+      _.concat(
+        _.map(differential.diffExp, val => val[4]),
+        _.map(differential.diffExp, val => val[5])
       )
     );
 
@@ -269,35 +267,16 @@ class Heatmap extends React.Component {
           <p style={{ marginRight: 20 }}>2</p>
           <p>ave diff</p>
         </div>
-        {topGenesForCellSet1.topgenes.map((gene, i) => {
+        {topGenes.map(g => {
+          const { geneName, avgDiff, set1AvgExp, set2AvgExp } = g;
           return (
             <HeatmapRow
-              key={gene}
-              gene={gene}
+              key={geneName}
+              gene={geneName}
               greyColorScale={greyColorScale}
-              aveDiff={topGenesForCellSet1.ave_diff[i]}
-              set1exp={Math.floor(
-                topGenesForCellSet1.mean_expression_cellset1[i]
-              )}
-              set2exp={Math.floor(
-                topGenesForCellSet1.mean_expression_cellset2[i]
-              )}
-            />
-          );
-        })}
-        {topGenesForCellSet2.topgenes.map((gene, i) => {
-          return (
-            <HeatmapRow
-              key={gene}
-              gene={gene}
-              greyColorScale={greyColorScale}
-              aveDiff={topGenesForCellSet2.ave_diff[i]}
-              set1exp={Math.floor(
-                topGenesForCellSet2.mean_expression_cellset1[i]
-              )}
-              set2exp={Math.floor(
-                topGenesForCellSet2.mean_expression_cellset2[i]
-              )}
+              aveDiff={avgDiff}
+              set1exp={set1AvgExp}
+              set2exp={set2AvgExp}
             />
           );
         })}

client/src/components/expression/expressionButtons.js

@@ -6,14 +6,12 @@ import * as globals from "../../globals";
 import actions from "../../actions";
 import CellSetButton from "./cellSetButtons";
 
-@connect(state => {
-  return {
-    differential: state.differential,
-    world: state.controls.world,
-    crossfilter: _.get(state.controls, "crossfilter", null),
-    selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
-  };
-})
+@connect(state => ({
+  differential: state.differential,
+  world: state.controls.world,
+  crossfilter: state.controls.crossfilter,
+  selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
+}))
 class Expression extends React.Component {
   constructor(props) {
     super(props);
@@ -22,24 +20,27 @@ class Expression extends React.Component {
 
   handleClick(gene) {
     return () => {
-      this.props.dispatch({
+      const { dispatch } = this.props;
+      dispatch({
         type: "color by expression",
-        gene: gene
+        gene
       });
     };
   }
 
   computeDiffExp() {
-    this.props.dispatch(
+    const { dispatch, differential } = this.props;
+    dispatch(
       actions.requestDifferentialExpression(
-        this.props.differential.celllist1,
-        this.props.differential.celllist2
+        differential.celllist1,
+        differential.celllist2
       )
     );
   }
 
   render() {
-    if (!this.props.differential) {
+    const { differential } = this.props;
+    if (!differential) {
       return null;
     }
     return (
@@ -47,6 +48,7 @@ class Expression extends React.Component {
         <CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
         <CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
         <button
+          type="button"
           style={{
             fontSize: 14,
             fontWeight: 400,
@@ -55,14 +57,12 @@ class Expression extends React.Component {
             height: 30,
             borderRadius: 2,
             backgroundColor:
-              this.props.differential.celllist1 &&
-              this.props.differential.celllist2
+              differential.celllist1 && differential.celllist2
                 ? globals.brightBlue
                 : globals.lightGrey,
             border: "none",
             cursor:
-              this.props.differential.celllist1 &&
-              this.props.differential.celllist2
+              differential.celllist1 && differential.celllist2
                 ? "pointer"
                 : "auto"
           }}
@@ -76,13 +76,3 @@ class Expression extends React.Component {
 }
 
 export default Expression;
-
-// <div style={{ marginBottom: 15, width: 300 }}>
-//   There are currently
-//   {" " +
-//     (this.props.crossfilter
-//       ? this.props.crossfilter.cells.countFiltered()
-//       : 0) +
-//     " "}
-//   cells selected, click a cell set button to store them.
-// </div>

client/src/components/leftsidebar.js

@@ -1,4 +1,5 @@
 // jshint esversion: 6
+import _ from "lodash";
 import React from "react";
 import { connect } from "react-redux";
 import Categorical from "./categorical/categorical";
@@ -9,7 +10,8 @@ import * as globals from "../globals";
 import DynamicScatterplot from "./scatterplot/scatterplot";
 
 @connect(state => ({
-  responsive: state.responsive
+  responsive: state.responsive,
+  datasetTitle: _.get(state.config, "displayNames.dataset")
 }))
 class LeftSideBar extends React.Component {
   constructor(props) {
@@ -21,7 +23,7 @@ class LeftSideBar extends React.Component {
 
   render() {
     const { currentTab } = this.state;
-    const { responsive } = this.props;
+    const { responsive, datasetTitle } = this.props;
 
     /*
     this magic number should be made less fragile,
@@ -39,8 +41,8 @@ class LeftSideBar extends React.Component {
             width: "100%"
           }}
         >
-          cellxgene
-          {globals.datasetTitle}{" "}
+          cellxgene  
+          {datasetTitle}
         </p>
         <div style={{ padding: 10 }}>
           <button

client/src/components/scatterplot/scatterplot.js

@@ -17,6 +17,7 @@ import _drawPoints from "./drawPointsRegl";
 import { scaleLinear } from "../../util/scaleLinear";
 
 import { margin, width, height } from "./util";
+import { kvCache } from "../../util/stateManager";
 
 @connect(state => {
   const {
@@ -27,11 +28,11 @@ import { margin, width, height } from "./util";
   } = state.controls;
   const expressionX =
     world && scatterplotXXaccessor
-      ? state.controls.world.varDataCache[scatterplotXXaccessor]
+      ? kvCache.get(world.varDataCache, scatterplotXXaccessor)
       : null;
   const expressionY =
     world && scatterplotYYaccessor
-      ? state.controls.world.varDataCache[scatterplotYYaccessor]
+      ? kvCache.get(world.varDataCache, scatterplotYYaccessor)
       : null;
 
   return {