fork/cellxgene
1
0
Fork 0
mirror of https://github.com/chanzuckerberg/cellxgene.git synced 2026-09-15 20:57:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

Restv2 feature branch merge to master (#284)

Browse Source 
Move to new REST v0.2 communication between front and back-end.   This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc.    Protocol spec is in docs directory.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Fix merge errors

- import warnings was improperly deleted
- scanpy engine tests were totally wrong

* Fix merge error with driver

* PUT /annotations (#235)

* Add query param for annotation name

* fix descriptions, eliminate else clause

* first cut at initial data load on rest 0.2 api

* Annotation var (#248)

* Fix bug strings are always objects in pandas

* Add axis to annotation method

* Add /annotation/var to REST api

* Csweaver/expressiondata (#242)

* Refactor expression method for REST v2

* Add message to QueryStringError

* Fix range filters

* Add GET route for /data

* /data PUT route

* rename expression to data_frame

* clarification of error

* Improve accept type handling

* support all schema types for 0.2 REST API

* remove REST 0.1 code; connect var annotations loading

* config reducer; use config to set data set title; remove obsolete templating code for data set title

* REST 0.2 expression conversion support

* partial port of expression to REST 0.2

*  diffexp (#273)

* Add diffexp method to scanpy

and test

* Minor tweaks to diffexp

Get a minimal working version to unblock FE development

* Fixing things git deleted

* cleanup print statements

* Add index test

* additional, partial REST 0.2 bring up of diffexp

* Ignore unstructured annotations for data (#275)

This is a temp hack, need to figure out how to include data.uns if there is only one gene

* diffexp REST 0.2 port finish

* ignore unstructured annotaitons on all routes except layout

* correctly use varDataCache; maintain state during world rebuild

* correct varDataCache use

* temporarily disable all memoization

* refinements to expression data caching

* clear cell sets upon regraph/reset

* update version of REST to 0.2

* Travis build fixes

- comment out cache import
- fix duplicate test name

* Remove dependency from travis

* clarify semantics of config variables

* move generic action helpers into util
This commit is contained in:
Bruce Martin
2018-10-01 14:58:46 -07:00
committed by GitHub
parent f0d9d873be
commit eeec842ad0
31 changed files with 2139 additions and 1373 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.travis.yml
View File

@@ -7,13 +7,11 @@ cache:
pip: true
install:
- set -eo pipefail
- pip install flake8 httpie
- pip install flake8
- ./bin/build-client
- pip install -e .
- pip install -r server/requirements-dev.txt
script:
- set -eo pipefail
- flake8 server/app/
- pytest -s server/test/test_filter.py server/test/test_scanpy_engine.py
- cellxgene scanpy example-dataset/ &
- for i in {1..90}; do if http :5005/api/v0.1/initialize > /dev/null; then break; else echo "Waiting for server..."; sleep 1; fi; done
- pytest server/test/test_api.py
- pytest -s server/test

3
client/index_template.html
View File

@@ -16,9 +16,8 @@
window.CELLXGENE = {};
window.CELLXGENE.API = {
prefix: "{{ prefix | safe }}",
version: "v0.1/"
version: "v0.2/"
};
window.CELLXGENE.datasetTitle = "{{ datasetTitle }}";
</script>
<noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>, please <strong>enable JS</strong> to make this app work.</noscript>

262
client/src/actions/index.js
View File

@@ -1,60 +1,30 @@
// jshint esversion: 6
import _ from "lodash";
import memoize from "memoize-one";
import * as globals from "../globals";
import store from "../reducers";
import { Universe } from "../util/stateManager";
import { Universe, kvCache } from "../util/stateManager";
import { catchErrorsWrap, doJsonRequest } from "../util/actionHelpers";
/*
Catch unexpected errors and make sure we don't lose them!
*/
function catchErrorsWrap(fn) {
return (dispatch, getState) => {
fn(dispatch, getState).catch(error => {
console.error(error);
dispatch({ type: "UNEXPECTED ERROR", error });
});
};
}
async function doRequestInitialize() {
const res = await fetch(
`${globals.API.prefix}${globals.API.version}initialize`,
{
method: "get",
headers: new Headers({
"Content-Type": "application/json"
})
}
);
return res.json();
}
async function doRequestCells(query) {
const res = await fetch(
`${globals.API.prefix}${globals.API.version}cells${query}`,
{
method: "get",
headers: new Headers({
"Content-Type": "application/json"
})
}
);
return res.json();
}
function doInitialDataLoad(query = "") {
return catchErrorsWrap(async dispatch => {
const doInitialDataLoad = () =>
catchErrorsWrap(async dispatch => {
dispatch({ type: "initial data load start" });
try {
const res = await Promise.all([
doRequestInitialize(),
doRequestCells(query)
]);
const universe = Universe.createUniverseFromRESTv01Response(
res[0],
res[1]
);
const requests = _([
"config",
"schema",
"annotations/obs",
"annotations/var",
"layout/obs"
])
.map(r => `${globals.API.prefix}${globals.API.version}${r}`)
.map(url => doJsonRequest(url))
.value();
const results = await Promise.all(requests);
const universe = Universe.createUniverseFromRestV02Response(...results);
dispatch({
type: "configuration load complete",
config: results[0].config
});
dispatch({
type: "initial data load complete (universe exists)",
universe
@@ -63,7 +33,6 @@ function doInitialDataLoad(query = "") {
dispatch({ type: "initial data load error", error });
}
});
}
// XXX TODO - this is the old code for doing a regraph. Preserving it solely
// until we port to 0.2 API. The new UX for regraph can't be implemented on
@@ -131,66 +100,56 @@ const resetGraph = () => (dispatch, getState) =>
universe: getState().controls.universe
});
// This code defends against the case where /expression returns a cellname
// never seen before (ie, not returned by /cells). This should not happen
// (see https://github.com/chanzuckerberg/cellxgene-rest-api/issues/34) but
// occasionally does.
//
// XXX TODO - this code is only relevant in v0.1 REST API, and can be retired
// when we port to 0.2.
//
const makeMetadataMap = memoize(metadata => _.keyBy(metadata, "CellName"));
function cleanupExpressionResponse(data) {
const s = store.getState();
const { universe } = s.controls;
const metadata = makeMetadataMap(universe.obsAnnotations);
let errorFound = false;
data.data.cells = _.filter(data.data.cells, cell => {
if (!errorFound && !metadata[cell.cellname]) {
errorFound = true;
console.error(
"Warning: /expression REST API returned unexpected cell names -- discarding surprises."
);
}
return metadata[cell.cellname];
});
return data;
}
/*
Fetch [gene, ...] from V0.1 API. Not an action function - just a helper
which implements the new expression data caching.
Fetch expression vectors for each gene in genes. This is NOT an action
function, but rather a helper to be called from an action helper that
needs expression data.
Transparently utilizes cached data if it is already present.
*/
async function _doRequestExpressionData(dispatch, getState, genes) {
const state = getState();
/* check cache and only fetch data we do not already have */
const { universe } = state.controls;
const genesToFetch = _.filter(genes, g => !universe.varDataCache[g]);
/* preload data already in cache */
let expressionData = _.transform(genes, (expData, g) => {
const data = kvCache.get(universe.varDataCache, g);
if (data) {
expData[g] = data;
}
}); // --> { gene: data }
/* make a list of genes for which we do not have data */
const genesToFetch = _.filter(genes, g => expressionData[g] === undefined);
dispatch({ type: "expression load start" });
let expressionData = {}; // { gene: data }
/* Fetch data for any genes not in cache */
if (genesToFetch.length) {
try {
// XXX: TODO - this could be using /data/var rather than /data/obs,
// as that would simplify the transformation in
// convertExpressionRESTv02ToObject
const res = await fetch(
`${globals.API.prefix}${globals.API.version}expression`,
`${globals.API.prefix}${globals.API.version}data/obs`,
{
method: "POST",
method: "PUT",
body: JSON.stringify({
genelist: genes
filter: {
var: {
annotation_value: [{ name: "name", values: genesToFetch }]
}
}
}),
headers: new Headers({
accept: "application/json",
"Accept-Encoding": "gzip, deflate, br",
"Content-Type": "application/json"
})
}
);
let data = await res.json();
data = cleanupExpressionResponse(data);
data = Universe.convertExpressionRESTv01ToObject(universe, data);
const data = await res.json();
expressionData = {
...expressionData,
...data
...Universe.convertExpressionRESTv02ToObject(universe, data)
};
} catch (error) {
dispatch({ type: "expression load error", error });
@@ -198,26 +157,24 @@ async function _doRequestExpressionData(dispatch, getState, genes) {
}
}
// add the cached values
_.forEach(genes, g => {
if (expressionData[g] === undefined) {
expressionData[g] = universe.varDataCache[g];
}
});
return dispatch({ type: "expression load success", expressionData });
dispatch({ type: "expression load success", expressionData });
return expressionData;
}
function requestSingleGeneExpressionCountsForColoringPOST(gene) {
return async (dispatch, getState) => {
dispatch({ type: "get single gene expression for coloring started" });
try {
await _doRequestExpressionData(dispatch, getState, [gene]);
const expressionData = await _doRequestExpressionData(
dispatch,
getState,
[gene]
);
dispatch({
type: "color by expression",
gene,
data: {
[gene]: getState().controls.world.varDataCache[gene]
[gene]: expressionData[gene]
}
});
} catch (error) {
@@ -232,65 +189,74 @@ function requestSingleGeneExpressionCountsForColoringPOST(gene) {
const requestGeneExpressionCountsPOST = genes => async (dispatch, getState) => {
dispatch({ type: "get expression started" });
try {
await _doRequestExpressionData(dispatch, getState, genes);
const expressionData = await _doRequestExpressionData(
dispatch,
getState,
genes
);
return dispatch({
type: "get expression success",
genes,
data: _.transform(
genes,
(res, gene) => {
res[gene] = getState().controls.world.varDataCache[gene];
},
{}
)
data: expressionData
});
} catch (error) {
return dispatch({ type: "get expression error", error });
}
};
const requestDifferentialExpression = (
celllist1,
celllist2,
num_genes = 7
) => dispatch => {
const requestDifferentialExpression = (set1, set2, num_genes = 10) => async (
dispatch,
getState
) => {
dispatch({ type: "request differential expression started" });
fetch(`${globals.API.prefix}${globals.API.version}diffexpression`, {
method: "POST",
body: JSON.stringify({
celllist1,
celllist2,
num_genes
}),
headers: new Headers({
accept: "application/json",
"Content-Type": "application/json"
})
})
.then(res => res.json())
.then(
data => {
/*
kick off a secondary action to get all expression counts for all cells
now that we know what the top expressed are
*/
dispatch(
requestGeneExpressionCountsPOST(
_.union(data.data.celllist1.topgenes, data.data.celllist2.topgenes)
)
);
/* then send the success case action through */
return dispatch({
type: "request differential expression success",
data
});
},
error =>
dispatch({
type: "request differential expression error",
error
try {
/*
Steps:
1. get the most differentially expressed genes
2. get expression data for each
*/
const state = getState();
const { universe } = state.controls;
const set1ByIndex = _.map(set1, s => universe.obsNameToIndexMap[s]);
const set2ByIndex = _.map(set2, s => universe.obsNameToIndexMap[s]);
const diffExpFetch = await fetch(
`${globals.API.prefix}${globals.API.version}diffexp/obs`,
{
method: "POST",
headers: new Headers({
Accept: "application/json",
"Accept-Encoding": "gzip, deflate, br",
"Content-Type": "application/json"
}),
body: JSON.stringify({
mode: "topN",
count: num_genes,
set1: { filter: { obs: { index: set1ByIndex } } },
set2: { filter: { obs: { index: set2ByIndex } } }
})
}
);
const data = await diffExpFetch.json();
// result is [ [varIdx, ...], ... ]
const topNGenes = _.map(data, r => universe.varAnnotations[r[0]].name);
/*
Kick off secondary action to fetch all of the expression data for the
topN expressed genes.
*/
dispatch(requestGeneExpressionCountsPOST(topNGenes));
/* then send the success case action through */
return dispatch({
type: "request differential expression success",
data
});
} catch (error) {
return dispatch({
type: "request differential expression error",
error
});
}
};
export default {

179
client/src/components/expression/diffExpHeatmap.js
View File

@@ -3,15 +3,13 @@ import React from "react";
import _ from "lodash";
import memoize from "memoize-one";
import { connect } from "react-redux";
import * as globals from "../../globals";
import styles from "./expression.css";
import SectionHeader from "../framework/sectionHeader";
import actions from "../../actions";
import ReactAutocomplete from "react-autocomplete"; /* http://emilebres.github.io/react-virtualized-checkbox/ */
import getContrast from "font-color-contrast"; // https://www.npmjs.com/package/font-color-contrast
import FaPaintBrush from "react-icons/lib/fa/paint-brush";
import * as d3 from "d3";
import { interpolateGreys } from "d3-scale-chromatic";
import * as globals from "../../globals";
import actions from "../../actions";
class HeatmapSquare extends React.Component {
constructor(props) {
@@ -22,9 +20,10 @@ class HeatmapSquare extends React.Component {
}
render() {
const { backgroundColor, text } = this.props;
const contrastColor = getContrast(
this.props.backgroundColor
.substring(4, this.props.backgroundColor.length - 1)
backgroundColor
.substring(4, backgroundColor.length - 1)
.replace(/ /g, "")
.split(",")
);
@@ -38,10 +37,10 @@ class HeatmapSquare extends React.Component {
flexShrink: 0,
fontSize: 12,
margin: 0,
backgroundColor: this.props.backgroundColor
backgroundColor
}}
>
{this.props.text}
{text}
</p>
);
}
@@ -54,13 +53,11 @@ class HeatmapSquare extends React.Component {
***********************************
***********************************
**********************************/
@connect(state => {
return {
scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
colorAccessor: state.controls.colorAccessor
};
})
@connect(state => ({
scatterplotXXaccessor: state.controls.scatterplotXXaccessor,
scatterplotYYaccessor: state.controls.scatterplotYYaccessor,
colorAccessor: state.controls.colorAccessor
}))
class HeatmapRow extends React.Component {
constructor(props) {
super(props);
@@ -69,35 +66,44 @@ class HeatmapRow extends React.Component {
};
}
handleGeneColorScaleClick(gene) {
handleGeneColorScaleClick() {
return () => {
this.props.dispatch(
actions.requestSingleGeneExpressionCountsForColoringPOST(
this.props.gene
)
);
const { dispatch, gene } = this.props;
dispatch(actions.requestSingleGeneExpressionCountsForColoringPOST(gene));
};
}
handleSetGeneAsScatterplotX(gene) {
handleSetGeneAsScatterplotX() {
return () => {
this.props.dispatch({
const { dispatch, gene } = this.props;
dispatch({
type: "set scatterplot x",
data: this.props.gene
data: gene
});
};
}
handleSetGeneAsScatterplotY(gene) {
handleSetGeneAsScatterplotY() {
return () => {
this.props.dispatch({
const { dispatch, gene } = this.props;
dispatch({
type: "set scatterplot y",
data: this.props.gene
data: gene
});
};
}
render() {
const {
gene,
aveDiff,
set1exp,
set2exp,
greyColorScale,
scatterplotXXaccessor,
scatterplotYYaccessor,
colorAccessor
} = this.props;
return (
<div
style={{
@@ -113,35 +119,33 @@ class HeatmapRow extends React.Component {
fontSize: 14
}}
>
{this.props.gene}
{gene}
</span>
</div>
<HeatmapSquare
backgroundColor={this.props.greyColorScale(this.props.set1exp)}
text={this.props.set1exp}
backgroundColor={greyColorScale(set1exp)}
text={set1exp.toPrecision(2)}
/>
<HeatmapSquare
backgroundColor={this.props.greyColorScale(this.props.set2exp)}
text={this.props.set2exp}
backgroundColor={greyColorScale(set2exp)}
text={set2exp.toPrecision(2)}
/>
<span
title={this.props.aveDiff}
title={aveDiff}
style={{
fontSize: 12,
marginLeft: 10,
marginRight: 10
}}
>
{this.props.aveDiff.toFixed(2)}
{aveDiff.toFixed(2)}
</span>
<span
onClick={this.handleSetGeneAsScatterplotX(this.props.gene).bind(this)}
onClick={this.handleSetGeneAsScatterplotX(gene).bind(this)}
style={{
fontSize: 16,
color:
this.props.scatterplotXXaccessor === this.props.gene
? "white"
: globals.brightBlue,
scatterplotXXaccessor === gene ? "white" : globals.brightBlue,
cursor: "pointer",
position: "relative",
top: 1,
@@ -150,21 +154,17 @@ class HeatmapRow extends React.Component {
borderRadius: 3,
padding: "2px 3px",
backgroundColor:
this.props.scatterplotXXaccessor === this.props.gene
? globals.brightBlue
: "inherit"
scatterplotXXaccessor === gene ? globals.brightBlue : "inherit"
}}
>
X
</span>
<span
onClick={this.handleSetGeneAsScatterplotY(this.props.gene).bind(this)}
onClick={this.handleSetGeneAsScatterplotY(gene).bind(this)}
style={{
fontSize: 16,
color:
this.props.scatterplotYYaccessor === this.props.gene
? "white"
: globals.brightBlue,
scatterplotYYaccessor === gene ? "white" : globals.brightBlue,
cursor: "pointer",
position: "relative",
top: 1,
@@ -173,15 +173,13 @@ class HeatmapRow extends React.Component {
borderRadius: 3,
padding: "2px 3px",
backgroundColor:
this.props.scatterplotYYaccessor === this.props.gene
? globals.brightBlue
: "inherit"
scatterplotYYaccessor === gene ? globals.brightBlue : "inherit"
}}
>
Y
</span>
<span
onClick={this.handleGeneColorScaleClick(this.props.gene).bind(this)}
onClick={this.handleGeneColorScaleClick(gene).bind(this)}
style={{
fontSize: 16,
cursor: "pointer",
@@ -189,14 +187,9 @@ class HeatmapRow extends React.Component {
marginRight: 6,
borderRadius: 3,
padding: "0px 2px 2px 2px",
color:
this.props.colorAccessor === this.props.gene
? "white"
: "inherit",
color: colorAccessor === gene ? "white" : "inherit",
backgroundColor:
this.props.colorAccessor === this.props.gene
? globals.brightBlue
: "inherit"
colorAccessor === gene ? globals.brightBlue : "inherit"
}}
>
<FaPaintBrush style={{ display: "inline-block" }} />
@@ -214,12 +207,10 @@ class HeatmapRow extends React.Component {
***********************************
**********************************/
@connect(state => {
return {
differential: state.differential,
world: state.controls.world
};
})
@connect(state => ({
differential: state.differential,
world: state.controls.world
}))
class Heatmap extends React.Component {
constructor(props) {
super(props);
@@ -228,24 +219,31 @@ class Heatmap extends React.Component {
};
}
getAllGeneNames = memoize(world =>
_.map(this.props.world.varAnnotations, "name")
);
// XXX TODO unused at the moment
// getAllGeneNames = memoize(world =>
// _.map(this.props.world.varAnnotations, "name")
// );
render() {
if (!this.props.differential.diffExp)
const { world, differential } = this.props;
if (!differential.diffExp) {
return <p>Select cells & compute differential to see heatmap</p>;
}
const topGenesForCellSet1 = this.props.differential.diffExp.data.celllist1;
const topGenesForCellSet2 = this.props.differential.diffExp.data.celllist2;
// const allGeneNames = this.getAllGeneNames(this.props.world);
// summarize the information for display.
const topGenes = _.map(differential.diffExp, val => ({
varIndex: val[0],
geneName: world.varAnnotations[val[0]].name,
avgDiff: val[1],
set1AvgExp: val[4],
set2AvgExp: val[5]
}));
// average expression extent
const extent = d3.extent(
_.union(
topGenesForCellSet1.mean_expression_cellset1,
topGenesForCellSet1.mean_expression_cellset2,
topGenesForCellSet2.mean_expression_cellset1,
topGenesForCellSet2.mean_expression_cellset2
_.concat(
_.map(differential.diffExp, val => val[4]),
_.map(differential.diffExp, val => val[5])
)
);
@@ -269,35 +267,16 @@ class Heatmap extends React.Component {
<p style={{ marginRight: 20 }}>2</p>
<p>ave diff</p>
</div>
{topGenesForCellSet1.topgenes.map((gene, i) => {
{topGenes.map(g => {
const { geneName, avgDiff, set1AvgExp, set2AvgExp } = g;
return (
<HeatmapRow
key={gene}
gene={gene}
key={geneName}
gene={geneName}
greyColorScale={greyColorScale}
aveDiff={topGenesForCellSet1.ave_diff[i]}
set1exp={Math.floor(
topGenesForCellSet1.mean_expression_cellset1[i]
)}
set2exp={Math.floor(
topGenesForCellSet1.mean_expression_cellset2[i]
)}
/>
);
})}
{topGenesForCellSet2.topgenes.map((gene, i) => {
return (
<HeatmapRow
key={gene}
gene={gene}
greyColorScale={greyColorScale}
aveDiff={topGenesForCellSet2.ave_diff[i]}
set1exp={Math.floor(
topGenesForCellSet2.mean_expression_cellset1[i]
)}
set2exp={Math.floor(
topGenesForCellSet2.mean_expression_cellset2[i]
)}
aveDiff={avgDiff}
set1exp={set1AvgExp}
set2exp={set2AvgExp}
/>
);
})}

46
client/src/components/expression/expressionButtons.js
View File

@@ -6,14 +6,12 @@ import * as globals from "../../globals";
import actions from "../../actions";
import CellSetButton from "./cellSetButtons";
@connect(state => {
return {
differential: state.differential,
world: state.controls.world,
crossfilter: _.get(state.controls, "crossfilter", null),
selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
};
})
@connect(state => ({
differential: state.differential,
world: state.controls.world,
crossfilter: state.controls.crossfilter,
selectionUpdate: _.get(state.controls, "crossfilter.updateTime", null)
}))
class Expression extends React.Component {
constructor(props) {
super(props);
@@ -22,24 +20,27 @@ class Expression extends React.Component {
handleClick(gene) {
return () => {
this.props.dispatch({
const { dispatch } = this.props;
dispatch({
type: "color by expression",
gene: gene
gene
});
};
}
computeDiffExp() {
this.props.dispatch(
const { dispatch, differential } = this.props;
dispatch(
actions.requestDifferentialExpression(
this.props.differential.celllist1,
this.props.differential.celllist2
differential.celllist1,
differential.celllist2
)
);
}
render() {
if (!this.props.differential) {
const { differential } = this.props;
if (!differential) {
return null;
}
return (
@@ -47,6 +48,7 @@ class Expression extends React.Component {
<CellSetButton {...this.props} eitherCellSetOneOrTwo={1} />
<CellSetButton {...this.props} eitherCellSetOneOrTwo={2} />
<button
type="button"
style={{
fontSize: 14,
fontWeight: 400,
@@ -55,14 +57,12 @@ class Expression extends React.Component {
height: 30,
borderRadius: 2,
backgroundColor:
this.props.differential.celllist1 &&
this.props.differential.celllist2
differential.celllist1 && differential.celllist2
? globals.brightBlue
: globals.lightGrey,
border: "none",
cursor:
this.props.differential.celllist1 &&
this.props.differential.celllist2
differential.celllist1 && differential.celllist2
? "pointer"
: "auto"
}}
@@ -76,13 +76,3 @@ class Expression extends React.Component {
}
export default Expression;
// <div style={{ marginBottom: 15, width: 300 }}>
// There are currently
// {" " +
// (this.props.crossfilter
// ? this.props.crossfilter.cells.countFiltered()
// : 0) +
// " "}
// cells selected, click a cell set button to store them.
// </div>

10
client/src/components/leftsidebar.js
View File

@@ -1,4 +1,5 @@
// jshint esversion: 6
import _ from "lodash";
import React from "react";
import { connect } from "react-redux";
import Categorical from "./categorical/categorical";
@@ -9,7 +10,8 @@ import * as globals from "../globals";
import DynamicScatterplot from "./scatterplot/scatterplot";
@connect(state => ({
responsive: state.responsive
responsive: state.responsive,
datasetTitle: _.get(state.config, "displayNames.dataset")
}))
class LeftSideBar extends React.Component {
constructor(props) {
@@ -21,7 +23,7 @@ class LeftSideBar extends React.Component {
render() {
const { currentTab } = this.state;
const { responsive } = this.props;
const { responsive, datasetTitle } = this.props;
/*
this magic number should be made less fragile,
@@ -39,8 +41,8 @@ class LeftSideBar extends React.Component {
width: "100%"
}}
>
cellxgene
{globals.datasetTitle}{" "}
cellxgene &nbsp;
{datasetTitle}
</p>
<div style={{ padding: 10 }}>
<button

5
client/src/components/scatterplot/scatterplot.js
View File

@@ -17,6 +17,7 @@ import _drawPoints from "./drawPointsRegl";
import { scaleLinear } from "../../util/scaleLinear";
import { margin, width, height } from "./util";
import { kvCache } from "../../util/stateManager";
@connect(state => {
const {
@@ -27,11 +28,11 @@ import { margin, width, height } from "./util";
} = state.controls;
const expressionX =
world && scatterplotXXaccessor
? state.controls.world.varDataCache[scatterplotXXaccessor]
? kvCache.get(world.varDataCache, scatterplotXXaccessor)
: null;
const expressionY =
world && scatterplotYYaccessor
? state.controls.world.varDataCache[scatterplotYYaccessor]
? kvCache.get(world.varDataCache, scatterplotYYaccessor)
: null;
return {

13
client/src/globals.js
View File

@@ -49,19 +49,14 @@ export const bolder = 700;
export let API = {
// prefix: "http://api.clustering.czi.technology/api/",
//prefix: "http://tabulamuris.cxg.czi.technology/api/",
prefix: "http://api-staging.clustering.czi.technology/api/",
version: "v0.1/"
// prefix: "http://tabulamuris.cxg.czi.technology/api/",
// prefix: "http://api-staging.clustering.czi.technology/api/",
prefix: "http://localhost:5005/api/",
version: "v0.2/"
};
if (window.CELLXGENE && window.CELLXGENE.API) API = window.CELLXGENE.API;
export let datasetTitle = "";
if (window.CELLXGENE && window.CELLXGENE.datasetTitle)
datasetTitle = window.CELLXGENE.datasetTitle;
export const accentFont = "Georgia,Times,Times New Roman,serif";
export const maxParagraphWidth = 600;
export const maxControlsWidth = 800;

33
client/src/reducers/config.js Normal file
View File

@@ -0,0 +1,33 @@
// jshint esversion: 6
const Config = (
state = {
displayNames: null,
features: null
},
action
) => {
switch (action.type) {
case "initial data load start":
return {
...state,
loading: true,
error: null
};
case "configuration load complete":
return {
...state,
loading: false,
error: null,
...action.config
};
case "initial data load error":
return {
...state,
error: action.error
};
default:
return state;
}
};
export default Config;

9
client/src/reducers/differential.js
View File

@@ -1,4 +1,6 @@
// jshint esversion: 6
import _ from "lodash";
const Differential = (
state = {
diffExp: null,
@@ -39,6 +41,13 @@ const Differential = (
...state,
celllist2: action.data
};
case "reset World to eq Universe":
case "set World to current selection":
return {
...state,
celllist1: null,
celllist2: null
};
default:
return state;
}

2
client/src/reducers/index.js
View File

@@ -4,11 +4,13 @@ import thunk from "redux-thunk";
import updateURLMiddleware from "../middleware/updateURLMiddleware";
import updateCellColors from "../middleware/updateCellColors";
import config from "./config";
import differential from "./differential";
import responsive from "./responsive";
import controls from "./controls";
const Reducer = combineReducers({
config,
responsive,
controls,
differential

28
client/src/util/actionHelpers.js Normal file
View File

@@ -0,0 +1,28 @@
/*
Catch unexpected errors and make sure we don't lose them!
*/
export function catchErrorsWrap(fn) {
return (dispatch, getState) => {
fn(dispatch, getState).catch(error => {
console.error(error);
dispatch({ type: "UNEXPECTED ERROR", error });
});
};
}
/*
Bootstrap application with the initial data loading.
* /config - application configuration
* /schema - schema of dataframe
* /annotations/obs - all metadata annotation
*/
export const doJsonRequest = async url => {
const res = await fetch(url, {
method: "get",
headers: new Headers({
"Content-Type": "application/json",
"Accept-Encoding": "gzip, deflate, br"
})
});
return res.json();
};

24
client/src/util/stateManager/keyvalcache.js
View File

@@ -4,13 +4,12 @@ import _ from "lodash";
/*
Very simple key/value cache for use by World & Universe.
* constructor(lowWatermark, cachekey):
* constructor(lowWatermark, minTTL):
- lowWatermark defines the number of cache elements below which
flushing will not occur.
- minTTL defines minimum time in MS that cache entries will live.
- minTTL defines minimum time in milliseconds that cache entries will live.
A value of -1 disables automatic flushing (flush() can still
be called by external user).
- cachekey is a key that will be assigned to any value to track age
* set() - add a key/val pair.
* get() - get a value or undefined if not present.
* flush(minAgeMs) - flush cache entries in excess of lowWatermark if those
@@ -69,4 +68,21 @@ function flush(kvcache, minAgeMs = 0) {
return kvcache;
}
export { create, get, set, flush };
/*
use to create a cache that is a transformation of another cache.
*/
function map(srcKvCache, cb, createOptions) {
const keysInSrcKvCache = _(srcKvCache)
.keys()
.filter(k => k !== cachePrivateKey)
.value();
const newKvCache = create(createOptions.lowWatermark, createOptions.minTTL);
_.forEach(keysInSrcKvCache, key => {
const val = cb(get(srcKvCache, key));
newKvCache[key] = val;
val[cachePrivateKey] = Date.now();
});
return newKvCache;
}
export { create, get, set, flush, map };

344
client/src/util/stateManager/universe.js
View File

@@ -3,6 +3,44 @@
import _ from "lodash";
import * as kvCache from "./keyvalcache";
/*
Private helper function - create and return a template Universe
*/
function templateUniverse() {
/* default universe template */
/* varDataCache config - see kvCache for semantics */
const VarDataCacheLowWatermark = 32; // cache element count
const VarDataCacheTTLMs = 1000; // min cache time in MS
return {
api: null,
finalized: false, // XXX: may not be needed
nObs: 0,
nVar: 0,
schema: {},
/*
Annotations
*/
obsAnnotations: [] /* all obs annotations, by obs index */,
varAnnotations: [] /* all var annotations, by var index */,
obsNameToIndexMap: {} /* reverse map 'name' to index */,
varNameToIndexMap: {} /* reverse map 'name' to index */,
obsLayout: { X: [], Y: [] } /* xy layout */,
/*
Cache of var data (expression), by var annotation name. Data can be
accesses as a POJO, but if you want caching semantics, use the kvCache
API (eg., kvCache.get(), kvCache.set(), ...), which will maintain the
LRU semantics.
*/
varDataCache: kvCache.create(VarDataCacheLowWatermark, VarDataCacheTTLMs)
};
}
/*
This module implements functions that support storage of "Universe",
aka all of the var/obs data and annotations.
@@ -12,130 +50,9 @@ build an internal POJO for use by the rendering components.
*/
/*
Cherry pick from /api/v0.1 response format to make somethign similar
to the v0.2 schema, which we use for internal interfaces.
generate any client-side transformations or summarization that
is independent of REST API response formats.
*/
function RESTv01ResponseToSchema(response) {
/*
Annotation schemas in V02 (our target) look like:
annotations: {
obs: [
{ name: "name", type: "string" },
{ name: "num_reads", type: "int32" },
{
name: "clusters",
type: "categorical",
categories=[ 99, 1, "unknown cluster" ]
},
{ name: "QScore", type: "float32" }
],
var: [
{ "name": "name", "type": "string" },
{ "name": "gene", "type": "string" }
]
}
In V01, our source, it looks like:
"schema": {
"CellName": {
"displayname": "Name",
"include": true,
"type": "string",
"variabletype": "categorical"
},
"Cluster_2d": {
"displayname": "Cluster2d",
"include": true,
"type": "string",
"variabletype": "categorical"
},
"ERCC_reads": {
"displayname": "ERCC Reads",
"include": true,
"type": "int",
"variabletype": "continuous"
},
...
}
Mapping between the two assumes:
- V01 only has schema for observations
- CellName is mapped to 'name'
- type conversion: float->float32, int->int32, string->string
*/
return {
annotations: {
obs: _.map(response.data.schema, (val, key) => {
const name = key === "CellName" ? "name" : key;
let { type } = val;
if (type === "int") {
type = "int32";
}
if (type === "float") {
type = "float32";
}
return {
name,
type
};
}),
var: [{ name: "name", type: "string" }]
}
};
}
function RESTv01ResponseToVarAnnotations(response) {
/*
v0.1 initialize response contains 'genes' - names of all genes
in order.
*/
return _.map(response.data.genes, (g, i) => ({ __varIndex__: i, name: g }));
}
function RESTv01ResponseToObsAnnotations(response) {
/*
v0.1 format for metadata:
metadata: [ { key: val, key: val, ... }, ... ]
Target format is essentially the same, except the CellName key becomes name.
*/
return _.map(response.data.metadata, (c, i) => ({
__obsIndex__: i,
name: c.CellName,
...c
}));
}
function RESTv01ResponseToLayout(obsAnnotations, response) {
/*
v0.1 format for the graph is:
[ [ 'cellname', x, y ], [ 'cellname', x, y, ], ... ]
NOTE XXX: this code does not assume any particular array ordering in the V0.1
response. But for Universe initial load, the layout will be in the same
order as annotations, so this extra work isn't really necessary.
*/
const obsAnnotationsByName = _.keyBy(obsAnnotations, "name");
const { graph } = response.data;
const layout = {
X: new Float32Array(graph.length),
Y: new Float32Array(graph.length)
};
for (let i = 0; i < graph.length; i += 1) {
const [name, x, y] = graph[i];
const anno = obsAnnotationsByName[name];
const idx = anno.__obsIndex__;
layout.X[idx] = x;
layout.Y[idx] = y;
}
return layout;
}
function finalize(universe) {
/* A bit of sanity checking! */
const { nObs, nVar } = universe;
@@ -147,7 +64,14 @@ function finalize(universe) {
) {
throw new Error("Universe dimensionality mismatch - failed to load");
}
// TODO: add more sanity checks, such as:
// - all annotations in the schema
// - layout has supported number of dimensions
// - ...
/*
Create all derived (convenience) data structures.
*/
universe.obsNameToIndexMap = _.transform(
universe.obsAnnotations,
(acc, value, idx) => {
@@ -166,85 +90,135 @@ function finalize(universe) {
return universe;
}
function templateUniverse() {
/* default universe template */
const VarDataCacheLowWatermark = 32;
const VarDataCacheTTLMs = 1000;
function RESTv02AnnotationsResponseToInternal(response) {
/*
Source per the spec:
{
names: [
'tissue_type', 'sex', 'num_reads', 'clusters'
],
data: [
[ 0, 'lung', 'F', 39844, 99 ],
[ 1, 'heart', 'M', 83, 1 ],
[ 49, 'spleen', null, 2, "unknown cluster" ],
// [ obsOrVarIndex, value, value, value, value ],
// ...
]
}
return {
api: "0.1",
finalized: true, // XXX: may not be needed
nObs: 0,
nVar: 0,
schema: {},
/*
Annotations
*/
obsAnnotations: [] /* all obs annotations, by obs index */,
varAnnotations: [] /* all var annotations, by var index */,
obsNameToIndexMap: {} /* reverse map 'name' to index */,
varNameToIndexMap: {} /* reverse map 'name' to index */,
obsLayout: { X: [], Y: [] } /* xy layout */,
varDataCache: kvCache.create(
VarDataCacheLowWatermark,
VarDataCacheTTLMs
) /* cache of var data (expression) */
};
Internal (target) format:
[
{ __index__: 0, tissue_type: "lung", sex: "F", ... },
...
]
*/
const { names, data } = response;
const keys = ["__index__", ...names];
return _(data)
.map(obs => _.zipObject(keys, obs))
.sortBy("__index__")
.value();
}
export function createUniverseFromRESTv01Response(initResponse, cellsResponse) {
function RESTv02LayoutResponseToInternal(response) {
/*
build & return universe from a REST 0.1 /init and /cells response
*/
Source per the spec:
{
layout: {
ndims: 2,
coordinates: [
[ 0, 0.284483, 0.983744 ],
[ 1, 0.038844, 0.739444 ],
// [ obsOrVarIndex, X_coord, Y_coord ],
// ...
]
}
}
Target (internal) format:
{
X: Float32Array(numObs),
Y: Float32Array(numObs)
}
In the same order as obsAnnotations
*/
const { ndims, coordinates } = response.layout;
if (ndims !== 2) {
throw new Error("Unsupported layout dimensionality");
}
const layout = {
X: new Float32Array(coordinates.length),
Y: new Float32Array(coordinates.length)
};
for (let i = 0; i < coordinates.length; i += 1) {
const [idx, x, y] = coordinates[i];
layout.X[idx] = x;
layout.Y[idx] = y;
}
return layout;
}
export function createUniverseFromRestV02Response(
configResponse,
schemaResponse,
annotationsObsResponse,
annotationsVarResponse,
layoutObsResponse
) {
/*
build & return universe from a REST 0.2 /config, /schema and /annotations/obs response
*/
const { schema } = schemaResponse;
const universe = templateUniverse();
/* extract information from init OTA response */
universe.schema = RESTv01ResponseToSchema(initResponse);
universe.varAnnotations = RESTv01ResponseToVarAnnotations(initResponse);
universe.nVar = universe.varAnnotations.length;
/* constants */
universe.api = "0.2";
/* extract information fron cells REST json response */
/*
NOTE: this code *assumes* that cell order in data.metadata and data.graph
are the same. TODO: error checking.
*/
universe.obsAnnotations = RESTv01ResponseToObsAnnotations(cellsResponse);
universe.nObs = universe.obsAnnotations.length;
universe.obsLayout = RESTv01ResponseToLayout(
universe.obsAnnotations,
cellsResponse
/* schema related */
universe.schema = schema;
universe.nObs = schema.dataframe.nObs;
universe.nVar = schema.dataframe.nVar;
/* annotations */
universe.obsAnnotations = RESTv02AnnotationsResponseToInternal(
annotationsObsResponse
);
universe.varAnnotations = RESTv02AnnotationsResponseToInternal(
annotationsVarResponse
);
/* layout */
universe.obsLayout = RESTv02LayoutResponseToInternal(layoutObsResponse);
return finalize(universe);
}
export function convertExpressionRESTv01ToObject(universe, response) {
export function convertExpressionRESTv02ToObject(universe, response) {
/*
v0.1 ota looks like:
{
genes: [ "name1", "name2", ... ],
cells: [
{ cellname: 'cell1', e: [ 3, 4, n, x, y, ... ] },
...
]
}
/data/obs response looks like:
{
var: [ varIndices fetched ],
obs: [
[ obsIndex, evalue, ... ],
...
]
}
convert expression to a simple Float32Array, and return
[ [geneName, array], [geneName, array], ... ]
*/
convert expression toa simple Float32Array, and return
{ geneName: array, geneName: array, ... }
NOTE: geneName, not varIndex
*/
const vars = response.var;
const { obs } = response;
const result = {};
const { genes, cells } = response.data;
for (let idx = 0; idx < genes.length; idx += 1) {
const gene = genes[idx];
// XXX TODO: could this use _.unzip and have less code?
for (let varIdx = 0; varIdx < vars.length; varIdx += 1) {
const gene = universe.varAnnotations[vars[varIdx]].name;
const data = new Float32Array(universe.nObs);
for (let c = 0; c < cells.length; c += 1) {
const obsIndex = universe.obsNameToIndexMap[cells[c].cellname];
data[obsIndex] = cells[c].e[idx];
for (let obsIdx = 0; obsIdx < obs.length; obsIdx += 1) {
data[obsIdx] = obs[obsIdx][varIdx + 1];
}
result[gene] = data;
}

119
client/src/util/stateManager/world.js
View File

@@ -28,7 +28,7 @@ obs/cell.
NOTE: world.obsAnnotation should be identical to the old state.cells value,
EXCEPT that
* __cellIndex__ renamed to __obsIndex__
* __cellIndex__ renamed to __index__
* __x__ and __y__ are now in world.obsLayout
* __color__ and __colorRBG__ should be moved to controls reducer
@@ -46,47 +46,50 @@ obs/cell.
*/
/*
Summary information for each annotation, keyed by annotation name.
Value will be an object, containing either 'range' or 'options' object,
depending on the annotation schema type (categorical or continuous).
/* varDataCache config - see kvCache for semantics */
const VarDataCacheLowWatermark = 32; // cache element count
const VarDataCacheTTLMs = 1000; // min cache time in MS
Summarize for BOTH obs and var annotations. Result format:
{
obs: {
annotation_name: { ... },
...
},
var: {
annotation_name: { ... },
...
}
}
Example:
{
"Splice_sites_Annotated": {
"range": {
"min": 26,
"max": 1075869
}
},
"Selection": {
"options": {
"Astrocytes(HEPACAM)": 714,
"Endothelial(BSC)": 123,
"Oligodendrocytes(GC)": 294,
"Neurons(Thy1)": 685,
"Microglia(CD45)": 1108,
"Unpanned": 665
}
}
}
*/
function summarizeAnnotations(schema, obsAnnotations) {
/*
Build and return obs/var summary using any annotation in the schema
Summary information for each annotation, keyed by annotation name.
Value will be an object, containing either 'range' or 'options' object,
depending on the annotation schema type (categorical or continuous).
Summarize for BOTH obs and var annotations. Result format:
{
obs: {
annotation_name: { ... },
...
},
var: {
annotation_name: { ... },
...
}
}
Example:
{
"Splice_sites_Annotated": {
"range": {
"min": 26,
"max": 1075869
}
},
"Selection": {
"options": {
"Astrocytes(HEPACAM)": 714,
"Endothelial(BSC)": 123,
"Oligodendrocytes(GC)": 294,
"Neurons(Thy1)": 685,
"Microglia(CD45)": 1108,
"Unpanned": 665
}
}
}
*/
const obsSummary = _(schema.annotations.obs)
.keyBy("name")
@@ -115,7 +118,8 @@ function summarizeAnnotations(schema, obsAnnotations) {
})
.value();
const varSummary = {}; // TODO XXX - not currently used, so skip it
// TODO XXX - not currently used, so skip it
const varSummary = {};
return {
obs: obsSummary,
@@ -124,9 +128,6 @@ function summarizeAnnotations(schema, obsAnnotations) {
}
function templateWorld() {
const VarDataCacheLowWatermark = 32;
const VarDataCacheTTLMs = 1000;
return {
// map from universe obsIndex to world offset.
// Undefined / null indicates identity mapping.
@@ -186,6 +187,13 @@ export function createWorldFromEntireUniverse(universe) {
/* derived data & summaries */
world.summary = summarizeAnnotations(world.schema, world.obsAnnotations);
/* build the varDataCache */
world.varDataCache = kvCache.map(
universe.varDataCache,
val => subsetVarData(world, universe, val),
{ lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
);
return world;
}
@@ -227,13 +235,21 @@ export function createWorldFromCurrentSelection(universe, world, crossfilter) {
// build index to our world offset
newWorld.worldObsIndex.fill(-1); // default - aka unused
for (let i = 0; i < newWorld.nObs; i += 1) {
newWorld.worldObsIndex[newWorld.obsAnnotations[i].__obsIndex__] = i;
newWorld.worldObsIndex[newWorld.obsAnnotations[i].__index__] = i;
}
/* derived data & summaries */
newWorld.summary = summarizeAnnotations(
newWorld.schema,
newWorld.obsAnnotations
);
/* build the varDataCache */
newWorld.varDataCache = kvCache.map(
universe.varDataCache,
val => subsetVarData(newWorld, universe, val),
{ lowWatermark: VarDataCacheLowWatermark, minTTL: VarDataCacheTTLMs }
);
return newWorld;
}
@@ -243,20 +259,19 @@ export function createWorldFromCurrentSelection(universe, world, crossfilter) {
*/
function deduceDimensionType(attributes, fieldName) {
let dimensionType;
if (attributes.type === "string") {
const { type } = attributes;
if (type === "string" || type === "categorical" || type === "boolean") {
dimensionType = "enum";
} else if (attributes.type === "int32") {
} else if (type === "int32") {
dimensionType = Int32Array;
} else if (attributes.type === "float32") {
} else if (type === "float32") {
dimensionType = Float32Array;
} else {
/*
Currently not supporting boolean and categorical types.
*/
console.error(
`Warning - REST API returned unknown metadata schema (${
attributes.type
}) for field ${fieldName}.`
`Warning - REST API returned unknown metadata schema (${type}) for field ${fieldName}.`
);
// skip it - we don't know what to do with this type
}
@@ -286,11 +301,11 @@ export function createObsDimensionMap(crossfilter, world) {
*/
const worldIndex = worldObsIndex ? idx => worldObsIndex[idx] : idx => idx;
dimensionMap.x = crossfilter.dimension(
r => obsLayout.X[worldIndex(r.__obsIndex__)],
r => obsLayout.X[worldIndex(r.__index__)],
Float32Array
);
dimensionMap.y = crossfilter.dimension(
r => obsLayout.Y[worldIndex(r.__obsIndex__)],
r => obsLayout.Y[worldIndex(r.__index__)],
Float32Array
);
@@ -309,7 +324,7 @@ export function subsetVarData(world, universe, varData) {
const newVarData = new Float32Array(world.nObs);
for (let i = 0; i < world.nObs; i += 1) {
newVarData[i] = varData[world.obsAnnotations[i].__obsIndex__];
newVarData[i] = varData[world.obsAnnotations[i].__index__];
}
return newVarData;
}

15
server/app/app.py
View File

@@ -9,11 +9,13 @@ from flask_cors import CORS
from flask_restful_swagger_2 import get_swagger_blueprint
from .rest_api.rest import get_api_resources
from .util.utils import Float32JSONEncoder
from .web import webapp
REACTIVE_LIMIT = 1_000_000
app = Flask(__name__, static_folder="web/static")
app.json_encoder = Float32JSONEncoder
cache = Cache(app, config={"CACHE_TYPE": "simple", "CACHE_DEFAULT_TIMEOUT": 860000})
Compress(app)
CORS(app)
@@ -54,9 +56,12 @@ def run_scanpy(args):
)
from .scanpy_engine.scanpy_engine import ScanpyEngine
app.data = ScanpyEngine(args.data_directory, schema="data_schema.json",
graph_method=args.layout, diffexp_method=args.diffexp)
app.run(host="127.0.0.1", debug=True, port=args.port)
app.data = ScanpyEngine(args.data_directory, layout_method=args.layout, diffexp_method=args.diffexp)
if args.bind_all:
host = "0.0.0.0"
else:
host = "127.0.0.1"
app.run(host=host, debug=True, port=args.port)
def main():
@@ -64,6 +69,10 @@ def main():
parser.add_argument("--title", "-t", help="Title to display -- if this is omitted the title will be the name "
"of the directory from the data_directory arg")
parser.add_argument("--port", help="Port to run server on.", type=int, default=5005)
parser.add_argument(
"--bind-all",
help="Bind to all interfaces (this makes the server accessible beyond this computer)",
action="store_true")
subparsers = parser.add_subparsers(dest="cellxgene_command")
try:
from .scanpy_engine.scanpy_engine import ScanpyEngine

81
server/app/driver/driver.py
View File

@@ -2,18 +2,38 @@ from abc import ABCMeta, abstractmethod
class CXGDriver(metaclass=ABCMeta):
def __init__(self, data, schema=None, graph_method=None, diffexp_method=None):
def __init__(self, data, layout_method=None, diffexp_method=None):
self.data = self._load_data(data)
self.layout_method = layout_method
self.diffexp_method = diffexp_method
self.cluster = None
@property
def features(self):
features = {
"cluster": {"available": False},
"layout": {
"obs": {"available": False},
"var": {"available": False},
},
"diffexp": {"available": False}
}
# TODO - Interactive limit should be generated from the actual available methods see GH issue #94
if self.layout_method:
# TODO handle "var" when gene layout becomes available
features["layout"]["obs"] = {"available": True, "interactiveLimit": 15000}
if self.diffexp_method:
features["diffexp"] = {"available": True, "interactiveLimit": 5000}
if self.cluster:
features["cluster"] = {"available": True, "interactiveLimit": 45000}
return features
@staticmethod
@abstractmethod
def _load_data(data):
pass
@abstractmethod
def _load_or_infer_schema(data):
pass
@abstractmethod
def cells(self):
pass
@@ -23,33 +43,32 @@ class CXGDriver(metaclass=ABCMeta):
pass
@abstractmethod
def filter_cells(self, filter):
def filter_dataframe(self, filter):
"""
Filter cells from data and return a subset of the data
A filter is a dictionary where the key is a metadatata category
Value is dictionary
value_type: int, float, string
variable_type: continuous, categorical
query: filter value, for categorical [val1, val2], for continuous {min: x, max:y}
Filters are combined with the and operator
:param filter:
:return: filtered dataframe
Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
indexing and filtering by annotation value. Filters are combined with the and operator.
See REST specs for info on filter format:
https://docs.google.com/document/d/1Fxjp1SKtCk7l8QP9-7KAjGXL0eldi_qEnNT0NmlGzXI/edit#heading=h.8qc9q57amldx
:param filter: dictionary with filter parames
:return: View into scanpy object with cells/genes filtered
"""
pass
@abstractmethod
def metadata(self, df, fields=None):
def annotation(self, df, axis, fields=None):
"""
Gets metadata key:value for each cells
Gets annotation value for each observation
:param axis:
:param df: from filter_cells, dataframe
:param fields: list of keys for metadata to return, returns all metadata values if not set.
:return: list of metadata values
:param fields: list of keys for annotation to return, returns all annotation values if not set.
:return: dict: names - list of fields in order, data - list of lists or metadata [idx, val1, val2...]
"""
pass
@abstractmethod
def create_graph(self, df):
def layout(self, df):
"""
Computes a n-d layout for cells through dimensionality reduction.
:param df: from filter_cells, dataframe
@@ -58,24 +77,26 @@ class CXGDriver(metaclass=ABCMeta):
pass
@abstractmethod
def diffexp(self, df1, df2):
def diffexp(self, df1, df2, genes):
"""
Computes the top differentially expressed genes between two clusters
:param df1: from filter_cells, dataframe containing first set of cells
:param df2: from filter_cells, dataframe containing second set of cells
:return: top genes, stats and expression values for top genes
Computes the top differentially expressed variables between two observation sets. If dataframes
contain a subset of variables, then statistics for all variables will be returned, otherwise
only the top N vars will be returned.
:param df1: from filter_cells, dataframe containing first set of observations
:param df2: from filter_cells, dataframe containing second set of observations
:param topN: Limit results to top N (Top var mode only)
:return: top genes, stats and expression values for variables
"""
pass
@abstractmethod
def expression(self, df):
def data_frame(self, df):
"""
Retrieves expression for each gene for cells in data frame
:param df:
:param df: from filter_cells, dataframe
:return: {
"genes": list of genes,
"cells": list of cells and expression list,
"nonzero_gene_count": number of nonzero genes
"var": list of variable ids,
"obs": [cellid, var1 expression, var2 expression, ...],
}
"""
pass

880
server/app/rest_api/rest.py
View File

@@ -1,435 +1,539 @@
from http import HTTPStatus
import pkg_resources
from flask import (
Blueprint, request, current_app
Blueprint, current_app, jsonify, make_response, request
)
from flask_restful_swagger_2 import Api, swagger, Resource
from werkzeug.datastructures import ImmutableMultiDict
from server.app.util.utils import make_payload
from server.app.util.filter import parse_filter
from server.app.util.constants import Axis, DiffExpMode
from server.app.util.filter import parse_filter, QueryStringError
from server.app.util.models import FilterModel
class InitializeAPI(Resource):
class SchemaAPI(Resource):
@swagger.doc({
"summary": "get metadata schema, ranges for values, and cell count to initialize cellxgene app",
"summary": "get schema for dataframe and annotations",
"tags": ["initialize"],
"parameters": [],
"responses": {
"200": {
"description": "initialization data for UI",
"description": "schema",
"examples": {
"application/json": {
"data": {
"cellcount": 3589,
"options": {
"Sample.type": {
"options": {
"Glioblastoma": 3589
}
},
"Selection": {
"options": {
"Astrocytes(HEPACAM)": 714,
"Endothelial(BSC)": 123,
"Microglia(CD45)": 1108,
"Neurons(Thy1)": 685,
"Oligodendrocytes(GC)": 294,
"Unpanned": 665
}
},
"Splice_sites_AT.AC": {
"range": {
"max": 1025,
"min": 152
}
},
"Splice_sites_Annotated": {
"range": {
"max": 1075869,
"min": 26
}
}
"schema": {
"dataframe": {
"nObs": 383,
"nVar": 19944,
"type": "float32"
},
"schema": {
"CellName": {
"displayname": "Name",
"type": "string",
"variabletype": "categorical"
},
"Class": {
"displayname": "Class",
"type": "string",
"variabletype": "categorical"
},
"ERCC_reads": {
"displayname": "ERCC Reads",
"type": "int",
"variabletype": "continuous"
},
"ERCC_to_non_ERCC": {
"displayname": "ERCC:Non-ERCC",
"type": "float",
"variabletype": "continuous"
},
"Genes_detected": {
"displayname": "Genes Detected",
"type": "int",
"variabletype": "continuous"
}
},
"genes": ["1/2-SBSRNA4", "A1BG", "A1BG-AS1"]
},
"status": {
"error": False,
"errormessage": ""
"annotations": {
"obs": [
{"name": "name", "type": "string"},
{"name": "tissue_type", "type": "string"},
{"name": "num_reads", "type": "int32"},
{"name": "sample_name", "type": "string"},
{
"name": "clusters",
"type": "categorical",
"categories": [99, 1, "unknown cluster"]
},
{"name": "QScore", "type": "float32"}
],
"var": [
{"name": "name", "type": "string"},
{"name": "gene", "type": "string"}
]
}
}
}
}
}
}
})
def get(self):
from server.app.app import REACTIVE_LIMIT
return make_payload({
"schema": current_app.data.schema,
"cellcount": current_app.data.cell_count,
"reactivelimit": REACTIVE_LIMIT,
"genes": current_app.data.genes(),
"ranges": current_app.data.metadata_ranges(),
})
return make_response(jsonify({"schema": current_app.data.schema}), 200)
class CellsAPI(Resource):
class ConfigAPI(Resource):
@swagger.doc({
"summary": "filter based on metadata fields to get a subset cells, expression data, and metadata",
"tags": ["cells"],
"description": "Cells takes query parameters defined in the schema retrieved from the /initialize enpoint. "
"<br>For categorical metadata keys filter based on `key=value` <br>"
" For continuous metadata keys filter by `key=min,max`<br> Either value "
"can be replaced by a \*. To have only a minimum value `key=min,\*` To have only a maximum "
"value `key=\*,max` <br>Graph data (if retrieved) is normalized"
" To only retrieve cells that don't have a value for the key filter by `key`",
"summary": "Configuration information to assist in front-end adaptation"
" to underlying engine, available functionality, interactive time limits, etc",
"tags": ["initialize"],
"parameters": [],
"responses": {
"200": {
"description": "initialization data for UI",
"description": "schema",
"examples": {
"application/json": {
"data": {
"badmetadatacount": 0,
"cellcount": 0,
"cellids": ["..."],
"metadata": [
"config": {
"features": [
{"method": "POST", "path": "/cluster/", "available": False},
{
"CellName": "1001000173.G8",
"Class": "Neoplastic",
"Cluster_2d": "11",
"Cluster_2d_color": "#8C564B",
"Cluster_CNV": "1",
"Cluster_CNV_color": "#1F77B4",
"ERCC_reads": "152104",
"ERCC_to_non_ERCC": "0.562454470489481",
"Genes_detected": "1962",
"Location": "Tumor",
"Location.color": "#FF7F0E",
"Multimapping_reads_percent": "2.67",
"Neoplastic": "Neoplastic",
"Non_ERCC_reads": "270429",
"Sample.name": "BT_S2",
"Sample.name.color": "#AEC7E8",
"Sample.type": "Glioblastoma",
"Sample.type.color": "#1F77B4",
"Selection": "Unpanned",
"Selection.color": "#98DF8A",
"Splice_sites_AT.AC": "102",
"Splice_sites_Annotated": "122397",
"Splice_sites_GC.AG": "761",
"Splice_sites_GT.AG": "125741",
"Splice_sites_non_canonical": "56",
"Splice_sites_total": "126660",
"Total_reads": "1741039",
"Unique_reads": "1400382",
"Unique_reads_percent": "80.43",
"Unmapped_mismatch": "2.15",
"Unmapped_other": "0.18",
"Unmapped_short": "14.56",
"housekeeping_cluster": "2",
"housekeeping_cluster_color": "#AEC7E8",
"recluster_myeloid": "NA",
"recluster_myeloid_color": "NA"
"method": "POST",
"path": "/layout/obs",
"available": True,
"interactiveLimit": 10000
},
],
"reactive": True,
"graph": [
[
"1001000173.G8",
0.93836,
0.28623
],
[
"1001000173.D4",
0.1662,
0.79438
]
{"method": "POST", "path": "/layout/var", "available": False}
],
"status": {
"error": False,
"errormessage": ""
}
},
}
},
},
"400": {
"description": "bad query params",
}
}
})
def get(self):
payload = {
"metadata": [],
"cellcount": 0,
"graph": [],
"ranges": {},
}
# get query params
cells_filter = parse_filter(request.args, current_app.data.schema)
filtered_data = current_app.data.filter_cells(cells_filter)
payload["metadata"] = current_app.data.metadata(filtered_data)
payload["ranges"] = current_app.data.metadata_ranges(filtered_data)
payload["graph"] = current_app.data.create_graph(filtered_data)
payload["cellcount"] = current_app.data.cell_count
return make_payload(payload)
class ExpressionAPI(Resource):
@swagger.doc({
"summary": "Json with gene list and expression data by cell, limited to first 40 cells",
"tags": ["expression"],
"parameters": [
{
"name": "include_unexpressed_genes",
"description": "Include genes that have 0 expression across all cells in set",
"in": "path",
"type": "bool",
}
],
"responses": {
"200": {
"description": "Json for heatmap",
"examples": {
"application/json": {
"data": {
"cells": [
{
"cellname": "1/2-SBSRNA4",
"e": [0, 0, 214, 0, 0]
},
],
"genes": [
"1001000173.G8",
"1001000173.D4",
"1001000173.B4",
"1001000173.A2",
"1001000173.E2"
],
"nonzero_gene_count": 2857
},
"status": {
"error": False,
"errormessage": ""
}
}
}
}
}
})
def get(self):
expression_data = current_app.data.expression()
return make_payload(expression_data)
@swagger.doc({
"summary": "Json with gene list and expression data by cell",
"tags": ["expression"],
"parameters": [
{
"name": "body",
"in": "body",
"schema": {
"example": {
"celllist": ["1001000173.G8", "1001000173.D4"],
"genelist": ["1/2-SBSRNA4", "A1BG", "A1BG-AS1", "A1CF", "A2LD1", "A2M", "A2ML1", "A2MP1",
"A4GALT"],
"include_unexpressed_genes": True,
}
}
},
],
"responses": {
"200": {
"description": "Json for expressiondata",
"examples": {
"application/json": {
"data": {
"cells": [
{
"cellname": "1001000173.D4",
"e": [0, 0]
},
{
"cellname": "1001000173.G8",
"e": [0, 0]
}
],
"genes": [
"ABCD4",
"ZWINT"
],
"nonzero_gene_count": 2857
},
"status": {
"error": False,
"errormessage": ""
}
}
}
},
"400": {
"description": "Required parameter missing/incorrect",
}
}
})
def post(self):
args = request.get_json()
cell_list = args.get("celllist", [])
gene_list = args.get("genelist", [])
if not cell_list and not gene_list:
return make_payload([], "must include celllist and/or genelist parameter", 400)
expression_data = current_app.data.expression(cell_list, gene_list)
if cell_list and len(expression_data["cells"]) < len(cell_list):
return make_payload([], "Some cell ids not available", 400)
if gene_list and len(expression_data["genes"]) < len(gene_list):
return make_payload([], "Some genes not available", 400)
return make_payload(expression_data)
class DifferentialExpressionAPI(Resource):
@swagger.doc({
"summary": "Get the top expressed genes for two cell sets. Calculated using t-test",
"tags": ["expression"],
"parameters": [
{
"name": "body",
"in": "body",
"schema": {
"example": {
"celllist1": ["1001000176.C12", "1001000176.C7", "1001000177.F11"],
"celllist2": ["1001000012.D2", "1001000017.F10", "1001000033.C3", "1001000229.D4"],
"num_genes": 5,
"pval": 0.000001,
},
}
}
],
"responses": {
"200": {
"description": "top expressed genes for cellset1, cellset2",
"examples": {
"application/json": {
"data": {
"celllist1": {
"ave_diff": [
432.0132935431362,
12470.5623982637,
957.0246880086814
],
"mean_expression_cellset1": [
438.6185567010309,
13315.536082474227,
1076.5773195876288
],
"mean_expression_cellset2": [
6.605263157894737,
844.9736842105264,
119.55263157894737
],
"pval": [
3.8906598089944563e-35,
1.9086226376018916e-25,
7.847480544069826e-21
],
"topgenes": [
"TMSB10",
"FTL",
"TMSB4X"
]
"displayNames": {
"engine": "ScanPy version 1.33",
"dataset": "/home/joe/mouse/blorth.csv"
},
"celllist2": {
"ave_diff": [
-6860.599158979924,
-519.1314432989691,
-10278.328269126423
],
"mean_expression_cellset1": [
2.8350515463917527,
0.6185567010309279,
23.09278350515464
],
"mean_expression_cellset2": [
6863.434210526316,
519.75,
10301.421052631578
],
"pval": [
4.662891833748732e-44,
3.6278087029927103e-37,
8.396825170618402e-35
],
"topgenes": [
"SPARCL1",
"C1orf61",
"CLU"
]
}
},
"status": {
"error": False,
"errormessage": ""
}
}
}
}
}
})
def get(self):
config = {
"config": {
"features": [
{"method": "POST", "path": "/cluster/", **current_app.data.features["cluster"]},
{"method": "POST", "path": "/layout/obs", **current_app.data.features["layout"]["obs"]},
{"method": "POST", "path": "/layout/var", **current_app.data.features["layout"]["var"]},
{"method": "POST", "path": "/diffexp/", **current_app.data.features["diffexp"]},
],
"displayNames": {
"engine": f"cellxgene Scanpy engine version {pkg_resources.get_distribution('cellxgene').version}",
"dataset": current_app.config["DATASET_TITLE"]
}
}
}
return make_response(jsonify(config), 200)
class LayoutObsAPI(Resource):
@swagger.doc({
"summary": "Get the default layout for all observations.",
"tags": ["layout"],
"parameters": [],
"responses": {
"200": {
"description": "layout",
"examples": {
"application/json": {
"layout": {
"ndims": 2,
"coordinates": [
[0, 0.284483, 0.983744],
[1, 0.038844, 0.739444]
]
}
}
}
}
}
})
def get(self):
return make_response((jsonify({"layout": current_app.data.layout(current_app.data.data)})))
@swagger.doc({
"summary": "Observation layout for filtered subset.",
"tags": ["layout"],
"parameters": [
{
"name": "filter",
"description": "Complex Filter",
"in": "body",
"schema": FilterModel
}
],
"responses": {
"200": {
"description": "layout",
"examples": {
"application/json": {
"layout": {
"ndims": 2,
"coordinates": [
[0, 0.284483, 0.983744],
[1, 0.038844, 0.739444]
]
}
}
}
}
}
})
def put(self):
df = current_app.data.filter_dataframe(request.get_json()["filter"])
return make_response((jsonify({"layout": current_app.data.layout(df)})))
class AnnotationsObsAPI(Resource):
@swagger.doc({
"summary": "Fetch annotations (metadata) for all observations.",
"tags": ["annotations"],
"parameters": [{
"in": "query",
"name": "annotation-name",
"type": "string",
"description": "list of 1 or more annotation names"
}],
"responses": {
"200": {
"description": "annotations",
"examples": {
"application/json": {
"names": [
"tissue_type", "sex", "num_reads", "clusters"
],
"data": [
[0, "lung", "F", 39844, 99],
[1, "heart", "M", 83, 1],
[49, "spleen", None, 2, "unknown cluster"],
]
}
}
}
}
})
def get(self):
fields = request.args.getlist("annotation-name", None)
try:
annotation_response = current_app.data.annotation(current_app.data.data, "obs", fields)
except KeyError:
return make_response(f"Error bad key in {fields}", 404)
return make_response(jsonify(annotation_response))
@swagger.doc({
"summary": "Fetch annotations (metadata) for filtered subset of observations.",
"tags": ["annotations"],
"parameters": [
{
"in": "query",
"name": "annotation-name",
"type": "string",
"description": "list of 1 or more annotation names"
},
{
"name": "filter",
"description": "Complex Filter",
"in": "body",
"schema": FilterModel
}
],
"responses": {
"200": {
"description": "annotations",
"examples": {
"application/json": {
"names": [
"tissue_type", "sex", "num_reads", "clusters"
],
"data": [
[0, "lung", "F", 39844, 99],
[1, "heart", "M", 83, 1],
[49, "spleen", None, 2, "unknown cluster"],
]
}
}
}
}
})
def put(self):
fields = request.args.getlist("annotation-name", None)
df = current_app.data.filter_dataframe(request.get_json()["filter"], include_uns=False)
try:
annotation_response = current_app.data.annotation(df, "obs", fields)
except KeyError:
return make_response(f"Error bad key in {fields}", 404)
return make_response(jsonify(annotation_response))
class AnnotationsVarAPI(Resource):
@swagger.doc({
"summary": "Fetch annotations (metadata) for all variables.",
"tags": ["annotations"],
"parameters": [{
"in": "query",
"name": "annotation-name",
"type": "string",
"description": "list of 1 or more annotation names"
}],
"responses": {
"200": {
"description": "annotations",
"examples": {
"application/json": {
"names": [
"name", "category"
],
"data": [
[0, "ATAD3C", 1],
[1, "RER1", None],
[49, "S100B", 6]
]
}
}
}
}
})
def get(self):
fields = request.args.getlist("annotation-name", None)
try:
annotation_response = current_app.data.annotation(current_app.data.data, "var", fields)
except KeyError:
return make_response(f"Error bad key in {fields}", 404)
return make_response(jsonify(annotation_response))
@swagger.doc({
"summary": "Fetch annotations (metadata) for filtered subset of variables.",
"tags": ["annotations"],
"parameters": [
{
"in": "query",
"name": "annotation-name",
"type": "string",
"description": "list of 1 or more annotation names"
},
{
"name": "filter",
"description": "Complex Filter",
"in": "body",
"schema": FilterModel
}
],
"responses": {
"200": {
"description": "annotations",
"examples": {
"application/json": {
"names": [
"name", "category"
],
"data": [
[0, "ATAD3C", 1],
[1, "RER1", None],
[49, "S100B", 6]
]
}
}
}
}
})
def put(self):
fields = request.args.getlist("annotation-name", None)
df = current_app.data.filter_dataframe(request.get_json()["filter"], include_uns=False)
try:
annotation_response = current_app.data.annotation(df, "var", fields)
except KeyError:
return make_response(f"Error bad key in {fields}", 404)
return make_response(jsonify(annotation_response))
class DiffExpObsAPI(Resource):
@swagger.doc({
"summary": "Generate differential expression (DE) statistics for two specified subsets of data, "
"as indicated by the two provided observation complex filters",
"tags": ["diffexp"],
# TODO sort out params
# "parameters": [
# # {
# # "in": "body",
# # "name": "mode",
# # "type": "string",
# # "required": True,
# # "description": "topN or varFilter"
# # },
# {
# "in": "query",
# "name": "count",
# "type": "int32",
# "description": "TopN mode: how many vars to return"
# },
# {
# "in": "body",
# "name": "varFilter",
# "schema": FilterModel,
# "description": "varFilter: Complex filter, only var for which vars to return"
# },
# {
# "in": "body",
# "name": "set1",
# "schema": FilterModel,
# "required": True,
# "description": "Complex filter, only obs - observations in set1"
# },
# {
# "in": "body",
# "name": "set2",
# "schema": FilterModel,
# "description": "Complex filter, only obs - observations in set2. If not included, inverse of set1."
# },
# ],
"responses": {
"200": {
"description": "Statistics are encoded as an array of arrays, with fields ordered as: "
"varIndex, avgDiff, pVal, pValAdj, set1AvgExp, set2AvgExp",
"examples": {
"application/json": [
[328, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9],
[1250, -2.569489, 2.655706e-63, 3.642036e-57, 383.393, 583.9],
]
}
}
}
})
def post(self):
args = request.get_json()
cell_list_1 = args.get("celllist1", [])
cell_list_2 = args.get("celllist2", [])
num_genes = args.get("num_genes", 7)
pval = args.get("pval", 0.5)
if not (cell_list_1 and cell_list_2):
return make_payload([],
"must include celllist1 and celllist2 parameters",
400)
data = current_app.data.diffexp(cell_list_1, cell_list_2, pval, num_genes)
return make_payload(data)
# confirm mode is present and legal
try:
mode = DiffExpMode(args["mode"])
except KeyError:
return make_response("Error: mode is required", 400)
except ValueError:
return make_response(f"Error: invalid mode option {args['mode']}", 400)
# Validate filters
if mode == DiffExpMode.VAR_FILTER:
if "varFilter" not in args:
return make_response("varFilter is required when mode is set to varFilter ", 400)
if Axis.OBS in args["varFilter"]["filter"]:
return make_response("Obs filter not allowed in varFilter", 400)
if "set1" not in args:
return make_response("set1 is required.", 400)
if Axis.VAR in args["set1"]["filter"]:
return make_response("Var filter not allowed for set1", 400)
# set2
if "set2" not in args:
return make_response("Set2 as inverse of set1 is not implemented", 501)
if Axis.VAR in args["set2"]["filter"]:
return make_response("Var filter not allowed for set2", 400)
set1_filter = args["set1"]["filter"]
set2_filter = args.get("set2", {"filter": {}})["filter"]
if "varFilter" in args:
set1_filter[Axis.VAR] = args["varFilter"]["filter"][Axis.VAR]
set2_filter[Axis.VAR] = args["varFilter"]["filter"][Axis.VAR]
df1 = current_app.data.filter_dataframe(set1_filter, include_uns=False)
# TODO inverse
df2 = current_app.data.filter_dataframe(set2_filter, include_uns=False)
# exceeds size limit
if df1.shape[0] + df2.shape[0] > current_app.data.features["diffexp"]["interactiveLimit"]:
return make_response("Non-interactive request", 403)
# mode
count = args.get("count", None)
try:
diffexp = current_app.data.diffexp(df1, df2, count)
except ValueError as ve:
return make_response(ve.message, 400)
return make_response(jsonify(diffexp))
class DataObsAPI(Resource):
@swagger.doc({
"summary": "Get data (expression values) from the dataframe.",
"tags": ["data"],
"parameters": [
{
"in": "query",
"name": "filter",
"type": "string",
"description": "axis:key:value"
},
{
"in": "query",
"name": "accept-type",
"type": "string",
"description": "MIME type"
},
],
"responses": {
"200": {
"description": "expression",
"examples": {
"application/json": {
"var": [0, 20000],
"obs": [
[1, 39483, 3902, 203, 0, 0, 28]
]
}
}
},
"400": {
"description": "Malformed filter"
},
"406": {
"description": "Unacceptable MIME type"
},
}
})
def get(self):
# request.args is immutable
args = dict(request.args)
accept_type = args.pop("accept-type", None)
try:
filter_ = parse_filter(ImmutableMultiDict(args), current_app.data.schema['annotations'])
except QueryStringError as e:
return make_response(e.message, HTTPStatus.BAD_REQUEST)
df = current_app.data.filter_dataframe(filter_, include_uns=False)
if accept_type and accept_type[0] == "application/json":
return make_response((jsonify(current_app.data.data_frame(df))))
# TODO support CSV
else:
return make_response(f"Unsupported accept-type: {accept_type}", HTTPStatus.NOT_ACCEPTABLE)
@swagger.doc({
"summary": "Get data (expression values) from the dataframe.",
"tags": ["data"],
"parameters": [
{
'name': 'filter',
'description': 'Complex Filter',
'in': 'body',
'schema': FilterModel
}
],
"responses": {
"200": {
"description": "expression",
"examples": {
"application/json": {
"var": [0, 20000],
"obs": [
[1, 39483, 3902, 203, 0, 0, 28]
]
}
}
},
"400": {
"description": "Malformed filter"
},
"406": {
"description": "Unacceptable MIME type"
},
}
})
def put(self):
if not request.accept_mimetypes.best_match(["application/json", "text/csv"]):
return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
# TODO catch error for bad filter
df = current_app.data.filter_dataframe(request.get_json()["filter"], include_uns=False)
if request.accept_mimetypes.best_match(['application/json']):
return make_response((jsonify(current_app.data.data_frame(df))))
# TODO support CSV
else:
return make_response(f"Unsupported MIME type '{request.accept_mimetypes}'", HTTPStatus.NOT_ACCEPTABLE)
def get_api_resources():
bp = Blueprint("api", __name__, url_prefix="/api/v0.1")
bp = Blueprint("api", __name__, url_prefix="/api/v0.2")
api = Api(bp, add_api_spec_resource=False)
api.add_resource(InitializeAPI, "/initialize")
api.add_resource(CellsAPI, "/cells")
api.add_resource(ExpressionAPI, "/expression")
api.add_resource(DifferentialExpressionAPI, "/diffexpression")
api.add_resource(SchemaAPI, "/schema")
api.add_resource(ConfigAPI, "/config")
api.add_resource(LayoutObsAPI, "/layout/obs")
api.add_resource(AnnotationsObsAPI, "/annotations/obs")
api.add_resource(DiffExpObsAPI, "/diffexp/obs")
api.add_resource(AnnotationsVarAPI, "/annotations/var")
api.add_resource(DataObsAPI, "/data/obs")
return api

421
server/app/scanpy_engine/scanpy_engine.py
View File

@@ -1,36 +1,68 @@
import os
import warnings
import numpy as np
from pandas import DataFrame, Series
import scanpy.api as sc
from scipy import stats
from server.app.app import cache
# TODO fix memoization so that it correctly identifies the same request
# from server.app.app import cache
from server.app.driver.driver import CXGDriver
from server.app.util.schema_parse import parse_schema
from server.app.util.constants import Axis, DEFAULT_TOP_N, DiffExpMode
class ScanpyEngine(CXGDriver):
def __init__(self, data, schema=None, graph_method="umap", diffexp_method="ttest"):
self.data = self._load_data(data)
self.schema = self._load_or_infer_schema(data, schema)
self._set_cell_names()
def __init__(self, data, layout_method=None, diffexp_method=None):
super().__init__(data, layout_method=layout_method, diffexp_method=diffexp_method)
self._validatate_data_types()
self._add_mandatory_annotations()
self.cell_count = self.data.shape[0]
self.gene_count = self.data.shape[1]
self.graph_method = graph_method
self.diffexp_method = diffexp_method
self._create_schema()
self.layout(self.data)
def _set_cell_names(self):
self.data.obs["cell_name"] = list(self.data.obs.index)
def _create_schema(self):
self.schema = {
"dataframe": {
"nObs": self.cell_count,
"nVar": self.gene_count,
"type": str(self.data.X.dtype)
},
"annotations": {
"obs": [],
"var": []
}
}
for ax in Axis:
curr_axis = getattr(self.data, str(ax))
for ann in curr_axis:
ann_schema = {"name": ann}
data_kind = curr_axis[ann].dtype.kind
if data_kind == 'f':
ann_schema["type"] = "float32"
elif data_kind in ['i', 'u']:
ann_schema["type"] = "int32"
elif data_kind == "?":
ann_schema["type"] = "boolean"
elif data_kind == "O" and curr_axis[ann].dtype == "object":
ann_schema["type"] = "string"
elif data_kind == "O" and curr_axis[ann].dtype == "category":
ann_schema["type"] = "categorical"
ann_schema["categories"] = curr_axis[ann].dtype.categories.tolist()
else:
raise TypeError(f"Annotations of type {curr_axis[ann].dtype} are unsupported by cellxgene.")
self.schema["annotations"][ax].append(ann_schema)
@classmethod
def add_to_parser(cls, subparsers, invocation_function):
scanpy_group = subparsers.add_parser("scanpy", help="run cellxgene using the scanpy engine")
# TODO these choices should be generated from the actual available methods
# TODO these choices should be generated from the actual available methods see GH issue #94
scanpy_group.add_argument("-l", "--layout", choices=["umap", "tsne"], default="umap",
help="Algorithm to use for graph layout")
scanpy_group.add_argument("-d", "--diffexp", choices=["ttest"], default="ttest",
help="Algorithm to use to calculate differential expression")
help="Algorithm to used to calculate differential expression")
scanpy_group.add_argument("data_directory", metavar="dir", help="Directory containing data and schema file")
scanpy_group.set_defaults(func=invocation_function)
return scanpy_group
@@ -39,206 +71,227 @@ class ScanpyEngine(CXGDriver):
def _load_data(data):
return sc.read(os.path.join(data, "data.h5ad"))
def _load_or_infer_schema(self, data, schema):
if not os.path.isfile(os.path.join(data, schema)):
# Initialize with cell name which is built off the index
data_schema = {
"CellName": {
"type": "string",
"variabletype": "categorical",
"displayname": "Name",
"include": True
}
}
metadata_fields = list(self.data.obs)
for m in metadata_fields:
# Since there are many type of float/int in numpy datatypes the kind attribute of a datatype object
# offers a decent insight into whether it can be lumped in with floats or ints, which is what we
# care about here.
data_kind = self.data.obs[m].dtype.kind
variable_type = "categorical"
data_type = "string"
if data_kind == 'f':
variable_type = "continuous"
data_type = "float"
elif data_kind in ['i', 'u']:
data_type = "int"
if self.data.obs[m].nunique() > 50:
variable_type = "continuous"
data_schema[m] = {
"type": data_type,
"variabletype": variable_type,
"displayname": m,
"include": True
}
else:
data_schema = parse_schema(os.path.join(data, schema))
return data_schema
@staticmethod
def _top_sort(values, sort_order, top_n=None):
"""
Sorts an iterable in sort order limited by top_n
:param values: iterable of values to sort
:param sort_order: ndarray order to sort in
:param top_n: cutoff number to return
:return: values sorted by sort_order limited by top_n
"""
return values[sort_order][:top_n]
def _add_mandatory_annotations(self):
# ensure gene
self.data.var["name"] = Series(list(self.data.var.index), dtype="unicode_", index=self.data.var.index)
self.data.var.index = Series(list(range(self.data.var.shape[0])), dtype="category")
# ensure cell name
self.data.obs["name"] = Series(list(self.data.obs.index), dtype="unicode_", index=self.data.obs.index)
self.data.obs.index = Series(list(range(self.data.obs.shape[0])), dtype="category")
def _validatate_data_types(self):
if self.data.X.dtype != "float32":
warnings.warn(f"Scanpy data matrix is in {self.data.X.dtype} format not float32. "
f"Precision may be truncated.")
for ax in Axis:
curr_axis = getattr(self.data, str(ax))
for ann in curr_axis:
datatype = curr_axis[ann].dtype
downcast_map = {'int64': 'int32',
'uint32': 'int32',
'uint64': 'int32',
'float64': 'float32',
}
if datatype in downcast_map:
warnings.warn(f"Scanpy annotation {ax}:{ann} is in unsupported format: {datatype}. "
f"Data will be downcast to {downcast_map[datatype]}.")
def cells(self):
return list(self.data.obs.index)
return self.data.obs.index.tolist()
def genes(self):
return self.data.var.index.tolist()
# Can't seem to cache a view of a dataframe, need to investigate why
def filter_cells(self, filter):
def filter_dataframe(self, filter, include_uns=True):
"""
Filter cells from data and return a subset of the data
A filter is a dictionary where the key is a metadatata category
Value is dictionary
value_type: int, float, string
variable_type: continuous, categorical
query: filter value, for categorical [val1, val2], for continuous {min: x, max:y}
Filters are combined with the and operator
:param filter:
:return: filtered dataframe
Filter cells from data and return a subset of the data. They can operate on both obs and var dimension with
indexing and filtering by annotation value. Filters are combined with the and operator.
See REST specs for info on filter format:
# TODO update this link to swagger when it's done
https://docs.google.com/document/d/1Fxjp1SKtCk7l8QP9-7KAjGXL0eldi_qEnNT0NmlGzXI/edit#heading=h.8qc9q57amldx
:param filter: dictionary with filter parames
:param include_uns: bool, include unstructured annotations
:return: View into scanpy object with cells/genes filtered
"""
cell_idx = np.ones((self.cell_count,), dtype=bool)
for key, value in filter.items():
if value["variable_type"] == "categorical":
key_idx = np.in1d(getattr(self.data.obs, key), value["query"])
cell_idx = np.logical_and(cell_idx, key_idx)
cells_idx = np.ones((self.cell_count,), dtype=bool)
genes_idx = np.ones((self.gene_count,), dtype=bool)
if Axis.OBS in filter:
if "index" in filter["obs"]:
cells_idx = self._filter_index(filter["obs"]["index"], cells_idx, Axis.OBS)
if "annotation_value" in filter["obs"]:
cells_idx = self._filter_annotation(filter["obs"]["annotation_value"], cells_idx, Axis.OBS)
if Axis.VAR in filter:
if "index" in filter["var"]:
genes_idx = self._filter_index(filter["var"]["index"], genes_idx, Axis.VAR)
if "annotation_value" in filter["var"]:
genes_idx = self._filter_annotation(filter["var"]["annotation_value"], genes_idx, Axis.VAR)
# Due to anndata issues we can't index into cells and genes at the same time
cell_data = self.data[cells_idx, :]
data = cell_data[:, genes_idx]
# TODO: tmp hack to avoid problems with filter that is limited to single gene
if include_uns:
data.uns = cell_data.uns
return data
def _filter_index(self, filter, index, axis):
"""
Filter data based on index. ex. [1, 3, [111:200]]
:param filter: subset of filter dict for obs/var:index
:param index: np logical vector containing true for passing false for failing filter
:param axis: Axis
:return: np logical vector for whether the data passes the filter
"""
if axis == Axis.OBS:
count_ = self.cell_count
elif axis == Axis.VAR:
count_ = self.gene_count
idx_filter = np.zeros((count_,), dtype=bool)
for i in filter:
if type(i) == list:
idx_filter[i[0]:i[1]] = True
else:
min_ = value["query"]["min"]
max_ = value["query"]["max"]
idx_filter[i] = True
return np.logical_and(index, idx_filter)
def _filter_annotation(self, filter, index, axis):
"""
Filter data based on annotation value
:param filter: subset of filter dict for obs/var:annotation_value
:param index: np logical vector containing true for passing false for failing filter
:param axis: string obs or var
:return: np logical vector for whether the data passes the filter
"""
d_axis = getattr(self.data, axis.value)
for v in filter:
if d_axis[v["name"]].dtype.name in ["boolean", "category", "object"]:
key_idx = np.in1d(getattr(d_axis, v["name"]), v["values"])
index = np.logical_and(index, key_idx)
else:
min_ = v.get("min", None)
max_ = v.get("max", None)
if min_ is not None:
key_idx = np.array((getattr(self.data.obs, key) >= min_).data)
cell_idx = np.logical_and(cell_idx, key_idx)
key_idx = (getattr(d_axis, v["name"]) >= min_).ravel()
index = np.logical_and(index, key_idx)
if max_ is not None:
key_idx = np.array((getattr(self.data.obs, key) <= max_).data)
cell_idx = np.logical_and(cell_idx, key_idx)
return self.data[cell_idx, :]
key_idx = (getattr(d_axis, v["name"]) <= max_).ravel()
index = np.logical_and(index, key_idx)
return index
@cache.memoize()
def metadata_ranges(self, df=None):
metadata_ranges = {}
if not df:
df = self.data
for field in self.schema:
if self.schema[field]["variabletype"] == "categorical":
group_by = field
if group_by == "CellName":
group_by = "cell_name"
metadata_ranges[field] = {"options": df.obs.groupby(group_by).size().to_dict()}
else:
metadata_ranges[field] = {
"range": {
"min": df.obs[field].min(),
"max": df.obs[field].max()
}
}
return metadata_ranges
@cache.memoize()
def metadata(self, df, fields=None):
# @cache.memoize()
def annotation(self, df, axis, fields=None):
"""
Gets metadata key:value for each cells
Gets annotation value for each observation
:param axis:
:param df: from filter_cells, dataframe
:param fields: list of keys for metadata to return, returns all metadata values if not set.
:return: list of metadata values
:param fields: list of keys for annotation to return, returns all annotation values if not set.
:return: dict: names - list of fields in order, data - list of lists or metadata
[observation ids, val1, val2...]
"""
metadata = df.obs.to_dict(orient="records")
for idx in range(len(metadata)):
metadata[idx]["CellName"] = metadata[idx].pop("cell_name", None)
return metadata
df_axis = getattr(df, axis)
if not fields:
fields = df_axis.columns.tolist()
annotations = DataFrame(df_axis[fields], index=df_axis.index)
return {
"names": fields,
"data": annotations.reset_index().values.tolist()
}
@cache.memoize()
def create_graph(self, df):
# @cache.memoize()
def layout(self, df):
"""
Computes a n-d layout for cells through dimensionality reduction.
:param df: from filter_cells, dataframe
:return: [cellid, x, y]
:return: [cellid, x, y, ...]
"""
getattr(sc.tl, self.graph_method)(df, random_state=123)
graph = df.obsm["X_{graph_method}".format(graph_method=self.graph_method)]
normalized_graph = (graph - graph.min()) / (graph.max() - graph.min())
return np.hstack((df.obs["cell_name"].values.reshape(len(df.obs.index), 1), normalized_graph)).tolist()
@cache.memoize()
def diffexp(self, cell_list_1, cell_list_2, pval, num_genes):
"""
Computes the top differentially expressed genes between two clusters
:param df1: from filter_cells, dataframe containing first set of cells
:param df2: from filter_cells, dataframe containing second set of cells
:return: top genes, stats and expression values for top genes
"""
cells_idx_1 = np.in1d(self.data.obs["cell_name"], cell_list_1)
cells_idx_2 = np.in1d(self.data.obs["cell_name"], cell_list_2)
expression_1 = self.data.X[cells_idx_1, :]
expression_2 = self.data.X[cells_idx_2, :]
diff_exp = stats.ttest_ind(expression_1, expression_2)
# TODO break this up into functions
set1 = np.logical_and(diff_exp.pvalue < pval, diff_exp.statistic > 0)
set2 = np.logical_and(diff_exp.pvalue < pval, diff_exp.statistic < 0)
stat1 = diff_exp.statistic[set1]
stat2 = diff_exp.statistic[set2]
sort_set1 = np.argsort(stat1)[::-1]
sort_set2 = np.argsort(stat2)
pval1 = diff_exp.pvalue[set1][sort_set1]
pval2 = diff_exp.pvalue[set2][sort_set2]
mean_ex1_set1 = np.mean(expression_1[:, set1], axis=0)[sort_set1]
mean_ex2_set1 = np.mean(expression_2[:, set1], axis=0)[sort_set1]
mean_ex1_set2 = np.mean(expression_1[:, set2], axis=0)[sort_set2]
mean_ex2_set2 = np.mean(expression_2[:, set2], axis=0)[sort_set2]
mean_diff1 = mean_ex1_set1 - mean_ex2_set1
mean_diff2 = mean_ex1_set2 - mean_ex2_set2
genes_cellset_1 = self.data.var_names[set1][sort_set1]
genes_cellset_2 = self.data.var_names[set2][sort_set2]
# TODO Filtering cells is fine, but filtering genes does nothing because the neighbors are
# calculated using the original vars (geneset) and this doesn’t get updated when you use less.
# Need to recalculate neighbors (long) if user requests new layout filtered by var
getattr(sc.tl, self.layout_method)(df, random_state=123)
df_layout = df.obsm[f"X_{self.layout_method}"]
normalized_layout = DataFrame((df_layout - df_layout.min()) / (df_layout.max() - df_layout.min()),
index=df.obs.index)
return {
"celllist1": {
"topgenes": genes_cellset_1.tolist()[:num_genes],
"mean_expression_cellset1": mean_ex1_set1.tolist()[:num_genes],
"mean_expression_cellset2": mean_ex2_set1.tolist()[:num_genes],
"pval": pval1.tolist()[:num_genes],
"ave_diff": mean_diff1.tolist()[:num_genes]
},
"celllist2": {
"topgenes": genes_cellset_2.tolist()[:num_genes],
"mean_expression_cellset1": mean_ex1_set2.tolist()[:num_genes],
"mean_expression_cellset2": mean_ex2_set2.tolist()[:num_genes],
"pval": pval2.tolist()[:num_genes],
"ave_diff": mean_diff2.tolist()[:num_genes]
},
"ndims": normalized_layout.shape[1],
# reset_index gets obs' id into output
"coordinates": normalized_layout.reset_index().values.tolist()
}
@cache.memoize()
def expression(self, cells=None, genes=None):
# @cache.memoize()
def diffexp(self, df1, df2, top_n=None):
"""
Retrieves expression for each gene for cells in data frame
:param df:
Computes the top differentially expressed variables between two observation sets. If dataframes
contain a subset of variables, then statistics for all variables will be returned, otherwise
only the top N vars will be returned.
:param df1: from filter_cells, dataframe containing first set of observations
:param df2: from filter_cells, dataframe containing second set of observations
:param topN: Limit results to top N (Top var mode only)
:return: top genes, stats and expression values for variables
"""
# If not the same genes, test is wrong!
if np.any(df1.var.index != df2.var.index):
raise ValueError("Variables ares not the same in set1 and set2")
# If not all genes, they used a var filter
if df1.var.shape[0] < self.gene_count:
mode = DiffExpMode.VAR_FILTER
if top_n:
raise Warning("Top N was specified but will not be used in 'Var Filter' mode")
else:
mode = DiffExpMode.TOP_N
if not top_n:
top_n = DEFAULT_TOP_N
genes_idx = df1.var.index
diffexp_result = stats.ttest_ind(df1.X, df2.X)
pval = diffexp_result.pvalue
bonferroni_pval = 1 - (1 - pval) ** self.gene_count
ave_exp_set1 = np.mean(df1.X, axis=0)
ave_exp_set2 = np.mean(df2.X, axis=0)
ave_diff = ave_exp_set1 - ave_exp_set2
if mode == DiffExpMode.TOP_N:
sort_order = np.argsort(np.abs(diffexp_result.statistic))[::-1]
# If top_n > length it will just return length
genes = self._top_sort(genes_idx, sort_order, top_n)
pval = self._top_sort(pval, sort_order, top_n)
bonferroni_pval = self._top_sort(bonferroni_pval, sort_order, top_n)
ave_exp_set1 = self._top_sort(ave_exp_set1, sort_order, top_n)
ave_exp_set2 = self._top_sort(ave_exp_set2, sort_order, top_n)
ave_diff = self._top_sort(ave_diff, sort_order, top_n)
# varIndex, avgDiff, pVal, pValAdj, set1AvgExp, set2AvgExp
result = []
for i in range(len(genes)):
result.append([genes[i], ave_diff[i], pval[i], bonferroni_pval[i], ave_exp_set1[i], ave_exp_set2[i]])
# Results need to be returned in var index order
return sorted(result, key=lambda gene: gene[0])
# @cache.memoize()
def data_frame(self, df):
"""
Retrieves data for each variable for observations in data frame
:param df: from filter_cells, dataframe
:return: {
"genes": list of genes,
"cells": list of cells and expression list,
"nonzero_gene_count": number of nonzero genes
"var": list of variable ids,
"obs": [cellid, var1 expression, var2 expression, ...],
}
"""
if cells:
cells_idx = np.in1d(self.data.obs["cell_name"], cells)
else:
cells_idx = np.ones((self.cell_count,), dtype=bool)
if genes:
genes_idx = np.in1d(self.data.var_names, genes)
else:
genes_idx = np.ones((self.gene_count,), dtype=bool)
index = np.ix_(cells_idx, genes_idx)
expression = self.data.X[index]
if not genes:
genes = self.data.var.index.tolist()
if not cells:
cells = self.data.obs["cell_name"].tolist()
cell_data = []
for idx, cell in enumerate(cells):
cell_data.append({
"cellname": cell,
"e": list(expression[idx]),
})
var_index = df.var.index.tolist()
expression = DataFrame(df.X, index=df.obs.index)
return {
"genes": genes,
"cells": cell_data,
"nonzero_gene_count": int(np.sum(expression.any(axis=0)))
"var": var_index,
"obs": expression.reset_index().values.tolist()
}

27
server/app/util/constants.py Normal file
View File

@@ -0,0 +1,27 @@
from enum import Enum
DEFAULT_TOP_N = 10
class AugmentedEnum(Enum):
def __hash__(self):
return self.value.__hash__()
def __eq__(self, other):
if isinstance(other, type(self)) or isinstance(other, str):
return self.value == other
return False
def __str__(self) -> str:
return self.value
class Axis(AugmentedEnum):
OBS = "obs"
VAR = "var"
class DiffExpMode(AugmentedEnum):
TOP_N = "topN"
VAR_FILTER = "varFilter"

103
server/app/util/filter.py
View File

@@ -1,5 +1,16 @@
import json
from collections import defaultdict
from numpy import float32, int32
from server.app.util.constants import Axis
class QueryStringError(Exception):
pass
def __init__(self, key, message):
self.key = key
self.message = message
def _convert_variable(datatype, variable):
@@ -9,62 +20,68 @@ def _convert_variable(datatype, variable):
:param datatype: type to convert to
:param variable (string or None): value of variable
:return: converted variable
:raises: ValueError
:raises: AssertionError
"""
try:
if variable is None:
return variable
if datatype == "int":
variable = int(variable)
elif datatype == "float":
variable = float(variable)
assert datatype in ["boolean", "categorical", "float32", "int32", "string"]
if variable is None:
return variable
except ValueError:
raise
if datatype == "int32":
variable = int32(variable)
elif datatype == "float32":
variable = float32(variable)
elif datatype == "boolean":
variable = json.loads(variable)
assert isinstance(variable, bool)
return variable
def parse_filter(filter, schema):
def parse_filter(query_filter, schema):
"""
The filter comes in as arguments from a GET/POST request
For categorical metadata keys filter based on key=value
For continuous metadata keys filter by key=min,max
Either value can be replaced by a * To have only a minimum value key=min, To have only a maximum value key=*,max
The filter comes in as arguments from a GET request
For categorical metadata keys filter based on axis:key=value
For continuous metadata keys filter by axis:key=min,max
Either value can be replaced by a * To have only a minimum
value axis:key=min,* To have only a maximum value axis:key=*,max
They combine via AND so a cell's metadata would have to match every filter
The results is a matrix with the cells the pass the filter and at this point all the genes
:param filter: flask's request.args
:param query_filter: flask's request.args
:param schema: dictionary schema
:raises QueryStringError
:return:
"""
query = {}
for key in filter:
value = filter.getlist(key)
if key not in schema:
raise QueryStringError("Error: key {} not in metadata schema".format(key))
query[key] = {
"variable_type": schema[key]["variabletype"],
"value_type": schema[key]["type"]
}
if query[key]["variable_type"] == "categorical":
query[key]["query"] = [_convert_variable(query[key]["value_type"], v) for v in value]
elif query[key]["variable_type"] == "continuous":
value = value[0]
query = defaultdict(lambda: defaultdict(list))
for key in query_filter:
axis, annotation = key.split(":", 1)
try:
Axis(axis)
except ValueError:
raise QueryStringError(key, f"Error: key {key} not in metadata schema")
ann_filter = {"name": annotation}
for ann in schema[axis]:
if ann["name"] == annotation:
dtype = ann["type"]
break
else:
raise QueryStringError(key, f"Error: {annotation} not a valid annotation name")
if dtype in ["string", "categorical", "boolean"]:
ann_filter["values"] = [_convert_variable(dtype, i) for i in query_filter.getlist(key)]
else:
value = query_filter.get(key)
try:
min, max = value.split(",")
min_, max_ = value.split(",")
except ValueError:
raise QueryStringError("Error: min,max format required for range for key {}, got {}".format(key, value))
if min == "*":
min = None
if max == "*":
max = None
raise QueryStringError(key, f"Error: min,max format required for range for {annotation}, got {value}")
if min_ == "*":
min_ = None
if max_ == "*":
max_ = None
try:
query[key]["query"] = {
"min": _convert_variable(query[key]["value_type"], min),
"max": _convert_variable(query[key]["value_type"], max)
}
ann_filter["min"] = _convert_variable(dtype, min_)
ann_filter["max"] = _convert_variable(dtype, max_)
except ValueError:
raise QueryStringError(
"Error: expected type {} for key {}, got {}".format(query[key]["type"], key, value)
)
raise QueryStringError(key, f"Error: expected type {query[key]['type']} for key {key}, got {value}")
query[axis]["annotation_value"].append(ann_filter)
return query

65
server/app/util/models.py Normal file
View File

@@ -0,0 +1,65 @@
from flask_restful_swagger_2 import Schema
class AnnotationModel(Schema):
type = "object"
description = "Filter by annotation key: value"
properties = {
"name": {
"type": "string"
},
# TODO update to OpenAPI v3.0 when a library is available that supports it
# Unfortunately 2.0 doesn't have a way to have a schema that accepts multiple types
# Overloading the type key with a list seems to work ok and makes it to the page
"values": {
"type": "array",
"items": {
"type": ["float32", "string", "int32", "bool"]
}
},
"min": {
"type": ["int32", "float32"],
},
"max": {
"type": ["int32", "float32"],
}
}
required = ["name"]
class IndexModel(Schema):
type = "object"
description = "Filter by index of observation/variable ex. [0, 5, 15]"
properties = {
"index": {
"type": "array",
"items": {
"format": "int32",
"type": "integer"
}
}
}
class AxisModel(Schema):
type = "object"
description = "Axis of data -- obs or var"
properties = {
"index": IndexModel,
"annotation_value": AnnotationModel.array()
}
class FilterModel(Schema):
type = "object"
description = "Complex filter"
properties = {
"filter": {
"type": "object",
"properties": {
"obs": AxisModel,
"var": AxisModel
}
}
}

7
server/app/util/schema_parse.py
View File

@@ -1,7 +0,0 @@
import json
def parse_schema(filename):
with open(filename) as fh:
schema = json.load(fh)
return schema

28
server/app/util/utils.py
View File

@@ -1,7 +1,6 @@
import json
from numpy import float32, integer
from flask import make_response, jsonify, Response
class Float32JSONEncoder(json.JSONEncoder):
@@ -11,30 +10,3 @@ class Float32JSONEncoder(json.JSONEncoder):
elif isinstance(obj, integer):
return int(obj)
return json.JSONEncoder.default(self, obj)
def make_payload(data, errormessage="", errorcode=200):
"""
Creates JSON respons for requests
:param data: json data
:param errormessage: error message
:param errorcode: http error code
:return: flask json repsonse
"""
error = False
if errormessage:
error = True
# Questionable
data = json.loads(json.dumps(data, cls=Float32JSONEncoder))
return make_response(jsonify({
"data": data,
"status": {
"error": error,
"errormessage": errormessage,
}
}), errorcode)
def make_streaming_response(data_generator, errorcode=200, content_type="application/json"):
# TODO headers
return Response(data_generator, status=errorcode, content_type=content_type)

3
server/requirements-dev.txt Normal file
View File

@@ -0,0 +1,3 @@
pytest
requests
-r requirements.txt

2
server/requirements.txt
View File

@@ -1,4 +1,4 @@
anndata==0.6.1
anndata
Flask==0.12.4
Flask-Caching==1.4.0
Flask-Compress==1.4.0

51
server/test/schema.json Normal file
View File

@@ -0,0 +1,51 @@
{
"dataframe": {
"nObs": 2638,
"nVar": 1838,
"type": "float32"
},
"annotations": {
"obs": [
{
"name": "n_genes",
"type": "int32"
},
{
"name": "percent_mito",
"type": "float32"
},
{
"name": "n_counts",
"type": "float32"
},
{
"name": "louvain",
"type": "categorical",
"categories": [
"CD4 T cells",
"CD14+ Monocytes",
"B cells",
"CD8 T cells",
"NK cells",
"FCGR3A+ Monocytes",
"Dendritic cells",
"Megakaryocytes"
]
},
{
"name": "name",
"type": "string"
}
],
"var": [
{
"name": "n_cells",
"type": "int32"
},
{
"name": "name",
"type": "string"
}
]
}
}

361
server/test/test_api.py
View File

@@ -1,60 +1,339 @@
import unittest
import requests
import json
from subprocess import Popen
import unittest
import time
LOCAL_URL = "http://127.0.0.1:5005/"
VERSION = "v0.2"
URL_BASE = f"{LOCAL_URL}api/{VERSION}/"
class EndPoints(unittest.TestCase):
"""Test Case for endpoints"""
@classmethod
def setUpClass(cls):
cls.ps = Popen(["cellxgene", "scanpy", "example-dataset/"])
session = requests.Session()
for i in range(90):
try:
session.get(f"{URL_BASE}schema")
except requests.exceptions.ConnectionError:
time.sleep(1)
@classmethod
def tearDownClass(cls):
try:
cls.ps.terminate()
except ProcessLookupError:
pass
def setUp(self):
# Local
self.local_url = "http://127.0.0.1:5005/"
self.version = "v0.1"
self.url_base = "{local_url}api/{version}/".format(local_url=self.local_url, version=self.version)
self.session = requests.Session()
def test_cells(self):
url = "{base}{endpoint}?{params}".format(base=self.url_base, endpoint="cells", params="&".join(
["louvain=B cells"]))
result = self.session.get(url)
assert result.status_code == 200
result_data = result.json()
assert "B cells" in result_data["data"]["ranges"]["louvain"]["options"]
url = "{base}{endpoint}?{params}".format(base=self.url_base, endpoint="cells", params="&".join(
["louvain=B cells", "louvain=Megakaryocytes"]))
result = self.session.get(url)
assert result.status_code == 200
result_data = result.json()
assert "Megakaryocytes" in result_data["data"]["ranges"]["louvain"]["options"]
def test_initialize(self):
url = "{base}{endpoint}".format(base=self.url_base, endpoint="initialize")
endpoint = "schema"
url = f"{URL_BASE}{endpoint}"
result = self.session.get(url)
assert result.status_code == 200
self.assertEqual(result.status_code, 200)
result_data = result.json()
assert result_data["data"]["cellcount"] == 2638
assert len(result_data["data"]['ranges']['CellName']['options']) == 2638
self.assertEqual(result_data["schema"]["dataframe"]["nObs"], 2638)
self.assertEqual(len(result_data["schema"]["annotations"]["obs"]), 5)
def test_expression_get(self):
url = "{base}{endpoint}".format(base=self.url_base, endpoint="expression")
def test_config(self):
endpoint = "config"
url = f"{URL_BASE}{endpoint}"
result = self.session.get(url)
assert result.status_code == 200
def test_expression_post(self):
url = "{base}{endpoint}".format(base=self.url_base, endpoint="expression")
result = self.session.post(url, data=json.dumps({"celllist": ["AAACATACAACCAC-1", "AACCGATGGTCATG-1"], "genelist": ["BACH1", "MIS18A", "ATP5O"]}), headers={'content-type': 'application/json'})
assert result.status_code == 200
self.assertEqual(result.status_code, 200)
result_data = result.json()
assert len(result_data["data"]["cells"]) == 2
assert len(result_data["data"]["cells"][0]['e']) == 3
self.assertEqual(result_data["config"]["displayNames"]["dataset"], "example-dataset")
self.assertEqual(len(result_data["config"]["features"]), 4)
def test_diffexp(self):
url = "{base}{endpoint}".format(base=self.url_base, endpoint="diffexpression")
result = self.session.post(url, data=json.dumps({"celllist1": ["AAACATACAACCAC-1", "AACCGATGGTCATG-1"], "celllist2": ["CCGATAGACCTAAG-1", "GGTGGAGAAGTAGA-1"]}), headers={'content-type': 'application/json'})
assert result.status_code == 200
def test_get_layout(self):
endpoint = "layout/obs"
url = f"{URL_BASE}{endpoint}"
result = self.session.get(url)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["layout"]["ndims"], 2)
self.assertEqual(len(result_data["layout"]["coordinates"]), 2638)
def test_put_layout(self):
endpoint = "layout/obs"
url = f"{URL_BASE}{endpoint}"
obs_filter = {
"filter": {
"obs": {
"annotation_value": [
{"name": "louvain", "values": ["NK cells", "CD8 T cells"]},
{"name": "n_counts", "min": 3000},
],
"index": [1, 99, [1000, 2000]]
}
}
}
result = self.session.put(url, json=obs_filter)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(len(result_data["layout"]["coordinates"]), 15)
def test_get_annotations_obs(self):
endpoint = "annotations/obs"
url = f"{URL_BASE}{endpoint}"
result = self.session.get(url)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_genes", "percent_mito", "n_counts", "louvain", "name"])
self.assertEqual(len(result_data["data"]), 2638)
self.assertEqual(len(result_data["data"][0]), 6)
def test_get_annotations_obs_keys(self):
endpoint = "annotations/obs"
query = "annotation-name=n_genes&annotation-name=percent_mito"
url = f"{URL_BASE}{endpoint}?{query}"
result = self.session.get(url)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_genes", "percent_mito"])
self.assertEqual(len(result_data["data"][0]), 3)
def test_get_annotations_obs_error(self):
endpoint = "annotations/obs"
query = "annotation-name=notakey"
url = f"{URL_BASE}{endpoint}?{query}"
result = self.session.get(url)
self.assertEqual(result.status_code, 404)
def test_put_annotations_obs(self):
endpoint = "annotations/obs"
url = f"{URL_BASE}{endpoint}"
obs_filter = {
"filter": {
"obs": {
"annotation_value": [
{"name": "louvain", "values": ["NK cells", "CD8 T cells"]},
{"name": "n_counts", "min": 3000},
],
"index": [1, 99, [1000, 2000]]
}
}
}
result = self.session.put(url, json=obs_filter)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_genes", "percent_mito", "n_counts", "louvain", "name"])
self.assertEqual(len(result_data["data"]), 15)
def test_filter_put_annotations_obs(self):
endpoint = "annotations/obs"
query = "annotation-name=n_genes&annotation-name=percent_mito"
url = f"{URL_BASE}{endpoint}?{query}"
obs_filter = {
"filter": {
"obs": {
"annotation_value": [
{"name": "louvain", "values": ["NK cells", "CD8 T cells"]},
{"name": "n_counts", "min": 3000},
],
"index": [1, 99, [1000, 2000]]
}
}
}
result = self.session.put(url, json=obs_filter)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_genes", "percent_mito"])
self.assertEqual(len(result_data["data"][0]), 3)
self.assertEqual(len(result_data["data"]), 15)
def test_diff_exp(self):
endpoint = "diffexp/obs"
url = f"{URL_BASE}{endpoint}"
params = {
"mode": "topN",
"set1": {
"filter": {
"obs": {"annotation_value": [
{"name": "louvain", "values": ["NK cells"]}
]
}
}
},
"set2": {
"filter": {
"obs": {"annotation_value": [
{"name": "louvain", "values": ["CD8 T cells"]}
]
}
}
},
"count": 7
}
result = self.session.post(url, json=params)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(len(result_data), 7)
def test_diff_exp_indices(self):
endpoint = "diffexp/obs"
url = f"{URL_BASE}{endpoint}"
params = {
"mode": "topN",
"set1": {
"filter": {
"obs": {
"index": [[0, 500]]
}
}
},
"set2": {
"filter": {
"obs": {
"index": [[500, 1000]]
}
}
}
}
result = self.session.post(url, json=params)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(len(result_data), 10)
def test_get_annotations_var(self):
endpoint = "annotations/var"
url = f"{URL_BASE}{endpoint}"
result = self.session.get(url)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_cells", "name"])
self.assertEqual(len(result_data["data"]), 1838)
self.assertEqual(len(result_data["data"][0]), 3)
def test_get_annotations_var_keys(self):
endpoint = "annotations/var"
query = "annotation-name=n_cells"
url = f"{URL_BASE}{endpoint}?{query}"
result = self.session.get(url)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_cells"])
self.assertEqual(len(result_data["data"][0]), 2)
def test_get_annotations_var_error(self):
endpoint = "annotations/var"
query = "annotation-name=notakey"
url = f"{URL_BASE}{endpoint}?{query}"
result = self.session.get(url)
self.assertEqual(result.status_code, 404)
def test_put_annotations_var(self):
endpoint = "annotations/var"
url = f"{URL_BASE}{endpoint}"
var_filter = {
"filter": {
"var": {
"annotation_value": [
{"name": "name", "values": ["ATAD3C", "RER1"]},
]
}
}
}
result = self.session.put(url, json=var_filter)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_cells", "name"])
self.assertEqual(len(result_data["data"]), 2)
def test_filter_put_annotations_var(self):
endpoint = "annotations/var"
query = "annotation-name=n_cells"
url = f"{URL_BASE}{endpoint}?{query}"
var_filter = {
"filter": {
"var": {
"annotation_value": [
{"name": "name", "values": ["ATAD3C", "RER1"]},
]
}
}
}
result = self.session.put(url, json=var_filter)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(result_data["names"], ["n_cells"])
self.assertEqual(len(result_data["data"][0]), 2)
self.assertEqual(len(result_data["data"]), 2)
def test_get_data(self):
endpoint = "data/obs"
query = "accept-type=application/json"
url = f"{URL_BASE}{endpoint}?{query}"
result = self.session.get(url)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(len(result_data["obs"]), 2638)
def test_data_mimetype_error(self):
endpoint = "data/obs"
query = "accept-type=xxx"
url = f"{URL_BASE}{endpoint}?{query}"
result = self.session.get(url)
self.assertEqual(result.status_code, 406)
# no accept type
url = f"{URL_BASE}{endpoint}"
result = self.session.get(url)
self.assertEqual(result.status_code, 406)
def test_data_filter(self):
endpoint = "data/obs"
query = "accept-type=application/json&obs:louvain=NK cells&obs:louvain=CD8 T cells&obs:n_counts=3000,*"
url = f"{URL_BASE}{endpoint}?{query}"
result = self.session.get(url)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(len(result_data["obs"]), 38)
def test_data_put(self):
endpoint = "data/obs"
url = f"{URL_BASE}{endpoint}"
header = {"Accept": "application/json"}
obs_filter = {
"filter": {
"obs": {
"annotation_value": [
{"name": "louvain", "values": ["NK cells", "CD8 T cells"]},
{"name": "n_counts", "min": 3000},
],
"index": [1, 99, [1000, 2000]]
}
}
}
result = self.session.put(url, headers=header, json=obs_filter)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(len(result_data["obs"]), 15)
def test_data_put_single_var(self):
endpoint = "data/obs"
url = f"{URL_BASE}{endpoint}"
header = {"Accept": "application/json"}
var_filter = {
"filter": {
"var": {
"annotation_value": [
{"name": "name", "values": ["RER1"]},
]
}
}
}
result = self.session.put(url, headers=header, json=var_filter)
self.assertEqual(result.status_code, 200)
result_data = result.json()
self.assertEqual(len(result_data["obs"][0]), 2)
def test_static(self):
url = "{url}{endpoint}/{file}".format(url=self.local_url, endpoint="static", file="js/service-worker.js")
endpoint = "static"
file = "js/service-worker.js"
url = f"{LOCAL_URL}{endpoint}/{file}"
result = self.session.get(url)
assert result.status_code == 200
self.assertEqual(result.status_code, 200)

115
server/test/test_filter.py
View File

@@ -1,75 +1,80 @@
import json
from os import path
import unittest
from unittest.mock import MagicMock
from server.app.util.filter import _convert_variable, parse_filter
from numpy import float32, int32
from werkzeug.datastructures import ImmutableMultiDict
from server.app.util.filter import _convert_variable, parse_filter, QueryStringError
class UtilTest(unittest.TestCase):
"""Test Case for endpoints"""
def setUp(self):
self.schema = {
"cluster": {
"displayname": "Cluster",
"include": True,
"type": "int",
"variabletype": "categorical"
},
"louvain": {
"displayname": "Louvain Cluster",
"include": True,
"type": "string",
"variabletype": "categorical"
},
"n_genes": {
"displayname": "Num Genes",
"include": True,
"type": "int",
"variabletype": "continuous"
}
}
with open(path.join(path.dirname(__file__), "schema.json")) as fh:
schema = json.load(fh)
self.schema = schema["annotations"]
def test_convert(self):
five = _convert_variable("int", "5")
assert five == 5
five = _convert_variable("int32", "5")
self.assertEqual(five, int32(5))
def test_convert_zero(self):
zero = _convert_variable("int", "0")
assert zero == 0
zero = _convert_variable("int32", "0")
self.assertEqual(zero, 0)
def test_convert_float(self):
str_to_convert = "4.38719237129"
val = _convert_variable("float32", str_to_convert)
self.assertAlmostEqual(val, float32(str_to_convert))
def test_convert_bool(self):
str_to_convert = "false"
val = _convert_variable("boolean", str_to_convert)
self.assertFalse(val)
str_to_convert = "true"
val = _convert_variable("boolean", str_to_convert)
self.assertTrue(val)
str_to_convert = "0"
with self.assertRaises(AssertionError):
val = _convert_variable("boolean", str_to_convert)
def test_empty_convert(self):
empty = _convert_variable("int", None)
assert empty is None
empty = _convert_variable("int32", None)
self.assertIsNone(empty)
def test_bad_convert(self):
with self.assertRaises(ValueError):
_convert_variable("int", "5.5")
_convert_variable("int32", "5.5")
def test_filter_categorical(self):
filterMock = MagicMock()
filterMock.__iter__.return_value = iter(["louvain"])
filterMock.getlist.return_value = ["B cells", "T cells"]
query = parse_filter(filterMock, self.schema)
assert query == {"louvain": {"variable_type": "categorical", "value_type": "string", "query": ["B cells", "T cells"]}}
filterMock.__iter__.return_value = iter(["cluster"])
filterMock.getlist.return_value = ["1", "2"]
query = parse_filter(filterMock, self.schema)
assert query == {"cluster": {"variable_type": "categorical", "value_type": "int", "query": [1, 2]}}
def test_bad_datatype(self):
with self.assertRaises(AssertionError):
_convert_variable("jkasdslkja", 1)
def test_filter_contiunous(self):
filterMock = MagicMock()
filterMock.__iter__.return_value = iter(["n_genes"])
filterMock.getlist.return_value = ["0,100"]
query = parse_filter(filterMock, self.schema)
assert query == {"n_genes": {"variable_type": "continuous", "value_type": "int", "query": {"min": 0, "max": 100}}}
filterMock.__iter__.return_value = iter(["n_genes"])
filterMock.getlist.return_value = ["*,100"]
query = parse_filter(filterMock, self.schema)
assert query == {"n_genes": {"variable_type": "continuous", "value_type": "int", "query": {"min": None, "max": 100}}}
filterMock.__iter__.return_value = iter(["n_genes"])
filterMock.getlist.return_value = ["0,*"]
query = parse_filter(filterMock, self.schema)
assert query == {"n_genes": {"variable_type": "continuous", "value_type": "int", "query": {"min": 0, "max": None}}}
def test_complex_filter(self):
filter_dict = ImmutableMultiDict(
[("obs:louvain", "NK cells"), ("obs:louvain", "CD8 T cells"), ("obs:n_counts", "3000,*")])
filter_ = parse_filter(filter_dict, self.schema)
self.assertIn("obs", filter_)
self.assertEqual(filter_["obs"]["annotation_value"], [{"name": "louvain",
"values": ["NK cells", "CD8 T cells"]},
{"name": "n_counts",
"max": None, "min": 3000.0}])
if __name__ == '__main__':
unittest.main()
def test_bad_filter(self):
bad_annotation_type = ImmutableMultiDict([("obs:tissue", "lung")])
with self.assertRaises(QueryStringError):
parse_filter(bad_annotation_type, self.schema)
bad_axis = ImmutableMultiDict([("xyz:n_genes", "100,1000")])
with self.assertRaises(QueryStringError):
parse_filter(bad_axis, self.schema)
def test_boolean_filter(self):
schema = {
"obs": [{"name": "bool_filter", "type": "boolean"}]
}
filter_dict = ImmutableMultiDict([("obs:bool_filter", "false")])
filter_ = parse_filter(filter_dict, schema)
self.assertIn("obs", filter_)
self.assertEqual(filter_["obs"]["annotation_value"], [{"name": "bool_filter", "values": [False]}])

248
server/test/test_scanpy_engine.py
View File

@@ -1,69 +1,229 @@
import json
from os import path
import pytest
import time
import unittest
import numpy as np
from pandas import Series
from server.app.scanpy_engine.scanpy_engine import ScanpyEngine
class UtilTest(unittest.TestCase):
def setUp(self):
self.data = ScanpyEngine("example-dataset/", schema="data_schema.json")
self.data = ScanpyEngine("example-dataset/", layout_method="umap", diffexp_method="ttest")
self.data._create_schema()
def test_init(self):
self.assertEqual(self.data.cell_count, 2638)
self.assertEqual(self.data.gene_count, 1838)
epsilon = 0.000005
self.assertTrue(self.data.data.X[0,0] - -0.17146951 < epsilon)
self.assertTrue(self.data.data.X[0, 0] - -0.17146951 < epsilon)
def test_mandatory_annotations(self):
self.assertIn("name", self.data.data.obs)
self.assertEqual(list(self.data.data.obs.index), list(range(2638)))
self.assertIn("name", self.data.data.var)
self.assertEqual(list(self.data.data.var.index), list(range(1838)))
@pytest.mark.filterwarnings("ignore:Scanpy data matrix")
def test_data_type(self):
self.data.data.X = self.data.data.X.astype("float64")
self.assertWarns(UserWarning, self.data._validatate_data_types())
def test_filter_idx(self):
filter_ = {
"filter": {
"var": {
"index": [1, 99, [200, 300]]
},
"obs": {
"index": [1, 99, [1000, 2000]]
}
}
}
data = self.data.filter_dataframe(filter_["filter"])
self.assertEqual(data.shape, (1002, 102))
def test_filter_annotation(self):
filter_ = {
"filter": {
"obs": {
"annotation_value": [
{"name": "louvain", "values": ["NK cells", "CD8 T cells"]},
]
}
}
}
data = self.data.filter_dataframe(filter_["filter"])
self.assertEqual(data.shape, (470, 1838))
filter_ = {
"filter": {
"obs": {
"annotation_value": [
{"name": "n_counts", "min": 3000},
]
}
}
}
data = self.data.filter_dataframe(filter_["filter"])
self.assertEqual(data.shape, (497, 1838))
def test_filter_annotation_no_uns(self):
filter_ = {
"filter": {
"var": {
"annotation_value": [
{"name": "name", "values": ["RER1"]},
]
}
}
}
data = self.data.filter_dataframe(filter_["filter"], include_uns=False)
self.assertEqual(data.shape[1], 1)
def test_filter_complex(self):
filter_ = {
"filter": {
"var": {
"index": [1, 99, [200, 300]]
},
"obs": {
"annotation_value": [
{"name": "louvain", "values": ["NK cells", "CD8 T cells"]},
{"name": "n_counts", "min": 3000},
],
"index": [1, 99, [1000, 2000]]
}
}
}
data = self.data.filter_dataframe(filter_["filter"])
self.assertEqual(data.shape, (15, 102))
def test_obs_and_var_names(self):
self.assertEqual(np.sum(self.data.data.var["name"].isna()), 0)
self.assertEqual(np.sum(self.data.data.obs["name"].isna()), 0)
def test_schema(self):
self.assertEqual(self.data.schema, {'CellName': {'type': 'string', 'variabletype': 'categorical', 'displayname': 'Name', 'include': True}, 'n_genes': {'type': 'int', 'variabletype': 'continuous', 'displayname': 'Num Genes', 'include': True}, 'percent_mito': {'type': 'float', 'variabletype': 'continuous', 'displayname': 'Mitochondrial Percentage', 'include': True}, 'n_counts': {'type': 'float', 'variabletype': 'continuous', 'displayname': 'Num Counts', 'include': True}, 'louvain': {'type': 'string', 'variabletype': 'categorical', 'displayname': 'Louvain Cluster', 'include': True}})
with open(path.join(path.dirname(__file__), "schema.json")) as fh:
schema = json.load(fh)
self.assertEqual(self.data.schema, schema)
def test_cells(self):
cells = self.data.cells()
self.assertIn("AAACATACAACCAC-1", cells)
self.assertEqual(len(cells), 2638)
def test_schema_produces_error(self):
self.data.data.obs["time"] = Series(list([time.time() for i in range(self.data.cell_count)]),
dtype="datetime64[ns]")
with pytest.raises(TypeError):
self.data._create_schema()
def test_genes(self):
genes = self.data.genes()
self.assertIn("SEPT4", genes)
self.assertEqual(len(genes), 1838)
def test_config(self):
self.assertEqual(self.data.features["layout"]["obs"], {'available': True, 'interactiveLimit': 15000})
def test_filter_categorical(self):
filter = {"louvain": {"variable_type": "categorical", "value_type": "string", "query": ["B cells"]}}
filtered_data = self.data.filter_cells(filter)
self.assertEqual(filtered_data.shape, (342, 1838))
louvain_vals = filtered_data.obs['louvain'].tolist()
self.assertIn("B cells", louvain_vals)
self.assertNotIn("NK cells", louvain_vals)
def test_layout(self):
layout = self.data.layout(self.data.data)
self.assertEqual(layout["ndims"], 2)
self.assertEqual(len(layout["coordinates"]), 2638)
self.assertEqual(layout["coordinates"][0][0], 0)
for idx, val in enumerate(layout["coordinates"]):
self.assertLessEqual(val[1], 1)
self.assertLessEqual(val[2], 1)
def test_filter_continuous(self):
# print(self.data.data.obs["n_genes"].tolist())
filter = {"n_genes": {"variable_type": "continuous", "value_type": "int", "query": {"min": 300, "max": 400}}}
filtered_data = self.data.filter_cells(filter)
self.assertEqual(filtered_data.shape, (71, 1838))
n_genes_vals = filtered_data.obs['n_genes'].tolist()
for val in n_genes_vals:
self.assertTrue(300 <= val <= 400)
def test_annotations(self):
annotations = self.data.annotation(self.data.data, "obs")
self.assertEqual(annotations["names"], ["n_genes", "percent_mito", "n_counts", "louvain", "name"])
self.assertEqual(len(annotations["data"]), 2638)
annotations = self.data.annotation(self.data.data, "var")
self.assertEqual(annotations["names"], ["n_cells", "name"])
self.assertEqual(len(annotations["data"]), 1838)
def test_metadata(self):
metadata = self.data.metadata(df=self.data.data)
self.assertEqual(len(metadata), 2638)
self.assertIn('louvain', metadata[0])
def test_annotation_fields(self):
annotations = self.data.annotation(self.data.data, "obs", ["n_genes", "n_counts"])
self.assertEqual(annotations["names"], ["n_genes", "n_counts"])
self.assertEqual(len(annotations["data"]), 2638)
annotations = self.data.annotation(self.data.data, "var", ["name"])
self.assertEqual(annotations["names"], ["name"])
self.assertEqual(len(annotations["data"]), 1838)
@unittest.skip("Umap not producing the same graph on different systems, even with the same seed. Skipping for now")
def test_create_graph(self):
graph = self.data.create_graph(df=self.data.data)
self.assertEqual(graph[0][1], 0.5545382653143183)
self.assertEqual(graph[0][2], 0.6021833809031731)
def test_filtered_annotation(self):
filter_ = {
"filter": {
"obs": {
"annotation_value": [
{"name": "n_counts", "min": 3000},
]
},
"var": {
"annotation_value": [
{"name": "name", "values": ["ATAD3C", "RER1"]},
]
}
}
}
data = self.data.filter_dataframe(filter_["filter"])
annotations = self.data.annotation(data, "obs")
self.assertEqual(annotations["names"], ["n_genes", "percent_mito", "n_counts", "louvain", "name"])
self.assertEqual(len(annotations["data"]), 497)
annotations = self.data.annotation(data, "var")
self.assertEqual(annotations["names"], ["n_cells", "name"])
self.assertEqual(len(annotations["data"]), 2)
def test_filtered_layout(self):
filter_ = {
"filter": {
"obs": {
"annotation_value": [
{"name": "n_counts", "min": 3000},
]
}
}
}
data = self.data.filter_dataframe(filter_["filter"])
layout = self.data.layout(data)
self.assertEqual(len(layout["coordinates"]), 497)
def test_diffexp(self):
diffexp = self.data.diffexp(["AAACATACAACCAC-1", "AACCGATGGTCATG-1"], ["CCGATAGACCTAAG-1", "GGTGGAGAAGTAGA-1"], 0.5, 7)
self.assertEqual(diffexp["celllist1"]["topgenes"], ['EBNA1BP2', 'DIAPH1', 'SLC25A11', 'SNRNP27', 'COMMD8', 'COTL1', 'GTF3A'])
f1 = {
"filter": {
"obs": {
"index": [[0, 500]]
}
}
}
df1 = self.data.filter_dataframe(f1["filter"])
f2 = {
"filter": {
"obs": {
"index": [[500, 1000]]
}
}
}
df2 = self.data.filter_dataframe(f2["filter"])
result = self.data.diffexp(df1, df2)
self.assertEqual(len(result), 10)
var_idx = [i[0] for i in result]
self.assertEqual(var_idx, sorted(var_idx))
result = self.data.diffexp(df1, df2, 20)
self.assertEqual(len(result), 20)
def test_expression(self):
expression = self.data.expression(cells=["AAACATACAACCAC-1"])
data_exp = self.data.data[["AAACATACAACCAC-1"], :].X
for idx in range(len(expression["cells"][0]["e"])):
self.assertEqual(expression["cells"][0]["e"][idx], data_exp[idx])
def test_data_frame(self):
data_frame = self.data.data_frame(self.data.data)
self.assertEqual(len(data_frame["var"]), 1838)
self.assertEqual(len(data_frame["obs"]), 2638)
def test_filtered_data_frame(self):
filter_ = {
"filter": {
"obs": {
"annotation_value": [
{"name": "n_counts", "min": 3000},
]
}
}
}
data = self.data.filter_dataframe(filter_["filter"])
data_frame = self.data.data_frame(data)
self.assertEqual(len(data_frame["var"]), 1838)
self.assertEqual(len(data_frame["obs"]), 497)
if __name__ == '__main__':
unittest.main()
if __name__ == '__main__':
unittest.main()

20
setup.py
View File

@@ -3,18 +3,18 @@ from setuptools import setup, find_packages
with open("README.md", "r") as fh:
long_description = fh.read()
with open('server/requirements.txt') as fh:
with open("server/requirements.txt") as fh:
requirements = fh.read().splitlines()
setup(
name='cellxgene',
version='0.0.1',
name="cellxgene",
version="0.0.1",
packages=find_packages(),
url='https://github.com/chanzuckerberg/cellxgene',
license='MIT',
author='Colin Megill, Charlotte Weaver',
author_email='cweaver@chanzuckerberg.com',
description='Web application for exploration of large scale scRNA-seq datasets',
url="https://github.com/chanzuckerberg/cellxgene",
license="MIT",
author="Colin Megill, Charlotte Weaver",
author_email="cweaver@chanzuckerberg.com",
description="Web application for exploration of large scale scRNA-seq datasets",
long_description=long_description,
install_requires=requirements,
include_package_data=True,
@@ -24,7 +24,7 @@ setup(
"License :: OSI Approved :: MIT License",
),
entry_points={
'console_scripts':
['cellxgene = server.app.app:main']
"console_scripts":
["cellxgene = server.app.app:main"]
}
)