Commit Graph
74 Commits
Author SHA1 Message Date
Charlotte Weaver efb55a6332 Renaming bind-all option to listen-all (#353)
See discussion here https://docs.google.com/document/d/1tcAvODhdlUUAOJPHoGQB3fWOPmnw3XjK7MVhDRwNUnk/edit#
2018-10-19 15:26:19 -07:00
Charlotte Weaver 2a3577f32b flask debug as hidden option (#352)
* Remove flask debug flag

* Add debug flag as a hidden option to keep reload on change for developers
2018-10-19 10:24:15 -07:00
Charlotte Weaver 06f402ab0a Do not calculate layout, used saved layout instead (#343)
* Do not calculate layout, used saved layout instead

See for rationale: https://docs.google.com/document/d/1HJFvbdDHxxgkCW0DZzdTZ9CUMgc2ef2rQFukATBQWvE/edit

* Error handling for when layout has not been precomputed

* Server error (500) not client error (400) for unprepared data
2018-10-18 11:51:38 -07:00
Charlotte Weaver 7c8b89eba3 Launch browser to cellxgene (#325)
* Launch browser to cellxgene

Also added  --no-launch command line parameter

* Don't launch browser for tests

* rename no-launch to no-open

* Rejigger -no-open to positive destination

so later logic looks cleaner
2018-10-17 16:46:56 -07:00
Bruce Martin 76ec29734a Performance work (#334)
* range encode filter range lists

* speed up data load

* add comment on scanpy read params

* update to latest scanpy/anndata

* performance improvments in data loading

* fix typo

* work around scanpy bug

* remove debugging print statements
2018-10-16 15:49:40 -07:00
Charlotte Weaver 2e9d576890 Correct capitalization in swagger template (#320) 2018-10-11 16:43:26 -07:00
Charlotte Weaver 3229babd14 Add caching back in (#312)
* Add caching back in

Works now against filter instead of dataframe view

* Layout cache and move post->put

* Fix tests
2018-10-11 16:29:46 -07:00
Charlotte Weaver b90292ce4c Bump interactive limits (#319) 2018-10-11 15:56:58 -07:00
Charlotte Weaver 210787cb5d Filter in engine (#307)
* Add empty filter case

* Filtering dataframes moved to engine instead of rest

* minor changes from PR review

* Minor fixes from PR review

Pass {} instead of none if no filter
chain exceptions
typos
2018-10-11 15:23:31 -07:00
Bruce Martin ab76c57fdb updated anndata dependency version (#314)
Update Python requirements.txt to include correct anndata version
2018-10-11 13:02:07 -07:00
Charlotte Weaver eb3b3fc59a Reorganize order of methods in engine and rest (#305) 2018-10-10 16:44:05 -07:00
Charlotte Weaver 11ef5ae51b REST Error handling (#299)
* Use HTTPStatus for all responses

More informative than just the code as an int

* Fill out REST error handling

* Test error routes

* Use HTTP Status for tests too

* factor mime type request into function

* Better mimetype errors

* 404 -> 400 error for bad key
2018-10-08 14:17:56 -07:00
Charlotte Weaver 8be0833d23 /layout put-> post (#301) 2018-10-08 13:38:13 -07:00
Philipp A 8674ad1f7f CLI improvements (#221)
* CLI improvements

* Circumvent apparent bug with subparsers.required
2018-10-04 16:41:14 -07:00
Charlotte Weaver 187bbfdcf7 /data/var (#295)
* Upgrade version of scanpy

* /data/var

This works for everything except the case where there is only one gene. Anndata flattens X when there is only one var thus causing the transpose to fail.

* Fix edge case when an axis (obs/var) only contains 1 element
2018-10-03 15:09:46 -07:00
Bruce Martin eeec842ad0 Restv2 feature branch merge to master (#284)
Move to new REST v0.2 communication between front and back-end.   This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc.    Protocol spec is in docs directory.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Fix merge errors

- import warnings was improperly deleted
- scanpy engine tests were totally wrong

* Fix merge error with driver

* PUT /annotations (#235)

* Add query param for annotation name

* fix descriptions, eliminate else clause

* first cut at initial data load on rest 0.2 api

* Annotation var (#248)

* Fix bug strings are always objects in pandas

* Add axis to annotation method

* Add /annotation/var to REST api

* Csweaver/expressiondata (#242)

* Refactor expression method for REST v2

* Add message to QueryStringError

* Fix range filters

* Add GET route for /data

* /data PUT route

* rename expression to data_frame

* clarification of error

* Improve accept type handling

* support all schema types for 0.2 REST API

* remove REST 0.1 code; connect var annotations loading

* config reducer; use config to set data set title; remove obsolete templating code for data set title

* REST 0.2 expression conversion support

* partial port of expression to REST 0.2

*  diffexp (#273)

* Add diffexp method to scanpy

and test

* Minor tweaks to diffexp

Get a minimal working version to unblock FE development

* Fixing things git deleted

* cleanup print statements

* Add index test

* additional, partial REST 0.2 bring up of diffexp

* Ignore unstructured annotations for data (#275)

This is a temp hack, need to figure out how to include data.uns if there is only one gene

* diffexp REST 0.2 port finish

* ignore unstructured annotaitons on all routes except layout

* correctly use varDataCache; maintain state during world rebuild

* correct varDataCache use

* temporarily disable all memoization

* refinements to expression data caching

* clear cell sets upon regraph/reset

* update version of REST to 0.2

* Travis build fixes

- comment out cache import
- fix duplicate test name

* Remove dependency from travis

* clarify semantics of config variables

* move generic action helpers into util
2018-10-01 14:58:46 -07:00
Charlotte Weaver 39414503bd Revert "Format loaded dataset" 2018-08-14 10:40:21 -07:00
Charlotte Weaver b884487789 Merge branch 'master' into csweaver/api-v2-init 2018-08-13 15:48:06 -07:00
Charlotte Weaver ddaa7016a5 Quote formatting
' => "
2018-08-13 15:00:23 -07:00
Charlotte Weaver a29ba8da4e Limit pytest ignore warning to only one specific warning 2018-08-13 15:00:06 -07:00
Charlotte Weaver a4464d107f Add test to make sure static files are served
this required that I break up the base url from the api route.
2018-08-13 10:33:59 -07:00
Charlotte Weaver d44d267bda Initial tests for the rest v2 refactor
testing the annotations and the data format validation
2018-08-10 16:57:41 -07:00
Charlotte Weaver 55fa9b892a Splitting the format validation and mandatory annotations 2018-08-10 16:38:07 -07:00
Charlotte Weaver 21a1f00a64 Set static folder to correct location 2018-08-09 14:25:11 -07:00
Charlotte Weaver e96653b826 Serving favicon correctly 2018-08-09 14:14:45 -07:00
Charlotte Weaver 58000c1815 Format loaded dataset
- create cell and gene ids
- recast numbers to float32/int32
2018-08-09 11:23:43 -07:00
Charlotte Weaver 246f4e41f0 Merge pull request #156 from chanzuckerberg/csweaver/infer-metadata
Infer schema if none exists
2018-08-08 11:56:13 -07:00
Charlotte Weaver af85792cce Don't forget unsigned ints! 2018-08-08 11:20:01 -07:00
Charlotte Weaver f4780a9806 Moved add to parser out of try block. 2018-08-08 11:18:06 -07:00
Charlotte Weaver 2a11af2550 Formatting 2018-08-07 17:01:19 -07:00
Charlotte Weaver e5c7c0c44a Warn instead if print when scanpy isn't available 2018-08-07 16:57:07 -07:00
Charlotte Weaver bf19bb7dce Formatting 2018-08-07 16:38:25 -07:00
Charlotte Weaver c2b3f331cf Add layout and diffexp calculations to cli options 2018-08-07 16:37:47 -07:00
Charlotte Weaver 499b9551f1 moved scanpy parser to scanpy class 2018-08-07 16:37:02 -07:00
Charlotte Weaver 0969854e9e Merge pull request #162 from chanzuckerberg/csweaver/require
Trimming out secondary dependencies from requirements
2018-08-07 16:14:28 -07:00
Charlotte Weaver ccee3e0a38 api host 0.0.0.0 -> 127.0.0.1 2018-08-07 15:43:13 -07:00
Charlotte Weaver c11ae77de4 Trimming out secondary dependencies from requirements 2018-08-07 12:53:48 -07:00
Charlotte Weaver a2b699b05d Infer schema if none exists 2018-08-06 16:39:18 -07:00
Charlotte Weaver e682028b8e Use flask's current_app 2018-08-03 10:54:43 -07:00
Charlotte Weaver 8dbb131afd Move data to a property of the app object 2018-08-02 16:18:17 -07:00
Charlotte Weaver e11b905836 Cleaning up import statements 2018-08-02 11:34:00 -07:00
Charlotte Weaver 116cef551c Switch to default timeout for cache 2018-08-02 11:28:44 -07:00
Charlotte Weaver 489a2dd5db Add flask-caching to requirements 2018-08-01 16:29:38 -07:00
Charlotte Weaver 0a36fac142 10 day cache (down from 1000)
I can expect a scientist to leave this running over the weekend on their laptop and still expect fast results. If someone leaves it running for a few years the data can probably be safely recalculated.
2018-08-01 15:48:56 -07:00
Charlotte Weaver 69fca7ea8b Using flask_caching instead
More recent module, same API
2018-08-01 15:46:04 -07:00
Charlotte Weaver 7642251d44 Add simple cache to backend 2018-08-01 13:32:04 -07:00
Charlotte Weaver de5e0f4ea7 Address PR review issues 2018-07-20 11:43:31 -07:00
Charlotte Weaver 2c7a11ab80 Added cli command for run 2018-07-19 17:21:28 -07:00
Charlotte Weaver 45f61d87d7 Initial CLI
added basic cli for running
2018-07-18 21:45:07 -07:00
Charlotte Weaver 895cfcebe0 Added initial travis config file 2018-07-18 10:21:24 -07:00