Commit Graph

17 Commits

Author SHA1 Message Date
Bruce Martin
57c4e9ff33 Flatbuffer cleanup (#598)
* dead code and route removal

* more dead code cleanup

* fix scanpy_engine tests

* lint

* add missing catch in filter parsing

* update scanpy NaN tests

* more fbs tests and dead test removal

* remove forced default for content type negotiation

* bit of cleanup

* more fbs test cleanup

* lint

* remove swagger

* swagger cleanup

* lint

* correctly handle lack of templates

* more dead code removal

* remove unused files

* fix dev build

* lint
2019-02-19 08:50:29 -08:00
Charlotte Weaver
10693b08cc Add __init__ file so fbs can be imported (#562) 2019-01-17 17:11:31 -08:00
Bruce Martin
5d60505407 remove --nan-to-num CLI parameter (#548)
* remove --nan-to-num CLI parameter

* factor tests better

* lint - remove unused variables
2019-01-10 15:03:03 -08:00
Bruce Martin
b90447c387 binary wire format with flatbuffers (#509)
* first flatbuffer schema

* do not lint auto-generated files

* add flatbuffers package

* add flatbuffer module

* wire up /data/X/T route

* use flatbuffers for matrix data fetc

* clarity and comments

* add flatbuffer layout route

* clean up obsolete code

* fix tests

* move flake8 config to setup.cfg

* add comments

* lint

* rework layout routes for fbs

* add more type support to fbs

* lint

* add flatbuffer support for annotations

* function name improvements

* fix botched merge with master

* remove unused import

* route cleanup for flatbuffers

* rename function for clarity

* add missing globals to Jest tests

* fix client JS tests

* fix routes for Python tests

* comments for clarity

* non-finite floating point hardening

* more non-finite number handling

* lint

* fix tests for summarizeAnnotations

* harden diffexp calculation against FP errors

* cleanup unused code

* lint

* add encoding tests for flatbuffers

* application type specified as strings

* fix spelling error

* improve variable names

* add note about documentation gap

* rename FBS DataFrame to Matrix
2019-01-09 14:26:05 -08:00
Charlotte Weaver
f0f7200f0b Move jsonification to engine level (#511) 2018-12-18 21:14:24 -08:00
Charlotte Weaver
83154577e4 Black -- formatter for python (#508)
* Add black

* use black to format code

* Black version
2018-12-12 09:44:47 -08:00
Bruce Martin
3bfeadc2b9 improve handling of non-finite floating point values (#490)
* improve HTTP error reporting

* generate standards-compatible JSON

* add --nan-to-num work-around for non-finite floating point values

* lint

* update tests

* correctly set Infinities to min/max

* REAMDE update for --nan-to-num

* define constant for repetitive warning message

* clarify where NaN errors will occure
2018-12-04 14:56:16 -08:00
Charlotte Weaver
455d987398 Add custom format for warnings (#477)
[cellxgene] Warning: <message>
2018-11-28 12:47:11 -08:00
Charlotte Weaver
7d5c054b90 Launch validation (#414)
* WIP

* Cleanup

* Validation

* typo

* Max category limit 100 -> 500
2018-11-07 13:01:13 -08:00
Charlotte Weaver
94f95d6565 CLI Launch (#366)
* Scanpy engine now required

Without the --engine param we need to error if scanpy engine cannot be imported rather than waiting for all engines

* CLI options and help matches proposal

(but not all options hooked up yet)

* Flesh out top level args

* Move computation args to engine

* CLI input file (#374)

* Fix test command

(tests still won't work)

* Input is file instead of directory
- also renamed example file

* Csweaver/debug (#376)

* Respect debug flag for logging flask calls

* Add loading messages

* max categories (#377)

* Add max categories

* Rename max_categories to category_selection_limit

* ensure whole numbers
2018-10-24 19:28:59 -07:00
Charlotte Weaver
06f402ab0a Do not calculate layout, used saved layout instead (#343)
* Do not calculate layout, used saved layout instead

See for rationale: https://docs.google.com/document/d/1HJFvbdDHxxgkCW0DZzdTZ9CUMgc2ef2rQFukATBQWvE/edit

* Error handling for when layout has not been precomputed

* Server error (500) not client error (400) for unprepared data
2018-10-18 11:51:38 -07:00
Charlotte Weaver
210787cb5d Filter in engine (#307)
* Add empty filter case

* Filtering dataframes moved to engine instead of rest

* minor changes from PR review

* Minor fixes from PR review

Pass {} instead of none if no filter
chain exceptions
typos
2018-10-11 15:23:31 -07:00
Charlotte Weaver
11ef5ae51b REST Error handling (#299)
* Use HTTPStatus for all responses

More informative than just the code as an int

* Fill out REST error handling

* Test error routes

* Use HTTP Status for tests too

* factor mime type request into function

* Better mimetype errors

* 404 -> 400 error for bad key
2018-10-08 14:17:56 -07:00
Bruce Martin
eeec842ad0 Restv2 feature branch merge to master (#284)
Move to new REST v0.2 communication between front and back-end.   This is a first cut implementation which is functional, but will need follow-up enhancements for performance, error checking, etc.    Protocol spec is in docs directory.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Add filtering via indexing

* Using new filter specs

Indexing working

* Added filtering by annotation value

* factor out common methods

* Documentation

* create enum for axis (obs/var)

* Better description for filter's return

* Add boolean to enumerated types

* Augmented enum for scanpy axis

* Create schema for annotations

Based on datatype within scanpy/anndata
+ tests

* remove obsolete schema parse script

* Update rest api to remove old routes and add schema route

* Separate development requirements

* Warning for unsupported datatypes

* include -r requirements.txt in dev

* Merged downcast warnings

* Fixed bug where names were NaNs

Needed to include the index too when creating the series

* Add config endpoint

* Generate app features from CLI selections

* Move features to driver

* Add tests for schema

* Clearer version wording

* python3 version of super

* version from engine to package level

* move features to driver

* Revise layout function to match the new spec

* GET for layout/obs

* PUT Layout (#211)

* PUT Layout

* Csweaver/annotations (#212)


* Update scanpy engine to support the rest v0.2 annotation requests

* GET endpoint for obs annotations + tests

* Documentation

* Test annotations in scanpy engine

* Description for annotation-keys param

* annotation->annotations

* clarified return for annotations

* Use URL query list for annotations fields

* parse_filter parses v0.2 GET filters (#215)

* parse_filter parses v0.2 GET filters

* Don't allow index filters from query params

* Better variable conversion

* Parse filter improvements

- uses default dict
- renamed filter -> query_filter

* Cleanup Tasks (#216)

* Add test_api back into travis build

* Do custom JSON encoding the correct way

* Run cellxgene server in test setup

* Cleanup new tests too

* Option to bind to all interfaces (#225)

app.run("0.0.0.0") instead of app.run("127.0.0.1") binds to all interfaces.

Note: There are comments on the internet that says that the flask server is not up to the task of production serving.  I don't think that such scalability concerns apply here, but I was able to get cellxgene working with twistd relatively easily, and we could switch to that if there are scalability concerns.

Test plan: browsed to <ip>:5005/api/v0.2/config on a different host.

* Fix merge errors

- import warnings was improperly deleted
- scanpy engine tests were totally wrong

* Fix merge error with driver

* PUT /annotations (#235)

* Add query param for annotation name

* fix descriptions, eliminate else clause

* first cut at initial data load on rest 0.2 api

* Annotation var (#248)

* Fix bug strings are always objects in pandas

* Add axis to annotation method

* Add /annotation/var to REST api

* Csweaver/expressiondata (#242)

* Refactor expression method for REST v2

* Add message to QueryStringError

* Fix range filters

* Add GET route for /data

* /data PUT route

* rename expression to data_frame

* clarification of error

* Improve accept type handling

* support all schema types for 0.2 REST API

* remove REST 0.1 code; connect var annotations loading

* config reducer; use config to set data set title; remove obsolete templating code for data set title

* REST 0.2 expression conversion support

* partial port of expression to REST 0.2

*  diffexp (#273)

* Add diffexp method to scanpy

and test

* Minor tweaks to diffexp

Get a minimal working version to unblock FE development

* Fixing things git deleted

* cleanup print statements

* Add index test

* additional, partial REST 0.2 bring up of diffexp

* Ignore unstructured annotations for data (#275)

This is a temp hack, need to figure out how to include data.uns if there is only one gene

* diffexp REST 0.2 port finish

* ignore unstructured annotaitons on all routes except layout

* correctly use varDataCache; maintain state during world rebuild

* correct varDataCache use

* temporarily disable all memoization

* refinements to expression data caching

* clear cell sets upon regraph/reset

* update version of REST to 0.2

* Travis build fixes

- comment out cache import
- fix duplicate test name

* Remove dependency from travis

* clarify semantics of config variables

* move generic action helpers into util
2018-10-01 14:58:46 -07:00
Charlotte Weaver
5b9ddfa1a6 Added documentation to filter 2018-07-06 15:40:21 -07:00
Charlotte Weaver
56c932a443 Fixed categorical filter not being converted to type, added tests 2018-06-29 17:16:00 -07:00
Charlotte Weaver
dd5fa57259 renaming backend, cellxgene to server, client respectively 2018-06-26 11:41:32 -07:00