* work around anndata bug 344
* fix accidental cut and paste error
* Use modified make_index_unique function
Temporarily copy code from https://github.com/theislab/anndata/pull/345
until the issue is resolved and released.
* Add notes and test for make_index_unique
* Lint fix
* Format python
Co-authored-by: Matt Weiden <538456+mweiden@users.noreply.github.com>
* Introduce a config file to cellxgene
The config file format is in yaml. The default config is located
in server/common/default_config.py. A user may create a yaml file
that contains a subset of these fields. It can be used during cellxgene
launch, or for hosted cellxgene.
The code has also been refactored. Much of the logic to check arguments
has moved from launch to app config.
It is now possible to set the tiledb context parameters using the config
file. Other feature will soon be handled in a similar way.
* first cut at re-embedding route and back-end support
* update and expand config route tests
* add scanpy_umap
* add reembedding to config route parameters
* front-end support for reembedding fetch and UI
* remove unused imports
* add loading state
* save reembedding in reducer state
* improve withColsFrom
* transmit reembed schema to client; pick unique embedding names
* display embeddings
* format
* lint
* spaces, tab size 2
* lint
* test hack for smoke-test race
* back out hack sleep
* add check for backed mode
* add unit test for reembedding
* lint
* hide re-embedding CLI param from help
* Add undo/redo tests for annotations
Fixes https://github.com/chanzuckerberg/cellxgene/issues/969
... also refactor the tests for DRY.
* Add done()
* Make e2e annotations tests safer to concurrency
* Add data-testclass for save state.
* Simplify tests and make them dependent on save state
* Add user-generated annotations tests to the server
Partially completes https://github.com/chanzuckerberg/cellxgene/issues/969
* Auto-format python code
* @skip_if: passing lambdas > than property strings
* Respond to feedback from @bkmartinjr
This PR contains a refactoring to make adding new features easier.
The new features include supporting the tiledb format, and the multi dataset application.
The refactoring includes
Simplifying the directory structure and files.
a class structure to handle annotations (currently one type: AnnotationsLocalFile).
a class to handle application configuration
a class structure to handle matrix data (currently AnndataAdaptor and CxgAdaptor). CxgAdaptor uses tiledb.
Algorithms that were previously dependent on the scanpy anndata object are now generalized to work with an abstract interface.
The multi dataset option is not fully supported yet, and so the option to use it is hidden.
Use "cli launch --dataroot ..."
To access this feature.
All combinations of app single dataset/ app multi dataset and AnndataAdaptor/CxgAdaptor work with all the features, such as annotations, ontologies, diffexp.
* revert MatrixProxy; replace with correct use of adata slicing
* work around 0.6 adata slicing bug
* fix incorrect var slice
* simplify slicing of X
* add warning about performance impact of anndata<=0.7
* lint and remove unused code
* improve comment
* lint
* correctly parse versions
* temp files should preserve file suffix if possible - anndata 0.7 compat
* update anndata dependency to 0.6.20
* resolve PR review comments
* Add smoke test for annotations features
* Do not save during annotations tests
* Fix botched rebase in dev guidelines
* Revert "Do not save during annotations tests"
This reverts commit f0bd970bb2.
* Respond to feedback from @bkmartinjr
* add sample ontologies file
* add ontologies reducer
* Move select category to own component
* Dialog and Input factored out
* refactoring categorical, partway
* validationn
* anno
* suggest populates input
* frontend for ontology working
* initial implementation of back-end support for ontologies
* edit is now dialog again
* autosuggest working on edit
* part way through create arbitrary label
* handle choice in function
* pass duplicate cat prop
* editing works
* update test to match new CLI params
* fix occupancy alignment
* edit category as dialogue
* secondary button
* remove stubbed out ontologies
* add label setting upon new label creation
* Update legal characters for labels (#1119)
* Allow any term in the ontology (bypass legal name check)
* Add hyphens and parens to legal characters in names
* improve performance for large ontologies
* correctly handle case where ontologies are disabled
* fix logic error in CLI
Co-authored-by: Bruce Martin <bruce@chanzuckerberg.com>
* PR cleanup 1
* lint
* validate user generated labels
* finish hooking up connected suggest component
* protect against undefined callbacks
* Fix illegal characters error message
* break out npm run commands
* fix error detection on label edit
Co-authored-by: Bruce Martin <bruce@chanzuckerberg.com>
Co-authored-by: Sidney Bell <sidneymbell@users.noreply.github.com>
* Notify users of new versions of cellxgene
Fixes https://github.com/chanzuckerberg/cellxgene/issues/683
* Do not use PyGithub client
* Protect against AttributeError
* Document that all version tags must follow SemVer
* Release tags `should -> MUST` follow semantic versioning
* Fix Makefile whitespace and .PHONY use
* Fix Makefile filename
* Modularize Makefile into client and server Makefiles
Part of the reason that the Makefile in the root directory is a bit
complicated is that it tries to handle tasks that can be handled
separately in the client and server modules.
This commit pushes some of the make logic specific to each module into
their own makefiles and calls out to those makefiles from that in the
project root.
* Add auto-formatting to client and server modules
One thing that can make linting faster is auto-formatting. This commit
adds the yapf auto-formatting tool to the server module and uses
eslint's "fix" functionality to speed up the linting/formatting process.
* Add yapf for automatic code formatting
* Add a root test target that calls sub-tests
* Apply yapf to python files
* Do not duplicate npm commands, simply pass through
* Update documentation
* Do not shadow reserved word len
* Add general test target
* Fix make call in dev-env
* Use black instead of yapf
* Run flake8 from the root directory
* Revert "Apply yapf to python files"
This reverts commit cdca128a01.
* Apply black to python code
* Resolve lint errors resulting from black format
* Add explanation of server unit tests in dev guidelines
* rename config param label-file
* annotations rework - CLI params, file naming and backups
* lint
* improve cli option error checks
* enable session cookies
* enable session cookies
* add session id
* name annotations file in multi-dataset and multi-user safe manner
* pass data user hash to front-end
* add annotation collection name support to front-end
* add constant for annotation data collection name
* parameterize annotation collection name; make it sticky in the session
* clarify comments
* hard wire a temporary data collection name for testing
* prettier
* test comment
* package command
* set annotations filename dialog
* name and hash are visible
* wire up data collection capture
* add last mod time lookup to data locator
* store data locator in Driver
* save metadata header in labels csv
* lint
* fix tests
* change datetime format to second precision
* initial cut at backed mode
* make flask multithreading conditional on debug flag
* update X access to support backed mode
* lint
* improve help message for backed mode
* fix tests
* add MatrixProxy to normalize supported matrix types
* add FAQ entry for --backed
* remove use of matrix.T
* clean up
* add ability to disable diffexp from CLI; add hueristic to detect likely slow diffexp calculation, and warn user
* fix tests
* do not print diffexp speed warning if diffexp is disabled
* tweak wording of diffexp speed messages
* add FAQ entry on --disable-diffexp
* revise heuristic for warning about slow diffexp
* use quick tooltip delay on diffexp button
* initial commit of URL support for launch
* lint
* modify tests to use new data locator
* add locator unit tests
* fix typo in faq
* more lint
* update faq per PR review
* enforce column name uniqueness for obs and var
* parameterize the column name containing obs and var user-readable names
* use the new annotation index value from schema
* update f/e unit tests
* PR review suggestions
* lint
* WIP
* import find_available_port method
* move method to utils
so I can add to eventually add to gui
* add fixed-port flag to tests
* Update server/utils/utils.py
Co-Authored-By: Tony Tung <tonytung@merly.org>
* pr review suggestions
* pr review suggestions
* fix outdated package.json
* update error message
* simplify find_available_port function
* Auto scan for ports unless port is specified.
* fix tests
* fix comment for find_available_port
* lint error
* differentiate port error from generic os error
* add errno to OSerror
* pr review fixes
* raise e -> raise
* oserror -> socket error
* move app creation to function
* create engine without load
* flake 8 fixes
* cleanup original scanpy test
* add default config
* handle missing data
* test data changes
* unify update
* load data isn't static anymore
* make app a class
* dead code and route removal
* more dead code cleanup
* fix scanpy_engine tests
* lint
* add missing catch in filter parsing
* update scanpy NaN tests
* more fbs tests and dead test removal
* remove forced default for content type negotiation
* bit of cleanup
* more fbs test cleanup
* lint
* remove swagger
* swagger cleanup
* lint
* correctly handle lack of templates
* more dead code removal
* remove unused files
* fix dev build
* lint
* first flatbuffer schema
* do not lint auto-generated files
* add flatbuffers package
* add flatbuffer module
* wire up /data/X/T route
* use flatbuffers for matrix data fetc
* clarity and comments
* add flatbuffer layout route
* clean up obsolete code
* fix tests
* move flake8 config to setup.cfg
* add comments
* lint
* rework layout routes for fbs
* add more type support to fbs
* lint
* add flatbuffer support for annotations
* function name improvements
* fix botched merge with master
* remove unused import
* route cleanup for flatbuffers
* rename function for clarity
* add missing globals to Jest tests
* fix client JS tests
* fix routes for Python tests
* comments for clarity
* non-finite floating point hardening
* more non-finite number handling
* lint
* fix tests for summarizeAnnotations
* harden diffexp calculation against FP errors
* cleanup unused code
* lint
* add encoding tests for flatbuffers
* application type specified as strings
* fix spelling error
* improve variable names
* add note about documentation gap
* rename FBS DataFrame to Matrix
* add cutoff for low expression genes in topN selection
* remove debugging printfs
* change cli param name for diffexp cutoff
* change CLI param name
* second try at diffexp - using lfc sort with pval cutoff
* use lfc cutoff
* update comments to match code; cap p-value adjustment to max of 1
* lint
* explain diffexp in readme
* add link
* add diffexp-lfc-cutoff to test config
* update test to match revised diffexp spec
* fix latent bug in GET arg handling that was breaking tests
* lint
* comment cleanup
* fix variance overestimation so it is symmetric
* lint
* new diffexp REST API spec
* new diffexp REST API; faster diffexp and dataframe slicing
* first draft of fast diffexp
* convert variance calculation to two-pass method
* lint
* update front-end use of API
* fix typo in spec
* disable content compression
* catch index filter format errors
* clean up of dead code
* resolve PR review comments
* refactor cli to improve ux and enable easy incorporation of prepare as a subcommand
* switches to use click, which removes some boilerplate and gets us some improved ux for free
* changes the entry point for the cli
* changes the name of the browser option to --open and makes the default false
* add --obs-names and --var-names CLI params
* fix lint
* performance improvements in scanpy engine
* fix lint
* fix typo
* correctly handle sparse formats in diffexp
* fix diffexp and 1d slicing
* diffexp uses t-stat, not pval; clean up arg handling
* make _slice a static method
* revise scanpy tests to match new API
* Scanpy engine now required
Without the --engine param we need to error if scanpy engine cannot be imported rather than waiting for all engines
* CLI options and help matches proposal
(but not all options hooked up yet)
* Flesh out top level args
* Move computation args to engine
* CLI input file (#374)
* Fix test command
(tests still won't work)
* Input is file instead of directory
- also renamed example file
* Csweaver/debug (#376)
* Respect debug flag for logging flask calls
* Add loading messages
* max categories (#377)
* Add max categories
* Rename max_categories to category_selection_limit
* ensure whole numbers
* Launch browser to cellxgene
Also added --no-launch command line parameter
* Don't launch browser for tests
* rename no-launch to no-open
* Rejigger -no-open to positive destination
so later logic looks cleaner
* Add empty filter case
* Filtering dataframes moved to engine instead of rest
* minor changes from PR review
* Minor fixes from PR review
Pass {} instead of none if no filter
chain exceptions
typos
