* Add basic authentication in the server
A pattern for creating authentication methods is introduced, with three
authentication types defined:
none - no authentication
session - like the current session based auth used for user annotations
test - used to test the login/logout process end to end
The config endpoint now returns informations about the authentication, like if
the user is authenticated and their username. The redirect uri's for login and
logout are also returned if the authentication type requires login
This is the first a several PRs for authentication.
*. Update server tests to avoid hardcoded ports
test_api and test_nan_rest now use a common function for starting a test server,
than will initially choose a random port.
This will give us the ability to specify different config options for
different dataroots.
the key of the dataroot dictionary is no longer the same as the dataroot_url.
Previously key==dataroot_url, and now those are separated.
Added an "is_multi_dataset" function to simplify logic where it branched on single vs multi.
Simplified the rest.py interface by no longer passing in the user annotations object, since
that can be retrieved from the dataset.
* Add user-defined category-label colors
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1152
As described in https://github.com/chanzuckerberg/cellxgene/issues/1307
* Respond to feedback from @bkmartinjr in nodejs
* Respond to feedback from @bkmartinjr in python
* Add tests to the server module
* Autoformat python, run linter
* Make colors_get error handling specific
* Respond to feedback from @bkmartinjr
* Respond to feedback from @bkmartinjr
* Fix whitespace
* Fix python lint errrors
* Update documentation
* Add --disable-user-colors option to launch and cxgtool.py
* Fix python formatting
* Rename '--disable-user-colors' to '--disable-custom-colors'
fixes an issue with "cellxgene launch" which had a bad interaction between
command line parameters and config file parameters.
Now, the config files are applied first, followed by the parameters that
were provided in the command line.
There is also now a check that each of the config attributes is type checked.
* Improvements to the matrix cache
- Add a timelimit for the matrix in the cache.
Once the timelimit is reached, the matrix can be removed.
- If a DatasetAccessError occurs, then remove the dataset
from the matrix cache.
Fixes#1322
* Added a config hook for secret key into the app.
the server first looks in an environment variable,
then looks in a config file.
For the cellxgene launch app, a default key is used if none is provided.
For the eb app, a secret key must be provided.
* black reformat
* tighten up error reporting
* lint
* fine tuning
* additional improvements in exception handling
* lint
* include exception and traceback in log
* fix typo
* Introduce a config file to cellxgene
The config file format is in yaml. The default config is located
in server/common/default_config.py. A user may create a yaml file
that contains a subset of these fields. It can be used during cellxgene
launch, or for hosted cellxgene.
The code has also been refactored. Much of the logic to check arguments
has moved from launch to app config.
It is now possible to set the tiledb context parameters using the config
file. Other feature will soon be handled in a similar way.
* Improve hosted cellxgene
- option to turn off the test index page, or supply a page for redirect.
For EB, The default is to return 404. For cli launch, the default is the test page.
- option to select which matrix types are allowed for multi dataset servers.
For EB, The default is CXG only. For cli launch, the default is any matrix type.
- Return early with an error response if diffexp is requested when not configured
- Verified that reembedings and user annotations also return with an error response
if used when not enabled.
TODO: The new options cannot currently be set by the user.
I plan to add a configuration file where these and all other settings can be set.
Fixes#1210Fixes#1228Fixes#1229
* Fix for favicon with --dataroot
* fix static assets in hosted cxg
The web proxy at aws eb was not finding the static assets.
The solution here is very simple: just copy the directory
containing the static assets to the top level of the artifact.zip.
This is not really the ideal solution. According to the AWS
docs you can make a mapping to the correct location in an
an ebextentions config file. I tried this and many combinations but
was not able to get this to work following that pattern.
Since we control the construction of the zip file, the solution
here isn't bad, but it could probably be made better.
* first cut at re-embedding route and back-end support
* update and expand config route tests
* add scanpy_umap
* add reembedding to config route parameters
* front-end support for reembedding fetch and UI
* remove unused imports
* add loading state
* save reembedding in reducer state
* improve withColsFrom
* transmit reembed schema to client; pick unique embedding names
* display embeddings
* format
* lint
* spaces, tab size 2
* lint
* test hack for smoke-test race
* back out hack sleep
* add check for backed mode
* add unit test for reembedding
* lint
* hide re-embedding CLI param from help
* Add user-generated annotations tests to the server
Partially completes https://github.com/chanzuckerberg/cellxgene/issues/969
* Auto-format python code
* @skip_if: passing lambdas > than property strings
* Respond to feedback from @bkmartinjr
This PR contains a refactoring to make adding new features easier.
The new features include supporting the tiledb format, and the multi dataset application.
The refactoring includes
Simplifying the directory structure and files.
a class structure to handle annotations (currently one type: AnnotationsLocalFile).
a class to handle application configuration
a class structure to handle matrix data (currently AnndataAdaptor and CxgAdaptor). CxgAdaptor uses tiledb.
Algorithms that were previously dependent on the scanpy anndata object are now generalized to work with an abstract interface.
The multi dataset option is not fully supported yet, and so the option to use it is hidden.
Use "cli launch --dataroot ..."
To access this feature.
All combinations of app single dataset/ app multi dataset and AnndataAdaptor/CxgAdaptor work with all the features, such as annotations, ontologies, diffexp.
* revert MatrixProxy; replace with correct use of adata slicing
* work around 0.6 adata slicing bug
* fix incorrect var slice
* simplify slicing of X
* add warning about performance impact of anndata<=0.7
* lint and remove unused code
* improve comment
* lint
* correctly parse versions
* temp files should preserve file suffix if possible - anndata 0.7 compat
* update anndata dependency to 0.6.20
* resolve PR review comments
* add sample ontologies file
* add ontologies reducer
* Move select category to own component
* Dialog and Input factored out
* refactoring categorical, partway
* validationn
* anno
* suggest populates input
* frontend for ontology working
* initial implementation of back-end support for ontologies
* edit is now dialog again
* autosuggest working on edit
* part way through create arbitrary label
* handle choice in function
* pass duplicate cat prop
* editing works
* update test to match new CLI params
* fix occupancy alignment
* edit category as dialogue
* secondary button
* remove stubbed out ontologies
* add label setting upon new label creation
* Update legal characters for labels (#1119)
* Allow any term in the ontology (bypass legal name check)
* Add hyphens and parens to legal characters in names
* improve performance for large ontologies
* correctly handle case where ontologies are disabled
* fix logic error in CLI
Co-authored-by: Bruce Martin <bruce@chanzuckerberg.com>
* PR cleanup 1
* lint
* validate user generated labels
* finish hooking up connected suggest component
* protect against undefined callbacks
* Fix illegal characters error message
* break out npm run commands
* fix error detection on label edit
Co-authored-by: Bruce Martin <bruce@chanzuckerberg.com>
Co-authored-by: Sidney Bell <sidneymbell@users.noreply.github.com>
* Fix Makefile whitespace and .PHONY use
* Fix Makefile filename
* Modularize Makefile into client and server Makefiles
Part of the reason that the Makefile in the root directory is a bit
complicated is that it tries to handle tasks that can be handled
separately in the client and server modules.
This commit pushes some of the make logic specific to each module into
their own makefiles and calls out to those makefiles from that in the
project root.
* Add auto-formatting to client and server modules
One thing that can make linting faster is auto-formatting. This commit
adds the yapf auto-formatting tool to the server module and uses
eslint's "fix" functionality to speed up the linting/formatting process.
* Add yapf for automatic code formatting
* Add a root test target that calls sub-tests
* Apply yapf to python files
* Do not duplicate npm commands, simply pass through
* Update documentation
* Do not shadow reserved word len
* Add general test target
* Fix make call in dev-env
* Use black instead of yapf
* Run flake8 from the root directory
* Revert "Apply yapf to python files"
This reverts commit cdca128a01.
* Apply black to python code
* Resolve lint errors resulting from black format
* Add explanation of server unit tests in dev guidelines
* rename config param label-file
* annotations rework - CLI params, file naming and backups
* lint
* improve cli option error checks
* enable session cookies
* enable session cookies
* add session id
* name annotations file in multi-dataset and multi-user safe manner
* pass data user hash to front-end
* add annotation collection name support to front-end
* add constant for annotation data collection name
* parameterize annotation collection name; make it sticky in the session
* clarify comments
* hard wire a temporary data collection name for testing
* prettier
* test comment
* package command
* set annotations filename dialog
* name and hash are visible
* wire up data collection capture
* add last mod time lookup to data locator
* store data locator in Driver
* save metadata header in labels csv
* lint
* fix tests
* change datetime format to second precision
* initial cut at backed mode
* make flask multithreading conditional on debug flag
* update X access to support backed mode
* lint
* improve help message for backed mode
* fix tests
* add MatrixProxy to normalize supported matrix types
* add FAQ entry for --backed
* remove use of matrix.T
* clean up
* add ability to disable diffexp from CLI; add hueristic to detect likely slow diffexp calculation, and warn user
* fix tests
* do not print diffexp speed warning if diffexp is disabled
* tweak wording of diffexp speed messages
* add FAQ entry on --disable-diffexp
* revise heuristic for warning about slow diffexp
* use quick tooltip delay on diffexp button
* add about arg
* add simple url validator
* attach about link to config api
* add links to configDefaults
* add conditional link in top left and menu item
* whitespace
* change to lower case
* move --about arg before click.command()
if this fixes it I have no idea why
* change link>URL
* be more descriptive about URL
* Make error more explicit
* refactor attach_data to accept about
* format
* change icon
* add trailing parenthesis
* whitespace
* initial commit of URL support for launch
* lint
* modify tests to use new data locator
* add locator unit tests
* fix typo in faq
* more lint
* update faq per PR review
