This splits the backend into two parts: the local backend for desktop cellxgene and the AWS backend for hosted cellxgene. The local backend is in local_server while the hosted remains in server. The general idea is to copy everything from server to local_server, pull unneeded stuff out of local_server, and keep server as-is for this PR. Not touching server means all the infra and deployment code will continue working just as it did before so we can make those changes incrementally.
* split out config
* add tests for base and app config, refactor client config out of app config
* refactor default config retrieval
* create config test class and helper functions
* move default_config into server to fix import issue
* Fixes from frontend/backend url separation
This fixes the CORS and CSP headers.
Also, in thie commit, I removed the cors_supports_credentials config parameter,
which was recently introduced.
Instead, the logic determines the need to use CORS headers if the
web_page_url is set.
#1778
* separate backend base url from frontend
This is needed for auth, and to support a different location for the backend api server,
than the frontend.
part of chanzuckerberg/cellxgene#1778
new server config parameters: app__api_base_url, app__web_base_url
Also changed api_base_url in the oauth config section to "oauth_api_base_url" to
be less confusing with the app's api_base_url
Other minor changes:
changed how the jwt decode options are handled.
Previously they needed to be set in a test case, and there was some extra logic to handle that.
Now they are handled through comfig parameters, which makes it more general.
Also, add a feature to set the CORS support credentials, which seems
to be necessary for the backend/frontend separation, at least when run
locally. This part is sort of experimental, and may be removed or changed later.
* save tiledb array to s3, dont cache user annotations
* Add option to disable annotation filename prompt (#1787)
Co-authored-by: Madison Dunitz <dunitzm@gmail.com>
* set tiledb default context in cxg_adaptor
Co-authored-by: maniarathi <arathi.mani@chanzuckerberg.com>
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
* Separate userinfo from the config endpoint
previously information about if the user was logged in and their username
was part of the config endpoint.
However, the config endpoint was previously static, and has a cache control.
Rather than not caching the config, a new endpoint called "userinfo"
is created to handle that information.
The config endpoint still has the non-changing part of the authentication:
config:
authentication:
requires_client_login: True/False
login: <uri to login endoint if requires_client_login is True>
logout: <uri to logout endoint if requires_client_login is True>
The userinfo endpoint returns this information:
userinfo:
is_authenticated: True/False
username: <string if is_authenticated>
if authentication is not enabled then the config does not have an authentication key,
and userinfo returns None.
Also in the PR are a few minor code improvements and bug fixes
Co-authored-by: Colin Megill <colinmegill@gmail.com>
* add oauth authentication
Add support for OAuth2.
Change the interface to AuthTypeBase
- better handling of config parameters
- add a complete_setup function for additional setup steps
Added a function wrapper to enforce authentication for the
routes that require authenticaiton.
* change fsspec requirement
fsspec 0.8.0 breaks our tests
it imports a module that is does not require.
* Add basic authentication in the server
A pattern for creating authentication methods is introduced, with three
authentication types defined:
none - no authentication
session - like the current session based auth used for user annotations
test - used to test the login/logout process end to end
The config endpoint now returns informations about the authentication, like if
the user is authenticated and their username. The redirect uri's for login and
logout are also returned if the authentication type requires login
This is the first a several PRs for authentication.
*. Update server tests to avoid hardcoded ports
test_api and test_nan_rest now use a common function for starting a test server,
than will initially choose a random port.
This will give us the ability to specify different config options for
different dataroots.
the key of the dataroot dictionary is no longer the same as the dataroot_url.
Previously key==dataroot_url, and now those are separated.
Added an "is_multi_dataset" function to simplify logic where it branched on single vs multi.
Simplified the rest.py interface by no longer passing in the user annotations object, since
that can be retrieved from the dataset.
* Add user-defined category-label colors
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1152
As described in https://github.com/chanzuckerberg/cellxgene/issues/1307
* Respond to feedback from @bkmartinjr in nodejs
* Respond to feedback from @bkmartinjr in python
* Add tests to the server module
* Autoformat python, run linter
* Make colors_get error handling specific
* Respond to feedback from @bkmartinjr
* Respond to feedback from @bkmartinjr
* Fix whitespace
* Fix python lint errrors
* Update documentation
* Add --disable-user-colors option to launch and cxgtool.py
* Fix python formatting
* Rename '--disable-user-colors' to '--disable-custom-colors'
fixes an issue with "cellxgene launch" which had a bad interaction between
command line parameters and config file parameters.
Now, the config files are applied first, followed by the parameters that
were provided in the command line.
There is also now a check that each of the config attributes is type checked.
* Improvements to the matrix cache
- Add a timelimit for the matrix in the cache.
Once the timelimit is reached, the matrix can be removed.
- If a DatasetAccessError occurs, then remove the dataset
from the matrix cache.
Fixes#1322
* Added a config hook for secret key into the app.
the server first looks in an environment variable,
then looks in a config file.
For the cellxgene launch app, a default key is used if none is provided.
For the eb app, a secret key must be provided.
* black reformat
* tighten up error reporting
* lint
* fine tuning
* additional improvements in exception handling
* lint
* include exception and traceback in log
* fix typo
* Introduce a config file to cellxgene
The config file format is in yaml. The default config is located
in server/common/default_config.py. A user may create a yaml file
that contains a subset of these fields. It can be used during cellxgene
launch, or for hosted cellxgene.
The code has also been refactored. Much of the logic to check arguments
has moved from launch to app config.
It is now possible to set the tiledb context parameters using the config
file. Other feature will soon be handled in a similar way.
* Improve hosted cellxgene
- option to turn off the test index page, or supply a page for redirect.
For EB, The default is to return 404. For cli launch, the default is the test page.
- option to select which matrix types are allowed for multi dataset servers.
For EB, The default is CXG only. For cli launch, the default is any matrix type.
- Return early with an error response if diffexp is requested when not configured
- Verified that reembedings and user annotations also return with an error response
if used when not enabled.
TODO: The new options cannot currently be set by the user.
I plan to add a configuration file where these and all other settings can be set.
Fixes#1210Fixes#1228Fixes#1229
* Fix for favicon with --dataroot
* fix static assets in hosted cxg
The web proxy at aws eb was not finding the static assets.
The solution here is very simple: just copy the directory
containing the static assets to the top level of the artifact.zip.
This is not really the ideal solution. According to the AWS
docs you can make a mapping to the correct location in an
an ebextentions config file. I tried this and many combinations but
was not able to get this to work following that pattern.
Since we control the construction of the zip file, the solution
here isn't bad, but it could probably be made better.
* first cut at re-embedding route and back-end support
* update and expand config route tests
* add scanpy_umap
* add reembedding to config route parameters
* front-end support for reembedding fetch and UI
* remove unused imports
* add loading state
* save reembedding in reducer state
* improve withColsFrom
* transmit reembed schema to client; pick unique embedding names
* display embeddings
* format
* lint
* spaces, tab size 2
* lint
* test hack for smoke-test race
* back out hack sleep
* add check for backed mode
* add unit test for reembedding
* lint
* hide re-embedding CLI param from help
* Add user-generated annotations tests to the server
Partially completes https://github.com/chanzuckerberg/cellxgene/issues/969
* Auto-format python code
* @skip_if: passing lambdas > than property strings
* Respond to feedback from @bkmartinjr
