* make testing plan
* create annotaions sets for different num categories/dataset size
* annotation creation testing
* create scale and perf tests for annotations
* create make commands for tests
* get cell count if not set in test_datasets dict
* split out config
* add tests for base and app config, refactor client config out of app config
* refactor default config retrieval
* create config test class and helper functions
* move default_config into server to fix import issue
The config file had a bug where it expected both a "server" and "dataset" section.
If one didn't exist, then it would raise an exception.
It should use the default server config or the defaul dataset config in those cases.
Added a test case that would have caught this.
* app config bug fix:
When reading a config file that included per_dataset_config,
the dataroot specializations were applied, but not the default config.
This PR fixes that and also includes a test for this case.
* separate backend base url from frontend
This is needed for auth, and to support a different location for the backend api server,
than the frontend.
part of chanzuckerberg/cellxgene#1778
new server config parameters: app__api_base_url, app__web_base_url
Also changed api_base_url in the oauth config section to "oauth_api_base_url" to
be less confusing with the app's api_base_url
Other minor changes:
changed how the jwt decode options are handled.
Previously they needed to be set in a test case, and there was some extra logic to handle that.
Now they are handled through comfig parameters, which makes it more general.
Also, add a feature to set the CORS support credentials, which seems
to be necessary for the backend/frontend separation, at least when run
locally. This part is sort of experimental, and may be removed or changed later.
* save tiledb array to s3, dont cache user annotations
* Add option to disable annotation filename prompt (#1787)
Co-authored-by: Madison Dunitz <dunitzm@gmail.com>
* set tiledb default context in cxg_adaptor
Co-authored-by: maniarathi <arathi.mani@chanzuckerberg.com>
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
* Handle the refresh token in oauth authentication
If the token has expired, then it can be refreshed to get a new token.
This is automatically handled by the server without the client being aware.
Also in the PR:
- refactor the auth_oauth.py file to more simply handle the save/restore of the token,
and the refresh token
- added an end2end test for oauth, which also tests refresh.
* adding python-jose and Authlib to requirements-dev.txt
They are needed in the auth_oauth test
* Separate userinfo from the config endpoint
previously information about if the user was logged in and their username
was part of the config endpoint.
However, the config endpoint was previously static, and has a cache control.
Rather than not caching the config, a new endpoint called "userinfo"
is created to handle that information.
The config endpoint still has the non-changing part of the authentication:
config:
authentication:
requires_client_login: True/False
login: <uri to login endoint if requires_client_login is True>
logout: <uri to logout endoint if requires_client_login is True>
The userinfo endpoint returns this information:
userinfo:
is_authenticated: True/False
username: <string if is_authenticated>
if authentication is not enabled then the config does not have an authentication key,
and userinfo returns None.
Also in the PR are a few minor code improvements and bug fixes
Co-authored-by: Colin Megill <colinmegill@gmail.com>
* Add basic authentication in the server
A pattern for creating authentication methods is introduced, with three
authentication types defined:
none - no authentication
session - like the current session based auth used for user annotations
test - used to test the login/logout process end to end
The config endpoint now returns informations about the authentication, like if
the user is authenticated and their username. The redirect uri's for login and
logout are also returned if the authentication type requires login
This is the first a several PRs for authentication.
*. Update server tests to avoid hardcoded ports
test_api and test_nan_rest now use a common function for starting a test server,
than will initially choose a random port.
This will give us the ability to specify different config options for
different dataroots.
the key of the dataroot dictionary is no longer the same as the dataroot_url.
Previously key==dataroot_url, and now those are separated.
Added an "is_multi_dataset" function to simplify logic where it branched on single vs multi.
Simplified the rest.py interface by no longer passing in the user annotations object, since
that can be retrieved from the dataset.
Many of our matrices are log normalized, which tends to eliminate
the number of non zero values (if there were any). This prevents
the matrix from being stored as a sparse matrix. The solution here
is to use a simple transformation to make it sparse again. The most
common value from each column is subtracted from that column. These
values that were subtracted are saved in an array called X_col_shift.
The cellxgene code needs to understand how to undo the transformation when
operating over the X matrix.
- added script to create a synthetic dataset for testing
- added a script to convert an existing CXG dataset to a sparse CXG dataset
Support for sparse tiledb arrays for the X matrix
1. cxgtool can now output sparse matrices
2. cxg_adaptor and diffexp_cxg updated to handle sparse matrices
3. added a test in test_diffexp to test sparse diffexp and get_X_array
* app_config, fix bug with list/tuple command line arguments.
There was a error caused by pyyaml using lists, and click using tuples.
Now tuples are automatically converted to lists when the config is
updated.
* Add server plugin system
Plugins are optional modules loaded at runtime. Specification:
* Plugins are loaded from the server.plugins module (directory
server/plugins)
* The import_plugins method is run as part of the initialization of the
server module in __init__.py
* Add plugins to the EB build process
* Remove bit of dead code
* Respond to feedback from @bmccandless
* Return empty colors for .cxg v0.0 files
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1440
The CxgAdaptor.get_colors method currently assumes that the .cxg file has
cxg_group_metadata. As a result, the /api/v0.2/colors endpoint always fails for
.cxg v0.0 files.
* Add test fixture
* Add user-defined category-label colors
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1152
As described in https://github.com/chanzuckerberg/cellxgene/issues/1307
* Respond to feedback from @bkmartinjr in nodejs
* Respond to feedback from @bkmartinjr in python
* Add tests to the server module
* Autoformat python, run linter
* Make colors_get error handling specific
* Respond to feedback from @bkmartinjr
* Respond to feedback from @bkmartinjr
* Fix whitespace
* Fix python lint errrors
* Update documentation
* Add --disable-user-colors option to launch and cxgtool.py
* Fix python formatting
* Rename '--disable-user-colors' to '--disable-custom-colors'
* Improve diffexp for tiledb
- The rows from the A and B sets are gathered and processed at the same time. In this
way the matrix is only accessed once instead of twice for each tile.
- There is now a single thread queue that gets shared between all callers of the diffexp.
This will slow down work if diffexp gets too busy.
- There is a target_workunit amount of work given to each thread. Previously the
workunit was (rows selected * width of tile), which could be small. Now multiple
column tiles can be combined into one workunit. If the target is too small then
thread and other overheads may reduce performance. If target_workunit is too large
then the size of the gathered sub matrix may take up too much memory.
- add configuration parameters (max_workers, cpu_multiplier, and target_workunit)
* Specialize diffexp for tiledb
This patch adds a new diffexp algorithm which is tuned for tiledb.
This algorithm was written by Bruce and is adapted here to plug into the
current framework. The anndata_adaptor still calls the original
algotithm (which was move from diffexp.py to diffexp_generic.py).
The cxg_adaptor now calls the new diffexp_tiledb version. Some
code is shared between the two.
This is part 1 of the diffexp for tiledb. Further tuning and
global throttles are still needed.
A script to run and time diffexp with various options is also
added: test/run_diffexp.py.
* warning on maxCount for diffexp
* cleanup logging
* clarification
* make the limits configurable
* make diff exp limit work
* danger!
* remove debugging code
* fix merge with master
* fix unit tests
Co-authored-by: Colin Megill <colinmegill@gmail.com>
fixes an issue with "cellxgene launch" which had a bad interaction between
command line parameters and config file parameters.
Now, the config files are applied first, followed by the parameters that
were provided in the command line.
There is also now a check that each of the config attributes is type checked.
