Github Actions Workflow updates compatibility matrix:
- Added MacOS Catalina and Big Sur to test compat matrix
- Added Python 3.9 to test compat matrix, but avoid running 3.9 for matrix jobs that do not have `tables` pypi build available for the given env
- Maintains running tests on both cellxgene main branch and latest pypi release.
- Add explicit matrix exclusions for matrix combinations that will never pass (see comments).
- Numerous refactorings to the workflow config to simplify matrix. Basically a rewrite.
- The anndata pkg is now tested at a pinned release and at latest release, but no longer using `master` branch version. To limit cross-product explosion of matrix jobs, the pinned anndata version is only tested on py3.8 and cellxgene latest release.
- Run unit and smoke tests in a single job, to improve speed, reduce workflow complexity and the number of jobs. Also fixes the redundant testing of unit tests. Within each job, the unit and smoke tests are run in separate steps for ease of troubleshooting.
- Fixed termination of backend server to allow both smoke tests to run within a single job (both attempt to use 5005 port, sequentially, but first server was not being terminated).
- Replaced `continue-on-error: true` with `fail-fast: false`, which allows all matrix jobs to run independently, while also ensuring the that entire workflow is flagged as failed if any matrix job fails
- The `smoke-test-annotations` fail intermittently and have been disabled. Fix will be addressed in story: https://app.zenhub.com/workspaces/single-cell-5e2a191dad828d52cc78b028/issues/chanzuckerberg/cellxgene/2433
This splits the backend into two parts: the local backend for desktop cellxgene and the AWS backend for hosted cellxgene. The local backend is in local_server while the hosted remains in server. The general idea is to copy everything from server to local_server, pull unneeded stuff out of local_server, and keep server as-is for this PR. Not touching server means all the infra and deployment code will continue working just as it did before so we can make those changes incrementally.
Until numpy version 1.20.0, numpy.unicode was an alias for str in python3. In 1.20.0, it's fully deprecated and is an int. This is bad and breaks things. This commit drops the np.unicode alias and just uses str, as is advised here:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
In this solution, all the server requirements are installed.
This is a slightly overkill, but it avoid having to restructure
any of the server or test code to avoid unnecessary imports.
#2019
* Convert float annotations if possible.
The client converts all arrays to floats.
If a category contains integer labels, and that category is copied, it will contains floats (e.g 1.0 instead of 1).
When that category is put back to the server, it fails in the tiledb code, which does not accept floats.
The solution is to convert a float category to integer, if possible.
#1984
* updates
Previously if the user remove all annotations, the code would still generate a tiledb uri
in the write_labels call, and add that to the database. A tiledb array would not be written in this case.
When the read_labels was then called, it would find the entry in the database, attempt to open
the tiledb array, then fail.
The patch here will set the tiledb_uri to the empty string if all categories are removed.
When read_labels is called, it will see the empty uri and return None.
Furthermore, if the database does have a tiledb_uri that does not exist, or cannot be read,
then the code will now log a warning, and return None (instead of throwing an exception,
which results in a server error).
#1932
Add the `cellxgene schema apply` and `cellxgene schema validate` subcommands.
The first takes an h5ad file and a yaml with config information and produces a new h5ad that follows the cellxgene data integration schema.
The second takes an h5ad and checks if it follows the schema version written into its metadata.
Both are currently marked as "experimental" as the primary intended users are still at CZI.
* Updates due dependency version changes.
h5py recently changes and now values once returned as str are now returned as bytes.
This would have caused a much larger change, so instead the version is restricted to <3.0.0.
This caused the bulk of the testing failues.
A few other changes were needed to make a few other tests pass.
#1959
* Fix bug in oauth.
The error checking was too specific, and missed a case.
Make the error checking catch all exceptions.
#1947
* Add logging when the cookie cannot be processed
* add tests
* run black
* run black and add disclaimer that tweaked errors on server
* lint
* change to get so it will return None
* tweak existing token instead of new one
* Trigger
* token is dict
* jsonify dict before encoding
* json dump instead of jsonify
* encode into bytes object
* use correct id token
* decode byte to string
* Update readme for eb server.
Update the README with new way of handling secrets.
Update portions that were out of date.
Add a section for Authentication and a placeholder for User Annotations.
Also remove an obsolete function that processes the AWS secrets.
#1522
Co-authored-by: Madison Dunitz <madison.dunitz@chanzuckerberg.com>
* Enhance the AppConfig with external config sources.
The external config sources are currently environment variables
and AWS secrets manager.
The config file can be augmented with a section describing how
environmen variables and secrets can update config parameters.
benefits:
- it will enable the config to draw from more than one secret. This is useful
for shared secrets between cellxgene and data portal, as well as auth0 secrets.
- it will make it very straightforward to check the config before a deployment.
Part of #1859
* make testing plan
* create annotaions sets for different num categories/dataset size
* annotation creation testing
* create scale and perf tests for annotations
* create make commands for tests
* get cell count if not set in test_datasets dict
* Remove door icon from log in button
* Move log in and info buttons from the top bar to in line with the cellxgene icon and dataset name
* Hover over on login button should say "Log in to cellxgene"
* Show email
closes#1830
* split out config
* add tests for base and app config, refactor client config out of app config
* refactor default config retrieval
* create config test class and helper functions
* move default_config into server to fix import issue
The config file had a bug where it expected both a "server" and "dataset" section.
If one didn't exist, then it would raise an exception.
It should use the default server config or the defaul dataset config in those cases.
Added a test case that would have caught this.
