Commit Graph

355 Commits

Author SHA1 Message Date
maniarathi
a63bf9d5a3 Change psycopg to be binary (#1842) 2020-09-16 14:46:59 -07:00
Madison Dunitz
4f339e89b1 dont cache schema (#1836) 2020-09-15 11:20:00 -05:00
bmccandless
342a9d774c app config bug fix: (#1833)
* app config bug fix:

When reading a config file that included per_dataset_config,
the dataroot specializations were applied, but not the default config.
This PR fixes that and also includes a test for this case.
2020-09-14 13:15:47 -07:00
bmccandless
6a7ae8bc8e Fixes from frontend/backend url separation (#1829)
* Fixes from frontend/backend url separation

This fixes the CORS and CSP headers.

Also, in thie commit, I removed the cors_supports_credentials config parameter,
which was recently introduced.
Instead, the logic determines the need to use CORS headers if the
web_page_url is set.

 #1778
2020-09-12 10:56:31 -07:00
bmccandless
a7a4580944 separate backend base url from frontend (#1819)
* separate backend base url from frontend

This is needed for auth, and to support a different location for the backend api server,
than the frontend.

 part of chanzuckerberg/cellxgene#1778

new server config parameters:   app__api_base_url,   app__web_base_url

Also changed api_base_url in the oauth config section to "oauth_api_base_url" to
be less confusing with the app's api_base_url

Other minor changes:

changed how the jwt decode options are handled.
Previously they needed to be set in a test case, and there was some extra logic to handle that.
Now they are handled through comfig parameters, which makes it more general.

Also, add a feature to set the CORS support credentials, which seems
to be necessary for the backend/frontend separation, at least when run
locally.  This part is sort of experimental, and may be removed or changed later.
2020-09-11 09:50:16 -07:00
bmccandless
3f20f4a1f4 Change modify upgrade message to print to stderr instead of stdout (#1827)
When generating a config file, you can do this:
 > cellxgene launch --dump-default-config > myconfig.yaml

And then modify the myconfig.yaml.

However, if an upgrade is available then you would get extra lines in the yaml
file, which are not yaml code:

  There's a new version of cellxgene available (0.16.4)!
  To upgrade, run the following: pip install --upgrade cellxgene

To solve this problem, the upgrade messages are sent to stderr instead,
so they will appear on the screen and not in the config file.

Alternatives:
  One workaround is "cellxgene --no-upgrade-check launch --dump-default-config > myconfig.yaml"

But that's a bit verbose and not user friendly.

The way we've setup the upgrade check to be separate and before the launch sub command,
makes other code changes more involved.

 #1826
2020-09-11 09:24:25 -07:00
bmccandless
0a27b2923a Add error message and exit if reembeddings is enabled and scanpy is n… (#1812)
* Add error message and exit if reembeddings is enabled and scanpy is not installed

  fixes #1811
2020-09-02 15:35:50 -07:00
bmccandless
54b42607ae Update the location of deployment assets for the eb server (#1806)
put deploy scripts in /static/cellxgene/deploy instead of /static/deploy

  fixed chanzuckerberg/corpora-data-portal#558
2020-08-31 18:26:35 -07:00
maniarathi
ed865e9a57 Update the release process for community release to include release candidate versioning (#1802) 2020-08-31 16:16:21 -07:00
bmccandless
f8cdb12892 Fix frontend mishandling of null userinfo (#1795)
* Fix frontend mishandling of null userinfo

If the authentication is disabled, the userinfo endpoint returns null.
This case needs to be handled.

 #1780

* Small fix for handling refesh tokens in auth
2020-08-26 13:01:50 -07:00
bmccandless
ab1b9368a0 fix pca call in reembeddings (#1793)
This had the wrong dim passed into n_comps,
and so failed when the number of genes was less than 50.
2020-08-25 17:21:39 -07:00
Madison Dunitz
65ea1b673f Dunitz 1685 hosted annotations (#1789)
* save tiledb array to s3, dont cache user annotations

* Add option to disable annotation filename prompt (#1787)

Co-authored-by: Madison Dunitz <dunitzm@gmail.com>

* set tiledb default context in cxg_adaptor

Co-authored-by: maniarathi <arathi.mani@chanzuckerberg.com>
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
2020-08-24 18:26:08 -05:00
maniarathi
5dfe0043c3 Serves static assets from each dataset root URL and switch the publicPath to be a relative path. (#1786) 2020-08-22 10:04:40 -07:00
maniarathi
bc150a8469 Fixing bugs in cxg conversion tool (#1782) 2020-08-22 09:53:59 -07:00
maniarathi
a5c9ffa880 When reading annotations from tiledb, check if the values are byte literals and if so, decode them. Also pin s3f3 to 0.4.2. (#1788) 2020-08-22 09:42:11 -07:00
bmccandless
924aaf9aef Allow user_annotations in the eb app (#1781) 2020-08-18 17:13:57 -07:00
bmccandless
950be4426d Handle the refresh token in oauth authentication (#1766)
* Handle the refresh token in oauth authentication

If the token has expired, then it can be refreshed to get a new token.
This is automatically handled by the server without the client being aware.

Also in the PR:
  - refactor the auth_oauth.py file to more simply handle the save/restore of the token,
    and the refresh token
  - added an end2end test for oauth, which also tests refresh.

* adding python-jose and Authlib to requirements-dev.txt

They are needed in the auth_oauth test
2020-08-18 14:41:15 -07:00
maniarathi
053f39d49e Cleaning up one script that makes use of the non-existent cxgtool. (#1765) 2020-08-17 18:40:26 -07:00
maniarathi
994c20c094 Move cxgtool into CLI and modularize conversion functions (#1701) 2020-08-17 17:28:29 -07:00
bmccandless
298924fef5 Separate userinfo from the config endpoint (#1728)
* Separate userinfo from the config endpoint

previously information about if the user was logged in and their username
was part of the config endpoint.
However, the config endpoint was previously static, and has a cache control.
Rather than not caching the config, a new endpoint called "userinfo"
is created to handle that information.

The config endpoint still has the non-changing part of the authentication:

  config:
    authentication:
        requires_client_login:  True/False
        login: <uri to login endoint if requires_client_login is True>
        logout: <uri to logout endoint if requires_client_login is True>

The userinfo endpoint returns this information:

  userinfo:
    is_authenticated:  True/False
    username: <string if is_authenticated>

if authentication is not enabled then the config does not have an authentication key,
and userinfo returns None.

Also in the PR are a few minor code improvements and bug fixes

Co-authored-by: Colin Megill <colinmegill@gmail.com>
2020-08-17 13:41:03 -07:00
maniarathi
508889f74b Refactoring cxg utility classes in preparation for CXG conversion tooling (#1739) 2020-08-14 16:51:13 -07:00
Madison Dunitz
b034055c35 update to get_secrets_key (#1755)
* raise exception when get_secrets fails, get db_uri and set as a default_dataset_config var

* log as info not an error
2020-08-14 18:17:21 -05:00
maniarathi
263e893b30 Revert "Patching (#1744)" (#1748)
This reverts commit 6848f7a8b2.
2020-08-14 11:22:39 -07:00
Madison Dunitz
6a82030558 remove db_uri secret (#1751)
* remove db_uri secret

* add test to catch bug in future
2020-08-14 12:38:46 -05:00
bmccandless
905308e09f Move psycopg2==2.7.7 from requirements.txt to requirements-dev.txt (#1747) 2020-08-13 21:20:13 -07:00
bmccandless
3c04529523 Fix error message when datapath and dataroot are not provided (#1746)
* Fix error message when datapath and dataroot are not provided

Previously:
$ cellxgene launch
cellxgene] Starting the CLI...
AttributeError: 'NoneType' object has no attribute 'startswith'

With this fix:
$ cellxgene launch
[cellxgene] Starting the CLI...
Error: missing datapath

* lint
2020-08-13 21:10:02 -07:00
Madison Dunitz
2689d8d2c0 Create hosted user annotations [1685] (#1726)
* add function to retrieve latest annotation from db, db updates

* read and write tiledb arrays

* adding tests
2020-08-13 19:07:17 -05:00
Severiano Badajoz
1c4bb84f35 Properly generate hash and provide how-to (#1745)
* properly generate hash and provide how-to

* Add link to this PR
2020-08-13 16:50:54 -07:00
maniarathi
6848f7a8b2 Patching (#1744) 2020-08-13 14:36:42 -07:00
Severiano Badajoz
a23aaa131d regenerate hash and fix url (#1742)
The script hash had a typo in it and was incorrectly generated.  The URL in the `img-src` directive also did not need to be encased in single-quotes.

Reviewers please double-check my hash generation against the inline-script here: https://github.com/chanzuckerberg/cellxgene/blob/main/client/configuration/webpack/obsoleteHTMLTemplate.html
2020-08-13 11:17:13 -07:00
Severiano Badajoz
53a268fb71 Surround script hash in single quotes (#1735)
* add disclaimer about changing the script

* add hash for obsolete browser script

* add explicit domain img-src

* add single quotes

* add quotes to `data:`

* change order and remove single quotes

* lint, remove extra slash, and make hash an array

* add links to S3

* surround script has in single quotes
2020-08-12 11:31:12 -07:00
maniarathi
44c738e2f4 Bumping version of cellxgene for desktop release (#1733) 2020-08-12 09:58:52 -07:00
Severiano Badajoz
f221856ae1 add CSP sources for obsolete browser prompt (#1731)
Adds script hash and explicit domain to `img-src` directive
2020-08-11 16:50:46 -07:00
Madison Dunitz
80f6137528 retrieve latest annotation from db (#1723)
* add function to retrieve latest annotation from db, db updates

* dont create directory in s3
2020-08-11 15:48:12 -05:00
bmccandless
4291f6005d fix logout for auth0 (#1719)
* fix logout for auth0

the redirect from logout needs to be registered, and therefore cannot
be an arbitrary path in the server.  When the user logs out, they are
redirected to the index page

* no need to provide dataset to logout url
2020-08-06 11:52:19 -07:00
Severiano Badajoz
8d96477fae Remove hash source from CSP style-src directive (#1717)
* remove style csp hash generation + lint

* remove references to style_hashes
2020-08-05 16:43:35 -07:00
bmccandless
b5e5ee0168 Update hosted app to get the oauth client secret from the secret manager (#1713)
* Update the hosted app to get the oauth client secret from the secret manager

* fix to eb app, and set no cache on oauth endpoints
2020-08-05 12:00:35 -07:00
maniarathi
0d94c9e092 DRY-ing flatbuffer code (#1716) 2020-08-05 11:56:34 -07:00
maniarathi
cdae4f9f10 Reorganize the server testing directory (#1705) 2020-08-05 08:31:02 -07:00
Madison Dunitz
f632a8db91 Dunitz/db setup (#1619)
* initial database setup
2020-08-03 17:54:06 -05:00
bmccandless
ce13a9c7ca oauth support, add the token in a configuration specified cookie (#1702)
* oauth support, add the token in a configuration specified cookie

Previously, the id token was stored in the session token.
Now, it can be placed in a different cookie with different properties.
2020-08-03 10:45:21 -07:00
bmccandless
2afa48cf11 add oauth authentication (#1681)
* add oauth authentication

Add support for OAuth2.

Change the interface to AuthTypeBase
  - better handling of config parameters
  - add a complete_setup function for additional setup steps

Added a function wrapper to enforce authentication for the
routes that require authenticaiton.

* change fsspec requirement

fsspec 0.8.0 breaks our tests
it imports a module that is does not require.
2020-07-31 18:16:57 -07:00
Bruce Martin
f2fbeff511 add support for corpora default_embedding field (#1696)
* fix mispelling

* re-implement re-embedding

* always load base embedding to fetch counts

* format

* lint

* fix tests

* lint

* fix accept handling

* test log

* more debug

* more

* more

* more

* more

* remove logging

* logging

* jsonify

* remove debugging logs

* lint

* clean up errors a bit

* fix issue found in PR review

* add support for corpora default_embedding

* fix botched merge

* PR review

* PR review
2020-07-31 07:36:48 -07:00
Bruce Martin
75cb513dd9 re-implement re-embeddings (#1679)
* fix mispelling

* re-implement re-embedding

* always load base embedding to fetch counts

* format

* lint

* fix tests

* lint

* fix accept handling

* test log

* more debug

* more

* more

* more

* more

* remove logging

* logging

* jsonify

* remove debugging logs

* lint

* clean up errors a bit

* fix issue found in PR review

* PR review changes
2020-07-30 12:31:36 -07:00
bmccandless
bd147abb3f Fix eb logging. (#1692)
It now logs the requests to the file

Fixes #1611
2020-07-29 16:03:47 -07:00
bmccandless
5633d7c761 Fix server exception classes (#1683)
str(e) and e.message will both show the error message.
refactored the error.py file to simplify our exception class definitions
2020-07-29 13:05:59 -07:00
Bruce Martin
59f989d26f initial support for corpora schema conventions (#1676)
* initial support for corpora schema conventions

* remove debugging print

* add corpora util module

* tests

* lint

* PR review edits

* PR changes

* more PR changes

* more PR chnages

* PR fixes

* formatting

* PR updates

* lint

* PR review
2020-07-28 17:32:27 -07:00
bmccandless
5285556415 Add basic authentication in the server (#1670)
* Add basic authentication in the server

A pattern for creating authentication methods is introduced, with three
authentication types defined:
  none - no authentication
  session - like the current session based auth used for user annotations
  test - used to test the login/logout process end to end

The config endpoint now returns informations about the authentication, like if
the user is authenticated and their username.  The redirect uri's for login and
logout are also returned if the authentication type requires login

This is the first a several PRs for authentication.

*. Update server tests to avoid hardcoded ports

test_api and test_nan_rest now use a common function for starting a test server,
than will initially choose a random port.
2020-07-28 13:28:30 -07:00
bmccandless
0344cfacce Fix the app config for reembedding and add a test (#1664)
* Fix the app config for reembedding and add a test

* Add scanpy to requirements-dev
2020-07-23 12:27:53 -07:00
bmccandless
2cb3ae7046 Fix error handling in plugin module (#1645) 2020-07-17 08:14:10 -07:00