Commit Graph

326 Commits

Author SHA1 Message Date
Severiano Badajoz
a23aaa131d regenerate hash and fix url (#1742)
The script hash had a typo in it and was incorrectly generated.  The URL in the `img-src` directive also did not need to be encased in single-quotes.

Reviewers please double-check my hash generation against the inline-script here: https://github.com/chanzuckerberg/cellxgene/blob/main/client/configuration/webpack/obsoleteHTMLTemplate.html
2020-08-13 11:17:13 -07:00
Severiano Badajoz
53a268fb71 Surround script hash in single quotes (#1735)
* add disclaimer about changing the script

* add hash for obsolete browser script

* add explicit domain img-src

* add single quotes

* add quotes to `data:`

* change order and remove single quotes

* lint, remove extra slash, and make hash an array

* add links to S3

* surround script has in single quotes
2020-08-12 11:31:12 -07:00
maniarathi
44c738e2f4 Bumping version of cellxgene for desktop release (#1733) 2020-08-12 09:58:52 -07:00
Severiano Badajoz
f221856ae1 add CSP sources for obsolete browser prompt (#1731)
Adds script hash and explicit domain to `img-src` directive
2020-08-11 16:50:46 -07:00
Madison Dunitz
80f6137528 retrieve latest annotation from db (#1723)
* add function to retrieve latest annotation from db, db updates

* dont create directory in s3
2020-08-11 15:48:12 -05:00
bmccandless
4291f6005d fix logout for auth0 (#1719)
* fix logout for auth0

the redirect from logout needs to be registered, and therefore cannot
be an arbitrary path in the server.  When the user logs out, they are
redirected to the index page

* no need to provide dataset to logout url
2020-08-06 11:52:19 -07:00
Severiano Badajoz
8d96477fae Remove hash source from CSP style-src directive (#1717)
* remove style csp hash generation + lint

* remove references to style_hashes
2020-08-05 16:43:35 -07:00
bmccandless
b5e5ee0168 Update hosted app to get the oauth client secret from the secret manager (#1713)
* Update the hosted app to get the oauth client secret from the secret manager

* fix to eb app, and set no cache on oauth endpoints
2020-08-05 12:00:35 -07:00
maniarathi
0d94c9e092 DRY-ing flatbuffer code (#1716) 2020-08-05 11:56:34 -07:00
maniarathi
cdae4f9f10 Reorganize the server testing directory (#1705) 2020-08-05 08:31:02 -07:00
Madison Dunitz
f632a8db91 Dunitz/db setup (#1619)
* initial database setup
2020-08-03 17:54:06 -05:00
bmccandless
ce13a9c7ca oauth support, add the token in a configuration specified cookie (#1702)
* oauth support, add the token in a configuration specified cookie

Previously, the id token was stored in the session token.
Now, it can be placed in a different cookie with different properties.
2020-08-03 10:45:21 -07:00
bmccandless
2afa48cf11 add oauth authentication (#1681)
* add oauth authentication

Add support for OAuth2.

Change the interface to AuthTypeBase
  - better handling of config parameters
  - add a complete_setup function for additional setup steps

Added a function wrapper to enforce authentication for the
routes that require authenticaiton.

* change fsspec requirement

fsspec 0.8.0 breaks our tests
it imports a module that is does not require.
2020-07-31 18:16:57 -07:00
Bruce Martin
f2fbeff511 add support for corpora default_embedding field (#1696)
* fix mispelling

* re-implement re-embedding

* always load base embedding to fetch counts

* format

* lint

* fix tests

* lint

* fix accept handling

* test log

* more debug

* more

* more

* more

* more

* remove logging

* logging

* jsonify

* remove debugging logs

* lint

* clean up errors a bit

* fix issue found in PR review

* add support for corpora default_embedding

* fix botched merge

* PR review

* PR review
2020-07-31 07:36:48 -07:00
Bruce Martin
75cb513dd9 re-implement re-embeddings (#1679)
* fix mispelling

* re-implement re-embedding

* always load base embedding to fetch counts

* format

* lint

* fix tests

* lint

* fix accept handling

* test log

* more debug

* more

* more

* more

* more

* remove logging

* logging

* jsonify

* remove debugging logs

* lint

* clean up errors a bit

* fix issue found in PR review

* PR review changes
2020-07-30 12:31:36 -07:00
bmccandless
bd147abb3f Fix eb logging. (#1692)
It now logs the requests to the file

Fixes #1611
2020-07-29 16:03:47 -07:00
bmccandless
5633d7c761 Fix server exception classes (#1683)
str(e) and e.message will both show the error message.
refactored the error.py file to simplify our exception class definitions
2020-07-29 13:05:59 -07:00
Bruce Martin
59f989d26f initial support for corpora schema conventions (#1676)
* initial support for corpora schema conventions

* remove debugging print

* add corpora util module

* tests

* lint

* PR review edits

* PR changes

* more PR changes

* more PR chnages

* PR fixes

* formatting

* PR updates

* lint

* PR review
2020-07-28 17:32:27 -07:00
bmccandless
5285556415 Add basic authentication in the server (#1670)
* Add basic authentication in the server

A pattern for creating authentication methods is introduced, with three
authentication types defined:
  none - no authentication
  session - like the current session based auth used for user annotations
  test - used to test the login/logout process end to end

The config endpoint now returns informations about the authentication, like if
the user is authenticated and their username.  The redirect uri's for login and
logout are also returned if the authentication type requires login

This is the first a several PRs for authentication.

*. Update server tests to avoid hardcoded ports

test_api and test_nan_rest now use a common function for starting a test server,
than will initially choose a random port.
2020-07-28 13:28:30 -07:00
bmccandless
0344cfacce Fix the app config for reembedding and add a test (#1664)
* Fix the app config for reembedding and add a test

* Add scanpy to requirements-dev
2020-07-23 12:27:53 -07:00
bmccandless
2cb3ae7046 Fix error handling in plugin module (#1645) 2020-07-17 08:14:10 -07:00
bmccandless
f223d7504e Small fix for handling display versions (#1625)
* Small fix for handling display versions

Making a distinction between __version__ and the version we display in the info panel (displayr_version).
The hosted cellxgene can overwrite the display_version using a plugin.

Improve version handling in the customized assets
2020-07-16 15:11:37 -07:00
Bruce Martin
2f700b377f permit NaN in embedding coordinates (#1631)
* permit NaN in embedding coordinates

* lint
2020-07-16 11:43:56 -07:00
bmccandless
3c6d90a4db Better error checking on bad params (#1626) 2020-07-16 11:08:27 -07:00
bmccandless
f69d141336 refactor config to support different config options for datasets in different dataroots. (#1596)
This will give us the ability to specify different config options for
different dataroots.

the key of the dataroot dictionary is no longer the same as the dataroot_url.
Previously key==dataroot_url, and now those are separated.

Added an "is_multi_dataset" function to simplify logic where it branched on single vs multi.

Simplified the rest.py interface by no longer passing in the user annotations object, since
that can be retrieved from the dataset.
2020-07-10 16:21:40 -07:00
bmccandless
f3d5081d38 exclude a tiledb version that has a known bug (#1604)
exclude a tiledb version that has a known bug
2020-07-07 08:39:50 -07:00
Matt Weiden
e5670d1732 Do not cache index.html (#1603)
* Fix typo

* Run black for code formatting

* Do not cache index.html
2020-07-01 14:35:44 -07:00
bmccandless
5015c5ca3f Return float32 for embedding data type (#1597) 2020-06-30 14:16:55 -07:00
Matt Weiden
152c07a28e Change default branch from 'master' to 'main' (#1589)
The default branch has been changed. This PR cleans up references to
`master` and fixes links that would otherwise be broken.

For more background see the following references:
* https://www.independent.co.uk/life-style/gadgets-and-tech/news/github-master-slave-slavery-whitelist-language-inclusive-a9568576.html
* https://tools.ietf.org/id/draft-knodel-terminology-00.html
2020-06-25 14:04:30 -07:00
Timmy Huang
83376627e8 1510-smoke-test (#1548)
* 1510-smoke-test

* config default

* update tests

* update test config

* fix linter errors

* more comments

* address comments

* use npm install in push_tests.yml

* use environment.default.json

* adding docs

* Take care of @mweiden's nits

* Save screenshots in the __tests__/screenshots/ directory

* typo

* docs

* Add chart tests (#1580)

* merge tests

* check if bin creation returned null before rendering charts (#1576)

* check if bin creation returned null before rendering charts

* refactor chart rendering into functions (#1577)

* little fixes from PR

* reintroduce fix to check for null values

* change getAllByClass to return element

* slice instead

* new stackedbar test

* feedback-1573-test (#1579)

* feedback-1573-test

* enable whole test set

* revert tests

Co-authored-by: Timmy Huang <tihuan@users.noreply.github.com>

* tweak test to actually render chart

* include snapshot

* remove async

* fix getAllHistograms

* properly grab id

Co-authored-by: Timmy Huang <tihuan@users.noreply.github.com>

Co-authored-by: Matt Weiden <538456+mweiden@users.noreply.github.com>
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
2020-06-24 11:45:39 -07:00
bmccandless
e22e671f10 Return 404 when the dataset does not exist (#1581)
Also in this PR, restructure the exception handling.

fixes #1566
2020-06-23 17:14:42 -07:00
bmccandless
99d004d1f0 add config option to handle multiple dataroots (#1531)
#1513
2020-06-04 19:29:37 -07:00
Matt Weiden
101f446f8d Add .gitignore file added by elastic beanstalk (#1506) 2020-06-03 09:14:24 -07:00
bmccandless
76523d4f32 sparse column shift encoding. (#1502)
Many of our matrices are log normalized, which tends to eliminate
the number of non zero values (if there were any).  This prevents
the matrix from being stored as a sparse matrix.  The solution here
is to use a simple transformation to make it sparse again.  The most
common value from each column is subtracted from that column.  These
values that were subtracted are saved in an array called X_col_shift.

The cellxgene code needs to understand how to undo the transformation when
operating over the X matrix.

- added script to create a synthetic dataset for testing
- added a script to convert an existing CXG dataset to a sparse CXG dataset
2020-06-02 08:23:52 -07:00
bmccandless
d0577b94af Return an empty matrix if no rows or columns are selected (#1501)
Return an empty matrix if no rows or columns are selected

Fixes #1499
2020-05-29 14:47:46 -07:00
bmccandless
f7585eef1e Support for sparse tiledb arrays for the X matrix (#1496)
Support for sparse tiledb arrays for the X matrix

1. cxgtool can now output sparse matrices
2. cxg_adaptor and diffexp_cxg updated to handle sparse matrices
3. added a test in test_diffexp to test sparse diffexp and get_X_array
2020-05-28 18:36:02 -07:00
bmccandless
030eea1898 Update run_diffexp.py (#1494)
You can now select labels within groups instead of random rows.
2020-05-26 16:06:33 -07:00
Matt Weiden
51758a0be9 Remove the trusted types requirement (#1495)
There seem to be breaking changes in Chrome that are causing this to
fail. We've weighed the risk of disabling the feature with the issues
we've encountered using it and decided to disable it for now.
2020-05-26 12:31:56 -07:00
bmccandless
4d100d4507 Upgrade to tiledb 2.0 (#1485)
* Upgrade to tiledb 2.0
2020-05-21 14:55:11 -07:00
Matt Weiden
e21997799c Upgrade python requirements to click>=7.1.2 (#1472)
6.7 does not have the `hidden` flag used in the code. Users building the
app with an older version of click within the current range specified by
requirements.txt may fail.
2020-05-14 10:04:28 -07:00
Matt Weiden
730410c5e1 Autoformat python to fix lint errors (#1470)
* Autoformat python to fix lint errors

* Fix lint errors not caught by black
2020-05-12 13:19:38 -07:00
Bruce Martin
c1bf491a96 Hooks for sentry integration (#1460)
* add sentry webpack plugin

* allow override of webpack config

* work around cheerio inability to parse jinga templates

* webpack can not minify jinja templates

* allow script injection to specify other attributes

* allow script injection to specify other attributes

* Adjustments to make plugin systems work

* Add sourcemaps for javascript in prod webpack

* Update .gitignore

* Fix spelling errors

Co-authored-by: Matt Weiden <538456+mweiden@users.noreply.github.com>
2020-05-08 15:59:56 -07:00
bmccandless
f49507f18a app_config, fix bug with list/tuple command line arguments. (#1464)
* app_config, fix bug with list/tuple command line arguments.

There was a error caused by pyyaml using lists, and click using tuples.
Now tuples are automatically converted to lists when the config is
updated.
2020-05-06 12:00:02 -07:00
Matt Weiden
5947306ca0 Add server plugin system (#1447)
* Add server plugin system

Plugins are optional modules loaded at runtime. Specification:
* Plugins are loaded from the server.plugins module (directory
  server/plugins)
* The import_plugins method is run as part of the initialization of the
  server module in __init__.py

* Add plugins to the EB build process

* Remove bit of dead code

* Respond to feedback from @bmccandless
2020-05-05 17:05:42 -07:00
bmccandless
55a7967001 Add a process for specifying exact requirements for an EB deployment (#1451)
Add a process for keeping specifying exact requirements for an EB deployment
2020-05-05 10:51:02 -07:00
bmccandless
61ca75e846 hosted, update order to look for config file. (#1452)
* hosted, update order to look for config file.

The app now uses a local config.yaml file bundled with the artifact
(if present), if it exists, then looks in the CXG_CONFIG_FILE
environment variable.  This is the reverse of previous behavior.
The purpose of this change is to move away from using the
config file on s3, since that could lead to problem where an older
version of the app uses a newer version of the config.

Also in this PR:
1. Changed documentation around dataroot, to describe the posibility of using lustre.
2. Added a few improvements around the secret manager region name.  If we use lustre for dataroot and a local config file, then we will no longer be able to
auto determine the region for the secret manager.  I plan to start using the
environment variable option for hosted cellxgene.

* small edit to README

Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
2020-05-05 10:50:28 -07:00
Matt Weiden
1ee76826c7 Pin tiledb version 0.5.9 (#1453)
There is a regression in 0.6.0
2020-05-05 10:03:57 -07:00
Bruce Martin
f42f5151a6 Refactor build and CSP headers for Safari compat (#1442)
* add unsafe-inline directive to style-src

* debugging - turn on csp reporting

* revert reporting only csp

* do not inline JS and CSS in build

* enable HTTPs only when in production mode

* remove debug printf

* fix clean target

* revert force_https removal
2020-05-04 12:47:35 -07:00
bmccandless
b255e32548 Add a more consistent way to customize the eb deployment (#1446)
* Add a more consistent way to customize the eb deployment

Plus a few minor bugfixes
2020-05-04 09:52:03 -07:00
Matt Weiden
a17fff83cf Return empty colors for .cxg v0.0 files (#1441)
* Return empty colors for .cxg v0.0 files

Fixes https://github.com/chanzuckerberg/cellxgene/issues/1440

The CxgAdaptor.get_colors method currently assumes that the .cxg file has
cxg_group_metadata. As a result, the /api/v0.2/colors endpoint always fails for
.cxg v0.0 files.

* Add test fixture
2020-04-30 17:21:20 -07:00