hosted cellxgene, add an environment variable to specify the config file (#1288)

Fixes #1272

server/eb/README.md

@@ -38,34 +38,57 @@ There are many more options to these commands that may be important or necessary
     eb init -p python-3.6 $EB_APP
     ```
 
-3. Create the artifact.zip file for the application
+3. Configuring cellxgene
+
+   All the cellxgene configuration options can be set from a configuration file.
+   This file can be generated like this:
+   
+   ```cellxgene launch --dump-default-config > myconfig.yaml```
+   
+   The config file may then be customized before the app is deployed.
+   
+   If your config file is named "config.yaml" and exists in this directory, then it will be bundled with the 
+   application zip file and installed along side the app on the EB servers.
+   
+   However, a potentially more flexible approach is to place your config file in a location accessible to the EB 
+   servers, such as along side the matrix files in S3.  For example:  s3://my-bucket/my-datasets/config.yaml.
+   Set the CXG_CONFIG_FILE environment variable to specify this location.  
+   
+   Another option is to set the CXG_DATAROOT environment variable.  The dataroot 
+   is the location where the matrix files are located. 
+   This environment variable will override the dataroot in the config file (if specified).
+   
+   - Note:  Certain features, such as diffexp and user annotations, are automatically disabled by the EB app,
+   and cannot be enabled using configuration.  They may be enabled manually by modifying app.py, however
+   this is not supported or recommended at this time.
+   
+4. Create the artifact.zip file for the application
 
-    If you have additional configuration for the application,
-    place that application in server/eb/config.yaml.  This will
-    be included in the eb deployment.
-    
    ```
    make build
    ```
    
-4. Create an environment
+5. Create an environment
 
     ```
     # name of the environment
     EB_ENV=cellxgene-env
     # type of ec2 instance to run the cellxgene server.     
     EB_INSTANCE=m5.large 
+    # One or both of the following environment variables needs to be set
     CXG_DATAROOT=<location to your S3 bucket>
+    CXG_CONFIG_FILE=<location to your config file>
    
-    eb create $EB_ENV --instance-type $EB_INSTANCE --envvars CXG_DATAROOT=$CXG_DATAROOT
+    eb create $EB_ENV --instance-type $EB_INSTANCE \
+       --envvars CXG_DATAROOT=$CXG_DATAROOT,CXG_CONFIG_FILE=$CXG_CONFIG_FILE
     ```
 
-5. Give the elastic beanstalk environment access to the S3 bucket.
+6. Give the elastic beanstalk environment access to the S3 bucket.
 
     This link may provide some useful information:
     https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-s3-bucket-instance/
     
-6. Deploy the application
+7. Deploy the application
 
     ```
     eb deploy $EB_ENV 

server/eb/app.py

@@ -24,6 +24,7 @@ try:
     from server.common.app_config import AppConfig
     from server.app.app import Server
     from server.data_common.matrix_loader import MatrixDataCacheManager
+    from server.common.data_locator import DataLocator
 except Exception:
     logging.critical("Exception importing server modules", exc_info=True)
     sys.exit(1)
@@ -39,13 +40,28 @@ class WSGIServer(Server):
 
 
 try:
-    dataroot = os.getenv("CXG_DATAROOT")
     app_config = AppConfig()
 
-    config_file = "config.yaml"
+    dataroot = os.getenv("CXG_DATAROOT")
-    if os.path.exists(config_file):
+    config_file = os.getenv("CXG_CONFIG_FILE")
-        logging.info(f"Configuration from {config_file}")
+
-        app_config.update_from_config_file(config_file)
+    if config_file:
+        config_location = DataLocator(config_file)
+        if config_location.exists():
+            with config_location.local_handle() as lh:
+                logging.info(f"Configuration from {config_file}")
+                app_config.update_from_config_file(lh)
+        else:
+            logging.critical(f"Configuration file not found {config_file}")
+            sys.exit(1)
+    else:
+        # no config file specified, try "config.yaml" in the current working directory
+        config_file = "config.yaml"
+        config_location = DataLocator(config_file)
+        if config_location.exists():
+            with config_location.local_handle() as lh:
+                logging.info(f"Configuration from {config_file}")
+                app_config.update_from_config_file(lh)
 
     if dataroot:
         logging.info(f"Configuration from CXG_DATAROOT")