actinia startup error

[neteler]

I have prepared an installer for "actinia_core" (a new OSGeo community project, ​https://www.osgeo.org/projects/actinia/) which we will present at FOSS4G 2019, Bucharest.

Pull request of installer script: ​https://github.com/OSGeo/OSGeoLive/pull/266

Background:

Actinia is an open source REST API for scalable, distributed, high performance processing of geographical data that uses GRASS GIS for computational tasks. It provides a REST API to process satellite images, time series of satellite images, arbitrary raster data with geographical relations and vector data.

[neteler]

Initial docs (actinia overview and quickstart) added as well:

​https://github.com/OSGeo/OSGeoLive-doc/pull/455

[kalxas]

Actinia debian package now available in out ppa.

The installer needs a review.

[kalxas]

Actinia debian package needs more dependencies to work properly:

python3-google-api-core, python3-google-cloud-core, python3-google-auth, python3-google-cloud-bigquery, python3-google-cloud-storage, python3-rq

[neteler]

In my view there is no need to include these dependencies since Google Cloud Processing requires an API key which isn't to be included in OSGeoLive

[kalxas]

user@osgeolive:~$ actinia_start.sh 
[2019-08-01 17:59:42 +0000] [2933] [INFO] Starting gunicorn 19.7.1
[2019-08-01 17:59:42 +0000] [2933] [INFO] Listening at: http://0.0.0.0:8088 (2933)
[2019-08-01 17:59:42 +0000] [2933] [INFO] Using worker: sync
[2019-08-01 17:59:42 +0000] [2936] [INFO] Booting worker with pid: 2936
[2019-08-01 17:59:42 +0000] [2936] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 578, in spawn_worker
    worker.init_process()
  File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 135, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python3/dist-packages/gunicorn/util.py", line 377, in import_app
    __import__(module)
  File "/usr/lib/python3/dist-packages/actinia_core/main.py", line 30, in <module>
    from .endpoints import create_endpoints
  File "/usr/lib/python3/dist-packages/actinia_core/endpoints.py", line 33, in <module>
    from .resources.location_management import ListLocationsResource, LocationManagementResourceUser
  File "/usr/lib/python3/dist-packages/actinia_core/resources/location_management.py", line 39, in <module>
    from .common.response_models import ProcessingResponseModel
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/response_models.py", line 33, in <module>
    from .process_chain import GrassModule
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/process_chain.py", line 35, in <module>
    from .sentinel_processing_library import Sentinel2Processing
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/sentinel_processing_library.py", line 30, in <module>
    from .google_satellite_bigquery_interface import GoogleSatelliteBigQueryInterface
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/google_satellite_bigquery_interface.py", line 28, in <module>
    from google.cloud import bigquery
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery/__init__.py", line 35, in <module>
    from google.cloud.bigquery.client import Client
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery/client.py", line 52, in <module>
    from google.cloud.bigquery import _pandas_helpers
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery/_pandas_helpers.py", line 24, in <module>
    from google.cloud import bigquery_storage_v1beta1
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery_storage_v1beta1/__init__.py", line 25, in <module>
    from google.cloud.bigquery_storage_v1beta1 import types
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery_storage_v1beta1/types.py", line 23, in <module>
    from google.cloud.bigquery_storage_v1beta1.proto import arrow_pb2
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2.py", line 26, in <module>
    '\n7google/cloud/bigquery/storage_v1beta1/proto/arrow.proto\x12%google.cloud.bigquery.storage.v1beta1"(\n\x0b\x41rrowSchema\x12\x19\n\x11serialized_schema\x18\x01 \x01(\x0c"F\n\x10\x41rrowRecordBatch\x12\x1f\n\x17serialized_record_batch\x18\x01 \x01(\x0c\x12\x11\n\trow_count\x18\x02 \x01(\x03\x42\x85\x01\n)com.google.cloud.bigquery.storage.v1beta1B\nArrowProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storageb\x06proto3'
TypeError: __new__() got an unexpected keyword argument 'serialized_options'
[2019-08-01 17:59:42 +0000] [2936] [INFO] Worker exiting (pid: 2936)
[2019-08-01 17:59:42 +0000] [2933] [INFO] Shutting down: Master
[2019-08-01 17:59:42 +0000] [2933] [INFO] Reason: Worker failed to boot.

[kalxas]

Without the google dependencies, there are several import errors on startup. Is there a way to drop these google dependencies by configuration?

[neteler]

The dependencies cannot be dropped without major refactoring (may come in future).

Probably Google Cloud API version differences are the reason:

​https://github.com/mundialis/actinia_core/blob/master/requirements.txt

• google-cloud==0.32.0
• google-cloud-bigquery==0.28.0
• google-cloud-storage==1.6.0

(known to work)

vs ​https://launchpad.net/~gcpp-kalxas/+archive/ubuntu/actinia

• google-cloud 0.32.0-0~bionic1 (the same)
• google-cloud-bigquery 1.17.0-0~bionic0 - different
• google-cloud-bigquery-storage 0.6.0-0~bionic0
• google-cloud-core 1.0.3-0~bionic1
• google-cloud-storage 1.17.0-0~bionic0 - different

[kalxas]

These are the requirements of google-cloud 0.32 according to the setup.py here:

​https://files.pythonhosted.org/packages/95/86/e04e00948b3347498500d8a7413d1d8423d0068697719d2c17f9a5fdbcbb/google-cloud-0.32.0.tar.gz

REQUIREMENTS = [
    'google-api-core >= 0.1.2, < 0.2.0dev',
    'google-cloud-bigquery >= 0.28.0, < 0.29dev',
    'google-cloud-bigquery-datatransfer >= 0.1.0, < 0.2dev',
    'google-cloud-bigtable >= 0.28.1, < 0.29dev',
    'google-cloud-container >= 0.1.0, < 0.2dev',
    'google-cloud-core >= 0.28.0, < 0.29dev',
    'google-cloud-datastore >= 1.4.0, < 1.5dev',
    'google-cloud-dns >= 0.28.0, < 0.29dev',
    'google-cloud-error-reporting >= 0.28.0, < 0.29dev',
    'google-cloud-firestore >= 0.28.0, < 0.29dev',
    'google-cloud-language >= 1.0.0, < 1.1dev',
    'google-cloud-logging >= 1.4.0, < 1.5dev',
    'google-cloud-monitoring >= 0.28.0, < 0.29dev',
    'google-cloud-pubsub >= 0.30.0, < 0.31dev',
    'google-cloud-resource-manager >= 0.28.0, < 0.29dev',
    'google-cloud-runtimeconfig >= 0.28.0, < 0.29dev',
    'google-cloud-spanner >= 0.29.0, < 0.30dev',
    'google-cloud-speech >= 0.30.0, < 0.31dev',
    'google-cloud-storage >= 1.6.0, < 1.7dev',
    'google-cloud-trace >= 0.17.0, < 0.18dev',
    'google-cloud-translate >= 1.3.0, < 1.4dev',
    'google-cloud-videointelligence >= 1.0.0, < 1.1dev',
    'google-cloud-vision >= 0.29.0, < 0.30dev',
]

[kalxas]

Fixed versions in ​https://launchpad.net/~gcpp-kalxas/+archive/ubuntu/osgeolive/+packages

Lets see if that will do it.

[kalxas]

user@osgeolive:/opt$ actinia_start.sh 
[2019-08-05 08:45:16 +0000] [7710] [INFO] Starting gunicorn 19.7.1
[2019-08-05 08:45:16 +0000] [7710] [INFO] Listening at: http://0.0.0.0:8088 (7710)
[2019-08-05 08:45:16 +0000] [7710] [INFO] Using worker: sync
[2019-08-05 08:45:16 +0000] [7713] [INFO] Booting worker with pid: 7713
from actinia_statistic_plugin.endpoints import create_endpoints as create_plugin_endpoints
create_plugin_endpoints(flask_api=flask_api)
    
ERROR: Could not connect to redis with localhost 6379 pass Client sent AUTH, but no password is set
ERROR: Could not connect to redis with localhost 6379 pass Client sent AUTH, but no password is set
ERROR: Could not connect to redis with localhost 6379 pass Client sent AUTH, but no password is set

[neteler]

An idea from (via huhabla):

​https://stackoverflow.com/questions/26356435/rediscommanderror-err-client-sent-auth-but-no-password-is-set

Can you check if the REDIS_URL environment variable is set? If that's the case, the used Redis client may be picking the password from it.

[kalxas]

Fixed in ​https://github.com/OSGeo/OSGeoLive/commit/91f338f2ccba59975e9fd5aebf860602beb964d2

[kalxas]

Fixed in build228

[neteler]

Replying to kalxas:

Fixed in build228

Thank you for your amazing work!!