Opened 4 months ago

Closed 2 months ago

Last modified 2 months ago

#2150 closed defect (fixed)

actinia startup error

Reported by: neteler Owned by: osgeolive@…
Priority: major Milestone: OSGeoLive13.0
Component: OSGeoLive Keywords: actinia
Cc: osgeolive@…

Description

I have prepared an installer for "actinia_core" (a new OSGeo community project, https://www.osgeo.org/projects/actinia/) which we will present at FOSS4G 2019, Bucharest.

Pull request of installer script: https://github.com/OSGeo/OSGeoLive/pull/266

Background:

Actinia is an open source REST API for scalable, distributed, high performance processing of geographical data that uses GRASS GIS for computational tasks. It provides a REST API to process satellite images, time series of satellite images, arbitrary raster data with geographical relations and vector data.

Change history (17)

comment:1 Changed 3 months ago by neteler

Initial docs (actinia overview and quickstart) added as well:

https://github.com/OSGeo/OSGeoLive-doc/pull/455

comment:2 Changed 3 months ago by kalxas

Actinia debian package now available in out ppa.

The installer needs a review.

comment:3 Changed 3 months ago by kalxas

Cc: osgeolive@… added
Priority: normalcritical

comment:4 Changed 3 months ago by kalxas

Actinia debian package needs more dependencies to work properly:

python3-google-api-core, python3-google-cloud-core, python3-google-auth, python3-google-cloud-bigquery, python3-google-cloud-storage, python3-rq

comment:5 Changed 3 months ago by kalxas

Priority: criticalmajor
Summary: New installer for actiniaactinia missing dependencies
Type: enhancementdefect

comment:6 Changed 3 months ago by neteler

In my view there is no need to include these dependencies since Google Cloud Processing requires an API key which isn't to be included in OSGeoLive

comment:7 Changed 3 months ago by kalxas

user@osgeolive:~$ actinia_start.sh 
[2019-08-01 17:59:42 +0000] [2933] [INFO] Starting gunicorn 19.7.1
[2019-08-01 17:59:42 +0000] [2933] [INFO] Listening at: http://0.0.0.0:8088 (2933)
[2019-08-01 17:59:42 +0000] [2933] [INFO] Using worker: sync
[2019-08-01 17:59:42 +0000] [2936] [INFO] Booting worker with pid: 2936
[2019-08-01 17:59:42 +0000] [2936] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 578, in spawn_worker
    worker.init_process()
  File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 135, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python3/dist-packages/gunicorn/util.py", line 377, in import_app
    __import__(module)
  File "/usr/lib/python3/dist-packages/actinia_core/main.py", line 30, in <module>
    from .endpoints import create_endpoints
  File "/usr/lib/python3/dist-packages/actinia_core/endpoints.py", line 33, in <module>
    from .resources.location_management import ListLocationsResource, LocationManagementResourceUser
  File "/usr/lib/python3/dist-packages/actinia_core/resources/location_management.py", line 39, in <module>
    from .common.response_models import ProcessingResponseModel
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/response_models.py", line 33, in <module>
    from .process_chain import GrassModule
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/process_chain.py", line 35, in <module>
    from .sentinel_processing_library import Sentinel2Processing
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/sentinel_processing_library.py", line 30, in <module>
    from .google_satellite_bigquery_interface import GoogleSatelliteBigQueryInterface
  File "/usr/lib/python3/dist-packages/actinia_core/resources/common/google_satellite_bigquery_interface.py", line 28, in <module>
    from google.cloud import bigquery
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery/__init__.py", line 35, in <module>
    from google.cloud.bigquery.client import Client
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery/client.py", line 52, in <module>
    from google.cloud.bigquery import _pandas_helpers
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery/_pandas_helpers.py", line 24, in <module>
    from google.cloud import bigquery_storage_v1beta1
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery_storage_v1beta1/__init__.py", line 25, in <module>
    from google.cloud.bigquery_storage_v1beta1 import types
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery_storage_v1beta1/types.py", line 23, in <module>
    from google.cloud.bigquery_storage_v1beta1.proto import arrow_pb2
  File "/usr/lib/python3/dist-packages/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2.py", line 26, in <module>
    '\n7google/cloud/bigquery/storage_v1beta1/proto/arrow.proto\x12%google.cloud.bigquery.storage.v1beta1"(\n\x0b\x41rrowSchema\x12\x19\n\x11serialized_schema\x18\x01 \x01(\x0c"F\n\x10\x41rrowRecordBatch\x12\x1f\n\x17serialized_record_batch\x18\x01 \x01(\x0c\x12\x11\n\trow_count\x18\x02 \x01(\x03\x42\x85\x01\n)com.google.cloud.bigquery.storage.v1beta1B\nArrowProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storageb\x06proto3'
TypeError: __new__() got an unexpected keyword argument 'serialized_options'
[2019-08-01 17:59:42 +0000] [2936] [INFO] Worker exiting (pid: 2936)
[2019-08-01 17:59:42 +0000] [2933] [INFO] Shutting down: Master
[2019-08-01 17:59:42 +0000] [2933] [INFO] Reason: Worker failed to boot.

comment:8 Changed 3 months ago by kalxas

Without the google dependencies, there are several import errors on startup. Is there a way to drop these google dependencies by configuration?

comment:9 Changed 3 months ago by neteler

The dependencies cannot be dropped without major refactoring (may come in future).

Probably Google Cloud API version differences are the reason:

https://github.com/mundialis/actinia_core/blob/master/requirements.txt

  • google-cloud==0.32.0
  • google-cloud-bigquery==0.28.0
  • google-cloud-storage==1.6.0

(known to work)

vs https://launchpad.net/~gcpp-kalxas/+archive/ubuntu/actinia

  • google-cloud 0.32.0-0~bionic1 (the same)
  • google-cloud-bigquery 1.17.0-0~bionic0 - different
  • google-cloud-bigquery-storage 0.6.0-0~bionic0
  • google-cloud-core 1.0.3-0~bionic1
  • google-cloud-storage 1.17.0-0~bionic0 - different

comment:10 Changed 3 months ago by kalxas

These are the requirements of google-cloud 0.32 according to the setup.py here:

https://files.pythonhosted.org/packages/95/86/e04e00948b3347498500d8a7413d1d8423d0068697719d2c17f9a5fdbcbb/google-cloud-0.32.0.tar.gz

REQUIREMENTS = [
    'google-api-core >= 0.1.2, < 0.2.0dev',
    'google-cloud-bigquery >= 0.28.0, < 0.29dev',
    'google-cloud-bigquery-datatransfer >= 0.1.0, < 0.2dev',
    'google-cloud-bigtable >= 0.28.1, < 0.29dev',
    'google-cloud-container >= 0.1.0, < 0.2dev',
    'google-cloud-core >= 0.28.0, < 0.29dev',
    'google-cloud-datastore >= 1.4.0, < 1.5dev',
    'google-cloud-dns >= 0.28.0, < 0.29dev',
    'google-cloud-error-reporting >= 0.28.0, < 0.29dev',
    'google-cloud-firestore >= 0.28.0, < 0.29dev',
    'google-cloud-language >= 1.0.0, < 1.1dev',
    'google-cloud-logging >= 1.4.0, < 1.5dev',
    'google-cloud-monitoring >= 0.28.0, < 0.29dev',
    'google-cloud-pubsub >= 0.30.0, < 0.31dev',
    'google-cloud-resource-manager >= 0.28.0, < 0.29dev',
    'google-cloud-runtimeconfig >= 0.28.0, < 0.29dev',
    'google-cloud-spanner >= 0.29.0, < 0.30dev',
    'google-cloud-speech >= 0.30.0, < 0.31dev',
    'google-cloud-storage >= 1.6.0, < 1.7dev',
    'google-cloud-trace >= 0.17.0, < 0.18dev',
    'google-cloud-translate >= 1.3.0, < 1.4dev',
    'google-cloud-videointelligence >= 1.0.0, < 1.1dev',
    'google-cloud-vision >= 0.29.0, < 0.30dev',
]

comment:11 Changed 3 months ago by kalxas

Fixed versions in https://launchpad.net/~gcpp-kalxas/+archive/ubuntu/osgeolive/+packages

Lets see if that will do it.

comment:12 Changed 3 months ago by kalxas

user@osgeolive:/opt$ actinia_start.sh 
[2019-08-05 08:45:16 +0000] [7710] [INFO] Starting gunicorn 19.7.1
[2019-08-05 08:45:16 +0000] [7710] [INFO] Listening at: http://0.0.0.0:8088 (7710)
[2019-08-05 08:45:16 +0000] [7710] [INFO] Using worker: sync
[2019-08-05 08:45:16 +0000] [7713] [INFO] Booting worker with pid: 7713
from actinia_statistic_plugin.endpoints import create_endpoints as create_plugin_endpoints
create_plugin_endpoints(flask_api=flask_api)
    
ERROR: Could not connect to redis with localhost 6379 pass Client sent AUTH, but no password is set
ERROR: Could not connect to redis with localhost 6379 pass Client sent AUTH, but no password is set
ERROR: Could not connect to redis with localhost 6379 pass Client sent AUTH, but no password is set

comment:13 Changed 3 months ago by kalxas

Summary: actinia missing dependenciesactinia startup error

comment:14 Changed 3 months ago by neteler

An idea from (via huhabla):

https://stackoverflow.com/questions/26356435/rediscommanderror-err-client-sent-auth-but-no-password-is-set

Can you check if the REDIS_URL environment variable is set? If that's the case, the used Redis client may be picking the password from it.

comment:16 Changed 2 months ago by kalxas

Resolution: fixed
Status: newclosed

Fixed in build228

comment:17 in reply to:  16 Changed 2 months ago by neteler

Replying to kalxas:

Fixed in build228

Thank you for your amazing work!!

Note: See TracTickets for help on using tickets.