tracking: Generic Database connection rollout

[sbernauer]

We want a generic way of for database connections like mysql, postgres or redis

Prerequisite tasks

• ADR 29: Standardize database connections documentation#321
• https://github.com/stackabletech/decisions/issues/72
• feat: Add generic database connection mechanism operator-rs#1163
• Update ADR with the new - chosen implementation

Rollout

Note: I expect many of the following tools don't use a database, and thus are a NOOP

• feat: Add generic database connection mechanism airflow-operator#754
• feat: Implement generic database connection druid-operator#814 @maltesander
• HBase No database connection
• Hadoop HDFS No database connection
• Implement generic database connection hive-operator#677 / feat: Implement generic database connection hive-operator#674
• Kafka No database connection
• NiFi No database connection
• OpenSearch No database connection
• Spark No database connection
• feat: Add generic database connection mechanism superset-operator#722 @adwk67
• Trino (Does not really need it, we could offer typed postgres / mysql connectors, but currently covered via generic connector)
• ZooKeeper No database connection
• Commons No database connection
• Listener No database connection
• OPA No database connection
• Secret No database connection

[fhennig]

@razvan I'm not sure this can be closed, there are still open tasks

[razvan]

It was closed automatically by merging the ADR PR. I missed the comment.

[Maleware]

As long as this is not implemented you can workaround with Podovrerrites e.g. like this:

To avoid searching, this is the line added to args: sed -i "s|XX_PASS_XX|$(cat /tmp/postgres-password/password)|g" /stackable/config/hive-site.xml

---
apiVersion: hive.stackable.tech/v1alpha1
kind: HiveCluster
metadata:
  name: hive
  namespace: my-namespace
spec:
  image:
    productVersion: 3.1.3
    stackableVersion: "23.7.0"
  clusterConfig:
    database:
      connString: jdbc:postgresql://my-postgress-connector/hive
      user: hive
###### This is what will be taken from a secret ########
      password: XX_PASS_XX
      dbType: postgres
    s3:
      reference: ionos
  metastore:
    config:
      resources:
        memory:
          limit: '1Gi'
####### Podoverrides to mount the volume and sed the pw from the secret
    podOverrides:
      spec:
        containers:
          - name: hive
            args:
              - echo copying /stackable/mount/config to /stackable/config && cp -RL /stackable/mount/config/* /stackable/config && echo copying /stackable/mount/log-config/hive-log4j2.properties to /stackable/config/hive-log4j2.properties && cp -RL /stackable/mount/log-config/hive-log4j2.properties /stackable/config/hive-log4j2.properties && keytool -importkeystore -srckeystore /etc/pki/java/cacerts -srcstoretype jks -srcstorepass changeit -destkeystore /stackable/truststore.p12 -deststoretype pkcs12 -deststorepass changeit -noprompt && echo replacing xxx_access_key_xxx and xxx_secret_key_xxx with secret values. && sed -i "s|xxx_access_key_xxx|$(cat /stackable/secrets/accessKey)|g" /stackable/config/hive-site.xml && sed -i "s|xxx_secret_key_xxx|$(cat /stackable/secrets/secretKey)|g" /stackable/config/hive-site.xml && sed -i "s|XX_PASS_XX|$(cat /tmp/postgres-password/password)|g" /stackable/config/hive-site.xml && bin/start-metastore --config /stackable/config --db-type postgres --hive-bin-dir bin
            volumeMounts:
              - mountPath: /tmp/postgres-password/
                name: postgres-password
        volumes:
          - name: postgres-password
            secret:
              secretName: hive-postgres-secret
      default:
        replicas: 1

[ruslanguns]

• echo copying /stackable/mount/config to /stackable/config && cp -RL /stackable/mount/config/* /stackable/config && echo copying /stackable/mount/log-config/hive-log4j2.properties to /stackable/config/hive-log4j2.properties && cp -RL /stackable/mount/log-config/hive-log4j2.properties /stackable/config/hive-log4j2.properties && keytool -importkeystore -srckeystore /etc/pki/java/cacerts -srcstoretype jks -srcstorepass changeit -destkeystore /stackable/truststore.p12 -deststoretype pkcs12 -deststorepass changeit -noprompt && echo replacing xxx_access_key_xxx and xxx_secret_key_xxx with secret values. && sed -i "s|xxx_access_key_xxx|$(cat /stackable/secrets/accessKey)|g" /stackable/config/hive-site.xml && sed -i "s|xxx_secret_key_xxx|$(cat /stackable/secrets/secretKey)|g" /stackable/config/hive-site.xml && sed -i "s|XX_PASS_XX|$(cat /tmp/postgres-password/password)|g" /stackable/config/hive-site.xml && bin/start-metastore --config /stackable/config --db-type postgres --hive-bin-dir bin

I would like to comment that this workaround has worked perfectly for me. Although I would like to understand a little bit how it works underneath., as I can appreciate that there is more to it than just replacing the database password with the placeholder, @Maleware could you comment on why?

[razvan]

Hive expects the password in hive-site.xml. We have a policy of not reading product related secrets in the operators. So we put a placeholder in hive-site.xml and use sed to replace it in the container.

When we have implemented this ADR, the mechanism will hopefully be a bit more elegant and transparent. Until then, this is all there is to it.