https://grails.org/ logo
Join Slack
Powered by
Is there a new approach to working with external c...
# questions
m
Is there a new approach to working with external config.yml files in Grails 7? See https://stackoverflow.com/questions/79828816/external-config-yml-file-not-working-in-grails-7
m
I had issues with the dataSource being read in for TEST environment, but I still use an external YML file that is read in in a similar fashion. However, I think they did make changes to the processing of the file, so I couldn't have multiple, same-name root items. It did seem to be a hit-and-miss process for me to get my configs loading again from 6 to 7.
m
Yeah, it's more of a timing issue. The properties of the external config file seem to load, but too late in the process for some things.
m
I load mine in the Application.groovy file overriding the setEnvironment() method. I'm able to change the dataSource values, so I assume it's pretty early in the pipeline.
m
What version of Grails are you using? We are upgrading to 7.0.2 and are also overriding the setEnvironment() method in Application.groovy.
m
I went from 6.2.3 to 7.0.2. I did look at your question so I guess the question is, is it actually loading the external file? My biggest issue w/ every upgrade is the change in configs. For some reason, when moving to 7.0.2, it lost the ability to read the TEST ENV dataSource and I had to add in a block of code in the setEnvironment to load in yet another YML file w/ the test dataSource config.
Copy code
@Override
    void setEnvironment(org.springframework.core.env.Environment environment) {
        String env = grails.util.Environment.getCurrent().name
        def configBase = System.getenv('GRAILS_CONFIG_LOCATION') ?: System.getProperty('grails.config.location') ?: "/app/config/config-file"
        configBase = "${configBase}-${env}"
        boolean configLoaded = false

        def ymlConfig = new File(configBase + '.yml')

        //Update to 7.0.2 - Can't seem to figure out how to get the test config to load properly otherwise
        //So created a copy of the 'development' config for 'test' environment, modified as needed.
        //then put it in the classpath to be loaded here.
        if (Environment.current == Environment.TEST) {
            InputStream ymlFileStream = this.class.classLoader.getResourceAsStream("config-file-test.yml")

            <http://log.info|log.info> "Loading TEST configuration from YAML: conf/config-file-test.yml"
            Resource resourceConfig = new InputStreamResource(ymlFileStream)
            YamlPropertiesFactoryBean ypfb = new YamlPropertiesFactoryBean()
            ypfb.setResources(resourceConfig)
            ypfb.afterPropertiesSet()
            Properties properties = ypfb.getObject()
            environment.propertySources.addFirst(new PropertiesPropertySource("testYamlConfig", properties))
            <http://log.info|log.info> "TEST YAML configuration loaded."
        }

        if (ymlConfig.exists()) {
            <http://log.info|log.info> "Loading external configuration from YAML: ${ymlConfig.absolutePath}"
            Resource resourceConfig = new FileSystemResource(ymlConfig)
            YamlPropertiesFactoryBean ypfb = new YamlPropertiesFactoryBean()
            ypfb.setResources(resourceConfig)
            ypfb.afterPropertiesSet()
            Properties properties = ypfb.getObject()
            environment.propertySources.addFirst(new PropertiesPropertySource("externalYamlConfig", properties))
            <http://log.info|log.info> "External YAML configuration loaded."
            configLoaded = true
        }

        def groovyConfig = new File(configBase + '.groovy')
        if (groovyConfig.exists()) {
            <http://log.info|log.info> "Loading external configuration from Groovy: ${groovyConfig.absolutePath}"
            def config = new ConfigSlurper().parse(groovyConfig.toURI())
            environment.propertySources.addFirst(new MapPropertySource("externalGroovyConfig", config))
            <http://log.info|log.info> "External Groovy configuration loaded."
            configLoaded = true
        }

        if (!configLoaded) {
            <http://log.info|log.info> "External config could not be found, checked ${ymlConfig.absolutePath} and ${groovyConfig.absolutePath}"
        }

        Boolean tomcatLogging = environment.getProperty("eti.tomcat-logging", Boolean, Boolean.FALSE)
        if (tomcatLogging) {
            //get the log directory...
            String basedir = environment.getProperty('server.tomcat.basedir')
            Thread t = new Thread(new TomcatLogReader(basedir + "/logs"))
            t.daemon = true
            t.start()
        }
    }
Copy code
#ETI TESTING CONFIG FILE - DO NOT USE FOR PRODUCTION
dataSource:
  pooled: ${DB_POOLED:true}
  jmxExport: ${DB_JMX_EXPORT:true}
  dialect: ${DB_DIALECT:org.hibernate.dialect.MariaDBDialect}
  driverClassName: ${DB_DRIVER_CLASS_NAME:org.mariadb.jdbc.Driver}

  username: ${DB_USERNAME:REDACTED}
  password: ${DB_PASSWORD:REDACTED}

  dbCreate: ${DB_CREATE_MODE:create-drop}
  url: ${DB_URL:jdbc:<mariadb://localhost:3306/etiPortalTest?autoReconnect=true&useSSL=false>}
  properties:
    jmxEnabled: true
    initialSize: 5
    maxActive: 50
    minIdle: 5
    maxIdle: 25
    maxWait: 10000
    maxAge: 600000
    timeBetweenEvictionRunsMillis: 5000
    minEvictableIdleTimeMillis: 60000
    validationQuery: SELECT 1
    validationQueryTimeout: 3
    validationInterval: 15000
    testOnBorrow: true
    testWhileIdle: true
    testOnReturn: false
    jdbcInterceptors: ConnectionState
    defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED

dataSources:
  logging:
    pooled: ${DB_POOLED:true}
    jmxExport: ${DB_JMX_EXPORT:true}
    dialect: ${DB_DIALECT:org.hibernate.dialect.MariaDBDialect}
    driverClassName: ${DB_DRIVER_CLASS_NAME:org.mariadb.jdbc.Driver}

    username: ${DB_USERNAME:REDACTED}
    password: ${DB_PASSWORD:REDACTED}

    dbCreate: ${DB_CREATE_MODE:update}
    url: ${LOG_DB_URL:jdbc:<mariadb://localhost:3306/etiPortalTest?autoReconnect=true&useSSL=false>}
    properties:
      jmxEnabled: true
      initialSize: 5
      maxActive: 50
      minIdle: 5
      maxIdle: 25
      maxWait: 10000
      maxAge: 600000
      timeBetweenEvictionRunsMillis: 5000
      minEvictableIdleTimeMillis: 60000
      validationQuery: SELECT 1
      validationQueryTimeout: 3
      validationInterval: 15000
      testOnBorrow: true
      testWhileIdle: true
      testOnReturn: false
      jdbcInterceptors: ConnectionState
      defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED

grails:
  redis:
    useSSL: ${REDIS_SSL:false}
    host: ${REDIS_HOST:localhost}
    port: ${REDIS_port:6379}

spring:
  jms:
    cache:
      enabled: false
  jmx:
    unique-names: true
  main:
    banner-mode: "console"
  groovy:
    template:
      check-template-location: false
  autoconfigure:
    exclude:
      - org.springframework.boot.autoconfigure.security.oauth2.client.servlet.OAuth2ClientAutoConfiguration
      - org.springframework.boot.autoconfigure.security.servlet.SecurityFilterAutoConfiguration
      - org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration
      - org.springframework.boot.actuate.autoconfigure.security.servlet.ManagementWebSecurityAutoConfiguration
      - org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientAutoConfiguration
      - org.springframework.boot.autoconfigure.security.oauth2.resource.servlet.OAuth2ResourceServerAutoConfiguration
      - org.springframework.boot.autoconfigure.security.servlet.UserDetailsServiceAutoConfiguration
  session:
    store-type: redis
    redis:
      flush-mode: ON_SAVE
      namespace: ${REDIS_NAMESPACE:local-dev}
    pidfile: application.pid
# FOR V1.3, goes here. (spiring)
# For v1.4, goes to spring.data?
  redis:
    ssl: ${REDIS_SSL:false}
    host: ${REDIS_HOST:localhost}
    port: ${REDIS_port:6379}
It took me 1 week solid to get the upgrade to build and run. It's a big lift, all other upgrades since 3 have been pretty straight forward.
m
Most of my code looks identical to yours. What does your application.yml file look like for the datasources? Also, I noticed you are using "url" for the datasources. When I try to use "url" for my datasources Grails complains and says something about Hikari needing "jdbcUrl" instead.
m
Copy code
info:
  app:
    name: '@info.app.name@'
    version: '@info.app.version@'
    grailsVersion: '@info.app.grailsVersion@'
grails:
  mail:
    host: "<http://smtp.gmail.com|smtp.gmail.com>"
    port: 465
    username: ""
    password: ""
    props:
      mail.smtp.auth: "true"
      mail.smtp.socketFactory.port: "465"
      mail.smtp.socketFactory.class: "javax.net.ssl.SSLSocketFactory"
      mail.smtp.socketFactory.fallback: "false"
  redis:
    useSSL: ${REDIS_SSL:false}
    host: ${REDIS_HOST:localhost}
    port: ${REDIS_PORT:6379}
  mime:
    disable:
      accept:
        header:
          userAgents:
#          - Gecko
#          - WebKit
#          - Presto
#          - Trident
    types:
      all: '*/*'
      atom: application/atom+xml
      css: text/css
      csv: text/csv
      xlsx: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
      form: application/x-www-form-urlencoded
      html:
        - text/html
        - application/xhtml+xml
      js: text/javascript
      json:
        - application/json
        - text/json
      multipartForm: multipart/form-data
      pdf: application/pdf
      rss: application/rss+xml
      text: text/plain
      hal:
        - application/hal+json
        - application/hal+xml
  urlmapping:
    cache:
      maxsize: 1000
  controllers:
    defaultScope: singleton
    upload:
      maxFileSize: 26214400
      maxRequestSize: 26214400
  converters:
    encoding: UTF-8
  views:
    default:
      codec: html
    gsp:
      encoding: UTF-8
      htmlcodec: xml
      codecs:
        expression: html
        scriptlet: html
        taglib: none
        staticparts: none
  codegen:
    defaultPackage: com.einstein.tech.portal
  profile: web
  plugin:
    springsecurity:
      password:
        algorithm: pbkdf2
      logout.postOnly: false
      rest:
        token:
          storage:
            jwt:
              useEncryptedJwt: ${JWT_USE_ENCYPTED:true}
              privateKeyPath: ${JWT_PUBLIC_KEY:/pub/private_key.der}
              publicKeyPath: ${JWT_PRIVATE_KEY:/pub/public_key.der}
              secret: "${JWD_SECRET:REDACTED}"
dataSource:
  pooled: ${DB_POOLED:true}
  jmxExport: ${DB_JMX_EXPORT:true}
  dialect: ${DB_DIALECT:org.hibernate.dialect.MariaDBDialect}
  driverClassName: ${DB_DRIVER_CLASS_NAME:org.mariadb.jdbc.Driver}
  username: "${DB_USERNAME:REDACTED}"
  password: "${DB_PASSWORD:REDACTED}"

jasypt:
  algorithm: ${JASYPT_ALGORITHM:PBEWITHSHA256AND256BITAES-CBC-BC}
  providerName: ${JASYPT_PROVIDER:BC}
  password: "${JASYPT_PASSSWORD:REDACTED}"
  keyObtentionIterations: ${JASYPT_ITERATIONS:1000}

#<https://docs.spring.io/spring-boot/docs/2.1.4.RELEASE/reference/htmlsingle/#boot-features-session>
spring:
  activemq:
    non-blocking-redelivery: true
    broker-url: <tcp://localhost:61616>
    user: artemis
    password: artemis
    pool:
      enabled: false
  jms:
    cache:
      enabled: false
  #  templates:
  #    standard:
  #      connectionFactoryBean: jmsConnectionFactory
  #      messageConverterBean: standardJmsMessageConverter
  jmx:
    unique-names: true
  main:
    banner-mode: "console"
  groovy:
    template:
      check-template-location: false
  devtools:
    restart:
      additional-exclude:
        - '*.gsp'
        - '*.sql'
        - '**/*.gsp'
        - '*.gson'
        - '**/*.gson'
        - 'logback.groovy'
        - '*.properties'
      exclude:
        - grails-app/views/**
        - grails-app/i18n/**
        - grails-app/conf/**
  autoconfigure:
    exclude:
      - org.springframework.boot.autoconfigure.security.oauth2.client.servlet.OAuth2ClientAutoConfiguration
      - org.springframework.boot.autoconfigure.security.servlet.SecurityFilterAutoConfiguration
      - org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration
      - org.springframework.boot.actuate.autoconfigure.security.servlet.ManagementWebSecurityAutoConfiguration
      - org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientAutoConfiguration
      - org.springframework.boot.autoconfigure.security.oauth2.resource.servlet.OAuth2ResourceServerAutoConfiguration
      - org.springframework.boot.autoconfigure.security.servlet.UserDetailsServiceAutoConfiguration
  session:
    store-type: redis
    redis:
      flush-mode: ON_SAVE
      namespace: ${REDIS_NAMESPACE:local-dev}
    pidfile: application.pid
  data:
    redis:
      ssl:
        enabled: ${REDIS_SSL:false}
      host: ${REDIS_HOST:localhost}
      port: ${REDIS_PORT:6379}

environments:
  development:
    dataSource:
      dbCreate: update
      url: jdbc:<mariadb://localhost:3306/etiPortal?autoReconnect=true&useSSL=false>
      properties:
        jmxEnabled: true
        initialSize: 5
        maxActive: 50
        minIdle: 5
        maxIdle: 25
        maxWait: 10000
        maxAge: 600000
        timeBetweenEvictionRunsMillis: 5000
        minEvictableIdleTimeMillis: 60000
        validationQuery: SELECT 1
        validationQueryTimeout: 3
        validationInterval: 15000
        testOnBorrow: true
        testWhileIdle: true
        testOnReturn: false
        jdbcInterceptors: ConnectionState
        defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
    dataSources:
      logging:
        pooled: true
        jmxExport: true
        dialect: org.hibernate.dialect.MariaDBDialect
        driverClassName: org.mariadb.jdbc.Driver
        username: REDACTED
        password: REDACTED
        dbCreate: update
        url: jdbc:<mariadb://localhost:3306/etiPortal?autoReconnect=true&useSSL=false>
        properties:
          jmxEnabled: true
          initialSize: 5
          maxActive: 50
          minIdle: 5
          maxIdle: 25
          maxWait: 10000
          maxAge: 600000
          timeBetweenEvictionRunsMillis: 5000
          minEvictableIdleTimeMillis: 60000
          validationQuery: SELECT 1
          validationQueryTimeout: 3
          validationInterval: 15000
          testOnBorrow: true
          testWhileIdle: true
          testOnReturn: false
          jdbcInterceptors: ConnectionState
          defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
  test:
    dataSource:
      dbCreate: create-drop
      url: jdbc:<mariadb://localhost:3306/etiPortalTest?autoReconnect=true&useSSL=false>
      properties:
        jmxEnabled: true
        initialSize: 5
        maxActive: 50
        minIdle: 5
        maxIdle: 25
        maxWait: 10000
        maxAge: 600000
        timeBetweenEvictionRunsMillis: 5000
        minEvictableIdleTimeMillis: 60000
        validationQuery: SELECT 1
        validationQueryTimeout: 3
        validationInterval: 15000
        testOnBorrow: true
        testWhileIdle: true
        testOnReturn: false
        jdbcInterceptors: ConnectionState
        defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
    dataSources:
      logging:
        pooled: true
        jmxExport: true
        dialect: org.hibernate.dialect.MariaDBDialect
        driverClassName: org.mariadb.jdbc.Driver
        username: REDACTED
        password: REDACTED
        dbCreate: update
        url: jdbc:<mariadb://localhost:3306/etiPortalTest?autoReconnect=true&useSSL=false>
        properties:
          jmxEnabled: true
          initialSize: 5
          maxActive: 50
          minIdle: 5
          maxIdle: 25
          maxWait: 10000
          maxAge: 600000
          timeBetweenEvictionRunsMillis: 5000
          minEvictableIdleTimeMillis: 60000
          validationQuery: SELECT 1
          validationQueryTimeout: 3
          validationInterval: 15000
          testOnBorrow: true
          testWhileIdle: true
          testOnReturn: false
          jdbcInterceptors: ConnectionState
          defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
  production:
    dataSource:
      dbCreate: update
      url: jdbc:<mariadb://localhost:3306/etiPortalProd?autoReconnect=true&useSSL=false>
      properties:
        jmxEnabled: true
        initialSize: 5
        maxActive: 50
        minIdle: 5
        maxIdle: 25
        maxWait: 10000
        maxAge: 600000
        timeBetweenEvictionRunsMillis: 5000
        minEvictableIdleTimeMillis: 60000
        validationQuery: SELECT 1
        validationQueryTimeout: 3
        validationInterval: 15000
        testOnBorrow: true
        testWhileIdle: true
        testOnReturn: false
        jdbcInterceptors: ConnectionState
        defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
    dataSources:
      logging:
        pooled: true
        jmxExport: true
        dialect: org.hibernate.dialect.MariaDBDialect
        driverClassName: org.mariadb.jdbc.Driver
        username: REDACTED
        password: REDACTED
        dbCreate: update
        url: jdbc:<mariadb://localhost:3306/etiPortalProd?autoReconnect=true&useSSL=false>
        properties:
          jmxEnabled: true
          initialSize: 5
          maxActive: 50
          minIdle: 5
          maxIdle: 25
          maxWait: 10000
          maxAge: 600000
          timeBetweenEvictionRunsMillis: 5000
          minEvictableIdleTimeMillis: 60000
          validationQuery: SELECT 1
          validationQueryTimeout: 3
          validationInterval: 15000
          testOnBorrow: true
          testWhileIdle: true
          testOnReturn: false
          jdbcInterceptors: ConnectionState
          defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
eti:
  async-metric-update: false
  async-compliant-update: false
  # DEV
  use-submit-event-escalate-stored-proc: true

  use-unsubmit-stored-proc: true
  use-time-entry-invoice-init-stored-proc: true

  # BETA
  use-pay-period-update-comp-time-stored-proc: true # DEFAULT false
  use-pay-period-update-stored-proc: true # DEFAULT false
  use-submit-stored-proc: true
  use-pay-period-create-stored-proc: true

  # PROD LEVEL
  IBA_MEDICAL_OUTSIDE_OF_DATE_RANGE: true
  allow-query-read-write: true
  use-experimental: false
  use-stored-procedure-for-iba: true
  use-stored-procedure-company-pay-period: true
  use-end-of-pay-period-for-iba-balance: true
  tomcat-logging: false
  init-file: data/initialization.json
hibernate:
  cache:
    queries: false
    use_second_level_cache: false
    use_query_cache: false
quartz.autoStartup: true
management:
  info:
    git:
      mode: full
I don't know, I never got a warning about not using the same old configs, but as I said, when I used this as the application.yml, it did not like it for the test environment. my dev env. may have been similar, but I loaded in the external config file for that so I never saw it.
Above: re: url specification of database
To explain: what was happening in my TEST env. was that if I used the application.yml as presented above, it thought it was supposed to be connecting to a mysql 8 db and the URL it printed out as an exception was not one that I specified anywhere. So I figured it wasn't reading in the settings, so I basically duplicated my method for the devevlopment envinronment, but for test and it worked for me. So instead of tracking any further, I wrote a block of code and moved on.
m
I think I have it working now. The main datasource is using "url" and the secondary datasource for a Spring Boot subproject is using "jdbcUrl". No build/Hikari errors and the app seems to run fine. Below is a snippet from an external config.yml file I'm using for testing. Thanks for your help! Much appreciated.
Copy code
dataSource:
    dbCreate: update

    #MariaDB database settings
    driverClassName: org.mariadb.jdbc.Driver
    dialect: org.hibernate.dialect.MariaDB106DBDialect
    url: jdbc:<mysql://localhost:3306/main_app>
    username: 'root'
    password: 'REDACTED'
    properties:
          jmxEnabled: true
          initialSize: 5
          maxActive: 50
          minIdle: 5
          maxIdle: 25
          maxWait: 10000
          maxAge: 600000
          validationQuery: SELECT 1 #Non-Oracle
          validationQueryTimeout: 3
          validationInterval: 15000
          defaultTransactionIsolation: 2 # ORACLE AND MYSQL
          dbProperties:
                autoReconnect: true
dataSources:
  datasourceadmin:
    jdbcUrl: jdbc:<mysql://localhost:3306/admin>
    username: 'root'
    password: 'REDACTED'
    driverClassName: org.mariadb.jdbc.Driver
👍 1
j
Grails 7 merged the external config plugin and it's possible to define external configuration files (optionally) by setting the list property grails.config.locations. More information is in the grails guide: https://grails.apache.org/docs/latest/guide/conf.html#externalConfiguration
I haven't tested it with environments, but I assume if you specify your configuration in an external file & you specify it in the test environments block, it would only be applied to test
👍 2
d
im running into the same issue and seems like the jdbcUrl isnt fix isnt working
6 Views
Open in Slack
PreviousNext
Powered by