Fv3-jedi: Adding an additional analysis variable in the fv3-jedi hyb-3dvar test

JEDI
1

Hello! I am new to JEDI and I have been testing for adding a new analysis variable into a regular hyb-3dvar test in fv3-jedi. More specifically, the following are the original variables in the yaml file

analysis variables: &3dvars [ua,va,T,DELP,sphum,ice_wat,liq_wat,o3mr]

and then I changed it to

analysis variables: &3dvars [ua,va,T,DELP,sphum,ice_wat,liq_wat,o3mr,smc]

Besides, in the yaml file, I changed the covariance to 100% from the ensemble members. However, I got some errors such as the following

!!! ABORT in atlas_field_to_array_real_r2 on task #000008: not enough levels in ATLAS field

I wonder how I can better address such issues. Thank you very much!

Liaofan

2

Hi Liaofan,

I’m not sure what could cause this off the top of my head.

One thing I notice is that the field “smc” has 4 levels (src/fv3jedi/FieldMetadata/FieldsMetadataDefault.h lines 607-615), so one possibility is that the code for the analysis variables sees a 3D field and assumes it will have the full number of atmospheric levels.

Could you give some more details to help diagnose? What version of fv3-jedi/oops/etc are you using? What HPC/machine are you running on? Is there any backtrace information along with the error message you’ve pasted?

Thanks!

Francois H

3

Hello Francois,

Thank you for the help.

Yes, smc has 4 levels while other atmospheric variables have 127 levels. Below are what I have found so far:

In $fv3-bundle/saber/src/saber/util/type_fieldset.fypp, subroutine fieldset_set_metadata has the following lines:

! Get number of levels
fieldset%nl0 = size(gmask,2)

This variable appears to be 127 for this hyb-3dvar case.

In $fv3-bundle/saber/src/saber/util/type_atlas.fypp, subroutine atlas_field_to_array_* has the following line:

if (nl0>afield%levels()) call mpl%abort(’${subr}$’,’not enough levels in ATLAS field’)

For atmospheric variables, “afield%levels()=127” but for smc, “afield%levels()=4”. Therefore, the test crashed. I wonder if there are ways of using analysis variables that are 3D but have different vertical levels.

For your other questions, the following shows where my relevant repositories are from:

  ecbuild_bundle( PROJECT oops  GIT "https://github.com/jcsda/oops.git"  TAG 72d2c05 )
  ecbuild_bundle( PROJECT saber GIT "https://github.com/jcsda/saber.git" TAG e742e4f )
  ecbuild_bundle( PROJECT fv3-jedi    GIT "https://github.com/jcsda/fv3-jedi.git"             TAG f79ed41 )

I am running my tests on the NOAA Hera HPC. Let me know if you like to know anything else.

Thanks!
Liaofan

4

Hi @liafoanlin, sorry for this late reply. I can give you a temporary solution to run the hybrid 3DVar with 100% on the static B (and 0% on the ensemble covariance). You have to run two steps:

  • Compute a static B matrix for the new variable smc with 4 levels (see bumpparameters_nicas_gfs.yaml attached).
  • Use this new component of the static B matrix in the hybrid 3DVar (see hyb-3dvar.yaml attached).

In both yaml files, please replace smc with the long name of the variable that is now expected (e.g. air_temperature instead of T).

To use the ensemble B with a variable that has a different number of levels is tricky. Indeed, to keep cross-correlations between variables, all increment fields should be added up on the same grid. 2D fields like surface pressure can be added at the bottom level of the model, but I don’t know how to deal with smc and its 4 levels. Are these levels related to a given height? Anyway, some significant developments will be needed.

5

Looks like I can’t attached yaml files. Here they are (not tested):

bumpparameters_nicas_gfs.yaml:

geometry:
  fms initialization:
    namelist filename: Data/fv3files/fmsmpp.nml
    field table filename: Data/fv3files/field_table_gfdl
  akbk: Data/fv3files/akbk127.nc4
  npx: 13
  npy: 13
  npz: 127
  field metadata override: Data/fieldmetadata/gfs-restart.yaml
input variables: [air_temperature,smc,surface_pressure]
background:
  datetime: 2020-12-14T21:00:00Z
  filetype: fms restart
  datapath: Data/inputs/gfs_c12/bkg/
  filename_core: 20201214.210000.fv_core.res.nc
  filename_trcr: 20201214.210000.fv_tracer.res.nc
  filename_sfcd: 20201214.210000.sfc_data.nc
  filename_sfcw: 20201214.210000.fv_srf_wnd.res.nc
  filename_cplr: 20201214.210000.coupler.res
  state variables: [air_temperature,smc,surface_pressure]
bump:
  verbosity: main
  universe_rad: 2500.0e3
  strategy: specific_univariate
  new_nicas: true
  write_nicas_local: true
  resol: 6
  forced_radii: true
  grids:
  - prefix: Data/bump/fv3jedi_bumpparameters_nicas_3D_gfs
    variables: [air_temperature]
    rh:
      air_temperature: [2500000.0]
    rv:
      air_temperature: [0.3]
    io_keys: [air_temperature-air_temperature]
    io_values: [fixed_2500km_0.3]
  - prefix: Data/bump/fv3jedi_bumpparameters_nicas_3DL4_gfs
    variables: [smc]
    rh:
      smc: [2500000.0] # Choose the horizontal length-scale here
    rv:
      smc: [0.3] # Choose the vertical length-scale here
    io_keys: [smc-smc]
    io_values: [fixed_3DL4]
  - prefix: Data/bump/fv3jedi_bumpparameters_nicas_2D_gfs
    variables: [surface_pressure]
    rh:
      surface_pressure: [2500000.0]
    rv:
      surface_pressure: [0.0]
    io_keys: [surface_pressure-surface_pressure]
    io_values: [fixed_2500km]
output:
- parameter: cor_rh
  file:
    filetype: fms restart
    datapath: Data/bump/
    filename_core: bumpparameters_nicas_gfs.cor_rh.fv_core.res.nc
    filename_trcr: bumpparameters_nicas_gfs.cor_rh.fv_tracer.res.nc
    filename_sfcd: bumpparameters_nicas_gfs.cor_rh.sfc_data.nc
    filename_sfcw: bumpparameters_nicas_gfs.cor_rh.fv_srf_wnd.res.nc
    filename_cplr: bumpparameters_nicas_gfs.cor_rh.coupler.res
- parameter: cor_rv
  file:
    filetype: fms restart
    datapath: Data/bump/
    filename_core: bumpparameters_nicas_gfs.cor_rv.fv_core.res.nc
    filename_trcr: bumpparameters_nicas_gfs.cor_rv.fv_tracer.res.nc
    filename_sfcd: bumpparameters_nicas_gfs.cor_rv.sfc_data.nc
    filename_sfcw: bumpparameters_nicas_gfs.cor_rv.fv_srf_wnd.res.nc
    filename_cplr: bumpparameters_nicas_gfs.cor_rv.coupler.res

test:
  reference filename: testoutput/bumpparameters_nicas_gfs.ref
  test output filename: testoutput/bumpparameters_nicas_gfs.test.out

hyb-3dvar.yaml:

cost function:
  cost type: 3D-Var
  window begin: 2020-12-14T21:00:00Z
  window length: PT6H
  analysis variables: &3dvars [eastward_wind,northward_wind,air_temperature,air_pressure_thickness,specific_humidity,cloud_liquid_ice,cloud_liquid_water,ozone_mass_mixing_ratio,smc]
  geometry:
    fms initialization:
      namelist filename: Data/fv3files/fmsmpp.nml
      field table filename: Data/fv3files/field_table_gfdl
    akbk: Data/fv3files/akbk127.nc4
    npx: 25
    npy: 25
    npz: 127
    field metadata override: Data/fieldmetadata/gfs-restart.yaml
  background:
    datetime: 2020-12-15T00:00:00Z
    filetype: fms restart
    datapath: Data/
    prefix: 20201215.000000.c24
    state variables: [ua,va,T,DELP,sphum,ice_wat,liq_wat,o3mr,phis,
                      slmsk,sheleg,tsea,vtype,stype,vfrac,stc,smc,snwdph,
                      u_srf,v_srf,f10m]
  background error:
    covariance model: hybrid
    components:
    - covariance:
        covariance model: SABER
        saber central block:
          saber block name: BUMP_NICAS
          active variables: [eastward_wind,northward_wind,air_temperature,air_pressure_thickness,specific_humidity,cloud_liquid_ice,cloud_liquid_water,ozone_mass_mixing_ratio,smc]
          bump:
            method: cor
            strategy: specific_univariate
            load_nicas_local: true
            verbosity: main
            grids:
            - prefix: Data/bump/fv3jedi_bumpparameters_nicas_3D_gfs
              variables: [eastward_wind,northward_wind,air_temperature,air_pressure_thickness,specific_humidity,cloud_liquid_ice,cloud_liquid_water,ozone_mass_mixing_ratio]
              io_keys: [eastward_wind-eastward_wind,northward_wind-northward_wind,air_temperature-air_temperature,air_pressure_thickness-air_pressure_thickness,specific_humidity-specific_humidity,cloud_liquid_ice-cloud_liquid_ice,cloud_liquid_water-cloud_liquid_water,ozone_mass_mixing_ratio-ozone_mass_mixing_ratio]
              io_values: [fixed_2500km_0.3,fixed_2500km_0.3,fixed_2500km_0.3,fixed_2500km_0.3,fixed_2500km_0.3,fixed_2500km_0.3,fixed_2500km_0.3,fixed_2500km_0.3]
            - prefix: Data/bump/fv3jedi_bumpparameters_nicas_3DL4_gfs
              variables: [smc]
              io_keys: [smc-smc]
              io_values: [fixed_3DL4]
      weight:
        value: 1.0
  observations:
    observers:
    - obs space:
        # The input file does not contain a variable called 'air_temperature', but contains one
        # called 'renamed_air_temperature'. We use filters to copy renamed_air_temperature@ObsValue
        # to air_temperature@DerivedObsValue and renamed_air_temperature@ObsError to
        # air_temperature@ObsError. In this way we verify derived variables can be assimilated.
        name: Aircraft
        obsdatain:
          engine:
            type: H5File
            obsfile: Data/obs/testinput_tier_1/aircraft_obs_2020121500_m_renamed_var.nc4
        obsdataout:
          engine:
            type: H5File
            obsfile: Data/hofx/aircraft_hyb-3dvar-gfs_2020121500_m.nc4
        simulated variables: [eastward_wind, northward_wind, air_temperature]
        observed variables: [eastward_wind, northward_wind]
        derived variables: [air_temperature]
      obs operator:
        name: VertInterp
      obs error:
        covariance model: diagonal ufo
      obs filters:
      # Create a derived variable
      - filter: Variable Assignment
        assignments:
        - name: air_temperature@DerivedObsValue
          type: float
          source variable:
            name: renamed_air_temperature@ObsValue
      # Populate its ObsError estimates (filled with missing values up to now)
      - filter: Perform Action
        filter variables: [air_temperature]
        action:
          name: assign error
          error function:
            name: renamed_air_temperature@ObsError
      # Populate its PreQC flags
      - filter: Variable Assignment
        assignments:
        - name: air_temperature@PreQC
          type: int
          source variable:
            name: renamed_air_temperature@PreQC
      - filter: PreQC
        maxvalue: 3
      - filter: Background Check
        filter variables:
        - name: eastward_wind
        - name: northward_wind
        - name: air_temperature
        threshold: 6.0
    - obs space:
        name: GnssroBndNBAM
        obsdatain:
          engine:
            type: H5File
            obsfile: Data/obs/testinput_tier_1/gnssro_obs_2020121500_m.nc4
        simulated variables: [bending_angle]
      obs operator:
        name: GnssroBndNBAM
        obs options:
          sr_steps: 2
          vertlayer: full
          use_compress: 1
          super_ref_qc: NBAM
      obs error:
        covariance model: diagonal ufo
      obs filters:
      - filter: Domain Check
        filter variables:
        - name: bending_angle
        where:
        - variable:
            name: impact_height@MetaData
          minvalue: 0
          maxvalue: 50000
      - filter: ROobserror
        filter variables:
        - name: bending_angle
        errmodel: NBAM
      - filter: Background Check RONBAM
        filter variables:
        - name: bending_angle
    - obs space:
        name: AMSUA-NOAA19
        obsdatain:
          engine:
            type: H5File
            obsfile: Data/obs/testinput_tier_1/amsua_n19_obs_2020121500_m.nc4
        simulated variables: [brightness_temperature]
        channels: 1-15
      obs operator:
        name: CRTM
        Absorbers: [H2O,O3]
        obs options:
          Sensor_ID: amsua_n19
          EndianType: little_endian
          CoefficientPath: Data/crtm/
      obs bias:
        input file: Data/obs/testinput_tier_1/satbias_amsua_n19.nc4
        output file: Data/analysis/satbias_amsua_n19_out.nc4
        variational bc:
          predictors:
          - name: constant
          - name: lapse_rate
            order: 2
            tlapse: &amsua19tlap Data/obs/testinput_tier_1/amsua_n19_tlapmean.txt
          - name: lapse_rate
            tlapse: *amsua19tlap
          - name: emissivity
          - name: scan_angle
            order: 4
          - name: scan_angle
            order: 3
          - name: scan_angle
            order: 2
          - name: scan_angle
        covariance:
          minimal required obs number: 20
          variance range: [1.0e-6, 10.0]
          step size: 1.0e-4
          largest analysis variance: 10000.0
          prior:
            input file: Data/obs/testinput_tier_1/satbias_amsua_n19.nc4
            inflation:
              ratio: 1.1
              ratio for small dataset: 2.0
          output file: Data/analysis/satbias_cov_amsua_n19_out.nc4
      obs error:
        covariance model: diagonal
      obs filters:
      - filter: Bounds Check
        filter variables:
        - name: brightness_temperature
          channels: 1-15
        minvalue: 100.0
        maxvalue: 500.0
      - filter: Background Check
        filter variables:
        - name: brightness_temperature
          channels: 1-15
        threshold: 3.0
    - obs space:
        name: SfcObs
        obsdatain:
          engine:
            type: H5File
            obsfile: Data/obs/testinput_tier_1/sfc_obs_2020121500_m.nc4
        simulated variables: [surface_pressure]
      obs operator:
        name: SfcPCorrected
        da_psfc_scheme: UKMO
      linear obs operator:
        name: Identity
      obs error:
        covariance model: diagonal ufo
      obs filters:
      - filter: Background Check
        threshold: 1000
    - obs space:
        name: Radiosonde
        obsdatain:
          engine:
            type: H5File
            obsfile: Data/obs/testinput_tier_1/sondes_obs_2020121500_m.nc4
        simulated variables: [eastward_wind, northward_wind, air_temperature]
      obs operator:
        name: VertInterp
      obs error:
        covariance model: diagonal ufo
      obs filters:
      - filter: PreQC
        maxvalue: 3
      - filter: Background Check
        filter variables:
        - name: eastward_wind
        - name: northward_wind
        - name: air_temperature
        threshold: 6.0
      monitoring only: true
variational:
  minimizer:
    algorithm: DRIPCG
  iterations:
  - ninner: 10
    gradient norm reduction: 1e-10
    test: on
    geometry:
      akbk: Data/fv3files/akbk127.nc4
      npx: 13
      npy: 13
      npz: 127
      field metadata override: Data/fieldmetadata/gfs-restart.yaml
    diagnostics:
      departures: ombg
    online diagnostics:
      write increment: true
      increment:
        filetype: fms restart
        datapath: Data/analysis
        prefix: iter1.hyb-3dvar.c24
  - ninner: 5
    gradient norm reduction: 1e-10
    test: on
    geometry:
      akbk: Data/fv3files/akbk127.nc4
      npx: 13
      npy: 13
      npz: 127
      field metadata override: Data/fieldmetadata/gfs-restart.yaml
    diagnostics:
      departures: ombg
    online diagnostics:
      write increment: true
      increment:
        filetype: fms restart
        datapath: Data/analysis
        prefix: iter2.hyb-3dvar.c24
final:
  diagnostics:
    departures: oman
output:
  filetype: fms restart
  datapath: Data/analysis/
  prefix: 20201215.000000.hyb-3dvar.c24
  frequency: PT1H

test:
  reference filename: testoutput/hyb-3dvar.ref
  test output filename: testoutput/hyb-3dvar.test.out
  float relative tolerance: 1.0e-3
  float absolute tolerance: 1.0e-6
6

Hello @benjaminmenetrier, thank you very much for the help! I will give it a try with your yaml files using 100% static B. But at the same time, could you suggest to me how the coding can be done to use 100% ensemble covariance for this case? The smc can temporarily be treated as at the bottom level of the model, for all its four layers.

You could also send the yaml files to my email (liaofan.lin@noaa.gov) in the future, in case that attachments are not allowed here.