Runtime error with MetOffice GNSSRO Bending Angle Operator

JEDI
1

I have been able to successfully run a test run of the hofx using the NBAM GNSSRO operator. Next I tried running the same experiment again using the MetOffice GNSSRO operator. It failed with errors I do not understand.

Here is my yaml input file:

# hofx3d jedi config.yaml
window begin: 2021-01-25T03:00:00Z
window length: PT6H
forecast length: PT6H
# model/gfs/hofx3d.jedi.yaml
geometry:
  nml_file_mpp: config/fmsmpp.nml
  trc_file: config/field_table
  akbk: config/akbk_gfs.nc4
  nml_file: config/geometry_gfs_c768.nml
  layout: [1,2]
  io_layout: [1,1]
  npx: 769
  npy: 769
  npz: 127
  ntiles: 6
  fieldsets:
  - fieldset: config/fieldsets/dynamics.yaml
  - fieldset: config/fieldsets/ufo.yaml
state:
  filetype: gfs
  datapath: input/bg/window_center
  filename_core: 20210125.060000.fv_core.res.nc
  filename_trcr: 20210125.060000.fv_tracer.res.nc
  filename_sfcd: 20210125.060000.sfc_data.nc
  filename_sfcw: 20210125.060000.fv_srf_wnd.res.nc
  filename_cplr: 20210125.060000.coupler.res
  state variables: [u,v,ua,va,T,DELP,sphum,ice_wat,liq_wat,o3mr,phis,
                    slmsk,sheleg,tsea,vtype,stype,vfrac,stc,smc,snwdph,
                    u_srf,v_srf,f10m]
observations: 
- obs operator:
    name: GnssroBendMetOffice
    obs options:
      vert_interp_ops: true
      pseudo_ops: true
      min_temp_grad: 1.0e-6
  obs space:
    name: GnssroBnd
    obsdatain:
      obsfile: input/obs/ioda_nomads_gnssro_PT6H_20210125_0600Z_v2.nc4
    obsdataout:
      obsfile: output/hofx/hofx3d_gfs_c48_nomads_gnssro_metbnd_PT6H_20210125_0600Z.nc4
    simulated variables: [bending_angle]
#  geovals:
#    filename: Data/ufo/testinput_tier_1/gnssro_geoval_2019050700_1obs.nc4
  obs filters:
  - filter: Background Check
    filter variables:
    - name: bending_angle
    threshold: 3.0
  norm ref: MetOfficeHofX
  tolerance: 1.0e-5
  linear obs operator test:
    coef TL: 1.0e-4
    iterations TL:  10
    tolerance TL: 1.5e-14
    tolerance AD: 1.5e-14

And here is what I think is the most relevant part of the error message:

Test     : H(x): 
Test     : GnssroBnd nobs= 104870 Min=0, Max=0.0204072, RMS=-nan

Test     : End H(x)
GnssroBnd: save database to output/hofx/hofx3d_gfs_c48_nomads_gnssro_metbnd_PT6H_20210125_0600Z.nc4
free(): invalid pointer
[comet-30-01:07852] *** Process received signal ***
[comet-30-01:07852] Signal: Aborted (6)
[comet-30-01:07852] Signal code:  (-6)
free(): invalid pointer
[comet-30-01:07857] *** Process received signal ***
[comet-30-01:07857] Signal: Aborted (6)
[comet-30-01:07857] Signal code:  (-6)
[comet-30-01:07852] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x46210)[0x2af4aca86210]
[comet-30-01:07852] [ 1] [comet-30-01:07857] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x46210)[0x2af314f59210]
[comet-30-01:07857] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x2af4aca8618b]
[comet-30-01:07852] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x2af314f5918b]
[comet-30-01:07857] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x2af314f38859]
[comet-30-01:07857] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x903ee)[0x2af314fa33ee]
[comet-30-01:07857] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x9847c)[0x2af314fab47c]
[comet-30-01:07857] [ 5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x2af4aca65859]
[comet-30-01:07852] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x903ee)[0x2af4acad03ee]
[comet-30-01:07852] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x9847c)[0x2af4acad847c]
[comet-30-01:07852] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x99cac)[0x2af4acad9cac]
[comet-30-01:07852] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0x99cac)[0x2af314faccac]
[comet-30-01:07857] [ 6] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(__fv3jedi_fields_mod_MOD_
delete+0x138)[0x2af3114d0d68]
[comet-30-01:07857] [ 7] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(__fv3jedi_fields_mod_MOD_
delete+0x138)[0x2af4a8ffdd68]
[comet-30-01:07852] [ 7] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(fv3jedi_state_delete_f90+
0x55)[0x2af31167e755]
[comet-30-01:07857] [ 8] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(fv3jedi_state_delete_f90+
0x55)[0x2af4a91ab755]
[comet-30-01:07852] [ 8] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(_ZN7fv3jedi5StateD1Ev+0x2
3)[0x2af4a91a8203]
[comet-30-01:07852] [ 9] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(_ZN7fv3jedi5StateD1Ev+0x2
3)[0x2af31167b203]
[comet-30-01:07857] [ 9] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(_ZN7fv3jedi5StateD0Ev+0xd
)[0x2af4a91a832d]
[comet-30-01:07852] [10] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/libfv3jedi.so(_ZN7fv3jedi5StateD0Ev+0xd
)[0x2af31167b32d]
[comet-30-01:07857] [10] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x82010)[0x55995a4b101
0]
[comet-30-01:07852] [11] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x82010)[0x558f596dd01
0]
[comet-30-01:07857] [11] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x11211a)[0x55995a5411
1a]
[comet-30-01:07852] [12] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/liboops.so(_ZN4oops3Run7executeERKNS_11
ApplicationE+0x123)[0x2af4ac654863]
[comet-30-01:07852] [13] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x11211a)[0x558f5976d1
1a]
[comet-30-01:07857] [12] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/../lib/liboops.so(_ZN4oops3Run7executeERKNS_11
ApplicationE+0x123)[0x2af314b27863]
[comet-30-01:07857] [13] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x4c8f2)[0x55995a47b8f
2]
[comet-30-01:07852] [14] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x2af4aca670b3]
[comet-30-01:07852] [15] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x4c8f2)[0x558f596a78f
2]
[comet-30-01:07857] [14] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x2af314f3a0b3]
[comet-30-01:07857] [15] /cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x4c95e)[0x55995a47b95
e]
[comet-30-01:07852] *** End of error message ***
/cw3e/mead/projects/cwp106/projects/hiaper/mjmurphy/modeling/jedi/build/fv3jedi/bin/fv3jedi_hofx_nomodel.x(+0x4c95e)[0x558f596a795e]
[comet-30-01:07857] *** End of error message ***
free(): invalid pointer

Thanks for any help you can provide!

Michael

2

Hi @stephenh could you or someone you know who is familiar with this operator please take a look at my runtime problem when you get a chance.

Thanks,
Michael

3

Hi @mjm, sorry about the lack of response. I’ve forwarded this issue to the JEDI core team in our Slack messaging. Hopefully, someone will pick it up soon.

1 Like
4

Where can I go to get help with my issue, this forum seems to not be the place @mmiesch

5

Hi @mjm, Again I apologize for the lack of response. Just so you know, Mark Miesch has left JCSDA. I have again put out a request to the core team for someone to help you with this issue. Hopefully, someone will respond soon.

1 Like
6

Two possible issues while using FV3 and UKMet RO operator:

State/control variables are different for FV3 and the operator

Vertical levels: UKMet models are staggering (uv are not at the same levels as for T and humidity) in a vertical direction but FV3 doesn’t.

It would be good to check these two parts first.

7

Hi @mjm is this problem resolved?

I have run the UK metoffice operator using the same yaml as yours except that I have a different date and JEDI version.
The version I am using is a little bit different from yours in the geometry section. please see
https://github.com/JCSDA-internal/fv3-jedi/blob/develop/test/testinput/hofx_nomodel.yaml

While I have the same configuration in the entire “observations” section, I run through the operator successfully. Sorry that I can not reproduce your error. If you want more discussion please feel free to send direct email to me at hailingz@ucar.edu and I may use your data for further debugging.

1 Like