IODA v2 format problem

JEDI
1

We convert our AOD observations in IODA v1 with ioda-upgrade.x to obtain file in IODA v2.
This step runs fine and the new file looks as could be expected with ncdump. When we run UFO it will crash with segfault as below. The same yaml with run fine with obs that are used in the ctest.
Help/suggestions much appreciated, thanks

[h25c19:60786:0:60786] Caught signal 11 (Segmentation fault: address not m
apped to object at address (nil))

2

I can help debug this.

Please send me the following so I can reproduce the seg fault.

  • The ioda file
  • The ufo application you are running
    • The ufo yaml file
    • The geovals file (if one is being used)

Thanks!

3

Thanks for offering to help.
It is a little bit tricky as we don’t have geovals because we run UFO within fv3-jedi’s 3d-var.
I think the easiest to reproduce the experiment failure would be to replace the current obs file used in ctest -VV -R fv3jedi_test_tier1_hyb-3dvar_gfs_aero with the one that we produced (20 obs, matching date, same one channel etc.). Would it work if I sent it to you, put it somewhere?

4

Thanks for the explanation. It should work to send only the new ioda obs file. I can then add it to the test you indicated and proceed with the debugging.

5

Just in case I attached both v1 and v2 files

(Attachment aod_viirs_obs_2018041500_20_v1.nc is missing)

(Attachment aod_viirs_obs_2018041500_20_v2.nc is missing)

6

OK, will send separately

7

For now, the ioda-v2 reader is getting confused with the nlocs dimension being marked as an “UNLIMITED” size dimension. The issue is that the HDF5 API returns a zero (instead of the actual size) when you query the dimension size.

The file works if you change nlocs to a fixed size dimension. This can be done with the NCO ncks tool:

ncks -4 --fix_rec_dmn all input-file output-file

The “-4” option says to write the output in netcdf4 format, and the “–fix_rec_dmn all” option says to change all unlimited size dimensions to fixed size.

This is definitely a bug (unfortunately in the HDF5 library) and I’ll make sure there is a ZenHub issue submitted to allow for dimensions with unlimited size.

8

Thanks, we are still checking if all works as expected but so far the new AOD files in IODA v2 with limited nlocs

appear to be correct

9

The ZenHub issue related to this is ioda #266.

10

I just created and closed a ZenHub issue about this particular issue with the new AOD obs file and the workaround using ncks. I just wanted to have a record of what happened and how we fixed it.