Trouble running mpasjedi_hofx3d.x with mpas-bundle 3.0.2

JEDI
1

I’m having trouble getting past the initialization phases when running mpasjedi_hofx3d.x using mpas-bundle 3.0.2.

First, I created input data by converting a 27-hour sequence of of 3-hourly GFS GRIB data on pressure coordinates into WRF intermediate format using the ungrib utility. Then I converted that data into a MPAS initialization file with the mpas_init_atmosphere program for the x1.40962 mesh.

I then tried to run mpasjedi_hofx3d.x, where is ran into my first runtime problem with the initialization of the calendar in the ESMF subsystem. I worked around this by recompiling MPAS with to ensure the ESMF_Initialize function was properly called, as described in this post in the WRF & MPAS support forum.

Since then, I’ve run into a variety of other runtime errors. I’ve tried to deduce the causes of the problems using a debugger, but I keep running into more problems. I hope people on this forum can help me get unstuck.

My current error has these properties:

  • This diagnostic message to stderr:
    → vars_nlevels: specific_humidity not available in MPAS domain or self % templated_fields
  • A WARNING in log.atmosphere.0000.out
    WARNING: Error: Field scalars has more than one time level, but no timeLevel argument given.
    This is followed by an abort.

I am running the software in this JEDI container:
docker.io/jcsda/docker-gnu-openmpi-dev:skylab-v8
This is the latest published docker image for the gnu environment and was created 12 months ago.

I run hofx with this command (where $jedibin is the full path the mpas-bundle/build/bin directory):
mpirun -np 1 $jedibin/mpasjedi_hofx3d.x test_hofx3d.yaml
I’ve also tried running with more processes, but it does not change the error.

I’ve created a tar file with all the configuration I’m using. I also included the full log.atmosphere.0000.out log and the output of ncdump -h x1.40962.init.nc, in case there is a problem with the input data I generated. A log.atmosphere.0000.err file was not generated.
However, tar.gz does not seem to be an allowed files type for me to upload. These are the allowed types: pg, jpeg, png, gif, heic, heif, webp, avif.
Please let me know if these is a better way to provide this information.

The run directory also has what I think are the necessary links to auxiliary files:

CAM_ABS_DATA.DBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/CAM_ABS_DATA.DBL
CAM_AEROPT_DATA.DBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/CAM_AEROPT_DATA.DBL
COMPATIBILITY -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/COMPATIBILITY
GENPARM.TBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/GENPARM.TBL
LANDUSE.TBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/LANDUSE.TBL
OZONE_DAT.TBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/OZONE_DAT.TBL
OZONE_LAT.TBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/OZONE_LAT.TBL
OZONE_PLEV.TBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/OZONE_PLEV.TBL
RRTMG_LW_DATA -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/RRTMG_LW_DATA
RRTMG_LW_DATA.DBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/RRTMG_LW_DATA.DBL
RRTMG_SW_DATA -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/RRTMG_SW_DATA
RRTMG_SW_DATA.DBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/RRTMG_SW_DATA.DBL
SOILPARM.TBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/SOILPARM.TBL
VEGPARM.TBL -> /home/smarshall/jedi/mpas-bundle/rel3/build/MPAS/core_atmosphere/VEGPARM.TBL
atmosphere_model -> /home/smarshall/jedi/mpas-bundle/rel3/build/bin/mpas_atmosphere
geovars.yaml -> /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/test/testinput/namelists/geovars.yaml
keptvars.yaml -> /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/test/testinput/namelists/keptvars.yaml
obsop_name_map.yaml -> /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/test/testinput/obsop_name_map.yaml
x1.40962.graph.info.part.128 -> graph/x1.40962.graph.info.part.128
x1.40962.graph.info.part.16 -> graph/x1.40962.graph.info.part.16
x1.40962.graph.info.part.2 -> graph/x1.40962.graph.info.part.2
x1.40962.graph.info.part.4 -> graph/x1.40962.graph.info.part.4
x1.40962.static.nc -> ../mesh/x1.40962/x1.40962.static.nc
2

I made a small bit of progress, but did not change the abort behavior.

I reduced the time window from 6 hour to 3 hour in test_hofx3d.yaml. This eliminated the time warning:
WARNING: Error: Field scalars has more than one time level, but no timeLevel argument given.

However, the message about specific_humidity and the abort are unchanged.

3

Hi @steve.marshall,

In NCAR/MMM, we have no experiance with the container you are using. But it seems that you have built the mpas-bundle 3.0.2 there. Did you successfully run all mpas-jedi ctest after building the code ?

Also, can you post your yaml (test_hofx3d.yaml) in this threads? Though I have no immediate idea, I can take a look at it.

Thank you,
BJ

4

Thank you for your willingness to help. I really appreciate it! :slightly_smiling_face:

You are correct that I built mpas-bundle-3.0.2 in the gnu container I mentioned in the first post.

First, regarding unit tests:
All the ctest succeeded except for about 2/3 of the tests in mpas-jedi. I looked into several and the problem stems from the same thing: the Geometry class in MPAS calls the MPI_finalize function in its destructor. That is fine for most normal mpas-jedi programs where the Geometry object is a singleton (one object per process). However, several unit test programs have multiple tests within them and each test constructs a Geometry object, does the test, and destroys the Geometry object. So in the second test in each such unit test program, the test fails when the second Geometry object is constructed with an error about calling a MPI function after MPI_finalize has been called.

Overall, I don’t think my problems with hofx are related to this. However, it is a significant problem with testability, and needs to be addressed. I can create another ticket with more details about that problem, if that would be helpful.

Second, I would like to provide all my config (not just test_hofx3d.yaml), as well as log files. However, the upload function seems to only accept image formats, not text formats like yaml or compressed tar files. Is there a good way to post text files?

For now, I’ll just copy the text of test_hofx3d.yaml into this message. This version uses a 3 hour time window centered on 2024-01-01 0Z, as I specified in my update to the original post.

time window:
  begin: '2023-12-31T22:30:00Z'
  length: PT3H
geometry:
  nml_file: "./namelist.hofx"
  streams_file: "./streams.hofx_new"
  deallocate non-da fields: false
  interpolation type: unstructured
state:
  state variables:
  - air_temperature
  - water_vapor_mixing_ratio_wrt_moist_air
  - eastward_wind
  - northward_wind
  - air_pressure_at_surface
    #  - cloud_liquid_water
    # - cloud_liquid_ice
    # - rain_water
    # - snow_water
    # - graupel
  - air_potential_temperature
  - dry_air_density
    # - u
    # - water_vapor_mixing_ratio_wrt_dry_air
  - air_pressure
    # - landmask
    # - seaice_fraction
    # - snowc
    # - skin_temperature_at_surface
    # - ivgtyp
    # - isltyp
    # - snowh
    # - vegetation_area_fraction
    # - eastward_wind_at_10m
    # - northward_wind_at_10m
    # - lai
    # - smois
    # - tslb
  filename: "/home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/120km_uniform_gfs/x1.40962.init.nc"
  date: '2024-01-01T00:00:00Z'
observations:
  observers:
  - obs space:
      name: GnssroBndNBAM
      obsdatain:
        engine:
          type: H5File
          obsfile: /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/input_obs/PIQ/piq_20250309/piq_vres_1p0/20240101/obs.gnssro.20240101T000000Z.nc4
        obsgrouping:
          group variables: [ sequenceNumber ]
          sort variable: impactHeightRO
          sort order: ascending
      obsdataout:
        engine:
          type: H5File
          obsfile: ./ioda.BndNBAM.20240101T000000Z.nc4
      simulated variables: [bendingAngle]
    obs operator:
      name: GnssroBndNBAM
      obs options:
        use_compress: 1
        sr_steps: 2
        vertlayer: full
        super_ref_qc: NBAM
        modeltop: 30 # model top height in km
        #nlevadd: 13 # extended model layers on top of the existing model levels
        #ngrd: 80 # number of integration grids for bending angle
5

@byoung-joo - The test_hofx3d.yaml references the state variable “water_vapor_mixing_ratio_wrt_moist_air”, while the error is about the variable “specific_humidity”. These variables are two different names for the same thing. Is there some subsystem of JEDI that creates a mapping between these two names for the same variable?

6

For reference, changing water_vapor_mixing_ratio_wrt_dry_air to specific_humidity in test_hofx3d.yaml causes a similar error, but one that happens earlier in the runtime of hofx. The program still aborts.

Instead of this warning:
→ vars_nlevels: specific_humidity not available in MPAS domain or self % templated_fields

I get this warning:
→ da_template_pool: specific_humidity not available in MPAS domain or geom % templated_fields

7

Thank you for attaching your yaml file.

I agree that the failing ctests seem to be a separate issue. I wonder if this is related to the workaround with the ESMF library, but I have no knowledge on it.

From the yaml, I noticed that the model variable names are not compatible with a mpas-bundle v3.0.2.

The model variable names in your yaml (such as water_vapor_mixing_ratio_wrt_moist_air) is more generic ones, which is introduced recently. You need to use the model variable names, compatible to your code version (i.e., v3.0.2). You can refer to mpas-jedi/test/testinput/3dvar_bumpcov.yaml at release/3.0.2 · JCSDA/mpas-jedi · GitHub . For example, spechum and uReconstructZonal can replace water_vapor_mixing_ratio_wrt_moist_air
and eastward_wind in your yaml.

There is no simple mapping between old and new names, because it also requires some changes in the mpas-jedi source code (see the link above).

@gibbsp or other admin team.
This user has an issue with uploading some files to forum. Can you take a look at this issue?
Thank you,

BJ

8

I changed variables to internal names suggested by @byoung-joo, referring to this test configuration.

state variables: [temperature, spechum, uReconstructZonal, uReconstructMeridional, surface_pressure,
theta, rho, u, qv, pressure, landmask, xice, snowc, skintemp, ivgtyp, isltyp,
snowh, vegfra, u10, v10, lai, smois, tslb, pressure_p]

This generated some more errors and ultimately seg faulted. Here is the end of log.atmosphere.0000.out:

-- Called MPAS_stream_mgr_get_property()
WARNING: Error: Field u has more than one time level, but no timeLevel argument given.
mpas_expand_string: inString = /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/120km_uniform_gfs/x1.40962.init.nc
mpas_expand_string: set curTime using timeStamp 2024-01-01_00:00:00
-- Called MPAS_stream_mgr_set_property()
-- Called MPAS_stream_mgr_read()
-- Handling read of stream background
** Building Filename
** when is: 2024-01-01_00:00:00
** ref_time is: 2024-01-01_00:00:00
** intv is: 000000000_00:00:00.000000000
** filetime start is: 2024-01-01_00:00:00
mpas_expand_string: inString = /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/120km_uniform_gfs/x1.40962.init.nc
mpas_expand_string: set curTime using timeStamp 2024-01-01_00:00:00
-- Stream filename is: /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/120km_uniform_gfs/x1.40962.init.nc
Is field 'pressure_p' active in stream 'background? T
WARNING: Variable pressure_p not in input file.
Is field 'rho' active in stream 'background? T
Is field 'scalars' active in stream 'background? T
WARNING: Variable qi not in input file.
WARNING: Variable qs not in input file.
WARNING: Variable qg not in input file.
Is field 'surface_pressure' active in stream 'background? T
Is field 'theta' active in stream 'background? T
Is field 'u' active in stream 'background? T
Is field 'uReconstructZonal' active in stream 'background? T
WARNING: Variable uReconstructZonal not in input file.
Is field 'uReconstructMeridional' active in stream 'background? T
WARNING: Variable uReconstructMeridional not in input file.
Seeking time of 2024-01-01_00:00:00
Seeked record is: 1 out of 1 with a time stamp of 2024-01-01_00:00:00 filename was /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/120km_uniform_gfs/x1.40962.init.nc
-- Exchange halo for pressure_p
-- Exchange halo for rho
-- Exchange halo for scalars
-- Exchange halo for surface_pressure
-- Exchange halo for theta
-- Exchange halo for u
-- Exchange halo for uReconstructZonal
-- Exchange halo for uReconstructMeridional
WARNING: Error: Field u has more than one time level, but no timeLevel argument given.
WARNING: Error: Field air_temperature not found in pool.
WARNING: Error: Field scalars has more than one time level, but no timeLevel argument given.
WARNING: Error: Field water_vapor_mixing_ratio_wrt_moist_air not found in pool.

Here is the stack trace

#0 0x000014b31c30bdf2 in mpas_fields_mod::update_diagnostic_fields (geom=0x55e812c91b78, subfields=0x55e842fb6950, ngrid=40962) at /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/src/mpasjedi/Fields/mpas_fields_mod.F90:517
#1 0x000014b31c30dfde in mpas_fields_mod::read_fields (self=..., f_conf=..., vdate=...) at /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/src/mpasjedi/Fields/mpas_fields_mod.F90:478
#2 0x000014b31c3fc9ad in mpas_state_interface_mod::mpas_state_read_file_c (c_key_state=1, c_conf=<error reading variable: Attempt to dereference a generic pointer.>,
c_dt=<error reading variable: Attempt to dereference a generic pointer.>) at /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/src/mpasjedi/State/mpas_state_interface_mod.F90:200
#3 0x000014b31c3f6daf in mpas::State::read (this=0x55e81fe28700, config=...) at /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/src/mpasjedi/State/State.cc:234
#4 0x000014b31c3f5906 in mpas::State::State (this=0x55e81fe28700, geom=..., config=...) at /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/src/mpasjedi/State/State.cc:57
#5 0x000055e7f5ac295a in oops::interface::State<mpas::Traits>::State (this=0x7ffe57ecb310, resol=..., config=...) at /home/smarshall/jedi/mpas-bundle/rel3/src/oops/src/oops/interface/State.h:140
#6 0x000055e7f5a9247f in oops::State<mpas::Traits>::State (this=0x7ffe57ecb310, resol=..., conf=...) at /home/smarshall/jedi/mpas-bundle/rel3/src/oops/src/oops/base/State.h:85
#7 0x000055e7f5a6674b in oops::HofX3D<mpas::Traits, ufo::ObsTraits>::execute (this=0x7ffe57ecbef0, fullConfig=..., validate=true) at /home/smarshall/jedi/mpas-bundle/rel3/src/oops/src/oops/runs/HofX3D.h:106
#8 0x000014b309591a51 in oops::Run::execute (this=0x7ffe57ecbf00, app=..., comm=...) at /home/smarshall/jedi/mpas-bundle/rel3/src/oops/src/oops/runs/Run.cc:186
#9 0x000055e7f5972242 in main (argc=2, argv=0x7ffe57ecc0c8) at /home/smarshall/jedi/mpas-bundle/rel3/src/mpas-jedi/src/mains/mpasHofX3D.cc:20

The crash is at this line of code in function update_diagnostic_fields() in source file mpas-jedi/src/mpasjedi/Fields/mpas_fields_mod.F90, attempting to convert water vapor mixing ratio to specific humidity:

call w_to_q( scalars % array(index_qv,:,1:ngrid) , specific_humidity % array(:,1:ngrid) )

In this case, we already have specific humidity as a native parameter, so I’m not sure why this is being done.

Running this through a debugger, and stopping at line 516, I can see both temperature and specific_humidity are NULL pointers before the calls to theta_to_temp and w_to_q in the code below. Temperature is still NULL at line 517, so it was not allocated by the theta_to_temp function.

I’m not sure what needs to be done to allocate those fields.

subroutine update_diagnostic_fields(geom, subFields, ngrid)
implicit none
type (mpas_geom), pointer, intent(in) :: geom
type (mpas_pool_type), pointer, intent(inout) :: subFields
integer, intent(in) :: ngrid
type (field2DReal), pointer :: theta, pressure, temperature, specific_humidity
type (field3DReal), pointer :: scalars
type (mpas_pool_type), pointer :: state
integer, pointer :: index_qv

!(1) copy all to subFields
call da_copy_all2sub_fields(geom, subFields)

!(2) diagnose temperature
!Special case: Convert theta and pressure to temperature
! Convert water vapor mixing ratio to specific humidity [ q = w / (1 + w) ]
!NOTE: This formula is somewhat different with MPAS one's (in physics, they use "exner")
! : If T diagnostic is added in, for example, subroutine atm_compute_output_diagnostics,
! : we need to include "exner" in stream_list.for.reading
call mpas_pool_get_field(geom % domain % blocklist % allFields, 'theta', theta)
call mpas_pool_get_field(geom % domain % blocklist % allFields, 'pressure', pressure)
call mpas_pool_get_field(subFields, 'air_temperature', temperature)
call mpas_pool_get_field(geom % domain % blocklist % allFields, 'scalars', scalars)
call mpas_pool_get_field(subFields, 'water_vapor_mixing_ratio_wrt_moist_air', specific_humidity)
call mpas_pool_get_subpool(geom % domain % blocklist % structs,'state',state)
call mpas_pool_get_dimension(state, 'index_qv', index_qv)

call theta_to_temp(theta % array(:,1:ngrid), pressure % array(:,1:ngrid), temperature % array(:,1:ngrid))
call w_to_q( scalars % array(index_qv,:,1:ngrid) , specific_humidity % array(:,1:ngrid) )

end subroutine update_diagnostic_fields
9

Hi @steve.marshall,

I am a bit confused with reading subroutine update_diagnostic_fields(...).
In the attached subroutine, it seems that the mpas-jedi repository may not use the version of v3.0.2. This is because there exist the generic model variable names, such as air_temperature or water_vapor_mixing_ratio_wrt_moist_air, which IS NOW used in develop branch.

Can you double check if your mpas-jedi repository uses the 3.0.2.mmm tag from GitHub - JCSDA/mpas-jedi: MPAS Interface for JEDI ? Or you may check the git hash as c9159fb.

Thank you,
BJ

10

@byoung-joo - You are correct. I am using different versions of mpas-jedi and mpas-jedi-data than what is specified in mpas-bundle, release/3.0.2 branch. I am using the develop branch for each of these packages.

Looking back at my notes, I’m not sure why I changed this, but it must have been an attempt to fix some problem. Usually, my notes are pretty complete, but I cannot find the reason for this one.

These are the two version changes I have in the top-level CMakeLists.txt in mpas-bundle (relative to the release/3.0.2 branch):

Original:
ecbuild_bundle( PROJECT mpas-jedi-data GIT “GitHub - JCSDA-internal/mpas-jedi-data: tier 1 test files for mpas-jedi” TAG 12cdc56 )
ecbuild_bundle( PROJECT mpas-jedi GIT “GitHub - JCSDA/mpas-jedi: MPAS Interface for JEDI” TAG 3.0.2.mmm )

Modified:
ecbuild_bundle( PROJECT mpas-jedi-data GIT “GitHub - JCSDA-internal/mpas-jedi-data: tier 1 test files for mpas-jedi” BRANCH develop )
ecbuild_bundle( PROJECT mpas-jedi GIT “GitHub - JCSDA/mpas-jedi: MPAS Interface for JEDI” BRANCH develop )

I have recloned the original mpas-bundle release, which uses the tag you specified, and I am rebuilding it now. Then I’ll test if I still see the same errors as before, or run into a different issue.

11

Running with a completely unmodifed version of mpas-bundle, release 3.0.2 branch, my test still aborts.

These are the final message emitted to stderr:

OOPS_TRACE[0] Geometry<MODEL>::variableSizes starting
 --> vars_nlevels: specific_humidity not available in MPAS domain or self % templated_fields
OOPS Ending   2025-05-20 23:26:26 (UTC+0000)
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD

It is not clear what is causing the abort. The application is not core dumping, even with ulimit -c unlimited set.

The end of log.atmosphere.0000.out has these WARNINGs, but nothing fatal:

 --- initialize NOAH LSM tables
     skipping over lutype = USGS
     landuse type = MODIFIED_IGBP_MODIS_NOAH found          20  categories
     end read VEGPARM.TBL
     input soil texture classification = STAS
     soil texture classification = STAS found          19  categories
     end read SOILPARM.TBL
     end read GENPARM.TBL
 --- end initialize NOAH LSM tables

 min/max of meshScalingDel2 = 1.00000000000000 1.00000000000000
 min/max of meshScalingDel4 = 1.00000000000000 1.00000000000000
WARNING: Variable pressure_p not in input file.
WARNING: Variable qi not in input file.
WARNING: Variable qs not in input file.
WARNING: Variable qg not in input file.
WARNING: Variable uReconstructZonal not in input file.
WARNING: Variable uReconstructMeridional not in input file.
12

Hi @steve.marshall ,

I have seen the error before; sometime related to “not-yet-implemented” configuration for new variable or sometimes a mistake in the configuration.

Can you check if the geovar.yaml file is linked/copied from mpas-jedi v3.0.2 (not from its develop branch) ?

If your ctest with mpas-jedi/test/testinput/hofx3d_nbam.yaml at release/3.0.2 · JCSDA/mpas-jedi · GitHub was successful, I believe that your test configuration should work in principle.

Thank you,
BJ

13

Thanks @byoung-joo . This was good advice. I checked the geovars.yaml I was using and it was different from the 3.0.2 version from mpas-jedi.

Unfortunately, mpasjedi-hofx3d.x is still aborting, and I don’t see any log messages indicating the configured forward operator is being run. If you have any further advice on how to proceed, I would be really grateful.

Core dump and stack trace
After changing the links to geovars.yaml, keptvars.yaml, and obsop_name_map.yaml to point to the correct mpas-jedi version, I reran my test. I got farther, but still aborted. However, this time it generated a core file. Here is the stack trace.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000014873c768859 in __GI_abort () at abort.c:79
#2  0x000014873c7d326e in __libc_message (action=action@entry=do_abort, fmt=0x7ffc3c402860 "", fmt@entry=0x14873c8fd298 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#3  0x000014873c7db2fc in malloc_printerr (str=str@entry=0x14873c8ff2a8 "corrupted size vs. prev_size in fastbins") at malloc.c:5347
#4  0x000014873c7dbacc in malloc_consolidate (av=av@entry=0x14873c932b80 <main_arena>) at malloc.c:4493
#5  0x000014873c7dcfe0 in _int_free (av=0x14873c932b80 <main_arena>, p=0x5653b8211280, have_lock=<optimized out>) at malloc.c:4400
#6  0x000014874d07b6ca in std::default_delete<eckit::Channel>::operator() (this=0x14874d1286c0 <eckit::LibEcKit::instance()::libeckit+224>, __ptr=<optimized out>)
    at /usr/include/c++/9/bits/unique_ptr.h:75
#7  std::unique_ptr<eckit::Channel, std::default_delete<eckit::Channel> >::~unique_ptr (this=0x14874d1286c0 <eckit::LibEcKit::instance()::libeckit+224>, 
    __in_chrg=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:292
#8  eckit::system::Library::~Library (this=0x14874d1285e0 <eckit::LibEcKit::instance()::libeckit>, __in_chrg=<optimized out>)
    at /tmp/root/spack-stage/spack-stage-eckit-1.24.5-ntzkqwn6jxizijmu57t7lgyjvvfy3bjf/spack-src/src/eckit/system/Library.cc:61
#9  0x000014873c78cfde in __cxa_finalize (d=0x14874d1281c0) at cxa_finalize.c:83
#10 0x000014874cf8f587 in __do_global_dtors_aux ()
    at /tmp/root/spack-stage/spack-stage-eckit-1.24.5-ntzkqwn6jxizijmu57t7lgyjvvfy3bjf/spack-src/src/eckit/system/Library.cc:224

The abort seems to be coming from some cleanup code that deallocates memory, with this message near the end of message written to stderr:

OOPS Ending   2025-05-22 12:31:21 (UTC+0000)
corrupted size vs. prev_size in fastbins

Progress of processing from the logs
The log information in log.atmosphere.0000.out show very similar warnings to my previous aborted runs, but there are a few more of WARNINGs, indicating the processing got at least a little farther. Also, the logs show a more complete exit where timing information is provided, even though the exit status of the program was 134. Here is the end of the log:

 min/max of meshScalingDel2 = 1.00000000000000 1.00000000000000
 min/max of meshScalingDel4 = 1.00000000000000 1.00000000000000
WARNING: Error: Field Time has more than one time level, but no timeLevel argument given.
WARNING: Error: Field u has more than one time level, but no timeLevel argument given.
WARNING: Variable pressure_p not in input file.
WARNING: Variable qi not in input file.
WARNING: Variable qs not in input file.
WARNING: Variable qg not in input file.
WARNING: Variable uReconstructZonal not in input file.
WARNING: Variable uReconstructMeridional not in input file.
WARNING: Error: Field scalars has more than one time level, but no timeLevel argument given.
WARNING: Error: Field u has more than one time level, but no timeLevel argument given.
WARNING: Error: Field scalars has more than one time level, but no timeLevel argument given.
WARNING: Error: Field u has more than one time level, but no timeLevel argument given.
WARNING: Error: Field w has more than one time level, but no timeLevel argument given.
WARNING: Error: Field w has more than one time level, but no timeLevel argument given.

 ********************************************************
    Finished running the atmosphere core
 ********************************************************


  Timer information:
     Globals are computed across all threads and processors

  Columns:
     total time: Global max of accumulated time spent in timer
     calls: Total number of times this timer was started / stopped.
     min: Global min of time spent in a single start / stop
     max: Global max of time spent in a single start / stop
     avg: Global max of average time spent in a single start / stop
     pct_tot: Percent of the timer at level 1
     pct_par: Percent of the parent timer (one level up)
     par_eff: Parallel efficiency, global average total time / global max total time


    timer_name                                            total       calls        min            max            avg      pct_tot   pct_par     par_eff
  1 total time                                          19.26860         1       19.26860       19.26860       19.26860   100.00       0.00       1.00
  2  initialize                                         14.38378         1       14.38378       14.38378       14.38378    74.65      74.65       1.00
  3   read_ICs                                           8.66910         1        8.66910        8.66910        8.66910    44.99      60.27       1.00

 -----------------------------------------
 Total log messages printed:
    Output messages =                  424
    Warning messages =                  19
    Error messages =                     0
    Critical error messages =            0
 -----------------------------------------
 Logging complete.  Closing file at 2025/05/22 12:01:24

Missing streams
The log also has some information about non-existent streams I have configured in streams.hofx_new. I’m not sure if these are relevant to the abort behavior.

Below are the relevant log messages. The input stream x1.40962.init.nc was created by mpas_init_atmosphere from a WRF intermediate file which was created from GFS GRIB files using the WRF ungrib utility. The background stream x1.40962.init.2024-01-01_00.00.00.nc is a copy of the input stream file. The files referenced by the other streams do not exist, and I’m not sure if I should omit the config entries, or if I should create files for them. If I need to create those files, I’m not sure how or what contents they should have.

 ** Validating streams


 Reading dimensions from input streams ...

   *** unable to open input file invariant.nc for stream 'invariant'

 ----- reading dimensions from stream 'input' using file x1.40962.init.nc
        nCells = 40962
        nEdges = 122880
        nVertices = 81920
        TWO = 2
        maxEdges = 10
        maxEdges2 = 20
        R3 = 3
        vertexDegree = 3
        nVertLevelsP1 = 56
        nVertLevels = 55
        FIFTEEN = 15
        nMonths = 12
        nSoilLevels = 4 
 
   *** unable to open input file restart.2024-01-01_00.00.00.nc for stream 'restart'
 
   *** unable to open input file x1.40962.AmB.2024-01-01_00.00.00.nc for stream 'iau'
 
   *** unable to open input file lbc.2024-01-01_00.00.00.nc for stream 'lbc_in'
 
   *** unable to open input file mpasout.2024-01-01_00.00.00.nc for stream 'da_state'
 
 ----- reading dimensions from stream 'background' using file x1.40962.init.2024-01-01_00.00.00.nc
        nVertLevels = 55
        nCells = 40962
        nEdges = 122880
 
 ----- done reading dimensions from input streams -----
14

@steve.marshall ,

The warning messages in log.atmosphere.0000.out and the missing streams in your streams file (streams.hofx_new) should be fine. But,

I noticed that pressure_p, uReconstructZonal, uReconstructMeridional are not available in your files. Are you using the MPAS initial file as the state for the hofx3d application ? I am not sure if it is documented somewhere, but we usually do not use the MPAS initial files for mpas-jedi. We usually use the MPAS files of da_state stream as mpas-jedi’s background state. → This may not be the reason for the encountered error, however.

Can you rerun the ctest with mpas-jedi/test/testinput/hofx3d_nbam.yaml at release/3.0.2 · JCSDA/mpas-jedi · GitHub ,which would be similar to your hofx configuration, and check if it goes through the end without similar error message?
The following command will run that single ctest. ctest -R test_mpasjedi_hofx3d_nbam

If the configuration is all correct, I wonder if this is somewhat related to ESMF…

Thank you,
BJ

15

Thank you @byoung-joo for the advice about running mpas-jedi programs using the da_state stream. How do I create an appropriate file for the da_state stream? I created my initialization file with the MPAS init_atmosphere_model program (called mpas_init_atmosphere within mpas-jedi). The input to init_atmosphere model was a file in the WRF intermediate format created by the WRF ungrib utility from a GFS data in GRIB2 format. I configured the stream called “input” to point to that file. I also made a copy of the same file and pointed the “background” stream to it. I would be grateful for suggestions on how I should generate input data and configure the streams for running mpas-jedi programs.

Regarding the test program, it crashes when I run it. It seems to run completely and without error until the cleanup step (__run_exit_handlers) where it fails while freeing up memory with this message: “corrupted size vs. prev_size in fastbins”.

The error output and the stack trace from the core dump are provided.

ctest --output-on-failure -R test_mpasjedi_hofx3d_nbam
Test project /home/smarshall/jedi/mpas-bundle/rel3.0.2/build_debug
    Start 2132: test_mpasjedi_hofx3d_nbam
1/1 Test #2132: test_mpasjedi_hofx3d_nbam ........***Failed    2.56 sec
OOPS Starting 2025-05-29 18:37:37 (UTC+0000)
[TestReference] Comparing to reference file: testoutput/hofx3d_nbam.ref
Relative float tolerance for tests : 1e-08
Absolute float tolerance for tests : 0
[TestReference] Saving Log output to: testoutput/hofx3d_nbam.run
[TestReference] Saving Test output to: testoutput/hofx3d_nbam.run.ref
Configuration input file is: testinput/hofx3d_nbam.yaml
Full configuration is:YAMLConfiguration[path=testinput/hofx3d_nbam.yaml, root={test => {float relative tolerance => 1e-08 , integer tolerance => 0 , reference filename => testoutput/hofx3d_nbam.ref , log output filename => testoutput/hofx3d_nbam.run , test output filename => testoutput/hofx3d_nbam.run.ref} , time window => {begin => 2018-04-14T21:00:00Z , length => PT6H} , geometry => {nml_file => ./Data/480km/namelist.atmosphere_2018041500 , streams_file => ./Data/480km/streams.atmosphere} , state => {state variables => (temperature,spechum,uReconstructZonal,uReconstructMeridional,surface_pressure,qc,qi,qr,qs,qg,theta,rho,u,qv,pressure,landmask,xice,snowc,skintemp,ivgtyp,isltyp,snowh,vegfra,u10,v10,lai,smois,tslb) , filename => ./Data/480km/bg/restart.2018-04-15_00.00.00.nc , date => 2018-04-15T00:00:00Z} , observations => {observers => ({obs space => {name => GnssroBndNBAM , obsdatain => {engine => {type => H5File , obsfile => Data/ufo/testinput_tier_1/gnssro_obs_2018041500_s.nc4}} , obsdataout => {engine => {type => H5File , obsfile => Data/os/obsout_hofx3d_gnssrobndnbam.nc4}} , simulated variables => (bendingAngle)} , obs operator => {name => GnssroBndNBAM , obs options => {use_compress => 1 , vertlayer => mass , modeltop => 30}}})}}]
OOPS_STATS ObjectCountHelper started.
OOPS_STATS Run start                                - Runtime:      0.33 sec,  Memory: total:   111.67 Mb, per task: min =   111.67 Mb, max =   111.67 Mb
Run: Starting oops::HofX3D<MPAS, UFO and IODA observations>
HofX3D observation window: TimeWindow: start = 2018-04-14T21:00:00Z, end = 2018-04-15T03:00:00Z, inclusive upper bound.

==> create geom
Test     : State: 
Test     :   Valid time: 2018-04-15T00:00:00Z
Test     :   Resolution: nCellsGlobal = 2562, nFields = 28
Test     : Fld=1  Min=1.9961034302148411e+02, Max=3.0524750469187848e+02, RMS=2.4396516847118451e+02 : temperature
Test     : Fld=2  Min=0.0000000000000000e+00, Max=1.8968370803990039e-02, RMS=4.6305151896304034e-03 : spechum
Test     : Fld=3  Min=-4.4161102375550811e+01, Max=8.3002884639199976e+01, RMS=1.7654622452843618e+01 : uReconstructZonal
Test     : Fld=4  Min=-4.5535885706116773e+01, Max=5.8607941256754437e+01, RMS=9.0017723000806011e+00 : uReconstructMeridional
Test     : Fld=5  Min=5.6985528898815101e+04, Max=1.0469254955758584e+05, RMS=9.8674507031394081e+04 : surface_pressure
Test     : Fld=6  Min=0.0000000000000000e+00, Max=4.7620185144607553e-04, RMS=3.5929903152108541e-05 : qc
Test     : Fld=7  Min=0.0000000000000000e+00, Max=1.4305271318894910e-04, RMS=7.3326340773610577e-06 : qi
Test     : Fld=8  Min=0.0000000000000000e+00, Max=2.2368046851574421e-04, RMS=6.2856631162687630e-06 : qr
Test     : Fld=9  Min=0.0000000000000000e+00, Max=9.5083864732163734e-04, RMS=2.0385194681801095e-05 : qs
Test     : Fld=10  Min=0.0000000000000000e+00, Max=3.9926870323757948e-04, RMS=7.4792927573773849e-06 : qg
Test     : Fld=11  Min=2.5182402460806750e+02, Max=7.2533093936813066e+02, RMS=4.4103127630366970e+02 : theta
Test     : Fld=12  Min=2.4838976284701046e-02, Max=1.3339283306870393e+00, RMS=6.2017444683729051e-01 : rho
Test     : Fld=13  Min=-9.0229125044006892e+01, Max=8.0580849609313418e+01, RMS=1.4345978280747147e+01 : u
Test     : Fld=14  Min=0.0000000000000000e+00, Max=1.9335126655942059e-02, RMS=4.6933265629772857e-03 : qv
Test     : Fld=15  Min=1.4722705745925159e+03, Max=9.8066892701442659e+04, RMS=4.9313616641863038e+04 : pressure
Test     : Fld=16  Min=0.0000000000000000e+00, Max=1.0000000000000000e+00, RMS=5.0369439206058919e-01 : landmask
Test     : Fld=17  Min=0.0000000000000000e+00, Max=1.0000000000000000e+00, RMS=2.2176638128637186e-01 : xice
Test     : Fld=18  Min=0.0000000000000000e+00, Max=1.0000000000000000e+00, RMS=3.3369892680607488e-01 : snowc
Test     : Fld=19  Min=2.1225740203621450e+02, Max=3.1557139113785132e+02, RMS=2.8841407340681502e+02 : skintemp
Test     : Fld=20  Min=2.0000000000000000e+00, Max=2.4000000000000000e+01, RMS=1.6129501290050165e+01 : ivgtyp
Test     : Fld=21  Min=1.0000000000000000e+00, Max=1.6000000000000000e+01, RMS=1.2913269805916933e+01 : isltyp
Test     : Fld=22  Min=0.0000000000000000e+00, Max=1.9652943056661127e+00, RMS=2.2509902783740057e-01 : snowh
Test     : Fld=23  Min=0.0000000000000000e+00, Max=8.7847616781386378e+01, RMS=1.7428710491483386e+01 : vegfra
Test     : Fld=24  Min=-2.6260571889495036e+01, Max=2.1062703748152455e+01, RMS=6.0695126263221821e+00 : u10
Test     : Fld=25  Min=-2.0220716817104563e+01, Max=2.2314186411640573e+01, RMS=5.2369952425733164e+00 : v10
Test     : Fld=26  Min=0.0000000000000000e+00, Max=6.4799321595370545e+00, RMS=1.3338709394557591e+00 : lai
Test     : Fld=27  Min=2.0000000000000000e-02, Max=1.0000000000000000e+00, RMS=8.9039092466516556e-01 : smois
Test     : Fld=28  Min=2.1861984093432565e+02, Max=3.1450799148116488e+02, RMS=2.7519634154960795e+02 : tslb
WARNING: the reference/offset style datetime will be deprecated soon.
WARNING: Please update your datetime data to the epoch style representation using the new variable: MetaData/dateTime.
GnssroBndNBAM: read database from Data/ufo/testinput_tier_1/gnssro_obs_2018041500_s.nc4 (io pool size: 1)
GnssroBndNBAM processed vars: 2 Variables: atmosphericRefractivity, bendingAngle
GnssroBndNBAM assimilated vars: 1 Variables: bendingAngle
No bias-correction is performed for this ObsSpace.
ObsErrorDiag covariance updated 20
Observer::finalize QC = 
GnssroBndNBAM atmosphericRefractivity nlocs = 20, nobs = 20, min = 0, max = 0, avg = 0
GnssroBndNBAM bendingAngle nlocs = 20, nobs = 20, min = 0, max = 0, avg = 0
H(x): 
GnssroBndNBAM nobs= 8 Min=0.000590765, Max=0.00835701, RMS=0.00410091

End H(x)
Test     : H(x): 
Test     : GnssroBndNBAM nobs= 8 Min=5.9076454470258254e-04, Max=8.3570141100731878e-03, RMS=4.1009103726672947e-03

Test     : End H(x)
GnssroBndNBAM: save database to Data/os/obsout_hofx3d_gnssrobndnbam.nc4 (io pool size: 1)
==> destruct MPAS corelist and domain:  0

Run: Finishing oops::HofX3D<MPAS, UFO and IODA observations>
OOPS_STATS  
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS --------------------------- Object counts ----------------------------------------
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS                                      Total  Simult. Remain    Avg (Mb)    HWM (Mb)
OOPS_STATS ioda::ObsDataVector             :        5       5        
OOPS_STATS ioda::ObsVector                 :        5       5        
OOPS_STATS mpas::Geometry                  :        1       1        
OOPS_STATS mpas::State                     :        3       3        
OOPS_STATS mpas::VarChaModel2GeoVars       :        1       1        
OOPS_STATS oops::GeoVaLs                   :        1       1        
OOPS_STATS oops::Geometry                  :        1       1        
OOPS_STATS oops::GetValues                 :        1       1        
OOPS_STATS oops::ObsAuxControl             :        1       1        
OOPS_STATS oops::ObsDataVector             :        5       5        
OOPS_STATS oops::ObsDiagnostics            :        1       1        
OOPS_STATS oops::ObsError                  :        1       1        
OOPS_STATS oops::ObsOperator               :        1       1        
OOPS_STATS oops::ObsSpace                  :        1       1                6.28        6.28
OOPS_STATS oops::ObsSpaces                 :        1       1        
OOPS_STATS oops::ObsVector                 :        5       5                0.00        0.00
OOPS_STATS oops::SampledLocations          :        2       2        
OOPS_STATS oops::State                     :        2       2                2.38        4.75
OOPS_STATS oops::UnstructuredInterpolator  :        1       1        
OOPS_STATS oops::VariableChange            :        1       1        
OOPS_STATS ufo::GeoVaLs                    :        2       2        
OOPS_STATS ufo::ObsBias                    :        1       1        
OOPS_STATS ufo::ObsGnssroBndNBAM           :        1       1        
OOPS_STATS ufo::SampledLocations           :        1       1        
OOPS_STATS ----------------------------- Object counts --------------------------------------
OOPS_STATS  
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS Name                                                :   total (ms)   count    time/call (ms)
OOPS_STATS mpas::VarChaModel2GeoVars::VarChaModel2GeoVars      :         0.01       1            0.0078
OOPS_STATS mpas::VarChaModel2GeoVars::changeVar                :         0.84       1            0.8412
OOPS_STATS mpas::VarChaModel2GeoVars::~VarChaModel2GeoVars     :         0.00       1            0.0019
OOPS_STATS oops::GeoVaLs::GeoVaLs                              :         0.09       1            0.0943
OOPS_STATS oops::GeoVaLs::fill                                 :         0.76       5            0.1523
OOPS_STATS oops::GeoVaLs::~GeoVaLs                             :         0.00       1            0.0025
OOPS_STATS oops::Geometry::Geometry                            :       753.11       1          753.1079
OOPS_STATS oops::Geometry::variableSizes                       :         1.11       3            0.3693
OOPS_STATS oops::Geometry::~Geometry                           :         5.00       1            4.9961
OOPS_STATS oops::GeometryData::containingTriangleAndBarycentricCoords:         0.94      20            0.0470
OOPS_STATS oops::GeometryData::setGlobalTree                   :         5.30       1            5.2986
OOPS_STATS oops::GeometryData::setLocalTree                    :         8.35       1            8.3511
OOPS_STATS oops::GeometryData::setMeshAndTriangulation         :         0.55       1            0.5521
OOPS_STATS oops::GetValues::GetValues                          :         1.34       1            1.3417
OOPS_STATS oops::GetValues::fillGeoVaLs                        :         0.79       1            0.7920
OOPS_STATS oops::GetValues::finalize                           :         0.04       1            0.0426
OOPS_STATS oops::GetValues::process                            :         0.14       1            0.1411
OOPS_STATS oops::ObsAuxControl::ObsAuxControl                  :        19.92       1           19.9168
OOPS_STATS oops::ObsAuxControl::requiredHdiagnostics           :         0.00       1            0.0003
OOPS_STATS oops::ObsAuxControl::requiredVars                   :         0.00       2            0.0005
OOPS_STATS oops::ObsAuxControl::~ObsAuxControl                 :         0.02       1            0.0193
OOPS_STATS oops::ObsDataVector::ObsDataVector                  :         0.33       5            0.0657
OOPS_STATS oops::ObsDataVector::mask                           :         0.00       1            0.0011
OOPS_STATS oops::ObsDataVector::operator=                      :         0.01       4            0.0035
OOPS_STATS oops::ObsDataVector::print                          :         0.11       1            0.1103
OOPS_STATS oops::ObsDataVector::save                           :         0.83       2            0.4127
OOPS_STATS oops::ObsDataVector::~ObsDataVector                 :         0.01       5            0.0020
OOPS_STATS oops::ObsDiagnostics::ObsDiagnostics                :         0.02       1            0.0167
OOPS_STATS oops::ObsDiagnostics::~ObsDiagnostics               :         0.01       1            0.0084
OOPS_STATS oops::ObsError::ObsErrors                           :         0.26       1            0.2590
OOPS_STATS oops::ObsError::obserrors                           :         0.01       1            0.0094
OOPS_STATS oops::ObsError::update                              :         0.04       1            0.0410
OOPS_STATS oops::ObsError::~ObsError                           :         0.01       1            0.0086
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::ObsOperator       :        37.20       1           37.2030
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::computeReducedVars:         0.05       1            0.0532
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::locations         :         2.21       1            2.2103
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::requiredVars      :         0.02       1            0.0176
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::simulateObs       :         1.54       1            1.5391
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::~ObsOperator      :         0.03       1            0.0336
OOPS_STATS oops::ObsSpace::ObsSpace                            :        56.12       1           56.1165
OOPS_STATS oops::ObsSpace::obsvariables                        :         0.00       3            0.0003
OOPS_STATS oops::ObsSpace::save                                :        42.93       1           42.9260
OOPS_STATS oops::ObsSpace::~ObsSpace                           :         0.24       1            0.2398
OOPS_STATS oops::ObsVector::ObsVector                          :         0.15       5            0.0300
OOPS_STATS oops::ObsVector::invert                             :         0.00       2            0.0006
OOPS_STATS oops::ObsVector::operator*=                         :         0.00       2            0.0016
OOPS_STATS oops::ObsVector::operator=                          :         0.02       4            0.0041
OOPS_STATS oops::ObsVector::save                               :         0.43       2            0.2174
OOPS_STATS oops::ObsVector::zero                               :         0.00       1            0.0010
OOPS_STATS oops::ObsVector::~ObsVector                         :         0.02       5            0.0032
OOPS_STATS oops::Parameters::deserialize                       :         0.78      13            0.0598
OOPS_STATS oops::Parameters::serialize                         :         0.23      10            0.0234
OOPS_STATS oops::Parameters::validate                          :        75.22       6           12.5369
OOPS_STATS oops::SampledLocations::SampledLocations            :         0.00       1            0.0013
OOPS_STATS oops::SampledLocations::latitudes                   :         0.00       1            0.0002
OOPS_STATS oops::SampledLocations::longitudes                  :         0.00       1            0.0001
OOPS_STATS oops::SampledLocations::times                       :         0.00       1            0.0001
OOPS_STATS oops::SampledLocations::~SampledLocations           :         0.17       2            0.0844
OOPS_STATS oops::State::State                                  :        27.64       2           13.8218
OOPS_STATS oops::State::print                                  :         1.34       1            1.3416
OOPS_STATS oops::State::toFieldSet                             :         0.34       1            0.3404
OOPS_STATS oops::State::~State                                 :         0.07       2            0.0365
OOPS_STATS oops::UnstructuredInterpolator::UnstructuredInterpolator:         1.21       1            1.2057
OOPS_STATS oops::UnstructuredInterpolator::apply               :         0.10       1            0.1049
OOPS_STATS oops::VariableChange::VariableChange                :         0.17       1            0.1656
OOPS_STATS oops::VariableChange::changeVar                     :         2.28       1            2.2762
OOPS_STATS oops::VariableChange::~VariableChange               :         0.01       1            0.0124
OOPS_STATS util::Timers::Total                                 :       983.22       1          983.2222
OOPS_STATS util::Timers::measured                              :       975.30       1          975.3047
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------

OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS ---------------------------------- Parallel Timing Statistics (   1 MPI tasks) -----------------------------------
OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS Name                                                :     min (ms)    max (ms)    avg (ms)     % total   imbal (%)
OOPS_STATS oops::Geometry::Geometry                            :       753.11      753.11      753.11       76.60        0.00
OOPS_STATS oops::Geometry::variableSizes                       :         1.11        1.11        1.11        0.11        0.00
OOPS_STATS oops::Geometry::~Geometry                           :         5.00        5.00        5.00        0.51        0.00
OOPS_STATS oops::GeometryData::setGlobalTree                   :         5.30        5.30        5.30        0.54        0.00
OOPS_STATS oops::GeometryData::setLocalTree                    :         8.35        8.35        8.35        0.85        0.00
OOPS_STATS oops::GetValues::GetValues                          :         1.34        1.34        1.34        0.14        0.00
OOPS_STATS oops::ObsAuxControl::ObsAuxControl                  :        19.92       19.92       19.92        2.03        0.00
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::ObsOperator       :        37.20       37.20       37.20        3.78        0.00
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::locations         :         2.21        2.21        2.21        0.22        0.00
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::simulateObs       :         1.54        1.54        1.54        0.16        0.00
OOPS_STATS oops::ObsSpace::ObsSpace                            :        56.12       56.12       56.12        5.71        0.00
OOPS_STATS oops::ObsSpace::save                                :        42.93       42.93       42.93        4.37        0.00
OOPS_STATS oops::Parameters::validate                          :        75.22       75.22       75.22        7.65        0.00
OOPS_STATS oops::State::State                                  :        27.64       27.64       27.64        2.81        0.00
OOPS_STATS oops::State::print                                  :         1.34        1.34        1.34        0.14        0.00
OOPS_STATS oops::UnstructuredInterpolator::UnstructuredInterpolator:         1.21        1.21        1.21        0.12        0.00
OOPS_STATS oops::VariableChange::changeVar                     :         2.28        2.28        2.28        0.23        0.00
OOPS_STATS util::Timers::Total                                 :       983.22      983.22      983.22      100.00        0.00
OOPS_STATS util::Timers::measured                              :       975.30      975.30      975.30       99.19        0.00
OOPS_STATS ---------------------------------- Parallel Timing Statistics (   1 MPI tasks) -----------------------------------

OOPS_STATS Run end                                  - Runtime:      1.31 sec,  Memory: total:   298.14 Mb, per task: min =   298.14 Mb, max =   298.14 Mb
Run: Finishing oops::HofX3D<MPAS, UFO and IODA observations> with status = 0
[TestReference] Comparison is done
OOPS Ending   2025-05-29 18:37:38 (UTC+0000)
corrupted size vs. prev_size in fastbins
[f851876e8e90:01127] *** Process received signal ***
[f851876e8e90:01127] Signal: Aborted (6)
[f851876e8e90:01127] Signal code:  (-6)
(gdb) where
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000014ce26968859 in __GI_abort () at abort.c:79
#2  0x000014ce269d326e in __libc_message (action=action@entry=do_abort, fmt=0x7ffe6f461110 "P7\370\066\316\024", 
    fmt@entry=0x14ce26afd298 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#3  0x000014ce269db2fc in malloc_printerr (str=str@entry=0x14ce26aff2a8 "corrupted size vs. prev_size in fastbins")
    at malloc.c:5347
#4  0x000014ce269dbacc in malloc_consolidate (av=av@entry=0x14ce26b32b80 <main_arena>) at malloc.c:4493
#5  0x000014ce269dcfe0 in _int_free (av=0x14ce26b32b80 <main_arena>, p=0x558f996292f0, have_lock=<optimized out>)
    at malloc.c:4400
#6  0x000014ce2698c953 in __run_exit_handlers (status=0, listp=0x14ce26b32718 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:124
#7  0x000014ce2698ca60 in __GI_exit (status=<optimized out>) at exit.c:139
#8  0x000014ce2696a08a in __libc_start_main (main=0x558f914e41b3 <main(int, char**)>, argc=2, argv=0x7ffe6f461428, 
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe6f461418)
    at ../csu/libc-start.c:342
#9  0x0000558f914e061e in _start ()
16

From the log file, I see that it actually calculated the hofx values. I.e.,

Test     : H(x): 
Test     : GnssroBndNBAM nobs= 8 Min=5.9076454470258254e-04, Max=8.3570141100731878e-03, RMS=4.1009103726672947e-03

Test     : End H(x)

with an indication of successful OOPS application

Run: Finishing oops::HofX3D<MPAS, UFO and IODA observations> with status = 0

So, I wonder if the error at the very end of execution is related to the ESMF, which I don’t know much about.

In your own testing, can you check similar lines from the log file OR the contents of obsdataout file (i.e., ioda.BndNBAM.20240101T000000Z.nc4) ? The hofx group from ioda.BndNBAM.20240101T000000Z.nc4 would be the major outcome from that application.

How do I create an appropriate file for the da_state stream?

Usually we initialize the MPAS initial file at previous 6 hours (T-6 hr), run the MPAS model for 6 hour with writing out the da_state stream file.
In your streams file for MPAS model forecast, you can add the following section:

<immutable_stream name="da_state"
                  type="output"
                  precision="single"
                  clobber_mode="truncate"
                  filename_template="mpasout.$Y-$M-$D_$h.$m.$s.nc"
                  packages="jedi_da"
                  io_type="pnetcdf,cdf5"
                  output_interval="6:00:00" />

And the mpasout.$Y-$M-$D_$h.$m.$s.nc can be used as your background file in test_hofx3d.yaml.

We are preparing more of these procedures in the upcoming tutorial.

Thank you,
BJ