Thank you @byoung-joo for the advice about running mpas-jedi programs using the da_state stream. How do I create an appropriate file for the da_state stream? I created my initialization file with the MPAS init_atmosphere_model program (called mpas_init_atmosphere within mpas-jedi). The input to init_atmosphere model was a file in the WRF intermediate format created by the WRF ungrib utility from a GFS data in GRIB2 format. I configured the stream called “input” to point to that file. I also made a copy of the same file and pointed the “background” stream to it. I would be grateful for suggestions on how I should generate input data and configure the streams for running mpas-jedi programs.
Regarding the test program, it crashes when I run it. It seems to run completely and without error until the cleanup step (__run_exit_handlers) where it fails while freeing up memory with this message: “corrupted size vs. prev_size in fastbins”.
The error output and the stack trace from the core dump are provided.
ctest --output-on-failure -R test_mpasjedi_hofx3d_nbam
Test project /home/smarshall/jedi/mpas-bundle/rel3.0.2/build_debug
Start 2132: test_mpasjedi_hofx3d_nbam
1/1 Test #2132: test_mpasjedi_hofx3d_nbam ........***Failed 2.56 sec
OOPS Starting 2025-05-29 18:37:37 (UTC+0000)
[TestReference] Comparing to reference file: testoutput/hofx3d_nbam.ref
Relative float tolerance for tests : 1e-08
Absolute float tolerance for tests : 0
[TestReference] Saving Log output to: testoutput/hofx3d_nbam.run
[TestReference] Saving Test output to: testoutput/hofx3d_nbam.run.ref
Configuration input file is: testinput/hofx3d_nbam.yaml
Full configuration is:YAMLConfiguration[path=testinput/hofx3d_nbam.yaml, root={test => {float relative tolerance => 1e-08 , integer tolerance => 0 , reference filename => testoutput/hofx3d_nbam.ref , log output filename => testoutput/hofx3d_nbam.run , test output filename => testoutput/hofx3d_nbam.run.ref} , time window => {begin => 2018-04-14T21:00:00Z , length => PT6H} , geometry => {nml_file => ./Data/480km/namelist.atmosphere_2018041500 , streams_file => ./Data/480km/streams.atmosphere} , state => {state variables => (temperature,spechum,uReconstructZonal,uReconstructMeridional,surface_pressure,qc,qi,qr,qs,qg,theta,rho,u,qv,pressure,landmask,xice,snowc,skintemp,ivgtyp,isltyp,snowh,vegfra,u10,v10,lai,smois,tslb) , filename => ./Data/480km/bg/restart.2018-04-15_00.00.00.nc , date => 2018-04-15T00:00:00Z} , observations => {observers => ({obs space => {name => GnssroBndNBAM , obsdatain => {engine => {type => H5File , obsfile => Data/ufo/testinput_tier_1/gnssro_obs_2018041500_s.nc4}} , obsdataout => {engine => {type => H5File , obsfile => Data/os/obsout_hofx3d_gnssrobndnbam.nc4}} , simulated variables => (bendingAngle)} , obs operator => {name => GnssroBndNBAM , obs options => {use_compress => 1 , vertlayer => mass , modeltop => 30}}})}}]
OOPS_STATS ObjectCountHelper started.
OOPS_STATS Run start - Runtime: 0.33 sec, Memory: total: 111.67 Mb, per task: min = 111.67 Mb, max = 111.67 Mb
Run: Starting oops::HofX3D<MPAS, UFO and IODA observations>
HofX3D observation window: TimeWindow: start = 2018-04-14T21:00:00Z, end = 2018-04-15T03:00:00Z, inclusive upper bound.
==> create geom
Test : State:
Test : Valid time: 2018-04-15T00:00:00Z
Test : Resolution: nCellsGlobal = 2562, nFields = 28
Test : Fld=1 Min=1.9961034302148411e+02, Max=3.0524750469187848e+02, RMS=2.4396516847118451e+02 : temperature
Test : Fld=2 Min=0.0000000000000000e+00, Max=1.8968370803990039e-02, RMS=4.6305151896304034e-03 : spechum
Test : Fld=3 Min=-4.4161102375550811e+01, Max=8.3002884639199976e+01, RMS=1.7654622452843618e+01 : uReconstructZonal
Test : Fld=4 Min=-4.5535885706116773e+01, Max=5.8607941256754437e+01, RMS=9.0017723000806011e+00 : uReconstructMeridional
Test : Fld=5 Min=5.6985528898815101e+04, Max=1.0469254955758584e+05, RMS=9.8674507031394081e+04 : surface_pressure
Test : Fld=6 Min=0.0000000000000000e+00, Max=4.7620185144607553e-04, RMS=3.5929903152108541e-05 : qc
Test : Fld=7 Min=0.0000000000000000e+00, Max=1.4305271318894910e-04, RMS=7.3326340773610577e-06 : qi
Test : Fld=8 Min=0.0000000000000000e+00, Max=2.2368046851574421e-04, RMS=6.2856631162687630e-06 : qr
Test : Fld=9 Min=0.0000000000000000e+00, Max=9.5083864732163734e-04, RMS=2.0385194681801095e-05 : qs
Test : Fld=10 Min=0.0000000000000000e+00, Max=3.9926870323757948e-04, RMS=7.4792927573773849e-06 : qg
Test : Fld=11 Min=2.5182402460806750e+02, Max=7.2533093936813066e+02, RMS=4.4103127630366970e+02 : theta
Test : Fld=12 Min=2.4838976284701046e-02, Max=1.3339283306870393e+00, RMS=6.2017444683729051e-01 : rho
Test : Fld=13 Min=-9.0229125044006892e+01, Max=8.0580849609313418e+01, RMS=1.4345978280747147e+01 : u
Test : Fld=14 Min=0.0000000000000000e+00, Max=1.9335126655942059e-02, RMS=4.6933265629772857e-03 : qv
Test : Fld=15 Min=1.4722705745925159e+03, Max=9.8066892701442659e+04, RMS=4.9313616641863038e+04 : pressure
Test : Fld=16 Min=0.0000000000000000e+00, Max=1.0000000000000000e+00, RMS=5.0369439206058919e-01 : landmask
Test : Fld=17 Min=0.0000000000000000e+00, Max=1.0000000000000000e+00, RMS=2.2176638128637186e-01 : xice
Test : Fld=18 Min=0.0000000000000000e+00, Max=1.0000000000000000e+00, RMS=3.3369892680607488e-01 : snowc
Test : Fld=19 Min=2.1225740203621450e+02, Max=3.1557139113785132e+02, RMS=2.8841407340681502e+02 : skintemp
Test : Fld=20 Min=2.0000000000000000e+00, Max=2.4000000000000000e+01, RMS=1.6129501290050165e+01 : ivgtyp
Test : Fld=21 Min=1.0000000000000000e+00, Max=1.6000000000000000e+01, RMS=1.2913269805916933e+01 : isltyp
Test : Fld=22 Min=0.0000000000000000e+00, Max=1.9652943056661127e+00, RMS=2.2509902783740057e-01 : snowh
Test : Fld=23 Min=0.0000000000000000e+00, Max=8.7847616781386378e+01, RMS=1.7428710491483386e+01 : vegfra
Test : Fld=24 Min=-2.6260571889495036e+01, Max=2.1062703748152455e+01, RMS=6.0695126263221821e+00 : u10
Test : Fld=25 Min=-2.0220716817104563e+01, Max=2.2314186411640573e+01, RMS=5.2369952425733164e+00 : v10
Test : Fld=26 Min=0.0000000000000000e+00, Max=6.4799321595370545e+00, RMS=1.3338709394557591e+00 : lai
Test : Fld=27 Min=2.0000000000000000e-02, Max=1.0000000000000000e+00, RMS=8.9039092466516556e-01 : smois
Test : Fld=28 Min=2.1861984093432565e+02, Max=3.1450799148116488e+02, RMS=2.7519634154960795e+02 : tslb
WARNING: the reference/offset style datetime will be deprecated soon.
WARNING: Please update your datetime data to the epoch style representation using the new variable: MetaData/dateTime.
GnssroBndNBAM: read database from Data/ufo/testinput_tier_1/gnssro_obs_2018041500_s.nc4 (io pool size: 1)
GnssroBndNBAM processed vars: 2 Variables: atmosphericRefractivity, bendingAngle
GnssroBndNBAM assimilated vars: 1 Variables: bendingAngle
No bias-correction is performed for this ObsSpace.
ObsErrorDiag covariance updated 20
Observer::finalize QC =
GnssroBndNBAM atmosphericRefractivity nlocs = 20, nobs = 20, min = 0, max = 0, avg = 0
GnssroBndNBAM bendingAngle nlocs = 20, nobs = 20, min = 0, max = 0, avg = 0
H(x):
GnssroBndNBAM nobs= 8 Min=0.000590765, Max=0.00835701, RMS=0.00410091
End H(x)
Test : H(x):
Test : GnssroBndNBAM nobs= 8 Min=5.9076454470258254e-04, Max=8.3570141100731878e-03, RMS=4.1009103726672947e-03
Test : End H(x)
GnssroBndNBAM: save database to Data/os/obsout_hofx3d_gnssrobndnbam.nc4 (io pool size: 1)
==> destruct MPAS corelist and domain: 0
Run: Finishing oops::HofX3D<MPAS, UFO and IODA observations>
OOPS_STATS
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS --------------------------- Object counts ----------------------------------------
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS Total Simult. Remain Avg (Mb) HWM (Mb)
OOPS_STATS ioda::ObsDataVector : 5 5
OOPS_STATS ioda::ObsVector : 5 5
OOPS_STATS mpas::Geometry : 1 1
OOPS_STATS mpas::State : 3 3
OOPS_STATS mpas::VarChaModel2GeoVars : 1 1
OOPS_STATS oops::GeoVaLs : 1 1
OOPS_STATS oops::Geometry : 1 1
OOPS_STATS oops::GetValues : 1 1
OOPS_STATS oops::ObsAuxControl : 1 1
OOPS_STATS oops::ObsDataVector : 5 5
OOPS_STATS oops::ObsDiagnostics : 1 1
OOPS_STATS oops::ObsError : 1 1
OOPS_STATS oops::ObsOperator : 1 1
OOPS_STATS oops::ObsSpace : 1 1 6.28 6.28
OOPS_STATS oops::ObsSpaces : 1 1
OOPS_STATS oops::ObsVector : 5 5 0.00 0.00
OOPS_STATS oops::SampledLocations : 2 2
OOPS_STATS oops::State : 2 2 2.38 4.75
OOPS_STATS oops::UnstructuredInterpolator : 1 1
OOPS_STATS oops::VariableChange : 1 1
OOPS_STATS ufo::GeoVaLs : 2 2
OOPS_STATS ufo::ObsBias : 1 1
OOPS_STATS ufo::ObsGnssroBndNBAM : 1 1
OOPS_STATS ufo::SampledLocations : 1 1
OOPS_STATS ----------------------------- Object counts --------------------------------------
OOPS_STATS
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS Name : total (ms) count time/call (ms)
OOPS_STATS mpas::VarChaModel2GeoVars::VarChaModel2GeoVars : 0.01 1 0.0078
OOPS_STATS mpas::VarChaModel2GeoVars::changeVar : 0.84 1 0.8412
OOPS_STATS mpas::VarChaModel2GeoVars::~VarChaModel2GeoVars : 0.00 1 0.0019
OOPS_STATS oops::GeoVaLs::GeoVaLs : 0.09 1 0.0943
OOPS_STATS oops::GeoVaLs::fill : 0.76 5 0.1523
OOPS_STATS oops::GeoVaLs::~GeoVaLs : 0.00 1 0.0025
OOPS_STATS oops::Geometry::Geometry : 753.11 1 753.1079
OOPS_STATS oops::Geometry::variableSizes : 1.11 3 0.3693
OOPS_STATS oops::Geometry::~Geometry : 5.00 1 4.9961
OOPS_STATS oops::GeometryData::containingTriangleAndBarycentricCoords: 0.94 20 0.0470
OOPS_STATS oops::GeometryData::setGlobalTree : 5.30 1 5.2986
OOPS_STATS oops::GeometryData::setLocalTree : 8.35 1 8.3511
OOPS_STATS oops::GeometryData::setMeshAndTriangulation : 0.55 1 0.5521
OOPS_STATS oops::GetValues::GetValues : 1.34 1 1.3417
OOPS_STATS oops::GetValues::fillGeoVaLs : 0.79 1 0.7920
OOPS_STATS oops::GetValues::finalize : 0.04 1 0.0426
OOPS_STATS oops::GetValues::process : 0.14 1 0.1411
OOPS_STATS oops::ObsAuxControl::ObsAuxControl : 19.92 1 19.9168
OOPS_STATS oops::ObsAuxControl::requiredHdiagnostics : 0.00 1 0.0003
OOPS_STATS oops::ObsAuxControl::requiredVars : 0.00 2 0.0005
OOPS_STATS oops::ObsAuxControl::~ObsAuxControl : 0.02 1 0.0193
OOPS_STATS oops::ObsDataVector::ObsDataVector : 0.33 5 0.0657
OOPS_STATS oops::ObsDataVector::mask : 0.00 1 0.0011
OOPS_STATS oops::ObsDataVector::operator= : 0.01 4 0.0035
OOPS_STATS oops::ObsDataVector::print : 0.11 1 0.1103
OOPS_STATS oops::ObsDataVector::save : 0.83 2 0.4127
OOPS_STATS oops::ObsDataVector::~ObsDataVector : 0.01 5 0.0020
OOPS_STATS oops::ObsDiagnostics::ObsDiagnostics : 0.02 1 0.0167
OOPS_STATS oops::ObsDiagnostics::~ObsDiagnostics : 0.01 1 0.0084
OOPS_STATS oops::ObsError::ObsErrors : 0.26 1 0.2590
OOPS_STATS oops::ObsError::obserrors : 0.01 1 0.0094
OOPS_STATS oops::ObsError::update : 0.04 1 0.0410
OOPS_STATS oops::ObsError::~ObsError : 0.01 1 0.0086
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::ObsOperator : 37.20 1 37.2030
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::computeReducedVars: 0.05 1 0.0532
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::locations : 2.21 1 2.2103
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::requiredVars : 0.02 1 0.0176
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::simulateObs : 1.54 1 1.5391
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::~ObsOperator : 0.03 1 0.0336
OOPS_STATS oops::ObsSpace::ObsSpace : 56.12 1 56.1165
OOPS_STATS oops::ObsSpace::obsvariables : 0.00 3 0.0003
OOPS_STATS oops::ObsSpace::save : 42.93 1 42.9260
OOPS_STATS oops::ObsSpace::~ObsSpace : 0.24 1 0.2398
OOPS_STATS oops::ObsVector::ObsVector : 0.15 5 0.0300
OOPS_STATS oops::ObsVector::invert : 0.00 2 0.0006
OOPS_STATS oops::ObsVector::operator*= : 0.00 2 0.0016
OOPS_STATS oops::ObsVector::operator= : 0.02 4 0.0041
OOPS_STATS oops::ObsVector::save : 0.43 2 0.2174
OOPS_STATS oops::ObsVector::zero : 0.00 1 0.0010
OOPS_STATS oops::ObsVector::~ObsVector : 0.02 5 0.0032
OOPS_STATS oops::Parameters::deserialize : 0.78 13 0.0598
OOPS_STATS oops::Parameters::serialize : 0.23 10 0.0234
OOPS_STATS oops::Parameters::validate : 75.22 6 12.5369
OOPS_STATS oops::SampledLocations::SampledLocations : 0.00 1 0.0013
OOPS_STATS oops::SampledLocations::latitudes : 0.00 1 0.0002
OOPS_STATS oops::SampledLocations::longitudes : 0.00 1 0.0001
OOPS_STATS oops::SampledLocations::times : 0.00 1 0.0001
OOPS_STATS oops::SampledLocations::~SampledLocations : 0.17 2 0.0844
OOPS_STATS oops::State::State : 27.64 2 13.8218
OOPS_STATS oops::State::print : 1.34 1 1.3416
OOPS_STATS oops::State::toFieldSet : 0.34 1 0.3404
OOPS_STATS oops::State::~State : 0.07 2 0.0365
OOPS_STATS oops::UnstructuredInterpolator::UnstructuredInterpolator: 1.21 1 1.2057
OOPS_STATS oops::UnstructuredInterpolator::apply : 0.10 1 0.1049
OOPS_STATS oops::VariableChange::VariableChange : 0.17 1 0.1656
OOPS_STATS oops::VariableChange::changeVar : 2.28 1 2.2762
OOPS_STATS oops::VariableChange::~VariableChange : 0.01 1 0.0124
OOPS_STATS util::Timers::Total : 983.22 1 983.2222
OOPS_STATS util::Timers::measured : 975.30 1 975.3047
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------
OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 1 MPI tasks) -----------------------------------
OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS Name : min (ms) max (ms) avg (ms) % total imbal (%)
OOPS_STATS oops::Geometry::Geometry : 753.11 753.11 753.11 76.60 0.00
OOPS_STATS oops::Geometry::variableSizes : 1.11 1.11 1.11 0.11 0.00
OOPS_STATS oops::Geometry::~Geometry : 5.00 5.00 5.00 0.51 0.00
OOPS_STATS oops::GeometryData::setGlobalTree : 5.30 5.30 5.30 0.54 0.00
OOPS_STATS oops::GeometryData::setLocalTree : 8.35 8.35 8.35 0.85 0.00
OOPS_STATS oops::GetValues::GetValues : 1.34 1.34 1.34 0.14 0.00
OOPS_STATS oops::ObsAuxControl::ObsAuxControl : 19.92 19.92 19.92 2.03 0.00
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::ObsOperator : 37.20 37.20 37.20 3.78 0.00
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::locations : 2.21 2.21 2.21 0.22 0.00
OOPS_STATS oops::ObsOperator::GnssroBndNBAM::simulateObs : 1.54 1.54 1.54 0.16 0.00
OOPS_STATS oops::ObsSpace::ObsSpace : 56.12 56.12 56.12 5.71 0.00
OOPS_STATS oops::ObsSpace::save : 42.93 42.93 42.93 4.37 0.00
OOPS_STATS oops::Parameters::validate : 75.22 75.22 75.22 7.65 0.00
OOPS_STATS oops::State::State : 27.64 27.64 27.64 2.81 0.00
OOPS_STATS oops::State::print : 1.34 1.34 1.34 0.14 0.00
OOPS_STATS oops::UnstructuredInterpolator::UnstructuredInterpolator: 1.21 1.21 1.21 0.12 0.00
OOPS_STATS oops::VariableChange::changeVar : 2.28 2.28 2.28 0.23 0.00
OOPS_STATS util::Timers::Total : 983.22 983.22 983.22 100.00 0.00
OOPS_STATS util::Timers::measured : 975.30 975.30 975.30 99.19 0.00
OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 1 MPI tasks) -----------------------------------
OOPS_STATS Run end - Runtime: 1.31 sec, Memory: total: 298.14 Mb, per task: min = 298.14 Mb, max = 298.14 Mb
Run: Finishing oops::HofX3D<MPAS, UFO and IODA observations> with status = 0
[TestReference] Comparison is done
OOPS Ending 2025-05-29 18:37:38 (UTC+0000)
corrupted size vs. prev_size in fastbins
[f851876e8e90:01127] *** Process received signal ***
[f851876e8e90:01127] Signal: Aborted (6)
[f851876e8e90:01127] Signal code: (-6)
(gdb) where
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x000014ce26968859 in __GI_abort () at abort.c:79
#2 0x000014ce269d326e in __libc_message (action=action@entry=do_abort, fmt=0x7ffe6f461110 "P7\370\066\316\024",
fmt@entry=0x14ce26afd298 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#3 0x000014ce269db2fc in malloc_printerr (str=str@entry=0x14ce26aff2a8 "corrupted size vs. prev_size in fastbins")
at malloc.c:5347
#4 0x000014ce269dbacc in malloc_consolidate (av=av@entry=0x14ce26b32b80 <main_arena>) at malloc.c:4493
#5 0x000014ce269dcfe0 in _int_free (av=0x14ce26b32b80 <main_arena>, p=0x558f996292f0, have_lock=<optimized out>)
at malloc.c:4400
#6 0x000014ce2698c953 in __run_exit_handlers (status=0, listp=0x14ce26b32718 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:124
#7 0x000014ce2698ca60 in __GI_exit (status=<optimized out>) at exit.c:139
#8 0x000014ce2696a08a in __libc_start_main (main=0x558f914e41b3 <main(int, char**)>, argc=2, argv=0x7ffe6f461428,
init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe6f461418)
at ../csu/libc-start.c:342
#9 0x0000558f914e061e in _start ()