# FV3-JEDI ctest failed with core dumped after turning on "debug" **URL:** \ **Category:** JEDI\ **Created:** [April 5, 2021, 1:46pm UTC](https://forums.jcsda.org/t/fv3-jedi-ctest-failed-with-core-dumped-after-turning-on-debug/84 "2021-04-05T13:46:33Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ytangnoaa](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/ytangnoaa/32/58_2.png) [@ytangnoaa](https://forums.jcsda.org/u/ytangnoaa)\ **Post date:** [April 5, 2021, 1:46pm UTC](https://forums.jcsda.org/t/fv3-jedi-ctest-failed-with-core-dumped-after-turning-on-debug/84/1 "2021-04-05T13:46:33Z")
I tried to turn on compiler’s debug to check the Nan value issue in my case ecbuild --build=debug … It passed compiling, but some unit tests failed, such as geometry test, in ctest ``` Start 1: fv3jedi_test_tier1_coding_norms ``` 1/71 Test #1: fv3jedi\_test\_tier1\_coding\_norms … Passed 2.94 sec Start 2: fv3jedi\_test\_tier1\_get\_test\_data\_fv3-jedi 2/71 Test #2: fv3jedi\_test\_tier1\_get\_test\_data\_fv3-jedi … Passed 21.53 sec Start 3: fv3jedi\_test\_tier1\_get\_test\_data\_ioda 3/71 Test #3: fv3jedi\_test\_tier1\_get\_test\_data\_ioda … Passed 10.39 sec Start 4: fv3jedi\_test\_tier1\_get\_test\_data\_crtm 4/71 Test #4: fv3jedi\_test\_tier1\_get\_test\_data\_crtm … Passed 184.67 sec Start 5: fv3jedi\_test\_tier1\_geometry\_gfs 5/71 Test #5: fv3jedi\_test\_tier1\_geometry\_gfs …\*\*\*Failed 8.54 sec Start 6: fv3jedi\_test\_tier1\_geometry\_gfs127 6/71 Test #6: fv3jedi\_test\_tier1\_geometry\_gfs127 …\*\*\*Failed 17.31 sec Start 7: fv3jedi\_test\_tier1\_geometry\_geos 7/71 Test #7: fv3jedi\_test\_tier1\_geometry\_geos …\*\*\*Failed 16.11 sec Start 8: fv3jedi\_test\_tier1\_geometry\_lam\_cmaq 8/71 Test #8: fv3jedi\_test\_tier1\_geometry\_lam\_cmaq …\*\*\*Failed 20.25 sec Start 9: fv3jedi\_test\_tier1\_geometry\_2d 9/71 Test #9: fv3jedi\_test\_tier1\_geometry\_2d …\*\*\*Failed 15.21 sec BTW, the same tests could pass when turning off the “–build=debug”. Has anyone encountered the similar issue? Thanks ---
**Author:** ![mmiesch](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/mmiesch/32/39_2.png) [@mmiesch](https://forums.jcsda.org/u/mmiesch)\ **Post date:** [April 5, 2021, 9:03pm UTC](https://forums.jcsda.org/t/fv3-jedi-ctest-failed-with-core-dumped-after-turning-on-debug/84/2 "2021-04-05T21:03:46Z")
Hi @ytangnoaa - what system are you running on, and what compiler suite? Also, have you made sure you have sufficient memory with ulimit -s unlimited ulimit -v unlimited Also, what is the error message from the `fv3jedi_test_tier1_geometry_gfs` test? ---
**Author:** ![ytangnoaa](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/ytangnoaa/32/58_2.png) [@ytangnoaa](https://forums.jcsda.org/u/ytangnoaa)\ **Post date:** [April 5, 2021, 9:16pm UTC](https://forums.jcsda.org/t/fv3-jedi-ctest-failed-with-core-dumped-after-turning-on-debug/84/3 "2021-04-05T21:16:22Z")
I am running on Hera, with intel/18.0.5.274 (the newer intel/2020.2 gave me compiler internal error during the compiling). I tried ulimit -s unlimited bash: ulimit: stack size: cannot modify limit: Operation not permitted ulimit -v unlimited (OK) With debug option, the gfs run still failed ctest -VV -R fv3jedi\_test\_tier1\_geometry\_gfs UpdateCTestConfiguration from :/home/YouHua.Tang/noscrub/test/build2/DartConfiguration.tcl Parse Config file:/home/YouHua.Tang/noscrub/test/build2/DartConfiguration.tcl UpdateCTestConfiguration from :/home/YouHua.Tang/noscrub/test/build2/DartConfiguration.tcl Parse Config file:/home/YouHua.Tang/noscrub/test/build2/DartConfiguration.tcl Test project /home/YouHua.Tang/noscrub/test/build2 Constructing a list of tests Done constructing a list of tests Updating test list for fixtures Added 0 tests to meet fixture requirements Checking test dependency graph… Checking test dependency graph end test 1017 Start 1017: fv3jedi\_test\_tier1\_geometry\_gfs 1017: Test command: /apps/slurm/default/bin/srun “-n” “6” “-q” “debug” “/home/YouHua.Tang/noscrub/test/build2/bin/test\_fv3jedi\_geometry.x” “testinput/geometry\_gfs.yaml” 1017: Environment variables: 1017: OOPS\_TRAPFPE=0 1017: OMP\_NUM\_THREADS=1 1017: Test timeout computed to be: 1500 1017: srun: job 17685159 queued and waiting for resources 1017: srun: job 17685159 has been allocated resources 1017: OOPS Starting 2021-04-05 21:09:16 (UTC+0000) 1017: OOPS Starting 2021-04-05 21:09:16 (UTC+0000) 1017: OOPS Starting 2021-04-05 21:09:16 (UTC+0000) 1017: OOPS Starting 2021-04-05 21:09:16 (UTC+0000) 1017: OOPS Starting 2021-04-05 21:09:16 (UTC+0000) 1017: OOPS Starting 2021-04-05 21:09:16 (UTC+0000) 1017: Configuration input file is: testinput/geometry\_gfs.yaml 1017: Full configuration is:YAMLConfiguration[path=testinput/geometry\_gfs.yaml, root={geometry =\> {nml\_file\_mpp =\> Data/fv3files/fmsmpp.nml , trc\_file =\> Data/fv3files/field\_table , akbk =\> Data/fv3files/akbk64.nc4 , layout =\> (1,1) , io\_layout =\> (1,1) , npx =\> 13 , npy =\> 13 , npz =\> 64 , ntiles =\> 6 , do\_write\_geom =\> true , fieldsets =\> ({fieldset =\> Data/fieldsets/aerosols\_gfs.yaml},{fieldset =\> Data/fieldsets/dynamics.yaml},{fieldset =\> Data/fieldsets/ufo.yaml})}}] 1017: OOPS\_STATS ObjectCountHelper started. 1017: Run: Starting oops::Test running test::Geometry 1017: Running 1 tests: 1017: Running 1 tests: 1017: Running 1 tests: 1017: Running 1 tests: 1017: Running 1 tests: 1017: Running case “interface/Geometry/testConstructor” … 1017: Running case “interface/Geometry/testConstructor” … 1017: Running case “interface/Geometry/testConstructor” … 1017: Running case “interface/Geometry/testConstructor” … 1017: Running 1 tests: 1017: Running case “interface/Geometry/testConstructor” … 1017: Running case “interface/Geometry/testConstructor” … 1017: srun: error: h23c08: tasks 0-5: Segmentation fault (core dumped) 1017: srun: launch/slurm: \_step\_signal: Terminating StepId=17685159.0 1/2 Test #1017: fv3jedi\_test\_tier1\_geometry\_gfs …\*\*\*Failed 11.91 sec test 1018 Start 1018: fv3jedi\_test\_tier1\_geometry\_gfs127 1018: Test command: /apps/slurm/default/bin/srun “-n” “6” “-q” “debug” “/home/YouHua.Tang/noscrub/test/build2/bin/test\_fv3jedi\_geometry.x” “testinput/geometry\_gfs127.yaml” 1018: Environment variables: 1018: OOPS\_TRAPFPE=0 1018: OMP\_NUM\_THREADS=1 1018: Test timeout computed to be: 1500 1018: srun: job 17685208 queued and waiting for resources 1018: srun: job 17685208 has been allocated resources 1018: OOPS Starting 2021-04-05 21:09:28 (UTC+0000) 1018: OOPS Starting 2021-04-05 21:09:28 (UTC+0000) 1018: OOPS Starting 2021-04-05 21:09:28 (UTC+0000) 1018: OOPS Starting 2021-04-05 21:09:28 (UTC+0000) 1018: OOPS Starting 2021-04-05 21:09:28 (UTC+0000) 1018: OOPS Starting 2021-04-05 21:09:28 (UTC+0000) 1018: Configuration input file is: testinput/geometry\_gfs127.yaml 1018: Full configuration is:YAMLConfiguration[path=testinput/geometry\_gfs127.yaml, root={geometry =\> {nml\_file\_mpp =\> Data/fv3files/fmsmpp.nml , trc\_file =\> Data/fv3files/field\_table , akbk =\> Data/fv3files/akbk127.nc4 , layout =\> (1,1) , io\_layout =\> (1,1) , npx =\> 13 , npy =\> 13 , npz =\> 127 , ntiles =\> 6 , do\_write\_geom =\> true , fieldsets =\> ({fieldset =\> Data/fieldsets/aerosols\_gfs.yaml},{fieldset =\> Data/fieldsets/dynamics.yaml},{fieldset =\> Data/fieldsets/ufo.yaml})}}] 1018: OOPS\_STATS ObjectCountHelper started. 1018: Run: Starting oops::Test running test::Geometry 1018: Running 1 tests: 1018: Running 1 tests: 1018: Running 1 tests: 1018: Running case “interface/Geometry/testConstructor” … 1018: Running case “interface/Geometry/testConstructor” … 1018: Running case “interface/Geometry/testConstructor” … 1018: Running 1 tests: 1018: Running 1 tests: 1018: Running case “interface/Geometry/testConstructor” … 1018: Running case “interface/Geometry/testConstructor” … 1018: Running 1 tests: 1018: Running case “interface/Geometry/testConstructor” … 1018: srun: error: h25c01: tasks 0-5: Segmentation fault (core dumped) 1018: srun: launch/slurm: \_step\_signal: Terminating StepId=17685208.0 2/2 Test #1018: fv3jedi\_test\_tier1\_geometry\_gfs127 …\*\*\*Failed 11.95 sec 0% tests passed, 2 tests failed out of 2 Label Time Summary: fv3-jedi = 23.86 sec_proc (2 tests) fv3jedi = 23.86 sec_proc (2 tests) mpi = 23.86 sec_proc (2 tests) script = 23.86 sec_proc (2 tests) Total Test time (real) = 24.25 sec The following tests FAILED: 1017 - fv3jedi\_test\_tier1\_geometry\_gfs (Failed) 1018 - fv3jedi\_test\_tier1\_geometry\_gfs127 (Failed) Errors while running CTest ---
**Author:** ![CoryMartin-NOAA](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/corymartin-noaa/32/46_2.png) [@CoryMartin-NOAA](https://forums.jcsda.org/u/CoryMartin-NOAA)\ **Post date:** [April 6, 2021, 12:52pm UTC](https://forums.jcsda.org/t/fv3-jedi-ctest-failed-with-core-dumped-after-turning-on-debug/84/4 "2021-04-06T12:52:09Z")
is this the same problem @TingLei-NOAA has found? ---
**Author:** ![ytangnoaa](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/ytangnoaa/32/58_2.png) [@ytangnoaa](https://forums.jcsda.org/u/ytangnoaa)\ **Post date:** [April 6, 2021, 1:00pm UTC](https://forums.jcsda.org/t/fv3-jedi-ctest-failed-with-core-dumped-after-turning-on-debug/84/5 "2021-04-06T13:00:20Z")
Seems not. This issue only occurred after turning on --build=debug option in FV3-JEDI test, such as geometry. Other units tests, like qg and l95, can pass with debug on. ---
**Author:** ![TingLei-NOAA](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/tinglei-noaa/32/45_2.png) [@TingLei-NOAA](https://forums.jcsda.org/u/TingLei-NOAA)\ **Post date:** [April 6, 2021, 1:25pm UTC](https://forums.jcsda.org/t/fv3-jedi-ctest-failed-with-core-dumped-after-turning-on-debug/84/6 "2021-04-06T13:25:18Z")
@CoryMartin-NOAA , thanks for letting me notice this discussion. @ytangnoaa , I am not sure if you are having the same issue as I encountered. On hera, when I first turned on debug mode, quite a few tests failed like hyb-gfs and, traced to the fms part. Quote my words "It was found the issue occurred in mpp\_util\_mpi.inc of FMS when fms compiled with option use\_libMPI (for more details, we can discuss off-line) . Changes to use\_libSMA would avoid this problem. " The tricky part is use\_libSMA seems coming from the recent changes in the 19.0.5 module setup by @Ryan.Honeyager . I haven’t found where to change it from my side. So, if you will trace the problem to fms mpi initialization step, you can try what works for me.