# Multiple atlas failures during ctest **URL:** \ **Category:** JEDI\ **Created:** [October 26, 2022, 6:13pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405 "2022-10-26T18:13:34Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![awheeler](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/awheeler/32/162_2.png) [@awheeler](https://forums.jcsda.org/u/awheeler)\ **Post date:** [October 26, 2022, 6:13pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/1 "2022-10-26T18:13:34Z")
I have tried various ways to solve this issue myself, but have not been able to resolve 24 test failures. I have provided some relevant information below that will hopefully help diagnose my mistake quickly. **Receiving the following failures from ctest…** The following tests FAILED: 158 - atlas\_test\_metadata (Failed) 162 - atlas\_test\_haloexchange\_adjoint (Failed) 163 - atlas\_test\_setcomm (Failed) 164 - atlas\_test\_haloexchange (Failed) 165 - atlas\_test\_gather (Failed) 178 - atlas\_fctest\_grids (Failed) 193 - atlas\_test\_distribution\_regular\_bands (Failed) 234 - atlas\_fctest\_elements (Failed) 235 - atlas\_test\_parfields (Failed) 236 - atlas\_test\_halo (Failed) 237 - atlas\_test\_distmesh (Failed) 239 - atlas\_test\_mesh\_node2cell (Failed) 249 - atlas\_fctest\_functionspace (Failed) 252 - test\_structuredcolumns\_biperiodic (Failed) 253 - atlas\_test\_structuredcolumns (Failed) 257 - atlas\_test\_stencil\_parallel\_mpi4 (Failed) 258 - atlas\_test\_stencil\_parallel\_mpi16 (Failed) 259 - atlas\_test\_polygons\_structuredcolumns (Failed) 260 - atlas\_test\_polygons\_nodecolumns (Failed) 261 - atlas\_test\_polygons\_projections\_structuredcolumns (Failed) 262 - atlas\_test\_polygons\_projections\_nodecolumns (Failed) 263 - atlas\_test\_structuredcolumns\_haloexchange (Failed) 291 - atlas\_test\_interpolation\_structured2D\_to\_unstructured (Failed) 1261 - test\_femps\_csgrid (Failed) **Running on DISCOVER using the following modules…** @discover32[jedi\_spack-stack] JEDI\_OPT=/discover/swdev/jcsda/modules @discover32[jedi\_spack-stack] export JEDI\_OPT @discover32[jedi\_spack-stack]$ module use JEDI\_OPT/modulefiles/core @discover32[jedi\_spack-stack] module use JEDI\_OPT/modulefiles/apps @discover32[jedi\_spack-stack] module load jedi/intel-impi @discover32[jedi\_spack-stack]$ module list Currently Loaded Modules: 1. git/2.38.1 2. git-lfs/2.10.0 3. jedi-python/3.8.3 4. comp/gcc/9.2.0 5. comp/intel/19.1.0.166 6. jedi-intel/19.1.0.166 7. szip/2.1.1 8. zlib/1.2.11 9. udunits/2.2.26 10. mpi/impi/19.1.0.166 11. jedi-impi/19.1.0.166 12. hdf5/1.12.0 13. pnetcdf/1.12.1 14. netcdf/4.7.4 15. nccmp/1.8.7.0 16. boost-headers/1.68.0 17. eigen/3.3.7 18. bufr/noaa-emc-11.5.0 19. cmake/3.23.1 20. ecbuild/ecmwf-3.6.1 21. eckit/ecmwf-1.16.0 22. fckit/ecmwf-0.9.2 23. atlas/ecmwf-0.24.1 24. nco/4.9.9 25. pio/2.5.1-debug 26. gsl\_lite/0.37.0 27. gptl/8.0.3 28. pybind11/2.7.0 29. jedi/intel-impi/ecbuild35 **Cloned the fv3-bundle and using the attached screenshot of CMakeLists.txt below** ![CMakeLists](https://canada1.discourse-cdn.com/flex031/uploads/jcsda/original/1X/97b14176aab61993691e47f54f8855287f663ca9.png) Any help identifying my error is much appreciated, THANK YOU! ---
**Author:** ![maryamao](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/maryamao/32/6_2.png) [@maryamao](https://forums.jcsda.org/u/maryamao)\ **Post date:** [October 26, 2022, 6:21pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/2 "2022-10-26T18:21:31Z")
Hi, Please try using the latest JEDI code. I recommend using [jedi-bundle](https://github.com/jcsda/jedi-bundle)’s latest tag (skylab-v2). Also, please follow the instruction [here](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/using/jedi_environment/modules.html#discover) to use the latest modules on Discover. Atlas is already available in the latest modules and you don’t need to build it again. Please let me know if you have any further questions. Maryam ---
**Author:** ![awheeler](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/awheeler/32/162_2.png) [@awheeler](https://forums.jcsda.org/u/awheeler)\ **Post date:** [October 26, 2022, 8:18pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/3 "2022-10-26T20:18:07Z")
Thank you so much for your quick response and guidance. I am now using the jedi-bundle instead and currently executing “make -j4”…this step always takes quite a while but after almost 1.5 hrs it is only 50% completed. I have previously tried to execute this step differently to speed up the process…similar to using “salloc --nodes=1 --time=30” for running the entire full ctest…however I began to worry that sometimes it lead to make failures so I resorted back to basic. Do you have any suggestions? Or is it best to suffer out the long wait. Thank you again for your help ---
**Author:** ![maryamao](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/maryamao/32/6_2.png) [@maryamao](https://forums.jcsda.org/u/maryamao)\ **Post date:** [October 26, 2022, 10:09pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/4 "2022-10-26T22:09:40Z")
Unfortunately it takes more than an hour for me to build it on Discover too. One way to save a bit of time is to comment out `mom6` and `soca` in `CMakeLists.txt` (Lines 51 and 52) if you don’t have plans to use those. ---
**Author:** ![awheeler](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/awheeler/32/162_2.png) [@awheeler](https://forums.jcsda.org/u/awheeler)\ **Post date:** [October 27, 2022, 1:48am UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/5 "2022-10-27T01:48:34Z")
I expected that was the case, thank you for the suggestion. After completing the full ctest following the instructions you provided above, I am only left with a single test failure. **1143 - test\_femps\_csgrid (Failed)** **forrtl: severe (174): SIGSEGV, segmentation fault occurred** > > > see screenshot below as well > > > > > > ![test_femps_csgrid_SINGLE FAIL](https://canada1.discourse-cdn.com/flex031/uploads/jcsda/original/1X/578b9dd37eec04b6e4d62d0e8c4ec789e0d83100.png) I have not been able to find any relevant notes in the online JCSDA forums or documentation, but I will continue to explore… Please let me know if you have any suggestions or know the solution to this ctest failure. Otherwise THANK YOU again for your help! ---
**Author:** ![climbfuji](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/climbfuji/32/154_2.png) [@climbfuji](https://forums.jcsda.org/u/climbfuji)\ **Post date:** [October 27, 2022, 2:25am UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/6 "2022-10-27T02:25:29Z")
We’ve seen this before with intel, an intermittent failure. Usually goes away when you run it again. ---
**Author:** ![awheeler](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/awheeler/32/162_2.png) [@awheeler](https://forums.jcsda.org/u/awheeler)\ **Post date:** [October 27, 2022, 6:10pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/7 "2022-10-27T18:10:59Z")
**Thank you for your response…** I have rebuilt and ran through the entire process two more times with the same failure. _ **Should I pivot and use GNU instead?** _ I understand that these types of intermittent failures occur and are often out of our control, so I have no problem if the solution is “it is what it is” and continuing to try until it passes is the reality…I just want to make sure I am not making things harder on myself or causing the issue myself **Truly appreciate your assistance** …I have been trying to build and run JEDI successfully on discover through trial and error myself, and the feedback provided via this forum has been extremely helpful to search through ---
**Author:** ![climbfuji](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/climbfuji/32/154_2.png) [@climbfuji](https://forums.jcsda.org/u/climbfuji)\ **Post date:** [October 27, 2022, 6:35pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/8 "2022-10-27T18:35:15Z")
If your workflow allows it, you may also simply ignore this one failing test or exclude it when running ctest (ctest -E femps\_csgrid) ---
**Author:** ![awheeler](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/awheeler/32/162_2.png) [@awheeler](https://forums.jcsda.org/u/awheeler)\ **Post date:** [October 31, 2022, 4:51pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/9 "2022-10-31T16:51:20Z")
I was able to successfully build jedi-bundle and complete ctest without failure using the instructions provided in the initial response from Maryam (Thank you!) Now I have hit another block… When I originally submitted my initial ctest question to this forum I was trying to build the fv3-bundle. I need to be able to make changes and rerun JEDI in various configurations…so that I can compare T/q soundings from a) different satellite instruments, b) using both forward models (crtm and rttov), and c) processed with specific modifications to the retrieval algorithm and/or channel selection coefficients. For this reason, I thought it made more sense to build and run JEDI using the fv3-bundle instead of the jedi-bundle. I may or may not be incorrect in this…I feel like I am still very much in the transitioning stage between “Youngling and Padawan” hahah I have tried to build and run JEDI with the fv3-bundle on Discover multiple ways (environ. modules, spack-stack, singularity, …) and using tutorials but I can’t seem to get any to work, despite the excellent and seemingly straightforward documentation provided online. Right now I think I am my biggest issue, bouncing around between options from one to another multiple times in a day and crossing my own wires… Regardless…I am still unsure which way to build/run JEDI will be the best for my specific goals. I also read somewhere that RTTOV must be pulled individually from EUMETSAT after signing a license agreement – I am not sure if that is still the case – but I assume this would also influence which option need. Please also feel free to contact me via email: [ashley.a.wheeler@nasa.gov](mailto:ashley.a.wheeler@nasa.gov) ---
**Author:** ![maryamao](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/maryamao/32/6_2.png) [@maryamao](https://forums.jcsda.org/u/maryamao)\ **Post date:** [November 1, 2022, 5:26pm UTC](https://forums.jcsda.org/t/multiple-atlas-failures-during-ctest/405/10 "2022-11-01T17:26:59Z")
I suggest you use Spack modules on Discover. Switching to GNU may help with reducing the build time. If I understand your research goals correctly then you only need the obs operators? If yes then you can comment out more repositories from jedi-bundle/CMakeLists.txt. For example you can remove fv3-jedi repos (fms, fv3, femps, fv3-jedi-lm, fv3-jedi, fv3-jedi-data) and also mom6 and soca. This will reduce the build and test time.