# Nicas keeps getting terminated

**URL:** <https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485>\
**Category:** Uncategorized\
**Created:** [July 5, 2023, 8:00pm UTC](https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485 "2023-07-05T20:00:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![xtian15](https://avatars.discourse-cdn.com/v4/letter/x/ecb155/32.png) [@xtian15](https://forums.jcsda.org/u/xtian15)\
**Post date:** [July 5, 2023, 8:00pm UTC](https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485/1 "2023-07-05T20:00:09Z")

</div>

Hi,  
I am trying to train a static B at c192 resolution with release v4. The nicas step for ozone mixing ratio and specific humidity keeps getting terminated with the following messages

```auto
Info : Count and find nearest neighbors, compute distances: 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Info : Compute weights: 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Info : Compute internal normalization
Info : Compute normalization
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 0 on node tomorrowbe-cn-8 exited on signal 9 (Killed).
--------------------------------------------------------------------------
real 52m13.253s
user 219m59.659s
sys 1640m26.638s
Wed Jul 5 19:48:55 UTC 2023
slurmstepd: error: Detected 3 oom-kill event(s) in StepId=36426.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

```

The stopping point is always immediately after the Compute normalization.  
Is there any parameter in the yaml file that can be tuned for this behavior? Thanks in advance!

---

<div class="post-metadata">

**Author:** ![benjaminmenetrier](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/benjaminmenetrier/32/9_2.png) [@benjaminmenetrier](https://forums.jcsda.org/u/benjaminmenetrier)\
**Post date:** [July 7, 2023, 6:59am UTC](https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485/2 "2023-07-07T06:59:01Z")

</div>

As the `oom-kill` suggests, this is an “Out Of Memory” issue. To reduce the memory footprint, you can reduce the `nicas.resolution` parameter, or limit the convolution length-scales by specifying a reasonable maximum length-scale with the parameter `general.universe length-scale`. You can send me the full log if these options are not working.

---

<div class="post-metadata">

**Author:** ![xtian15](https://avatars.discourse-cdn.com/v4/letter/x/ecb155/32.png) [@xtian15](https://forums.jcsda.org/u/xtian15)\
**Post date:** [July 7, 2023, 12:33pm UTC](https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485/3 "2023-07-07T12:33:07Z")

</div>

Thanks for the suggestion! What would the decreased nicas.resolution imply or where can I find related documentations?

---

<div class="post-metadata">

**Author:** ![benjaminmenetrier](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/benjaminmenetrier/32/9_2.png) [@benjaminmenetrier](https://forums.jcsda.org/u/benjaminmenetrier)\
**Post date:** [July 7, 2023, 1:28pm UTC](https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485/4 "2023-07-07T13:28:06Z")

</div>

Reducing `nicas.resolution` would make the correlation or localization function represented by NICAS more “bumpy”, less smooth. But the computational cost and the memory footprint scale like this parameter squared… So it is a good trade-off parameter between cost and accuracy.

---

<div class="post-metadata">

**Author:** ![xtian15](https://avatars.discourse-cdn.com/v4/letter/x/ecb155/32.png) [@xtian15](https://forums.jcsda.org/u/xtian15)\
**Post date:** [July 7, 2023, 2:00pm UTC](https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485/5 "2023-07-07T14:00:27Z")

</div>

Would it be viable to merge variables trained with different nicas resolutions together for 3dvar? All other variables are trained with resolution=10, only ozone and specific humidities with resolution=4.

---

<div class="post-metadata">

**Author:** ![benjaminmenetrier](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.jcsda.org/benjaminmenetrier/32/9_2.png) [@benjaminmenetrier](https://forums.jcsda.org/u/benjaminmenetrier)\
**Post date:** [July 7, 2023, 3:27pm UTC](https://forums.jcsda.org/t/nicas-keeps-getting-terminated/485/6 "2023-07-07T15:27:31Z")

</div>

Yes, NICAS applications for each variable are independent. Not a problem!