Skip to content

lower bootstrax resources and use xenon_sectets

Luca Scotto Lavina requested to merge bootstrax_resources into master

Created by: jorana

Two minor additions are made to bootstrax:

  • lower the resources available to process a run to be more lenient on CPU/RAM when a run has failed 10 times or more. Was raised in https://github.com/XENONnT/straxen/issues/138. I could have implemented a roll-of where each fail gradually lowers the amount of resources. I choose not to do this as this might waste too much time on a problem that actually needs a proper fix or is simply not related to the CPU/RAM usage. Please do note that sub-optimal performance of the eventbuilders may stay undetected if this happens regularly! For example a few days I restarted the eventbuilders as their swap and RAM were clogged up with all kind of stuff. One could imagine that if this happens very often one would not notice after this PR. However, especially comming year, I will be paying close attention to the rundb and often query the runsdb with the following syntax to check that this doesn't happen too often:
{"bootstrax.n_failures":{"$gt":1}, "number":{"$gt":8000}}
  • use xenon_sectrets.py as a resource for rundb passwords. This is a technical issue of not being able to restart bootstrax as the environment variables are not available over ssh -t 'bootstrax'. Using this syntax allow us to store the passwords in the xenon_sectrets.py without having to mingle with the sshd_config of the eventbuilders.
  • Finally I've also upgraded the default of the number of cores and max messages as these values were ridiculously low.

Merge request reports