Acceptable Use of Interactive Nodes - Please Review

Important information about use of the general interactive cluster nodes

CHPC has had a longstanding policy regarding the acceptable use of the cluster interactive nodes (details below). Up to date, we have on occasion noticed and killed processes that were in violation of this policy, most typically in reaction to a noticeable impact on the usability of the node, either noticed by CHPC staff or reported to us by an user.

In order to automate this process, we have developed a script to monitor the interactive nodes, identifying processes that are in violation of this policy and generating a daily log of these events. The script also has the capability of emailing the user that they are in violation of the acceptable use policy as well as the ability of killing the process(es) that are in violation.  This script has acceptable processes whitelisted, so that they do not trigger the warnings.

Over the last couple of weeks, we have been testing the script, with the emails only going to CHPC staff, sometimes followed by staff killing process(es) and reaching out to users via email.  On February 1, 2016 we will start to have the emails sent to the user along with CHPC staff, and we will start to use these messages to kill the processes without sending further notifications. These messages can be identified by the subject line: “ALERT: Interactive tasks might be killed on NODENAME”.

On a future date, we will enable the auto kill capability.; when this date has been chosen, we will let users know and update our policy.  In addition, we will also be using this script to monitor the usage of the frisco nodes, and using this information to decide on the usage policy of these nodes.

CHPC interactive node acceptable use policy (for more information see  https://www.chpc.utah.edu/documentation/policies/2.1GeneralHPCClusterPolicies.php#Pol2.1.1):

Interactive nodes are your interface with the computational nodes and are where you interact with the batch system. Please see our User Guides for details. Processes that are run directly on these nodes should be limited to tasks such as editing, data transfer and management, data analysis, compiling codes and debugging -- as long as they are not resource intensive (memory, cpu, network and/or i/o). Any resource intensive work must be run on the compute nodes through the batch system.  

Any process that is consuming extensive resources on the interactive node may be killed, especially when it begins to impact other users on that node.

1. CHPC will usually allow up to 15 minutes CPU time before considering killing a process, unless the resource usage is impacting other users. If the process is creating significant problems on the system, the process will be killed immediately and the user will be contacted via email.
2. Owners of the process are notified via a tty message (if possible) and an email message is sent when the process is killed.
3. Repeated abuse of interactive nodes may result in notification of your PI and potentially locking your account.

Note that this applies to the general interactive nodes, specifically:

• kingspeak1.chpc.utah.edu and kingspeak2.chpc.utah.edu (accessible via kingspeak.chpc.utah.edu)
• ember1.chpc.utah.edu and ember2.chpc.utah.edu (accessible via ember.chpc.utah.edu)
• lonepeak1.chpc.utah.edu and lonepeak2.chpc.utah.edu (accessible via lonepeak.chpc.utah.edu)
• tangent1.chpc.utah.edu and tangent2.chpc.utah.edu (accessible via tangent.chpc.utah.edu)
• ash5.chpc.utah.edu and ash6.chpc.utah.edu (accessible via ash-guest.chpc.utah.edu)

Please let us know of any concerns or questions you may have,