From 4b5e7c290d6aad22e146a4b7c20724e53df862ea Mon Sep 17 00:00:00 2001
From: Blake Leverington <leverington@physi.uni-heidelberg.de>
Date: Wed, 28 Jul 2021 11:15:12 +0200
Subject: [PATCH] Update 'computing_htcondor'

---
 computing_htcondor.md | 68 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 62 insertions(+), 6 deletions(-)

diff --git a/computing_htcondor.md b/computing_htcondor.md
index 566aa2a..445f29a 100644
--- a/computing_htcondor.md
+++ b/computing_htcondor.md
@@ -4,22 +4,73 @@
 Basic use, see the manual entry from the command line: 
 'man condor_submit'
 
-You can find a job.sub example submission file in /auto/work/zhelezov/singularity/batch_centos7 .
+You can find example submission files in /auto/work/zhelezov/singularity/batch_centos7 .
 
 
 
+An example .sub file:
+
+```vim
+# Generic job which will run under local CentOS7 conteiner, on modern servers only
+
+# What to run and arguments, can (and probably always should be) a script
+executable            = ./job.sh
+arguments             = 
+
+# Safe option, so files are not transfer when not required (our servers have access
+# to the same storage, /auto/work, /auto/data, /home).
+should_transfer_files = IF_NEEDED
+
+# Without variable part, output files will be the same for all jobs.
+# That can be confusing.
+output                = out/job.$(ClusterId).$(ProcId).out
+error                 = err/job.$(ClusterId).$(ProcId).err
+log                   = log/job.$(ClusterId).log
+
+# It is possible apply current environment variables in jobs, with the following
+# flag. But in general when particular environment is required, safe option
+# is directly call the script which setup correct environment (lb-run, etc). In LHCb case
+# that can also set optimal platform.
+#getenv                = True
+
+
+# The following flag by itself set 2 following options. It is set by default
+# when submiting from CentOS7 container, but setting it explicitly produce
+# desired result from Ubuntu as well (when submited from hosts supporting it).
++FromVIRTC = "CentOS7"
+
+# Ask to run inside local "centos7" container on working node
+#  +WantVIRTC = "CentOS7"
+# Execute on modern servers only
+#  +WantCPUCap = 2020
+
+# To run the job under Ubuntu, set
+#  +FromVIRTC = ""
+# or
+#  +WantVIRTC = ""
+
+
+# For any job (Ubuntu/CentOS) it is possible limit used servers with
+#  +WantCPUCap = 2020
+# Or run on any server with
+#  +WantCPUCap = 0
+
+# The number is the number of subjobs
+queue 2
+
+```
+
+
 A simple job.sh file to see if ROOT is available and which version.
 
 ```bash
-
 #!/bin/bash
 
-# Demonstrate that environment is correct
 hostname
 pwd
 /work/software/os_version
 
-#setup a simple lhcb environment
+#setup a simple lhcb environment and check if ROOT is the correct version
 source /cvmfs/lhcb.cern.ch/lib/LbEnv
 env #outputs the current environment in stdout
 which lb-run #check lb-run is there
@@ -28,9 +79,14 @@ lb-run Urania/latest root-config --cflags #check the ROOT compilation flags, out
 
 ```
 
-Then submit the job via the command line:
-```console
 
+
+Then submit the job via the command line:
+
+```console
+mkdir -p err #setup the output directories to keep things clean
+mkdir -p out
+mkdir -p log
 condor_submit -file job.sub 
 
 ```