A generic infrastructure monitor will tell you a Siebel host is up while users are being thrown out of the application. Our team has run Siebel CRM in production since 2004, and the outages we see are rarely a dead server. They are an object manager that has quietly gone Unavailable, a component sitting at its task limit, processes crashing on a database that has run out of connections, or core files filling a filesystem. This is the checklist we use: what to watch, why it matters and how to collect it.
1. Host signals and the processes that must be up
Start with the operating system, because every Siebel symptom eventually shows up there. CPU and load, memory and swap, and disk usage on the Siebel filesystem, the log directories and the Siebel Server bin directory (where crash dumps land) are the minimum. Memory needs particular attention. Each interactive object manager multithreaded process typically takes from a few hundred MB to around 1 GB. An undersized host does not fail cleanly. It slows down, swaps and eventually stops serving sessions.
A host being up does not mean Siebel is up. Check by process name for each tier:
- Gateway registry: a native process (
siebsvc -s gtwyns), separate from the Gateway's Tomcat container. - Gateway / Siebel Server application container: the Tomcat JVM under
applicationcontainer_internal(this is also where SMC runs). - Application Interface (AI): the Tomcat JVM under
applicationcontainer_external. - Siebel Server: one
siebsvc -s siebsrvrprocess per Siebel Server, plus its component shells. Interactive object managers run insiebmtshmw. Background and batch components run in shells such assiebprocorsiebprocmw.
# Gateway registry (native) and Gateway container pgrep -af 'siebsvc -s gtwyns|zookeeper|QuorumPeer' || echo "no registry process" pgrep -f '[j]ava .*applicationcontainer_internal' || echo "internal container down" # Application Interface pgrep -f '[j]ava .*applicationcontainer_external' || echo "AI container down" # Siebel Server siebsvc for one server, and its object manager shells pgrep -fa '^siebsvc -s siebsrvr .* -e <enterprise> -s siebel-app1( |$)' ps -ef | grep -c '[s]iebmtsh' # Listeners: read the socket table ss -ltn | grep -E ':(<gateway_port>|<registry_port>|<ai_https_port>) '
Two rules from our own runbooks. First, check Gateway listeners by reading the socket table (ss -ltn) and the process list, not by sending curl or openssl handshakes to the Gateway TLS and registry ports. Second, use a check like the siebsvc one above to decide whether start_server may run at all. On a server that is already running, start_server re-initialises the shared memory segment under the live components (SBL-SVR-00027), and every component on that server goes down. A monitoring or auto-restart script that does not check first can cause the outage it was meant to detect.
2. Siebel Server and component state
Running processes are necessary, but they do not prove that Siebel will serve users. The authoritative view comes from Server Manager. Oracle's System Monitoring and Diagnostics Guide documents list servers, list component, list component groups and list tasks for this purpose. Use the show clause to return only the columns you parse:
list servers show SBLSRVR_NAME, HOST_NAME, SBLSRVR_STATE, START_TIME list compgrps show SV_NAME, CG_ALIAS, CG_DISP_ENABLE_ST, CA_RUN_STATE list comp show SV_NAME, CC_ALIAS, CP_DISP_RUN_STATE, CP_STARTMODE, CP_NUM_RUN_TASKS, CP_MAX_TASKS
In srvrmgr, show must be the last clause of a list command. A where after show fails with SBL-ADM-02532. Filter by name pattern instead (list comp FINSO%), or spool the full list and filter it in the shell. Copy column names from the header of a bare list comp, never from memory. The column is CP_MAX_TASKS, not CP_MAX_TASK. An unknown column does not always produce a parse error. Some list commands abort the whole script with a generic error while srvrmgr still exits 0.
Server state. SBLSRVR_STATE should be Running. The states that point to a real problem are Connect Failed (the Gateway gave Server Manager the connect string but the server could not be reached), Handshake Failed (this includes the ServerMgr component reaching its own MaxTasks), Login Failed, Disconnected (the ServerMgr task on that server exited) and Not available.
Component state. The documented states are Starting Up, Online, Running, Offline, Shutting Down, Shutdown and Unavailable. The state to alert on immediately is Unavailable. For a multithreaded component it means MinMTServers is greater than zero but no process is running: the component failed to initialise or exited with an error. Oracle also notes that components can move to Unavailable when the database connection is down, and that they must be restarted once the connection is back. Partially Offline is the early warning: fewer processes than MinMTServers, or for background components fewer than DfltTasks.
Compare against what should be running, not against an empty list. If srvrmgr reports Connected to 0 server(s), list comp returns every component definition in the Gateway registry with blank state columns. That is a catalogue, not a status report. If a component returns 0 rows while connected, its component group is usually disabled. Keep an expected list per server (component alias and start mode) and alert on any difference.
3. Task utilization and MaxTasks
A component can be Running and still refuse work. MaxTasks is the ceiling on concurrent tasks. For an object manager, a task is a user session, and MaxTasks divided by MaxMTServers is the number of threads each multithreaded process can host. When CP_NUM_RUN_TASKS reaches CP_MAX_TASKS, new logins fail. Users see that as “the server is busy”, while the server itself looks healthy.
- Track
CP_NUM_RUN_TASKS / CP_MAX_TASKSper component and per server as a percentage. Warn well before 100% (we typically use 80% and 90%). - Track
CP_ACTV_MTS_PROCSagainstCP_MAX_MTS_PROCS. A drop in active processes without a restart usually means processes are dying. - Watch the trend, not only the level. Sessions that rise steadily without falling back often mean tasks are being left behind by failed requests. Check with
list tasks for comp <alias>. - For SCBroker,
list statistics for component SCBrokerincludes counters for connections refused because the target component was maxed out (SCBCompMaxeOut) or unavailable (SCBCompOffline).
list tasks for comp <alias> show SV_NAME, TK_TASKID, TK_DISP_RUNSTATE, TK_START_TIME list statistics for component SCBroker
MaxTasks also matters outside Siebel. The Performance Tuning Guide sizes the UNIX latch variables from the cumulative MaxTasks of all components on the host (SIEBEL_OSD_LATCH = 1.2 × cumulative MaxTasks, SIEBEL_OSD_NLATCH = 7 × cumulative MaxTasks + 1000). Each session also needs a database connection. When someone raises MaxTasks, the latch settings and the database's process limit should be reviewed with it, and your dashboard is the obvious place to see the change.
4. Logs, SBL- error codes and crash files
Siebel writes its useful evidence to many files. The Siebel Server log is named <enterprise>.<server>.log. Component task logs (<alias>_*.log) sit under <siebel_root>/siebsrvr/enterprises/<enterprise>/<server>/log. A busy server can produce thousands of these files a day, so scan them incrementally: remember the offset per file and read only the new bytes. Re-grepping everything on every pass makes the monitor a load problem of its own.
Group what you find by error code, not by line. These are the codes we alert on in almost every environment, all taken from real incidents:
- SBL-SVR-00027: shared memory problems on the Siebel Server. Treat it as an outage of every component on that server.
- SBL-DAT-00175 “Database connection was lost”: despite the wording, the cause is often on the database side (for example a Resource Manager directive cancelling long calls, or the instance hitting its PGA limit with ORA-04036). Read the database alert log before you blame the network.
- SBL-JCA-00200 / SBL-JCA-00208: the AI could not get a session from the object manager, which users see as “the server you are trying to access is either busy or experiencing difficulties”. Check the component state and capacity (sections 2 and 3) first.
- SBL-SEC-10001 / SBL-SEC-10018: authentication subsystem failures. A sudden burst across several object managers usually points to the directory or security adapter rather than to individual users.
- ORA- codes in Siebel logs, especially ORA-12520 (the listener refused the connection because the database had no free process) and ORA-00020 (maximum number of processes exceeded).
Crash files deserve their own alert. In the Siebel Server bin directory, look for siebel_crash*.txt (on UNIX, a process malfunctioned or failed), siebel_assert*.txt (a fatal condition), core* files, and flight data recorder files named T<YYYYMMDDHHMM>_P<pid>.fdr. For the Tomcat containers, look for JVM hs_err_pid* files. Oracle's guidance is that these files can be ignored when they appear during normal running without errors. When they appear alongside failures, keep them for the service request.
# New crash evidence in the last 15 minutes find <siebel_root>/siebsrvr/bin -maxdepth 1 -mmin -15 \ \( -name 'core*' -o -name 'siebel_crash*.txt' -o -name 'siebel_assert*.txt' -o -name '*.fdr' \) -ls find <siebel_root>/applicationcontainer_internal <siebel_root>/applicationcontainer_external \ -maxdepth 2 -name 'hs_err_pid*' -mmin -15 -ls
Why this matters: an object manager process hosts many sessions, so one crash drops all of them at the same moment. To users it looks as if “the web server went down”. We have seen dozens of such crashes an hour, with core files growing fast enough to fill the Siebel filesystem. In that case the root cause was the database refusing new connections, not Siebel. The crash count is often the first signal that separates “slow” from “broken”. Also put a size alert on the bin directory.
One more source is worth enabling. Setting SIEBEL_STDERROUT=1 writes process standard error and standard output to <siebsrvr_root>/log/StdErrOut. This is where you find, for example, a siebmtshmw that cannot start because of a library path problem. It is not enabled by default, and the Siebel Server host must be restarted for it to take effect.
5. Application Interface HTTP errors and database connections
The AI is the closest thing you have to the user's view. Its Tomcat access log, <siebel_root>/applicationcontainer_external/logs/localhost_access_log.<date>.txt, records every request with its HTTP status. Count requests and 5xx responses per interval and alert on the 5xx rate, not on single errors. Oracle documents an extended AccessLogValve pattern in server.xml that adds begin and end time, response commit time and the SOAPAction header. With that pattern in place, the same log also shows which inbound web service is failing and how slowly.
When a Siebel Server rejects an unusual number of requests, Oracle's load-balancing troubleshooting points to SISNAPI Connection Refused messages in the AI logs. The documented causes are SCBroker not running or listening on a different port, the target object manager not running or unable to run more tasks, a hung task or thread, or the object manager being unable to reach the database. Each of those corresponds to a check in this article. Also watch catalina.<date>.log on both containers for start-up failures and out-of-memory errors.
Connection counts complete the picture:
- Established connections on the AI HTTPS/HTTP ports, which show user load and connection leaks.
- Established connections to the SCBroker port on each Siebel Server, which show the AI-to-object-manager traffic.
- Database sessions from the Siebel hosts, compared with the database's processes limit as well as its sessions limit. These are separate ceilings. We have seen processes reach its limit while sessions still had plenty of room, and the result was ORA-12520, crashing object managers and a DBA truthfully reporting that “max sessions was never reached”.
# Run on the relevant host: AI port on the AI host, SCBroker and DB on each Siebel Server ss -tn state established '( sport = :<ai_https_port> or sport = :<scbroker_port> )' | wc -l ss -tn state established '( dport = :<db_listener_port> )' | wc -l
6. Alert design: states versus events
Most Siebel monitoring set-ups fail at this point. They collect the right data, then page people until everyone ignores the alerts. We separate two kinds of alert:
State alerts describe a condition that is either true or false right now: a process missing, a server not Running, a component Unavailable, utilization above 90%, a disk above threshold. They should open once when the condition starts, stay open without repeating, and resolve themselves when the next poll sees the condition gone. Require two consecutive failed polls before opening, so a component restart or a slow srvrmgr response does not page anyone. Suppress them during planned maintenance.
Event alerts describe things that happened: a log line with an SBL- or ORA- code, a new core file, a burst of 5xx. They have nothing to resolve, so do not model them as states. Aggregate them by error code (and server or component) over a window, and alert on new codes and on rate changes. “SBL-DAT-00175 seen 140 times on siebel-app1 in 10 minutes, first seen 09:42” is something an on-call engineer can act on. 140 separate emails are not.
- Every alert names the server, component or file, and links to the evidence.
- Codes that are known to be harmless in your environment go on a reviewed allow-list. Do not raise the threshold to hide them.
- Crash files are always events, and the first one after a quiet period is always worth a notification.
7. Doing it agentlessly over SSH, with the installation's own srvrmgr
Siebel hosts are change-controlled. Installing agents on them means change requests, root access and another component to patch. Everything in this article can be collected over SSH as the Siebel OS user with standard tools, plus the srvrmgr binary already in the installation. The Siebel environment (siebenv.sh) must be sourced first. Use the installation's own srvrmgr so the client version always matches the server version.
# One srvrmgr session per poll: commands from a file, batch mode, stdin closed
cat > <workdir>/poll.cmd <<'EOF'
list servers show SBLSRVR_NAME, HOST_NAME, SBLSRVR_STATE, START_TIME
list comp show SV_NAME, CC_ALIAS, CP_DISP_RUN_STATE, CP_STARTMODE, CP_NUM_RUN_TASKS, CP_MAX_TASKS
EOF
. <siebel_root>/siebsrvr/siebenv.sh
srvrmgr -g <gateway_host>:<port> -e <enterprise> -u <admin_user> -ip <password_file> \
-b -i <workdir>/poll.cmd -o <workdir>/poll.out < /dev/null
echo "rc=$?"Details that save you trouble:
- Use
-iwith-b. Without-b, srvrmgr returns 0 even when a command in the input file fails. With-b, any error gives a nonzero exit code. Still check the output for expected headers, because an invalidshowcolumn can abort a script while the exit code stays 0. - Prefer an input file to
-c. Oracle recommends-ifor scripted use, and we have seen srvrmgr silently truncate a long-ccommand and then wait forever for input. Always close stdin (< /dev/null) in automation. - Keep passwords off the command line.
-ipreads an encrypted password file created withencryptstring, whereas-pexposes the password in the process list. - Connect at enterprise level for
list servers. Oracle's guide warns against starting with-sfor enterprise status. For frequent per-server lists, one session per server with-sand aggregation outside srvrmgr is faster, because srvrmgr serialises operations across servers. - Parse fixed columns from a defined
showlist, or use the-kdelimiter flag, instead of relying on the default layout. - Run the poll at low priority and keep it read-only. A monitor should never be the thing that starts or stops a Siebel Server.
8. CoreOps: all of this, out of the box
We built CoreOps because we kept writing these checks again for each environment. It does what this article describes, every 30 seconds, with minimal dependencies. A small static probe (a single Go binary of a few MB) is copied over SSH into the Siebel OS user's home and runs only when called. Nothing is installed, it needs no root and runs no daemon. Siebel state comes from the installation's own srvrmgr.
- Host CPU, memory, disk and load. Expected processes: Gateway, registry,
siebsvc, component processes, AI and management Tomcat. - Siebel servers and every component, with task utilization, from
list serversandlist comp. - Incremental error scanning across thousands of Siebel and Tomcat logs (only new bytes are read), plus crash dumps:
crash*.txt,core*,hs_err_pid*. - AI request and 5xx counts from the access log. Open connections on the AI ports, database sessions and SCBroker.
- State alerts that resolve themselves, and log and crash events aggregated by error code.
The control plane runs in Docker on a separate management host, so the Siebel servers stay exactly as they were. See the CoreOps page for the full feature list and deployment requirements.
Written by Mahmoud Ali, CoretiTech.
CoretiTech provides Siebel CRM support, upgrades and development, and builds CoreOps.
Contact us →