Siebel CRM

Restarting a Siebel Server Safely: A Staged Checklist

By ··11 min read

Our team has worked with Siebel CRM since 2004, and the restarts that turned into outages almost never failed because of Siebel itself. They failed because a step was skipped: someone ran start_server on a server that was still up, a component process outlived stop_server, or nobody checked the database before bringing the server back. This is the checklist we use instead.

Why ad-hoc restarts fail

Running stop_server all and then start_server all looks like a restart. Most of the time it works. When it doesn't, the problem is nearly always one of four things.

start_server on a running server. start_server runs a cleanup step (SrvrCleanup) before it starts anything. If it misreads a live server as dead, it resets that server's shared memory while the components are still using it. The components then crash together, the logs fill with SBL-SVR-00027 and shared-memory errors, Server Manager sessions drop, and the only way out is a full stop and start. This happens when a script is run twice, when automation is re-run, or when someone "makes sure" the server is up by starting it again.

Leftover processes. stop_server returns before everything has exited, and it can leave component processes (siebmtsh, siebmtshmw, siebproc, siebprocmw) behind, for example when the Server Request Broker is already down. Those processes still hold the server's shared-memory segment, and they block the next start.

Orphaned shared memory and semaphores. If a process is killed with kill -9, its shared-memory file and IPC semaphores are left behind. The next start then fails with SBL-OSD-00220: Internal: pthread_mutex_trylock () failed with error. (22).

Stale temp and cache files. The per-server OS data file and shared-memory file, and caches such as the dictionary cache, survive a restart. After an unclean stop they describe a state that no longer exists.

The answer is not more care at 2 a.m. It is a fixed sequence in which every stage checks a condition before moving on, and stops when the check fails. On UNIX and Linux the sequence below is the one our runbooks use on Siebel CRM 19.x through the current 26.x monthly updates, with the Gateway registry, application containers and Application Interface architecture. On Windows the services are different, but the order and the checks are the same.

The enterprise order: who goes down first

Restarting one Siebel Server does not touch the Gateway. You still need the enterprise order, though, for host reboots, patching and any restart that goes beyond one server. The Oracle System Administration Guide is clear about the fixed points. The Siebel Gateway starts first and stops last, because everything else registers with it. The Siebel Server must reach both the Gateway and the database when it starts. The application container for each installed module has to be running before the Siebel Server system service starts.

We apply that as:

SHUTDOWN:  Application Interface
           -> Siebel Servers (stop_server, then each host's management container)
           -> Gateway (internal container, then the name server / registry: stop_ns)

STARTUP:   Gateway (start_ns, then its internal container)
           -> Siebel Servers (management container, then start_server)
           -> Application Interface
           -> application login test

We take the Application Interface down first and bring it up last on purpose. While it is down, users get a clean "unavailable" page. They don't get routed to object managers that are halfway through shutting down or haven't started yet. The management container (applicationcontainer_internal) on each Siebel Server host is a different matter. It belongs to the server tier and must be up before start_server runs.

Only stop the Gateway registry for a full shutdown such as a reboot or cutover. An ordinary container restart leaves it running. On Windows, stopping the Siebel Gateway Name Server service also offers to stop the Siebel Server service, so read that dialog before you click Yes.

Stage 1: capture a baseline before you stop anything

Start by writing down what "healthy" looks like for this server. After the restart you need something to compare against, and memory is not good enough. Use an enterprise-level Server Manager session (no -s flag) so the same commands still work while the server is down. If you leave out -p, srvrmgr prompts for the password, which keeps it out of your shell history.

srvrmgr -g <gateway_host>:<port> -e <enterprise> -u SADMIN

srvrmgr> list servers show SBLSRVR_NAME, SBLSRVR_STATE
srvrmgr> list compgrps for server <server> show CG_ALIAS, CA_RUN_STATE
srvrmgr> list comp

# Or spool the full component list to a file for the comparison later
# (-o logs the output of the command given with -c)
srvrmgr -g <gateway_host>:<port> -e <enterprise> -u SADMIN -p "$ADMIN_PW" \
  -c "list comp" -o <work_dir>/comp-before-<server>.txt

For each row where SV_NAME is your server, record CC_ALIAS, CP_DISP_RUN_STATE and CP_STARTMODE. Two groups matter most. The first is components that are Online or Running now: the restart is not finished until they are back. The second is components with start mode Manual that are running now. These are usually a Web Tools object manager or a batch component someone started by hand. They will not come back on their own after a restart, and that is how a "successful" restart turns into a ticket the next morning.

We filter the full list in the shell rather than writing clever list commands. In srvrmgr, show must be the last clause of a list command. Writing list ... show ... where ... fails with SBL-ADM-02532, and a mistyped column name aborts the command. Copy column names from the header of a bare list comp. Don't type them from memory.

If the server sits behind a load balancer with others, drain it first and restart one server at a time so the application stays available.

Stage 2: stop, and prove it is really stopped

Source the Siebel Server environment, stop the server, then wait for the processes to go. Don't trust the prompt coming back. siebsvc and the component processes need time to exit. Two to three minutes is normal on a busy server.

. <siebel_root>/siebsrvr/siebenv.sh
stop_server <server>          # or "stop_server all" if this host runs only one server

# Repeat until both commands print nothing (allow ~3 minutes)
ps -ef | grep "[s]iebsvc -s siebsrvr" | grep -- "-e <enterprise> -s <server>"
ps -ef | grep -E "[s]ieb(mtsh|mtshmw|proc|procmw) " | grep "<enterprise>\.<server>\.shm"

The bracket in [s]iebsvc stops grep from matching its own command line. Keep it in every pattern you script.

If processes are still there after the wait, read the server log under <siebel_root>/siebsrvr/enterprises/<enterprise>/<server>/log/ before you do anything else. If you have to remove them, send a normal termination signal first. kill -9 is a last resort, and anything killed that way leaves state behind that you have to clean up before the next start:

# 1. Terminate leftovers - anchored patterns, never broader than this server
pkill -f "^siebsvc -s siebsrvr .* -e <enterprise> -s <server>( |$)"
pkill -f "^sieb(mtsh|mtshmw|proc|procmw) .*<enterprise>\.<server>\.shm"
sleep 5
pkill -9 -f "^sieb(mtsh|mtshmw|proc|procmw) .*<enterprise>\.<server>\.shm"   # last resort

# 2. Only if something had to be killed: remove the orphaned OS data files
rm -f <siebel_root>/siebsrvr/admin/<enterprise>.<server>.shm \
      <siebel_root>/siebsrvr/sys/osdf.<enterprise>.<server>

# 3. Only if NO other Siebel process of this OS user runs on the host
#    (Linux ipcs layout: column 2 = id, column 3 = owner)
ipcs -m | awk -v u="$(id -un)" '$3==u {print $2}' | xargs -r ipcrm -m
ipcs -s | awk -v u="$(id -un)" '$3==u {print $2}' | xargs -r ipcrm -s

Be strict about the conditions. Deleting the .shm and osdf files under a healthy, running server corrupts its state. The IPC cleanup removes every segment and semaphore the OS user owns, so it is only safe when nothing else runs as that user, such as a Gateway registry or a second Siebel Server on the same host. On AIX the ipcs output has a different column layout, so check it before you adapt the filter.

Be just as clear about what you never clean up. The Oracle guide warns against copying or renaming the svc.siebsrvr.* files or files in the siebsrvr sys directory by hand. It also says not to delete the MW* files in /tmp while a Siebel Server is running, because components reuse them when they restart.

Stage 3: pre-start checks (disk, database, Gateway)

Each of these takes seconds, and each one catches a failure that would otherwise show up as a server stuck in "Starting" or a set of object managers that never come online.

Disk space. A full log or temp filesystem is one of the most common reasons a Siebel Server starts and then dies. Check the filesystem that holds the installation and logs against a minimum you set for each environment:

df -Pk <siebel_root> | awk 'NR==2 {printf "%.1f GB free on %s\n", $4/1048576, $6}'

Database login with odbcsql. The Siebel Server connects to the database through its ODBC data source as it starts. Test that exact path with the installation's own odbcsql utility from siebsrvr/bin, after sourcing siebenv.sh (which sets the ODBC environment). The DSN is normally named after the enterprise, as in <enterprise>_DSN.

. <siebel_root>/siebsrvr/siebenv.sh
cd <siebel_root>/siebsrvr/bin
timeout 45 ./odbcsql /s <enterprise>_DSN /u <db_user> /p "$DB_PW" <<'SQL'
select count(*) from <table_owner>.S_APP_VER;
quit
SQL

You should get one row with a count. Do not start the server if the output contains an ORA- error (for example ORA-01017 for invalid credentials, ORA-28000 for a locked account, or a TNS/listener error), an ODBC error, or a segmentation fault. Treat a timeout as a failure too. Don't rely on the exit code alone: we have seen ODBC test wrappers crash and still print a success message. Read what came back.

Gateway reachability. On the Gateway host, check that the name server and registry are listening by reading the socket table. Don't open a TLS connection to the Gateway port with curl or openssl. We have seen a failed handshake against the Gateway TLS listener stop it accepting connections, and the only fix was a container restart. From the Siebel Server host, an enterprise-level srvrmgr login proves both connectivity and credentials:

# Gateway host: processes and listeners, never a TLS probe
pgrep -af 'siebsvc -s gtwyns'
ss -ltn | grep -E ':(<gateway_port>|<registry_port>) '

# Siebel Server host: authenticate through the Gateway
srvrmgr -g <gateway_host>:<port> -e <enterprise> -u SADMIN
srvrmgr> list servers show SBLSRVR_NAME, SBLSRVR_STATE

While your server is down, the banner says something like "Connected to 0 server(s) out of a total of N server(s)", and a bare list comp returns every component definition with blank states. That is expected. It is the Gateway's catalogue of what the enterprise defines, not a status report.

Stage 4: start from a clean environment

Siebel's native tools (start_server, stop_server, start_ns, srvrmgr, odbcsql) resolve their bin, log and admin directories from the SIEBEL_ROOT variable. That variable must be the component root, <siebel_root>/siebsrvr or <siebel_root>/gtwysrvr, and it is set by that component's siebenv.sh. If a profile or wrapper script exports SIEBEL_ROOT with some other value, processes can start against the wrong paths. The Gateway service in particular can exit without writing a single log line. So source the component's siebenv.sh last, after anything else, and check the result.

Gotcha: never use set -u in scripts that source siebenv.sh

siebenv.sh (and cfgenv.sh) refer to variables that may not be set yet. LD_LIBRARY_PATH is the usual one on a clean login. With set -u, bash stops with an "unbound variable" error partway through sourcing the file, and you end up with a half-built environment. Leave set -u out of any wrapper that sources Siebel environment scripts, and check the variables you depend on explicitly.

For the Siebel Server, the start is guarded. If siebsvc for this server is already running, the script refuses to start rather than letting start_server run its cleanup against a live server. Also check that the management container on the host is running before you call start_server.

#!/bin/bash
# no "set -u" - see above
. <siebel_root>/siebsrvr/siebenv.sh
printenv SIEBEL_ROOT                 # must end in /siebsrvr

if pgrep -f "^siebsvc -s siebsrvr .* -e <enterprise> -s <server>( |$)" >/dev/null; then
  echo "siebsvc already running for <server> - refusing start_server" >&2
  exit 3
fi
pgrep -f '[j]ava .*applicationcontainer_internal' >/dev/null \
  || { echo "management container not running - start it first" >&2; exit 4; }

start_server <server>

If the Gateway is part of the restart, start it from an empty environment so that nothing from your login shell gets into the name server and registry processes:

env -i HOME="$HOME" PATH=/usr/bin:/bin:/usr/sbin:/sbin bash -c '
  . <siebel_root>/gtwysrvr/siebenv.sh
  printenv SIEBEL_ROOT               # must end in /gtwysrvr
  <siebel_root>/gtwysrvr/bin/start_ns -r <siebel_root>/gtwysrvr'
# add JAVA_HOME inside the env -i list if your gtwysrvr environment does not set it

# A zero exit is not success - wait for the listeners
ss -ltn | grep -E ':(<gateway_port>|<registry_port>) '

The same rule applies to every start script here. A zero exit code from start_ns or start_server only means the launch worked. The daemon can still die a few seconds later. Success is the process and the listener, and for a Siebel Server, the Server Manager state.

Stage 5: verify against the baseline, then log in

siebsvc typically needs a couple of minutes before it spawns components. Allow about five minutes for the server to reach Running. Poll Server Manager until it does:

srvrmgr> list servers show SBLSRVR_NAME, SBLSRVR_STATE
srvrmgr> list compgrps for server <server> show CG_ALIAS, CA_RUN_STATE
srvrmgr> list comp

Running only tells you the server process is up. Object manager groups often show Offline for the first few minutes after start_server while they load. Groups made up of batch-only components (EAI, Remote, Workflow) normally stay at "Part offline". That is fine as long as it matches the baseline. Compare the component list with what you captured in Stage 1: every component that was Online or Running before should be Online or Running again, with nothing stuck in a starting state.

Next, restore the manual-mode components that were running before the restart. For each one:

srvrmgr> startup component <component_alias> for server <server>

Leave the start mode as it is. If a component should really come back on its own every time, change it once with auto start comp. Don't paper over it in every restart.

Last, test what users actually do: log in through the Application Interface. A login that goes through the AI exercises the whole chain: AI, Gateway, object manager, database authentication. Where the Siebel REST API is enabled, a single authenticated request is enough. curl prompts for the password when you pass only the user name.

curl -s --cacert <ca_bundle.pem> -u SADMIN -o /dev/null -w '%{http_code}\n' \
  "https://<ai_host>:<ai_port>/siebel/v1.0/data/Employee/Employee?PageSize=1"

HTTP 200 means the login worked. So does an SBL-EAI-50297 response, which is raised only after authentication and means the object is not exposed over REST. HTTP 401 means authentication failed. Any other result in the first few minutes usually means the object managers are still starting, so retry for a few minutes before you call it a failure. Without REST, log in through the browser with a real user and open a view backed by the restarted server. Then put the server back into the load balancer.

The checklist on one card

1. Capture the baseline: server state, component groups, full component list, manual-mode components that are running. 2. Drain the server from the load balancer. 3. stop_server, then wait until siebsvc and the component processes are gone. 4. Clean leftovers only if needed, and if anything was killed, remove the shm and osdf files. 5. Check disk space, the odbcsql login and Gateway listeners. 6. Guarded start_server from a clean environment, with no set -u. 7. Server Manager reports Running. 8. Start the manual-mode components that were running before. 9. Components match the baseline. 10. Application login through the AI, then return the server to the load balancer.

How CoreOps runs this as an audited staged job

We wrote this checklist down because we kept seeing the same failures, and then we built it into CoreOps, our monitoring and operations platform for Siebel CRM. A Siebel Server restart in CoreOps is a queued job with named stages, run over SSH as the Siebel OS user with the installation's own scripts. Nothing is installed on the Siebel hosts.

The stop stages are stop_server, waiting for the processes to exit, cleaning up leftovers, and confirming that Server Manager no longer reports the server Running. The start stages are the not-already-running check, free disk space, the odbcsql database login, temp and cache cleanup, the management container, start_server, waiting for the processes, Server Manager reporting Running, restoring manual-mode components that were online before, comparing components with the last healthy baseline, and the application login test through the Application Interface. If a stage fails, the job stops there. If the database login fails, the server is not started. start_server is never sent to a server whose siebsvc is running. Enterprise-wide restarts follow the same order described above.

Every step shows its live output, and every job is recorded with who ran it, when and with what result, under role-based permissions for each environment. When someone asks what happened during last night's restart, the answer is a job record, not someone's memory.

About CoretiTech

Written by Mahmoud Ali, CoretiTech.

CoretiTech provides Siebel CRM support, upgrades and development, and builds CoreOps for fast Siebel monitoring and safe day-to-day operations.

Contact us →
Restarting a Siebel Server Safely: A Staged Checklist | CoretiTech