systemd Restart=always vs on-failure: Examples and How to Prevent Infinite Restarts

The process died and nobody brought it back

Your app process dies quietly in the middle of the night, and you only find out in the morning when someone asks, "the service isn't working." If you run custom daemons on Linux servers, you've probably been there. Fortunately, if you're on systemd, this is a one- or two-line unit file fix.

Containers and Kubernetes may look like the default, but for workloads on bare metal and VMs, systemd is still the de facto process supervisor. Before you move on to Kubernetes restartPolicy, or in hybrid environments, making a service that "comes back on its own when it dies" is still a core skill.

This post starts with copy-paste unit file templates, then covers how to pick the right Restart option among the six available, and the practical settings that stop infinite restart loops when the config is wrong.

• Scope: systemd-based distros (Ubuntu 18.04+, RHEL/Rocky 8+, Debian 10+, and similar), targeting systemd 245+. We focus on the StartLimitIntervalSec spelling that stabilized in 250+.

Minimal copy-paste template: auto-restart in three lines

Start with the simplest form. Create /etc/systemd/system/myapp.service and paste the following.

[Unit]
Description=My App Daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The important part is three lines in the [Service] section.

• ExecStart — the command to run. Always use an absolute path. A relative path like myapp, or anything that depends on PATH, will fail to start.
• Restart=on-failure — bring it back on an unclean exit.
• RestartSec=5 — wait 5 seconds after it dies before restarting. If omitted, the default is 100ms, which can burn CPU in a crash loop, so set it explicitly.

Production-ready version with loop protection

In production, add a safety net against infinite restarts. The unit file below is complete enough to paste into production as-is.

[Unit]
Description=My App Daemon
After=network.target
# Allow at most 3 restarts inside a 60-second window
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /opt/myapp/config.yaml
Restart=on-failure
RestartSec=5
# Send logs to the journal
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Version note: StartLimitIntervalSec and StartLimitBurst belong in the [Unit] section (not [Service]). On systemd older than 230, the option was StartLimitInterval (no Sec). Current versions accept both, but for new files use StartLimitIntervalSec. Check your version with systemctl --version first.

Restart options compared: always for web servers, on-failure for batch jobs

Restart= has six values, and whether a restart happens depends on how the process exited (clean / unclean / signal). Pick the wrong one and you get "it keeps coming back after a clean exit" or "it died and never came back."

Use this selection guide.

• Always-on daemons that must never stay down, such as web/API servers → Restart=always. Restart no matter why it exited.
• Batch/worker jobs where exit 0 means "the work is done" → Restart=on-failure. Retry only on failure; stay down after a clean finish.
• When you don't care about exit codes and only want to recover from crashes or forced kills → on-abnormal.

A common point of confusion: systemctl stop does not trigger a restart, regardless of the Restart value. A stop signal systemd sends on purpose is treated as a normal stop. So even with always, if an admin stops the service, it stays down. That is by design, not a bug.

Preventing infinite restart loops: how StartLimit works

You set Restart=always, but the app dies immediately because of a config error? It will restart forever at RestartSec intervals, chewing through journal logs and CPU. StartLimitIntervalSec and StartLimitBurst stop that.

The rule is simple.

If restarts exceed StartLimitBurst (count) inside the StartLimitIntervalSec (time window), systemd gives up and pins the service in the failed state.

For example, StartLimitIntervalSec=60 and StartLimitBurst=3 from the unit above work like this:

• Look at a 60-second window.
• Allow up to 3 start attempts inside it.
• If a 4th attempt falls inside that 60-second window → stop restarting, start-limit-hit.

If the app dies immediately and RestartSec=5: start at 0s → die → restart at 5s → die → restart at 10s → die → 4th attempt at 15s... that 4th attempt is still inside the 60-second window, so it stops there.

Checking the start-limit-hit logs

When the limit is hit, systemctl status looks like this:

● myapp.service - My App Daemon
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
     Active: failed (Result: start-limit-hit) since Fri 2026-07-10 09:14:22 KST
   Duration: 15s

Jul 10 09:14:22 host systemd[1]: myapp.service: Scheduled restart job, restart counter is at 3.
Jul 10 09:14:22 host systemd[1]: Stopped My App Daemon.
Jul 10 09:14:22 host systemd[1]: myapp.service: Start request repeated too quickly.
Jul 10 09:14:22 host systemd[1]: myapp.service: Failed with result 'start-limit-hit'.
Jul 10 09:14:22 host systemd[1]: Failed to start My App Daemon.

The key phrases are Start request repeated too quickly and Result: start-limit-hit. When you see these, the service is not "dead and not coming back"—systemd gave up because it died too often. Restarting again without fixing the root cause (bad config, port conflict, and so on) will just hit the limit again.

Recovery: reset-failed

After fixing the cause, reset the counter and start again:

# Reset the failure counter (clear start-limit-hit)
sudo systemctl reset-failed myapp
# Start again
sudo systemctl start myapp

If you start without reset-failed, you can still hit the limit again if the time window has not expired.

Apply and verify: a command runbook

Once the unit file is saved, apply and verify in this order. Order matters.

# 1) Always run this after creating or editing a unit file
#    systemd re-reads the .service file from disk into memory
sudo systemctl daemon-reload

# 2) Enable on boot and start immediately (--now) in one shot
sudo systemctl enable --now myapp

# 3) Check current status — is it active (running), and did Restart take effect?
systemctl status myapp

# 4) Follow logs in real time — watch crash causes and the restart flow
journalctl -u myapp -f

# 5) After a failed state such as start-limit-hit, reset the counter and recover
sudo systemctl reset-failed myapp

Expected healthy results for each command:

• daemon-reload — no output means success. If you get an error, it is a unit file syntax problem; check the line number in the message.
• status — Active: active (running) in green is healthy. Loaded: ... enabled means boot autostart is on.
• journalctl -u myapp -f — you should see the app's normal startup logs. Repeated Scheduled restart job lines are a crash-loop signal.

When results don't match:

• status shows inactive (dead) → you never ran start, or you only ran enable. Run sudo systemctl start myapp.
• failed (Result: exit-code) → the app itself exited uncleanly. Check app errors with journalctl -u myapp -n 50.
• failed (Result: start-limit-hit) → see the StartLimit section above; fix the cause, then reset-failed.

Frequently asked questions (FAQ)

Q1. I edited the unit file but nothing changed. Even after I change Restart, it stays the same.

You did not run sudo systemctl daemon-reload. systemd does not re-read the .service file from disk every time; it uses the copy loaded in memory. After editing the file, you must daemon-reload and then re-apply with sudo systemctl restart myapp. Remember the three-step set: edit → daemon-reload → restart.

Q2. I set Restart=always, but the process does not come back after a clean exit (exit 0)—or the opposite, it keeps coming back.

always restarts on every exit, including a clean one. If a batch job finishing cleanly and then starting again is the problem, switch to Restart=on-failure. If it does not come back after a clean exit, you are either on Restart=no (the default), or an admin stopped it with systemctl stop. Remember that stop does not trigger a restart for any Restart value.

Q3. It restarts a few times and then suddenly stops.

You hit the StartLimitIntervalSec / StartLimitBurst limit. Result: start-limit-hit in systemctl status myapp confirms it. Fix the root cause, then recover with sudo systemctl reset-failed myapp → sudo systemctl start myapp. To loosen the limit itself, raise StartLimitBurst or adjust StartLimitIntervalSec in the [Unit] section.

Q4. What is the difference between enable and start?

enable registers the unit to start on boot. start runs it right now. They are independent: enable alone does nothing until the next reboot, and start alone runs it now but it will not come back after a reboot. If you want both, the production default is sudo systemctl enable --now myapp.

Once this is in place, you have a service that comes back on its own when it dies—without falling into an infinite loop. To dig into the crash itself, follow up with "Finding service crash causes by analyzing journalctl logs"; for status values, "How to read systemctl status (failed/activating/dead)"; for boot-order dependencies, "Linux boot-time service start order (After=/Requires=)". Together those complete a troubleshooting runbook. For exact option behavior, check man systemd.service and the official systemd docs (freedesktop.org) for the version you are running.