systemd Auto-Restart Guide: Restart=always vs on-failure and Fixing start-limit-hit

The Complete systemd Auto-Restart Guide: Restart=always vs on-failure and Fixing start-limit-hit

Nobody Restarts the Process When It Dies

It's 3 a.m. and the service died quietly. If you've ever SSH'd in and run systemctl start only after a monitoring alert fired, you're not alone. On a single VM or bare metal without an orchestrator like Kubernetes, someone has to own "bring it back if it dies." These days the de facto standard is to standardize on systemd rather than running supervisord or pm2 separately. The unit is already there, and you get boot-time autostart, log collection (journald), and dependency management in one place.

Plenty of people slap on Restart=always and then fall into the trap where the service never comes back because of "start request repeated too quickly". This post covers the exact differences among Restart options, how to diagnose and fix the infinite-restart trap, and a full copy-paste .service sample.

The 7 Restart Options and RestartSec Compared

The most confusing part is the difference between always and on-failure. The key question is whether the service should be started again when the process exits cleanly on its own (exit 0). Restart= accepts the 7 values below. systemd.service(5)

A "please shut down cleanly" signal like SIGTERM is treated as a clean exit by default, so on-failure will not restart. SIGKILL, SIGSEGV, SIGABRT, and similar are treated as abnormal and will restart. To treat specific exit codes as success use SuccessExitStatus=; to exclude them from restarts use RestartPreventExitStatus=.

systemctl stop never triggers a restart, whatever the Restart value. A stop requested by an administrator with systemctl stop (or systemctl kill) is a stop that systemd itself carried out, so it is not a restart case. Even with always, the service stays down after a stop.

When to use which?

• Web servers, APIs, daemons — long-running services that should always be up → both on-failure and always are used. They only differ when the process exits cleanly on its own. If a clean exit 0 from a long-running service also counts as an outage, pick always; if it counts as a normal shutdown, pick on-failure.
• Must always be running, and a clean exit is itself abnormal → always.
• Batch jobs → on-failure or no, because they must not run again after completing successfully (exit 0).
• Services health-checked with a watchdog (WatchdogSec=) where you only want to recover from hangs → on-watchdog.

RestartSec: Pause Before Restarting

RestartSec=5s

This is the wait before a restart. The default is 100ms. Leave that as-is and, when the DB is down or a port is blocked, the service will try to boot dozens of times per second and cause CPU spikes and a flood of DB connections. In production, RestartSec=5 (seconds) is a good default. It gives external dependencies (DB, message queue) time to recover.

The Infinite-Restart Trap: Why start-limit-hit Appears

This is the heart of the article. systemd has a safety valve: if restarts happen too quickly, it gives up.

[Unit]
# systemd defaults
StartLimitIntervalSec=10
StartLimitBurst=5

That means "if you restart more than 5 times within 10 seconds, stop trying to bring it up." When that trips, the logs look like this.

myapp.service: Start request repeated too quickly.
myapp.service: Failed with result 'start-limit-hit'.
Failed to start myapp.service.

If ExecStart fails immediately (typo in config, port conflict, etc.) and RestartSec is short, you blow past 5 attempts in 10 seconds and land in this state. From then on, even systemctl start will not bring it up. The counter is locked.

When the limit trips, systemctl status looks like this.

● myapp.service - My App Daemon
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
     Active: failed (Result: start-limit-hit) since Fri 2026-07-10 09:14:22 KST

Jul 10 09:14:22 host systemd[1]: myapp.service: Scheduled restart job, restart counter is at 3.
Jul 10 09:14:22 host systemd[1]: myapp.service: Start request repeated too quickly.
Jul 10 09:14:22 host systemd[1]: myapp.service: Failed with result 'start-limit-hit'.

If you see Result: start-limit-hit, the service isn't "dead and failing to come back"; it died so often that systemd gave up restarting it.

Three Fixes

① Adjust the limits themselves

[Unit]
StartLimitIntervalSec=60
StartLimitBurst=3

② Increase RestartSec to avoid hitting the burst — there is a key formula.

RestartSec × StartLimitBurst > StartLimitIntervalSec

If this inequality holds, it is physically impossible to fill the burst count inside the interval, so you never hit start-limit-hit. For example, with RestartSec=5 and StartLimitBurst=3, 5×3=15, so if you set StartLimitIntervalSec smaller than 15 (e.g. 10), you will not hit the limit. Conversely, if you want a truly broken service to stop quickly, you can deliberately break the formula so it stops after a fixed number of attempts.

③ Disable the limit entirely — if you want it to keep restarting no matter what:

[Unit]
StartLimitIntervalSec=0

When this is 0, counting is disabled and it retries forever. (If there is a real bug it will chew CPU, so always give RestartSec a generous value.)

systemctl reset-failed: Unlock the Counter

Once you are in start-limit-hit, fixing the unit and running start is not enough. You must reset the counter first.

# 1) unit file after
sudo systemctl daemon-reload

# 2) failed ( !)
sudo systemctl reset-failed myapp.service

# 3) when
sudo systemctl start myapp.service

# 4) failed check
systemctl status myapp.service
systemctl is-failed myapp.service # 'active' or 'inactive'

A lot of people wander around wondering "why won't start work?" without reset-failed. If you see a start-limit log, think of this command first.

Diagnosing Why It Died, plus a Production Unit File

Before you turn on auto-restart, look at why it died. journald already recorded it.

# 50 ( without )
journalctl -u myapp.service -n 50 --no-pager

# 10
journalctl -u myapp.service --since "10 min ago"

# quit /when
systemctl status myapp.service

# when check (·daemon-reload )
systemctl show myapp.service -p Restart -p RestartSec -p StartLimitBurst -p StartLimitIntervalUSec

The last lines of systemctl status give clues like Main PID exited, code=exited, status=1 or code=killed, signal=SEGV. Figure out the exit reason first, then choose a Restart policy.

A note from production: I once left RestartSec short, hit an external DB outage, and watched restart storms drain the DB connection pool even faster. Since then I start long-running services with on-failure + RestartSec=5 + a generous StartLimit as the default. "Restart safely" beats "restart fast" for availability.

A Complete .service Sample You Can Copy

/etc/systemd/system/myapp.service:

[Unit]
Description=My Application Service
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5
# / journald
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Apply it:

# file save after systemd
sudo systemctl daemon-reload

# boot when + when
sudo systemctl enable --now myapp.service

# check
systemctl status myapp.service

Note: StartLimitIntervalSec/StartLimitBurst are [Unit] section directives (under these names since systemd 230). StartLimitIntervalSec under [Service] is ignored as an "Unknown key"; for compatibility [Service] only accepts the old names StartLimitInterval and StartLimitBurst. Check the values in effect with systemctl show myapp.service -p StartLimitIntervalUSec -p StartLimitBurst, and syntax warnings with systemd-analyze verify /etc/systemd/system/myapp.service. systemd.unit(5)

Expected results after applying, and what to do if they differ:

• daemon-reload — no output means success.
• status showing Active: active (running) is healthy; Loaded: ... enabled means start at boot is on too.
• inactive (dead) → you didn't start it, or only ran enable. Run sudo systemctl start myapp.service.
• failed (Result: exit-code) → the app itself exited abnormally. Check the app error with journalctl -u myapp.service -n 50.
• failed (Result: start-limit-hit) → fix the cause as in the StartLimit section above, then reset-failed.
• If journalctl -u myapp.service -f keeps printing Scheduled restart job, you are in a crash loop.

Conclusion: Auto-Restart Checklist

For a long-running service, these are the recommended defaults to start from.

• ✅ Restart=on-failure — do not start again when the process exits cleanly on its own (exit 0) (systemctl stop never restarts, whatever the value)
• ✅ RestartSec=5 — prevent CPU/DB storms
• ✅ StartLimitIntervalSec=60, StartLimitBurst=3 in [Unit] — stop a truly broken service
• ✅ If you truly need infinite retries, StartLimitIntervalSec=0
• ✅ If you hit start-limit-hit → daemon-reload → reset-failed → start
• ✅ Before choosing a policy, check the exit reason with journalctl -u

Remember just the formula RestartSec × StartLimitBurst > StartLimitIntervalSec and it will help you avoid the infinite-restart trap.

FAQ

Q. "start request repeated too quickly" appears and systemctl start will not bring the service up. A. The start-limit counter is locked. Reset it with sudo systemctl reset-failed <service-name>, then run sudo systemctl start <service-name> again. If you do not fix the root cause (ExecStart failing immediately), you will hit the limit again, so check the exit reason with journalctl -u as well.

Q. Should I use Restart=always or on-failure? A. systemctl stop triggers no restart under either value, so the only difference is what happens when the process exits cleanly on its own (exit 0). Use always if it must be started again in that case, and on-failure if that should count as a normal shutdown (batch jobs, for example).

Q. I changed the settings but they are not taking effect. A. After editing a unit file you must run sudo systemctl daemon-reload first. Then confirm the values actually in effect with systemctl show <service> -p Restart -p RestartSec -p StartLimitBurst.

Q. What is the difference between enable and start? A. enable registers the service to start at boot; start runs it right now. They are independent: with only enable it won't run until the next reboot, and with only start it won't come back after a reboot. If you want both, use sudo systemctl enable --now <service-name>.