systemd Auto-Restart Guide: Restart=always vs on-failure and Fixing start-limit-hit

The Complete systemd Auto-Restart Guide: Restart=always vs on-failure and Fixing start-limit-hit

Nobody Restarts the Process When It Dies

It's 3 a.m. and the service died quietly. If you've ever SSH'd in and run systemctl start only after a monitoring alert fired, you're not alone. On a single VM or bare metal without an orchestrator like Kubernetes, someone has to own "bring it back if it dies." These days the de facto standard is to standardize on systemd rather than running supervisord or pm2 separately. The unit is already there, and you get boot-time autostart, log collection (journald), and dependency management in one place.

Plenty of people slap on Restart=always and then fall into the trap where the service never comes back because of "start request repeated too quickly". This post covers the exact differences among Restart options, how to diagnose and fix the infinite-restart trap, and a full copy-paste .service sample.

Restart Options and RestartSec: A Full Comparison

The most confusing part is the difference between always and on-failure. The key question is whether a clean exit (exit 0) should also trigger a restart.

A "please shut down cleanly" signal like SIGTERM is treated as a clean exit by default, so on-failure will not restart. SIGKILL, SIGSEGV, SIGABRT, and similar are treated as abnormal and will restart.

When to use which?

• Web servers, APIs, daemons — long-running services that should always be up → on-failure is the standard. If an operator intentionally stops it with systemctl stop (exit 0), it will not come back, which is safer.
• Must always be running, and a clean exit is itself abnormal → always. Note that it can come back even after systemctl stop, which can confuse operators.
• Batch jobs → on-failure or no.

RestartSec: Pause Before Restarting

RestartSec=5s

This is the wait before a restart. The default is 100ms. Leave that as-is and, when the DB is down or a port is blocked, the service will try to boot dozens of times per second and cause CPU spikes and a flood of DB connections. In production, RestartSec=5 (seconds) is a good default. It gives external dependencies (DB, message queue) time to recover.

The Infinite-Restart Trap: Why start-limit-hit Appears

This is the heart of the article. systemd has a safety valve: if restarts happen too quickly, it gives up.

StartLimitIntervalSec=10
StartLimitBurst=5

That means "if you restart more than 5 times within 10 seconds, stop trying to bring it up." When that trips, the logs look like this.

myapp.service: Start request repeated too quickly.
myapp.service: Failed with result 'start-limit-hit'.
Failed to start myapp.service.

If ExecStart fails immediately (typo in config, port conflict, etc.) and RestartSec is short, you blow past 5 attempts in 10 seconds and land in this state. From then on, even systemctl start will not bring it up. The counter is locked.

Three Fixes

① Adjust the limits themselves

StartLimitIntervalSec=60
StartLimitBurst=3

② Increase RestartSec to avoid hitting the burst — there is a key formula.

RestartSec × StartLimitBurst > StartLimitIntervalSec

If this inequality holds, it is physically impossible to fill the burst count inside the interval, so you never hit start-limit-hit. For example, with RestartSec=5 and StartLimitBurst=3, 5×3=15, so if you set StartLimitIntervalSec smaller than 15 (e.g. 10), you will not hit the limit. Conversely, if you want a truly broken service to stop quickly, you can deliberately break the formula so it stops after a fixed number of attempts.

③ Disable the limit entirely — if you want it to keep restarting no matter what:

StartLimitIntervalSec=0

When this is 0, counting is disabled and it retries forever. (If there is a real bug it will chew CPU, so always give RestartSec a generous value.)

systemctl reset-failed: Unlock the Counter

Once you are in start-limit-hit, fixing the unit and running start is not enough. You must reset the counter first.

# 1) unit 파일 수정 후
sudo systemctl daemon-reload

# 2) 실패 카운터 리셋 (이게 핵심!)
sudo systemctl reset-failed myapp.service

# 3) 다시 기동
sudo systemctl start myapp.service

# 4) failed 상태가 풀렸는지 확인
systemctl status myapp.service
systemctl is-failed myapp.service   # 'active' 또는 'inactive'면 정상

A lot of people wander around wondering "why won't start work?" without reset-failed. If you see a start-limit log, think of this command first.

Diagnosing Why It Died, plus a Production Unit File

Before you turn on auto-restart, look at why it died. journald already recorded it.

# 최근 50줄 (페이저 없이 바로 출력)
journalctl -u myapp.service -n 50 --no-pager

# 최근 10분간 로그만
journalctl -u myapp.service --since "10 min ago"

# 현재 상태와 마지막 종료 코드/시그널
systemctl status myapp.service

# 실제 적용된 재시작 관련 값 확인 (오타·daemon-reload 누락 점검)
systemctl show myapp.service -p Restart -p RestartSec -p StartLimitBurst -p StartLimitIntervalUSec

The last lines of systemctl status give clues like Main PID exited, code=exited, status=1 or code=killed, signal=SEGV. Figure out the exit reason first, then choose a Restart policy.

A note from production: I once left RestartSec short, hit an external DB outage, and watched restart storms drain the DB connection pool even faster. Since then I start long-running services with on-failure + RestartSec=5 + a generous StartLimit as the default. "Restart safely" beats "restart fast" for availability.

A Complete .service Sample You Can Copy

/etc/systemd/system/myapp.service:

[Unit]
Description=My Application Service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3
# 표준출력/에러를 journald로
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Apply it:

# 파일 저장 후 systemd에 변경 알림
sudo systemctl daemon-reload

# 부팅 자동 시작 등록 + 즉시 기동
sudo systemctl enable --now myapp.service

# 잘 떴는지 확인
systemctl status myapp.service

Note: StartLimitIntervalSec/StartLimitBurst are originally [Unit] section directives. Putting them under [Service] as above works on most recent systemd versions, but if behavior looks wrong, move them to [Unit].

Conclusion: Auto-Restart Checklist

For a long-running service, starting with these defaults is safe about 90% of the time.

• ✅ Restart=on-failure — do not come back after an intentional stop (exit 0)
• ✅ RestartSec=5 — prevent CPU/DB storms
• ✅ StartLimitIntervalSec=60, StartLimitBurst=3 — stop a truly broken service
• ✅ If you truly need infinite retries, StartLimitIntervalSec=0
• ✅ If you hit start-limit-hit → daemon-reload → reset-failed → start
• ✅ Before choosing a policy, check the exit reason with journalctl -u

Remember just the formula RestartSec × StartLimitBurst > StartLimitIntervalSec and you will dodge half of the infinite-restart trap.

FAQ

Q. "start request repeated too quickly" appears and systemctl start will not bring the service up. A. The start-limit counter is locked. Reset it with sudo systemctl reset-failed <service-name>, then run sudo systemctl start <service-name> again. If you do not fix the root cause (ExecStart failing immediately), you will hit the limit again, so check the exit reason with journalctl -u as well.

Q. Should I use Restart=always or on-failure? A. For a typical long-running service where it would be a problem if it came back after an operator ran systemctl stop, on-failure is safer. Use always only in the special case where even a clean exit (exit 0) must immediately start the process again.

Q. I changed the settings but they are not taking effect. A. After editing a unit file you must run sudo systemctl daemon-reload first. Then confirm the values actually in effect with systemctl show <service> -p Restart -p RestartSec -p StartLimitBurst.