Fixing UnicodeDecodeError: 'utf-8' codec can't decode byte (cp949 · euc-kr)

A Complete Fix for UnicodeDecodeError: 'utf-8' codec can't decode byte (cp949 · euc-kr)

Python Development Guide, Part 3

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xec in position 0

If you copied that error message straight into a search box, you're in the right place. The short version: don't swap encodings by guesswork. Cycling utf-8 → cp949 → euc-kr → latin1 might luckily decode once, but then you still don't know why it worked, so the next file will fail again. This post finishes the job in three steps: (1) diagnose the real encoding in one line → (2) classify the cause from the byte value and position → (3) copy-paste snippets for each situation.

Step 1: No guessing — confirm the actual encoding first

Run this first. Just swap in your file path in the terminal.

# charset-normalizer (요즘 사실상 표준, chardet보다 빠름)
pip install charset-normalizer
python -m charset_normalizer your_file.csv

If you want to use it inside a script:

from charset_normalizer import from_path
result = from_path("your_file.csv").best()
print(result.encoding)   # 예: 'cp949', 'utf_8', 'euc_kr'

# (구버전 chardet을 쓴다면)
import chardet
with open("your_file.csv", "rb") as f:
    print(chardet.detect(f.read()))
    # {'encoding': 'EUC-KR', 'confidence': 0.99, ...}

Once you have a diagnosis, drop that encoding straight into encoding= and you're done. If you still want a feel for it, you can roughly classify from the byte value in the error message.

Cause classification table: byte value and position

If it blows up at position 0, it's the first character or a BOM, so diagnosis is easy. If the position is a large number, the leading ASCII (English) decoded fine and it broke on the first Hangul character.

Step 2: Four copy-paste fixes by situation

Plug the encoding you diagnosed into each situation.

① Reading a regular file — open()

# ❌ Before: encoding 미지정 → OS 기본값(윈도우=cp949)에 의존
with open("data.txt") as f:
    text = f.read()   # UnicodeDecodeError 발생

# ✅ After: encoding을 명시
with open("data.txt", encoding="cp949") as f:   # 또는 "utf-8"
    text = f.read()

② pandas read_csv

import pandas as pd

# ❌ Before
df = pd.read_csv("sales.csv")   # 'utf-8' codec can't decode byte 0xb0 ...

# ✅ After: 엑셀이 만든 한글 CSV는 보통 cp949
df = pd.read_csv("sales.csv", encoding="cp949")

# ✅ "CSV UTF-8(쉼표로 분리)"로 저장한 파일 → BOM 제거
df = pd.read_csv("sales.csv", encoding="utf-8-sig")

💡 Hangul CSVs from Excel are almost always cp949. If you saved from Excel with the "CSV UTF-8" option, use utf-8-sig.

③ subprocess

import subprocess

# ❌ Before: bytes로 받아 디코딩 시 깨짐
out = subprocess.run(["git", "log"], capture_output=True).stdout
# out.decode()  # 윈도우 한글 출력에서 에러

# ✅ After: text=True + encoding 명시
out = subprocess.run(
    ["git", "log"],
    capture_output=True,
    text=True,
    encoding="utf-8",   # 윈도우 콘솔 출력이면 "cp949"가 맞을 때도 많음
).stdout

④ requests web responses

import requests
r = requests.get("https://example.co.kr/legacy.html")

# 케이스 A: 서버가 euc-kr인데 헤더가 부실해 requests가 잘못 추측
text = r.content.decode("euc-kr")   # bytes를 직접 디코딩

# 케이스 B: r.text를 쓰고 싶으면 인코딩을 강제 지정
r.encoding = "utf-8"   # 이 줄 이후 r.text가 올바르게 디코딩됨
text = r.text

r.text is decoded with r.encoding, so if the header guess is wrong, pin r.encoding first or decode r.content yourself. That's the safe path.

Step 3: When it still fails — the errors option and BOM

Use these only as a temporary measure when you truly don't know the encoding, or a single file mixes encodings. Understand the data-loss risk before you do.

with open("dirty.txt", encoding="utf-8", errors="replace") as f:
    text = f.read()

errors option comparison

Key warning: errors="ignore" is not "no error, so we're good." It is the most dangerous option because characters vanish silently. Never use it on amounts, quantities, or similar data. The only reasonable use is a brief backslashreplace while debugging so you can see byte positions.

Handling a BOM (0xef 0xbb 0xbf)

If an invisible character like \ufeff is glued to the first column name at position 0 and you get a KeyError, that's a BOM. Read with utf-8-sig instead of utf-8 and it is stripped automatically.

with open("excel_export.csv", encoding="utf-8-sig") as f:
    ...

A note from the field

When I build a new pipeline, I always run python -m charset_normalizer on the first line and log the encoding. In production, about 80% of "the batch that worked yesterday suddenly broke" incidents are because the data vendor switched from UTF-8 to cp949 (or the other way around). Automate the diagnosis and root-cause time drops from 5 minutes to 5 seconds.

Windows tips (PEP 686, PYTHONUTF8)

Per PEP 686, Python is moving toward UTF-8 mode as the default starting in 3.15. Until then, if Windows cp949 defaults are making you miserable, you can force it with an environment variable.

# 윈도우(PowerShell)
$env:PYTHONUTF8 = "1"
python app.py

# 또는 실행 시 옵션
python -X utf8 app.py

That makes UTF-8 the default encoding for open(). But if your code reads a cp949 file, you still must specify encoding="cp949". Changing the default does not change the file's own encoding.

Wrap-up: diagnose → fix, a 3-step checklist

1. Diagnose: confirm the real encoding with python -m charset_normalizer filename (no guessing)
2. Classify: infer the cause from the error's byte value — 0xec range = UTF-8 Hangul, 0xb0–0xc8 range = cp949/euc-kr, 0xef bb bf = BOM
3. Fix: open(encoding=) / read_csv(encoding="cp949"|"utf-8-sig") / subprocess(text=True, encoding=) / for requests, r.content.decode() or set r.encoding

errors="ignore" is not even a last resort. Data disappears — avoid it.

FAQ

Q. A Hangul .txt saved from Windows Notepad won't read. A. Notepad often saves as cp949 (ANSI) by default. Read with open("f.txt", encoding="cp949"). Re-save from Notepad via "Save As → Encoding: UTF-8" and you can standardize on utf-8 after that.

Q. Hangul in an Excel-made CSV is garbled in pandas. A. Excel Hangul CSVs are usually encoding="cp949". If you saved as "CSV UTF-8 (Comma delimited)", encoding="utf-8-sig" handles the BOM as well.

Q. I have a mix of euc-kr legacy files and utf-8 files. How do I handle them in one go? A. Detect encoding per file with charset_normalizer first, then open with that value — automate it. If encodings are mixed inside a single file, use errors="backslashreplace" to find the broken byte positions and ask the source vendor to unify the encoding. That's the proper approach.

Coming next — Part 4: Troubleshooting SSL/certificate errors that often blow up after Python venv and package installs