LLM 에이전트와 에러 피드백 재시도 기능을 사용하여 정돈되지 않은 채용 공고를 스키마 검증이 완료된 타입 지정 JSON으로 변환합니다.
다른 이름 nilayp-codes/typecast, typecast-mcp
- 스타
- 1
- Tools
- —
- 다운로드, 30일
- 180
- 최신 버전
- 0.1.0
- 마지막 커밋
- 2026년 11월 6일
- 라이선스
- MIT
Typecast MCP란?
Typecast은(는) MCP server입니다. LLM 에이전트와 에러 피드백 재시도 기능을 사용하여 정돈되지 않은 채용 공고를 스키마 검증이 완료된 타입 지정 JSON으로 변환합니다.
프로젝트 README에서 생성된 요약
Typecast MCP server 정보
비정형 채용 공고를 구조화된 타입 데이터로 변환해야 하는 개발자는 Typecast를 사용하여 추출 프로세스를 자동화할 수 있습니다. 이 도구는 LLM 에이전트를 활용해 정돈되지 않은 텍스트를 스키마 검증이 완료된 JSON으로 변환하며, 특히 신뢰할 수 있는 채용 공고 모델을 생성하는 데 특화되어 있습니다. OpenRouter를 통해 Qwen과 같은 오픈 웨이트 모델과 연동하여 작동하도록 설계되었으며, 다양한 프롬프트 전략의 정확도와 비용 효율성을 측정할 수 있는 방법을 제공합니다.
시스템은 Pydantic 모델로부터 프롬프트를 생성하여 스키마를 수동으로 작성할 필요가 없도록 운영됩니다. 일반 프롬프팅, 프로바이더별 스키마 모드, 그리고 검증 오류를 수정하기 위해 최대 3회까지 시도할 수 있는 에러 피드백 재시도 루프를 포함한 여러 운영 모드를 지원합니다. 사용자는 저비용 또는 높은 엔드투엔드 정확도 중 우선순위에 따라 설정을 선택할 수 있으며, MCP server가 가공되지 않은 문자열을 예상되는 JSON 스키마와 대조하여 검증하고 추출 실패를 기록하는 복잡한 작업을 처리합니다.
활용 사례
- 1채용 텍스트를 JSON으로 변환정돈되지 않은 채용 공고를 구조화되고 스키마 검증이 완료된 JSON 객체로 변환합니다.
- 2오류 수정 자동화에러 피드백 재시도 기능을 사용하여 LLM 출력의 검증 오류를 자동으로 수정합니다.
- 3추출 정확도 벤치마크다양한 모델 설정 및 프롬프트 방식에 따른 비용과 필드 정확도를 비교합니다.
- 4엄격한 데이터 제약 조건 적용스키마 내 아이템 제한과 같은 특정 제약 조건에 모델 출력이 부합하도록 보장합니다.
프로젝트의 README를 바탕으로 작성되었습니다. 모든 활용 사례는 해당 내용의 인용문과 대조하여 확인합니다.
Typecast MCP 설치 방법
형식 확인: 2026년 11월 10일Cursor에 추가Goose에 추가Claude Code에 Typecast MCP 설치
$ claude mcp add typecast -- uvx typecast-mcpCursor에 Typecast MCP 설치
~/.cursor/mcp.json
{
"mcpServers": {
"typecast": {
"command": "uvx",
"args": [
"typecast-mcp"
]
}
}
}VS Code에 Typecast MCP 설치
.vscode/mcp.json
{
"servers": {
"typecast": {
"type": "stdio",
"command": "uvx",
"args": [
"typecast-mcp"
]
}
}
}VS Code CLI에 Typecast MCP 설치
$ code --add-mcp '{"name":"typecast","type":"stdio","command":"uvx","args":["typecast-mcp"]}'Claude Desktop에 Typecast MCP 설치
claude_desktop_config.json
{
"mcpServers": {
"typecast": {
"command": "uvx",
"args": [
"typecast-mcp"
]
}
}
}Codex에 Typecast MCP 설치
$ codex mcp add typecast -- uvx typecast-mcpCodex config.toml에 Typecast MCP 설치
~/.codex/config.toml
[mcp_servers.typecast] command = "uvx" args = ["typecast-mcp"]
Gemini CLI에 Typecast MCP 설치
$ gemini mcp add -s user typecast uvx -- typecast-mcpGemini settings.json에 Typecast MCP 설치
~/.gemini/settings.json
{
"mcpServers": {
"typecast": {
"command": "uvx",
"args": [
"typecast-mcp"
]
}
}
}자동 검사
2026년 11월 14일에 실행됨전체 보고서 →자동 검사를 통과했습니다. 이는 보안 감사가 아닙니다. 코드를 직접 읽고 서버에 필요한 최소한의 권한만 부여하십시오. 검사 방식
- 소스 저장소 연결 및 접근 가능nilayp-codes/typecast is public and reachablegithub.com/nilayp-codes/typecast ↗통과
- 패키지가 이 저장소를 소스로 명시typecast-mcp names nilayp-codes/typecast as sourcepypi.org/project/typecast-mcp/ ↗통과
- 빌드 출처(Sigstore 증명)No build attestation published for this release검사되지 않음
- 오픈 소스 라이선스 (OSI)MITspdx.org/licenses/MIT.html ↗통과
- 최근 1년 내 커밋 또는 릴리스Last commit or release 2026-11-06github.com/nilayp-codes/typecast ↗통과
- OpenSSF Scorecard검사되지 않음
- 패키지 이름이 인기 패키지와 유사하지 않음No popular package one edit away from typecast-mcp통과
- 패키지 및 해당 종속성에 중요한 보안 권고 없음29 resolved dependencies checked; no critical advisoryosv.dev ↗통과
- 모델에 대한 숨겨진 지침 없음0 tool descriptions and the README read; no override phrases, credential paths or hidden characters통과
- 파괴적인 도구에 라벨 표시됨검사되지 않음
- 원격 엔드포인트가 MCP 핸드셰이크에 응답검사되지 않음
README
GitHub에서 보기Typecast
LLM agent that turns messy job descriptions into schema-validated, typed JSON — and measures how often that fails.
Key findings (two Qwen models via OpenRouter, 30 real job descriptions):
- Schema mode makes output valid, not correct. In setup C both models were valid 98.9% of the time, at lower cost than retry, yet end-to-end accuracy was 67.1% (7B) and 84.7% (235B).
- Error-feedback retry fixed the small model, not the large one. Setup B was 98.9% valid for the 7B but 88.9% for the 235B, which failed on jd_002 in all four setup-B runs. A count-aware retry message did not change that.
- No run reached my 90% field-accuracy goal (best: 87.3%), and runs vary. The same setup moved by about 8 points end to end between runs, so a gap between two single runs is not conclusive.
60-second walkthrough:
https://github.com/user-attachments/assets/c4a2572c-88a2-49a3-8e20-176010877154
CLI and package name: soa · MCP server on PyPI: typecast-mcp
Results
Two open-weight models through OpenRouter, 30 real job descriptions, four setups. Field accuracy is scored against 20 hand-verified gold labels (see Limitations for how they were made). With 3 runs, the figure is the mean and the brackets are min–max across the runs; with 1 run it is that single run. A = plain prompt, no retry; B = plain prompt + error-feedback retry; C = provider schema mode, no retry; D = schema mode + retry.
| model | setup | runs | final valid | end-to-end accuracy | cost per 30-doc run |
|---|---|---|---|---|---|
| qwen-2.5-7b-instruct | A · plain | 1 | 83.3% | 50.4% | $0.0081 |
| qwen-2.5-7b-instruct | B · plain + retry | 3 | 98.9% (96.7%-100.0%) | 67.1% (62.9%-70.4%) | $0.0103 ($0.0101-$0.0105) |
| qwen-2.5-7b-instruct | C · schema | 3 | 98.9% (96.7%-100.0%) | 67.1% (65.0%-69.8%) | $0.0076 ($0.0076-$0.0076) |
| qwen-2.5-7b-instruct | D · schema + retry | 1 | 100.0% | 70.0% | $0.0080 |
| qwen3-235b-a22b-2507 | A · plain | 1 | 70.0% | 60.4% | $0.0120 |
| qwen3-235b-a22b-2507 | B · plain + retry | 3 | 88.9% (86.7%-93.3%) | 75.6% (72.8%-80.9%) | $0.0211 ($0.0199-$0.0234) |
| qwen3-235b-a22b-2507 | C · schema | 3 | 98.9% (96.7%-100.0%) | 84.7% (82.8%-85.8%) | $0.0136 ($0.0134-$0.0140) |
| qwen3-235b-a22b-2507 | D · schema + retry | 1 | 100.0% | 85.4% | $0.0143 |
| model | setup | runs | first-try valid | final valid | avg calls | field accuracy (valid outputs) | end-to-end accuracy | cost per 30-doc run |
|---|---|---|---|---|---|---|---|---|
| qwen-2.5-7b-instruct | A: plain prompt, no retry | single run | 83.3% | 83.3% | 1.00 | 62.9% | 50.4% | $0.0081 |
| qwen-2.5-7b-instruct | B: plain prompt + error-feedback retry | mean of 3 runs (min–max) | 82.2% (80.0%-83.3%) | 98.9% (96.7%-100.0%) | 1.20 (1.17-1.23) | 68.2% (66.0%-70.4%) | 67.1% (62.9%-70.4%) | $0.0103 ($0.0101-$0.0105) |
| qwen-2.5-7b-instruct | C: schema mode, no retry | mean of 3 runs (min–max) | 98.9% (96.7%-100.0%) | 98.9% (96.7%-100.0%) | 1.00 (1.00-1.00) | 68.2% (66.5%-69.8%) | 67.1% (65.0%-69.8%) | $0.0076 ($0.0076-$0.0076) |
| qwen-2.5-7b-instruct | D: schema mode + retry | single run | 96.7% | 100.0% | 1.03 | 70.0% | 70.0% | $0.0080 |
| qwen3-235b-a22b-2507 | A: plain prompt, no retry | single run | 70.0% | 70.0% | 1.00 | 86.6% | 60.4% | $0.0120 |
| qwen3-235b-a22b-2507 | B: plain prompt + error-feedback retry | mean of 3 runs (min–max) | 76.7% (73.3%-80.0%) | 88.9% (86.7%-93.3%) | 1.34 (1.33-1.37) | 85.6% (85.1%-86.0%) | 75.6% (72.8%-80.9%) | $0.0211 ($0.0199-$0.0234) |
| qwen3-235b-a22b-2507 | C: schema mode, no retry | mean of 3 runs (min–max) | 98.9% (96.7%-100.0%) | 98.9% (96.7%-100.0%) | 1.00 (1.00-1.00) | 86.1% (85.6%-87.0%) | 84.7% (82.8%-85.8%) | $0.0136 ($0.0134-$0.0140) |
| qwen3-235b-a22b-2507 | D: schema mode + retry | single run | 100.0% | 100.0% | 1.00 | 85.4% | 85.4% | $0.0143 |
How to read it:
- "Mean of 3 runs (min–max)" (B and C) is the mean of three runs on the same 30 documents, with the lowest and highest run in brackets. It is a spread across runs, not an interval estimate. The B runs are the three runs made after the retry-feedback fix described below. The C runs are the first C run plus two repeats. "Single run" (A and D) is one draw each: the repeats show the same setup moving by about 8 points end to end between runs (235B setup B: 72.8% to 80.9%). Each run's own 95% interval (Wilson for the validity columns, bootstrap for the accuracy columns) is in the per-run table at the end of this section; the single runs of A and D have none, because they were made before the interval code existed.
- First-try / final valid: share of the 30 documents whose first reply, or any reply within 3 attempts, passed validation. Avg calls: model calls per document.
- Field accuracy (valid outputs) is measured only on documents that produced a valid posting, so it flatters a setup that fails often (235B setup A: 86.6% field accuracy, but only 70.0% of the labelled documents had a valid output to score). End-to-end accuracy counts a failed document as wrong on every field and is the number to trust when setups differ in validity.
- Cost is the provider-reported cost of one run over the 30 documents. Temperature was not set (the provider default applied, value unknown).
My targets, and whether they were met
I set three targets for setup B before running anything: final validity of 30/30, at most 1.3 calls per document on average, and at least 90% field accuracy. Result, per run (the three runs after the retry fix):
| target (setup B) | qwen-2.5-7b-instruct | qwen3-235b-a22b-2507 |
|---|---|---|
| final validity 30/30 | met in 2 of 3 runs (100.0%, 100.0%, 96.7%) | missed in all 3 (86.7%, 93.3%, 86.7%) |
| avg calls <= 1.3 | met (1.20, 1.17, 1.23) | missed (1.37, 1.33, 1.33) |
| field accuracy >= 90% | missed (70.4%, 68.1%, 66.0%) | missed (85.7%, 85.1%, 86.0%) |
The 90% field-accuracy target was missed by every run of both models, in every setup. The best number anywhere is 87.3% (235B, setup B, first-round run).
Rows 1-8: the first round, one run per model and setup (A, B, C, D for the 7B, then for the 235B). These were written
before the interval code existed, so they have none. Rows 9-18: the repeat round, in order 7B B x3, 7B C x2, 235B B x3,
235B C x2. Intervals are in brackets. Columns: field acc is over valid outputs, coverage is the share of the 20
labelled documents with a valid output, e2e acc counts failures as wrong.
| model | setup | n | first-try | final | avg calls | field acc (norm) | coverage | e2e acc (norm) | cost |
|---|---|---|---|---|---|---|---|---|---|
| openrouter/qwen/qwen-2.5-7b-instruct | A | 30 | 83.3% | 83.3% | 1.00 | 62.9% | 80.0% | 50.4% | $0.0081 |
| openrouter/qwen/qwen-2.5-7b-instruct | B | 30 | 86.7% | 96.7% | 1.17 | 68.3% | 95.0% | 64.8% | $0.0102 |
| openrouter/qwen/qwen-2.5-7b-instruct | C | 30 | 100.0% | 100.0% | 1.00 | 69.8% | 100.0% | 69.8% | $0.0076 |
| openrouter/qwen/qwen-2.5-7b-instruct | D | 30 | 96.7% | 100.0% | 1.03 | 70.0% | 100.0% | 70.0% | $0.0080 |
| openrouter/qwen/qwen3-235b-a22b-2507 | A | 30 | 70.0% | 70.0% | 1.00 | 86.6% | 70.0% | 60.4% | $0.0120 |
| openrouter/qwen/qwen3-235b-a22b-2507 | B | 30 | 83.3% | 86.7% | 1.33 | 87.3% | 85.0% | 74.3% | $0.0186 |
| openrouter/qwen/qwen3-235b-a22b-2507 | C | 30 | 100.0% | 100.0% | 1.00 | 85.8% | 100.0% | 85.8% | $0.0140 |
| openrouter/qwen/qwen3-235b-a22b-2507 | D | 30 | 100.0% | 100.0% | 1.00 | 85.4% | 100.0% | 85.4% | $0.0143 |
| openrouter/qwen/qwen-2.5-7b-instruct | B | 30 | 80.0% [62.7-90.5] | 100.0% [88.6-100.0] | 1.20 | 70.4% [64.4-76.9] | 100.0% | 70.4% [64.4-76.9] | $0.0104 |
| openrouter/qwen/qwen-2.5-7b-instruct | B | 30 | 83.3% [66.4-92.7] | 100.0% [88.6-100.0] | 1.17 | 68.1% [61.4-74.7] | 100.0% | 68.1% [61.4-74.7] | $0.0101 |
| openrouter/qwen/qwen-2.5-7b-instruct | B | 30 | 83.3% [66.4-92.7] | 96.7% [83.3-99.4] | 1.23 | 66.0% [59.6-72.2] | 95.0% | 62.9% [54.5-70.5] | $0.0105 |
| openrouter/qwen/qwen-2.5-7b-instruct | C | 30 | 100.0% [88.6-100.0] | 100.0% [88.6-100.0] | 1.00 | 66.5% [59.6-73.8] | 100.0% | 66.5% [59.6-73.8] | $0.0076 |
| openrouter/qwen/qwen-2.5-7b-instruct | C | 30 | 96.7% [83.3-99.4] | 96.7% [83.3-99.4] | 1.00 | 68.4% [61.3-75.0] | 95.0% | 65.0% [55.2-73.3] | $0.0076 |
| openrouter/qwen/qwen3-235b-a22b-2507 | B | 30 | 76.7% [59.1-88.2] | 86.7% [70.3-94.7] | 1.37 | 85.7% [81.3-89.6] | 85.0% | 72.8% [58.4-84.8] | $0.0234 |
| openrouter/qwen/qwen3-235b-a22b-2507 | B | 30 | 73.3% [55.6-85.8] | 93.3% [78.7-98.2] | 1.33 | 85.1% [81.1-88.9] | 95.0% | 80.9% [71.1-87.9] | $0.0199 |
| openrouter/qwen/qwen3-235b-a22b-2507 | B | 30 | 80.0% [62.7-90.5] | 86.7% [70.3-94.7] | 1.33 | 86.0% [82.0-89.6] | 85.0% | 73.1% [58.7-84.9] | $0.0202 |
| openrouter/qwen/qwen3-235b-a22b-2507 | C | 30 | 96.7% [83.3-99.4] | 96.7% [83.3-99.4] | 1.00 | 87.0% [83.7-89.9] | 95.0% | 82.8% [72.3-89.0] | $0.0136 |
| openrouter/qwen/qwen3-235b-a22b-2507 | C | 30 | 100.0% [88.6-100.0] | 100.0% [88.6-100.0] | 1.00 | 85.6% [82.0-88.8] | 100.0% | 85.6% [82.0-88.8] | $0.0134 |
The A and D figures in the main table are rows 1-8 (A and D) above. The first-round B rows (7B: 86.7% / 96.7%,
235B: 83.3% / 86.7% first-try / final) are the runs from before the retry fix; the first-round C rows are the first
of the three C runs in the main table. The aggregate over the repeats is in results/aggregate.md; the raw per-run
reports are results/2026-11-04_*.json.
How it works
text ──► build prompt (schema + instructions)
│
▼
LLM call ──► raw string
│
▼
JobPosting.model_validate_json(raw)
│ ok │ ValidationError
▼ ▼
log success log failure (error + raw)
return model attempt < MAX? ──► append error to messages, retry
│ no
▼
raise ExtractionFailed (logged)
The schema shown to the model is generated from the Pydantic model (JobPosting.model_json_schema()), never written
by hand, and the reply is checked with the same model. A failed reply goes back to the model with the validator's
errors, one line per error, for up to 3 attempts. After a valid reply, two deterministic tools tidy the result
(canonical skill names, a location check against the document header). They are plain code, not model calls, so they
add no LLM calls and the setups below stay comparable. Details: docs/ARCHITECTURE.md.
The four setups: A plain prompt, no retry. B plain prompt, errors fed back (up to 3 attempts). C the provider's strict JSON-schema mode, no retry. D schema mode plus retry.
Why not just use native structured outputs?
Schema mode helped a lot, and it is cheaper. It also does not do the whole job.
What it fixed. Setup C was valid on the first try 98.9% (96.7%-100.0%) of the time for both models, against a
single plain-prompt run (A) of 83.3% for the 7B and 70.0% for the 235B. The 30-item cap on skills was the constraint
that models actually broke in plain mode; under schema mode the report counts replies that broke a constraint the
schema carries, and it was 0 in every C and D run (6 C runs, 2 D runs). C also cost less than B: $0.0076 against
$0.0103 for the 7B, $0.0136 against $0.0211 for the 235B (means per run).
What it does not do:
- It guarantees shape, not meaning. The remaining failures in C were rules no JSON schema can express. Two of the six C runs had one document rejected for duplicate skill names ("LangChain" and "Langchain"; "Python" twice; both rejections are in results/failure_examples.jsonl), which is why C's final validity is 98.9% (96.7%-100.0%) and not 100%. The intern-implies-no-experience rule has the same property (it cannot be stated in a schema); it did not fire in any of the 637 logged attempt lines of this project (the last line of that file records the count), so I have no evidence either way on it.
- Valid is not correct. Schema mode made the 235B valid 98.9% (96.7%-100.0%) of the time, and its field accuracy is 86.1% (85.6%-87.0%), with end-to-end accuracy 84.7% (82.8%-85.8%): below my 90% goal. The 7B in C is just as valid and gets 68.2% (66.5%-69.8%) field accuracy; its seniority accuracy was 55.0%, 55.0% and 52.6% over the three C runs. Validity says nothing about whether the seniority is right.
- It may trade recall for validity. The cap is enforced by the model emitting shorter skill lists, and skills are where accuracy is weakest. For the 235B, skills F1 was 66.9%, 72.1% and 65.7% in the three C runs, against 71.4%, 71.6% and 68.3% in the three B runs (means 68.2% vs 70.4%, with overlapping ranges). In the first-round single runs it was 72.1% (A) and 73.8% (B) against 66.9% (C) and 63.4% (D). The repeats show this is a tendency, not a clean effect, and with n=20 labelled documents I would not claim more.
- It depends on the provider. Both models accepted strict
json_schemahere, but only because requests were sent with OpenRouter'srequire_parametersso they could only route to providers that support it. The 7B was always served by one provider; the 235B was split across three.
Most common failure types, and what fixed them
From the plain-prompt runs and the retry runs (counts are failing attempts whose first error was at that field,
failures.by_first_loc in the reports):
| failure | setup A (one run) | setup B, 3 runs | what fixed it |
|---|---|---|---|
skills over the 30-item cap |
7B: 3 of 5 failing docs; 235B: 8 of 9 failing docs | 7B: 5, 5, 3 failing attempts; 235B: 14, 11, 14 | Retry feedback fixes most of it. Schema mode removes it. |
skill name over 40 characters (skills.*.name) |
7B 2, 235B 1 | 7B: 0, 0, 3; 235B: 1, 1, 0 | Retry |
null instead of true/false (skills.*.required, salary_mentioned) |
none | 7B: 2 attempts at skills.*.required in one run, 1 at salary_mentioned in another |
Retry |
duplicate skill names (also counted under skills, since that is where the error is reported) |
not counted separately | not counted separately | Retry fixes it: all 5 duplicate-name rejections in setup B were valid on the next attempt (each rejected attempt and its valid successor are in results/failure_examples.jsonl). Schema mode cannot: it was the only failure left in C (two documents, same file). |
| malformed JSON, markdown fences, prose, extra fields, invalid enum values | none: no run has a failure at the root, at seniority, mode or any other enum or extra field |
none | Nothing needed |
Almost everything that goes wrong is a count cap, not a formatting problem. In the first 11-document baseline every
failure of both models was a too-long skills list (results/2026-11-03_baseline_*.md). The first thing I tried was
to make the feedback more specific. The default feedback line is the validator's own wording (for example skills: List should have at most 30 items after validation, not 42). I replaced it for over-long lists with a count and a
fix, for example skills: has 39 items, maximum is 30 - remove 9, starting with required=false ones. The result is
mixed:
- It did not lift the 235B's final validity in setup B: 86.7% before, then 86.7%, 93.3% and 86.7% across three runs. Each run's interval contains the old figure, so at n=30 no effect is shown.
- The four 235B documents that failed in the first B run (jd_002, jd_003, jd_005, jd_023) passed in 6 of 12 document-runs afterwards (jd_002 0 of 3, jd_003 1 of 3, jd_005 2 of 3, jd_023 3 of 3), but other documents (jd_029, jd_030, jd_004) failed instead. I cannot separate the fix from run-to-run variation. The first attempt is identical before and after the fix, yet first-try validity moved from 83.3% to 76.7%, 73.3% and 80.0%; that spread is run-to-run variation (sampling plus a provider mix I did not log in the first round).
- jd_002 failed for the 235B in all four setup-B runs. Every attempt is in
results/failure_examples.jsonl: 42, 31, 31 skills in the first run (before the
fix), then 70, 31, 31 and 47, 31, 31 and 44, 31, 31 in the three runs after it. The model goes from 42-70 skills down
to 31 on the first retry and then stays at 31. For a 31-item reply the new feedback line is
skills: has 31 items, maximum is 30 - remove 1, starting with required=false ones(built by the code; the prompt text itself is not logged). This is the clearest open failure: the model does not make the last cut.
Design decisions
strict=Trueon every model. Pydantic's default lax mode quietly turns"3"into3and"yes"intoTrue, so a sloppy reply passed and the retry loop had nothing to do. Strict mode rejects both.- Required-but-nullable fields (
company,city,country,min_years_experience). No default: the model has to writenullexplicitly, so "not stated" is a decision the model made, and a forgotten key is an error rather than a silentnull. - One line per error in the retry prompt,
loc: msg (got: <input, clipped to 60 chars>), rendered fromValidationError.errors()instead ofstr(error). The raw string carries a version-specific prefix, a truncatedinput_value, and a documentation URL, which are noise for a model. The one exception is an over-long list, which gets the count-aware line shown above. I did not run a blind-retry control (retry with no error text), so I have no measurement of feedback versus plain resampling; setup A against B is retry-with-feedback against nothing. - Tools are deterministic post-processing, not model-called. Canonical skill names and a location check run
after validation as plain code. That keeps every setup at one prompt and the retry/attempt/cost counts clean, and
the model's own output is kept (
raw_posting) so validity is measured on what the model actually said. Letting the model call the tools is a separate experiment I have not run. - API errors are separate from validation errors. A 429 or 5xx is retried with backoff (2, 4, 8 s) and never uses up one of the 3 validation attempts.
- The 30-skill cap stays. The first plain-prompt baseline showed every failure was this one constraint, so I kept it as the test of what retries and schema modes can do rather than loosening it to make numbers look better.
Limitations
- 30 job descriptions, 20 gold labels. One document moves a validity rate by 3.3 points and a field accuracy by 5 points; the intervals above are wide for that reason (for example 235B setup B final validity 93.3% [78.7-98.2] in one run). Five of the 30 are non-technical or semi-technical roles, kept on purpose as an out-of-domain slice; two of them (jd_012, jd_019) are among the 20 labelled.
- The gold labels were drafted by a model (Claude) and then checked by me. I verified every field and made 17 corrections (docs/label_corrections.log). One person, one pass, so any systematic mistake of mine is in the labels. The labelling rules were written into the schema's field descriptions after the first baseline, so the models and the labels follow the same rules; runs on the earlier prompt are not comparable.
- Noise floor on accuracy, from a blind check. On 5 other documents (jd_004, 005, 008, 014, 015, none of them in the gold set) I labelled blind and compared against Claude's sealed labels written beforehand, with the same per-field metrics as the eval (my labels as the reference): title 80.0%, company 100.0%, seniority 80.0%, mode 80.0%, salary_mentioned 100.0%, min_years_experience 100.0%, city 80.0%, country 100.0%, skills F1 61.0% (precision 52.9%, recall 72.0%), overall 86.8% (results/label_agreement.md). Five documents is a sanity check, not a rate: one disagreement moves a field by 20 points. It does suggest that on skills two careful labelers disagree about as much as the models disagree with the gold labels, and that the 235B's roughly 86% field accuracy is close to the level two labelers reach with each other, so the remaining gap to my 90% target may partly be label noise rather than model error. I have not shown that.
- Runs are from one day, two models of one family, one provider layer. Temperature was not set. The 7B was always routed to one provider; the 235B to three, and the provider was not logged in the first round, so provider mix and sampling noise are mixed into the run-to-run spread.
- The post-processing tools were not tuned against the final labelling rules, so the location check and the labels can disagree on some documents.
- Setup D is a single run per model; only B and C were repeated.
- Data. The 30 job descriptions in
data/jds/were copied from public postings (LinkedIn 13, Work at a Startup 11, Wellfound 6, counted fromdata/jds/manifest.csv) for evaluation only; I will remove any of them on request. Five rows have no source URL recorded inmanifest.csv: jd_001, jd_002, jd_003, jd_004 and jd_010 (all LinkedIn; their notes say "URL not recorded").
Run it yourself
Needs Python 3.11+ and uv.
git clone https://github.com/nilayp-codes/typecast.git
cd typecast
uv sync
cp .env.example .env # then set OPENROUTER_API_KEY in .env (never commit it)
uv run soa extract data/jds/jd_001.txt # the JSON on stdout, a status line on stderr
uv run soa extract data/jds/jd_001.txt --json-mode # provider schema mode
uv run soa extract data/jds/jd_001.txt > out.json
uv run soa eval --model openrouter/qwen/qwen-2.5-7b-instruct --setup B --limit 5
uv run soa eval --model openrouter/qwen/qwen3-235b-a22b-2507 --setup all
Known issue: on Windows, clone into a short path, because LiteLLM writes a cache file under .venv and a very deep path exceeds Windows' path-length limit (soa extract then fails with APIConnectionError ... [Errno 2]).
soa eval makes real model calls: it prints the plan and the estimated number of calls and asks before sending
anything (--yes skips the question). In my runs one setup over the 30 documents cost between $0.0076 and $0.0234.
It saves results/<date>_<model>_<setup>.json and appends a row to results/summary.md; --resume <eval id>
continues an interrupted run. To combine repeated runs: uv run python scripts/aggregate_runs.py results/<files>.json
(writes results/aggregate.md). The 30 job descriptions are in data/jds/ and the gold labels in data/labels/.
Tests and type checks (no real model is ever called in tests):
uv run pytest
uv run mypy --strict src/ packages/typecast-mcp/src/
Typecast MCP 질문
Typecast MCP를 Claude Code에 추가하려면 어떻게 하나요?
터미널에서 claude mcp add typecast -- uvx typecast-mcp를 실행합니다. 모든 프로젝트에서 사용할 수 있게 하려면 --scope user를 추가합니다.
Typecast MCP를 Cursor에 추가하려면 어떻게 하나요?
~/.cursor/mcp.json를 열고 mcpServers 아래에 서버를 추가합니다. 위 설치 섹션에 표시된 Cursor 구성을 사용합니다.
검증 프로세스는 어떻게 작동하나요?
이 MCP server는 Pydantic 모델을 사용하여 JSON 스키마를 생성하고, LLM의 가공되지 않은 문자열 출력을 검증합니다. 검증에 실패할 경우 에러 피드백과 함께 최대 3회까지 재시도를 수행합니다.
스키마 모드를 사용하면 데이터의 정확성이 보장되나요?
아니요, 스키마 모드는 JSON의 형태와 구조를 보장하지만, 내용의 의미론적 정확성까지 보장하지는 않습니다.
사용 가능한 실행 설정에는 어떤 것들이 있나요?
A(일반 프롬프트, 재시도 없음), B(일반 프롬프트, 에러 피드백 재시도 포함), C(프로바이더 스키마 모드, 재시도 없음), D(스키마 모드, 재시도 포함)의 네 가지 설정이 있습니다.