Using Text Diffing for API Regression Testing

Regression Testing with Text Diffing

When an API response changes between releases, you want to know exactly what changed — not just that the response is different. Text diffing provides that answer.

The Basic Pattern

1. Record the baseline response (known-good)
2. After each deployment, fetch the current response
3. Diff the two — any change is flagged for review

This works for JSON responses, HTML pages, generated configs, and CLI output.

API Usage

curl -X POST https://api.toolkitapi.io/v1/devtools/diff \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "a": "foo\nbar\nbaz",
    "b": "foo\nqux\nbaz"
  }'

Response:

{
  "changed": true,
  "additions": 1,
  "deletions": 1,
  "unified": "  foo\n- bar\n+ qux\n  baz"
}

Normalising Before Diffing

For JSON API responses, normalise before diffing to avoid false positives:

import json, httpx

def normalise(response_text: str) -> str:
    data = json.loads(response_text)
    # Remove volatile fields before diffing
    data.pop("timestamp", None)
    data.pop("request_id", None)
    return json.dumps(data, sort_keys=True, indent=2)

baseline = normalise(Path("fixtures/baseline.json").read_text())
current  = normalise(httpx.get("https://api.example.com/v1/users").text)

r = httpx.post(
    "https://api.toolkitapi.io/v1/devtools/diff",
    headers={"X-API-Key": API_KEY},
    json={"a": baseline, "b": current},
)
if r.json()["changed"]:
    print(r.json()["unified"])
    raise AssertionError("API response changed!")

Diff Formats