Build an API Test Matrix From HTTP Semantics, Not Endpoint Counts

Treat status codes as semantics, not labels

A status code is only one part of a response, but choosing the right one communicates what a client may do next. A 401 response concerns missing or invalid authentication and commonly carries a challenge; 403 means the server understood the credentials but refuses the action. A 404 can mean the target resource is absent, while some systems deliberately use it to avoid disclosing that a protected resource exists. Tests should reflect the documented security policy rather than forcing a universal rule.

Similar distinctions matter for creation and asynchronous work. A successful POST may return 201 when a resource was created and identify it through a Location header or representation. A 202 means processing was accepted but not completed; a test that immediately asserts the final state misunderstands that contract. A 204 has no response content, so checking for a JSON body is itself an invalid expectation.

• Method and resource state: existing, missing, deleted, or conflicting.
• Identity and permission: unauthenticated, expired credential, allowed role, denied role.
• Representation: accepted media type, unsupported media type, malformed body, valid alternative shape.
• Outcome: status, required headers, body schema, side effect, and idempotency behavior.

Design a compact risk matrix

Begin with one canonical success case, then add boundaries for each independent contract. For GET /orders/{id}, that might be an existing order, a syntactically invalid identifier, a well-formed missing identifier, an order owned by another tenant, and a conditional request with an ETag. These rows protect different behavior even if two of them intentionally return the same public status.

Do not automatically cross every payload, role, and resource state. Select interactions where one dimension changes the meaning of another. A locked account may matter only for mutations; an Accept header may matter only when multiple response representations exist. Record excluded combinations in the test-design note if their absence would otherwise look accidental.

Verify errors as contracts

Error bodies are client-facing interfaces. RFC 9457 defines a Problem Details format with standard members such as type, title, status, detail, and instance, while allowing extension members. A test can validate that the response media type and stable fields match the contract without freezing every sentence. Human-readable detail text often changes; a machine-readable type URI or error code is usually a better automation anchor.

Negative tests should also verify absence of side effects. A rejected transfer must not debit an account. An unauthorized update must not modify a timestamp. This requires a second observation—through a read endpoint, event probe, or database boundary—not merely an expected 4xx response. Otherwise, the suite can miss a system that returns an error after partially applying the operation.

Use OpenAPI as a map, not as the only oracle

An OpenAPI document can enumerate paths, operations, parameters, request bodies, and response schemas. It is excellent input for discovering missing coverage and validating representations. It does not necessarily express business invariants, authorization rules, eventual consistency, or all side effects. Generated schema checks should therefore complement scenario tests rather than replace them.

The contract also needs version awareness. If an optional response field is added, strict whole-body equality can create false failures even though the change is backward compatible. Conversely, silently accepting a changed type can hide a breaking change. Assert required fields, types, constrained values, and explicitly stable headers; decide how additional fields are handled according to the client compatibility policy.

Make timeout behavior explicit

A request can fail while connecting, waiting for bytes to be read, writing a request, or waiting for a connection from the pool. HTTPX exposes these as separate timeout categories. A single generous timeout can make tests slow and conceal where an integration is blocked. Configure finite values deliberately and report the category of failure.

Timeout tests need a controlled dependency such as a stub server; relying on a public API or arbitrary sleep makes results unstable. Also separate transport failures from valid HTTP error responses. A 503 is a response that the client received and can inspect. A read timeout means no complete response arrived within the configured condition. Retry expectations should depend on method semantics, idempotency, and product policy—not on a blanket rule that every failure deserves another attempt.

import httpx

timeout = httpx.Timeout(8.0, connect=2.0, read=5.0, write=5.0, pool=1.0)

with httpx.Client(base_url="https://api.example.test", timeout=timeout) as client:
    response = client.get("/orders/42", headers={"Accept": "application/json"})
    assert response.status_code == 200
    assert response.headers["content-type"].startswith("application/json")