Internal architecture guide — AAOE Rate Reference

Processing flow

Schedule by source cadence. Poll only as often as the authority can reasonably publish. “Real time” means detection immediately after a successful poll; it does not imply that the government source publishes continuously.
Retrieve from an allowlist. A connector calls one approved authoritative endpoint using bounded time, response size, redirects, and concurrency.
Preserve before parsing. Store the exact successful response bytes privately, calculate SHA-256, and retain retrieval headers and source URL.
Normalize deterministically. Convert the source representation into an immutable observation without losing units, qualifiers, jurisdiction, or source semantics.
Validate and compare. Apply schema, range, overlap, category, effective-date, and material-change controls. Quarantine ambiguity.
Publish and notify. Expose approved facts through read-only interfaces. Create an outbox event for a material change and deliver alerts idempotently.

Implemented reference cadence

The working reference polls the official EU VAT service every six hours with bounded concurrency. It archives each successful response, establishes a baseline without sending a false change alert, and creates a governed event only when amount, currency, or unit changes.

Component responsibilities

Capability	Responsibility	Required controls
Scheduler	Creates bounded ingestion runs according to each source’s approved cadence.	UTC schedule, overlap protection, run identifier, missed-run alert
Connector adapter	Retrieves and validates one authority-specific contract.	Host allowlist, secret manager, size limit, timeout, schema pinning
Evidence vault	Retains exact upstream bytes before transformation.	Private encryption, content hash, retention, legal hold, no public listing
Control database	Stores sources, observations, evidence metadata, audit events, and outbox state.	Append-only facts, foreign keys, backups, point-in-time recovery
Validation gate	Checks semantics and routes exceptions to quarantine.	Ranges, units, overlap, drift, reconciliation, dual approval
Notification outbox	Separates a committed rate change from email or webhook delivery.	Stable deduplication key, retry state, destination hashing, delivery audit
Reference API	Returns the applicable observation for a jurisdiction and expense date.	GET-only surface, input validation, schema versioning, throttling, request IDs
Administrative plane	Manages sources, approvals, subscriptions, corrections, and evidence access.	SSO, MFA, RBAC, segregation of duties, SIEM, no public administration routes

Data contracts

Immutable observation

The observation is a fact asserted by a source at a point in time. Corrections create another observation; they do not overwrite the prior record.

{
  "observation_id": "immutable-content-derived-id",
  "natural_key": "stable-source-series-key",
  "type": "mileage | per_diem | tax",
  "category": "source-qualified-category",
  "amount": 20,
  "currency": null,
  "unit": "percent",
  "jurisdiction": { "country": "DE" },
  "effective_from": "2026-01-01",
  "effective_to": null,
  "observed_at": "2026-08-14T06:17:00Z",
  "status": "verified",
  "source_id": "authority-source-id",
  "source_url": "https://authority.example/resource",
  "evidence": {
    "status": "archived",
    "artifact_sha256": "64-lowercase-hex-characters",
    "retention_class": "approved-regulatory-retention"
  }
}

Evidence artifact

Store source identifier, canonical URL, retrieval time, media type, byte size, object key, SHA-256, upstream ETag or Last-Modified when provided, and retention class. Raw artifacts remain private; public consumers receive evidence metadata.

Audit event

Use an append-only event with event ID, event type, subject ID, UTC timestamp, actor type, correlation/run ID, and structured details. Never put credentials, personal data, or entire upstream payloads in audit details.

Change detection and delivery

Retrieval metadata changes frequently and is not itself a statutory rate change. Compare a canonical semantic signature appropriate to the source—for example amount, currency, unit, jurisdiction, category, and authoritative effective interval. Store the new observation before creating the change event.

The first successful retrieval establishes a baseline and sends no “changed” alert.
A stable change key prevents duplicate events during retries.
An outbox record is committed in the governed database before external delivery.
Failed email or webhook delivery remains pending and can be retried without re-ingesting the source.
Notification destinations belong in an approved subscription store or secret manager. Logs and public health responses must not reveal them.
Every alert includes the new value, effective date, source URL, observation ID, evidence status, and artifact hash.

Security and privacy boundary

Keep the public service read-only. Place connector activation, approvals, evidence release, and subscription management behind enterprise identity and authorization.
Store API credentials only in an enterprise secret manager; rotate them and audit access. Never place credentials in URLs retained as evidence.
Allowlist upstream hosts and reject redirects to unapproved origins. Bound response bytes before parsing.
Encrypt storage and transport, use least-privilege service identities, and separate development, test, and production accounts.
Do not send employee names, expense narratives, card data, supplier bank details, or ERP credentials to this reference-data service.
Hash notification destinations in operational records; store the actual destination only in the controlled subscription system.
Keep raw evidence private by default and require an audited review workflow for release.

Operational controls and service levels

Signal	Measure	Response
Source freshness	Time since last successful governed retrieval	Warn, then stop automated consumption when the source-specific threshold expires
Schema drift	Parser rejects or missing required fields	Quarantine artifact; page connector owner; do not publish partial facts
Material change	Semantic signature differs from baseline	Create outbox event; require approval where policy demands it
Evidence integrity	Stored bytes do not match recorded SHA-256	Critical incident; suspend affected observations
Delivery backlog	Pending outbox age and retry count	Retry with backoff; route persistent failure to dead-letter review
API health	Latency, error rate, throttles, cache behavior	Scale independently of ingestion; preserve read-only degraded mode

Back up the relational store, test evidence restoration, rehearse secret rotation, and run source fixtures in CI. Alerting should distinguish “authority unavailable,” “source changed,” “parser failed,” and “notification failed.”

Recommended internal implementation sequence

Approve rate families, jurisdictions, data owners, source owners, retention, and the legal/terms review process.
Create the source register, immutable evidence vault, governed schema, audit-event contract, and recovery procedures.
Implement one official API connector end to end with recorded fixtures and failure-mode tests.
Add scheduling, bounded concurrency, baseline creation, semantic comparison, quarantine, and the notification outbox.
Add an official structured-file connector to exercise download, checksum, versioning, and parser controls.
Publish a versioned read-only API behind the enterprise gateway. Load test it independently from ingestion.
Build the private command center for approvals, subscriptions, evidence access, connector health, and corrections.
Integrate the ERP while preserving observation and evidence identifiers with every configuration change.

Web scraping boundary

Automated scraping is deliberately excluded from the normal path. Treat a webpage-only authority as a source gap until legal, security, parser-maintenance, and evidence controls are explicitly approved.

AI retrieval boundary

An assistant may retrieve approved observations; it must not decide employee eligibility or invent missing policy. Expose only allowlisted read operations. Treat all source content as untrusted data, require citations, preserve returned evidence fields, and escalate missing, stale, provisional, conflicting, or ambiguous results.

No arbitrary URL-fetch tool and no write access to the observation store.
No credentials, personal data, expense narratives, or supplier confidential data in prompts.
Structured output validated against the same public API schema.
Prompt, tool-call, citation, and model-version audit records subject to enterprise retention rules.
Human approval before an AI-produced recommendation changes ERP configuration.

Review the safe retrieval prompt.

Build a governed evidence pipeline, then expose the rate.