Clean domain and site data from official sources, without the guesswork

If you have ever wired a DNS, WHOIS, or sitemap scraper into a pipeline, you know the failure modes. Some actors guess values when a lookup is slow or empty. Some hand back a raw dump and leave the parsing to you. Many charge a per-run start fee, so a batch of 10,000 domains where half return nothing still bills you for the misses.

There is a calmer way to do this. The protocols you actually want to read are already standardized and already public: DNS over HTTPS for resolution, IANA RDAP for registration data, and sitemap.xml plus robots.txt for site structure. Read those directly, return one typed row per result, put a source URL on every row, and bill only when you deliver a clean result. The three actors below follow that pattern.

DNS records, read over DNS-over-HTTPS

Bulk DNS Lookup resolves A, AAAA, MX, NS, TXT, CAA and more for many domains in one run, over DNS-over-HTTPS. It decodes CAA records into readable fields rather than leaving you the wire format, derives an SPF and DMARC email posture from the relevant TXT records, and can do reverse DNS.

Input is a list of domains. You get back one row per domain with the record types you asked for, and each row carries the source URL of the DoH query that produced it.

input:  { "domains": ["example.com"], "recordTypes": ["A", "MX", "CAA", "TXT"] }
output: { "domain": "example.com", "a": ["93.184.216.34"],
          "mx": [...], "caa": [...], "emailPosture": {...},
          "sourceUrl": "https://..." }

If a record does not exist, you see that it does not exist. You do not see a guess.

Registration data through official RDAP servers

Legacy WHOIS is free text that every registrar formats differently. RDAP is its structured-JSON successor, served by the official IANA-pointed servers. Domain WHOIS via RDAP follows the IANA bootstrap to the right RDAP endpoint for each TLD and returns the registrar, the key dates, the status codes, and the nameservers as clean JSON.

input:  { "domains": ["example.com"] }
output: { "domain": "example.com", "registrar": "...",
          "createdDate": "...", "expiresDate": "...",
          "status": ["clientTransferProhibited"],
          "nameservers": [...], "sourceUrl": "https://..." }

Because it reads RDAP rather than scraping a registrar web page, the fields are consistent across TLDs, and the source URL on each row is the RDAP endpoint you can re-query yourself to confirm.

Site structure from sitemaps and robots.txt

Sitemap Extractor reads sitemap.xml, sitemap index files that point to other sitemaps, gzipped (.gz) sitemaps, and the Sitemap directives inside robots.txt. It walks the index tree for you and returns one clean row per URL, with the lastmod and changefreq values when the sitemap provides them.

input:  { "startUrls": ["https://example.com/robots.txt"] }
output: { "url": "https://example.com/page",
          "lastmod": "2026-05-01", "changefreq": "weekly",
          "sourceUrl": "https://example.com/sitemap.xml" }

The source URL tells you which sitemap each URL came from, which matters when a site splits content across several sitemaps under an index.

Why official-source plus provenance matters for AI agents

These three share the same approach. They read official sources only. They do not fill empty fields with guessed values. Every row carries a source URL you can open and verify. And you are charged only on a clean delivered result, so a miss or an error costs you nothing.

That last part is not just billing hygiene. It is provenance. The actors run as agent tools over the Apify MCP server with a typed output schema, so an agent can call them and get back rows it can reason about by field name instead of parsing prose.

When an agent makes a claim, the question is always where the claim came from. A row that says a domain expires on a given date is more useful when it also says that date came from a specific RDAP endpoint. A row that lists a page in a sitemap is more useful when it names the sitemap file. Provenance on every row lets a downstream system check the work instead of trusting it, and it lets you debug a surprising result by going straight to the source.

Close

None of this is novel. The protocols have been public for years. The contribution here is discipline: read the official source, return a typed row, attach the source URL, and do not charge for a miss. If that fits how you build, the three actors above are a place to start.

A note on pricing, since it shapes the design: these are pay-per-result with no subscription and no per-run start fee. Bulk DNS Lookup and Domain WHOIS via RDAP bill per domain resolved; Sitemap Extractor bills per URL returned.