Defaults.Exposed

You found DefaultsExposed/1.0 in your logs

It is our crawler. It made one request to your home page, read the response headers, and left. Below is exactly what it does, how often, and how to stop it.

What it is

We publish an open measurement of domain security configuration across the internet, covering DNS, email anti-spoofing, TLS and HTTP response headers, at defaults.exposed/data. This crawler is how that measurement is taken. It runs roughly once a month.

Full User-Agent string: Mozilla/5.0 (compatible; DefaultsExposed/1.0; +https://defaults.exposed/headers)

What it actually did to your server

  • One HEAD or GET to /. If that redirects to another page on the same site, it follows the redirect once.
  • Read the response headers, things like Content-Security-Policy, Strict-Transport-Security, X-Frame-Options.
  • Made public DNS queries for your domain: SPF, DMARC, DKIM, DNSSEC, MX, CAA.
  • Completed a TLS handshake to read your certificate.

What it did not do

  • No login attempts, no form submissions, no POST requests.
  • No vulnerability probing, no path guessing, no admin-panel hunting.
  • No crawling beyond your home page. It does not walk your site.
  • Nothing was downloaded beyond the response to that one request.

In practice that is a handful of packets, roughly once a month. If it showed up as a load problem on your server, we would genuinely like to know. See below.

It does not read your robots.txt, and here is why

We would rather say this plainly than have you find out. The measurement is of what a server presents to the public internet, so it requires actually fetching a response. And a crawler that skipped every site with a restrictive robots.txt would produce a census biased toward less-protected sites, and the published numbers would be wrong in a direction we could not correct for.

So instead of a rule we would be quietly breaking, there is a real opt-out that we honour. It is below.

How to stop it

Ask us to exclude your domain

Email [email protected] from an address at the domain you want excluded, and we will remove it from future scans and from the published data. We ask for an address at the domain only so that people cannot exclude domains they do not run.

Or block it yourself

Blocking the User-Agent at your edge works and we will not route around it. One caveat worth knowing: a blocked request looks like a failed request, so your domain may then appear worse in the published data than it really is. Exclusion is the cleaner option, which is why it is first.

While you are here: what did it find?

Same checks, same scoring, run live against your domain right now. Free, no sign-up.

Scores against the same methodology as the published census. How the scoring works.

Who runs it

defaults.exposed. Questions, complaints and load problems all go to [email protected] and reach a person.

Analytics opt-out for this website is separate and lives at /optout. It has nothing to do with the crawler.