This article discusses DNS configuration problems that prevent Google-managed SSL certificates from being issued for Global HTTP(S) Load Balancers.

How do you detect that there is a problem with a Google-managed SSL certificate?

The certificate status is Provisioning.

The Domain status has a warning sign. A warning of FAILED_NOT_VISIBLE indicates a DNS problem.

Google-managed SSL Certificate Permission Required

The following permissions are required:

  • compute.sslCertificates.*
  • compute.targetHttpsProxies.*

Those permissions are the roles:

  • Owner
  • Editor
  • Compute Security Admin and Compute Network Admin (both are required)

What types of SSL certificates are supported?

  • Google-managed SSL certificates
  • Self-managed SSL certificates – certificates you upload to Google Cloud

DNS Troubleshooting Steps

This section provides details on how to verify the correct settings to ensure a Google-managed SSL certificate can be issued.

DNS Query Errors

There are four common types of DNS query errors:

  • NOERROR – The DNS query successfully completed
  • NXDOMAIN – The domain name does not exist. The most common DNS query error
  • SERVFAIL – This DNS server is misconfigured, failing, or under attack
  • REFUSED – The DNS query failed because the server refused to answer.

Anything besides NOERROR requires investigation. NXDOMAIN is a common problem that you can solve. SERVFAIL and REFUSED require assistance from the DNS server technical staff.

For more details on DNS query errors reference RFC 2308.

NXDOMAIN: Broken Resolvers. NXDOMAIN is incorrectly managed by a large number of resolvers. So many that there is a section in the RFC: RFC 2308 Section 2.1.1.

NXDOMAIN: Caching Negative Answers: RFC 2308 Section 5.

Verify that you are updating and resolving from the correct DNS servers

An important technical item is to understand the difference between “authoritative” and “non-authoritative”  DNS lookups. When debugging DNS resource records and SSL certificate problems, ensure you are making authoritative lookups. Then compare against the non-authoritative lookups to confirm that there is not a DNS caching issue.

Registration Lookup Tool

ICANN, Internet Corporation for Assigned Names and Numbers, has a domain name lookup service. This service provides many details about a registered domain. Use their site to fetch the nameservers for the domain that you are configuring an SSL certificate. ICANN Registration data lookup tool

The results for my site jhanley.com:

Check DNS Resolution

Verify that the domain resolves to the load balancer IP address.

There are many tools available for testing DNS resolution. One that I often use is the DNS Lookup by MxToolbox. From the MxToolbox menu select DNS Lookup. Enter the domain name. Verify that the DNS lookup resolves to the load balancer IP address. If you are also deploying IPv6, verify the AAAA resource records with the AAAA Lookup test.

Another very good tool is Google Admin Toolbox – Dig.

You can also use the command line tool nslookup.

Note the following line from the previous command:

The non-authoritative answer means that the answer came from a DNS resolver’s cache and not from one of the domain’s nameservers. The value(s) might not be correct if you recently changed resource records at the domain’s authoritative DNS servers. To perform a check without DNS cache, use the domain’s nameservers:

That answer is authoritative and does not come from a cached response.

Verify that the line containing Address: 209.23.11.234 is correct for your domain.

Review the results from this command:

Notice the NXDOMAIN response.

NXDOMAIN is a DNS message type received by the DNS resolver (i.e. client) when a request to resolve a domain is sent to the DNS and cannot be resolved to an IP address. An NXDOMAIN error message means that the domain does not exist. [source]

An important point to remember is that NXDOMAIN is a DNS response that is cached by DNS resolvers. This is called Negative Caching. After you create the DNS resource record to correct that error, you will often need to wait for the DNS cache to refresh. NXDOMAIN response will be cached as specified in the SOA resource record. The smaller value from the SOA TTL and SOA MINIMUM values will be used. A common value is 300 seconds or 3,600 seconds. Therefore, it is important to create the DNS resource records correctly before attempting to create a Google-managed SSL certificate.

Once you have the correct DNS resource records at the domain’s authoritative nameservers, you might have to wait a few hours or days for DNS caching to refresh.

Note: if you created new DNS resource records instead of changing existing ones, there will be no cached records to update. The new resource records will be cached on the first query so the wait time is zero.

If instead of receiving NOERROR or NXDOMAIN, you receive SERVFAIL, then the domain’s DNS server has a technical problem. If you are using DNS servers managed by a vendor, you should open a problem report with their technical staff. Your DNS servers might be misconfigured, failing, or under attack.

DNS Record Propagation

Propagation is the time it tasks for DNS changes to be updated across the global Internet. DNS resource records have a TTL value. That value is the Time-To-Live in seconds. The TTL is a recommendation to DNS resolvers to cache that resource record for the TTL number of seconds. A DNS resolver is not required to honor the TTL. Some DNS resolvers refuse to honor short TTLs.

There are Internet tools that can help you determine if a domain resolves to the same IP address such as DNS Checker. They accomplish this test by querying multiple DNS servers around the world. After you update a DNS resource record, these tools can help you check for stale caches.

Global DNS Server Flush

Several very popular DNS servers support DNS cache flush. This can help reduce the DNS propagation time.

For Google-managed SSL certificates, flushing the Google DNS cache is usually enough.

Local DNS Resolver Flush

After you update your DNS servers and flush the global DNS servers caches, clear the local cache on your computer. This step will not affect Google-managed SSL certificates, but you do not want your local computer to be out of sync while testing.

Windows:

Linux:

The command to use depends on the distribution.

Google Chrome DNS Flush

Google Chrome has its own DNS resolver. Go to chrome://net-internals/#dns to clear the host cache.

Suggestions

  • If you are updating an existing service, plan to make changes during the slowest time. If you are a business, that time might start late Friday afternoon. I usually plan to make changes Friday evening and then validate Saturday morning. This allows me time to recover before Monday morning. Just do not plan to make changes during the busiest time. Customers do not like errors or unexpected problems when browsing your site.
  • If you plan to modify an existing DNS resource record, make a note of the TTL’s current value. Then change the resource record’s TTL to a small value such as 30 seconds. Wait for the original TTL to expire before using a service that must query for the DNS domain such as creating an SSL certificate.
  • If you are creating a new DNS resource record, double-check the resource record’s value before writing it to the server. A mistake here can require waiting because of DNS record propagation and negative caching.
  • Validate your DNS server resource records before creating a Google-managed SSL certificate. If you have the DNS settings correct, the certificate will issue very quickly – typically in a couple of minutes. Make a mistake with the DNS server and you might find yourself in a multi-day problem.
  • Use reasonable values for a DNS resource records TTL. Do not set short TTL values except when preparing to update them. Short TTL values decrease site performance and increase DNS server costs. 24 hours is usually a good value once you have everything working. While testing and debugging problems, set the TTL to be very short. Then increase the value once the problem is solved.
  • Remember this item: make a change, verify that change, and then wait. DNS is a global system that caches DNS resource records. DNS does not care if you are in a hurry. If you are in a hurry, DNS will punish mistakes by making you wait even longer.
  • Do not make repeated changes. You will not know which change is being propagated to which servers.

More Information

Photography Credit

I write free articles about technology. Recently, I learned about Pexels.com which provides free images. The image in this article is courtesy of Pixabay at Pexels.

Related Posts