[Total: 0    Average: 0/5]

Date created: March 10, 2018
Last updated: March 10, 2019

Google Stackdriver is a very good product for monitoring and logging your compute instances on Google Cloud, AWS, Azure, Alibaba, etc.

This article covers Stackdriver logging for Google Compute instances running Debian 9.

To make sure that Stackdriver is installed on each instance, I create instance templates that contain a script in the custom metadata section to automate Stackdriver installation and setup.

An important item to remember, startup scripts are executed every time an instance starts and not just on instance creation.

In my startup script for Debian 9 Stretch, I install Google Stackdriver logging and monitoring agents.

If you are manually creating a Compute instance, copy this script into the Automation -> Startup script section when creating the instance.

Installing the Logging Agent

Installing the Monitoring Agent

Sending a test Stackdriver log message

logger "Hello Stackdriver

This message is sent to Stackdriver and can be found in Stackdriver Logging -> GCE VM Instance -> Instance Name. If you do not see this message after about 15 seconds, check for Stackdriver errors in the logfile on the instance.

Stackdriver logfile

To see the latest logs in the Stackdriver logfile for debugging:

tail /var/log/google-fluentd/google-fluentd.log

Common Stackdriver errors

No service account assigned to the VM instance

Missing IAM permission to write to Stackdriver

Compute instances without a public IP address

For instances without external IP addresses, you must enable Private Google Access to allow the Stackdriver Logging agent to send logs.

Verify that your instance can resolve the following DNS hostnames:

      • oauth2.googleapis.com
      • monitoring.googleapis.com
      • stackdriver.googleapis.com
Google Stackdriver service account file location

Stackdriver will check for the following location and use these credentials if present instead of the metadata service account credentials.

/etc/google/auth/application_default_credentials.json

IAM Permissions required for Stackdriver

Stackdriver Monitoring

Your VM instance needs the permission roles/monitoring.metricWriter which can be added via the role roles/monitoring.metricWriter. Link.

Stackdriver Logging

Your VM instance needs the permission logging.logEntries.create which can be added via the role roles/logging.logWriter. Link.

Stackdriver Error Reporting

Your VM instance needs the permission errorreporting.errorEvents.create which can be added via the role roles/errorreporting.writer. Link.

Stackdriver Profiler

Your VM instance needs the permission cloudprofiler.profiles.create and
cloudprofiler.profiles.update which can be added via the role roles/cloudprofiler.agent. Link.

Stackdriver Trace

Your VM instance needs the permission cloudtrace.traces.patch which can be added via the role roles/cloudtrace.agent. Link.

Stackdriver Debugger

You don’t directly give members permissions; instead, you grant them one or more roles on a GCP resource, which have one or more permissions bundled within them. Refer to this document.

To determine the currently installed versions:

Stackdriver Monitoring

Output:

 

Stackdriver Logging

Output:

Add a startup Script remotely

You can add a startup-script for a running instance from the CLI. Note: this command will replace the existing startup script.

Copy the following to a local file. In this example startup.script. Modify to fit your requirements:

 

Execute the following command from your desktop:

 

You can also store your scripts in Google Storage:

 

The startup script will be executed the next time the instance reboots.

Restarting the Stackdriver agent

sudo service google-fluentd restart

Stackdriver agent status

sudo service google-fluentd status

Upgrading the Stackdriver agent – Debian & Ubuntu systems

sudo apt-get install --only-upgrade google-fluentd

Note: This command does not change the agent’s configuration files. To get the latest default configuration and catch-all configuration files run the following command instead.

Uninstall the Stackdriver agent – Debian & Ubuntu systems