Google Associate Cloud Engineer – Operations in Google Cloud Platform – GCP Part 2

6. Step 04b – Demo – Playing with Cloud Logging

Back. It took a little while for the function to be created, but finally it’s there. And now what we want to do, we want to put some objects into our cloud storage. So I’ll go back to Cloud Storage browser and I would select our specific bucket, my bucket tied with CF in 28 minutes and this is what would trigger our cloud function. So what I would want to do is to upload a few objects. You can actually upload anything that you would want. What I would make use of is the things that we made use of earlier. So I’ll go to downloads. Inside cloud storage is where we have a number of objects. Actually we have a number of files. I’ll drag them in here and I would start the upload process. So what would happen as soon as an upload here is done, a cloud function will be triggered and if you go to the cloud function you can see the logs for it.

So you can go to the logs for the cloud function and you would see that there are a lot of logs due to the triggering of the cloud function. Whenever an object is, you can see function execution started processing file a specific file and function execution took four milliseconds, so on and so forth. You can see that there are a number of function execution messages present in here. Now the important thing for you to understand is the fact that all these logs are coming in because of the integration of cloud functions with Cloud Logging. Cloud Logging is the logging service. So I’ll go to Cloud logging. The cloud. Logging is the logging service in Google cloud. All the services in Google. Cloud integrate with Cloud logging. And that’s where you can see all the logs related to all the services.

Services like cloud functions, cloud run app engine, Google Kubernetes engine and compute engine. All of these have very very good integration with Cloud Logging. As soon as I go into Logs Explorer, you should be able to see all the recent logs that we have triggered. So you can see that there are a lot of logs which are present in here. So if I actually maximize this by clicking this in here you can see all the query results. If you want, you can also filter these logs. So let’s say I would want to filter by something, let’s say I would want to filter by the resource label. I can click that in here so I can say click. Once you click that you’d be able to say I only want to see matching entries and I can say show matching entries and this would actually create a filter and based on which we can see the data for that specific filter. So the filter is based on resource labels.

Function name is equal to the function which is being invoked. So all the logs for this function is what we are able to see right now, similar to this, you can filter logs based on a number of criteria. So if you click this specific icon which is present in here, you’d be able to see that each log entry contains a lot of details. It contains the text payload, it contains, which resources generating that log cloud function, and a label around that. So the label is function name is this, which project, which region, and there’s a timestamp associated with it, the severity. And also you can see a few more details around that. You can use any of these details to filter your log messages and you can explore them. So if you have a problem in production, you are trying to find out what the problem is, you can come over here, search for it and see if you are able to find the right logs for that specific thing.

Now where are these logs getting stored? The logs are getting stored in logs storage. So you can see that by default there are two log buckets. Default and required default is the default bucket where most of the application logs would be stored. And you can see that the retention period for default bucket is 30 days. The required bucket is the audit bucket. These are for most of the administration operations which happen in Google Cloud. And that’s the reason why these logs cannot be deleted. You cannot delete this bucket or you cannot change the retention period. The retention period will always be 400 days. Now, how does the logs get into this specific buckets? You have an application sending the logs.

How does the log get into the log buckets? That’s where we create routers. When any log comes in, these routers are executed. What I would do is I’ll go and say edit sync and this is where you can actually see the rules around that. Okay, a log entry is coming in, I want to see if it matches this rule. So the matching condition is, is it a cloud audit activity entry or is it an external audit activity entry? Or is it a system event from cloud audit or external audit? Or is it something related to access transparency in those kind of situations? The log should not be sent to this specific sync only. When it is not any of these, it should be sent to the default sync. So to the default only logs which are not any of these will be sent.

If I go back and say discard, if you go over to the required bucket and say edit sync, you can see that you cannot even edit the sync. Let’s do view sync details. You can see the inclusion filter in here. You can see that anything which is related to cloud audit, external audit, you can see that all the things that were excluded from default are what are the things which are included into the required filter whenever entry comes into cloud logging. All the rules which are configured for these log routers are executed and if the rule matches, then it would be sent to that specific sync. One important thing to remember is the same log entry can be sent to multiple syncs. If, let’s say I have five rules in here and three rules allow, then the entry will be sent to three buckets.

Another important thing to remember is you can also disable these rules. So if I don’t want any of the logs to go to default, if I think it’s unnecessary information, I can go in here and disable this sync as well. The last important thing that we would look at is how to create a sync. What are the different syncs you can create? Let’s go to create sync and you can give a name in here. We are not going to create this sync. So whatever name you’d give is fine. So let’s go and say test. And this is where you can say sync service. So where do you want to send the log entry to? In addition to the log storage buckets that we looked at earlier, you can send the logs out to a number of other sync services. You can send it to Cloud logging bucket, that’s the one which we looked at in here.

You can send to BigQuery data set. So you can store the log as data in BigQuery. You can send the log to a file in cloud storage bucket, or you can send it to Cloud Pub Subtopic or you can send it to a service like Splunk, or you can even send it to a destination in another project. So whatever log entries which are coming in, you can send it to different types of destinations and you configure in here which destination you’d want to send it to. For example, let’s just choose one. So let’s just say I want to send it to Cloud storage bucket. We are not going to create it, so it does not really matter. So I’ll choose one of the buckets which is present in here and let’s say I would want to send all data to that specific bucket. And over here is where we would define the filter.

Just like we saw earlier, you can define a condition on the logs that decides if the log should be sent to the sync or not. Once you configure the inclusion filter, you can also configure an exclusion filter. What are the things that should be excluded? So you can configure both an inclusion and an exclusion filter and then you can say create sync. I’m not going to create this sync. I would just say cancel. And I’ll say this card in this step we looked at cloud logging. Whenever you want to find the logs related to your applications or your services, this is where you would come to. And in cloud logging, the default place where most of the logs would go to are the cloud logging buckets. In addition to cloud logging buckets, you can create specific rules redirecting to specific syncs. I’m sure you’re having a wonderful time and I’ll see you in the next step.

7. Step 04c – Demo – Playing with Cloud Monitoring

Back in this step. Let’s look at cloud monitoring. So let’s go to monitoring. So, cloud monitoring is one of the most important monitoring services in Google Cloud. That’s where you can actually monitor all the resources that you are creating. In Google Cloud, we saw a lot of metrics related to compute engine, app engine. All of the metrics are actually coming in from cloud monitoring. And over here there are a few things that you can look at once you log in in here, once we create a workspace, we already talked about the fact that a single workspace of cloud monitoring can actually have data from multiple projects. So you can have all the projects configured in here and you can also have data from multiple AWS accounts. You can add the AWS accounts, which you want to monitor in here as well.

Now, if I go back to the overview, this is where I can see what are the different resources that are being monitored. You can see the resource dashboard, app engine, one resource cloud pub sub, one resource cloud storage, eight resources. So you can see all the details around that and you can see if there are any incidents or events or if you have any APPTime checks which are configured. You can see the details in here. You can also group resources by creating groups. You can say these resources, these virtual machines are a single group and I would want to look at data for it in a group. In those kind of situations, you can also create groups. You can go to dashboards and look at the precustomized dashboards that are present in here if you want.

You can also create your own dashboard. You can go to create Dashboard and create your own dashboard. You can also look at all the details related to all the metrics. So if I go to Metrics Explorer and let’s say I would want to look at function, so I want to look at cloud function related metrics because that’s what we did a few steps ago. And I would say I would want to look at and let’s look at, let’s take the first entry which is present in here, which is active instances. So how many cloud function resources were active at a specific point in time? And the graph data is loading in and you can see how many active instances are there at any specific point in time. If you want, you can also add a filter and you can group data as well. So this is how you can look at the metrics.

A couple of other important things that you need to remember as far as cloud monitoring are concerned are alerting and uptime checks. Alerting is very, very important feature of cloud monitoring, where you can set alerts, you can say this is the condition. When that condition happens, I would want to send an alert on different notification channels. You can edit the notification channels in here. What are the different options to send alerts? So there are different integrations which are supported like pager duty, Slack, web hooks, email, SMS and Pub sub notifications, so you can use any of them. So you can configure a notification channel in here. Once you have a notification channel in here, you can configure the condition under which a message should be sent to the notification channel. You can go in here and create a policy.

You can say I want to monitor this specific metrics. So I can say add condition. I want to look at cloud functions for example. So I would want to say cloud function and let’s just take something at random. I’ll say active instances when cloud function. Active instances go over a specific range of values. When cloud function if if I scroll down I can see the period five minutes and I can configure the trigger in here. I can say if the cloud function instances in a five minute time period go over 1000, I would want to be alerted. Let’s say add. So that’s the condition that we have configured. If you want, you can include multiple conditions and you can say next and this is where you configure the notification channels.

Right now I don’t have any notification channels, but if you have any notification channels configured, then the message would be sent on that specific notification channels. Once you configure the notification channel, you can also add a little bit of documentation around how to fix a specific issue so that’s alerting alerting is when a metric does not meet a specific criteria. You’d want to send an immediate alert to somebody. The other one is uptime checks. You want to ensure that your applications run all the time. If there is any downtime in your application, you’d want to find that out. And that’s where you can configure an uptime check. You can come here and say create uptime check and you can give it a name and you can say I would want to configure a target.

There are different types of targets that can be configured. So you can say I would want to look at a http URL or https URL or you can say I would want to monitor the uptime of app engine or a specific VM instance. Or you can also look at AWS elastic load balancers. So you want to monitor the uptime of an elastic load balancer. You can also configure frequency. How often do you want to send a request to this? And there are a lot of other options that you can configure in here. You can send custom headers, you can send authentication headers if you need, and after that you can configure how you want to validate the response. Do you want to check for status of it? Let’s just configure. What I’ll do is I’ll just configure a simple URL, let’s just say https google. com we are not going to create this uptime check.

So don’t worry about it. Let’s go and say next. And then you can say you can specify the response validation details. You can say, I would want the response to come within 10 seconds. And you can also say I would want to enable content matching and the response should contain this content. Once you have this, I’ll disable content matching for now and say Next and next you can configure the notification how do you want to receive a notification? So you can configure your notification channel else. So alerting is when some metrics don’t meet your criteria. Uptime checks is when a specific application is down in the step. We looked at cloud monitoring. I’m sure you’re having a wonderful time and I’ll see you in the next step.

8. Step 05 – Getting Started with Google Cloud Trace

Back in this step. Let’s look at Cloud Trace. Whenever we talk about microservices architecture, you have multiple microservices involved in a single request. How do you trace a request across microservices? How do you trace a request across multiple Google services? That’s where you can go for Cloud Trace. It’s a distributed tracing system for GCP. You can collect latency data from supported Google Cloud Services and instrumented applications. If you want to trace your request across microservices, you need to instrument your microservices using the tracing libraries.

Or you can actually use the Cloud Trace API directly. Using Cloud Trace, you’ll be able to find out how long does a service take to handle requests? What is the average latency of requests? How are we doing over a period of time? Are we improving or not? Cloud Trace is supported for Compute engine, GKE app engine, both flexible and standard. If you have applications, you can also integrate Trace client libraries into them. Trace client libraries are available for C, Sharp, Go, Java, Node, JS, PHP, Python and Ruby. If you want to perform distributed tracing in Google Cloud, the service you need to go for is Cloud Trace.

9. Step 06 – Getting Started with Google Cloud Debugger

Next up, let’s look at cloud debugger. How do you debug issues that are only happening in test or production environments? They are not happening in your local environment at all. You’d want to find out why something is going wrong in test or production environments. One of the options you have is cloud debugger. You can actually capture the state of a running application directly in your test or production environments. So you can inspect the state of the application directly directly in the GCP environment. You can take snapshots of variables and you can also look at the call stack. There is no need to add additional logging statements in your code.

So whatever code is running, you can directly use that and you can look at the variables, call stack and you can look at the different values which are present in there. Because there is no need to change the code, there is no need to redeploy as well. Cloud debugger. Also very lightweight. You can enable the debugger without impacting your users at all. There is very, very little impact for your end users. And because of that, you can use it in any environment, even in production. So if you don’t want to directly debug issues in your environment, you can use cloud debugger. I’ll.

10. Step 07 – Getting Started with Google Cloud Profiler

In this step, let’s look at Cloud Profiler. How do you identify performance bottlenecks in production? One of the options is cloud profiler. It is a low overhead profiler. It continuously gathers CPU and memory usage from production systems. You can connect the profiling data which you capture using Cloud Profiler with the application source code directly. And this would help you to identify performance bottlenecks directly in your source code. There are two major components that are important for Cloud Profiler. One is the profiling agent. The profiling agent is the one which collects the profiling information. And the second component is the interface. The interface of the profiler provides the visualization around the information which is gathered.

If you are struggling with identifying a performance bottleneck in production, one of the tools you can consider using is Profiler. The great thing about the profiler is it has very low overhead. So you can actually run the profiler even in a production system. I’ll see you in the next step. Welcome back. In this step, let’s talk about error reporting. How do you identify production problems in real time? That’s where the real time exception monitoring that’s a feature of error reporting comes into picture. You can aggregate and display errors reported from Cloud Services. These errors are directly captured from stacked traces of the applications. So you can see this error happened this many times and you can identify the specific exceptions which might need your attention.

So error reporting provides a centralized error management console. Identify and manage top errors or recent errors. If you are interested in errors from the Android mobile applications or iOS mobile applications, then you can go for Firebase Crash Reporting. Error reporting is supported for Go, Java, net, NodeJS, PHP, Python and Ruby. Errors can be reported by sending them to Cloud Logging or by directly calling the Error Reporting API. The Error Reporting tool can also be accessed directly from Desktop. Or you can also use the Cloud Console mobile app for iOS and Android. And error reporting is part of that mobile app. If you want to track your production problems in real time, the best place to go is error reporting.

11. Step 09 – Scenarios – Operations in Google Cloud Platform

Let’s look at different scenarios related to cloud operations. You’d like to record all operations or requests on all objects in a bucket for auditing. What can you do? You can turn on data access, audit logging for the bucket. You want to trace a request across multiple microservices. You can go to Cloud Trace. You want to identify prominent exceptions for a specific Microsoft service. You can go to error reporting. You want to debug a problem in production by executing step by step. You want to look at the variable information and see what’s happening. You can go to Cloud Debugger. You want to look at the logs for a specific request. You need to go to Cloud Logging. In this section we played around with Cloud operations. I’m sure you had a wonderful time and I’ll see you in the next step.

• Category: Uncategorized