Organizing Objects in a Namespace

Cloud Storage uses buckets to group objects. Objects are different from files in that they cannot be updated. They are versioned and accessed as a single unit; that is, you cannot access a subset of an object the way you can access a block of a file in a filesystem. A bucket is a group of objects that share access controls at the bucket level. For example, the service account assigned to a virtual machine may have permissions to write to one bucket and read from another bucket. Individual objects within buckets can have their own access controls as well.

Google Cloud Storage uses a global namespace for bucket names, so all bucket names must have unique names. Object names do not have to be unique. A bucket is named when it is created and cannot be renamed. To simulate renaming a bucket, you will need to copy the contents of the bucket to a new bucket with the desired name and then delete the original bucket.

The following are best practice suggestions for bucket naming:

• Do not use personally identifying information, such as names, email addresses, IP addresses, and so forth in bucket names. That kind of information could be useful to an attacker.
• Follow DNS naming conventions because bucket names can appear in a CNAME record in DNS.
• Use globally unique identifiers (GUIDs) if creating many buckets.
• Do not use sequential names or timestamps if uploading files in parallel. Files with sequentially close names will likely be assigned to the same server. This can create a hotspot when writing files to Cloud Storage.
• Bucket names can also be subdomain names, such as mybucket.example.com.

The Cloud Storage service does not provide a filesystem. This means that there is no ability to navigate a path through a hierarchy of directories and files. The object store does support a naming convention that allows for the naming of objects in a way that looks similar to the way that a hierarchical filesystem would structure a file path and filename. If you would like to use Google Cloud Storage as a filesystem, the Cloud Storage FUSE open source project provides a mechanism to map from object storage systems to filesystems.

Cloud Storage FUSE

Filesystem in Userspace (FUSE) is a framework for exposing a filesystem to the Linux kernel. FUSE uses a stand-alone application that runs on Linux and provides a filesystem API along with an adapter for implementing filesystem functions in the underlying storage system. Cloud Storage FUSE is an open source adapter that allows users to mount Cloud Storage buckets as filesystems on Linux and macOS platforms.

Cloud Storage FUSE is not a filesystem like NFS. It does not implement a filesystem or a hierarchical directory structure. It does interpret / characters in filenames as directory delimiters.

For example, when using Cloud Storage FUSE, a user could mount a Cloud Storage bucket to a mount point called gcs. The user could then interact with the local operating system to save a file named mydata.csv to /gcs/myproject/mydirectory/mysubdirectory. The user could execute the ls command at the command line to list the contents of the simulated mysubdirectory to see the listing for mydata.csv along with any other files with a name prefixed by myproject/mydirectory. The user could then use a gsutil command or cloud console to list the contents of the mounted bucket, and they would see an object named myproject/mydirectory/mysubdirectory.

Cloud Storage FUSE is useful when you want to move files easily back and forth between Cloud Storage and a Compute Engine VM, a local server, or your development device using filesystem commands instead of gsutil commands or the cloud console.

Cloud Storage FUSE is a Google-developed and community-supported open source project under the Apache license. Cloud Storage FUSE is available at github.com/GoogleCloudPlatform/gcsfuse.

Storage Tiers

Cloud Storage offers four tiers or types of storage. It is essential to understand the characteristics of each tier and when it should be used for the Professional Cloud Architect exam. The four types of Cloud Storage are as follows:

• Standard
• Nearline
• Coldline
• Archive

Standard storage is well suited for use with frequently accessed (“hot”) data or for data that is stored briefly. Standard storage has three location types: region, dual-region, and multiregion.

The region location type stores multiple copies of an object in multiple zones in one region. All Cloud Storage options provide high durability, which means the probability of losing an object during any period is extremely low. Cloud Storage provides 99.999999999 percent (eleven 9s) annual durability.

This level of durability is achieved by keeping redundant copies of the object. Availability is the ability to access an object when you want it. An object can be durably stored but unavailable. For example, a network outage in a region would prevent you from accessing an object stored in that region, although it would continue to be stored in multiple zones.

Dual-region and multiregion storage mitigate the risk of a regional outage by storing replicas of objects in two or multiple regions, respectively. This can also improve access time and latency by distributing copies of objects to locations that are closer to the users of those objects. Consider a user in California in the western United States accessing an object stored in us-west1, which is a region located in the northwest state of Oregon in the United States. That user can expect under 5 ms latency with a user in New York, in the United States northeast, and would likely experience latencies closer to 30ms.

Dual-region and multiregion storage are also known as geo-redundant storage.

The latencies mentioned here are based on public internet network infrastructure. Google offers two network tiers: Standard and Premium. With Standard network tier, data is routed between regions using public internet infrastructure and is subject to network conditions and routing decisions beyond Google's control. The Premium network tier routes data over Google's global high-speed network. Users of Premium tier networking can expect lower latencies.

Nearline and Coldline storage are used for storing data that is not frequently accessed. Data that is accessed less than once in 30 days is a good candidate for Nearline storage. Data that is accessed less than once in 90 days is a good candidate for Coldline storage. All storage classes have the same latency to return the first byte of data. Archive storage is optimal for data accessed less than once per year.

Nearline, Coldline, and Archive storage have slightly lower availability than Standard storage, lower at-rest storage costs, and higher data access costs.

Cloud Storage also has three other classes: Multi-Regional storage, Regional storage, and Durable Reduced Availability (DRA) storage. Google Cloud recommends using Standard storage instead of these unless you are already using one of these additional classes.

Cloud Storage Use Cases

Cloud Storage is used for a few broad use cases.

• Storage of data shared among multiple instances that does not need to be on persistent attached storage. For example, log files may be stored in Cloud Storage and analyzed by programs running in a Cloud Dataproc Spark cluster.
• Backup and archival storage, such as persistent disk snapshots, backups of on-premises systems, and data kept for audit and compliance requirements but not likely to be accessed.
• As a staging area for uploaded data. For example, a mobile app may allow users to upload images to a Cloud Storage bucket. When the file is created, a Cloud Function could trigger to initiate the next steps of processing.

Each of these examples fits well with Cloud Storage's treatment of objects as atomic units. If data within the file needs to be accessed and processed, that is done by another service or application, such as a Spark analytics program.

Different tiers are better suited for some use cases. For example, Archive class storage is best used for archival storage, but Multiregional storage may be the best option for uploading user data, especially if users are geographically dispersed.

As an architect, it is important to understand the characteristics of the four storage tiers, their relative costs, the assumption of atomicity of objects by Cloud Storage, and how Cloud Storage is used in larger workflows.

Cloud Filestore Service Tiers

Cloud Filestore is especially useful when organizations are lifting and shifting applications that require a filesystem and cannot use object storage systems, such as Cloud Storage. Some typical use cases for Cloud Filestore are home directories and shared directories, web server content, and migrated applications that require a filesystem.

Cloud Filestore has three service tiers.

• Filestore Basic
• Filestore High Scale
• Filestore Enterprise

Filestore Basic is designed for file sharing, software, and Google Kubernetes Engine workloads that benefit from a persistent, managed filesystem. Filestore Basic is a zonal resource.

Filestore High Scale is designed to meet the demands of high-performance computing workloads, such as genome analysis and financial services applications requiring low-latency file operations. Filestore High Scale is a zonal resource.

Filestore Enterprise is for mission-critical applications and Google Kubernetes Engine workloads. This version provides 99.99 percent regional availability by provisioning multiple NFS shares across multiple zones within a region.

Filestore provides for snapshots of the filesystem, which can be taken periodically. If you need to recover a filesystem from a snapshot, it would be available within 10 minutes.

Cloud Filestore Networking

Cloud Filestore connects to VPC networks using either VPC Network Peering or private services access. VPC Network Peering is used when creating a filesystem instance with a stand-alone VPC network, when creating an instance within the host project of a Shared VPC, or when accessing the filesystem from an on-premises network using Cloud VPN or Cloud Interconnect. Private service access is used when creating an instance on a Shared VPC network from a service project, not the host project, or when you are using centralized IP range management for multiple Google Cloud services. Note, Cloud Filestore does not support transitive peering.

Cloud Filestore Access Controls

Access controls to Cloud Filestore and to files on the filesystem are managed with a combination of IAM roles and POSIX file permissions.

The roles/file.editor and roles/file.viewer are used to grant permissions on the Cloud Filestore instance to users. The viewer role allows users to see details about an instance, its location, backups, and snapshots as well as the operational status of the instance. The editor role includes the viewer permissions as well as permissions to create and delete instances, backups, and snapshots. These roles do not provide access to the files in the Filestore instance.

When a Filestore instance is created, it has default POSIX permissions of rwxr-xr-x permissions that are changed using operating system commands such as chmod. Access control lists are also supported using operating systems commands such as setfacl.

Relational Database Overview

Relational databases are highly structured data stores that are designed to store data in ways that minimize the risk of data anomalies and to support a comprehensive query language. An example of a data anomaly is inserting an employee record into a database department ID that does not exist in the department table. Relational databases support data models that include constraints that help prevent anomalies.

Another common characteristic of relational databases is support for ACID transactions. (ACID stands for atomicity, consistency, isolation, and durability.) NoSQL databases may support ACID transactions as well, but not all do.

Isolation

Isolation refers to ensuring that the effects of transactions that run at the same time leave the database in the same state as if they ran one after the other. Let's consider an example.

Transaction 1 is as follows:

Transaction 2 is as follows:

When high isolation is in place, the value of C will be either 5 or 15, which are the results of either of the transactions. The data will appear as if the three operations of Transaction 1 executed first and the three operations of Transaction 2 executed next. In that case, the value of C is 15. If the operations of Transaction 2 execute first followed by the Transaction 1 operations, then C will have the value 5.

What will not occur is that some of the Transaction 1 operations will execute followed by some of the Transaction 2 operations and then the rest of the Transaction 1 operations.

Here is an execution sequence that cannot occur when isolation is in place:

This sequence of operations would leave C with the assigned value of 7, which would be an incorrect state for the database to be in.

Cloud SQL

Cloud SQL is a managed service that provides MySQL, SQL Server, and PostgreSQL databases.

Cloud SQL allows users to deploy MySQL, SQL Server, and PostgreSQL on managed virtual servers. GCP manages patching database software, backups, and failovers. Key features of Cloud SQL include the following:

• All data is encrypted at rest and in transit.
• Data is replicated across multiple zones for high availability.
• GCP manages failover to replicas.
• Support for standard database connectors and tools is provided.
• The service provides integrated monitoring and logging.

Cloud SQL is an appropriate choice for regional databases up to 30 TB.

Customers can choose the machine type and disk size for the virtual machine running the database server. MySQL, SQL Server, and PostgreSQL all support the SQL query language.

Relational databases, like MySQL, SQL Server, and PostgreSQL, are used with structured data. Both require well-defined schemas, which can be specified using the data definition commands of SQL. These databases also support strongly consistent transactions, so there is no need to work around issues with eventual consistency. Cloud SQL is appropriate for transaction processing applications, such as e-commerce sales and inventory systems.

By default, Cloud SQL creates an instance in a single zone, but Cloud SQL provides for high availability by optionally maintaining a failover replica of the primary instance. Any changes to the primary instance are mirrored on the failover replica. If the primary instance fails, Cloud SQL will detect the failure and promote the failover replica to the primary instance.

Cloud SQL also supports read replicas. A read replica is a copy of the primary instance data stored in the same region or different region from the primary instance. When a read replica is in a different region, it is known as a cross-region replica. Cross-region replicas provide support for disaster recovery and migration of data between regions.

Database Migration Service is a managed service for migrating MySQL and PostgreSQL databases to Cloud SQL. Support for SQL Server migration is expected soon. The service uses change data capture mechanisms to create the initial snapshot and provide for ongoing replications. The Database Migration Service uses the replication mechanisms of the underlying database. The service is useful for both lift-and-shift migrations and for multicloud continuous replication.

A limiting factor of Cloud SQL is that databases can scale only vertically, that is, by moving the database to a larger machine. For use cases that require horizontal scalability or support, a globally accessed database, Cloud Spanner, is an appropriate choice.

Cloud Spanner

Cloud Spanner is a managed database service that supports horizontal scalability across regions. This database supports common relational features, such as schemas for structured data and SQL for querying. Cloud Spanner supports both Google Standard SQL (ANSI 2011 with extensions) and Postgres dialects.

It supports strong consistency, so there is no risk of data anomalies caused by eventual consistency. Cloud Spanner also manages replication. Cloud Spanner is used for applications that require strong consistency on a global scale. Here are some examples:

• Financial trading systems require a globally consistent view of markets to ensure that traders have a consistent view of the market when making trades.
• Logistics applications managing a global fleet of vehicles need accurate data on the state of vehicles.
• Global inventory tracking requires global-scale transaction to preserve the integrity of inventory data.

Cloud Spanner provides 99.999 percent availability, which guarantees less than 5 minutes of downtime per year. Like Cloud SQL, all patching, backing up, and failover management is performed by GCP.

Data is encrypted at rest and in transit. Cloud Spanner is integrated with Cloud Identity to support the use of user accounts across applications and with Cloud Identity and Access Management to control authorizations to perform operations on Cloud Spanner resources.

As with any distributed database, there is the potential for hot spotting. That is skewing the database workload so that a small number of nodes are doing a disproportionate amount of work. When using Cloud Spanner, it is recommended that you use primary keys that do not lead to hotspotting. Incremented values, time stamps, and other values that monotonically increase in the first part of the key should not be used as a primary key since it will lead to writes being directed to a single server instead of more evenly distributed across all servers.

Cloud Spanner supports secondary indexes in addition to primary key indexes.

Cloud Spanner stores data encrypted at rest and by default uses Google-managed encryption. If you need to manage your encryption, you have the option to use Cloud Key Management Service (KMS) with a symmetric key, a Cloud HSM key, or a Cloud External Key Manager key.

Analytical Database: BigQuery

BigQuery is a managed data warehouse and analytics database solution. It is designed to support queries that scan and return large volumes of data, and it performs aggregations on that data. BigQuery uses SQL as a query language. Customers do not need to choose a machine instance type or storage system. BigQuery is a serverless application from the perspective of the user.

NoSQL Databases

GCP offers three NoSQL databases: Bigtable, Datastore, and Cloud Firestore. All three are well suited to storing data that requires flexible schemas. Cloud Bigtable is a wide-column NoSQL database. Cloud Firestore and Cloud Datastore are document NoSQL databases. Cloud Firebase is the next generation of Cloud Datastore, and in the future existing Datastore databases will be automatically upgraded to Firestore using Datastore mode.

Caching with Cloud Memorystore

Cloud Memorystore is a managed cache service supporting both Redis and Memcached. Memorystore caches are used for storing data in nonpersistent memory, particularly when low-latency access is important. Stream processing and database caching are both common use cases for Memorystore.

1. You need to store a set of files for an extended period. Anytime the data in the files needs to be accessed, it will be copied to a server first, and then the data will be accessed. Files will not be accessed more than once a year. The set of files will all have the same access controls. What storage solution would you use to store these files?
   1. Cloud Storage Archive
   2. Cloud Storage Nearline
   3. Cloud Filestore
   4. Bigtable
2. You are uploading files in parallel to Cloud Storage and want to optimize load performance. What could you do to avoid creating hotspots when writing files to Cloud Storage?
   1. Use sequential names or time stamps for files.
   2. Do not use sequential names or time stamps for files.
   3. Configure retention policies to ensure that files are not deleted prematurely.
   4. Configure lifecycle policies to ensure that files are always using the most appropriate storage class.
3. As a consultant on a cloud migration project, you have been asked to recommend a strategy for storing files that must be highly available even in the event of a regional failure. What would you recommend?
   1. BigQuery
   2. Cloud Datastore
   3. Multiregional Cloud Storage
   4. Regional Cloud Storage
4. As part of a migration to Google Cloud Platform, your department will run a collaboration and document management application on Compute Engine virtual machines. The application requires a filesystem that can be mounted using operating system commands. All documents should be accessible from any instance. What storage solution would you recommend?
   1. Cloud Storage
   2. Cloud Filestore
   3. A document database
   4. A relational database
5. Your team currently supports seven MySQL databases for transaction processing applications. Management wants to reduce the amount of staff time spent on database administration. What GCP service would you recommend to help reduce the database administration load on your teams?
   1. Bigtable
   2. BigQuery
   3. Cloud SQL
   4. Cloud Filestore
6. Your company is developing a new service that will have a global customer base. The service will generate large volumes of structured data and require the support of a transaction processing database. All users, regardless of where they are on the globe, must have a consistent view of data. What storage system will meet these requirements?
   1. Cloud Spanner
   2. Cloud SQL
   3. Cloud Storage
   4. BigQuery
7. Your company is required to comply with several government and industry regulations, which include encrypting data at rest. What GCP storage services can be used for applications subject to these regulations?
   1. Bigtable and BigQuery only
   2. Bigtable and Cloud Storage only
   3. Any of the managed databases, but no other storage services
   4. Any GCP storage service
8. As part of your role as a data warehouse administrator, you occasionally need to export data from the data warehouse, which is implemented in BigQuery. What command-line tool would you use for that task?
   1. gsutil
   2. gcloud
   3. bq
   4. cbt
9. Another task that you perform as data warehouse administrator is granting authorizations to perform tasks with the BigQuery data warehouse. A user has requested permission to view table data but not change it. What role would you grant to this user to provide the needed permissions but nothing more?
   1. dataViewer
   2. admin
   3. metadataViewer
   4. dataOwner
10. A developer is creating a set of reports and is trying to minimize the amount of data each query returns while still meeting all requirements. What bq command-line option will help you understand the amount of data returned by a query without actually executing the query?
    1. --no-data
    2. --estimate-size
    3. --dry-run
    4. --size
11. A team of developers is choosing between using NoSQL or a relational database. What is a feature of NoSQL databases that is not available in relational databases?
    1. Fixed schemas
    2. ACID transactions
    3. Indexes
    4. Flexible schemas
12. A group of venture capital investors has hired you to review the technical design of a service that will be developed by a startup company seeking funding. The startup plans to collect data from sensors attached to vehicles. The data will be used to predict when a vehicle needs maintenance and before the vehicle breaks down. Thirty sensors will be on each vehicle. Each sensor will send up to 5 KB of data every second. The startup expects to start with hundreds of vehicles, but it plans to reach 1 million vehicles globally within 18 months. The data will be used to develop machine learning models to predict the need for maintenance. The startup is considering using a self-managed relational database to store the time-series data but wants your opinion. What would you recommend for a time-series database?
    1. Continue to plan to use a self-managed relational database.
    2. Use Cloud SQL.
    3. Use Cloud Spanner.
    4. Use Bigtable.
13. A Bigtable instance increasingly needs to support simultaneous read and write operations. You'd like to separate the workload so that some nodes respond to read requests and others respond to write requests. How would you implement this to minimize the workload on developers and database administrators?
    1. Create two instances, and separate the workload at the application level.
    2. Create multiple clusters in the Bigtable instance, and use Bigtable replication to keep the clusters synchronized.
    3. Create multiple clusters in the Bigtable instance, and use your own replication program to keep the clusters synchronized.
    4. It is not possible to accomplish the partitioning of the workload as described.
14. As a database architect, you've been asked to recommend a database service to support an application that will make extensive use of JSON documents. What would you recommend to minimize database administration overhead while minimizing the work required for developers to store JSON data in the database?
    1. Cloud Storage
    2. Cloud Firestore
    3. Cloud Spanner
    4. Cloud SQL
15. Your Cloud SQL database is experiencing high query latency. You could vertically scale the database to use a larger instance, but you do not need additional write capacity. What else could you try to reduce the number of reads performed by the database?
    1. Switch to Cloud Spanner.
    2. Use Cloud Bigtable instead.
    3. Use Cloud Memorystore to create a database cache that stores the results of database queries. Before a query is sent to the database, the cache is checked for the answer to the query.
    4. Add read replicas to the Cloud SQL database.
16. You would like to move objects stored in Cloud Storage automatically from regional storage to Nearline storage when the object is six months old. What feature of Cloud Storage would you use?
    1. Retention policies
    2. Lifecycle policies
    3. Bucket locks
    4. Multiregion replication
17. A customer has asked for help with a web application. Static data served from a data center in Chicago in the United States loads slowly for users located in Australia, South Africa, and Southeast Asia. What would you recommend to reduce latency?
    1. Distribute data using Cloud CDN.
    2. Use Premium Network from the server in Chicago to client devices.
    3. Scale up the size of the web server.
    4. Move the server to a location closer to those users.
18. A data pipeline ingests performance monitoring data about a fleet of vehicles using Cloud Pub/Sub. The data is written to Cloud Bigtable to enable queries about specific vehicles. The data will also be written to BigQuery and BigQuery ML will be used to build predictive models about failures in vehicle components. You would like to provide high throughput ingestion and exactly-once delivery semantics when writing data to BigQuery. How would you load that data into BigQuery?
    1. BigQuery Transfer Service
    2. Cloud Storage Transfer Service
    3. BigQuery Storage Write API
    4. BigQuery Load Jobs