Introduction: We used GCP to carry out platform migration before, and encountered many problems in the process of using GCP platform, some of which were due to unskilled operation, some due to unclear official documents, and some problems encountered in practice.

Today we will focus on the problems encountered in the use of Storage in Google Cloud Platform.


Google Cloud Storage (GCS) is an integrated object Storage space for developers and enterprises. The website has details about it, including a description of its various features (almost all of them positive).
Data in a Storage is stored in buckets.

Each Bucket can store a variety of files: images, documents, audio, and so on.

Actual application scenarios: Download various files from the old platform through the API, and these files will be downloaded to the machine running the script, so you need to upload the local data to the intermediate Storage (Storage), then calculate the correct number of files, and then synchronize the data stored on Storage to the new platform (Google Drive or whatever).

How can resources on a Bucket be accessed and manipulated locally

Storage has a utility called gsutil. You can use it to manipulate Storage, for example:

How can I upload a local file to a Storage

The first is to manually upload via the page console.
The second way is much more convenient, using the command line through the gsutil tool.

#Upload the files from the local folder to the bucket
gsutil -m cp -r /path/to/local/folder gs://bucket_name
#Copy copies from one Bucket to another
gsutil -m cp -r   gs://bucket_name/* gs://another_bucket_name

How to calculate the number of files on Storage

The ls command in the gsutil tool with options -l (long listing) and -r (recursive listing) will display all the contents of the file and a total:

$ gsutil ls -lR gs://pub

    104413  2011-04-03T20:58:02Z  gs://pub/SomeOfTheTeam.jpg

       172  2012-06-18T21:51:01Z  gs://pub/cloud_storage_storage_schema_v0.json

      1379  2012-06-18T21:51:01Z  gs://pub/cloud_storage_usage_schema_v0.json

   1767691  2013-09-18T07:57:42Z  gs://pub/gsutil.tar.gz

   2445111  2013-09-18T07:57:44Z  gs://pub/

      1136  2012-07-19T16:01:05Z  gs://pub/gsutil_2.0.ReleaseNotes.txt

... <snipped> ...


     10372  2013-06-10T22:52:58Z  gs://pub/apt/pool/main/p/python-socksipy-branch/python-socksipy-branch_1.01_all.deb


        84  2010-05-07T23:36:25Z  gs://pub/shakespeare/rose.txt

TOTAL: 144 objects, 102723169 bytes (97.96 MB)

You can use the tail command if you only want the total

$ gsutil ls -lR gs://pub | tail -n 1

TOTAL: 144 objects, 102723169 bytes (97.96 MB)


Gsutil now also has a du command. This command makes it easier to get a total.

$ gsutil du gs://pub | wc -l


How do I mount GCS content locally

In the requirements, you need to run a script to back up the data on the GCS to another location. So first you need to get the GCS data. This process uses a command called gcsfuse

Prerequisites: You have obtained authorization

mkdir /path/to/mount/point
gcsfuse my-bucket /path/to/mount/point

Mount only the first layer of content to local, if you want to mount all the content of the sub-layer to local need to add parameters

gcsfuse --implicit-dirs "my-bucket" /path/to/mount/point

The file can then be accessed locally via /path/to/mount/point:
For example, by reading and writing files, uploading files to Google Drive or some cloud platform.

These are the usages related to Storage Bucket operations.