Introduction: We used GCP to carry out platform migration before, and encountered many problems in the process of using GCP platform, some of which were due to unskilled operation, some due to unclear official documents, and some problems encountered in practice.
Today we will focus on the problems encountered in the use of Storage in Google Cloud Platform.
Storage
Google Cloud Storage (GCS) is an integrated object Storage space for developers and enterprises. The website has details about it, including a description of its various features (almost all of them positive).
Data in a Storage is stored in buckets.
Each Bucket can store a variety of files: images, documents, audio, and so on.
Actual application scenarios: Download various files from the old platform through the API, and these files will be downloaded to the machine running the script, so you need to upload the local data to the intermediate Storage (Storage), then calculate the correct number of files, and then synchronize the data stored on Storage to the new platform (Google Drive or whatever).
How can resources on a Bucket be accessed and manipulated locally
Storage has a utility called gsutil. You can use it to manipulate Storage, for example:
- Create a bucket:
gsutil mb gs://bucket-name - Copy files to a bucket
gsutil cp /path/to/my/local/file gs://bucket-name - Copy the files in the bucket to local
gsutil cp gs://bucket-name/folder/filename . - Delete the files on the bucket
gsutil rm gs://bucket-name/folder/filename - For more gsutil usage, see：official document
How can I upload a local file to a Storage
The first is to manually upload via the page console.
The second way is much more convenient, using the command line through the gsutil tool.
#Upload the files from the local folder to the bucket
gsutil -m cp -r /path/to/local/folder gs://bucket_name
#Copy copies from one Bucket to another
gsutil -m cp -r gs://bucket_name/* gs://another_bucket_nameHow to calculate the number of files on Storage
The ls command in the gsutil tool with options -l (long listing) and -r (recursive listing) will display all the contents of the file and a total:
$ gsutil ls -lR gs://pub
104413 2011-04-03T20:58:02Z gs://pub/SomeOfTheTeam.jpg
172 2012-06-18T21:51:01Z gs://pub/cloud_storage_storage_schema_v0.json
1379 2012-06-18T21:51:01Z gs://pub/cloud_storage_usage_schema_v0.json
1767691 2013-09-18T07:57:42Z gs://pub/gsutil.tar.gz
2445111 2013-09-18T07:57:44Z gs://pub/gsutil.zip
1136 2012-07-19T16:01:05Z gs://pub/gsutil_2.0.ReleaseNotes.txt
... <snipped> ...
gs://pub/apt/pool/main/p/python-socksipy-branch/:
10372 2013-06-10T22:52:58Z gs://pub/apt/pool/main/p/python-socksipy-branch/python-socksipy-branch_1.01_all.deb
gs://pub/shakespeare/:
84 2010-05-07T23:36:25Z gs://pub/shakespeare/rose.txt
TOTAL: 144 objects, 102723169 bytes (97.96 MB)
You can use the tail command if you only want the total
$ gsutil ls -lR gs://pub | tail -n 1
TOTAL: 144 objects, 102723169 bytes (97.96 MB)
UPDATE
Gsutil now also has a du command. This command makes it easier to get a total.
$ gsutil du gs://pub | wc -l
232
How do I mount GCS content locally
In the requirements, you need to run a script to back up the data on the GCS to another location. So first you need to get the GCS data. This process uses a command called gcsfuse。
Prerequisites: You have obtained authorization
- Mount a Bucket to the local PC
mkdir /path/to/mount/point
gcsfuse my-bucket /path/to/mount/point
Mount only the first layer of content to local, if you want to mount all the content of the sub-layer to local need to add parameters
gcsfuse --implicit-dirs "my-bucket" /path/to/mount/point
The file can then be accessed locally via /path/to/mount/point:
For example, by reading and writing files, uploading files to Google Drive or some cloud platform.
These are the usages related to Storage Bucket operations.