How to Upload Image to Google Cloud Storage Using Python
Uploading and Downloading Zip Files In GCP Cloud Storage Using Python
GCP (Google Cloud Platform) cloud storage is the object storage service provided past Google for storing many data formats from PNG files to zipped source code for web apps and cloud functions. The information is stored in a flat, key/value-like data structure where the key is your storage object'south name and the value is your data.
Object storage is groovy for storing massive amounts of data every bit a single entity, data that volition later be accessed all at in one case as opposed to information that will exist read and written in small subsets as is the case with relational and not-relational databases.
If you're looking to store a collection of files as a unmarried unit, either to archive a big number of log files for hereafter audits or to bundle and store code equally a part of an automatic deployment bike, it'due south likely yous will do then by packing all of information technology together as a cipher file.
Using an awarding to automate the procedure of creating, altering, or unzipping a zip file in memory is a useful skill to have nonetheless working with memory streams and bytes rather than integers, strings, and objects can be daunting when it is unfamiliar territory.
Whether you are specifically looking to upload and download zip files to GCP deject storage or you merely take an interest in learning how to work with zilch files in memory, this postal service will walk you through the procedure of creating a new zip file from files on your local machine and uploading them to cloud storage likewise every bit downloading an existing cypher file in cloud storage and unzipping it to a local directory.
Establishing Credentials
Before you tin can begin uploading and downloading local files to cloud storage as zip files, you will demand to create the customer object used in your Python code to communicate with your project's cloud storage resources in GCP.
There are various ways to establish credentials that will grant the client object admission to a cloud storage bucket, the near mutual of which is to create a service account and assign it to your application in one of two ways.
The first choice is to assign the service account to a particular resources upon deployment. For example, if your code is being deployed as a GCP deject office, you would attach the service account to the application upon deployment using either the gcloud sdk:
# using powershell and the gcloud sdk to deploy a python cloud function gcloud functions deploy my-cloud-function ` --entry-point my_function_name ` --runtime python38 ` --service-account my-cloud-function @ my-project-id.iam.gserviceaccount.com ` --trigger-http
Or by using an IAC (infrastructure as code) solution similar Terraform:
resource "google_service_account" "my_cloud_func_sa" { account_id = "my-cloud-function" display_name = "Cloud Office Service Business relationship" } resource "google_project_iam_binding" "cloud_storage_user" { project = "my-project-id" role = "roles/storage.objectAdmin" members = [ "serviceAccount: ${ google_service_account . my_cloud_func_sa . e-mail } " , ] } resource "google_cloud_functions_function" "my_cloud_func" { proper noun = "my-cloud-function" entry_point = "my_function_name" runtime = "python38" service_account_email = google_service_account . my_cloud_func_sa . electronic mail trigger_http = true }
Annotation that the service account as defined in Terraform is also being referenced in a google_project_iam_binding resource equally a member that will exist assigned the role of storage.objectAdmin. You lot will need to assign a similar function (or ideally one with the minimal permissions required for your code to perform its tasks) if yous cull to create a service account using the GCP console.
For code being deployed with an assigned service account, creating the GCP deject storage client in Python requires only the project id be passed as an argument to the customer constructor.
from google.cloud import storage client = storage . Customer ( project = GCP_PROJECT_ID , )
However if yous would like to upload and download to deject storage using a CLI application or to test your deject role before deploying it, you will want to use a locally stored JSON credentials file.
To create the file, open up the GCP console and select IAM & Admin from the Navigation carte, accessed through the hamburger carte du jour icon in the top left corner.
From the IAM & Admin bill of fare, select the Service Accounts page and either create a new service business relationship or click on the link of an existing one, establish under the Email column of the service accounts table.
At the lesser of the Details page for the selected service account, click Add Fundamental > Create New Key and select the JSON option.
This volition download the JSON credentials file to your machine.
Anyone with access to this file will have the credentials necessary to make changes to your cloud resources co-ordinate to the permissions of this service business relationship. Store it in a secure identify and exercise not check this file into source control. If y'all do, immediately delete the central from the aforementioned menu used to create it and remove the JSON file from source command.
To allow your client object to use these credentials and admission GCP cloud storage, initializing the client will require a few extra steps. You lot will need to create a credentials object using the from_service_account_file method on the service_account.Credentials grade of the google.oauth2 library. The just required argument for this method is the absolute or relative file path to your JSON credentials file.
This credentials object will exist passed as a second argument to the storage.Client class constructor.
from google.deject import storage from google.oauth2 import service_account credentials = service_account . Credentials . from_service_account_file ( SERVICE_ACCOUNT_FILE ) customer = storage . Client ( project = GCP_PROJECT_ID , credentials = credentials )
Uploading Local Files to Cloud Storage as a Nix File
At present that your client object has the required permissions to access cloud storage yous can begin uploading local files as a zip file.
Assuming that the files you intend to upload are all in the same directory and are not already zipped, you will upload the files to GCP cloud storage every bit a nix file by creating a zip archive in retentivity and uploading it as bytes.
from google.cloud import storage from zipfile import ZipFile , ZipInfo def upload (): source_dir = pathlib . Path ( SOURCE_DIRECTORY ) archive = io . BytesIO () with ZipFile ( archive , 'w' ) every bit zip_archive : for file_path in source_dir . iterdir (): with open ( file_path , 'r' ) as file : zip_entry_name = file_path . proper name zip_file = ZipInfo ( zip_entry_name ) zip_archive . writestr ( zip_file , file . read ()) annal . seek ( 0 ) object_name = 'super-important-data-v1' bucket = client . bucket ( BUCKET_NAME ) hulk = storage . Hulk ( object_name , bucket ) blob . upload_from_file ( annal , content_type = 'application/zip' )
io.BytesIO() creates an in retention binary stream used by the ZipFile object to shop all the data from your local files as bytes.
The files in the source directory are iterated over and for each one a ZipInfo object is created and written to the ZipFile object forth with the contents of the source file. The ZipInfo object corresponds to an private file entry inside a zip file and will be labeled with whatever file proper noun and extension you use in the constructor to instantiate the ZipInfo object. Using zip_entry_name = file_path.name every bit in the instance above volition set the file name and extension in the zippo file to match the proper noun and extension of the local file.
The in memory binary stream (the archive variable) is what you will be uploading to GCP cloud storage, nonetheless a prerequisite for uploading an in memory stream is that the stream position be gear up to the start of the stream. Without moving the position of the stream dorsum to nix with annal.seek(0) yous will go an error from the Google API when y'all try to upload the data.
With the in memory binary stream ready to be delivered, the remaining lines of lawmaking create a new Saucepan object for the specified bucket and a Blob object for the storage object. The zipped files are then uploaded to cloud storage and can later on retrieved using the storage object name you lot used to create the Blob instance.
A bucket in deject storage is a user defined partition for the logical separation of data and a blob (as the Python course is chosen) is another name for a storage object.
Downloading a Zip File Blob in Deject Storage to a Local Directory
To download a nix file storage object and unzip information technology into a local directory, yous volition need to contrary the process by starting time creating a bucket object and a blob object in order to download the nada file as bytes.
def download (): target_dir = pathlib . Path ( TARGET_DIRECTORY ) object_name = 'super-important-data-v1' bucket = client . bucket ( BUCKET_NAME ) blob = storage . Blob ( object_name , bucket ) object_bytes = hulk . download_as_bytes () archive = io . BytesIO () annal . write ( object_bytes ) with ZipFile ( archive , 'westward' ) every bit zip_archive : zip_archive . extractall ( target_dir )
Once downloaded, the bytes tin exist written to an in retentiveness stream which will in turn be used to create a ZipFile object in guild to extract the files to your target directory. io.BytesIO() is again used to create the in memory binary stream and the write method on the BytesIO object is used to write the downloaded bytes to the stream. The ZipFile object has a method for extracting all of its contents to a specified directory, making the concluding step a uncomplicated one.
With these two functions and the appropriate credentials you should have everything yous need to start uploading and downloading your ain cypher files into cloud storage using Python.
And if y'all'd similar to run across all the Python code in one place, you can discover it here equally a Gist on my Github account.
Source: https://dev.to/jakewitcher/uploading-and-downloading-zip-files-in-gcp-cloud-storage-using-python-2l1b