Google Cloud Storage component

Google Cloud Storage Component is designed to connect to Google Cloud Storage JSON API

Description

Google Cloud Storage Component is designed to connect to Google Cloud Storage JSON API The current release of component support JSON API v1.

Environment variables

Name	Mandatory	Description	Values
`MAX_FILE_SIZE`	false	Maximum file size that can be downloaded from cloud storage to platform in megabytes (mb) (100MB by default)	any `integer` above 0

Credentials

Before building any integration flow you must at first configure the app from inside the Google Developers Console.

Go to the APIs & Services -> Enabled APIs & services page and enable the following:
- Cloud Storage
- Google Cloud Storage JSON API
Go to the Credentials section and create a new credential of type OAuth client ID.
- Set Application type to Web application
- Add Authorized redirect URI as: https://{your-tenant-address}/callback/oauth2
Create new or link existing Billing account to your project in Billing section - it must be valid (Status: active) to use this service

In case of new domain you may get message like This app isn't verified. Please refer to this doc to check how to proceed: https://support.google.com/cloud/answer/7454865?hl=en

Now you can create new credentials for component:

Credential

Type (dropdown, required) - OAuth2
Choose Auth Client (dropdown, required) - select one of created before or Add New Auth Client:
- Name (string, required) - provide any name you want
- Client ID (string, required) - put here Client ID from Web application in Google Developers Console
- Client Secret (string, required) - put here Client Secret from Web application in Google Developers Console
- Authorization Endpoint (string, required) - Google oauth2 authorization endpoint https://accounts.google.com/o/oauth2/v2/auth
- Token Endpoint (string, required) - Google refresh token endpoint https://oauth2.googleapis.com/token
Name Your Credential (string, required) - provide any name you want
Scopes (Comma-separated list) (string, required) - Put here scopes to get access to your Storage - https://www.googleapis.com/auth/devstorage.read_write, more info
Additional parameters (Comma-separated list) (string, required) - set it as access_type:offline,prompt:consent to make component works properly
Project ID (string, required) - Get project ID from Google Developers Console
Number of retries (number, optional, 5 by default) - How many times component should retry to make request
Delay between retries (number ms, optional, 10000 by default) - How much time wait until new try

Triggers

Get New and Updated Objects Polling

Retrieve all the updated or created objects within a given time range.

Configuration Fields

Bucket - (dropdown, required): select one of the available buckets
Enable File Attachments - (checkbox, optional): If checked, the file will be uploaded to local storage and the link provided in response
Timestamp field to poll on - (dropdown, optional, default Last Modified): Select which date will be used to track files - Last Modified or Created
Emit Behavior - (dropdown, optional, default Emit individually): Defines the way result objects will be emitted, one of Emit page or Emit individually.
Page Size - (number, optional, defaults to 999, max 999): Indicates the size of pages to be fetched per request
Start Time - (string, optional): The timestamp to start polling from (inclusive) - using ISO 8601 Date time utc format - YYYY-MM-DDThh:mm: ssZ. The default value is the beginning of time (January 1, 1970, at 00:00).
End Time - (string, optional): The timestamp to stop polling (exclusive) - using ISO 8601 Date time utc format - YYYY-MM-DDThh:mm:ssZ. Default value is flow execution time.

Input Metadata

There is no input metadata in this trigger.

Output Metadata

Depends on Enable File Attachments and Emit behavior fields.

If Emit behavior field is equal to Emit page - object with the property results that contains array of files
If Emit behavior field is equal to Emit individually, file information will fulfill the whole message
If Enable File Attachments is checked, for each file there will be an additional field - attachmentUrl

Limitations

Google Cloud Storage API doesn’t support filtering by dates - as result we collect information about all files from selected Bucket and filter them locally (inside component) for each trigger execution
From the point above, option Emit page does not always emit records according to Page Size

Webhook

This trigger allows tracking changes in the bucket

Configuration Fields

Bucket - (dropdown, required): select one of the available buckets
Enable File Attachments - (checkbox, optional, false by default): If selected, the contents of the file will be exported in addition to the attachment

Input Metadata

There is no Input metadata in this trigger.

Output Metadata

operation - (object, required): One of Create’, Update or Delete
fileMetadata - (object, required): File metadata from Google Cloud storage

If selected Enable File Attachments there will be additional field:

attachmentUrl - (string, required): Url to file in internal storage

Limitations

attachmentUrl not available for Deleted objects
In an ordinary flow you need to follow the webhook URL after a flow started in order to correctly initialize it (no extra actions needed if a flow is real-time)

Actions

Delete File

Delete file from Google Cloud Storage

Configuration Fields

Bucket - (dropdown, required): select one of the available buckets
Don’t throw error - (checkbox, optional, false by default): If got error (like file not found) emit empty message instead of throwing an error

Input Metadata

File name - (string, required): Name of the file in Cloud Storage

Output Metadata

name - name of deleted file

Download file

Download file from Google Cloud Storage to internal storage

Configuration Fields

Bucket - (dropdown, required): select one of the available buckets

Input Metadata

File name - (string, required): Name of the file in Cloud Storage

Output Metadata

attachmentUrl - (string, required): Url to file in internal storage
fileMetadata - (object, required): File metadata from google cloud storage

Upload file

Upload files from internal or external storage to Google Cloud Storage

Configuration Fields

Bucket - (dropdown, required): select one of the available buckets

Input Metadata

Url - (string, required): Link to the file, can be from the platform or external direct
File name - (string, required): How to name this file in Cloud Storage

Output Metadata

kind - (string, required): The kind of item this is. For objects, this is always “storage#object”.
id - (string, required): The ID of the object, including the bucket name, object name, and generation number.
selfLink - (string, required): A URL for this object.
mediaLink - (string, required): A URL for downloading the object’s data.
name - (string, required): The name of the object.
bucket - (string, required): The name of the bucket containing this object.
generation - (string, required): The content generation of this object. Used for object versioning.
metageneration - (string, required): The version of the metadata for this object at this generation. Used for preconditions and for detecting changes in metadata. A metageneration number is only meaningful in the context of a particular generation of a particular object.
contentType - (string, required): Content-Type of the object data
storageClass - (string, required): Storage class of the object.
size - (string, required): Content-Length of the data in bytes.
md5Hash - (string, required): MD5 hash of the data, encoded using base64
crc32c - (string, required): CRC32c checksum, as described in RFC 4960, Appendix B; encoded using base64 in big-endian byte order.
etag - (string, required): HTTP 1.1 Entity tag for the object
timeCreated - (string, required): The creation time of the object in RFC 3339 format.
updated - (string, required): The modification time of the object metadata in RFC 3339 format. Set initially to object creation time and then updated whenever any metadata of the object changes.
timeStorageClassUpdated - (string, required): The time at which the object’s storage class was last changed. When the object is initially created, it will be set to timeCreated.

Limitations

It is not possible to set the File/Object Metadata
Maximum file size: 5 TiB
If a file with provided name is already exist, it will be overwritten by new one

Make Raw Request

Executes custom request.

Configuration Fields

Don’t throw error on 404 Response - (optional, boolean): Treat 404 HTTP responses not as error, defaults to false.

Input Metadata

Url - (string, required): Path of the resource relative to the base URL.
Method - (string, required): HTTP verb to use in the request, one of GET, POST, PUT, PATCH, DELETE.
Request Body - (object, optional): Body of the request to send.

Output Metadata

Status Code - (number, required): HTTP status code of the response.
HTTP headers - (object, required): HTTP headers of the response.
Response Body - (object, optional): HTTP response body.