Cloud Function Sync protocol

The GCP integration supports syncing resources from any HTTP endpoint implementing the Fivetran Cloud Function Sync protocol. With this approach, you can ingest data from external systems (such as employee records, DWH data, custom metrics, etc.) served by a Cloud Run service.

Overview

Cloud Function Sync lets you extend the GCP integration to pull data from HTTP endpoints that implement a simple request-response protocol. This is useful when your data lives outside GCP or in systems not supported by the Cloud Asset Inventory API.

Use cases

• Sync employee/HR data from internal services.
• Ingest DWH records (customers, transactions, products).
• Fetch custom metrics from metrics services.
• Sync data from SaaS platforms via Cloud Run proxy.

Configuration

Example blueprint

Below is an example blueprint definition for an employee resource type. You can adapt this structure to suit your own use case:

Example employee blueprint

{
  "identifier": "ACMEEmployee",
  "description": "ACME Employee",
  "title": "ACME Employee",
  "icon": "Hibob",
  "schema": {
    "properties": {
      "employeeId": {
        "title": "Employee ID",
        "type": "number"
      },
      "tribe": {
        "title": "Tribe",
        "type": "string"
      },
      "tribeOneLiner": {
        "title": "Tribe Description",
        "type": "string"
      },
      "squad": {
        "title": "Squad",
        "type": "string"
      },
      "pod": {
        "title": "Pod",
        "type": "string"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {}
}

Example mapping configuration

Here is a sample mapping you can add to your GCP integration configuration. Update the details as needed for your data source and blueprint:

Sample GCP integration resource mapping

resources:
  - kind: gcpCloudFunction
    selector:
      query: 'true'
      functionUrl: http://localhost:9000
      resource: employees
    port:
      entity:
        mappings:
          identifier: .email
          blueprint: '"ACMEEmployee"'
          title: .email
          properties:
            employeeId: .employee_id
            tribe: .tribe
            tribeOneLiner: .tribe_one_liner
            squad: .squad
            pod: .pod
createMissingRelatedEntities: false
deleteDependentEntities: true

Example templates

The blueprint and mapping shown above are examples to help you get started. You should replace the property names, identifiers, and URLs with values matching your own data structures and system.