Ingest Slack channels data into Port via Fivetran, S3 and webhook

This guide will demonstrate how to ingest Slack channels and channel membership data into Port using Fivetran, S3 and a webhook integration.

Disclaimer

S3 integrations lack some of the features (such as reconciliation) found in Ocean or other Port integration solutions.

As a result, if a record ingested during the initial sync is later deleted in the data source, there’s no automatic mechanism to remove it from Port. The record simply won’t appear in future syncs, but it will remain in Port indefinitely.

If the data includes a flag for deleted records (e.g., is_deleted: "true"), you can configure a webhook delete operation in your webhook’s mapping configuration to remove these records from Port automatically.

Prerequisites

Ensure you have a Port account and have completed the onboarding process.
This feature is part of Port's limited-access offering. To obtain the required S3 bucket, please contact our team directly via chat, Slack, or e-mail, and we will create and manage the bucket on your behalf.

Important

In order to configure your Fivetran ARN Role, you will need to provide us the "External ID" that Fivetran will produce in the UI when configuring the S3 destinstaion. For that purpose, you can start creating the Destination, copy the External ID value, and Click on "Save for later". You can resume the Destination setup once the rest of the details will be provided by Port.
To set up multiple Destinsations with different S3 Prefix Path's for each data source (Fivetran does not allow multiple more than one prefix for the same Destination) - you will need to provide the each External ID to Port's representative to add it to the IAM Role.

Access to an available Fivetran Account (for the purpose of this guide, it could be completed within the free trial period)
Access to Slack account with privileges to approve Fivetran Slack app installation

Data model setup

Add Blueprints

Add the Slack Channel Membership blueprint:

Go to the Builder page of your portal.
Click on "+ Blueprint".
Click on the {...} button in the top right corner, and choose "Edit JSON".
Paste the following JSON schema into the editor:

Slack Channel Membership (Click to expand)

{
  "identifier": "slack_channel_membership",
  "description": "Slack Channel Membership",
  "title": "Slack Channel Membership",
  "icon": "Slack",
  "schema": {
    "properties": {
      "member_id": {
        "type": "string",
        "description": "ID of the user who is a member of the channel."
      },
      "channel_id": {
        "type": "string",
        "description": "ID of the channel the user belongs to."
      }
    },
    "required": [
      "member_id",
      "channel_id"
    ]
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {}
}

Add the Slack Channel blueprint in the same way:

Slack Channel (Click to expand)

{
  "identifier": "slack_channel",
  "description": "Slack Channel",
  "title": "Slack Channel",
  "icon": "Slack",
  "schema": {
    "properties": {
      "is_private": {
        "type": "boolean",
        "description": "Indicates if the channel is private."
      },
      "context_team_id": {
        "type": "string",
        "description": "ID of the team the channel belongs to."
      },
      "is_channel": {
        "type": "boolean",
        "description": "Indicates if this is a channel (true) or a direct message (false)."
      },
      "is_shared": {
        "type": "boolean",
        "description": "Indicates if the channel is shared across teams."
      },
      "previous_names": {
        "type": "array",
        "description": "List of previous names of the channel."
      },
      "creator": {
        "type": "string",
        "description": "ID of the user who created the channel."
      },
      "createdAt": {
        "type": "string",
        "description": "Timestamp of when the channel was created."
      },
      "is_ext_shared": {
        "type": "boolean",
        "description": "Indicates if the channel is externally shared."
      },
      "is_group": {
        "type": "boolean",
        "description": "Indicates if this is a group DM."
      },
      "is_archived": {
        "type": "boolean",
        "description": "Indicates if the channel is archived."
      },
      "shared_team_ids": {
        "type": "array",
        "description": "List of teams the channel is shared with."
      },
      "is_org_shared": {
        "type": "boolean",
        "description": "Indicates if the channel is shared across the entire organization."
      },
      "num_members": {
        "type": "number",
        "title": "num_members"
      },
      "purpose": {
        "type": "string",
        "description": "Information about the channel's purpose."
      },
      "topic": {
        "type": "string",
        "description": "Information about the channel's topic."
      }
    },
    "required": []
  },
  "mirrorProperties": {
    "member_id": {
      "title": "member_id",
      "path": "users.member_id"
    }
  },
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {
    "users": {
      "title": "Users",
      "target": "slack_channel_membership",
      "required": false,
      "many": true
    }
  }
}

Add the Slack User blueprint in the same way:

Slack User (Click to expand)

{
  "identifier": "slack_user",
  "description": "Slack User",
  "title": "Slack User",
  "icon": "Slack",
  "schema": {
    "properties": {
      "tz": {
        "type": "string",
        "description": "The user's time zone."
      },
      "is_restricted": {
        "type": "boolean",
        "description": "Indicates if the user is restricted."
      },
      "is_primary_owner": {
        "type": "boolean",
        "description": "Indicates if the user is the primary owner."
      },
      "real_name": {
        "type": "string",
        "description": "The user's real name."
      },
      "team_id": {
        "type": "string",
        "description": "The user's team ID."
      },
      "is_admin": {
        "type": "boolean",
        "description": "Indicates if the user is an admin."
      },
      "is_app_user": {
        "type": "boolean",
        "description": "Indicates if the user is an app user."
      },
      "deleted": {
        "type": "boolean",
        "description": "Indicates if the user is deleted."
      },
      "is_bot": {
        "type": "boolean",
        "description": "Indicates if the user is a bot."
      },
      "email": {
        "type": "string",
        "title": "email"
      }
    },
    "required": []
  },
  "mirrorProperties": {
    "channel_id": {
      "title": "channel_id",
      "path": "membership.channel_id"
    }
  },
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {
    "user": {
      "title": "User",
      "target": "_user",
      "required": false,
      "many": false
    },
    "membership": {
      "title": "membership",
      "target": "slack_channel_membership",
      "required": false,
      "many": true
    }
  }
}

Create Webhook Integration

Create a webhook integration to ingest the data into Port:

Go to the Data sources page of your portal.
Click on "+ Data source".
In the top selection bar, click on Webhook, then select Custom Integration.
Enter a name for your Integration (for example: "Hibob Integration"), enter a description (optional), then click on Next.
Copy the Webhook URL that was generated and include set up the airbyte connection (see Below).
Scroll down to the section titled "Map the data from the external system into Port" and paste the following mapping:

Slack Webhook Mapping (Click to expand)

[
  {
    "blueprint": "slack_channel",
    "operation": "create",
    "filter": "(.body | has(\"_PORT_SOURCE_OBJECT_KEY\")) and ( .body._PORT_SOURCE_OBJECT_KEY | split(\"/\") | .[3] | IN(\"channel\") )",
    "entity": {
      "identifier": ".body.id | tostring",
      "title": ".body.name_normalized | tostring",
      "properties": {
        "purpose": ".body.purpose_value",
        "is_private": ".body.is_private",
        "context_team_id": ".body.context_team_id",
        "is_shared": ".body.is_shared",
        "previous_names": ".body.previous_names",
        "creator": ".body.creator",
        "createdAt": ".body.created",
        "is_ext_shared": ".body.is_ext_shared",
        "is_group": ".body.is_group",
        "is_archived": ".body.is_archived",
        "num_members": ".body.num_members | tonumber? // .",
        "topic": ".body.topic.value",
        "shared_team_ids": ".body.shared_team_ids",
        "is_org_shared": ".body.is_org_shared"
      },
      "relations": {
        "users": {
          "combinator": "'and'",
          "rules": [
            {
              "property": "'channel_id'",
              "operator": "'='",
              "value": ".body.id | tostring"
            }
          ]
        }
      }
    }
  },
  {
    "blueprint": "slack_user",
    "operation": "create",
    "filter": "(.body | has(\"_PORT_SOURCE_OBJECT_KEY\")) and ( .body._PORT_SOURCE_OBJECT_KEY | split(\"/\") | .[3] | IN(\"users\") )",
    "entity": {
      "identifier": ".body.id | tostring",
      "title": ".body.name | tostring",
      "properties": {
        "tz": ".body.tz",
        "is_restricted": ".body.is_restricted",
        "is_primary_owner": ".body.is_primary_owner",
        "real_name": ".body.real_name",
        "team_id": ".body.team_id",
        "is_admin": ".body.is_admin",
        "is_app_user": ".body.is_app_user",
        "deleted": ".body.deleted",
        "is_bot": ".body.is_bot",
        "email": ".body.profile.email"
      },
      "relations": {
        "user": ".body.profile.email"
      }
    }
  },
  {
    "blueprint": "slack_channel_membership",
    "operation": "create",
    "filter": "(.body | has(\"_PORT_SOURCE_OBJECT_KEY\")) and ( .body._PORT_SOURCE_OBJECT_KEY | split(\"/\") | .[3] | IN(\"channel_member\")  )",
    "entity": {
      "identifier": ".body.channel_id + \"_\" + .body.user_id | tostring",
      "title": ".body.channel_id + \"_\" + .body.user_id | tostring",
      "properties": {
        "member_id": ".body.user_id",
        "channel_id": ".body.channel_id"
      }
    }
  }
]

Fivetran Setup

Set up S3 Destination

When using Fivetran with Port's S3-integration, you will have to set up S3 Data Lake Destination for each data source you woul like to integrate:

Login to your Fivetran account, and navigate to Destinations.
Click on Add Destination.
Locate the S3 Data Lake option under Fivetran Managed Data Lake Service, and click set up.
Give the destination a name, and Click Add.
Input the S3 Credentials provided to you by Port:
- Under Storage Provider select AWS.
- Under Bucket enter the bucket name (example: "org-xxx").
- Under Fivetran Role ARN enter your IAM Role ARN.
- Under S3 Bucket Path enter data/<Webhook URL> of the Webhook you have copied when setting up the webhook, for example: data/wSLvwtI1LFwQzXXX
- Under S3 Bucket Region enter the appropriate region.
- Keep Update AWS Glue Catalog toggled off. Data catalog updates are not supported.
- Keep Update Databricks Unity Catalog toggled off. Data catalog updates are not supported.
Click Save & Test and wait for Fivetran to confirm the Destination is set up correctly.

Set up Slack Connection

Follow Fivetran's guide to set up a Slack connector.

Private Channels
Fivetran will not read information from private channels by default. If you wish to include private channels: tick the "include private channels" option, and manually add the Slack-export App to your desired private channels.
For Destination, choose the S3 Destination you have set up in the previous step.
In the Schema tab, make sure only "channel", "users" are marked for synchronization.
Click Start Initial Sync to apply and start the Integration process!

Important

If for any reason you have entered different values than the ones specified in this guide, inform us so we can assist to ensure the integration will run smoothly.

Prerequisites​

Data model setup​

Add Blueprints​

Create Webhook Integration​

Fivetran Setup​

Set up S3 Destination​

Set up Slack Connection​

Additional relevant guides​