Full resync

A full resync is the default sync mechanism, and the one referred to as "periodic sync" in the installation methods comparison table. On each run, the integration fetches all data available to it from the third-party source, maps it according to your mapping configuration, and ingests it into Port.

• The first sync after installing an integration is always a full resync.
• It runs on the interval configured for your installation method (user-selectable for hosted integrations, custom for self-hosted and CI), or when triggered manually from the data sources page or the resync API endpoint.
• It includes a reconciliation phase: entities that are no longer present in the data source are deleted, and entities that were waiting on related entities are reingested.
• Full resync progress, per-kind status, and reconciliation results are visible in the monitoring and sync status section below.

Monitoring and sync status

Every integration has a dedicated page that displays detailed monitoring metrics and sync status, helping you track the data ingestion process. This monitoring view is available for all Ocean integrations.

To access the integration's monitoring metrics and sync status, navigate to the data sources page. Under the Exporters section, select your integration.

The Sync Status tab offers a detailed overview of the data ingestion and reconciliation processes. It updates on initial installation, manual resyncs, and at scheduled intervals. Note that it does not reflect changes from live events.

It includes the following sections:

Sync by kind

A kind represents a specific type of resource or entity ingested by the integration. For example, if you are integrating with GitHub, a kind could be a repository, a pull request, or a commit.

This section shows the monitoring metrics and sync status for each kind.

When triggering a sync, the Save & resync button offers two options:

• Run full sync: Runs all three stages below - fetches fresh data from the source, maps it, and ingests it into Port. The first sync is always a full resync.
• Apply mapping: Skips the fetch stage and reruns only the mapping and ingestion using already-fetched data. This is useful after editing your mapping configuration - if you don't need fresh data from the source, apply mapping is much faster since it reuses existing data rather than re-fetching everything.

Sync stages overview:

• Fetch data: The first step in the data ingestion process is fetching the data from the data source. This stage only runs during a Run full sync and is skipped when using Apply mapping. In this section, you will find the following fields:
  • Fetched: Number of objects fetched from the data source.
  • Failed: If the fetch fails, this field will show 1 and Fetched will show 0. Otherwise, it shows 0.
• Map data: The next step is mapping these objects according to the integration's mapping configuration. In this section, you will find the following fields:
  • Transformed: Number of objects that were mapped to blueprint entities.
  • Filtered out: Number of objects that were filtered out according to the mapping conditions.
  • Failed: Number of objects that failed to map, you can find more information about the failure reasons in the Troubleshooting tips section below in the UI or refer to the Audit log or Event log tabs, available in the same window.
• Ingest to Port: Final step is ingesting the mapped objects to Port. In this section, you will find the following fields:
  • Ingested: Number of objects that were inserted or updated successfully.
  • Not changed: Number of objects that were already up to date, so no changes were made to them.
  • Failed: Number of objects that failed to ingest. To learn more about the failure reasons, see the Audit log or Event log tabs, available in the same window.

Details: Displays the duration of the sync.

Kind status

While a resync is in progress, each kind can show the following in-progress statuses:

• Pending: The kind is waiting to start.
• Syncing: The kind is currently being processed.

After the ingestion process finishes for each kind, Port assigns a final status based on the metrics from each sync stage. Available final statuses: Aborted, Failed, Synced with issues, and Synced.

A kind is marked as Failed when any of the following apply:

• Could not fetch data: The fetch step reported failures and no objects were fetched.
• All objects failed to map: At least one object was fetched, every fetched object failed to map, and none were transformed.
• Failed to ingest all mappable objects: At least one object was fetched, not all objects were filtered out, not all objects failed to map, at least one object was transformed, and none were ingested while ingest reported failures.

A kind is marked as Synced with issues when it is not Failed, and at least one sync stage reported failures.

A kind is marked as Synced when it is not Failed and no sync stage reported failures.

A kind is marked as Aborted when:

• A newer resync starts before the current sync completes for that kind.
• Metrics for the kind are not available (for example, the sync did not complete).

Reconciliation

This section provides visibility into both the reingestion and deletion processes.

When the createMissingRelatedEntities flag is set to false, Port will not automatically create related entities that are referenced in the mapping but not yet present in Port. The reingestion process ensures that once the missing related entities are ingested, the dependent entities will be reingested.

The deletion process ensures that entities that are no longer present in the data source are removed.

In this section, you will find the following data:

Ingestion retry:

• Reingested entities: Displays the number of entities that were ingested successfully on retry after dependencies resolved.
• Failed: Displays the number of entities that failed to reingest.

Delete:

• Deleted entities: Displays the number of entities that were removed from Port to match the data source.
• Failed: Displays the number of entities that failed to delete.

Details: Displays the duration of the reconciliation.

Reconciliation status

While a resync is in progress, reconciliation can show the following in-progress statuses:

• Pending: Reconciliation is waiting to start. This happens when one or more kind ingestion processes have not finished yet.
• Syncing: Reconciliation is currently running.

After the reingestion and deletion processes finish, Port assigns a final reconciliation status based on the metrics from both stages and the status of each kind. Available final statuses: Aborted, Failed, Synced with issues, and Synced.

Reconciliation is marked as Aborted when at least one kind is Failed.

Reconciliation is marked as Failed when both stages fully failed:

• Ingestion retry fully failed: At least one entity failed to reingest and none were reingested successfully.
• Delete fully failed: At least one entity failed to delete and none were deleted successfully.

Reconciliation is marked as Synced with issues when it is not Aborted or Failed, and any of the following apply:

• One stage fully succeeded, the other did not: One stage completed with no failures and at least one success, while the other stage either fully failed or partially succeeded.
• Both stages partially succeeded: Both Ingestion retry and Delete reported failures and successes.

Reconciliation is marked as Synced when it is not Aborted, Failed, or Synced with issues. This includes cases where both stages completed with no failures, or when a stage had no work to perform.

Sync status

While a sync is in progress, the integration shows the following in-progress statuses:

• Syncing: The sync is currently running.

After all kind ingestion processes and reconciliation finish, Port calculates a final overall resync status for the integration from the status of each kind and the reconciliation status. Available final statuses: Aborted, Failed, Synced with issues, and Synced.

The resync is marked as Failed when every kind is Failed.

The resync is marked as Synced when every kind is Synced and reconciliation is Synced.

The resync is marked as Synced with issues when at least one kind is Failed or Synced with issues, or when reconciliation is Failed or Synced with issues.

The resync is marked as Aborted when none of the above conditions apply. For example, when one or more kind statuses are Aborted and no kind or reconciliation issue triggered Synced with issues.

Note: The same data is also available through the Get an integration's metrics and sync status API endpoint.

Prometheus Metrics Endpoint

If you are using the self-hosted integration method, you can get raw Prometheus metrics by accessing the following endpoint: {your_integration's_base_url}/metrics/.

Understanding the metrics (click to expand)

Types of Metrics Available

• duration_seconds: Measures how long it takes for a phase to complete. Labels: kind, phase
• object_count: Counts the number of objects handled in each phase. Labels: kind, phase, object_count_type
• success : Indicates whether the phase completed successfully. Labels: kind, phase

Types of Labels Available

• kind: The specific type of resource handled by the integration (varies per integration).
• phase: The step in the ETL process:
  • extract: Fetch data from the source.
  • transform: Map or modify data before loading.
  • load: Ingest the data into Port.
  • resync: Full ETL process for a specific kind.
  • delete: Part of the reconciliation phase.
• object_count_type: Subtype of object count, which varies by phase:
  • For transform: transformed, filtered_out, failed.
  • For load: loaded, failed, skipped.
  • For extract: raw_extracted, failed.
  • For deletion: deleted.

FAQ