Retrieving changes from Salesforce (flow in)

A core feature of OrgFlow is the ability to identify metadata changes in a Salesforce org, retrieve them and commit them to a Git repository. We call this process flow in or sometimes inbound flow. Together, flow in and flow out are key processes that help keep a Salesforce org in sync with a Git branch.

Retrieving metadata

OrgFlow uses Salesforce's metadata API to retrieve metadata from a Salesforce org. There are two retrieval modes — full and partial — and both modes support automatic change detection. By default, OrgFlow will use partial retrieve when it can, falling back to full retrieve where necessary. This behavior can be changed if required.

Full

A full retrieve involves retrieving all of the metadata that is included by the include specs.

At a high level, the process looks like this:

1. OrgFlow determines the metadata types that are included by the include specs
2. OrgFlow asks Salesforce for a list of all the metadata items of those types
3. All of the metadata items that match the include specs are retrieved from Salesforce

This is the most thorough and well supported retrieval mode (because it doesn't require source tracking), but it can be substantially slower than a partial retrieve due to the potentially large volume of metadata that is retrieved.

Partial (source tracking)

Partial retrieves utilize Salesforce's source tracking functionality to identify which metadata items have changed since the last flow in. This can result in only a small subset of metadata needing to be retrieved, greatly reducing the time it takes for the retrieve to run.

At a high level, the process looks like this:

1. OrgFlow maintains a baseline source tracking version number for each source tracked environment
2. OrgFlow asks Salesforce for all the source tracking records since the baseline version
3. Those source tracking records are filtered to include only the metadata items that are included by the include specs
4. Those metadata items are retrieved from Salesforce using the metadata API

However - partial retrieves may not always be possible, and there are some known issues with source tracking that may mean that metadata changes in the Salesforce org are incorrectly omitted from Salesforce's source tracking.

OrgFlow is able to automatically detect whether or not a partial retrieve is possible and fall back to a full retrieve in that case. For a partial retrieve to be possible, all of the following criteria must be met:

• Source tracking must be enabled in the Salesforce org
• OrgFlow must have previously done a full retrieve of the environment in order to establish a baseline
• The include specs must not have changed since the last flow in
• Force mode must not be enabled
• The environment must have been flowed in in the past 30 days
• There must not have been an external deployment since the last flow in of the environment

An external deployment is a deployment that has been done with a tool other than OrgFlow, for example using changesets. External deployments may not update source tracking records for the metadata items that they touch, so OrgFlow cannot be confident that the source tracking history is complete if there has been an external deployment since the last flow in.

Support for non-source tracked types

Some metadata types do not support source tracking; OrgFlow is still able to support these types during a partial retrieve.

It does this by effectively doing a full retrieve of the types that are not supported by source tracking.

The types that do not support source tracking varies with each Salesforce API version and can be seen in Salesforce's metadata coverage report. Because these types vary between Salesforce API versions, the types affected in each environment will be impacted by the API version selection mode that you have chosen for the environment.

Known issues with source tracking

OrgFlow's partial retrieve functionality is powered by Salesforce's source tracking functionality, and it is only as good as Salesforce's ability to accurately maintain these source tracking records.

Unfortunately there are some known issues with Salesforce that result in some changes not being reflected in Salesforce's source tracking. One example of this is deleting a custom field from a custom object. In this case, Salesforce should create a source tracking record to indicate that the field is deleted, but instead it simply does not create the record. If you encounter this issue, you should raise a case with Salesforce as we are not in a position do anything about it.

This same issue will impact any tool that uses source tracking, including the Salesforce CLI.

For this reason we suggest that you run flow in with full retrieve enabled from time to time in order to catch any changes that source tracking may have missed.

Batching

Regardless of retrieval mode, OrgFlow will usually split the metadata to be retrieved into several retrieval batches. There are two main advantages to doing this:

• It's usually quicker to retrieve several smaller batches in parallel from the Salesforce metadata API than it is to retrieve one large metadata archive
• The metadata API imposes limits on the number of items and the total overall download size per request, so splitting the request into several smaller batches allows OrgFlow to retrieve more than 10,000 metadata items at once.

When deciding how to split the metadata to be retrieved into different retrieval batches, OrgFlow must take into account several factors. Firstly, OrgFlow will determine an optimal batch size and from that the number of retrieval batches to create. Then, OrgFlow must identify metadata items that need to be retrieved in the same batch. This is a crucial step because Salesforce's metadata API can return inconsistent results for some metadata types if those types take dependencies on other types or items that are not included in the same batch.

The result of these batching requirements is that you may see some retrieval batches that are larger or smaller than others during a retrieve. This is normal behavior and is to be expected.

There are some exceptions where multiple batches will not be employed:

• If the number of metadata items to be retrieved is very small and creating more than one batch would be inefficient
• If all of the metadata items to be retrieved must be retrieved in the same batch because of the limitations with Salesforce's metadata API

Automatic change detection

OrgFlow is capable of automatically detecting metadata changes in a Salesforce org. After retrieving metadata from the Salesforce org, OrgFlow compares it to the metadata in the Git repository (at the correct point in time), and identifies the components that have changed.

This approach works regardless of retrieval mode, which means that you no longer need to worry about remembering which components need to be added to a deployment — even for orgs that do not have source tracking enabled (such as a production org).

Committing changes

Once the metadata has been retrieved, OrgFlow will identify any changes, commit them, and then push those commits back up to the remote repository (unless you're using check-only mode).

Merging changes

Part of the magic of OrgFlow is its ability to merge changes in a Git repository with the changes in a Salesforce org, without one side overwriting the other.

For example, if the help text of a custom field has been changed in Git, and someone else changes the description of the same field in the Org, the flow in process will merge those two changes and neither change will be overwritten by the other. This is one of the fundamental pieces that enable OrgFlow to merge changes between environments without clobbering changes in one org with the changes from another org.

See understanding metadata version control for more in-depth information on how OrgFlow utilizes Git to achieve this and other benefits.

To enable this, OrgFlow maintains some state information about each included metadata item in an environment. The state information is stored in OrgFlow's cloud. Part of this state information includes commit hashes that OrgFlow can use to compare the current state of the metadata item in the Git repository to the state of the metadata item in the Salesforce org. This is what enables automatic change detection, and also allows the Git repository to be ahead of the Salesforce org without risking data loss during a flow in.

What's stored in OrgFlow's cloud?

The state information that is stored in OrgFlow's cloud does not include the metadata items themselves, but it does include the names and types of the metadata items, commit hashes, and source tracking version numbers.

This feature can be bypassed by enabling force mode.

Author attribution

When committing metadata to Git, OrgFlow attempts to identify the Salesforce user who last changed each metadata item that is to be committed. A combination of source tracking data (if available) and the last modified by property on the metadata item is used to determine this.

OrgFlow will then commit changes by each author separately, assigning the identified Salesforce user as the author of each commit. The Name and Email fields from the User object in Salesforce are used to create the Git author string (i.e. user.Name <user.Email>).

Author attribution helps to build up a rich history in Git, recording a name on each change to the Salesforce metadata.

Attributing multiple changes

Author attribution is only able to record a single author per component and flow in. If two users have changed the same metadata item since the last flow in, then both changes will be attributed to the user who made their change last.

Occasionally it won't be possible to attribute an author to a change (for example when a metadata item has been deleted and source tracking is not available for the org). In these cases the author will default to Unknown Author <unknownauthor@orgflow.io>.

Force mode

Force mode resets the metadata in the Git repository to match the state of the metadata in the Salesforce org.

It does this by bypassing the branching strategies used to support merging changes. The metadata that is retrieved is applied to and committed onto the head of the environment's branch. Author attribution is still supported in force mode.

Be careful with undeployable components

Force mode will reset everything in the Git branch to match the state of the Salesforce org. This means that undeployable components will also be reset.

Undeployable components will almost certainly be ahead in the Git repository compared to the Salesforce org. Resetting these components to match the org versions will revert these changes.

Check-only mode

You can run a flow in with check-only mode. In this mode, the state in OrgFlow's cloud is unchanged and the committed changes are never pushed to the remote repository. The result is a dry run that does not impact the state of the environment or Git repository in any way, but still allows you to inspect the would-be result.