Labeling Pipelines for Ingest, Transform, and Load

C

Annotations are additional, informative tags that you can add to specific factory resources: pipelines, datasets, linked services, and triggers. By adding annotations, you can easily filter and search for specific factory resources.

To label each pipeline in Azure Data Factory with its main purpose of either ingest, transform, or load, you should add a user property to each pipeline.

User properties are custom name/value pairs that you can add to resources in Azure Data Factory. By adding a user property to each pipeline, you can provide metadata that describes the purpose of the pipeline.

To add a user property to a pipeline in Azure Data Factory, follow these steps:

1. Open the Azure Data Factory UI and navigate to the Author & Monitor section.
2. Click on the pipeline that you want to label.
3. In the pipeline details pane, click on the "Properties" tab.
4. Scroll down to the "User Properties" section and click the "Add Property" button.
5. Enter a name for the property, such as "Purpose".
6. Enter a value for the property, such as "Ingest", "Transform", or "Load".
7. Save the pipeline.

Once you have added a user property to each pipeline, you can use the monitoring experience in Azure Data Factory to group and filter the pipelines based on their purpose. To do this, follow these steps:

1. Navigate to the Monitor section in Azure Data Factory.
2. Click on the "Pipeline runs" tab.
3. Click the "Filter" button.
4. In the filter pane, click the "Add filter" button.
5. Choose the "User property" filter type.
6. Select the user property that you added to label the pipelines (e.g. "Purpose").
7. Choose the value that corresponds to the purpose of the pipelines that you want to view (e.g. "Ingest").
8. Click the "Apply" button to filter the list of pipeline runs.

In summary, to label each pipeline in Azure Data Factory with its main purpose of either ingest, transform, or load, you should add a user property to each pipeline. User properties are custom name/value pairs that you can use to provide metadata about resources in Azure Data Factory. Once you have added user properties to your pipelines, you can use them to group and filter pipelines in the monitoring experience in Azure Data Factory.