http_request operation

The http_request operation executes HTTP requests integrated with data sources. While sources are typically databases, they can be any source for which we have a built-in connector, such as file-based sources (e.g. Amazon S3).

The http_request operation consists of three main section:

• for_each - to iterate over an input table
• request - HTTP call configuration
• response - to process the HTTP response

for_each section

The for_each section is optional. It allows you to specify an input table. http_request will iterate over the table executing the request and response sections for each input row.

The for_each section has the following parameters:

• source (type: string, required) - source name
• database (type: string, optional) - database name
• namespace (type: string, optional) - namespace name
• table (type: string, required) - table name
• as (type: string, required) - variable name for a local variable that will be set to the current row. You can access this variable from Jinja templates and Python functions in the request and response sections.

The value of the variable specified in the as parameter is a Row object that provides dictionary-style access to column values. The following methods are available:

request section

The request section contains HTTP call configuration with the following parameters:

• source (type: str, optional) - name of a source of the http type. The source defines authentication parameters and source-level variables that can be access from all sections of the http_request operation. For more information on the http source, see the http source section.
• url (type: str, required) - URL to call.
• method (type: str, required) - HTTP method. Values are: GET, POST, DELETE, or PATCH.
• parameters (type: Dict[str, str], optional) - URL parameters.
• header (type: Dict[str, str], optional) - HTTP header
• body_format (type: str, required if body is present) - defines the expected type of the body parameter value and the format to which to convert the body value. Values are:
  • json - the body parameter must be dictionary that will be converted to JSON format
  • form_urlencoded - the body parameter must be a dictionary that will be encoded in URL-encoded format. Note: you must set the Content-Type: application/x-www-form-urlencoded header yourself.

response section

The response section defines how to process the HTTP response:

• parser_expression (type: str, required) - a Python function evaluate that takes two parameters:
  • context - provides access to variables and other runtime information
  • response - contains response information. For information method of the context object, see the Python functions subsection in Dynamic parametrization. The response object has the following methods and properties:
  • status_code (type: int) - HTTP status of the response (e.g. 200 or 404)
  • headers (type: Dict[str, str]) - case-insensitive dictionary of the response headers. For example, headers[‘content-encoding’] will return the value of a ‘Content-Encoding’ response header.
  • text (type: str) - content of the response, in unicode.
  • json() - parses the response body as JSON and returns the corresponding Python data structure (dict, list, etc).

Examples

This example demonstrates most parameters described of the http_request operations. It creates customers in a Bigcommerce e-commerce system from an input table by executing a POST API call for each customer and parses the response of each call to store the newly generated customer IDs. In the response parsing, it stores the ID from the input row to preserve lineage information.

- op: http_request
    id: create_customers
    for_each:
      source: "stage"
      table: "bc_customers_to_insert"
      as: customer
    request:
      source: "bigcommerce_rest"
      url: "https://api.bigcommerce.com/stores/{{ var('store_hash') }}/{{ var('api_version') }}/customers"
      method: POST
      headers:
        "Content-Type": "application/json"
      body_format: json
      body:
        - "first_name": "{{ var('customer').get('first_name', '__NULL__') }}"
          "last_name": "{{ var('customer').get('last_name', '__NULL__') }}"
          "email": "{{ var('customer').get('email', '__NULL__') }}"
      # Equivalent version using body_expression
      # body_expression: |
      #   def evaluate(context):
      #     return [{
      #       "first_name": context.var("customer").get("first_name"),
      #       "last_name": context.var("customer").get("last_name"),
      #       "email": context.var("customer").get("email")
      #     }]          
    response:
      parser_expression: |
        def evaluate(context, response):
          if response.status_code != 200:
            raise Exception(f"Error status code: {response.status_code}, body: {response.text}")
          customers_created = response.json()['data']
          customers_created['source_id'] = context.var("customer").get("id")
          return {
            "tables": [{  
              "source": "stage",
              "table": "bc_customers_inserted",
              "data": customers_created,  
              "model": {
                "columns": [
                {"name": "id", "type": "text"},
                {"name": "source_id", "type": "text"},
                {"name": "first_name", "type": "text"},
                {"name": "last_name", "type": "text"},
                {"name": "email", "type": "text"}
                ]
              }
            }]
          } 

You can find more examples of HTTP request operations for various systems and use cases in the Sequor integrations  GitHub repository.