Skip to main content

Set Up a Product Collector

Updated this week

To create personalized product recommendations in BlueConic dialogues, you start by collecting product data using the BlueConic Product Collector Connection. This connection collects data about your products and stores it in a BlueConic product store to be used for personalization in BlueConic.

Note about personalization:

If you have a limited number of products in your catalog, you may prefer to upload product information using a flat file rather than this connection.

Make smarter personalized product recommendations with BlueConic

You can use this information to make smarter product recommendations in dialogues. Deliver one-to-one product recommendations using toolbar plugins that use filters and product recommendation algorithms to choose the optimal products to recommend based on individual customers' behaviors, interests, preferences, and previous purchases.

Use personalization to target customers with abandoned shopping carts

By detecting the number of items in the customer's shopping cart and the last update, you can target customers who have abandoned shopping carts to complete their purchases. Please note that this is a relatively complex use case, and you can reach out to the BlueConic Customer Support team if you need assistance.

How to use the BlueConic product store to create personalized product recommendations in BlueConic for individualized marketing and personalization

Note: The Product Collector needs to be active on the page that fires the `shoppingcart` or `order` event on your site, such as the shopping cart or order page (not just on the product pages).

Configure the Product Collector to collect items for recommendations

To create a Product Collector connection, select Connections from the main BlueConic menu and click the Add connection button. Search for 'product collector' and create the connection. When the Product Collector connection appears, provide a name at the top of the page.

How to collect product data in a product store to feed BlueConic's product recommendation algorithms for personalization

To set up the connection, select "Collect data from your channels" in the lefthand panel.

  1. Select the channel(s) to collect product data from.
    Optionally, define URL rules to specify which areas of the sites to collect product data from.

  2. Next, paste a product URL into the "Test URL" area, and click 'Test' to review the metadata that would be collected for your product:

    How to review metadata for BlueConic product recommendations and personalized marketing


    BlueConic requires a number of metadata fields out of the box. You can select the checkbox next to other metadata fields to mark them as also required.

    Default required fields:

    • ID, Name, URL, and Type are all required by default.

    • If any of these are not scraped via the configured selectors, the item will not be added to the Product Store.

    • For the Product Collector the page type must equal product.

    If required metadata fields are not populated, the webpage will no be scrapped by the content collector. Make sure that the information on the page is in a supported format. The default BlueConic selector will automatically detect:

    To add custom metadata fields, select Add data field. For example, if your products have a typical gender association, or are tagged based on associated brands, influencers, or sources, you can use this data in recommendations placement filtering.

  3. Some algorithms are based on a look-back time frame, for example the "top products" algorithm.
    You can configure this time frame based on hours or days here.

    How to use product recommendations powered by Machine Learning and AI for personalization in BlueConic
  4. Optional: Add request headers, to be able to scrape content for webpages that are only available after login, after a paywall, or on development sites.

  5. Detect products in the shopping cart, order data, and last time the cart was updated.
    A. Add shopping cart data to profile properties.
    Select profile properties to store the number of unique products in the customer's shopping cart. You can choose an existing profile property or create one here.

    How to use shopping cart data to feed personalized product recommendations in BlueConic using the product store


    B. Implement the JavaScript shopping cart event.
    Add the following JavaScript code to your website to let BlueConic know which products the customer adds to or removes from the cart, and when the cart is emptied. Use this data in dialogues for recommendations.

    // Replace the contents of the productIds array
    // with the IDs of all products that are currently in the shopping cart.

    blueConicClient.event.publish("shoppingcart", {
    productIds: ["id1", "id2", "id3"]
    });

    C. Implement the JavaScript order event.
    Add the following JavaScript code to your website to let BlueConic know which products the customer ordered. Use this data in dialogues to improve recommendations, using the "bought items" algorithm and collaborative filters and algorithms for order data.

    // Replace the contents of the productsIds array
        // with the IDs of all products the customer just ordered.
       
        blueConicClient.event.publish("order", {
        productIds: ["id1", "id2", "id3"]
        });

    Note: For both JavaScript options B and C, the product IDs in the event should exist in the content store. The configuration of the "ID" field should match the ID that is sent in the event.

    Learn more about BlueConic recommendation algorithms for personalization

    For details on the recommendation algorithms that power personalized product recommendations in BlueConic dialogues, see BlueConic recommendation algorithms.


FAQs

How can I test which items or products are in the collector?

In Step 2 of the Product Collector, you can test to see which values are being returned. Paste an item's URL into the Test URL field, click 'Test' to review the metadata that would be collected for its content, and then confirm whether all 'Required' metadata fields are returning values.

If values are not being returned for required fields, then items are not being collected successfully.

What happens if I change my product collector settings?

Changing or updating the selector field for your product collector could result in unwanted data duplication. When collecting products, the product collector checks the product store for a matching ID. If it finds a matching ID, it will update the existing item in the product store with the new metadata. For this reason, we do not recommend changing the ID selector on your product collector, as this can lead to duplicate items being created in your product store.

Why is my Product Collector not scraping my data successfully?

Product Collectors can only retrieve data that is present in the original page source (the raw HTML returned by the server). If data is visible when using Inspect but is not included in the page source, it cannot be collected by the Product Collector.

To check what’s actually available to the collector, right-click the page and choose View Page Source. Only content you see there is eligible for collection.

Important limitations

  • Anything inside <style> or <script> tags is stripped by the collector.

  • We cannot scrape metadata from inline JavaScript objects (e.g., variables embedded in <script>), unless the data is provided in JSON-LD format.

  • Data that renders client-side (after the initial HTML is loaded) is not available to the collector unless it is also present in the original page source.

Why is a specific page not getting collected?

If a specific page isn’t being collected by the Content Collector, it’s usually due to a missing or incorrect required field (for example, the page “type” not matching the collector’s expected value or is missing).

How to diagnose

  1. Install and enable the BlueConic Chrome extension.

  2. Open the page you expect the collector to scrape.

  3. Open the browser console.

  4. Look for BlueConic messages. If the collector is skipping the page because of a required field, you’ll see a message similar to:

    [BC] Item will not be scraped due to incorrect type:: !== product

    This indicates the page’s type does not match the collector’s requirement (in this example, the collector expects product).

Why aren’t my images rendering in recommendations?

BlueConic currently supports PNG and JPG/JPEG images for the image attribute.
If an image is served in another format (for example WebP, AVIF, SVG, GIF), the Product recommendations will not be able to render it.

Avoid dynamic “auto-format” transformations

Many CDNs and image services change the file type on the fly based on the user agent or query parameters. Examples include parameters like:

  • auto=format, f=auto, fm=auto, format=auto, tr=f-auto

These can cause the image URL to return WebP/AVIF to BlueConic, resulting in file-type mismatches and missing images.

Did this answer your question?