Loop54

Developer guide - Generating the product feed

Introduction

The Loop54 search engine learns about your products from the product feed. The better the product feed, the better the search results, hence the quality of the product feed is paramount.

Loop54 needs to have the entire product catalogue, including all relevant metadata, indexed. Submitting your product data to Loop54 in the right format is important for creating a successful and relevant search and category navigation experience for your customers.

Why is it Needed?

Loop54 uses the product metadata to:

  • make products searchable
  • build relationships between products
  • incorporate business logic into the search and navigation experience
  • enable filtering logic
  • build faceted filters

Optional: many of our customers use Loop54 as their data source for rendering all information displayed on the search results page, product pages, etc. For this reason, they include more information in the product feed than Loop54 requires.

Examples

Below are examples of typical product feeds that contain all required product attributes.

The product feed simply contains a list of objects each representing a product. The object can be as simple as JSON and XML examples shown here.

{
"Products" : [
{
    "Id" : 123456,
    "GroupId" : 1000,
    "SKU" : "N0507",
    "Name" : "Trek shorts",
    "Description" : "A perfect pair of shorts, for the summer jungle trek.",
    "Brand" : "Markys",
    "ProductURL" : "https://www.asiteaddress.com/products/product123456",
    "ImageURL" : "https://www.asiteaddress.com/images/products/123456.jpg",
    "Tags" : ["short", "pants", "khaki"],
    "CategoryPath" : [["Clothes", "Pants", "Shorts"], ["Swimwear", "Swimming Trunks"]],
    "Price" : 100.1,
    "CampaignPrice" : null,
    "Campaign" : false,
    "NewUntil" : "2019-01-30",
    "InStock" : false,
    "Attributes" : {
        "Color" : ["Green", "Black"],
        "Size" : [32, 34, 36, 38],
        "Gender" : "Female"
    },
    "HiddenKeywords" : ["trekking", "hiking", "trunks"]
},
{...}]
}
<?xml version="1.0" encoding="UTF-8"?>
<root>
    <Product>
       <Attributes>
          <Color>
             <element><![CDATA[Green]]></element>
             <element><![CDATA[Black]]></element>
          </Color>
          <Gender><![CDATA[Female]]></Gender>
          <Size>
             <element>32</element>
             <element>34</element>
             <element>36</element>
             <element>38</element>
          </Size>
       </Attributes>
       <Brand><![CDATA[Markys]]></Brand>
       <Campaign>false</Campaign>
       <CampaignPrice null="true" />
       <CategoryPaths>
          <element>
             <element><![CDATA[Clothes]]></element>
             <element><![CDATA[Pants]]></element>
             <element><![CDATA[Shorts]]></element>
          </element>
          <element>
             <element><![CDATA[Swimwear]]></element>
             <element><![CDATA[Swimming Trunks]]></element>
          </element>
       </CategoryPaths>
       <Description><![CDATA[A perfect pair of shorts, for the summer jungle trek.]]></Description>
       <GroupId>1000</GroupId>
       <Id>123456</Id>
       <ImageURL><![CDATA[https://www.asiteaddress.com/images/products/123456.jpg]]></ImageURL>
       <InStock>false</InStock>
       <Name><![CDATA[Trek shorts]]></Name>
       <NewUntil><![CDATA[2019-01-30]]></New>
       <Price>100.1</Price>
       <ProductURL><![CDATA[https://www.asiteaddress.com/products/product123456]]></ProductURL>
       <SKU><![CDATA[N0507]]></SKU>
       <Tags>
          <element><![CDATA[short]]></element>
          <element><![CDATA[pants]]></element>
          <element><![CDATA[khaki]]></element>
       </Tags>
       <HiddenKeywords>
          <element><![CDATA[trekking]]></element>
          <element><![CDATA[hiking]]></element>
          <element><![CDATA[trunks]]></element>
       </HiddenKeywords>  
    </Product>
    <Product>
    ...
    </Product>
</root>

Definitions

Definition of terms you will see throughout this document:

  • Product
    Word that describes the actual product that your potential customers would be searching for on your website.
  • Attribute
    Anything that stores data on the product, not just distinct data. Often characteristics of a product which make it distinct from other products. Attributes include size, color, functionality, components and features that affect the product’s appeal or acceptance in the market.
  • Variant
    Word that describes a product that comes in different variations. For example, a shirt that comes in different sizes has size variants.
  • Entity
    Entities are what the engine uses to store products, categories, brands, etc. Loop54 will save all structural attributes as entities.

The Feed Specification

The feed is simply a list of all your products including all relevant attributes. Some of the attributes are required which means that they must be included for the search engine to work.

The list of products can be generated in common markup languages such as JSON or XML.

Product Attributes

The product information you submit using these attributes is the foundation we use to create your search engine. Please make sure everything you submit is of the quality you would show to a customer.

If you don’t have a relevant value for a product attribute then leave it empty, do not provide a generic value like: N/A, Generic, No brand, or Does not exist.

The sections below describe what product attributes are required, recommended and optional.

Required Product Attributes

There are five required product attributes: identifier, title, brand, image link and category. They are explained in detail here.

  • Product identifier
    Your product’s unique identifier. For example, you can use your SKU numbers or the ID generated by your store’s e-commerce platform. If you’re setting up feeds for the same country in multiple languages, you must use the same product ID for the same item in different languages.
    • Use a unique value for each product. Use the product’s SKU where possible. The ID for the same product must stay the same even across countries or languages.
    • Keep the ID the same when updating your data
    • Use only valid UTF-8 characters. Avoid invalid characters like control, function, or private area characters
    • Only one field will be used to uniquely identify the product in the Loop54 engine, however we can index any amount of ID fields, so it is fine, and even encouraged to supply several. If there is an Article number, SKU, EAN and product Id for example, we would take all of them, but only choose one to use as the ID of the product.
  • Title
    Your product’s name. Accurately describe your product and match the title from your landing page.
    • Don’t include promotional text like “free shipping”, “Limited offer”, all capital letters, or gimmicky foreign characters (emojis).
    • For variants, include a distinguishing feature such as color or size. If possible, avoid HTML or other formatting tokens - the engine will strip them before indexing.
  • Brand
    Your product’s brand name. Provide the name of the company that created the product (i.e. manufacturer or supplier). Use the brand name of the product generally recognized by consumers.
  • Image link
    The URL of an image for your product. This is the main product image. If you have multiple images, use the additional_image_link attribute. To increase site load speed, please have small file sizes, anything from 200x200 to 600x600 would reasonable.
  • Category
    Product categories that you define for your product. A product category is not a physical attribute, like color or dimension; nor should it specify that the product is new or on sale (e.g. “Gifts for him” or “Black Friday Sales” are not categories).
    The engine uses categories to infer similarity between products, which is then used to determine relevance and show related results, so putting dissimilar products into one category can lead to irrelevant search results.
    Make sure to include the full category path. For example: if you sell a jersey for cycling, don’t settle for the general category: Activewear, rather be as specific as possible: Apparel and Accessories > Clothing > Activewear > Bicycle Activewear > Bicycle Jerseys.
    • The engine can support a full path as an array and a list of lists - where outer list is all categories the product belongs to and inner list are the hierarchies for each category. The example above should be encoded as [["Apparel and Accessories", "Clothing", "Activewear", "Bicycle Activewear", "Bicycle Jerseys"]]
    • If the product falls under multiple categories paths and one path is primary, give it a name like: primary_category_path

We use these attributes to build additional logic, faceted filters and to improve the relevance of search results and/or category listings. The attributes listed below are highly recommended but not required for the proper functioning of your Loop54 engine.

  • Description
    Your product’s description. This is where you give all the product details a customer cares about: features, dimensions, use cases, etc. The description helps to tell Loop54 what other keywords your product would be relevant to appear for. As with the title, avoid promotional text.
    • Clearly explain what the product does or its intended use. Offer added value or benefits. Include all relevant information.
    • Match the description found on the product landing page.
    • Don’t include promotional text like “free shipping,” all capital letters, or gimmicky foreign characters like emojis.
    • Include only information about the product. Don’t include links to your store, sales information, details about competitors, other products, accessories or standard text that isn’t exclusively a description of the product.
  • Price
    Your product’s price (excluding currency). Loop54 uses price for distinct and and range faceting, therefore please don’t include currency in the price value. If you intend to use the same feed (and engine) for several countries and have different prices for different currencies, please use multiple currency attributes instead. For example: PriceEUR:5.99, PriceUSD: 7.99
  • Variant group id (where applicable)
    Connection ID between parent SKU and child variants. This attribute groups together different product variants, like items in different colors or sizes. Used to associate articles and products and to make all metadata searchable even if only one variant is displayed in the search response.
    • Use a unique value for each group of variants
    • Use the parent SKU as the group ID where possible
    • Keep the value the same when updating your product data
    • Use only valid UTF-8 characters
    • Use a variant group ID for a set of products that differ by one or more of these attributes:
      • color
      • size
      • pattern
      • material
      • age group
      • gender
      • etc.
    • Include the same attributes for each product in the variant group. For example, if a product varies by size and color, submit size and color for every product that shares the same value for variant_group_id
  • Color
    Your product’s color as the full name of the color in the target language, e.g. “Red”, “Dark Green”, “Neon Yellow”. For multiple colors use an array, e.g [“Red,”Dark Green“,”Neon Yellow"]. For variants: On each variant, put the same value for variant_group_id, but a different value for color. If there are multiple variants with similar but differently named color types (e.g. canary, yellow and neon yellow) and color will be used for facetting, group colors into a new attribute called color_group (e.g. color_group: “Yellow”).
  • Availability
    Availability status of an item (as a string or boolean). It is useful when Loop54 handles paging. We can use it to bury out-of-stock products in results and category listings. Example: Preorder, In Stock, Out of Stock.
  • Campaigns
    Products on sale or products being promoted for any reason. We can use it to give products on campaign higher weight/prominence in search results and/or category listings.
  • New Until
    When a new product is added to your catalogue, this is the date that the product is no longer considered new. Should you wish to boost new products, we use this attribute to give new products higher weight/prominence in search results and/or category listings until the date provided. Please use ISO 8601 format.
  • Product series name (or similar)
    Your product’s Global Trade Item Number (GTIN). Gtin or Global Trade Item Numbers are unique product identifiers. Often called barcodes. Depending on where you are located or source your products from, you’ll have a different set of numbers: UPC: 12 digit number mainly used in North-America; EAN/UCC: 8,13 or 14 digit number mainly used outside North America; JAN: 8 or 13 digit number used in Japan; ISBN: unique identifier for books. If you have multiple options, for example a UPC and a EAN number, you can use either one of them.
  • Hidden Keywords
    Words that you can supply to help boost discoverability in search results. Use terms that would help someone find a product if they hadn’t searched for what you visibly provided on the search result page.
  • Attributes that say something about the relationship between specific products
    Since Loop54 uses all product metadata to build relationships between products, a product attribute like “Gear Type” for Bicycles could be used to group all bicycles with the same gear type. Any attribute that describes the product and would be used by shoppers to narrow or expand their search should be included.
  • Attributes that describe a unique product feature
    Any other attribute shoppers might use to compare products or use to narrow results through faceting. For example: gender, age group or size. (e.g. Wheel Size for bicycles)

Optional Product Attributes

Include these optional attributes if they are useful to you.

  • Link
    Your product’s landing page. Provide a direct link to the product page (not your homepage). If you have a specific page for mobile traffic, supply that link via the mobile_link attribute.
    • Use your verified domain name
    • Start with http or https
    • Use an in-domain tracking URL if needed. Don’t use too many redirects, which can increase latency

Synchronization

We can theoretically synchronize the product feed every 5-10 minutes (depending on the size of the feed) but this can be very resource intensive for the e-commerce platform.

Hybrid Solutions in Synchronization

There are also hybrid solutions available where delta-information/changes is pushed to our API continuously in real time. Partial API syncs should always be complemented with a complete feed that Loop54 can synchronize with at least once a day.

Decisions to Make

This is a non-exhaustive list of decisions you should be considering when implementing your feed.

Synchronization Interval

How often should Loop54 sync against the product feed? This is often dependent on the logic that Loop54 should incorporate. E.g. if Loop54 shall filter out of stock items, the synchronization intervals needs to be short.

Value-adding Attributes

Which attributes should Loop54 incorporate business logic from? The usual ones are campaigns and news.

Filtering

Should Loop54 filter out results (e.g. out of stock) before the result is sent back?

Faceting

All attributes that are needed for faceting need to be in the feed.

Parent or Children, or Both?

Many product catalogues contains a product hierarchy where one product contains several variants or articles (On Magento, these are often called configurable products). Loop54 can be configured to send back either type, depending on how the users interact with the products. It is also possible to return variants/articles on a narrow search (e.g. a model number) and products on broader search phrases (e.g. product category). So depending on the situation it needs to be decided which is most suitable in terms of use case, available meta data and platform capabilities.