Company Logo
MCIP

Menu

Why Adapters Matter

Here's the reality of e-commerce: every platform speaks its own language. Shopify has its REST conventions, Vendure uses GraphQL, and that enterprise system your company built in 2015? It probably has a SOAP API that makes developers cry.

Adapters are MCIP's universal translators. They take the unique dialect of each platform and convert it into a language that AI agents understand. Think of them like those electrical plug adapters you use when traveling—your device (MCIP) stays the same, but the adapter handles whatever outlet (platform API) you encounter.

Without adapters, you'd need custom integration code for every store. With adapters, you write the translation logic once, and MCIP handles everything else: semantic search, session handling, and parallel execution across multiple platforms.

Understanding the Adapter Interface

The ProductMapper Contract

Every adapter in MCIP implements the ProductMapper interface. It's elegantly simple:

// src/modules/ingestion/mapper/product-mapper.interface.ts

import { UnifiedProduct } from "../../../domain/product.schema";

export interface ProductMapper {
  map(raw: any): Promise<UnifiedProduct>;
}

The map() method receives raw product data from your platform and returns a UnifiedProduct. That's it. One method, infinite possibilities.

The UnifiedProduct Schema

This is the common language all adapters must speak. Every product, regardless of its source, gets normalized to this structure:

interface UnifiedProduct {
  // Identity - Where does this product live?
  externalId: string;          // Unique ID from source (e.g., "prod_abc123")
  url: string;                 // Product page URL for humans

  // Core Content - What is this product?
  title: string;               // Clean product name (min 3 characters)
  description: string;         // Plain text, no HTML

  // Categorization - How do we find it?
  brand?: string;              // "Nike", "Apple", "Acme Corp"
  category?: string;           // "Laptops", "Running Shoes", "Coffee"

  // Commercial - What does it cost?
  price: {
    amount: number;            // 99.99 (not cents!)
    currency: "UAH" | "USD" | "EUR";
  };

  // Visuals - What does it look like?
  mainImage: string;           // Primary image URL

  // Details - What are the specifics?
  attributes: Array<{
    name: string;              // "Color", "Size", "Material"
    value: string | number | boolean;
  }>;

  variants: Array<{
    sku: string;               // Stock keeping unit
    title: string;             // "Red / XL", "64GB / Space Gray"
    price: { amount: number; currency: string } | null;
    available: boolean;
  }>;

  // AI Metadata - What should search understand?
  keywords: string[];          // 5-10 SEO keywords for vector search
}

Why this structure? It captures everything AI agents need to make intelligent shopping decisions while remaining simple enough that any platform's data can map to it.

Part 1: Building a REST Adapter

Let's build a real adapter for a REST-based e-commerce platform. We'll use a fictional "MegaStore" API, but the patterns apply to Shopify, WooCommerce, BigCommerce, and any REST endpoint.

Step 1: Create the Adapter File

Start by creating the directory structure:

mkdir -p src/modules/ingestion/mapper/strategies/megastore
touch src/modules/ingestion/mapper/strategies/megastore/megastore.mapper.ts
touch src/modules/ingestion/mapper/strategies/megastore/types.ts

Step 2: Define Your Platform's Types

Understanding your source data is half the battle. Document the shape of your platform's API response:

// src/modules/ingestion/mapper/strategies/megastore/types.ts

export interface MegaStoreProduct {
  product_id: string;
  product_name: string;
  description_html: string;
  manufacturer: string;
  category_path: string[];      // ["Electronics", "Computers", "Laptops"]
  base_price: number;           // In cents: 99999 = $999.99
  currency_code: string;
  primary_image_url: string;
  product_url: string;
  specifications: Array<{
    spec_name: string;
    spec_value: string;
  }>;
  variants: Array<{
    variant_id: string;
    variant_name: string;
    price_adjustment: number;   // Additional cost in cents
    in_stock: boolean;
    stock_quantity: number;
  }>;
  tags: string[];
  is_active: boolean;
}

Step 3: Implement the Mapper

Now for the main event—the actual adapter:

// src/modules/ingestion/mapper/strategies/megastore/megastore.mapper.ts

import { Injectable, Logger } from "@nestjs/common";
import { ConfigService } from "@nestjs/config";
import { 
  UnifiedProduct, 
  UnifiedProductSchema 
} from "../../../../../domain/product.schema";
import { ProductMapper } from "../../product-mapper.interface";
import { MegaStoreProduct } from "./types";

@Injectable()
export class MegaStoreMapper implements ProductMapper {
  private readonly logger = new Logger(MegaStoreMapper.name);

  constructor(private readonly configService: ConfigService) {}

  async map(raw: any): Promise<UnifiedProduct> {
    // Step 1: Validate we received something usable
    if (!raw || typeof raw !== "object" || !raw.product_id) {
      throw new Error("Invalid MegaStore product: missing product_id");
    }

    // Step 2: Cast to our typed interface
    const source = raw as MegaStoreProduct;
    
    // Step 3: Skip inactive products
    if (!source.is_active) {
      throw new Error(`Product ${source.product_id} is inactive, skipping`);
    }

    this.logger.debug(`Mapping MegaStore product: ${source.product_name}`);

    // Step 4: Build the unified product
    const product: UnifiedProduct = {
      externalId: source.product_id,
      url: source.product_url,
      
      title: this.cleanTitle(source.product_name),
      description: this.stripHtml(source.description_html),
      
      brand: source.manufacturer || undefined,
      category: this.extractCategory(source.category_path),
      
      price: {
        amount: this.centsToDecimal(source.base_price),
        currency: this.normalizeCurrency(source.currency_code),
      },
      
      mainImage: source.primary_image_url,
      
      attributes: this.mapAttributes(source.specifications),
      variants: this.mapVariants(source.variants, source.base_price),
      
      keywords: this.generateKeywords(source),
    };

    // Step 5: Validate with Zod schema (catches mistakes early!)
    return UnifiedProductSchema.parse(product);
  }

  // ─────────────────────────────────────────────────────────────
  // Helper Methods - Each handles one transformation
  // ─────────────────────────────────────────────────────────────

  private cleanTitle(title: string): string {
    return title
      .replace(/\s+/g, " ")           // Collapse whitespace
      .replace(/[\r\n]/g, "")         // Remove line breaks
      .trim()
      .substring(0, 200);              // Reasonable length limit
  }

  private stripHtml(html: string): string {
    if (!html) return "";
    
    return html
      // Remove script and style tags completely
      .replace(/<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi, "")
      .replace(/<style\b[^<]*(?:(?!<\/style>)<[^<]*)*<\/style>/gi, "")
      // Convert common entities
      .replace(/&nbsp;/g, " ")
      .replace(/&amp;/g, "&")
      .replace(/&lt;/g, "<")
      .replace(/&gt;/g, ">")
      .replace(/&quot;/g, '"')
      // Strip remaining tags
      .replace(/<[^>]*>/g, "")
      // Clean up whitespace
      .replace(/\s+/g, " ")
      .trim();
  }

  private extractCategory(categoryPath: string[]): string | undefined {
    // Take the most specific category (last in path)
    if (!categoryPath || categoryPath.length === 0) return undefined;
    return categoryPath[categoryPath.length - 1];
  }

  private centsToDecimal(cents: number): number {
    // Many platforms store prices in cents to avoid floating point issues
    return Math.round(cents) / 100;
  }

  private normalizeCurrency(code: string): "UAH" | "USD" | "EUR" {
    const mapping: Record<string, "UAH" | "USD" | "EUR"> = {
      "usd": "USD",
      "us": "USD",
      "dollar": "USD",
      "$": "USD",
      "eur": "EUR",
      "euro": "EUR",
      "€": "EUR",
      "uah": "UAH",
      "грн": "UAH",
      "hryvnia": "UAH",
    };
    
    const normalized = code.toLowerCase().trim();
    return mapping[normalized] || "USD"; // Default to USD if unknown
  }

  private mapAttributes(
    specs: MegaStoreProduct["specifications"]
  ): UnifiedProduct["attributes"] {
    if (!specs) return [];
    
    return specs.map(spec => ({
      name: spec.spec_name,
      value: spec.spec_value,
    }));
  }

  private mapVariants(
    variants: MegaStoreProduct["variants"],
    basePrice: number
  ): UnifiedProduct["variants"] {
    if (!variants) return [];
    
    return variants.map(v => {
      const variantPrice = basePrice + (v.price_adjustment || 0);
      const basePriceDecimal = this.centsToDecimal(basePrice);
      const variantPriceDecimal = this.centsToDecimal(variantPrice);
      
      return {
        sku: v.variant_id,
        title: v.variant_name,
        available: v.in_stock && v.stock_quantity > 0,
        // Only include price if different from base
        price: variantPriceDecimal !== basePriceDecimal
          ? { amount: variantPriceDecimal, currency: "USD" }
          : null,
      };
    });
  }

  private generateKeywords(product: MegaStoreProduct): string[] {
    const keywords: string[] = [];
    
    // Extract words from title
    const titleWords = product.product_name
      .toLowerCase()
      .replace(/[^a-z0-9\s]/g, "")
      .split(/\s+/)
      .filter(word => word.length > 3);
    keywords.push(...titleWords);
    
    // Add tags
    if (product.tags) {
      keywords.push(...product.tags.map(t => t.toLowerCase()));
    }
    
    // Add brand and category
    if (product.manufacturer) {
      keywords.push(product.manufacturer.toLowerCase());
    }
    if (product.category_path) {
      keywords.push(...product.category_path.map(c => c.toLowerCase()));
    }
    
    // Deduplicate and limit
    return [...new Set(keywords)].slice(0, 10);
  }
}

Step 4: Register Your Adapter

Add your mapper to the ingestion module:

// src/modules/ingestion/ingestion.module.ts

import { MegaStoreMapper } from "./mapper/strategies/megastore/megastore.mapper";

@Module({
  providers: [
    {
      provide: PRODUCT_MAPPER,
      useFactory: (
        configService: ConfigService,
        customAiMapper: CustomAiMapper,
        vendureMapper: VendureMapper,
        megaStoreMapper: MegaStoreMapper  // Add here
      ) => {
        const provider = configService.get<string>("STORE_PROVIDER", "VENDURE");
        
        switch (provider.toUpperCase()) {
          case "VENDURE":
            return vendureMapper;
          case "MEGASTORE":           // Add case
            return megaStoreMapper;
          default:
            return customAiMapper;
        }
      },
      inject: [ConfigService, CustomAiMapper, VendureMapper, MegaStoreMapper],
    },
    MegaStoreMapper,  // Add to providers
    // ... other providers
  ],
})
export class IngestionModule {}

Step 5: Configure Environment

STORE_PROVIDER=MEGASTORE
SOURCE_URL=https://api.megastore.com/v2/products
SOURCE_API_KEY=your-api-key
STOREFRONT_URL=https://megastore.com

Congratulations! You've built a complete REST adapter. Time for a coffee break—you've earned it.

Part 2: Building a GraphQL Adapter

GraphQL platforms like Vendure, Shopify Storefront API, and modern headless systems require a slightly different approach. Let's build one.

The Key Difference

REST adapters deal with whatever the API gives you. GraphQL adapters let you request exactly what you need. This is both a superpower and a responsibility.

Complete GraphQL Adapter Example

// src/modules/ingestion/mapper/strategies/modernshop/modernshop.mapper.ts

import { Injectable, Logger } from "@nestjs/common";
import { ConfigService } from "@nestjs/config";
import { 
  UnifiedProduct, 
  UnifiedProductSchema 
} from "../../../../../domain/product.schema";
import { ProductMapper } from "../../product-mapper.interface";

// The GraphQL response shape (matches your query)
interface ModernShopProduct {
  id: string;
  name: string;
  slug: string;
  description: string;
  featuredAsset: {
    preview: string;
    source: string;
  } | null;
  variants: Array<{
    id: string;
    sku: string;
    name: string;
    price: number;           // Already in decimal form
    priceWithTax: number;
    currencyCode: string;
    stockLevel: string;      // "IN_STOCK", "OUT_OF_STOCK", "LOW_STOCK"
    options: Array<{
      code: string;
      name: string;
    }>;
  }>;
  facetValues: Array<{
    name: string;
    facet: {
      name: string;
    };
  }>;
  collections: Array<{
    name: string;
    slug: string;
  }>;
}

@Injectable()
export class ModernShopMapper implements ProductMapper {
  private readonly logger = new Logger(ModernShopMapper.name);
  private readonly storefrontUrl: string;

  constructor(private readonly configService: ConfigService) {
    this.storefrontUrl = this.configService.get<string>(
      "STOREFRONT_URL", 
      "https://shop.example.com"
    );
  }

  async map(raw: any): Promise<UnifiedProduct> {
    if (!raw?.id) {
      throw new Error("Invalid ModernShop product: missing id");
    }

    const source = raw as ModernShopProduct;
    this.logger.debug(`Mapping: ${source.name}`);

    // Extract brand from facet values
    const brand = this.extractFacetValue(source.facetValues, "brand");
    
    // Get primary category from collections
    const category = source.collections?.[0]?.name;
    
    // Use first variant for base pricing
    const primaryVariant = source.variants[0];
    if (!primaryVariant) {
      throw new Error(`Product ${source.id} has no variants`);
    }

    const product: UnifiedProduct = {
      externalId: source.id,
      url: `${this.storefrontUrl}/products/${source.slug}`,
      
      title: source.name,
      description: this.stripHtml(source.description || ""),
      
      brand,
      category,
      
      price: {
        amount: this.normalizePrice(primaryVariant.priceWithTax),
        currency: this.normalizeCurrency(primaryVariant.currencyCode),
      },
      
      mainImage: source.featuredAsset?.preview 
        || source.featuredAsset?.source 
        || "",
      
      attributes: this.extractAttributes(source.facetValues),
      variants: this.mapVariants(source.variants, primaryVariant.priceWithTax),
      
      keywords: this.generateKeywords(source),
    };

    return UnifiedProductSchema.parse(product);
  }

  private extractFacetValue(
    facetValues: ModernShopProduct["facetValues"],
    facetName: string
  ): string | undefined {
    const facet = facetValues?.find(
      fv => fv.facet.name.toLowerCase() === facetName.toLowerCase()
    );
    return facet?.name;
  }

  private extractAttributes(
    facetValues: ModernShopProduct["facetValues"]
  ): UnifiedProduct["attributes"] {
    if (!facetValues) return [];
    
    // Group by facet name, skip "brand" (handled separately)
    return facetValues
      .filter(fv => fv.facet.name.toLowerCase() !== "brand")
      .map(fv => ({
        name: fv.facet.name,
        value: fv.name,
      }));
  }

  private mapVariants(
    variants: ModernShopProduct["variants"],
    basePrice: number
  ): UnifiedProduct["variants"] {
    return variants.map(v => {
      const optionString = v.options
        .map(o => o.name)
        .join(" / ");
      
      return {
        sku: v.sku || v.id,
        title: optionString || v.name,
        available: v.stockLevel !== "OUT_OF_STOCK",
        price: v.priceWithTax !== basePrice
          ? { 
              amount: this.normalizePrice(v.priceWithTax), 
              currency: this.normalizeCurrency(v.currencyCode) 
            }
          : null,
      };
    });
  }

  private normalizePrice(price: number): number {
    // Vendure stores prices in cents, others might not
    // Adjust based on your platform
    return Math.round(price) / 100;
  }

  private normalizeCurrency(code: string): "UAH" | "USD" | "EUR" {
    const upperCode = code.toUpperCase();
    if (["UAH", "USD", "EUR"].includes(upperCode)) {
      return upperCode as "UAH" | "USD" | "EUR";
    }
    return "USD";
  }

  private stripHtml(html: string): string {
    return html.replace(/<[^>]*>/g, "").trim();
  }

  private generateKeywords(product: ModernShopProduct): string[] {
    const words: string[] = [];
    
    // Title words
    words.push(
      ...product.name.toLowerCase().split(/\s+/).filter(w => w.length > 3)
    );
    
    // Collection names
    product.collections?.forEach(c => {
      words.push(c.name.toLowerCase());
    });
    
    // Facet values
    product.facetValues?.forEach(fv => {
      words.push(fv.name.toLowerCase());
    });
    
    return [...new Set(words)].slice(0, 10);
  }
}

The GraphQL Query

Store this in your environment configuration:

{
  products {
    items {
      id
      name
      slug
      description
      featuredAsset {
        preview
        source
      }
      variants {
        id
        sku
        name
        price
        priceWithTax
        currencyCode
        stockLevel
        options {
          code
          name
        }
      }
      facetValues {
        name
        facet {
          name
        }
      }
      collections {
        name
        slug
      }
    }
  }
}

Part 3: Custom Protocol Adapters

Sometimes you encounter APIs that don't follow REST or GraphQL conventions. Maybe it's a SOAP service, an XML-RPC endpoint, or a proprietary binary protocol. Don't panic—the adapter pattern handles these too.

Strategy: Normalize at the Edge

The key insight: your mapper doesn't care how data arrives. It just needs a JavaScript object to transform. Handle the protocol weirdness in a separate layer, then pass clean objects to your mapper.

// src/modules/ingestion/services/soap-client.service.ts

import { Injectable } from "@nestjs/common";
import * as soap from "soap";

@Injectable()
export class SoapClientService {
  private client: soap.Client | null = null;

  async initialize(wsdlUrl: string): Promise<void> {
    this.client = await soap.createClientAsync(wsdlUrl);
  }

  async getProducts(): Promise<any[]> {
    if (!this.client) {
      throw new Error("SOAP client not initialized");
    }

    // Call the SOAP method
    const [result] = await this.client.GetProductCatalogAsync({});
    
    // SOAP responses are often deeply nested
    const products = result?.GetProductCatalogResult?.Products?.Product;
    
    // Normalize to array (SOAP might return single item differently)
    if (!products) return [];
    return Array.isArray(products) ? products : [products];
  }
}

The Mapper Remains Simple

// src/modules/ingestion/mapper/strategies/legacy/legacy-soap.mapper.ts

@Injectable()
export class LegacySoapMapper implements ProductMapper {
  async map(raw: any): Promise<UnifiedProduct> {
    // By the time data reaches here, it's just an object
    // The SOAP complexity is handled elsewhere
    
    return UnifiedProductSchema.parse({
      externalId: raw.ProductCode,
      url: `https://legacy.company.com/item/${raw.ProductCode}`,
      title: raw.ProductName,
      description: raw.LongDescription || raw.ShortDescription || "",
      brand: raw.ManufacturerName,
      category: raw.CategoryName,
      price: {
        amount: parseFloat(raw.ListPrice),
        currency: "USD",
      },
      mainImage: raw.ImageURL,
      attributes: this.parseAttributes(raw.Specifications),
      variants: [],  // Legacy system doesn't support variants
      keywords: this.extractKeywords(raw),
    });
  }

  private parseAttributes(specs: string): UnifiedProduct["attributes"] {
    // Legacy system stores specs as "Key1:Value1|Key2:Value2"
    if (!specs) return [];
    
    return specs.split("|").map(pair => {
      const [name, value] = pair.split(":");
      return { name: name.trim(), value: value.trim() };
    });
  }

  private extractKeywords(raw: any): string[] {
    const text = `${raw.ProductName} ${raw.CategoryName} ${raw.ManufacturerName}`;
    return text.toLowerCase().split(/\s+/).filter(w => w.length > 3).slice(0, 10);
  }
}

Part 4: Testing Your Adapter

Adapters that aren't tested are adapters waiting to fail in production. Here's how to test thoroughly.

Unit Tests

Test the mapper in isolation with known inputs:

// src/modules/ingestion/mapper/strategies/megastore/megastore.mapper.spec.ts

import { Test, TestingModule } from "@nestjs/testing";
import { ConfigService } from "@nestjs/config";
import { MegaStoreMapper } from "./megastore.mapper";

describe("MegaStoreMapper", () => {
  let mapper: MegaStoreMapper;

  beforeEach(async () => {
    const module: TestingModule = await Test.createTestingModule({
      providers: [
        MegaStoreMapper,
        {
          provide: ConfigService,
          useValue: {
            get: jest.fn((key: string) => {
              const config: Record<string, string> = {
                STOREFRONT_URL: "https://test.megastore.com",
              };
              return config[key];
            }),
          },
        },
      ],
    }).compile();

    mapper = module.get<MegaStoreMapper>(MegaStoreMapper);
  });

  describe("successful mapping", () => {
    it("should map a complete product correctly", async () => {
      const rawProduct = {
        product_id: "MEGA-001",
        product_name: "Ultra Gaming Laptop",
        description_html: "<p>Powerful <strong>gaming</strong> laptop</p>",
        manufacturer: "TechBrand",
        category_path: ["Electronics", "Computers", "Laptops"],
        base_price: 149999,  // $1499.99 in cents
        currency_code: "USD",
        primary_image_url: "https://cdn.megastore.com/laptop.jpg",
        product_url: "https://megastore.com/products/ultra-gaming-laptop",
        specifications: [
          { spec_name: "RAM", spec_value: "32GB" },
          { spec_name: "Storage", spec_value: "1TB SSD" },
        ],
        variants: [
          {
            variant_id: "MEGA-001-16",
            variant_name: "16GB RAM",
            price_adjustment: -20000,  // -$200
            in_stock: true,
            stock_quantity: 10,
          },
          {
            variant_id: "MEGA-001-32",
            variant_name: "32GB RAM",
            price_adjustment: 0,
            in_stock: true,
            stock_quantity: 5,
          },
        ],
        tags: ["gaming", "laptop", "high-performance"],
        is_active: true,
      };

      const result = await mapper.map(rawProduct);

      expect(result.externalId).toBe("MEGA-001");
      expect(result.title).toBe("Ultra Gaming Laptop");
      expect(result.description).toBe("Powerful gaming laptop");
      expect(result.brand).toBe("TechBrand");
      expect(result.category).toBe("Laptops");
      expect(result.price.amount).toBe(1499.99);
      expect(result.price.currency).toBe("USD");
      expect(result.attributes).toHaveLength(2);
      expect(result.variants).toHaveLength(2);
      expect(result.variants[0].price?.amount).toBe(1299.99);
      expect(result.keywords).toContain("gaming");
    });

    it("should handle missing optional fields", async () => {
      const minimalProduct = {
        product_id: "MEGA-002",
        product_name: "Simple Product",
        description_html: "",
        manufacturer: "",
        category_path: [],
        base_price: 999,
        currency_code: "USD",
        primary_image_url: "",
        product_url: "https://megastore.com/products/simple",
        specifications: [],
        variants: [
          {
            variant_id: "MEGA-002-DEF",
            variant_name: "Default",
            price_adjustment: 0,
            in_stock: true,
            stock_quantity: 1,
          },
        ],
        tags: [],
        is_active: true,
      };

      const result = await mapper.map(minimalProduct);

      expect(result.externalId).toBe("MEGA-002");
      expect(result.brand).toBeUndefined();
      expect(result.category).toBeUndefined();
      expect(result.attributes).toHaveLength(0);
    });
  });

  describe("error handling", () => {
    it("should throw on null input", async () => {
      await expect(mapper.map(null)).rejects.toThrow("Invalid MegaStore product");
    });

    it("should throw on missing product_id", async () => {
      await expect(mapper.map({ name: "No ID" })).rejects.toThrow();
    });

    it("should throw on inactive products", async () => {
      const inactiveProduct = {
        product_id: "MEGA-003",
        product_name: "Inactive Item",
        is_active: false,
        // ... other fields
      };

      await expect(mapper.map(inactiveProduct)).rejects.toThrow("inactive");
    });
  });

  describe("edge cases", () => {
    it("should handle HTML entities in descriptions", async () => {
      const product = createValidProduct({
        description_html: "Price is &lt;$100 &amp; includes &quot;free&quot; shipping",
      });

      const result = await mapper.map(product);

      expect(result.description).toBe('Price is <$100 & includes "free" shipping');
    });

    it("should normalize various currency formats", async () => {
      const currencies = ["usd", "USD", "$", "dollar"];
      
      for (const curr of currencies) {
        const product = createValidProduct({ currency_code: curr });
        const result = await mapper.map(product);
        expect(result.price.currency).toBe("USD");
      }
    });
  });
});

// Helper to create valid products with overrides
function createValidProduct(overrides: Partial<any> = {}): any {
  return {
    product_id: "TEST-001",
    product_name: "Test Product",
    description_html: "<p>Description</p>",
    manufacturer: "TestBrand",
    category_path: ["Category"],
    base_price: 1000,
    currency_code: "USD",
    primary_image_url: "https://example.com/img.jpg",
    product_url: "https://example.com/product",
    specifications: [],
    variants: [{
      variant_id: "VAR-001",
      variant_name: "Default",
      price_adjustment: 0,
      in_stock: true,
      stock_quantity: 10,
    }],
    tags: [],
    is_active: true,
    ...overrides,
  };
}

Integration Tests

Test with real API responses (mocked or recorded):

// src/modules/ingestion/mapper/strategies/megastore/megastore.integration.spec.ts

import { Test } from "@nestjs/testing";
import { MegaStoreMapper } from "./megastore.mapper";
import * as realApiResponse from "./fixtures/real-api-response.json";

describe("MegaStoreMapper Integration", () => {
  let mapper: MegaStoreMapper;

  beforeAll(async () => {
    // Setup with real config
    const module = await Test.createTestingModule({
      imports: [ConfigModule.forRoot()],
      providers: [MegaStoreMapper],
    }).compile();

    mapper = module.get<MegaStoreMapper>(MegaStoreMapper);
  });

  it("should map all products from real API response", async () => {
    const products = realApiResponse.products;
    const results: UnifiedProduct[] = [];
    const errors: Error[] = [];

    for (const product of products) {
      try {
        const mapped = await mapper.map(product);
        results.push(mapped);
      } catch (error) {
        errors.push(error as Error);
      }
    }

    // Report on success rate
    console.log(`Mapped ${results.length}/${products.length} products`);
    console.log(`Errors: ${errors.length}`);
    
    if (errors.length > 0) {
      console.log("Error samples:", errors.slice(0, 3).map(e => e.message));
    }

    // Expect high success rate
    expect(results.length / products.length).toBeGreaterThan(0.95);
  });
});

Troubleshooting Common Issues

"Zod validation failed"

Symptom: ZodError: ... when mapping products

Cause: Your mapped data doesn't match the UnifiedProduct schema

Solution: Check which field failed validation:

try {
  return UnifiedProductSchema.parse(product);
} catch (error) {
  if (error instanceof z.ZodError) {
    console.error("Validation failed:", error.errors);
    console.error("Product data:", JSON.stringify(product, null, 2));
  }
  throw error;
}

Common causes:

  • title shorter than 3 characters
  • price.currency not in allowed values
  • Missing required fields

"Products not appearing in search"

Symptom: Sync succeeds but products don't show in search results

Cause: Embeddings weren't generated or stored correctly

Solution: Check the ingestion processor logs:

docker-compose logs mcip | grep -i "embedding\|vector\|qdrant"

Verify products exist in Qdrant:

curl http://localhost:6333/collections/products/points/count

"Price shows wrong value"

Symptom: Prices are 100x too high or too low

Cause: Mismatch between cents and decimal representation

Solution: Check your platform's price format and adjust:

// If API returns cents (12999 = $129.99)
private normalizePrice(price: number): number {
  return price / 100;
}

// If API returns decimals (129.99)
private normalizePrice(price: number): number {
  return price; // No conversion needed
}

"Keywords are empty or wrong"

Symptom: Semantic search returns irrelevant results

Cause: Poor keyword extraction affects vector quality

Solution: Review and improve your generateKeywords method:

private generateKeywords(product: YourProduct): string[] {
  const keywords: string[] = [];
  
  // Don't just split title - use meaningful parts
  // Bad: "The" "Best" "Gaming" "Laptop" "Ever"
  // Good: "gaming" "laptop" "rgb" "nvidia"
  
  // Include searchable attributes
  if (product.specs?.gpu) {
    keywords.push(product.specs.gpu.toLowerCase());
  }
  
  // Include category synonyms
  const categoryMap: Record<string, string[]> = {
    "Laptops": ["notebook", "portable computer"],
    "Phones": ["mobile", "smartphone", "cell phone"],
  };
  
  if (categoryMap[product.category]) {
    keywords.push(...categoryMap[product.category]);
  }
  
  return [...new Set(keywords)].slice(0, 10);
}

Performance Tips

Batch Processing

For large catalogs, process products in batches:

const BATCH_SIZE = 50;

for (let i = 0; i < products.length; i += BATCH_SIZE) {
  const batch = products.slice(i, i + BATCH_SIZE);
  await Promise.all(batch.map(p => this.processProduct(p)));
  
  // Small delay to avoid rate limits
  await new Promise(resolve => setTimeout(resolve, 100));
}

Lazy Loading Images

Don't validate image URLs during mapping—it's slow:

// Don't do this in the mapper:
const isValidImage = await fetch(imageUrl).then(r => r.ok);

// Do this instead:
mainImage: source.image_url || "",  // Accept as-is, validate later

Cache Expensive Operations

If your mapper makes external calls (rare but sometimes needed), cache results:

private readonly categoryCache = new Map<string, string>();

private async resolveCategory(categoryId: string): Promise<string> {
  if (this.categoryCache.has(categoryId)) {
    return this.categoryCache.get(categoryId)!;
  }
  
  const name = await this.fetchCategoryName(categoryId);
  this.categoryCache.set(categoryId, name);
  return name;
}

What's Next

You've mastered adapter development! Here's where to go from here: