Feed Services Documentation

Overview

Microservice Name: eq-foundation-service
Version: 9.0
API Reference: Swagger UI

Description

The Equinox Foundation Feed Services is the data orchestration and batch processing backbone of the Equinox 9.x commerce platform — enabling businesses to automate data imports, exports, synchronizations, and transformations across all platform services. This service provides centralized control over data processing pipelines, from catalog imports to GDPR exports, with flexible execution engines and comprehensive monitoring.

Business Value

Capability Business Outcome
Unified Feed Orchestration One centralized platform for all data import/export operations — no scattered batch scripts across services
Multi-Engine Support Choose the right execution engine (Lambda, Step Functions, Spring Batch, AWS Glue) for each job type
Automated Scheduling Configure recurring feeds with EventBridge integration — schedule once, run forever
Real-Time Monitoring Track every feed execution with detailed history, progress reporting, and error diagnostics
Multi-Tenant Isolation Complete data segregation per tenant with business and instance-level filtering
Event-Driven Architecture Fire-and-forget execution via EventBridge — services remain decoupled and independently scalable

Who Uses It

Persona How They Interact
Data Operations Teams Configure and schedule feeds, monitor execution history, troubleshoot failures
Commerce Managers Trigger on-demand catalog imports/exports, sync inventory data, generate GDPR reports
Platform Administrators Define feed templates, configure retry policies, manage execution clusters
Business Analysts Access feed execution metrics, success rates, and performance statistics
Service Developers Integrate new feed types, implement custom processors, extend Step Functions

Related Resources

  • Glossary: Section 13
  • Batch API Reference: Section 5
  • Feed Architectures: docs/feed-architectures.md
  • BRD Document: BRD-FEED-20260224-V1.0.md

Orchestrate Your Data. Automate Your Operations.

One platform. Every feed. Complete control.

The Equinox Foundation Feed Services gives commerce platforms a unified, scalable infrastructure for all data processing operations. Whether importing product catalogs, exporting inventory snapshots, or fulfilling GDPR data requests, Feed Services handles scheduling, execution, monitoring, and error recovery — so your teams can focus on business outcomes rather than managing batch infrastructure.

Why Equinox Feed Services?

Modern commerce platforms require reliable, automated data pipelines. Catalog updates, inventory synchronizations, price imports, and compliance exports must run on schedule, at scale, without manual intervention. Feed Services provides the orchestration layer that makes it possible.

Key Capabilities

Multi-Engine Execution
Support for Lambda functions, AWS Step Functions with Distributed Map, Spring Batch, and AWS Glue. Choose the right engine based on data volume, complexity, and processing requirements.

Event-Driven Architecture
Publish-subscribe pattern via EventBridge. Foundation Service publishes batch.job.initiated events; Job Initiator Lambda subscribes and starts Step Functions. Services remain decoupled and independently scalable.

Intelligent Scheduling
EventBridge-native scheduling with cron expressions and rate limits. Supports recurring schedules, one-time schedules, and on-demand execution. Pause, resume, or stop schedules without code changes.

Comprehensive Monitoring
Track every execution with detailed history records. Monitor status (INITIATED → RUNNING → COMPLETED/FAILED), progress percentage, chunk-level success rates, and error diagnostics. 90-day history retention with TTL.

Parallel Execution Control
Configure whether feeds can run concurrently. When allowParallelExecution=false, Foundation Service prevents resource contention by blocking new executions until the current one completes.

Automatic Retry & Error Recovery
Configurable retry policies (exponential backoff, fixed interval, circuit breaker). Error threshold enforcement with toleration policies. Failed chunks retry independently without reprocessing successful data.

Multi-Tenant by Design
Business-level and instance-level data isolation. Every batch, job, schedule, and execution is scoped to a specific tenant. Filters enforce access control at query time.

S3-Based Data Handoff
Stateless Lambda functions coordinate via S3. Upload raw files to imports/, process in chunks, write results to results/, generate error files in source format. No shared memory or database transactions across execution boundaries.

Who Benefits

Persona Business Benefit
Data Operations Automated feed execution with scheduling, monitoring, and error recovery — reduce manual processing by 80%
Commerce Managers On-demand data imports/exports via UI; track progress in real time; download error files for corrections
Platform Teams Event-driven architecture with decoupled services; horizontally scalable execution; standard API contracts
Business Analysts Access execution metrics, success rates, and performance statistics for reporting and optimization
Compliance Officers Automated GDPR export and deletion workflows with complete audit trails

At a Glance

Metric Detail
Execution Engines Lambda, Step Functions (Distributed Map), Spring Batch, AWS Glue
Scheduling EventBridge rules with cron/rate expressions
Data Storage S3 (raw files, chunk files, result files, error files)
Feed Types Import (Catalog, Inventory, Price, Category) / Export (Catalog, Inventory, GDPR)
Max Concurrent Chunks 20 per Step Function Map (configurable per job)
History Retention 90 days with TTL
Error Recovery Exponential backoff, circuit breaker, error threshold enforcement
Deployment Cloud-native — serverless (AWS Lambda + Step Functions)
Version 9.0
API Reference Swagger UI

Core Features

# Domain Feature
1 Batch Create, read, update, delete batch definitions (execution configuration, parallel control, max runtime)
2 Batch Job Create, read, update, delete job definitions within a batch (job parameters, retry policy, error threshold)
3 Batch Schedule Create, read, update, delete schedules for jobs (cron/rate expressions, EventBridge integration)
4 Batch Execution Trigger job execution (on-demand or scheduled), track status, view progress, stop running jobs
5 Execution History Complete audit trail of all executions with start/end times, status, result, error diagnostics
6 File Upload Presigned S3 URLs for batch file uploads with validation and metadata tracking
7 Schedule State Management Activate, pause, resume, stop schedules without deleting EventBridge rules
8 Progress Reporting Real-time chunk-level progress updates with success/failure counts
9 Metrics & Analytics Execution statistics, success rates, performance metrics per batch/job
10 PIM Catalog Import Multi-sheet Excel/CSV import for products, SKUs, categories, attributes, prices
11 PIM Catalog Export Multi-sheet Excel/CSV export with entity extraction order and locale support
12 PIM Category Import Hierarchical category import with parent-child validation (sequential processing)
13 PIM Price Import Price list import with date range validation and currency checks
14 Inventory Import Multi-sheet import for locations, SKUs, lots, policies, positions, transactions
15 Inventory Export Multi-sheet export with 9-entity hierarchy (3 levels: master, policy, transactional)
16 GDPR Export User PII export as Excel (user, addresses, payments, lists, consents, orders)
17 GDPR Delete Anonymize user PII after retention period (Step Functions + Lambda)
18 Error File Generation Produces error files in source format with _importErrors column for failed rows
19 Distributed Map Processing AWS Step Functions Distributed Map for parallel chunk processing (max 20 concurrent)
20 Retry Orchestration Automatic retry with configurable policies (exponential backoff, max retries, circuit breaker)

Batch & Job Concepts

Hierarchy

Batch (Execution Group)
  ├── Metadata: batchId, name, serviceType, executionType
  ├── Configuration: allowParallelExecution, maxRuntime, jobEndpoint
  └── BatchJob (Executable Unit)
       ├── Metadata: jobId, name, status (ACTIVE/INACTIVE)
       ├── Configuration: jobParams (jobType, instanceId, catalogId, fileUrl, etc.)
       ├── Retry Policy: maxRetries, retryDelay, backoffMultiplier
       ├── Error Handling: errorThreshold, tolerationPolicy
       └── BatchSchedule (Trigger Mechanism)
            ├── Metadata: scheduleId, name, status, state
            ├── Schedule: repeatInterval (cron/rate), startDate, endDate
            ├── State: ACTIVE, PAUSED, STOPPED
            └── EventBridge: ruleName, targetArn, lastRunTime, nextRunTime

Core Entities

Batch

Purpose: Logical grouping of related jobs (e.g., “PIM Import Batch”, “Inventory Export Batch”).

Key Properties:

  • batchId — Unique identifier (kebab-case, e.g., pim-catalog-import)
  • serviceType — Service domain (e.g., pim-import, inventory-export)
  • executionType — Execution engine (LAMBDA, STEP_FUNCTION, SPRING_BATCH, GLUE)
  • allowParallelExecution — Boolean flag controlling concurrent job instances
  • maxRuntime — Maximum execution time in seconds (enforced by Step Functions timeout)
  • jobEndpoint — Lambda ARN or API endpoint for execution

BatchJob

Purpose: Specific executable unit within a batch (e.g., “Product Full Refresh”, “Inventory Stock Sync”).

Key Properties:

  • jobId — Unique identifier within the batch

  • status — ACTIVE (can execute) or INACTIVE (disabled)

  • jobParams — Flat key-value object with execution parameters

    • jobType — Feed type (e.g., CATALOG_IMPORT, INVENTORY_EXPORT, GDPR_EXPORT)
    • instanceId — Tenant identifier (multi-tenant isolation)
    • catalogId — PIM catalog identifier (for catalog-scoped jobs)
    • fileUrl — S3 path for import files (for import jobs)
    • chunkSize — Number of rows per chunk (default: 100)
    • Custom parameters per job type

  • Retry Policy:

    • maxRetries — Maximum retry attempts (default: 3)
    • retryDelay — Initial delay in seconds (default: 60)
    • backoffMultiplier — Exponential backoff factor (default: 2)
    • retryStrategy — exponential, fixed, circuitBreaker

  • Error Handling:

    • errorThreshold — Percentage of failures allowed before job fails (default: 10)
    • tolerationPolicy — STRICT (fail fast) or LENIENT (continue with errors)

BatchSchedule

Purpose: Trigger mechanism for recurring or on-demand job execution.

Key Properties:

  • scheduleId — Unique identifier within the job
  • status — SCHEDULED (cron/rate) or ONDEMAND (manual trigger)
  • state — ACTIVE, PAUSED, STOPPED (lifecycle state)
  • repeatInterval — EventBridge expression
    • Cron: cron(0 10 * * ? *) — Daily at 10:00 AM UTC
    • Rate: rate(1 day) — Every 24 hours
  • startDate / endDate — Schedule validity window
  • eventBridgeRuleName — Generated rule name (format: {stage}-fnd-{batchId}-{jobId}-{scheduleId})
  • lastRunTime / nextRunTime — Execution tracking

Special Status: ONDEMAND

  • Creates schedule record but skips EventBridge rule creation
  • Immediately triggers executeBatchJob() and returns runRequestId
  • Used for UI-triggered imports/exports and GDPR requests

Execution Lifecycle

1. INITIATED   — Job record created in MongoDB, EventBridge event published
                 └─► Job Initiator Lambda subscribes to event
2. PENDING     — Job Initiator Lambda invokes Step Function
                 └─► Step Function starts execution
3. RUNNING     — Step Function processes chunks via Distributed Map
                 └─► Lambda functions validate, parse, process data
4. COMPLETED   — All chunks processed successfully
     └─► Result file(s) written to S3
     └─► History record updated with end time, duration, result
     └─► EventBridge event: batch.execution.completed
5. FAILED      — Execution failed (validation error, threshold exceeded, timeout)
     └─► Error message recorded in history
     └─► EventBridge event: batch.execution.failed
     └─► Retry policy determines if retry is triggered

Parallel Execution Control

When allowParallelExecution = false:

  • Foundation Service checks for existing RUNNING executions for same (batchId, jobId)
  • If found: Throws BATCH_JOB_ALREADY_RUNNING error (HTTP 409 Conflict)
  • Purpose: Prevent resource contention and ensure sequential execution

When allowParallelExecution = true:

  • Multiple concurrent executions allowed for same job
  • Each execution tracked independently by runRequestId
  • Use case: Tenant-specific imports running in parallel

API Endpoints

Swagger UI: https://eq9-pdn1.equinox.shop/foundation-service-prod/documentation
API prefix: /v1

Security: Bearer JWT token in Authorization header. Privileges are enforced per endpoint via preHandler hooks.

Batch Management (/src/routes/batch.routes.ts)

Method Path Privilege Description
POST /v1/batches foundation:createBatch Create a new batch entity
GET /v1/batches foundation:viewBatch List batches (paginated, filterable by instanceId)
GET /v1/batches/:batchId foundation:viewBatch Get specific batch by batchId
PATCH /v1/batches/:batchId foundation:updateBatch Partial update of batch properties
DELETE /v1/batches/:batchId foundation:deleteBatch Soft-delete batch (cascades to jobs and schedules)

Batch Job Management (/src/routes/batchJob.routes.ts)

Method Path Privilege Description
POST /v1/batches/:batchId/jobs foundation:createBatch Create a new job within a batch
GET /v1/batches/:batchId/jobs foundation:viewBatch List jobs for a batch (paginated)
GET /v1/batches/:batchId/jobs/:jobId foundation:viewBatch Get specific job by jobId
PATCH /v1/batches/:batchId/jobs/:jobId foundation:updateBatch Partial update of job properties
DELETE /v1/batches/:batchId/jobs/:jobId foundation:deleteBatch Soft-delete job (cascades to schedules)

Batch Schedule Management (/src/routes/batchSchedule.routes.ts)

Method Path Privilege Description
POST /v1/batches/:batchId/jobs/:jobId/schedules foundation:createBatch Create schedule (EventBridge rule or ONDEMAND trigger)
GET /v1/batches/:batchId/jobs/:jobId/schedules foundation:viewBatch List schedules for a job (paginated)
GET /v1/batches/:batchId/jobs/:jobId/schedules/:scheduleId foundation:viewBatch Get specific schedule by scheduleId
PATCH /v1/batches/:batchId/jobs/:jobId/schedules/:scheduleId foundation:updateBatch Update schedule properties (resyncs EventBridge rule)
DELETE /v1/batches/:batchId/jobs/:jobId/schedules/:scheduleId foundation:deleteBatch Delete schedule (removes EventBridge rule)
POST /v1/schedules/:scheduleId/state foundation:updateBatch Change schedule state (ACTIVE → PAUSED → STOPPED)

Batch Execution (/src/routes/batchExecution.routes.ts)

Method Path Privilege Description
POST /v1/batches/:batchId/jobs/:jobId/execute foundation:executeBatch Trigger job execution (on-demand)
POST /v1/batches/:batchId/jobs/:jobId/history/:runRequestId/status Internal (Lambda callback) Update execution status from Lambda
POST /v1/batches/:batchId/jobs/:jobId/history/:runRequestId/progress Internal (Lambda callback) Update execution progress (chunk-level)
POST /v1/batches/:batchId/jobs/:jobId/history/:runRequestId/stop foundation:executeBatch Stop running execution (Step Functions StopExecution API)
GET /v1/batches/:batchId/jobs/:jobId/history foundation:viewBatchHistory Get execution history (filterable by status, date range)
GET /v1/batches/:batchId/jobs/:jobId/history/:runRequestId foundation:viewBatchHistory Get specific execution by runRequestId
GET /v1/batches/:batchId/jobs/:jobId/metrics foundation:viewBatchHistory Get execution metrics (success rate, avg duration, failure count)

File Upload (/src/routes/fileUpload.routes.ts)

Method Path Privilege Description
POST /v1/batch-uploads foundation:uploadBatchFile Generate presigned S3 URL for file upload
GET /v1/batch-uploads foundation:viewBatchUpload List uploaded files (paginated)
GET /v1/batch-uploads/:uploadId foundation:viewBatchUpload Get specific upload metadata

Available Feeds Summary

PIM Feeds (Product Information Management)

Feed Type Job Type Direction Input Format Output Format Max Concurrency Processing Order
Catalog Import CATALOG_IMPORT Import CSV, XLSX Error CSV/XLSX 20 Unordered
Category Import CATEGORY_IMPORT Import CSV, XLSX Error CSV/XLSX 1 (Sequential) Hierarchical (depth-first)
Price Import PRICE_IMPORT Import CSV, XLSX Error CSV/XLSX 20 Unordered
Catalog Export PIM_CATALOG_EXPORT Export — Excel (multi-sheet) 20 Entity extraction order

Key Characteristics:

  • Chunk Size: 100 rows per chunk (configurable)
  • Entity Extraction Order (Export): 
    Level 1: products, categories, attributes
Level 2: skus, prices, media
Level 3: productAttributes, categoryAssociations
  • Validation: Schema validation + reference validation (categories exist, SKUs exist, etc.)
  • Error Handling: Row-level errors collected, source file regenerated with _importErrors column

Inventory Feeds

Feed Type Job Type Direction Input Format Output Format Max Concurrency Processing Order
Inventory Multi-Sheet Import INVENTORY_MULTI_SHEET_IMPORT Import Excel (multi-sheet) Error Excel 5 Sheet order: locations → SKUs → lots → policies → positions
Inventory Export INVENTORY_EXPORT Export — Excel (9 sheets) 20 Entity extraction order (3 levels)

Entity Extraction Order (Export):

Level 1 (Master):      locations, skus, lots
Level 2 (Policy):      locationChannelPolicy, replenishmentPolicy, reservations
Level 3 (Transactional): inventoryPositions, inventoryTransactions, transfers

Key Characteristics:

  • Database: MySQL (relational) — uses SQL queries with WHERE instance_id = ?
  • Tenant Isolation: SQL-based filtering (vs MongoDB composite IDs in PIM)
  • Chunk Size: 100 rows per entity (configurable)
  • Reference Validation: SKUs validated against master SKU table, locations validated, etc.

GDPR Feeds (Compliance)

Feed Type Job Type Direction Services Output Format Retention Processing Order
GDPR Export GDPR_EXPORT Export User, Payment, Order, List Excel (2 sheets: User, Orders) S3 90-day presigned URL Parallel (User + Orders)
GDPR Delete GDPR_DELETE Delete User, Payment, List, Consent — Retention period (configurable) Sequential (anonymize + hard-delete)

GDPR Export Details:

  • Orchestrator: Foundation Service (Step Functions coordinator)
  • Data Fetchers:
    • gdpr.export.fetchUser → UserServices Lambda (reads userdb_*)
    • gdpr.export.fetchOrders → OMS Lambda (reads omsdb_*)
  • Output Sheets:
    • User Sheet: First Name, Last Name, Email, Phone, Gender, DOB, Status, Addresses, Payment Cards, Custom Properties, Consents, Marketing Preferences
    • Orders Sheet: Order ID, Order Date, Order Status, SKU ID, SKU Name, Quantity, Unit Price, Shipping Method, Item Status, Shipping Charges, Taxes, Discounts, Order Total
  • Trigger: ONDEMAND schedule via UI or API
  • Delivery: Presigned S3 URL (90-day expiry)

GDPR Delete Details:

  • Trigger: Step Functions (scheduled retention check)
  • Retention Period: Configurable days (passed as Step Function input)
  • Actions:
    1. gdpr.delete.initialize — Calculate expiry date, chunk size
    2. gdpr.delete.fetchEligibleUsers — Query users with status=DELETED and deletedAt < expiryDate
    3. gdpr.delete.anonymizeUser — Anonymize PII fields (name, email, phone → hashed values)
    4. gdpr.delete.aggregate — Summarize results (users anonymized, lists deleted)
  • Outcome: User records remain in database with anonymized data; lists hard-deleted

Feed Type Comparison

Dimension PIM Catalog Import Inventory Import GDPR Export GDPR Delete
Database MongoDB MySQL MongoDB (multi-DB) MongoDB (multi-DB)
Tenant Isolation Composite _id SQL WHERE instanceId instanceId filter instanceId filter
Processing Model Distributed Map Sequential + Parallel Parallel branches Sequential chunks
Error Recovery Chunk-level retry Sheet-level retry Full job retry User-level retry
Output File Source format + errors Source format + errors Excel (2 sheets) None (anonymization)
Max Concurrency 20 5 2 (User + Orders) 10 (user chunks)

Events

All events are published to AWS EventBridge. Source: "foundation-service". Bus: EVENT_BUS_NAME env var.
Published via fastify.publishEvent({ type, data }) from @equinox/event-bridge-implementation plugin.

Published Events (EventBridge)

Batch Lifecycle Events

Event Type Trigger Key Data Fields
batch.created Batch creation success batchId, instanceId, name, serviceType, executionType, createdBy, timestamp
batch.updated Batch update batchId, instanceId, updatedFields, updatedBy, timestamp
batch.deleted Batch deletion (soft-delete, cascades to jobs/schedules) batchId, instanceId, deletedBy, timestamp

Batch Job Lifecycle Events

Event Type Trigger Key Data Fields
batch.job.created Job creation success batchId, jobId, instanceId, name, status, createdBy, timestamp
batch.job.updated Job update batchId, jobId, instanceId, updatedFields, updatedBy, timestamp
batch.job.deleted Job deletion (cascades to schedules) batchId, jobId, instanceId, cascadedSchedules, deletedBy, timestamp

Batch Schedule Lifecycle Events

Event Type Trigger Key Data Fields
batch.schedule.created Schedule creation (EventBridge rule created) batchId, jobId, scheduleId, instanceId, name, status, state, repeatInterval, eventBridgeRuleName, createdBy, timestamp
batch.schedule.updated Schedule update (EventBridge rule updated) batchId, jobId, scheduleId, instanceId, updatedFields, eventBridgeRuleName, updatedBy, timestamp
batch.schedule.deleted Schedule deletion (EventBridge rule removed) batchId, jobId, scheduleId, instanceId, eventBridgeRuleName, deletedBy, timestamp
batch.schedule.state_changed Schedule state transition batchId, jobId, scheduleId, instanceId, previousState, newState, eventBridgeRuleName, timestamp

Batch Execution Events

Event Type Trigger Key Data Fields
batch.job.initiated Job execution initiated (ONDEMAND or scheduled) batchId, jobId, runRequestId, instanceId, jobConfig, s3KeyPrefix, callbackBaseUrl, timestamp
batch.execution.started Step Function execution started runRequestId, batchId, jobId, scheduleId, status: "RUNNING", timestamp
batch.execution.completed Execution completed successfully runRequestId, batchId, jobId, status: "COMPLETED", result, duration, timestamp
batch.execution.failed Execution failed (validation, threshold, timeout) runRequestId, batchId, jobId, status: "FAILED", errorMessage, timestamp
batch.execution.threshold_exceeded Error threshold exceeded during execution runRequestId, batchId, jobId, recordsFailed, threshold, timestamp
batch.execution.stopped Execution manually stopped runRequestId, batchId, jobId, status: "STOPPED", stoppedBy, timestamp
batch.execution.max_retries_exhausted All retry attempts failed runRequestId, batchId, jobId, attemptNumber, maxRetries, timestamp
batch.execution.retry_requested Retry triggered by policy runRequestId, batchId, jobId, attemptNumber, retryDelay, timestamp
batch.execution.retry_started Retry Lambda invoked successfully runRequestId, batchId, jobId, attemptNumber, timestamp
batch.execution.retry_failed Retry Lambda invocation failed runRequestId, batchId, jobId, attemptNumber, error, timestamp

Consumed Events (Inbound)

Inbound events arrive via the EventBridge Lambda handler (/src/handlers/eventbridge.handler.ts).

Detail Type Source Handler Action
batch.job.initiated foundation-service (self-emitted) Job Initiator Lambda starts Step Function
app.* (e.g., app.enabled, app.disabled) marketplace-service Marketplace event handler — updates installed connector configurations

Payload Models (Key Events)

All events are wrapped in the EventBridge envelope. The data field below is the detail.data payload.

batch.job.initiated

Published when: ONDEMAND schedule created or scheduled execution triggers.

{
  "type": "batch.job.initiated",
  "data": {
    "batchId": "pim-catalog-import",
    "jobId": "product-full-refresh",
    "runRequestId": "550e8400-e29b-41d4-a716-446655440000",
    "instanceId": "prod-us-east",
    "jobConfig": {
      "jobType": "CATALOG_IMPORT",
      "instanceId": "prod-us-east",
      "catalogId": "master",
      "fileUrl": "s3://bucket/imports/550e8400.../catalog.xlsx",
      "chunkSize": 100,
      "validateMode": "full",
      "mapMaxConcurrency": 20
    },
    "s3KeyPrefix": "imports/550e8400-e29b-41d4-a716-446655440000",
    "callbackBaseUrl": "https://api.equinox.shop/foundationservices",
    "timestamp": "2026-08-07T10:00:00.000Z"
  }
}

batch.execution.completed

Published when: Step Function completes successfully.

{
  "type": "batch.execution.completed",
  "data": {
    "runRequestId": "550e8400-e29b-41d4-a716-446655440000",
    "batchId": "pim-catalog-import",
    "jobId": "product-full-refresh",
    "scheduleId": "daily-midnight-sync",
    "status": "COMPLETED",
    "result": {
      "totalRecords": 5000,
      "successRecords": 5000,
      "failedRecords": 0,
      "totalChunks": 50,
      "processedChunks": 50,
      "failedChunks": 0,
      "outputFiles": ["s3://bucket/results/550e8400.../catalog_result.json"]
    },
    "duration": 180,
    "timestamp": "2026-08-07T10:03:00.000Z"
  }
}

batch.execution.failed

Published when: Execution fails (validation, threshold exceeded, timeout).

{
  "type": "batch.execution.failed",
  "data": {
    "runRequestId": "550e8400-e29b-41d4-a716-446655440000",
    "batchId": "pim-catalog-import",
    "jobId": "product-full-refresh",
    "status": "FAILED",
    "errorMessage": "Validation failed: Required column 'productId' missing from input file",
    "errorCode": "BATCH_VALIDATION_FAILED",
    "timestamp": "2026-08-07T10:01:00.000Z"
  }
}

Feed Architecture Patterns

Common Import Flow

User/UI → Foundation API → S3 Upload → EventBridge → Job Initiator Lambda
                                                           ↓
                                               Step Function (Distributed Map)
                                                           ↓
                            ┌─────────────────────────────┼─────────────────────────────┐
                            ▼                             ▼                             ▼
                      VALIDATE                        PARSE + CHUNK              PROCESS CHUNKS
                  (validate.lambda)                 (parse.lambda)             (process.lambda × N)
                Check file format                  Stream CSV/XLSX             Validate rows
                Verify required columns            Split into chunks           Transform to domain
                Validate file size                 Write to S3 chunks/         Insert to MongoDB/MySQL
                                                   Write manifest.jsonl        Write results to S3
                            ↓                             ↓                             ↓
                            └─────────────────────────────┼─────────────────────────────┘
                                                           ↓
                                                      AGGREGATE
                                                  (aggregate.lambda)
                                                  Collect all results
                                                  Generate error file
                                                  Update history: COMPLETED
                                                           ↓
                                                      FINALIZE
                                                  (finalize.lambda)
                                                  Publish completion event
                                                  Clean up temp files

Key Principles:

  1. Stateless Lambdas: No shared memory or database transactions across execution boundaries
  2. S3-Based Coordination: Raw files, chunk files, result files, error files all stored in S3
  3. Chunk-Level Retry: Failed chunks retry independently without reprocessing successful data
  4. Error Collection: Row-level errors collected, source file regenerated with _importErrors column
  5. Event-Driven: EventBridge decouples services — Foundation publishes, Job Initiator subscribes

Distributed Map Pattern

AWS Step Functions Distributed Map enables high-throughput parallel processing:

{
  "Type": "Map",
  "ItemReader": {
    "Resource": "arn:aws:states:::s3:getObject",
    "ReaderConfig": {
      "InputType": "MANIFEST",
      "CSVHeaderLocation": "FIRST_ROW"
    },
    "Parameters": {
      "Bucket": "batch-bucket",
      "Key.$": "$.manifestKey"
    }
  },
  "MaxConcurrency": 20,
  "ItemProcessor": {
    "ProcessorConfig": {
      "Mode": "DISTRIBUTED",
      "ExecutionType": "STANDARD"
    },
    "StartAt": "ProcessChunk",
    "States": {
      "ProcessChunk": {
        "Type": "Task",
        "Resource": "arn:aws:lambda:us-east-1:123456789012:function:process-chunk",
        "Retry": [
          {
            "ErrorEquals": ["States.ALL"],
            "IntervalSeconds": 2,
            "MaxAttempts": 2,
            "BackoffRate": 2
          }
        ],
        "End": true
      }
    }
  }
}

Configuration:

  • MaxConcurrency: 20 (default) — How many chunks process in parallel
  • Retry: 2 attempts with exponential backoff — Chunk-level failure recovery
  • Manifest File: S3 JSONL file listing all chunk S3 keys
  • Mode: DISTRIBUTED — High-scale processing (up to 10,000 parallel executions)

Batch Execution Flow

ONDEMAND Schedule Flow

Trigger: User clicks “Import Now” in UI or API call with status: "ONDEMAND"

1. POST /v1/batches/:batchId/jobs/:jobId/schedules
   Body: { status: "ONDEMAND", parameters: { ... } }
        ↓
2. batchSchedule.service.ts — createBatchSchedule()
        ├── Validate parent batch + job exist and are ACTIVE
        ├── Validate required parameters (instanceId, fileUrl, etc.)
        ├── Save BatchSchedule record { status: "ONDEMAND", state: "ACTIVE" }
        ├── Detect status === "ONDEMAND"
        └── Immediately call executeBatchJob()
                ↓
3. batchExecution.service.ts — executeEventDrivenJob()
        ├── Generate runRequestId (UUID)
        ├── Save batchHistory record { status: "INITIATED" }
        ├── Enrich jobConfig with s3KeyPrefix, callbackBaseUrl, foundationLambdaArn
        └── Publish EventBridge event: batch.job.initiated
                ↓
4. EventBridge Rule → Job Initiator Lambda (subscribed to batch.job.initiated)
        ↓
5. Job Initiator Lambda
        ├── Validate jobConfig
        ├── Determine Step Function ARN based on jobType
        ├── Start Step Function execution
        └── Return (fire-and-forget)
                ↓
6. Step Function (e.g., CatalogImport.json)
        ├── NormalizeInput (Pass state)
        ├── VALIDATE → Foundation Lambda (action: batch.validate)
        ├── PARSE + CHUNK → Foundation Lambda (action: batch.parseChunk)
        ├── DISTRIBUTED MAP → process.lambda × N (action: batch.processChunk)
        ├── AGGREGATE → Foundation Lambda (action: batch.aggregate)
        └── FINALIZE → Foundation Lambda (action: batch.finalize)
                ├── Update batchHistory { status: "COMPLETED", result: {...} }
                └── Publish EventBridge event: batch.execution.completed

Scheduled Execution Flow

Trigger: EventBridge rule fires based on cron/rate expression

1. EventBridge Rule (e.g., dev-fnd-pim-catalog-import-product-daily)
   Cron: cron(0 10 * * ? *)  — Daily at 10:00 AM UTC
        ↓
2. Target: Foundation Lambda (registered as EventBridge rule target)
   Event Payload: { batchId, jobId, scheduleId, parameters: {...} }
        ↓
3. Foundation Lambda — eventbridge.handler.ts
        ├── Parse incoming EventBridge event
        ├── Extract { batchId, jobId, scheduleId }
        └── Call executeBatchJob(batchId, jobId, scheduleId, parameters)
                ↓
4. batchExecution.service.ts — executeEventDrivenJob()
        └── [Same flow as ONDEMAND starting from step 3 above]

Step Functions Integration

Step Function Definitions

The services that include jobs keep Step Function JSON definitions in the step-functions/ directory:

Step Function File Purpose Max Duration
CatalogImport catalog-import.asl.json PIM catalog import with Distributed Map 1 hour
CategoryImport category-import.asl.json Hierarchical category import (sequential) 30 minutes
PriceImport price-import.asl.json Price list import with date validation 30 minutes
CatalogExport catalog-export.asl.json Multi-sheet PIM catalog export 1 hour
InventoryMultiSheetImport inventory-multi-sheet-import.asl.json Multi-sheet inventory import 1 hour
InventoryExport inventory-export.asl.json Multi-sheet inventory export (9 entities) 1 hour
GdprExport gdpr-export.asl.json GDPR user data export (User + Orders) 15 minutes
GdprDelete gdpr-delete.asl.json GDPR PII anonymization after retention 2 hours

Step Function State Types

State Type Purpose Example
Pass Transform input, add metadata NormalizeInput, PrepareChunkConfig
Choice Conditional branching CheckJobType, CheckInitialization
Task Lambda invocation VALIDATE, PARSE, ProcessChunk
Map Distributed parallel processing DISTRIBUTED MAP (ItemReader: S3 manifest)
Parallel Independent parallel branches FetchData (User branch + Orders branch)
Fail Terminal failure state ValidationFailed, ThresholdExceeded
Succeed Terminal success state SlimOutput

Lambda Action Routing

Foundation Lambda handler (src/handlers/lambda.handler.ts) routes Step Function tasks based on action field:

const action = event.action; // From Step Function Task state Parameters

switch (action) {
  // Import actions
  case "batch.validate":
    return validateHandler(event);
  case "batch.parseChunk":
    return parseChunkHandler(event);
  case "batch.processChunk":
    return processChunkHandler(event);
  case "batch.aggregate":
    return aggregateHandler(event);
  case "batch.finalize":
    return finalizeHandler(event);

  // Export actions
  case "batch.export.initialize":
    return exportInitializeHandler(event);
  case "batch.export.extract":
    return exportExtractHandler(event);
  case "batch.export.buildFile":
    return exportBuildFileHandler(event);
  case "batch.export.finalize":
    return exportFinalizeHandler(event);

  // GDPR actions
  case "gdpr.export.initialize":
    return gdprInitializeHandler(event);
  case "gdpr.export.buildExcel":
    return gdprBuildExcelHandler(event);
  case "gdpr.export.finalize":
    return gdprFinalizeHandler(event);
  case "gdpr.delete.initialize":
    return gdprDeleteInitHandler(event);
  case "gdpr.delete.anonymizeUser":
    return gdprAnonymizeUserHandler(event);
  case "gdpr.delete.aggregate":
    return gdprDeleteAggregateHandler(event);

  default:
    throw new Error(`Unknown action: ${action}`);
}

Error Handling & Retry

Retry Policy Configuration

Batch-level retry policy (default for all jobs in the batch):

{
  "retryPolicy": {
    "maxRetries": 3,
    "retryDelay": 60,
    "backoffMultiplier": 2,
    "retryStrategy": "exponential"
  }
}

Job-level retry policy (overrides batch-level):

{
  "jobParams": {
    "retryPolicy": {
      "maxRetries": 5,
      "retryDelay": 30,
      "backoffMultiplier": 1.5,
      "retryStrategy": "fixed"
    }
  }
}

Retry Strategies:

  • exponential — Delay doubles on each retry (60s → 120s → 240s)
  • fixed — Constant delay between retries (60s → 60s → 60s)
  • circuitBreaker — Stop retrying after threshold failures (prevents cascading failures)

Error Threshold & Toleration Policy

Error Threshold: Percentage of failures allowed before job fails overall.

{
  "errorThreshold": 10, // 10% of chunks can fail
  "tolerationPolicy": "LENIENT" // Continue processing remaining chunks
}

Example:

  • Total chunks: 100
  • Threshold: 10%
  • Maximum allowed failures: 10 chunks
  • If 11th chunk fails → Job status: FAILED, EventBridge event: batch.execution.threshold_exceeded

Toleration Policies:

  • STRICT — Fail immediately on first chunk error (default for critical data)
  • LENIENT — Continue processing remaining chunks, collect all errors (default for imports)

Error File Generation

For import jobs, Foundation Service generates an error file in the same format as the input file with an additional _importErrors column:

Original File (catalog.csv):

productId,name,status,categoryCode
PRD-001,T-Shirt,ACTIVE,CAT-TOPS
PRD-002,,ACTIVE,CAT-TOPS
PRD-003,Jeans,ACTIVE,CAT-NONE

Error File (catalog_errors.csv):

productId,name,status,categoryCode,_importErrors
PRD-002,,ACTIVE,CAT-TOPS,"name: Required field is missing"
PRD-003,Jeans,ACTIVE,CAT-NONE,"categoryCode: Category 'CAT-NONE' does not exist"

Error File Location:

  • S3 path: s3://bucket/results/{runRequestId}/catalog_errors.csv
  • Available via: GET /v1/batches/.../history/{runRequestId} → result.errorFileUrl
  • Presigned URL: 7-day expiry (configurable)

FAQs

Q: What is the difference between a Batch, Job, and Schedule?

Batch is a logical grouping (e.g., “PIM Import Batch”). Job is an executable unit within a batch (e.g., “Product Full Refresh”). Schedule is a trigger mechanism for the job (cron, rate, or on-demand). One batch can have multiple jobs; one job can have multiple schedules.

Q: How does ONDEMAND execution work without EventBridge?

When status: "ONDEMAND" is specified during schedule creation, Foundation Service creates the schedule record but skips EventBridge rule creation. Instead, it immediately calls executeBatchJob() and returns the runRequestId. This allows UI-triggered imports/exports without waiting for the next scheduled run.

Q: What happens if a job fails midway through chunk processing?

When tolerationPolicy: "LENIENT", remaining chunks continue processing and an error file is generated with failed rows. When tolerationPolicy: "STRICT", the job stops immediately on first chunk failure with status FAILED.

Q: How does retry work for failed executions?

The retry policy (configured at batch or job level) determines retry behavior. After a job fails, Foundation Service checks the policy: if attemptNumber < maxRetries, it publishes a batch.execution.retry_requested event, waits for retryDelay seconds (exponential backoff if configured), then re-invokes the Step Function with attemptNumber incremented.

Q: Can I import categories and products in the same feed?

Yes, using PIM Multi-Sheet Import. Categories must be in one sheet, products in another. Step Function processes sheets in order: categories first (with sequential MaxConcurrency=1 for hierarchy), then products (with MaxConcurrency=20). This ensures parent categories exist before child products reference them.

Q: How does Foundation Service handle multi-tenant isolation?

Every API requires instanceId in the query or body. MongoDB queries filter by instanceId. S3 file paths include instanceId in the prefix (s3://bucket/imports/{instanceId}/{runRequestId}/...). EventBridge events include instanceId in the payload. Job Initiator Lambda validates instanceId before starting Step Functions.

Q: What is the maximum file size for imports?

The validation step checks file size. Default limits:

  • CSV/Excel imports: 100MB (configurable via batch maxFileSize property)
  • Multi-sheet imports: 50MB per sheet (total 200MB for 4 sheets)
  • Larger files require chunked upload with multipart S3 API (not currently supported)

Q: How long are execution history records retained?

Batch history records have a 90-day TTL (Time To Live). MongoDB automatically deletes records older than 90 days. This ensures compliance with data retention policies and prevents unbounded database growth.

Q: Can I cancel a running execution?

Yes, via POST /v1/batches/:batchId/jobs/:jobId/history/:runRequestId/stop. Foundation Service calls AWS Step Functions StopExecution API, which terminates the execution. Status updates to STOPPED. Partial results (already processed chunks) remain in S3.

Glossary

Term Definition
Batch Logical grouping of related jobs (e.g., “PIM Import Batch”). Configuration includes execution type, parallel execution flag, max runtime.
BatchJob Executable unit within a batch (e.g., “Product Full Refresh”). Contains job parameters, retry policy, error threshold.
BatchSchedule Trigger mechanism for job execution. Types: SCHEDULED (cron/rate with EventBridge rule) or ONDEMAND (immediate execution).
runRequestId Unique UUID identifying a single execution instance. Used to track status, retrieve results, and correlate logs.
jobType Feed type identifier (e.g., CATALOG_IMPORT, INVENTORY_EXPORT, GDPR_EXPORT). Determines which Step Function to invoke.
instanceId Tenant identifier. All queries, S3 paths, and EventBridge events are scoped to instanceId for multi-tenant isolation.
jobParams Flat key-value object on BatchJob containing execution parameters (catalogId, fileUrl, chunkSize, etc.). Schedule parameters override job parameters.
eventBridgeRuleName Auto-generated name for EventBridge schedule rule. Format: {stage}-fnd-{batchId}-{jobId}-{scheduleId}.
Distributed Map AWS Step Functions feature for high-throughput parallel processing. Reads S3 manifest file, spawns up to 10,000 concurrent executions.
Chunk Subset of input file rows (default: 100 rows). Each chunk processed independently by Lambda function. Failed chunks retry without reprocessing successful chunks.
s3KeyPrefix S3 path prefix for job execution files. Format: imports/{runRequestId} or exports/{runRequestId} or gdpr-temp/{runRequestId}.
errorThreshold Percentage of chunks allowed to fail before overall job fails. Example: 10% threshold = 10 failed chunks out of 100 total.
tolerationPolicy Error handling strategy. STRICT = fail immediately on first error; LENIENT = continue processing, collect all errors.
retryPolicy Configuration for automatic retry on failure. Includes maxRetries, retryDelay, backoffMultiplier, retryStrategy.
ONDEMAND Schedule status for immediate one-time execution. No EventBridge rule created; executeBatchJob() called directly.
INITIATED Initial execution status when job record created in MongoDB. Transitions to RUNNING when Step Function starts.
callbackBaseUrl Foundation Service API base URL (e.g., https://api.equinox.shop/foundationservices). Lambda functions call back to update status and progress.
Job Initiator Lambda Lambda function subscribed to batch.job.initiated EventBridge events. Starts appropriate Step Function based on jobType.
entityExtractionOrder (Export only) Array of entity types defining extraction order. Example: ["products", "categories", "skus", "prices"].
exportConfig (Export only) Configuration object for export behavior. Includes sheets array, locale, onlyAttributes, filters, etc.
manifest.jsonl S3 file listing all chunk S3 keys (JSONL format, one key per line). Used by Distributed Map ItemReader.

Common Execution Sequence Diagram

Framework: Fastify + TypeScript + AWS Step Functions

sequenceDiagram
    autonumber
    participant User
    participant FoundationAPI
    participant MongoDB
    participant EventBridge
    participant JobInitiator
    participant StepFunction
    participant Lambda
    participant S3

    User->>FoundationAPI: POST /v1/batches/:batchId/jobs/:jobId/schedules<br/>{status: "ONDEMAND", parameters: {...}}
    FoundationAPI->>MongoDB: Save BatchSchedule record
    FoundationAPI->>FoundationAPI: Detect ONDEMAND → executeBatchJob()
    FoundationAPI->>MongoDB: Save batchHistory {status: "INITIATED"}
    FoundationAPI->>EventBridge: Publish batch.job.initiated<br/>{runRequestId, jobConfig, s3KeyPrefix}
    FoundationAPI-->>User: 202 Accepted<br/>{runRequestId, status: "INITIATED"}
    EventBridge->>JobInitiator: EventBridge Rule triggers<br/>Job Initiator Lambda
    JobInitiator->>StepFunction: StartExecution(stateMachineArn, input)
    StepFunction->>Lambda: VALIDATE action
    Lambda->>S3: Read first 8KB (header check)
    Lambda-->>StepFunction: {valid: true}
    StepFunction->>Lambda: PARSE + CHUNK action
    Lambda->>S3: Stream file, write chunks/
    Lambda->>S3: Write manifest.jsonl
    Lambda-->>StepFunction: {totalChunks: 50, manifestKey}
    StepFunction->>Lambda: DISTRIBUTED MAP (20 concurrent)<br/>ProcessChunk action × 50
    loop For each chunk
        Lambda->>S3: Read chunk file
        Lambda->>Lambda: Validate rows, transform to domain
        Lambda->>MongoDB: bulkWrite (insert/update)
        Lambda->>S3: Write result file
    end
    Lambda-->>StepFunction: All chunks complete
    StepFunction->>Lambda: AGGREGATE action
    Lambda->>S3: Read all result files
    Lambda->>S3: Generate error file (if failures)
    Lambda-->>StepFunction: {errorFileUrl, totalSuccess, totalFailed}
    StepFunction->>Lambda: FINALIZE action
    Lambda->>MongoDB: Update batchHistory<br/>{status: "COMPLETED", result, duration}
    Lambda->>EventBridge: Publish batch.execution.completed
    Lambda-->>StepFunction: Success
    StepFunction-->>JobInitiator: Execution complete
    User->>FoundationAPI: GET /v1/batches/.../history/:runRequestId
    FoundationAPI->>MongoDB: Query batchHistory
    MongoDB-->>FoundationAPI: {status: "COMPLETED", result: {...}}
    FoundationAPI-->>User: 200 OK<br/>{status, result, errorFileUrl}

MongoDB Collections

Database: foundationdb_<STAGE> (e.g., foundationdb_dev).

Collection: batches

Repository: src/repository/batch.repository.ts

Field Type Description
_id ObjectId Auto-generated MongoDB ID
batchId string Unique batch identifier (kebab-case)
instanceId string Tenant identifier (multi-tenancy)
businessId string Business identifier (deprecated, use instanceId)
name string Human-readable batch name
description string Batch purpose and usage notes
serviceType string Service domain (e.g., pim-import, inventory-export)
executionType string LAMBDA, STEP_FUNCTION, SPRING_BATCH, GLUE
jobEndpoint string Lambda ARN or API endpoint for execution
allowParallelExecution boolean false prevents concurrent executions for same job
maxRuntime number Maximum execution time in seconds
executionCluster string Execution environment identifier
jobInformation object Additional job metadata (legacy, use jobParams)
metadata object Custom key-value pairs for batch metadata
isDeleted boolean Soft-delete flag
deletedAt Date Soft-delete timestamp
createdAt Date Creation timestamp
updatedAt Date Last update timestamp
createdBy string User identifier
updatedBy string User identifier

Indexes:

  • { batchId: 1, instanceId: 1 } — Unique compound index
  • { instanceId: 1, isDeleted: 1 } — List query optimization

Collection: batch_jobs

Repository: src/repository/batchJob.repository.ts

Field Type Description
_id ObjectId Auto-generated MongoDB ID
batchId string Parent batch identifier
jobId string Unique job identifier within batch
instanceId string Tenant identifier
businessId string Business identifier (deprecated)
name string Human-readable job name
description string Job purpose and usage notes
status string ACTIVE (can execute) or INACTIVE (disabled)
jobParams object Flat key-value object with execution parameters
Required keys: jobType, instanceId
Optional keys: catalogId, fileUrl, chunkSize, etc.
retryPolicy object { maxRetries, retryDelay, backoffMultiplier, retryStrategy }
errorThreshold number Percentage of failures allowed (0-100)
tolerationPolicy string STRICT (fail fast) or LENIENT (collect errors)
metadata object Custom key-value pairs
isDeleted boolean Soft-delete flag (cascades from parent batch)
deletedAt Date Soft-delete timestamp
createdAt Date Creation timestamp
updatedAt Date Last update timestamp
createdBy string User identifier
updatedBy string User identifier

Indexes:

  • { batchId: 1, jobId: 1, instanceId: 1 } — Unique compound index
  • { instanceId: 1, status: 1, isDeleted: 1 } — List query optimization

Collection: batch_schedules

Repository: src/repository/batchSchedule.repository.ts

Field Type Description
_id ObjectId Auto-generated MongoDB ID
batchId string Parent batch identifier
jobId string Parent job identifier
scheduleId string Unique schedule identifier within job
instanceId string Tenant identifier
businessId string Business identifier (deprecated)
name string Human-readable schedule name
description string Schedule purpose
status string SCHEDULED (cron/rate) or ONDEMAND (manual)
state string ACTIVE, PAUSED, STOPPED (lifecycle state)
repeatInterval string EventBridge cron/rate expression
(null for ONDEMAND)
startDate Date Schedule validity start (null = immediate)
endDate Date Schedule validity end (null = no expiry)
eventBridgeRuleName string Auto-generated EventBridge rule name
lastRunTime Date Most recent execution timestamp
nextRunTime Date Next scheduled execution timestamp
parameters object Schedule-specific parameters (override jobParams)
metadata object Custom key-value pairs
isDeleted boolean Soft-delete flag (cascades from parent job)
deletedAt Date Soft-delete timestamp
createdAt Date Creation timestamp
updatedAt Date Last update timestamp
createdBy string User identifier
updatedBy string User identifier

Indexes:

  • { batchId: 1, jobId: 1, scheduleId: 1, instanceId: 1 } — Unique compound index
  • { instanceId: 1, state: 1, isDeleted: 1 } — List query optimization

Collection: batch_history

Repository: src/repository/batchHistory.repository.ts

Field Type Description
_id ObjectId Auto-generated MongoDB ID
runRequestId string Unique execution identifier (UUID)
batchId string Parent batch identifier
jobId string Parent job identifier
scheduleId string Parent schedule identifier (null for on-demand)
instanceId string Tenant identifier
businessId string Business identifier (deprecated)
status string INITIATED → PENDING → RUNNING → COMPLETED/FAILED/STOPPED
jobConfig object Complete job configuration (merged jobParams + schedule parameters)
totalChunks number Total number of chunks (set after PARSE)
processedChunks number Successfully processed chunks
failedChunks number Failed chunks
progress number Percentage complete (0-100)
startTime Date Execution start timestamp
endTime Date Execution end timestamp
duration number Execution duration in seconds
result object Execution result (success output, file URLs, statistics)
errorMessage string Failure details (validation errors, exceptions)
payload object Original execution request payload (deprecated)
executionType string event-driven (Step Functions) or direct (Lambda)
triggeredBy string User identifier who triggered execution
attemptNumber number Retry attempt number (1 = first attempt)
ttl Date Time to live (90 days from endTime, auto-delete)
createdAt Date Creation timestamp
updatedAt Date Last update timestamp

Indexes:

  • { runRequestId: 1 } — Unique index for lookup
  • { batchId: 1, jobId: 1, status: 1 } — List query optimization
  • { instanceId: 1, status: 1, startTime: -1 } — History pagination
  • { ttl: 1 } — TTL index for automatic cleanup (expireAfterSeconds: 0)

Collection: batch_uploads

Repository: src/repository/batchUpload.repository.ts

Field Type Description
_id ObjectId Auto-generated MongoDB ID
uploadId string Unique upload identifier (UUID)
instanceId string Tenant identifier
businessId string Business identifier (deprecated)
fileName string Original file name
fileSize number File size in bytes
fileType string MIME type (e.g., text/csv, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
s3Key string S3 object key (path)
s3Bucket string S3 bucket name
presignedUrl string Presigned PUT URL for upload
urlExpiry Date Presigned URL expiration timestamp (15 minutes)
uploadStatus string PENDING, UPLOADING, COMPLETED, FAILED
metadata object Custom key-value pairs (batchId, jobId, etc.)
createdAt Date Creation timestamp
updatedAt Date Last update timestamp
createdBy string User identifier

Indexes:

  • { uploadId: 1 } — Unique index for lookup
  • { instanceId: 1, uploadStatus: 1, createdAt: -1 } — List query optimization

Configuration & Properties

Environment Variables

Variable Type Default Description
MONGODB_URI string Required MongoDB connection string (shared across services)
MONGODB_DATABASE string foundationdb_{STAGE} Database name
STAGE string development Environment stage (dev, staging, prod)
EVENT_BUS_NAME string Required AWS EventBridge bus name
ENABLE_EVENT_BRIDGE boolean true Enable EventBridge event publishing
S3_BUCKET_NAME string Required S3 bucket for batch file uploads and processing
BATCH_UPLOAD_BUCKET string Same as S3_BUCKET_NAME Legacy alias
FOUNDATION_LAMBDA_ARN string Required Foundation Lambda ARN for Step Functions
JOB_INITIATOR_LAMBDA_ARN string Required Job Initiator Lambda ARN for EventBridge target
STEP_FUNCTION_CATALOG_IMPORT_ARN string Required Catalog Import Step Function ARN
STEP_FUNCTION_INVENTORY_EXPORT_ARN string Required Inventory Export Step Function ARN
STEP_FUNCTION_GDPR_EXPORT_ARN string Required GDPR Export Step Function ARN
REGISTERED_PROCESSORS string CSV list Comma-separated list of registered processor aliases (e.g., pim-product-import,inventory-sync)
SKIP_LAMBDA_INVOCATION boolean false Skip Lambda invocation (dev/test only)
CALLBACK_BASE_URL string Auto-detect Foundation Service API base URL for Lambda callbacks
GDPR_EXPORT_PRESIGNED_URL_EXPIRY number 86400 GDPR export presigned URL expiry in seconds (90 days = 7776000)

Batch Configuration Templates

Foundation Service provides default batch configurations for common feed types. Seed data (Webbox migrations) pre-populates these templates.

Example: PIM Catalog Import Batch

{
  "batchId": "pim-catalog-import",
  "name": "PIM Catalog Import",
  "serviceType": "pim-import",
  "executionType": "STEP_FUNCTION",
  "allowParallelExecution": false,
  "maxRuntime": 3600,
  "jobEndpoint": "${FOUNDATION_LAMBDA_ARN}",
  "jobRequiredFields": ["jobType", "instanceId", "catalogId", "fileUrl"]
}

Example: Inventory Export Batch

{
  "batchId": "inventory-export",
  "name": "Inventory Export",
  "serviceType": "inventory-export",
  "executionType": "STEP_FUNCTION",
  "allowParallelExecution": true,
  "maxRuntime": 3600,
  "jobEndpoint": "${FOUNDATION_LAMBDA_ARN}",
  "jobRequiredFields": ["jobType", "instanceId"]
}

END OF DOCUMENT

Document Revision History

  • v1.0 (2026-08-07): Initial comprehensive feed services documentation
  • Based on eq-foundation-service v9.0 codebase
  • Modeled after EQ_USER_SERVICE_DOCUMENTATION 4.md structure
↑

Revision History
2026-08-12 | JP – Created the page and added the content.