Take the Pain Out of Data Imports with Laravel Ingest

Laravel Ingest by Robin Kopp is a configuration-driven ETL (Extract, Transform, Load) package that replaces one-off import scripts with declarative importer classes. It handles files from a few hundred to tens of millions of rows by processing them through PHP Generators and Laravel Queues, keeping memory usage consistent regardless of file size.

Main Features

• Declarative importer classes using a fluent IngestConfig builder
• Automatic resolution of BelongsTo and BelongsToMany relationships
• Duplicate handling strategies: SKIP, CREATE, UPDATE, and UPDATE_IF_NEWER
• Dry-run mode to validate imports before writing to the database
• Failed row tracking with downloadable CSV exports
• Column aliasing to map varying header names to a single field
• Dynamic model resolution based on row data
• Import sources: file upload, filesystem disks (including S3), URL, FTP, and SFTP
• Auto-generated Artisan commands and REST API endpoints per importer

Defining an Importer

After installing the package and running migrations, you create an importer class that implements IngestDefinition and returns an IngestConfig. By convention, these live in the App\Ingest namespace:

namespace App\Ingest;
 
use App\Models\Product;
use LaravelIngest\Contracts\IngestDefinition;
use LaravelIngest\DTOs\IngestConfig;
use LaravelIngest\Enums\DuplicateStrategy;
use LaravelIngest\Enums\SourceType;
 
class ProductImporter implements IngestDefinition
{
 public function getConfig(): IngestConfig
 {
 return IngestConfig::for(Product::class)
 ->fromSource(SourceType::UPLOAD)
 ->keyedBy('sku')
 ->onDuplicate(DuplicateStrategy::UPDATE)
 ->map('Product Name', 'name')
 ->relate('Category', 'category', Category::class, 'slug')
 ->validate([
 'sku' => 'required|string',
 'Product Name' => 'required|string|min:3',
 ]);
 }
}

Register the importer in your AppServiceProvider using the package’s tag:

use LaravelIngest\IngestServiceProvider;
 
$this->app->tag([ProductImporter::class], IngestServiceProvider::INGEST_DEFINITION_TAG);

Running Imports

Once registered, the package exposes both an Artisan command and an HTTP endpoint for each importer.

Via CLI:

php artisan ingest:run product-importer --file=products.csv

Via API (multipart form upload):

POST /api/v1/ingest/upload/product-importer

For dry runs, append the --dry-run flag to the Artisan command to validate the file and surface any errors without touching the database.

Monitoring

The package includes several Artisan commands for checking on running or completed imports:

php artisan ingest:list # List registered importers
php artisan ingest:status {id} # Show progress and row statistics
php artisan ingest:cancel {id} # Stop an in-progress import
php artisan ingest:retry {id} # Reprocess only the failed rows

Equivalent REST endpoints are also available:

• GET /api/v1/ingest — recent runs
• GET /api/v1/ingest/{id} — status and statistics
• GET /api/v1/ingest/{id}/errors/summary — aggregated error breakdown
• GET /api/v1/ingest/{id}/failed-rows/download — CSV of rows that failed

Events

The package dispatches events throughout the import lifecycle — IngestRunStarted, ChunkProcessed, RowProcessed, IngestRunCompleted, and IngestRunFailed — which you can listen to for notifications or custom side effects.

You can find Laravel Ingest on GitHub and read the full documentation at the Laravel Ingest docs.