Data

/laravel-agent:import:make

Create robust CSV/Excel importers with validation, chunking, and error handling

Overview

The /import:make command generates production-ready CSV/Excel importers with comprehensive features including validation, chunking for large files, error handling, and progress tracking. It supports both Maatwebsite Excel (full-featured, Laravel-specific) and Spatie Simple Excel (lightweight, streaming) packages.

This command creates a complete import system with:

  • Row-by-row validation with custom error messages
  • Chunked processing for memory-efficient handling of large files
  • Batch inserts for improved database performance
  • Error tracking with detailed failure reporting
  • Queued imports for background processing of massive files
  • Upsert support for update-or-create behavior
  • Upload UI with template downloads and error display

Usage

/laravel-agent:import:make <ImporterName> [--model=<Model>] [--package=<maatwebsite|spatie>]

Examples

# Create a product importer with validation
/laravel-agent:import:make ProductImport --model=Product

# Create a lightweight user importer using Spatie
/laravel-agent:import:make UserImport --package=spatie

# Create an order importer with queued processing
/laravel-agent:import:make OrderImport --model=Order

Supported File Formats

Format Extension Maatwebsite Spatie Notes
CSV .csv Best for large files, universal support
Excel 2007+ .xlsx Modern Excel format, multiple sheets
Excel Legacy .xls Older format, limited support

What Gets Created

A complete importer system includes the following components:

Component Location Description
Import Class app/Imports/ Core import logic with validation and row processing
Controller app/Http/Controllers/ Handles file uploads and triggers import
Form Request app/Http/Requests/ Validates file uploads (size, type, permissions)
Upload View resources/views/ User interface with file picker and error display
Queue Job app/Jobs/ Background processing for large files (optional)

Package Comparison

Feature Maatwebsite Excel Spatie Simple Excel
Laravel Integration ✅ Deep integration ⚠️ Manual setup
Built-in Validation ✅ Yes ❌ Manual
Queue Support ✅ Built-in ⚠️ Manual
Memory Usage ⚠️ Higher ✅ Lower (streaming)
Learning Curve ⚠️ Moderate ✅ Simple
Best For Complex imports, full features Simple imports, large files

Example Output Structure

For /laravel-agent:import:make ProductImport --model=Product:

app/
├── Imports/
│   └── ProductImport.php
├── Http/
│   ├── Controllers/
│   │   └── ProductImportController.php
│   └── Requests/
│       └── ProductImportRequest.php
└── Jobs/
    └── ProcessProductImport.php
resources/views/
└── products/
    └── import.blade.php
routes/
└── web.php (updated)

Generated Import Class (Maatwebsite)

<?php

declare(strict_types=1);

namespace App\Imports;

use App\Models\Product;
use Maatwebsite\Excel\Concerns\ToModel;
use Maatwebsite\Excel\Concerns\WithHeadingRow;
use Maatwebsite\Excel\Concerns\WithValidation;
use Maatwebsite\Excel\Concerns\SkipsOnFailure;
use Maatwebsite\Excel\Concerns\SkipsFailures;
use Maatwebsite\Excel\Concerns\WithBatchInserts;
use Maatwebsite\Excel\Concerns\WithChunkReading;

final class ProductImport implements
    ToModel,
    WithHeadingRow,
    WithValidation,
    SkipsOnFailure,
    WithBatchInserts,
    WithChunkReading
{
    use SkipsFailures;

    public function model(array $row): ?Product
    {
        return new Product([
            'name' => $row['name'],
            'sku' => $row['sku'],
            'description' => $row['description'] ?? null,
            'price_cents' => (int) ($row['price'] * 100),
            'stock' => (int) $row['stock'],
            'category_id' => $this->findCategoryId($row['category']),
        ]);
    }

    public function rules(): array
    {
        return [
            'name' => ['required', 'string', 'max:255'],
            'sku' => ['required', 'string', 'unique:products,sku'],
            'price' => ['required', 'numeric', 'min:0'],
            'stock' => ['required', 'integer', 'min:0'],
            'category' => ['required', 'string'],
        ];
    }

    public function customValidationMessages(): array
    {
        return [
            'sku.unique' => 'SKU :input already exists.',
        ];
    }

    public function batchSize(): int
    {
        return 1000;
    }

    public function chunkSize(): int
    {
        return 1000;
    }

    private function findCategoryId(string $categoryName): ?int
    {
        return \App\Models\Category::firstWhere('name', $categoryName)?->id;
    }
}

Generated Controller Example

<?php

declare(strict_types=1);

namespace App\Http\Controllers;

use App\Imports\ProductImport;
use App\Http\Requests\ProductImportRequest;
use Illuminate\Http\RedirectResponse;
use Illuminate\View\View;
use Maatwebsite\Excel\Facades\Excel;

final class ProductImportController extends Controller
{
    public function create(): View
    {
        return view('products.import');
    }

    public function store(ProductImportRequest $request): RedirectResponse
    {
        $import = new ProductImport();

        Excel::import($import, $request->file('file'));

        $failures = $import->failures();

        if ($failures->isNotEmpty()) {
            return back()
                ->with('warning', "Imported with {$failures->count()} errors.")
                ->with('failures', $failures);
        }

        return redirect()
            ->route('products.index')
            ->with('success', 'Products imported successfully.');
    }

    /**
     * Queued import for large files.
     */
    public function storeQueued(ProductImportRequest $request): RedirectResponse
    {
        $import = new ProductQueuedImport(auth()->id());

        Excel::queueImport($import, $request->file('file'));

        return redirect()
            ->route('products.index')
            ->with('info', 'Import started. You will be notified when complete.');
    }
}

Validation Patterns

The importer includes comprehensive validation rules:

Basic Validation

public function rules(): array
{
    return [
        'name' => ['required', 'string', 'max:255'],
        'sku' => ['required', 'string', 'unique:products,sku'],
        'email' => ['required', 'email', 'unique:users,email'],
        'price' => ['required', 'numeric', 'min:0'],
        'stock' => ['required', 'integer', 'min:0'],
        'status' => ['required', 'in:active,inactive'],
        'published_at' => ['nullable', 'date'],
    ];
}

Custom Error Messages

public function customValidationMessages(): array
{
    return [
        'sku.unique' => 'SKU :input already exists in the database.',
        'email.unique' => 'Email :input is already registered.',
        'price.min' => 'Price must be at least 0.',
    ];
}

Conditional Validation

public function rules(): array
{
    return [
        'type' => ['required', 'in:physical,digital'],
        'weight' => [
            'required_if:type,physical',
            'nullable',
            'numeric',
            'min:0',
        ],
        'download_url' => [
            'required_if:type,digital',
            'nullable',
            'url',
        ],
    ];
}

Large File Handling

For files with thousands of rows, use queued imports for background processing:

Queued Import Class

<?php

declare(strict_types=1);

namespace App\Imports;

use App\Models\Product;
use Maatwebsite\Excel\Concerns\ToModel;
use Maatwebsite\Excel\Concerns\WithHeadingRow;
use Maatwebsite\Excel\Concerns\WithChunkReading;
use Maatwebsite\Excel\Concerns\ShouldQueue;
use Illuminate\Contracts\Queue\ShouldQueue as ShouldQueueContract;
use Illuminate\Bus\Queueable;

final class ProductQueuedImport implements
    ToModel,
    WithHeadingRow,
    WithChunkReading,
    ShouldQueue,
    ShouldQueueContract
{
    use Queueable;

    public function __construct(
        public readonly int $userId,
    ) {}

    public function model(array $row): ?Product
    {
        return new Product([
            'name' => $row['name'],
            'sku' => $row['sku'],
            'price_cents' => (int) ($row['price'] * 100),
            'created_by_id' => $this->userId,
        ]);
    }

    public function chunkSize(): int
    {
        return 500; // Process 500 rows at a time
    }
}

Memory-Efficient Chunking (Spatie)

<?php

use Spatie\SimpleExcel\SimpleExcelReader;

SimpleExcelReader::create($filePath)
    ->useDelimiter(',')
    ->getRows()
    ->chunk(1000)
    ->each(function ($chunk) {
        $data = $chunk->map(fn ($row) => [
            'name' => $row['name'],
            'sku' => $row['sku'],
            'price_cents' => (int) ($row['price'] * 100),
            'created_at' => now(),
            'updated_at' => now(),
        ])->toArray();

        Product::insert($data); // Bulk insert
    });

Upsert Support (Update or Create)

Use the WithUpserts concern to update existing records:

<?php

declare(strict_types=1);

namespace App\Imports;

use App\Models\Product;
use Maatwebsite\Excel\Concerns\ToModel;
use Maatwebsite\Excel\Concerns\WithHeadingRow;
use Maatwebsite\Excel\Concerns\WithUpserts;

final class ProductUpsertImport implements ToModel, WithHeadingRow, WithUpserts
{
    public function model(array $row): ?Product
    {
        return new Product([
            'sku' => $row['sku'],
            'name' => $row['name'],
            'price_cents' => (int) ($row['price'] * 100),
            'stock' => (int) $row['stock'],
        ]);
    }

    public function uniqueBy(): string
    {
        return 'sku'; // Update existing records by SKU
    }
}

Error Handling

The importer tracks validation failures and allows partial imports:

Displaying Errors in View

@if(session('failures'))
    <div class="mt-4 p-4 bg-yellow-50 border border-yellow-200 rounded">
        <h3 class="font-bold">Import Errors</h3>
        <ul class="list-disc pl-5 mt-2">
            @foreach(session('failures') as $failure)
                <li>
                    Row :
                    
                </li>
            @endforeach
        </ul>
    </div>
@endif

Custom Error Handling

use Maatwebsite\Excel\Concerns\SkipsOnError;
use Maatwebsite\Excel\Concerns\SkipsErrors;

final class ProductImport implements SkipsOnError
{
    use SkipsErrors;

    public function onError(\Throwable $e)
    {
        // Log the error
        \Log::error('Import error: ' . $e->getMessage());

        // Notify admin
        \Notification::route('mail', 'admin@example.com')
            ->notify(new ImportErrorNotification($e));
    }
}

Progress Tracking

Track import progress for user feedback:

use Maatwebsite\Excel\Concerns\WithProgressBar;

final class ProductImport implements WithProgressBar
{
    // Progress bar automatically shown in console
}

Expected CSV Format

For a Product importer, the CSV should have headers matching the validation rules:

name,sku,description,price,stock,category
"Widget Pro","WGT-001","Premium widget",29.99,100,"Electronics"
"Gadget Max","GDG-002","Advanced gadget",49.99,50,"Electronics"
"Tool Set","TLS-003","Complete tool set",79.99,25,"Tools"

Interactive Prompts

When run without arguments, the command prompts for:

  1. Import name - The class name (e.g., ProductImport)
  2. Target model - Select from existing models or specify new
  3. Package preference - Maatwebsite Excel or Spatie Simple Excel
  4. Import behavior - Create only, Upsert, or Replace all
  5. File types to support - CSV, XLSX, XLS
  6. Large file support - Enable queued processing for massive files

Best Practices

  1. Always validate input - Never trust uploaded data; use comprehensive validation rules
  2. Use chunking for large files - Process in batches to avoid memory exhaustion
  3. Queue large imports - Don't block user requests; use background jobs for files with 1000+ rows
  4. Provide template downloads - Give users a sample CSV with correct headers and format
  5. Skip on failure - Allow partial imports instead of failing completely on first error
  6. Log import activity - Track who imported what and when for audit trails
  7. Set file size limits - Prevent abuse with reasonable upload size restrictions (e.g., 10MB)
  8. Use batch inserts - Improve performance with WithBatchInserts concern
  9. Handle relationships carefully - Resolve foreign keys before creating records
  10. Test with real data - Use actual customer data (anonymized) to catch edge cases

Performance Considerations

File Size Rows Recommended Approach Chunk Size
< 1 MB < 1,000 Synchronous import No chunking needed
1-5 MB 1,000-5,000 Synchronous with chunking 1,000 rows
5-20 MB 5,000-20,000 Queued import 500-1,000 rows
> 20 MB > 20,000 Queued with streaming 500 rows

Routes Added

The command automatically adds routes to routes/web.php:

use App\Http\Controllers\ProductImportController;

Route::middleware(['auth'])->group(function () {
    Route::get('/products/import', [ProductImportController::class, 'create'])
        ->name('products.import.create');

    Route::post('/products/import', [ProductImportController::class, 'store'])
        ->name('products.import.store');

    Route::get('/products/import/template', [ProductImportController::class, 'template'])
        ->name('products.import.template');
});

Related Commands