Getting Started

This tutorial walks through the complete workflow of creating a blob archive, pushing it to an OCI registry, and reading files lazily via HTTP range requests.

Prerequisites

Go 1.21 or later
Access to an OCI registry (Docker Hub, ghcr.io, or a local registry)
Docker configured with registry credentials (for WithDockerConfig())

What We Will Build

We will create a simple program that:

Creates and pushes an archive in a single call
Inspects archive metadata without downloading data
Pulls the archive and reads files lazily
Adds caching for improved performance

Step 1: Create a Project

Create a new directory and initialize a Go module:

mkdir blob-demo && cd blob-demo
go mod init blob-demo
go get github.com/meigma/blob

Create a main.go file:

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/meigma/blob"
)

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	// We'll fill this in step by step
	return nil
}

Step 2: Create Test Files

Create some files to archive. Add this to your run() function:

// Create a temporary source directory with test files
srcDir, err := os.MkdirTemp("", "blob-src-*")
if err != nil {
	return err
}
defer os.RemoveAll(srcDir)

// Create some test files
files := map[string]string{
	"readme.txt":       "Welcome to the blob demo!",
	"config/app.json":  `{"name": "demo", "version": "1.0"}`,
	"config/db.json":   `{"host": "localhost", "port": 5432}`,
	"src/main.go":      "package main\n\nfunc main() {}\n",
	"src/utils/log.go": "package utils\n\nfunc Log(msg string) {}\n",
}

for path, content := range files {
	fullPath := filepath.Join(srcDir, path)
	if err := os.MkdirAll(filepath.Dir(fullPath), 0o755); err != nil {
		return err
	}
	if err := os.WriteFile(fullPath, []byte(content), 0o644); err != nil {
		return err
	}
}

fmt.Printf("Created %d test files in %s\n", len(files), srcDir)

Step 3: Push to OCI Registry

Create a client and push the archive. The Push method creates the archive and pushes it in a single call:

ctx := context.Background()

// Create the OCI client
c, err := blob.NewClient(blob.WithDockerConfig())
if err != nil {
	return fmt.Errorf("create client: %w", err)
}

// Push to registry - creates archive from srcDir and pushes in one call
// Replace with your registry: docker.io/username/demo:v1, ghcr.io/org/demo:v1, etc.
ref := "localhost:5000/blob-demo:v1"  // Use a local registry for testing
if err := c.Push(ctx, ref, srcDir,
	blob.PushWithCompression(blob.CompressionZstd),
); err != nil {
	return fmt.Errorf("push archive: %w", err)
}

fmt.Printf("Pushed to %s\n", ref)

For local testing without a real registry, you can run a local registry:

docker run -d -p 5000:5000 --name registry registry:2

Note (macOS): Port 5000 may conflict with AirPlay Receiver. Use port 5001 instead: docker run -d -p 5001:5000 --name registry registry:2 and update references to localhost:5001.

And configure the client to use plain HTTP:

c, err := blob.NewClient(
	blob.WithDockerConfig(),
	blob.WithPlainHTTP(true),  // For local registries without TLS
)

Step 4: Inspect Archive Metadata

Before pulling the full archive, you can inspect its metadata without downloading the data blob:

// Inspect fetches only manifest and file index (no data blob)
result, err := c.Inspect(ctx, ref)
if err != nil {
	return fmt.Errorf("inspect archive: %w", err)
}

fmt.Printf("Archive digest: %s\n", result.Digest())
fmt.Printf("Files: %d\n", result.FileCount())
fmt.Printf("Data size: %d bytes\n", result.DataBlobSize())
fmt.Printf("Compression ratio: %.2f\n", result.CompressionRatio())

// List all files without downloading any data
fmt.Println("\nFiles in archive:")
for entry := range result.Index().Entries() {
	fmt.Printf("  %s (%d bytes)\n", entry.Path(), entry.OriginalSize())
}

This is useful for checking archive contents before deciding to pull, or for building file browsers that don't need the actual file data.

Step 5: Pull and Read Files Lazily

Pull the archive and read files. Data is fetched on demand via HTTP range requests:

// Pull the archive (downloads only the small index blob)
archive, err := c.Pull(ctx, ref)
if err != nil {
	return fmt.Errorf("pull archive: %w", err)
}

fmt.Printf("Pulled archive with %d files\n", archive.Len())

// Read a specific file (fetches only this file's bytes via range request)
content, err := archive.ReadFile("readme.txt")
if err != nil {
	return fmt.Errorf("read file: %w", err)
}
fmt.Printf("readme.txt: %s\n", content)

// Read another file
configContent, err := archive.ReadFile("config/app.json")
if err != nil {
	return fmt.Errorf("read config: %w", err)
}
fmt.Printf("config/app.json: %s\n", configContent)

// List directory contents
entries, err := archive.ReadDir("config")
if err != nil {
	return fmt.Errorf("read dir: %w", err)
}
fmt.Println("\nconfig/ directory:")
for _, entry := range entries {
	fmt.Printf("  %s\n", entry.Name())
}

Step 6: Add Caching

Add caching with a single option. WithCacheDir enables all cache layers:

// Create client with all caches enabled in one line
c, err := blob.NewClient(
	blob.WithDockerConfig(),
	blob.WithPlainHTTP(true),
	blob.WithCacheDir("/tmp/blob-cache"),
)
if err != nil {
	return err
}

// Second pull will use cached data - no network requests needed
archive, err := c.Pull(ctx, ref)
if err != nil {
	return err
}
fmt.Printf("Pulled (cached): %d files\n", archive.Len())

// Cached file reads are instant
content, err := archive.ReadFile("src/main.go")
if err != nil {
	return err
}
fmt.Printf("src/main.go: %s\n", content)

The cache directory structure is:

content/ - file content cache (deduplication across archives)
blocks/ - HTTP range block cache
refs/ - tag→digest mappings
manifests/ - parsed manifests
indexes/ - index blobs

Complete Example

Here is the complete program:

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/meigma/blob"
)

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	ctx := context.Background()

	// Step 1: Create test files
	srcDir, err := os.MkdirTemp("", "blob-src-*")
	if err != nil {
		return err
	}
	defer os.RemoveAll(srcDir)

	files := map[string]string{
		"readme.txt":       "Welcome to the blob demo!",
		"config/app.json":  `{"name": "demo", "version": "1.0"}`,
		"config/db.json":   `{"host": "localhost", "port": 5432}`,
		"src/main.go":      "package main\n\nfunc main() {}\n",
		"src/utils/log.go": "package utils\n\nfunc Log(msg string) {}\n",
	}

	for path, content := range files {
		fullPath := filepath.Join(srcDir, path)
		if err := os.MkdirAll(filepath.Dir(fullPath), 0o755); err != nil {
			return err
		}
		if err := os.WriteFile(fullPath, []byte(content), 0o644); err != nil {
			return err
		}
	}
	fmt.Printf("Created %d test files\n", len(files))

	// Step 2: Create cached client
	c, err := blob.NewClient(
		blob.WithDockerConfig(),
		blob.WithPlainHTTP(true), // For local registry
		blob.WithCacheDir("/tmp/blob-cache"),
	)
	if err != nil {
		return err
	}

	// Step 3: Push to registry (creates archive and pushes in one call)
	ref := "localhost:5000/blob-demo:v1"
	if err := c.Push(ctx, ref, srcDir,
		blob.PushWithCompression(blob.CompressionZstd),
	); err != nil {
		return fmt.Errorf("push: %w", err)
	}
	fmt.Printf("Pushed to %s\n", ref)

	// Step 4: Pull and read files
	archive, err := c.Pull(ctx, ref)
	if err != nil {
		return fmt.Errorf("pull: %w", err)
	}
	fmt.Printf("Pulled: %d files\n", archive.Len())

	content, _ := archive.ReadFile("readme.txt")
	fmt.Printf("\nreadme.txt: %s\n", content)

	entries, _ := archive.ReadDir("config")
	fmt.Println("\nconfig/ directory:")
	for _, entry := range entries {
		fmt.Printf("  %s\n", entry.Name())
	}

	return nil
}

Run the program (with a local registry running):

# Start local registry
docker run -d -p 5000:5000 --name registry registry:2

# Run the demo
go run main.go

Expected output:

Created 5 test files
Pushed to localhost:5000/blob-demo:v1
Pulled: 5 files

readme.txt: Welcome to the blob demo!

config/ directory:
  app.json
  db.json

Next Steps

Now that you have the basics, explore these guides:

OCI Client - Authentication options and client configuration
Creating Archives - Compression, change detection, and file limits
Caching - Cache configuration and sizing
Extracting Files - Advanced extraction options
Performance Tuning - Optimize for your workload

Production Security

For production deployments, add supply chain security with signing and verification.

Signing Archives (CI/CD)

Sign archives during build and push to create a verifiable chain of trust:

import (
    "github.com/meigma/blob"
    "github.com/meigma/blob/policy/sigstore"
)

// Create keyless signer (uses GitHub Actions OIDC)
signer, _ := sigstore.NewSigner(
    sigstore.WithEphemeralKey(),
    sigstore.WithFulcio("https://fulcio.sigstore.dev"),
    sigstore.WithRekor("https://rekor.sigstore.dev"),
    sigstore.WithAmbientCredentials(),
)

// Push and sign
c, _ := blob.NewClient(blob.WithDockerConfig())
c.Push(ctx, ref, srcDir, blob.PushWithCompression(blob.CompressionZstd))
c.Sign(ctx, ref, signer)  // Creates OCI 1.1 referrer signature

Verifying Archives (Consumers)

Configure policies to verify signatures and provenance on pull:

import (
    "github.com/meigma/blob"
    "github.com/meigma/blob/policy"
    "github.com/meigma/blob/policy/sigstore"
    "github.com/meigma/blob/policy/slsa"
)

// Verify signatures from GitHub Actions
sigPolicy, _ := sigstore.GitHubActionsPolicy("myorg/myrepo",
    sigstore.AllowBranches("main"),
    sigstore.AllowTags("v*"),
)

// Validate SLSA provenance (optional)
slsaPolicy, _ := slsa.GitHubActionsWorkflow("myorg/myrepo",
    slsa.WithWorkflowPath(".github/workflows/release.yml"),
)

// Combine policies (both must pass)
c, _ := blob.NewClient(
    blob.WithDockerConfig(),
    blob.WithPolicy(policy.RequireAll(sigPolicy, slsaPolicy)),
)

// Pull rejects archives that fail verification
archive, err := c.Pull(ctx, ref)

This ensures archives are signed by trusted workflows and built through authorized CI/CD pipelines.

See the Provenance & Signing guide for complete implementation details, including custom OPA policies for advanced use cases.

Prerequisites​

What We Will Build​

Step 1: Create a Project​

Step 2: Create Test Files​

Step 3: Push to OCI Registry​

Step 4: Inspect Archive Metadata​

Step 5: Pull and Read Files Lazily​

Step 6: Add Caching​

Complete Example​

Next Steps​

Production Security​

Signing Archives (CI/CD)​

Verifying Archives (Consumers)​