Content Prefetch

How prefetching works

传统流程：
用户请求 → 缓存未命中 → 回源获取 → 缓存存储 → 响应

预取流程：
预取任务 → 回源获取 → 缓存存储（就绪）
用户请求 → 缓存命中 → 即时响应

Advantages of prefetching

Prefetching method

URL list prefetching

Prefetch the specified URL:

{
  "urls": [
    "https://example.com/",
    "https://example.com/products",
    "https://example.com/about",
    "https://example.com/css/styles.css",
    "https://example.com/js/app.js"
  ]
}

Best for: Known key pages, landing pages, campaign pages

Sitemap prefetch

Automatically prefetch URLs from your sitemap:

站点地图 URL：https://example.com/sitemap.xml

Sudun parses your sitemap and prefetches all listed URLs:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/page1</loc>
    <lastmod>2024-01-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/page2</loc>
  </url>
</urlset>

Best for: Site-wide warm-ups, SEO-critical pages

Crawler pre-fetched

Automatically discover and prefetch link content:

{
  "start_url": "https://example.com/",
  "depth": 2,
  "max_urls": 100
}

Crawlers follow links to discover content:

深度 0：首页
深度 1：首页链接的页面
深度 2：深度 1 页面链接的页面

Best for: Dynamic websites, discovering new content

Configure prefetch

Dashboard settings

1. Go to Domains → and select your domain name
2. Navigate to Cache → Prefetch
3. Click Create Prefetch Task
4. Configuration:

- Source: URL list, sitemap, or crawler

- Plan: One-time or periodic

- Options: Request header, region, number of concurrency

Prefetch options

Region-specific prefetching

Prefetch to a specified geographic region:

{
  "urls": ["https://example.com/"],
  "regions": ["us-east", "eu-west", "asia-pacific"]
}

Available Regions:

Pre-Fetch Plan

One-time pre-pickup

Run prefetch immediately or at a scheduled time:

{
  "schedule": {
    "type": "once",
    "run_at": "2024-01-15T06:00:00Z"
  }
}

Periodic pre-fetch

Set up automatic periodic prefetching:

{
  "schedule": {
    "type": "recurring",
    "interval": "daily",
    "time": "04:00",
    "timezone": "UTC"
  }
}

Plan Options:

Cron expressions

For advanced scheduling:

{
  "schedule": {
    "type": "cron",
    "expression": "0 */6 * * *"
  }
}

Common patterns:

• 0 4 * * * - 4 a.m. every day
• 0 */2 * * * - Every 2 hours
• 0 0 * * 0 - Every Sunday

Prefetch the request header

Custom request headers

Send specific headers at prefetching:

{
  "headers": {
    "Accept-Language": "en-US",
    "Accept-Encoding": "gzip, br",
    "X-Prefetch": "true"
  }
}

User agent

Configure the prefetch user agent:

{
  "user_agent": "Sudun-Prefetch/1.0"
}

Tip: Your origin server can detect prefetch requests through a user agent or custom headers and log them separately.

Prefetch triggers

Deploy triggers

Automatic prefetching after deployment:

# CI/CD 集成
- name: 部署
  run: deploy-script.sh

- name: 预热缓存
  run: |
    curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch \
      -H "Authorization: Bearer $API_KEY" \
      -d '{"sitemap": "https://example.com/sitemap.xml"}'

Webhook triggers

Triggering prefetching via webhook:

curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch/trigger \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"job_id": "prefetch-job-123"}'

Content update triggers

Pre-fetch when content changes:

{
  "trigger": {
    "type": "purge",
    "action": "prefetch_after_purge"
  }
}

Monitor prefetching

Task status

Track prefetching task progress:

Prefetch metrics

View metrics in the dashboard:

• Processed URLs: The total number of URLs prefetched
• Success rate: The percentage of successful prefetching
• Average latency: The origin response time during the prefetching period
• Cache fill: The amount of content that has been cached

Pre-logging

{
  "job_id": "pf-abc123",
  "status": "completed",
  "started_at": "2024-01-15T04:00:00Z",
  "completed_at": "2024-01-15T04:05:32Z",
  "stats": {
    "total_urls": 150,
    "successful": 148,
    "failed": 2,
    "skipped": 0
  }
}

Best practices

Prioritize key content

Start by prefetching the most important pages:

{
  "urls": [
    {"url": "https://example.com/", "priority": "high"},
    {"url": "https://example.com/products", "priority": "high"},
    {"url": "https://example.com/blog", "priority": "medium"},
    {"url": "https://example.com/about", "priority": "low"}
  ]
}

Respect the origin capacity

Configure concurrency to avoid overloading the origin server:

{
  "concurrency": 3,
  "rate_limit": "10/秒"
}

Scheduling during off-peak hours

Run prefetching when traffic is at its lowest:

高峰时段：上午 9 点 - 下午 6 点
预取窗口：凌晨 2 点 - 凌晨 5 点

Exclude dynamic content

Skip URLs that shouldn't be prefetched:

{
  "exclude_patterns": [
    "/api/*",
    "/admin/*",
    "*.json",
    "*?session=*"
  ]
}

Troubleshooting

Pre-fetch does not take effect

1. Check origin accessibility: Ensure that the origin server responds to prefetch requests
2. Verify URL formatting: URLs must be fully qualified (https://... ）/)
3. Check robots.txt: Enable "Ignore Robots" if blocked
4. Review the error logs: Check for specific URL failure reasons

The load on the origin station is too high

If prefetching overloads the origin:

1. Lower concurrency settings
2. Add a rate limit
3. Scheduled to run during off-peak hours
4. Use incremental prefetching

The cache is not warmed up

If the contents are not cached after prefetching:

1. Validate that caching rules allow caching
2. Check the origin response header
3. Make sure the TTL is not too short
4. Confirm that the prefetch has completed successfully

API Reference

Create a prefetch task

curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/",
      "https://example.com/products"
    ],
    "regions": ["us-east", "eu-west"],
    "concurrency": 5
  }'

Prefetch from the sitemap

curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "sitemap": "https://example.com/sitemap.xml",
    "schedule": {
      "type": "daily",
      "time": "04:00"
    }
  }'

Get the prefetch status

curl -X GET https://api.Sudun.com/v1/domains/example.com/prefetch/job-123 \
  -H "Authorization: Bearer YOUR_API_KEY"

Need help with content prefetching? Please contact admin@jindundun.com