aicost crawler health

FAILURES
๐Ÿ“ˆ Price changes (7d)
79
across all crawlers
๐Ÿ†• New models (7d)
281
need review / catalog
๐Ÿงน Sanity catches (7d)
39
crawler self-heal working
โŒ Failures (7d)
1
crawl or extraction errors

1. Token API Pricing Crawler

8 LLM/embedding/TTS API vendors ยท per-token rates ยท manually-curated source of truth at lib/pricing-data.js (crawler reads this for diff, does not auto-write)
AUTOMATED DAILY ยท 03:00 EDT
SourceLast modifiedVendors coveredStatus
lib/pricing-data.js 13h ago anthropic, openai, google, deepseek, xai, mistral, cohere, together โœ“ clean

2. Product / Subscription Pricing Crawler

14 consumer/team plan vendors ยท seat tiers ยท slug stored as product:vendor
AUTOMATED DAILY ยท 03:00 EDT
VendorLast runModelsChangesNew๐ŸงนStatus
anthropic4d ago14โ€”14โ€”โš  review
openai4d ago6โ€”6โ€”โš  review
google43d ago4โ€”4โ€”โš  review
google-ai4d ago8โ€”8โ€”โš  review
google-workspace4d ago4โ€”4โ€”โš  review
github4d ago4โ€”4โ€”โš  review
cursor4d ago7โ€”7โ€”โš  review
windsurf4d ago5โ€”5โ€”โš  review
perplexity4d ago2โ€”2โ€”โš  review
midjourney4d ago5โ€”5โ€”โš  review
runway4d ago5โ€”5โ€”โš  review
elevenlabs4d ago7โ€”7โ€”โš  review
replit4d agoโ€”โ€”โ€”โ€”โœ— failed
vercel4d ago4โ€”4โ€”โš  review

3. Hybrid Plan Pricing Crawler

16 seat+overage vendors ยท slug stored as hybrid:vendor ยท CF /markdown + Gemini extract
AUTOMATED DAILY ยท 03:00 EDT
VendorLast runPlansChangesNew๐ŸงนStatus
anthropic34d ago731โ€”โš  partial
aws34d ago5โ€”5โ€”โš  partial
cursorneverโ€”โ€”โ€”โ€”โธ pending
github34d ago3โ€”โ€”โ€”โš  partial
google34d ago733โ€”โš  partial
hubspotneverโ€”โ€”โ€”โ€”โธ pending
microsoft34d ago5โ€”3โ€”โš  partial
mistral34d ago2โ€”2โ€”โš  partial
notionneverโ€”โ€”โ€”โ€”โธ pending
openai34d ago714โ€”โš  review
salesforceneverโ€”โ€”โ€”โ€”โธ pending
selfhostneverโ€”โ€”โ€”โ€”โธ pending
servicenowneverโ€”โ€”โ€”โ€”โธ pending
slackneverโ€”โ€”โ€”โ€”โธ pending
together34d ago4โ€”4โ€”โš  partial
windsurfneverโ€”โ€”โ€”โ€”โธ pending

4. Manual Maintenance

1 vendor(s) blocked from auto-scrape
MANUAL QUARTERLY ยท OR ON ANNOUNCEMENTS
VendorLast verifiedDays sincePlansNext dueStatusProcedure
microsoft
Microsoft 365 / Copilot / Sales Copilot
2026-05-0135d72026-08-01โœ“ cleandocs/MICROSOFT-MANUAL-UPDATES.md

5. Benchmark Crawler

10 mechanism topics ยท prompt-cache rates, batch discounts, latencies ยท writes aicost_benchmarks table
AUTOMATED WEEKLY ยท WED 03:00 EDT
TopicLast verifiedTopicsNeeds reviewStatus
anthropic-caching
Anthropic Prompt Caching
3d ago 4 00000 โœ“ clean
anthropic-batch
Anthropic Message Batches
3d ago 2 000 โœ“ clean
openai-batch
OpenAI Batch API
3d ago 2 000 โœ“ clean
openai-caching
OpenAI Prompt Caching
3d ago 2 000 โœ“ clean
google-gemini
Google Gemini
3d ago 3 0000 โœ“ clean
google-vertex
Google Vertex AI
35d ago 1 00 ! stale
aws-bedrock
AWS Bedrock
3d ago 1 00 โœ“ clean
mistral
Mistral
3d ago 1 00 โœ“ clean
pinecone
Pinecone
3d ago 2 000 โœ“ clean
cohere
Cohere
3d ago 2 000 โœ“ clean

๐Ÿ“Š Recent price changes (last 7 days ยท all crawlers)

TrackModelFieldOldNewฮ”%Detected
pricingelevenlabs/eleven-flash-v2-5costPerMinute0.180.17-5.6%6/5/2026
pricingmistral/mistral-large-3output61.5-75.0%6/5/2026
pricingmistral/mistral-large-3input20.5-75.0%6/5/2026
pricingxai/imagine-api-imagecostPerImage_standard0.02โ€”6/5/2026
pricinggoogle/gemini-3-1-flash-live-previewinput30.75-75.0%6/5/2026
pricinggoogle/gemini-3-pro-imagebatchOutput606-90.0%6/5/2026
pricinggoogle/gemini-3-1-flash-imagecostPerImage_standard0.0670.045-32.8%6/5/2026
pricinggoogle/gemini-2-5-flash-native-audio-preview-12-2025output122-83.3%6/5/2026
pricinggoogle/gemini-2-5-flash-native-audio-preview-12-2025input30.5-83.3%6/5/2026
pricinggoogle/gemini-3-1-flash-live-previewoutput124.5-62.5%6/5/2026
pricingopenai/gpt-5-5-prolongContextThreshold272000270000-0.7%6/5/2026
pricingopenai/gpt-5-5longContextThreshold272000270000-0.7%6/5/2026
pricingopenai/gpt-realtime-2costPerMinute0.034โ€”6/5/2026
pricinganthropic/claude-opus-4-8batchInput2.5โ€”6/5/2026
pricinganthropic/claude-opus-4-8batchOutput12.5โ€”6/5/2026
pricingxai/xai-voice-apinew_model_addedโ€”6/4/2026
pricingxai/xai-stt-batchnew_model_addedโ€”6/4/2026
pricingxai/xai-stt-streamingnew_model_addedโ€”6/4/2026
pricingmistral/mistral-large-3-25-12new_model_addedโ€”6/4/2026
pricingmistral/mistral-voxtral-mini-transcribe-realtimenew_model_addedโ€”6/4/2026
pricingelevenlabs/eleven-flash-v2-5costPerMinute0.180.17-5.6%6/4/2026
pricingdeepgram/nova-3-multilingualcostPerMinute0.0058โ€”6/4/2026
pricingdeepgram/nova-3-monolingualcostPerMinute0.0048โ€”6/4/2026
pricingcohere/command-routput0.61.5150.0%6/4/2026
pricingcohere/command-rinput0.150.5233.3%6/4/2026
pricingmistral/mistral-large-3-25-12input20.5-75.0%6/4/2026
pricingmistral/mistral-large-3-25-12output61.5-75.0%6/4/2026
pricingxai/imagine-api-imagecostPerImage_standard0.02โ€”6/4/2026
pricinggoogle/gemini-3-1-flash-imagecostPerImage_standard0.0670.045-32.8%6/4/2026
pricingopenai/gpt-realtime-2costPerMinute0.034โ€”6/4/2026

๐Ÿ“‹ TLDR โ€” what needs attention ๐Ÿ”ง Troubleshooting playbook โ†’

๐Ÿšจ Falling behind (10)

replit ยท Product/Subscription
Last run failed
cursor ยท Hybrid Plan Pricing
Never run
hubspot ยท Hybrid Plan Pricing
Never run
notion ยท Hybrid Plan Pricing
Never run
salesforce ยท Hybrid Plan Pricing
Never run
selfhost ยท Hybrid Plan Pricing
Never run
servicenow ยท Hybrid Plan Pricing
Never run
slack ยท Hybrid Plan Pricing
Never run
windsurf ยท Hybrid Plan Pricing
Never run
google-vertex ยท Benchmark
Last run 837h ago (>14d)

โœ… Healthy (32)

pricing-data.js (8 vendors) ยท anthropic (product/subscription) ยท openai (product/subscription) ยท google (product/subscription) ยท google-ai (product/subscription) ยท google-workspace (product/subscription) ยท github (product/subscription) ยท cursor (product/subscription) ยท windsurf (product/subscription) ยท perplexity (product/subscription) ยท midjourney (product/subscription) ยท runway (product/subscription) ยท elevenlabs (product/subscription) ยท vercel (product/subscription) ยท anthropic (hybrid) ยท aws (hybrid) ยท github (hybrid) ยท google (hybrid) ยท microsoft (hybrid) ยท mistral (hybrid) ยท openai (hybrid) ยท together (hybrid) ยท anthropic-caching (benchmark) ยท anthropic-batch (benchmark) ยท openai-batch (benchmark) ยท openai-caching (benchmark) ยท google-gemini (benchmark) ยท aws-bedrock (benchmark) ยท mistral (benchmark) ยท pinecone (benchmark) ยท cohere (benchmark) ยท microsoft (manual)

๐Ÿ›  Manual run commands

Copy-paste reference for triggering crawlers manually. Use these when scheduled jobs fail, after vendor announcements, or for ad-hoc data refreshes. Replace <slug> with the vendor slug. Add --dry-run to preview without DB writes.

๐Ÿ”ต TOKEN API PRICING (detects vendor changes โ€” pricing-data.js is manually-curated)
:: All vendors (full nightly run)
node scripts\aicost-pricing-crawler.js > D:\aicost\logs\pricing-now.log 2>&1

:: Single vendor
node scripts\aicost-pricing-crawler.js --vendor=anthropic

:: Dry run (preview, no DB writes)
node scripts\aicost-pricing-crawler.js --vendor=openai --dry-run
๐Ÿ”ด PRODUCT / SUBSCRIPTION PRICING (writes product:* rows)
:: All vendors (full weekly run)
node scripts\aicost-product-crawler.js > D:\aicost\logs\product-now.log 2>&1

:: Single vendor
node scripts\aicost-product-crawler.js --vendor=cursor

:: Dry run
node scripts\aicost-product-crawler.js --vendor=github --dry-run
๐ŸŸก HYBRID PLAN PRICING (writes hybrid:* rows)
:: All vendors (full weekly run)
node lib\aicost-hybrid-crawler.js > D:\aicost\logs\hybrid-now.log 2>&1

:: Single vendor
node lib\aicost-hybrid-crawler.js --vendor=anthropic

:: Dry run / Tier-A only / extract phase only
node lib\aicost-hybrid-crawler.js --vendor=openai --dry-run
node lib\aicost-hybrid-crawler.js --tier=A
node lib\aicost-hybrid-crawler.js --phase=extract
๐ŸŸข BENCHMARK CRAWLER (mechanism facts: cache rates, batch discounts, latencies)
:: All 10 topics (full weekly run)
node scripts\aicost-benchmark-crawler.js > D:\aicost\logs\benchmark-now.log 2>&1

:: Single topic
node scripts\aicost-benchmark-crawler.js --vendor=anthropic-caching
node scripts\aicost-benchmark-crawler.js --vendor=openai-batch

:: Dry run / phase-only
node scripts\aicost-benchmark-crawler.js --dry-run
node scripts\aicost-benchmark-crawler.js --phase=crawl
node scripts\aicost-benchmark-crawler.js --phase=extract

:: Available topic slugs:
::   anthropic-caching, anthropic-batch, openai-batch, openai-caching,
::   google-gemini, google-vertex, aws-bedrock, mistral, pinecone, cohere

:: Review flagged rows (Mistral couldn't verify claim from source text)
mysql -u root -p toolsinfo -e "SELECT topic_slug, claim, source_url FROM aicost_benchmarks WHERE needs_review = 1;"
๐ŸŸฃ V2 GUIDE REGENERATION (after pricing changes)
:: Single vendor
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --force

:: Multiple vendors
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --vendor=openai --force

:: All 16 V2 vendors (~$0.30, ~5-7 minutes)
node scripts\generate-vendor-pricing-guides-batch.js --all --force
โš™๏ธ MANUAL MAINTENANCE (Microsoft quarterly audit)
:: Open the procedure doc
notepad D:\aicost\docs\MICROSOFT-MANUAL-UPDATES.md

:: Edit verified plan data with current Microsoft pricing
notepad D:\aicost\lib\hybrid-pricing-data-verified-2026q2.js

:: Update last_verified date in routes/crawler-health.js MANUAL_VENDORS
notepad D:\aicost\routes\crawler-health.js

:: Then regen Microsoft V2
node scripts\generate-vendor-pricing-guides-batch.js --vendor=microsoft --force
๐Ÿ“… SCHEDULED TASK MANAGEMENT
:: Register all scheduled tasks (run as admin)
powershell -ExecutionPolicy Bypass -File D:\aicost\powershell\Run-AicostCrawler.ps1

:: List all aicost scheduled tasks
schtasks /query /tn "aicost*"

:: Manually trigger a scheduled task
schtasks /run /tn "aicost-pricing-crawler"

:: View recent task history (last 24h)
powershell "Get-WinEvent -LogName 'Microsoft-Windows-TaskScheduler/Operational' -MaxEvents 50 | Where-Object {$_.Message -like '*aicost*'}"
๐Ÿ” DIAGNOSTIC QUERIES
:: Last run per vendor (all crawlers)
mysql -u root -p toolsinfo -e "SELECT vendor_slug, COUNT(*) AS runs, MAX(started_at) AS last_run FROM aicost_crawler_runs GROUP BY vendor_slug ORDER BY last_run DESC;"

:: Recent price changes (last 7 days)
mysql -u root -p toolsinfo -e "SELECT provider_slug, model_slug, field_name, old_value, new_value, change_pct, detected_at FROM aicost_price_changelog WHERE detected_at >= DATE_SUB(NOW(), INTERVAL 7 DAY) ORDER BY detected_at DESC LIMIT 20;"

:: V2 guide refresh status
mysql -u root -p toolsinfo -e "SELECT slug, schema_version, last_refreshed_at FROM aicost_guides WHERE type='vendor-pricing' AND schema_version='v2' ORDER BY last_refreshed_at DESC;"

:: pricing-data.js last modified
dir D:\aicost\lib\pricing-data.js
Auto-refreshes on page reload ยท ๐Ÿ”ง Troubleshooting playbook ยท JSON API ยท Updated 6/5/2026, 4:25:51 PM