Skip to main content
SolarQna
Transparent Web Crawler & Cross-Validation Engine

SolarQnaBot/1.0 — How we index public offers

Reads public product pages on solar equipment shops so SolarQna can report verified market availability and cost per watt. One request at a time per host, strictly identified in headers, RFC 9309 robots.txt obeyed.

Rate ≥ 2.5s / hostBlock SLA ≤ 24h
Exact HTTP User-Agent Header
SolarQnaBot/1.0 (+https://solarqna.com/bot; respects robots.txt; opt-out there)
Request Pacing

1 req / ≥2.5s

Fields Read

Title, Price, Stock

Personal Data

None Collected

Opt-Out Contact

[email protected]

01 · Block or throttle in robots.txt

Paste into your-shop.example/robots.txt — takes effect automatically

Stop SolarQnaBot completely
User-agent: SolarQnaBot
Disallow: /
Block B2B or clearance paths
User-agent: SolarQnaBot
Disallow: /clearance/
Disallow: /b2b/
Slow crawler down (30s gap)
User-agent: SolarQnaBot
Crawl-delay: 30

Three common robots.txt mistakes to avoid:

  • Comments are ignored: # SolarQnaBot: Deny has zero effect on any compliant parser.
  • Groups do not stack: Adding a User-agent: SolarQnaBot group means we read that group instead of User-agent: *. Repeat any wildcard rules that still apply.
  • Per-host scope: A file on your apex domain does not govern shop.example.com — serve /robots.txt on the shop subdomain itself.
02 · What it reads vs. what it never touches

Reads (Public Product Metadata)

  • Product title & SKU / MPN identifier
  • Public offer price, currency, and VAT indicator
  • In-stock / out-of-stock availability flag
  • Public product image URL & canonical link

Never Touches

  • Login forms, cart sessions, or checkout routes
  • Customer-only B2B portals or paywalled pricing
  • Cookies, visitor telemetry, or personal user records
  • High-concurrency bursts (strict per-host token bucket)
03 · Cross-Validation Product Strategy

Why shop listings never pollute verified hardware specifications

Layer 1 · Primary Truth

Electrical metrics (Wp, Voc, Isc, Vmp, Imp, γPmax, dimensions) originate strictly from SHA-256 deduplicated OEM datasheets, NREL SAM / CEC lists, and MNRE ALMM schedules.

Layer 2 · 7-Law Physics

Every candidate SKU must satisfy Pmax ≈ Vmp × Imp, Voc > Vmp, computed aperture efficiency matching Pmax / (A × 1000), and physically plausible temperature coefficients.

Layer 3 · 4-Stage Resolver

SolarQnaBot matches shop listings by manufacturer bucket, wattage guardrail (preventing 580 W vs 585 W false merges), and SKU cosine similarity — separating price discovery from spec truth.

04 · Questions & Data Removal

Want your shop excluded or historical offer rows purged? Email [email protected] from a domain address — datasheet specs stay unchanged because they come from OEM PDFs, while your shop’s commercial offers are removed promptly. Want to explore verified market coverage? Inspect live Offers & Market Stats →

How fast does a robots.txt block take effect?

Within 24 hours. SolarQnaBot refreshes robots.txt per host at most once every 24 hours and immediately skips disallowed prefixes or entire domains.

Does a comment like '# SolarQnaBot: Deny' block the crawler?

No. Per RFC 9309, anything after '#' is ignored as a human comment. Use a User-agent: SolarQnaBot group with a Disallow line underneath.

Can a shop slow down SolarQnaBot instead of blocking it?

Yes. Add a Crawl-delay directive inside your User-agent: SolarQnaBot block. We respect any delay above our 2.5-second baseline and back off on 429/503 Retry-After headers.

Can shop listings ever overwrite panel or inverter datasheet specifications?

Never. Hardware electrical parameters come exclusively from verified manufacturer PDF datasheets, CEC/NREL registries, and our 7-law physics validator. Shop pages contribute public pricing and availability only.