How to Use This Tool

About this tool

This tool compares a page's raw HTML against what a crawler can actually read, then builds a list of actionable findings — not a single score. Search ranking depends on many other factors (content relevance, backlinks, domain authority, and more) that can't be measured from one URL, so this tool focuses on technical hygiene instead of promising a search position.

How to use it

1.Enter the URL of the page you want to check, then click Analyze.
2.Pick which user-agent to crawl as: Googlebot or Generic bot (full explanation below).
3.On success, you'll see a summary of findings by severity, the raw facts found (title, description, canonical, etc), and a full list of findings with why each matters and how to fix it.
4.If the page couldn't be fetched or was blocked, a single explanation banner appears instead of a findings list — see "When you see a banner" below.

Googlebot vs Generic bot

Googlebot

Mimics Googlebot: sends Googlebot's own User-Agent header, and is matched against the site's Googlebot-specific robots.txt group (if one exists). This answers "can Google crawl this page?" — the question most people actually want answered.

Generic bot

Uses this tool's own bot identity, matched against the generic group (User-agent: *) in robots.txt. This answers "what can a generic crawler, with no special treatment, see?" — useful when a site's robots.txt treats Googlebot differently from everyone else.

Results can differ by user-agent, since robots.txt can have different rules per user-agent. Important note: many large sites verify Googlebot's identity via reverse DNS / Google's real IP ranges, not just the User-Agent header — so even with Googlebot selected, a site like LinkedIn can still reject the request. That isn't a bug; that's their security working as designed.

When you see a banner instead of a findings list

This tool deliberately doesn't guess when a page couldn't be fetched. Rather than report "title not found" when the page was never actually read, it shows a single banner explaining why content checks were skipped entirely — not treated as failed, just never checked.

robots.txt disallows it: The site's robots.txt disallows this path for the user-agent you selected. The tool didn't proceed with fetching the page, out of respect for that rule. Try switching user-agent — results can differ if the site has separate rule groups per user-agent.
Server blocked it (anti-bot): The server responded with a status commonly used to reject automated requests (e.g. 403, 429, or LinkedIn's own 999), not the real page. This usually means the site has an anti-scraping system, not that the page has an SEO problem.
Other HTTP error: The page responded with a non-2xx status that isn't an anti-bot signal (e.g. 404 or 500). Worth checking directly whether the page is genuinely broken.
Too many redirects: The URL redirected repeatedly without ever reaching a final page. There may be a redirect loop worth fixing.
Timeout / connection failed: The server didn't respond in a reasonable time, or the connection failed (e.g. an SSL error or connection refused).
Domain not found: The domain name couldn't be resolved. Double-check the URL is correct and the domain is still active.

Audit limits

To prevent abuse, the number of audits per IP address is limited over a given time window. If you hit the limit, you'll see a message asking you to try again later.

SEO Development: how the expected output is built

The SEO Development page takes a /products path plus a mocked listing response (total products, out-of-stock count, lowest price, top brands) and works out what the page should serve. The rules below run top to bottom; the first one that decides something wins. The generated title and description are always Indonesian, since they're the site's own content.

Flow

1. Validate the path: The path must start with / and with /products. Anything else shows an error and no result.
2. Search goes first: If there's a ?search= query, the page is a search result: canonical is /products, robots is "-" (follows canonical), no internal links, and the title has no length limit. Nothing below applies.
3. Read the path: Segments after /products are read as /products/{category}/{vehicle}/{brand}. The first segment is the category; the next ones are classified by the known slug lists (vehicle: mobil, motor; brands: gs-astra, incoe, …). Unknown segments are ignored with a note, more than 3 segments are cut off, and brand-before-vehicle is flagged as the wrong order.
4. Fold query filters in: ?category=, ?vehicle= and ?brand= are merged with the path. A category from the query only counts when the path has none. Other params (utm, etc.) are ignored.
5. Pick the canonical: The canonical is the one official URL for that combination: /products/{category}/{vehicle}/{brand}, or /products?brand=… / /products?vehicle=… when there's no category. ?page=N is kept from page 2 onward; sort and page=1 are dropped. A filter with several values (brand=a,b), or brand + vehicle without a category, has no PLP: robots noindex, canonical to the nearest real PLP.
6. Decide index / noindex: Only when the canonical equals the typed path (self). It's index when total products ≥ the minimum (default 3) and the in-stock share ≥ the minimum (default 50%), otherwise noindex. When the canonical points elsewhere the URL is a variant and robots is "-": no meta robots at all (not a lone follow tag, since without a tag the default is already index, follow), just the canonical. Otherwise follow is always on.
7. Internal links: true only for leveling pages (there's a category) or when a brand filter is present. /products, vehicle-only pages and search get false.
8. Build the title: The template is chosen from the canonical entities (table below). From page 2 on, " Halaman N" goes before " | Astra Otopart". If the title is longer than the limit (default 60), "Berkualitas" is removed first, then "Jual". Search titles are exempt.
9. Build the description: One of six templates (table below), also chosen from the canonical entities. The Vehicle, Category + Brand and Brand templates show the in-stock count rounded down to 20+, 50+, 100+, 150+, then every 50; with 20 or fewer in stock the number is left out. From page 2 on, " (Halaman N)" is appended.

Templates

Meta title
Template	Pattern
Products	Produk Otomotif Original \| Astra Otopart
Brand	Jual Produk {Brand} Berkualitas Original \| Astra Otopart
Leveling	Jual {Kategori} {Kendaraan} {Brand} Berkualitas Original \| Astra Otopart
Search	Pencarian {query} \| Astra Otopart

Meta description
Template	Pattern
Products	Temukan produk dan sparepart original pilihan terbaik dengan garansi resmi hanya di Astra Otopart.
Category	Belanja {kategori} original untuk mobil dan motor: {topBrands}, dan lainnya. Bergaransi resmi, mulai {minPrice}.
Vehicle	Cari {kategori} [{Brand}] untuk {kendaraan}? Pilih [N+] {kategori} original sesuai spesifikasi, bergaransi resmi, mulai {minPrice}.
Category + Brand	Beli {kategori} {Brand} original. [N+ pilihan \| Pilihan] bergaransi resmi, mulai {minPrice}.
Brand	Belanja produk {Brand} original untuk mobil dan motor. [N+ pilihan bergaransi resmi \| Bergaransi resmi], mulai {minPrice}.
Search	Hasil pencarian untuk {query}. Temukan produk dan sparepart original pilihan terbaik dengan garansi resmi hanya di Astra Otopart.

Thresholds come from server env vars, read per request (restart, no rebuild): SEO_DEV_TITLE_MAX_LENGTH (default 60), SEO_DEV_MIN_PRODUCTS (default 3), SEO_DEV_MIN_IN_STOCK_PERCENT (default 50).