AF
Progress
#1236Medium~35 min

Web Crawler

Time O(\ · Space V\ · Official statement on LeetCode

design
cpppython

Solutions

// Time:  O(|V| + |E|)
// Space: O(|V|)

/**
 * // This is the HtmlParser's API interface.
 * // You should not implement it, or speculate about its implementation
 * class HtmlParser {
 *   public:
 *     vector<string> getUrls(string url);
 * };
 */
class Solution {
public:
    vector<string> crawl(string startUrl, HtmlParser htmlParser) {
        vector<string> result = {startUrl};
        unordered_set<string> lookup(result.cbegin(), result.cend());
        for (int i = 0; i < result.size(); ++i) {
            const auto& from_url = result[i];
            const auto& name = hostname(from_url);
            for (const auto& to_url: htmlParser.getUrls(from_url)) {
                if (!lookup.count(to_url) && name == hostname(to_url)) {
                    result.emplace_back(to_url);
                    lookup.emplace(to_url);
                }
            }
        }
        return result;
    }

private:
    string hostname(const string& url) {
        static const string scheme = "http://";
        return url.substr(0, url.find('/', scheme.length()));
    }
};

Beginner Explanation

What is Web Crawler?

Web Crawler (LeetCode #1236) is a Medium problem that primarily trains design.

How to think about it

  1. Restate the goal in your own words before coding.
  2. Work a tiny example by hand so the invariant becomes obvious.
  3. Identify the pattern — this problem aligns with general problem-solving.
  4. Only then translate the idea into code.

Why this problem matters

It sits in the sweet spot of interview difficulty: multiple valid approaches, clear trade-offs. Official solution notes mention: )_.

AlgoForge explanations are original teaching notes. Always open the official problem statement on LeetCode for constraints and examples.

Interview Walkthrough

Interview approach for Web Crawler

Opening (30–60 seconds)

  • Clarify inputs/outputs and edge cases (empty input, single element, duplicates, overflow).
  • State a brute force so the interviewer knows you can solve it naively.
  • Propose the optimal direction tied to general problem-solving.

Core solution narrative

  1. Define the state you track (pointers, DP cell, set membership, stack top, etc.).
  2. Explain the transition when you process the next element.
  3. Call out time (O() and space (V) before coding.
  4. Code cleanly; narrate variable names.

What interviewers listen for

  • Correctness on edge cases
  • Complexity honesty
  • Ability to discuss trade-offs (e.g., hash map space vs. sort + two pointers)

Follow-up questions they may ask

  • Can you solve it with less memory?
  • What if the input stream is infinite / doesn't fit in RAM?
  • How would tests look for adversarial inputs?

Optimized Approach

Optimized solution notes

The reference solutions on AlgoForge target *O(* time and *V* space.

Pattern focus: general problem-solving

Use the pattern as a checklist:

  • Identify the dominant pattern and stick to one clear invariant

Multiple methods appear in the source solutions — compare them and explain when each is preferable.

Implementation tips

  • Prefer readable names over micro-optimizations in interviews.
  • Extract helpers only when they clarify (e.g., expand-around-center, DFS visit).
  • After AC-level logic, re-scan for off-by-one and null checks.

Complexity Analysis

Complexity

Measure Bound
Time *O(*
Space *V*

How to justify this in an interview

  • Time: count loops, map/set operations, and recursive branching; state average vs worst case if relevant.
  • Space: include hash maps, recursion stack, and output allocation when the problem asks for it.

If your implementation differs from the reference, re-derive big-O from your code — never memorize a complexity you cannot defend.

Common Mistakes

Common mistakes on Web Crawler

  1. Skipping edge cases — empty collections, single-element inputs, max constraints.
  2. Wrong invariant for general problem-solving — updating state too early or too late.
  3. Mutating input unexpectedly when the problem forbids it.
  4. Off-by-one in windows, ranges, or binary search bounds.
  5. Ignoring overflow / precision for integer arithmetic problems.
  6. Overengineering — jumping to an advanced structure when a simpler approach works.

Alternative Approaches

Alternatives

The source file includes more than one method. Compare:

  1. Primary optimized path — best complexity for typical interviews.
  2. Secondary approach — often brute force, sorting-based, or space-optimized variant.

Practice articulating when you would pick each (constraints, readability, follow-ups).

Edge Cases

Edge cases checklist

  • Minimum input size
  • Maximum input size / time limits
  • Duplicates and already-sorted input
  • Negative numbers / zeros (if applicable)
  • Disconnected structures (graphs/trees)
  • Single path vs branching recursion depth

Pattern Recognition

Spotting this pattern

Signal phrases that point to general problem-solving:

  • Sorted input or ability to sort without changing the answer class
  • Need for contiguous subarray / substring → consider sliding window
  • Need for O(1) membership → hash set/map
  • Optimal substructure + overlapping subproblems → DP
  • Connectivity / components → graph DFS/BFS or Union-Find

Primary topics: design.

Follow-up Interview Questions

Follow-ups

  1. How does the solution change if the input is a stream?
  2. Can you solve it in-place?
  3. What if duplicates must be handled differently?
  4. How would you parallelize the approach?
  5. Design tests that would break a buggy implementation.

Practice Recommendations

What to practice next

  1. Re-solve Web Crawler in a second language (cpp, python).
  2. Drill 3–5 more problems tagged design.
  3. Teach the solution out loud in under 5 minutes.
  4. Add this problem to your revision calendar in 3 days and 14 days.

Visualization

Conceptual diagram for Web Crawler: show input structure (design), highlight the moving parts of the general problem-solving approach, and annotate each step with the maintained invariant and complexity.

Study checklist

  • Read the official problem statement on LeetCode
  • Solve on paper / whiteboard first
  • Implement the general problem-solving approach
  • Verify edge cases from the checklist
  • State time and space complexity aloud
  • Compare with the AlgoForge reference solution
  • Schedule a revision session

Revision notes

Web Crawler (#1236) — Medium. Pattern: general problem-solving. Complexity: O(\ time / V\ space. Re-derive the invariant before coding.

FAQs

What is the time complexity of Web Crawler?+

The reference solutions aim for O(\ time and V\ space. Always re-derive complexity from the code you write in the interview.

What pattern does Web Crawler use?+

It primarily maps to general problem-solving, within the broader topic of design.

Is Web Crawler good for interviews?+

Yes — as a Medium problem it is a solid practice target. Pair it with related problems in the same pattern family for spaced repetition.

Where can I read the official statement?+

Open the official LeetCode page for constraints and examples: https://leetcode.com/problems/web-crawler/