implement new requirements

TASK-urlshortener.md

@@ -12,4 +12,62 @@ I'd like to implement a URL shortener, mocking the de-facto `urlshortener` proje
   * ... other configs, if you think are necessary, ask me.
 * The API should be RESTful, and simple enough to be used by a human.
 * The shortened data should be stored within a sqlite database.
-* Other than the .py file, provide a docker compose file to run it. Remember to keep the data in a mount dir so it won't be lost.
+* Other than the .py file, provide a docker compose file to run it. Remember to keep the data in a mount dir so it won't be lost.
+
+# Additional requirements
+* API authentication should be purely done based on each API's api_key field, as described below. No session management or token management is needed. The API key can be passed as a query parameter or in the json request body, and it should be checked for every API call that requires authentication.
+* The base path should be configurable, the same as both in current implementation and in README.
+* Implement retention days feature where old URLs are deleted after a certain number of days. If 0 or not set for a shortened URL, it will never be deleted.
+* The API design should be like this, remember to deal with trailing slashes:
+  1. GET `<base_url>` - health check, same as current implementation in README.
+  2. GET `<base_url>/api/urls&api_key=<api_key>` - returns all shortened URLs in json, newest first. The API key is required. The return json body should be same as described in current README.
+  3. POST `<base_url>/api/shorten`, create new short URL. The fields needed can either be passed as query parameters (remember to deal with URL escaping/unescaping) or in the json request body.
+    The response should be only the short URL in plain text. The fields needed are listed as below: 
+    a) `api_key` (required) - the API key is required.
+    b) `url` (required) - the URL to shorten
+    c) `short_length` (optional) - the length of the shortened URLs, default to config value.
+    d) `retention_days` (optional) - the number of days to keep current shortened URLs, default to config value.
+    
+  4. GET `<base_url>/api/urls/<short_id>` - no API key required, returns metadata json for a shortened URL, includes:
+    ```json
+    {
+      "short_code": "aB3xYz",
+      "short_url": "<base_url>/aB3xYz",
+      "original_url": "https://example.com",
+      "created_at": 1710000000,
+      "visit_count": 5
+    }
+    ```
+  5. DELETE `<base_url>/api/urls/<short_id>&api_key=<api_key>` - deletes a shortened URL. API key is required. 
+    a) Returns 204 on success without body,
+    b) 404 if not found, without body, 
+    c) 403 if not authorized, without body.
+  6. GET `<base_url>/<short_id>` - redirects to the original URL (302). Increments the visit count on each hit.
+
+# Simple Frontend
+* Implement a clean and simple frontend, follow the same design and style requirements for the single page HTML located in the `navpage` folder.
+* Add an optional one-line API key input field to top left corner, so that admin can access to the restricted APIs.
+* There is a long search input bar on top center of the page, where user can input a URL to shorten once enter is hit. if shortening is successful, show all its metadata below in a table at center of the page below the search bar;
+* If the input URL is existing (even without hitting enter), also show all its metadata in the table.
+* If the input URL is malformed on hitting enter, display nothing below the search bar.
+* If the API key provided is not valid, show nothing and continue using the page as non admin.
+* Once the provided API key is valid, list the existing shortened URLs in a table below the search bar (if the search bar is empty), with the following columns, ordered by created_at descending:
+  1. shortened URL
+  2. original URL
+  3. visit count
+  4. created at
+  5. retention days
+* For this admin table, each row should have a delete button on end, only displayed on mouse hover, which will call the delete API to delete the shortened URL.
+* Each row should have a copy button on end, copying the shortened URL to clipboard.
+* The table should be sortable by any column.
+* The table should be paginated, with 20 rows per page by default, controlled by a dropdown of `20, 50, 100, 200, 500, all`, and the pagination controls should be displayed at the bottom of the table.
+* The search bar should also function as the same as for admin (hide the listing table once start typing) - display metadata for existing URLs, create new one for valid URL on hitting enter.
+* all static resources should be put under the `urlshort/static` folder.
+* the strict "local-serve" requirement is the same as `navpage`, reuse the script in `navpage` if possible.
+* Once the frontend implementation is done, rewrite the nginx config to serve it correctly. If possible, only write the locations config, for that I'm planning to deploy both frontend and backend in an existing vhost. Write the path mapping carefully to avoid possible conflicts with the existing vhost.
+
+# Extra requirements on backend
+* Also, implement simple but proper CORS to allow seamless redirection to the target URL.
+* In hindsight, the API backend should at least implement simple rate limiting/throttling, by IP address.
+* Field validation is also a must, checking and limiting ALL fields to reasonable values to prevent hacking, especially SQL injections and buffer overflows.
+* If needed, implement other necessary guard features on the backend to prevent XSS, CSRF, and other common web attacks.

config.json

@@ -1,9 +1,10 @@
 {
-  "base_url": "http://localhost:8080",
+  "base_url": "http://localhost:8080/s",
   "short_length": 6,
   "api_key": "change-this-secret-key",
   "host": "0.0.0.0",
   "port": 8080,
-  "db_path": "data/urlshort.db"
+  "db_path": "data/urlshort.db",
+  "retention_days": 0
 }
 

urlshort.py

@@ -3,12 +3,12 @@
 URL Shortener — stdlib-only, Python 3.8+
 Usage: python urlshort.py <config.json>
 
-API:
+API (all routes are prefixed with base_path, e.g. /s):
   GET  /                        Health check
   POST /api/shorten             Create a short URL  (API key required)
-  GET  /api/urls                List all short URLs (API key required)
+  GET  /api/urls                List all short URLs  (API key required)
-  GET  /api/urls/<code>         Get info for a code (API key required)
+  GET  /api/urls/<code>         Get info for a code  (no API key required)
-  DELETE /api/urls/<code>       Delete a short URL  (API key required)
+  DELETE /api/urls/<code>       Delete a short URL   (API key required)
   GET  /<code>                  Redirect to original URL
 """
 
@@ -38,6 +38,15 @@ def load_config(path: str) -> dict:
     cfg.setdefault("host", "0.0.0.0")
     cfg.setdefault("port", 8080)
     cfg.setdefault("db_path", "data/urlshort.db")
+    cfg.setdefault("retention_days", 0)
+
+    # Derive base_path from the path component of base_url.
+    # e.g. "http://example.com/s"  → "/s"
+    #      "http://example.com/s/" → "/s"
+    #      "http://example.com"    → ""
+    raw = urlparse(cfg["base_url"]).path.strip("/")
+    cfg["base_path"] = f"/{raw}" if raw else ""
+
     return cfg
 
 
@@ -53,14 +62,24 @@ def init_db(db_path: str) -> None:
     with sqlite3.connect(db_path) as conn:
         conn.execute("""
             CREATE TABLE IF NOT EXISTS urls (
-                short_code   TEXT    PRIMARY KEY,
+                short_code     TEXT    PRIMARY KEY,
-                original_url TEXT    NOT NULL,
+                original_url   TEXT    NOT NULL,
-                created_at   INTEGER NOT NULL,
+                created_at     INTEGER NOT NULL,
-                visit_count  INTEGER NOT NULL DEFAULT 0
+                visit_count    INTEGER NOT NULL DEFAULT 0,
+                retention_days INTEGER NOT NULL DEFAULT 0
             )
         """)
         conn.commit()
 
+        # Migrate: add retention_days column if missing (existing DB).
+        try:
+            conn.execute(
+                "ALTER TABLE urls ADD COLUMN retention_days INTEGER NOT NULL DEFAULT 0"
+            )
+            conn.commit()
+        except sqlite3.OperationalError:
+            pass  # column already exists
+
 
 def db_connect(db_path: str) -> sqlite3.Connection:
     conn = sqlite3.connect(db_path)
@@ -68,6 +87,18 @@ def db_connect(db_path: str) -> sqlite3.Connection:
     return conn
 
 
+def cleanup_expired(db_path: str) -> None:
+    """Delete URLs whose retention period has elapsed."""
+    now = int(time.time())
+    with db_connect(db_path) as conn:
+        conn.execute(
+            "DELETE FROM urls WHERE retention_days > 0 "
+            "AND (created_at + retention_days * 86400) < ?",
+            (now,),
+        )
+        conn.commit()
+
+
 # ---------------------------------------------------------------------------
 # Helpers
 # ---------------------------------------------------------------------------
@@ -97,6 +128,28 @@ class Handler(BaseHTTPRequestHandler):
     # Injected by main() before the server starts.
     cfg: dict = {}
 
+    # ------------------------------------------------------------------
+    # Routing helpers
+    # ------------------------------------------------------------------
+
+    def _local_path(self):
+        """Return the request path with base_path prefix stripped.
+
+        Returns None (→ 404) when the request path does not start with
+        the configured base_path at all.
+        """
+        raw = urlparse(self.path).path
+        base = self.cfg.get("base_path", "")
+        if base:
+            if raw == base or raw == base + "/":
+                # exact match on the prefix itself → treat as root
+                return "/"
+            if raw.startswith(base + "/"):
+                return raw[len(base):].rstrip("/") or "/"
+            # path is outside our prefix entirely
+            return None
+        return raw.rstrip("/") or "/"
+
     # ------------------------------------------------------------------
     # Logging
     # ------------------------------------------------------------------
@@ -120,6 +173,19 @@ class Handler(BaseHTTPRequestHandler):
         self.end_headers()
         self.wfile.write(body)
 
+    def _send_plain(self, status: int, text: str) -> None:
+        body = text.encode("utf-8")
+        self.send_response(status)
+        self.send_header("Content-Type", "text/plain; charset=utf-8")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def _send_empty(self, status: int) -> None:
+        self.send_response(status)
+        self.send_header("Content-Length", "0")
+        self.end_headers()
+
     def _error(self, status: int, message: str) -> None:
         self._send_json(status, {"error": message})
 
@@ -130,17 +196,20 @@ class Handler(BaseHTTPRequestHandler):
         self.end_headers()
 
     # ------------------------------------------------------------------
-    # Auth
+    # Auth — purely api_key based (query param or JSON body)
     # ------------------------------------------------------------------
 
-    def _authenticated(self) -> bool:
+    def _check_api_key(self, body: dict = None) -> bool:
-        """Accept key via Authorization: Bearer <key> header or ?api_key=."""
+        """Check api_key from query parameter or JSON body."""
-        auth = self.headers.get("Authorization", "")
-        if auth.lower().startswith("bearer "):
-            return auth[7:].strip() == self.cfg["api_key"]
-
         qs = parse_qs(urlparse(self.path).query)
-        return qs.get("api_key", [""])[0] == self.cfg["api_key"]
+        key = qs.get("api_key", [""])[0]
+        if key:
+            return key == self.cfg["api_key"]
+        if body and isinstance(body, dict):
+            key = str(body.get("api_key", ""))
+            if key:
+                return key == self.cfg["api_key"]
+        return False
 
     # ------------------------------------------------------------------
     # Body
@@ -161,7 +230,12 @@ class Handler(BaseHTTPRequestHandler):
     # ------------------------------------------------------------------
 
     def do_GET(self):  # noqa: N802
-        path = urlparse(self.path).path.rstrip("/") or "/"
+        cleanup_expired(self.cfg["db_path"])
+
+        path = self._local_path()
+        if path is None:
+            self._send_empty(404)
+            return
 
         if path == "/":
             self._send_json(200, {"status": "ok", "service": "url-shortener"})
@@ -178,37 +252,51 @@ class Handler(BaseHTTPRequestHandler):
             self._handle_redirect(code)
 
     def do_POST(self):  # noqa: N802
-        path = urlparse(self.path).path.rstrip("/")
+        cleanup_expired(self.cfg["db_path"])
+
+        path = self._local_path()
+        if path is None:
+            self._send_empty(404)
+            return
 
         if path == "/api/shorten":
             self._handle_shorten()
         else:
-            self._error(404, "Not found")
+            self._send_empty(404)
 
     def do_DELETE(self):  # noqa: N802
-        path = urlparse(self.path).path.rstrip("/")
+        cleanup_expired(self.cfg["db_path"])
+
+        path = self._local_path()
+        if path is None:
+            self._send_empty(404)
+            return
 
         if path.startswith("/api/urls/"):
             code = path[len("/api/urls/"):]
             self._handle_delete_url(code)
         else:
-            self._error(404, "Not found")
+            self._send_empty(404)
 
     # ------------------------------------------------------------------
     # Handlers
     # ------------------------------------------------------------------
 
     def _handle_shorten(self) -> None:
-        if not self._authenticated():
+        # Parse both query parameters and JSON body
-            self._error(401, "Unauthorized")
+        qs = parse_qs(urlparse(self.path).query)
-            return
-
         body = self._read_json()
         if body is None:
             self._error(400, "Invalid JSON body")
             return
 
-        original_url = str(body.get("url", "")).strip()
+        # Auth: check api_key from query param or body
+        if not self._check_api_key(body):
+            self._send_empty(403)
+            return
+
+        # url: query param takes precedence, then body
+        original_url = qs.get("url", [""])[0] or str(body.get("url", "")).strip()
         if not original_url:
             self._error(400, "Missing required field: url")
             return
@@ -216,41 +304,53 @@ class Handler(BaseHTTPRequestHandler):
             self._error(400, "Invalid URL — must start with http:// or https://")
             return
 
-        custom_code = str(body.get("custom_code", "")).strip()
+        # short_length: query param, then body, then config default
+        raw_sl = qs.get("short_length", [None])[0]
+        if raw_sl is None:
+            raw_sl = body.get("short_length")
+        if raw_sl is not None:
+            try:
+                short_length = int(raw_sl)
+            except (ValueError, TypeError):
+                self._error(400, "Invalid short_length")
+                return
+        else:
+            short_length = self.cfg["short_length"]
+
+        # retention_days: query param, then body, then config default
+        raw_rd = qs.get("retention_days", [None])[0]
+        if raw_rd is None:
+            raw_rd = body.get("retention_days")
+        if raw_rd is not None:
+            try:
+                retention_days = int(raw_rd)
+            except (ValueError, TypeError):
+                self._error(400, "Invalid retention_days")
+                return
+        else:
+            retention_days = self.cfg.get("retention_days", 0)
 
         with db_connect(self.cfg["db_path"]) as conn:
-            if custom_code:
+            short_code = self._unique_code(conn, short_length)
-                row = conn.execute(
+            if short_code is None:
-                    "SELECT short_code FROM urls WHERE short_code = ?", (custom_code,)
+                self._error(500, "Could not generate a unique short code — try again")
-                ).fetchone()
+                return
-                if row:
-                    self._error(409, "Custom code already in use")
-                    return
-                short_code = custom_code
-            else:
-                short_code = self._unique_code(conn)
-                if short_code is None:
-                    self._error(500, "Could not generate a unique short code — try again")
-                    return
 
             created_at = int(time.time())
             conn.execute(
-                "INSERT INTO urls (short_code, original_url, created_at) VALUES (?, ?, ?)",
+                "INSERT INTO urls (short_code, original_url, created_at, retention_days) "
-                (short_code, original_url, created_at),
+                "VALUES (?, ?, ?, ?)",
+                (short_code, original_url, created_at, retention_days),
             )
             conn.commit()
 
         short_url = f"{self.cfg['base_url'].rstrip('/')}/{short_code}"
-        self._send_json(201, {
+        # Response: only the short URL in plain text
-            "short_code": short_code,
+        self._send_plain(201, short_url)
-            "short_url": short_url,
-            "original_url": original_url,
-            "created_at": created_at,
-        })
 
     def _handle_list_urls(self) -> None:
-        if not self._authenticated():
+        if not self._check_api_key():
-            self._error(401, "Unauthorized")
+            self._send_empty(403)
             return
 
         with db_connect(self.cfg["db_path"]) as conn:
@@ -259,32 +359,41 @@ class Handler(BaseHTTPRequestHandler):
             ).fetchall()
 
         base = self.cfg["base_url"].rstrip("/")
-        urls = [
+        urls = []
-            {**dict(r), "short_url": f"{base}/{r['short_code']}"}
+        for r in rows:
-            for r in rows
+            urls.append({
-        ]
+                "short_code": r["short_code"],
+                "short_url": f"{base}/{r['short_code']}",
+                "original_url": r["original_url"],
+                "created_at": r["created_at"],
+                "visit_count": r["visit_count"],
+            })
         self._send_json(200, {"count": len(urls), "urls": urls})
 
     def _handle_get_url(self, code: str) -> None:
-        if not self._authenticated():
+        # No API key required
-            self._error(401, "Unauthorized")
-            return
-
         with db_connect(self.cfg["db_path"]) as conn:
             row = conn.execute(
                 "SELECT * FROM urls WHERE short_code = ?", (code,)
             ).fetchone()
 
         if row is None:
-            self._error(404, "Short code not found")
+            self._send_empty(404)
             return
 
         base = self.cfg["base_url"].rstrip("/")
-        self._send_json(200, {**dict(row), "short_url": f"{base}/{code}"})
+        self._send_json(200, {
+            "short_code": row["short_code"],
+            "short_url": f"{base}/{row['short_code']}",
+            "original_url": row["original_url"],
+            "created_at": row["created_at"],
+            "visit_count": row["visit_count"],
+        })
 
     def _handle_delete_url(self, code: str) -> None:
-        if not self._authenticated():
+        # API key required — 403 if not authorized (no body)
-            self._error(401, "Unauthorized")
+        if not self._check_api_key():
+            self._send_empty(403)
             return
 
         with db_connect(self.cfg["db_path"]) as conn:
@@ -292,12 +401,14 @@ class Handler(BaseHTTPRequestHandler):
                 "SELECT short_code FROM urls WHERE short_code = ?", (code,)
             ).fetchone()
             if row is None:
-                self._error(404, "Short code not found")
+                # 404 if not found (no body)
+                self._send_empty(404)
                 return
             conn.execute("DELETE FROM urls WHERE short_code = ?", (code,))
             conn.commit()
 
-        self._send_json(200, {"message": f"Deleted '{code}'"})
+        # 204 on success (no body)
+        self._send_empty(204)
 
     def _handle_redirect(self, code: str) -> None:
         with db_connect(self.cfg["db_path"]) as conn:
@@ -319,8 +430,9 @@ class Handler(BaseHTTPRequestHandler):
     # Internal utils
     # ------------------------------------------------------------------
 
-    def _unique_code(self, conn: sqlite3.Connection, attempts: int = 10):
+    def _unique_code(self, conn: sqlite3.Connection, length: int = None, attempts: int = 10):
-        length = self.cfg["short_length"]
+        if length is None:
+            length = self.cfg["short_length"]
         for _ in range(attempts):
             code = generate_code(length)
             exists = conn.execute(
@@ -342,6 +454,7 @@ def main() -> None:
 
     cfg = load_config(sys.argv[1])
     init_db(cfg["db_path"])
+    cleanup_expired(cfg["db_path"])
 
     Handler.cfg = cfg
 
@@ -349,8 +462,10 @@ def main() -> None:
     server = HTTPServer((host, port), Handler)
 
     print(f"URL Shortener listening on http://{host}:{port}", file=sys.stderr)
-    print(f"Base URL : {cfg['base_url']}", file=sys.stderr)
+    print(f"Base URL       : {cfg['base_url']}", file=sys.stderr)
-    print(f"DB path  : {cfg['db_path']}", file=sys.stderr)
+    print(f"Base path      : {cfg['base_path'] or '/'}", file=sys.stderr)
+    print(f"DB path        : {cfg['db_path']}", file=sys.stderr)
+    print(f"Retention days : {cfg['retention_days']}", file=sys.stderr)
 
     try:
         server.serve_forever()