Writing Code That Fetches User-Supplied URLs With LLMs in November 2026: Allow Only the Schemes You Need, Block Internal Addresses After DNS Lookup, and Cap Redirects and Response Size
Plenty of features start with "the user pastes a URL and we fetch it": link previews, webhook test buttons, "import from URL" for images or feeds, RSS readers, and AI agents that read a page on request. Ask an LLM for this code and you will usually get three working lines: take the string, call your HTTP library, return the body. It works on the first try with a real website. It also lets anyone on the internet make your server send requests to places only your server can reach.
That class of bug is called server-side request forgery (SSRF), and it is on the OWASP Top 10 for a reason. This post shows how to get an LLM to write URL-fetching code that is safe, what to put in the prompt, what to check in the result, and how to test it with inputs the model will not think of on its own.
A URL from a user should pass a checkpoint before your server goes anywhere on its behalf. Image: Checkpoint Charlie by The Central Intelligence Agency, public domain, via Wikimedia Commons.
Why the three-line version is dangerous
Your server sits inside a network that outsiders cannot see. When it fetches a URL on a user's behalf, the request comes from inside that network. A user who submits these instead of a normal website can reach things you never meant to expose:
http://169.254.169.254/, the cloud instance metadata address used by AWS, Google Cloud, Azure and others. On some setups it hands out temporary credentials for the machine's role.http://localhost:6379/orhttp://127.0.0.1:9200/, internal services like Redis, Elasticsearch, or an admin panel that only listens locally and assumes everyone who can reach it is trusted.http://10.0.3.17/orhttp://192.168.1.1/, other machines on your private network.file:///etc/passwdorgopher://..., if the library you use supports schemes other than HTTP.
Even if the response is never shown to the user, they can often learn things from timing and error messages, such as which internal ports are open. If the response is shown, as in a link preview, they can read internal data directly.
Step 1: Tell the model what the feature is actually for
The model cannot guess your threat model. Give it the purpose, the runtime, and the limits up front. A prompt that works well looks like this:
I need a Python function fetch_user_url(url: str) -> bytes for a link-preview
feature. The URL comes from untrusted users. Use the httpx library.
Requirements:
- Allow only http and https schemes. Reject everything else.
- Resolve the hostname with DNS and reject the request if ANY resolved address
is loopback, private, link-local, multicast, reserved, or unspecified,
for both IPv4 and IPv6 (including IPv4-mapped IPv6 like ::ffff:127.0.0.1).
- Connect to the IP address you checked, not a second fresh lookup, so DNS
cannot change between the check and the request.
- Do not follow redirects automatically. Follow at most 3 manually,
running every redirect target through the same checks.
- Allow only ports 80 and 443.
- Connect timeout 3s, total timeout 10s.
- Stop reading after 2 MB and raise an error.
- Do not send cookies, auth headers, or internal headers.
- Raise a specific exception type for each rejection reason.
Write the function and a pytest file that covers each rule.
Every line in that list is something the model tends to leave out when you just ask for "a function that fetches a URL." Spelling them out turns a security review into a checklist you can tick off.
Step 2: Check the scheme and port with a real URL parser
Look at how the generated code reads the URL. It should use a proper parser (urllib.parse.urlsplit in Python, the URL class in JavaScript) and check the parsed scheme and port. Reject any version that uses string tricks like url.startswith("http"), which also accepts httpfoo://, or a regex that looks for "localhost" in the text.
Watch for these details in the review:
- The scheme check should be an allowlist (
{"http", "https"}), not a blocklist of bad schemes. You will never list them all. - The port should come from the parsed URL, with the default filled in (80 for http, 443 for https) when it is missing.
- URLs with a username and password part, like
http://user:pass@host/, should usually be rejected. They are a common way to confuse a human reviewer about which host is really being contacted.
Step 3: Check the IP address, not the hostname
This is where most LLM drafts go wrong. A typical first attempt checks whether the hostname equals localhost or starts with 192.168.. That fails against inputs like these, all of which can point at your own machine:
http://2130706433/(127.0.0.1 written as one decimal number; many resolvers accept it)http://0x7f.0.0.1/orhttp://0177.0.0.1/(hex and octal forms)http://[::1]/andhttp://[::ffff:127.0.0.1]/(IPv6 loopback and IPv4-mapped IPv6)http://my-trick.example.com/, a normal-looking domain whose DNS record simply points to 127.0.0.1 or 10.0.0.5
The fix is to resolve the hostname first and check every address that comes back. In Python, the standard ipaddress module does the classification for you:
import ipaddress, socket
def is_public_ip(ip_str: str) -> bool:
ip = ipaddress.ip_address(ip_str)
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped:
ip = ip.ipv4_mapped
return ip.is_global and not ip.is_multicast
def resolve_public(host: str, port: int) -> list[str]:
infos = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
addrs = sorted({info[4][0] for info in infos})
if not addrs or not all(is_public_ip(a) for a in addrs):
raise BlockedAddress(host)
return addrs
Two things to verify in whatever the model writes. First, it should reject the request if any resolved address is internal, not just the first one. Second, it should use a positive test like is_global rather than chaining a few negative checks such as "not private and not loopback," which forgets link-local (where 169.254.169.254 lives) more often than you would expect.
Check the number you actually connect to, not just the one you were asked for. Image: Photograph of Women Working at a Bell System Telephone Switchboard by The U.S. National Archives, no known copyright restrictions, via Wikimedia Commons.
Step 4: Close the gap between checking and connecting
Even a correct IP check can be bypassed if the code looks up the hostname once to check it, then hands the hostname to the HTTP library, which looks it up again. An attacker who controls a domain's DNS can answer the first lookup with a public address and the second with 127.0.0.1. This is called DNS rebinding, and short DNS TTLs make it easy.
There are three reasonable ways to handle it. Ask the model which one its code uses, and make it say so in a comment:
- Connect to the checked IP directly. Open the connection to the IP address you validated and send the original hostname in the
Hostheader (and as the TLS server name for HTTPS). This is fiddly in some libraries, so test it against a real HTTPS site before you trust it. - Check at connect time. Some HTTP clients let you hook the step where the socket is opened. Running the IP check there means you are checking the exact address being connected to.
- Use an egress proxy. Route all user-URL fetches through a proxy that enforces the rules for you, such as Stripe's open-source Smokescreen. This also protects code paths you forget about, and it is the most robust option if many services fetch URLs.
Whichever you pick, it should be a conscious decision. If the model's code calls getaddrinfo for the check and then passes the hostname string to httpx.get, the rebinding gap is still open.
Step 5: Handle redirects, timeouts and size by hand
Most HTTP libraries follow redirects automatically. That means a public URL you checked can answer with 302 Location: http://169.254.169.254/, and the library will happily go there without running any of your checks. Turn automatic redirects off and loop manually, sending each Location through the full validation again, with a small maximum like 3 or 5.
Also check that the code sets limits on the cost of each request:
- Timeouts. Both a connect timeout and an overall timeout. Without them, a URL that drips one byte every few seconds can tie up a worker for a long time.
- Response size. Stream the body and stop once it passes your limit. Do not trust the
Content-Lengthheader alone, because the server can lie or leave it out. Also be aware that compressed responses can expand a lot when decoded, so count bytes after decompression. - Content type. If you only want HTML or images, check the response type before doing anything expensive with the body.
Step 6: Test with the inputs an attacker would try
The model will happily write tests that fetch https://example.com and assert success. Those prove almost nothing. Make it write a rejection test for each of these, and add any internal hostnames specific to your setup:
BAD_URLS = [
"file:///etc/passwd",
"gopher://127.0.0.1:6379/_INFO",
"ftp://example.com/",
"http://127.0.0.1/",
"http://localhost/",
"http://2130706433/",
"http://0x7f.0.0.1/",
"http://[::1]/",
"http://[::ffff:127.0.0.1]/",
"http://169.254.169.254/latest/meta-data/",
"http://10.0.0.1/",
"http://192.168.0.1/",
"http://0.0.0.0/",
"http://example.com:22/",
"http://user:pass@example.com/",
]
Then add tests that need a little setup:
- A redirect test: a small local test server, or a mocked response, that returns a 302 pointing at 127.0.0.1. The fetch must fail.
- A DNS test: mock
getaddrinfoto return one public and one private address for the same name. The fetch must fail. - A size test: a response larger than your limit. The fetch must stop reading and raise.
- A slow test: a server that never responds. The fetch must time out within your limit.
Run each test once against a deliberately broken version of the function (for example, with the IP check commented out) and confirm it fails. A test that cannot fail is not protecting you.
Defense in depth outside the code
Application checks are the first line, not the only one. A few infrastructure settings shrink the damage if something slips through:
- Lock down the metadata service. On AWS, require IMDSv2, which needs a session token obtained with a PUT request before metadata can be read. A plain GET from an SSRF bug then fails.
- Run fetchers in their own network zone. A worker that only fetches user URLs should have outbound internet access and nothing else: no route to your databases or internal services.
- Give that worker minimal credentials. If the fetcher's cloud role can do nothing interesting, stolen credentials are worth little.
- Log blocked attempts. A burst of rejections for internal addresses is a useful signal that someone is probing.
A short review checklist
Before you merge LLM-written URL-fetching code, check that:
- The scheme is parsed and checked against an allowlist of http and https.
- Only expected ports are allowed.
- The hostname is resolved, and every resolved address must be public, including IPv6 and IPv4-mapped forms.
- The connection goes to the checked address, or the check happens at connect time, or a filtering proxy is used.
- Redirects are followed manually, re-checked each time, with a small cap.
- Connect and total timeouts are set.
- The body is streamed with a hard size limit counted after decompression.
- No cookies or internal auth headers are sent with the request.
- Tests cover every bad URL above, plus redirect, DNS, size and timeout cases, and each test has been seen to fail.
The model is good at writing the plumbing once you tell it exactly what the rules are. What it will not do on its own is assume the person typing the URL is trying to reach your internal network. Write that assumption into the prompt, test for it, and the "paste a link" feature stops being a door into your servers.
Comments
Post a Comment