Quick answer
How to mirror a website on a VPS?
To mirror a website on a VPS, first make sure the site is yours or its licence allows copies, read its robots.txt, run wget --mirror --convert-links --page-requisites with a wait between requests, then serve the folder with nginx. On an IronKVM Anvil, 240 GB of NVMe holds large mirrors for $15.00 a month.
In brief
- Mirror only your own sites, or content whose licence allows copies and redistribution.
- wget respects robots.txt by default; keep it that way.
--waitand--random-waitspace requests out so the source server is not overloaded.--convert-linksrewrites links so the copy works from your server.- Ignoring DMCA notices does not make illegal content acceptable. What runs on an IronKVM VPS must be legal in the country of its datacenter.
Which wget options make a good mirror?
| Option | What it does |
|---|---|
| --mirror | Recursion with timestamps and infinite depth (-r -N -l inf --no-remove-listing) |
| --convert-links | Rewrites links after download for local viewing |
| --adjust-extension | Adds .html to HTML pages whose URL lacks it |
| --page-requisites | Downloads the images, CSS and scripts a page needs |
| --no-parent | Never climbs above the starting folder |
| --wait=2 --random-wait | Pauses between 1 and 3 seconds between requests |
| --limit-rate=500k | Caps the download speed |
What are the steps to mirror a website on a VPS?
Check that you may copy it
Mirror your own site, or content under a licence that allows copies, such as public domain or a Creative Commons licence. Keep the licence notice.
Read robots.txt
Open https://example.org/robots.txt and note the paths that crawlers are asked to avoid.
Install wget
Install wget on the VPS if the image does not include it.
Run the mirror politely
Use --mirror with --wait, --random-wait and --limit-rate, and a clear user agent.
Check the copy
Open a few pages from disk and look for broken images or links.
Serve it with nginx
Point a server block at the folder, test with nginx -t and reload.
Refresh it on a schedule
Run the same command weekly from cron; --mirror only fetches changed files.
sudo apt install wget
mkdir -p /var/www/mirror && cd /var/www/mirror
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent \
--wait=2 --random-wait --limit-rate=500k \
--user-agent="mirror-bot (contact: [email protected])" \
https://example.org/docs/server {
listen 80;
server_name mirror.your-domain.example;
root /var/www/mirror/example.org;
index index.html;
}When is it allowed to mirror a website on a VPS?
When the site is yours, or its licence or owner allows copies. Good cases are your own sites, documentation of free software, public domain texts and Creative Commons works whose licence permits sharing. A mirror of such content keeps it reachable if the original goes down.
Copying someone else’s protected work without permission is not one of them. The acceptable use policy applies to mirrors like any other content. IronKVM takes no action on DMCA takedown notices or on other foreign copyright notices: the VPS stays online and no customer data is handed over because of such a notice. Ignoring DMCA notices does not make illegal content acceptable. What runs on an IronKVM VPS must be legal in the country of its datacenter. Where a licence asks for attribution, keep the notice on every page of the copy.
Why respect robots.txt when you mirror a site?
Because it is how the site owner tells crawlers what to leave alone. RFC 9309, published in September 2022, standardises the Robots Exclusion Protocol: a /robots.txt file with user-agent groups and allow or disallow rules. The RFC also notes that these rules are not access control. Honouring them is a choice, and the right one.
The wget manual says it respects the Robot Exclusion Standard. Do not turn that off. Add a wait between requests and a speed limit, and put a contact address in the user agent so the owner can reach you. A polite mirror runs slower but never looks like an attack.
Which VPS plan is best for a website mirror?
Size the plan by disk first: a mirror needs room for every page and image. Billet has 40 GB NVMe, Anvil 240 GB NVMe and Forge 400 GB NVMe. Traffic counts too, since visitors download from you: 12 TB a month on Anvil.
According to IronKVM's survey of 16 November 2026, plans comparable to Forge cost a median of $142.32 a month across 6 hosts; IronKVM Forge costs $20.63 on 12 months. Serve the copy as shown in how to host a website on an offshore VPS, and keep a second copy with the VPS backup guide.
Mirroring a website on a VPS: questions
Can I mirror any website I like?
No. Mirror your own sites or content whose licence or owner allows copies, and respect robots.txt.
How do I keep a mirror up to date?
Run the same wget command from cron, for example weekly. --mirror uses timestamps and fetches only changed files.
Why are some pages missing from my mirror?
robots.txt may disallow them, or they load through JavaScript, which wget does not run.
What is the best tool to mirror a website in 2026?
wget is the standard choice for static sites. For sites that offer rsync or a Git repository, those copy faster and more exactly.
Does IronKVM allow archive and mirror sites?
Yes, for legal content. Archives and mirrors of free content are allowed uses; the acceptable use policy still applies.
Verdict
- Check the licence, respect robots.txt, wait between requests: a mirror is then useful and polite.
- IronKVM Anvil or Forge give the disk for large mirrors of your own or freely licensed sites.
Sources (4)
- wget(1), Debian trixie — Debian (read 2026-11-16)
- RFC 9309: Robots Exclusion Protocol — IETF (read 2026-11-16)
- Beginner’s guide — nginx (read 2026-11-16)
- IronKVM offshore VPS survey (CSV) — IronKVM (read 2026-11-16)
Host a mirror of free content
Anvil, 6 vCPU, 16 GB RAM, 240 GB NVMe, 12 TB traffic, $15.00 a month on 12 months.