Last winter I set up a cold backup for my NAS: two 4TB drives, rsync running for almost eleven hours, and when it finished I only checked that the exit code was 0 and called it done. Two months later I needed to restore a compressed archive of my bookkeeping records, and extraction failed with unexpected end of archive. The archive had been 2.1GB, but only 1.4GB was left on the drive; one of the writes had quietly stopped partway. I had no way to prove the other three hundred or so archives were still good. Since then my backup routine has one more step: compute the hashes, then store the manifest on another machine. What a hash can actually prove, and which problems it cannot help with, is something I only figured out afterwards.
1. A hash is a fingerprint of the content, not encryption
MD5 squeezes data of any length into 128 bits, usually written as a string of 32 hexadecimal characters. Three properties decide what it can do. The output is fixed length: a 3-byte text and a 4GB disk image produce results of the same length. It is one-way: given 5d41402abc4b2a76b9719d911017c592 you cannot work back to the original (it is the MD5 of hello; the empty string gives d41d8cd98f00b204e9800998ecf8427e). The avalanche effect: flip one bit and the result is unrecognisable.
So the phrase MD5 encryption is simply wrong. A hash only answers two questions: are these two pieces of data the same, and is this data still the copy I received? To keep something secret you need encryption; to store a password you need a slow hash with a salt. Neither should be replaced by MD5.
2. The real question in verification is where the expected value comes from
Many people remember the flow as download, hash, compare. But what decides whether that conclusion is worth anything is where the reference value in the third step came from. Split it apart:
| Source of the expected value | What a pass proves | What a failure proves |
|---|---|---|
| Published separately on the official HTTPS page | The file matches the official copy and did not break in transit | Truncated transfer, a swapped file or the wrong version; download again |
| Same third-party download site as the file | Only that the two shipped together, which says almost nothing about safety | The site's own notes were copied wrong |
| GPG or code signature with a separate public key | The source is trusted and unmodified, which is real tamper protection | The signature is invalid, do not install |
| A manifest I recorded at the last backup | The file has not changed or failed in the meantime | The media or the write has a problem; find another copy now |
The middle row is what I would most want written into the manual: when an attacker swaps an installer, editing the MD5 on the page takes only seconds. A hash guarantees integrity, not authenticity. Authenticity depends on signatures and on trusting the channel.
3. How I actually compute hashes, and with which commands
Start with the commands built into the system, which work without a network:
| Platform | Command | Output |
|---|---|---|
| Windows (cmd) | certutil -hashfile app.iso MD5 | 32 hex characters |
| Windows (PowerShell) | Get-FileHash -Algorithm SHA256 app.iso | 64 hex characters |
| macOS | md5 app.iso / shasum -a 256 app.iso | MD5 / SHA-256 |
| Linux, Git Bash | sha256sum app.iso; for a whole batch sha256sum -c SHA256SUMS | OK / FAILED per line |
When I only have one or two files and do not want to type paths, I drag them straight into File Hash Calculator. Drop in a batch and it returns MD5, SHA-1 and SHA-256 at once, along with file name, size, type and modified time, and files with identical hashes are marked in red. Clearing out duplicate installers last time, that red flag is how I picked them. The computation uses the browser's built-in interface and the files are not uploaded; for images of tens of GB I still trust the command line more.
On the backup side my routine has three fixed steps: copy, generate a SHA256SUMS.txt on the source side, then copy the manifest to another machine. Before a restore I run -c once and can see exactly which of the two hundred plus archives are broken. For short text strings (the plaintext used for API signatures) I open a separate page; MD5 Tool supports 16 and 32 characters, upper or lower case, and multi-line batches.
4. Can MD5 still be trusted
The answer depends on who your opponent is. MD5 collisions have been constructed in practice, and in 2008 researchers used them to forge a CA certificate, so it should no longer be used anywhere adversarial: software signatures, certificate chains and password storage should all move to SHA-256 at minimum. On the other hand, for a non-adversarial question such as whether my file was cut off during a copy, it is still fine, because a broken file is not something someone built specifically against your data.
| Use case | Suitable algorithm | Why |
|---|---|---|
| Download integrity, backup self-check | MD5 or SHA-256 | You only need to catch accidental damage |
| Installers and disk images published to the world | SHA-256 plus a signature | MD5 can be collided, so it does not stop a deliberate swap |
| Password storage | bcrypt / scrypt / Argon2 | MD5 is too fast; even a salt will not save it |
| API signing, replay protection | HMAC-SHA256 | You need a keyed authentication code; Hash Calculator returns six results from MD5 to SHA-512 and supports HMAC |
| File deduplication | Start with MD5 | It is fast; spot-check the content of important data after deduplicating |
5. A few traps that waste your debugging time
- 16 and 32 characters are not two algorithms: the 16-character form is just characters 9 to 24 of the 32-character result. Check the length before comparing, or you will conclude that nothing matches at all.
- Upper and lower case:
a-fandA-Fare the same value in a different display format. Normalise to lower case before a script compares them. - Line endings change a text hash: the same
README.mdis CRLF on Windows and LF on Linux, so the MD5 must differ, and Git'score.autocrlfproduces the same effect. Normalise line endings before comparing text hashes. - Renaming and changing the time does not affect the hash: a fingerprint covers content only. The intuition that renaming a file should change its hash is wrong, and that is exactly why a hash is a good way to tell whether two contents are the same.
- The outer archive and the files inside are different things: a matching zip hash only proves the archive is intact. Whether something inside was swapped needs a per-file manifest.
- A directory has no single hash: file order, permissions and timestamps all change the packed result. To verify a whole directory, use a per-file manifest or rsync's checksum mode.
6. Common questions
Q: If two files have the same MD5, are they certainly identical?
For everyday purposes you can treat it that way; the chance of random damage producing a match is negligible. You only need a stronger algorithm when you face an opponent who deliberately constructs collisions.
Q: The official site only lists SHA-256 but my machine only has md5sum. What then?
Change the tool, not the file: PowerShell's Get-FileHash, macOS's shasum -a 256 and Linux's sha256sum all compute it directly, so there is no need to download again.
Q: What should I do when a check does not match?
Download once more and hash it again. If it still does not match, treat the file as modified: do not run it, do not use anything inside it, and check whether you grabbed the wrong version or pulled it from a mirror site.
Q: Do backups really need a hash manifest? Can I just recopy on a schedule?
A recopy cannot tell you whether the copy on the drive has already failed; it is a gamble. A manifest costs a few tens of seconds of computing, and the payoff is knowing which archives are usable before you restore.
Q: How long does a large file take?
Disk reads dominate, not the algorithm; on the same drive the time difference between SHA-256 and MD5 is often smaller than the read speed itself. A few minutes on a mechanical drive is normal, and in a browser very large files also run into memory limits.
That broken archive caused no real loss in the end: the same batch of data had an older copy on another machine, only two months out of date. What really changed was the process. Now the moment a backup finishes I write the hash manifest to a second storage device, because having a backup and knowing the backup is good are two different things.