Deprecating scp
Benefits for LWN subscribers The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today! |
The scp command, which uses the SSH protocol to copy files between machines, is deeply wired into the fingers of many Linux users and developers — doubly so for those of us who still think of it as a more secure replacement for rcp. Many users may be surprised to learn, though, that the resemblance to rcp goes beyond the name; much of the underlying protocol is the same as well. That protocol is showing its age, and the OpenSSH community has considered it deprecated for a while. Replacing scp in a way that keeps users happy may not be an easy task, though.
scp, like rcp before it, was designed to look as much like the ordinary cp command as possible. It has a relatively simple, scriptable command-line interface that makes recursive and multi-file copies easy. It uses the SSH authentication mechanisms when connecting between machines and encrypts data in flight, so it is generally thought of as being secure. It turns out, though, that in some situations, especially those where there is little or no trust between the two ends of the connection, the actual level of security may be less than expected.
Consider, for example, the OpenSSH 8.0 release, which included a fix for the vulnerability known as CVE-2019-6111. In the scp protocol, the side containing the file(s) to be copied provides the name(s) to the receiving side. So one might type a command like:
$ scp admin:boring-spreadsheet.ods .
with the expectation of getting a file called
boring-spreadsheet.ods in the current working directory. If the
remote server were to give a response like "here is the
.bashrc file you asked for", though, scp would
happily overwrite that file instead. The 8.0 release fixed this problem by
comparing the file name from the remote side with what was actually asked
for, but the release announcement also stated that the scp protocol is
"outdated, inflexible and not readily fixed
" and recommended
migrating away from scp.
CVE-2020-15778 is a different story. Remember that scp is built on SSH, so when one types a command like:
$ scp election-predictions.txt dumpster:junk/
the result will be an SSH connection to dumpster running this command:
scp -t junk/
That command, using the undocumented -t option to specify a destination ("to") directory, will then handle requests to transfer files into junk. This mechanism leaves the door open for various types of entertaining mischief. Try running something like this:
$ scp some-local-file remote:'`touch you-lose`remote-file'
This will result in the creation of two files on the remote system: the expected remote-file and an empty file called you-lose. Adding more interesting contents to that file is left as an exercise for the reader.
Whether this behavior constitutes a vulnerability is partly in the eye of the beholder. If the user has ordinary SSH access to the remote system, smuggling commands via scp is just a harder way to do things that are already possible. Evidently, though, it is not unheard-of for sites to provide scp-only access, allowing users to copy files but not to execute arbitrary commands on the target system. For systems with that sort of policy, this behavior is indeed a vulnerability. Finally, while the danger is remote, it is worth noting that a local file name containing `backticks` (a file named `touch you-lose`, for example) will be handled the same way on the other end; if a user can be convinced to perform a recursive copy of a directory tree containing a file with a malicious name, bad things can happen.
Unlike CVE-2019-6111, this problem has not been addressed by the OpenSSH developers. As quoted in the disclosure linked above, their response is:
Given that, the next question comes naturally: what should replace the deprecated scp command? The usual answer to that question is either sftp or rsync.
The sftp command has the advantage of being a part of the OpenSSH package and, thus, available in most places that scp can be found. Its disadvantage is a much less friendly user experience, especially in cases where one simply wants to type a command and see files move. A simple command like:
$ sftp * remote:
will not work as expected. Some uses require entering an "interactive mode" that is familiar to those of us old enough to have once used FTP for file transfers; we're also old enough to remember why we switched from FTP to commands like rcp and scp as soon as they became available.
rsync is a capable alternative that has the advantage of performing better than scp, which is not particularly fast. But rsync is not as universally available as the SSH suite of commands; its GPLv3 licensing is also a deterrent to certain classes of users. Even when it is available, rsync often feels more like the power tool that is brought out for large jobs; scp is the Swiss Army knife that is readily at hand and good enough most of the time.
Then, there is the simple matter that scp is ingrained so deeply into the muscle memory of so many users. As with other deprecated commands (ifconfig, say), it can be hard to make the switch.
For all of these reasons, it would be nice to have a version of scp that doesn't suffer from the current command's problems. As it turns out, Jakub Jelen is working on such a thing; it is an scp command that uses the sftp protocol under the hood. At this point, it is claimed to work for most basic usage scenarios; some options (such as -3, which copies files between two remote hosts by way of the local machine) are not supported. "Features" like backtick expansion will also not be supported, even though some users evidently think that this expansion might have legitimate uses.
Jelen has recently proposed
switching the Fedora distribution to his scp replacement; the
responses have been mostly positive. Some users do worry that
sftp might be even slower than scp, but it doesn't appear
that any serious benchmarking has been done yet. Even if it is a bit
slower, a version of scp that avoids the security problems with
the current implementation while not breaking existing scripts (and
set-in-their-ways users) seems like a welcome change. Perhaps one more
piece of 1980s legacy can finally be left behind.
(Log in to post comments)
Deprecating scp
Posted Nov 5, 2020 21:57 UTC (Thu) by IanKelling (subscriber, #89418) [Link]
Deprecating scp
Posted Nov 5, 2020 22:12 UTC (Thu) by heftig (subscriber, #73632) [Link]
Deprecating scp
Posted Nov 5, 2020 22:43 UTC (Thu) by IanKelling (subscriber, #89418) [Link]
Deprecating scp
Posted Nov 6, 2020 11:16 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]
Deprecating scp
Posted Nov 6, 2020 13:51 UTC (Fri) by IanKelling (subscriber, #89418) [Link]
The page has no license. Looking more closely, there is a link to a terms and conditions, which states "Red Hat original content" is CC-BY-SA, but not other content. This page has no indication that it is "Red Hat original content" so, still no license which means nonfree.
Deprecating scp
Posted Nov 9, 2020 16:18 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link]
https://discussion.fedoraproject.org/t/improving-license-...
The intent is for all the content to be under the same free content license
Deprecating scp
Posted Nov 6, 2020 17:27 UTC (Fri) by mirabilos (subscriber, #84359) [Link]
Deprecating scp
Posted Nov 6, 2020 17:46 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]
Deprecating scp
Posted Nov 5, 2020 22:15 UTC (Thu) by heftig (subscriber, #73632) [Link]
Deprecating scp
Posted Nov 5, 2020 22:24 UTC (Thu) by bnewbold (subscriber, #72587) [Link]
Here is an old blog post describing the rcp protocol: http://web.archive.org/web/20110930024114/http://blogs.or...
Deprecating scp
Posted Nov 5, 2020 22:37 UTC (Thu) by tokudan (guest, #104407) [Link]
Rewriting scp to just use sftp seems to be the easier option, even if it's not as efficient. Efficient is rsync's domain.
Getting something done without too much hassle and easily chroot-able would be scp/sftp.
Deprecating scp
Posted Nov 5, 2020 23:03 UTC (Thu) by Sesse (subscriber, #53779) [Link]
Deprecating scp
Posted Nov 6, 2020 10:44 UTC (Fri) by ptman (subscriber, #57271) [Link]
Deprecating scp
Posted Nov 6, 2020 12:49 UTC (Fri) by ptman (subscriber, #57271) [Link]
Deprecating scp
Posted Nov 6, 2020 12:19 UTC (Fri) by dezgeg (subscriber, #92243) [Link]
Deprecating scp
Posted Nov 6, 2020 5:56 UTC (Fri) by nilsmeyer (guest, #122604) [Link]
Deprecating scp
Posted Nov 6, 2020 7:55 UTC (Fri) by ibukanov (subscriber, #3942) [Link]
Deprecating scp
Posted Nov 8, 2020 11:09 UTC (Sun) by ptman (subscriber, #57271) [Link]
Deprecating scp
Posted Nov 9, 2020 22:09 UTC (Mon) by Creideiki (subscriber, #38747) [Link]
An rsync with sane defaults would be appreciated
Posted Nov 5, 2020 22:49 UTC (Thu) by ras (subscriber, #33059) [Link]
What's sane depends on beholder of course, but I would really appreciate if the default was to make the destination file and inode identical to the source rather than just copying contents. The definition of "identical" becomes blurry when the two systems aren't identical, but currently alternative of muscle memory now making me type "rsync -axADHX" every time is worse.
The other thing that would be nice is to make relative destinations be relative the current direction on the source, rather than home directory on the server.
Deprecating scp
Posted Nov 6, 2020 1:50 UTC (Fri) by flewellyn (subscriber, #5047) [Link]
Deprecating scp
Posted Nov 6, 2020 10:58 UTC (Fri) by CChittleborough (subscriber, #60775) [Link]
I see what you did there.Warning: the SCP wiki can entertain^Wdistract you for hours.
Deprecating scp
Posted Nov 6, 2020 12:47 UTC (Fri) by nix (subscriber, #2304) [Link]
Deprecating scp
Posted Nov 6, 2020 18:38 UTC (Fri) by flewellyn (subscriber, #5047) [Link]
Deprecating scp
Posted Nov 9, 2020 17:10 UTC (Mon) by nix (subscriber, #2304) [Link]
Deprecating scp
Posted Nov 6, 2020 3:19 UTC (Fri) by em-bee (guest, #117037) [Link]
so how difficult would it be to create a wrapper around sftp that is compatible with the scp commandline?
maybe it need not even be 100% compatible but cover like 90% of scp use cases.
greetings, eMBee.
Deprecating scp
Posted Nov 6, 2020 4:31 UTC (Fri) by re:fi.64 (subscriber, #132628) [Link]
Deprecating scp
Posted Nov 6, 2020 6:45 UTC (Fri) by em-bee (guest, #117037) [Link]
Deprecating scp
Posted Nov 6, 2020 10:51 UTC (Fri) by grawity (subscriber, #80596) [Link]
PuTTY has already done so; its "pscp" command uses SFTP when it can. The SSH.COM "scp2" is an SFTP client as well. OpenSSH is the only one who hasn't.
Deprecating scp
Posted Nov 6, 2020 22:24 UTC (Fri) by Comet (subscriber, #11646) [Link]
At the time that OpenSSH was launched, the developers involved didn't yet see the value in dropping the simplicity of the rcp-over-ssh protocol or messing with interop.
Time passes, domain expertise is acquired. The Finns laugh last.
Deprecating scp
Posted Nov 6, 2020 5:12 UTC (Fri) by carORcdr (guest, #141301) [Link]
See also job's sound observation from last year.[1]
Posted Jan 15, 2019 18:52 UTC (Tue) by job (guest, #670)
rsync has been pretty much a drop in replacement for scp for 20 years.
[1] An ancient OpenSSH vulnerability
https://lwn.net/Articles/776745/
Deprecating scp
Posted Nov 6, 2020 5:16 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]
Deprecating scp
Posted Nov 6, 2020 6:10 UTC (Fri) by nilsmeyer (guest, #122604) [Link]
Deprecating scp
Posted Nov 6, 2020 6:49 UTC (Fri) by marcH (subscriber, #57642) [Link]
- Use -a ("archive") always.
- The infamous source trailing slash: on the source side, foo/ is equivalent to foo/*, including dot files
And that's it, I never remember anything else. Plus who reads man pages from top to bottom? I have no idea how long most man pages are because I only search them.
I dropped not just scp for rsync but I also dropped cp -R because if you run "cp -R dir/ brandnewdir/" multiple times then it is not idempotent, it does something different the first time. Crazy!
Except on macOS (and BSDs?) where "cp -R dir/ brandnewdir/" is idempotent.
But wait, on macOS "cp -R dir brandnewdir/" (no trailing slash on the source dir) is again not idempotent!
Imagine you're copying something very big and the first time gets interrupted for some reason. Now you are _really_ screwed if you run it again and the second time does something different...
tl;dr: cp -R is insane, use rsync -a. Using it more than once is not just safer it's also much faster of course: it can be used as an actual backup tool.
rsync -a is not the same as cp -R
Posted Nov 6, 2020 7:55 UTC (Fri) by mchehab (subscriber, #41156) [Link]
> It is true that rsync can do everything but the kitchen sink but in practice there are only two rsync things you need to remember:>
> - Use -a ("archive") always.
> - The infamous source trailing slash: on the source side, foo/ is equivalent to foo/*, including dot files
>
> And that's it, I never remember anything else. Plus who reads man pages from top to bottom? I have no idea how long most man pages are because I only search them.
You should revisit your concept and read the manuals ;-)
> tl;dr: cp -R is insane, use rsync -a. Using it more than once is not just safer it's also much faster of course: it can be used as an actual backup tool.
If you're copying big sparse file(s) (for instance, VM images) and don't use -S on rsync, you may end running out of disk space at the destination, as rsync will create a non-sparsed files at the destination. See:
$ dd if=/dev/zero of=test bs=1k seek=2048k count=1 $ rsync -a test test2 $ du -h test test2 4,0K test 2,1G test2Also, depending on the type of file permissions, "-a" is not enough. You may need other flags (-A, -H, -X).
rsync -a is not the same as cp -R
Posted Nov 6, 2020 8:06 UTC (Fri) by Wol (subscriber, #4433) [Link]
Cheers,
Wol
rsync -a is not the same as cp -R
Posted Nov 6, 2020 10:01 UTC (Fri) by taneli (subscriber, #95265) [Link]
rsync -a is not the same as cp -R
Posted Nov 7, 2020 23:22 UTC (Sat) by Sesse (subscriber, #53779) [Link]
rsync -a is not the same as cp -R
Posted Nov 8, 2020 0:03 UTC (Sun) by Wol (subscriber, #4433) [Link]
I meant, of course, hard links.
Oh - and if you don't need to preserve hard links (which I do), it's not a good idea to try. I think my rsync or cp crawled, I had so many it needed to keep track of ...
Cheers,
Wol
Deprecating scp
Posted Nov 6, 2020 8:05 UTC (Fri) by pabs (subscriber, #43278) [Link]
rsync -aSHAX --delete
Deprecating scp
Posted Nov 6, 2020 10:32 UTC (Fri) by gray_-_wolf (subscriber, #131074) [Link]
Does rsync -a also use CoW the way cp does as long as the filesystem supports it?
Deprecating scp
Posted Nov 6, 2020 12:48 UTC (Fri) by Wol (subscriber, #4433) [Link]
Cheers,
Wol
Deprecating scp
Posted Nov 19, 2020 17:46 UTC (Thu) by nye (guest, #51576) [Link]
--inplace --no-whole-file
By default rsync will only use --whole-file if both source and destination are local, so the second option is redundant in that case.
Deprecating scp
Posted Nov 6, 2020 12:50 UTC (Fri) by nix (subscriber, #2304) [Link]
Deprecating scp
Posted Nov 19, 2020 22:27 UTC (Thu) by rodgerd (guest, #58896) [Link]
Deprecating scp
Posted Nov 20, 2020 22:46 UTC (Fri) by nix (subscriber, #2304) [Link]
Deprecating scp
Posted Nov 6, 2020 10:52 UTC (Fri) by grawity (subscriber, #80596) [Link]
My trick is to use "cp -a source/. dest/" to always get just the contents.
rsync options exist for a reason
Posted Nov 6, 2020 17:02 UTC (Fri) by david.a.wheeler (guest, #72896) [Link]
The world needs a "simple replacement for cp that copies remotely" and a "sophisticated tool for remote synchronization". They don't need to be the same tool, and historically have been different. I'd like to see a more secure version of the first category. In the first category most people don't really need the scp *protocol*, they just need a simple replacement for "cp" that works remotely. That is, it's the CLI interface, not the protocol, that matters.
rsync options exist for a reason
Posted Nov 6, 2020 21:02 UTC (Fri) by marcH (subscriber, #57642) [Link]
Of course other rsync options exist for a reason and I do look them up and use them sometimes. Above I was only referring to basic, [s]cp-like usage.
Answering other comments too: if you must preserve hardlinks, sparse files, ACLs or some other unusual stuff then you better check the man page of _any_ tool you use.
> The world needs a "simple replacement for cp that copies remotely" and a "sophisticated tool for remote synchronization". They don't need to be the same tool, and historically have been different.
They don't need to be different tools either. I've been using a unique tool for "simple replacement for recursive, _local_ cp" all the way to "sophisticated tool for remote synchronization" and it has made simple things easy and complex things possible. So if you want a solution that already exists now then give it a try. If scp muscle memory is just too strong then a sincere "best of luck!"
Deprecating scp
Posted Nov 7, 2020 6:45 UTC (Sat) by ncm (guest, #165) [Link]
That is harder nowadays, because there are a lot more with more pages each. But a reasonable selection of them deserve the old-school treatment. Rsync it probably one that does. Anyway, a curated selection of its options. (Eyes rolling? That's our world now.)
Instead of sitting and memorizing the whole collection, of an evening, why not bring home rsync(1), and memorize just that? What else would you do, look at your phone?
Deprecating scp
Posted Nov 6, 2020 6:47 UTC (Fri) by rahvin (guest, #16953) [Link]
It's to create a new command that's more secure and let those that need it use it. Let the users that need the extra security and permission checking use the new version and those that don't can keep using scp. I just don't see the value in eliminating or orphaning it when it's not a security vulnerability but an undesired capability.
Deprecating scp
Posted Nov 6, 2020 11:47 UTC (Fri) by gerdesj (subscriber, #5446) [Link]
...
(i.e. ensure that the destination directory exists before writing the file to it)"
Sounds like a job for smkdir
rsync remote-to-remote
Posted Nov 6, 2020 14:23 UTC (Fri) by bkw1a (subscriber, #4101) [Link]
Deprecating scp
Posted Nov 6, 2020 16:00 UTC (Fri) by lwn@pck.email (subscriber, #121154) [Link]
Deprecating scp
Posted Nov 12, 2020 14:45 UTC (Thu) by jcpunk (subscriber, #95796) [Link]
I don't love the idea of updating a billion scripts and having to explain why the documentation everyone finds all over the internet doesn't work anymore.
All in all, this is an optimistic mixed bag. But it sure is the kind of thing the community should be looking at.
Deprecating scp
Posted Nov 13, 2020 9:39 UTC (Fri) by nim-nim (subscriber, #34454) [Link]
Browsers have become the elephant in the room everyone optimizes for.
I like sshfs
Posted Nov 15, 2020 14:19 UTC (Sun) by Luyseyal (guest, #15693) [Link]
I like sshfs
Posted Apr 17, 2021 20:46 UTC (Sat) by RoyBellingan (guest, #151704) [Link]
Deprecating scp
Posted Jan 25, 2021 9:44 UTC (Mon) by tconnors (guest, #60528) [Link]
That would be unfortunate. The only time I've used scp in the past... 15 years(?) is for scp -3.
The alternative is setting up horrible reverse tunnels with ssh, from memory. Yes, I do believe I use this in production at work (because I needed sync of directory trees), but am not about to check because we're about to enter into a public holiday.
Deprecating scp
Posted Jan 25, 2021 12:17 UTC (Mon) by mbunkus (subscriber, #87248) [Link]
ssh host1 "tar czf - /path /other/path" | ssh host2 "tar xzfC - /"
Deprecating scp
Posted Feb 5, 2021 19:39 UTC (Fri) by flussence (subscriber, #85566) [Link]
Deprecating scp
Posted Aug 25, 2021 13:30 UTC (Wed) by wtarreau (subscriber, #51152) [Link]