Solved Deprecating scp

[a6h]

There was some discussion about "Deprecating scp" topic at the Forums. Do you have any thoughts on this topic? Please read following links and give it some thought. Thanks.

Deprecating scp

View the details of “Deprecating scp” as bookmarked by ~lobsters. You can read user notes and if you are interested in this bookmark you can add it to your bookmarks.

www.tefter.io

Deprecating scp [LWN.net]

lwn.net

OpenSSH: for OpenBSD

OpenSSH for OpenBSD

www.openssh.com

 

[a6h]

I voted "bad", because I like scp! and I rely on it to work with vms and misc/computers on my LAN. In the meantime I'm open to consider using new tools.
It would be great to have some technical discussions about scp itself. I read those blogs, but it's better to listen to FreeBSD users' opinion on this subject.

 

[Cthulhux]

Is there a sanely licensed rsync replacement?

 

[msplsh]

Why not SFTP or Rsync for these operations?

 

[Cthulhux]

Why not SFTP or Rsync for these operations?

You might want to follow the links first.

 

[Phishfry]

What is good for me might be a security nightmare for another user.
So it is all in context. I only use scp on my LAN so I have no security fears.

We do seem to have a lot of opinionated media critters trying to sway the discussion.
Just like this recent LWN article and Phoronix article declaring Xorg as dead.

 

[kpedersen]

If scp was massive, then yeah deprecated and removal is fine. But scp is pretty tiny and portable and has no replacement (rsync is too large).

This recent "trend" for removal of useful software is simply Linux finally breaking away from UNIX and posix. Good luck to them.

 

[Jose]

This recent "trend" for removal of useful software is simply Linux finally breaking away from UNIX and posix. Good luck to them.

"As with other deprecated commands (ifconfig, say), it can be hard to make the switch." Yep, definitely a Linux-centric proposal.

 

[shkhln]

Did you miss https://github.com/openssh/openssh-portable/pull/194 somehow?

 

[kpedersen]

Did you miss https://github.com/openssh/openssh-portable/pull/194 somehow?

That is IMO the correct decision. Improve scp by replacing its internals but keeping the command compatible (i.e for existing scripts, workflows, etc).

It is Fedora that is apparently deprecating it (which usually leads to removal) which I find a little less sane.
(https://lists.fedoraproject.org/arc....org/thread/4CYOOLAWIIQTYAZXF2XXVALBS5OX4WM6/)

 

[shkhln]

No, they are replacing the underlying protocol with the proposed patch.

 

[msplsh]

You might want to follow the links first.

I did and the answers were unsatisfactory:
* Glob problems
* People who feel icky about GPL
* Can't execute commands, which is why you would stop using scp

These are such banal reasons that killing scp would be fine if only to cause somebody to create a satisfactory replacement.

 

[olli@]

I often use cpdup (sysutils/cpdup) for copying files from one machine to another. While it was designed to copy whole directory trees, it can as well be used to copy single files. The syntax is similar to scp, although not identical – In particular, when copying a single file, you must specify the target file name, not just the target directory. For example: cpdup remote:/dir/some_file local_copy

 

[Jose]

Did you miss https://github.com/openssh/openssh-portable/pull/194 somehow?

No. It is mentioned in the LWN article:

Jakub Jelen is working on such a thing; it is an scp command that uses the sftp protocol under the hood. At this point, it is claimed to work for most basic usage scenarios; some options (such as -3, which copies files between two remote hosts by way of the local machine) are not supported. "Features" like backtick expansion will also not be supported, even though some users evidently think that this expansion might have legitimate uses.

 

[Zvoni]

Didn't read the links.
Voted bad, because: Don't fix what's not broken.

 

[ralphbsz]

Is there a sanely licensed rsync replacement?

Whether GPL is a sane license is a question of taste, or religious conviction. I happen to know the main author of rsync reasonable well (used to be a colleague), and he has done an enormous amount of good work for the open software world and software freedom, not only rsync, but also Samba, getting the TiVo to work in other countries, and Wireshark. He is a seriously good guy, and one of the few counterweights to jerks like Linus T in the community. If he selected the GPL license for this work, that needs to be respected (whether I like GPL or not is not relevant).

Why not SFTP or Rsync for these operations?

Sftp: totally different command line and user interface. Rsync: Way too complex for the simple task of copying a few files. While rsync is great for when its complexity is needed, it can be overwhelming. Real-life example: On Saturday, I wanted to use rsync to copy files, but stop at mount points. I vaguely remembered that the correct flag is "-x", but confirming that in the man page took 5 minutes, because the man page has so much information; it's not badly written, just exceedingly long, because there are many dozen flags and operation modes to be documented. And rsync has many of the same problems that scp has, with file/directory names having to be quoted in bizarre ways.

Voted bad, because: Don't fix what's not broken.

Today's scp is broken. Real-world example: simplified. I'll use two machines, and the prompt is the machine name. Let's assume we have ssh keys set up for passwordless login:

A_# mkdir foo\ bar  # Execute these commands on machine A, that's why the prompt.
A_# ls -l foo\ bar/
total 0
B_# echo "Hallo" > blatz  # Execute these commands on machine B.
B_# ls -l blatz
-rw-r--r--  1 ralphbsz  ralphbsz  6 Nov  9 15:24 blatz
B_# scp blatz A:foo\ bar/ # This command will fail!
scp: ambiguous target
B_# mkdir foo\ bar
B_# cp blatz foo\ bar/
# Succeeds

So remote directory or file names that have spaces (or other special characters, like UTF-8) in them can't be used. That's INSANE. Yes, I know you can make it work (by double-quoting, for example copy to A:"foo\ bar"/), but even there you have to be careful and do a lot of trial and error, or understand exactly how shells unquote things.

So I agree: the implementation of scp needs to be fixed or replaced. If no volunteers can be found to do that, it needs to be obsoleted and withdrawn from service.

 

[richardtoohey2]

the implementation of scp needs to be fixed or replaced. If no volunteers can be found to do that, it needs to be obsoleted and withdrawn from service.

I think if you took that approach to software that is a bit confusing (and yes, sometimes allows you to blow your feet off!) then there wouldn't be much left on computers.

I can't think of any machine or OS that at some time or another I've not wanted to throw out the the window because of (seemingly-odd-but-presumably-there-was-a-reason-for-) design or implementation choices made.

 

[olli@]

[…] Real-life example: On Saturday, I wanted to use rsync to copy files, but stop at mount points. I vaguely remembered that the correct flag is "-x", but confirming that in the man page took 5 minutes

I don’t like rsync either, but confirming what -x does is rather easy: Type man rsync, then search for -x by pressing /-x<Enter>. It will jump directly to this line:
-x, --one-file-system don’t cross filesystem boundaries

Today's scp is broken. […]
So remote directory or file names that have spaces (or other special characters, like UTF-8) in them can't be used.

Again: I agree that scp is broken. But spaces can be used, you just have to keep in mind that the names are parsed twice – first by your local shell, and then by the remote shell, so you have to quote them twice (as you have noticed). UTF-8 characters can be used without problems if the locale is configured correctly on both sides, in this case you don’t even need quoting.

The real brokenness of scp appears when you look at commands like this:

scp somefile remote:'$(touch gotcha)'

It copies the file and executes the command (specified using the shell’s command substitution syntax) on the remote side. The reason for this behavior is the same as above: The arguments are parsed a second time on the remote side. This behavior was inherited from rcp, because scp is nothing more than the rcp protocol tunnelled through ssh.

The problem is that scp cannot be fixed without giving up backwards-compatibility. Therefore, the best option would be to write a new tool (with a different name!) that uses a different protocol (e.g. sftp, or a completely new protocol), while retaining a usage similar to scp. And either emulate scp’s broken syntax when called with the name “scp”, or allow for a transition period so people can adapt their scripts that depend on scp syntax.

 

[msplsh]

Sftp: totally different command line and user interface.

I guess just thunking SCP into SFTP on the wire is going to "solve" your problem then. Proceed with removal!

 

[Jose]

I don’t like rsync either, but confirming what -x does is rather easy: Type man rsync, then search for -x by pressing /-x<Enter>. It will jump directly to this line:
-x, --one-file-system don’t cross filesystem boundaries
...

I have to look up what exactly it is that -a does every time. I've spent far too much quality time with Rsync's man page, and I agree with Ralphbsz. Mo' better documentation is needed.

The problem is that scp cannot be fixed without giving up backwards-compatibility. Therefore, the best option would be to write a new tool (with a different name!) that uses a different protocol (e.g. sftp, or a completely new protocol), while retaining a usage similar to scp. And either emulate scp’s broken syntax when called with the name “scp”, or allow for a transition period so people can adapt their scripts that depend on scp syntax.

I disagree. I think the new SFTP-based scp should retain the name, and there should be a switch, something like -1 or -rcp that uses the old protocol for the tiny minority of scp users that rely on ancient misfeatures.

 

[ralphbsz]

I end up agreeing with you on everything, I'll just add details.

Type man rsync, then search for -x by pressing /-x<Enter>. It will jump directly to this line:
-x, --one-file-system don’t cross filesystem boundaries

Tried that, but didn't work: "/-x" jumps to the line that has "--xattrs, -X". Searching for "/filesystem" interestingly doesn't find anything (I wonder why not). So what I did is to go to the web-based man page on the FreeBSD site, and then search within the page. But that shouldn't be necessary; for a tool like scp (which is nothing but cp but with remote nodes via ssh), the man page should be dozens or a hundred lines long, not a confusing 4000.

But spaces can be used, you just have to keep in mind that the names are parsed twice – first by your local shell, and then by the remote shell, so you have to quote them twice (as you have noticed).

For me, who knows how shells work, that's reasonably easy. Typically I get it on the second try. Again, for a simple tool like scp, it should work on the first try: if the command works for cp, it should work for scp, just with "host:" or "user@host:" prefixed.

The real brokenness of scp appears when you look at commands like this: "scp somefile remote:'$(touch gotcha)'"

Absolutely. This just has to be disallowed scp. Fortunately, one hardly ever does that by mistake: few people have files in their file system that are called "$(touch gotcha)", although it is theoretically possible. But file names with spaces are (unfortunately) somewhat common.

All this could be fixed by a new version of scp not using simply the ssh protocol, but a dedicated protocol, that distinguishes between already parsed and expanded file names, and command line options. In reality, this all points out a more fundamental problem of the Unix CLI user interface: it doesn't distinguish parameters in general (for example file names, which are nothing but opaque strings) and options, and packs then all together in arguments. The classic joke is a person who create a file named "-R", and then wonders why "rm -f *" recursively deletes subdirectories too, even though the user clearly didn't intend that. Now take that known (and 50 year old) brokenness, and add double-parsing everything on commands that go over the network, and it becomes more user-inimical.

The problem is that scp cannot be fixed without giving up backwards-compatibility. Therefore, the best option would be to write a new tool (with a different name!) that uses a different protocol (e.g. sftp, or a completely new protocol), while retaining a usage similar to scp.

Agree. Something like scp is needed, just deleting the current one creates a hardship. But the new one has to be better, and that will make it different, so it also needs a different name.

 

[msplsh]

I disagree. I think the new SFTP-based scp should retain the name, and there should be a switch, something like -1 or -rcp that uses the old protocol for the tiny minority of scp users that rely on ancient misfeatures.

This is not a good idea for anyone, because then you are completely changing the behavior of a program that people use. They could update the system and think they're just getting fixes and suddenly nothing works (or worse, it appears to work). Imagine if people just updated the Python package from 2 to 3 without changing any names.

Second, people want to banish the SCP protocol because of insecurities. They don't want to maintain that code path anymore, so having an option to go back to the old way does not solve their problem.

New program, different name, (mostly) same command line UI, and throw in a "symlink scp to this new program" in the build options.

 

[Jose]

This is not a good idea for anyone, because then you are completely changing the behavior of a program that people use. They could update the system and think they're just getting fixes and suddenly nothing works (or worse, it appears to work). Imagine if people just updated the Python package from 2 to 3 without changing any names.

This is hardly a change "completely changing the behavior" of scp. A few odd, and judging from this thread, very infrequently used features would disappear. The example given in the mail list is simply not convincing:

(E)nsure that the destination directory exists before writing the file to it

scp sourcefile remoteserver:'`[ -d /a/b/c ] || mkdir -p /a/b/c`/a/b/c/targetfile'

Stop being cute, FFS, and use an ssh(1) command to ensure the directory exists before you copy to it.

Second, people want to banish the SCP protocol because of insecurities. They don't want to maintain that code path anymore, so having an option to go back to the old way does not solve their problem.

You're arguing against yourself here. You can't have both a new, secure protocol, and the weird insecure functionality of the old scp command. Pick one.

Besides, there's a precedent to the -1 command line switch. You could force the SSH client to select the old, insecure version 1 protocol with that option. It was deprecated and eventually removed.

 

[msplsh]

If people expect backticks to work, or have to quote things a certain way, this is not a backwards compatible way to do things. Yes, it's not a lot, but it will cause them a lot of problems and potentially data loss. Using a different command that basically implies "don't expect this to work the exact same" is enough warning before people symlink scp to it and go on with their day.

I'm not arguing against myself. People want to implement most of scp via the SFTP mechanism and call it a different thing, but have the user interface be mostly but not exactly like scp. That is not choosing both, it's choosing the former.

"It was deprecated and eventually removed." Yep, just like what the plan is for scp, except maybe they're going to skip right over the deprecated part.

 

[olli@]

I disagree. I think the new SFTP-based scp should retain the name, and there should be a switch, something like -1 or -rcp that uses the old protocol for the tiny minority of scp users that rely on ancient misfeatures.

Do you have any numbers to back the claim “tiny minority of scp users”?

Personally, when I use scp inside a script, I always try to write it in a way that it’ll work when used with file names that contain spaces or other special characters (double quoting). All of these scripts would break if you replaced the scp command with another command (but with the same name) that doesn’t use the rcp protocol by default. It would be confusing and a POLA-violation. And even when you adapt the scripts, they would now be unportable between systems that have the “real” scp and systems that have the “new” scp.

Therefore it would be better to give the new command a new name, so there is a clear distinction, and retain the old scp command for a transition period (and maybe let it print a warning).