Rsync link-dest problem

[scorpizz]

Question: Is there a special way to use symbolic links in Rsync "--link-dest" ?

Setup is a rsync deamon running on the FreeBSD server with a path to a module storage.
On the client, rsync is executed with the following command:
rsync -ah --hard-links --link-dest=../test /home/user module@server::user/tmp

IF "test" on the destination is a directory, everything is woking as expected
But if "test" is a symbolic link on destination pointing to a directory in the same desination directory, the result is "--link-dest arg does not exist: ../test"

Working: (ls -l on server: /mnt/files/rsync_storage/user)
drwxr-x---  user  user   Oct 1 10:00 tmp
drwxr-x---  user  user   Oct 1 10:00 test

Not working: (ls -l on server: /mnt/files/rsync_storage/user)
drwxr-x---  user  user   Oct 1 10:00 tmp
lrwxr-x---  user  user   Oct 1 10:05 test -> /mnt/files/rsync_storage/user/tmp

(symbolic link is created with "ln -sfh source link_name")

Is there something that I have overlooked or does rsync not support symbolic links in the --link-dest option ?

 

[aragats]

I guess, you should use a trailing slash for a directory, thus it won't be confused with the symlink itself.

 

[scorpizz]

I guess, you should use a trailing slash for a directory, thus it won't be confused with the symlink itself.

Thanks for the suggestion.
I have tried adding the tailing slash to the "../test/" and for that matter the "::user/tmp/", but unfortunally it didn't solved the problem.
Also re-checked that the symlink on the server is working when accessed from the local shell on the server.

 

[juan_r]

I hit a similar problem, and this thread is the only relevant discussion I've found.
My issue is with the --keep-dirlinks (short form: -K) option. The FreeBSD rsync server won't honor the option, which should follow symlinks to directories on the server side.

Example. In the source (client) directory /foo I have something like
/src/foo/bar/a.txt

I make a symlink in the corresponding destination (server) folder, such as
/dest/foo/bar/ -> ../baz/

I found no way to have /src/foo/bar/a.txt copied into /dest/baz/a.txt (or anywhere else, for that matter). The server ignores the -K option, so it either (a) clobbers soft links to directories into real directories, and goes on to happily copy files, or (b) fails because it can't clobber a link because of permissions.

It basically either creates /dest/foo/bar/a.txt (where bar is a real directory and the link I'd made to ../baz gets clobbered), or if it can't overwrite the symlink /dest/foo/bar it issues an error and fails.

I tried asking at superuser.com (https://superuser.com/questions/155...er-please-follow-symlinks-in-target-directory) with little luck.

 

[aragats]

Maybe I don't understand you correctly, but this works as expected:

rsync -a /src/foo/bar/ server:/dest/foo/bar/

with the directories layouts you described.

 

[juan_r]

Hello aragats, thanks for your interest and answer!
What I'm actually trying to do is, for example,

rsync -v -rltR -z -u --delete "/cygdrive/./C/a.txt" "nas01.lan::HB2-Z-raw"
rsync -v -rltR -z -u --delete "/cygdrive/./D/b.txt" "nas01.lan::HB2-Z-raw"

where on the server nas01.lan, the directory /HB2-Z-raw/ contains a real directory C/
and a symlink D/ --> ../HB2-raw/D/

The idea is to duplicate the source file tree into two destination branches
from (src)/cygdrive/C into (dest)/HB2-Z-raw/C
from (src)/cygdrive/D into (dest)/HB2-raw/D
always issuing same rsync command, with only the source file changing, so that
a.txt ends up in /HB2-Z-raw/C/a.txt
b.txt ends up in /HB2-raw/C/a.txt

(The reason for this intended behavior is that HB2:/D and HB2-Z:/D are actually the same disk - physical in one case, network disk in the other. On the contrary, HB2:/C and HB2-Z:/C are two distinct physical drives).

Whatever I try, the D -> ../HB2-raw link on the server isn't followed, but replaced by a freshly created directory only containing b.txt, which is not what I want. I would like b.txt to join the other files in /HB2-raw/D.

The server is running rsync version 3.1.3 protocol version 31.

 

[Jose]

Do you really mean to use --relative, -R and not --recursive, -r? In any case, it appears that's your problem

--relative, -R
...
Beginning with rsync 3.0.0, rsync always sends these implied directories as real directories in the file list, even if a path element is really a symlink on the sending side. This prevents some really unexpected behaviors when copying the full path of a file that you didn't realize had a symlink in its path. If you want to duplicate a server-side symlink, include both the symlink via its path, and referent directory via its real path. If you're dealing with an older rsync on the sending side, you may need to use the --no-implied-dirs option.

rsync(1) manpage

 

[juan_r]

Thanks Jose! I did know (and tried to use) --keep-dirlinks, -K, but I didn't know about --no-implied-dirs until I read your reply. Still no joy, though.

Do you really mean to use --relative, -R and not --recursive, -r? In any case, it appears that's your problem

rsync(1) manpage

I mean both -R and -r. Relative and recursive: I'm trying to sync whole subbranches.
The only relevant symlinks in this discussion are on the server side.

The rsync manpage you linked mentions --no-implied-dirs right before -K, --keep-dirlinks. I've emphasized the relevant part with italics.

--no-implied-dirs

This option affects the default behavior of the --relative option. When it is specified, the attributes of the implied directories from the source names are not included in the transfer. This means that the corresponding path elements on the destination system are left unchanged if they exist, and any missing implied directories are created with default attributes. This even allows these implied path elements to have big differences, such as being a symlink to a directory on the receiving side.

For instance, if a command-line arg or a files-from entry told rsync to transfer the file "path/foo/file", the directories "path" and "path/foo" are implied when --relative is used. If "path/foo" is a symlink to "bar" on the destination system, the receiving rsync would ordinarily delete "path/foo", recreate it as a directory, and receive the file into the new directory. With --no-implied-dirs, the receiving rsync updates "path/foo/file" using the existing path elements, which means that the file ends up being created in "path/bar". Another way to accomplish this link preservation is to use the --keep-dirlinks option (which will also affect symlinks to directories in the rest of the transfer).

The italicized part is just what I want to happen. However, it doesn't seem to work quite that way. Here's what happens to me.

$ rsync -v -rltR -z -u --delete -K --no-implied-dirs "/cygdrive/./D/a.txt" "nas01.lan::HB2-Z-raw/"
Hello, rsyncing to nas01!
sending incremental file list
default_perms_for_dir: sys_acl_get_file(D, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
D/a.txt
default_perms_for_dir: sys_acl_get_file(D, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/D/.a.txt.djavJo" (in HB2-Z-raw) failed: No such file or directory (2)

sent 105 bytes  received 28 bytes  266.00 bytes/sec
total size is 13  speedup is 0.10
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /home/lapo/packaging/rsync-3.0.7-1/src/rsync-3.0.7/main.c(1042) [sender=3.0.7]

$ rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

So client side it's rsync 3.0.7, server side it's rsync 3.1.3.

 

[Jose]

I mean both -R and -r. Relative and recursive: I'm trying to sync whole subbranches.
The only relevant symlinks in this discussion are on the server side.

I still don't understand why you need -R at all. Won't -a (expands to -rlptgoD) do exactly what you want?

 

[juan_r]

Thanks for bearing with me, Jose.

I need -R because I send whole pathnames to rsync via a custom right-click menu (Windows).

If I want to backup, say, c:\somefolder, I right click it and choose "my_rsync" (it's my custom script). Cygwin runs the sending rsync. The path seen by cygwin and rsync is actually /cygdrive/c/somefolder. My script modifies the path to become /cygdrive/./c/somefolder, so the receiving rsync knows the path is relative to the first component ./, so the server syncs the folder into, say, /HB2-Z/C/somefolder rather than /HB2-Z/cygdrive/c/somefolder.

The use or non use of -R doesn't seem to affect what happens to dirlinks on the receiving side. They get clobbered or not, but never followed anyway. The issue you reported above, on the other hand,

Do you really mean to use --relative, -R and not --recursive, -r? In any case, it appears that's your problem

Beginning with rsync 3.0.0, rsync always sends these implied directories as real directories in the file list, even if a path element is really a symlink on the sending side. This prevents some really unexpected behaviors when copying the full path of a file that you didn't realize had a symlink in its path. If you want to duplicate a server-side symlink, include both the symlink via its path, and referent directory via its real path. If you're dealing with an older rsync on the sending side, you may need to use the --no-implied-dirs option.

rsync(1) manpage

seems to be related to symlinks on the sending, not the receiving, side. I have no symlinks on the sending side.

I'm probably being thick, but I don't understand the following phrase. Is it relevant?

If you want to duplicate a server-side symlink, include both the symlink via its path, and referent directory via its real path.

 

[Jose]

Thanks for bearing with me, Jose.

I need -R because I send whole pathnames to rsync via a custom right-click menu (Windows).

If I want to backup, say, c:\somefolder, I right click it and choose "my_rsync" (it's my custom script). Cygwin runs the sending rsync. The path seen by cygwin and rsync is actually /cygdrive/c/somefolder. My script modifies the path to become /cygdrive/./c/somefolder, so the receiving rsync knows the path is relative to the first component ./, so the server syncs the folder into, say, /HB2-Z/C/somefolder rather than /HB2-Z/cygdrive/c/somefolder.

The use or non use of -R doesn't seem to affect what happens to dirlinks on the receiving side. They get clobbered or not, but never followed anyway. The issue you reported above, on the other hand,

seems to be related to symlinks on the sending, not the receiving, side. I have no symlinks on the sending side.

I'm probably being thick, but I don't understand the following phrase. Is it relevant?

I've never tried anything so advanced. I think that's the crux of the matter, yeah.

I believe you're going to have to send both /HB2-raw/D/ (the symlink) and /HB2-Z-raw/D (what you're symlinking to.)

 

[juan_r]

Actually my problem is kind of the opposite. HB2 and HB2-Z basically share the same disks (except C: - they do have distinct boot drives). The disks are physically connected to one machine and connected as as network disks on the other. Both client machines modify the files on all the disks. What I wanted to do is keep a full backup on my FreeBSD NAS, avoiding file duplication and using just one set of rsync scripts. The scripts would be invoked identically from either client. In my intentions, the redirection would happen through smart symlinking on the NAS.

Unfortunately, since the FreeBSD rsync server won't follow server symlinks to dirs no matter what, I'll have to recode the scripts on both machines so they use different server profiles according to the disk where the file(s) to backup are.