problems with pkg fetch -a to create local mirror

[MarkRB]

I'm struggling to create a local mirror, to use when internet access is not available or permitted by hosts, and to store custom packages.

The problem is checksum failures which stop the transfer.
Some times it runs for a while, but at other times fails on every package.

I have tried a smaller fetch, but it doesn't help.
bandwidth is not an issue with up to 700KiB/s, and wget on the failed file always transfers the file.
The server is running behind a firewall and as a vm on esxi 6.7

testing with a subset, same result as using pkg fetch -y -o /mnt/nas/mirror -a

pkg fetch -yUd -o /mnt/nas/mirror -x 'ne'

...
742 MiB to be downloaded.
DBG(1)[46845]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/minetest-0.4.17.1.txz with opts "i4"
Fetching minetest-0.4.17.1.txz: 100% 6 MiB 351.1kB/s 00:18
DBG(1)[46845]> Resuming fetch
DBG(1)[46845]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/minetest_game-0.4.17.txz with opts "i4"
Fetching minetest_game-0.4.17.txz: 100% 1 MiB 1.1MB/s 00:01
pkg: minetest_game-0.4.17 failed checksum from repository

repeat
...
The process will require 507 MiB more space.
736 MiB to be downloaded.
DBG(1)[47790]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/minetest_game-0.4.17.txz with opts "i4"
Fetching minetest_game-0.4.17.txz: 100% 1 MiB 279.6kB/s 00:04
DBG(1)[47790]> Resuming fetch
DBG(1)[47790]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/monotone-viz-1.0.2_7.txz with opts "i4"
Fetching monotone-viz-1.0.2_7.txz: 100% 719 KiB 736.6kB/s 00:01
pkg: monotone-viz-1.0.2_7 failed checksum from repository

repeat...

735 MiB to be downloaded.
DBG(1)[51018]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/monotone-viz-1.0.2_7.txz with opts "i4"
Fetching monotone-viz-1.0.2_7.txz: 100% 719 KiB 147.3kB/s 00:05
pkg: cached package ne-libreoffice-6.2.2: size mismatch, fetching from remote
DBG(1)[51018]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/ne-libreoffice-6.2.2.txz with opts "i4"
Fetching ne-libreoffice-6.2.2.txz: 100% 2 MiB 287.6kB/s 00:07
DBG(1)[51018]> Resuming fetch
DBG(1)[51018]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/nemo-extensions-2.4.0_3.txz with opts "i4"
Fetching nemo-extensions-2.4.0_3.txz: 100% 311 KiB 318.4kB/s 00:01
pkg: nemo-extensions-2.4.0_3 failed checksum from repository

use wget instead

wget http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/nemo-extensions-2.4.0_3.txz
--2019-05-31 10:40:37-- http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/quarterly/All/nemo-extensions-2.4.0_3.txz
Resolving pkgmir.geo.freebsd.org (pkgmir.geo.freebsd.org)... 149.20.1.201, 2001:4f8:1:11::50:1
Connecting to pkgmir.geo.freebsd.org (pkgmir.geo.freebsd.org)|149.20.1.201|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 318360 (311K) [application/octet-stream]
Saving to: 'nemo-extensions-2.4.0_3.txz'

nemo-extensions-2.4.0_3.txz 100%[========================================================================================>] 310.90K 180KB/s in 1.7s

2019-05-31 10:40:40 (180 KB/s) - 'nemo-extensions-2.4.0_3.txz' saved [318360/318360]



Can anyone shed some light on what might be the cause?

I have been trying to complete this for a week, with little progress.

 

[jbliao]

I met this problem too. I found that this problem would present if I fetch same package twice with some interval.
For example I fetch `vim` (pkg fetch -o /tmp/pkg vim ) at 6/25 and fetch it again at 6/27. The second time shows the checksum error message, and the old package file on disk just disappeared.
I assume the pkg repo provide difference package(maybe with some hotfix) with same name and pkg-fetch would compare new checksum(on repo) with old package file(on local). Obviously they would not match.
I'm trying to make experiment to proof my assumption .