From asaf at lingnu.com Wed Aug 3 20:18:21 2011 From: asaf at lingnu.com (Asaf) Date: Wed, 03 Aug 2011 23:18:21 +0300 Subject: [sbopkg-users] Using http, Curl as a repository tool Message-ID: <4E39AD0D.8080703@lingnu.com> Continued from: http://sbopkg.org/pipermail/sbopkg-users/2010-July/000492.html I've forked sbopkg to implement Curl repository tool: https://github.com/asafo/sbopkg there is also a sample repos.d/80-SBo-curl.repo I'll be happy for any suggestions to help it get merged Thanks -- Asaf Ohaion Lingnu Open Source Consulting Ltd http://www.lingnu.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From slakmagik at gmail.com Wed Aug 3 23:19:07 2011 From: slakmagik at gmail.com (slakmagik) Date: Wed, 3 Aug 2011 19:19:07 -0400 Subject: [sbopkg-users] Using http, Curl as a repository tool In-Reply-To: <4E39AD0D.8080703@lingnu.com> References: <4E39AD0D.8080703@lingnu.com> Message-ID: <20110803231907.GB10770@devbox> On 2011-08-03 (Wed) 23:18:21 [+0300], Asaf wrote: > Continued from: > http://sbopkg.org/pipermail/sbopkg-users/2010-July/000492.html > > I've forked sbopkg to implement Curl repository tool: > https://github.com/asafo/sbopkg > > there is also a sample repos.d/80-SBo-curl.repo > > I'll be happy for any suggestions to help it get merged > First, am I to understand that every use of 'sbopkg -r' using this would result in downloading the entire SBo repository? Because I don't think that's acceptable, both from a user's POV and from the SBo admin's POV. For less major points: your 80 repo file doesn't work because it attempts to sync to localhost. Even after correcting that to a full url, the protocol is duplicated in _curl_sync_package(). Even after correcting that, gunzip asks about whether it should overwrite SLACKBUILDS.TXT if that file already exists. For relatively minor points, we don't use the _func convention (not saying whether we should or not, but we don't) and your patches have trailing whitespace. Also we only use the '['-style test on pathnames and maybe not always then. We prefer '[[' for strings and '((' for numerics. We don't leave commented-out code in (if we do, it's a mistake). Error messages should announce that they're coming from $SCRIPT (and I don't mind even $FUNCNAME at times). While there's nothing in writing about and sbopkg is very bashistic, I'd personally still prefer to avoid stuff like BASH_REMATCH, though the use is really spiffy in this case and might well be the best thing to do. I'm not fond of _curl_fail() when it's only used once and its functionality could just be used directly in the _curl_sync_package() line where it's called. And I'm personally a <80 columns fanatic. Lines should fit within standard terminal widths wherever possible. Lastly, a major point, I'd raise first except that I wouldn't document something until I knew it was going in, myself, but anything that would go in would have to be thoroughly documented (man pages, always, and any relevant text files). Well, lastly lastly, I've only briefly looked at the patches (or patch, as it could be profitably 'squashed' into one, IMO) and only briefly tested but, while I'm testing, I'm noticing a lot of hangs and 'Curl Failed: 6's so this doesn't seem very efficient and effective even without the non-differencing problem. But, other than these points, it's pretty neat. I definitely appreciate your interest and this contribution but, if I understand it correctly, I don't think I'm in favor of it, at least at this point. We really do intend to get an lftp sync method in soon and, while that's not http, it might help with some people who have rsync issues while still only updating the differences. Or possibly something could be worked out via parsing the ChangeLog and only http'ing those packages. But maybe if Mauro's around, he'll have a different take. And, yeah, the http sync is still ongoing after having been started before I began this email (and I take forever with emails). ;) From asaf at lingnu.com Thu Aug 4 07:32:49 2011 From: asaf at lingnu.com (Asaf) Date: Thu, 04 Aug 2011 10:32:49 +0300 Subject: [sbopkg-users] Using http, Curl as a repository tool In-Reply-To: <20110803231907.GB10770@devbox> References: <4E39AD0D.8080703@lingnu.com> <20110803231907.GB10770@devbox> Message-ID: <4E3A4B21.6050409@lingnu.com> On 08/04/2011 02:19 AM, slakmagik wrote: > On 2011-08-03 (Wed) 23:18:21 [+0300], Asaf wrote: >> Continued from: >> http://sbopkg.org/pipermail/sbopkg-users/2010-July/000492.html >> >> I've forked sbopkg to implement Curl repository tool: >> https://github.com/asafo/sbopkg >> >> there is also a sample repos.d/80-SBo-curl.repo >> >> I'll be happy for any suggestions to help it get merged >> > First, am I to understand that every use of 'sbopkg -r' using this would > result in downloading the entire SBo repository? Because I don't think > that's acceptable, both from a user's POV and from the SBo admin's POV. Yes, the work is not complete yet, there are starting points for parsing the ChangeLog.txt file (Actually, the diff with the old ChangleLog.txt, if there is one) so that only new packages would get downloaded > For less major points: your 80 repo file doesn't work because it > attempts to sync to localhost. Even after correcting that to a full url, > the protocol is duplicated in _curl_sync_package(). Even after > correcting that, gunzip asks about whether it should overwrite > SLACKBUILDS.TXT if that file already exists. This is going to change, when ChangeLog.txt parsing implementation is complete, also, there is currently no public repository available (I hope that they will open one at slackbuilds.org), so I'm using localhost for testing > > For relatively minor points, we don't use the _func convention (not > saying whether we should or not, but we don't) and your patches have > trailing whitespace. Also we only use the '['-style test on pathnames > and maybe not always then. We prefer '[[' for strings and '((' for > numerics. We don't leave commented-out code in (if we do, it's a > mistake). Error messages should announce that they're coming from > $SCRIPT (and I don't mind even $FUNCNAME at times). While there's > nothing in writing about and sbopkg is very bashistic, I'd personally > still prefer to avoid stuff like BASH_REMATCH, though the use is really > spiffy in this case and might well be the best thing to do. I'm not fond > of _curl_fail() when it's only used once and its functionality could > just be used directly in the _curl_sync_package() line where it's > called. And I'm personally a<80 columns fanatic. Lines should fit > within standard terminal widths wherever possible. the use of BASH_REMATCH saves a lot of process forking (grep, cut, sed, etc..) (or is there a third way ?) > > Lastly, a major point, I'd raise first except that I wouldn't document > something until I knew it was going in, myself, but anything that would > go in would have to be thoroughly documented (man pages, always, and any > relevant text files). > > Well, lastly lastly, I've only briefly looked at the patches (or patch, > as it could be profitably 'squashed' into one, IMO) and only briefly > tested but, while I'm testing, I'm noticing a lot of hangs and 'Curl > Failed: 6's so this doesn't seem very efficient and effective even > without the non-differencing problem. > > But, other than these points, it's pretty neat. I definitely appreciate > your interest and this contribution but, if I understand it correctly, I > don't think I'm in favor of it, at least at this point. We really do > intend to get an lftp sync method in soon and, while that's not http, it > might help with some people who have rsync issues while still only > updating the differences. Or possibly something could be worked out via > parsing the ChangeLog and only http'ing those packages. lftp wouldn't solve the problem (restrictive firewalls) > > But maybe if Mauro's around, he'll have a different take. > > And, yeah, the http sync is still ongoing after having been started > before I began this email (and I take forever with emails). ;) > _______________________________________________ > sbopkg-users mailing list > sbopkg-users at sbopkg.org > http://sbopkg.org/mailman/listinfo/sbopkg-users > -- Asaf Ohaion Lingnu Open Source Consulting Ltd http://www.lingnu.com