Make repos go fastly
Closed, ResolvedPublic

Tokens
"Party Time" token, awarded by mcritchlow."Like" token, awarded by Riokei."Like" token, awarded by shameempk."Yellow Medal" token, awarded by euoar."Hungry Hippo" token, awarded by kyrios123."Love" token, awarded by miwilc."Like" token, awarded by Elvanex."Like" token, awarded by curantes.
Assigned To
Authored By
ikey, Sep 11 2017

Description

Right, tracking this for the week, to make sure it actually gets done..

@DataDrake has kindly reworked the foobarred job handling in ferryd so I can restart work on that.

  • Finish ferryd in a test-deployable form this week.
  • Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR) (Edit 2: THIS WEEKEND.)
  • Onboard community maintainers
  • Sort out the god-awful fuckery that is our current repo network speed. New host, CDN, something. Throw money at someone. For the love of cheese sandwiches, make network speeds to the repo great again.
  • Discuss and establish formal sync cadence. I'm inclined to say 2 days but for stuff like a GNOME update it's not enough time. Input requested here.

Long and short of it is, make downloads from repo faster, make syncs faster, make stuff go from dev portal to repos faster.

This blocks T4235: Meta: Post-release blues because without the fastly bits done, we can't move quick enough to make it all happen. Plus other high-p items need to wait for this to be done.

Re: onboarding: We'd really love to make some of you guys actual full blown maintainers if you're up for it. We need to also help you in transitioning to that ... "trusted "(?) role,
but we also need to make sure that we're all in sync with one another (i.e. #Solus-Dev IRC channel) and that all stuff hitting the repos is valid. Maybe we need to put together
a very simple ypkg-check-cranky script to ensure we're not making common mistakes, we can hit a great cadence.

There are a very large number of changes, so older changes are hidden. Show Older Changes
ikey triaged this task as "Unbreak Now!" priority.Sep 11 2017, 9:03 PM

Oh look, an Unbreak Now! that actually carries meaning. For realsies, this is our "blocker" bug. One whole week, let's get it done.

STiAT added a subscriber: STiAT.Sep 11 2017, 9:14 PM
Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR)

At least you didn't state which weekend. Could be the one in a year. It's Solus still :-).

Tearow added a subscriber: Tearow.Sep 11 2017, 9:15 PM
DataDrake added a comment.EditedSep 11 2017, 9:16 PM

I think we should have a mandatory weekend sync and then an optional mid-week sync that we usually do unless working on heavier things.

My $0.02.

ikey edited the task description. (Show Details)Sep 11 2017, 9:16 PM
In T4524#79827, @STiAT wrote:
Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR)

At least you didn't state which weekend. Could be the one in a year. It's Solus still :-).

Fixed it. :P

jalden added a subscriber: jalden.Sep 11 2017, 9:20 PM
ikey added a comment.Sep 11 2017, 9:23 PM

I think we should have a mandatory weekend sync and then an optional mid-week sync that we usually do unless working on heavier things.

My $0.02.

This sounds good, and targeting the weekend is most friendly to people with, uh whats them things called.. "lives". With Sunday typically being a Solus release day,
is Saturday a bit optimistic or should it be an update Sunday? Push our "normal" release day back to the Monday? Saturday is likely taking the piss a bit when it comes
to free time for most people to have stuff fixed and synced.

Friday mandatory syncs so people can have weekends :P

ikey added a comment.Sep 11 2017, 11:37 PM

Well I'm easy whichever way we do it so I'll let you lads decide on the actual day

Sort out the god-awful fuckery that is our current repo network speed. New host, CDN, something. Throw money at someone. For the love of cheese sandwiches, make network speeds to the repo great again.

I don't know why I'm so happy because of this sentence. 😄

Going with a major weekly sync (not to be skipped), then one to two (have to think of the time for security updates to reach stable) midweek when there's no blockers.

Major updates (say gnome stack) could start on a weekend (the odd working weekend xD) with a Friday sync schedule. Nothing major starts after Tuesday, and pushed to next week. This allows for skipping mid-week syncs when it's for the best, sets a timeframe to push major changes with time to stabilise.

miwilc added a subscriber: miwilc.

Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR) (Edit 2: THIS WEEKEND.)

All the eyes are turning towards @ikey We are all watching you now... 😛

Discuss and establish formal sync cadence. I'm inclined to say 2 days but for stuff like a GNOME update it's not enough time. Input requested here.

Perhaps this could be done with some "freeze" periods for big stuffs like GNOME or testing phase before releasing new ISOs. This being said, maybe it's better to define this after the first wave of community maintainers are onboarded because new maintainers will probably do a few mistake at the very beginning, there will certainly be some lessons to learn (after this wave new maintainer will join one by one so it will be easier to handle than many at a time, won't it?)

euoar awarded a token.Sep 12 2017, 9:19 AM
ikey added a comment.Sep 12 2017, 7:41 PM

@kyrios123 we'll definitely want coordinated freezes yes. I imagine it's gonna be a bit like weekly scrums, but without all the agile bullshit.

Are you open to having the repos mirrored? I can potentially help organise things within New Zealand if so at no cost - FSMG has a host with gigabit connectivity, but is georestricted to connections from NZ and I work with the maintainer. If interested, please let me know.

ikey added a comment.Sep 13 2017, 9:44 PM

@chrisweeksnz we'll take anything we can get! Once we move to CDN + new infra we'll need to add geoip mirror selection to the SC and push out an emergency update to help everyone jump across to mirrors

Tick added a subscriber: Tick.Sep 15 2017, 4:34 PM

Are there network speed problems?

I sent a couple emails to @JoshStrobl about hosing a Canadian package mirror. Is this the hold up? If so, what can I do to help out?

@stobbsm I haven't gotten any emails.

@JoshStrobl maybe in your spam box? Either way, I can host a Canadian mirror, just need to know how to set it up, and how to help ease the process.

ikey added a comment.Sep 15 2017, 7:17 PM
In T4524#80392, @Tick wrote:

Are there network speed problems?

...For serious?

In T4524#80398, @ikey wrote:
In T4524#80392, @Tick wrote:

Are there network speed problems?

...For serious?

With a 14.4k modem, it doesn't make any difference... really 😬

Tick added a comment.Sep 15 2017, 8:17 PM

I have not used Solus for a few days, so maybe I missed a big problem :p
In any case, I had no problems today.

wget https://packages.solus-project.com/shannon/libr/libreoffice/libreoffice-common-5.3.3-40-1-x86_64.eopkg -O /dev/null 
--2017-09-15 22:11:58--  https://packages.solus-project.com/shannon/libr/libreoffice/libreoffice-common-5.3.3-40-1-x86_64.eopkg
Résolution de packages.solus-project.com… 149.202.199.80
Connexion à packages.solus-project.com|149.202.199.80|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 126070203 (120M)
Sauvegarde en : « /dev/null »

/dev/null           100%[===================>] 120,23M   104MB/s    ds 1,2s    

2017-09-15 22:11:59 (104 MB/s) — « /dev/null » sauvegardé [126070203/126070203]

@Tick Download more than one package in a go, then we'll talk :P

The network speeds are quite slow, IMO. =_=

ikey added a comment.Sep 17 2017, 6:45 PM

Yep, I know, running late, here we are Sunday, no ferryd.

Long story short we actually have all the right code in place inside ferryd and its doing the right things, but boltdb (a key/value store) is the thing
actually making boltdb hard lock. So our super high priority "sequential" queue is blocked forever. Yknow, the whole thing that justified creating
ferryd in the first place.

Track the issue here if you're interested: https://github.com/solus-project/ferryd/issues/4
I'm gonna be hacking on this all night and get boltdb replaced with gorm, I don't intend to let Murphy knock me back that far.

ikey added a comment.Sep 20 2017, 3:09 AM

Initial testing will begin later today of ferryd after my next sleep (madly offset right now). Basically its internally feature complete but it needs a tiny bit
more polish and safety checks, as well as some kind of visual job tracker.

ikey changed the task status from "Open" to "In Progress".Sep 20 2017, 3:09 AM

I am from Nepal.
The repo downloading speed is too slow.
Please add a mirror here.!
I am loving Solus too much, due to this reason I may have to go back to Ubuntu.
Many Nepalese have faced this problem!
Please help!

In T4524#80133, @ikey wrote:

@chrisweeksnz we'll take anything we can get! Once we move to CDN + new infra we'll need to add geoip mirror selection to the SC and push out an emergency update to help everyone jump across to mirrors

I'd previously sent email to an address recommended by Josh with a domain name solus-project.com (not written here, to prevent web crawler / spam evilness). Who should I contact about mirroring and via what method (eg. email, IRC, etc..)? Happy for reply to be PM or in IRC. Alternatively, if the answer is wait until we have a mirror method posted in either patreon or the docs site, that's all fine. :)

I am from Nepal.
The repo downloading speed is too slow.

@maverick1 I think this will be taken care of soon. I would suggest to have an Asian mirror somewhere centrally located so many locations can benefit from infrastructure updates. When downloading from China I get very slow network speeds as well.

Please inform when mirrors are added!!

Let's get our main repo sorted first fellas :)

ikey added a comment.Sep 20 2017, 6:08 PM

I am from Nepal.
The repo downloading speed is too slow.
Please add a mirror here.!
I am loving Solus too much, due to this reason I may have to go back to Ubuntu.
Many Nepalese have faced this problem!
Please help!

Yeah alright dude. Read the issue, we're clearly fixing the issue, I don't appreciate ultimatums.

ikey added a comment.Sep 20 2017, 6:15 PM

ring0 is now registered in DNS to help with the transition (packages. will become the CDN)

Interestingly, I just downloaded at full speed... CDN isn't in place yet, is it?

Justin added a subscriber: Justin.Sep 21 2017, 1:08 AM
In T4524#80774, @STiAT wrote:

Interestingly, I just downloaded at full speed... CDN isn't in place yet, is it?

No, not yet. The repo has been inconsistent in speeds but the CDN is still going ahead.

ikey edited the task description. (Show Details)Sep 21 2017, 10:28 AM

Yea. See that? ferryd is in place. Yep.

ikey added a comment.Sep 22 2017, 1:32 AM

@kyrios123 ping me on the IRCs when you're around pls

ikey added a subscriber: joebonrichie.EditedSep 22 2017, 5:25 PM

Please welcome @joebonrichie and @kyrios123 as our first community global maintainers! Solus is now going to go through a massive evolutionary
phase, and with their help we can begin moving "moar fastly". Importantly, we can flesh out the first steps for our first generation of maintainers, so
that one day they can help to onboard the second, and so on. :)

We're happy to have you guys on board, and look forward to kicking ass together, thanks!

ikey edited the task description. (Show Details)Sep 22 2017, 5:25 PM

By way of a mild update, our CDN situation should be fixed on Wednesday.

euoar added a subscriber: euoar.EditedSep 22 2017, 7:33 PM

Welcome @kyrios123 and @joebonrichie !!

Nice to see the team growing!!

Thanks ! Happy to join the Solus ⛵

Welcome guys!

Welcome !

A huge thanks for your packaging effort @kyrios123 @joebonrichie

Awesome! Thanks for working so hard for the community and supporting the rise of a major operating system. @kyrios123 and @joebonrichie The Solus development pace never ceases to surprise me. I am just occasionally filing bugs and requesting packages but you guys are actually doing the packaging.

As for sync cycles, and when to update software here is my opinion on things

  • Syncs can be made once a week, preferably Wednesdays. Wednesday because as most people don't want to deal with potential package upgrade issues over the weekend and Wednesday is late enough into the working week if there are package upgrade issues it shouldn't affect their productivity too much if they are using Solus for work. Exceptions can be made for security updates but preferably these should be done on the weekend unless critical, meaning a max of two syncs to stable a week.
  • Syncs to stable can be skipped as many weeks as necessary, but no longer than a month.
  • It would be nice to be able to force security updates and small package fixes into stable but that would require a lot of ferryd plumbing. See here
  • Point .0 releases for major software should never make it into stable such as linux-current, mesa or a gnome stack upgrade. Putting .0 releases in unstable is okay for testing and helping upstream iron out issues, but major software should be updated to the first point release (x.x.1) before making it into stable.
  • Waiting two days after a software update before putting it into unstable is fine, but perhaps too lenient?
  • Point .0 releases for major software should never make it into stable such as linux-current, mesa or a gnome stack upgrade. Putting .0 releases in unstable is okay for testing and helping upstream iron out issues, but major software should be updated to the first point release (x.x.1) before making it into stable.

.0 releases are not always full of bugs and it may take time before a .1 is released, so I don't see the point of this, with the exception of the Linux kernel, however even with the kernel .1 releases _could_ take time to come all the same, and we shouldn't be stuck with a older kernel for long.

As for only including .0 releases in unstable, I really don't see the point since it only exists to test software/get newer software before it is merged with stable.

ikey added a comment.Oct 3 2017, 9:18 AM

Regarding .0 I think this would mostly apply to the dominant minor release for a stack, such as GNOME. For kernels this likely applies too. Outside of that most projects are OK with .0

ikey edited the task description. (Show Details)Oct 3 2017, 9:18 AM

If you are still deciding into multiple options, maybe you could do a poll on Google+ to finally choose one and close this task :p

We have a finalized sync cadence and will have a blog covering it soon. After that we'll close this task :)

ikey closed this task as "Resolved".Oct 6 2017, 3:44 PM

Release practices are now documented as part of the Solus Help Center: https://github.com/solus-project/help-center-docs/commit/3e75a0fe9d7cafc2b92b9e4095f92dbcfdc79fcb

TLDR Friday is the sync day (today) - and some minor syncs *may* happen in the week. Friday is mandatory. Closing this issue, expect blog post to follow along with the deployed help
center update.

ikey edited the task description. (Show Details)Oct 6 2017, 3:44 PM
cam34 reopened this task as "Open".Tue, Nov 21, 12:18 AM
cam34 added a subscriber: cam34.

Hello

Sorry for re-opening the closed ticket, but I thought you might like what I have to say.
I am able to mirror Solus, at a Tier 1 ISP in Oceania.

Please PM me with rsync details and I will get it done.

Thanks

JoshStrobl closed this task as "Resolved".Tue, Nov 21, 1:56 PM

Please speak with Ikey with that rather than re-opening this task. Thanks.

JoshStrobl changed the edit policy from "All Users" to "Triage Team (Project)".Tue, Nov 21, 1:56 PM
ikey added a comment.Tue, Nov 21, 1:59 PM

Hi @cam34 - there isn't a PM mechanism on Phabricator. Please email me: ikey@solus-project.com :)