Make repos go fastly
Open, In Progress, Unbreak Now!Public

Tokens
"Party Time" token, awarded by mcritchlow."Like" token, awarded by Riokei."Like" token, awarded by shameempk."Yellow Medal" token, awarded by euoar."Hungry Hippo" token, awarded by kyrios123."Love" token, awarded by miwilc."Like" token, awarded by Elvanex."Like" token, awarded by curantes.
Assigned To
Authored By
ikey, Mon, Sep 11

Description

Right, tracking this for the week, to make sure it actually gets done..

@DataDrake has kindly reworked the foobarred job handling in ferryd so I can restart work on that.

  • Finish ferryd in a test-deployable form this week.
  • Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR) (Edit 2: THIS WEEKEND.)
  • Onboard community maintainers
  • Sort out the god-awful fuckery that is our current repo network speed. New host, CDN, something. Throw money at someone. For the love of cheese sandwiches, make network speeds to the repo great again.
  • Discuss and establish formal sync cadence. I'm inclined to say 2 days but for stuff like a GNOME update it's not enough time. Input requested here.

Long and short of it is, make downloads from repo faster, make syncs faster, make stuff go from dev portal to repos faster.

This blocks T4235: Meta: Post-release blues because without the fastly bits done, we can't move quick enough to make it all happen. Plus other high-p items need to wait for this to be done.

Re: onboarding: We'd really love to make some of you guys actual full blown maintainers if you're up for it. We need to also help you in transitioning to that ... "trusted "(?) role,
but we also need to make sure that we're all in sync with one another (i.e. #Solus-Dev IRC channel) and that all stuff hitting the repos is valid. Maybe we need to put together
a very simple ypkg-check-cranky script to ensure we're not making common mistakes, we can hit a great cadence.

ikey created this task.Mon, Sep 11, 9:01 PM
Herald removed ikey as the assignee of this task. · View Herald TranscriptMon, Sep 11, 9:01 PM
ikey triaged this task as "Unbreak Now!" priority.Mon, Sep 11, 9:03 PM

Oh look, an Unbreak Now! that actually carries meaning. For realsies, this is our "blocker" bug. One whole week, let's get it done.

STiAT added a subscriber: STiAT.Mon, Sep 11, 9:14 PM
Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR)

At least you didn't state which weekend. Could be the one in a year. It's Solus still :-).

Tearow added a subscriber: Tearow.Mon, Sep 11, 9:15 PM
DataDrake added a comment.EditedMon, Sep 11, 9:16 PM

I think we should have a mandatory weekend sync and then an optional mid-week sync that we usually do unless working on heavier things.

My $0.02.

ikey edited the task description. (Show Details)Mon, Sep 11, 9:16 PM
In T4524#79827, @STiAT wrote:
Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR)

At least you didn't state which weekend. Could be the one in a year. It's Solus still :-).

Fixed it. :P

jalden added a subscriber: jalden.Mon, Sep 11, 9:20 PM
ikey added a comment.Mon, Sep 11, 9:23 PM

I think we should have a mandatory weekend sync and then an optional mid-week sync that we usually do unless working on heavier things.

My $0.02.

This sounds good, and targeting the weekend is most friendly to people with, uh whats them things called.. "lives". With Sunday typically being a Solus release day,
is Saturday a bit optimistic or should it be an update Sunday? Push our "normal" release day back to the Monday? Saturday is likely taking the piss a bit when it comes
to free time for most people to have stuff fixed and synced.

Friday mandatory syncs so people can have weekends :P

ikey added a comment.Mon, Sep 11, 11:37 PM

Well I'm easy whichever way we do it so I'll let you lads decide on the actual day

Sort out the god-awful fuckery that is our current repo network speed. New host, CDN, something. Throw money at someone. For the love of cheese sandwiches, make network speeds to the repo great again.

I don't know why I'm so happy because of this sentence. 😄

Going with a major weekly sync (not to be skipped), then one to two (have to think of the time for security updates to reach stable) midweek when there's no blockers.

Major updates (say gnome stack) could start on a weekend (the odd working weekend xD) with a Friday sync schedule. Nothing major starts after Tuesday, and pushed to next week. This allows for skipping mid-week syncs when it's for the best, sets a timeframe to push major changes with time to stabilise.

miwilc added a subscriber: miwilc.

Get it deployed (For realsies!!!!111) for the weekend (not on, not after, FOR) (Edit 2: THIS WEEKEND.)

All the eyes are turning towards @ikey We are all watching you now... 😛

Discuss and establish formal sync cadence. I'm inclined to say 2 days but for stuff like a GNOME update it's not enough time. Input requested here.

Perhaps this could be done with some "freeze" periods for big stuffs like GNOME or testing phase before releasing new ISOs. This being said, maybe it's better to define this after the first wave of community maintainers are onboarded because new maintainers will probably do a few mistake at the very beginning, there will certainly be some lessons to learn (after this wave new maintainer will join one by one so it will be easier to handle than many at a time, won't it?)

euoar awarded a token.Tue, Sep 12, 9:19 AM
ikey added a comment.Tue, Sep 12, 7:41 PM

@kyrios123 we'll definitely want coordinated freezes yes. I imagine it's gonna be a bit like weekly scrums, but without all the agile bullshit.

Are you open to having the repos mirrored? I can potentially help organise things within New Zealand if so at no cost - FSMG has a host with gigabit connectivity, but is georestricted to connections from NZ and I work with the maintainer. If interested, please let me know.

ikey added a comment.Wed, Sep 13, 9:44 PM

@chrisweeksnz we'll take anything we can get! Once we move to CDN + new infra we'll need to add geoip mirror selection to the SC and push out an emergency update to help everyone jump across to mirrors

Tick added a subscriber: Tick.Fri, Sep 15, 4:34 PM

Are there network speed problems?

I sent a couple emails to @JoshStrobl about hosing a Canadian package mirror. Is this the hold up? If so, what can I do to help out?

@stobbsm I haven't gotten any emails.

@JoshStrobl maybe in your spam box? Either way, I can host a Canadian mirror, just need to know how to set it up, and how to help ease the process.

ikey added a comment.Fri, Sep 15, 7:17 PM
In T4524#80392, @Tick wrote:

Are there network speed problems?

...For serious?

In T4524#80398, @ikey wrote:
In T4524#80392, @Tick wrote:

Are there network speed problems?

...For serious?

With a 14.4k modem, it doesn't make any difference... really 😬

Tick added a comment.Fri, Sep 15, 8:17 PM

I have not used Solus for a few days, so maybe I missed a big problem :p
In any case, I had no problems today.

wget https://packages.solus-project.com/shannon/libr/libreoffice/libreoffice-common-5.3.3-40-1-x86_64.eopkg -O /dev/null 
--2017-09-15 22:11:58--  https://packages.solus-project.com/shannon/libr/libreoffice/libreoffice-common-5.3.3-40-1-x86_64.eopkg
Résolution de packages.solus-project.com… 149.202.199.80
Connexion à packages.solus-project.com|149.202.199.80|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 126070203 (120M)
Sauvegarde en : « /dev/null »

/dev/null           100%[===================>] 120,23M   104MB/s    ds 1,2s    

2017-09-15 22:11:59 (104 MB/s) — « /dev/null » sauvegardé [126070203/126070203]

@Tick Download more than one package in a go, then we'll talk :P

The network speeds are quite slow, IMO. =_=

ikey added a comment.Sun, Sep 17, 6:45 PM

Yep, I know, running late, here we are Sunday, no ferryd.

Long story short we actually have all the right code in place inside ferryd and its doing the right things, but boltdb (a key/value store) is the thing
actually making boltdb hard lock. So our super high priority "sequential" queue is blocked forever. Yknow, the whole thing that justified creating
ferryd in the first place.

Track the issue here if you're interested: https://github.com/solus-project/ferryd/issues/4
I'm gonna be hacking on this all night and get boltdb replaced with gorm, I don't intend to let Murphy knock me back that far.

ikey added a comment.Wed, Sep 20, 3:09 AM

Initial testing will begin later today of ferryd after my next sleep (madly offset right now). Basically its internally feature complete but it needs a tiny bit
more polish and safety checks, as well as some kind of visual job tracker.

ikey changed the task status from "Open" to "In Progress".Wed, Sep 20, 3:09 AM

I am from Nepal.
The repo downloading speed is too slow.
Please add a mirror here.!
I am loving Solus too much, due to this reason I may have to go back to Ubuntu.
Many Nepalese have faced this problem!
Please help!

In T4524#80133, @ikey wrote:

@chrisweeksnz we'll take anything we can get! Once we move to CDN + new infra we'll need to add geoip mirror selection to the SC and push out an emergency update to help everyone jump across to mirrors

I'd previously sent email to an address recommended by Josh with a domain name solus-project.com (not written here, to prevent web crawler / spam evilness). Who should I contact about mirroring and via what method (eg. email, IRC, etc..)? Happy for reply to be PM or in IRC. Alternatively, if the answer is wait until we have a mirror method posted in either patreon or the docs site, that's all fine. :)

I am from Nepal.
The repo downloading speed is too slow.

@maverick1 I think this will be taken care of soon. I would suggest to have an Asian mirror somewhere centrally located so many locations can benefit from infrastructure updates. When downloading from China I get very slow network speeds as well.

Please inform when mirrors are added!!

Let's get our main repo sorted first fellas :)

ikey added a comment.Wed, Sep 20, 6:08 PM

I am from Nepal.
The repo downloading speed is too slow.
Please add a mirror here.!
I am loving Solus too much, due to this reason I may have to go back to Ubuntu.
Many Nepalese have faced this problem!
Please help!

Yeah alright dude. Read the issue, we're clearly fixing the issue, I don't appreciate ultimatums.

ikey added a comment.Wed, Sep 20, 6:15 PM

ring0 is now registered in DNS to help with the transition (packages. will become the CDN)

Interestingly, I just downloaded at full speed... CDN isn't in place yet, is it?

Justin added a subscriber: Justin.Thu, Sep 21, 1:08 AM
In T4524#80774, @STiAT wrote:

Interestingly, I just downloaded at full speed... CDN isn't in place yet, is it?

No, not yet. The repo has been inconsistent in speeds but the CDN is still going ahead.

ikey edited the task description. (Show Details)Thu, Sep 21, 10:28 AM

Yea. See that? ferryd is in place. Yep.

ikey added a comment.Fri, Sep 22, 1:32 AM

@kyrios123 ping me on the IRCs when you're around pls

ikey added a subscriber: joebonrichie.EditedFri, Sep 22, 5:25 PM

Please welcome @joebonrichie and @kyrios123 as our first community global maintainers! Solus is now going to go through a massive evolutionary
phase, and with their help we can begin moving "moar fastly". Importantly, we can flesh out the first steps for our first generation of maintainers, so
that one day they can help to onboard the second, and so on. :)

We're happy to have you guys on board, and look forward to kicking ass together, thanks!

ikey edited the task description. (Show Details)Fri, Sep 22, 5:25 PM

By way of a mild update, our CDN situation should be fixed on Wednesday.

euoar added a subscriber: euoar.EditedFri, Sep 22, 7:33 PM

Welcome @kyrios123 and @joebonrichie !!

Nice to see the team growing!!

Thanks ! Happy to join the Solus ⛵

Welcome guys!

Welcome !

A huge thanks for your packaging effort @kyrios123 @joebonrichie