Yet another systemd comment

At this point, the whole discussion has started to annoy me. I started so many replies to so many comments on reddit, posts on Arch’s forums and or mails on [arch-general] – and I aborted most of them. The stupidity of the whole discussion makes me sick. But in the end, you cannot change who you are, and I am a man of many angry words.

As with every discussion, there’s two extremes: Morons who praise systemd as if it would save the world, and retards that think systemd is the reason the world will end this year. Of course, all of them are wrong. If you belong to one of those groups: Fuck you! Everyone else: Keep reading.

Write programs that do one thing and do it well.

I won’t pretend to know the internal design of systemd. I also won’t argue that it actually does or does not follow that principle.

Let us instead remember that this UNIX principle was formulated decades ago. Back then and now, what was one of the biggest problem with computers? Bugs in your software. How do you avoid bugs? Divide your work into small tasks, only write the code for each task once, put it together as you need it. How? Use one tool, let it write its output to a file or the standard output, let the next tool read the output and process it. Does it sound like a good idea? Maybe, but here’s what’s wrong with it:

  • The only way of communication between tasks are strings. Every tool has to serialize its output, and the next tool has to parse it again. The communication is one-way only, parsing can be complex and error-prone.
  • cmd1 | cmd2 | cmd3 – an error occurs in cmd2, how do you handle that?
  • For performing a complex task, you need to create a shitload of processes and call a shitload of tools. Sounds efficient.
  • Syntax is checked on runtime.

I’m no expert, but to me it sounds like that could be improved. So, what’s changed in the last few decades? Shared libraries. Let me repeat that: shared fucking libraries. Here’s what you get:

  • Libraries contain code for common tasks that can be integrated into any program. Bug fixes in the library automatically affect all programs that use it.
  • Well-defined APIs, structured data, less data copying.
  • Proper error reporting.
  • No bugs from output and parsing errors.
  • Syntax is checked on compile time.

I’m not going to say more about this. Does systemd follow the classical UNIX principle? I don’t care. It’s the fucking 21st century. If we have better ways of doing things, let us please use them! Let us not follow a guideline that is decades old and mostly obsolete.

systemd is not easily hackable, Arch’s initscripts are more flexible

Let us have a look at the old initscripts. They boot up your system. They start your daemons. And they don’t do that very well. They don’t nearly cover all use cases. The code is complex. Extending them means changing the scripts and doing it again after the next update. There are bugs which are hard or even impossible to fix.

A lot has improved since Tom took over maintainership of the scripts. However, it has become clear that they can only be an intermediate solution: If we want them to remain simple, we will keep missing important features. If we keep extending them, they will become so complex that nobody understands them anymore.

Now, to the people that claim systemd is not hackable: Have you looked at /usr/lib/systemd/system? You can change, disable or extend almost every single detail of it. You can override almost all of its behaviour by placing files in /etc/systemd/system and it will be persistent to upgrades. All of that without changing a single line of C or shell code. If you want to, you can even mask all of its units and instead make it call a script very similar to our old rc.sysinit.

In short: systemd’s hackability and flexibility supercedes initscripts’ considerably.

systemd – why not?

I have only been running pure systemd on my machines for a few weeks. I like it. It has some rough edges, its support in Arch Linux is incomplete. There’s room for improvement. It will not save the world. It will not eat your children either.

A final message to the people who keep complaining

You can yell all you want, yet Arch Linux will slowly move towards systemd. Initscripts will probably keep working for a long time, but they will eventually disappear. It doesn’t help if you insult us for it. It doesn’t help if you state a thousand times that you leave Arch over “the systemd issue”. We don’t care. You can either embrace systemd and enjoy all its advantages, provide an alternative or use another distribution. We don’t care. We make Arch for ourselves, and for the ones that like it like we do. Whether we have a million users or one hundred users – we will keep making it the distro we like. Deal with it.

26 Comments

  1. Luke says:

    I couldn’t agree more. I’ve had to rummage for a few .service files, and wrote one or two that I needed customized, but so far I haven’t run into anything that I would call out a doomsday. I believe I read that arch bang is going to stay anti-systemd, but I can’t for the life of me think of why?

  2. lee wrangler says:

    As always there will be people that dont llike the change no matter how positive it is. Those users dont do their research and get their knowledge from gossip.

    Also: nice post, I like your style.

  3. Tim says:

    I have been reading up all bits about systemd in gentoo/arch ecosystem since months. Your post seems to hit the nail perfectly. Thank you for these words, lets hope people read it.

  4. desapararecido says:

    all the time we prone linux like the OS where you can choice.. and is that.. I’m archuser from 5 years ago and all this evolutions is good, I learn a lot, for make the change in my desktop I waited some to take the time between job, party and move.. and finally I did it and I’m happy. I still need read about systemd, but was exactly the reason to choice Archlinux.. I want understand my OS, how it works, KISS principle, bleeding edge.. so thanks for this nice post and thank to dev’s, nice work too..
    …. and …
    like @lee wrangler say… I like your style ;)

  5. njanja says:

    Well, that’s what everyone else who doesn’t like it and has bothered to actually understand what’s wrong with the fucking thing is doing. The thing is, those others? They aren’t Arch developers. So they can’t FORCE OTHER PEOPLE TO DO WHAT THEY THINK IS BEST. From that perspective, fuck you, sir. Fuck you very much. Systemd is broken by design. The whim of one idiot has pulled in *everyone* into a nice mess. Did you think you need another init system before you ran into systemd? I don’t remember seeing the Arch devs give Upstart a good thinking over. I’m sure runit or OpenRC weren’t published yesterday. Did you bother to look into those, and how they remedy the things you (rightfully) criticise? Nope nope nope. Anyway. You’re free to ruin the distro for the rest of us. Unfortunately. Those who dislike that are free to jump ship. I don’t think most will, since what the fuck do they know, they just use the thing, I mean this is an Appl.. oh no, wait, this is fucking Arch Linux. I hope your myopia gets better. Not bloody likely, though – if there’s one thing that will related to *nix that will persist beyond what makes *nix what is it, it’s the steretypical, obnoxious, knows-what-is-best-for-everyone, Asperger-but-not-really devs. Congratulations?

    • brain0 says:

      You Sir, are exactly the kind of reader whom I told to stop reading at the beginning. I don’t need your opinion.

      Did you pay anything for Arch? Did you contribute a single line of code to any of its projects? Have you even read a single line of code on any of the projects? Of initscripts? No? What did you do? You came here to insult me! And on what basis? Lies. Yes, lies.

      Is anyone forcing you to use systemd (as you claim)? No. You can easily use initscripts, they work, and they are still being developped, nothing changed. Did Arch suddenly become a closed system? No. It is still the operating system where it’s the easiest to replace any core component.

      So why are you here? Because I think systemd is the right design for an init system? Because I think progress is good? Because I have actually read the systemd design document and know that it makes sense? Because I think that systemd is the init system that will get the most development in the coming years and that it will be actively maintained and extended?

      No, you are here because you are an asshole and want to insult me. I have a message for you: FUCK YOU. You know why Arch is where it is now? Because we do WHAT THE FUCK WE WANT with it. If you don’t like it, that’s not our problem, because YOU DON’T PAY US.

    • DoDo says:

      Dear njanja,

      if you find Archlinux difficult to use, please use then Ubuntu or Windows.

  6. Renato says:

    There’s one thing that I never understood. If systemd is so bad, why every single distribution is adopting? Did Lennart Poettering put a gun on head of every single distro developer?

    • brain0 says:

      systemd was originally created by Lennart Poettering (Redhat) in cooperation with Kay Sievers (udev-Maintainer, Novell) – originally in their free time, if I understand this right. Thus, it is not surprising that both Fedora and openSuSE use systemd these days (openSuSE still offers sysvinit, IIRC).

      What other distros (besides Arch) have switched to systemd?

      • teho says:

        Mandriva, Mageia and Chakra (still in-progress) on desktop. On mobile side there’s at least Mer and Tizen that use systemd. It’s also popular on embedded systems and can be found on Ångström, MontaVista Linux and it’s part of every GENEVI (IVI systems) approved operating system including Ubuntu IVI.

  7. Leonid.I says:

    Throughout this discussion of systemd 2 things surprise me:
    1. Criticisms along the lines “I am a serious Unix/BSD sysadmin since 1960 and systemd
    sucks because I have developed a 10^23 bash scripts and yet can’t spend 20 min
    to read man 1 systemd”.
    2. Amount of energy you guys spend reasoning with trolls. To me personally,
    arch has been nothing but useful and most importantly instructive, so THANK YOU. But given your
    position above, why do you even care to reply to personal insults?

    • brain0 says:

      In part, there’s some bullshit I just can’t not comment. I am also guilty of https://xkcd.com/386/. At times, I even enjoy arguing with trolls.

      But the worst thing is: I get really angry when someone is being an ass. I’m working on that.

      • Hi,

        Just a quick comment to say thumbs up! I also find it very hard not to reply to trolls and spend far too much time and energy getting worked up about it when I could just set an ignore on IRC and be done with it.

        We’ve had surprising little opposition to systemd in Mageia. The current release (Mageia 2) uses systemd by default and Mageia 3 is systemd only and I’m loving removing some of the old cruft and getting a much neater and easier to understand system.

        I find that in pretty much all the cases the people complaining are simply joining the bandwagon without any kind of proper technical research. As I know Lennart quite well, I knew he was working on the systemd project for a long time before it was announced, but I certainly didn’t assume it was awesome. I spend several hours reading the very informative blog posts and documentation created for the project before making up my own mind. Obviously I’m not a big supporter of it and I struggle to understand people who don’t feel the same.

        Lennart seems to have the right attitude to the nay-sayers. Just ignore them! The only issue with that is that occasionally you’ll get valid bug reports that do need fixing, but they are done in such a trollish way that it really puts you off trying to help. I had one like this just the other day with the guy spouting a whole bunch of bullshit about systemd, it’s documentation and Lennart himself. I did engage the troll for a while, but it was going nowhere. I took a deep breath and did end up investigating the issue… where was the bug? polkit (primarily because of our half transition to logind rather than consolekit in Mageia 2 due to still supporting sysvinit. Lennart’s fault? No, not at all, but that doesn’t stop him or his projects getting blamed.

        Anyway, that’s some of my own steam being let off too now!

        Kudos on the xkcd reference, love it! Keep up the good work!

        • stqn says:

          I haven’t used Mageia but I think the difference is that Mageia users don’t need to configure a lot of stuff to get a working system, unlike Arch users. So they don’t care about the init system. They possibly don’t even know that there is an init system and wtf it is. Arch users have to read a lot about systemd and change their config files and hunt for fixes in the forums in order to get a working system again, which I find a good enough reason to be upset. (Which doesn’t mean that the switch is a bad choice…)

          • Marvn says:

            Well, but that’s the way Arch is and always have been. If you are installing Arch for the first time and you read the wiki wrong, you can end up with system which doesn’t really work (and then you have to hunt for fixes ofc). People who have an issue with that probably shouldn’t use Arch…

  8. Enjolras says:

    What a wonderful post. I’m tired of the troll, who from the main part don’t even know what they are talking about. I spent some time reading systemd source code. Although this is not the kissier software i ever saw, it does its job, and it does it quite well. Hacky shell scripts do too, but they are harder to maintain and distro specific. And seriously, whoever in the world find rc.d scripts KISS ? Anyway, i will not advocate yet another time for or against systemd.

    The main point here, is that developers want to switch systemd because this better for THEM. And that is KISS. Developers do what they want to, whether users like it or not. People tend to believe than opensource developers are at their service, but opensource projects are not democracy ! If you don’t like it you have two alternatives :
    - submiting patchs/packages to support old initscripts
    - go where the fuck you want to go !

    Complaining again and again is pointless, and boring. I totally agree with these harsh words.

    Last but not least, the first comment from the openBSD installer script :

    # OpenBSD installation script.
    # In a perfect world, this would be a nice C program, with a reasonable
    # user interface.

  9. I don’t care if systemd is hackable. What I care about is debuggability.

    E.g., one of the new inventions that comes with systemd is pam_systemd that is supposed to put the session in a new cgroup, register it as active with systemd, and do nothing else. Now see https://bugs.gentoo.org/show_bug.cgi?id=435042 – it makes a login impossible, with no logs that help.

  10. syrinx says:

    very well said. systemd isn’t nearly as bad as the trolls make it out to be. don’t like it, go use ubuntu.

  11. Kevin Chadwick says:

    How long do you think it would take for the OpenBSD devs to convert /etc/rc to C but they wouldn’t and certainly not in anything like systemd that crams features that already exist to subvert and overwhelm objectors. OpenBSD install script, what another pointless argument to add to the list above in the face of real issues. Try searching the OpenBSD list for systemd, you won’t get a lot as it’s not worth discussing even if it was cross platform but it is basically laughed at. Shared libraries, how silly. Sorry but this blog isn’t even worth spending time on. I will say this, if these methods are so slow how come systemd saves so little boot time despite wild claims of shells slowness that you repeat more mildly with ‘sounds efficient’. If that was a real issue, rc would be in perl, but there are good reasons why it isn’t.

  12. dammannj says:

    tl;dr Keep to the way, don’t feed the Borg if avoidable, keep up the good work.

    You have arrived at the only right and meaningful conclusion:
    systemd – “It will not save the world. It will not eat your children either.”

    Some comments:

    “A final message to the people who keep complaining” and “We don’t care”

    Bravo. THIS – IS – OSS. Don’t like it? Leave it, fork it, don’t bother me anymore (or do commit beneficial changes even though you forked… whatever).
    Still, this sounded harsh. You know why? People, and I, love “Arch” (that is the philosophy – i still hate stuff about Arch, but that’s stuff I would get with other distros, too).
    I would hate to see “it” lose it’s way – even though of course anyone could adopt the way and make a new distro (and even use yours as a starting point at any time).

    Let those who say that they’ll jump ship because of system jump ship, but: remember the way!

    “without unnecessary additions, modifications, or complications”
    The point of Arch I love the most.
    Other Distros impose their brand on software, you try not to – keep it up (or I might actually try and jump ship – this is what brought me, and probably countless oters, to Arch).

    “Complexity without complication.”
    At the moment, you might be doing “the right thing” in choosing systemd, as dodging it adds complexity.

    “Arch Linux targets and accommodates competent GNU/Linux users by giving them complete control and responsibility over the system.”
    “By keeping the system simple, Arch Linux provides the freedom to make any choice about the system.”

    What I don’t like about this decision is that it empowers the systemd developers and the systemd ecosystem.
    I would not mind this if I hadn’t had the gut feeling that “systemd” will try to make many parts of the system’s software dependent on it.
    There might come a point where you simply can’t live without systemd (e.g. replace with your init, log whatever of choice). I’d hate to see that.
    systemd might still eat your children in the future.

    Some minor complaints about your arguments:

    Shared libraries: You pit them against Pipelining simple programs.
    Your argument should be: Libraries
    Shared or static, benefits are the same imho. Static relies on build-time (packager) work, dynamic mostly on run-time work (loader).

    Don’t hate on The Unix Way. Programs which need IPC can use other mechanisms than “cmd1 | cmd2 | cmd3″.
    Imo they should use that way though if data is interesting for users to use, or if it can be desirable to “filter” the data.

    • Marvn says:

      “What I don’t like about this decision is that it empowers the systemd developers and the systemd ecosystem.”

      Yeah, but in the same way, Arch empowers linux developers, glibc developers and many other developers of unreplacable Arch linux parts. This argument is quite pointless I’m afraid…

  13. [...] “I AM LEAVING!” email on the arch-general mailing list this morning citing this article;http://archlinux.me/brain0/2012/09/10/yet-another-systemd-comment/and the reason why the author of the email was [...]

  14. James says:

    Awesome just Awesome! “If you belong in one of these groups Fuck you! ha ha

    After 10 years of Windows and then trying what seems like every distro under the sun “The Arch Way” is now my way, I am home.

    • phani says:

      same here; i’ve been using openSUSE for a few years, now i came to arch and find it refreshingly simple & straightforward.

      openSUSE has to lug a lot of baggage around because of it’s affiliation with novel / SUSE enterprise, and while it’s a great distro, this makes things more difficult & complex than i care for.

      came to arch only recently, and for a while i couldn’t believe my luck: without using [testing] at all, i get a system that’s pretty much ‘bleeding edge’ and stable at the same time. until now i was holding my breath, expecting the whole thing to come crashing down around my ears eventually, but it doesn’t!

      and congrats to your style as well. you won’t find replies like this on openSUSE. there, everything has to be vetted by lawyers and the novel mothership before it gets published.

      my advice: stop worrying about those anti-systemd trolls; there’s nothing to discuss anymore. ban them from the mailing list, or just ignore their posts, whatever comes easiest. don’t waste your time or drive your blood pressure into orbit; isn’t worth it.

  15. Gimmy says:

    if you write a book about Arch Linux I will buy it :)