Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A new upstream project to break up Docker into independent components #32691

Merged
merged 2 commits into from Apr 20, 2017

Conversation

shykes
Copy link
Contributor

@shykes shykes commented Apr 18, 2017

Work has been ongoing to break Docker into modular components for some time, with runc and containerd as examples. Containerd for example has been donated to the CNCF. We are now completing this work with the goal being that the monolithic docker repo eventually ceases to exist, instead being assembled from a set of components.

Docker is, and will remain, a open source product that lets you build, ship and run containers. It is staying exactly the same from a user’s perspective. Users can download Docker from the docker.com website.

Moby is a project which provides a “Lego set” of dozens of components, the framework for assembling them into custom container-based systems, and a place for all container enthusiasts to experiment and exchange ideas. You can learn more at http://mobyproject.org

Docker the product will be assembled from components that are packaged by the Moby project.

The Moby project provides a command-line tool called moby which assembles components. Currently it assembles bootable OS images, but soon it will also be used by Docker for assembling Docker out of components, many of which will be independent projects.

As the Docker Engine continues to be split up into more components the Moby project will also be the home for those components until a more appropriate location is found.

Docker is transitioning all of its open source collaborations to the Moby project going forward.
During the transition, all open source activity should continue as usual.

This pull request changes the README and kicks off the process of breaking up the engine under Moby. Again: if you are a Docker user, nothing changes.

README.md Outdated
@@ -276,29 +84,5 @@ For more information, please see https://www.bis.doc.gov
Licensing
=========
Docker is licensed under the Apache License, Version 2.0. See
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docker -> Moby?

@fsouza
Copy link
Contributor

fsouza commented Apr 18, 2017

Is updating the import paths within the moby project part of the scope of this change?

Right now if I do go get -d github.com/moby/moby/... this repo will get cloned twice because moby's source is still using the import path github.com/docker/docker. And since go get doesn't do shallow clones, cloning this repo twice can take a lot of time, specially if you're using some conference's wifi :)

Thanks!

@ghost
Copy link

ghost commented Apr 19, 2017

Did that means docker will all to moby

docker run   =>  moby run 
docker build => moby build

Am i right?

@mwarkentin
Copy link

@netroby I think docker CE / EE will continue to exist, and the CLI will remain the same - however they'll be built out of these moby components.

@omeid
Copy link

omeid commented Apr 19, 2017

This is not gonna work nice for all the projects that depend on github.com/docker/docker. No notice, nothing, and just breaking everything. Not cool.

@ghost
Copy link

ghost commented Apr 19, 2017

The docker/docker repository does not exists. only moby/moby..

Where is the docker ce code? inside the moby project?

@shykes
Copy link
Contributor Author

shykes commented Apr 19, 2017

@omeid nothing should break. We have moved the repo before (from the dotcloud org) and your go get should handle the redirect gracefully.

In parallel we are setting up a facade URL for go packages (which we should have done long ago) to separate package names from git hosting.

@shykes
Copy link
Contributor Author

shykes commented Apr 19, 2017

@netroby the Docker CE code is spread out in dozens of repos: containerd, libnetwork, swarmkit etc. Some of it is in moby/moby, and is being spun out into new components until moby/moby only includes the tooling to assemble Docker CE from the components (or your custom alternative to Docker CE).

@kunalkushwaha
Copy link
Contributor

kunalkushwaha commented Apr 19, 2017

What are the plans for Infrakit and Swarmkit? Will they be part of docker.org or may come to Moby project, considering they are vital part of docker container platform.

@frekele
Copy link

frekele commented Apr 19, 2017

It no makes sense to change the name of the project already widespread in the community, many projects will be broken because of this, change without any warning.
I think there is a lack of common sense with a community.

I'm very confused by this change.
Moby = Open Source Development?
Docker CE = Open Source Release to Production? Or commercial now? or Dead?
Docker EE = Commercial Release to Production? or Dead?

Where is the Docker CE now?

Sorry, but it's not clear to me. Could you explain the structure of the project better and how will the licenses stay?

@shykes
Copy link
Contributor Author

shykes commented Apr 19, 2017

@frekele

Moby = open source development
Docker CE = free product release based on Moby
Docker EE = commercial product release based on Docker CE.

Nothing is dead; and everything that was open-source remains open-source. In fact we are open-sourcing new things.

@frekele
Copy link

frekele commented Apr 19, 2017

Thank you @shykes for the feedback.

It would be interesting to create a migration guide from docker to moby, so we can better understand what it affects, before the next release.

@duffn
Copy link

duffn commented Apr 19, 2017

Moby = open source development
Docker CE = free product release based on Moby
Docker EE = commercial product release based on Docker EE.

Nothing is dead; and everything that was open-source remains open-source. In fact we are open- sourcing new things.

@shykes This should be made markedly clearer then. A few line GitHub issue without additional reference is not enough for your many, many users.

@shykes
Copy link
Contributor Author

shykes commented Apr 19, 2017

@duffn understood we will refine the explanation. We wanted to get things started in the open as early as possible, so that we can prepare the transition in the open woth all the contributors and maintainers.

Note: for the huge majority of Docker users nothing changes at all: the binary packages remain exactly the same. The github repo is used by thousands of people - the binary packages by millions.

@shykes
Copy link
Contributor Author

shykes commented Apr 19, 2017

@frekele good idea. the short version is that Docker CE and EE users are not affected at all. The binary releases remain exactly the same.

Moby will get its own release process - probably source releases instead of binary. We need to discuss with the maintainers what it will be. We have a summit thursday to get things started.

@shykes
Copy link
Contributor Author

shykes commented Apr 19, 2017

@kunalkishwaha SwarmKit and InfraKit will remain standalone like today and will not be affected. We would like to move them into separate standalone github orgs to clarify that we want them to be open and not dependent on Docker. We will discuss that move with the maintainers first. We will use Moby to develop and integrate SwarmKit and InfraKit, but they will not be "owned" by Moby.

@panuhorsmalahti
Copy link

Will Docker CE be in part proprietary in the future? (Maybe it already is and I missed the news).

@Vanuan
Copy link

Vanuan commented Apr 19, 2017

@panuhorsemalahti AFAIK there was never a way to build Docker CE from sources. Now it seams there is.

I think it will be like Chrome vs Chromium.

@krasi-georgiev
Copy link
Contributor

It took me a while and a bit of reading, but I think now I understand the idea and really like it.

The "docker" idea just goes a level deeper by using the Moby project to build the base OS on which to run the containers.

I think container oriented kernel changes would be one of good the side effects for this transition.

@gaocegege
Copy link
Contributor

gaocegege commented Apr 19, 2017

Since moby and docker focus on different aspects, why not keep two repos 🤔

And, could moby build the unikernel, instead of minimal linux distribution?

Anyway, congratulations 🎉

@krasi-georgiev
Copy link
Contributor

@gaocegege if I understand it correctly moby will use linuxkit as the moby base cli #32693 and looking at the linuxkit repo there is already support for unikernels.

https://github.com/linuxkit/linuxkit/tree/master/projects/miragesdk

@catap
Copy link

catap commented Apr 19, 2017

@shykes

Moby = open source development
Docker CE = free product release based on Moby
Docker EE = commercial product release based on Docker EE.

Am I right that Docker EE will be something not related to Docker CE/Moby?

@thaJeztah
Copy link
Member

Am I right that Docker EE will be something not related to Docker CE/Moby?

The last line should probably be;

Docker EE = commercial product release based on Docker CE.

Docker EE is on the same code base as Docker CE, so also built from Moby, with commercial components added, such as "docker data center / universal control plane"

@zaquestion
Copy link

classic

@shykes
Copy link
Contributor Author

shykes commented Apr 20, 2017

@gaocegege we moved the repository because the first task of the Moby project is to break up the engine monolith into smaller components, and spin them out into separate repos. Once that is complete, Moby will be an integration repository, with the tools necessary to combine upstream components into a complete system.

What we want to do (and clearly we need to communicate this better) is ask for the contributor's help in making this happen. Let's finish breaking up the monolith together - and then let's make Moby the best place to assemble the components in many different ways, so that everyone can do whatever they want with them.

@shykes
Copy link
Contributor Author

shykes commented Apr 20, 2017

To clarify the "production chain", from upstream to downstream:

upstream components (containerd, linuxkit etc) -> Moby -> Docker CE -> Docker EE.

@AkihiroSuda
Copy link
Member

+1 for technical modularization of current engine, but I fear separating GitHub repo can cause vendoring hell.
How about creating multiple "module" directories on single repo so as to avoid vendor hell?

@omeid
Copy link

omeid commented Apr 21, 2017

Let me ask it like it is, I depend on docker (the current standard distribution) both as a Go package and as a running daemon for a lot of my projects and my clients. What should I plan ahead? How much work should I expect to put in?

@Wirone
Copy link

Wirone commented Apr 21, 2017

Isn't Moby too busy with making music so he won't be able to handle all the building stuff? How this guy will do this single handedly.. All tech people, all the companies will rely on one bald guy with glasses.. I guess at some point he will say "I just cannot contain this" (pun intended).

@apakhomov
Copy link

Did you try to use google when choosing new name? Now everybody will have to filter a lot of search rubbish about musician, moby dick, moby explorer, mobile applications etc when searching moby-related issues? Awesome

@shykes
Copy link
Contributor Author

shykes commented Apr 21, 2017

@omeid no work to plan. Nothing will change for you. Just keep installing Docker from the docker.com packages as usual. Same for the Go packages.

@brettdh
Copy link
Contributor

brettdh commented Apr 21, 2017

For further clarification on the respective roles of Docker and Moby, please see this link.

@shykes did you mean to include a link there? 🙂 Maybe this one?

@JonET
Copy link

JonET commented Apr 21, 2017

Appropriate:
https://www.youtube.com/watch?v=5Og42ldefVc

@lukemarsden
Copy link
Contributor

@brettdh yes that's the right link

@frekele
Copy link

frekele commented Apr 22, 2017

As the thread was unfolding with community input and comments from @shykes I began to understand the real purpose of this change. And I truly fully agree that the moby project only brings more advantages to the open source community.

This will bring more Collaboration of the community, and each time we will have a more incredible tool beyond what it already is.

An interesting post about this.
https://thenewstack.io/docker-seeds-two-new-projects-building-containerized-infrastructure/

I believe this image says a lot about docker/moby changes.
88fd7e29-docker-upstream 1

*Thus far, Docker already has a library of more than 80 containerized components, including Swarm, containerd, Docker Build and even LinuxKit. Many third-party components are now hard-wired into Linux distributions. Docker is welcoming more contributions from the community.here

Now with moby we can also have a third-party Moby projects.
With moby, now many components that were previously attached to the docker, will be linked to the moby, thereby generating more feedback, issues, PR of the community.

I think it's going to be this way now.

moby-structure-build

Correct me if I'm wrong. Please. 🙏 😄

@fatherlinux
Copy link

@frekele while, I really want to believe this is a good move for the community, there is one critical component missing. In the drawings referenced, there is no mention of the docker command, docker daemon, and Dockerfile. There is no mention of whether they stay in the Moby project/repo or are separated out into another repository.

Either way, if they remain named docker daemon, docker command, and Dockerfile, it prevents any other company from extracting any value from their contributions to these projects because only Docker Inc. retains the rights to use the docker name.

As somebody who has targeted and refined my career over the last 4 years based on Docker/Containers, I can no longer extract value from that participation directly without becoming a Docker EE reseller of some kind. I can't build a company doing consulting on the docker daemon, Dockerfiles because that would be a breach Docker Inc's usage policy.

In fact, nobody can build anything downstream from Moby that includes the docker daemon, command or Dockerfile and talk about it in any sane way. The docker daemon absolutely needs renamed or Docker Inc is the only company that can extract value.

Open source projects like Linux, Apache, KDE, etcd, etc allow ANYBODY to extract downstream value. If the goal is openness, then I ask Docker Inc. to rename the docker daemon to mobyd, the command to moby, and the Dockerfile to Mobyfile. Then, anybody can build downstream value. I personally, am very upset. (opinions here my own, not those of my employer Red Hat).

@shykes
Copy link
Contributor Author

shykes commented Apr 22, 2017

@frekele that's exactly right! Thank you for the extended drawing, and for taking the time to write this.

@shykes
Copy link
Contributor Author

shykes commented Apr 22, 2017 via email

@fatherlinux
Copy link

fatherlinux commented Apr 22, 2017 via email

@zouyee
Copy link

zouyee commented Apr 24, 2017

Moby Dick

@ozbillwang
Copy link
Contributor

ozbillwang commented Apr 24, 2017

our model is closer to the Red Hat model: we do everything in the open, in a fast-moving project where everyone can participate. Then we integrate, test and harden downstream into a supported enterprise product.

so docker as Red Hat, moby as centos?

@Vanuan
Copy link

Vanuan commented Apr 24, 2017

so docker as Red Hat, moby as centos?

I would think it's more like this:

Docker EE = RHEL
Docker CE = CentOS
Moby project = Fedora project

@fatherlinux
Copy link

Docker CE can't be CentOS because CentOS is a downstream build of Red Hat Enterprise Linux. The Red Hat relationship is below, each step is a selection, packaging and/or hardening process of the upstream because they are ALL open source:

Linux & Community Projects-> Fedora -> Red Hat -> CentOS

With Docker it's more like:

Community Projects (Containerd, Dockerd (maybe), Runc) -> Moby -> Docker CE -> Docker EE -> Dead End (because Docker EE is not made of all open source components. Docker EE adds proprietary components, RHEL does not).

@jberkus
Copy link

jberkus commented Apr 24, 2017

@fatherlinux although to be fair, years ago, Red Hat used to have proprietary components (I'm talking about 2002 here).

We don't know that Docker EE will have proprietary components; @shykes has said they'll be open-sourcing some of the things which are currently proprietary. Let's give them a chance, eh?

For my part, a complete Moby kit will make it easier to release the version of Docker (or Moby) we want in Fedora. At least, theoretically. Some of us have been after Docker for the last year and more to "break up the monolith", and now they're doing it, so I think at least a bit of cheering them on is in order.

@jberkus
Copy link

jberkus commented Apr 24, 2017

BTW, @shykes, we have a bunch of community experts in Fedora-land, happy to discuss the new setup if we can help!

@shykes
Copy link
Contributor Author

shykes commented Apr 25, 2017

@jberkus I would love that, thank you for offering. I will contact you on your profile email to pick your brains, if that's OK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet