rfc 3.0 architecture documentation by egernst · Pull Request #543 · clearcontainers/runtime

egernst · 2017-09-12T04:43:23Z

This is very much work in progress, and shouldn't be merged in this state. I'm pushing this as a PR so folks can easily get their eyes on the doc/pictures and start to make suggestions edits. This is based on Graham's initial branch with some quick cleanup on my end. Note, the PNGs are exports from our (sorry, internal at the moment) google doc. Once we have settled on images we can export the odp and put on to the repo as well...

Of note, I'd like to see someone write the section on the agent so it is of equivalent or more detail compared to what we had for hyperstart in 2.1. Not sure, but perhaps @amshinde would be well suited for this? Thoughts?

There are a few fixmes as well, which I figure will be easier to discuss during this RFC PR's review process.

Once we have input from folks, I'd recommend doing a giant squash and adding signed offs from the contributors.

Fixes #392

egernst · 2017-09-12T04:46:53Z

docs/architecture/architecture.md

+time initializing devices of no use for containers.
+- Skipping the guest BIOS/firmware and jumping straight to the Clear Containers kernel.
+
+#### Agent


@amshinde ; can you help with this section?

@egernst I'll take a look.

clearcontainersbot · 2017-09-12T04:49:11Z

Popular Images qa-passed 👍

egernst · 2017-09-12T04:47:55Z

docs/architecture/architecture.md

+multiplexes and demultiplexes those commands and streams for all container virtual machines.
+There is only one `cc-proxy` instance running per Clear Containers host.
+
+On the host, each container process is reaped by a Docker specific (`containerd-shim`) monitoring


This is pretty docker specific. We may want to make this more generic so it can cover !docker as well.

Yes, this needs to be made more generic. At the moment, we always have a reaper between us and the higher layers of the stack. It's either containerd-shim (Docker) or conmon (CRI-O). So we could mention that and just basically replace containerd-shim with container process reaper.

egernst · 2017-09-12T04:48:22Z

docs/architecture/architecture.md

+wants to run within an already running container (`docker exec`).
+
+The container workload, i.e. the actual OCI bundle rootfs, is exported from the host to
+the virtual machine via a 9pfs virtio mount point. Hyperstart uses this mount point as the root


s/Hyperstart/agent

egernst · 2017-09-12T04:49:17Z

docs/architecture/architecture.md

+Although Clear Containers can run with any recent QEMU release, containers boot time and memory
+footprint are significantly optimized by using a specific QEMU version called [`qemu-lite`](https://github.com/clearcontainers/qemu/tree/qemu-lite-v2.9.0).
+
+`qemu-lite` improvements comes through a new `pc-lite` machine type, mostly by:


We are using PC machine type for 3.0, I think for release perhaps it makes sense to just remove all of the permutations and just describe what we actually have?

Yes. Describe what we use by default, and what we support (q35, pc-lite). We should also explain why we go with pc by default.

took a pass at this @ https://github.com/clearcontainers/runtime/wiki/Clear-Containers-Architecture#hypervisor - not sure how much detail we should go into features we aren't actually using though....

egernst · 2017-09-12T04:49:40Z

docs/architecture/architecture.md

+`qemu-lite` improvements comes through a new `pc-lite` machine type, mostly by:
+- Removing many of the legacy hardware devices support so that the guest kernel does not waste
+time initializing devices of no use for containers.
+- Skipping the guest BIOS/firmware and jumping straight to the Clear Containers kernel.


not really the case for 3.0 release, right?

With 2.7, we're not skipping firmware with the pc machine type while we are with pc-lite. Right now our packages install 2.7 and since we switched to pc, we're indeed no longer skipping the firmware.
With 2.9 we could skip the firmware with the pc machine type, we're tracking why we haven't switched to 2.9 yet: clearcontainers/packaging#28

egernst · 2017-09-12T04:50:18Z

docs/architecture/architecture.md

+   virtio I/O serial one).
+3. Run all the [OCI hooks](https://github.com/opencontainers/runtime-spec/blob/master/config.md#hooks) in the container namespaces,
+as described by the OCI container configuration file.
+4. **fixme** [Set up the container networking](https://github.com/clearcontainers/runtime/blob/master/documentation/architecture.md#networking).


fixme is just for the link location, right @grahamwhaley ?

Looks like it. I believe we can change it to a 'local reference', so just #networking should resolve to the section in this document.

egernst · 2017-09-12T05:10:24Z

@sameo @jodh-intel @mcastelino - if you have a chance to start taking a look. @jcvenegas perhaps we can reference the proxy protocol in the proxy section?

@iphutch -- early heads up and FYI that this is (in early stage of) the pipeline

jodh-intel · 2017-09-12T07:46:26Z

docs/architecture/architecture.md

+This is an architectural overview of Clear Containers, based on the 3.0 release.
+
+The [Clear Containers runtime (cc-runtime)](https://github.com/clearcontainers/runtime)
+complies with the [OCI](https://github.com/opencontainers) specifications and thus


I'd say "compatible" rather than "complies with".

agree - 'compatible'

jodh-intel · 2017-09-12T07:47:05Z

docs/architecture/architecture.md

+
+The container process is then spawned by an [agent](https://github.com/clearcontainers/agent),
+running as a daemon on the guest operating system.
+Hyperstart opens 2 virtio serial interfaces (Control and I/O) on the guest, and QEMU exposes them


This should say "The agent", not Hyperstart now.

Small numbers should be spelt out in full, so can you change this to "two".

I'd say "in the guest" rather than "on the guest" to reinforce that we are talking about what is happening inside the VM.

jodh-intel · 2017-09-12T07:54:59Z

docs/architecture/architecture.md

+`cc-runtime` creates a QEMU/KVM virtual machine for each container the Docker engine creates.
+
+The container process is then spawned by an [agent](https://github.com/clearcontainers/agent),
+running as a daemon on the guest operating system.


I'd add "inside the virtual machine" at the end of this sentence to be clearer here.

jodh-intel · 2017-09-12T07:57:10Z

docs/architecture/architecture.md

+`stderr`, `stdin`) between the guest and the Docker Engine.
+
+For any given container, both the init process and all potentially executed commands within that
+container, together with their related I/O streams, need to go through 2 virtio serial interfaces


jodh-intel · 2017-09-12T07:58:28Z

docs/architecture/architecture.md

+
+#### Hypervisor
+
+Clear Containers use [KVM](http://www.linux-kvm.org/page/Main_Page)/[QEMU](http://www.qemu-project.org/) to


About it was QEMU/KVM so I'd stick with that ordering for consistency.

jodh-intel · 2017-09-12T08:06:20Z

docs/architecture/architecture.md

+For more details about `cc-proxy`'s protocol, theory of operations or debugging tips, please read
+[`cc-proxy` README](https://github.com/clearcontainers/proxy).
+
+#### Shim


@amshinde - can you review this section please?

jodh-intel · 2017-09-12T08:08:15Z

docs/architecture/architecture.md

+Clear Containers utilises the Linux kernel DAX (Direct Access filesystem)
+feature to efficiently map some host side files into the guest VM space.
+In particular, Clear Containers uses the `QEMU` nvdimm feature to provide a
+memory mapped virtual device that can be used to DAX map the mini-OS root


This seems to be the first mention of the mini-OS. I think it might be helpful to introduce it when we first start taking about the VM, pointing out that there is a mini-OS and the 9p-mapped docker image (ubuntu, busybox, etc).

jodh-intel · 2017-09-12T08:11:32Z

docs/architecture/architecture.md

+file and device mapping mechanisms:
+
+- Mapping as a direct access devices allows the guest to directly access
+the memory pages (such as via eXicute In Place (XIP)), bypassing the guest


s/the memory pages/the host memory pages/ ?

jodh-intel · 2017-09-12T08:12:28Z

docs/architecture/architecture.md

+host to be demand loaded using page faults, rather than having to make requests
+via a virtualised device (causing expensive VM exits/hypercalls), thus providing
+a speed optimisation.
+- Utilising shmem MAP_SHARED on the host allows the host to efficiently


s/shmem MAP_SHARED/MAP_SHARED shared memory/ ?

jodh-intel · 2017-09-12T08:14:20Z

docs/architecture/architecture.md

+Information on the use of nvdimm via QEMU is available in the QEMU source code
+[here](http://git.qemu-project.org/?p=qemu.git;a=blob;f=docs/nvdimm.txt;hb=HEAD)
+
+### Previous releases


Maybe this should be called "Architectural changes by release" or something? We can then document here what changed between 2.1 and 3.0.

good idea @jodh-intel . I made this update on the wiki: https://github.com/clearcontainers/runtime/wiki/Clear-Containers-Architecture#architectural-changes-by-release

grahamwhaley · 2017-09-12T09:40:29Z

docs/architecture/architecture.md

+
+![QEMU/KVM](qemu.png)
+
+** fixme - discuss different QEMUs - lite, q35, pc etc. **


I think for this release we can drop this 'fixme' and just mention qemu-lite.

We can also mention that we support several QEMU machine types.

sameo · 2017-09-12T09:53:49Z

docs/architecture/architecture.md

+wants to run within an already running container (`docker exec`).
+
+The container workload, i.e. the actual OCI bundle rootfs, is exported from the host to
+the virtual machine via a 9pfs virtio mount point. Hyperstart uses this mount point as the root


Here we should mention that we go for virtio-blk if we find a block based graph driver.

sameo · 2017-09-12T09:58:25Z

docs/architecture/architecture.md

+multiplexes and demultiplexes those commands and streams for all container virtual machines.
+There is only one `cc-proxy` instance running per Clear Containers host.
+
+On the host, each container process is reaped by a Docker specific (`containerd-shim`) monitoring


Yes, this needs to be made more generic. At the moment, we always have a reaper between us and the higher layers of the stack. It's either containerd-shim (Docker) or conmon (CRI-O). So we could mention that and just basically replace containerd-shim with container process reaper.

sameo · 2017-09-12T09:59:23Z

docs/architecture/architecture.md

+
+![QEMU/KVM](qemu.png)
+
+** fixme - discuss different QEMUs - lite, q35, pc etc. **


We can also mention that we support several QEMU machine types.

sameo · 2017-09-12T10:00:23Z

docs/architecture/architecture.md

+Although Clear Containers can run with any recent QEMU release, containers boot time and memory
+footprint are significantly optimized by using a specific QEMU version called [`qemu-lite`](https://github.com/clearcontainers/qemu/tree/qemu-lite-v2.9.0).
+
+`qemu-lite` improvements comes through a new `pc-lite` machine type, mostly by:


Yes. Describe what we use by default, and what we support (q35, pc-lite). We should also explain why we go with pc by default.

sameo · 2017-09-12T10:01:35Z

docs/architecture/architecture.md

+by a set of namespaces (UTS, PID, mount and IPC). Although a pod can hold several containers,
+`cc-runtime` always runs a single container per pod. **fixme** incorrect<--
+
+**todo** add details on the agent protocol


Also, explain that the agent is based on libcontainers, the runc library, and reference it.

sameo · 2017-09-12T13:19:55Z

docs/architecture/architecture.md

+1. `cc-runtime` connects to `cc-proxy` and sends it the `attach` command to let it know which pod
+we want to use to run the `exec` command.
+2. `cc-runtime` sends the allocateIO command to the proxy, for getting the `agent` I/O sequence
+numbers for the `exec` command I/O streams.


This has also changed with 3.0. We connect to the proxy and get a token back. We then create a shim with the given token and the shim connects to the proxy.

sameo · 2017-09-12T13:22:43Z

docs/architecture/architecture.md

+agent instance running in the appropriate guest.
+3. After deleting the last running pod, the `agent` will gracefully shut the virtual machine
+down.
+4. `cc-runtime` sends the `BYE` command to `cc-proxy`, to let it know that a given virtual


BYE has been replaced with UnregisterVM

sameo · 2017-09-12T13:25:54Z

docs/architecture/architecture.md

+
+- A UNIX, named socket for all `cc-runtime` instances on the host to send commands to `cc-proxy`.
+- One socket pair per `cc-shim` instance, to send stdin and receive stdout and stderr I/O streams. See the
+[cc-shim section](#shim)


That has changed as well and is now simplified. There is one single socket (UNIX or TCP) for all runtimes and all shims.

@jcvenegas -- can you take a look and update?

@egernst I just check it is updated describing the interaction between proxy and shim in the new protocol probablyt @amshinde updated :D

sameo · 2017-09-12T13:26:17Z

docs/architecture/architecture.md

+the `AllocateIO` command to `cc-proxy` to have it request the `agent` to allocate those sequence numbers.
+They will be passed as command line arguments to `cc-shim`, who will then use them to e.g. prepend its stdin
+stream packets with the right sequence number.
+- `Hyper`: This command is used by both `cc-runtime` and `cc-shim` to forward `agent` specific


This is well out of date as well for 3.0.

sameo · 2017-09-12T13:26:44Z

docs/architecture/architecture.md

+- `Hyper`: This command is used by both `cc-runtime` and `cc-shim` to forward `agent` specific
+commands.
+
+For more details about `cc-proxy`'s protocol, theory of operations or debugging tips, please read


We should definitely have a link to it.

mcastelino · 2017-09-12T16:18:35Z

docs/architecture/architecture.md

+The [Clear Containers runtime (cc-runtime)](https://github.com/clearcontainers/runtime)
+complies with the [OCI](https://github.com/opencontainers) specifications and thus
+works seamlessly with the [Docker Engine](https://www.docker.com/products/docker-engine)
+pluggable runtime architecture. In other words, one can transparently replace the


I would not say replace. We can add another runtime. We cannot satisfy all possible types of containers like priv ones.

mcastelino · 2017-09-12T16:20:08Z

docs/architecture/architecture.md

+running as a daemon on the guest operating system.
+Hyperstart opens 2 virtio serial interfaces (Control and I/O) on the guest, and QEMU exposes them
+as serial devices on the host. `cc-runtime` uses the control device for sending container
+management commands to the agent while the I/O serial device is used to pass I/O streams (`stdout`,


Would be good to point to a section that contain the container management command or maybe the source file where the command protocol is defined.

Added url to to godoc proxy api protocol, and added link to UML sequence diagram.

mcastelino · 2017-09-12T16:20:58Z

docs/architecture/architecture.md

+management commands to the agent while the I/O serial device is used to pass I/O streams (`stdout`,
+`stderr`, `stdin`) between the guest and the Docker Engine.
+
+For any given container, both the init process and all potentially executed commands within that


Would be good to document how exec is handled too.

mcastelino · 2017-09-12T16:23:45Z

docs/architecture/architecture.md

+
+The `agent` execution unit is the pod. An `agent` pod is a container sandbox defined
+by a set of namespaces (UTS, PID, mount and IPC). Although a pod can hold several containers,
+`cc-runtime` always runs a single container per pod. **fixme** incorrect<--


This is not accurate

cc-runtime always runs a single container per pod.

mcastelino · 2017-09-12T16:24:26Z

docs/architecture/architecture.md

+
+Here we will describe how `cc-runtime` handles the most important OCI commands.
+
+##### `create`


Would be nice to link to the source code file that implements create

egernst · 2017-09-12T18:33:23Z

Moved this document to https://github.com/clearcontainers/runtime/wiki/Clear-Containers-Architecture to facilitate better collaboration at this point in the review process.

egernst · 2017-09-14T15:35:35Z

@iphutch - Can you start taking a look at this PR?

iphutch

Line length and indentation are the biggies here. It's a good looking doc. See suggestions for other changes too.

iphutch · 2017-09-14T16:46:06Z

docs/architecture/architecture.md

+is compatible with the [OCI](https://github.com/opencontainers) [runtime specification](https://github.com/opencontainers/runtime-spec)
+and thus works seamlessly with the
+[Docker Engine](https://www.docker.com/products/docker-engine) pluggable runtime
+architecture. It also supports the [Kubernetes Container Runtime Interface (CRI)](https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/apis/cri/v1alpha1/runtime) through the [CRI-O](https://github.com/kubernetes-incubator/cri-o) implementation. In other words, one can transparently select between the


Line length

Docker, Kubernetes, and CRI-O need to be followed by an * in their first instance in the doc.

Let's avoid "one". Saying "you" is acceptable when referring to the reader/user:
In other words, you can transparently...

iphutch · 2017-09-14T16:55:08Z

docs/architecture/architecture.md

+![Runtime and virtcontainers](arch-images/runtime-vc-relationship.png)
+![Docker and Clear Containers](arch-images/docker-cc.png)
+
+`cc-runtime` creates a QEMU/KVM virtual machine for each container the Docker


iphutch · 2017-09-14T17:01:06Z

docs/architecture/architecture.md

+
+The [Clear Containers runtime (cc-runtime)](https://github.com/clearcontainers/runtime)
+is compatible with the [OCI](https://github.com/opencontainers) [runtime specification](https://github.com/opencontainers/runtime-spec)
+and thus works seamlessly with the


replace/thus/therefore

iphutch · 2017-09-14T19:15:24Z

docs/architecture/architecture.md

+to the container process on the guest and pass the container `stdout` and `stderr`
+streams back up the stack to CRI-O or Docker via the container process reaper.
+`cc-runtime` creates a `cc-shim` daemon for each container and for each OCI command
+received to run within an already running container (i.e. `docker exec`).


Line length on 66 and 68

iphutch · 2017-09-14T19:17:42Z

docs/architecture/architecture.md

+`cc-runtime` creates a `cc-shim` daemon for each container and for each OCI command
+received to run within an already running container (i.e. `docker exec`).
+
+The container workload, i.e. the actual OCI bundle rootfs, is exported from the


Avoid latin abbreviations:
The container workload, that is, the actual OCI bundle rootfs, is...

iphutch · 2017-09-14T20:04:12Z

docs/architecture/architecture.md

+2. Get CNI plugin information
+
+3. Start the plugin (providing previously created netns) to add a network
+described into /etc/cni/net.d/ directory. At that time, the CNI plugin will


use literal text for the directory name

iphutch · 2017-09-14T20:07:33Z

docs/architecture/architecture.md

+5. Start VM inside the netns and start the container
+
+## Storage
+Container workloads are shared with the virtualized environment through 9pfs.


can we link to information on 9pfs?

iphutch · 2017-09-14T20:09:42Z

docs/architecture/architecture.md

+
+## DAX
+
+Clear Containers utilises the Linux kernel DAX (Direct Access filesystem)


Ditto, can we link to DAX info here?:
Direct Access filesystem

iphutch · 2017-09-14T20:11:00Z

docs/architecture/architecture.md

+share pages.
+
+Clear Containers uses the following steps to set up the DAX mappings:
+- QEMU is configured with an nvdimm memory device, with a memory file


Use numbers instead of - here. Also, indentation

iphutch · 2017-09-14T20:13:04Z

docs/architecture/architecture.md

+More information about DAX can be found in the Linux Kernel
+[documentation](https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/dax.txt)
+
+Information on the use of nvdimm via QEMU is available in the QEMU source code


make "QEMU source code" the link and remove "here"

egernst · 2017-09-14T20:29:13Z

@iphutch -- Thanks! Will start to address these now...

egernst · 2017-09-14T21:36:54Z

docs/architecture/architecture.md

+- Connects to `cc-proxy` using a token obtained by calling the `cc-proxy` `ConnectShim` command. The token is passed from `cc-runtime` to `cc-shim` when the former spawns the latter and is used to identify the true container process that the shim process will be shadowing or representing.
+- Fragments and encapsulates the standard input stream from the container process reaper into `cc-proxy` stream frames:
+```
+                     1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3


@amshinde -- what's the intention for the above line?

good question , I had asked the same to damien with no answer on that .
This is a direct copy of the frame format from his documented proxy protocol :)

sameo · 2017-09-14T22:05:27Z

docs/architecture/architecture.md

+
+- Moved from hyperstart to `cc-agent` as an agent inside the VM.
+- Moved from `qemu-lite` to `pc` QEMU machine type.
+- Rewrite of runtime in go, leveraging virtcontainers.


I'd add a few items here:

virtio-blk for block based graph drivers

New simplified protocol between the shim, proxy and runtime

KSM throttling

Agreed, though if we bring up KSM throttlng, we should probably describe its setup in the proxy section, right?

iphutch

You're a champ for getting the bulk of these indentation issues, a few more to go!

iphutch · 2017-09-14T22:34:02Z

docs/architecture/architecture.md

+3. Call the prestart hook (from inside the netns)
+
+4. Scan network interfaces inside netns and get the name of the interface
+created by prestart hook


Indentation :( From here down. line 525, 538, 543, 592, 595,599, 651, and finally (fittingly) 666.

iphutch

Three things to fix and I'm happy to approve :)

iphutch · 2017-09-14T23:10:00Z

docs/architecture/architecture.md

+[default Docker and CRI-O runtime (runc)](https://github.com/opencontainers/runc)
+and `cc-runtime`.
+
+![Runtime and virtcontainers](arch-images/runtime-vc-relationship.png)


This .png isn't included in the PR or has a different name. It will bring a 404 error.

y'ouch - git blame says that wa sfrom me, but I don't recall adding this, nor can I find this image anywehere... removing now...

iphutch · 2017-09-14T23:12:55Z

docs/architecture/architecture.md

+the [Clear Containers QEMU repo](https://github.com/clearcontainers/qemu/tree/qemu-lite-v2.9.0).
+This transition has been delayed until after the release of Clear Containers 3.0
+due to regressions, as described in [runtime issue 407]
+(https://github.com/clearcontainers/runtime/issues/407). Once support for


Breaking up this runtime issue 407 breaks the link's markup. Don't worry about going over 78chars when it's code or markup forcing you over:
due to regressions, as described in runtime issue 407.
Once support for

I meant to fix all those -- thanks! Done

iphutch · 2017-09-14T23:15:32Z

docs/architecture/architecture.md

+## DAX
+
+Clear Containers utilises the Linux kernel DAX [(Direct Access filesystem)]
+(https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/dax.txt)


Build error. Ignore line length rules for markup OR bring the whole link to 575.

Whoops - thanks

iphutch

Thanks for all the changes! Thumbs up from me.

jodh-intel · 2017-09-15T07:42:12Z

docs/architecture/architecture.md

+## Agent
+
+[`cc-agent`](https://github.com/clearcontainers/agent) is a daemon running in the
+guest as a supervisor for managing containers and processes potentially running


The word "potentially" is redundant so I'd drop it.

agreed - was just about to say the same thing.

jodh-intel · 2017-09-15T09:15:04Z

You might like to add in some Contributions-by: in the commit like we have here: 56e96d3.

grahamwhaley

Nothing serious, mostly stylistic etc.

grahamwhaley · 2017-09-15T10:46:00Z

docs/architecture/architecture.md

+
+![Docker and Clear Containers](arch-images/docker-cc.png)
+
+`cc-runtime` creates a QEMU\*/KVM virtual machine for each container the Docker


Would it be more accurate to say 'for each container or pod', rather than just 'container' - as for k8s I believe we have one-VM per pod?

Sure, agreed.

grahamwhaley · 2017-09-15T11:02:52Z

docs/architecture/architecture.md

+## Agent
+
+[`cc-agent`](https://github.com/clearcontainers/agent) is a daemon running in the
+guest as a supervisor for managing containers and processes potentially running


agreed - was just about to say the same thing.

grahamwhaley · 2017-09-15T11:06:05Z

docs/architecture/architecture.md

+- A control serial channel over which the `cc-agent` sends and receives specific
+  commands for controlling and managing pods and containers. Detailed information
+  about the commands can be found at [`cc-agent` API](https://github.com/clearcontainers/agent/tree/master/api).
+- An I/O serial channel for passing the container processes output streams (stdout,


earlier in the doc we backtick std[in|out|err] to stdin - we should consider doing that consistently through the document.

grahamwhaley · 2017-09-15T11:14:59Z

docs/architecture/architecture.md

+
+1. Create the networking container namespace on the host, according to the container
+  OCI configuration file. We only support networking namespaces for now, but
+  will support more of them later.


s/of them/namespaces/

or even s/more of them/other namespaces/

grahamwhaley · 2017-09-15T11:16:56Z

docs/architecture/architecture.md

+2. Run all the [prestart OCI hooks](https://github.com/opencontainers/runtime-spec/blob/master/config.md#hooks)
+  in the host namespaces created in step 1, as described by the OCI container
+  configuration file.
+3. [Set up the container networking namespace up](#networking). This is when


'Set up the container networking namespace up': Is that 'up' at the end a typo/redundant?

agreed. done

grahamwhaley · 2017-09-15T11:47:08Z

docs/architecture/architecture.md

+  `cc-proxy` and let it know that they stop monitoring their container process.
+
+For more details about `cc-proxy`'s protocol, theory of operations or debugging
+tips, please read


s/read/read the/

grahamwhaley · 2017-09-15T11:48:19Z

docs/architecture/architecture.md

+
+## Shim
+
+A container process reaper, such as Docker's `containerd-shim` or crio's `conmon`,


s/crios/CRI-O's/

grahamwhaley · 2017-09-15T11:50:15Z

docs/architecture/architecture.md

+
+`cc-shim` has an implicit knowledge about which VM agent will handle those streams
+and signals and thus acts as an encapsulation layer between the container process
+reaper and the `cc-agent`. `cc-shim`:


Maybe put cc-shim: on a new paragraph, and expand a little to something like 'cc-shim performs the following steps:' or similar.

grahamwhaley · 2017-09-15T11:51:36Z

docs/architecture/architecture.md

+- Fragments and encapsulates the standard input stream from the container process
+  reaper into `cc-proxy` stream frames:
+```
+                  1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3


As I think @egernst and @amshinde have noted - the '1 1 1 1 ' line seems redundant and makes no sense. Yes, we queried this on the original doc, and I have a feeling the answer was 'oops, a copy/paste issue' :-)

I should note - if anything then I think it would have been denoting byte lanes - but it is failing at that on many fronts right now. It might be nice to show it as byte lanes though (1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3

Now that it is adjusted, it is showing bit number, the 1's aren't redundant, they are the decimal MSB for each bit. Turn your head clockwise when you look at it :)

grahamwhaley · 2017-09-15T12:02:51Z

docs/architecture/architecture.md

+Users can check to see if the container uses devicemapper block device as its
+rootfs by calling `mount(8)` within the counter.  If devicemapper block device
+is used, '/' will be mounted on `/dev/vda`.
+


We could note that devicemapper block device mode can be disabled in the runtime config file if necessary.

grahamwhaley · 2017-09-15T12:05:13Z

This is looking really good - a visible improvement on the already rather good CC2.x document - so, kudos to everybody involved in the update.

jodh-intel · 2017-09-15T14:25:28Z

docs/architecture/architecture.md

+This is an architectural overview of Clear Containers, based on the 3.0 release.
+
+The [Clear Containers runtime (cc-runtime)](https://github.com/clearcontainers/runtime)
+is compatible with the [OCI](https://github.com/opencontainers) [runtime specification](https://github.com/opencontainers/runtime-spec)


Hi @iphutch - does OCI need an asterisk here?

Yes, we will need an asterisk here. Good catch.

jodh-intel · 2017-09-15T14:26:56Z

docs/architecture/architecture.md

+
+In the future, Clear Containers plan to move to a 2.9 based version of QEMU,
+available at 
+the [Clear Containers QEMU repo](https://github.com/clearcontainers/qemu/tree/qemu-lite-v2.9.0).


s/repo/repository/.

jodh-intel · 2017-09-15T14:28:12Z

docs/architecture/architecture.md

+within those containers.
+
+The `cc-agent` execution unit is the pod. A `cc-agent` pod is a container sandbox
+defined by a set of namespaces (NS, UTS, IPC and PID). `cc-runtime` can run several


s/NS/mount, net, cgroup/ I think @sameo?

jodh-intel · 2017-09-15T14:30:36Z

docs/architecture/architecture.md

+the [Clear Containers QEMU repo](https://github.com/clearcontainers/qemu/tree/qemu-lite-v2.9.0).
+This transition has been delayed until after the release of Clear Containers 3.0
+due to regressions, as described in [runtime issue 407](https://github.com/clearcontainers/runtime/issues/407).
+Once support for features like hotplug are available in `Q35`, the project will


s/Q35/q35/ for consistency with all other mentions of machine types.

jodh-intel · 2017-09-15T14:32:29Z

docs/architecture/architecture.md

+Most users will not need to modify the configuration file.
+
+The file is well commented and provides a few "knobs" that can modify the
+behaviour of the runtime.


I wrote this but on re-reading, I think it would be better to say, "that can be used to modify the behaviour of the runtime."

jodh-intel · 2017-09-15T14:49:58Z

docs/architecture/architecture.md

+5. Start VM inside the netns and start the container
+
+## Storage
+Container workloads are shared with the virtualized environment through [9pfs](https://www.kernel.org/doc/Documentation/filesystems/9p.txt).


s/through 9pfs/using the 9pfs filesystem/

jodh-intel · 2017-09-15T14:50:57Z

docs/architecture/architecture.md

+Clear Containers utilises the Linux kernel DAX [(Direct Access filesystem)](https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/dax.txt)
+feature to efficiently map some host side files into the guest VM space.
+In particular, Clear Containers uses the `QEMU` nvdimm feature to provide a
+memory mapped virtual device that can be used to DAX map the virtual machine's


hyphenate "memory-mapped"?

I generally try to avoid hyphenation but I'm on board with memory-mapped and host-side.

jodh-intel · 2017-09-15T14:51:21Z

docs/architecture/architecture.md

+feature to efficiently map some host side files into the guest VM space.
+In particular, Clear Containers uses the `QEMU` nvdimm feature to provide a
+memory mapped virtual device that can be used to DAX map the virtual machine's
+root filesystem into the guest space.


s/guest space/guest memory address space/

jodh-intel · 2017-09-15T14:51:50Z

docs/architecture/architecture.md

+  host to be demand loaded using page faults, rather than having to make requests
+  via a virtualised device (causing expensive VM exits/hypercalls), thus providing
+a speed optimisation.
+- Utilising MAP_SHARED shared memory on the host allows the host to efficiently


s/MAP_SHARED/MAP_SHARED/

jodh-intel · 2017-09-15T14:52:20Z

docs/architecture/architecture.md

+
+Clear Containers utilises the Linux kernel DAX [(Direct Access filesystem)](https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/dax.txt)
+feature to efficiently map some host side files into the guest VM space.
+In particular, Clear Containers uses the `QEMU` nvdimm feature to provide a


Remove backticks around QEMU.

grahamwhaley · 2017-09-15T15:00:00Z

I think we ( @iphutch ) may have mentioned the high-level-overview.png is not referenced anywhere - to note, it was referenced from the proxy README.md in the cc2.x repo, and thus I believe has been un-necessarily carried over (and there is a copy over in the clearcontainers/proxy/docs dir I think). Yes, it can be dropped from here.

grahamwhaley · 2017-09-15T15:34:47Z

docs/architecture/architecture.md

+
+The `delete` code path differs significantly between having to delete one container
+inside a pod (as is typical in Docker) and having to delete an entire pod (which
+is unique to Kubernetes). In the former case, `cc-runtime` will only send a


rather than 'is unique to' I might say 'such as from'.

grahamwhaley · 2017-09-15T15:35:20Z

docs/architecture/architecture.md

+The `delete` code path differs significantly between having to delete one container
+inside a pod (as is typical in Docker) and having to delete an entire pod (which
+is unique to Kubernetes). In the former case, `cc-runtime` will only send a
+`SIGKILL` signal to the container process.  In the latter case, the whole thing


s/whole thing/whole pod/ or something similar - 'all components' or something

further work to do: -upload text used to create UMLs -add UML for other key OCI commands -add crio/conmon/k8s diagram for parity w/ docker -Describe KSM throttling feature in the proxy or its own section once the feature lands. Contributions-by: Graham Whaley <graham.whaley@intel.com> Contributions-by: James O. D. Hunt <james.o.hunt@intel.com> Contributions-by: Samuel Ortiz <sameo@linux.intel.com> Contributions-by: Archana Shinde <archana.m.shinde@intel.com> Contributions-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> Signed-off-by: Eric Ernst <eric.ernst@intel.com>

ericwadams · 2017-09-20T16:48:35Z

I recommend adding a Kubernetes overall architecture guide including the shim/proxy/agent and how it works with crio.

To use the filepath.Join() instead of the simple string append method to form the file path, otherwise it will lose the "/" between the two parts. Fixes clearcontainers#543. Signed-off-by: Fupan Li <lifupan@gmail.com>

egernst added in progress CC 3.0 do-not-merge domain:documentation release gating labels Sep 12, 2017

egernst commented Sep 12, 2017

View reviewed changes

jodh-intel reviewed Sep 12, 2017

View reviewed changes

grahamwhaley reviewed Sep 12, 2017

View reviewed changes

sameo reviewed Sep 12, 2017

View reviewed changes

mcastelino reviewed Sep 12, 2017

View reviewed changes

egernst requested a review from iphutch September 14, 2017 15:35

iphutch suggested changes Sep 14, 2017

View reviewed changes

egernst commented Sep 14, 2017

View reviewed changes

sameo reviewed Sep 14, 2017

View reviewed changes

iphutch suggested changes Sep 14, 2017

View reviewed changes

iphutch approved these changes Sep 14, 2017

View reviewed changes

jodh-intel reviewed Sep 15, 2017

View reviewed changes

grahamwhaley approved these changes Sep 15, 2017

View reviewed changes

clearcontainers deleted a comment from iphutch Sep 15, 2017

jodh-intel reviewed Sep 15, 2017

View reviewed changes

egernst removed the do-not-merge label Sep 15, 2017

grahamwhaley reviewed Sep 15, 2017

View reviewed changes

mcastelino merged commit c9ccbb7 into clearcontainers:master Sep 15, 2017

amyleeland removed in progress release gating labels Sep 15, 2017

jodh-intel mentioned this pull request Apr 18, 2018

libreoffice architecture doc is corrupt #1098

Open


		#### Hypervisor

		Clear Containers use [KVM](http://www.linux-kvm.org/page/Main_Page)/[QEMU](http://www.qemu-project.org/) to


		![QEMU/KVM](qemu.png)

		fixme - discuss different QEMUs - lite, q35, pc etc.


		Here we will describe how `cc-runtime` handles the most important OCI commands.

		##### `create`


		## DAX

		Clear Containers utilises the Linux kernel DAX (Direct Access filesystem)


		![Docker and Clear Containers](arch-images/docker-cc.png)

		`cc-runtime` creates a QEMU\*/KVM virtual machine for each container the Docker


		## Shim

		A container process reaper, such as Docker's `containerd-shim` or crio's `conmon`,

Conversation

egernst commented Sep 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clearcontainersbot commented Sep 12, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egernst commented Sep 12, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egernst commented Sep 12, 2017 •

edited

Loading