Skip to content

Introduction Into Docker and Continous Delivery with Docker - Docker Usergroup Hamburg 09.06.2014

Docker UserGroup Hamburg

Introduction Docker

General

By Johannes Ziemke from docker

What is Docker

  • version 1.0 released a few hours ago - api stable (a bit)
  • written in go
  • build, packs and ships applications as lightweight containers
  • build once an run almost everywhere (since kernel 3.8.*)
  • rewrite of docCloud PaaS code

Benefits

  • start always from clean slate
  • spawn up complete test infrastructures in seconds
  • run your code against multiple versions easily
  • easy up setup of software environment
  • distribute complex setups as self-contained containers
  • build own infrastructure
  • easy ups deployment per day
  • scaling and mantaining of infrastructure (state convergence, manage everything)
  • helps you to solve managing complexity
    • similar problems
      • modules, classes, plugins
      • human communication: named concepts like car, cat or container
      • shipping goods: intermodal containers
    • solution: create an abstraction
      • does not matter what is in the container nor where it runs
      • clear separation of concerns (reduce communication between developers and ops)
      • developer can take care about used libraries, packagemanager, ...
      • op can take care about the system (alos resource planning and monitoring)
  • running commandos on top of:
    • immutable, shipable, layered images (defined by dockerfile, built by builder and pushed/pulled to registry)
    • copy-on-write storage on top (provides writeable layer on top of read only images to persists changes done by running container)
      • aufs
      • btrfs
      • devicemapper
    • isolated environment
      • using kernal features
        • namespaces (isolation by scoping, available: pid, mnt, net, uts, jpc, user)
        • cgroups (limit, accound and isolated general devices, cou, memory, ...)
        • future (solaris zones ...)
    • via RESTish API
    • missing pieces
      • docker can not address service discovery or dynamic scheduling
      • projects to close the gap
        • openstack
        • coreOs
        • mesos + marathon
        • flynn.io

Demotime

  • --privileged - can break out of container but control host
  • images are read only and creating a container on each start
  • docker commit - creates new image out of running container
  • use boot2docker

Continous Delivery with Docker

By Tobias Schwab from Dynport GmbH

Philosophie

  • continous delivery
  • canarien releases
  • "never touch running system", create a new one when something has to change
  • immutable releases (once created, never change it)

Why They used Docker

  • privacy concerns, aws not an option
  • hoster they could not pick
  • unreliable hoster API
  • flash based structure management
  • limited capacity
  • they are the biggest customer

Learnings

  • image based deployments -> use dockerfiles (do not add dockerfile generators on top)
  • use syslog and rsyslog to log out of the container
  • >>CMD ["/sbin/init"]<< to start debian based image
  • caching
  • configuration management
    • store configuration in environment
    • dependency injected with start of container (use same container for testing, stageing, testing ...)
    • logfiles and data outside of the container
  • do not
    • full blown vm (best case, one service per container)
    • ssh deamon inside containers
    • syslog deamon inside containters (sometimes needed)
    • user management (everything can run as root)
    • chef/puppet makes caching useles
    • rely on external services (github, gem page)
  • build management tools
    • bundler, pip, carton, composer, ...
    • problem is, they are slow on "clean slate"
    • ways to solve
      • add mainifest (like composer.lock) before code and use cached build files
      • pre-bundles base images
  • multi host
    • image distribution via docker registry
    • weighted load balancing via HAProxy
    • SSL termination via HAProxy (container exists by Johannes Ziemke)
  • load balancing
    • HAProxy
    • pool configuration stored in redis/etcd
    • configuration update
      • created
      • uploaded via ssh to host
      • verify configuration
      • replace configuration (old with new)
      • reload configuration
  • logging
    • putting container id, revision of image (host, code, request) via rsyslog out of the container
  • metrics
    • openTSDB
      • distributed, scalable time series database
      • hbase
      • tags/diminsions
      • from syslog via udp
      • rickshaw.js for graphs
      • compare status cudes, counts and times between actions of two revisions
  • benefits (reduces)
    • external dependencies
    • "did work on my machine"
    • unused cpu cycles
    • number of hosts
    • feedback times (number of deploys per day)
    • time to get new host online
    • hosts are more flexible
    • controlled revision change (controlled on each host)
    • faster build

Translate to de es fr it pt ja

Reusability Of Symfony Component - Mindworks

By Oliver
Mindworks
Jarrestraße 42A
Take a look to knpbundles.

Hurdles

  • problem is solved in other language
  • problem is solved in other licence
  • can be solved by design patterns (abstract solution)
  • write a good bundle/component that is general enough takes much more time

Anti-Pattern

Reuseage by copy and paste.

  • decreases maintainability
  • decreases testability
  • violates dry principle

Scenario

  • community web application
  • users seek other users by interests
  • how to input data to easy up matching?

What Do We Need?

  • form widget (js and css based)
  • data transformer (converts format from storage to human readable presentation and back)
  • form type (provides name for form widget, called in FormTypes, gets TagManager injected and defines DataTransformer)
  • Entity Trait (avoid copy and paste)
  • Taggable Subscriber
  • Make it reusable (dedicated bundle, created composer project)
  • Use JavaScript and CSS via "composer component" (see web/bundles, RobLoach/component-installer)

Live Demo

Available on github

Translate to de es fr it pt ja

High-Performance Websites - PHPUG and WebPerformanceUG

Arne Blankerts, Stefan Priebsch - High-Performance Websites

This talk contains information about architecture.

Classical Architecture

When have done this for quite a while.

Browser->WebServer->PHP-Database

This works best in the past. General solution was to add more hardware and that leads to more errors.

  • monolithic architecture
  • normalized data (in database)
  • pull-principle (browser is build per request)
  • full page cache is bad
  • edge site include (for varnish/ESI) is also not the best solution

We are fast when:

  • no real workload per request
  • everything from memory
  • denomalized data
  • snippets

How Often Does Data Change (From Rare To Often)?

Based on the example of a shop system.

  • catalog
  • product information
  • price
  • availability

Who Changes Data?

  • editor
  • product manager
  • receipt of goods / outgoing goods

New Architecture (CQRS-Architecture)

  • images and static content is deliverd by static content webserver
  • only dynamic part is done by php
  • snippets are deliverd, not full pages

PHP->Key-Value-Store->Backend-Process->Database
or
PHP->Web-Service->Database

Benefit of key-value-store (for example redis) is, that the database itself can die and the only thing that happens is, that the entries are becoming obsolete.

But What About Filternavigation?

Use a searchengine that returns simple a collection of product keys. Use this product keys and ask the key value store to fetch product data.

But What About Personalization?

  • use snippets with variables and default values
  • fetch this snippet in the key value store
  • thanks to "time to death" feature provided by many key-value-stores, you can easily define "special offers" per day and so on

Nice meetup, incredible how many people are attending already.

Translate to de es fr it pt ja

Define The Unspeakable - PHPUG and WebPerformanceUG

2013-09-10 Hamburg, Otto, Loft 06

Judith Andresen - Make Mistakes The Right Way

Define The Unspeakable

Result

  • establishment
  • project
  • uniform

moral concepts (negotiable)

  • hanseatic
  • freaked out / nerdic
  • cutting edge

basic assumption

They are teached/ already at school level (errors are bad, don't do something wrong).

  • what can i do
  • who am i able to add critic to
  • natur
  • fellow men
  • industry

From the school already, we get trained to:

  • it is bad to say "that was my fault"
  • shift the know error to others
  • there is one majority out there who decides if this is a bad error
  • be perfect
  • always have a fallback (not to lose the face)

Define A Good Error Culture

This goals can be achived by exemplify the following steps (best by upper management but important for all).

  • "good" is enough
  • no fear of mistakes
  • many/fast update cycles (don't try to do big jumps, doing small steps results in small mistakes)
  • straightforwardness
  • to take the blame is not important
  • do not repeat your mistakes
  • do small and usefull steps (validate if your change contains a "only if")
  • use and understand the Twelve Principles of Agile Software
  • celebrate reached goals

Slides are available on judiths page. Again a nice talk and incredible what kind of easy mistakes we are doing in our regular live.

Translate to de es fr it pt ja

meetup - attraktor - selinux

Just arrived at home from back to hack selinux. I tried my best to keep up the speed of slides. Nevertheless, there are some gaps in between.

Big thanks to the presenter. I turned my fear into anticipation for using it!

History

  • 1976 LaPadula paper released
  • DTMach as construction idea released
  • Based on FLASK, first patches to linux kernel
  • 2002 implemented as linux security module (LSM)
  • Since 2003 part of the mainline kernel
  • Fedora core was first distribution (shipped with policy)
  • Since 2007 shipped with RHEL5 EAL4+

Bell LaPadula

  • MAC
  • TE
  • RBAC
  • No read up, no write down

Features

  • Extends DAC (directory access control)
  • mode at runtime changeable (enforcing vs permissive)
  • flexible policy
  • AVC (access vector cache)
  • root is not god anymore
  • process running in own security domain

Discretionary Access Control

  • classical security concept
  • object based access control
  • unflexible
  • rights per user
  • user can ship rights
  • uid can be changed by using suid

MAC

  • access control by using policy

TE - Type Enforcement

  • all resources have a typ/domain
  • whitelisting ruleset

Security Context

  • securityarea
  • object (user:role:domain:level)
  • subject

Policy

  • heart of selinux
  • common policis are targeted, strict and MLS
  • defines rules for access from subject of object
  • defines domain transition of a subject (when can a subject change its domain)

Policies

Targeted

  • default policy
  • contains a understandable complexity
  • well choosen processes running in own domain
  • you can tweak the policy by using booleans
  • this policy should be enough for daily work
  • if you use the filesystem hirachy standard you are quite save
  • Unknown software is stored in context "unconfined"

Multi Level Security (mls)

  • everything is running under selinux policies
  • tweakable by booleans
  • only for high risk security aspects

Booleans

  • enables tweaking the policy without reload
  • no knowledge about policy needed
  • booleans can set permanent
  • semanage boolean -l lists a lot of informations

Userlandtools

  • -Z is selinux switch (ls -Z, netstat -Z)
  • cp, mkdir and so on are linked against selinux to support setting needed bytes
  • getenforec/setenforce - switching selinux mode
  • restorecon/fixfiles/chcon - changing context of subjects
  • sestatus - status
  • avcstat - status about avc
  • libselinux - the selinux
  • libsepol - all binaries linked against

Policy Management

  • setsebool/getsebool - show and set of selinux booleans
  • semanage - policy management
  • semodule -

Policy Development

  • audit2allow - builds selinux rules based on audit log
  • checkmodule - transforms rulse into binary
  • semodule_package -
  • audit2why - analyse selinux

Linux Audit Framework (LAF)

  • should be used for logging

Backup

  • you are loosing the context if your backup software does
  • tar and rsync supports :-)

Translate to de es fr it pt ja

usergroup - meetup - phpug - hamburg - 130409

betahaus hamburg Ruben Explains what betahaus is about - betahaus can be your office if you have none or if you want to have a second one - betahaus could be used for freelancer as well as for persons in the it area - it is more then a mackerspace

Ole Michaelis What is wrong with php - by github charts, php is the sixth most used language (why not number one? :-D) - global players like "the fruit company", flickr, yahoo or "the other big blue" are using php - search for "php sample codes" and investigate the first results, can you see whats wrong (why they are the "best fitting" results :-()?

can we fix it? - yes we can! - join the community - fix core bugs - create usefull tools or use them (like behat, mink, composer, phpspec, symfony, phpphp or php the right way) - tell "them" that php is not php3 anymore

Judith Andresen money value of test driven (php) development why we should test - no manager can motivate with money only - you can only motivate by surrounding

additional informations - use atam to transform developer language into non technical (product owner) language - use risc management (bot team but company based to remove interface problems) - try tu use timeboxing to reduce switch times

books - Drive: The Surprising Truth About What Motivates Us by Daniel H. Pink - Softwarequalität in PHP-Projekten by Sebastian Bergmann and Stefan Priebsch

Translate to de es fr it pt ja

usergroup - meetup - phpug - hamburg - 130320

Grunar+Jahr Condition driven quality management / Test Pattern Nils Langner and Torsten Franz - three developers to one qa - what is quality? Reaching defined goals - Balanced score card - define threshold of errors (with which bugs is it still releasable) - also check ISO/IEC 9126 - differ between technical qa and functional qa - tools: jmeter, phpunit, bamboo, phplint, phpcs - reviews/analyse: code, architecture, flow of data - explore with teams how to test the application (try to break it) - github: livetests

couchDB jan lehnardt

  • example by using node.js proxy and two couchDB's with replication.
  • changes stream can be used to implement event based jobs, pull services
  • PouchDB is couchDB for mobils, based on javascript ->stores data in client as long as couchDB backend is not available
  • TouchDB for smartphones (ios, android)
  • BigCouch implements googles dynamo implementation
  • You can restrict document types after implementation (add/cover with a schema)
  • Queries in couchDB are json queries named views ->map functions are equals to selects ->reduce functions are calculating stuff like sum, count, stats or own functions. Can be extended (e.g. lucene)
  • CouchDB is written in ERLANG

    Pretty cool talks. Especially Jan lights my fire to play with couchDb :-D.

    Translate to de es fr it pt ja

usergroup - meetup - webPerformance - hamburg 130220

Attraktor e.v. mexikoring 21, hamburg

Soundcloud - Arbo Von Monkiewitsch

html5 audio playback widget - measuring for velocity

measure first

how to?

  • synthetic user monitoring (by phantom.js)
    • repeaable
    • doesn't show real user experience
    • good for benchmarking
  • real user monitoring (google analytics)
    • shows real user experience
    • change of user behavior impacts repeateability (e.g. side slow in asia but fast in euro)
    • impact of embedding site

what?

  • http ping
  • fully page load
  • include all transactions
  • follow K.I.S.S., just render the page

define your goals

  • what is fast
  • don't define random goals on yourself
  • public benchmarks are good as orientation (for e.g. set up a benchmark for youtube or similar)
  • response time
  • don't simple trust your metrics

best practices

  • httparchive, pagespeed, yslow (100 - 200 ms as full page loading speed is good enough)
  • compress each connection (use varnishs or something similar for uncompressable content)
  • availability fallback -> dc
  • performance - put api in cdn (also for http or https handshake)
  • consitency - load balanced cdns

rap it up

  • measure first
  • keep it simple
  • team autonomy

phantomJs - Raj Dudi

http://rajdudi.com

what it is - created by ariya hidayat

  • can be integrated into jenkins
  • automation and headless testing
  • can be used for metrics measurement
  • render and dump client side html
  • current version is 1.8
  • headless (Qt)Webkit/Browser
  • javascript driven
  • ideal for
    • headless website testing -> but it is not a test framework
    • screen capture
    • page automation
    • network monitoring
  • projects
    • casperjs
    • ghostbusters
    • poltergeist

code evaluation

  • evaluate(function, arg1, arg2, ...) {object}
  • returns simple objects but no functions and closures
  • execution is sandboxed
  • dom scripting and css selector works
  • use onConsoleMessage callback to receive the console messages

get it from

  • phantomjs.org
  • github.com/ariya/phantomjs
  • https://aur.archlinux.org/packages/phantomjs/

examples

https://github.com/ariya/phantomjs/tree/master/examples

hello world

console.log("Hello world"); phantom.exit();

screen capture

  • Page.render to render a screenshot
  • supports ong, jpeg, gif and pdf
  • options
    • page.clipRect = { top: 14, left: 3, width: 400, height: 300 };
    • page.paperSize = { width: '200px', height: '300px' } //for pdf output e.g., also supports borders and much more
    • page.zoomFactor = 0.25 // 1 is default
  • projects based on phantomjs
    • screenr
    • mediaqueri
    • chromanope
  • webpage api
    • var page = require('webpage').create();
    • page.open(url, callback) {void}
    • page.close() {void}
    • goBack(), goForward(), title, url
    • settings{object} //enable javascript, load images, username or password, websecurity
    • page.includeJs(url, callback) {void}
    • page.injectJs(filename) {boolean}
  • webpage callbacks
    • onLoadStarted
    • onLoadFinished
    • onAlert
    • onCallback //experimental
    • onClosing
    • onConfirm
    • onConsoleMessage
    • onError
  • system API
    • var system = require('system');
    • ...
  • filesstem api
    • var fs = require('fs);
    • separator {string}
    • workingDirectory
    • ...

troubleshooting

  • allows network sniffing
    • onResourceRequested
    • onResoureceReceived
    • TLS and SSL required for encrypted data - are they enabled?
  • error handling

cons

  • uses qtWebKit (not used by chrome, safari and so one)
  • asynchronous api - difficult to handle for many
  • not supported
    • plugins
    • webGL
    • video and audio
    • css 3-D
    • xpath

What could i add? Thanks to the talks, yes very much. Also thanks to the attraktor, you are doing it right!

Translate to de es fr it pt ja