artodeto's blog about coding, politics and the world

First talk finished :-). No proofreading done so far. Next talk will start in a few seconds.

Its All About Scale

• size
• features
• non-functional requirements
• Language agnostic - language you are using doesn't matter

Background Tasks

User can't wait and server can't handle it

• sending mail
• calling 3rd party APIs / services
• converting media (image, video, audio)
• updating caches

Direct Approach

• ignoreuserabort()
• forking process like using exec()

Don't do that since your are losing control

Slightly Less Bad Idea

• Write jobs into sql database
• have some workers that poll the database for jobs

Don't use it

Queus To The Rescue

• write jobs into message queue
• have some workers for the queue
• many services available (ZeroMQ, RabbitMQ, Redis Pub/Sub, Amazon SQS ...)

Good idea but lot of work and hard to get right

Beanstalkd

• create a job
• process a job

RQ

• simple job queue in python
• backed by redis
• enqeueu function call
• run worker how stores result in database

Celery

• python
• multiple brokers
• primarily python with clients for ruby and php
• highly available (HA)
• fast
• flexible
• monitoring
• workflows
• time and rate limits
• scheduling
• autoscaling

Search

Like query

• no lingustic suport
• no ranking
• not indexed (mysql has fulltext index and postgresql)

Fulltext Search Engines - Solr Vs. Elastic Search

• based on lucene
• full linguistic support
• faceted search
• result highlighting
• HTTP API
• similar but different
• easy to set up
• clients available like PHP: solarium, symfony component or PHP Elastica
• integrate search into framework (map objects into documents and back) or handeling updates and deletes

NoSQL

• key-value stores
• column based
• document stores
• graph

Errors

• don't let it happen
• send an email (overload mailboxes)
• log errors
  • php symfony provides a component that you log only all messages if error occures (so only info logging if something happens)
  • properly configure logging
  • use event aggregator
    • sentry
    • errbit

Sentry (event aggregator)

• started as OSS
• now available as SaSS
• udp
• written in django
• php client available

Errbit (event aggretator)

• still OSS

Deployment

• ftp
• git pull / app server by ssh
• but you want a automated deployment like fab deploy
• automated deployment with webistrano (just a single button)

Wrap Up

• use background tasks
• use full text search
• try to use nosql where needed
• do error logging
• think about deployment