marlou knowledge base

# Génie Logiciel

# principes

Découper un problème en plusieurs problèmes plus simples

Low coupling, high cohesion, maintainability, extensibility, reusability.

  • Loi de Hofstadter

Il faut toujours plus de temps que prévu, même en tenant compte de la Loi de Hofstadter.

  • Donald Knuth à propos de l'optimisation prématurée

Premature optimization is the root of all evil

Ajouter des personnes à un projet en retard accroît son retard

Adding manpower to a late software project makes it later

Neuf femmes ne font pas un enfant en un mois

80 % des effets sont le produit de 20 % des causes

Les organisations à l’origine des architectures doivent refléter le système en lui-même.

organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations

“The structure of a problem reflects the structure of the organization that created it.” Bill Corcoran’s version of Conway’s Law

Pour un projet innovant, il faut multiplier l’estimation initiale des coûts par 3,14.

  • Loi de Parkinson

Plus on a de temps pour réaliser une tâche, plus cette tâche prend du temps.

  • Loi de Murphy

Toute chose prend plus de temps qu'on ne l'avait prévu

  • Loi d'Illich

Au delà d'un certain seuil de travail, l'efficacité décroit

  • Loi de Carson

Faire un travail de façon continue prend moins de temps que de le faire en plusieurs fois

  • Loi de Fraisse

1 heure n'est pas toujours égale à 1 heure

Ne parlez qu'à vos amis immédiats

La notion fondamentale est qu'un objet devrait faire aussi peu d'hypothèses que possible à propos de la structure de quoi que ce soit d'autre, y compris ses propres sous-composants.

Exemple concret par Misko Hevery

  • YAGNI (You aren't gonna need it)

Always implement things when you actually need them, never when you just foresee that you need them. (Ron Jeffries)

You aren't gonna need it : en.wikipedia.org

# Inversion of Control

# normes

# pratiques

# monorepo

Plusieurs solutions de gestion des repos/projets :

  • 1 repo = 1 projet
  • 1 repo = 1 micro-service
  • monorepo (tout le SI est contenu dans un seul repo)

défaut des 2 premiers : Dependency Hell.

avantages du monorepo :

  • version unifiée du code
  • unique source de vérité
  • réutilisation intensive
  • gestion des dépendances simplifiées (tous les projets ont les mêmes versions)
  • collaboration inter-équipes améliorées (mais nécessite meilleure communication)
  • frontières des équipes et code ownership flexibles (résoud le problème levé par la loi de Conway)

inconvénients du monorep :

  • difficulté à faire des montées de version des dépendances (puisque tout le monde est impacté)
  • investissement très lourd du tooling pour gérer les projets
  • sécurité du code (tout le monde a tout le code)

impact Git :

  • généralement on fait du feature branching (rend difficile la collaboration)
  • monorepo impose le trunk based development (commit quotidiens sur le trunk + feature toggle)

Le monorepo est la pratique actuelle des géants comme Google et Facebook.

Exemple Google (janvier 2015) :

  • 1 repo (exclus : Chrome, Android, ...)
  • 25K ingés
  • 1 milliard de fichiers
  • 35 millions de commits
  • 9 millions de fichiers sources
  • 2 milliards de lignes de code

Pourquoi votre entreprise devrait stocker l'ensemble de son code dans un seul repo (Thierry Abaléa) Devoxx2017

Why Google Stores Billions of Lines of Code in a Single Repository : Rachel Potvin@Google 20150914

# continuous integration

wikipedia

Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies to a shared mainline several times a day.

# continuous delivery

wikipedia

Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

Continuous Delivery - Martin Fowler - XConf - 20150131

A developer needs to do some change to the software, and another, and another, and it comes to a new version.

The question is how good are the changes ?

Continuous Delivery says : We need to prove the changes are OK. We begin to throw more and more tests (units, integration, functionnals, etc ...)

Until will be confident it is good.

And it goes through environments closer and closer to production.

Until you are confident it will be good when goin to production.

Continuous Delivery is : You can deploy a change to production (but you may choose not to because it is a business decision)

Continuous Deployment is : You do deploy a change to production

Needed ingredients :

  • Automating everything (build, deploy, tests, even provisioning your machines)
  • Configuration management (you have to be able to checkout any version and instantly put it into a dedicated environment) Everything must be in the configuration management
  • DevOps Culture (Dev and Ops must collaborate. No separate walls.)
  • Continuous Integration (Every developer integrates with everybody elses work frequently)
    • Everyboy commits everyday to mainline (trunk, master)
    • Tests give enough confidence to say it's OK
    • If it fails, you fix it immediatly

Benefits :

  • Lowering risks (the smaller the change, the smaller the risk)
  • Definition Of Done (live in prod) : Real progress
  • Learning from frequent delivery from users feedback

Not easy to make it, can takes months to put in place.

Probably one of the most critical practice.

Continuous Delivery - Jez Humble - 2012

Jez Humble RETEX about a case :

  • top management decide to go agile
  • they hire scrum master
  • they send team into 2 days formation
  • and now they should be agile

The organisation is now something like that :

  • Customer push user stories to the "agile dev team".
  • Dev team analyse, design, dev them test and makes showcases. They iterates several sprints inside that box
  • The result is pushed to a centralized QA team
  • They iterates with dev team to fix things
  • Then the result is pushed to IT Operations team who deploy.

The 3 lasts steps are called "the last mile".

The end user get any value just at the end.

Flickr was the first organization to deploy several times a day.

They got bought buy Yahoo. Yahoo engineers were stonished buy the deployment frequency.

They compare the service uptime with their core services and found Flickr uptime was far better then any of them.

Need to kill a myth : Deploy frequently is not killing stability. It is the opposite.

Why do we care about releasing frequently ?

  • building the right thing (far more then 50% of functionnality in software are rarely used)
    • Scientific method : create hypothesis, deliver MVP, get feedback, repeat. This is lead time.
    • the main work is to optimize a software delivery process around this loop.
  • reduce risk of release
    • release frequently means delivering smaller batch size
    • when Flickr release a change it is about 50 to 100 LoC. Easy to debug a problem.
    • MTBF versus MTRS (Mean Time Before Failure vs Mean Time to Restore Service)
  • real project progress (done means it works on my machine, done done means it works in production)
    • if you want to reduce cycle times (and reduce lead time) you can
      • increase capacity (more peoples) but see mythical man month (it's trying to speed up pregnancy by adding 8 women)
      • reduce demand (reduce the scope)
      • the easiest way is to reduce batch size (this is why Kanban is important)

Continuous Delivery

  • software always production ready. Keeping the system working is more important then working on other features
  • releases tied to business needs, not operational constraints
  • production-ready software (fast, automated feedback on the production readiness of your applicatons every time there is a change - to code, infra or config)
  • 3 pillars :
    • automation
    • patterns and practices
    • collaboration (ux, tests, business, ...)
  • a single command should install everything i need to run the system on a virgin machine
  • continuous integration (merge commit to the trunk every day + build and test each revision)
  • W. Edwards Deming : "Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place."
    • testing is not something we do after development (we should do all the time)
    • QA is not responsable for quality. Their job is to represent the user.
    • See the Marick Test Matrix
  • deployment pipeline (an automated impl of your system's build, deploy, test, release process)
    • visibility
    • feedback
    • control
    • every build is a release candidate
    • if you fear to deploy in production by just pushing a button, it means you need to fix your validation process
  • release != deployment (release with feature toggle, so it is already deployed, but not released to end user)
  • don't implements continuous delivery in one batch, do it with continuous improvement

# continuous deployment

wikipedia

Continuous deployment means that every change is automatically deployed to production.

# continuous delivery vs continuous deployment

Continuous Delivery Vs. Continuous Deployment: What's the Diff? - puppet.com/blog - 20130830

They are the same except in continuous delivery the prod deploy is manual. Everything else is automated.

In continuous deployment everything is automated from commit to prod.

# continuous delivery / deployment and semver

Semantic Versioning with Continuous Deployment by Mark Seemann : blog.ploeh.dk

# feature flipping

feature-flipping : blog.octo.com

feature flags explained : featureflags.io (very good link, with lots of use case, eg : early access)

# drawback

Settings are evil - MPJ's Musings - FunFunFunction #62

TLDR : Feature toggle generate combinatorial explosion of test cases -> lower quality

Martin Fowler deal with this question in his feature toggle article :

A common question we hear about feature toggles concerns testing - does using feature toggles mean a combinatorial explosion of tests? In general there's no need to test all combinations of features. For release toggles it's usually sufficient to run two combinations :

  • all the toggles on that are expected to be on in the next release
  • all toggles on

This is pretty much the same as what you need to do with feature branches if you want to find any integration bugs.

It's very important to retire release toggles once the pending features have bedded down in production. This involves removing the definitions on the configuration file and all the code that uses them. Otherwise you will get a pile of toggles that nobody can remember how to use. In one memorable example I heard of, it required making a special recompilation of the linux kernel to handle enough command line switches.

# feature branches vs feature toggles

Feature branches vs. feature toggles : fournines.wordpress.com : 20111120

The biggest mistake you can make when deciding on how to use branches is trying to define "one branching strategy to rule them all".

Feature toggle should not be systematicly used. It should be only if the team experience pain merges.

Features branches cons :

  • the feature branch is not merged in the mainline, so there is no CI on it
  • concurrent feature branch works can lead to big conflicts when merging. The last team to merge will have to handle all the issues.

Conflicts could symptomatic of underlying problems :

  • poor design
  • no cohesive view of the domain (no or bad modelization)
  • or multiple logical views of the domain
  • refactor-rage (a dev doing lot's of cow boy refactoring)

Feature toggle can be impl at build time or at runtime. Build time means you need several CI pipeline, run time increase increase code complexity.

Feature toggles are certainly an interesting concept, and seem to mesh well with the principle of building composable applications. However, they are not without their pitfalls, so if you are thinking about trying them out, make sure that they are the most appropriate solution to the actual problem that you’re trying to solve.

# méthodes

# waterfall

# cycle en V

# agiles

# auteurs

a été titulaire de la chaire de génie logiciel du Cnam. A dirigé les développements logiciels des ordinateurs Bull DPS7. Consultant en systèmes d’information auprès de grandes entreprises et administrations. Auteur de nombreux ouvrages sur le génie logiciel.

inventeur du datagramme et concepteur du premier réseau à commutation de paquets, à la base de ce qui deviendra Internet. Lauréat du Queen Elizabeth Prize for Engineering en 2013. Militant du multilinguisme dans l’espace numérique et d’une nouvelle gouvernance d’Internet

a été titulaire de la chaire des techniques fondamentales de l'informatique du Cnam, responsable de recherche à l'Imag, Bull et l'Ecole Polytechnique. Pionnier en France des architectures de circuits intégrés, en particulier sur leur preuve formelle et leur synchronisation interne.

Précurseur à l’INRIA de la notion de routeur de réseau, puis du poste de travail intelligent, en dirigeant le projet Kayak. Il a ensuite exercé des responsabilités de direction dans l’Industrie (Bull, EDS, Sabre, Prologue), et maintenant de conseil en management.

# histoire

  • Software crisis Fin des années 70, les ordinateurs sont montés en puissance, donc permettent le développement de programmes plus puissants et plus complexe. Dégradation massive de la qualité du à la complexité. C'est le début des recherches en génie logiciel pour définir les bonnes pratiques du développement logiciel (gestion qualité, méthodes, invention programmation procédurale puis objet).
  • bulle internet Fin 90 début 2000 le crash boursier entraine de nombreux licenciements de développeurs suite aux réductions du couts et aux sociétés déposants le bilan. Les développeurs ont alors tendance à se transformer en oui:oui pour éviter de perdre leur emploi, entrainant une dégradation massive de la qualité des productions. Ce phénomène associé à l'arrivée de nombreux jeunes développeurs sur le marché du travail quelques années après sont les deux phénomènes responsables de l'immaturité de l'industrie du développement logiciel dénoncée par Robert C. Martin (cf Martin pour une explication plus détaillée dans The Future of Programming)
  • Jeff Bezos API Mandate explained by Google employee
  1. All teams will henceforth expose their data and functionality through service interfaces.
  2. Teams must communicate with each other through these interfaces.
  3. There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
  4. It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.
  5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
  6. Anyone who doesn't do this will be fired.

Bret Victor give a presentation as we were in late sixties and imagine the future (our present)

There is always massive resistance to change.

  • The 4 ideas the author wants to talk about (about recent researchs) :

    • coding (telling the computer what to do) -> direct manipulation of data (manipulation of data structure implicitly build the program)
    • procedures (list of things to do) -> goals and constraints (telling the computer what you want, not what to do)
    • text dump (we use text file as source code) -> spatial representations (hooking videos displays to computers)
    • sequential (the control flow) -> concurrent (soon there will be material really adapted to concurrency)

  • coding -> direct manipulation of data

    • sketchpad in 1962. Creating a rectangle not by programming it in source code but by manipulating the data and applying constraints to them.

  • procedures -> goals and constraints

    • planner in 1969, prolog in 1972
    • pattern matching : snobol in 1962, regular expressions in 1967
    • joke about internet : programs in a shared network should figure out by themselves how to talk to each others (the goal). They should not rely on API (imperatively programmed by humans and subject to breaking changes).

  • text dump -> spatial representations

    • NLS in 1968 (presenting data in different views, a list then a graph)
    • Grail in 1968 (programming with flow charts)
    • smalltalk in 1972 (text source code but organized spatially)

  • sequential -> concurrent

    • today program are sequence of instruction. The future should be parallel programming.
    • the reason we write sequential programs is because of the hardware
    • hardware is built upon the Von Neumann architecture : The CPU is fetching words from the memory.
    • the sequential programming model makes sense when you have only one CPU, a CPU can process only one thing at a time.
    • one characteristic of Von Neumann architecture is that most of the memory is idle most of the time.
    • it makes sense with old hardware architecture (tubes, mecanisms, etc ...)
    • 1971, 4004 microprocessor from Intel. CPU and memory are only be made by transistors on silicon.
    • if you want to maximize the usage, you should start looking to massively parallel processors array
    • it scales
    • this is the architecture of the future instead Intel stranghold the market by pushing his architecture 😉
    • how do we program on that architecture ?
    • today we program with threads and locks and this is really hard when working with the same shared memory
    • 1973 : The actor model, inspired by physics

  • it would be a shame if in 40 years we will still programming in procedures with text files in sequential model

  • the worst would be if that ideas from 60/70 have been forgotten

  • the worst would be if the next programmers generations grows up without being exposed to theses ideas

  • they will think that programming model is solved and they grow up with dogma

  • The most dangerous thought you can have as a creative person is to think you know what you're doing.

# courants

# Software Craftmanship

Agile : Are we building the right thing ?

Craftsmanship : Are we building the thing right ?

from Software Craftsmanship by Sandro Mancuso @ Devoxx France 20??

# Conférences

# exemples concrets

odyssée du Continuous Delivery Diego Lemos David Caramelo (Nicolas Bourgeois)

Présentation Devoxx2016 sur le passage du code legacy au continuous delivery à la Société Générale. Juste génial.

SOAT Agile Day - Keynote de Romain Niccoli - Etre agile dans un contexte de forte croissance

# architecture

Architectural patterns - www.dossier-andreas.net

  • Model-View-Controller
  • Presentation-Abstraction-Control
  • Pipe-And-Filter
  • Layered Systems
  • Microkernel
  • Client-Server and N-Tier Systems
  • Repository
  • Blackboard
  • Finite State Machine
  • Process Control
  • Multi Agent System
  • Broker a.k.a. Service Oriented Architecture
  • Master-Slave
  • Interpreter a.k.a. Virtual Machine
  • Hub-And-Spoke
  • Message Bus a.k.a. Event Bus
  • Structural Model
  • Ports-And-Adapters a.k.a. Hexagonal Architecture
  • Peer-to-peer
  • Event sourcing
  • CQRS

# divers

The worst mistake of computer science - www.lucidchart.com/techblog - 20150831

In commemoration of the 50th anniversary of Sir Hoare’s null, this article explains what null is, why it is so terrible, and how to avoid it.

architecture →