Python Advanced (7.9.2020 - 10.9.2020)

Training Information

Start

Montag 7.9.2020, 8:00

Official Description

Python: More Language Features

Material

Python material	Python Programming
PDF Slides (Design Patterns)	Design Patterns

Github Private Repository

https://github.com/jfasch/2020-09-07

This is a training for a team which is already experienced in Python programming. There were some special requirements for the training, such as to loose a couple of words about AI/machinelearning and other topics, as well as a fuzzy “bring us advanced stuff”.

The training was accompanied by a lot of improvised live hacking.

Preparation

Workspace Setup

• Python Installation

• Visual Studio Code for Python Programming

Github for Exercises

Please create Github accounts and send them to training@faschingbauer.co.at. We will do exercises via a private repository on Github.

Agenda

Advanced Core Python Topics

Python is easy; one can solve nontrivial problems in only a few lines of code, in no time. This does not mean that you have to fully understand the language - which is good because this is what makes the language easy.

On the other hand, there’s always a line to cross where you wish you knew more. Here the more advanced core Python topics that shall be covered.

• Iteration, Generators, And yield

• Context Managers: The with Statement

• Encoding

• Exception Handling

• Modules and Packages

• Inheritance

• Operator Overloading

• Positional and Keyword Arguments

• Closures

• Decorators

Special Topics

• Machine Learning, Artificial Intelligence

  • Machine Learning: Concepts and Terminology

  • Linear Regression

  • K-Means

• Unit Testing, Test Driven Development, Design Patterns

  • Excerpt from Unittesting and Test Driven Development (/trainings/material/pdf/060-design-patterns-unittests.pdf)

  • The unittest Module

• Web programming with Jango and/or Flask

Log

Following is a recap of what has happened. File names are relative to the root of the Github repository we were working from.

Day 1

Rushed through basics,

• Variables

• Datatypes

• Compound Datatypes

• The if Statement

Put a strong focus on Python features,

• References, (Im)mutability

• Sequential Datatypes

• Indexing and Slicing

• while Loops. The else clause, mainly - everyone was supposed to know what a while loop is.

• for Loops.

  • else clause

  • The range Function. Whetting appetite for the Iteration livehacking session on day 2.

Day 2

• Livehacking: Iteration, Comprehensions, and Generators, covering

  • List comprehensions

  • The range function

  • Writing generators using yield

  • Generator expressions

• Livehacking OO

  • joerg-livehacking/oo.py

    • initially: Thermometer, ctor, get_temperature()

    • __init__, self, etc. (self can also read this)

    • th.__dict__

    • private, pros and cons

    • properties (r/w)

  • inheritance

    • joerg-livehacking/abstract-base-class.py, joerg-livehacking/abstract-base-class.cpp

    • Insert one step without abc before it, and then show what abc can do

    • explain “check errors as early as can” ⟶ at ctor time, rather than at method call time.

• Livehacking design patterns

  • joerg-livehacking/composite.py. Using the thermometer hierarchy, a “composite” thermometer was created. That thermometer uses (has) a set of concrete thermometers to calculate the average room temperature.

  • joerg-livehacking/adapter.py. Fictional scenario …

    • The Unser thermometer framework contains a number of thermometer implementations which all support the get_temperature_celsius() method.

    • A collaboration with a competitor is launched. That competitor has a similar set of thermometer implementations. The difference between Eana and Unser is that Eana thermometers do not support get_temperature_celsius(), but rather only get_temperature_fahrenheit().

    • We employ the adapter pattern and create one special thermometer in the Unser hierarchy

      class EanaAdapter(UnserThermometer):
          ...
      

Day 3

• Revisit abstract base classes

  • Discuss duck typing.

  • Morph joerg-livehacking/abstract-base-class.py into joerg-livehacking/duck-typing.py.

  • Explaining how abc shifts duck errors from call to initialization

• Visitor pattern. Not every pattern in the “Gang of Four” book should be considered a real pattern. For example, the Visitor pattern turn into an idiom for languages that don’t support generators.

  • joerg-livehacking/visitor.py. Classic OO implementation of the visitor pattern as a DFS traversal. Together with callbacks and all convolutions. Took sideways like,

    • show how __call__ makes a class callable.

    • show how __str__ and __repr__ work together in print()

  • joerg-livehacking/visitor-generator.py. “I don’t want to implement a visitor!”, poor user says. “I only want to iterate over the tree in DFS order!”

    Implement DFS iteration using yield from which delegates iteration into recursion.

• TDD and Unit Testing theory; using excerpts from the Design Patterns deck of slides. Explain terminology; fixtures and such.

• Start hacking on project. Agreed upon myself doing live hacking. Doing TDD.

  • Project/sensor/sensordata.py, and Project/tests/sensordata_tests.py. Prepare TDD; explain suites, cases, fixture, assertions.

  • While writing data classes (holding only attributes and no functionality), explain namedtuple. Use that to implement sensor.sensordata.HistoryData.

  • Slowly fix things, in a test driven way. Discuss, team giving input, all really fine.

Day 4

• Decorators theory, and livehacking. Mainly to see how flask routes work.

  • Positional and Keyword Arguments

  • Closures

  • Decorators

• Continue project; add CSV import.

  Project/programs/csvreport.py

• sqlite3 export, trying out the DBAPI 2.0 interface.

  Project/programs/csv2sqlite3.py

• Flask frontend, reading sqlite3 db.

  Project/programs/flaskerl.py

Recommended Tutorials

We couldn’t cover everything we would have liked to. Here is a random list of tutorial to watch in quiet moments.

• Transforming Code into Beautiful, Idiomatic Python. Raymond Hettinger, reiterating his favorite phrase: “There must be a better way”. (Hettinger is a “Python Core Developer”.)

• Python Tutorial: Duck Typing and Asking Forgiveness, Not Permission (EAFP). Corey Schafer about Duck Typing, and the word Pythonic. Corey Schafer has a number of very good Tutorials; he manages to keep those short and to the point, and rarely exceeds 15 minutes.

• Python Tutorial: Unit Testing Your Code with the unittest Module. Our project was guided by unit tests; here’s Corey Schafer about the unittest module.

• Python Tutorial: Decorators - Dynamically Alter The Functionality Of Your Functions. Corey Schafer about decorators. Mine was better :-)

• Built in Super Heroes. David Beazley in an entertaining keynote to the “PyData Chicago 2016” conference. He has a number of very good and entertaining (and very advanced) videos. You have to spend an entire evening with him though.

• Concurrency: Raymond Hettinger covering most if not all aspects and possiblities of concurrency. Very informative, very concise, covering

  • Multithreading

  • Multiprocessing

  • Async; I didn’t even mention that. asyncio. Me big fan.

• Understanding the Python GIL: David Beazley dissecting the Global Interpreter Lock, explaining why multiprocessing is better. At around minute 45, in the questions/answers, there a mention that using NumPy operations in multiple threads is truly parallel.

• Modules and Packages. David Beazley has a three hour (!) really cool and in-depth look into the seemingly simple import mechanism.

• Virtual Environments Tutorial: Corey Schafer again. Virtual environments are kind of an isolated development sandbox, solving a similar problem as containers do, but much more lightweight and Python only.

• Packaging, Deployment, PyPI, and pip: Chris Wilcox (of Google) talking about packaging and deployment, and related topics

• Generators: The Final Frontier: David Beazley, again a bit (a whopping four hours) more precise on that topic.

• NumPy Tutorial: Keith Galli has a number of good data science tutorials.

After Work Party

From my point of view, the training went really fine. Not everyone is equally satisfied with the outcome (we didn’t get to the AI topics, for example), but I have the impression that I brought it over.

As a gift to myself, I had reserved Thursday night at Gmundner Hütte. After the training I went back to Hoisn Wirt (which is where I stayed during the training - really fine), changed clothes, and started to climb the Traunstein via Zierlersteig.

See here for the description and pictures from this extraordinary hike.