Python Advanced (2023-10-24 - 2023-10-26)

Original Agenda

• Agenda: Python Advanced (2023-10-24 - 2023-10-26)

Github Repository

• https://github.com/jfasch/2023-10-24

Notebook

• Structure: Jupyter Notebook (Structure): Python Advanced (2023-10-24 - 2023-10-26)

• Entire notebook: notebook.ipynb

Material

• Python Programming

Day 1

Python Basics Recap

All optional. But we should not miss the central points like Python’s referencing scheme, im/mutability, memory management, typing. And iteration of course 😎. Everything’s an object, alas.

The Very Basics

• Variables

• Datatypes

• More About Strings

• String Formatting

• Miscellaneous String Methods

• Compound Datatypes

• More About Lists

• More About Dictionaries

• More About Sets

• Sequential Datatypes

• Indexing and Slicing

• References, (Im)mutability

Exercise time …

• Exercise: Digit to English Word

range(), And Iteration In General

• The range Function

• The Iterator Protocol

• Implementing A Fibonacci Number Generator

Modules

• Modules and Packages

Group Project Kickoff

• Draw “Data Logger” sketch

• Sensor classes intro

• Main loop ⟶ application, simply printing to stdout

Object Oriented Programming

From Object Oriented Programming …

• Classes And Dictionaries

• Constructor

• Methods

Day 2

Leftovers From Yesterday

• float.__int__() (via Person.__str__())

• Where does the exit() function come from? It is not documented in the builtin functions documentation, nonetheless it is present as a global function.

  • Built-in Constants: at the bottom, there is a note about the site — Site-specific configuration hook module doing that

  • site — Site-specific configuration hook does not mention that

  ⟶ a-ha

• frozendict? There is no such thing, but there’s a frozenset

• async/await? ⟶ Pointless Blinking

Miscellaneous

• Comprehensions (List, Dictionary, Set)

Object Oriented Programming, Continued

From Object Oriented Programming …

• str() And repr()

• Properties

• collections.namedtuple

Group Project, OO-ified

• FileSensor

• CSVSink

• AcquisitionLoop

Bring them live …

See also

• Permissions: Mode, User and Group Ownership

• pytest Introduction, By Example

• Coding guidelines? ⟶ PEP-8

• Integration into datalogger.py

• Make datalogger.py a real program (⟶ Permissions: Mode, User and Group Ownership)

• Problems

  • Test data (the test temperature file for FileSensor) ⟶ pytest Introduction, By Example

  • Testing AcquisitionLoop

    • Number of iterations

    • Timestamps?

Test Driven Development

• pytest Introduction, By Example

• Look into tests

• AcquisitionLoop?

  • Externalize timestamps ⟶ iterables and generators

  • itertools — Functions creating iterators for efficient looping

Object Oriented Programming, Continued

• Inheritance

  Apply inheritance

  • Create Sensor base class (test first)

  • assert being-a Sensor before entering main loop

• Abstract Base Classes (abc), And Duck Typing

  • Require Sensor to be-a abc.ABC (test first)

• Put all sensor implementations under an abstract base class, sensor.Sensor.

Decorator Toolcase: *args, **kwargs

• Positional and Keyword Arguments

Day 3

Morning Greeting

• How does pytest know which fixtures are requested? ⟶ inspect — Inspect live objects

• Sergey’s OPC-UA sensor

• Revisit test_acquisition_loop.test_mock_csv_sink. Replace class MyMockedThing with a function that uses nonlocal

• ⟶ Functions: Global and Local Scope (Livehacking Screenplay) (enter Closures)

• Further plan (see below)

Decorator Toolcase, Continued (Including Function Scoping and global)

• Functions: Global and Local Scope (Livehacking Screenplay)

• Closures

• Decorators

Multithreading

• Multithreading (Python)

• Hack small Load-Modify-Store race (Thread Creation, Race Condition and Synchronization in C++

• Same in Python ⟶ nothing

• ⟶ The GIL (Global Interpreter Lock), and its way out of the Python interpreter

• Race Conditions, and Mutexes

Exercises

Sensor Implementation That Receives UDP Datagrams

See also

• Multithreading (Python)

• Race Conditions, and Mutexes

• queue — A synchronized queue class

• Datagrams contain temperature in millcelsius (just like FileSensor)

• Look into acquisition_loop_utils.py for how to receive datagrams on a port

• Acquisition rate does ont necessarily match datagram rate. It might well be that

  • Datagrams are received much faster (10ms, for example) than the acquisition loop acquires measurements (1s interval, for example). Samples gather up inside the sensor (roughly 100 samples per measurement with the above rates).

    In this case (multiple samples available at the time sensor.get_temperature() is called) a call to sensor.get_temperature() must

    • compute the average across available samples

    • remove the gathered samples to make room for new incoming data

  • Datagrams are received at a slower rate than the loop acquires. In this case there is nothing available to return from sensor.get_temperature(). To overcome that,

    • If a measurement from a previous measurement is available, return that

    • If no measurement has yet been made, raise an exception

• To not drop any packets while the main acquisition loop is busy doing other things (block on file IO, for example, or wait for another sensor to complete its measurement), the sensor implementation must spawn a thread, internally, that does the network communication.

  In fact, the whole point of the exercise is to correctly communicate the data to the main (acquisition) thread.

• Is it possible to TDD that? Try hard.

CompositeSink

• AcquisitionLoop knows about only one Sink. That is the plan!

• Create one Sink implementation, CompositeSink, that dispatches write_measurement() onto multiple sinks that it owns

• Write a test for that, together.

MQTT Sink Implementation

See also

• MQTT

• json — JSON encoder and decoder (dumps() and loads())

• monkeypatch fixture

• Publish measurement dictionary to an MQTT broker

• Define format of transmitted data

• TDD?!

Timestamps?

Bring timestamps (datatime.datatime) into the game …

• Formally define measurement dictionary (test)

• Wrap sensors dictionary that is used in datalogger.py into a class, say, Sensors.

• Write a method Sensors.read_measurements() -> Measurements

• class Measurements is an encapsulation of the raw dictionary that is composed in AcquisitionLoop, and consumed across all Sink implementations.

Data Science

Pandas

• Pandas: Basics (DataFrame And Series)

• Pandas: Selecting Rows (And Columns) With iloc[]

• Pandas: Selecting Rows (And Columns) With loc[]

• Pandas: Indexes

• Pandas: Filters

• Pandas: Adding/Modifying Columns

• Pandas: Adding Rows

NumPy

• Python (12.11.2019 - 14.11.2019 in Graz)

• Discrete Fourier Transform (numpy.fft)

  NumPy Tutorials : 011 : Fast Fourier Transforms - FFT and IFFT

Error Handling And Recovery

• Exception Handling

• Context Managers: The with Statement

Project Management

• Python Package Index

• Virtual Environments