Python programming

Python basics

Python IDE

In 2016 I started using PyCharm by JetBrains for Python IDE. JetBrains has a Community Edition version which is free.

Before PyCharm I used PyDev for my Python IDE. See My PyDev Notes section.

Escape sequences

Insert list or link to list of escape sequences

Type conversions

int() float()

Python string formatting

A nice collection of new and old Python format examples. I should really add some examples as this site has helped me out more than a few times. I like the examples and style used by the site.

Format String Syntax

Format Specification Mini-Language

time.strftime and use %f to get microseconds

f string Python usage guide

Python f-string tutorial

comma and fixed-point notation examples

For this example I transferred 2 billion bytes over my home LAN between a Ubuntu 18.04 server and Windows 10 client using Python networking scripts. It took 18.92 (rounding) seconds using Wireshark timestamps.

Plain formatting

numBytes = 2000000000
seconds = 18.92
bps = (numBytes / seconds) * 8
print('{} bps'.format(bps))

845665961.9450316 bps

Using comma as a thousands separator

numBytes = 2000000000
seconds = 18.92
bps = (numBytes / seconds) * 8
print('{:,} bps'.format(bps))

845,665,961.9450316 bps

Using fixed-point notation with 3 precision (6 is default)

numBytes = 2000000000
seconds = 18.92
bps = (numBytes / seconds) * 8
print('{:,.3f} bps'.format(bps))

845,665,961.945 bps

Convert hexadecimal string to integer then format integer into string with number commas and pad some hex

>>> int('f31d', 16)
62237

>>> '{:,}'.format(int('f31d', 16))
'62,237'

>>> port = int('f31d', 16)

>>> port
62237

>>> hex(port)
'0xf31d'

>>> hex(port)[2:] < -- remove the '0x' characters, they are simply in the way when I work in hexadecimal
'f31d'

>>> s = hex(port)[2:]

>>> s
'f31d'

>>> '{:>08}'.format(s) < -- pad hexadecimal value to be 8 characters wide
'0000f31d'

>>> '{:>06x}'.format(41082)  < -- pad hexadecimal value straight from integer without splicing
'00a07a'

Formatting with bits, decimals, and hexadecimals

Convert text numbers into text hexadecimal values

>>> f = '{:02x}'

>>> f.format(255)
'ff'

>>> f.format('255') < -- cannot directly convert from string number to hexadecimal number
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    f.format('255')
ValueError: Unknown format code 'x' for object of type 'str'

>>> f.format(int('255')) < -- cast string number into integer then apply string formatting
'ff' < -- hexadecimal string

Randomly create some bits then format as hexadecimal. Then dynamically specify format using nested parametrized formats.

>>> from random import getrandbits

>>> '0x{0:08x}'.format(getrandbits(64))
'0x43c5e5b009f7dc1b'

>>> '0x{0:08x}'.format(getrandbits(64))
'0xce3d5ffeaf7d693b'

>>> '0x{:08x}'.format(getrandbits(64))
'0x9b3fcfc7b31b9c1a'

>>> '0x{:0{width}x}'.format(getrandbits(64), width=8)
'0x8f6aae777885c555'

>>> '{:0{width}x}'.format(getrandbits(64), width=8)
'f5f900fba39f78ef'

>>> '{:0{width}x}'.format(getrandbits(64), width=8)
'4f3f855b2d6f4abd'

>>> '{:0{width}x}'.format(getrandbits(32), width=8)
'7791644a'

>>> '{:0{width}x}'.format(getrandbits(32), width=16)
'00000000b498791e'

>>> def get_random_hex_string(length_of_hex):
	return '{:0{length}x}'.format(getrandbits(length_of_hex * 4), length=length_of_hex)

>>> get_random_hex_string(8)
'c61e3d06'

>>> get_random_hex_string(16)
'0c81d7dda113c0fd'

>>> get_random_hex_string(32)
'270cb00a17f5dfc79f65b15ebfd57195'

>>> get_random_hex_string(64)
'eb06490b3d208afed79c1f380ab6c7f9c23c16319aa0928d6ac3609fa2c620af'

Create a function to randomly return a hexadecimal string of varied length with integer parameter.

def get_random_hex_string(length_of_hex):
    """
    Function takes the desired total length of hexadecimal characters and returns text string composed of random hexadecimal characters.
    :param length_of_hex: integer representing the total number of hexadecimal digits desired
    :return: text string of precise length and random hexadecimal values
    """
    return '{:0{length}x'.format(getrandbits(length_of_hex * 4), length=length_of_hex)

Create random IPv4 address an octet at a time

>>> from random import getrandbits
	
>>> hf = '{:02x}'
	
>>> hl = list()
	
>>> for i in range(4):
	hl.append(hf.format(getrandbits(8))) < -- build a list of four random octets (32 bits)

>>> hl
['73', 'dc', 'd9', '36']

>>> ''.join(hl) < -- join the octets
	
'73dcd936' < -- IPv4 representation in compact hexadecimal format

Conditionals

Operator	Conditions when true
x == y	x and y have same value
x != y	x and y don't have same value
x < y	x value is less than y value
x <= y	x value is less than or equal to y value
x > y	x value is greater than y value
x >= y	x value is greater than or equal to y value

string methods (s1 and s2 are strings)	conditions when true
s1.startswith(s2)	String s1 starts with s2
s1.endswith(s2)	String s1 ends with s2
s1.isalnum()	All characters in s1 are alphanumeric and there is >= 1 character
s1.isalpha()	All characters in s1 are alphabetic and there is >= 1 character
s1.isdigit()	All characters in s1 are digits and there is >= 1 character
s1.islower()	All cased characters in s1 are lowercase and there is >= 1 cased character
s.isupper()	All cased characters in s1 are uppercase and there is >= 1 cased character

Lists, Tuples, Sets, Dicts

My Python dictionary notes

My Python list notes

Create empty lists, dictionaries, sets

>>> d = dict()

>>> type(d)
<class 'dict'>

>>> l = list()

>>> type(l)
<class 'list'>

>>> s = set()

>>> type(s)
<class 'set'>

>>> d2 = {}

>>> type(d2)
<class 'dict'>

>>> l2 = []

>>> type(l2)
<class 'list'>

Convert a list into a set.

>>> my_list = [1, 2, 3, 4, 1, 2, 3, 4]

>>> my_list
[1, 2, 3, 4, 1, 2, 3, 4]

>>> type(my_list)
<class 'list'>

>>> my_set = set(my_list)

>>> my_set
{1, 2, 3, 4}

>>> type(my_set)
<class 'set'>

Loops

While loop

How to exit while loop using Enter key

while True:
    i = input("Enter text (or Enter to quit): ")
    if not i:
        break
    print("Your input:", i)
print("While loop has exited")

For loop

How to cycle through elements viewing each one by pressing Enter key or any character to exit

for element_in_list in big_list:
    if not input("Press Enter to see next item or any other character to exit: "):
        print(element_in_list)
    else:
        break

Python datetime module

datetime module supplies classes for manipulating dates and times in both simple and complex ways.

Examples of datetime usage

strftime and strptime behavior

Stop using utcnow and utcfromtimestamp

HOWTO get an ISO 8601 or ASN.1 GeneralizedTime time format with Python datetime object

>>> import datetime

>>> datetime.MINYEAR		  
1

>>> datetime.MAXYEAR		  
9999

>>> datetime.datetime.now()		  
datetime.datetime(2019, 3, 31, 21, 29, 47, 993877)

>>> from datetime import datetime
		  
>>> datetime.now()
		  
datetime.datetime(2019, 3, 31, 21, 30, 4, 999219)

>>> d = datetime.now()
		  
>>> d
		  
datetime.datetime(2019, 3, 31, 21, 30, 51, 362520)

>>> f = '{:%Y%m%d%H%M%S.%f}'  < -- Setup a string format to control the datetime format.
		  
>>> f.format(d)
		  
'20190331213051.362520' < -- microsecond accuracy

>>> d2 = datetime(2019, 3, 31, 21, 30, 51, 000000)  < -- Create datetime object with exactly zero microseconds to verify string format.
		  
>>> f.format(d2)
		  
'20190331213051.000000' < -- verified that datetime displays six zeros

>>> '20190331213051.000000'[:-3] < -- truncate last three digits
		  
'20190331213051.000' < -- Now we need timezone offset

>>> z = datetime.datetime.utcnow() < -- I want UTC time so I use datetime class method utcnow()
		  
>>> z
		  
datetime.datetime(2019, 4, 1, 3, 28, 53, 152240) < -- not in format I want

>>> f.format(z) < -- apply format specification used above
		  
'20190401032853.152240' < -- want to remove least 3 insignificant digits from microseconds

>>> f.format(z)[:-3]
		  
'20190401032853.152' < -- missing offset

>>> f.format(z)[:-3]+'Z' < -- simple string concatenate
		  
'20190401032853.152Z' < -- final product

Summary of string formatting to get to ISO 8601 from Python datetime object

• Format string: '{:%Y%m%d%H%M%S.%f}'
• Truncate last three digits from formatted string: [:-3]
• Concatenate 'Z' for Zulu time + 'Z'

Complete example (without references)

>>> '{:%Y%m%d%H%M%S.%f}'.format(datetime.datetime.utcnow())[:-3] + 'Z'
							  
'20190401034018.177Z'

Same example with f-string

f'{datetime.utcnow():%Y%m%d%H%M%S.%f}'[:-3]+'Z'

Working with JSON in Python

Working With JSON Data in Python by Lucas Lofaro

JSON encoder and decoder including making JSON beautiful.

Writing data to CSV

Python write CSV file tutorial

Python Network Programming

HOWTO send UTF-8 text strings as text or hexadeciaml

>>> import socket

>>> socket_object = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

>>> socket_object.connect(('192.168.1.109', 1060))

>>> socket_object.send(bytes('Hello There', 'utf-8')) < -- Wireshark shows TCP payload data as 48656c6c6f205468657265
11 < -- number of bytes sent (this means "sent" to operating system network functions, this does NOT MEAN the remote end received these bytes)

>>> socket_object.send(bytes.fromhex('48656C6C6F205468657265')) < -- Wireshark shows TCP payload data as 48656c6c6f205468657265
11 < -- number of bytes sent (this means "sent" to operating system network functions, this does NOT MEAN the remote end received these bytes)

>>> socket_object.close()

If you have text string data encoded as UTF-8 characters then use bytes(<string>, 'utf-8')

If you have text string data encoded as hexadecimal values (2 hexadecimal characters map to a UTF-8 character) then use bytes.fromhex()

My Python Network Programming Notes

Converting binary string and text string

The binascii module contains a number of methods to convert between binary and various ASCII-encoded binary representations.

>>> import binascii

>>> b = binascii.hexlify(b'binary_string') < -- convert from binary string to binary hexadecimal representation...similar to bytes.hex() except this returns binary string
							  
>>> b
							  
b'62696e6172795f737472696e67' < -- binary hexadecimal representation
>>> binascii.a2b_hex(b) <-- convert from binary hexadecimal to binary string
							  
b'binary_string'

>>> bytes.hex(b) < -- convert binary hexadecimal b'62696e6172795f737472696e67' to octet strings (takes up twice as much space)
							  
'3632363936653631373237393566373337343732363936653637'

>>> s = '1234567890'  < -- create sample string
							  
>>> bo = bytes(s, 'ascii') < -- create bytes object of string (binary string)
							  
>>> bo
							  
b'1234567890' < -- notice the little b in front of the first single quote
>>> octet_string = bytes.hex(bo)  < -- convert bytes object (binary string) to octet string
							  
>>> octet_string
							  
'31323334353637383930'

Converting back and forth with binascii, bytes, ASCII symbols, and ASCII as hexadecimal

>>> import datetime

>>> dt = datetime.datetime.utcnow()

>>> dt
datetime.datetime(2019, 4, 2, 1, 36, 8, 931977)

>>> f = '{:%Y%m%d%H%M%S.%f}'

>>> f.format(dt)
'20190402013608.931977'

>>> f.format(dt)[:-3]+'Z'
'20190402013608.931Z'

>>> t = f.format(dt)[:-3]+'Z'

>>> t
'20190402013608.931Z'

>>> bytes(t,'ascii')
b'20190402013608.931Z'

>>> import binascii

>>> binascii.hexlify(bytes(t,'ascii'))
b'32303139303430323031333630382e3933315a'

>>> binascii.hexlify(bytes(t,'ascii'))
b'32303139303430323031333630382e3933315a'

>>> bt = binascii.hexlify(bytes(t,'ascii'))

>>> bt
b'32303139303430323031333630382e3933315a'

>>> bt.decode()
'32303139303430323031333630382e3933315a'

>>> '32303139303430323031333630382e3933315a'.encode()
b'32303139303430323031333630382e3933315a'

>>> b'32303139303430323031333630382e3933315a'.decode()
'32303139303430323031333630382e3933315a'

>>> binascii.unhexlify(bt)  < -- change from binary ASCII encoded as hexadecimals to binary string of ASCII symbols	
b'20190402013608.931Z'

>>> binascii.unhexlify(bt).decode()  < -- do the same as above but also change from binary string to text string
	
'20190402013608.931Z'

>>> st = bt.decode()  < -- convert binary string to text string

>>> st
'32303139303430323031333630382e3933315a'

>>> bytearray.fromhex(st).decode() < -- change from text string ASCII encoded as hexadecimals to text string of ASCII symbols
'20190402013608.931Z'

Miscellaneous stuff

Difference Between Carriage Return, Line Feed and End of Line Characters

To Python