Editing Python programming

== Python basics ==

=== Python IDE ===

In 2016 I started using [https://www.jetbrains.com/pycharm/ PyCharm] by [https://www.jetbrains.com/ JetBrains] for Python IDE. JetBrains has a Community Edition version which is free.

Before PyCharm I used [http://pydev.org/ PyDev] for my Python IDE. See [[My PyDev Notes]] section.

=== Escape sequences ===

Insert list or link to list of escape sequences

=== Type conversions ===

int()
float()

=== Python string formatting ===

A nice collection of new and old Python [https://pyformat.info/ format examples]. I should really add some examples as this site has helped me out more than a few times. I like the examples and style used by the site.

[https://docs.python.org/3/library/string.html#format-string-syntax Format String Syntax]

[https://docs.python.org/3/library/string.html#formatspec Format Specification Mini-Language]

[https://docs.python.org/3/library/time.html#time.strftime time.strftime] and use %f to get microseconds

[https://saralgyaan.com/posts/f-string-in-python-usage-guide/ f string Python usage guide]

[https://zetcode.com/python/fstring/ Python f-string tutorial]

==== comma and [https://docs.python.org/3.7/library/string.html#format-specification-mini-language fixed-point notation] examples ====

For this example I transferred 2 billion bytes over my home LAN between a Ubuntu 18.04 server and Windows 10 client using Python networking scripts. It took 18.92 (rounding) seconds using Wireshark timestamps.

'''Plain formatting'''
<pre>numBytes = 2000000000
seconds = 18.92
bps = (numBytes / seconds) * 8
print('{} bps'.format(bps))

845665961.9450316 bps
</pre>

'''Using comma as a thousands separator'''
<pre>numBytes = 2000000000
seconds = 18.92
bps = (numBytes / seconds) * 8
print('{:,} bps'.format(bps))

845,665,961.9450316 bps</pre>

'''Using fixed-point notation with 3 precision (6 is default)'''
<pre>numBytes = 2000000000
seconds = 18.92
bps = (numBytes / seconds) * 8
print('{:,.3f} bps'.format(bps))

845,665,961.945 bps</pre>

=== Convert hexadecimal string to integer then format integer into string with number commas and pad some hex ===

<pre>>>> int('f31d', 16)
62237

>>> '{:,}'.format(int('f31d', 16))
'62,237'

>>> port = int('f31d', 16)

>>> port
62237

>>> hex(port)
'0xf31d'

>>> hex(port)[2:] < -- remove the '0x' characters, they are simply in the way when I work in hexadecimal
'f31d'

>>> s = hex(port)[2:]

>>> s
'f31d'

>>> '{:>08}'.format(s) < -- pad hexadecimal value to be 8 characters wide
'0000f31d'

>>> '{:>06x}'.format(41082)  < -- pad hexadecimal value straight from integer without splicing
'00a07a'</pre>

=== Formatting with bits, decimals, and hexadecimals ===

<big>Convert text numbers into text hexadecimal values</big>
<pre>>>> f = '{:02x}'

>>> f.format(255)
'ff'

>>> f.format('255') < -- cannot directly convert from string number to hexadecimal number
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    f.format('255')
ValueError: Unknown format code 'x' for object of type 'str'

>>> f.format(int('255')) < -- cast string number into integer then apply string formatting
'ff' < -- hexadecimal string</pre>

<big>Randomly create some bits then format as hexadecimal. Then dynamically specify format using nested parametrized formats.</big>
<pre>>>> from random import getrandbits

>>> '0x{0:08x}'.format(getrandbits(64))
'0x43c5e5b009f7dc1b'

>>> '0x{0:08x}'.format(getrandbits(64))
'0xce3d5ffeaf7d693b'

>>> '0x{:08x}'.format(getrandbits(64))
'0x9b3fcfc7b31b9c1a'

>>> '0x{:0{width}x}'.format(getrandbits(64), width=8)
'0x8f6aae777885c555'

>>> '{:0{width}x}'.format(getrandbits(64), width=8)
'f5f900fba39f78ef'

>>> '{:0{width}x}'.format(getrandbits(64), width=8)
'4f3f855b2d6f4abd'

>>> '{:0{width}x}'.format(getrandbits(32), width=8)
'7791644a'

>>> '{:0{width}x}'.format(getrandbits(32), width=16)
'00000000b498791e'

>>> def get_random_hex_string(length_of_hex):
	return '{:0{length}x}'.format(getrandbits(length_of_hex * 4), length=length_of_hex)

>>> get_random_hex_string(8)
'c61e3d06'

>>> get_random_hex_string(16)
'0c81d7dda113c0fd'

>>> get_random_hex_string(32)
'270cb00a17f5dfc79f65b15ebfd57195'

>>> get_random_hex_string(64)
'eb06490b3d208afed79c1f380ab6c7f9c23c16319aa0928d6ac3609fa2c620af'</pre>

<big>Create a function to randomly return a hexadecimal string of varied length with integer parameter.</big>
<pre>def get_random_hex_string(length_of_hex):
    """
    Function takes the desired total length of hexadecimal characters and returns text string composed of random hexadecimal characters.
    :param length_of_hex: integer representing the total number of hexadecimal digits desired
    :return: text string of precise length and random hexadecimal values
    """
    return '{:0{length}x'.format(getrandbits(length_of_hex * 4), length=length_of_hex)</pre>

<big>Create random IPv4 address an octet at a time</big>
<pre>>>> from random import getrandbits
	
>>> hf = '{:02x}'
	
>>> hl = list()
	
>>> for i in range(4):
	hl.append(hf.format(getrandbits(8))) < -- build a list of four random octets (32 bits)

>>> hl
['73', 'dc', 'd9', '36']

>>> ''.join(hl) < -- join the octets
	
'73dcd936' < -- IPv4 representation in compact hexadecimal format</pre>

=== split function ===

HOWTO use [https://docs.python.org/3/library/stdtypes.html#string-methods Python's split()] on text using ASCII decimal 29 [https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_separators Group Separator] for separator.

Define your ASCII char <code>.split(chr(29))</code>

Example:
<pre>text_to_split.split(chr(29))</pre>

=== Conditionals ===

{| border="1" cellspacing="0" cellpadding="5" align="center"
|-
! Operator
! Conditions when true
|- 
| x == y
| x and y have same value
|-
| x != y
| x and y don't have same value
|-
| x < y
| x value is less than y value
|-
| x <= y
| x value is less than or equal to y value
|-
| x > y
| x value is greater than y value
|-
| x >= y
| x value is greater than or equal to y value
|}


{| border="1" cellspacing="0" cellpadding="5" align="center"
|-
| string methods (s1 and s2 are strings)
| conditions when true
|-
| s1.startswith(s2)
| String s1 starts with s2
|-
| s1.endswith(s2)
| String s1 ends with s2
|-
| s1.isalnum()
| All characters in s1 are alphanumeric and there is >= 1 character
|-
| s1.isalpha()
| All characters in s1 are alphabetic and there is >= 1 character
|-
| s1.isdigit()
| All characters in s1 are digits and there is >= 1 character
|-
| s1.islower()
| All cased characters in s1 are lowercase and there is >= 1 cased character
|-
| s.isupper()
| All cased characters in s1 are uppercase and there is >= 1 cased character
|}

=== Lists, Tuples, Sets, Dicts ===

[[My Python dictionary notes]]

[[My Python list notes]]

<big>Create empty lists, dictionaries, sets</big>

<pre>>>> d = dict()

>>> type(d)
<class 'dict'>

>>> l = list()

>>> type(l)
<class 'list'>

>>> s = set()

>>> type(s)
<class 'set'>

>>> d2 = {}

>>> type(d2)
<class 'dict'>

>>> l2 = []

>>> type(l2)
<class 'list'></pre>

<big>Convert a list into a set.</big>

<pre>>>> my_list = [1, 2, 3, 4, 1, 2, 3, 4]

>>> my_list
[1, 2, 3, 4, 1, 2, 3, 4]

>>> type(my_list)
<class 'list'>

>>> my_set = set(my_list)

>>> my_set
{1, 2, 3, 4}

>>> type(my_set)
<class 'set'></pre>

== Python operators ==

== Loops ==

=== While loop ===

How to exit while loop using Enter key

<pre>
while True:
    i = input("Enter text (or Enter to quit): ")
    if not i:
        break
    print("Your input:", i)
print("While loop has exited")
</pre>


=== For loop ===

How to cycle through elements viewing each one by pressing Enter key or any character to exit

<pre>
for element_in_list in big_list:
    if not input("Press Enter to see next item or any other character to exit: "):
        print(element_in_list)
    else:
        break
</pre>

== Python datetime module ==

[https://docs.python.org/3/library/datetime.html datetime] module supplies classes for manipulating dates and times in both simple and complex ways.

[https://docs.python.org/3/library/datetime.html#examples-of-usage-datetime Examples of datetime usage]

[https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior strftime and strptime behavior]

[https://blog.ganssle.io/articles/2019/11/utcnow.html Stop using utcnow and utcfromtimestamp]

=== HOWTO get an [https://en.wikipedia.org/wiki/ISO_8601 ISO 8601] or [https://www.obj-sys.com/asn1tutorial/node14.html ASN.1 GeneralizedTime] time format with Python datetime object ===

<pre>>>> import datetime

>>> datetime.MINYEAR		  
1

>>> datetime.MAXYEAR		  
9999

>>> datetime.datetime.now()		  
datetime.datetime(2019, 3, 31, 21, 29, 47, 993877)

>>> from datetime import datetime
		  
>>> datetime.now()
		  
datetime.datetime(2019, 3, 31, 21, 30, 4, 999219)

>>> d = datetime.now()
		  
>>> d
		  
datetime.datetime(2019, 3, 31, 21, 30, 51, 362520)

>>> f = '{:%Y%m%d%H%M%S.%f}'  < -- Setup a string format to control the datetime format.
		  
>>> f.format(d)
		  
'20190331213051.362520' < -- microsecond accuracy

>>> d2 = datetime(2019, 3, 31, 21, 30, 51, 000000)  < -- Create datetime object with exactly zero microseconds to verify string format.
		  
>>> f.format(d2)
		  
'20190331213051.000000' < -- verified that datetime displays six zeros

>>> '20190331213051.000000'[:-3] < -- truncate last three digits
		  
'20190331213051.000' < -- Now we need timezone offset

>>> z = datetime.datetime.utcnow() < -- I want UTC time so I use datetime class method utcnow()
		  
>>> z
		  
datetime.datetime(2019, 4, 1, 3, 28, 53, 152240) < -- not in format I want

>>> f.format(z) < -- apply format specification used above
		  
'20190401032853.152240' < -- want to remove least 3 insignificant digits from microseconds

>>> f.format(z)[:-3]
		  
'20190401032853.152' < -- missing offset

>>> f.format(z)[:-3]+'Z' < -- simple string concatenate
		  
'20190401032853.152Z' < -- final product</pre>

<big>Summary of string formatting to get to [https://en.wikipedia.org/wiki/ISO_8601 ISO 8601] from [https://docs.python.org/3/library/datetime.html#module-datetime Python datetime object]</big>
* Format string: <code>'{:%Y%m%d%H%M%S.%f}'</code>
* Truncate last three digits from formatted string: <code>[:-3]</code>
* Concatenate 'Z' for Zulu time <code>+ 'Z'</code>

<big>Complete example (without references)</big>
<pre>>>> '{:%Y%m%d%H%M%S.%f}'.format(datetime.datetime.utcnow())[:-3] + 'Z'
							  
'20190401034018.177Z'</pre>

Same example with f-string

<pre>f'{datetime.utcnow():%Y%m%d%H%M%S.%f}'[:-3]+'Z'</pre>

== Working with [http://json.org JSON] in Python ==

[https://realpython.com/python-json/ Working With JSON Data in Python] by Lucas Lofaro

[https://docs.python.org/3/library/json.html JSON encoder and decoder] including making JSON beautiful.

== Writing data to CSV ==

[https://www.pythontutorial.net/python-basics/python-write-csv-file/ Python write CSV file tutorial]

== Python Network Programming ==

=== HOWTO send UTF-8 text strings as text or hexadeciaml ===

<pre>>>> import socket

>>> socket_object = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

>>> socket_object.connect(('192.168.1.109', 1060))

>>> socket_object.send(bytes('Hello There', 'utf-8')) < -- Wireshark shows TCP payload data as 48656c6c6f205468657265
11 < -- number of bytes sent (this means "sent" to operating system network functions, this does NOT MEAN the remote end received these bytes)

>>> socket_object.send(bytes.fromhex('48656C6C6F205468657265')) < -- Wireshark shows TCP payload data as 48656c6c6f205468657265
11 < -- number of bytes sent (this means "sent" to operating system network functions, this does NOT MEAN the remote end received these bytes)

>>> socket_object.close()</pre>

If you have text string data encoded as UTF-8 characters then use bytes(<string>, 'utf-8')

If you have text string data encoded as hexadecimal values (2 hexadecimal characters map to a UTF-8 character) then use bytes.fromhex()

[[My Python Network Programming Notes]]

== Converting binary string and text string ==

The [https://docs.python.org/3/library/binascii.html binascii] module contains a number of methods to convert between binary and various ASCII-encoded binary representations.

<pre>>>> import binascii

>>> b = binascii.hexlify(b'binary_string') < -- convert from binary string to binary hexadecimal representation...similar to bytes.hex() except this returns binary string
							  
>>> b
							  
b'62696e6172795f737472696e67' < -- binary hexadecimal representation
>>> binascii.a2b_hex(b) <-- convert from binary hexadecimal to binary string
							  
b'binary_string'

>>> bytes.hex(b) < -- convert binary hexadecimal b'62696e6172795f737472696e67' to octet strings (takes up twice as much space)
							  
'3632363936653631373237393566373337343732363936653637'

>>> s = '1234567890'  < -- create sample string
							  
>>> bo = bytes(s, 'ascii') < -- create bytes object of string (binary string)
							  
>>> bo
							  
b'1234567890' < -- notice the little b in front of the first single quote
>>> octet_string = bytes.hex(bo)  < -- convert bytes object (binary string) to octet string
							  
>>> octet_string
							  
'31323334353637383930'</pre>

=== Converting back and forth with binascii, bytes, ASCII symbols, and ASCII as hexadecimal ===

<pre>>>> import datetime

>>> dt = datetime.datetime.utcnow()

>>> dt
datetime.datetime(2019, 4, 2, 1, 36, 8, 931977)

>>> f = '{:%Y%m%d%H%M%S.%f}'

>>> f.format(dt)
'20190402013608.931977'

>>> f.format(dt)[:-3]+'Z'
'20190402013608.931Z'

>>> t = f.format(dt)[:-3]+'Z'

>>> t
'20190402013608.931Z'

>>> bytes(t,'ascii')
b'20190402013608.931Z'

>>> import binascii

>>> binascii.hexlify(bytes(t,'ascii'))
b'32303139303430323031333630382e3933315a'

>>> binascii.hexlify(bytes(t,'ascii'))
b'32303139303430323031333630382e3933315a'

>>> bt = binascii.hexlify(bytes(t,'ascii'))

>>> bt
b'32303139303430323031333630382e3933315a'

>>> bt.decode()
'32303139303430323031333630382e3933315a'

>>> '32303139303430323031333630382e3933315a'.encode()
b'32303139303430323031333630382e3933315a'

>>> b'32303139303430323031333630382e3933315a'.decode()
'32303139303430323031333630382e3933315a'

>>> binascii.unhexlify(bt)  < -- change from binary ASCII encoded as hexadecimals to binary string of ASCII symbols	
b'20190402013608.931Z'

>>> binascii.unhexlify(bt).decode()  < -- do the same as above but also change from binary string to text string
	
'20190402013608.931Z'

>>> st = bt.decode()  < -- convert binary string to text string

>>> st
'32303139303430323031333630382e3933315a'

>>> bytearray.fromhex(st).decode() < -- change from text string ASCII encoded as hexadecimals to text string of ASCII symbols
'20190402013608.931Z'</pre>

== Miscellaneous stuff ==

[https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019KZDSA2&l=en-US Difference Between Carriage Return, Line Feed and End of Line Characters]

<center>[[Python|To Python]]</center>