python►c-pandas

Python-c02-Getting Started with Pandas

I will show a simple example of csv with pandas, you can download the files from here:
survey_results_public.csv
survey_results_schema.csv

Notebook

1
import pandas as pd
1
df = pd.read_csv('data/survey_results_public.csv')
1
df

Respondent Professional ProgramHobby Country University EmploymentStatus FormalEducation MajorUndergrad HomeRemote CompanySize ... StackOverflowMakeMoney Gender HighestEducationParents Race SurveyLong QuestionsInteresting QuestionsConfusing InterestedAnswers Salary ExpectedSalary
0 1 Student Yes, both United States No Not employed, and not looking for work Secondary school NaN NaN NaN ... Strongly disagree Male High school White or of European descent Strongly disagree Strongly agree Disagree Strongly agree NaN NaN
1 2 Student Yes, both United Kingdom Yes, full-time Employed part-time Some college/university study without earning ... Computer science or software engineering More than half, but not all, the time 20 to 99 employees ... Strongly disagree Male A master's degree White or of European descent Somewhat agree Somewhat agree Disagree Strongly agree NaN 37500.0
2 3 Professional developer Yes, both United Kingdom No Employed full-time Bachelor's degree Computer science or software engineering Less than half the time, but at least one day ... 10,000 or more employees ... Disagree Male A professional degree White or of European descent Somewhat agree Agree Disagree Agree 113750.0 NaN
3 4 Professional non-developer who sometimes write... Yes, both United States No Employed full-time Doctoral degree A non-computer-focused engineering discipline Less than half the time, but at least one day ... 10,000 or more employees ... Disagree Male A doctoral degree White or of European descent Agree Agree Somewhat agree Strongly agree NaN NaN
4 5 Professional developer Yes, I program as a hobby Switzerland No Employed full-time Master's degree Computer science or software engineering Never 10 to 19 employees ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
19097 19098 Professional developer Yes, I program as a hobby Canada No Employed full-time Bachelor's degree A business discipline A few days each month 10 to 19 employees ... Disagree Male Some college/university study, no bachelor's d... White or of European descent Somewhat agree Agree Disagree Agree NaN NaN
19098 19099 Student Yes, I program as a hobby India No Not employed, and not looking for work Secondary school NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
19099 19100 Professional non-developer who sometimes write... Yes, I program as a hobby United Kingdom No Independent contractor, freelancer, or self-em... Bachelor's degree Computer science or software engineering Never NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
19100 19101 Professional developer Yes, I program as a hobby United States No Employed full-time Some college/university study without earning ... A humanities discipline Less than half the time, but at least one day ... 100 to 499 employees ... Disagree Male Some college/university study, no bachelor's d... White or of European descent Somewhat agree Somewhat agree Disagree Agree 110000.0 NaN
19101 19102 Professional developer Yes, I program as a hobby France No Employed full-time Master's degree Computer science or software engineering All or almost all the time (I'm full-time remote) 100 to 499 employees ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

19102 rows × 154 columns

1
2
# shape give us the number of rows and columns of the dataframe
df.shape

(19102, 154)

1
df.info()

<class ‘pandas.core.frame.DataFrame’>
RangeIndex: 19102 entries, 0 to 19101
Columns: 154 entries, Respondent to ExpectedSalary
dtypes: float64(6), int64(1), object(147)
memory usage: 22.4+ MB

1
2
3
# set the number of columns and rows displayed
pd.set_option('display.max_columns', 154)
pd.set_option('display.max_rows', 154)
python►c-pandas

Python-c01-Jupyter

How to install Jupyter on local machine?

Here we use docker to do the installation.

  1. install docker on the windows, in this example, we use docker on windows
    download the installer from https://docs.docker.com/desktop/install/windows-install/ and install it simply
    NB: you have to use the admin account
  2. open the terminal with admin and install the jupyter
    1
    docker run -p 8888:8888 -v C:\dev\softwares\docker_data\jupyter_data:/home/jovyan/work jupyter/scipy-notebook
  3. run jupyter in a browser
    You can find the url from the logs or from the desktop
  4. set the password
    you have to set a password for the jupyter, remember it as you have to type it for the next times
  5. the interface

Website Recommended

Share a website which you can view

  • headlines of major French media in real time
  • changes in the epidemic
  • markets, food delivery shops, drug stores, and banks that are open during the epidemic
  • some very useful links, you can enjoy online concerts for free, visit online museums
  • some rumor information, etc.

highly recommended http://daccfrance.com/

git►commands

Git Commands

Git modify history

We have 3 old commits pushed as following, we want to modify them and squash the last 2 commits

command:

1
git rebase -i HEAD~3

we got:

Change the last 2 commits to squash

1
2
Ech
:wq

Comment the last 2 commit messages

1
2
Ech
:wq
1
git push -f

And we can find that the commit history have been changed

gcp►c-data-architect►infrastructure-foundation

gcp-c01-vpc

4 ways to interact with GCP

Projects and networks

Project

  • Associate objects and services with billing
  • Contain networks(5 by default) that can be shared/peered

Network

  • Has no IP address range
  • Is global and spans all available regions
  • Contains subnetworks
  • Is available as default, auto or custom

3 VPC network types

The projects can communicate over internal IPs when they are in the same network even though they are in different regions.
They can not communicate over internal IPs when they are in different networks even though they are in the same region

Subnetworks cross zones

  • VMs can be on the same subnet but in different zones
  • A single firewall rule can apply to both VMs

Expand subnets

  • Cannot overlap with other subnets
  • Can expand but not shrink
  • Avoid large subnets

Internal and External Ip address

Every VM need an internal ip address but the external ip address is optional and by default it is ephemeral

gcp►b-data-engineer

b09-01-BigQuery Load From Json

How to copy dataset between different projects in BigQuery ?

1. BigQuery Data Transfer Service.

This is the simplest solution and with it, we can copy a dataset within a region or from one region to another, without
extracting, moving, and reloading data into BigQuery

2. Copy manually

If our BigQuery Data Transfer Service is not active, we can use another way to copy the table.
The steps are as following:

1.

python►a-basics

Python-a11-pythonic

This article shows the examples of pythonic in python

replace map, filter by list comprehension

1
2
3
4
5
6
7
8
# no pythonic
nums = [1, 2, 3, 4, 5]
result = map(lambda x: x ** 2, filter(lambda x: x % 2 == 0, nums))
print(list(result))     # [4, 16]

# pythonic
result2 = [x ** 2 for x in nums if x % 2 == 0]
print(list(result2))     # [4, 16]

replace list comprehension by generator expression

Generator expression takes less memory than list comprehension

1
2
3
4
5
6
# no pythonic
my_comp = [x * 5 for x in range(1000)]
print(getsizeof(my_comp))   # 8856
# pythonic
my_gen = (x * 5 for x in range(1000))
print(getsizeof(my_gen))    # 112

using of enumerate

1
2
3
4
5
6
7
8
9
10
11
12
# no pythonic
fruits = ['apple','banana', 'orange']
for i in range(len(fruits)):
    print(fruits[i], ':', i)    #   apple : 0
                                #   banana : 1
                                #   orange : 2
                                
# pythonic
for i, v in enumerate(fruits):
    print(v, ':', i)            #   apple : 0
                                #   banana : 1
                                #   orange : 2

use with

With the ‘with’ statement, we get better syntax and exception handing

1
2
3
4
5
6
7
8
9
# no pythonic
f = open('test.txt')
try:
    data = f.read()
finally:
    f.close()        # we have to close the file by ourself
# pythonic
with open('test.txt') as f:
    data = f.read() # python close the file for us

use zip

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# no pythonic
a = [1, 5, 7]
b = [2, 4, 6]
for i in range(len(a)):
    if a[i] > b[i]:
        print(a[i])
    else:
        print(b[i])         # 2
                            # 5
                            # 7
# pythonic
a = [1, 5, 7]
b = [2, 4, 6]
for i, j in zip(a, b):
    if i > j:
        print(i)
    else:
        print(j)            # 2
                            # 5
                            # 7

exchange two variables

1
2
3
4
5
6
7
8
9
10
11
# no pythonic
a = 'hello'
b = 'world'
a,b=b,a
print(a, b) # world hello

# pythonic
a = 'hello'
b = 'world'
a,b=b,a
print(a, b) # world hello
python►a-basics

Python-a10-Functional Programming

This article shows the examples of Functional Programming in python

function composition

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 1
def inner():
    print('this is the inner function')
    
def outer(func):
    func()
    
outer(inner)  # this is the inner function

# 2
def outer():
    def inner():
        print('this is the inner function')
    return inner

outer()() # this is the inner function

# sorted
animals = ["ferret", "vole", "dog", "gecko"]
print(sorted(animals, key=len)) # ['dog', 'vole', 'gecko', 'ferret']

lambda

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# callable Return whether the object is callable
print(callable(lambda s: s[::-1])) # True

# 2
reverse = lambda s: s[::-1]
print(reverse('This is a test')) # tset a si sihT<

# 3
print((lambda s: s[::-1])('This is a test'))  # tset a si sihT

# 4
print((lambda x, y, z: x + y + z / 3)(1, 2, 3)) # 4.0

# 5
def func(x):
    return x, x ** 2, x ** 3

print(func(3)) #(3, 9, 27)

# 6
print((lambda x: (x, x ** 2, x ** 3))(3)) #(3, 9, 27)

map

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# map returns an iterator
def reverse(s):
    return s[::-1]

animals = ["cat", "dog", "hedgehog", "gecko"]
iterator = map(reverse, animals)
print(iterator)         # <map object at 0x000001ADCF5CFCD0>
print(list(iterator)    # ['tac', 'god', 'gohegdeh', 'okceg']

# 2
print(list(map(lambda s: s[::-1], animals)))    # ['tac', 'god', 'gohegdeh', 'okceg']

# map with multiple iterables
def func(a, b, c):
    return a + b + c

print(list(map(func, [1, 2, 3], [10, 20, 30], [100, 200, 300])))    # [111, 222, 333]

filter

1
2
3
4
5
6
# 1
def greater_than_100(x):
    return x > 100

print(list(filter(greater_than_100, [1, 3, 5, 100, 200, 300])))     # [200, 300]

reduce

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 1
from functools import reduce

def add(x, y):
    return x + y

print(reduce(add, [1, 3, 5]))   # 9

# complex
def custom_map(function, iterable):
    from functools import reduce

    return reduce(
        lambda items, value: items + [function(value)],
        iterable,
        []
    )

print(list(custom_map(str, [2, 3, 3, 4, 5]))) # ['2', '3', '3', '4', '5']
python►b-OOP

Python-b05-Property Decorators

This article shows the examples of in python

email does not change when first changes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class Employee:
    raise_amount = 2.04

    def __init__(self, first, last):
        self.first = first
        self.last = last
        self.email = first + '.' + last + '@company.com'

    def fullname(self):
        return '{} {}'.format(self.first, self.last)


emp_1 = Employee('Bob', 'Gobdo')

emp_1.first = 'Jim'

print(emp_1.first)          # Jim
print(emp_1.email)          # Bob.Gobdo@company.com
print(emp_1.fullname())     # Jim Gobdo

email move into method

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
class Employee:
    raise_amount = 2.04

    def __init__(self, first, last):
        self.first = first
        self.last = last

    def email(self):
        return '{}.{}@company.com'.format(self.first, self.last)

    def fullname(self):
        return '{} {}'.format(self.first, self.last)


emp_1 = Employee('Bob', 'Gobdo')

emp_1.first = 'Jim'

print(emp_1.first)          # Jim
print(emp_1.email())        # Jim.Gobdo@company.com
print(emp_1.fullname())     # Jim Gobdo

add @property on method

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class Employee:
    raise_amount = 2.04

    def __init__(self, first, last):
        self.first = first
        self.last = last

    @property
    def email(self):
        return '{}.{}@company.com'.format(self.first, self.last)

    @property
    def fullname(self):
        return '{} {}'.format(self.first, self.last)


emp_1 = Employee('Bob', 'Gobdo')

emp_1.first = 'Jim'

print(emp_1.first)        # Jim
print(emp_1.email)        # Jim.Gobdo@company.com
print(emp_1.fullname)     # Jim Gobdo

setter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class Employee:
    raise_amount = 2.04

    def __init__(self, first, last):
        self.first = first
        self.last = last

    @property
    def email(self):
        return '{}.{}@company.com'.format(self.first, self.last)

    @property
    def fullname(self):
        return '{} {}'.format(self.first, self.last)

    @fullname.setter
    def fullname(self, name):
        first, last = name.split(' ')
        self.first = first
        self.last = last

emp_1 = Employee('Bob', 'Gobdo')

emp_1.fullname = 'John Smith'

print(emp_1.first)        # John
print(emp_1.email)        # John.Smith@company.com
print(emp_1.fullname)     # John Smith

deleter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
class Employee:
    raise_amount = 2.04

    def __init__(self, first, last):
        self.first = first
        self.last = last

    @property
    def email(self):
        return '{}.{}@company.com'.format(self.first, self.last)

    @property
    def fullname(self):
        return '{} {}'.format(self.first, self.last)

    @fullname.setter
    def fullname(self, name):
        first, last = name.split(' ')
        self.first = first
        self.last = last

    @fullname.deleter
    def fullname(self):
        print('Delete name')
        self.first = None
        self.last = None

emp_1 = Employee('Bob', 'Gobdo')

del emp_1.fullname

print(emp_1.first)        # None
print(emp_1.email)        # None.None@company.com
print(emp_1.fullname)     # None None