Getting started with Python

Reference resources   Cython introduction - cloud + Community - Tencent cloud

setup I met many times this year , With python The constant depth of programming learning is important for python The understanding of is also increasing , Here is a simple section .

Related tools ：distutils,cython

1.Cython brief introduction

We usually use python, Also called CPython, Because he uses C Written language , Generally speaking , our python Source code (.py Watson piece ), First compile it into bytecode (.pyc file ), And then .pyc Files in python Running on a virtual machine , there python Virtual machines are so-called “python Interpreter ”. To make a long story short , pure python The running speed of the code is between the traditional compilation language and the traditional interpretation language .

2.1 What is bytecode ？

Bytecode stay python The virtual machine program corresponds to PyCodeObject object ; .pyc File is the representation of bytecode on disk .

Every one with py At the end of the python Source code files are all modules , The file that can run the whole program after startup is called the top-level file . The top-level file is imported into other modules （ file ）, You must find the file and compile it into bytecode , And run bytecode .

The bytecode compiled when the file is imported is automatically saved , The timestamp is also saved . If they exist at the same time .py and .pyc,python Will use .pyc function , If .pyc The compilation time of is earlier than .py Time for , Then recompile .py, And update the .pyc file .

If python Unable to write bytecode on machine , The program still works , Bytecode is generated in memory and discarded at the end of the program .（ Strictly speaking , The bytecode is saved only when the file is imported , Not for top-level files ）.

If you want to generate test.pyc, We can use python Built-in module py_compile To compile the . You can also execute commands python -m test.py such , It was generated test.pyc.

typedef struct {
    PyObject_HEAD
    int co_argcount;        /*  Number of position parameters  */
    int co_nlocals;         /*  Number of local variables  */
    int co_stacksize;       /*  Stack size  */
    int co_flags;   
    PyObject *co_code;      /*  Bytecode instruction sequence  */
    PyObject *co_consts;    /*  All constant sets  */
    PyObject *co_names;     /*  A collection of all symbol names  */
    PyObject *co_varnames;  /*  Set of local variable names  */
    PyObject *co_freevars;  /*  Collection of variable names for closures  */
    PyObject *co_cellvars;  /*  A collection of variable names referenced by internally nested functions  */
    /* The rest doesn’t count for hash/cmp */
    PyObject *co_filename;  /*  The file name where the code is located  */
    PyObject *co_name;      /*  Module name | Function name | Class name  */
    int co_firstlineno;     /*  The starting line number of the code block in the file  */
    PyObject *co_lnotab;    /*  Correspondence between bytecode instruction and line number  */
    void *co_zombieframe;   /* for optimization only (see frameobject.c) */
} PyCodeObject;

2.2 Execution bytecode ？

Python The principle of virtual machine is to simulate executable program and then X86 Operation on the machine ,X86 The runtime stack frame of is shown in the figure below ： When a function call occurs , Create a new stack frame , Corresponding Python The realization of PyFrameObject object .PyFrameObject Object to create dynamic information when the program runs , That is, the execution environment , The relevant source code is as follows ：

typedef struct _frame{  
    PyObject_VAR_HEAD //" Runtime stack " The size of is uncertain   
    struct _frame *f_back; // Execute the previous in the environment chain frame, Many PyFrameObject Connected to form a linked list of execution environments   
    PyCodeObject *f_code; //PyCodeObject  object , This frame This is this. PyCodeObject Object context   
    PyObject *f_builtins; //builtin Namespace   
    PyObject *f_globals;  //global Namespace   
    PyObject *f_locals;   //local Namespace   
    PyObject **f_valuestack; //" Runtime stack " At the bottom of the stack   
    PyObject **f_stacktop;   //" Runtime stack " The top of the stack   
    //...  
    int f_lasti;  // The last bytecode instruction is in f_code Offset position in   
    int f_lineno; // The source code line corresponding to the current bytecode   
    //...  
      
    // Dynamic memory , maintain ( local variable +cell A collection of objects +free A collection of objects + Runtime stack ) The space needed   
    PyObject *f_localsplus[1];    
} PyFrameObject;

every last PyFrameObject Objects maintain a PyCodeObject object , This shows that every PyFrameObject The dynamic memory space objects in are the same as a section in the source code Code Corresponding .

2.2 What is? Cython？

Cython yes Python Language extension module , His aim is to be called python The superset of language (superset), by python Provide advanced , Object oriented , Functional and dynamic programming . Its main function is to support （ Optional ） Some static types are declared as Cython Part of the language . such cython The source code can be transformed into optimized C/C++ Code , You can then program this code as python Expansion module for .Cython Code in CPython Execute in the runtime environment , But in compiled C Speed execution of , And can directly call C library . meanwhile , It retains Python The original interface of the source code , This allows it to be directly from Python The code uses .

2.3 structure Cython

Cython The code must be compiled , It includes two steps ：

　　 First step , take A.pyx For documents Cython Compile to a .c In file , It contains python Extension module code

　　 The second step , take .c File compiled into .so file （Windows for .pyd file ）, This file can   direct import Get into Python session.Distutils or setuptools Responsible for this part . although Cython They can be called for you in some cases .

Specific examples , stay faser rcnn in ,bbox_overlaps Function is OK cython Written .

             

Specific construction Cython Several ways to ：

1. Write a distutils / setuptools setup.py.

setyp.py:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("helloworld.pyx")
)

Command line ：

$ python setup.py build_ext --inplace

2. Use Pyximport, Import Cython .pyx Files are like they are .py file （ Use distutils Compile and build in the background ）. This method is better than writing a setup.py It's easier . But not very flexible , such as , You need some compilation options .（ Actually , I don't need any compilation options without learning the principles of compilation , Obviously this method is excellent ）. however , When my computer uses this method , failed , May be VS Environmental reasons , So I used the first method , Directly compiled a pyd Files use .

import pyximport; pyximport.install()
import helloworld

# Hello World

　　　　3.Jupyter notebook Allow inline Cython Code . This is the beginning of writing Cython The easiest way to code and run it .

2.distutils brief introduction

except Cython Used in setup.py, To write python Third party library , It is also necessary to write setup.py Of . In fact, if we downloaded some source code files of the third Library , After opening, there is usually a setup.py, perform python setup.py install You can install this library .setup.py How to write a lot of content , Please refer to the official documents ：https://wiki.python.org/moin/Distutils/Tutorial?highlight=%28setup.py%29. A typical setup.py Let's write it as follows ( Refer to the official document ):

 The file structure is ：
top
|-- package
|   |-- __init__.py  #  Empty 
|   |-- module.py  #  This package Module files in 
|   `-- things
|       |-- cross.png
|       |-- fplogo.png
|       `-- tick.png
|-- README
`-- setup.py

setup.py：

from distutils.core import setup

#This is a list of files to install, and where
#(relative to the 'root' dir, where setup.py is)
#You could be more specific.
files = ["things/*"]

setup(name = "appname",
    version = "100",
    description = "yadda yadda",
    author = "myself and I",
    author_email = "[email protected]",
    url = "whatever",
    #Name the folder where your packages live:
    #(If you have other packages (dirs) or modules (py files) then
    #put them into the package directory - they will be found 
    #recursively.)
    packages = ['package'],
    #'package' package must contain files (see list above)
    #I called the package 'package' thus cleverly confusing the whole issue...
    #This dict maps the package name =to=> directories
    #It says, package *needs* these files.
    package_data = {'package' : files },
    #'runner' is in the root.
    scripts = ["runner"],
    long_description = """Really long text here.""" 
    #
    #This next part it for the Cheese Shop, look a little down the page.
    #classifiers = []     
) 

Making the installation tarball

python setup.py sdist

My compilation process ：

setup.py:

import sys
import numpy as np
A=sys.path.insert(0, "..")
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext

# ext_module = cythonize("TestOMP.pyx")
ext_module = Extension(
                        "bbox",
            ["bbox.pyx"],
            extra_compile_args=["/openmp"],
            extra_link_args=["/openmp"],
            )

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [ext_module],
    # Note that this sentence must have , Otherwise, it will only be compiled into C Code , Cannot compile to pyd file 
    include_dirs=[np.get_include()]
)