################################################################# py001_resources.txt ################################################################# Hello, This is the first issue of "Python Of The Day" - daily email to familiarize beginners with Python programming language. We will go slowly in small pieces day by day. Today's email lists resources. Main web site is python.org. It has the official tutorial http://docs.python.org/tutorial/ It is a good reference, but too detailed for a short intro. There are many tutorials (including video tutorials). just google python tutorial python video tutorial or look here: http://stackoverflow.com/questions/2675772/video-tutorial-for-learning-python Lists of links to numerous tutorials: http://wiki.python.org/moin/BeginnersGuide/Programmers http://wiki.python.org/moin/BeginnersGuide/NonProgrammers Official tutorial is here (but it is too long and detailed) - http://docs.python.org/tutorial/ I recommend this short book as a starter: "A Byte of Python" http://www.ibiblio.org/g2swap/byteofpython/read/index.html It is written by a unix shell/perl programmer who switched to Python. You can also get the latest version of this book here: http://www.swaroopch.com/notes/Python And there is also a version for python 3.0. But - I recommend to stay with the old version of the book. Because it is shorter, and because it is cut in small pieces. I also recommend this O'Reilly book: Python for Data Analysis - by Wes McKinney This book has the best tutorials for python, ipython, and numpy I have ever seen. More books online: Dive Into Python - http://www.diveintopython.net/ and you can also google and download this book: Beginning Python: From Novice to Professional, Second Edition # --------------------------------------------------------------- Note that this email series is based on python 2.7. I don't cover new version 3.0 becasue: - nobody is using it yet. - it is not backward compatible with versions 2.x # --------------------------------------------------------------- Getting and installing python # --------------------------------------------------------------- You can install a good version from here http://enthought.com/products/epd_free.php http://enthought.com/repo/free/ This version is available for Windows, Mac, and Linux. It includes ipython,scipy, numpy, and lots of other good stuff. http://www.enthought.com/products/epdlibraries.php Students can get even fuller version from enthought.com You can also install by downloading from python.org http://python.org/download/ For example, if you are a Mac user - http://www.python.org/getit/mac/ But this version will not have many additional packages which we need for analytical work. So I really recommend to go with enthought.com # --------------------------------------------------------------- Installation on unix Python is usually available on all unix boxes - just type: which python If you start python - it will show you the version - and give you the prompt for interractive session. To exit back to unix shell prompt, press ctrl-D. Unfortunately the python installed by default with the operating system is is usually old. We want a newer version - 2.7.x (for example, 2.7.1). If it is already available on network, for example at /opt/py27/bin then you can make it work by adding the following line to your .bashrc file: export PATH=/opt/py27/bin:$PATH # --------------------------------------------------------------- Installing pandas pandas stands for "Panel Data" http://pandas.pydata.org/ It is a python module which is great for analytical calculations. Installation: - install enthought python http://enthought.com/products/epd_free.php http://enthought.com/repo/free/ Then: on Windows - use binary installer from this page: http://pypi.python.org/pypi/pandas on Mac and unix - run those 2 commands (wait - let them finish) easy_install Cython easy_install pandas # --------------------------------------------------------------- When you have a question - jsut google it. Usually helps if you add the word stackoverflow, (because http://stackoverflow.com/ usually has the best answers). Or use some of these links: http://docs.python.org/library/functions.html http://docs.python.org/tutorial/stdlib.html http://docs.python.org/modindex.html http://effbot.org/ http://stackoverflow.com/ http://code.activestate.com/recipes/langs/python/ - Cookbook (recipes) http://gnosis.cx/publish/tech_index_cp.html - Charming Python # --------------------------------------------------------------- Warm Regards Lev ################################################################# py002_first_steps.txt ################################################################# Hello, python of the day - day 2 Today's topic: first steps with python. You will learn: - how to run your first python script, - how to work interactively - how to get help To get maximum benefit from the lesson, spend 5-10 min to actually try the commands by yourself. # --------------------------------------------------------------- In terminal window: which python python -h ... displays usage (different options and env. vars) python -V  Python 2.7.2 Note - use capital "V", because small "v" has different meaning ("verbose") which ipython ipython is an improved version of interractive python. I spend almost 100% of my python time using ipython. There are many GUI tools, but I find ipython much better as a productivity tool. http://ipython.org/ipython-doc/stable/interactive/tutorial.html # --------------------------------------------------------------- If you type python (or ipython) and press , you will get a prompt where you can work interractively. a=2 b=3 c=a+b print c 5 a = 'Hello ' + "World" print a Hello World a = a.upper() print a HELLO WORLD for ii in xrange(1,5): print ii 1 2 3 4 To exit back to the shell prompt, press Ctrl-D (or - if you are on MS Windows computer) On regular python prompt you can also do this: >>> import sys >>> sys.exit() If you are using ipython (which you should), then jsut type exit command # --------------------------------------------------------------- ipython commands: exit - to exit ipython TAB-completion - this is awesome feature ? - typing ? after a name will give info about it ?? - will print even more info (including source code defining this object) Example getting help in ipython: import string string. string ? string ?? string.strip ?? See some examples here: http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/ipython.html IPython "magic" commands are commands starting with %. Flag %automagic is usually set "ON" by default, allowing to call magic commands without the preceding %. So these 2 commands on ipython prompt will usually do the same thing run test.py %run test.py Other magic commands: %who - list names of objects, functions, etc. which were added in the current namespace %whos - includes lists type adn data info %whos function - list only functions %hist - list history of commands %hist -r - shows commands exactly as you have typed them Previous commands are stored in the list "In", outputs are stored in list "Out". exec In[37] will re-execute this specific line from history - While typing a command, press to see a list of previous commands with the same beginning Repeatedly pressing cycles through the list %macro - assignes a name to a set of several commands %edit somefile.py - opens file in the editor %edit func_name - opens file which has this function %lsmagic - lists all ipython magic commands. %store - stores variables, functions, etc. that you've defined in your .ipython/ipythonrc file (for use in future sessions) %pdb - configures ipython to automatically open the python debugger pdb when an error occurs %time and %timeit - to time functions (measure execution time) %logstart, %logon, %logoff, and %logstate - to log ipython input and/or output to files %cd, %pushd, %popd, and %bookmark - to change directories, manipulate directory stacks, and create directory "bookmarks" See more tips here: http://ipython.scipy.org/doc/manual/node4.html # --------------------------------------------------------------- Create file test.py with 3 lines in it like this: #!/path/to/your/python/bin/python # Filename : helloworld.py print 'Hello World' Now you can run it: $ python test.py Hello World If you make the file executable, you can run it like this: ./test.py Running it from python prompt: execfile("test.py") Running it from ipython prompt: run test.py # --------------------------------------------------------------- Importing modules - and showing what is inside import sys list(sys.modules.keys()) Listing available modules >>> help() help> modules Please wait a moment while I gather a list of all available modules... Note: inside help you often need to press 'q' to exit (or "quit" to exit help altogether) Listing builtin functions from interractive session >>> dir(__builtins__) # --------------------------------------------------------------- convenient alias to get history from ipython alias ihist='sqlite3 ~/.ipython/profile_default/history.sqlite "select * from history" > ~/hist_default.txt;sqlite3 ~/.ipython/profile_nexus/history.sqlite "select * from history" > ~/hist_nexus.txt;' # --------------------------------------------------------------- pdb module (debugger) & profile module (profiler - times spent in functions) import pdb pdb.run('Networks.FindPathLengthsFromNode(g, 0)') import profile profile.run('Networks.FindPathLengthsFromNode(g, 0)') # --------------------------------------------------------------- There is also local documentation "pydoc". I never use it because I usually use google. But here how to use it: Starting documentation GUI >>> import pydoc >>> pydoc.gui() Note - on Windows for this to work you will need to start Xwindos (Exceed) # --------------------------------------------------------------- Warm Regards Lev ################################################################# py003_ipython_notebook.txt ################################################################# Hello, If you installed enthought python, you can use ipython html notebook to do presentations with graphs. Open terminal window and execute the following command: ipython notebook --pylab=inline This should pop-up a browser window In it click on "New Notebook" - it open a new window. On the right you will have interactive prompt. Type there any command and press to execute it. See how it works. Here is a list of shortcuts for notebook  try them: Shift-Enter : run cell Ctrl-Enter : run cell in-place Ctrl-m d : delete cell Ctrl-m a : insert cell above Ctrl-m b : insert cell below Ctrl-m t : toggle output Ctrl-m l : toggle line numbers Ctrl-m s : save notebook Ctrl-m j : move cell down Ctrl-m k : move cell up Ctrl-m c : code cell Ctrl-m m : markdown cell Ctrl-m p : select previous Ctrl-m n : select next Ctrl-m i : interrupt kernel Ctrl-m . : restart kernel Ctrl-m h : show keyboard shortcuts To see an example of making a graph, copy/paste the following code and execute it. ############################## import numpy as np import matplotlib.pyplot as plt fig = plt.figure() ax1 = fig.add_subplot(111) t = np.arange(0.01, 10.0, 0.01) s1 = np.exp(t) ax1.plot(t, s1, 'b-') ax1.set_xlabel('time (s)') # Make the y-axis label and tick labels match the line color. ax1.set_ylabel('exp', color='b') for tl in ax1.get_yticklabels(): tl.set_color('b') ax2 = ax1.twinx() s2 = np.sin(2*np.pi*t) ax2.plot(t, s2, 'r.') ax2.set_ylabel('sin', color='r') for tl in ax2.get_yticklabels(): tl.set_color('r') plt.show() ############################## Many more examples of graphs with source code: http://matplotlib.sourceforge.net/gallery.html Warm Regards someuser someuser@domain.com | ################################################################# py004_gui_compile.txt ################################################################# Hello, Python of the day - day 4 - ipython, Eclipse, Emacs, etc., also how to compile your script ======================================== ipython - http://ipython.org/ ipython is a single greatest IDE for python. Very simple and intuitive. That's all you need for this email course. Even if you use emacs or Eclipse, you will probably have ipython session open in a separate window for short tests and for getting help. ======================================== Eclipse - probably the best Graphical IDE for paython - it has "pydev" plugin for python for all platforms (Windows, Mac, Linux). - google for installation instructions, for example: Mac Eclipse pydev - Find if you have 32 bit or 64 bit operating system - Download proper Eclipse from http://www.eclipse.org/downloads/ - Download pydev from http://pydev.org/download.html - follow installation instructions ======================================== Emacs - has python mode. ======================================== Other GUI IDE choices: Wing IDE (its commercial version). Spyder More (much more) GUI choices: - http://wiki.python.org/moin/GuiProgramming - http://wiki.python.org/moin/GUI%20Programming%20in%20Python ================================================= How to compile/test python script without running it. In perl we test scripts like this: perl -wc myscript.pl which means - turn on all warnings, compile only (don't run). ( w - warnings, c - compile only) Here are ways of doing similar things in python: python -m py_compile test.py This will compile the script into a file test.pyc in the same directory Another way of doing this: python -c "import py_compile; py_compile.compile('test.py')" or explicitly: python /you_path_to_python/py_compile.py test.py There are also special utilities one can use: pyflakes - http://pypi.python.org/pypi/pyflakes pychecker - http://pychecker.sourceforge.net/ pylint - http://www.logilab.org/857 Note: if you use a good GUI (for example Eclipse with python plugin) - it will report syntax errors as you edit. Package "compiler" - deprecated, doesn't exist in ver. 3. It is still available in ver. 2.7. Example of usage (in interactive session): compiler.compileFile(source) http://docs.python.org/library/compiler.html ================================================= Warm Regards Lev ################################################################# py005_script_structure.txt ################################################################# Hello, folks Python of the day - day 5 - the structure of a python script Simple example: #!/opt/py27/bin/python print "Hello" 1st line - "shebang" line - tells which python we want to use. After that lines of executable commands Each command usually on a separate line. Empty lines are OK Comments start with # We can create procedures (subroutines, functions) using the word "def". They should be placed before they are used. For example: #!/opt/py27/bin/python def func1(): print "I am in func1" print "I am in func1" print "I am in func1" def func2(): for ii in xrange(1,3): print "I am in func2" # now execute them func1() func2() # ------------------------------ In the above script we define 2 simple functions - and run them. Note: we use indentation to define blocks of commands which go together. No BEGIN/END markers are needed. Note: avoid using tabs, use only spaces for indentation. We can separate functions into another file (let's call it mylib.py) and then inport them into our script using "import" command. Homework: create 2 files: test.py and mylib.py as described below - and make it work Here are those files on my local hard drive (note shebang line of enthought python): ############################# # test.py ############################# #!/Library/Frameworks/Python.framework/Versions/Current/bin/python from mylib import * func1() func2() ############################# # mylib.py ############################# #!/Library/Frameworks/Python.framework/Versions/Current/bin/python def func1(): print "I am in func1" print "I am in func1" print "I am in func1" def func2(): print "I am in func2" print "I am in func2" print "I am in func2" Warm Regards someuser someuser@domain.com | ################################################################# py006_indentation.txt ################################################################# Hello, Python of the day - day 6 - indentation. In python indentation is used to create blocks of code (instead of curlies {} or begin/end keywords). Thus you can NOT arbitrarily add spaces at the beginning of the line - python will not compile. Also do not mix tabs and spaces. Better use spaces everywhere. # -------------------------------------- # Indentation example 1 # -------------------------------------- a=1 b=2 if a == b: print 'equal' elif a > b: print 'a > b' else: print 'a < b' print 'Done' # -------------------------------------- # Indentation example 2 # -------------------------------------- def sayHello(): print 'Hello World!' # block belonging to the function # End of function sayHello() # call the function # -------------------------------------- It is a good idea to put an empty line after the end of the block to show that the block ended. So code may look like this XXXXXXXXXXX XXXXXXXX XXXXXXXXXXX XXXXXXXX XXXXXXXXXXX XXXXXXXX # -------------------------------------- Exception: If your block of statements contains only one single statement, you can put it on the same line. Example: if flag: print 'Yes' This practice is discouraged. It is better to put the statement on a separate line (indented). Why? Because it is easier to debug, easier to add statements into the block, easier to use in interactive mode (in the above case after you enter the if-line and press , python gives you '....' prompt telling you that the statement is not complete. ################################################################# py007_data_types.txt ################################################################# Hello, Python of the day - day 7 - data types in python. Examples: a=5 along = 5L # long integer - unlimited precision b=1.23 c=9.25e-3 s1 = "String in single quotes" s2 = "String in double quotes is the same" s3 = """Use tripple-quotes for multiline text""" cn = -5+4j # complex number ================================================= Integers are implemented using Long precision (in C-language, at least 32 bit precision). Long integers have unlimited precision. float usually implemented as a double value (in C-language). You can see the implentation like this: import sys print sys.maxint 9223372036854775807 print sys.float_info sys.float_info(max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1) f=(-5+4j) # complex number f.real f.imag The constructors int(), long(), float(), and complex() can be used to produce numbers of a specific type. Some useful functions: abs(x) int(x) long(x) float(x) complex(re,im) # x % y # remainder of x / y x // y # (floored) quotient of x and y divmod(x, y) # the pair (x // y, x % y) math.trunc(x) # truncated to integral round(x[, n]) # x rounded to n digits, rounding half to even. If n is omitted, it defaults to 0. math.floor(x) # the greatest integral float <= x math.ceil(x) # the least integral float >= x ================================================= Strings - single quotes, double guotes, tripple quotes d = 'What\'s your name' e = "It's a string!" # double quoted string is the same as single quote text = """Use tripple quotes to create multiline string""" Note - backslash at the end means that string continues on next line - but no new line is added "Hello \ World" is equivalent to "Hello World" raw strings (escapes are not processed) - use them for regular expressions to avoid extra backslashes: r"Newlines are indicated by \n" unicode strings: u"This is a Unicode string." string concatenation - just put them side by side, or put plus sign between them: a = 'Hello' "World"; print a a = 'Hello'+"World"; print a ================================================= Example - printing documentation for functions and modules functions are usually documented by a multiline string which immediately follows the "def" line. You can access this string as "__doc__" attribute. Example: def rstrip(s, chars=None): """rstrip(s [,chars]) -> string Return a copy of the string s with trailing whitespace removed. If chars is given and not None, remove characters in chars instead. """ return s.rstrip(chars) Homework: run the following commands from ipython prompt - and look at the output: import string print string.__doc__ string.rstrip?? print string.rstrip.__doc__ ================================================= Warm Regards Lev ################################################################# py008_string.txt ################################################################# Hello, python of the day - day 8 - introduction to string operations. String literals can be written with single, double, or tripple quotes (tripple quotes are used for multiline strings) To concatenate strings together you can use spaces or "+" signs In [3]: a='s1' "s2" """s3""" In [4]: a Out[4]: 's1s2s3' In [5]: a='s1' + "s2" + """s3""" In [6]: a Out[6]: 's1s2s3' Or you can use a join str.join() string.join() (there will be examples below) ======================================= str vs string ======================================= str is a built-in class with many string functions available for any string object. string is a standard module with even more functions - you have to import it to use it. Here is discussion of the differences between them: http://stackoverflow.com/questions/2026038/relationship-between-string-module-and-str In [12]: str. str.capitalize str.isalnum str.lstrip str.split str.center str.isalpha str.mro str.splitlines str.count str.isdigit str.partition str.startswith str.decode str.islower str.replace str.strip str.encode str.isspace str.rfind str.swapcase str.endswith str.istitle str.rindex str.title str.expandtabs str.isupper str.rjust str.translate str.find str.join str.rpartition str.upper str.format str.ljust str.rsplit str.zfill str.index str.lower str.rstrip In [13]: import string In [14]: string. string.Formatter string.expandtabs string.replace string.Template string.find string.rfind string.ascii_letters string.hexdigits string.rindex string.ascii_lowercase string.index string.rjust string.ascii_uppercase string.index_error string.rsplit string.atof string.join string.rstrip string.atof_error string.joinfields string.split string.atoi string.letters string.splitfields string.atoi_error string.ljust string.strip string.atol string.lower string.swapcase string.atol_error string.lowercase string.translate string.capitalize string.lstrip string.upper string.capwords string.maketrans string.uppercase string.center string.octdigits string.whitespace string.count string.printable string.zfill string.digits string.punctuation ======================================= Below are some common examples. Try some of them. ======================================= http://docs.python.org/library/string.html string.replace(str, old, new[, maxreplace]) string.rstrip(s[, chars]) # kind of like chomp in perl string.strip(s[, chars]) # trims from both sides string.join(words[, sep]) Try this: import string string.strip?? s='\n\na\n\n';s=s.strip();print s+s string.rstrip?? string.join?? # example alist=["a","b","c"] print '|'.join(alist) a|b|c ================================= str() - Converting argument (usually a number) to a string s1 = 'abc'+str(123);print s1 ================================= Converting string into nubmers: a = 1 + int('2'); print a a = 1.1 + float('1.1'); print a ================================= Python doesn't have printf or sprintf functions, but you can use formats in print() in a similar fashion. s2 = "Hello, %s\n" % 'World!';print s2 or print "Hello, %s\n" % 'World!' s3="I have %d cats with in %g colors %s" % (5,3.5,'mama'); print s3 or print "I have %d cats with in %g colors %s" % (5,3.5,'mama') See many more string formatting operations here: http://docs.python.org/library/stdtypes.html#string-formatting Try this: a= {"language": "Python", "number": 2}; print a['language'] print '%(language)s has %(number)03d quote types.' % {"language": "Python", "number": 2} print '%(language)s has %(number)03d quote types.' % a ================================= REGEX - searching for patterns in strings (or searching/replacing) http://docs.python.org/library/re.html import re re.compile(pattern, flags=0) a = re.compile(r"""\d + # the integral part \. # the decimal point \d * # some fractional digits""", re.X) b = re.compile(r"\d+\.\d*") re.split(pattern, string, maxsplit=0, flags=0) re.search(pattern, string, flags=0) # like perl...re.match only starts from start of string ================================================= Some examples using built-in str class: ================================================= str.capitalize?? str.upper?? str.title?? str.index?? str.startswith?? str.rsplit?? str.strip?? etc. Example: a = ' mama ' # a is a string, so we can use 'str' methods on it b = a.replace('mama','papa') # ' papa ' name = 'Peter' if name.startswith('Pet'): print 'startswith - "Pet"' if 'te' in name: print 'in - "te"' if name.find('te') != -1: print 'name.find - "te"' delimiter = ', ' mylist = ['cat', 'dog', 'crocodile', 'elephant'] print delimiter.join(mylist) # Output: startswith - "Pet" in - "te" name.find - "te" cat, dog, crocodile, elephant ================================================= Warm Regards Lev ################################################################# py009_False_None_NaN.txt ################################################################# Hello, python of the day - day 9 - True and False, None, NaN Today we talk about True, False, None, NaN ======================================== Any object can be tested for truth value. # -------------------------------------- The following values are considered false: - None - False - zero of any numeric type, for example, 0, 0L, 0.0, 0j. - any empty sequence, for example, '', (), []. - any empty mapping, for example, {}. - instances of user-defined classes, if the class defines a __nonzero__() or __len__() method, when that method returns the integer zero or bool value False. All other values are considered true. (Even NaN (Not A Number) is true - see below.) # -------------------------------------- Operations and built-in functions that have a Boolean result always return: 0 or False for false 1 or True for true, unless otherwise stated (Important exception: the Boolean operations "or" and "and" always return one of their operands.) # -------------------------------------- # Boolean Operations: and, or, not # -------------------------------------- or - short-circuit operator, so it only evaluates the second argument if the first one is False. x or y -- returns value of x if it is not false. Otherwise returns value of y. and - short-circuit operator, so it only evaluates the second argument if the first one is True. x and y -- if x is false, then x, else y not - negation. It has a lower priority than non-Boolean operators, so not a == b is interpreted as not (a == b), a == not b is a syntax error. not x -- if x is false, then True, else False ================================================= # None - a keyword which is used to represent the absence of a value >> if (None) : print "True" >> if (not None) : print "False" False >> if (None == None) : print "Equal" Equal >> a = None; b=None; if (a == b) : print "Equal" Equal >> type(a) NoneType ================================================= # A variable may contain a number with a value NaN (Not A Number) # NaN is different from None. # Main feature - it is not equal to itself. # For testing, you can create NaN using different methods, for example: a = 1e400*0 b = a a == b # false, not equal to itself type(a) # float import numpy as np a = np.nan # or np.NaN b = a a == b | false type(a) # float # You can check for NaN using np.isnan(a) or math.isnan(a) function np.isnan(a) # True import math math.isnan(a) # True x=float('nan') # another way to create nan math.isnan(x) # True # pandas module has a isnull() function: a=np.nan pandas.isnull(a) # True # You can create your own function to test for NaN: def is_nan(num): return num != num # http://en.wikipedia.org/wiki/NaN # http://stackoverflow.com/questions/944700/how-to-check-for-nan-in-python # Note: floats are encoded in hardware like this, according to IEEE 754: # http://en.wikipedia.org/wiki/Single_precision_floating-point_format # Without going too much into it, one bit is for the sign, a small portion # go towards the exponent (the power you're raising to), and the majority of # the bits represent the mantissa (the base of the exponentiation # expression). Also remember this is all in base2. When either the # exponent or the mantissa is all 0 or all 1 (inf), things get crazy. # This also explains why there is no such thing as an # integer NaN - any sequence of bits can always encode an integer. ================================================= Warm Regards Lev ################################################################# py010_operators.txt ################################################################# Hello, Do you find 5 min per email to play with the examples in those emails ?? If you do, then in 30 days you will be fluent in Python. If you dont, in 30 days you will have many emails - and no skill. Do it - just 5 min per day. python of the day - day 10 - Operators Operatos mostly act the same as in other languages. Here is the list with some notes: + - adds 2 objects (for strings - concatenates them) - - minus * - multiply ** - power ( 3**2 == 9 ) / - divide // - floor division (4 // 3 == 1.0) % - Modulo (remainder) (8 % 3 == 2. , -1%3 == 2) << - left bit-shift (2<<2 == 8) >> - right bit-shift & - bitwise AND | - bitwise OR ^ - bitwise XOR ~ - bitwise invert < - less than (can chain: 3 < 5 < 7 gives True) > - greater than <= >= == - equal != - not equal (can also be written as <> - obsolete) not - Boolean NOT and - Boolean AND (short-circuit evaluation) or - Boolean OR (short-circuit evaluation) is - object identity is not - negated object identity You can see more operators - and table of their precedence in docs here: http://docs.python.org/reference/expressions.html#summary See also docs here: http://docs.python.org/library/stdtypes.html You will see operators, and also common math functions like: math.trunc(x) x truncated to Integral round(x[, n]) x rounded to n digits, rounding half to even. If n is omitted, it defaults to 0. math.floor(x) the greatest integral float <= x math.ceil(x) the least integral float >= x divmod(x, y) the pair (x // y, x % y) , pow(x, y), x ** y, log() abs() bin() float.hex() float.fromhex() etc. etc. Example: a=99999 print int.bit_length(a) 17 n=-37 print bin(n) '-0b100101' ================================================= Operators can be overloaded for objects of different types. For example, for standard iterable objects (like lists and strings) operators '+' and '*' do concatenation and repetition a1 = [1,2] * 3 # [1,2,1,2,1,2] a2 = [0] * 10000 # [0,0, ... 0,0] - list with 10,000 elements s1 = 'abc' * 3 # 'abcabcabc' s2 = 'x' * 10000 # # 'xxx ... xxx' - string fileld with 10,000 characters 'x' To overload operators for your own classes, you define member functions like these: __add__(self, other) __sub__(self, other) __mul__(self, other) __div__(self, other) __pow__(self, other[, modulo]) __floordiv__(self, other) __mod__(self, other) __and__(self, other) __xor__(self, other) __or__(self, other) __repr__(self) __str__(self) __lt__(self, other) __le__(self, other) __eq__(self, other) __ne__(self, other) __gt__(self, other) __ge__(self, other) etc. - see more here: http://docs.python.org/reference/datamodel.html ================================================= The backtick operator is deprecated and doesn't exist in Python 3.0. In Python 2.x it is basically a synonym for the repr() function, which converts its argument to a string suitable for a programmer to view. ================================================= Ternary operator in C : result = (a > b) ? r1 : r2; in python: result = r1 if (a > b) else r2 ================================================= operators for testing collection membership in not in http://docs.python.org/reference/expressions.html#notin Example: > a = [1,2,3] > b = 2 > if b in a: print 'contains' 'contains' > b = 5 > if b not in a: print 'not contains' > a = 'crocodile Michael' > b = 'crocodile' > if b in a: print 'contains', b Membership operators work for sequences (lists, tuples, strings), sets, keys of dictionaries, user-defined classes. For user-defined classes if __contains__() method is defined - it is used if __iter__() method is defined, then x in y is true if some value z with x == z is produced while iterating over y. if __getitem__(), x in y is true if and only if there is a non-negative integer index i such that x == y[i], and all lower integer indices do not raise IndexError exception. (If any other exception is raised, it is as if in raised that exception). The operator "not in" is defined to have the inverse true value of in. ================================================= Operators to check object identity: is is not x is y is true if and only if x and y are the same object. ================================================= More operators: += -= /= *= etc. For example a = 2 a *= 3 # 6 a = 'abc' a += 'def' # 'abcdef' ================================================= Warm Regards Lev ################################################################# py011_control_flow.txt ################################################################# Hello, Please spend 5 min right now to play with simple examples in this email. If you haven't done examples in previous emails - that's OK. Just do this one right now. Just 5 min. It is not enough to read those emails. You have to play with the examples - actually typing the lines of code on the ipython prompt If you spend 5 min for each email, then in 30 days you will be fluent in Python. If you dont, in 30 days you will have many emails - and no skill. Just do it - just 5 min per day. # -------------------------------------- python of the day - day 11 - Control Flow - if/else, for, while, break/continue Today you will play with the following keywords: if/elif/else for, while range, xrange break, continue # -------------------------------------- # example - if else # -------------------------------------- a = int(raw_input('Enter an integer : ')) b=2 c=3 if a == b: print 'equal to b' if a == b: print 'equal to b' else: print 'not equal' if a == b: print 'equal to b' elif a == c: print 'equal to c' else: print 'not equal' # -------------------------------------- # Note - there is no 'switch' statement in Python. # But you can easily emulate it. # http://simonwillison.net/2004/may/7/switch/ in PHP - C-like syntax: switch ($value) { case 'a': $result = $x * 5; break; case 'b': $result = $x + 7; break; case 'c': $result = $x - 2; break; } # In Python: if value == 'a': result = x * 5 elif value == 'b': result = x + 7 elif value == 'c': result = x - 2 # or result = { 'a': lambda : x * 5, 'b': lambda : x + 7, 'c': lambda : x - 2 }[value]() # You can also use the get method. It would be (not tested) {'option1': function1, 'option2': function2, 'option3': function3, 'option4': function4, }.get(value, defaultfunction)() # or for value in [value]: if value == 'a': result = x * 5 break if value == 'b': result = x + 7 break if value == 'c': result = x - 2 break # default result = x break # -------------------------------------- # Ternary operation # Ternary operation like this (in C, etc.) result = (a > b) ? r1 : r2; # has different syntax in Python: result = r1 if (a > b) else r2 # -------------------------------------- # example - while loop # -------------------------------------- number = 23 running = True while running: guess = int(raw_input('Enter an integer : ')) if guess == number: print 'Congratulations, you guessed it.' running = False # this causes the while loop to stop elif guess < number: print 'No, it is a little higher than that.' else: print 'No, it is a little lower than that.' else: print 'The while loop is over.' # Do anything else you want to do here print 'Done' # example while not v1 and v2 < 20: # do something # -------------------------------------- # example - for loop # -------------------------------------- for i in [0,1,2]: print i, for i in range(1, 5): print i, else: print 'The for loop is over' Note - python for loop is similar to foreach loop in perl (and other languages) Note - range() function returns a list of numbers. It can accept 1,2, or 3 arguments: range([start,] stop[, step]), where starts defaults to 0, step defaults to 1 range(5) # [0, 1, 2, 3, 4] range(2,5) # [2, 3, 4] range(2,5,2) # [2,4] Note that last index of the range (4 in our example) is less than stop = 5 Note: if the list is very big, then it is better to use xrange() instead of range(), because xrange() doesn't generate the whole list at once thus saving memory: http://docs.python.org/library/functions.html#xrange for i in xrange(0,999999): # do something # -------------------------------------- # example - break statement # -------------------------------------- while True: s = raw_input('Enter something : ') if s == 'quit': break print 'Length of the string is', len(s) print 'Done' # -------------------------------------- # example - continue statement (skip the rest of the statements in the block) # -------------------------------------- while True: s = raw_input('Enter something : ') if s == 'quit': break if len(s) < 3: continue print 'Input is of sufficient length' # Do other kinds of processing here... Warm Regards Lev ################################################################# py012_2_pandas_installation.txt ################################################################# Hello, folks python of the day - day 12 - 2nd email today - installing pandas pandas stands for "Panel Data" (I think) Author - Wes McKinney http://pandas.pydata.org/ Tutorial video (3 hrs 16 min) http://www.youtube.com/watch?v=w26x-z-BdWQ Link to download notebook used in pandas tutorial http://ow.ly/9w92O Other videos: http://www.youtube.com/watch?v=3vrCCjmVnIk 4 min http://www.youtube.com/watch?v=MxRMXhjXZos 56 min Docs: http://pandas.pydata.org/pandas-docs/stable/basics.html ================================================= Installing pandas Pandas is built on top of numpy and some other packages. The easiest way to install it is to start with enthought package, which already includes most of the prerequisites: http://enthought.com/products/epd_free.php http://enthought.com/repo/free/ Once you have enthought python, then: On Unix or Mac - run the following 2 commands: easy_install Cython easy_install pandas On Windows - use binary installer from this page: http://pypi.python.org/pypi/pandas Once pandas is installed, try this from ipython prompt: from pandas import * import pandas pandas. pandas.DataFrame. ================================================= ################################################################# py012_functions.txt ################################################################# Hello, Please spend 5 min right now to play with simple examples in this email. If you haven't done examples in previous emails - that's OK. Just do this one right now. Just 5 min. It is not enough to read those emails. You have to play with the examples - actually typing the lines of code on the ipython prompt If you spend 5 min for each email, then in 30 days you will be fluent in Python. If you dont, in 30 days you will have many emails - and no skill. Just do it - just 5 min per day. # -------------------------------------- python of the day - day 12 - Functions # -------------------------------------- # Functions are defined using the def keyword # -------------------------------------- def sayHello(): print 'Hello World!' # block belonging to the function # End of function sayHello() # call the function # -------------------------------------- # parameters / arguments # -------------------------------------- def printMax(a, b): if a > b: print a, 'is maximum' else: print b, 'is maximum' printMax(3, 4) # directly give literal values x = 5 y = 7 printMax(x, y) # give variables as arguments Note about parameters of functions: all objects are passed by reference, so when you change the object inside the function - it also changes outside. simple types (int, float, etc.) a passed "by value", so changing them inside doesn't change them outside. strings are passed by reference, but changing the string inside doesn't change it outside. You have to "return" the string from the function (or pass it inside of a container - for example in a list). See more details at the end of this email. # -------------------------------------- # local variables # -------------------------------------- def func(x): print 'x is', x x = 2 print 'Changed local x to', x x = 50 func(x) print 'x is still', x # -------------------------------------- # Using the global statement to tell that we are dealing with variable defined outside the function # -------------------------------------- def func(): global x,y print 'x is', x x = 2 print 'Changed global x to', x x = 50 y = 60 func() print 'Value of x is', x # -------------------------------------- # Using Default Argument Values # -------------------------------------- def say(message, times = 1): print message * times say('Hello') say('World', 5) # -------------------------------------- # Using Keyword (named) Arguments # -------------------------------------- def func(a, b=5, c=10): print 'a is', a, 'and b is', b, 'and c is', c func(3, 7) func(25, c=24) func(c=50, a=100) # -------------------------------------- # Using the return statement # -------------------------------------- def maximum(x, y): if x > y: return x else: return y print maximum(2, 3) Note: a return statement without a value is equivalent to return None. Every function implicitly contains a return None statement at the end unless you have written your own return statement. You can see this by running print someFunction() where the function someFunction does not use the return statement. None is a special type in Python that represents nothingness. pass indicate an empty block def someFunction(): pass print someFunction() # prints None # -------------------------------------- # Using DocStrings - a first logical line of a function (also apply to modules and classes). # -------------------------------------- def printMax(x, y): '''Prints the maximum of two numbers. The two values must be integers.''' x = int(x) # convert to integers, if possible y = int(y) if x > y: print x, 'is maximum' else: print y, 'is maximum' printMax(3, 5) print printMax.__doc__ help(printMax) Note: help() is a built-in function to print docstrings for functions, modules, etc. press "q" to exit help # -------------------------------------- objects are passed by reference, so it is easy to make a function to modify a list in place. For example: def modifyList( aList ): for i in range( len( aList ) ): aList[ i ] *= 2 a = [ 1, 2, 3, 4, 5 ] modifyList( a ) # [2, 4, 6, 8, 10] # -------------------------------------- simple types (int, float, etc) are passed by value. If you want function to change them - you need to wrap them into some object. # -------------------------------------- Strings is a strange case. They are not treated as a regular object. Strings are immutable in Python, so you cannot change them directly. You can pass them into a function and change/re-assign inside this function. But this will not change the string outside this function. You have to return the string to get it changed outside. This works: def mytrim(s): return s.strip() a = mytrim(a) But we can't change string "in place" without returning it. def mytrim(s): s=s.strip() print "in function >"+s+"<" a=' bbb ' mytrim(a) in function >bbb< a ' bbb ' # outside it is still unchanged # -------------------------------------- If you want to change a string "in place" - you need to put it inside of some object container (for example, a list): def mytrim(s): s[0]=s[0].strip() a=[' abc '] print 'before ',a # ' abc ' mytrim(a) print 'after ',a # 'abc' ====================================== Warm Regards Lev ################################################################# py013_modules.txt ################################################################# Hello, Please open ipython, copy/paste examples, spend 5 min. Just do it - just 5 min per day. # -------------------------------------- python of the day - day 13 - Modules Your script can use functions and variables defined in other python files. These other files are called modules. They usually have extension .py Python comes with a big library of modules. To see a list of all installed modules, on unix/dos prompt type: pydoc modules on ipython prompt type: import or: help('modules') You can create your own modules (decsribed below). You can import individual modules into your program to use their functionality, for example: import sys # this command tells python to load module sys.py # (or byte-compile sys.pyc file) # and to execute its 'main' block print 'The command line arguments are:' for i in sys.argv: print i print '\n\nThe PYTHONPATH is', sys.path, '\n' print sys.path # prints a list of paths where python searches for modules # first path is empty '' - this means 'current directory' Note: sys.argv[0] contains the name of the script, sys.argv[1] - 1st argument, etc. # -------------------------------------- # from ... import statement # -------------------------------------- from sys import argv # import just one object There is a difference between these 2 forms of importing: import sys # sys is imported completely, # you have to use "sys" prefix from sys import * # sys is imported AND names of its objects are imported # sys prefix is not required # (possible problem - name collisions) # -------------------------------------- # A module's __name__ # -------------------------------------- if __name__ == '__main__': print 'This program is being run by itself' else: print 'I am being imported from another module' # -------------------------------------- # Creating your own Modules # -------------------------------------- # Filename: mymodule.py """Simple test module Functions: sayhi() """ def sayhi(): """Function to test that module is working.""" print 'Hi, this is mymodule speaking.' version = '0.1' # End of mymodule.py # -------------------------------------- # Filename: mymodule_demo.py import mymodule mymodule.sayhi() print 'Version', mymodule.version # -------------------------------------- #!/usr/bin/python # Filename: mymodule_demo2.py from mymodule import sayhi, version # Alternative: from mymodule import * sayhi() # note - can use function without module prefix print 'Version', version # -------------------------------------- # using dir() function to list attributes of a module # -------------------------------------- >>> import sys >>> dir(sys) shows big list here >>> dir() # get list of attributes for current module ['__builtins__', '__doc__', '__name__', 'sys'] >>> a = 5 # create a new variable 'a' >>> dir() ['__builtins__', '__doc__', '__name__', 'a', 'sys'] >>> >>> del a # delete/remove a name >>> >>> dir() ['__builtins__', '__doc__', '__name__', 'sys'] # -------------------------------------- Warm Regards Lev ################################################################# py014_data_structures.txt ################################################################# Hello, Please spend 5 min right now to play with simple examples in this email. Go to terminal prompt, start ipython - and copy/paste the examples - see how things work. If you haven't done examples in previous emails - that's OK. Just do this one right now. Just 5 min. # -------------------------------------- python of the day - day 14 - Data Structures list [1,2,3] - mutable tuple (1,2,3) - immutable sequences (something with [index]) set (unique, unordered, mutable) dictionary - similar to hash in Perl http://docs.python.org/tutorial/datastructures.html # -------------------------------------- # list - class to create mutable lists # -------------------------------------- help(list) a = ['apple', 'mango', 'carrot', 'banana'] b = ['abc',123] print 'There are', len(a), 'items in a' for item in a: print item, # comma at the end prevents from printing a new-line symbol # empty line above useful when pasting code to ipython - it tells to remove indentation print '\n' # concatenating lists using "+" operation # Note - you should convert scalars, tupples or sets to list a = [1,2,3] b = [4,5,6] c = a + b + [7] + list((8,9)) print c [1, 2, 3, 4, 5, 6, 7, 8, 9] m = [None] * 1000 # creates a list of 1000 None's m = range(10) # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] m = range(1,10) # [1, 2, 3, 4, 5, 6, 7, 8, 9] # list() - method to create a list from any iterable object (like tuple, set, string) # - for example, when applied to a string, it splits the string into individual characters: a= [1,2,3] c = a + list("mama") # [1,2,3,'m','a','m','a'] c = a + list(["mama"]) # [1,2,3,"mama"] # indexing and slicing: print 'first element (with index 0) is:',a[0] del a[0] print 'I deleted first element:', a a1 = a.pop(2) # deleting (and returning) 3rd element a1 a= [1,2,3,4,5,6] a[2:2] = ['a','b','c'] a [1, 2, 'a', 'b', 'c', 3, 4, 5, 6] print 'methods append(), insert(), extend()' a.append('rice') a.extend([1,2,3]) a.insert(2,'mama') # sorting a.sort() # sorts list in place print 'a now is sorted:', a # -------------------------------------- # tuple (round parentheses) - class to create immutable lists (you can not modify tuples) # -------------------------------------- Tuples are just like lists except that they are immutable like strings i.e. you cannot modify tuples. help(tuple) zoo = ('wolf', 'elephant', 'penguin') print 'Number of animals in the zoo is', len(zoo) new_zoo = ('monkey', 'dolphin', zoo) print 'Number of animals in the new zoo is', len(new_zoo) print 'All animals in new zoo are', new_zoo print 'Animals brought from old zoo are', new_zoo[2] print 'Last animal brought from old zoo is', new_zoo[2][2] print 'Fist 2 animals in new zoo are %s, %s' % (new_zoo[0],new_zoo[1]) output: Number of animals in the zoo is 3 Number of animals in the new zoo is 3 All animals in new zoo are ('monkey', 'dolphin', ('wolf', 'elephant', 'penguin')) Animals brought from old zoo are ('wolf', 'elephant', 'penguin') Last animal brought from old zoo is penguin Fist 2 animals in new zoo are monkey, dolphin Note for Perl programmers A list within a list are not flattened as in Perl. The same applies to a tuple within a tuple, or a tuple within a list, or a list within a tuple, etc. As far as Python is concerned, they are just objects stored using another object, that's all. Note: To specify a tuple with one element, add a comma: ('a',) Without this comma it will be treated simply as parentheses around something. # -------------------------------------- # dictionary - dict class # -------------------------------------- a dictionary is a set of key-value pairs d = {'key1' : value1, 'key2' : value2 } accessing element by key: d['key1'] ab = { 'Swaroop' : 'swaroopch@byteofpython.info', 'Larry' : 'larry@wall.org', 'Matsumoto' : 'matz@ruby-lang.org', 'Spammer' : 'spammer@hotmail.com' } print "Swaroop's address is %s" % ab['Swaroop'] # Adding a key/value pair ab['Guido'] = 'guido@python.org' # Deleting a key/value pair del ab['Spammer'] # or ab.pop('Spammer') if 'Guido' in ab: # OR ab.has_key('Guido') print "\nGuido's address is %s" % ab['Guido'] # len() function: print '\nThere are %d contacts in the address-book\n' % len(ab) # iterating through ha dictionary for name in ab: print name + ' => ' + ab[name] for name, address in ab.items(): print 'Contact %s at %s' % (name, address) # syntax above causes python to first generate a big list of keys (or keys/values). # For big dictionaries this creates a delay and consumes resources. # It is better to use iterator syntax to avoid building this list. # Iterator is a construct to generate and provide one element at a time as needed for key, value in my_dict.iteritems(): print key, value for key in my_dict.iterkeys(): print key for value in my_dict.itervalues(): print value # -------------------------------------- dictionary can be key-ed by a number, a string, a boolean, or even by a tuple (but not by a list): d={('k1','k2'):'v1'} d['k3']='v3' print d {'k3': 'v3', ('k1', 'k2'): 'v1'} d[('k4','k5')] = 'v4' print d {'k3': 'v3', ('k1', 'k2'): 'v1', ('k4', 'k5'): 'v4'} # -------------------------------------- # sequences # -------------------------------------- Sequences are things that can be indexed and sliced. Lists, tuples and strings are examples of sequences. Here are examples of indexing / slicing [i1:i2] # i1, i1+1, ... i2-1 (does NOT include i2) [i:] # i, i+1, ... to the end [:i] # 0,1,2,... i-1 (does NOT include i) [:] # everything [i] # returns list with just one element [i:i] # returns [] empty list (i:i indicates empty position right after i) - splicing, append/pop, shift, unshift? a=[1,2,3] del a[1] a.insert(1,7) print a [1,7,3] # splicing # splicing syntax is like [s:e] # google "Python splicing". Very powerful feature. Widlely used. a = [1,2,3,4,5] a[2:3] [3] a[2:5] [3, 4, 5] # -------------------------------------- # array - module to create arrays # -------------------------------------- # array is smilar to list, but all elements must be of the same type. # Type is indicated with a type character: import array myarr1 = array('l') # empty array of signed long values myarr2 = array('c', 'hello world') # array of characters myarr3 = array('u', u'hello \u2641') # array of myarr4 = array('l', [1, 2, 3, 4, 5]) myarr5 = array('d', [1.0, 2.0, 3.14]) myarr5.append(8) print myarr5[1] # 2.0 # See many array methods here: # http://docs.python.org/library/array.html # -------------------------------------- # set # -------------------------------------- # set is unordered collection of unique elements # http://docs.python.org/library/stdtypes.html#set # # Common usage - to remove duplicates a = [1,2,3,3,4,1] a2=list(set(a)) # returns [1,2,3,4] if len(a) == len(a2): print 'List is unique!' # http://www.siafoo.net/article/52 # attempt to add non-unique element to a set is simply ignored. s = set() s.add(1) s.add(2) print s s.add(1) print s # -------------------------------------- # references and copying # -------------------------------------- In python structures are objects, and refered by references. An assignment doesn't copy the object, it only copies the reference. One way to create a copy of any sequence (list,string, etc.) is to use a slice. Here is an example: print 'Simple Assignment' shoplist = ['apple', 'mango', 'carrot', 'banana'] mylist = shoplist # mylist is just another name pointing to the same object! del shoplist[0] # I purchased the first item, so I remove it from the list print 'shoplist is', shoplist print 'mylist is', mylist # notice that both shoplist and mylist both print the same list without # the 'apple' confirming that they point to the same object print 'Copy by making a full slice' mylist = shoplist[:] # make a copy by doing a full slice del mylist[0] # remove first item print 'shoplist is', shoplist print 'mylist is', mylist # notice that now the two lists are different Another way to copy is to use standard module "copy" http://docs.python.org/library/copy.html import copy y = copy.copy(x) # a shallow copy of x. y = copy.deepcopy(x) # a deep copy of x. exception copy.error # Raised for module specific errors # ------------------------------------- Warm Regards Lev ################################################################# py015_filter_map_reduce.txt ################################################################# Hello, Please spend 5 min right now to play with simple examples in this email. Go to terminal prompt, start ipython - and copy/paste the examples - see how things work. If you haven't done examples in previous emails - that's OK. Just do this one right now. Just 5 min. # -------------------------------------- python of the day - day 15 - filter(), map(), reduce() http://docs.python.org/tutorial/datastructures.html#functional-programming-tools # -------------------------------------- filter(function, sequence) works similar to unix "grep" command returns a sequence consisting of those items from the input sequence for which function(item) is true. Example: def f(x): return x % 2 != 0 and x % 3 != 0 a = filter(f, range(2, 25)) # [5, 7, 11, 13, 17, 19, 23] Note: returns string for string, tuple for tuple, list for anything else # -------------------------------------- map(function, sequence) execute function for each element of a sequence - and returns the result def cube(x): return x*x*x a = map(cube, range(1, 5)) # [1, 8, 27, 64, 125] Note: More than one sequence may be passed. Example - adding 2 lists item by item: a = [1,2,3] b = [4,5,6] def myadd(x, y): return x+y a = map(myadd, a, b) # [5, 7, 9] # -------------------------------------- reduce(function, sequence) - returns a single value applies function to first 2 elements, then to the result and next element, etc. For example, to compute the sum of the numbers 1 through 10: def add(x,y): return x+y reduce(add, range(1, 11)) # 55 Note: If there’s only one item in the sequence, its value is returned; If the sequence is empty, an exception is raised. Note: A third argument can be passed to indicate the starting value. In this case the starting value is returned for an empty sequence, and the function is first applied to the starting value and the first sequence item, then to the result and the next item, and so on. For example, def mysum(seq): def add(x,y): return x+y return reduce(add, seq, 0) mysum(range(1, 11)) # 55 mysum([]) # 0 Note: python has its own built-in function sum(sequence). a= [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] sum(a) # 55 # -------------------------------------- Warm Regards Lev ################################################################# py015_ipython_notebook.txt ################################################################# Hello, Python of the day - day 15 - ipython notebook Hello, People have asked me about this command: ipython notebook --pylab=inline ipython is an application with many features. Mostly it is known for being an interactive shell for python. notebook is one of subcommands of ipython --pylab is an option of notebook subcommand. It makes it possible to create graphics using commands like plot(range(10)) Execute the following command from unix/dos prompt: ipython --help and you will see that there are several possible subcommands: profile - Create and manage IPython profiles. kernel - Start a kernel without an attached frontend. notebook - Launch the IPython HTML Notebook Server. console - Launch the IPython terminal-based Console. qtconsole - Launch the IPython Qt Console. To see help on using "notebook" subcommand, run this command: ipython notebook --help Note: if you run ipython notebook without --pylab=inline, then graphics commands will not work. To learn more about pylab: http://www.scipy.org/PyLab To learn more about matplotlib: http://matplotlib.sourceforge.net/ This is docs for ipython notebook: http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html try for example these commands from ipython notebook - and see the graphs: y=[x*x for x in range(0,20)] ; plot(y) y=randn(100);plot(y) y=randn(10000);hist(y, 100) Warm Regards Lev ################################################################# py016_2_ipython_notebook_stdin.txt ################################################################# Hello, python of the day - day 16 2nd email - ipython html notebook stdin problem. Try to run this line in ipython html notebook (in the browser): line = raw_input('Enter an integer : ') You will see that it fails with the error like this: StdinNotImplementedError: raw_input was called, but this frontend does not support stdin. If you run code from ipython terminal prompt - everything works fine. The error only exists when you run ipython notebook in the browser. https://github.com/ipython/ipython/issues/1268/ In fact, any function which tries to get stdin will not work. For example, you can try this magic ipython command: %reset # -------------------------------------- Warm Regards Lev ################################################################# py016_dictionary_loops.txt ################################################################# Hello, Please spend 5 min right now to play with simple examples in this email. Go to terminal prompt, start ipython - and copy/paste the examples - see how things work. If you haven't done examples in previous emails - that's OK. Just do this one right now. Just 5 min. # -------------------------------------- python of the day - day 16 - looping through a dictionary http://yuji.wordpress.com/2008/05/14/python-basics-of-python-dictionary-and-looping-through-them/ my_dict = { 'key1': 'value1', 'key2': 'value2', 'key3': 'value3' } print my_dict['key1'] # Out: 'value1' Interesting that Python itself uses dictionaries to store variables. Whenever you declare a variable (x=3), that variable is accessible by its string name ('x') via the dictionary returned by the builtin function 'locals()' or 'vars()'. Try this: a = 'some value' # we define a local variable locals_dict = locals() # get all local variables into a dictionary print locals_dict['a'] # Out: some value # -------------------------------------- # looping d = {'k1':'v1','k2':'v2','k3':'v3'} for k in d: print k for k in d.keys(): print k # k3 # k2 # k1 for k in d.values(): print k # v3 # v2 # v1 for k in d: print d[k] # v3 # v2 # v1 for k in d.items(): print k # ('k3', 'v3') # ('k2', 'v2') # ('k1', 'v1') for k, v in d.items(): print k,v # k3 v3 # k2 v2 # k1 v1 # -------------------------------------- # more efficient method - using "iter" syntax to avoid building the whole list for k, v in d.iteritems(): print k, v for k in d.iterkeys(): print k for v in d.itervalues(): print v # -------------------------------------- # you can't delete an item during iteration for x, y in d: del d[x] # Out: RuntimeError: dictionary changed size during iteration # -------------------------------------- # sorting # dictionary can't be sorted. But we can make a list of tuples - and sort this list # Next 2 examples use special syntax (list comprehension, lambda) which will be explained in future emails. # I uncluded them here because they are typical mylist = [x for x in d.iteritems()] mylist.sort(key=lambda x: x[0]) # sort in place by key mylist.sort(key=lambda x: x[1]) # sort in place by value # to reverse the sort (in place) mylist.reverse() mylist mylist.reverse() mylist # -------------------------------------- # checking if a key exists if 'key1' in d: print "blah" else: print "boo" # another way: if d.has_key('k1'): print "Hurraa" # -------------------------------------- # accessing a key which doesn't exist d = {'key1':'value1'} d.get('key1') # out: 'value1' d.get('key2') # out: None d.get('key2', "Key doesn't exist") # out: Key doesn't exist # -------------------------------------- # Get or insert into a dictionary if key doesn’t exist d= dict() d[key] = d.get(key,0)+1 # increases value by one. #If it didn't exist - first creates it with zero value. # another way - using defaultdict() from the collections module # defaultdict() function accepts a method which will be used to assign default value. from collections import defaultdict d = defaultdict(lambda: 0) d['key'] += 1 # another way - using setdefault() d = {} d.setdefault('key1', 'value1') # Out: value1 print d # Out: { 'key1': 'value1' } # This can be extremely useful if you need to append # to a list if it exists or otherwise create a blank list. key_value_pairs = [ ('key1', 'value'), ('key1', 'value2'), ('key1', 'value3'), ] d = {} for key, value in key_value_pairs: d.setdefault(key, []).append(value) print d # Out: { 'key1': ['value','value2','value3'] } # -------------------------------------- # Generate a dictionary from a list of tuples using dict() function: key_value_pairs = [ ('k1', 'v1'), ('k2', 'v2'), ('k3', 'v3'), ] d = dict(key_value_pairs) print d # Out: {'k1': 'v1', 'k2': 'v2', 'k3': 'v3'} # -------------------------------------- Warm Regards Lev ################################################################# py017_ed_in_ipython.txt ################################################################# Hello, python of the day - day 17 - 2nd email - "ed" command in ipython How to edit a multi-line snippet of code in ipython? 1st way to edit: use up-arrow to get the multi-line command, then use left and right arrows (and ctrl-left, ctrl-right for faster movement) to move in your multiline code. Note: on a Mac ctrl-arrow can be programmed to switch between desktop spaces If this is the case, I recommend to go here: Apple Menu > System Preferences > Expose & Spaces > Spaces and change the keys used to switch between spaces (use Apple-arrow instead of ctrl-arrow) 2nd way to edit: use "ed" (or "edit") command from ipytohn prompt to open/reopen a previous command (by command [in] or [out] number) Examples of usage: ed ed 55 # edit and execute command entered on line 55 (also refered to as In [55]) ed _65 # edit and execute Out [65] ed -x _65 # edit only (no execution) of Out [65] ed -x _ # edit only (no execution) of the last "ed" edit ed junk # edit file "junk" if it exists Note: if you give a parameter to "ed", it will try to interpret it as an edit number or a file. If file doesn't exist - it will NOT create it. If you don't provide a parameter - it will open editor - and save your edits into a new file # -------------------------------------- Warm Regards Lev ################################################################# py017_script.txt ################################################################# Hello, python of the day - day 17 - small demo script Please spend 5 min right now to implement the script below. Note - raw_input command will not work in ipython notebook. You have to copy/paste the code into a file with .py extension, enter real paths and select propper command for compressing files. Also add propper shebang line at the top (if you are on unix or Mac). Make it work. Become proud of yourself. In case of problems - use google to figure out what is wrong. # -------------------------------------- But before that, start ipython - and type this command: import this Read the output. Enjoy. # -------------------------------------- #!/Library/Frameworks/Python.framework/Versions/Current/bin/python #!/opt/py27/bin/python """ script to create a dated backup """ import os, time source = ['/path/to/src/dir', '/another/path'] # list of directories to backup target_dir = '/path/to/backup/dir/' today = target_dir + time.strftime('%Y%m%d') now = time.strftime('%H%M%S') comment = raw_input('Enter a comment --> ') if len(comment) == 0: # no comment was entered target = today + os.sep + now + '.zip' else: target = today + os.sep + now + '_' + \ comment.replace(' ', '_') + '.zip' if not os.path.exists(today): os.mkdir(today) print 'Successfully created directory', today cmd = "zip -qr '%s' %s" % (target, ' '.join(source)) # cmd = 'tar -cvzf %s %s -X /somepath/excludes.txt' % (target, ' '.join(source)) if os.system(cmd) == 0: print 'Successful backup to', target else: print 'Backup FAILED' # -------------------------------------- Warm Regards Lev ################################################################# py018_classes_objects.txt ################################################################# Hello, Please spend 5 min right now to play with simple examples in this email. Go to terminal prompt, start ipython - and copy/paste the examples - see how things work. If you haven't done examples in previous emails - that's OK. Just do this one right now. Just 5 min. # -------------------------------------- python of the day - day 18 - classes and objects Python is an object-oriented (O.O.) language. Almost everything is an object in Python. You can easily create your own classes (also referred to as "types") and create objects of those types/classes. In many cases I prefer to use an object instead of a dictionary simply because it makes code easier to read. Compare: bag.df # using object bag['df'] # using dictionary Below I provide simple examples of OO syntax in Python. (see end of this email for some definitions and links) # -------------------------------------- class Person: def sayHi(self): print 'Hello, how are you?' p = Person() p.sayHi() # -------------------------------------- # self - reference to the object # __init__ - method will run as soon as an object is instantiated. # __del__ - garbage collection - no guarantee when it will actually happen # -------------------------------------- class Person: def __init__(self, name): self.name = name def sayHi(self): print 'Hello, my name is', self.name p = Person('John') p.sayHi() # -------------------------------------- # class variables and object variables # -------------------------------------- Class variables are shared by all objects (instances) of that class. Object variables are owned by each individual object/instance of the class. All class members (including the data members) are public. All methods are virtual. One exception: If you use data members with names starting with 2 underscores, then Python uses name-mangling to effectively make it a private variable. class Person: '''Represents a person.''' population = 0 # class variable def __init__(self, name): '''Initializes the person's data.''' self.name = name # object variable print '(Initializing %s)' % self.name # When this person is created, he/she # adds to the population Person.population += 1 def __del__(self): '''I am dying.''' print '%s says bye.' % self.name Person.population -= 1 if Person.population == 0: print 'I am the last one.' else: print 'There are still %d people left.' % Person.population def sayHi(self): '''Greeting by the person. Really, that's all it does.''' print 'Hi, my name is %s.' % self.name def howMany(self): '''Prints the current population.''' if Person.population == 1: print 'I am the only person here.' else: print 'We have %d persons here.' % Person.population john = Person('John') john.sayHi() john.howMany() gary = Person('Gary Halbert') gary.sayHi() gary.howMany() john.sayHi() john.howMany() # -------------------------------------- # Inheritance # -------------------------------------- Below is a simple example. Note - multiple inheritance allowed (simply specify more than one parent class in parenthesis). Note - Python doesn't automatically call the constructor of the base class - you have to call it explicitly. Note - as functions are virtual, you can apply method tell() in the axample below to both teachers and students, and correct method will be called. # Filename: inherit.py class SchoolMember: '''Represents any school member.''' def __init__(self, name, age): self.name = name self.age = age print '(Initialized SchoolMember: %s)' % self.name def tell(self): '''Tell my details.''' print 'Name:"%s" Age:"%s"' % (self.name, self.age), class Teacher(SchoolMember): '''Represents a teacher.''' def __init__(self, name, age, salary): SchoolMember.__init__(self, name, age) self.salary = salary print '(Initialized Teacher: %s)' % self.name def tell(self): SchoolMember.tell(self) print 'Salary: "%d"' % self.salary class Student(SchoolMember): '''Represents a student.''' def __init__(self, name, age, marks): SchoolMember.__init__(self, name, age) self.marks = marks print '(Initialized Student: %s)' % self.name def tell(self): SchoolMember.tell(self) print 'Marks: "%d"' % self.marks t = Teacher('Mrs. Margaret Thatcher', 40, 30000) s = Student('John', 22, 75) print # prints a blank line members = [t, s] for member in members: member.tell() # works for both Teachers and Students Output $ python inherit.py (Initialized SchoolMember: Mrs. Margaret Thatcher) (Initialized Teacher: Mrs. Margaret Thatcher) (Initialized SchoolMember: John) (Initialized Student: John) Name:"Mrs. Margaret Thatcher" Age:"40" Salary: "30000" Name:"John" Age:"22" Marks: "75" # -------------------------------------- Note: Variables declared inside the class definition, but not inside a method, are called class or static variables # -------------------------------------- Here is a short list of common terms used in Object Oriented programming: class - a construct which serves as a template for creating objects (instances of this class). http://en.wikipedia.org/wiki/Class_(computer_science) object - an instance of some class http://en.wikipedia.org/wiki/Object_(computer_science) member - any contents of a class: member variables (also called properties or data fields) attributes (metadata - http://en.wikipedia.org/wiki/Attribute_(computing) ) member methods - functions defined in a class which usually have access to member variables inner classes (classes defined inside other classes) static methods - defined on the class level, do not refer any instance http://en.wikipedia.org/wiki/Method_(computer_science) static variables - (class variables) defined on the class level. Only a single copy of this variable exists regardless of how many objects were created (if any). http://en.wikipedia.org/wiki/Class_variable constructor - method to craete an instance of the class (an object). Usually has the same name as the class. destructor - method to clean out the object when it is no longer needed inheritance - a way to create sub-classes http://en.wikipedia.org/wiki/Inheritance_(object-oriented_programming) multiple inheritance - inherit behavious and features from several classes http://en.wikipedia.org/wiki/Multiple_inheritance polymorphism - a way to allow functions, containers, and other constructs to work on any object as long as it has certain interface. For example, the function is defined to work with any Fruite, so it can work on objects of types "Apple", "Pear", etc. http://en.wikipedia.org/wiki/Type_polymorphism virtual function - a method which can be overridden within a child class. http://en.wikipedia.org/wiki/Virtual_function private, public, protected - levels of protecting member variables and methods http://en.wikipedia.org/wiki/Encapsulation_(object-oriented_programming) See more terms here: http://en.wikipedia.org/wiki/List_of_object-oriented_programming_terms # -------------------------------------- Warm Regards Lev ################################################################# py019_file_in_out.txt ################################################################# Hello, Please spend 5 min right now to play with simple examples in this email. Go to terminal prompt, start ipython - and copy/paste the examples - see how things work. If you haven't done examples in previous emails - that's OK. Just do this one right now. Just 5 min. # -------------------------------------- python of the day - day 19 - file in/out http://docs.python.org/tutorial/inputoutput.html # -------------------------------------- mytext = """ some text """ f = file('myfile.txt', 'w') # open for writing f.write(mytext) # mytext may be a multiline string f.close() # -------------------------------------- f = file('myfile.txt') # if no mode is specified, 'r'ead mode is assumed by default while True: line = f.readline() if len(line) == 0: # Zero length indicates EOF break print line, # Notice comma to avoid automatic newline added by Python f.close() # -------------------------------------- # Note - possible modes # r - read # r+ - read and write # w - write # a - append # -------------------------------------- # using r+ # http://docs.python.org/tutorial/inputoutput.html f = open('/tmp/workfile', 'r+') f.write('0123456789abcdef') f.seek(5) # Go to the 6th byte in the file f.read(1) # 5 f.seek(-3, 2) # Go to the 3rd byte before the end f.read(1) # 'd' # -------------------------------------- # shorter ways of reading a text file # -------------------------------------- f = open('/tmp/workfile', 'r+') # r+ allows you to both read and write for line in f: print line, words = line.strip().split() print words f.close() # I've even seen it like this (it automatically closes): for line in open('/tmp/workfile', 'r+'): print line, print; Here is a simple working example - reading a column of numbers from a text file for line in open('data.txt'): ss = line.strip() if ss: ff=float(line.strip()) print ff # -------------------------------------- # example - reading pipe-separated file into a dictionary # to test, first create a text CSV file 'test'csv' with following lines: # h1|h2|h3|h4|h5|h6 # 8|2|mama|papa|4.5|5e2 # 6|7|cook|sook|7.3|333 fh = file('test.csv', 'r') lines = fh.read().split('\n')[1:-1] # read the whole file, split into lines, skip 0th row (header) print lines # ['8|2|mama|papa|4.5|5e2', '6|7|cook|sook|7.3|333'] mydict = {} for line in lines: v1,v2,v3 = line.split('|')[:3] # take only columns 0,1,2 mydict[(v1,v2)] = v3 print mydict # {('6', '7'): 'cook', ('8', '2'): 'mama'} # -------------------------------------- Of course in real life you don't have to do splitting yourself like in the above example. You can use 'csv' module which methods to read/write csv files. http://docs.python.org/library/csv.html http://www.linuxjournal.com/content/handling-csv-files-python import csv # reading CSV file myreader = csv.reader(open('test.csv', 'rb'), delimiter='|', quotechar='"') for row in myreader: print row # row is a python list of cell values # saving list of lists into CSV file fh = open('out.csv','wb') mywriter=csv.writer(fh,delimiter='|', quotechar='"', quoting=csv.QUOTE_MINIMAL) # mywriter.writerow([1,2,3]) for row in my_list_of_rows: mywriter.writerow(row) fh.close() # Note how we create separate file handle fh to be able to explicitly close the file. # Another way to do the same is to use "with" block - fh will close automatically on exiting this block: with open('out.csv', 'w') as fh: mywriter = csv.writer(fh,delimiter='|', quotechar='"', quoting=csv.QUOTE_MINIMAL) mywriter.writerow([1,2,3]) # -------------------------------------- # 'pandas' module has its own methods for reading/writing DataFrame object from/to CSV file df = read_csv('myfile.csv',sep=',') df.to_csv('outfile.csv', sep='|', header=True) # -------------------------------------- # Pickling and Unpickling # -------------------------------------- Python provides a standard module called 'pickle' using which you can store any Python object in a file - and restore it back. import cPickle as p p.dump(myobj,fh) myobj = p.load(fh) There is another module called cPickle which functions exactly same as the pickle module except that it is written in the C language and is faster (upto 1000 times). import cPickle as p #import pickle as p myfile = 'shoplist.data' # file to store the object mylist = ['apple', 'mango', 'carrot'] # Write to the file (pickle) f = file(myfile, 'w') p.dump(mylist, f) # dump the object to a file f.close() del mylist # Read back from the storage (unpickle) f = file(myfile) stored_list = p.load(f) print stored_list ======================================== Comon practices: print function is good and understands different types. If I output into CSV file - I may use join() to avoid joining things in print. a=['1','2','3','4','5'] s='|'.join(a) f = open("test.txt","w",0) print >>f,s # this doesn't insert spaces print >>f,6,7,8,9 # this inserts spaces between numbers f.close() Normally just use open() to read + write files. If you don't want to put a newline in the file, do something like fh.write(). For joining strings with non-strings you can use list comprehension: "|".join([str(s) for s in [1,2,3,4,5]]). You can also apply str() function to every element of a list using map(str,LST) with open(fname, 'w') as f: f.writelines(','.join(str(j) for j in i) + '\n' for i in matrix) for row in matrix: file.write(" ".join(map(str,row))+"\n") # -------------------------------------- # how to unbuffer the output to make sure everthing is pushed out immediately. myfile = open(filename,"w",0) # here 0 indicates unbuffered output for name in namelist: myfile.write(name) myfile.close() # -------------------------------------- m = [[1.1, 2.1, 3.1], [4.1, 5.1, 6.1], [7.1, 8.1, 9.1]] file.write(str(m)) # -------------------------------------- # Try this from ipython prompt: m = [[1.1, 2.1, 3.1], [4.1, 5.1, 6.1], [7.1, 8.1, 9.1]] def format(value): return "%.3f" % value formatted = [[format(v) for v in r] for r in m] # file.write(str(formatted)) formatted print formatted str(formatted) formatted.__str__() # note that simply typing "formatted" on ipython prompt produces nice matrix-lie output, whereas all other call produce single line output. # -------------------------------------- Warm Regards Lev ################################################################# py020_debug_pyc.txt ################################################################# Hello, python of the day - day 20 - pyc, debugging # -------------------------------------- How to avoid pyc files When you start creating and running python scripts, you will notice files with extension *.pyc You can tell python to stop writing byte code into these files Method 1 - using -B option in the shebang line #!/opt/py27/bin/python -B Method 2 - setting flag in sys module import sys; sys.dont_write_bytecode = True Method 3 - setting the environment variable export PYTHONDONTWRITEBYTECODE=1 http://stackoverflow.com/questions/154443/how-to-avoid-pyc-files # -------------------------------------- How to debug your program. There are many ways to debug python scripts - depending on which development tools you are using. python has a debugger "pdb" http://docs.python.org/library/pdb.html ipython has a built-in enhanced debuger "ipdb" Eclipse (and other GUI tools) have their own ways to set breakpoints, etc. http://pydev.org/manual_adv_debugger.html There are also useful modules, like traceback (to print stack traces) http://docs.python.org/library/traceback.html I use ipython ipdb. I found description here: http://www.electricmonk.nl/log/2008/06/25/breakpoint-induced-python-debugging-with-ipython/ Note that in our version of ipython we have IPython.core.debugger instead of IPython.Debugger So I add this line in the script: from IPython.core.debugger import Tracer; debug_here = Tracer() And then to set up a breakpoint, I add line like this: debug_here() Then I run the script from ipython prompt using magic command "run", for example run myscript.py It will break at the debug_here(), and you will get enhanced ipython debugger, which understands tab-autocomplete, etc. On ipdb prompt: ? - to see a list of commands (also h or help) EOF bt cont enable jump pdef r tbreak w a c continue exit l pdoc restart u whatis alias cl d h list pinfo return unalias where args clear debug help n pp run unt b commands disable ignore next q s until break condition down j p quit step up ? command - to see help for this command (also "h command" or "help command") q - quit debugger and return to ipython prompt To play with ipdb, go to ipython prompt, and run these 2 lines: from IPython.core.debugger import Tracer; debug_here = Tracer() debug_here() help # -------------------------------------- Warm Regards Lev ################################################################# py021_exceptions.txt ################################################################# Hello, python of the day - day 21 - exceptions Exceptions are anomalous situations requiring special processing. Usually - errors. But you can throw your own exceptions when you want. http://en.wikipedia.org/wiki/Exception_handling http://docs.python.org/tutorial/errors.html Usual pattern of handling exeptions is to set up a trap (a try / catch block) to catch an exception and process it differently depending on its type (may need to "re-throw" it). # -------------------------------------- # example of catching # -------------------------------------- import sys try: s = raw_input('Enter something --> ') except EOFError: print '\nWhy did you do an EOF on me?' sys.exit() # exit the program except: print '\nSome error/exception occurred.' # here, we are not exiting the program print 'Done' # -------------------------------------- # raising exceptions # -------------------------------------- # Filename: raising.py class ShortInputException(Exception): '''A user-defined exception class.''' def __init__(self, length, atleast): Exception.__init__(self) self.length = length self.atleast = atleast try: s = raw_input('Enter something --> ') if len(s) < 3: raise ShortInputException(len(s), 3) # Other work can continue as usual here except EOFError: print '\nWhy did you do an EOF on me?' except ShortInputException, x: print 'ShortInputException: The input was of length %d, \ was expecting at least %d' % (x.length, x.atleast) else: print 'No exception was raised.' # -------------------------------------- # Try..Finally # -------------------------------------- # finally - finally is a block to be executed whether or not an exception was raised # Filename: finally.py import time try: f = file('poem.txt') while True: # our usual file-reading idiom line = f.readline() if len(line) == 0: break time.sleep(2) print line, finally: f.close() print 'Cleaning up...closed the file' # How It Works: # Program is slowed down using sleep() method. # Press Ctrl-c to interrupt/cancel the program. # A KeyboardInterrupt exception is thrown, the finally close is executed. # -------------------------------------- # Re-throwing exceptions # -------------------------------------- # read examples and comments here: http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html try: do_something_dangerous() except: do_something_to_apologize() raise # -------------------------------------- Warm Regards Lev ################################################################# py022_ipython_tips.txt ################################################################# Hello, python of the day - day 22 - several useful ipython tips =================================== Q: How to run a shell command from ipython prompt A: use !! !! ls -alF =================================== Q: How to save output of a shell command into a variable A: use ! a = !ls -alF print a =================================== Q: how to write any variable to a file A: Here is how I do it (Unix/Mac) cd cd .ipython vi myinit.py def tofile(s,fname): f=open(fname,'w') f.write(s.__str__()) f.close() :wq cd ~/.ipython/profile_default/startup/ ln -s ~/.ipython/myinit.py myinit.py cd ~/.ipython/profile_nexus/startup/ ln -s ~/.ipython/myinit.py myinit.py That's it. Now file myinit.py will be imported when ipython starts - and you will have function tofile() always available to write to a file any object (including string, number, list, dictionary, DataFrame, etc.) tofile(something,'somefile') If you want to format it first - use your own function to stringify your object, for example tofile(df.head().to_string(),'junk.txt') !!head junk =================================== Note - history is always available in variable _ih It is an ipython special variable - a list of commands. You can save it to a file like this: tofile(_ih,'somefile') but ... the output will be one long line. It is better to do this: tofile('\n'.join(_ih),'somefile') Also ipython always keeps history in the sqlite database. You can have an alias to extract it as following: alias ihist='sqlite3 ~/.ipython/profile_default/history.sqlite "select * from history" > ~/hist_default.txt" =================================== ################################################################# py023_list_comprehension.txt ################################################################# Hello, python of the day - day 23 - list comprehension [ ... for ... ] [ ... for ... if ... ] etc. A list comprehension is a way to construct a list using "for" loop(s) and some logic. It consists of brackets [ ] containing an expression followed by a "for" clause, which can be followed by additional "for" or "if" clause(s). # -------------------------------------- list1 = [2, 3, 4] list2 = [2*i for i in list1 if i > 2] # list comprehension print list2 # [6, 8] In Perl I would do the same using map & grep functions @list2 = map {; 2*$_ ;} grep { $_ > 2 } @list1; or for (@list1) { next if $ <=2; push @list2, 2*$_; } # -------------------------------------- http://docs.python.org/tutorial/datastructures.html#list-comprehensions http://docs.python.org/tutorial/datastructures.html#nested-list-comprehensions # -------------------------------------- # More advanced example - swap rows and columns of a 2-dim matrix mat = [ [1, 2, 3], [4, 5, 6], [7, 8, 9], ] rev = [[row[i] for row in mat] for i in [0, 1, 2]] # [[1, 4, 7], [2, 5, 8], [3, 6, 9]] print rev # [[1, 4, 7], [2, 5, 8], [3, 6, 9]] # -------------------------------------- # reversing dictionary ( keys become values, and values become keys): num_map={"MAR":3, "JUN":6, "SEP":9,"DEC":12} mo_map = dict([ (v,k) for k, v in num_map.iteritems() ]) mo_map # {3: 'MAR', 6: 'JUN', 9: 'SEP', 12: 'DEC'} # -------------------------------------- # flattening a list of lists into a 1-dim list list_of_list = [ [1,2,3], [4,5,6], [7,8,9] ] print [elem for list in list_of_list for elem in list] # [1, 2, 3, 4, 5, 6, 7, 8, 9] Here is how to understand it. You start with this: for list in list_of_list for elem in list return elem and then move "elem" to the beginning # -------------------------------------- Warm Regards Lev ################################################################# py024_lambda_forms.txt ################################################################# Hello, python of the day - day 24 - lambda forms http://www.secnetix.de/olli/Python/lambda_functions.hawk http://pythonconquerstheuniverse.wordpress.com/2011/08/29/lambda_tutorial/ Python supports the creation of anonymous functions (i.e. functions that are not bound to a name) at runtime, using a construct called "lambda". # regular function: def f (x): return x**2 print f(8) # 64 # lambda function: g = lambda x: x**2 print g(8) # 64 # Note - lambda definition does not include a "return" statement. # Instead it always contains an expression which is returned. # Note - even a print statement cannot be used inside a lambda form, only expressions. # Note - you can put a lambda definition anywhere a function is expected # Note - you don't have to assign lambda function to a variable. # # The lambda function objects can have several arguments # # lambda a, b: a+b. # -------------------------------------- Example: create a dictionary. Take keys from a list, set all values to 1. import pprint lst = ['a', 'b', 'c', 'd', 'e'] d = dict( map( lambda x: (x, 1), lst ) ) from pprint import pprint pprint(d) {'a': 1, 'b': 1, 'c': 1, 'd': 1, 'e': 1} # -------------------------------------- # You can also create named functions. # Example: # Step1 - create a constructor function for lambda objects def my_creator(s,n): return lambda mypar: s + str(par*n) # Step2 - create a lambda function object twice = my_creator("Result: ", 2) # step3 - use this function object print twice('word') # Result: wordword print twice(5) # Result: 10 # -------------------------------------- Warm Regards Lev ################################################################# py025_pass_list_to_function.txt ################################################################# Hello, python of the day - day 25 - passing lists or tuples to functions # -------------------------------------- # Objects are passed by reference. # List is an object, so it is passed by reference. # Thus it is easy to make a function to modify a list in place. # For example: def modifyList( aList ): for i in range( len( aList ) ): aList[i] *= 2 a = [1, 2, 3, 4, 5] modifyList(a) print a # [2, 4, 6, 8, 10] # -------------------------------------- # Reminder - there is no built-in way to pass variables of simple # types (like int or float) by reference. # For example this function will change the element locally, # but will not change the element outside of the function def modifyElement( element ): element *= 2 modifyElement(a[2]) # no effect on a modifyList(a[2:4]) # no effect on a # Workaround: # You can wrap your simple variables into some object (for example, a list), # and pass the object to a function. # Note: module "ctypes" gives you access to C-language data types and pointers # http://stackoverflow.com/questions/2783489/python-copy-by-reference # -------------------------------------- # Reminder about passing strings to functions (see email#12 - functions). # You have to return the string to get it changed outside. # Or pass it inside some container (a list, for example). # Strings is a strange case - they are not treated as a regular object. # Strings are immutable in Python, so you cannot change them directly. # When you pass a string into a function - you actually passing a reference to an immutable string. # When you change the string inside the function, you create a new immutable string, # and point local variable to it. But outside variable still points to the old immutable string. # -------------------------------------- using * to unpack list into list of function arguments using ** to unpack dictionary into named function arguments http://docs.python.org/tutorial/controlflow.html#unpacking-argument-lists If a function accepts several arguments, you can prepare arguments in some external list and then pass this list to the function all at once (instead of passing individual argumentss). When passing the list to a function, prepend it with '*' to tell python to expand the list into function arguments. Example: using * range(3, 6) # normal call with separate arguments # [3, 4, 5] args = [3, 6] range(*args) # call with arguments unpacked from a list # [3, 4, 5] Note: this technique allows to implement variable number of function parameters. For example: def powersum(power, *args): '''Return the sum of each argument raised to specified power.''' total = 0 for i in args: total += pow(i, power) return total powersum(2, 10) # 100 powersum(2, 3, 4) # 25 # -------------------------------------- Example: using ** to unpack a dictionary. Keys should correspond to names of function parameters. Values will be passed into the function. def myfunc(p1, p2, p3): print p1, p2, p3 d = {"p3":"v3", "p2":"v2", "p1": "v1"} # note that the order is irrelevant myfunc(**d) # outputs v1 v2 v3 Note that the keys should match the list of parameters in function definition. For example, if we rename "p1" into "p0" and call the function again: d = {"p0":"v1", "p2":"v2", "p3": "v3"} myfunc(**d) TypeError: myfunc() got an unexpected keyword argument 'p0' Note: you will also get an error if you add additional key/val pair to the dictionary, or if you remove one pair. # -------------------------------------- Warm Regards Lev ################################################################# py026_re.txt ################################################################# Hello, python of the day - day 26 - re - regular expressions A regular expression is a special sequence of characters forming a pattern to search in a text. regexs are very common to Unix worl. Perl programming language is known to introduce really rich and powerful set of regex rules. http://perldoc.perl.org/perlre.html Python has basically borrowed most of regex rules from perl. http://docs.python.org/library/re.html regex is a big topic (as you can see from the length of the above web pages). I highly recommend this book: Mastering Regular Expressions - by Jeffrey Friedl (3rd edition - 534 pages, August 2006, O'Reilly) http://regex.info/ You can find many tutorials and examples online by just googling. For example: http://www.tutorialspoint.com/python/python_reg_expressions.htm Here are some basic examples showing how to invoke regexes for searching/matching, extracting, and replacing/substituting. # ------------------------------------- Matching/Searching: syntax: import re res = re.search(pattern, string, flags) # search for a pattern in a string flags: re.I (2) - case-insensitive re.L (4) - use current localce re.M (8) - Makes ^, $ match the beginning, end of a line inside the text re.S (16) - Makes a period (dot) match any character, including a newline. re.U (32) - Interprets letters according to the Unicode character set. re.X (64) - Ignore whitespace (except inside a set [] or when escaped by a backslash) Treat unescaped # as a comment marker. You can combine these binary flags using bitwize OR "|" # ------------------------------------- Extracting - use round parenthesis in your pattern to indicate the group you want to extract/remember. Then use group() method to access extracted pieces: line = "Cats are smarter than dogs"; res = re.search( r'(.*) are (.*)', line, re.M|re.I) # the above regex matches and remembers first word, followed by word ' are' if res: print "res.group() : ", res.group() # Cats are smarter than dogs print "res.group(1) : ", res.group(1) # Cats print "res.group(2) : ", res.group(2) # smarter than dogs else: print "No match!!" # ------------------------------------- # Example - extracting multiple words using findall method ss = 'aa bb cc' res = re.findall(r'(\b\w+\b)',ss) print res # ['aa', 'bb', 'cc'] # ------------------------------------- # Example - extracting multiple "quoted" words using non-gready match with ? ss = 'mama "papa" sister "crocodile" peter "Chris" Lev' result = re.findall(r'"(.*?)"', ss) print result ['papa', 'crocodile', 'Chris'] # ------------------------------------- # Substituting (search and replace) # syntax: # changed = re.sub(pattern, repl, string, max=0) phone = "2004-959-559 # This is Phone Number" # Delete comments num = re.sub(r'#.*$', "", phone) print "Phone Num : ", num # 2004-959-559 # Remove anything other than digits num = re.sub(r'\D', "", phone) print "Phone Num : ", num # 2004959559 # Note - 3rd parameter to re.sub() allows to specify the nubmer of substitutions. # (the default is zero, means - replace all. # ------------------------------------- # Example - Abbreviate long name by taking no more than 3 first letters of every word def first3(match): if match: matches = match.groups() return ''.join([x[0:3].title() for x in matches]) else: return '' def abbreviate(ss): ss = re.sub(r'(\w+)', first3, ss) # each word substitute by first 3 chars by applying first3(match) function ss = re.sub(r'\s+','',ss) # remove spaces return ss # ------------------------------------- Note: in many cases you can use standard python string functions instead of regexes # startswith(), in, find() name = 'Peter' if name.startswith('Pet'): print 'startswith - "Pet"' if 'te' in name: print 'in - "te"' if name.find('te') != -1: print 'name.find - "te"' # replace(old, new[, count]) s1='mama mama mama' s2=s1.replace('mama','papa',1) # 'papa mama mama' s2=s1.replace('mama','papa') # 'papa papa papa' s1 = ' mama ' s2 = s1.strip() # removing starting and trailing white space # Look also at other standard methods, like # str.split # str.splitlines # etc. # ------------------------------------- Warm Regards Lev ################################################################# py026_re2_regex_email.txt ################################################################# Hello, Python of the day - day 26 - regular expressions - email # 2 I want to again recommend the classic book "Mastering Regular Expressions" by Jeffrey E.F. Friedl It explains how regex engine works inside. regex actually builds a tree, trying different paths, backtracking, trying other paths. Depending on your pattern, the matching procedure can take forever. Here is a simple demo how this could happen: http://www.regular-expressions.info/catastrophic.html So you better keep your regex simple. Here is a famous example from a Friedl's regex book. This is a regex to match an email address: http://www.diablotin.com/librairie/autres/mre/chBB.html The web page explains it. I never seen it used anywhere. But it is an interesting achievement. Here it is - enjoy. [\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\ xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^( \040)<>@,;:".\\\[\]\000-\037\x80-\xff])|"[^\\\x80-\xff\n\015"]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015"]*)*")[\04 0\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\ n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:\.[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\ xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t] *)*(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])|"[^\\\x80-\xff\n \015"]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015"]*)*")[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\( [^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*)*@[ \040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\x ff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040 )<>@,;:".\\\[\]\000-\037\x80-\xff])|\[(?:[^\\\x80-\xff\n\015\[\]]|\\[^\x80-\xff])*\])[\040\t]*(?:\([^\\\x80-\x ff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x8 0-\xff\n\015()]*)*\)[\040\t]*)*(?:\.[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\ n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:" .\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])|\[(?:[^\\\x80-\xff\n\015\[\]]|\\[^\x8 0-\xff])*\])[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xf f][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*)*|(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80- \xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])|"[^\\\x80-\xff\n\015"]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\01 5"]*)*")[^()<>@,;:".\\\[\]\x80-\xff\000-\010\012-\037]*(?:(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([ ^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)|"[^\\\x80-\xff\ n\015"]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015"]*)*")[^()<>@,;:".\\\[\]\x80-\xff\000-\010\012-\037]*)*<[\040\t]* (?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015 ()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:@[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\ ([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?: [^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])|\[(?:[^\\\x80-\xff\n\0 15\[\]]|\\[^\x80-\xff])*\])[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]* (?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:\.[\040\t]*(?:\([^\\\x80 -\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\ \x80-\xff\n\015()]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\ 037\x80-\xff])|\[(?:[^\\\x80-\xff\n\015\[\]]|\\[^\x80-\xff])*\])[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[ ^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[ \040\t]*)*)*(?:,[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80 -\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*@[\040\t]*(?:\([^\\\x80-\xff\n\015()]* (?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015 ()]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])| \[(?:[^\\\x80-\xff\n\015\[\]]|\\[^\x80-\xff])*\])[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([ ^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:\. [\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\ xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\04 0)<>@,;:".\\\[\]\000-\037\x80-\xff])|\[(?:[^\\\x80-\xff\n\015\[\]]|\\[^\x80-\xff])*\])[\040\t]*(?:\([^\\\x80-\ xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x 80-\xff\n\015()]*)*\)[\040\t]*)*)*)*:[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff \n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*)?(?:[^(\040)<>@, ;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])|"[^\\\x80-\xff\n\015"]*(?:\\[^\x80 -\xff][^\\\x80-\xff\n\015"]*)*")[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\01 5()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:\.[\040\t]*(?:\([^\ \\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\) )[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\ 000-\037\x80-\xff])|"[^\\\x80-\xff\n\015"]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015"]*)*")[\040\t]*(?:\([^\\\x80-\ xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x 80-\xff\n\015()]*)*\)[\040\t]*)*)*@[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n \015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:". \\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])|\[(?:[^\\\x80-\xff\n\015\[\]]|\\[^\x80 -\xff])*\])[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff ][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*(?:\.[\040\t]*(?:\([^\\\x80-\xff\n\015()]*( ?:(?:\\[^\x80-\xff]|\([^\\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015( )]*)*\)[\040\t]*)*(?:[^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff]+(?![^(\040)<>@,;:".\\\[\]\000-\037\x80-\xff])|\ [(?:[^\\\x80-\xff\n\015\[\]]|\\[^\x80-\xff])*\])[\040\t]*(?:\([^\\\x80-\xff\n\015()]*(?:(?:\\[^\x80-\xff]|\([^ \\\x80-\xff\n\015()]*(?:\\[^\x80-\xff][^\\\x80-\xff\n\015()]*)*\))[^\\\x80-\xff\n\015()]*)*\)[\040\t]*)*)*>) Warm Regards Lev ################################################################# py027_argparse.txt ################################################################# Hello, python of the day - day 27 - parsing cmd options Here are common ways of parsing cmd options: sys.argv getopt - similar to C getopt() optparse - deprecated since ver. 2.7 - use argparse instead argparse - preferred way to parse agrs since ver. 2.7 # -------------------------------------- sys.argv - allows access to cmd options via 2-dm array sys.argv[][] # file test.py #!/usr/bin/env python import sys print sys.argv sys.exit() # invoking the file ./test.py aa bb cc # ['./test.py', 'aa', 'bb', 'cc'] # Note - very first element of sys.argv is the name of the script # So to check if there are cmd options, you should check that length of sys.argv is >1. # For example if len(sys.argv) > 1 and sys.argv[1] == "test": # do something # -------------------------------------- getopt - gives a much nicer interface to handle both short and long options http://docs.python.org/library/getopt.html import getopt, sys def main(): try: opts, args = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="]) except getopt.GetoptError, err: # print help information and exit: print str(err) # will print something like "option -a not recognized" usage() sys.exit(2) output = None verbose = False for o, a in opts: if o == "-v": verbose = True elif o in ("-h", "--help"): usage() sys.exit() elif o in ("-o", "--output"): output = a else: assert False, "unhandled option" # ... if __name__ == "__main__": main() # -------------------------------------- optparse is a newer and better than getopt, but deprecated starting ver. 2.7 in favor of argparse http://docs.python.org/library/optparse.html from optparse import OptionParser parser = OptionParser() parser.add_option("-r", "--ref", dest="ref", help="The reference app.") parser.add_option("-t", "--tst", dest="tst", help="The test app.") (options, args) = parser.parse_args() if (options.ref): ref = options.ref if (options.tst): tst = options.tst # -------------------------------------- argparse - best way to parse options starting ver. 2.7 http://docs.python.org/library/argparse.html#module-argparse import argparse parser = argparse.ArgumentParser(description='Some description string.') parser.add_argument('--file', '-f', '-i', '--input', action="store", dest="lev", help='store input file name to lev') results = parser.parse_args() print results.lev # try this script like this: > python test.py -f 5 Hello World 5 > python test.py -f5 Hello World 5 > python test.py -h Hello World usage: test.py [-h] [--file LEV] Some description string. optional arguments: -h, --help show this help message and exit --file LEV, -f LEV, -i LEV, --input LEV store input file name to lev # -------------------------------------- Warm Regards Lev ################################################################# py028_sys_os.txt ################################################################# Hello, python of the day - day 28 - sys, os Here is a short overview of two standard modules - sys & os These modules are used very often. Please spend 5 min reviewing avilable methods. in ipython use to list the methods, and use ?? after the method name to see description. # -------------------------------------- import sys sys.exit() if sys.argv[1].startswith('--'): option = sys.argv[1][2:] if option == 'version': print 'Version 1.2' sys.version sys.version_info sys. sys.api_version sys.float_repr_style sys.path sys.argv sys.getcheckinterval sys.path_hooks sys.builtin_module_names sys.getdefaultencoding sys.path_importer_cache sys.byteorder sys.getfilesystemencoding sys.platform sys.call_tracing sys.getprofile sys.prefix sys.callstats sys.getrecursionlimit sys.py3kwarning sys.copyright sys.getrefcount sys.setcheckinterval sys.displayhook sys.getsizeof sys.setprofile sys.dllhandle sys.gettrace sys.setrecursionlimit sys.dont_write_bytecode sys.getwindowsversion sys.settrace sys.exc_clear sys.hexversion sys.stderr sys.exc_info sys.last_traceback sys.stdin sys.exc_type sys.last_type sys.stdout sys.excepthook sys.last_value sys.subversion sys.exec_prefix sys.long_info sys.version sys.executable sys.maxint sys.version_info sys.exit sys.maxsize sys.warnoptions sys.exitfunc sys.maxunicode sys.winver sys.flags sys.meta_path sys.float_info sys.modules # -------------------------------------- import os os.name - 'nt' for Windows, 'posix' for Linux/Unix. os.getcwd() - current working directory os.getenv(varname[, value]) - return the value of the env variable varname if it exists, or value if it doesn't. os.putenv('PATH', '/path1:/path2:/path3') - set env var os.listdir() - directory listing os.remove(file) - delete a file or directory os.unlink(file) os.rmdir(dir) os.rename(path) os.system(cmd) function is used to run a shell command. os.stat(path) - returns object with many fields st_nlink - number of hard links, st_uid - user id of owner, st_gid - group id of owner, st_size - size of file, in bytes, st_atime - time of most recent access, st_mtime - time of most recent content modification, st_ctime - platform dependent; time of most recent metadata change on Unix, or the time of creation on Windows) etc. os.sep - dir separator (Windows - '\\', Unix - '/') os.linesep - line terminator (Windows - '\r\n', Linux - '\n', Mac - '\r'). os.path.split('/home/swaroop/byte/code/poem.txt') # ('/home/swaroop/byte/code', 'poem.txt') os.path.isfile(path) os.path.isdir(path) os.path.exists(path) import os os. os.F_OK os.devnull os.popen2 os.O_APPEND os.dup os.popen3 os.O_BINARY os.dup2 os.popen4 os.O_CREAT os.environ os.putenv os.O_EXCL os.errno os.read os.O_NOINHERIT os.error os.remove os.O_RANDOM os.execl os.removedirs os.O_RDONLY os.execle os.rename os.O_RDWR os.execlp os.renames os.O_SEQUENTIAL os.execlpe os.rmdir os.O_SHORT_LIVED os.execv os.sep os.O_TEMPORARY os.execve os.spawnl os.O_TEXT os.execvp os.spawnle os.O_TRUNC os.execvpe os.spawnv os.O_WRONLY os.extsep os.spawnve os.P_DETACH os.fdopen os.startfile os.P_NOWAIT os.fstat os.stat os.P_NOWAITO os.fsync os.stat_float_times os.P_OVERLAY os.getcwd os.stat_result os.P_WAIT os.getcwdu os.statvfs_result os.R_OK os.getenv os.strerror os.SEEK_CUR os.getpid os.sys os.SEEK_END os.isatty os.system os.SEEK_SET os.kill os.tempnam os.TMP_MAX os.linesep os.times os.UserDict os.listdir os.tmpfile os.W_OK os.lseek os.tmpnam os.X_OK os.lstat os.umask os.abort os.makedirs os.unlink os.access os.mkdir os.unsetenv os.altsep os.name os.urandom os.chdir os.open os.utime os.chmod os.pardir os.waitpid os.close os.path os.walk os.closerange os.pathsep os.write os.curdir os.pipe os.defpath os.popen user = os.environ['USER'] os.putenv - doesn't work as expected os.environ["MY_PATH"]="/path/to/program" # here how to add env. var (and check if it exists first) keys = os.environ.keys() from re import search for key in keys: if not search("MY_PATH", key): os.environ["MY_PATH"]="/path/to/program" print os.getenv('MY_PATH') # -------------------------------------- Warm Regards Lev ################################################################# py029__name__.txt ################################################################# Hello, python of the day - day 29 - __name__ # -------------------------------------- Typical python script has 3 parts: import statements definitions of functions and classes commands to run You can run the script by itself, or you can use it as a library (import from another script). In both cases the 3rd part ("commands to run") will be executed. Note - python keeps record of which files were already imported, and will not import the same script more than once (there is a reload() built-in function for that). What if we want to run the "commands to run" only if the program was used by itself and not when it was imported from another module? This can be achieved by using the __name__ attribute of the module. If the module is run by itself, then __name__ == "__main__" If module is imported from another module, then __name__ is set to the name of this module (without "py" extension). http://docs.python.org/library/__main__.html # run this simple test: if __name__ == '__main__': print 'This program is being run by itself' else: print 'I am being imported from another module' # -------------------------------------- Similarly you can do something like this to force exit only if script runs as stand-alone: if (__name__ == '__main__'): sys.exit(main()) # -------------------------------------- Here is an example how __name__ works. If you have a script a.py with only one line of code print __name__ Then if you run this script independently, it will print '__main__' but if you import it from module b.py, it will print 'a' cat a.py print __name__ cat b.py import a python a.py __main__ python b.py a # -------------------------------------- Warm Regards Lev ################################################################# py030_unbuffer.txt ################################################################# Hello, python of the day - day 30 - how to unbuffer output By default file output (even to stdout when it is redirected to a file) is buffered. This means that you will not see the output until the buffer is explicitly flushed. Here are several ways to unbuffer outputs. # -------------------------------------- When you start python with option -u - it will unbuffer stdout and stderr. You can also include it in the shebang line #!/usr/bin/python -uB # -------------------------------------- You can also add the following line to your .bashrc file export PYTHONUNBUFFERED=1 Note - this setting somehow makes ipython slow. So I don't use it. I use this though (to avoid creating *.pyc files): export PYTHONDONTWRITEBYTECODE=1 # -------------------------------------- # To unbuffer file output - open file handler with zero as the buffer size import time fh = open('file.log','w',0) for ii in xrange(20): t= time.time() print t fh.write(str(t)) time.sleep(0.2) # sleep 0.2 seconds fh.close() # -------------------------------------- # To unbuffer stdout - reopen it with zero buffer size import sys import os sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) print "unbuffered text" # or import sys import os unbuffered = os.fdopen(sys.stdout.fileno(), 'w', 0) unbuffered.write('test\n') sys.stdout = unbuffered print "unbuffered text" # -------------------------------------- # You can always explicitly flush buffer import sys sys.stdout.flush() So you can wrap sys.stdout in an object that flushes after every write # -------------------------------------- Several ways to print to STDERR print >> sys.stderr, 'some error message' # deprecated, removed in ver.3 of python # -------------------- sys.stderr.write('some error message\n') # -------------------- from __future__ import print_function print('some error message', file=sys.stderr) # -------------------------------------- # On Windows you can do something like this: import msvcrt msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) see some Windows stuff here: http://docs.python.org/library/msvcrt.html http://effbot.org/librarybook/msvcrt.htm # -------------------------------------- Warm Regards Lev ################################################################# py031_repr_dump.txt ################################################################# Hello, python of the day - day 31 - __repr__() function # -------------------------------------- repr - string repr-esentation of an object which you can print or eval(). Backticks do the same thing, but they are deprecated - and don't exist in python ver. 3. In most cases (but not required by standards) eval(repr(object)) == object Note: see PEP docs - repr string should be unambiguous (but not necessarily eval-able) http://www.python.org/dev/peps/pep-3138/ # Example: i = [] i.append('item') `i` # "['item']" repr(i) # "['item']" For user-defined classes you can define __str__ and __repr__ methods Read nice discussion here about the difference betwen them: http://stackoverflow.com/questions/1436703/difference-between-str-and-repr-in-python Main points of this discussion: - The default implementation is useless - __repr__ goal is to be unambiguous - __str__ goal is to be readable - Container’s __str__ uses contained objects’ __repr__ Advice: - Implement __repr__ for any class you implement. This should be second nature. - Implement __str__ if you think it would be useful to have a string version which errs on the side of more readability in favor of more ambiguity. Exercise: Create a simple class, and define a __repr__ amd __str__ methods like this: # --------------------------------- def __repr__(self: return "%s(%r)" % (self.__class__, self.__dict__) # --------------------------------- def __str__(self): return self.__repr__() See what will happen if you crete an object of this class and try to print it # -------------------------------------- How to dump out the object? Standard objects (lists, dictionaries) can be output with str() or print (although for better formatting you can inport pprint - see below) for your own classes you can define your own __repr__() or dump() method, for example: # --------------------------------- def dump(self): for attr in dir(self): print "self.%s = %s" % (attr, getattr(self, attr)) # -------------------------------------- http://stackoverflow.com/questions/192109/is-there-a-function-in-python-to-print-all-the-current-properties-and-values-of # -------------------------------------- pprint - pretty printer for data structures from pprint import pprint # create a list of tuples (i,d), where d is a dictionary data = [ (i, {'a':'A','b':'B','c':'C',}) for i in xrange(3)] # pretty printing pprint(data) [(0, {'a': 'A', 'b': 'B', 'c': 'C'}), (1, {'a': 'A', 'b': 'B', 'c': 'C'}), (2, {'a': 'A', 'b': 'B', 'c': 'C'})] # regular printing - no formatting or sorting print data [(0, {'a': 'A', 'c': 'C', 'b': 'B'}), (1, {'a': 'A', 'c': 'C', 'b': 'B'}), (2, {'a': 'A', 'c': 'C', 'b': 'B'})] # str - no formatting or sorting str(data) "[(0, {'a': 'A', 'c': 'C', 'b': 'B'}), (1, {'a': 'A', 'c': 'C', 'b': 'B'}), (2, {'a': 'A', 'c': 'C', 'b': 'B'})]" # -------------------------------------- Warm Regards Lev ################################################################# py032_email.txt ################################################################# Hello, python of the day - day 32 - email sending emaiil: email, smtplib http://docs.python.org/library/email-examples.html smtplib email email.MIMEText email.Errors email.MIMEBase email.MIMEMultipart receiving email: mailbox http://docs.python.org/library/mailbox.html # -------------------------------------- # example - (old style using popen()) sending email via piping into sendmail unix command def sendMail(): sendmail_location = "/usr/sbin/sendmail" # sendmail location p = os.popen("%s -t" % sendmail_location, "w") p.write("From: %s\n" % "from@somewhere.com") p.write("To: %s\n" % "to@somewhereelse.com") p.write("Subject: thesubject\n") p.write("\n") # blank line separating headers from body p.write("body of the mail") status = p.close() if status != 0: print "Sendmail exit status", status # -------------------------------------- # better - sending email using subprocess and mutt http://en.wikipedia.org/wiki/Mutt_(email_client) mutt is an excellent binary to send emails - both text and with attachments. It is easy to send multiple attachments - and to multiple recipients. echo "one line text" | /usr/bin/mutt -s "test message with 1 line" john.smith@domain.com echo "line1 line2 line3" | /usr/bin/mutt -s "test message with 3 lines" john.smith@domain.com cat some_text_file | /usr/bin/mutt -s "test message getting text from file" john.smith@domain.com # attaching several files - and sending to several people - this is what I use in my bash scripts echo "some text" | /usr/bin/mutt -s "some subject" -a file1.zip -a file2.jpg -a file3.csv a@b.com c@d.com e@f.com # Note: you need to pipe something into mutt, because if you don't - then mutt terminal application will popup in interactive mode. Try to do this from the prompt - and see what will happen: mutt -s "test message" -a somefile john.smith@domain.com Note: If you provide a filename using "-a" option, and this file doesn't exist - mutt will exit with error without sending an email. # Here is a simple python script import string import subprocess def send_mail_via_mutt(to, subject="", msg=""): mail_prog = "/usr/bin/mutt" sub_str=" " if subject != "": sub_str = " -s \"" + subject + "\" " msg=string.replace(msg, "\n", "\\n") subprocess.call ("echo -e \"" + msg + "\" | " + mail_prog + sub_str + to, shell=True) send_mail_via_mutt('john.smith@somedomain.com',"my test", "my text\nmy text2") # -------------------------------------- # sending email to lists with attachments import smtplib import mimetypes from email import Encoders from email.MIMEText import MIMEText from email.MIMEMultipart import MIMEMultipart from email.MIMEBase import MIMEBase def construct_email_message(sender, to_list, cc_list, subject, attachment_list): outer = MIMEMultipart() outer['From'] = sender outer['To'] = ', '.join(to_list) outer['Cc'] = ', '.join(cc_list) outer['Subject'] = subject outer.preample = 'You will not see this in a MIME-aware mail reader.\n' outer.epilogue = '' # plain text body msg = MIMEText('Please see attached files.\n\n') msg.add_header('Content-Disposition', 'inline') outer.attach(msg) # file attachments for att in attachment_list: if not os.path.isfile(att): continue ctype, encoding = mimetypes.guess_type(att) maintype, subtype = ctype.split('/',1) fp = open(att, 'rb') msg = MIMEBase(maintype, subtype) msg.set_payload(fp.read()) Encoders.encode_base64(msg) msg.add_header('Content-Disposition', 'attachment', filename=att) outer.attach(msg) return outer def send_email_message(sender, recip_list, outer): s = smtplib.SMTP() s.connect('mymail.domain.net') # provide correct server s.sendmail(sender, recip_list, outer.as_string()) s.close() def main(): print 'sending email' sender = 'John.Smith@domain.com' subject = 'some subject' to_list = ['a@b.com', 'c@d.com'] cc_list = ['a2@b.com', 'c2@d.com'] attach_list = ['file1.pdf', 'file2.pdf'] outer = construct_email_message(sender, to_list, cc_list, subject, attach_list) send_email_message(sender, to_list + cc_list, outer) # -------------------------------------- # Example - sending a simple email using smtplib import smtplib from email.mime.text import MIMEText fp = open(textfile, 'rb') msg = MIMEText(fp.read()) fp.close() email_from = 'john.smith@gmail.com' email_to = 'gena.crocodil@gmail.com' msg['Subject'] = 'The contents of %s' % textfile msg['From'] = email_from msg['To'] = email_to # Send the message via our own SMTP server, but don't include the envelope header. s = smtplib.SMTP('localhost') s.sendmail(email_from, [email_to], msg.as_string()) s.quit() # -------------------------------------- # sending html email using smtplib # http://stackoverflow.com/questions/882712/sending-html-email-in-python import smtplib from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText email_from = "my@email.com" email_to = "your@email.com" # Create message container - the correct MIME type is multipart/alternative. msg = MIMEMultipart('alternative') msg['Subject'] = "Link" msg['From'] = email_from msg['To'] = email_to # Create the body of the message (a plain-text and an HTML version). text = "Hi!\nHow are you?\nHere is the link you wanted:\nhttp://www.python.org" html = """\

Hi!
How are you?
Here is the link you wanted.

""" # Record the MIME types of both parts - text/plain and text/html. part1 = MIMEText(text, 'plain') part2 = MIMEText(html, 'html') # Attach parts into message container. # According to RFC 2046, the last part of a multipart message is preferred. msg.attach(part1) msg.attach(part2) # Send the message via local SMTP server. s = smtplib.SMTP('localhost') s.sendmail(email_from, email_to, msg.as_string()) s.quit() # -------------------------------------- # Example - sending email via specific server # http://stackoverflow.com/questions/64505/sending-mail-from-python-using-smtp SMTPserver = 'smtp.att.yahoo.com' sender = 'me@my_email_domain.net' destination = ['recipient@her_email_domain.com'] USERNAME = "USER_NAME_FOR_INTERNET_SERVICE_PROVIDER" PASSWORD = "PASSWORD_INTERNET_SERVICE_PROVIDER" # typical values for text_subtype are plain, html, xml text_subtype = 'plain' content="""\ Test message """ subject="Sent from Python" import sys import os import re from smtplib import SMTP_SSL as SMTP # this invokes the secure SMTP protocol (port 465, uses SSL) # from smtplib import SMTP # use this for standard SMTP protocol (port 25, no encryption) from email.MIMEText import MIMEText try: msg = MIMEText(content, text_subtype) msg['Subject']= subject msg['From'] = sender # some SMTP servers will do this automatically, not all conn = SMTP(SMTPserver) conn.set_debuglevel(False) conn.login(USERNAME, PASSWORD) try: conn.sendmail(sender, destination, msg.as_string()) finally: conn.close() except Exception, exc: sys.exit( "mail failed; %s" % str(exc) ) # give a error message # -------------------------------------- # sending email with reply_to MAIL_SERVER = 'smtp.domain.com' TO_ADDRESS = 'you@gmail.com' FROM_ADDRESS = 'email@domain.com' REPLY_TO_ADDRESS = 'email2@domain2.com' import smtplib import email.mime.multipart msg = email.mime.multipart.MIMEMultipart() msg['to'] = TO_ADDRESS msg['from'] = FROM_ADDRESS msg['subject'] = 'testing reply-to header' msg.add_header('reply-to', REPLY_TO_ADDRESS) server = smtplib.SMTP(MAIL_SERVER) server.sendmail(msg['from'], [msg['to']], msg.as_string()) # -------------------------------------- mimetypes - Map filenames to MIME types http://docs.python.org/library/mimetypes.html This module defines two useful functions: guess_type(url, strict=1) -- guess the MIME type and encoding of a URL. guess_extension(type, strict=1) -- guess the extension for a given MIME type. It also contains the following, for tuning the behavior: Data: knownfiles -- list of files to parse inited -- flag set when init() has been called suffix_map -- dictionary mapping suffixes to suffixes encodings_map -- dictionary mapping suffixes to encodings types_map -- dictionary mapping suffixes to types Functions: init([files]) -- parse a list of files, default knownfiles (on Windows, the default values are taken from the registry) read_mime_types(file) -- parse one file, return a dictionary or None # -------------------------------------- # marrow.mailer (formerly TurboMail) - a highly efficient and modular # mail delivery framework for Python 2.6+ and 3.1+ # https://github.com/marrow/marrow.mailer from marrow.mailer import Mailer, Message mailer = Mailer(dict( transport = dict( use = 'smtp', host = 'localhost'))) mailer.start() message = Message(author="user@example.com", to="user-two@example.com") message.subject = "Testing Marrow Mailer" message.plain = "This is a test." mailer.send(message) mailer.stop() ============================================================ To get maximum benefit from these emails, play with examples. ============================================================ Warm Regards Lev ################################################################# py033_exec_eval.txt ################################################################# Hello, python of the day - day 33 - exec(), eval(), compile(), execfile() # -------------------------------------- exec - execute statement(s) from a string eval - returns the value of an expression compile() - compiles string into a code object execfile(filename[, globals[, locals]]) - executes statements from a file # -------------------------------------- >>> exec 'print "Hello World"' Hello World >>> eval('2*3') 6 What is the difference ? # -------------------------------------- exec is a statement, not an expression . exec 'print 5' exec 'print 5; print6' exec 'print 5\nprint 6' exec 'if True: print 6' exec '5' # does nothing and returns nothing. # -------------------------------------- eval evaluates an expression (not a statement) and returns the value that expression produces. x = eval('5') # 5 x = eval('%d + 6' % x) # 11 x = eval('abs(%d)' % -100) # 100 x = eval('print 5') # INVALID; print is a statement, not an expression x = eval('if 1: x = 4') # INVALID; if is a statement, not an expression. read more: http://stackoverflow.com/questions/2220699/whats-the-difference-between-eval-exec-and-compile-in-python # -------------------------------------- compile is a lower level version of exec and eval. It does not execute or evaluate your statements or expressions, but returns a code object that can do it. http://docs.python.org/library/functions.html#compile Try this on iPython prompt: x = 1 y = 2 a = compile("print 'x+y=', x+y", "", "single") exec a # outputs x+y= 3 # -------------------------------------- execfile('myfile.py') - reads the file and executes all statements. It is similar to "import", but simpler, because execfile() doesn't do any "module administration" - does not create a new module. The arguments are a file name two optional dictionaries to be used as global and local namespace. execfile() doesn't exist in Python 3. instead you are supposed to do 3 steps: - read the file - compile the contents into a code object - exec it http://docs.python.org/library/functions.html#execfile # -------------------------------------- Warm Regards Lev ################################################################# py034_graphics.txt ################################################################# Hello, python of the day - day 34 - graphics # -------------------------------------- There are many libraries available do draw graphics. see for example some offers here: http://stackoverflow.com/questions/326300/python-best-library-for-drawing Google for python graph library python graphcs library and you will see a lot of choices. # -------------------------------------- You most probably will use matplotlib - a library for a MATLAB-like plotting interface in Python. Integrates with ipython, can show graphics using X-windows or in the browser. Can export in many different image formats. import matplotlib If you are using enthought python, then you can do this: ipython notebook --pylab=inline and then in the notebook in the browser do this: plot(range(10)) See multiple examples here: Many more examples of graphs with source code: http://matplotlib.sourceforge.net/gallery.html http://www.scipy.org/Cookbook/Matplotlib http://www.scipy.org/Cookbook/Matplotlib/Animations pandas works with matplotlib very well. # -------------------------------------- Chaco - http://code.enthought.com/chaco/ It is developed at Enthought, more complicated and richer (in comparison with matplotlib): http://code.enthought.com/projects/chaco/docs/html/index.html http://www.enthought.com/ mayavi - 3D graphics toolkit, integrates with ipython Other: PyQwt, Veusz, gnuplot-py, biggles, etc. # -------------------------------------- Main python web site discusses some libraries, for example Tk: http://docs.python.org/library/tk.html?highlight=graphics # -------------------------------------- Yet another: http://www.wxwidgets.org/ # -------------------------------------- Gnuplot.py http://www.gnuplot.info/ http://gnuplot-py.sourceforge.net/ Gnuplot.py is a Python package that interfaces to gnuplot, the popular open-source plotting program. # -------------------------------------- Example of a graphical application for Windows. This is a demo of famous Hodgkin-Huxley Model (1952) of a nerve fiber membrane. (received Nobel Prize in Physiology or Medicine in 1963) Download and install the following on your PC: http://www.python.org/download/releases/2.6.6/ - ver. 2.6.6 of Python. Follow default options - it will install into C:\Python26 Then add this directory to the PATH (Control Panel - System, Environment, Path) Then download and install the following libraries: http://sourceforge.net/projects/numpy/files/ http://sourceforge.net/projects/scipy/files/ http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.0/ Then go here: http://www.neurdon.com/2011/01/26/neural-modeling-with-python-part-2/ and you will see the script hh.py Click on the link "hh.py" above the script. Copy/paste this script into a text file called hh.py in a directory with no spaces in the names. For example: C:\model\hh.py open DOS window, go to this directory and run the script using this command: python hh.py Warm Regards Lev ################################################################# py035_slices_misc.txt ################################################################# Hello, python of the day - day 35 - slices Here is just a collection of common expressions. Many of them you have already seen in previous emails. # -------------------------------------- # using slices with dict (using list comprehension or map/lambda): ar = [5, 4, 3, 1, 2] hh = dict([(x, 1) for x in ar]) # or using map and lambda hh = dict( map( lambda x: (x, 1), ar ) ) # or using "for" loop for kk in ar : hh[kk] = 1 # For dict slices use list comprehension, i.e., myslice = [ hh[k] for k in [2,3,4] ] # -------------------------------------- list.append(x) # lista[len(list):] = [x]. list.extend(L) # list[len(list):] = L, where L is a list (similar to push(@list, @L) in perl) list.insert(i, x) # Insert an item x at a given position i. list.insert(0, x) # inserts at the front of the list, like "unshift" in perl a.insert(len(a), x) # is equivalent to list.append(x), or to "push" in perl list.remove(x) # remove the first item from the list whose value is x. list.pop([i]) # remove the item at the given position in the list list.pop() # removes and returns the last item in the list. list.pop(0) # removes and returns the very frist item in the list. del list[i] # remove i-th element list[i:i+1] = [] # remove ith element list[i:j] = [] # remove several elements list[i:j] = [1,2,3,4,5,6,7,8,9] # remove elements - and then insert a new list instead of them (may be more or less elements) list[i:i] = [1,2,3,4,5,6,7,8,9] # insert new list just before item with index i list = [1,2,3,4,5] list[1:3] = [7,8] # [1, 7, 8, 4, 5] list[:0] = [11,12,13] # [11, 12, 13, 1, 7, 8, 4, 5] # -------------------------------------- # dictionaries - getting/updating several elements # create a dictionary d = dict( ( k, k ) for k in xrange( 5 ) ) print d # {0: 0, 1: 1, 2: 2, 3: 3, 4: 4} # update many values at once d.update( ( k, k*k ) for k in xrange(5) ) print d # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16} # copy key/values to another dictionary d2 = dict( (k, d[k]) for k in [3,4]) print d2 # {3: 9, 4: 16} # remove many keys except for some d= dict( (k, d[k]) for k in [3,4]) print d2 # {3: 9, 4: 16} # deleting del d[3] d.pop(3) # deletes the key/value and return it d2={} d2['mama']=d.pop(3) # -------------------------------------- Warm Regards Lev ################################################################# py036_date_time.txt ################################################################# Hello, python of the day - day 36 - date & time time - time access and conversions http://docs.python.org/library/time.html datetime - Basic date and time types http://docs.python.org/library/datetime.html calendar http://docs.python.org/library/calendar.html mxDateTime - open source http://www.egenix.com/products/python/mxBase/mxDateTime/ http://www.egenix.com/products/python/mxBase/ parsedatetime - module that is able to parse 'human readable' date/time expressions. http://code.google.com/p/parsedatetime/wiki/Examples parse_datetime - module by Paul Harrison, 2006 http://www.logarithmic.net/pfh-files/blog/01162445830/parse_datetime.py # -------------------------------------- # some examples # -------------------------------------- http://pleac.sourceforge.net/pleac_python/datesandtimes.html from datetime import date, timedelta ONE_DAY = timedelta(days=1) # ONE_DAY is now datetime.timedelta(1) # try those: print "Today is day", time.localtime()[7], "of the current year" # Today is day 218 of the current year today = datetime.date.today() print "Today is day", today.timetuple()[7], "of ", today.year # Today is day 218 of 2003 print "Today is day", today.strftime("%j"), "of the current year" # Today is day 218 of the current year today = datetime.date.today() print "The date is", today # The date is 2003-08-06 # -------------------------------------- # Dates produce timedeltas when subtracted. d1=datetime.date(2012,2,1) d11=datetime.date(2012,2,11) diff = d11-d1 print diff # 10 days, 0:00:00 print diff.days # 10 dt1 = datetime.datetime(1973, 1, 18, 3, 45, 50) dt2 = datetime.datetime(1981, 6, 16, 4, 35, 25) diff = dt2 - dt1 print diff # 3071 days, 0:49:35 # -------------------------------------- import time time.sleep(10) - sleep 10 seconds time.sleep(0.2) - sleep 0.2 seconds time.time() - epoch seconds (in UTC, floating point number) time.localtime() - returns a structure time.gmtime() - returns a structure time.asctime() - convert a tuple or struct_time representing a time to a string like this: 'Mon Feb 20 14:03:32 2012' time.strptime("30 Nov 00", "%d %b %y") - parse a string representing a time according to a format, returns a structure time.strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime()) # 'Thu, 28 Jun 2001 14:17:15 +0000' strftime() convert a tuple or struct_time representing a time as returned by gmtime() or localtime() to a string as specified by the format argument The time value as returned by gmtime(), localtime(), and strptime(), and accepted by asctime(), mktime() and strftime(), may be considered as a sequence of 9 integers. The return values of gmtime(), localtime(), and strptime() also offer attribute names for individual fields. 0 tm_year (for example, 1993) 1 tm_mon range [1, 12] 2 tm_mday range [1, 31] 3 tm_hour range [0, 23] 4 tm_min range [0, 59] 5 tm_sec range [0, 61]; see (1) in strftime() description 6 tm_wday range [0, 6], Monday is 0 7 tm_yday range [1, 366] 8 tm_isdst 0, 1 or -1; see below # -------------------------------------- calendar http://docs.python.org/library/calendar.html calendar.timegm(tuple) An unrelated but handy function that takes a time tuple such as returned by the gmtime() function in the time module, and returns the corresponding Unix timestamp value, assuming an epoch of 1970, and the POSIX encoding. In fact, time.gmtime() and timegm() are each others’ inverse. # -------------------------------------- converting between struct_time and epoch seconds converting from epoch seconds: time.gmtime(epoch_seconds) => returns time.struct_time for UTC time.localtime(epoch_seconds) => returns time.struct_time for local time converting to epoch seconds: calendar.timegm() time.mktime() For example: # For the local timezone t = datetime.datetime.now() print "Epoch Seconds:", time.mktime(t.timetuple()) ============================================================ To get maximum benefit from these emails, play with examples. ============================================================ Warm Regards Lev ################################################################# py037_C_C++.txt ################################################################# Hello, python of the day - day 37 - C C++ Python is an excellent glue language, but it is ~ 70 times slower than C or Java. http://blog.dhananjaynene.com/2008/07/performance-comparison-c-java-python-ruby-jython-jruby-groovy/ and about the same speed as Perl or PHP. http://www.skitoy.com/p/performance-of-python-php-and-perl/160 This article shows examples of getting better performance using NymPy and other tools: http://www.scipy.org/PerformancePython Python can be made faster by using psyco: http://psyco.sourceforge.net/introduction.html Yet another faster implementation of Python (JIT compiler) http://pypy.org/ - fast http://en.wikipedia.org/wiki/PyPy Why would you need to invoke C form Python (or python from C)? May be for speed? May be to use legacy C/C++ libraries. This is a big topic going way beyond the scope of this short intro course. Below I provide links and a couple of simple examples. Note: Parameters are passed (between Python and C/C++) via the various Py* interfaces, or in the SWIG *.i interface file. Debugging is generally via gdb (on Linux), or Microsoft Visual Studio on Windows. # ------------------------------------------------ Here are some ways to integrate between Python and C/C++ http://docs.python.org/extending/extending.html - Python built-in native way http://docs.python.org/library/ctypes.html#module-ctypes # ------------------------------------------------ SWIG: http://www.swig.org/ # ------------------------------------------------ "Instant" module - C/C++ inlining into Python http://heim.ifi.uio.no/~kent-and/software/Instant/doc/Instant.html from Instant import inline add_func = inline("double add(double a, double b){ return a+b; }") print "The sum of 3 and 4.5 is ", add_func(3, 4.5) # ------------------------------------------------ # Example with NumPy arrays converted to C double arrays: from Instant import inline_with_numpy c_code = """ double sum (int n1, double* array1){ double tmp = 0.0; for (int i=0; i new sorted list http://docs.python.org/library/functions.html#sorted note: cmp parameter allows you to specify a custom comparison function which accepts 2 args to compare in Python 3.x cmp parameter is removed. You should use key parameter instead. key parameter allows you to specify a function which accepts one parameter - and returns the value used to compare/sort on a=[4,5,1,2,3] a=sorted(a) # [1,2,3,4,5] a=sorted(a,reverse=True) # [5,4,3,2,1] # -------------------------------------- # numeric sort def numeric_compare(x, y): return x - y a = sorted(a, cmp=numeric_compare) # # another way to achieve numeric sort is to specify a key function a=sorted(a,key=int) # int() will convert float or string to an int - and use it for sorting. In [7]: a=[' 5','4',' 3','2',' 1'] In [13]: b = sorted(a) In [14]: b Out[14]: [' 1', ' 3', ' 5', '2', '4'] # string comparison pays attention to spaces In [15]: b = sorted(a, key=int) In [16]: b Out[16]: [' 1', '2', ' 3', '4', ' 5'] # numeric (int()) comparison # -------------------------------------- # alphanumeric sort a=['C','b','a'] a=sorted(a) # ['C', 'a', 'b'] case-sensitive sort a=sorted(a,key=str.lower) # ['a', 'b', 'C'] alphanumeric sort # or a=sorted(a, cmp=lambda x,y: cmp(x.lower(), y.lower())) # ['a', 'b', 'C'] # -------------------------------------- >>> sorted({1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'}) [1, 2, 3, 4, 5] >>> sorted("This is a test string from Andrew".split(), key=str.lower) ['a', 'Andrew', 'from', 'is', 'string', 'test', 'This'] # -------------------------------------- >>> student_tuples = [ ('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10), ] >>> sorted(student_tuples, key=lambda student: student[2]) # sort by age [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)] # -------------------------------------- # Schwartzian transform - Decorate-Sort-Undecorate idiom >>> decorated = [(student.grade, i, student) for i, student in enumerate(student_objects)] >>> decorated.sort() >>> [student for grade, i, student in decorated] # undecorate [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)] # This idiom works because tuples are compared lexicographically; # the first items are compared; if they are the same then the second items are compared, and so on. # -------------------------------------- def sort_lines(text): """Return text sorted by line, remove empty lines and strip trailing whitespace.""" lines = text.split('\n') non_empty = [line.rstrip() for line in lines if line.strip()] non_empty.sort() return '\n'.join(non_empty) # -------------------------------------- list1=["1","10","3","22","23","4","2","200"] list1.sort() # ['1', '10', '2', '200', '22', '23', '3', '4'] list1 = [int(x) for x in list1] # or list1 = list(map(int, list1)) # [1, 10, 2, 200, 22, 23, 3, 4] list1.sort(key=int) # [1, 2, 3, 4, 10, 22, 23, 200] # -------------------------------------- # more examples: # http://stackoverflow.com/questions/2669059/how-to-sort-alpha-numeric-set-in-python sorted(data, key=lambda item: (int(item.partition(' ')[0]) if item[0].isdigit() else float('inf'), item)) # -------------------------------------- heapq http://docs.python.org/library/heapq.html This module provides an implementation of the heap queue algorithm, also known as the priority queue algorithm. # -------------------------------------- Warm Regards Lev ################################################################# py040_timeit.txt ################################################################# Hello, python of the day - day 40 - time.time(), timeit.Timer(), %time, %timeit # ------------------------------------- # measure wall-clock time in seconds import time t0 = time.time() # do something t1 = time.time() print "run time = ",t1-t0 # ------------------------------------- # note - you can also try time.clock() instead of time.time() on Unix, time.clock() is a tick counter with tick resolution 1-10 milliseconds. on Windows, time.clock() is a microsecond-resolution CPU counter # ------------------------------------- # measure run-time of small bits of Python code # http://docs.python.org/library/timeit.html import timeit s = """ # some python code here """ mytimer = timeit.Timer(stmt=s) # create a timer print "%.2f usec/pass" % (1000000 * mytimer.timeit(number=100000)/100000) # 17.09 usec/pass Note: timeit can be used from command line: python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...] See examples here: http://pleac.sourceforge.net/pleac_python/datesandtimes.html # ------------------------------------- # measuring time in ipython ipython provides 2 magic commands: %time & %timeit examples of usage: In [2]: time sum(range(0,1000000)) CPU times: user 0.11 s, sys: 0.00 s, total: 0.12 s Wall time: 0.12 s Out[2]: 499999500000L In [3]: timeit sum(range(0,1000000)) 10 loops, best of 3: 106 ms per loop # create a stand-alone script similar to this #!/path/to_your_python/python import time for ii in range(0,10): print ii time.sleep(0.2) then from ipython prompt run this command: %time %run test.py # ------------------------------------- Warm Regards Lev ################################################################# py041_pdf.txt ################################################################# Hello, python of the day - day 41 - pdf, TeX Below is a list of links showing/discussing how to use Python to create/manipulate PDF, PostScript, TeX PDFlib - tools for generating and manipulating PDF files http://www.pdflib.com/products/pdflib-family/pdflib/ # -------------------------------------- reportlab - create PDF reports http://www.reportlab.com/software/opensource/ http://code.google.com/p/rst2pdf/ - convert reStructuredText to PDF using reportlab. # -------------------------------------- xhtml to pdf converter written in python: http://www.xhtml2pdf.com/ # -------------------------------------- pyPDF - PDF toolkit http://pybrary.net/pyPdf/ # -------------------------------------- PyX - a Python package for the creation of PostScript and PDF files. http://pyx.sourceforge.net/ # -------------------------------------- PDFMiner is a tool for extracting information from PDF documents. Written entirely in Python. http://www.unixuser.org/~euske/python/pdfminer/ # -------------------------------------- http://stackoverflow.com/questions/6413441/python-pdf-library http://stackoverflow.com/questions/25665/python-module-for-converting-pdf-to-text # -------------------------------------- using TeX from Python http://www.pytex.org/ http://www.profv.de/texcaller/ - written in C http://www.profv.de/texcaller/group__python.html - Texcaller Python interface http://pypi.python.org/pypi/tex - obsolete http://www.texample.net/weblog/2008/oct/24/embedding-python-latex/ http://plastex.sourceforge.net/plastex/ http://tex.stackexchange.com/questions/885/how-can-i-use-latex-from-python Latexmk.py - https://bitbucket.org/ms4py/latexmk.py/wiki/Home # -------------------------------------- Sphinx - a tool to translate reStructuredText source files into various output formats, automatically producing cross-references, indices etc. http://sphinx.pocoo.org/contents.html # -------------------------------------- Note: for perl: http://stackoverflow.com/questions/556786/what-is-the-best-perl-module-to-use-for-creating-a-pdf-from-scratch PDF::API2 PDF:::Create PDF::Template Pdflib - commercial software (the best) Some other tools: http://www.investintech.com/prod_options.htm - C/C++ http://www.cogniview.com/pdf2xl.php http://www.cutepdf.com/products/cutepdf/Default.asp ============================================================ To get maximum benefit from these emails, play with examples. ============================================================ Warm Regards Lev ################################################################# py042_compression.txt ################################################################# Hello, python of the day - day 42 - file compression http://docs.python.org/library/archiving.html # -------------------------------------- gzip http://docs.python.org/library/gzip.html zipfile http://docs.python.org/library/zipfile.html tarfile http://docs.python.org/library/tarfile.html zlib compression http://docs.python.org/library/zlib.html bz2 http://docs.python.org/library/bz2.html # -------------------------------------- # Example - working with zip: import zipfile with ZipFile('spam.zip', 'w') as myzip: myzip.write('eggs.txt') ZipFile.close() # some other functions for zipfile zipFile.namelist() # Return a list of archive members by name. zipFile.open(name[, mode[, pwd]]) # Extract a member "name" from the archive as a file-like object. zipFile.extract(member[, path[, pwd]]) # Extract a member "name" from the archive to a directory zipFile.extractall([path[, members[, pwd]]]) # extract everything Lev ################################################################# py043_commify.txt ################################################################# Hello, python of the day - day 43 - commify commify - format a number with commas every 3 digits There are many solutions http://stackoverflow.com/questions/3909457/whats-the-easiest-way-to-add-commas-to-an-integer-in-python http://stackoverflow.com/questions/1823058/how-to-print-number-with-commas-as-thousands-separators-in-python-2-x Here is what I am using # -------------------------------------------------------------- def commify_test(): """ # tests/demonstrates how commify function is working """ mylist = [ 0, 1, 123, 1234, 1234567890, 123.0, 1234.5, 1234.04, 1234.56789, 1234.99, 1234.999, -0, -1234.5678 ] for n in mylist: print '%012f => %s' % (n,commify(n)) # -------------------------------------------------------------- def commify(n): """ # accepts money amount, adds commas, limits precision to 2 digits after the dot """ if n is None: return None if n < 0: return '-' + commify(-n) n = round(n,2) # only keep 2 digits for cents dollars = int(n) cents = round((n - dollars)*100) dollars = '%d' % dollars cents = '%02d' % cents groups = [] while dollars and dollars[-1].isdigit(): groups.append(dollars[-3:]) dollars = dollars[:-3] return dollars + ','.join(reversed(groups)) + '.' + cents # -------------------------------------------------------------- def clean_dollar2float(s): """ # removes spaces, $-sign, commas, () # ' ($1,234.56) ' => '-1234.56' """ s = re.sub(r'[$,]|\s','',s) if s[0] == '(' and s[-1] == ')': # (123.45) - negative number s = "-" + s[1:-1] return s # -------------------------------------- Here are some other solutions from the web: # -------------------------------------- import locale locale.setlocale(locale.LC_ALL, 'en_US') s= locale.format("%d", 1255000, grouping=True) print s # 1,255,000 # -------------------------------------- def intWithCommas(x): if type(x) not in [type(0), type(0L)]: raise TypeError("Parameter must be an integer.") if x < 0: return '-' + intWithCommas(-x) result = '' while x >= 1000: x, r = divmod(x, 1000) result = ",%03d%s" % (r, result) return "%d%s" % (x, result) # -------------------------------------- def group(number): s = '%d' % number groups = [] while s and s[-1].isdigit(): groups.append(s[-3:]) s = s[:-3] return s + ','.join(reversed(groups)) s = group(-23432432434.34) print s # -23,432,432,434 # -------------------------------------- starting with python 2.7 there is a format specifier: http://docs.python.org/dev/whatsnew/2.7.html#pep-378-format-specifier-for-thousands-separator '{:20,.2f}'.format(18446744073709551616.0) '18,446,744,073,709,551,616.00' # -------------------------------------- In python3.1 you can do the same thing like this: s = format(1234567, ',d') print s # 1,234,567 # -------------------------------------- more # -------------------------------------- import locale locale.setlocale(locale.LC_ALL, '') # empty string for platform's default setting print format(1234, "n") # 1,234 print "{:n}".format(1234) # 1,234 In other large parts of the world, periods instead of commas are used: print format(1234, "n") # 1.234 print "{:n}".format(1234) # 1.234 Output based on the 'd' or ',d' formatting type specifier is unaffected by the use (or non-use) of setlocale() Note - Doesn't work properly with unicode # -------------------------------------- Warm Regards Lev ################################################################# py044_generator_comprehensions.txt ################################################################# Hello, python of the day - day 44 - generator comprehensions List comprehension is a very common idiom. There are also other comprehensions: In [11]: # list comprehension In [12]: [3*x for x in range(10)] Out[12]: [0, 3, 6, 9, 12, 15, 18, 21, 24, 27] In [13]: In [13]: # set comprehension In [14]: {x for x in 'abracadabra' if x not in 'abc'} Out[14]: set(['r', 'd']) In [15]: In [15]: # dict comprehension In [16]: {x: x**2 for x in (2, 4, 6)} Out[16]: {2: 4, 4: 16, 6: 36} In [17]: {2: 4, 4: 16, 6: 36} Out[17]: {2: 4, 4: 16, 6: 36} In [18]: In [18]: # Generator comprehension In [19]: (x*2 for x in range(256)) Out[19]: at 0x234ea2d0> Generator comprehensions are similar to list comprehensions, but much better when you want to save memory. http://www.python.org/dev/peps/pep-0289/ http://stackoverflow.com/questions/4211209/remove-all-the-elements-that-occur-in-1-list-from-another http://stackoverflow.com/questions/47789/generator-expressions-vs-list-comprehension List comprehensions are better when you want to iterate over something multiple times. Or when you want to use any of the list methods. For example, the following code using generator comprehension won't work: def gen(): return (something for something in get_some_stuff()) print gen()[:2] # generators don't support indexing or slicing print [5,6] + gen() # generators can't be added to lists Basically, use a generator expression if all you're doing is iterating once and you care about performance/memory. Warm Regards Lev ################################################################# py045_email_examples.txt ################################################################# Hello, python of the day - day 45 - email examples This email contains 4 examples of how to send email. - test_email_text.py - plain text message to several recepients - test_email_html.py - html message (with DataFrame as html table) - test_email_attach.py - email with one or more attachments - test_email_embed_image.py - html with images embeded into the message itself # -------------------------------------------------------------- """ # test_email_text.py # using smtplib.SMTP.sendmail to send plain text message to several recepients """ import smtplib from email.mime.text import MIMEText smtp_server = 'mail.somedomain.net' mytext = "line 1\nline 2\nline 3\n" sender = 'you@somedomain.com' to_list = ['you@somedomain.com'] # list may contain several addresses to_str = ', '.join(to_list) # comma-separated list as a string mysubject = 'test 123' msg = MIMEText(mytext) # this is an object msg['Subject'] = mysubject msg['From'] = sender msg['To'] = to_str smtp = smtplib.SMTP(smtp_server) # smtp.connect(smtp_server) smtp.sendmail(sender, to_list, msg.as_string()) smtp.quit() # -------------------------------------------------------------- """ # test_email_html.py # send html or plain text message to several recepients # http://stackoverflow.com/questions/882712/sending-html-email-in-python # note - also includes a simple pandas dataframe as a table in the body of the message """ import smtplib from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText from pandas import * smtp_server = 'mail.somedomain.net' sender = 'you@somedomain.com' to_list = ['you@somedomain.com'] # list may contain several addresses to_str = ', '.join(to_list) # comma-separated list as a string mysubject = "Email with a link" # Create message container - the correct MIME type is multipart/alternative. msg = MIMEMultipart('alternative') msg['From'] = sender msg['To'] = ', '.join(to_list) # this is a comma-separated list msg['Subject'] = mysubject # Create the body of the message (a plain-text and an HTML version). mytext = "Hi!\nHow are you?\nHere is the link you wanted:\nhttp://www.python.org" myhtml = """\

Hi!
How are you?
Here is the link you wanted.

Here is a dataframe:

__DATAFRAME1__

Warm Regards
someuser

""" # create a simple dataframe - and form it's HTML df = DataFrame({'Name':['Client1','Client2','Client3'], 'Clicks':[123456,np.nan,4567], 'Money':[123456,np.nan,4567]}) df_html = df.to_html(columns=['Name','Clicks','Money'],na_rep=' - ') myhtml = myhtml.replace("__DATAFRAME1__", df_html) # Record the MIME types of both parts - text/plain and text/html. part1 = MIMEText(mytext, 'plain') part2 = MIMEText(myhtml, 'html') # Attach parts into message container. # According to RFC 2046, the last part of a multipart message is preferred. msg.attach(part1) msg.attach(part2) # Send the message s = smtplib.SMTP(smtp_server) s.sendmail(sender, to_list, msg.as_string()) s.quit() # -------------------------------------------------------------- """ # test_email_attach.py # sends one or more attachments """ import os import smtplib import mimetypes from email import Encoders from email.MIMEText import MIMEText from email.MIMEMultipart import MIMEMultipart from email.MIMEBase import MIMEBase # ------------------------------------- def construct_email_message(sender, to_list, cc_list, subject, attachment_list): outer = MIMEMultipart() outer['From'] = sender outer['To'] = ', '.join(to_list) outer['Cc'] = ', '.join(cc_list) outer['Subject'] = subject outer.preample = 'You will not see this in a MIME-aware mail reader.\n' outer.epilogue = '' # plain text body msg = MIMEText('Please see attached files.\n\n') msg.add_header('Content-Disposition', 'inline') outer.attach(msg) # file attachments for att in attachment_list: if not os.path.isfile(att): continue ctype, encoding = mimetypes.guess_type(att) maintype, subtype = ctype.split('/',1) fp = open(att, 'rb') msg = MIMEBase(maintype, subtype) msg.set_payload(fp.read()) Encoders.encode_base64(msg) msg.add_header('Content-Disposition', 'attachment', filename=att) outer.attach(msg) return outer # ------------------------------------- def send_email_message(sender, recip_list, outer, smtp_server): s = smtplib.SMTP() s.connect(smtp_server) s.sendmail(sender, recip_list, outer.as_string()) s.close() # ##################################### # main execution # ##################################### print 'sending email' smtp_server = 'mail.somedomain.net' sender = 'you@somedomain.com' to_list = ['you@somedomain.com'] # list may contain several addresses cc_list = [] subject = 'test email to send attachment(s)' attach_list = ['path/to/somefile.xls'] # list of several outer = construct_email_message(sender, to_list, cc_list, subject, attach_list) send_email_message(sender, to_list + cc_list, outer, smtp_server) # -------------------------------------------------------------- """ # test_email_embed_image.py # Send an HTML email with embedded images for email clients supporting html # and a plain text message for email clients that don't want to display the HTML. """ from email.MIMEMultipart import MIMEMultipart from email.MIMEText import MIMEText from email.MIMEImage import MIMEImage # Define these once; use them twice! smtp_server = 'mail.somedomain.net' sender = 'you@somedomain.com' to_list = ['you@somedomain.com'] # list may contain several addresses to_str = ', '.join(to_list) # comma-separated list as a string mysubject = "Email with embeded images" # Create the root message and fill in the from, to, and subject headers msg = MIMEMultipart('related') msg['Subject'] = mysubject msg['From'] = sender msg['To'] = to_str msg.preamble = 'This is a multi-part message in MIME format.' # ------------------------------------- # Encapsulate the plain and HTML versions of the message body in an # 'alternative' part, so message agents can decide which they want to display. msgAlternative = MIMEMultipart('alternative') msg.attach(msgAlternative) # first add plain text msgText = MIMEText('This is the alternative plain text message.') msgAlternative.attach(msgText) # then add html text (images referenced by id strings, for example: id_image_cat) msgText = MIMEText(""" Example of sending html message with embedded images
Some HTML text
catfish

Nifty!""", 'html') msgAlternative.attach(msgText) # ------------------------------------- # Read images from disk, associate with corresponding html id-s, and add to message fp = open('files/cat.jpg', 'rb') msg_image_cat = MIMEImage(fp.read()) fp.close() fp = open('files/fish.jpg', 'rb') msg_image_fish = MIMEImage(fp.read()) fp.close() msg_image_cat.add_header('Content-ID', '') msg.attach(msg_image_cat) msg_image_fish.add_header('Content-ID', '') msg.attach(msg_image_fish) # ------------------------------------- # Send the email (this example assumes SMTP authentication is required) import smtplib smtp = smtplib.SMTP() smtp.connect(smtp_server) smtp.sendmail(sender, to_list, msg.as_string()) smtp.quit() # -------------------------------------------------------------- ################################################################# py046_fork_popen_subprocess.txt ################################################################# Hello, python of the day - day 46 - fork, popen, subprocess # -------------------------------------- # subprocess module - allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as os.system os.spawn* os.popen* popen2.* - deprecated since 2.6 commands.* http://docs.python.org/library/subprocess.html # -------------------------------------- # typical usage: import subprocess output = subprocess.check_output("ls -alF", shell=True) print output # -------------------------------------- # replacing os.system() sts = os.system("mycmd myarg") # becomes retcode = subprocess.call("mycmd myarg", shell=True) # or try: retcode = subprocess.call("mycmd myarg", shell=True) if retcode < 0: print >>sys.stderr, "Child was terminated by signal", -retcode else: print >>sys.stderr, "Child returned", retcode except OSError, e: print >>sys.stderr, "Execution failed:", # try these retcode = subprocess.call("/bin/true", shell=True) print " true => ",retcode # 0 retcode = subprocess.call("/bin/false", shell=True) print " false => ",retcode # 1 # -------------------------------------- # a short demo for different commands including pipe and redirection. # (covers probably 99% of all usage cases you ever need import subprocess # list of sublists containing 2 strings - description and command mycmds = [ ['regular echo', """echo 'mama papa' """], ['using wildcard, pipe, xargs, grep', """ls -1 *.py | xargs grep -l mysql | grep test """], ['redirect stdout and stderr to file', """echo 'mama papa' 1>junk 2>&1 """], ['redirect stdout to stderr', """echo 'mama papa' 1>&2 """], ['redirect stderr to null, and stdout to stderr', """echo 'mama papa' 2>/dev/null 1>&2 """] ] for [dd,cc] in mycmds: print '==========' print "dsc = ",dd output = subprocess.check_output(cc, shell=True) print "cmd = ",cc print "out = ",output # -------------------------------------- More examples of using subprocess subprocess.call(["ls", "-l"]) subprocess.check_call(["ls", "-l"]) a = subprocess.check_output(["echo", "Hello World!"]) subprocess.CalledProcessError: some string here subprocess.check_output("ls non_existent_file; exit 0", stderr=subprocess.STDOUT, shell=True) # -------------------------------------- For more advanced custom handling of subprocesses use underlying Popen() method directly. It offers you a lot of args / flexibility: class subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0) p = subprocess.Popen(args) p.poll() # non-blocking check if child process has terminated. # Set and return returncode attribute. p.wait() # Wait for child process to terminate. Set and return returncode attribute. p.communicate() - returns a tuple (stdoutdata, stderrdata). p.terminate() # SIGTERM p.kill() # SIGKILL p.stdin p.stdout p.stderr p.returncode # set by poll() and wait(). a None value indicates that thte process hasn't terminated yet. # -------------------------------------- common real-life tasks may include: - running a command with a timeout (if it takes too long - kill it and all related (zombie) processes - forking several processes to run them in parallel (may be on different servers) There are more than one way to accomplish those tasks. # -------------------------------------- Excellent module to use: http://celeryproject.org/ - asynchronous task queue based on distributed message passing (multiprocessing). You can set things to run in parallel - and with a timeout. # -------------------------------------- How to do it yourself (timeout, parallel processes) is getting too complex for the scope of this emails. Here are some links with examples and discussions. http://stackoverflow.com/questions/1191374/subprocess-with-timeout # running parallel processes http://www.doughellmann.com/articles/pythonmagazine/completely-different/2007-10-multiprocessing/index.html http://www.parallelpython.com/ http://docs.python.org/release/3.1.5/library/multiprocessing.html # requires that the __main__ module be importable by the children http://stackoverflow.com/questions/320232/ensuring-subprocesses-are-dead-on-exiting-python-program http://stackoverflow.com/questions/2760652/how-to-kill-or-avoid-zombie-processes-with-subprocess-module http://stackoverflow.com/questions/4600460/python-run-multi-command-in-the-same-time http://keramida.wordpress.com/2010/01/19/parallel-downloads-with-python-and-gnu-wget/ http://stackoverflow.com/questions/3449901/how-to-close-file-objects-when-downloading-files-over-ftp-using-twisted https://github.com/petewarden/pyparallelcurl http://twistedmatrix.com/pipermail/twisted-python/2010-July/022534.html - uploading multiple files in parallel http://code.google.com/p/python-ssh/ http://code.google.com/p/parallel-ssh/ http://blog.crdlo.com/2010/07/parallel-ssh-tool-roundup.html http://www.thelupine.com/content/pssh-parallel-ssh # ------------------------------------- # here is one of the examples from # http://stackoverflow.com/questions/1191374/subprocess-with-timeout from os import kill from signal import alarm, signal, SIGALRM, SIGKILL from subprocess import PIPE, Popen def run(args, cwd = None, shell = False, kill_tree = True, timeout = -1, env = None): ''' Run a command with a timeout after which it will be forcibly killed. ''' class Alarm(Exception): pass def alarm_handler(signum, frame): raise Alarm p = Popen(args, shell = shell, cwd = cwd, stdout = PIPE, stderr = PIPE, env = env) if timeout != -1: signal(SIGALRM, alarm_handler) alarm(timeout) try: stdout, stderr = p.communicate() if timeout != -1: alarm(0) except Alarm: pids = [p.pid] if kill_tree: pids.extend(get_process_children(p.pid)) for pid in pids: # process might have died before getting to this line # so wrap to avoid OSError: no such process try: kill(pid, SIGKILL) except OSError: pass return -9, '', '' return p.returncode, stdout, stderr def get_process_children(pid): p = Popen('ps --no-headers -o pid --ppid %d' % pid, shell = True, stdout = PIPE, stderr = PIPE) stdout, stderr = p.communicate() return [int(p) for p in stdout.split()] if __name__ == '__main__': print run('find /', shell = True, timeout = 3) print run('find', shell = True) # -------------------------------------- # replacing backquotes # (backquotes are obsolete and should not be used in new code.) output=`mycmd myarg` # obsolete # becomes output = subprocess.check_output(["mycmd", "myarg"]) or output = subprocess.check_output("mycmd myarg", , shell=True) output=`dmesg | grep hda` # becomes output = subprocess.check_output("dmesg | grep hda", shell=True) # or p1 = subprocess.Popen(["dmesg"], stdout=PIPE) p2 = subprocess.Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] # -------------------------------------- # Here is what I use for simple cases (when I still want to see if there was an error): import subprocess as sub def run_cmd(cmd): print "running cmd = ", cmd p = sub.Popen(cmd, shell=True, stdout=sub.PIPE, stderr=sub.PIPE) output, errors = p.communicate() err_lst = [] if p.returncode != 0: err_lst.append("Error code = " + str(p.returncode)) if len(errors) > 0 : err_lst.append(errors) err_str = ', '.join(err_lst) return (output, err_str) cmd_lst = [ """echo;echo;echo papa;echo "mama" 2>&1 """, """ grep word b_*py | grep -v "#" """, """ grep word jjjjj """, """ touch /bin/11 """, """ cp aaa bbb """, """ /bin/false """ ] for cmd in cmd_lst : (output, err_str) = run_cmd(cmd) print "OUT = ",output print "ERR = >>" + err_str + "<<" print "" # -------------------------------------- # a python function that opens a subprocess, # collects (and shows) stdout adn stderr as it is running # waits for completion, outputs resutls. # http://stackoverflow.com/questions/7729336/how-can-i-print-and-display-subprocess-stdout-and-stderr-output-without-distorti # Helper function to add the O_NONBLOCK flag to a file descriptor def make_async(fd): fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK) # Helper function to read some data from a file descriptor, ignoring EAGAIN errors def read_async(fd): try: return fd.read() except IOError, e: if e.errno != errno.EAGAIN: raise e else: return '' process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) make_async(process.stdout) make_async(process.stderr) stdout = str() stderr = str() returnCode = None while True: # Wait for data to become available select.select([process.stdout, process.stderr], [], []) # Try reading some data from each stdoutPiece = read_async(process.stdout) stderrPiece = read_async(process.stderr) if stdoutPiece: print stdoutPiece, if stderrPiece: print stderrPiece, stdout += stdoutPiece stderr += stderrPiece returnCode = process.poll() if returnCode != None: return (returnCode, stdout, stderr) # -------------------------------------- # Replacing the os.spawn family¶ # P_NOWAIT example: pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg") # becomes pid = Popen(["/bin/mycmd", "myarg"]).pidP_WAIT example: retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg") # becomes retcode = call(["/bin/mycmd", "myarg"]) # -------------------------------------- Replacing os.popen(), os.popen2(), os.popen3() pipe = os.popen("cmd", 'r', bufsize) ==> pipe = Popen("cmd", shell=True, bufsize=bufsize, stdout=PIPE).stdout pipe = os.popen("cmd", 'w', bufsize) ==> pipe = Popen("cmd", shell=True, bufsize=bufsize, stdin=PIPE).stdin (child_stdin, child_stdout) = os.popen2("cmd", mode, bufsize) ==> p = Popen("cmd", shell=True, bufsize=bufsize, stdin=PIPE, stdout=PIPE, close_fds=True) (child_stdin, child_stdout) = (p.stdin, p.stdout) etc. # -------------------------------------- # example of old syntax # -------------------------------------- # popen - starting a process - and getting output Popen.fifo(…) p=os.popen("grep " + name + " " + config_file) return p.read().split()[1] def sys_exec (cmd): fhs = popen2.popen2 (cmd) result = string.strip(fhs[0].readline()) fhs[0].close fhs[1].close return result ret = os.system(cmd) # -------------------------------------- # forking using old os.fork() def counter(count): for i in range(count): time.sleep(10) print '[%s]--> %s ' %(os.getpid(), i) for i in range(10): pid = os.fork() if pid != 0: print 'Process %d spawned' %pid else: counter(10) os._exit(0) print 'Main process exiting.' # -------------------------------------- errno http://docs.python.org/library/errno.html Warm Regards Lev ################################################################# py047_iterator_generator_itertools.txt ################################################################# Hello, python of the day - day 47 - iterators, generators, itertools ======================================= iterable - something you can iterate through - list, string, file, etc. mylist = [x*x for x in range(3)] # the whole list is created and stored in memory for i in mylist : print(i) ======================================= iterator - an object which has method next() to return next element. A list is not an iterator. A list is an iterable. To create an iterator object from a list python uses function iter(mylist) So when you are doing "for i in mylist:", python actually does 2 steps: step1 - use iter() method to create iterator object step2 - repeatedly use next() method of this object to get next element ======================================= generator - an object which generates next value on the fly. Benefit - generator doesn't have to take memory to keep the whole list. mygenerator = (x*x for x in range(3)) # list is not created here, only a generator routine to generate values for i in mygenerator : print(i) Note - the syntax is basicaly the same as with the list, but we use () instead of []. generator expression is like list comprehension - but in round parentheses Another example: (x*y for x in range(10) for y in bar(x)) Note: the parentheses can be omitted on calls with only one argument. Note: you can NOT perform "for i in mygenerator" a second time since generators can only be used once. (we have looped through the inner loop inside the generator to the end - nothing left) ======================================= Note - to create a generator you can write a function instead of using round brackets. This would be a special function - it will use keyword "yield" instead of "return". def createGenerator(): i=-1 while i<2: i=i+1 yield i*i mygenerator = createGenerator() # create a generator object which knows how to give a sequence of 0,1,4 for i in mygenerator: print(i) # outputs 0, 1, 4 for i in mygenerator: print(i) # no output # You can think of a generator as an alias for a function, like this: for i in createGenerator(): print i ======================================= If you have your own class, and you want to make it iterable, you need to define method __iter__() ======================================= The above explanation was mainly derived from this page - which has more stuff: http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained More good links: http://www.dabeaz.com/generators/ http://wiki.python.org/moin/Generators http://docs.python.org/library/itertools.html - tools to create different types of iterators http://docs.python.org/reference/expressions.html#generator-expressions ======================================= Warm Regards Lev ################################################################# py048_generators_examples.txt ################################################################# Hello, python of the day - day 48 - generators examples # ------------------------------------- The most common way to generate a list of sequential numbers is to use functions range() or xrange() range() creates a list, xrange() gives a generator object. # xrange([start], stop[, step]) - built-in function internally implemented in C. # This function is very similar to range(), but returns an "xrange object" # instead of a list. This is an opaque sequence type which yields the same # values as the corresponding list, without actually storing them all # simultaneously. The advantage of xrange() over range() is minimal # (since xrange() still has to create the values when asked for them) # except when a very large range is used on a memory-starved machine # or when all of the range's elements are never used # (such as when the loop is usually terminated with break). # Let's create our own simplified version of xrange # ------------------------------------- def myxrange(a1=None,a2=None,step=1): if a1 == None or step == 0: return if a2 == None: start = 0 stop = a1 else: start = a1 stop = a2 while (step > 0 and start < stop) or (step < 0 and start > stop): start = start + step yield start-step for ii in myxrange(4): print ii for ii in myxrange(1,4): print ii for ii in myxrange(1,10,2): print ii for ii in myxrange(10,5,-1): print ii # ------------------------------------- # similarly let's create a generator function to return a list of characters def xchars(c1,c2): i1 = ord(c1) - 1 i2 = ord(c2) while i1 < i2: i1 += 1 yield chr(i1) for c in xchars('A','D'): print c # note - you can also create a generator using () syntax: xc = (chr(x) for x in xrange(ord('a'),ord('d')+1)) for c in xc: print c # ------------------------------------- # of course you can also get characters like this: import string list(string.uppercase[0:5]) [c for c in string.uppercase] string.ascii_letters # 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' string.ascii_lowercase # 'abcdefghijklmnopqrstuvwxyz' string.ascii_uppercase # 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' string.letters # 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' string.lowercase # 'abcdefghijklmnopqrstuvwxyz' string.uppercase # 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' string.digits # '0123456789' string.hexdigits # '0123456789abcdefABCDEF' string.octdigits # '01234567' string.printable # '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c' string.punctuation # '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~' string.whitespace # '\t\n\x0b\x0c\r ' # ------------------------------------- Warm Regards Lev ################################################################# py049_one_liners.txt ################################################################# Hello, python of the day - day 49 - one-liners # ------------------------------------- from sys import platform; print platform from socket import gethostname; print gethostname() import os; print os.uname()[1] import platform; print platform.uname()[1] import platform; print platform.node() # ------------------------------------- # start HTTP server to serve current directory as http://localhost:8000 python -m SimpleHTTPServer 8000 # ------------------------------------- import pprint;pprint.pprint(zip(('Byte', 'KByte', 'MByte', 'GByte', 'TByte'), (1 << 10*i for i in xrange(5)))) # ------------------------------------- from compileall import compile_dir; compile_dir("C:/Python24/") # ------------------------------------- # running python oneliner from unix prompt python -c "for ii in xrange(10):print'hello' " # ------------------------------------- # Editing a list of files in place # Warning: if you run this, it will edit all files in your directory # substitute all words 'mama' to words 'papa' in all files in current directory python -c "import sys,os,re,fileinput;a=[i[2] for i in os.walk('.') if i[2]] [0];[sys.stdout.write(re.sub('mama','papa',j)) for j in fileinput.input(a,inplace=1)]" # remove all '\r' characters (MS DOS carriage returns) python -c "import sys,os,os.path,re,fileinput;a=[f for f in os.listdir('.') if not os.path.isdir(f)];[sys.stdout.write(re.sub(r'\r','',j)) for j in fileinput.input(a,inplace=1)]" # note - in this particular case it is easier to do it using perl: perl -pi -e 's/\r//' * # ------------------------------------- # Reimplementing cut # Print every line from an input file but remove the first two fields. python -c "import sys;[sys.stdout.write(' '.join(line.split(' ')[2:])) for line in sys.stdin]" < input.txt # ------------------------------------- # Cramming Python into Makefiles # This script generates a "one-liner" from make's point of view. import sys,re def main(): fh = open(sys.argv[1],'r') lines = fh.readlines() print '\tpython2.2 -c "`printf \\"if 1:\\n\\' for line in lines: line = re.sub('[\\\'\"()]','\\\g<0>',line) # grab leading white space (should be multiples of 4) and makes them into # tabs wh_spc_len = len(re.match('\s*',line).group()) sys.stdout.write('\t') sys.stdout.write(wh_spc_len/4*'\\t'+line.rstrip().lstrip()) sys.stdout.write('\\n\\\n') print '\t\\"`"' if __name__=='__main__': main() # ------------------------------------- # echo unicode character: python -c "print unichr(234)" # ------------------------------------- # Apply regular expression to lines from stdin [another command] | python -c "import sys,re;[sys.stdout.write(re.sub('PATTERN', 'SUBSTITUTION', line)) for line in sys.stdin]" # ------------------------------------- # Modify lines from stdin using map python -c "import sys; tmp = lambda x: sys.stdout.write(x.split()[0]+'\t'+str(int(x.split()[1])+1)+'\n'); map(tmp, sys.stdin);" # ------------------------------------- # Display List of all users on Unix-like systems print '\n'.join(line.split(":",1)[0] for line in open("/etc/passwd")) # ------------------------------------- # Compress CSS file python -c 'import re,sys;print re.sub("\s*([{};,:])\s*", "\\1", re.sub("/\*.*?\*/", "", re.sub("\s+", " ", sys.stdin.read())))' # ------------------------------------- # Decode string written in Hex python -c "print ''.join(chr(int(''.join(i), 16)) for i in zip(*[iter('474e552773204e6f7420556e6978')]*2))" # ------------------------------------- # Retrieve content text from HTTP data python -c "import sys; print sys.stdin.read().replace('\r','').split('\n\n',2)[1]"; # ------------------------------------- # Prints file extension print '~/python/one-liners.py'.split('.')[-1] # ------------------------------------- # Escapes content from stdin # This can be used to convert a string into a "url safe" string python -c "import urllib, sys ; print urllib.quote_plus(sys.stdin.read())"; # ------------------------------------- # Reverse lines in stdin python -c "import sys; print '\n'.join(reversed(sys.stdin.read().split('\n')))" # ------------------------------------- # Print top 10 lines of stdin python -c "import sys; sys.stdout.write(''.join(sys.stdin.readlines()[:10]))" < /path/to/your/file # ------------------------------------- Sony's Open Source command line tool for performing python one liners using unix-like pipes They call it "The Pyed Piper" or pyp. It's pretty similar to the -c way of executing python, but it imports common modules and has it's own preset variable that help with splitting/joining, line counter, etc. You use pipes to pass information forward instead of nested parentheses, and then use your normal python string and list methods. Here is an example from the homepage: Here, we take a linux long listing, capture every other of the 5th through the 10th lines, keep username and file name fields, replace "hello" with "goodbye", capitalize the first letter of every word, and then add the text "is splendid" to the end: ls -l | pyp "pp[5:11:2] | whitespace[2], w[-1] | p.replace('hello','goodbye') | p.title(),'is splendid'" and the explanation: This uses pyp's built-in string and list variables (p and pp), as well as the variable whitespace and it's shortcut w, which both represent a list based on splitting each line on whitespace (whitespace = w = p.split()). The other functions and selection techniques are all standard python. Notice the pipes ("|") are inside the pyp command. http://code.google.com/p/pyp/ http://opensource.imageworks.com/?p=pyp # ------------------------------------- # Multiple Each Item in a List by 2 int map(lambda x: x * 2, range(1,11)) # ------------------------------------- # Sum a List of Numbers print sum(range(1,1001)) # ------------------------------------- # Verify if Exists in a String wordlist = ["scala", "akka", "play framework", "sbt", "typesafe"] tweet = "This is an example tweet talking about scala and sbt." print map(lambda x: x in tweet.split(),wordlist) # ------------------------------------- # Read in a File print open("ten_one_liners.py").readlines() # ------------------------------------- # Happy Birthday to You! print map(lambda x: "Happy Birthday to " + ("you" if x != 2 else "dear Name"),range(4)) # ------------------------------------- # Filter list of numbers print reduce(lambda(a,b),c: (a+[c],b) if c > 60 else (a,b + [c]), [49, 58, 76, 82, 88, 90],([],[])) # ------------------------------------- # Fetch and Parse an XML web service from xml.dom.minidom import parse, parseString import urllib2 # note - i convert it back into xml to pretty print it print parse(urllib2.urlopen("http://search.twitter.com/search.atom?&q=python")).toprettyxml(encoding="utf-8") # ------------------------------------- # Find minimum (or maximum) in a List print min([14, 35, -7, 46, 98]) print max([14, 35, -7, 46, 98]) # ------------------------------------- Parallel Processing import multiprocessing import math print list(multiprocessing.Pool(processes=4).map(math.exp,range(1,11))) # ------------------------------------- # Sieve of Eratosthenes # There is no Sieve of Eratosthenes operator, but that is hardly a constraint. n = 50 # We want to find prime numbers between 2 and 50 print sorted(set(range(2,n+1)).difference(set((p * f) for p in range(2,int(n**0.5) + 2) for f in range(2,(n/p)+1)))) # ------------------------------------- # largest number that can be represented by 8 bytes print '\n'.join("%i Byte = %i Bit = largest number: %i" % (j, j*8, 256**j-1) for j in (1 << i for i in xrange(8))) # ------------------------------------- # Function that returns the set of all subsets of its argument f = lambda x: [[y for j, y in enumerate(set(x)) if (i >> j) & 1] for i in range(2**len(set(x)))] >>>f([10,9,1,10,9,1,1,1,10,9,7]) [[], [9], [10], [9, 10], [7], [9, 7], [10, 7], [9, 10, 7], [1], [9, 1], [10, 1], [9, 10, 1], [7, 1], [9, 7, 1], [10, 7, 1], [9, 10, 7, 1]] # ------------------------------------- # Alternately (shorter, more functional version): f = lambda l: reduce(lambda z, x: z + [y + [x] for y in z], l, [[]]) # ------------------------------------- Decode a base64 encoded file import base64, sys; base64.decode(open(sys.argv[1], "rb"), open(sys.argv[2], "wb")) # ------------------------------------- Note - the above examples were collected from many pages on the web, including: http://wiki.python.org/moin/Powerful%20Python%20One-Liners/ http://codeblog.dhananjaynene.com/2011/06/10-python-one-liners-to-impress-your-friends/ http://linuxgazette.net/issue96/orr.html http://www.catonmat.net/blog/top-ten-one-liners-from-commandlinefu-explained/ Warm Regards someuser someuser@domain.com | ################################################################# py050_one_liners2.txt ################################################################# Hello, python of the day - day 50 - one-liners 2 # ------------------------------------- One way to fit a program with blocks into one-liner is to use exec function: exec """for ii in [1,2]:\n print'line1'\n print"line2"\nprint'DONE'\n""" # ------------------------------------- Another way is to use list comprehensions - each LC is a separate block. exec "def f(a):print a"; [(f("line1"),f("line2")) for ii in [1,2]]; print"mama"; [(f("line3"),f("line4")) for ii in [1,2]] Note - in list comprehension you can return any user-defined function f() - it is OK because it is an object. But you can NOT use a print statement directly in LC. A workaround is to wrap the print into your own function - as I did. # ------------------------------------- Here is a short idiom to parse a comma-separated string into a sorted list of integers (duplicates removed): import re s1 = '55 , junk 1 , more junk 2.0, 1, 55, 33' s1 = re.sub(r'[^0123456789,.]','',s1) # remove everything except numbers, dots and commas l1 = [int(float(x)) for x in s1.split(',') if x is not ''] # split, remove empty, convert to int l1 = sorted(list(set(l1))) # remove duplicates and sort print l1 # ------------------------------------- Warm Regards Lev ################################################################# py051_add_method_to_object.txt ################################################################# Hello, python of the day - day 51 - add properties and methods at runtime # ------------------------------------- Question: How to add a value property to a class or an object class A: pass # adding a class variable A.a = 5 print A.a # 5 print A.__dict__ # {'__doc__': None, '__module__': '__main__', 'a': 5} aa=A() print aa.a # 5 - class variable can be accessed from any object print aa.__dict__ # {} - no variables in the object itself aa.a = 77 # this actually creates a variable in the object which masks a class variable print aa.__dict__ # {'a': 77} print aa.__class__.__dict__ # {'__doc__': None, '__module__': '__main__', 'a': 5} # To change class variable you can do one of 2 things: A.a = 4 # set it from class, not from object aa.__class__.a = 6 # start from object, get its class - and then set the variable # adding an object variable aa.b=6 print aa.b # 6 print aa.__dict__ # {'a':77, 'b': 6} print aa.__class__.__dict__ # {'__doc__': None, '__module__': '__main__', 'a': 6} # ------------------------------------- Question: How to add a method to an object (not to a class) at run time? This can be separated into 3 questions: 1. how to add an object method which will be available for all objects of this class 2. how to add an object method only to a specific instance (specific object) (answer to this - use module "types") 3. how to add a class method at run time (use the "classmethod() function) Look how this is done in this script: # ------------------------------------- import types # allows to bind functions to objects # ----------------- class A: # instance method def im(cls, ss): print "im - ",ss # class method @classmethod def cm(cls, ss): print "cm - ",ss # ----------------- def foo1(self, ss): print "foo1 - ",ss # ----------------- def foo2(self,ss): print "foo2 - ",ss # ----------------- def foo3(self,ss): print "foo3 - ",ss # ----------------- A.cm("A.cm") # cm # A.im("A.im") # error, can not call instance method from a class # ----------------- # create an object a1 = A() a1.cm("a1.cm") # cm a1.im("a1.im") # im # ----------------- # add instance method to all objects (existing and future) A.foo1 = foo1 a1.foo1('a1.foo1') # foo1 # ----------------- # add instance method to specific object a1 a1.foo2 = types.MethodType(foo2,a1) a1.foo2('a1.foo2') # foo2 # ----------------- # testing on another object a2=A() a2.foo1('a2.foo1') # works # a2.foo2() # error, it only bound to a1 # ----------------- # adding a class method A.foo3 = classmethod(foo3) A.foo3('A.foo3') # works a2.foo3('a2.foo3') # works print A.__dict__ print a1.__class__.__dict__ print a1.__dict__ print a2.__dict__ # ----------------- del A.foo1 # remove method from all objects #a2.foo1('a2.foo1') # error, this method doesn't exist any more del a1.foo2 # a1.foo2('a1.foo2') # error, this moethod doesn't exist any longer for object a1 del A.foo3 # a1.foo3('a1.foo3') # error, this moethod doesn't exist any longer # ------------------------------------- # In the above script we used types.MethodType() to bind a method to a specific instance. a1.foo2 = types.MethodType(foo2,a1) # Another way to do the same is to use __get__() method. # All functions have this method. # It is used to bind functions (to an object or to a class): # http://docs.python.org/release/2.6.6/howto/descriptor.html#functions-and-methods a1.foo2 = foo2.__get__(a1) # ------------------------------------- # Here are some links used in preparing this email: # http://stackoverflow.com/questions/972/adding-a-method-to-an-existing-object # http://stackoverflow.com/questions/962962/python-changing-methods-and-attributes-at-runtime?lq=1 # http://stackoverflow.com/questions/533382/dynamic-runtime-method-creation-code-generation-in-python # http://stackoverflow.com/questions/1325673/python-how-to-add-property-to-a-class-dynamically # http://code.activestate.com/recipes/81732-dynamically-added-methods-to-a-class/ # ------------------------------------- Warm Regards Lev ################################################################# py052_binary_files.txt ################################################################# Hello, python of the day - day 52 - binary files Here are just few rules to follow: # ------------------------------------- When writing/reading binary data to/from file, you will need to pack/unpack it. For this you can use the "struct" module. http://docs.python.org/library/struct.html import struct struct.pack(fmt, v1, v2, ...) # packs values v1, v2, ... into a binary buffer according format, returns it. struct.unpack(fmt, string) # unpacks a binary string according to the format, returns a tuple # ------------------------------------- When opening the binary file for writing or reading, add letter 'b' to the mode ('wb' - write binary, 'ab' - append binary, 'rb' - read binary), for example: fh = open('junk', 'wb') # Typical use: fh = open(fname,'wb') while ...: # prepare buffer fh.write(buffer) # accepts string or binary buffer fh.close() # or fh = open(fname,'rb') while ...: buffer = fh.read(Nbytes) # process buffer fh.close() # ------------------------------------- # Note: do not mix text and binary file operations # ------------------------------------- # Example - writing to a binary file: import struct fh = open('junk', 'wb') fh.write('mama') buff = struct.pack('4i', 1,2,3,4) fh.write(buff) fh.write('mama') fh.close() # ------------------------------------- This is how file 'junk' now looks in vim: mama^A^@^@^@^B^@^@^@^C^@^@^@^D^@^@^@mama As you see, each integer number is recorded as a set of 4 bytes (^@ means byte filled with zero bits). The least significant byte goes first. Note: Intel CPUs are "little-endian" So most probably your computer is "little-endian" whether it is PC, or Mac or Linux. # To see byteorder import sys print sys.byteorder # read more about lettle endian vs big-endian http://en.wikipedia.org/wiki/Endianness http://stackoverflow.com/questions/4207326/parse-wav-file-header http://stackoverflow.com/questions/1698036/convert-little-endian-string-to-integer http://stackoverflow.com/questions/846038/convert-a-python-int-into-a-big-endian-string-of-bytes # ------------------------------------- # Example - reading from a binary file import struct fh = open('junk', 'wb') buff = struct.pack('4i', 1,2,3,4) fh.write(buff) fh.close() fh = open('junk', 'rb') buf= '' while buf = fh.read(4): print struct.unpack('i', buf) # ------------------------------------- More examples: http://stackoverflow.com/questions/1035340/reading-binary-file-in-python http://docs.python.org/tutorial/inputoutput.html http://stackoverflow.com/questions/7141091/parsing-binary-files-with-python http://stackoverflow.com/questions/1163459/reading-integers-from-binary-file-in-python http://stackoverflow.com/questions/7566825/python-parsing-binary-stl-file http://stackoverflow.com/questions/11647306/python-reading-parsing-binary-file http://stackoverflow.com/questions/9259861/python-read-array-in-binary-file # ------------------------------------- # example - hexdump in Python # ------------------------------------- #!/usr/bin/env python # Usage: xdump file1 file2 # cat filename | xdump import sys mymap = ['.' for ii in range(0,256)] for ii in range(32,127): mymap[ii] = chr(ii) # ------------------------------------- def hexdump (fh): global mymap pos = 0 while 1: buf = fh.read (16) length = len (buf) if length == 0: return hex = "" asc = "" for i in range (0, length): c = buf[i] if i > 0 and i % 4 == 0: hex = hex + " " hex = hex + ("%02x " % ord (c)) asc = asc + mymap[ord (c)] print "%05x: %-51s %s" % (pos, hex, asc) pos = pos + 16 # ------------------------------------- # main execution # ------------------------------------- if len (sys.argv) < 2: hexdump (sys.stdin) else: for name in sys.argv[1:]: f = open (name, "r") hexdump (f) f.close () # ------------------------------------- # Here are many examples of how different people have implemented hexdump http://code.activestate.com/recipes/577243-hex-dump/ http://code.activestate.com/recipes/142812-hex-dumper/ http://hex-dump.blogspot.com/2006/02/hex-dump-tools-written-in-python.html http://c2.com/cgi/wiki?HexDumpInManyProgrammingLanguages http://nedbatchelder.com/code/utilities/hexdump.py http://www.koders.com/python/fid5B237323AC1CCBC45CE3D0730091C92EFBE4B1E3.aspx?s=dbi http://code.google.com/p/pynfc/source/browse/branches/pynfc-0.0.7/rfid/libnfc/hexdump.py?spec=svn22&r=22 http://www.alexonlinux.com/hex-dump-functions http://stackoverflow.com/questions/10998079/scapy-hexdump http://jessekornblum.com/tools/hexdump/hexdump.py # ------------------------------------- # You can use binary file operations to write your own file copy utility. http://stackoverflow.com/questions/123198/how-do-i-copy-a-file-in-python def copyfile(source, dest, buffer_size=1024*1024): """ # Copy a file from source to dest. # source and dest can either be strings # or any object with a read or write method, # like StringIO for example. """ if not hasattr(source, 'read'): source = open(source, 'rb') if not hasattr(dest, 'write'): dest = open(dest, 'wb') while 1: copy_buffer = source.read(buffer_size) if copy_buffer: dest.write(copy_buffer) else: break source.close() dest.close() # Alternatively you can invoke external system command, # or use shell utilities frm module "shutil": http://docs.python.org/library/shutil.html import shutil shutil.copyfile(src, dst) shutil.move(src,dest) shutil.copy2('/dir/file.ext', '/new/dir/newname.ext') shutil.copy2('/dir/file.ext', '/new/dir') # Note - copy2 preserves modification and access info (mtime and atime) in the file metadata. # ------------------------------------- # yet another example of copying a file and a directory tree import os import shutil import tempfile filename1 = tempfile.mktemp (".txt") open (filename1, "w").close () filename2 = filename1 + ".copy" print filename1, "=>", filename2 shutil.copy (filename1, filename2) if os.path.isfile (filename2): print "Success" dirname1 = tempfile.mktemp (".dir") os.mkdir (dirname1) dirname2 = dirname1 + ".copy" print dirname1, "=>", dirname2 shutil.copytree (dirname1, dirname2) if os.path.isdir (dirname2): print "Success" # ------------------------------------- Warm Regards Lev ################################################################# py053_globals_locals_dir_dict.txt ################################################################# Hello, python of the day - day 53 - globals(), locals(), dir(), vars(), __dict__ Each module has its own private symbol table, where it keeps names of variables, functions, etc. and (important) which is used as the global symbol table by all functions defined in the module. Thus, the author of a module can use global variables in the module without worrying about accidental clashes with a users global variables. Note: you can always access globals using fully-qualifying name: modname.itemname # ------------------------------------- You can access a global variable in functions. But if you want to assign to this global variable inside the function, you will have to explicitly declare it as global within this function. Otherwise Python will assign value to a local var with the same name. myvar = 5 def func1(): myvar = 10 print myvar func1() # 10 print myvar # 5 def func2(): global myvar myvar = 10 print myvar func2() # 10 print myvar # 10 http://stackoverflow.com/questions/423379/using-global-variables-in-a-function-other-than-the-one-that-created-them # ------------------------------------- globals() built-in function, returns the dictionary of the current module's global symbol table. Note: inside a function or method, this is the module where it is defined, not the module from which it is called. http://docs.python.org/library/functions.html#globals # Try these commands: type(globals()) globals().keys() sorted(globals().keys()) for k, v in globals().items(): print k, "=", v # ------------------------------------- locals()¶ built-in function that update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks. Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter. http://docs.python.org/library/functions.html#locals # ------------------------------------- vars() returns either a dictionary of the current namespace (if called with no argument) or the dictionary of the argument: var(x) returns x.__dict__ # ------------------------------------- dir(x) returns x.__dict__.keys(), and also returns attributes of its class hierarchy (goes recursively up) http://stackoverflow.com/questions/7969949/whats-the-difference-between-globals-locals-and-vars http://stackoverflow.com/questions/980249/difference-between-dir-and-vars-keys-in-python # ------------------------------------- Be careful with exec and eval in Python. Especially inside or outside a function. Without any additional instructions, exec uses the current global and local namespaces to execute your code. But you can provide globals and locals dictionaries explicitly like this: exec("x+=1", globals(),locals()) exec "x+=1" in globals(),locals() # ----------------- x=5 exec("x+=1") print(x) # 6 # ----------------- x=5 def app(): x=5 exec("x+=1") print x app() # 6 print x # 5 # ----------------- x=5 def app(): exec("global x;x+=1") print x app() print x # ----------------- x=5 def app(): x=5 d = {'x': x} # or d=locals().copy() exec("x+=1", d) x = d['x'] print x app() print x # ----------------- http://late.am/post/2012/04/30/the-exec-statement-and-a-python-mystery http://lucumr.pocoo.org/2011/2/1/exec-in-python/ http://www.velocityreviews.com/forums/t335671-how-can-i-exec-in-global.html http://stackoverflow.com/questions/5657519/using-exec-with-python-3-2 # ------------------------------------- Warm Regards Lev ################################################################# py054_find_modules.txt ################################################################# Hello, python of the day - day 54 - find installed modules # ------------------------------------------------ Where and how you get modules ? Python web site - more than 400 modules http://docs.python.org/modindex.html PYPI - Python Package Index - more than 23,000 packages http://pypi.python.org/pypi PyPM - Python Package Management Utility from ActiveState - more than 16,000 packages http://code.activestate.com/pypm/ setuptools - utilities to download, build, install, upgrade, and uninstall python packages http://pypi.python.org/pypi/setuptools easy_install - main utility to do installs http://peak.telecommunity.com/DevCenter/EasyInstall http://packages.python.org/distribute/easy_install.html pip - tool to install/manage python packages found in PyPI http://www.pip-installer.org/en/latest/index.html # ------------------------------------------------ if you can import it - it is installed How to find which modules are installed and available for import. In ipython you can type import This is really cool: import pydoc pydoc.gui() (then click the 'open browser' button) import sys sys.modules # this shows only already imported modules also you can ge a list of modules used bt a script: http://docs.python.org/library/modulefinder.html # ------------------------------------------------ To find where the actual module is on your computer, you can run help on it or use __file__ : import pandas help('pandas') # or import pandas pandas.__file__ # '/opt/py27/lib/python2.7/site-packages/pandas-0.7.2-py2.7-linux-x86_64.egg/pandas/__init__.pyc' Note - this only works for pure-python modules. For compiled modules you may need to simply find through directory tree >>> import datetime >>> datetime.__file__ '/opt/py27/lib/python2.7/lib-dynload/datetime.so' cd /opt/py27 find . -follow -type f -name datetime\* -print ./lib/python2.7/site-packages/django/utils/datetime_safe.pyc ./lib/python2.7/site-packages/django/utils/datetime_safe.py ./lib/python2.7/lib-dynload/datetime.so ./include/python2.7/datetime.h ./lib64/python2.7/site-packages/django/utils/datetime_safe.pyc ./lib64/python2.7/site-packages/django/utils/datetime_safe.py ./lib64/python2.7/lib-dynload/datetime.so ================================================================ start python in verbose mode - you will see where it is reading from python -v You can also do this to see directories: import sys import os for p in sys.path: if os.path.isdir(p): print p Also at the command-line, you can use pydoc modules ================================================================ python -c "help('modules')" python -vc "help('modules')" Note - if the above command gives you an error, try to debug to find the offending module. put help('modules') into a python script - and run it under debugger: python -m pdb show_modules.py Once you know the problem - add exceptions to /usr/lib/pythonXXX/pkgutil.py ================================================================ if pip is installed, then from unix prompt run: pip freeze ================================================================ http://stackoverflow.com/questions/739993/get-a-list-of-installed-python-modules ================================================================ In a script: http://docs.python.org/library/pkgutil.html import pkgutil mod1 = sorted([x[1] for x in pkgutil.iter_modules()]) ================================================================ import sys import os for p in sys.path: if os.path.isdir(p): print os.listdir(p) ================================================================ http://code.activestate.com/recipes/440501-list-information-about-installed-python-packages-a/ http://stackoverflow.com/questions/269795/how-do-i-find-the-location-of-python-module-sources ================================================================ https://chris-lamb.co.uk/2010/04/22/locating-source-any-python-module/ A shell function allowing to easily navigate to localtion of a python library cdp () { cd "$(python -c "import os.path as _, ${1}; print _.dirname(_.realpath(${1}.__file__[:-1]))")" } # Here are the functions I have in my .bashrc where () { echo "$(python -c "import os.path as _, ${1}; print _.dirname(_.realpath(${1}.__file__[:-1]))")" } packages () { python -c "print '\n'.join(sorted([str(x) for x in __import__('pkg_resources').working_set]))" } modules () { python -c "import pkgutil; print '\n'.join(sorted([m[1] for m in pkgutil.iter_modules()]))" } use it like this form unix prompt: where pandas modules | grep pandas packages | grep pandas cdp pandas etc. ================================================================ http://trac.assembla.com/yolk/ Yolk is a Python command-line tool and library for getting info about packages installed by setuptools, easy_install, distutils, and it can also query pypi packages. ================================================================ script to locate standard modules http://svn.python.org/projects/python/tags/r252/Doc/tools/listmodules.py ================================================================ nice forum thread: https://groups.google.com/forum/?fromgroups#!topic/comp.lang.python/TQfB9720m5Q ================================================================ Warm Regards Lev