User Tools

Site Tools

Trace:
python:python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
python:python [2013/03/13 17:16] – dblumepython:python [2024/02/27 15:43] (current) – [Linux script that takes either stdin or files] dblume
Line 1: Line 1:
-====== Python ======+====== General Python Notes ======
  
 Man, there's too much to do and note.  Logging some stuff to investigate later... Man, there's too much to do and note.  Logging some stuff to investigate later...
Line 7: Line 7:
   * [[http://pytools.codeplex.com/releases/view/76089|Python Tools for Visual Studio 2010]].   * [[http://pytools.codeplex.com/releases/view/76089|Python Tools for Visual Studio 2010]].
   * [[http://wiki.python.org/moin/PythonSpeed/PerformanceTips|Python Performance Tips at the Python wiki]].   * [[http://wiki.python.org/moin/PythonSpeed/PerformanceTips|Python Performance Tips at the Python wiki]].
-  * Of course, there's getting Python running under Xampp. [[xampp]]+  * Of course, there's [[:xampp|getting Python running under Xampp]].
   * There's [[http://www.infoworld.com/d/application-development/pillars-python-six-python-web-frameworks-compared-169442|A comparison of CubicWeb, Django, Pyramid, Web.py, Web2py, and Zope 2]]   * There's [[http://www.infoworld.com/d/application-development/pillars-python-six-python-web-frameworks-compared-169442|A comparison of CubicWeb, Django, Pyramid, Web.py, Web2py, and Zope 2]]
-  * Look into the microframework [[http://flask.pocoo.org/|Flask]]. Here's a [[http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world|tutorial on making a blog with it]].+  * Look into the microframework [[http://flask.pocoo.org/|Flask]]. Here's a [[http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world|tutorial on making a blog with it]]. Consider flask with gunicorn and nginx. 
 +  * [[https://dataset.readthedocs.org/en/latest/|dataset]], an abstraction layer above relational databases. 
 +  * [[http://docopt.org/|docopt]] for a stupidly easy and correct command-line interface description language. 
 +  * [[http://stevenloria.com/python-best-practice-patterns-by-vladimir-keleshev-notes/|Best practices]] by Vladimir Keleshev. 
 +  * [[https://twitter.com/brandon_rhodes/status/449938490516860928|You can ask a new-style class for its .__subclasses__()]] It uses weak references. Look at [[https://gist.github.com/etrepum/9862280|this interesting example with a child class inside a method]]. 
 +  * A Power Point deck by Alex Martelli describing [[http://www.aleax.it/yt_pydi.pdf|Dependency Injection vs. Template Method (override base method) vs. Monkey Patching]]. 
 +  * [[http://stackoverflow.com/questions/101268/hidden-features-of-python|Hidden Features of Python]] at StackOverflow 
 +  * [[http://chriskiehl.com/article/parallelism-in-one-line|Parallelism in one line]]
  
 Would be nice to create a binary search in text files in Python.  Maybe based on an answer deep in [[http://stackoverflow.com/questions/744256/reading-huge-file-in-python|Reading Huge File in Python]]. Would be nice to create a binary search in text files in Python.  Maybe based on an answer deep in [[http://stackoverflow.com/questions/744256/reading-huge-file-in-python|Reading Huge File in Python]].
Line 16: Line 23:
  
 Would be good to experiment with [[http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/|least-squares polynomial fitting in Python]]. Would be good to experiment with [[http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/|least-squares polynomial fitting in Python]].
 +
 ===== Data Analysis ===== ===== Data Analysis =====
  
Line 23: Line 31:
   * [[http://pandas.pydata.org/|Pandas]]   * [[http://pandas.pydata.org/|Pandas]]
   * [[http://ipython.org/|IPython]].  (As seen above.)   * [[http://ipython.org/|IPython]].  (As seen above.)
 +
 ===== Template Files to Start With ===== ===== Template Files to Start With =====
  
Line 35: Line 44:
 </code> </code>
  
-===== Linux or Bash Tips ===== 
- 
-Useful bash command for finding strings within python files... 
- 
-<code bash> 
-find . -name \*.py -type f -print0 | xargs -0 grep -nI "timeit" 
-</code> 
- 
-Interesting way to use ''grep -v'' to remove paths from a list generated by ''find''.  Not sure about the escaped ''|'' character, though... 
- 
-<code bash> 
-#!/bin/bash 
-find $PWD -regex ".*\.[hcHC]\(pp\|xx\)?" | \ 
-    grep -v " \|unwantedpath/unwantedpath2\|unwantedpath3" > cscope.files 
-cscope -q -b 
-</code> 
- 
-And these two have nothing to do with Python.  Here's how to find if a symbol is in a library, and how to search lots of object files and print the filename above the search... 
- 
-<code bash> 
-nm obj-directory/libmyobject.a | c++filt | grep Initialize_my_obj 
-find bindirectory/ -name \*.a -exec nm /dev/null {} \; 2>/dev/null | \ 
-    c++filt | grep -P "(^bindirectory.*\.a|T Initialize_my_obj)" 
-</code> 
- 
-Also handy to merge two streams together... 
- 
-<code bash> 
-( cat file1 && cat file2 ) | sort 
-</code> 
- 
-When a little quick math is needed, use ''bc'' 
-<code bash> 
-$ bc <<< "obase=16;ibase=10;15" 
-F 
-$ bc -l <<< 1/3 
-.33333333333333333333 
-$ bc <<< "scale=2; 1/3" 
-.33 
-$ bc <<< "obase=10;ibase=16;B" 
-11 
-</code> 
- 
-and, when coverting from hex to dec... 
- 
-<code bash> 
-echo $((0x2dec)) 
-</code> 
- 
-But, then again, does that really seem easier than, 
- 
-<code> 
-python -c "print int('B',16)" 
-</code> 
- 
-There's a bash way to calculate how many days ago a date was: 
- 
-<code bash> 
-$ echo $(( ($(date +%s) - $(date -d "2012-4-16" +%s)) / 86400 )) 
-</code> 
- 
-And a Python way... 
- 
-<code python> 
-python -c "import datetime; print (datetime.date.today() - datetime.date( 2012, 4, 16 )).days" 
-</code> 
- 
-And for displaying lines to get cut instead of wrapped: 
- 
-<code bash> 
-cat_one_line_per_row() { 
-  cat "$@" | expand | cut -b1-$COLUMNS 
-} 
-</code> 
- 
-ctags's man page says that one of its bugs is that it has too many options.  Ain't that the truth.  Make note of the obscure flag here, ''--c++-kinds=+p'', that tells ctags to process prototypes and method declarations. 
- 
-<code bash> 
-ctags -n --if0=yes --c++-kinds=+p --langmap=c++:+.inl.lst \ 
-    --langmap=asm:+.inc --file-tags=yes -R --extra=fq \ 
-    --exclude=unwanted_file.lst \ 
-    --exclude='*unwanted-directory*/*' \ 
-    --regex-C++='/^.*CINIT.(.+),.*,.*,.*/CURLOPT_\1/' 
-</code> 
  
 ===== For / Else (Nobreak) ===== ===== For / Else (Nobreak) =====
Line 155: Line 80:
 ===== Linux script that takes either stdin or files ===== ===== Linux script that takes either stdin or files =====
  
 +Newer way, **use [[https://docs.python.org/3/library/fileinput.html|module fileinput]]**. Older way:
 <code python> <code python>
 if __name__=='__main__': if __name__=='__main__':
Line 163: Line 89:
             if not line:             if not line:
                 break                 break
-            my_process_line( line.rstrip() )+            my_process_line(line.rstrip())
     else:     else:
         # Process lines of the files specified.         # Process lines of the files specified.
         for fname in sys.argv[1:]:         for fname in sys.argv[1:]:
-            if not os.path.exists( fname ): +            if not os.path.exists(fname): 
-                treat_argument_as_literal( fname )+                treat_argument_as_literal(fname)
                 continue                 continue
-            with open( fname, 'r' ) as f:+            with open(fname, 'r') as f:
                 while 1:                 while 1:
                     line = f.readline()                     line = f.readline()
                     if not line:                     if not line:
                         break                         break
-                    my_process_line( line.rstrip() )+                    my_process_line(line.rstrip())
 </code> </code>
  
Line 181: Line 107:
  
 [[http://amix.dk/blog/post/19663#Making-ugly-code-more-beautiful-using-Pythons-with-statement|Making code more beautiful with "with"]].  (Also mentions yield.) [[http://amix.dk/blog/post/19663#Making-ugly-code-more-beautiful-using-Pythons-with-statement|Making code more beautiful with "with"]].  (Also mentions yield.)
 +
 +===== cProfile vs. line_profiler and kernprof =====
 +
 +Raymond Hettinger [[https://twitter.com/raymondh/status/341010756101341187|likes Robert Kern's line_profiler much better than profile and cProfile]].
 +
 +Here it is: [[http://pythonhosted.org/line_profiler/|line_profiler and kernprof]]
  
 ===== timeit ===== ===== timeit =====
Line 218: Line 150:
          
 if __name__ == '__main__': if __name__ == '__main__':
-    cProfile.run( "my_function()" )+    cProfile.run("my_function()")
 </code> </code>
  
Line 232: Line 164:
  
 # Split input data by row and then on spaces # Split input data by row and then on spaces
-rows = [ line.strip().split(' ') for line in data.split('\n') ]+rows = [line.strip().split(' ') for line in data.split('\n')]
  
 # Reorganize data by columns # Reorganize data by columns
Line 238: Line 170:
  
 # Compute column widths by taking maximum length of values per column # Compute column widths by taking maximum length of values per column
-col_widths = [ max(len(value) for value in col) for col in cols ]+col_widths = [max(len(value) for value in col) for col in cols]
  
 # Create a suitable format string # Create a suitable format string
-format = ' '.join(['%%%ds' % width for width in col_widths ])+format = ' '.join(['%%%ds' % width for width in col_widths])
  
 # Print each row using the computed format # Print each row using the computed format
Line 272: Line 204:
         self.__m_x = x         self.__m_x = x
  
-    x = property( getx, setx )+    x = property(getx, setx)
  
 class B: class B:
     """ Old; very small, was for for multitudes of objects. """     """ Old; very small, was for for multitudes of objects. """
-    __slots__ = [ "__m_x" ]+    __slots__ = ["__m_x"]
     def __init__(self):     def __init__(self):
         self.__m_x = 0         self.__m_x = 0
Line 287: Line 219:
         self.__m_x = x         self.__m_x = x
  
-    x = property( getx, setx )        +    x = property(getx, setx)        
  
 class C(object): class C(object):
-    """ New, reccommended. """+    """ New, recommended. """
     def __init__(self):     def __init__(self):
         self.__x = 0         self.__x = 0
Line 301: Line 233:
         self.__x = x         self.__x = x
  
-    x = property( getx, setx )+    x = property(getx, setx) 
 +</code> 
 + 
 +====== filelock ====== 
 + 
 +Evan Fosmark has a filelock module.  But here's a quick and dirty implementation of a lock that uses the current file: 
 + 
 +<code python> 
 +import os 
 +import fcntl 
 +import inspect 
 +# Maybe use os.path.abspath(__file__) ? 
 +with open(os.path.abspath inspect.getfile(inspect.currentframe())), 'r') as f: 
 +    try: 
 +        fcntl.flock(f, fcntl.LOCK_EX) 
 +        call_that_cannot_be_concurrent() 
 +    finally: 
 +        fcntl.flock(f, fcntl.LOCK_UN) 
 +</code> 
 + 
 +====== Various Approaches to threaded URL Requests ====== 
 + 
 +  * [[https://stackoverflow.com/questions/2632520/what-is-the-fastest-way-to-send-100-000-http-requests-in-python|Use Queue and threading's Thread]] 
 +  * [[https://www.shanelynn.ie/using-python-threading-for-multiple-results-queue/|Use threading and store results in a pre-allocated list]]. Then use Queue for lots of URLs. 
 +  * [[https://dev.to/rhymes/how-to-make-python-code-concurrent-with-3-lines-of-code-2fpe|Use concurrent.futures' ThreadPoolExecutor and map()]]. 
 +  * Or, use the doc's [[https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example|ThreadPoolExecutor Example]]. 
 +  * And, as mentioned in Parallelism in One Line, [[https://chriskiehl.com/article/parallelism-in-one-line|multiprocessing.dummy's Pool and its own map()]]. 
 + 
 +====== Fibonacci Generator with Itertools ====== 
 + 
 +<code python> 
 +import itertools 
 + 
 +def fib(n): 
 +    """Print a Fibonacci series up to n.""" 
 +    a, b = 0, 1 
 +    while True: 
 +        yield a 
 +        b = a + b 
 +        yield b 
 +        a = a + b 
 + 
 +if __name__ == '__main__':  
 +    for x in itertools.islice(fib(), 5): 
 +        print x 
 + 
 +#    for i in range( 5 ): 
 +#        print i, fib( i ) 
 +</code> 
 + 
 +====== Parse C++ Code with nm and graphviz's dot ====== 
 + 
 +<code bash> 
 +sudo apt-get install graphviz 
 +</code> 
 + 
 +<code python> 
 +    # Read `nm` output 
 +    output = subprocess.check_output(['nm', '--demangle', entry.path]).decode('utf-8') 
 +    for line in output.split('\n'): 
 +        if 'Singleton' in line: 
 + 
 +    # Make a .dot file 
 +    with open(filename + '.dot', 'w') as file: 
 +        file.write('digraph {\n') 
 +        for f, destinations in self.graph.items(): 
 +            for d in destinations: 
 +                file.write(f'  "{f}" -> "{d}"\n') 
 + 
 + 
 +def create_png_image(filename): 
 +    with open(filename + '.png', 'wb') as fout: 
 +        p1 = subprocess.run(['dot', filename + '.dot', '-Tpng'], stdout=fout) 
 </code> </code>
  
Line 307: Line 312:
  
 TODO Link to my tips from LiveJournal and GMail, and why I chose which timing modules. TODO Link to my tips from LiveJournal and GMail, and why I chose which timing modules.
-Touching this page to see if the indexmenu will update its data. 
python/python.1363220210.txt.gz · Last modified: 2023/04/12 20:44 (external edit)