RSS

Category Archives: Computers

Distinguishing between False and 0 (zero) in a Django template

You may occasionally find yourself in a Django template attempting to distinguish if False is not equal to 0 (bool vs. int).

There are multiple ways to accomplish this, but if you don’t want to create a new template tag, and you don’t want to add logic to the view (because you’re a rebel), you can accomplish it using the “add” filter.

Take the variable “my_var”

If my_var is a ‘bool’ False:

my_var|add:"f" == "f"

 

If my_var is an ‘int’ 0:

my_var|add:"f" == ""

 

And all together now:

{% if my_var|add:"f" == "f" %}
    I have a boolean False
{% else %}
    I have a 0
{% endif %}

 

Keep in mind that this logic may vary depending what you’re comparing.  It’s useful as is ONLY if you know you’re trying to distinguish between False and zero (0).  If you can have values for my_var like “” or [] or other data types that have certain scenarios for equating with False, you’ll have to play around, but I personally had difficulty finding answers that address this that didn’t involve building logic outside of the template file itself.

 

Posted by on June 27, 2014 in Computers, Programming, Uncategorized

Leave a comment

Tags: , , , , , ,

Reading Compound File Binary Format Files generated by PHPExcel with Python/Django xlrd

This was a monster for me to track down, so I’m hoping it helps many others.

The issue I ran into was that when trying to read an Excel file using Python’s xlrd package that was generated by PHPExcel, the following error was generated:

File read error [row 1]: Workbook corruption: seen[2] == 4

In search of a fix, preferably one that didn’t assume PHPExcel was just buggy (which it may be, but that’s another topic), I perused dozens of online articles with the actual developer of xlrd (John Machin) commenting on threads from people with nearly identical issues, but all the solutions were just specific workarounds that didn’t seem to apply to me.

Somewhere, I caught the tiniest glimpse of the phrase “Compound File Binary.” At first it didn’t lead to much, but it ultimately led me here:

OleFileIO_PL – a Python module to read MS OLE2 files

A quick “pip install OleFileIO-PL” and the deployment of this code:


data = file.read()
try:
f = StringIO.StringIO(data)
ole = OleFileIO_PL.OleFileIO(f)
    if ole.exists('Workbook'):
         d = ole.openstream('Workbook')
         data = d.read()

except Exception as e:
     # try to read it the normal way
     pass

And I was on my way.

Note that “Workbook” stream might be specific to PHPExcel, but there is a list_dir() method on the ole object that will show you the streams available.  There’s also decent documentation in the zip download from the above link.

 

Posted by on September 19, 2013 in Computers, How-To, Programming

1 Comment

XSL Template for converting XSD to complete XML prototype with inline specifications

For a project relating to Amazon, I found a need for an XSL template that could convert their XSD files–which are both numerous and meaty–into complete XML prototypes.  This would be useful for two reasons.

First, it would generate a complete sample call for basically any API request for which the provide an XSD.  Second, it would give me XML that I could programmatically convert to a self-generating form if necessary.

After digging around the web, to my surprise I was unable to find an XSL already made that accomplishes this, so I set about making my own.

The resulting XSL is included below.

Please note that while this appears (as of this writing) to be 100% functional for what I need on Amazon, it does not cover all possible XSD configurations, so may need to be modified for more exotic XSDs.  Also, it’s possible (and quite likely) that the XML generated will not validate to the given XSD. This is because I had to find ways to pass value restriction meta data in a way that made sense in XML.

The two obvious points are nodes with enumerations in which the valid values are passed as <Value> subnodes and nodes with extendable attributes that had to be constructed in the way of other nodes.  In both cases, a flag is set as an attribute in the parent node.

If you make enhancements or fixes or if you find problems, feel free to let me know, and I will revise this.

The XSL is free for anyone to use, modify, etc. without restraint, but I’d certainly be happy to hear about it if it helps you out.

Here is the code:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
version="1.0"
>

<xsl:output method=”xml”/>

<!– Ignore nodes that may have text values as they will output directly otherwise –>
<xsl:template match=”*[not(xs:element) and not(xs:include)]” />

<!– Uncomment to process any included files inline–>
<!–
<xsl:template match=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)/xs:schema/xs:element”>
<xsl:call-template name=”element” />
</xsl:for-each>
</xsl:template>
–>

<!– ######################
## XML ELEMENTS TEMPLATE ##
####################### –>

<xsl:template match=”/xs:schema/xs:element” name=”element”>

<xsl:choose>

<!– Node is referencing a base element – use that for the prototype either from this file or from any included files–>
<xsl:when test=”@ref”>
<xsl:variable name=”elementRef” select=”current()/@ref” />
<xsl:for-each select=”//xs:element[@name=$elementRef]”><xsl:call-template name=”element”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:element[@name=$elementRef]”><xsl:call-template name=”element”/></xsl:for-each>
</xsl:for-each>
</xsl:when>

<!– The node is a standard element – output its XML Prototype –>
<xsl:otherwise>

<xsl:element name=”{@name}”>

<!– Carry all attributes through other than name (which is used for the element nodeName) and type (which we have to handle cases for) –>
<xsl:copy-of select=”@*[local-name() != ‘name’ and local-name() != ‘type’]” />

<!– Bring in any annotations for the element as a “note” attribute –>
<xsl:if test=”./xs:annotation/xs:documentation”>
<xsl:attribute name=”note”><xsl:value-of select=”./xs:annotation/xs:documentation” /></xsl:attribute>
</xsl:if>

<xsl:choose>

<!– If there is a type attribute, it implies that it’s a complexType node and so we’ll be handling its subNodes independently –>
<xsl:when test=”contains(@type,’:’)”>
<xsl:variable name=”elementType” select=”substring-after(@type,’:’)” />
<xsl:for-each select=”//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:when>
<xsl:when test=”@type”>
<xsl:variable name=”elementType” select=”@type” />
<xsl:for-each select=”//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:when>

<!– This is a direct element definition but it still may have inline definitions –>
<xsl:otherwise>
<xsl:for-each select=”./xs:complexType”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”./xs:simpleType”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”./xs:element”><xsl:call-template name=”element”/></xsl:for-each>
</xsl:otherwise>

</xsl:choose>

</xsl:element>

</xsl:otherwise>

</xsl:choose>

</xsl:template>

<!– ######################
## COMPLEX TYPE TEMPLATE ##
####################### –>

<xsl:template name=”complexType”>

<!– Pass along all attributes –>
<xsl:copy-of select=”@*[local-name() != ‘name’]” />

<!– If this complexType needs additional attributes, pass a way to use those –>
<xsl:if test=”./xs:simpleContent/xs:extension/xs:attribute”>

<!– Custom node to indicate that this is node has extended attributes –>
<xsl:attribute name=’isExtended’>1</xsl:attribute>

<!– Element is complexType because it’s an extends the attribute set of a simpleType –>
<xsl:if test=”./xs:simpleContent/xs:extension/@base”>
<xsl:choose>
<xsl:when test=”contains(./xs:simpleContent/xs:extension/@base,’:’)”>
<xsl:variable name=”baseSimpleType” select=”substring-after(./xs:simpleContent/xs:extension/@base,’:’)” />
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:when>
<xsl:otherwise>
<xsl:variable name=”baseSimpleType” select=”./xs:simpleContent/xs:extension/@base” />
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:otherwise>
</xsl:choose>
</xsl:if>

<!– Pass along all other attributes (other than @base) –>
<xsl:copy-of select=”./xs:simpleContent/xs:extension/@*[local-name() != ‘base’]” />

<!– Give a container for the actual value of the node –>
<xsl:element name=”NodeValue” />

<!– Show all attributes as sub nodes –>
<xsl:for-each select=”./xs:simpleContent/xs:extension/xs:attribute”>
<xsl:element name=”{@name}”>
<xsl:variable name=”elementType” select=”current()/@type” />
<xsl:for-each select=”//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
</xsl:for-each>
</xsl:element>
</xsl:for-each>

</xsl:if>

<!– Handle any sub nodes –>
<xsl:for-each select=”./xs:sequence/xs:element | ./xs:choice/xs:element”><xsl:call-template name=”element” /></xsl:for-each>

</xsl:template>

<!– #####################
## SIMPLE TYPE TEMPLATE ##
###################### –>

<xsl:template name=”simpleType”>

<!– If there is a base type, analyze and output its attributes first –>
<xsl:choose>
<xsl:when test=”contains(./xs:restriction/@base,’:’)”>
<xsl:variable name=”baseSimpleType” select=”substring-after(./xs:restriction/@base,’:’)” />
<xsl:attribute name='{$baseSimpleType}’>{type}</xsl:attribute>
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
</xsl:for-each>
</xsl:when>
<xsl:when test=”./xs:restriction/@base”>
<xsl:variable name=”baseSimpleType” select=”./xs:restriction/@base” />
<xsl:attribute name='{$baseSimpleType}’>{type}</xsl:attribute>
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
</xsl:for-each>
</xsl:when>
</xsl:choose>

<!– Pass all type attributes back –>
<xsl:copy-of select=”./@*[local-name() != ‘name’ and local-name() != ‘id’]” />

<!– Pass all non-enumeration nodes as attributes (minLength, etc.) –>
<xsl:for-each select=”./xs:restriction/*[local-name() != ‘enumeration’]”>
<xsl:attribute name=”{local-name()}”><xsl:value-of select=”@value” /></xsl:attribute>
</xsl:for-each>

<!– If enumerations exist, pass an attribute indicating so and pass the values as sub nodes (must come last because it outputs XML directly as text) –>
<xsl:for-each select=”./xs:restriction/xs:enumeration”>
<xsl:if test=”position() = 1″><xsl:attribute name=”isList”>1</xsl:attribute></xsl:if>
<xsl:element name=”Value”><xsl:value-of select=”current()/@value” /></xsl:element>
</xsl:for-each>

</xsl:template>

</xsl:stylesheet>

 

Posted by on April 13, 2012 in Computers, How-To, Programming

4 Comments

Tags: , , , , , , , ,

The Solution: Use PHP’s pcntl_fork to limit execution time of MySQL queries

One of the worst parts of a web application can be the variability of mysql queries that get sent into your database.  You can add indices, tweak hardware configurations, etc., but wouldn’t it be nice to simply kill any database query that takes longer than whatever you deem is “too long”?

Well, no, not any query.  I would never want to kill a write — just a read-only query; in particular: search queries.

So, as it turns out, the hurdles for this are immense, and, because my solution uses PHP’s pcntl_fork() function, even my solution, while it works, it has to make assumptions and is not perfect.

That being said, it would seem easy enough for this to be built into PHP or for some similar mechanism to be built into MySQL: Execute this query but only if it takes less than n seconds.  If not, kill it.  This is not the case, however, so we’re left to our own cleverness.

There are hundreds of reasons why you would never want to do this, but I only need one reason to want to do it to try to implement it.

So here is my solution steps in techno-layman’s terms, followed by the necessary code:

  • Call a function to execute a MySQL query (again, preferably read-only)
  • Open a shared memory space so that we can pass the query results back to the parent from the child
  • Store process state information in a database
  • Fork, and execute the query in the child process
  • Keep time in the parent process
  • Kill the child process if it takes longer than n seconds
  • Return the results

 

Just to forwarn, I have tested this as proof of concept, but I am uncertain about the particulars of PHP’s shared memory and am not confident how reliable the shared memory implementation will in a production environment.  I plan to try it out, but right now I’m just getting the info out there.

So the state database will be defined as follows:

CREATE TABLE IF NOT EXISTS `pcntlFork` (
`idx` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`status` enum('0','1') NOT NULL DEFAULT '0',
`output` longblob NOT NULL,
PRIMARY KEY (`idx`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=58 ;

The PHP class is available to download here:  http://mysql-restrictor.googlecode.com/files/mysqlRestrictor.class.php (I had it inline, but WordPress did not want to format it properly)

And you would use it something like this

$m = new mysqlRestrictor();
$results = $m->dbQuery("SELECT * FROM `table`");
var_dump($results);

Since this took me an extremely long time to build and just verify that it even works, I welcome comments for improvement, and by all means use it for yourself. Let me know how it does in production!

 

Posted by on December 16, 2011 in Computers, Programming

Leave a comment

Tags: , , ,

PHP Random String and POST Form Generator

Just a little snippet of sample code.  Sometimes you just need a random string generator, and on top of that, a random form to test a page.

Maybe I’m just keeping this for my own future reference, but maybe someone else out there could use it too.  🙂

The function generates a string containing numbers and letters only (it’s easily customizable to contain other chars).  The form just creates inputs with random names from the string generator and random values from the string generator.


function randomString() {
        $chars = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
        for ($i=0; $i<rand(10,20); $i++) { $string .= $chars[rand(0,strlen($chars))]; }
        return $string;
}

?><form action="" method="POST"><?php
        for ($i=1; $i<=rand(5,10); $i++) { ?><p /><input name="<?php echo randomString(); ?>" value="<?php echo randomString(); ?>" /><?php }
        ?><p /><input type="submit" /><?php
?></form><?php

 

Posted by on July 8, 2011 in Computers, How-To, Programming

Leave a comment

Tags: , , ,

How to process PayPal Express Checkout for third party merchants

This is a very simple one, but one that can take a lot of google-search-query-nuance tweaking to find.

It’s easy to find the documentation to do PayPal Express Checkout, but to find that one little field where you send an alternate user as the recipient of the payments, well that’s downright impossible.  It is not in the docs (at least not as of this writing 5-18-2011).

How simple is it? Very — the variable is “SUBJECT”

Yes, you specify an alternate “SUBJECT” of the transaction.

Normally your SetExpressCheckout request looks something like:

METHOD=<method_name>&VERSION=<version>&PWD=<API_Password>&USER=<API_UserName>&SIGNATURE=<API_Signature>&...

Now it will look like this:

METHOD=<method_name>&VERSION=<version>&PWD=<API_Password>&USER=<API_UserName>&SIGNATURE=<API_Signature>&SUBJECT=<Payee_PayPal_Account>...

Payee_PayPal_Account is the email address/username the user uses to log in.

Hope this helps!  Took us valuable time to find …

 

Posted by on May 18, 2011 in Computers, How-To, Programming

7 Comments

Tags: , , , ,

Nano global search and replace tabs to spaces or spaces to tabs

I have been an avid user of nano/pico since about 1999, and yes, many naysayers think it is crap for programming, but it works for me, and I like it.

That being said, one of the major issues I’ve had is that the Xorg select/paste always copies tab characters as the corresponding number of spaces.  So, when I select text in one file, paste into another, I have to replace all the spaces with tabs.


Typically I paste into gedit first, do the replace there, then c/p into the file.  This preserves the tabs.

(If you’re still wondering why I use nano, I just like having my editor accessible as long as I have server access.  I never got used to vi, and nano is more than effective for me.)

I have always thought nano should be able to search and replace tabs and spaces, but I could never get it to work.  Even without the gedit technique, I would typically just replace all double spaces to nothing, then manually insert the tabs.  My workarounds, again, are generally sufficient.  I have not NEEDED search and replace of tabs within nano, but today I decided I wanted to really find out if I could.

And it required some digging …

But, I finally found it: verbatim input!

Nano has a feature to disable character interpretation, and for one character, accept input literally.

To turn it on (again it’s for just the first character typed), hit alt-SHIFT-V (alt-V without shift may trigger x-windows menus), then just hit the tab key (it may or may not show a note that you’re in verbatim input mode).

You only need to do this in the search / replace prompts.  Obviously, you can type tabs directly into the file.

So an example — let’s say you want to convert any instance of 8 spaces to a tab character.

Here is the command stack:

control-W (search)
control-R (replace)
hit space 8 times, then hit enter
alt-shift-V (verbatim input)
hit the tab key, hit enter

Proceed as normal.

Hopefully this post is easier to find than the seemingly impossible digging I just undertook …

 

Posted by on May 17, 2011 in Computers, Programming

12 Comments

Tags: , , ,

Linux BASH Script: Search current directory for files containing text

I have written a simple BASH script to scan the current directory for all files containing certain text.

I use this all the time as I don’t know of a better Linux alternative, but it’s simple enough.  Just put save it as “/sbin/find-file-with” with executable permissions, and use it like so:

find-file-with sometext

If a matching file is found, it will print the full path of the file as well as the lines matching using grep (so the “some text” is passed directly into grep if you need help on how to vary your search)

Obviously some advancements can be performed, but without this script, I always just type (each time):

for I in `find ./*` ; do echo $I; cat $I | grep sometext; done

This script just keeps me from filling the screen with directory errors and unmatching files.

As a side note, if you want to include quotes (as in grep “some text”), you need to escape them:

find-file-with \"some text\"

Hope others find it useful … but the main reason it is here is so I have an online backup.  😀

#!/bin/sh
echo "";
for I in `find ./*` ; do
        if [ ! -d "$I" ] ; then
                if [ -e "$I" ] ; then
                        CMD="cat $I | grep $* | wc -c";
                        C=`eval $CMD`
                        if [ $C -gt 0 ] ; then
                                echo "\033[1m[ "$I" ]\033[0m"
                                cat $I | grep $* -n --color=auto;
                                echo ""
                        fi;
                fi;
        fi;
done;
 

Posted by on December 29, 2010 in Computers, How-To

Leave a comment

Tags: , ,

HOWTO: Linux SSH to Remote Server, execute a command and stay logged in

One of the shortcuts I always keep in my gnome taskbar is a link to a gnome-terminal with 3 tabs, each in the websites base directory of my web development server.

It took my quite a long time to figure out how to get it to change to that directory then stay logged in, even though it is rather simple.  It also took me a long time again recently because I reinstalled without backing that up.  So, this post is mainly so I don’t have to do that again, but maybe it will help others too.

The trick is to make SSH behave like a terminal (with the oft-overlooked -t flag), then to execute a login bash shell using normal SSH command execution.

So:

ssh -t ‘cd /path/to/go/to; bash -l’

So for me, my shortcut in the gnome taskbar is

gnome-terminal –tab -e “ssh -t ‘cd /path; bash -l'” –tab -e “ssh -t ‘cd /path; bash -l'” –tab -e “ssh -t ‘cd /path; bash -l'”

There — saved myself (and you) some time!

 

Posted by on December 14, 2010 in Computers, How-To, Programming

1 Comment

Tags: , , ,