RSS

Category Archives: How-To

Reading Compound File Binary Format Files generated by PHPExcel with Python/Django xlrd

This was a monster for me to track down, so I’m hoping it helps many others.

The issue I ran into was that when trying to read an Excel file using Python’s xlrd package that was generated by PHPExcel, the following error was generated:

File read error [row 1]: Workbook corruption: seen[2] == 4

In search of a fix, preferably one that didn’t assume PHPExcel was just buggy (which it may be, but that’s another topic), I perused dozens of online articles with the actual developer of xlrd (John Machin) commenting on threads from people with nearly identical issues, but all the solutions were just specific workarounds that didn’t seem to apply to me.

Somewhere, I caught the tiniest glimpse of the phrase “Compound File Binary.” At first it didn’t lead to much, but it ultimately led me here:

OleFileIO_PL – a Python module to read MS OLE2 files

A quick “pip install OleFileIO-PL” and the deployment of this code:


data = file.read()
try:
f = StringIO.StringIO(data)
ole = OleFileIO_PL.OleFileIO(f)
    if ole.exists('Workbook'):
         d = ole.openstream('Workbook')
         data = d.read()

except Exception as e:
     # try to read it the normal way
     pass

And I was on my way.

Note that “Workbook” stream might be specific to PHPExcel, but there is a list_dir() method on the ole object that will show you the streams available.  There’s also decent documentation in the zip download from the above link.

 

Posted by on September 19, 2013 in Computers, How-To, Programming

Leave a comment

XSL Template for converting XSD to complete XML prototype with inline specifications

For a project relating to Amazon, I found a need for an XSL template that could convert their XSD files–which are both numerous and meaty–into complete XML prototypes.  This would be useful for two reasons.

First, it would generate a complete sample call for basically any API request for which the provide an XSD.  Second, it would give me XML that I could programmatically convert to a self-generating form if necessary.

After digging around the web, to my surprise I was unable to find an XSL already made that accomplishes this, so I set about making my own.

The resulting XSL is included below.

Please note that while this appears (as of this writing) to be 100% functional for what I need on Amazon, it does not cover all possible XSD configurations, so may need to be modified for more exotic XSDs.  Also, it’s possible (and quite likely) that the XML generated will not validate to the given XSD. This is because I had to find ways to pass value restriction meta data in a way that made sense in XML.

The two obvious points are nodes with enumerations in which the valid values are passed as <Value> subnodes and nodes with extendable attributes that had to be constructed in the way of other nodes.  In both cases, a flag is set as an attribute in the parent node.

If you make enhancements or fixes or if you find problems, feel free to let me know, and I will revise this.

The XSL is free for anyone to use, modify, etc. without restraint, but I’d certainly be happy to hear about it if it helps you out.

Here is the code:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
version="1.0"
>

<xsl:output method=”xml”/>

<!– Ignore nodes that may have text values as they will output directly otherwise –>
<xsl:template match=”*[not(xs:element) and not(xs:include)]” />

<!– Uncomment to process any included files inline–>
<!–
<xsl:template match=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)/xs:schema/xs:element”>
<xsl:call-template name=”element” />
</xsl:for-each>
</xsl:template>
–>

<!– ######################
## XML ELEMENTS TEMPLATE ##
####################### –>

<xsl:template match=”/xs:schema/xs:element” name=”element”>

<xsl:choose>

<!– Node is referencing a base element – use that for the prototype either from this file or from any included files–>
<xsl:when test=”@ref”>
<xsl:variable name=”elementRef” select=”current()/@ref” />
<xsl:for-each select=”//xs:element[@name=$elementRef]”><xsl:call-template name=”element”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:element[@name=$elementRef]”><xsl:call-template name=”element”/></xsl:for-each>
</xsl:for-each>
</xsl:when>

<!– The node is a standard element – output its XML Prototype –>
<xsl:otherwise>

<xsl:element name=”{@name}”>

<!– Carry all attributes through other than name (which is used for the element nodeName) and type (which we have to handle cases for) –>
<xsl:copy-of select=”@*[local-name() != ‘name’ and local-name() != ‘type’]” />

<!– Bring in any annotations for the element as a “note” attribute –>
<xsl:if test=”./xs:annotation/xs:documentation”>
<xsl:attribute name=”note”><xsl:value-of select=”./xs:annotation/xs:documentation” /></xsl:attribute>
</xsl:if>

<xsl:choose>

<!– If there is a type attribute, it implies that it’s a complexType node and so we’ll be handling its subNodes independently –>
<xsl:when test=”contains(@type,’:’)”>
<xsl:variable name=”elementType” select=”substring-after(@type,’:’)” />
<xsl:for-each select=”//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:when>
<xsl:when test=”@type”>
<xsl:variable name=”elementType” select=”@type” />
<xsl:for-each select=”//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:when>

<!– This is a direct element definition but it still may have inline definitions –>
<xsl:otherwise>
<xsl:for-each select=”./xs:complexType”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”./xs:simpleType”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”./xs:element”><xsl:call-template name=”element”/></xsl:for-each>
</xsl:otherwise>

</xsl:choose>

</xsl:element>

</xsl:otherwise>

</xsl:choose>

</xsl:template>

<!– ######################
## COMPLEX TYPE TEMPLATE ##
####################### –>

<xsl:template name=”complexType”>

<!– Pass along all attributes –>
<xsl:copy-of select=”@*[local-name() != ‘name’]” />

<!– If this complexType needs additional attributes, pass a way to use those –>
<xsl:if test=”./xs:simpleContent/xs:extension/xs:attribute”>

<!– Custom node to indicate that this is node has extended attributes –>
<xsl:attribute name=’isExtended’>1</xsl:attribute>

<!– Element is complexType because it’s an extends the attribute set of a simpleType –>
<xsl:if test=”./xs:simpleContent/xs:extension/@base”>
<xsl:choose>
<xsl:when test=”contains(./xs:simpleContent/xs:extension/@base,’:’)”>
<xsl:variable name=”baseSimpleType” select=”substring-after(./xs:simpleContent/xs:extension/@base,’:’)” />
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:when>
<xsl:otherwise>
<xsl:variable name=”baseSimpleType” select=”./xs:simpleContent/xs:extension/@base” />
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
</xsl:for-each>
</xsl:otherwise>
</xsl:choose>
</xsl:if>

<!– Pass along all other attributes (other than @base) –>
<xsl:copy-of select=”./xs:simpleContent/xs:extension/@*[local-name() != ‘base’]” />

<!– Give a container for the actual value of the node –>
<xsl:element name=”NodeValue” />

<!– Show all attributes as sub nodes –>
<xsl:for-each select=”./xs:simpleContent/xs:extension/xs:attribute”>
<xsl:element name=”{@name}”>
<xsl:variable name=”elementType” select=”current()/@type” />
<xsl:for-each select=”//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType”/></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:complexType[@name=$elementType]”><xsl:call-template name=”complexType”/></xsl:for-each>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$elementType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
</xsl:for-each>
</xsl:element>
</xsl:for-each>

</xsl:if>

<!– Handle any sub nodes –>
<xsl:for-each select=”./xs:sequence/xs:element | ./xs:choice/xs:element”><xsl:call-template name=”element” /></xsl:for-each>

</xsl:template>

<!– #####################
## SIMPLE TYPE TEMPLATE ##
###################### –>

<xsl:template name=”simpleType”>

<!– If there is a base type, analyze and output its attributes first –>
<xsl:choose>
<xsl:when test=”contains(./xs:restriction/@base,’:’)”>
<xsl:variable name=”baseSimpleType” select=”substring-after(./xs:restriction/@base,’:’)” />
<xsl:attribute name='{$baseSimpleType}’>{type}</xsl:attribute>
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
</xsl:for-each>
</xsl:when>
<xsl:when test=”./xs:restriction/@base”>
<xsl:variable name=”baseSimpleType” select=”./xs:restriction/@base” />
<xsl:attribute name='{$baseSimpleType}’>{type}</xsl:attribute>
<xsl:for-each select=”//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
<xsl:for-each select=”//xs:include”>
<xsl:for-each select=”document(@schemaLocation)//xs:simpleType[@name=$baseSimpleType]”><xsl:call-template name=”simpleType” /></xsl:for-each>
</xsl:for-each>
</xsl:when>
</xsl:choose>

<!– Pass all type attributes back –>
<xsl:copy-of select=”./@*[local-name() != ‘name’ and local-name() != ‘id’]” />

<!– Pass all non-enumeration nodes as attributes (minLength, etc.) –>
<xsl:for-each select=”./xs:restriction/*[local-name() != ‘enumeration’]”>
<xsl:attribute name=”{local-name()}”><xsl:value-of select=”@value” /></xsl:attribute>
</xsl:for-each>

<!– If enumerations exist, pass an attribute indicating so and pass the values as sub nodes (must come last because it outputs XML directly as text) –>
<xsl:for-each select=”./xs:restriction/xs:enumeration”>
<xsl:if test=”position() = 1″><xsl:attribute name=”isList”>1</xsl:attribute></xsl:if>
<xsl:element name=”Value”><xsl:value-of select=”current()/@value” /></xsl:element>
</xsl:for-each>

</xsl:template>

</xsl:stylesheet>

 

Posted by on April 13, 2012 in Computers, How-To, Programming

4 Comments

Tags: , , , , , , , ,

PHP Random String and POST Form Generator

Just a little snippet of sample code.  Sometimes you just need a random string generator, and on top of that, a random form to test a page.

Maybe I’m just keeping this for my own future reference, but maybe someone else out there could use it too.  🙂

The function generates a string containing numbers and letters only (it’s easily customizable to contain other chars).  The form just creates inputs with random names from the string generator and random values from the string generator.


function randomString() {
        $chars = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
        for ($i=0; $i<rand(10,20); $i++) { $string .= $chars[rand(0,strlen($chars))]; }
        return $string;
}

?><form action="" method="POST"><?php
        for ($i=1; $i<=rand(5,10); $i++) { ?><p /><input name="<?php echo randomString(); ?>" value="<?php echo randomString(); ?>" /><?php }
        ?><p /><input type="submit" /><?php
?></form><?php

 

Posted by on July 8, 2011 in Computers, How-To, Programming

Leave a comment

Tags: , , ,

How to process PayPal Express Checkout for third party merchants

This is a very simple one, but one that can take a lot of google-search-query-nuance tweaking to find.

It’s easy to find the documentation to do PayPal Express Checkout, but to find that one little field where you send an alternate user as the recipient of the payments, well that’s downright impossible.  It is not in the docs (at least not as of this writing 5-18-2011).

How simple is it? Very — the variable is “SUBJECT”

Yes, you specify an alternate “SUBJECT” of the transaction.

Normally your SetExpressCheckout request looks something like:

METHOD=<method_name>&VERSION=<version>&PWD=<API_Password>&USER=<API_UserName>&SIGNATURE=<API_Signature>&...

Now it will look like this:

METHOD=<method_name>&VERSION=<version>&PWD=<API_Password>&USER=<API_UserName>&SIGNATURE=<API_Signature>&SUBJECT=<Payee_PayPal_Account>...

Payee_PayPal_Account is the email address/username the user uses to log in.

Hope this helps!  Took us valuable time to find …

 

Posted by on May 18, 2011 in Computers, How-To, Programming

7 Comments

Tags: , , , ,

Linux BASH Script: Search current directory for files containing text

I have written a simple BASH script to scan the current directory for all files containing certain text.

I use this all the time as I don’t know of a better Linux alternative, but it’s simple enough.  Just put save it as “/sbin/find-file-with” with executable permissions, and use it like so:

find-file-with sometext

If a matching file is found, it will print the full path of the file as well as the lines matching using grep (so the “some text” is passed directly into grep if you need help on how to vary your search)

Obviously some advancements can be performed, but without this script, I always just type (each time):

for I in `find ./*` ; do echo $I; cat $I | grep sometext; done

This script just keeps me from filling the screen with directory errors and unmatching files.

As a side note, if you want to include quotes (as in grep “some text”), you need to escape them:

find-file-with \"some text\"

Hope others find it useful … but the main reason it is here is so I have an online backup.  😀

#!/bin/sh
echo "";
for I in `find ./*` ; do
        if [ ! -d "$I" ] ; then
                if [ -e "$I" ] ; then
                        CMD="cat $I | grep $* | wc -c";
                        C=`eval $CMD`
                        if [ $C -gt 0 ] ; then
                                echo "\033[1m[ "$I" ]\033[0m"
                                cat $I | grep $* -n --color=auto;
                                echo ""
                        fi;
                fi;
        fi;
done;
 

Posted by on December 29, 2010 in Computers, How-To

Leave a comment

Tags: , ,

HOWTO: Linux SSH to Remote Server, execute a command and stay logged in

One of the shortcuts I always keep in my gnome taskbar is a link to a gnome-terminal with 3 tabs, each in the websites base directory of my web development server.

It took my quite a long time to figure out how to get it to change to that directory then stay logged in, even though it is rather simple.  It also took me a long time again recently because I reinstalled without backing that up.  So, this post is mainly so I don’t have to do that again, but maybe it will help others too.

The trick is to make SSH behave like a terminal (with the oft-overlooked -t flag), then to execute a login bash shell using normal SSH command execution.

So:

ssh -t ‘cd /path/to/go/to; bash -l’

So for me, my shortcut in the gnome taskbar is

gnome-terminal –tab -e “ssh -t ‘cd /path; bash -l'” –tab -e “ssh -t ‘cd /path; bash -l'” –tab -e “ssh -t ‘cd /path; bash -l'”

There — saved myself (and you) some time!

 

Posted by on December 14, 2010 in Computers, How-To, Programming

1 Comment

Tags: , , ,

WordPress 3 Auto-Update & Unable To Create Directory wp-content/upgrade/wordpress-3.tmp

Many, many, many users have apparently been experiencing an issue with this in WordPress — for some reason their auto update doesn’t work anymore.  I use the word “anymore,” because, for some users, it never worked.  For you, the solution is this: change permissions on the <install-dir>/wp-content/upgrade/ directory to 777 (only that directory for very obvious security reasons).

For users like me, however, there is an equally stupid result, and that is that VSFTP apparently disabled all FTP write commands in new versions.  Are you running VSFTP and using FTP to do your WordPress upgrades?

Well just edit your /etc/vsftpd.conf (exact location may vary by distro) and uncomment the line:

#write_enable=YES

(Just remove the #, save, and restart VSFTP.)

This worked for me, and I just figured it out on my own.  Stupid, yes, but for those of use who have become search-engine-instant-success-addicts, impossible to find.  Hopefully not so for you.

 

Posted by on December 6, 2010 in Computers, How-To

4 Comments

Tags: ,

Howto: Enable PCNTL in Ubuntu PHP installations

PCNTL in PHP allows for some handy advanced “trickery” using the OS process functions inherent in Linux (*nix?).  I believe some features are available in Windows, but I know for certain that pcntl_fork() is not.

Anyway, it is not enabled by default, so if you want to take advantage of the functions on your Ubuntu LAMP server, you might spend hours searching the web for that magic aptitude command.  But, as far as I can tell, it doesn’t exist.

Luckily, I stumbled across this article on the Ubuntu forums, so I’m dedicating a post here with the hopes that other will find it more easily.


Please note that you’ll probably need build-essentials and a few other source compilation basics, but as long as you have that, the following code will get you what you want.

First, in your home directory:

mkdir php
cd php
apt-get source php5
cd php5-(WHATEVER_RELEASE)/ext/pcntl
phpize
./configure
make

Then:

cp modules/pcntl.so /usr/lib/php5/WHEVER_YOUR_SO_FILES_ARE/
echo "extension=pcntl.so" > /etc/php5/conf.d/pcntl.ini

FYI: “make install” does not appear to put the files in the correct place.

Btw, please direct any thanks/praise to skout23 on the Ubuntu forums.

 

Posted by on July 30, 2010 in Computers, How-To

24 Comments

Tags: , , , , , , ,

FedEx Shipping XML API Idiosyncrasies — the saga continues (PHP & Soap envelopes)

I am doing a fairly large implementation using the exposed APIs of all 4 major shippers.  I’ve posted a few notes on USPS, and I’ve been using UPS for so long it almost seems to make perfect sense.

So, while I have to applaud FedEx for the granularity and versatility of the API they have exposed, it has caused me severe headaches due to the PHP SimpleXML issues it brings up with SOAP envelopes — particularly nested soap envelopes.

Long story short, the jist of this post is to help others deal with the responses they receive from the FedEx servers (if, like me, their pre-packaged methods do not implement tidily into your application structure).

Here is a sample response of a rate request (NOTE: this is a “Warning” response, only here for demo purposes):


   <?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
    <env:Header xmlns:env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" />
    <env:Body xmlns:env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <v8:RateReply xmlns:v8="http://fedex.com/ws/rate/v8">
            <v8:HighestSeverity>WARNING</v8:HighestSeverity>
            <v8:Notifications>
                <v8:Severity>WARNING</v8:Severity>
                <v8:Source>crs</v8:Source>
                <v8:Code>556</v8:Code>
                <v8:Message>There are no valid services available. </v8:Message>
                <v8:LocalizedMessage>There are no valid services available. </v8:LocalizedMessage>
            </v8:Notifications>
            <v8:TransactionDetail xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/">
                <v8:CustomerTransactionId>TC05_Rate_Package_Groups_POS</v8:CustomerTransactionId>
            </v8:TransactionDetail>
            <v8:Version xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/">
                <v8:ServiceId>crs</v8:ServiceId>
                <v8:Major>8</v8:Major>
                <v8:Intermediate>0</v8:Intermediate>
                <v8:Minor>0</v8:Minor>
            </v8:Version>
        </v8:RateReply>
    </env:Body>
</soapenv:Envelope>
    

There are lots of ways you might envision accessing that precious “RateReply” tag, but to keep it simple and avoid inferences, you get to it like this:

$xml = simplexml_load_string ($xml_from_above);
$xml->children(‘http://schemas.xmlsoap.org/soap/envelope/’)->Body->children(‘http://fedex.com/ws/rate/v8’)->RateReply;

I do suppose this is obvious enough, but believe me, the nuances that can throw it off can still cause you to inadvertently lose ridiculous amounts of time on it.

Furthermore, if you prefer to do direct PHP cURL XML calls (which you probably do given that you’re reading this), you’ll find that FedEx is not terribly interested in your type of programmer. To find a simple sample XML call, you’ll have a tough time (although I managed to find it in the WSDL downloadable ZIP). However, this is a very basic call. To add additional data (such as the exotic field that apparently no one would ever think to use — “InsuredValue”) — this field has the form (EXACTLY, stuff in brackets is what can be changed) “<InsuredValue><Currency>[currencytype]</Currency><Amount>[float]</Amount></InsuredValue>”

The “InsuredValue” node/field must go immediately before the “Weight” node in the “RequestedPackageLineItems” node. (This of course reference FedEx API v8 — things may be different in future releases.)

 

Posted by on May 24, 2010 in Computers, How-To, Personal

8 Comments

Tags: , , , , , , , , ,