Help with Python Script please

[Tico]

First some background to the problem -

I had two php scripts on the Loxberry that were activated from the Miniserver to control a feature of an inverter/battery combination. The first script was a POST command (with Digest Authentication) that suspended a battery calibration feature. The second script was a similar POST command but re-enabled the calibration.

The security of the inverter was upgraded with a recent firmware update. The firmware update broke the existing facility to send POST commands and control the calibration feature. This was acknowledged by the company.

I've been communicating with the local engineering department of the company. They forwarded a Python script that 'should' work with the new firmware to suspend the calibration feature.

I hope someone can provide some guidance on how this script should be run on the Loxberry (or if it's even viable to use on the Loxberry?). I've tried the command -

python post.py

and get the following error message -

Traceback (most recent call last):
File "./post.py", line 12, in <module>
from importlib.util import spec_from_file_location, module_from_spec
ImportError: No module named util

The engineering department advised that the error message means that the python environment doesn't have the module called util. It will need to be installed.

I've googled the error 'ImportError: No module named util' and there's many confusing options. Some suggest I need Python 3.4, (Python version 2.7.13 is on the Loxberry).

Any ideas on how to proceed from here? I can provide the full Python script if it helps, but I think the problem is possibly just the Python environment?

[Christian Fenzl]

LoxBerry ships with both versions of Python (2 and 3).

Try python3 post.py instead of python post.py

Install Libs with
pip install <lib>
or (for Py3)
pip3 install <Lib>

[Tico]

Thanks Christian. That got me a bit further. I used 'python3 post.py' and received the following -

usage: post.py [-h] query content
post.py: error: the following arguments are required: query, content

I think that gets me into the substance of the script (of which I don't have a full understanding). I would expect I need to define the IP address of the inverter, username, password etc in the script. I don't know where they should go.

import socket
import subprocess
import time
import os
import datetime
import argparse
import random
import hashlib
import subprocess
import json

from importlib.util import spec_from_file_location, module_from_spec
from urllib.request import Request
from urllib.request import urlopen
from urllib.error import HTTPError
from urllib.error import URLError

import urllib
import socket
import sys
import getpass
import json


def getRandomByte():
    """
    Internal method.
    :return:
    """
    return str(hex(random.randrange(0, 256, 2))).replace('0x', '')


def randomBytes(count):
    """
    Internal method.
    :param count:
    :return:
    """
    rb = ''
    for i in range(0, count):
        if i > 0:
            rb += ','
        rb += getRandomByte()
    return rb


def calcHash(username, realm, password):
    """
    Internal method, not required.

    :param username:
    :param realm:
    :param password:
    :return:
    """
    hashGenerator = hashlib.md5()
    hashGenerator.update((username + ':' + realm + ':' + password).encode('utf-8'))
    print(hashGenerator.hexdigest())
    return hashGenerator.hexdigest()


def calculateDigestAuthHeader(method, url, wwwAuthHeader, userpass):
    """
    This function calculates the custom Coyote Digest Authentication Header for the given URL.
    To get the wwwAuthHeader content, call getDigestAuthNonceInfo before.

    :param method: GET/POST
    :param url: the url for which the digest header should be calculated (without protocol, host)
    :param wwwAuthHeader: return value of getDigestAuthNonceInfo
    :param userpass: '<user>:<password>'
    :return: The custom digest auth header string to be used in HTTP header Authorization
    """
    realm = ''
    nonce = ''
    qop = ''
    opaque = ''
    cnonce = randomBytes(8)

    parts = wwwAuthHeader.strip().split(',')
    for part in parts:
        entries = part.split('=')
        if 'nonce' in entries[0].strip():
            nonce = entries[1].replace('"', '').replace('\'', '').strip()
        if 'realm' in entries[0].strip():
            realm = entries[1].replace('"', '').replace('\'', '').strip()
        if 'qop' in entries[0].strip():
            qop = entries[1].replace('"', '').replace('\'', '').strip()
        if 'opaque' in entries[0].strip():
            opaque = entries[1].replace('"', '').replace('\'', '').strip()

    if (realm is '') or (nonce is ''):
        return ''

    userpass = userpass.split(':')
    nc = str(1).zfill(8)

    hash1 = hashlib.md5((userpass[0] + ':' + realm + ':' + userpass[1]).encode('utf-8')).hexdigest()
    hash2 = hashlib.md5((method.upper() + ':' + url).encode('utf-8')).hexdigest()
    s = hash1 + ':' + nonce

    if qop is not '':
        qop = qop.split(',')[0]
        s += ':' + nc + ':' + cnonce + ':' + qop

    s += ':' + hash2

    response = hashlib.md5(s.encode('utf-8')).hexdigest()
    authString = 'Digest username="' + userpass[0] + '", realm="' + realm + '", nonce="' + nonce + \
                 '", uri="' + url + '", response="' + response + '"'

    if opaque is not '':
        authString += ', opaque="' + opaque + '"'
    if qop is not '':
        authString += ', qop=' + qop + ', nc=' + nc + ', cnonce="' + cnonce + '"'

    return authString

def dump(obj):
   for attr in dir(obj):
       if hasattr( obj, attr ):
           print( "obj.%s = %s" % (attr, getattr(obj, attr)))

def getDigestAuthNonceInfo(url):
    """
    Initial call to Coyote WebUI in order to get the initial nonce.
    Use the returned string in method calculateDigestAuthHeader

    :param url: some URL of Coyote WebUI
    :return: the x-www-Authenticate header string
    """

    try:
        request = Request(url,method='POST')
        response = urlopen(request)
        response = str(response.read().decode())
        print (response);
        return False, json.loads(response)
    except HTTPError as e:
        if hasattr(e,'headers') and e.headers.get('X-WWW-Authenticate') is not None:
            return True, e.headers.get('X-WWW-Authenticate')

        print ("query failed: is hostname and query correct?")
    return 'no auth'


def post(query, content):

    if (query.find('http://') != 0):
        query = 'http://'+query

    (authRequired, data) = getDigestAuthNonceInfo(query)

    if ( not authRequired ):
        return data;
    ## else
    xAuthChellange = data;

    user = input('user (customer | technician): ')
    passwd = getpass.getpass('password:')

    apiQuery = query.split('http://')[1]
    apiQuery = apiQuery[apiQuery.index('/'):]

    authHeader = calculateDigestAuthHeader('POST', apiQuery, xAuthChellange, user+":"+passwd)

    reuqestHeaders = {
       'Content-Type': 'application/json',
       'Authorization': authHeader,
    }

    try:    
        request = Request(query, headers=reuqestHeaders, method='POST',data=bytes(json.dumps(content),encoding="utf-8"))
        response = urlopen(request)
        response = str(response.read().decode())
        print (response)
        return json.loads(response)
    except HTTPError as e:
        print('Status: ' + str(e.code) + ', ' + e.reason)
        if (e.code != 401):
            err = str(e.read().decode() )
            if (e.code == 400):
                err = json.loads( err ) 
            print ( err )
            return err


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('query', metavar='query', type=str, nargs=1,
                        help='target host + REST API Query')

    parser.add_argument('content', metavar='content', type=str, nargs=1,
                        help='JSON content which has to be transmitted')

    args = parser.parse_args()
    query = args.query[0]
    content = json.loads(args.content[0])
    post(query, content)

[Christian Fenzl]

I would use ‘query’ (1st param) as the full url, and ‘content’ (2nd param) as the post data, both you before had sent in your php post.

[Tico]

The original php post script had the following -





In the Python script attached, I've tried many variations of where to put the data for 'query' and 'content'.





I'm getting further, but not there yet -

[Christian Fenzl]

Try = instead of => and remove the blanks around =.
A blank is the parameter separator (you had 4 params not 2)

[Christian Fenzl]

And no comma
This is command line

[Tico]

No success yet. I could keep guessing variations on the format for a long, long time.

I've emailed back to the engineering department and requested a specific example for the 'suspend' command (rather than the generic POST shell they've provided).

[Tico]

Hi Christian,

I received a new script from the engineering department that works. There are a few lines of code they've changed and they also added a shebang as line 1 -

#!/usr/bin/python3

I can activate it successfully from the terminal using -

python3 Suspend_Calibration.py

If I use the command 'python Suspend_Calibration.py', it fails with the original error (I'm guessing due to using python2). I thought the shebang would make it use python3 regardless?

The last hurdle I'm stuck at is activating the script from the Virtual Output.

The Virtual Output 'Command for ON' that's not working is -

/legacy/Suspend_Calibration.py


I've also tried -

python3 /legacy/Suspend_Calibration.py
/usr/bin/python3 /legacy/Suspend_Calibration.py


I suspect it's defaulting to using python2 when called from the Miniserver. Any suggestions?

[Tico]

Just to close the loop -

The activation of the Python script was not successful when done directly from the Miniserver. ie. Command for ON = /legacy/Suspend_Calibration.py

I found an old post from svethi with the solution.

Create a php script on the Loxberry with contents -

<?php
echo exec('python3 Suspend_Calibration.py');
?>

Then direct the 'Command for ON' to the php script. ie /legacy/script_Suspend_Calibration.php

I don't understand why it works this way (and direct activation of the .py script doesn't work). Any explanation would be welcome.

[Christian Fenzl]

I try to explain:

Test 1:
python3 /legacy/Suspend_Calibration.py
/usr/bin/python3 /legacy/Suspend_Calibration.py

This commands in an HTTP command cannot work, because you do not talk to a shell/bash, but to Apache. You cannot give Apache an http request to start an executable, and even less an executable from an absolute file system path.

Test 2:
#!/usr/bin/python3

The shebang is correct, and if you chmod +x your file, you even can run it by ./Suspend_Calibration.py

If you place the .py file to the legacy folder and run curl http://localhost/legacy/Suspend_Calibration.py you only get the content of the script. This is of Apache configuration, that is not configured to run any file extension but *.cgi. Any other file will be responded as plain text.

Option 1:
In the legacy folder, and with the shebang from above, you rename your file to Suspend_Calibration.cgi
If you now run
curl http://localhost/legacy/Suspend_Calibration.cgi
you will get an HTTP 500 INTERNAL SERVER ERROR (and an error html from LoxBerry) but the script was executed.
The reason for the error is, that your script does not respond with a valid HTTP response, and Apache blocks your response.

Option 1 PLUS:
If you don't like the HTTP 500, you can add on top of the Python3 cgi script, but below the shebang, the following http header:
print("Content-Type: text/plain\r\n")
This is a valid response header, and tells Apache and the receiver that the following output is plain text.

Option 2:
You have now running the script with a PHP wrapper. *.php is not directly allowed in /legacy, but Apaches mod_php catches all calls for *.php files and executes them.

Option 3:
The Any-Plugin would have been a help too, it can execute any shell command from an incoming TCP call from a VO. It responds with the script output via udp. It is quite similar to the TCP2UDP plugin, but it does not connect to an external tcp device, but calls shell commands.

[Tico]

Thank-you very much for the explanation. That does help my general understanding.